Impact of a national basic skills in colonoscopy course on trainee performance: An interrupted time series analysis

Abstract

BACKGROUND

The Joint Advisory Group on Gastrointestinal Endoscopy basic skills in colonoscopy (BSC) course was introduced in 2009 to improve colonoscopy training within the United Kingdom, but its impact on trainee performance is unknown.

AIM

To assess whether attendance of the BSC could improve colonoscopy performance.

METHODS

Trainees awarded colonoscopy certification between 2011-2016 were stratified into 3 groups according to pre-course procedure count (< 70, 70-140 and > 140). Study outcomes, comprising the unassisted caecal intubation rate (CIR) and the performance indicator of colonic intubation (PICI), were studied over the 50 procedures pre and post- course. Interrupted time series analyses were performed to detect step-change changes attributable to the course.

RESULTS

A total of 369 trainees with pre-course procedure counts of < 70 (n = 118), 70-140 (n = 121) and > 140 (n = 130) were included. Over the 50 pre-course procedures, significant linear improvements in CIR were found, with average increases of 4.2, 3.6 and 1.7 percentage points (pp) per 10 procedures performed in the < 70, 70-140 and > 140 groups respectively (all P < 0.001). The < 70 procedures group saw a significant step-change improvement in CIR, increasing from 46% in the last pre-course procedure, to 51% in the first procedure post-course (P = 0.005). The CIR step-change was not significant in the 70-140 (68% to 71%; P = 0.239) or > 140 (86% to 87%; P = 0.354) groups. For PICI, significant step-change improvements were seen in all three groups, with average increases of 5.6 pp (P < 0.001), 5.4 pp (P = 0.003) and 3.9 pp (P = 0.014) respectively.

CONCLUSION

Attendance of the BSC was associated with a significant step-change improvement in PICI, regardless of prior procedural experience. However, CIR data suggest that the optimal timing of course attendance appears to be at earlier stages of training (< 70 procedures).

Key Words: Colonoscopy; Training; Course; Competency development; Skills; Endoscopy

Core tip: In this nationwide study from the United Kingdom, attendance of a 3-d hands-on basic skills in colonoscopy course was associated with a significant step-change improvement in colonoscopy competence, equivalent to performing an additional 17-30 procedures. Moreover, this study provides data on the optimal timing of course attendance, which appears to be at earlier stages of training (< 70 procedures).

INTRODUCTION

Colonoscopy is a complex procedure which requires prolonged training time to acquire the necessary competencies for independent practice. The United Kingdom colonoscopy audit (performed in 1999 and published in 2004) exposed substandard levels of performance and safety concerns arising from excessive sedation, triggering reforms in the quality assurance (QA) of endoscopy training[1]. The Joint Advisory Group on Gastrointestinal Endoscopy (JAG)[2], which oversees QA of training within the United Kingdom, was tasked with developing a standardised, national colonoscopy course replete with the essential theoretical and practical aspects of colonoscopy, to ensure that all course attenders would be sufficiently equipped with the basic principles for performing colonoscopy. This led to the development and roll-out of the basic skills in colonoscopy (BSC) course in 2001[2]. In order to ensure consistency and quality of delivery of BSC courses, JAG mandated for these courses to be delivered in JAG accredited training centres by accredited faculty who had attended endoscopy-related Train-the-Trainer courses[3]. Since the implementation of JAG endoscopy certification in 2011[4], attendance of the BSC has been mandatory for provisional colonoscopy certification[5], a requirement for unsupervised, independent colonoscopy practice in the United Kingdom.

Despite the implementation of BSC, its impact on subsequent real-world performance has not been assessed. Data on the effectiveness of hands-on training interventions on real-world trainee outcomes are limited, whilst the question of when trainees should attend such training courses to achieve maximal benefit also remains unanswered. We therefore aimed to evaluate the impact of BSC attendance on trainee colonoscopy performance using completion-based metrics, and to determine the optimal timing of course attendance.

MATERIALS AND METHODS

Study design

This was a retrospective cohort study which enrolled all United Kingdom trainees who had attended the BSC between 2011 and 2016 and received provisional certification in colonoscopy (PCC). This period was selected as electronic colonoscopy certification began in 2011 and the criteria for certification were amended in June 2016. Under JAG recommendations, United Kingdom endoscopy trainees are required to log training experience onto the JAG Endoscopic Training System (JETS) e-portfolio[6].

The PCC cohort was selected as they had demonstrated engagement with the JETS e-portfolio and completed a training pathway in colonoscopy. Trainees were excluded if they had < 25 procedures prior to the attendance of the BSC, as this gave insufficient data to estimate the trends in outcome rates over the pre-course period, or if there was a > 3 mo interval between BSC attendance and the first post-BSC procedural record on JETS.

Basic skills colonoscopy course

The BSC is an intensive, three-day course consisting of hands-on, supervised patient-based colonoscopy, complemented by mechanical simulator training, teaching of basic principles[7], small-group discussions and performance-enhancing feedback[8]. Under expert faculty supervision, trainees are observed when undertaking a number of patient-based colonoscopies (4-6 cases per delegate). Tri-split video streams comprising the endoscopic luminal view, in-room view of the endoscopist and patient, and the magnetic endoscopic imager configuration are transmitted to other delegates using local videoconferencing facilities to mediate group reflection and discussion. The one-to-one teaching focuses on the development of individualised learning objectives through performance enhancing feedback. In order to be eligible to attend the course, trainees require written support from their educational supervisor or endoscopy trainer, and for trainees to have at least one training list per week available at their base hospital following the course. Courses are quality assured by the JAG in response to trainee and faculty feedback.

Outcomes

The primary outcome was the unassisted caecal intubation rate (CIR), defined as the unadjusted rate of intubation to caecum or ileocolonic anastomosis without physical assistance by the trainer[9]. Whilst CIR is a recognised performance measure, it is regarded to be unidimensional in its measure of procedural completion. Recently, the performance indicator of colonic intubation (PICI) was proposed as a more comprehensive performance metric for colonic intubation, on the basis that repeatedly reaching the caecum, while using minimal sedation and causing little discomfort requires considerable skills in intubation of the colon to the caecum[10]. PICI is a validated composite endpoint comprising: unassisted caecal intubation, use of midazolam ≤ 2 mg, and acceptable rates of procedural discomfort (discomfort not rated as moderate or severe on the modified Gloucester comfort scale)[10,11]. The PICI reflects some of the key United Kingdom quality standards in colonoscopy, which recommend the use of age-adjusted sedation thresholds and that rates of moderate-severe discomfort should not exceed 10%[12]. Of note, trainees generally do not perform colonoscopy under propofol within the United Kingdom.

For each procedure, a number of variables were collected to enable analysis of the study outcomes, including procedural extent, level of trainer assistance, nurse reported pain scores (assessed on a five-point Likert scale) and sedation doses.

Statistical methods

Since performance during training improves with procedural experience[13,14], the cohort was first divided into three groups of experience based on the number of procedures performed prior to the basic skills course. This was used to denote early, intermediate or late attendance of the BSC. For each of the fifty procedures before and after the course, the outcome rates were calculated across all trainees within each of these groups.

The changes over time in the CIR were then assessed using an interrupted time series approach[15]. This took the form of a linear regression model, with the CIR set as the dependent variable, and three covariates. The first specified the procedure number relative to the basic skills course, with the course being procedure zero, and pre-course procedures taking negative values. The second variable was similar to the first but took the value of zero for all procedures prior to the basic skills course. The final variable was dichotomous, stating whether or not the basic skill course had been attended prior to each procedure. As such, the first variable measured the gradient in CIR by procedure number, the second quantified how this gradient changed in the procedures after the course, and the third looked for any step-change improvement immediately after the course. The analysis was then repeated for the outcome of PICI.

Trainees were then divided into two groups, based on whether or not an improvement in CIR was observed between the 25 procedures before and 25 procedures after the training course. Comparisons between these groups were then performed using Mann-Whitney U tests.

All analyses were performed using IBM SPSS 22 (IBM Corp. Armonk, NY, United States), with P < 0.05 deemed to be indicative of statistical significance throughout.

RESULTS

Demographics

A total of 709 trainees had attended the BSC and were awarded colonoscopy certification. Of these, 155 trainees were excluded due to having < 25 procedures prior to completing the BSC, and a further 185 trainees were excluded due to intervals of > 3 mo between completing the BSC and logging the subsequent procedure on their e-portfolio. A total of 369 trainees were eligible for analysis, contributing a total of 35716 procedures. These trainees attended 202 BSC courses over this period. Trainee specialties comprised: Gastroenterology (53.7%), colorectal surgery (23.3%), non-medical endoscopist (22.2%) and general practitioner (0.8%). Trainees were then divided into three groups based on approximate tertile of pre-course lifetime procedure counts, namely < 70 (n = 118), 70-140 (n = 121) and > 140 (n = 130) procedures.

CIR

Immediately prior to attending the course, the CIRs were 46%, 71% and 92% in the < 70, 70-140 and > 140 procedure groups, respectively. Trends over time in CIR are shown in Table 1 and Figure 1. Across the 50 procedures prior to the basic skills course, rates of CIR improved significantly in all three groups (all P < 0.001). However, the rate of improvement slowed with increasing experience, with average increase in CIR of 4.2, 3.6 and 1.7 percentage points (pp) per 10 procedures for the < 70, 70-140 and > 140 groups, respectively.

Table 1 Interrupted time series models of unassisted caecal intubation rate by procedure count.

	Gradient (95%CI)	P value
Pre-course lifetime procedure count: < 70
Pre-course gradient (per 10 procedures)	4.2 pp (3.4, 4.9)	< 0.001b
Step-change	4.5 pp (1.4, 7.6)	0.005b
Post-course gradient (per 10 procedures)	3.6 pp (2.8, 4.3)	0.2961
Pre-course lifetime procedure count: 70-140
Pre-course gradient (per 10 procedures)	3.6 pp (2.8, 4.4)	< 0.001b
Step-change	1.9 pp (-1.3, 5.2)	0.239
Post-course gradient (per 10 procedures)	1.5 pp (0.7, 2.3)	< 0.0011b
Pre-course lifetime procedure count: > 140
Pre-course gradient (per 10 procedures)	1.7 pp (1.1, 2.2)	< 0.001b
Step-change	1.1 pp (-1.3, 3.5)	0.354
Post-course gradient (per 10 procedures)	1.0 pp (0.5, 1.6)	0.1321

¹Denotes comparison of pre- and post-course gradients.

^bP < 0.01. Results are from interrupted time series models, using a linear regression approach. Gradients are reported per 10 procedure increase in experience and are given as percentage point changes with 95% confidence intervals. CI: Confidence interval; pp: Percentage point.

Figure 1 Changes over time in unassisted caecal intubation rate by lifetime procedure count at the basic skills course. Points represent the observed caecal intubation rate (CIR) at each procedure number. Coloured lines are from the interrupted time series models reported in Table 1. Grey lines are the corresponding forecasted trends in CIR, had the basic skills course not been performed.

In the immediate post-course period, there was a significant step-change improvement in the least experienced group (< 70 procedures), with the CIR increasing by 4.5 pp (95%CI: 1.4-7.6 pp, P = 0.005). The degree of improvement resulting from the course was approximately equivalent to that of performing an additional 11 procedures, based on the observed gradient of 4.2 pp per 10 procedures. No significant step-change was observed in either of the other groups (70-140: P = 0.239, > 140: P = 0.354). In the intermediate group (70-140 procedures), a significant change in the gradient was observed after the basic skills course, with the rate of improvement in CIR slowing significantly (P < 0.001). This was not observed in the other groups.

Predictors of improvements in CIR

Between the 25 pre-course procedures and the 25 post-course procedures, the CIR increased by a mean of 9.3 (standard deviation: 18.0) pp (mean CIR: 64% pre vs 73% post), with the CIR showing some degree of improvement in 64% of trainees (n = 236). Trainees with an improvement in post-course CIR had performed significantly fewer procedures prior to the course (median: 98 vs 122, P = 0.006) and, as such, had a significantly lower pre-course CIR (median: 36% vs 56%, P < 0.001). The proportion of trainees with improvements in CIR was not found to differ significantly by trainee specialty (χ²: P = 0.564).

PICI

The analysis was then repeated for the composite outcome of PICI (Table 2, Figure 2). Prior to attending the course, rates of PICI were improving significantly with experience in all three groups (all P < 0.001) by an average of 3.3, 3.2 and 1.3 pp per 10 procedures in the < 70, 70-140 and > 140 procedure groups, respectively, reaching rates of 31%, 48% and 61% just prior to the course. Significant step-change improvements were then observed in all three groups after the course, with PICI increasing by 5.6 pp, 5.4 pp and 3.9 pp in the < 70, 70-140 and > 140 procedure groups, respectively. Relative to the rate of improvement prior to the course, these changes are equivalent to that of performing between 17 and 30 additional procedures. The rate of improvement in PICI slowed significantly after the course in the 70-140 procedure group (P < 0.001), with no significant changes detected in the other two groups.

Table 2 Interrupted time series models of the performance indicator of colonic intubation by procedure count.

	Gradient (95%CI)	P value
Pre-course lifetime procedure count: < 70
Pre-course gradient (per 10 procedures)	3.3 pp (2.6, 4.1)	< 0.001b
Step-change	5.6 pp (2.5, 8.7)	< 0.001b
Post-course gradient (per 10 procedures)	2.3 pp (1.6, 3.1)	0.0651
Pre-course lifetime procedure count: 70-140
Pre-course gradient (per 10 procedures)	3.2 pp (2.4, 4.1)	< 0.001b
Step-change	5.4 pp (1.9, 8.9)	0.003b
Post-course gradient (per 10 procedures)	0.2 pp (-0.6, 1.1)	< 0.0011b
Pre-course lifetime procedure count: > 140
Pre-course gradient (per 10 procedures)	1.3 pp (0.6, 2.1)	0.001b
Step-change	3.9 pp (0.8, 7.0)	0.014a
Post-course gradient (per 10 procedures)	0.8 pp (0.1, 1.6)	0.3341

¹Denotes comparisons of pre- and post-course gradients.

^aP < 0.05.

^bP < 0.01. Results are from interrupted time series models, using a linear regression approach, as described in the methods. Gradients are reported per 10 procedure increase in experience and are given as percentage point changes with 95% confidence intervals. CI: Confidence interval; pp: Percentage point.

Figure 2 Changes over time in the performance indicator of colonic intubation by lifetime procedure count at the basic skills course. Points represent the observed performance indicator of colonic intubation (PICI) at each procedure number. Coloured lines are from the interrupted time series models reported in Table 2. Grey lines are the corresponding forecasted trends in PICI, had the basic skills course not been performed.

DISCUSSION

Beyond the realms of simulation-based training, efficacy data in support of training interventions, and the optimal timing of attendance, are lacking. In this United Kingdom-wide study, attendance of a three-day BSC encompassing fundamental theory and hands-on patient-based colonoscopy training was found to result in a significant step-change improvement in performance, as measured by PICI. Data from CIR analyses suggest that the optimal timing of course attendance appears to be at earlier stages of training (< 70 procedures).

Simulation-based training has been shown to improve patient-based outcomes in trainees with little or no experience in patient-based colonoscopy[16,17]. However, meta-analyses do not support its use as a substitute for patient-based training, which provides greatest benefit[18,19]. Thomas-Gibson et al[20] (n = 21) assessed the impact of a five-day hands-on colonoscopy training course on trainee outcomes. Immediately post-course, trainees demonstrated significant improvements in colonic intubation, as evidenced by improvements in direct observation of procedural skills (DOPS) assessment scores and efficiency metrics. Follow-up assessments undertaken after a median interval of 9 mo post-course (median of 46 procedures) showed improvements in simulator-based assessments, but not in DOPS scores. The authors concluded that the course could deliver sustained improvements in colonoscopy performance, although the study lacked a comparison arm and did not account for the natural trajectory of trainee competency development during the post-course period. As such, comparative studies with longer-term data on trainee outcomes remain necessary.

Our study compared performance using interrupted time series regression analyses in three subgroups of trainees with varying levels of colonoscopy experience. This method enables the estimation of post-course performance by accounting for pre-course trajectories, and detects immediate post-course improvements in the form of a step-change[15]. The step-change was significant for PICI and in the lowest experience group for CIR, suggesting greatest potential benefit in this group. The rates of improvement in both CIR and PICI slowed significantly in the intermediate experience group (70-140 procedures at the time of the course), but not in the > 140 group. This may be a limitation of the models used. Whilst fitting a straight line to the data gave good model fit over the 100 procedures being analysed in each group, a continuing linear trend would not be feasible as outcome rates approach a ceiling effect. Comparisons of the pre-course gradients showed a reduction in gradient as the procedure count at the time of the course increases (e.g., from 4.2 to 1.7 pp per 10 procedures for CIR). It is possible that in the most experienced group (> 140 procedures), trainees were already close to an upper limit of performance at the time of the first procedure included in the analysis, meaning that the reduced rate of improvement was already present at the beginning of follow up.

Attendance of the BSC improved the completion and patient-based metric of PICI in all trainees, with the effect equivalent to performing an additional 17-30 training procedures. This may be attributable to both didactic and hands-on course elements, which provides a holistic approach to training beyond technical skills. The course covers a colonoscopy curriculum ranging from sedation, lesion recognition, optimal positions and troubleshooting techniques. During hands-on sessions, trainees are subjected to rigorous performance review of technical aspects, e.g., scope handling, loop avoidance and resolution, to non-technical skills[21], e.g., situational awareness, which can impact on progress and patient comfort. Tailored high-performance feedback, based on the DOPS formative assessment tool[22], is provided for each trainee and backed by future goal setting to encourage practice changes within their usual training environment. In contrast, CIR analyses showed a significant benefit only in those at earlier stages of training (< 70 procedures). It is possible that for novice trainees, training on basic principles, e.g., ergonomics[23], scope handling and patient positioning may expedite procedural completion, whereas in more experienced trainees, refinements in technique and the focus on patient comfort and safe sedation could be reasons behind improvements in the PICI metric. Such efficacy data relates to the final step on the Kirkpatrick model (Level 4) for the evaluation of training interventions[24], which are lacking within the endoscopy training literature.

Several limitations should be noted. First was the non-randomised observational study design, which introduces the possibility of trainee bias. Analyses were exploratory and did not account for procedural factors, e.g., intubation technique[25], use of magnetic imaging[26], indication, bowel preparation, diagnoses, caseload difficulty, or trainee factors, e.g., breaks in training, which may affect outcomes. Trainee feedback in relation to the impact of the course, while invariably positive, was not analysed as responses were anonymised. Second, the study was limited to those who had received JAG certification to ensure that all trainees had completed a minimum number of pre- and post-course procedures. This may have introduced selection bias of more technically capable trainees. Third, PICI is a novel outcome measure without an existing standard for benchmarking. Based on data extrapolated from the last United Kingdom colonoscopy audit, PICI was achieved in 54.1% of procedures[10]. Therefore, we believe that the PICI rate should be at least 50% in United Kingdom practice, with 80% being an achievable aspirational goal. Finally, the outcome measures were calculated from self-entered trainee data from the JETS e-portfolio, which have previously been shown to be reliable[13]. The United Kingdom National Endoscopy Database aims to autopopulate procedural outcomes directly from endoscopy reporting systems into the JETS e-portfolio[27], thereby reducing possible entry bias in the future.

High-quality standardised training courses, delivered by accredited trainers, have been central to the JAG QA philosophy and have underpinned the transformation in colonoscopy quality over the last 15 years[28]. Worldwide, the concept of expert-led, hands-on courses in endoscopy is gaining traction. Our analyses provide efficacy data in support of the BSC and indicate these should be undertaken early in the training pathway. Similar analyses could be used to evaluate the longitudinal impact of training courses in the future.

In conclusion, attendance of the BSC improved the trainee outcome of PICI in all trainees. Data from CIR analyses suggest that the optimal timing of course attendance appears to be at earlier stages of training (< 70 procedures).

ARTICLE HIGHLIGHTS

Research background

Hands-on endoscopy training courses are increasingly available worldwide to improve endoscopy skills. To our knowledge, no studies have published data on how attendance of hands-on endoscopy courses have improved endoscopy competency in trainees. Moreover, the optimal timing of course attendance is unknown.

Research motivation

The basic skills in colonoscopy (course) is a 3-d, hands-on, training course for which attendance is mandated for United Kingdom colonoscopy certification. The efficacy of the course has not been previously investigated.

Research objectives

We aimed to evaluate the impact of BSC attendance on trainee colonoscopy performance using completion-based metrics, and to determine the optimal timing of course attendance.

Research methods

We performed a national analysis of trainees who attended the BSC. Trainees awarded colonoscopy certification between 2011-2016 were stratified into 3 groups according to tertile of pre-course procedure count (early, intermediate or late attenders). Study outcomes, comprising the unassisted caecal intubation rate (CIR) and the unassisted Performance Indicator of Colonic Intubation (PICI), were compared over the 50 procedures pre and post- course. Interrupted time series analyses were performed to measure step-change change, i.e., the difference in performance directly attributable to the course.

Research results

In total, 369 trainees with pre-course procedure counts of < 70 (n = 118), 70-140 (n = 121) and > 140 (n = 130) were included. Using the outcome of CIR, a significant step-change improvement was seen in the < 70 procedures group, increasing from 46% in the last pre-course procedure, to 51% in the first procedure post-course (P = 0.005). The CIR step-change was not significant in the 70-140 (68% to 71%; P = 0.239) or > 140 (86% to 87%; P = 0.354) groups. For PICI, significant step-change improvements were seen in all three groups, with average increases of 5.6 pp (P < 0.001), 5.4 pp (P = 0.003) and 3.9 pp (P = 0.014) respectively. Based on PICI data, attendance of the BSC was associated with a significant step-change improvement, equivalent to performing an additional 17-30 procedures within the trainee’s usual training environment.

Research conclusions

Attendance of the basic skills course was associated with a significant step-change improvement in PICI, regardless of prior procedural experience. However, CIR data suggest that the optimal timing of course attendance appears to be at earlier stages of training (< 70 procedures).

Research perspectives

The learning curves of trainees naturally increase within their own training environment over procedure count and time. The use of interrupted time-series analyses to measure a step-change represents is valid way of assessing the impact of training courses on trainee outcomes. This method may be used across different training courses for research and quality assurance purposes.

ACKNOWLEDGEMENTS

We wish to thank the JAG team and Weblogik United Kingdom for providing assistance with this study.