New colonoscopy skills assessment tool developed for trainees

Ensuring that gastroenterology fellows and surgery residents achieve competence in colonoscopy is one of the goals of every training program. Despite this, defining what constitutes "competence" in colonoscopy and having a standardized means to formally assess it have remained elusive. Training programs traditionally use informal global evaluation methods in which, toward the end of training, supervising staff make a subjective judgment as to the overall preparedness of the trainee to operate alone. The lack of standardized and measurable end points makes it difficult to reproduce such judgments.

To date, the vast majority of literature on competency in colonoscopy has focused on very limited performance parameters such as trying to define how many procedures are required for a trainee to reach a specified cecal intubation rate rather than defining the specific skills of competence. As a result, training guidelines based on these limited data suggest that trainees require hands-on experience with approximately 140 supervised colonoscopies before competency assessment can be attempted. Since these initial recommendations have come out, various other numbers have been cited in the published literature, ranging from 150 to as high as 500 procedures before competency may be assessed.

"What is needed is the development of an ongoing, formalized assessment method that assesses a broad range of both motor and cognitive skills in a standardized and measurable way. The benefits of such a tool would allow continuous monitoring of a trainee's ongoing progress and the establishment of normal learning curves for trainees," said study lead author Robert E. Sedlack, MD, MHPE, Mayo Clinic, Rochester, Minn. "The MCSAT form addresses this need and provides a valid means to objectively assess individual cognitive and motor skills in a continuous manner throughout colonoscopy training. The resultant data can eventually be used to establish average learning curves in colonoscopic skills and define competency thresholds based on performance scores."

Methods

The study was conducted at the Mayo Clinic, Rochester, Minn., routine outpatient endoscopy suites from July 2007 through May 2010. Study participants included all gastroenterology fellows performing colonoscopy as part of their training during this period. The study is primarily a descriptive report of the development of a novel colonoscopy skills assessment tool (MCSAT) and prospective evaluation of the data obtained from the use of this form for the purpose of tool validation.

After the process of MCSAT form development, teaching staff were requested to complete a MCSAT form assessing fellow performance of every colonoscopy performed in the routine outpatient endoscopy suites. The procedures performed in this area were primarily routine screening examinations. With the goal of this form intended to focus on assessing the minimal procedural skills required to be competent to perform routine colonoscopies, procedures performed in the therapeutic or "complex" endoscopy suites were excluded from this analysis because these tend to be much longer cases with more difficult anatomy. (The form is available on www.giejournal.org.)

To develop the assessment tool, the core skills of colonoscopy were identified and then a means to accurately and reliably measure each of these skills was created. A "blueprint" of the tool was designed that broke down colonoscopy skills into basic motor and cognitive competency areas. The blueprint was derived from a review of professional society recommendations and published reviews, and by a focus group of nine expert endoscopists at the Mayo Clinic (Rochester, Minn.).

After its development, the blueprint was reviewed by this panel and, after multiple revisions and refinement, was deemed to contain the core cognitive and motor domains of colonoscopy training. Based on these domain items, the expert panel next identified 14 distinct skills (six cognitive and eight motor) that are necessary to be minimally competent in routine colonoscopy. From this list of skills, the items found on the MCSAT were created through the development of survey questions and a categorical scoring system for each of the identified core skills.

The MCSAT was integrated into the Mayo GI endoscopy database software where all other procedure data are also recorded. The database keeps track of the trainee's order of procedure completion by date and time, thus pairing the individual MCSAT results with a specific procedure within the continuum of an individual's training experience. As a result, trainee performance can be monitored sequentially, and improvement from one point of their training to another can be tracked (ie, comparisons of the first, 150th, or 300th procedure).

Results

Over a nearly three-year period, 41 fellows performed 6,390 colonoscopies and 58 different staff endoscopists completed 3,936 MCSAT forms on these procedures. MCSAT stages were divided into three groups: novice (20 procedures), intermediate (150 procedures), and advanced (300 procedures). Researchers found an improvement in both cognitive and motor skills as the clinician progressed through the stages of colonoscopy training. The improvement of the clinician suggests MCSAT is able to demonstrate differences in each stage of training for both motor and cognitive skills. The study concluded that the MCSAT provides a valid means to objectively assess individual cognitive and motor skills in a continuous manner throughout colonoscopy training. The resultant data can eventually be used to establish average learning curves in colonoscopic skills and define competency thresholds based on performance scores rather than basing assessment simply on numbers of procedures performed.

Researchers noted that one of the limitations of these results is that this research is limited to a single institution, thus introducing possible biases. Once validated, a multicenter study would be advisable to prove broader generalizability of the MCSAT results across institutional boundaries. They also noted that this form does not provide a means to specifically track adenoma (precancerous polyp) detection rates. This was considered but found to be problematic because determining which polyps are eventually identified as adenomas would require considerable retroactive data entry once pathology results become available after a procedure. In the future, however, the addition of simple polyp detection rates should be easy for the form to accommodate.

In an accompanying editorial, Jonathan Cohen, MD, FASGE, Division of Gastroenterology, New York University, states, "Their (Sedlack et al) work represents an important step forward in the evolution of skills measurement in colonoscopy and holds promise both as a training tool in itself and as a means of better assessing the acquisition of competency in this area."

Provided by American Society for Gastrointestinal Endoscopy