Evaluation of surgical training in the era of simulation

Abstract

AIM: To assess where we currently stand in relation to simulator-based training within modern surgical training curricula.

METHODS: A systematic literature search was performed in PubMed database using keywords “simulation”, “skills assessment” and “surgery”. The studies retrieved were examined according to the inclusion and exclusion criteria. Time period reviewed was 2000 to 2013. The methodology of skills assessment was examined.

RESULTS: Five hundred and fifteen articles focussed upon simulator based skills assessment. Fifty-two articles were identified that dealt with technical skills assessment in general surgery. Five articles assessed open skills, 37 assessed laparoscopic skills, 4 articles assessed both open and laparoscopic skills and 6 assessed endoscopic skills. Only 12 articles were found to be integrating simulators in the surgical training curricula. Observational assessment tools, in the form of Objective Structured Assessment of Technical Skills (OSATS) dominated the literature.

CONCLUSION: Observational tools such as OSATS remain the top assessment instrument in surgical training especially in open technical skills. Unlike the aviation industry, simulation based assessment has only now begun to cross the threshold of incorporation into mainstream skills training. Over the next decade we expect the promise of simulator-based training to finally take flight and begin an exciting voyage of discovery for surgical trainees.

Key Words: Simulation; Surgical training; Surgery; Training; Objective Structured Assessment of Technical Skills; Observational tool; Surgical skills; Assessment; Skill assessment

Core tip: The nature of surgical training has teetered on the brink of a seismic change in how we can deliver the level of expertise required of a modern surgeon for over a decade. It is evolving from Halstedian’s apprenticeship model towards simulation-based training similar to the aviation industry. Since 2000 there have been approximately 173 studies about validation of simulators as assessment tools. As the technology grows, its translation into real changes in curriculum is still unclear. This review is focused upon where we currently stand in relation to the effective integration of simulation-based skills assessment into modern surgical training curricula.

INTRODUCTION

The nature of surgical training has teetered on the brink of a seismic change in how we can deliver the level of surgical training required of a modern surgeon for over a decade. The demands imposed by a zero complication ethos expected by patients and emphasised by the media has challenged us as surgical educators to continually assess our training paradigms. Traditionally, surgical training has been largely an opportunity-based learning approach based upon an apprenticeship in the operating room (OR). This Halstedian method[1] of surgical training is often exemplified as the “see one, do one, teach one” approach to training. This system which was reliant upon opportunistic encounters particularly of the complex case mix variety remains extremely time dependant. This apprenticeship model resulted in surgical training often being prolonged in order to gain sufficient surgical experience to reach a subjective level of operative experience. In the modern era of surgical training, trainees are continually restricted on the number of hours they can legally work. This may be as low as 48 h per week in Europe[2] or 80 h in North America[3]. These mandated reductions in working hours have been based upon safe guarding both patients and doctors alike in order to decrease potential errors in the health care system. This decrease in hours however will result in a fundamental reduction in the trainees’ opportunity for surgical operating time exposure with “real” patients. As a direct consequence of these challenges, interest in laboratories with formal curricula, specifically designed to teach surgical skills, has increased dramatically[4].

The use of surgical simulators and inanimate bench models for training and assessment has been the centre of attraction among the training bodies around the world for well over a decade. The use of simulation for clinical skills training, assessment and clinical scenario management provides educators the freedom of focused training in more controlled environment without risking the life of any patients. Trainees may also have the chance to practice the skills required of a modern surgeon to proficiency at their own pace. The greatest advantage of virtual reality medical simulation is the opportunity to try and fail without consequence for the patient[5]. The integration of simulation into training programmes would therefore seem the next most intuitive step for the design and implementation of any modern surgical training curriculum.

In tandem with the continued development of surgical skills in training surgeons of equal importance is our ability to assess the candidates’ proficiency in the performance of these very surgical skills that we have taught. Once again the assessment of surgical skills has been largely subjective and onto this horizon surgical simulation may also provide a solution. The objective characterisation of technical skills can be difficult. Technical performance assessment ranges from basic surgical skills such as knot tying and suturing, basic laparoscopic skills and endoscopy to a wide spectrum of evaluations that include performing complex procedures such as laparoscopic cholecystectomy, vessel anastomosis and tendon repair. Assessment can be defined as making a judgement against a predefined reference[6]. As surgical educators, it is important to assess trainees on their progress in surgical skills in order to ensure that they remain safe in the stressful environment of a real operating theatre. It allows the trainers to give a constructive feedback based on their performances and can be used for the award of certification or even credentialing. Despite its importance to surgeons, technical proficiency historically has been poorly evaluated[7]. A good assessment tool must possess reliability, validity, educational impact, acceptability and feasibility[8].

The aim of this review is to determine where we currently stand in relation to the use of simulation in surgical skills assessment within current training curricula. We focused upon the use of simulators in surgical curricula that embraced the concept of creating proficiency profiles using simulators. Technical performance assessment in laparoscopy, endoscopy and open surgical skills were included.

MATERIALS AND METHODS

This review encompassed a literature search in PubMed from January 2000 to November 2013. The keywords used to search the database were “simulation”, “skills assessment” and “surgery”. All search result titles and abstracts were reviewed by the authors, SS and PN. Full texts of compatible articles were examined for eligibility of inclusion as agreed by the two authors.

Inclusion criteria

Studies were included if simulators were used in laparoscopic and endoscopic skills assessment following an intervention such as skills training, courses, surgical curriculum and selection process. Also, studies using simulators to assess open technical skills such as knot tying, suturing or a basic open procedure, for example excisions of sebaceous cyst were included.

Exclusion criteria

The review was focused upon the use of simulators in assessment of surgical skills. Studies that aimed at validating their latest simulator alone were excluded. Studies were excluded if the surgical skills are of specific subspecialties such as ophthalmology, urology, gynaecology, cardiothoracic, ear, nose and throat (ENT), neurosurgery, trauma and orthopaedics, as well as non-validated methods, non-technical skills for example cognitive analysis and patient care simulation. Any non-English articles, reviews, conference abstracts, editorial, comments, supplements and case reports were excluded.

RESULTS

The keyword search yielded 515 articles, of which 201 articles were eligible. Following the application of our inclusion and exclusion criteria, there were 52 articles remained that dealt with technical skills assessment in general surgery. These selected articles were divided into 4 categories according to the skills assessed; open skills (Table 1), laparoscopic skill (Table 2), combination of open and laparoscopic skills (Table 3), and endoscopic skills (Table 4). Out of these articles only 12 studies integrated simulators in a surgical curriculum with technical skills being assessed (Table 5). Only 1 study was found using simulators in the selection process into surgical training programme.

Table 1 Study characteristics assessing open surgical skills (n = 5).

Ref.	Year	No. of trainees	Tasks	Assessment tool
Acton et al[9]	2010	157 clerkship	Suturing	OSATS
Brydges et al[10]	2008	38 trainees	One-handed knot tying	Motion analysis (ROVIMAS) and GRS
Chipman et al[11]	2009	24 trainees PGY 1	Excision of skin lesion and wound closure	OSATS
Jensen et al[12]	2008	45 PGY 1-2	Excision of skin lesion and bowel anastomosis	Video-based OSATS and FPA (wound closure aesthetic quality and anastomotic leak pressure)
Olson et al[13]	2012	11 intern	Open laparotomy and bowel anastomosis	OSATS and survey

OSATS: Objective Structured Assessment of Technical Skills; ROVIMAS: RObotics VIdeo and Motion Assessment Software; GRS: Global Rating Score; FPA: Final product analysis.

Table 2 Study characteristics of studies assessing laparoscopic skills (n = 37).

Ref.	Year	No. of participants	Tasks	Assessment tool
Aggarwal et al[14]	2007	20 trainees	Laparoscopic cholecystectomy	Motion analysis and video-based GRS
Arora et al[15]	2011	25 surgeons	Laparoscopic cholecystectomy	OSATS
Bennett et al[16]	2011	70 students	Camera navigation	Box trainer
Botden et al[17]	2009	18 students	Laparoscopic suturing	ProMIS™, FPA using 5-point Likert Scale
Buzink et al[18]	2012	25 trainees6 experts	Diagnostic laparoscopy, laparoscopic cholecystectomy and laparoscopic appendicectomy	LapMentor
Cope et al[19]	2008	22 interns	6 tasks on MIST VR	MIST VR
Crochet et al[20]	2011	26 trainees	Laparoscopic cholecystectomy	VR Simulator
Ganai et al[21]	2007	19 students	Angled telescope navigation	VR Simulator
Grantchar-ov et al[22]	2009	37 residents	Basic laparoscopic task	MIST VR
Heinrich et al[23]	2007	17 experts	26 modules	LapMentor, LapSim, ProMIS™, Surgical SIM
Kanumuri et al[24]	2008	16 students	Laparoscopic suturing and knot tying	Video-based performance assessment tool on live porcine
Kolozsvari et al[25]	2012	63 residents	FLS tasks1	FLS scoring system
Kurashima et al[26]	2013	17 residents	Laparoscopic inguinal hernia repair	GOALS-GH
Langelotz et al[27]	2005	150 surgeons	Navigation, coordination, grasping, cutting and clipping	VR simulators
LeBlanc et al[28]	2010	29 surgeons	Laparoscopic sigmoid colectomy	ProMIS™ simulator, OSATS and operative error
Lehmann et al[29]	2012	36 surgeons	2 LapSim tasks	LapSim
Lehmann et al[30]	2013	105 surgeons	Lifting and Grasping, Fine dissection	LapSim
Loukas et al[31]	2011	25 trainees	Adhesiolysis, bowel suturing, laparoscopic cholecystectomy	LapVR
Loukas et al[32]	2011	20 trainees	Adhesiolysis, bowel suturing, laparoscopic cholecystectomy	LapVR
Loukas et al[33]	2012	44 novices	Peg transfer, cutting, knot tying	LapVR and video trainer
Lucas et al[34]	2008	32 students	Laparoscopic cholecystectomy	OSATS
Mansour et al[35]	2012	48 trainees	Peg transfer, clipping	VR simulators
Munz et al[36]	2007	20 novices	Intracorporeal knot tying	ICSAD and checklist
Munz et al[37]	2004	24 novices	Cutting a shape on a glove and clipping a rubber tube	Motion analysis and error score
Palter et al[38]	2012	25 residents	Laparoscopic right colectomy (live and simulator)	Video-based procedure-specific evaluation tool, modified OSATS global rating scale and LapSim
Palter et al[39]	2013	20 trainees	Clipping, and lifting and grasping, laparoscopic cholecystectomy (actual OR)	Video-based procedure-specific evaluation tool, modified OSATS global rating scale and LapSim
Panait et al[40]	2011	42 applicants	Navigation, coordination, grasping, cutting and clipping	LapSim
Rinewalt et al[41]	2012	20 residents	FLS tasks	GOALS
Rosenthal et al[42]	2006	20 students	Clip and cut cystic duct	Xitact LS500 Virtual Patient
Seymour et al[43]	2002	16 trainees	Laparoscopic cholecystectomy (OR)	Video-based operative error scoring system
Sharma et al[44]	2013	19 trainees	Laparoscopic cholecystectomy	LAP MentorTM
Stefanidis et al[45]	2013	42 novices	Laparoscopic suturing (OR)	GOALS, speed, accuracy and inadvertent injuries
Stelzer et al[46]	2009	23 interns	Peg transfer, intracorporeal knot tying in dry lab,	MISTELS scoring system
			running the bowel, intracorporeal knot tying in live porcine model	Video-based modified GOALS
Tanoue et al[47]	2010	194 surgeons	Lifting and grasping	LapSim
Torkington et al[48]	2001	13 trainees	MIST VR tasks	ICSAD and MIST VR
van Rijssen et al[49]	2012	162 trainees	Intracorporeal knot tying	OSATS and Motion Analysis Parameter (MAP)
Varas et al[50]	2012	25 residents	Laparoscopic jejunojejunostomy	OSATS, ICSAD, FPA

¹FLS tasks are peg transfer, pattern cut, endoloop placement, suture with an extracorporeal knot and suture with an intracorporeal knot. GRS: Global rating scale; OSATS: Objective Structured Assessment of Technical Skills; FPA: Final product analysis; MIST-VR: Minimally invasive surgical trainer-virtual reality; VR: Virtual reality; FLS: Fundamentals of laparoscopic surgery; GOALS: Global operative assessment of laparoscopic skills; GOALS-GH: Global Operative Assessment of Laparoscopic Skills-Groin Hernia; OR: Operating theatre; MISTELS: The McGill Inanimate System for Training and Evaluation of Laparoscopic Skills; ICSAD: Imperial College Surgical Assessment Device.

Table 3 Study characteristics of studies in assessment of open and laparoscopic skills (n = 4).

Ref.	Year	Number of participants	Tasks	Assessment tool
Beard et al[51]	2011	85 trainees	Mixed tasks (OR)	Procedure-based assessment, OSATS
Fernandez et al[52]	2012	30 PGY 1	Knot-tying, suturing, laparoscopic skills	OSATS, computer metric-based performance assessments
Mittal et al[53]	2012	60 residents	Basic skills(knot tying,wound closure, enterotomy,vascular anastomosis) and FLS	OSATS and FLS
Parent et al[54]	2010	28 interns	Wound closure and FLS tasks	Essential item checklist, economy of time, global competence, FLS system

OR: Operating theatre; OSATS: Objective Structured assessment of technical skills; FLS: Fundamentals of laparoscopic surgery.

Table 4 Characteristics of studies in assessment of endoscopic skills (n = 6).

Ref.	Year	Number of participants	Tasks	Assessment tool
Ende et al[55]	2012	28 residents	OGD	Simulator and observation
Götzberger et al[56]	2011	13 trainees	No mention in abstract	Simulator (5-point Likert scale)
Haycock et al[57]	2010	36 trainees	Colonoscopy (simulator and OR)	Direct Observation of Procedural Skills and Global Scores sheet
Haycock et al[58]	2009	28 trainees	Polypectomy, control of upper GI bleeding and oesophageal dilation and PEG insertion	Station-specific checklist and global score
Shirai et al[59]	2008	20 residents	OGD	11 items 5-grade scale
Van Sickle et al[60]	2011	41 trainees	Colonoscopy	GI Mentor II and GAGES

OGD: Oesophago-gastro-duodenoscopy; GI: Gastrointestinal; PEG: Percutaneous Endoscopic Gastroscopy; GAGES: Global Assessment of Gastrointestinal Endoscopic Skills.

Table 5 Characteristics of studies integrating skills assessment tools in a simulation-based curricula and selection process (n =12).

Ref.	Year	No. of participants	Tasks	Assessment tool
Open skills
Chipman et al[11]	2009	24 trainees	Excision of skin lesion and wound closure	OSATS
Olson et al[13]	2012	11 interns	Open laparotomy, bowel anastomosis	OSATS and survey
Laparoscopic skills
Buzink et al[18]	2012	25 trainees	Diagnostic laparoscopy, laparoscopic cholecystectomy and laparoscopic appendicectomy	LapMentor
		6 experts
Palter et al[39]	2013	20 trainees	Clipping, and lifting and grasping, Laparoscopic cholecystectomy (actual OR)	Video-based procedure-specific evaluation tool, modified OSATS global rating scale and LapSim tasks
Panait et al[40]	2011	42 applicants	Navigation, coordination, grasping, cutting and clipping	LapSim
Rinewalt et al[41]	2012	20 residents	FLS tasks	GOALS
van Rijssen et al[49]	2012	162 trainees	Intracorporeal knot tying	OSATS and Motion Analysis Parameter(MAP)
Varas et al[50]	2012	25 residents	Laparoscopic jejunojejunostomy	OSATS, ICSAD, FPA
Open and laparoscopic skills
Fernandez et al[52]	2012	30 PGY 1	Knot-tying, suturing, laparoscopic skills	OSATS, computer metric-based performance assessments
Mittal et al[53]	2012	60 residents	Basic skills(knot tying,wound closure, enterotomy, vascular anastomosis), FLS tasks1	OSATS and FLS score
Parent et al[54]	2010	28 interns	Wound closure, FLS tasks1	essential item checklist, economy of time, global competence, FLS score
Endoscopic skills
Van Sickle et al[60]	2011	41 trainees	Colonoscopy	GI Mentor II and GAGES

¹FLS tasks are peg transfer, pattern cut, endoloop placement, suture with an extracorporeal knot and suture with an intracorporeal knot. OSATS: Objective Structured Assessment of Technical Skills; FLS: Fundamentals of Laparoscopic Surgery; GOALS: Global Operative Assessment of Laparoscopic Skills; ICSAD: Imperial College Surgical Assessment Device; FPA: Final Product Analysis.

With an increasing emphasis of surgical procedures being undertaken in a minimally invasive approach, it is not unsurprising that the assessment of laparoscopic skills dominate the articles included. This bias is also a result of the reality that laparoscopic skills assessment in a simulator has proved far easier than the assessment of open surgical skills. However, observational-type assessment tools remain the instrument of choice in all the skills, especially when assessing trainees in a real operating theatre (OR).

In the studies identified, 21 employed observational tools, mainly Objective Structured Assessment of Technical Skills (OSATS) as the main scoring system to evaluate their candidates’ technical skills performances in open and laparoscopic skills.

The use of simulators in the assessment of laparoscopic skills was evident in 23 publications. Nineteen studies utilised the objective metrics generated by the simulator only and 3 studies used FLS scoring system. One study[17] combined the objective metrics from the simulator with error or injury scores. A total of 13 studies that assessed laparoscopic skills in simulators were using OSATS or checklist-based tools, solely. Out of these, 2 studies[43,45] assessed trainees in the operating theatre (OR) using video-based observational tools following simulation-based training. Interestingly one study[39] combined the performance score on simulator with performance in the OR. Five studies[14,36,37,48,50] used ICSAD combined with other assessment tools or simulator-generated metrics in both open and laparoscopy.

Table 5 outlines reports that incorporated simulators as part of the course in their curriculum. Two of them were for open surgical skills, 6 studies were for laparoscopic skills, 3 studies were for both open and laparoscopic skills and only 1 for endoscopic skills assessment.

One study[40] used virtual reality laparoscopy simulator to assess general surgical applicants who were shortlisted for the residency interview. However, the scores were not used in ranking the candidates for acceptance into the training programme.

DISCUSSION

Simulation in surgery has been a hot topic among surgical educators for more than a decade. In the early millennium, there was an avalanche of studies using simulators that focused on validating the simulators and proving their reliability and fidelity. Since the year 2000 approximately 173 studies were published that specifically reported construct validity of a wide spectrum of surgical simulators. Many new technologies evolved to progressively improve the existing simulators to higher fidelity systems. However despite the plethora of validation studies being completed over a decade ago there is a glaring hiatus in the literature when one examines the results of the integration of these simulators into surgical training curricula. In particular, there is a lack of study showing the implementation of these simulators in the surgical training institutions across the globe, especially in the arena of surgical skills assessment for credentialing. From our review only 12 studies could be identified from the five hundred triaged that have integrated simulation into a surgical training curriculum. There were 52 studies that used simulators in surgical skills assessment within general surgery. The size of these studies was quite modest with 34 having less than 40 candidates and only 5 having greater than 100 candidates.

The main purpose of having simulators in the surgical training arena is for the acquisition of technical skills appropriate to the level of training. This may be undertaken in a safe training environment both from the trainees and patients’ viewpoint. Simulation-based surgical training is important in teaching the surgical trainees and to monitor their progress along the training programmes until they possess the essential technical skills without risking patients’ lives. In order to grasp this, continuous training and assessment is paramount. Traditionally, trainees’ surgical skills are being assessed by examining the logbook and supervisor feedback after certain amount of time in the service. However it is clear that a logbook records experience and is not a marker of expertise[61]. It contains the number of procedures and supervision code, rather than performance scores for a particular procedure. Therefore, logbooks lack content validity[62]. Supervisor feedback assesses the overall performance of a particular trainee and is not exclusively on the technical skills. It is largely subjective and influenced by multiple factors such as patients’ condition, theatre environment and hospital condition. Therefore, the need for a more robust assessment tool which is objective, reliable and feasible[63] remains.

In our institution surgical simulators are used as part of the initial selection process and thereafter for skills assessment and ongoing training. Irish surgical trainees are required to attend simulation-based operative skills classes throughout their training programme. Apart from the didactic teachings, practical sessions are provided which allow the trainees to practice their skills in open surgery, laparoscopy and endoscopy. Basic surgical trainees are assessed at the end of their training years. Trainees who underperform are required to attend a remedial day where their performances will be discussed with the faculty. For the past 6 years, all candidates shortlisted for Higher Surgical Training (HST) programme in general surgery, cardiothoracic and plastic surgery are required to go through surgical skills assessments prior to their interviews. Their scores carry 10% marks in their overall markings. Gallagher et al[64] showed that four out of five top performers on technical skills stations during selection of higher surgical trainee in general surgery were in the top-ranked applicants overall and subsequently succeeded in being selected into the HST programme. In plastic surgery, Carroll et al[65] proved those applicants selected for HST performed better in all six tasks (laceration repair, Z-plasty, lipoma excision, sebaceous cyst excision, tendon repair and arterial anastomosis) than those who were not.

OSATS remains the selected assessment tool of choice in the evaluation of surgical skills. In our own training programme it is used for all open surgical procedures with inanimate bench models such as bowel anastomosis, excision of lipoma or sebaceous cyst and laparotomy incision and closure. Each station is assessed by an expert surgeon relative to the specialty and all stations are run simultaneously within a time frame. For laparoscopic skills, OSATS assessment is combined with performance on ProMIS™ laparoscopic simulator (Haptica, Dublin, Ireland). The tasks for laparoscopic skills generally include object positioning and sharp dissection. Promis™ simulators score the trainees or candidates according to the total path length, smoothness, time and error. In general surgery and cardiothoracic skills assessment, the GI Mentor endoscopy simulator (Simbionix, Cleveland, OH, United States) and a 15-item checklist are used to assess candidates’ endoscopic skills. GI Mentor could provide time and the percentage of mucosa visualised as objective score in the assessment.

From this review, we identified that the main instruments utilised in practice remain observational tools for both open and laparoscopy. This is despite a myriad of validated computer-based simulators being available in laparoscopy. The most commonly used observational tool is the Objective Structured Assessment of Technical Skills or OSATS. It consists of 2 sets of evaluation checklist; operation-specific checklist and global rating scales. It is consistent with the format of the typical Objective Structured Clinical Examination (OSCE) in which examinees perform a series of clinical tasks at each of several time-limited stations[66]. In another study[41], a different type of observer-dependant assessment tool was used for assessing laparoscopic skills called Global Operative Assessment of Laparoscopic Skills (GOALS). It was developed by a group of researchers[67] in Quebec, Canada. This consists of a checklist and 2 visual analogue scales (VAS). All these observational tools require a minimum of two independent assessors in order to avoid bias in scoring the candidates by single assessor. Therefore, a group of expert surgeons should be recruited to use these assessment tools. This could be done either live during the assessment or by video recordings. Since multiple assessors are required to make these tools valuable, there should be a minimum discrepancy between the scores among the assessors. Otherwise, the scores can be open to critique. In order to prove the degree of agreement among the assessors, inter-rater (IR) reliability is used. IR value should be at 0.8, which means the assessors are in agreement in 80% of the scores but in disagreement in the rest of 20%. A high value of IR reliability indicates that the scores are homogenous and the assessment tool is both robust and of value. In one of the study[13], IR reliability was 0.67 which reflects significant differences of opinion of assessors in the subjective data they are evaluating. This emphasises the weakness of this scoring system, as well as the labour intensive nature of the scoring system. In all these studies the candidates could feel appropriately aggrieved if the arbitrators of success in any task undertaken demonstrated significant difference in opinion as evidenced by such a low IR reliability score. We would contend that the use of a truly objective assessment via simulation in real time must inherently be a stronger approach to assessment.

As with every technology there are a variety of simulators available on the market that has been used in surgical skills assessment. In laparoscopic training and assessment, computer-based simulators are able to provide objective metrics after completion of a laparoscopic task. Some examples of validated virtual reality (VR) simulators available in laparoscopy are MIST VR, LapSim, LapMentor and Xitact LS500[68]. These simulators are able to assess various laparoscopic skills such as camera navigation, object positioning and manipulation, intracorporeal suturing and sharp dissection. However, the main criticism on VR simulators is that they lack of real life representation such as delayed gravity effect and no haptic feedback, as found in LapSim[36].

A hybrid simulator, ProMIS^TM (Haptica, Dublin) used 100% VR for certain tasks and augmented reality that overlays graphics onto a task performed on a physical exercise[69]. It provided the tactile feedback which is lacking in most VR simulators. VR and hybrid simulators are able to quantify skills in terms of path length, smoothness, economy of movement and time. The simulators also are able to identify the errors performed specific to the procedures and include them in the final report. Various studies have shown their validity and reliability[70-76]. However, these simulators are largely used for learning and practising the skills but rarely used as an assessment tool. Only 56% of the studies in this review employed simulator-generated objective metrics in the laparoscopic skills assessment, either exclusively or combined with other assessment tools.

Endoscopic skills also can be trained and assessed using simulators. Training in endoscopy in a virtual environment is thought to be a good alternative to classical bedside teaching, but without its adverse effects, such as patient discomfort, risk of perforation, and longer examination time[77]. GI Mentor (Symbionix, Israel) is one of the commonest endoscopy simulators used in surgical training institution. After the performance of a case on the simulator, the trainee is presented with an evaluation of performance such as time taken, percentage of mucosa visualized, and percentage of time spent without clear vision (red-out)[78]. Recently, Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) developed Fundamental of Endoscopic Surgery^TM (FES^TM) as a training and assessment tool for basic skills in endoscopy[79].

There are fundamental differences in the skills required for laparoscopic surgery as compared to open surgery[80]. Without doubt it is clear from the literature that the use of simulators in open surgery represents a challenge. In general the progression of simulator development has tended to target minimally invasive surgery (MIS)[75]. Nonetheless, open surgery remains to be the paramount procedures across surgical specialties. It is vital to teach surgical trainees and assess their skills in open surgery during their training years. Inanimate bench models such as the laparotomy model from Simulab Corporation (Seattle, WA), skin pads and saphenofemoral junction model from Limbs and Things (Bristol, United Kingdom) are amongst most commonly used in training and assessment. Animal models, either cadaveric or live, have been used in some studies but plagued by ethical issues in regards to animal rights. In United Kingdom, the use of live animals is not permitted under the current law, unlike in Europe, United States and other countries[81]. Martin et al[82] showed bench top simulations gave equivalent results to the use of live animals.

The challenge for the assessment of open surgical skills is to decide what parameters should be evaluated. The role of simulators in the assessment of open surgery however may lie in the determination of a surgeon’s dexterity. The objective measurement of a surgeon’s technical skill or level of dexterity has proved to be very difficult. Surprisingly only 1 study combined OSATS with motion analysis system in an attempt to capture the essence of dexterity[10]. The technology behind the measurement of dexterity in surgery and in particular open surgery is however slowly evolving. The researchers in Imperial College London developed a motion tracking system called Imperial College Surgical Assessment Device (ICSAD). This is a combination of a commercially available electromagnetic tracking system (Isotrak II, Polhemus Inc,Colchester, VT) and bespoke computer software program[83]. It measures the time taken, path length and number of movements in open and laparoscopy skills assessment. This has been shown that the measurements were able to discriminate different level of surgical experience in laparoscopy[48] and open surgical procedures[84]. Then, RObotics VIdeo and Motion Assessment Software (ROVIMAS) replaced the former ICSAD motion analysis software and integrated an improved version of data acquisition module including real-time synchronized motion-video capture functionality[85]. Despite these technologies being now over a decade old it remains largely a research tool rather than incorporated into main stream curricula.

The measurement of dexterity alone is insufficient without it being in the appropriate context. In essence, dexterity may be independent of the quality of the end result. This represents surgical context. Errors such as slip knot and incorrect suture placement could cause horrendous morbidity towards patients. It is the appreciation of these errors that underpins the concept of placing skills assessment and the associated metrics in the correct context.

The majority of assessing errors or analysing the end product is observational. A crude assessment of the quality of the final product is by using a 5-point scale[86]. Scott et al[87] formulated a proficiency score which include a series of errors observed for knot tying and suturing skills and maximum allowable task duration as cutoff time. The formula used was as follow: Score = (cutoff time) - (completion time) - 10 (sum of errors); a higher score indicates superior performance[86]. A significant weight was given to the sum of errors showing the importance of the end-product quality in surgical skills assessment. Patel et al[88] developed low-fidelity exercises for basic skills training and assessment and proved its validity. The exercises were needle driving, knot tying, two-hand coordination and fine motor coordination. The metrics measured include time, accuracy and number of targets completed for needle driving exercise or number of appropriate knots for knot tying exercise. Again, this is open to bias and labour intensive. In practice, the quality of knots is easily tested by spreading the loop until they are either break or slip. However, this is hardly performed with a standardised force by surgical educators. Several studies have used tensiometers to assess the quality of the knots[89-92]. Brydges et al[93] developed a measurement for wound closure skill performance called ‘absolute symmetry error’, which measure related to the “bite size” on each suture placement. It does not require an expert assessor and feasible for self-training and assessment. A few studies assessed the end product of bowel anastomosis by measuring the leak pressure[12,50]. These studies combined the validated assessment tool with final product analysis (FPA). By combining these components in the assessment of skills, trainees’ appraisal is thought to be more accurate and apparent. From our review of the literature, only 2 studies[17,28] combined virtual reality simulator generating metrics combined with error scoring systems in their assessments. This approach would seem sensible when one is considering surgical skills assessment.

There is a vast quantity of published data available underpinning the validity of surgical simulators. However, it was abundantly evident from our review that only a small number of papers have outlined their use as part of a training curriculum. It should be noted that the literature search was restricted to English language publications only. In total, twelve studies were identified that incorporated simulation-based training in the curriculum. The participants in these studies went through various periods of time in training using simulators and their performances in technical skills were assessed at the end of the training phase. OSATS or other observational tools were used to assess open skills in 5 studies. For laparoscopic skills, only 2 of these articles used simulators alone in the assessment and 3 studies combined the simulator score with observational assessment tools. Only 1 study[50] assessed trainees’ performances using multi-modal assessment tools which were OSATS, ICSAD and final product analysis (FPA) (leakage and permeability of an anastomosis). In another study[60], endoscopic skills were assessed by a combination of the simulator-based scores and Global Assessment of Gastrointestinal Endoscopic Skills (GAGES) scores. Two studies developed intensive boot camp session for new residents in order to boost their basic technical skills at the start of their training programme[52,54]. Both studies assessed open technical skills using observational tools and for laparoscopic skills, one study[52] used computer-generated metrics and the other study[54] used FLS scoring system. Fernandez et al[52] proved that the new residents’ performances improved after the 9-wk intensive course. However, in the other study[54] the boot camp course ran for only 3 d and the performances did not show any significant difference compared to the control group. Interestingly, only 1 study[38] assessed trainees performances in a simulation lab and thereafter in OR. After training to proficiency with the simulators, the trainees were required to perform laparoscopic cholecystectomy with their supervisors and the performances were video-recorded. These recordings were then assessed using observational tools. This was the only study that seemed to report active integration of simulator based surgical skill training and translation into real time clinical practice.

It is clear that the assessment of surgical skills in simulation laboratories is robust. The critical question is whether the skills acquired from simulation-based curriculum are transferrable to real operations. The most recent systematic review by Buckley et al[94] demonstrated that simulation-based training has a positive impact on operative time and predefined performance scores in the OR but not the quantifiable measures such as ergonomics, hand dominance and smoothness of movement as measured by simulators. The fundamental assumption of simulation-based training is that the skills acquired in simulated settings are directly transferable to the operative setting[95]. If this assumption is proven to be true, simulation-based curriculum must be one of the main pillars in creating top-quality surgeons which in turn would guarantee an excellent patient care and safety.

Over the last decade, observational assessment tools, such as OSATS, remain the most used methodology to assess surgical skills. It has been over a decade since motion tracking systems were reported as effective tracking tools in assessing surgical skills[96]. Despite the advancement in simulation technology, this available technology has not been fully incorporated into surgical training curricula. This is particularly true for the assessment of open skills. One must therefore query why this is the case. We initially had a frenzy of validation studies since the turn of the millennium in relation to simulators. Following this, technology has only improved in terms of fidelity and reproducibility. The dearth of information in the literature regarding the efficacy of the use of simulators in training programmes may be related to the paucity of data on translating simulator based training into the real patient setting. Yet the conversion from VR to OR as coined by Professor Anthony Gallagher[97] perhaps is finally beginning to get traction. In the past 14 years there have been 12 articles that report their experience of simulators within their general surgical training programmes. One of these has now translated this VR training into OR in practice.

The integration of surgical skills assessment as part of the selection process for Higher Surgical Training (HST) selection in the Irish National Training Programme is a further example of the potential that simulation holds for the surgical training community. One can only hope that over the next decade, now that the validity of simulator based training has finally being accepted, the future of simulation-based surgical training will no longer stand on the precipice but finally take flight.

COMMENTS

Background

The traditional apprenticeship model for surgical training as described by William Halstead is reliant on opportunity in order to gain sufficient surgical skills. In the current climate, surgical training is focusing on integrating simulators in the formal curricula, including surgical skills assessment of the trainees.

Research frontiers

For the past 14 years, there is a plethora of published studies that involved validation of various simulators. However, the integration of these simulators in surgical skills assessment as an objective measurement is still minimal.

Innovations and breakthroughs

The authors identified that observer-dependant tool remains largely a tool of choice when assessing both laparoscopy and open technical skills. Some of the studies outlined the use of simulators in objective assessment of laparoscopic skills and minimum amount of studies showed the application of non-observer dependant tool in the assessment of endoscopic and open surgical skills.

Applications

The assessment of surgical skills using simulators is highly applicable in surgical curriculum. The next step is to engage the simulation technology in the assessment of technical skills in a real operative setting.

Terminology

Observational tool: Checklist-based assessment tool used by surgical experts; OSATS: Objective Structured Assessment of Technical Skills.

Peer review

This is a very nice review of the available literature on the results of simulation training on surgical residents.