A decision-making learning and assessment tool in laparoscopic cholecystectomy

Abstract

Making correct decisions is an integral part of surgical competency and excellence. The learning of this expert skill takes years to accumulate during training. To date there has not been an attempt to accelerate this learning process by developing a tool. In our present study we develop a self-appraisal computer software learning and assessment decision-making tool for laparoscopic surgery. It aims to accumulate several years of varied surgical experience, so the trainee can start to learn the complexities of surgical decision making in various types of cases. In this study we aim to validate the tool.

Three decision-making modules were developed in a computer program for laparoscopic cholecystectomy: knowledge of operation, operative surgical technique and operative task completion. The latter two modules were based on answering questions based on watching recorded live operations from a library of 100 recorded laparoscopic cholecystectomies of various grades. The questions were devised by two experienced surgeons with more than 14 years postgraduate surgical experience. To validate the tool two groups with varying surgical experience were assessed: intermediate and expert surgeons. These groups were determined by the number of laparoscopic cholecystectomies performed as well as of number of years of operative surgical experience.

A total of 20 subjects were assessed, 12 intermediate and 8 experts surgeons. Mean time to perform the programme was 21 min (range 18-45 min). Using the Mann-Whitney test, p < 0.05, construct validity was demonstrated in the surgical technique and completion of task modules as well as the total combined scores.

Our computer-based decision-making learning tool for laparoscopic cholecystectomy seems to have face, content, concurrent and construct validities. Surgical decision making is a multifaceted process; by assessing how and why decisions are made effectively, focussed surgical training may be achieved. We aim in the future to determine if the self-appraisal decision-making tool improves or accelerates surgical training. [PUBLICATION ABSTRACT]

Surg Endosc (2009) 23:197203 DOI 10.1007/s00464-008-9774-6

A decision-making learning and assessment tool in laparoscopic cholecystectomy

Sudip K. Sarker Saif Rehman Meera Ladwa Avril Chang Charles Vincent

Received: 17 July 2007 / Accepted: 6 November 2007 / Published online: 5 March 2008 Springer Science+Business Media, LLC 2008

AbstractAims Making correct decisions is an integral part of surgical competency and excellence. The learning of this expert skill takes years to accumulate during training. To date there has not been an attempt to accelerate this learning process by developing a tool. In our present study we develop a self-appraisal computer software learning and assessment decision-making tool for laparoscopic surgery. It aims to accumulate several years of varied surgical experience, so the trainee can start to learn the complexities of surgical decision making in various types of cases. In this study we aim to validate the tool.

Methods Three decision-making modules were developed in a computer program for laparoscopic cholecystectomy: knowledge of operation, operative surgical technique and operative task completion. The latter two modules were based on answering questions based on watching recorded live operations from a library of 100 recorded laparoscopic cholecystectomies of various grades. The questions were devised by two experienced surgeons with more than 14 years postgraduate surgical experience. To validate the tool two groups with varying surgical

experience were assessed: intermediate and expert surgeons. These groups were determined by the number of laparoscopic cholecystectomies performed as well as of number of years of operative surgical experience.

Results A total of 20 subjects were assessed, 12 intermediate and 8 experts surgeons. Mean time to perform the programme was 21 min (range 1845 min). Using the MannWhitney test, p \ 0.05, construct validity was demonstrated in the surgical technique and completion of task modules as well as the total combined scores. Conclusions Our computer-based decision-making learning tool for laparoscopic cholecystectomy seems to have face, content, concurrent and construct validities. Surgical decision making is a multifaceted process; by assessing how and why decisions are made effectively, focussed surgical training may be achieved. We aim in the future to determine if the self-appraisal decision-making tool improves or accelerates surgical training.

Keywords Assessment Cholecystectomy

Decision making Laparoscopic Surgery

Making accurate decisions is crucial to the delivery of effective and safe surgical healthcare, especially since surgeons have to make many complex decisions in continuously changing circumstances. While decision making has been studied in depth in high-risk industries such as aviation, the importance of this skill is being increasingly recognised by surgeons, hence the value in nding new and better ways of teaching trainees how to make accurate intraoperative decisions [1130].

The learning of this expert skill takes many years to accumulate during training as well as post training. To date

S. K. Sarker S. Rehman

University Department of Surgery, Royal Free Hospital, London, UK

S. K. Sarker (&)89 Cornwall Gardens, London SW7 4AX, UK e-mail: [email protected]

M. Ladwa C. Vincent

Clinical Safety Research Unit, Imperial College London, London, UK

A. ChangDepartment of Surgery, Kings College Hospital, London, UK

123

198 Surg Endosc (2009) 23:197203

there has not been an attempt to accelerate this learning process, to learn how and why surgeons make decisions whilst operating. In our present study we develop a self-appraisal learning decision making tool in laparoscopic cholecystectomy named LapSkill. Figure 1ac illustrates sample questions from each module. If validated it could accumulate several years of varied experience in different types and grades of laparoscopic cholecystectomy, so the trainee can start to learn the complexities of surgical decision making in laparoscopic cholecystectomy.

Research suggests that failures in decision making may contribute to adverse events in healthcare as much as failures in technical competence [1]. If accurate decisions lead to better patient outcomes, how can we use decision-making research to train surgeons to make the right decisions in the operating theatre?

The process by which surgeons make decisions has been described as the integration of evidence, inference and experience [2], for example, in deciding whether to perform a primary anastomosis or a Hartmanns procedure for a particular patient, the surgeon would use data from clinical trials (evidence), logical reasoning using his biological knowledge (inference) and awareness of his own capabilities and the patients preferences and needs (experience). All three sources of input can complement each other, but controversy arises when they are contradictory or when one is lacking, e.g. the absence of rigorous trial data. It is in these situations that the ability to make an accurate decision is most crucial.

Currently, surgeons are trained to make intraoperative decisions through feedback in the classroom, the laboratory, the clinic and in theatre. Advances include the use of virtual-reality simulation, particularly in the teaching of laparoscopic techniques, which have allowed trainees to develop their decision-making skills before they perform on real patients. Serious trauma is a situation that calls for rapid and effective decision making and there are various training courses available that aim to teach the recognition and management of trauma in a structured and systematic mannerthese include Advanced Trauma Life Support (ATLS), Care of the Critically Ill Surgical Patient (CCrISP) and Denitive Surgical Trauma Skills (DSTS). However, in general terms here has been a lack of emphasis on the structured teaching of the so-called nontechnical skills, with recognition that decision-making, as an entity in its own right, is poorly tutored [3]. Additionally, trainees often only get the chance to apply their decision-making skills when they are actually in the operating theatre, a high-pressure environment which is not ideal for learning; nor is it ideal in terms of safety, since the trainee is dealing with a live patient. With this in mind, the experience of making dynamic surgical decisions in a controlled environment may be valuable in the preparation and training of surgeons.

This is particularly the case since consultant surgeons have been shown to regard many cognitive skills as even more valuable than technical skills in surgical trainees [4] and have ranked decision-making ability as the most important personality trait required for a competent surgeon [5]. It has been said that 75% of the important events in an operation are related to making decisions and only 25% to manual skill [6]. Therefore, it has become necessary to look at innovative ways of training junior surgeons to develop skills that normally take many years to amass, especially considering the European working-time directive, which will limit the experience that trainees have in theatre.

Competency in surgery is said to be a combination of technical skill, teamwork and decision making [7] and to date research has largely concentrated on the assessment of technical skills in laparoscopic surgery. Now that the General Medical Council is currently moving towards a system of compulsory revalidation for all surgeons in the UK, there is a need for a reliable, valid and feasible means of teaching decision-making skills while avoiding subjective bias. LapSkill has been developed as one such tool by the rst two authors. It is based on laparoscopic cholecystectomy, an elective operation performed routinely in large numbers in the UK and worldwide [8], and consists of three modules, assessing the didactic knowledge of the operation, surgical technique and task-related decision-making ability of the operator during the procedure. These three key elements have been chosen because they are required in combination for a surgeon to be able to operate independently and competently.

As for any piece of software to be used as a structured learning and assessment tool, it must be reliable, valid and practical [9]. Validity is manifest in several different forms, e.g., concurrent validity refers to the extent to which the instrument correlates with other measures known to be valid, while predictive validity refers to the extent to which it predicts future performance. This study concentrates on construct validity, meaning the extent to which the tool actually measures decision-making ability between the various groups.

Methods

Development

The methodology for the development of the computer-based laparoscopic learning and assessment tool consisted of two stages: the cognitive task analysis of laparoscopic cholecystectomies and the implementation of the derived conceptual framework as a basis for the specications of the learning and assessment tool.

123

Surg Endosc (2009) 23:197203 199

Fig. 1 Sample questions from each LapSkill module

123

200 Surg Endosc (2009) 23:197203

Stage 1: cognitive task analysis of laparoscopic cholecystectomy

In order to understand how surgeons act during a surgical procedure, it is necessary to understand how they structure information and make decisions during task performance. Particularly when the tasks they are doing are complex, it is not enough simply to observe their behaviours. It is also important to nd out how they think and what they know, how they organize and structure information, and what they seek to understand better. For this purpose the cognitive task analysis (CTA) of laparoscopic cholecystectomy was performed. The purpose of carrying out CTA of laparoscopic cholecystectomies was to identify cognitive issues that will be important for training and assessment in real and/or simulated environments.

The basic CTA methodology consists of videotaping and analysing the laparoscopic view and a view of the surgeon, observational analysis, concurrent verbal protocols and postoperation interviews. This resulted in the development of an initial protocol for CTA of laparoscopic cholecystectomy as previously published [8].

Subsequent analysis of data gathered using the CTA protocol led to the development of the framework for characterising the cognitive aspects of surgery and mapping them to physical tasks (Table 1).

This analysis derived from using the protocol and the framework provides the basis for the specications of a cognitive training and assessment simulator. This prototype will be used to assess the cognitive and decision making skills of surgeons. It will be validated by seeing whether it can distinguish between intermediate and expert surgeons.

The CTA framework identies the surgical skills required for successful task completion. As apparent from the framework, successful task completion is dependent on how the surgeon perceives information both visually and through haptics, how he uses his procedural knowledge to advance through the procedure, how he uses his knowledge of anatomy to identify the correct structures and how he perceives hazards and make decisions accordingly in order to prevent those hazards.

Stage 2: Implementation

On the basis of CTA and surgical skills classication, the design for the learning and assessment tool was performed, which included the following:

1. Assessment of all columns of the CTA framework, i.e., perception, knowledge (anatomical and procedural), hazard perception and decision making according to the classication.

2. Ability to use different scenarios to assess decision making in complex and difcult surgical cases.

3. Use of video clips from the recorded operations in order to give more realistic picture of the scene.

4. The tool will comprise of three modules: knowledge of operation, operative surgical technique and operative task completion.

Each module consists of 15 questions at three levels (ve each). The computer software randomly changed the order of the questions each time the software was used. The knowledge-based module consists of questions on basic theory, including anatomical knowledge of the operation; the Royal College of Surgeons of England course manual on laparoscopic cholecystectomy was used as a reference. Each correct answer was based on the manual. The surgical technique module consists of questions based on various video clips of laparoscopic cholecystectomies. These video clips were from a library of 100 cases, consisting of various grades of gallbladders, various grades of surgeons performing the operations and various technical and patient anomalies. The operative task completion module also included video clips from the same library. It was used to assess decision making in completing a task in normal and critical situations for completion of the task in hand. For these two last modules the correct answers were based on two experienced surgeons assessing each video independently and concluding the best option as the correct answer. Inter-rater reliability for each correct answer had to be j = 1.00 for the video clip question to be included in the learning tool as previously published [8].

In order to carry out an analysis and for correlations, questions within each module were divided into simple, moderate and difcult levels of questions. This classication was kept blind from the user and used for analysis purposes. Each individual took all three modules and all three levels of questions. The level of question was not disclosed to the participant. Modules were in a different order for each user of the same group.

A simple two-tier application was developed, using a database and a simple graphical user interface (GUI). A Microsoft Access 2002 database was used for storing all information, questions and video links (the video les were stored on the hard disk in a separate directory). Microsoft Visual Basic (VB) 6.0 was used to develop a simple GUI.

The result were stored as percentage scores at each level (intermediate and expert surgeons) within the module as

Table 1 Cognitive framework for laparoscopic cholecystectomy

Physical tasks

Perceptual information

Procedural knowledge

Anatomical knowledge

Hazards

123

Surg Endosc (2009) 23:197203 201

well as a total percentage score for each module and a grand percentage for all three modules combined. Answers given by the user for each question in each module were also stored for individual tests, in order to analyse and correlate between and within groups.

Methods

Design of study

The research was a prospective nonrandomised clinical trial. Participants with varying levels of expertise completed LapSkill in order that their scores could be compared.

Subjects

Participants were divided into two categories: group 1 intermediates (senior house ofcers and specialist registrars 14 years), who had performed fewer than 50 laparoscopic cholecystectomies, and group 2experts (specialist registrars 56 years or consultants, with an interest in laparoscopic surgery), who had performed more than 150 laparoscopic cholecystectomies

Participants included surgeons from ve different London hospitals. The only exclusion criterion was prior experience with the LapSkill programme.

Procedure

Twenty participants were recruited by telephone, email and letter. The software was loaded onto a personal computer (PC) or laptop at the participants preferred location. They were asked to complete the programme in their own time and to follow the instructions as they appeared on screen. The investigator was present throughout as participants completed LapSkill, to ensure that no conferring took place and to make notes of any comments or problems experienced with the software as and when they occurred.

Ethical approval

Local Ethics Committee approval was secured and all surgeons and students gave verbal consent to take part in the study. They were all given the opportunity to decline to participate.

Data analysis

The data collected automatically by the software consisted of the percentage scores for each module and the total score for LapSkill. Data was tabulated as a spreadsheet and then analysed using the SPSS software statistical package (SPSS, Chicago, Illinois, USA). Mean scores between each category were compared using the MannWhitney test, taking a value of p \ 0.05 as signicant.

Results

Demographic data

A total of 20 surgeons completed LapSkill. The participants consisted of 8 experts and 12 intermediates surgeons. Eighteen of the participants were male and 2 were female. Mean time to complete the test was 29.12 8.55 min, 95% condence interval 1845 min.

LapSkill data

Scores achieved on LapSkill tended to rise with increasing expertise, as shown in the surgical technique module (Fig. 3), the completion of task module (Fig. 4) and in total (Fig. 5). This was not shown for the knowledge module (Fig. 2).

These differences were statistically signicant with the MannWhitney test between the expert and intermediate categories for the surgical technique module (p = 0.038), the completion of task module (p = 0.004) and for the total scores (p = 0.001). There was no signicant difference in

Fig. 2 Operative knowledge total scores-intermediate & expert surgeons

123

202 Surg Endosc (2009) 23:197203

scores for the knowledge-based module between the intermediate and expert categories (p = 0.35).

Discussion and conclusions

Construct validity was demonstrated for LapSkill by the signicant increase in overall scores between intermediate and experts surgeons. While there were no signicant differences between the scores of experts and intermediates in the knowledge-based module, experts scored signicantly higher than intermediates in the completion of task and surgical technique modules. This suggests that, while knowledge is gained at an earlier stage of training, decision-making task completion and surgical technique require more expertise and takes longer to accrue. It also suggests that accurate decision-making task completion and surgical technique are important qualities for trainees to learn in order to be capable of operating independently and becoming an expert.

There are several areas where further work could be undertaken to maximise the value of LapSkill.

It could be worthwhile to pilot a web-based version of the modied LapSkill with options for feedback, in order to collect data on how feasible and useful the tool is from a larger study group. The predictive validity of LapSkill might be demonstrated by retesting a cohort of intermediate surgeons after a designated period of surgical training and conrming a signicant rise in their scores.

Tools such as LapSkill have many strong points. They are accessible: being electronic they are easily distributed, making them available to surgeons all over the country; they are relatively quick and easy to use, making them ideal for self-assessment; and they are adaptable, lending themselves to a range of applications for the training of surgeons. While this particular model considers laparoscopic cholecystectomy, it could also be used as a template to develop several generic programmes dealing with various procedures, e.g. inguinal hernia repair, appendectomy or varicose veins. These programmes could be distributed as CD-ROMs with written material to provide an adjunct to the teaching of intraoperative decision making to novice and intermediate surgeons. It would also be possible to make LapSkill available as a web-based resource, so it could be easily accessed both by junior surgeons as a learning tool and by expert surgeons as a way of self-appraising their own decision-making skills.

Another prospect is the incorporation of LapSkill as a preliminary module into a virtual-reality laparoscopic simulator. Evidence suggests that watching videos of laparoscopic cholecystectomies in a structured, guided manner increases the subjects perceptual skills and is likely to be an effective way of teaching the recognition of critical cues

Fig. 3 Operative surgical technique total scores-intermediate & expert surgeons

Fig. 4 Operative completion of task total scores-intermediate & expert surgeons

Fig. 5 LapSkill combined total scores-intermediate & expert surgeons

123

Surg Endosc (2009) 23:197203 203

[10]. Use of LapSkill combined with a virtual-reality simulator could bring together training in technical skills, knowledge and decision-making task completion ability, a holistic approach which would reinforce skills acquisition for both junior and intermediate surgeons. It would also be an economic and efcient way of providing junior surgeons with teaching in different aspects of the curriculum within a single format.

Experts achieved signicantly higher scores in LapSkill compared to intermediates, demonstrating construct validity for LapSkill as a training and assessment tool in surgical decision making. Use of the tool was feasible and participants found LapSkill easy to use and largely relevant to the real-life performance of a laparoscopic cholecystectomy. Since competence in surgery requires good decision-making skills, it is hoped that LapSkill will allow trainees to learn how to make accurate surgical decisions in an effective, safe manner, and that it will ensure the objective and fair assessment of surgeons outside the theatre. It could be used in the training and reappraisal of surgeons to develop their ability to make good intraoperative decisions, therefore contributing to the ultimate goal of improving patient safety.

Statement of Disclosure None declared

References

1. Bogner M (ed) (2004) Misadventures in health care. LEA, Mahwah

2. Marshall JC (2006) Surgical decision making: integrating evidence, inference and experience. Surg Clin North Am 86(1):201 215

3. Youngson GG (2000) Surgical competence: acquisition, measurement, & retention. Royal College of Surgeons of Edinburgh, Edinburgh

4. Baldwin PJ, Paiseley AM, Brown SP (1999) Consultant surgeons opinion of the skills required of basic surgical trainees. Br J Surg 86(8):10781082

5. Cuschieri A, Francis N, Crosby J, Hanna GB (2001) What do master surgeons think of surgical competence and revalidation? Am J Surg 182(2):110116

6. Spencer F (1978) Teaching and measuring surgical techniques: the technical evaluation of competence. Bull Amer Coll Surg 63:912

7. Sarker SK (2003) Courses, cadavers, and counsellors: reducing errors in the operating theatre. BMJ 327(7418):s109

8. Sarker SK, Chang A, Vincent C, Darzi SA (2006) Development of assessing generic and specic technical skills in laparoscopic surgery. Am J Surg 191(2):238244

9. Sidhu RS, Grober ED, Musselman LJ, Reznick RK (2004) Assessing competency in surgery: where to begin? Surgery 135(1):620, Review

10. Guerlain S, Brook Green K, Luniewski MS, Mersch TC, Mitchell BA, Reed Poole G, Adams R, Forrest Calland J, Bovbjerg V, Chekan EG (2003) Training anatomy recognition through repetitive viewing of laparoscopic surgery video clips. Human Factors and Ergonomics Society. National Conference, Denver, Co

11. Hastie R, Dawes RM (2001) Rational choice in an uncertain world: the psychology of judgement and decision making. Sage Publications, Thousand Oaks, CA

12. Zsambok CE, Klein G (1997) Naturalistic decision making. Lawrence Erlbaum, Mahwah, NJ

13. Edwards W (1954) The theory of decision making. Psychol Bull 51:380417

14. Chapman GB, Sonnenberg FA (eds) (2000) Decision making in health care: theory psychology, and applications. Cambridge University Press, New York

15. Kahneman D, Tversky A (1974) Judgements under uncertainty: heuristics and biases. Science 185:11241131

16. Janis IL, Mann L (1977) Emergency decision making: a theoretical analysis of responses to disaster warnings. J Human Stress 3(2):3545

17. Sutton GC (1989) Computer-aided diagnosis: a review. Br J Surg 76(1):8285

18. Orasanu J (2005) Crew collaboration in space: a naturalistic decision-making perspective. Aviat Space Environ Med 76(6 Suppl):B1 5463, Review

19. Klein G (1998) The Recognition-primed decision model. In sources of power: how people make decisions. MIT Press, Boston

20. Dreyfus HL, Dreyfus SE, Athanasiou T (1986) Mind over machine: the power of human intuition and expertise in the era of the computer. Free Press, New York

21. Sarker SK, Chang A, Darzi A, Vincent C (2006) Decision Making in Laparoscopic Surgery. Presented at the 13th EAES annual meeting. Venice, Italy, June 2005. Surg Endosc 20:s10

22. Hammond K. Judgements under stress. Oxford Press23. Clarke JR (1989) Decision making in surgical practice. World J Surg 13(3):245251

24. Dominguez CO (2001) Expertise and metacognition in laparoscopic surgery. Human Factors Ergon Soc 45:12981304

25. Satish U, Streufert S, Marshall R., Smith JS, Powers S, Gorman P, Krummel T (2001) Strategic management simulation is a novel way to measure resident competencies. Am J Surg 181(6):557 561

26. Paisley AM, Baldwin P, Paterson-Brown S (2001) Feasibility, reliability and validity of a new assessment form for use with basic surgical trainees. Am J Surg 182(1):2429

27. Brothers TE, Cox MH, Robison JG, Elliott BM, Nietert P (2004) Prospective decision analysis modeling indicates that clinical decisions in vascular surgery often fail to maximize patient expected utility. J Surg Res 120(2):278287

28. Salem L, Veenstra DL, Sullivan SD, Flum DR (2004) The timing of elective colectomy in diverticulitis: a decision analysis. J Am Coll Surg 199(6):904912

29. Larson JL, Williams RG (2005) Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents. Surgery 138(4):640647; discussion 647649

30. Servais EL, Lamorte WW, Agarwal S, Moschetti W, Mallipattu SK, Moutlon SL (2006) Teaching surgical decision-making: an interactive, web-based approach. J Surg Res 134(1):102106

123