Is there a benefit for anesthesiologists of adding difficult airway scenarios for learning fiberoptic intubation skills using virtual reality training? A randomized controlled study

Abstract

Fiberoptic intubation for a difficult airway requires significant experience. Traditionally only normal airways were available for high fidelity bronchoscopy simulators. It is not clear if training on difficult airways offers an advantage over training on normal airways. This study investigates the added value of difficult airway scenarios during virtual reality fiberoptic intubation training. A prospective multicentric randomized study was conducted 2019 to 2020, among 86 inexperienced anesthesia residents, fellows and staff. Two groups were compared: Group N (control, n = 43) first trained on a normal airway and Group D (n = 43) first trained on a normal, followed by three difficult airways. All were then tested by comparing their ORSIM^® scores on 5 scenarios (1 normal and 4 difficult airways). The final evaluation ORSIM^® score for the normal airway testing scenario was significantly higher for group N than group D: median score 76% (IQR 56.5–90) versus 58% (IQR 51.5–69, p = 0.0039), but there was no difference in ORSIM^® scores for the difficult intubation testing scenarios. A single exposure to each of 3 different difficult airway scenarios did not lead to better fiberoptic intubation skills on previously unseen difficult airways, when compared to multiple exposures to a normal airway scenario. This finding may be due to the learning curve of approximately 5–10 exposures to a specific airway scenario required to reach proficiency.

Introduction

The management of difficult airways is associated with significant mortality in both the Intensive Care Unit (ICU) and the operating room [1, 2]. Airway management devices have evolved and guidelines in airway management have changed over the last decade [3–7]. Flexible bronchoscopy (FB) remains an accepted first choice for patients with known difficult intubation or as a rescue technique for non-anticipated difficult intubations [8]. However, learning FB remains challenging because of its infrequent use which is further compounded by increasing use of video-laryngoscopes [4, 8–11]. Simulation appears to be an efficient tool for training health care providers using otherwise rare clinical scenarios [12–14]. Many studies have indeed shown the value of simulation for bronchoscopy training but none of them have studied its use during difficult airway management [15–26].

Until the recent development of a virtual reality fiberoptic simulator with multiple difficult airway scenarios, mostly normal airways were available for high fidelity bronchoscopy simulators [15, 17, 21, 22, 27–29]. It is, however, not clear if training on difficult airway scenarios offers a significant advantage over training on normal airway scenarios [30].

We hypothesized that inexperienced anesthesia providers would improve by training on a virtual reality simulator with difficult airway scenarios when compared to training on normal airway scenarios (primary outcome: improved performance score, secondary outcome: success rates, times, collision avoidance and perceived level of difficulty).

Materials and methods

This study recruited anesthesia residents, fellows and staff physicians who had performed 10 or less fiberoptic intubations from June 2019 to March 2020. Participating centers were: Toulouse (France), Clermont-Ferrand (France), Coventry (United Kingdom) and Oxford (United Kingdom). The study protocol was not registered but communicated to all participating centers.

Consent

Each participating center obtained ethics approval through their local Research Ethics Committee. This study’s protocol has been registered with the Comité d’Ethique pour les Recherches (CER) of Toulouse University under the number: 2018–099 for the centers in Toulouse and Clermont-Ferrand (France) and by the Health and Research Authority in Wales (Ref 19/HRA/5142) for the centers in Oxford and Coventry (United Kingdom). All participants gave written informed consent.

Virtual reality high fidelity simulator

The ORSIM^® virtual reality bronchoscopy simulator (Airway Simulation Limited, Auckland, New Zealand) incorporates a replica video bronchoscope, desktop sensor module, and dedicated laptop computer. The ORSIM^® contains software recreating a high-fidelity FB intubation scenario for the user. It includes learning modules on airway anatomy and dexterity and uses both normal and difficult airway intubation scenarios. (https://www.orsim.co.nz/design).

ORSIM^® score

Validity and reliability of the ORSIM^® simulator as an assessment tool have been shown in a previous study by Baker et al. and relied on external scoring of a large number of video recording, which was not felt to be feasible for research [30]. Their results provided them the basis for establishing computer-generated metrics [30] and the development of a proprietary ORSIM^® score. This ORSIM^® Score (which has not been validated against expert scores) appears on the ORSIM^® Session Results screen after the participant finishes a given scenario and was recorded to grade their performance.

Gopher-module

To assess pre-test dexterity of each participant, the ‘Gopher’ module, contained in the ORSIM^®, was used: it consists of a challenge game to catch nine different gopher figures using the ORSIM^® bronchoscope.

Data collection

Demographic data were collected including age, gender, dominant hand, years of residency training, previous FB experience, previous ORSIM^® and video game experience. The ORSIM^® simulator’s data were collected including type of scenario, time to successful completion, collision avoidance percentage score and minimal oxygen saturation, ORSIM^® scores for each scenario and the degree of difficulty felt for each of the scenarios as judged by the study subjects.

Study design

The participants recruitment, randomization and allocation are displayed in Fig 1. After obtaining written informed consent, a study form was completed. Participants then watched a video explaining the functionalities of the ORSIM^® simulator and the flexible bronchoscope. A pre-test dexterity test (Gopher module) was done before and then again after five minutes of hands-on familiarization with the bronchoscope simulator.

[Figure omitted. See PDF.]

Afterwards participants were randomized (by random number drawing) into two groups: Group N trained on normal airway scenarios up to a maximum of 40 minutes and group D trained instead on one normal and then three difficult airway scenarios (a retropharyngeal abscess, an epiglottic cyst, and a macroglossia). Each of the participants in group D had up to 10 minutes per scenario before being prompted to move to the next scenario (i.e. up to a total of 40 minutes). Completion times, ORSIM^® scores, collision avoidance, lowest oxygen saturation, perceived level of difficulty on a Visual Analogic Scale (VAS) from 0 (easy) to 100 (very difficult) were recorded.

The final step consisted of testing the participants on their fiberoptic difficult intubation skills by using one normal and four difficult scenarios (airway trauma, severe epiglottitis, false cord cyst and angioneurotic oedema). Each participant had to try to succeed once in a maximum of ten minutes for each scenario. Times, ORSIM^® scores, collision avoidance, lowest oxygen saturation, perceived level of difficulty on a VAS from 0 (easy) to 100 (very difficult) were recorded. The investigator was not allowed to guide or help the participant to solve scenarios. The investigator’s role was strictly to observe and keep time, as well as manage any technical difficulty with the ORSIM^® simulator.

Primary outcome

ORSIM^® scores for the final test scenarios were compared between the two groups for each scenario.

Secondary outcomes

Times and ORSIM^® scores in normal airway scenarios before and after training were compared between the two groups. Successful rates, times, collision avoidance and perceived level of difficulty were also compared.

Statistical analyses

Redcap^® software (Research Electronic Data Capture^®) was used to collect data [31, 32]. R^® software (R Foundation for Statistical Computing, Vienna, Austria) was used for the statistical analyses. Power of this study had been estimated by using the data from the previous ORSIM^® study by Baker et al. [30]: 150 participants were needed to show a significant difference of 25% with a power of 90%. A Chi square test and Fisher’s exact test were used to test differences between group N and D concerning categorial variables. A T-test (or Wilcoxon-Mann-Whitney for not normally distributed data) was used to test differences concerning continuous or ordinal variables. Statistical significance was taken as p<0.05. Results are expressed in median and interquartile ranges or in numbers and percentages.

Results

Recruitment and exclusions

The study began in June 2019 and was stopped in March 2020 due to the Covid-19 pandemic. This analysis includes the 86 participants out of the 150 expected enrolments (43 in group N, 43 in group D) from Toulouse (n = 25) and Clermont-Ferrand (n = 25) in France, the University Hospitals Coventry and Warwickshire NHS Trust (n = 16) and Oxford University Hospitals NHS Foundation Trust (n = 20) in the UK.

105 participations had been recruited, but nine were excluded because of missing or invalid data and a further nine because of a protocol violation (they had already performed more than 10 fiberoptic intubations). Finally, one participant had been recruited twice: only the first of those attempts was kept for analysis.

Demographics and pre-test dexterity

Table 1 shows the demographic data and Table 2 shows the pre-text dexterity. There was no difference in experience level with the fiberoptic bronchoscope or video game expertise between groups N and D and there was no difference in pre-test dexterity between the two groups.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

Final ORSIM^® score (primary outcome)

Table 3 shows the comparison of both groups ORSIM^® scores during the final testing (primary outcome). The final test ORSIM^® score for the normal airway scenario was significantly higher for group N than group D: median score 76% (IQR 56.5–90) versus 58% (IQR 51.5–69) respectively (p = 0.0039). There was no statistically significant difference between the two groups for the final test ORSIM^® score on any of the difficult intubation scenarios (Table 3).

[Figure omitted. See PDF.]

Performance evolution on normal airways

During the first try, at the beginning of training, there was no difference between groups in their performance on the normal airway scenario for ORSIM^® scores, time to succeed and collision avoidance. During the final testing on the normal airway scenario (after completion of the training) the ORSIM^® score for group N was significantly higher and their times and estimated level of difficulty were significantly decreased when compared to Group D (Table 4).

[Figure omitted. See PDF.]

Comparison of perceived level of difficulty between groups during final testing

There was a significant difference of perceived level of difficulty on a VAS (0 to 100) between the two groups concerning the severe epiglottitis scenario as it appeared easier to group D than to group N: 66 (57–80) versus 79 (68–90) for the severe epiglottitis, p<0.05) (Table 5).

[Figure omitted. See PDF.]

Discussion

This study shows a significant performance improvement for fiberoptic intubations on normal airways for the group who trained only on normal airways. In contrast, training on a variety of difficult airways did not translate to performance improvement on (other) difficult airway scenarios or on a normal airway.

Simulation appears as a new tool allowing learners to achieve a satisfactory skill level before practice. Graeser et al. [26] showed that simulator training allows for entry of the learning curve of airway management at a higher level. J.E Smith et al. [33] described a learning curve of 10 fiberoptic intubations to achieve a satisfactory competence and 30–50 fiberoptic intubations to reach expert level. Repetition of a same scenario seems to be the main determinant of learning. In our study, participants in group N had performed a median of 12 successive normal intubations before being tested (over a median of 15 minutes, Table 1), compared to only one single attempt in group D. In group D, each scenario could only be done once and participants in group D had thus performed the balance of their training on various difficult airways (over a median total of 14 minutes, Table 1). As our study shows, that training did not translate into better performance on the normal airway scenario.

In their study on validating the ORSIM^® [30], Baker et al. proposed a score of 70 to characterize an expert level. Our study shows a median ORSIM^® score of 76% after the 12 attempts in the final testing phase for group N on the normal airway, suggesting an satisfactory competence, or even expert level had been acquired after their repetitive training with this same normal airway scenario. On the other hand, for group D, the median ORSIM^® score for each scenario remained well below 70%. It would be interesting to see if the ORSIM^® score during final testing for group D would have improved significantly if Group D also would have trained until achieving ‘expert level’ (score of at least 70%) on each of their difficult training scenarios, but our experimental design did not allow to study that (and would have likely led to group D training for a much longer total time compared to group N, thus introducing a confounder).

The main finding of our study is thus that the learning may initially be limited to the actual scenario and the degree with which learning extends to other scenarios remains unclear (but clinically important). It may be necessary to introduce repetitive training sessions of each difficult scenario before the full benefits would become apparent.

Although training on a variety of difficult airways for group D did not translate to performance improvement, it did seem to lead to a reduction of perceived difficulty level some scenarios (for the severe epiglottitis scenario). Similarly, acquisition of expertise level for group N significantly reduced the perception of difficulty on the normal airway scenario. In practice, we can often observe that stress and perception of difficulty can have a negative impact on performance—and training is probably the only way to remedy this [34]. Our study shows that training on difficult airway scenarios could indeed reduce the stress during subsequent attempts on difficult airway scenarios.

Yang et al. [35] showed that psychometric skills can predict the acquisition of procedural skill performance and Louridas et al. [36], looking at laparoscopic surgery, investigated the value of psychometric testing to predict the technical performance of new residents. It would be interesting to study the relationship between dexterity and time to reach an expert level for fiberoptic intubation skills for a ‘competence by design’ approach to teach fiberoptic intubation.

Limitations of the study

Due to the unforeseen circumstances posed by COVID-19 we were forced to halt trial recruitment and decided to perform an unplanned interim analysis of the results. We put in place a plan for a single interim analysis with a symmetrical stopping rule for a p< 0.002 using the Pocock rule. When we identified a p = 0.0039 for our primary outcome in the direction of improved outcomes for the normal group we decided to stop the trial based on futility. While the number of participants in our study was smaller than planned, the magnitude of the estimated effect sizes was consistently small. Furthermore, the group trained on the difficult scenarios were as likely to show worse results as they were to show improved results. For these two reasons we postulate that the inclusion of more participants would not change the primary outcome.

Some participants in both groups had already used the ORSIM^® simulator, however the numbers were small and also well distributed across both study groups, thus unlikely to create a bias. While it was originally contemplated to exclude participants with previous ORSIM^® experience, it was felt that this would unnecessarily restrict the recruitment in view of the fact that the use of ORSIM^® simulation was expanding rapidly.

In our study, we only assessed the level 2 of Kirkpatrick’s model of training evaluation: the testing of skills based on training [37]. It will be important to investigate actual changes in clinical practice (level 3) and ultimately changes in patient outcome (level 4) when evaluating the role of the ORSIM^® simulator. The latter steps present significant logistical and ethical challenges and may involve the creation of actual patient airway through 3 D printer technology as an interim step.

Conclusion

This study was not able to demonstrate a benefit of adding difficult airway scenarios for learner’s performance during intubation training with a replica flexible bronchoscope. It only showed that repeated training on the ORSIM^® simulator with the same scenario (a normal airway) led to increased intubation skills for the normal airway and that this skill was specific for the normal airway scenario. Training on normal airways did not predict better skills on difficult airways (and vice versa).

Subsequent studies should focus on the acquisition of a level of expertise in difficult airways and its reproducibility from one difficult airway scenario to another in order to validate this simulation model and better prepare the learner for clinical practice.

Acknowledgments

Professor Richard Cooper added valuable advice at the beginning of the study. Dr. Paul Baker assisted with the design and development of the study and provided a thoughtful review of the draft manuscript.

Citation: Cailleau L, Geeraerts T, Minville V, Fourcade O, Fernandez T, Bazin JE, et al. (2023) Is there a benefit for anesthesiologists of adding difficult airway scenarios for learning fiberoptic intubation skills using virtual reality training? A randomized controlled study. PLoS ONE 18(1): e0281016. https://doi.org/10.1371/journal.pone.0281016

About the Authors:

Loic Cailleau

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia and Intensive Care, University Toulouse 3 Paul Sabatier, Toulouse, France

Thomas Geeraerts

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Writing – original draft

Affiliation: Department of Anesthesia and Intensive Care, University Toulouse 3 Paul Sabatier, Toulouse, France

Vincent Minville

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia and Intensive Care, University Toulouse 3 Paul Sabatier, Toulouse, France

Olivier Fourcade

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia and Intensive Care, University Toulouse 3 Paul Sabatier, Toulouse, France

ORICD: https://orcid.org/0000-0002-0101-8208

Thomas Fernandez

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia and Intensive Care, University Clermont Auvergne, Clermont Ferrand, France

Jean Etienne Bazin

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia and Intensive Care, University Clermont Auvergne, Clermont Ferrand, France

Linden Baxter

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia, Oxford University, Oxford, United Kingdom

Vassilis Athanassoglou

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Supervision, Writing – original draft

Affiliation: Department of Anesthesia, Oxford University, Oxford, United Kingdom

Henry Jefferson

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Supervision, Writing – original draft

Affiliation: Department of Anesthesia, Oxford University, Oxford, United Kingdom

Anika Sud

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia, Oxford University, Oxford, United Kingdom

Tim Davies

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Validation, Writing – original draft

Affiliation: Department of Anesthesia, University of Warwick and Coventry, Coventry, United Kingdom

Cyprian Mendonca

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Supervision, Writing – original draft

Affiliation: Department of Anesthesia, University of Warwick and Coventry, Coventry, United Kingdom

Matteo Parotto

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Formal analysis, Methodology, Writing – original draft

Affiliation: Department of Anesthesia, University of Toronto, Toronto, Canada

Matt Kurrek

Contributed equally to this work with: Loic Cailleau, Thomas Geeraerts, Vincent Minville, Olivier Fourcade, Thomas Fernandez, Jean Etienne Bazin, Linden Baxter, Vassilis Athanassoglou, Henry Jefferson, Anika Sud, Tim Davies, Cyprian Mendonca, Matteo Parotto, Matt Kurrek

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Supervision, Validation, Writing – original draft

E-mail: [email protected]

Affiliations: Department of Anesthesia and Intensive Care, University Toulouse 3 Paul Sabatier, Toulouse, France, Department of Anesthesia, University of Toronto, Toronto, Canada

ORICD: https://orcid.org/0000-0003-0819-0171

References

1. Cook TM, MacDougall-Davis SR. Complications and failure of airway management. Br J Anaesth. 2012; 109:i68–i85. pmid:23242753

2. Cook TM, Woodall N, Harper J, Benger J. Major complications of airway management in the UK: results of the Fourth National Audit Project of the Royal College of Anaesthetists and the Difficult Airway Society. Part 2: intensive care and emergency departments. Br J Anaesth. 2011; 106(5):632–42. pmid:21447489

3. McNarry AF, Patel A. The evolution of airway management–new concepts and conflicts with traditional practice. Br J Anaesth. 2017;119(suppl_1):i154–i166. pmid:29161401

4. Wanderer JP, Ehrenfeld JM, Sandberg WS, Epstein RH. The changing scope of difficult airway management. Can J Anesth. 2013; 60(10):1022–4. pmid:23897492

5. Langeron O, Bourgain J-L, Francon D, Amour J, Baillard C, Bouroche G, et al. Difficult intubation and extubation in adult anesthesia. Anesth Crit Care Pain Med. 2018; 37(6):639–651.

6. Xu Z, Ma W, Hester DL, Jiang Y. Anticipated and unanticipated difficult airway management: Curr Opin Anaesthesiol. 2018; 31(1):96–103. pmid:29176376

7. Sklar MC, Detsky ME. Emergent airway management of the critically ill patient: current opinion in critical care. Curr Opin Crit Care. 2019; 25(6):597–604. pmid:31490206

8. Collins SR, Blank RS. Fiberoptic Intubation: An Overview and Update. Respir Care. 2014; 59(6):865–78. pmid:24891196

9. Ahmad I, Onwochei DN, Muldoon S, Keane O, El-Boghdadly K. Airway management research: a systematic review. Anaesthesia. 2019; 74(2):225–36. pmid:30460982

10. Moore A, Schricker T. Awake videolaryngoscopy versus fiberoptic bronchoscopy: Curr Opin Anaesthesiol. 2019; 32(6):764–768. pmid:31356364

11. Kovacs G, Sowers N. Airway Management in Trauma. Emerg Med Clin North Am. 2018; 36(1):61–84. pmid:29132582

12. Wheeler DW, Williams CE, Merry AF. Pulling the plug on ad hoc critical incident training. Br J Anaesth. 2009; 103(2):145–7. pmid:19596758

13. Benhamou D, Roulleau P, Trabold F. Simulation in healthcare: a new teaching method to improve patient safety and quality of care. Réanimation. 2013; 22(S2):383–90.

14. L’Her E, Geeraerts T, Desclefs JP, Benhamou D, Blanie A, Cerf C, et al. Simulation-based teaching in critical care, anesthesia and emergency medicine. Anesth Crit Care Pain Med. 2020; 39(2):311–326.

15. Colt HG, Crawford SW, Galbraith O. Virtual Reality Bronchoscopy Simulation. Chest. 2001;120(4):1333–9.

16. Samuelson ST, Burnett G, Sim AJ, Hofer I, Weinberg AD, Goldberg A, et al. Simulation as a set-up for technical proficiency: can a virtual warm-up improve live fibre-optic intubation? B J Anesth. 2016; 116(3):398–404. pmid:26821699

17. Latif RK, Bautista A, Duan X, Neamtu A, Wu D, Wadhwa A, et al. Teaching basic fiberoptic intubation skills in a simulator: initial learning and skills decay. J Anesth. 2016; 30(1):12–9. pmid:26493397

18. Binstadt E, Donner S, Nelson J, Flottemesch T, Hegarty C. Simulator Training Improves Fiber-optic Intubation Proficiency among Emergency Medicine Residents. Acad Emerg Med. 2008; 15(11):1211–4. pmid:18699826

19. Giglioli S, Boet S, De Gaudio AR, Linden M, Schaeffer R, Bould MD, et al. Self-directed deliberate practice with virtual fiberoptic intubation improves initial skills for anesthesia residents. Minerva Anestesiol. 2012; 78(4):456–61. pmid:22310190

20. Chandra DB, Savoldelli GL, Joo HS, Weiss ID, Naik VN. Fiberoptic Oral Intubation: The Effect of Model Fidelity on Training for Transfer to Patient Care. Anesthesiology. 2008; 109(6):1007–13. pmid:19034097

21. Moorthy K, Smith S, Brown T, Bann S, Darzi A. Evaluation of Virtual Reality Bronchoscopy as a Learning and Assessment Tool. Respiration. 2003; 70(2):195–9. pmid:12740517

22. Martin KM, Larsen PD, Segal R, Marsland CP. Effective Nonanatomical Endoscopy Training Produces Clinical Airway Endoscopy Proficiency: Anesth Analg. 2004; 99(3):938–44. pmid:15333435

23. Gopal M, Skobodzinski AA, Sterbling HM, Rao SR, LaChapelle C, Suzuki K, et al. Bronchoscopy Simulation Training as a Tool in Medical School Education. Ann Thorac Surg. 2018; 106(1):280–286. pmid:29530773

24. Goldmann K, Steinfeldt T. Acquisition of basic fiberoptic intubation skills with a virtual reality airway simulator. J Clin Anesth. 2006; 18(3):173–8. pmid:16731318

25. Marsland CP, Robinson BJ, Chitty CH, Guy BJ. Acquisition and Maintenance of Endoscopic Skills: Developing an endoscopic dexterity training system for anesthesiologists. J Clin Anesth 2002; 14(8):615–9 pmid:12565124

26. Graeser K, Konge L, Kristensen MS, Ulrich AG, Hornbech K, Ringsted C. Airway management in a bronchoscopic simulator based setting: An observational study. Eur J Anaesthesiol. 2014; 31(3):125–30. pmid:24247408

27. Rowe R, Cohen RA. An Evaluation of a Virtual Reality Airway Simulator. Anesth Analg. 2002; 95(1):62–6. pmid:12088944

28. Casso G, Schoettker P, Savoldelli GL, Azzola A, Cassina T. Development and Initial Evaluation of a Novel, Ultraportable, Virtual Reality Bronchoscopy Simulator: The Computer Airway Simulation System. Anesth Analg. 2019; 129(5):1258–64. pmid:29481426

29. Schebesta K, Hüpfl M, Rössler B, Ringl H, Müller MP, Kimberger O. Degrees of Reality: Airway Anatomy of High-fidelity Human Patient Simulators and Airway Trainers. Anesthesiology. 2012; 116(6):1204–9. pmid:22481118

30. Baker PA, Weller JM, Baker MJ, Hounsell GL, Scott J, Gardiner PJ, et al. Evaluating the ORSIM ® simulator for assessment of anaesthetists’ skills in flexible bronchoscopy: aspects of validity and reliability. Br J Anaesth. 2016; 117(S1):i87–i91.

31. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009; 42(2):377–81. pmid:18929686

32. Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O’Neal L, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. 2019; 95:103208. pmid:31078660

33. Smith JE, Jackson APF, Hurdley J, Clifton PJM. Learning curves for fibreoptic nasotracheal intubation when using the endoscopic video camera. Anaesthesia. 1997; 52(2):101–6. pmid:9059089

34. Barré J, Michelet D, Truchot J, Jolivet E, Recanzone T, Stiti S, et al. Virtual Reality Single-Port Sleeve Gastrectomy Training Decreases Physical and Mental Workload in Novice Surgeons: An Exploratory Study. Obes Surg. 2019; 29(4):1309–1316. pmid:30689172

35. Yang Y, Gupta A, Gillett L, Kenig M, Athanassoglou V, Chuan A. Innate visuospatial and psychomotor ability influences early learning of flexible fibreoptic videobronchoscopy. Br J Anaesth. 2019; 123(2):e164–e166. pmid:31202560

36. Louridas M, Quinn LE, Grantcharov TP. Predictive value of background experiences and visual spatial ability testing on laparoscopic baseline performance among residents entering postgraduate surgical training. Surg Endosc. 2016; 30(3):1126–1133. pmid:26092028

37. Kirkpatrick DL. Evaluation of training. In: Training and development handbook: A guide to human resource development. In Craig R. L. (Ed.) (2nd ed); McGraw Hill, New York., p. 301–319. 1976