Electronic learning methods compared to traditional learning methods in anaesthesia education: a systematic review

Abstract

Background

The use of electronic or digital resources has been shown to be effective in improving knowledge retention and skills development through many fields of medical education. Few systematic reviews have studied the effects of electronic learning (‘e-learning’) compared to traditional learning methods in anaesthesia education. We conducted a systematic review of randomised controlled trials and prospective cohort studies assessing learning outcomes from e-learning modalities compared to ‘traditional’, face-to-face or didactic teaching methods.

Methods

Medical databases (MEDLINE, EMBASE, ERIC, Scopus, CENTRAL, Google Scholar) were searched for applicable studies from January 2002 to January 2023 on the 12th of February 2023. Studies comparing an ‘e-learning arm’ against a ‘traditional learning’ arm including anaesthetic doctors of any level within the study population were considered and assessed for inclusion. The systematic review tool Covidence™ was utilised to track studies for inclusion, results were synthesised be each reviewer prior to independent assessment for risk of bias using the Cochrane Risk of Bias assessment tool ‘RoB-2’.

Results

Our search strategy identified 1681 papers for review. Thirteen studies were deemed eligible for inclusion, assessing 572 health practitioners at varying stages of their clinical careers. Four eligible studies showed a statistically significant difference in their primary outcomes favouring e-learning, while one study found e-learning non-inferior to traditional learning. Three studies found a statistically significant change in favour of traditional learning and the remaining five studies did not find a significant difference when comparing e-learning and traditional learning. Qualitative analysis of various secondary outcomes (where applicable) found considerable variation regarding participant preference in favour of both traditional and e-learning models.

Conclusion

E-learning is an important adjunct to traditional learning methods and when undertaken in the appropriate clinical teaching context, outcomes from e-learning programs can outperform those from traditional learning methodologies. However, there is considerable heterogeneity in the research and shows no consistent benefit to either e-learning or traditional learning. Further research is required to explore the most effective teaching contexts and the efficient implementation of different e-learning modalities in anaesthesia.

Registration

This systematic review was registered prospectively with PROSPERO (reference: CRD42023399129).

(https://www.crd.york.ac.uk/PROSPERO/view/CRD42023399129).

Twitter

https://x.com/abcsofa.

Introduction

Technological advancements in video communication, high fidelity simulation and online assessment has led to a revolution in accessibility to electronic learning or ‘e-learning’ globally, and medical education in anaesthesia is no exception [1, 2]. In their pursuit to develop the knowledge, skills and behaviours required to keep a patient safe throughout the perioperative period, an anaesthesia trainee must process a wide array of medical knowledge in real-time and apply this promptly in their clinical work. As such, education delivery for anaesthesia presents a number of unique challenges, such as interhospital variability between formal teaching programs and the uniqueness of the ‘apprenticeship’ model of informal theatre-based learning, contributing to a widely heterogenous experience between trainees [3, 4]. Increased penetration of e-learning in anaesthesia training within a formalised curriculum has emerged as a possible solution for some of the shortcomings of this model, however evaluation must be undertaken to ensure e-learning is sufficient to replace or supplement ‘traditional learning’ modalities currently in use.

An international 2012 consensus statement defined electronic learning or ‘e-learning’ as “an approach to teaching and learning, representing all or part of the educational model applied, that is based on the use of electronic media and devices as tools for improving access to training, communication and interaction and that facilitates the adoption of new ways of understanding and developing learning” [5]. E-learning studies will be appraised in our review if they are completely accessible on a screen-based platform, such as a phone or computer. For the purposes of this review, traditional education includes didactic lectures, face-to-face or simulation-based teaching or practical in-person classes.

E-learning offers many advantages to doctors in training when compared to traditional learning methods, including but not limited to; increased accessibility, flexibility, affordability, interactivity and user satisfaction [6, 7], ease of access, updateable materials and personalised pace of learning [8]. Those utilising e-learning platforms subjectively associate its use with achieving higher exam scores [8], ability to self-monitor knowledge gaps [9], improved knowledge retention from repeat exposure [10], and greater opportunities to practice exam technique [9].

One of the largest criticisms of e-learning however relates to the teaching of practical hands-on skills, which opponents suggest cannot be adequately taught or assessed through online resources alone. Skills such as advanced airway management, point-of-care ultrasound and neuraxial blockade have all been suggested to be inferior amongst those taught via online methods, rather than in-person education [11, 12]. Conversely, E-learning has also been criticised for its heterogeneity in content delivery between providers, and quality control remains difficult to implement; placing the burden on the learner to appraise the content they consume. Such shortcomings are exacerbated by lower quality resources. Despite high-quality resources being available online, an educational review paper of videos on Youtube depicting lumbar puncture and neuraxial blockade techniques demonstrated an incidence of sterility violations of 11%. A further 13% of these videos were assessed as presenting dangerously misleading information [12].

A number of reviews have compared the effectiveness of a variety of e-learning techniques compared to traditional learning methods in medical education [13,14,15]. A systematic review by Shih and colleagues published in 2021 evaluated digital learning modalities using specialised equipment such as virtual reality and haptic simulators [16] and were outside the scope of our search. This review collates recent research to assess the efficacy of e-learning compared to traditional teaching methods and assessed learning outcomes (knowledge or skill acquisition) for medical doctors in anaesthesia. A wide-ranging literature search was conducted and research papers evaluated for inclusion. We hypothesise that the use of e-learning modalities is non-inferior, or results in improved knowledge acquisition or improved proficiency in practical skills, when compared to didactic teaching methods for the same educational content. Assessment is measured by comparing results of quantitative pre- to post-intervention testing between both types of learning programs. The aim of this review is to ask how does e-learning compares to traditional teaching methods in trials assessing learning outcomes (knowledge or skill acquisition) of medical doctors in anaesthesia?

Methods

Search strategy

On the 12th of February 2023, an experienced research librarian conducted the systematic review by searching several medical databases (MEDLINE, EMBASE, ERIC, Scopus, CENTRAL, Google Scholar) using the search strategies outlined in Appendix 1, to identify studies comparing e-learning and traditional learning programs amongst doctors in anaesthesia. The following inclusion and exclusion criteria of eligible studies were registered with PROSPERO:

1. 1)

Any trials that include medical doctors in anaesthesia as study participants, undertaking education in clinical anaesthesia. Studies with no representation of anaesthetic doctors across the cohort, such as those comprised entirely of medical students, intensive care or emergency physicians, nurses or allied health staff were excluded. Anaesthetic doctors did not have to make up the entirety of the patient population to deem a study eligible for inclusion.

2. 2)

Trials comparing e-learning methods with traditional learning methods were included i.e. randomised controlled trials and prospective cohort studies. Due to an intrinsic lack of feasibility for blinding in this area of research, no requirement for participant or assessor blinding was placed on eligible studies.

3. 3)

The e-learning intervention arm must have been able to be completed at any time with a personal device alone, such as a computer or phone. Examples include instructional videos, asynchronous lecture programs, computer-based online simulators, online case-based learning. The use of additional physical resources such as mannequins, phantom limbs, simulators, haptics, online live teaching with feedback or any in-person teaching resulted in exclusion.

4. 4)

The traditional control arm may include didactic lectures, text-based learning, theatre-based teaching or in-person simulation (low or high fidelity, including in-person simulation using electronic technology). The use of any e-learning components throughout the implementation of the control arm, included blended learning models, were excluded.

5. 5)

Studies directly assessing quantitative or objective assessment results (knowledge or practical skill gain) after the e-learning program compared to assessment outcomes after the traditional learning program. Studies assessing only qualitative outcomes such as participant enjoyment were excluded, though studies assessing these domains as a secondary outcome remained eligible if the primary outcome was appropriate.

6. 6)

Studies published from January 2002 to January 2023 were eligible for the study.

7. 7)

Studies not originally published in English were eligible to be screened if an English translation from a medical research translator using the Cochrane database was able to be obtained.

The systematic review tool Covidence™ was utilised to amalgamate identified studies and track those at varying stages of review. 2060 results from database searches were collated and 381 duplicates excluded. Citations from relevant articles were also screened and one appropriate study not identified by the search strategy was added, totalling 1680 articles for analysis. Two authors (LB and ZH) independently screened these articles for inclusion using abstract and title. 18 studies proceeded to full text review. In cases of conflict, studies were reassessed by both reviewers and a consensus opinion sought for inclusion or exclusion. Using this process, no studies proceeded to mediation by an available third party (LA), as a consensus opinion was always able to be reached. Following full text review, 13 studies were selected for inclusion.

Data was collected from each of the included studies including sample size, the nature of the intervention and the control group, the tools used to assess the efficacy of the intervention and the results of each study. Additionally, each study was analysed and classified by its Kirkpatrick score which evaluates the impact of training, with increasing scope of analysis per Kirkpatrick level. Level 1 studies assessed the reaction of the student and their thoughts about the training experience; Level 2 studies assessed the student’s resulting learning and increase in knowledge post-intervention; Level 3 studies assessed the student’s behavioural change and improvement after applying the skills on the job; and Level 4 studies assessed the results or effects that the student’s performance has on the business or health service, or on future clinical outcomes [17].

Results

Study characteristics

As demonstrated in Fig. 1 and 13 studies were included for this review, comprising 572 participants across ten countries. Ten studies were randomised controlled trials (level of evidence 1b), while the remaining three studies were prospective cohort studies (level of evidence 2b). Studies published from 2002 onwards were eligible for inclusion. All but the study by Nyssen et al. was published after 2012, reflecting a trend of increasing interest in e-learning in anaesthesia. As per the listed inclusion criteria, all studies included doctors working in anaesthesia – ranging in experience from residents, trainees, fellows and consultants, while Weber et al. also included ICU medical staff in their research. The studies covered a wide range of topics including obstetric anaesthesia and obstetric cardiac arrest; various procedural skills such as vascular access, point-of-care ultrasound, effective mechanical ventilation strategies, lung isolation, EEG interpretation, and several anaesthetic emergencies such as anaphylaxis and malignant hyperthermia. From the 13 studies included, four assessed knowledge gain through MCQs, five measured practical skill gain through assessor-scored simulation and four evaluated both simulation-based practical skill and knowledge gain.

[IMAGE OMITTED: SEE PDF]

Risk of bias

All studies were assessed for risk of bias independently by both reviewers (LB and ZH) using the Cochrane Risk of Bias assessment [RoB-2] (for randomised studies), and Cochrane Risk Of Bias In Non-randomised Studies, Version 2 assessment [ROBINS-I V2] (for non-randomised studies) to establish the respective risk of bias for various aspects of trial design, author conduct and outcome reporting [18, 19]. Studies by Bensalem-Owen, Nyssen and Vasilopoulos did not use randomisation and were assessed by the ROBINS tool. Papers by Bensalem-Owen and Nyssen were judged at ‘serious’ risk of bias due to a failure to adjust for confounding factors, while the article by Vasilopoulous was judged to be ‘moderate’ risk of bias due to a lack of analysis to estimate the effect of assignment to intervention. All other studies underwent randomisation and were assessed by the RoB tool demonstrated a ‘low risk’ of bias. Of these however, with the exception of Andersson et al., none of the included studies undertook an intention-to-treat or per-protocol analysis, which may have raised concerns regarding bias stemming from deviation from the intended intervention. However, due to the nature of the intervention and the relatively small sample sizes, we believe this poses little to no risk to the results of our review. The risk of bias assessment tables for both randomised and non-randomised studies can be found in Appendix 2 and Appendix 3.

Outcomes

Overall, five studies demonstrated a statistically significant benefit of e-learning compared to traditional learning, while Bensalem-Owen et al. found e-learning non-inferior to traditional methods, in line with their study design. Conversely, three studies significantly favoured traditional learning over e-learning. The remaining five studies reported no difference when comparing learning outcomes between e-learning and traditional learning trial arms. E-learning consistently performed better than traditional learning methods such as didactic and/or face-to-lace learning superior knowledge retention (assessed via testing immediately following intervention, as well as delayed testing of several weeks to months following intervention) and skills acquisition. When traditional learning methods outperformed e-learning in some studies, we found mannequin-based simulation, with or without accompanying coursework, to be the modality most likely to be involved. The results demonstrate that e-learning can reliably improve knowledge and skills performance overall, and can produce comparable or better results in a significant proportion of clinical teaching environments. Authors frequently commented that e-learning modalities were easier to implement and the scope of some studies was reduced due to cost barriers to using mannequin simulators to gather reliable data, suggesting a practical role for e-learning in low resource settings. Table 1 provides further information on the e-learning and traditional learning methods employed, as well as the outcomes assessed.

[IMAGE OMITTED: SEE PDF]

Discussion

The purpose of this review was to evaluate the impact of e-learning versus traditional learning on knowledge and practical skill outcomes of doctors working in anaesthesia, as evaluated by pre- and post-intervention testing. This review suggests that e-learning likely demonstrates outcomes comparable, or non-inferior, to traditional learning in anaesthesia. However, the heterogeneity in study interventions and outcomes make direct comparisons between trials difficult, and more research is required to generate conclusions regarding specific e-learning activities and their comparative efficacy.

This review addresses a gap in the literature by assessing trials and cohort studies comparing the impact of e-learning interventions and traditional teaching methods on education outcomes of doctors undertaking anaesthesia. The e-learning interventions considered within this review are largely accessible to most doctors in developed countries, which is reflected in our inclusion criteria to assess interventions that are ‘able to be completed at any time with a personal device alone, such as a computer or phone.’ A comprehensive systematic review and meta-analysis by Cook and colleagues has been undertaken on medical education previously [33] while a meta-analysis examining technologies used in anaesthesia education was published by Shih and colleagues in 2021 [16]. The breadth of electronic-assisted education in anaesthesia is vast. The ‘digital learning’ in anaesthesia review by Shih et al. appraises educational mediums such as ‘gamification’, virtual reality, augmented reality, computer-assisted units and haptic feedback, rather than screen-based ‘e-learning’ [34]. This review is the first to compare e-learning and traditional learning in anaesthesia.

Shih and colleagues examined digital instruction in anaesthesia with advanced technologies and demonstrated a benefit with these e-learning modalities when compared to traditional teaching methods. The paper by Shih and colleagues analysed 15 papers in total, however only the paper by Sharma and colleagues met criteria for inclusion in this review. This was due to a range of factors: five papers did not include doctors working in anaesthesia, instead comprising medical students, pre-vocational doctors or nurse anaesthetists; two papers were published prior to 2002; five studies used physical simulators or teaching aides and one study used a point-of-care reference tool to represent an e-learning intervention; and finally, two studies lacked an appropriate control group to represent traditional learning. While some interventions such as screen-based simulators or case-based online modules are consistent with the interventions examined in this review, many of the digital instruction methods assessed are reliant on costly and specialised equipment that are inaccessible to most doctors learning remotely in anaesthesia and are thus beyond the scope of this review.

The studies retrieved by this review demonstrate the diversity of e-learning and traditional learning methods in use today. E-learning activities included didactic videos and podcasts, asynchronous lecture programs, computer-based online simulators and online case-based learning. By comparison, traditional learning methods included lectures, tutorials, mannequin-based simulation, low- and high-fidelity simulators and theatre-based teaching. Whilst necessarily representative of the range of methods available to teach anaesthesia online and in-person, a wide range of modalities hence limits the validity of commentary which broadly applies conclusions to all e-learning and traditional learning methods. Drawing conclusions from this review about the most effective types of e-learning and traditional learning methods for different learning outcomes on different topics is equally impractical. Utilising both approaches – both traditional and e-learning modalities, blended learning or ‘flipped classroom’ models help may also enable learners to process theoretical and knowledge-based constructs using e-learning resources at their own pace, prior to engaging in more traditional methods of learning practical skills in the simulated or clinical environment. An alternative flipped classroom model includes reading and offline tasks set by the author, in conjunction or in addition to an online learning portal [35]. A study comparing flipped classroom sessions to traditional lectures for residents in anaesthesia demonstrated an increase in knowledge retention in the flipped classroom group [36].

Analysis of the Kirkpatrick model scores applied across all included studies demonstrates that e-learning modules increase knowledge acquisition and retention, as evidenced by delayed post-intervention testing such as that conducted by Andersson et al., and Edrich et al., both studies classified as Level 2. The volume of studies judged as Kirkpatrick Level 3 further indicates that e-learning modules can not only increase knowledge, but also provide clinically significant and meaningful outcomes on clinical practice, such as proficiency in practical skills or behaviour in real-world scenarios. The lack of studies judged as Kirkpatrick Level 4 indicates a lack of research regarding long term patient outcomes or organisational change, however does not diminish the efficacy of e-learning modules.

All three of the randomised controlled trials that demonstrated a positive benefit of traditional teaching over e-learning utilised high-fidelity simulation/mannequins and as such, taught participants assigned to this intervention in the same environment that they used to assess them [25, 26, 30]. The e-learning group in these trials were not exposed to the simulation environment prior to their simulation assessment. Koens and colleagues describe a form of bias known as same-context advantage, whereby environmental cues facilitate memory recall in trial participants, which could account for some of the benefit demonstrated by the simulation groups in these studies [37]. In the absence of such a bias however, a meta-analysis by Lorello and colleagues found that the use of simulation in anaesthetics, particularly using high fidelity mannequins, does provide superior teaching outcomes compared to a screen- or electronic-based resource [38]. Unfortunately, access to these teaching resources is often limited to tertiary, academic hospitals or university centres and may not be able to be translated into a curriculum-wide intervention. Feasibility studies investigating mobile simulation centres have been conducted however they are unlikely to provide enough coverage for all required participants [39]. Additionally cost-utility analysis has found online simulation training results in no quantitative differences in learning or performance, but can be administered at as little as 29% of the cost of mannequin-based simulation training [40]. Conversely, a meta-analysis into simulation in anaesthesia education actively excluded cost from their analysis, as the same researchers have previously found cost was evaluated in just 2% of all comparative studies on simulation [38, 41]. While the results collected from this review suggest high-fidelity simulation is superior to e-learning, consideration of accessibility to these resources on a broader context must be considered. E-learning still appears to be non-inferior, or indeed in certain circumstances a superior alternative, to other modalities of didactic, traditional learning such as face-to-face lectures or in-person, mannequin simulation.

Two studies from our review by Edrich et al. and Andersson et al. demonstrated that anaesthesia residents that watched a video podcast performed better on a knowledge test than those who attended an in-person didactic lecture [21, 27]. One possible explanation for this difference is that e-learning platforms allow participants to navigate the material at their own pace, enabling doctors to invest more time and focus into areas of difficulty. Similarly, participants who were randomised to watch an ultrasound guided CVC insertion video in their own time, outperformed their colleagues in a simulated assessment of CVC insertion, who were randomised to a time-framed didactic lecture [20].

A primary strength of this review included the high quality of evidence assessed; achieved via rigorous inclusion criteria to appraise randomised controlled trials and prospective cohort studies. Well formulated studies and appropriate study designs improve the validity of conclusions presented in each trial and provide a strong foundation for ongoing research. The quality of each selected study is further evidenced by the relative paucity of bias amongst the included literature as assessed by the Cochrane RoB tool. The inclusion of an intention- to-treat, or similar analysis where applicable, would have further reduced the risk of bias, but the absence of such is unlikely to have generated a significant impact on the results. The exclusion of studies not involving doctors involved in anaesthesia also strengthened this review, limiting the heterogeneity of prior learning on study outcomes. Our study also focussed on mediums of e-learning which could be considered accessible to most learners, such as online videos or case studies. As such, it is hoped that these results can be more easily applied to a number of environments, including low-resource settings, where the availability of high-fidelity simulation and digital instruction methods such as virtual reality or haptic feedback models as examined by Shih and colleagues, is considerably less.

This review was limited by several factors. The power of many trials assessed in this review is limited by small participant cohorts, with only one study including over 55 participants. Similarly, none of the included studies were undertaken across multiple centres limiting the generalisability of this research across a wide range of learning contexts. Hence, many of these trials may thereby have been underpowered to detect a significant difference between e-learning and traditional learning methods, if in fact a true difference were to exist.

Conclusion

E-learning has emerged as a viable adjunct to traditional, didactic learning methods, and this review suggests that in several areas of clinical anaesthesia, it is a superior teaching tool. This review confirms previous research in other fields of medicine that e-learning is an efficacious and accessible alternative to didactic teaching in clinical anaesthesia education. A number of studies similarly found traditional learning superior, and a clear trend between either modality was not identified. Ongoing research is necessary to identify the specific modalities of didactic teaching in anaesthesia that e-learning can complement or replace, and conversely, areas where didactic teaching and simulation remain superior. The volume of production of e-learning resources, their quality and uptake, will only continue to grow over the coming years and it is imperative these resources are utilised to ensure the greatest outcomes for learners. Additional high-powered randomised controlled trials or cohort studies assessing the knowledge, skills and behaviours that are critical to becoming a competent anaesthetist will enable educators to provide evidence-based curriculums and cater their online teaching methods to their learners needs.

Data availability

This systematic review was registered prospectively with PROSPERO (https://www.crd.york.ac.uk/PROSPERO/view/CRD42023399129) and performed in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) and required assessment of extracted data. Data herein is available from the papers listed in the references section in addition to the search protocols listed in the appendix.

Abbreviations

e-learning:

Electronic learning

References

Chu LF, Kurup V. Graduate medical education in anaesthesiology and COVID-19: lessons learned from a global pandemic. Curr Opin Anesthesiol. 2021;34:726–34. https://doi.org/10.1097/ACO.0000000000001065

Cordato DJ, Fatima Shad K, Soubra W, Beran RG. Health research and education during and after the COVID-19 pandemic: an Australian clinician and researcher perspective. Diagnostics. 2023;13. https://doi.org/10.3390/diagnostics13020289

Yamamoto S, Tanaka P, Madsen MV, Macario A. Comparing anesthesiology residency training structure and requirements in seven different countries on three continents. Cureus. 2017;9:e1060–1060. https://doi.org/10.7759/cureus.1060

Jonker G, Manders LA, Marty AP, Kalkman CJ, ten Cate TJ, van Gessel EF, et al. Variations in assessment and certification in postgraduate anaesthesia training: a European survey. Br J Anaesth. 2017;119:1009–14. https://doi.org/10.1093/bja/aex196

Sangrà A, Vlachopoulos D, Cabrera N. Building an inclusive definition of E-Learning: an approach to the conceptual framework. Int Rev Res Open Distrib Learn. 2012;13:145–59. https://doi.org/10.19173/irrodl.v13i2.1161

Arkorful V, Abaidoo N. The role of e-learning, advantages and disadvantages of its adoption in higher education. Int J Educ Res. 2014;2:397–410.

Gray K, Tobin J. Introducing an online community into a clinical education setting: a pilot study of student and staff engagement and outcomes using blended learning. BMC Med Educ. 2010;10:6–6. https://doi.org/10.1186/1472-6920-10-6

Zhang J, Peterson RF, Ozolins IZ. Student approaches for learning in medicine: what does it tell Us about the informal curriculum? BMC Med Educ. 2011;11:87–87. https://doi.org/10.1186/1472-6920-11-87

Wynter L, Burgess A, Kalman E, Heron JE, Bleasel J. Medical students: what educational resources are they using? BMC Med Educ. 2019;19:36–36. https://doi.org/10.1186/s12909-019-1462-9

Larsen DP, Butler AC, Roediger HL III. Test-enhanced learning in medical education. Med Educ. 2008;42:959–66. https://doi.org/10.1111/j.1365-2923.2008.03124.x

Roe D, Simon Carley C, Sherratt. Potential and limitations of e-learning in emergency medicine. Emerg Med J. 2010;27:100. https://doi.org/10.1136/emj.2008.064915

Rössler B, Lahner D, Schebesta K, Chiari A, Plöchl W. Medical information on the internet: quality assessment of lumbar puncture and neuroaxial block techniques on YouTube. Clin Neurol Neurosurg. 2012;114:655–8. https://doi.org/10.1016/j.clineuro.2011.12.048

Liu Q, Peng W, Zhang F, Hu R, Li Y, Yan W. The effectiveness of blended learning in health professions: systematic review and Meta-Analysis. J Med Internet Res. 2016;18:e2. https://doi.org/10.2196/jmir.4807

Nicoll P, MacRury S, van Woerden HC, Smyth K. Evaluation of Technology-Enhanced learning programs for health care professionals: systematic review. J Med Internet Res. 2018;20:e131. https://doi.org/10.2196/jmir.9085

Al Shorbaji N, Atun R, Car J, Majeed A, Wheeler EL, Beck D, et al. eLearning for undergraduate health professional education: a systematic review informing a radical transformation of health workforce development. World Health Organization 2015.

Shih Y-CD, Liu C-C, Chang C-C, Lee Y-W. Effects of digital learning in anaesthesiology: A systematic review and meta-analysis. Eur J Anaesthesiol EJA. 2021;38. https://doi.org/10.1097/EJA.0000000000001262

Praslova LN. The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation. 2025. Available: https://methods.sagepub.com/ency/edvol/sage-encyclopedia-of-educational-research-measurement-evaluation/chpt/fourlevel-evaluation-model

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355. https://doi.org/10.1136/bmj.i4919

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366. https://doi.org/10.1136/bmj.l4898

Bertran S, Boby H, Bertrand PM, Pereira B, Perbet S, Lautrette A. Comparison of video-based learning and lecture-based learning for training of ultrasound-guided central venous catheterization: a randomized controlled trial. 2017;118:628–30. https://doi.org/10.1093/bja/aex059

Edrich T, Stopfkuchen-Evans M, Scheiermann P, Heim M, Chan W, Stone MB, et al. A comparison of Web-Based with traditional Classroom-Based training of lung ultrasound for the exclusion of pneumothorax. Anesth Analg. 2016;123. https://doi.org/10.1213/ANE.0000000000001383

Haskins SC, Feldman D, Fields KG, Kirksey MA, Lien CA, Luu TH et al. Teaching a Point-of-Care Ultrasound Curriculum to Anesthesiology Trainees With Traditional Didactic Lectures or an Online E-Learning Platform: A Pilot Study. J Educ Perioper Med JEPM. 2018;20:E624. Available: http://ovidsp.ovid.com/ovidweb.cgi?T=JS%26PAGE=reference%26D=pmnm6%26NEWS=N%26AN=30510972

Mamoun N, Bottiger B, Wright MC, Klinger R, Grantham A, Stolp B, et al. Effectiveness of Simulation- versus didactic Video-Based learning to teach advanced lung isolation techniques. J Cardiothorac Vasc Anesth. 2022. https://doi.org/10.1053/j.jvca.2022.12.016

Sharma V, Chamos C, Valencia O, Meineri M, Fletcher SN. The impact of internet and simulation-based training on transoesophageal echocardiography learning in anaesthetic trainees: a prospective randomised study. 2013;68:621–7. https://doi.org/10.1111/anae.12261

Spadaro S, Karbing DS, Fogagnolo A, Ragazzi R, Mojoli F, Astolfi L et al. Simulation training for residents focused on mechanical ventilation: a randomized trial using Mannequin-Based versus Computer-Based simulation. 2017;12:349–55. https://doi.org/10.1097/SIH.0000000000000249

Weber U, Zapletal B, Base E, Hambrusch M, Ristl R, Mora B. Resident performance in basic perioperative transesophageal echocardiography: comparing 3 teaching methods in a randomized controlled trial. 2019;98:e17072. https://doi.org/10.1097/MD.0000000000017072

Andersson ML, Duch P, Bessmann EL, Lundstrøm LH, Ekelund K. Preparing for obstetric anaesthesia—An educational randomised controlled trial comparing e-learning to written course material. Acta Anaesthesiol Scand. 2023;67:36–43. https://doi.org/10.1111/aas.14148

Bensalem-Owen M, Chau DF, Sardam SC, Fahy BG. Education research: evaluating the use of podcasting for residents during EEG instruction A pilot study. Neurology. 2011;77:e42–4. https://doi.org/10.1212/WNL.0b013e31822b0017

Hards A, Davies S, Salman A, Erik-Soussi M, Balki M. Management of simulated maternal cardiac arrest by residents: didactic teaching versus electronic learning. 2012;59:852–60. https://doi.org/10.1007/s12630-012-9752-2

Mejia V, Gonzalez C, Delfino AE, Altermatt FR, Corvetto MA. Acquiring skills in malignant hyperthermia crisis management: comparison of high-fidelity simulation versus computer‐based case study. 2018;68:292–8. https://doi.org/10.1016/j.bjan.2018.01.005

Nyssen AS, Larbuisson R, Janssens M, Pendeville P, Mayne A. A comparison of the training value of two types of anesthesia simulators: computer screen-based and mannequin-based simulators. Anesth Analg. 2002;94:1560–5. https://doi.org/10.1213/00000539-200206000-00035

Vasilopoulos T, Chau DF, Bensalem-Owen M, Cibula JE, Fahy BG. Prior podcast experience moderates efficacy of electroencephalography. Anesth Analg. 2015;120:S151. https://doi.org/10.1213/01.ane.0000470325.07465.0f

Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-Based learning in the health professions: A Meta-analysis. JAMA. 2008;300:1181–96. https://doi.org/10.1001/jama.300.10.1181

Basak SK, Wotto M, Bélanger P, E-learning. M-learning and D-learning: conceptual definition and comparative analysis. E-Learn Digit Media. 2018;15:191–216. https://doi.org/10.1177/2042753018785180

Marchalot A, Dureuil B, Veber B, Fellahi J-L, Hanouz J-L, Dupont H, et al. Effectiveness of a blended learning course and flipped classroom in first year anaesthesia training. Anaesth Crit Care Pain Med. 2018;37:411–5. https://doi.org/10.1016/j.accpm.2017.10.008

Martinelli SM, Chen F, DiLorenzo AN, Mayer DC, Fairbanks S, Moran K, et al. Results of a flipped classroom teaching approach in anesthesiology residents. J Grad Med Educ. 2017;9:485–90. https://doi.org/10.4300/JGME-D-17-00128.1

Koens F, Mann KV, Custers EJFM, Ten Cate OTJ. Analysing the concept of context in medical education. Med Educ. 2005;39:1243–9. https://doi.org/10.1111/j.1365-2929.2005.02338.x

Lorello GR, Cook DA, Johnson RL, Brydges R. Simulation-based training in anaesthesiology: a systematic review and meta-analysis. BJA Br J Anaesth. 2014;112:231–45. https://doi.org/10.1093/bja/aet414

Premkumar K, Umaefulam V, O’Brien J. Mobile medical simulation for rural anesthesia providers: A feasibility study. Can Med Educ J. 2020;11:e60–71. https://doi.org/10.36834/cmej.69572

Haerling KA. Cost-Utility analysis of virtual and Mannequin-Based simulation. Simul Healthc. 2018;13. https://doi.org/10.1097/SIH.0000000000000280

Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA, Cost. The missing outcome in simulation-based medical education research: A systematic review. Surgery. 2013;153:160–76. https://doi.org/10.1016/j.surg.2012.06.025