Effectiveness of non-technical skills training in intensive care units: a systematic review

Abstract

Background

Non-technical skills (NTS) are social, cognitive and psychological skills that are related to safety and efficiency in the healthcare process. These skills have an important role in the intensive care once are closely related to the occurrence of safety incidents. This study aims to examine the effectiveness of NTS training programs in intensive care units using Kirkpatrick’s evaluation model, encompassing reaction, learning, behaviors, and organizational results.

Method

We conduct a systematic review of original articles in the following databases: Pubmed (Medline), Scopus, Web of Science, Science Direct, CINHAL (EBSCOhost), EMBASE (Elsevier), and PsycINFO. We include randomized clinical trials, quasi-experimental studies, and those employing a before-and-after design. Two pairs of independent reviewers selected and assessed the studies, with any discrepancies resolved by a fifth reviewer. Risk of bias and quality assessments were performed using Cochrane Risk of Bias 2, Risk of Bias in Non- randomized Studies of Interventions and Quality Assessment Tool developed by the National Heart, Lung, and Blood Institute.

Results

Fourteen studies were included, with 11 employing a before-and-after design. Overall study quality was assessed as moderate or low. Simulation training was the main teaching strategy of the training programs. In the reaction dimension, participants (502 participants) generally perceived the training programs as beneficial, despite variations in teaching strategies. Only two studies evaluated the impact on the learning dimension (383 participants), with small improvements. A positive correlation was identified between the adoption of NTS training programs and improved teamwork (413 participants and 183 direct observations), situational awareness (444 participants and 183 direct observations), leadership (183 participants and 155 direct observations), and communication (183 participants and 1141 direct observations). Organizational results showed no significant changes in patient safety culture, length of stay or mortality rates (15776 patients). However, there was a reduction in response time to cardiopulmonary resuscitation and lower adjusted mortality and complications rates (7721 patients).

Conclusion

The implementation of NTS training programs in intensive care units may improve healthcare professionals’ performance, especially in teamwork, situational awareness and communication. However, it is necessary to conduct more robust experimental studies to establish causality and explore broader impacts on patient safety and costs.

Registration

CRD42021244769.

Background

Non-technical skills (NTS) are social, cognitive and psychological skills that play a fundamental role in the efficiency and safety of human operations. These skills encompass situational awareness, teamwork, task management, and decision making [1]. Traditionally incorporated into the training of professionals in high-reliability organizations such as aviation and nuclear energy, NTS have been introduced into healthcare over the past two decades with the aim of improving patient safety [1,2,3,4].

Studies indicate that intensive care units (ICUs) are particularly vulnerable places to safety incidents in healthcare services. From 19 to 62% of admissions record at least one safety incident [5,6,7,8,9], with deficiencies in NTS, such as teamwork and communication, identified as the root cause of up to 50% of these incidents [10,11,12].

Non-technical skills training programs are associated with significant improvements in teamwork performance, communication and patient safety culture, resulting in a reduction in safety incidents and even an improvement in clinical performance in several areas of healthcare services [13,14,15,16,17]. However, despite the importance of NTS for patient safety in ICUs, most review studies focus on surgical environments [18,19,20,21]. Therefore, this study proposes to evaluate, through a systematic literature review, the effectiveness of implementing non-technical skills training programs in ICUs.

Method

Protocol and registration

This systematic review study was registered in the International Prospective Register of Systematic Reviews (PROSPERO) (CRD42021244769) with a research protocol described in a previous publication [22]. The report of this review followed the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta- Analyses (PRISMA) [23], and the methodology was elaborated based on the recommendations of the Cochrane Handbook for Systematic Reviews of Interventions [24].

Eligibility criteria

We included randomized clinical trials (RCT), quasi-experimental with control group and before-and-after studies without a control group conducted with healthcare professionals working in ICUs. We considered studies that reported interventions focused on non-technical skills (NTS) training, which include components such as communication, teamwork, decision-making and leadership. These training programs must have been implemented with intensive care unit workers and may have used a variety of delivery methods, such as simulation-based training, workshops or blended learning approaches.

We considered the outcomes of the studies according to the four levels of Kirkpatrick’s scale [25], which included: reaction – the participant’s response to the training; learning – acquisition of knowledge, skills and attitudes after training; behavior – changes in participants’ behavior in real or controlled care environments, and; results – considered measurable impacts of the training program at the organizational and health level.

We excluded studies: in which the intervention was applied only to leaders and managers; that presented an incomplete description of the training program and evaluation methods; reported that the training program was within the scope of a broad quality improvement strategy with multiple interventions; or used educational strategies based on coaching or mentoring. Furthermore, observational, pilot, validation and qualitative studies, as well as theses, dissertations, case studies, annals of scientific events and other literature reviews were also excluded.

Search strategy

A summary of evidence from original studies was prepared using the following databases: Pubmed (Medline), Scopus, Web of Science, Science Direct, CINHAL (EBSCOhost), EMBASE (Elsevier) and PsycINFO. The search strategy combined terms present in the Medical Subject Headings (MeSH), Embase Subject Headings (EMTREE), as well as non-indexed terms, based on the Anaesthetists' Non- Technical Skills System (ANTS) [3, 26] The search equation for each database can be found as an Additional File (Additional File 1).

Furthermore, we conducted a manual search in the references of the papers included to identify relevant studies that were not reached by the main search. The retrieval of articles from the databases was completed on January 22, 2025. There were no restrictions on language or year of publication.

Study selection

After recovering the documents in all selected databases, a review was carried out to identify duplicates, which were properly removed. Four reviewers (IDSFP, INMB, RMPA and JLAS) worked in pairs who and independently selected the studies based on reading the title and abstract. Likewise, the selected studies had their full text evaluated to ensure compliance with the eligibility criteria. All reviewers were previously trained to select studies and in case of disagreement, a fifth reviewer (ANSM) was consulted. We used the Rayyan® software to perform the study selection and duplicate identification. Duplicates were removed manually by the main researcher (IDSFP).

Data collection

Four reviewers (IDSFP, INMB, RMPA and JLAS) worked in pairs and independently extracted data from the selected studies. For this purpose, we use an adapted version of Data Extraction and Assessment Form proposed by Cochrane Effective Practice and Organization of Care (EPOC) [27]. The data collected included identification of studies (main author, year of publication, study design, number and type of ICU), characteristics of the participants (sample size, age, sex and profession), description of the intervention (duration, follow-up, training content, teaching strategies, facilitators/instructors, co-interventions and formation of implementation committees), evaluation methods (questionnaires, observation methods) and outcomes (at the level of reaction, learning, behavior and organizational results).

Analysis of the risk of bias and methodological quality of studies

Two reviewers (IDSFP and ANSM) independently assessed the risk of bias in the selected studies. Disagreements in the assessment were resolved by a third reviewer (GP). Cochrane Risk of Bias 2 (RoB 2) was used for randomized clinical trials [28, 29]. For quasi-experimental studies with a control group, the Risk of Bias in Non- randomized Studies of Interventions (ROBINS-I) [30], while studies that tested interventions with a before-and-after design, without a control group, were evaluated using the Quality Assessment Tool developed by the National Heart, Lung, and Blood Institute [31]. The degree of agreement between reviewers for assessing the quality of studies was assessed using the Kappa index.

Data synthesis

The data were synthesized in a narrative form to produce summary tables in accordance with the data collection plan. The results of continuous outcomes were summarized using mean differences and effect size. We calculate the relative percentages of changes for each outcome. The p value was also considered [32]. We did not perform a meta-analysis of the included studies considering the high methodological heterogeneity and the low availability of controlled studies to produce a proper comparison [24].

Nevertheless, two reviewers (IDSFP and ANSM) independently assessed the certainty of the evidence (high, moderate, low and very low) using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) system with five factors (risk of bias, consistency of effect, imprecision, indirectness and publication bias) through the GRADEpro software [33]. We performed a narrative summary of findings considering the absence of a single estimate of effect [34].

Results

Study selection

The search in the databases, after removing duplicates, returned 3466 documents of which 36 had the full text evaluated. Following the selection process, 14 articles were included in the systematic review. Figure 1 describes the selection process according to the PRISMA flowchart.

[IMAGE OMITTED: SEE PDF]

Characteristics of studies and participants

Among the 14 articles, 12 studies [35,36,37,38,39,40,41,42,43,44,45,46] had a before-and-after design without a control group. Only one randomized controlled clinical trial study [47], and one controlled quasi-experimental study [48] were identified.

The studies encompassed 21 ICUs, with 11 dedicated to neonatal/pediatric patients, involving a total of 2,103 professionals (average of 150 professionals) in at least more than one professional category, with exception of Anderson et al [36]. and Bos-Boon et al. [37]., where interventions were exclusive to nurses.

Only four studies [41, 42, 46, 48] reported patient-related outcomes, with characteristics of the participants presented in three studies [41, 46, 48]. The studies involved 15,272 patients, predominantly men over the age of 60. Detailed characteristics are provided in the Additional File 2.

Characteristics of the interventions

Short-term interventions (averaging 9 h) that can be completed within a single working day were more frequent. The follow-up period ranged from two months to two years across eight studies, however selectively applied to specific outcomes. Additional File 3 provides details of the reported interventions.

Common training themes included teamwork elements (e.g., peer support, workload distribution, role clarity) in all 14 studies [35,36,37,38,39,40,41,42,43,44,45,46,47,48], situational awareness or its elements (assessment global and situational monitoring) in 12 studies [35,36,37,38,39,40,41,42,43, 45,46,47] and communication also in 10 studies [37,38,39,40,41,42,43,44,45, 48]. Other themes such as leadership (n = 9) [36, 38,39,40,41,42,43, 45, 48], decision making (n = 5) [36, 37, 41, 43, 48], stress management (n = 3) [41, 43, 48] and coping with fatigue (n = 1) [41], were less frequent.

Three studies integrated NTS training with broader topics like human factors, risk management, human errors and safety in complex organizations [43, 47, 48]. Another three studies linked NTS training with patient safety protocols, extracorporeal membrane oxygenation (ECMO) emergencies, and cardiorespiratory arrests [35, 36, 40].

Simulation training emerged as the predominant teaching strategy, often combined with lectures, game play, online modules, interactive activities, and the discussion of clinical cases in text or video format. Among the controlled studies, Jonsson et al. [47] carried out simulation sessions with briefing and debriefing before and after the intervention with the control group, however without focusing on the NTS of interest. Kemper et al. [48] did not carry out any intervention with the control group.

Facilitators/instructors were primarily professionals from the ICU or hospital education sector, except in the studies by Kemper et al. [48] and Haerkens et al. [41], where external companies provided the training. Eight studies [36,37,38,39, 41, 42, 44, 48] reported the support of implementation committees, mostly formed by ICU and hospital professionals.

Co-interventions were reported in five studies [38, 41, 42, 44, 48], employing various strategies, including protocol adoption, participation in team meetings by implementation committees, lectures, reinforcement simulations, and behavioral strategies for team recognition.

Risk of bias analysis

The study by Jonsson et al. [47], evaluated using RoB2, had the general risk of bias classified as “some concerns”. Still, most outcomes were assessed independently and blinded, which presents a low risk of bias. The study by Kemper et al. [48], was considered to have a moderate risk of bias according to ROBINS-I, due to measurement methods and missing data.

For before-and-after studies without a control group, nine were considered as fair [37, 38, 40,41,42,43,44,45,46] and three as of poor quality [35, 36, 39]. Quality concerns envolved participant recruitment clarity, blinding of outcome evaluators, follow-up losses, and the absence of time series designs for statistical robustness. Detailed assessments are provided in the Additional File 5. The kappa index between researchers showed a moderate result (0.639).

Effects of interventions

The results were classified following the Kirkpatrick levels of assessment [25]. We presented the effects of the interventions in a summary of findings table (Table 1), except for the reaction to the training, considering that this result has no comparison and was only assessed after the intervention. The detailed results of each study are described in the Additional File 4.

[IMAGE OMITTED: SEE PDF]

Participant reactions to training were reported in three studies [35, 40, 48] indicating positive perceptions despite variations in assessment instruments. Still, in the three-month follow-up carried out by Kemper et al. [48]., the professionals’ perception was that the training program had only a reasonable influence on performance.

Two studies assessed participant learning on a set of NTS and situational awareness. In the study of Sawyer et al. [45], learning of teamwork, leadership, situational monitoring, mutual support and communication were assessesed using a TeamSTEPPS® tool. Overall learning scores slightly increased, but we consider the certainty of evidence very low due to imprecision, considering the sample size. In the study of Kemper et al. [48], no significant changes were found in situational awareness learning, with a moderate certainty of evidence due to a missing data higher than 20%. Still, the baseline scores for both intervention and control group were already high [45].

All included studies, except for Murphy et al. [44], evaluated the impact of training on professional behaviors. Situational awareness behaviors were evaluated in six studies. Four of them showed increased scores using self-reported instruments [38] and external observations [39, 42, 46], however with a low or very low certainty of evidence. The studies of Jonsson et al. [47]. and Kemper et al. [48] found no significant differences in self-reported situational awareness assessments after training, both with a moderate level of certainty.

Ten studies assessed teamwork behaviors. In nine of them, which had a before and after design, NTS training led to an increase in teamwork behavior, both in self-assessments [35, 37, 38, 40] and through direct observations tools [36, 39, 42, 45, 46], however with a low certainty of evidence. In the RCT by Jonsson et al. [47], there was little difference in teamwork behavior between control and intervention group after the training, with a moderate certainty of evidence.

Anderson et al. [36]. assessed situational awareness, leadership, communication, and teamwork cumulatively, registering a significant increase. We classified the combined assessment of this study under teamwork behavior.

In five before-and-after studies, communcation improved signicantly in self-assessments [38, 40] and through direct observation [39, 42, 45]. Even so, the improvement varieted considerably among the studies, which reduces the certainty of evidence due to imprecision. Kemper et al. [48] was the only controlled study to assess communication and noted a reduction in oral communications in the intervention group, however without statistical significance. Certainty of evidence in observed communication for both desings was downgraded due to some concerns in risk of bias in the assessment.

Six studies evaluated leadership behaviors. Three of them reported significant improvements, primarily using external observation [39, 42, 45]. In self-reported measurements, Brodsky et al. [38] reported a small improvement, but not statiscally significant. Still, Figueroa et al. [40]. reported an significant increase in leadership scores, but in a smaller sample. Jonsson et al., in a RCT, did not found any significantly difference after the intervention. The certainty of evidence was very low to moderate, also motivated by the sample size and risk of bias in the assessment.

Colman et al. [39] and Mileder et al. [46] assessed decision-making behaviors, both with a similar number of direct observations, in before-and-after designs. Only Colman et al. [39] found a significant improvement to this behavior and overall certainty of evidence was very low.

Six studies [38, 41, 42, 44, 46, 48] evaluated organizational results, which we organized in three dimensions: patient safety culture – which consider also error culture and safety climate; satisfaction with work – that also consider stress, relation within the staff and general conditions; and patient outcomes.

Patient safety culture was mostly assessed using the Hospital Survey on Patient Safety Culture developed by the Agency for Healthcare Research and Quality (AHRQ). Mayer et al. [42]. and Murphy et al. [44]. reported improvements in some dimensions of the instrument, even after two years. The teamwork climate was the dimension that did not significantly improve in both studies. Kemper et al. [48]. only found a significant improvement in the dimensions of perception of adequate staff and managers’ actions to improve patient safety. Kemper et al. [48]. also have evaluated error culture and found a significant improvement in the intervention group.

Meurling et al. [43]. and Haerkens et al. [41]. evaluated patient safety climate and found significant increase after the intervention, both using the Safety Attitude Questionnaire. Evidence certainly was classified from low to moderate.

Six studies have shown that NTS training may improve staff satisfaction with work, elevating general satisfaction and reducing stress [38, 41,42,43,44, 48], but the certainty of evidence was considered very low for most of them [38, 41,42,43,44].

Four studies evaluated patient outcomes. Kemper et al. [48] and Haerkens et al. [41]. used ICU indicators from the National Intensive Care Evaluation (NICE) finding no significant differences in length of stay and overall mortality. Haerkens et al. [41]. also found no difference in readmission within 24 h. Mileder et al. [46]. was the only study that evaluted neonatal patients, still with a small sample. They also did not found any difference after the intervention in mortality rate or length of stay. Despite the differences in the certainty of evidence, the overall conclusion is that NTS training may result in little or no difference for those indicators.

Haerkens et al. [41]. observed a significant reduction in complication rates, cardiorespiratory arrest and mortality rates adjusted for risk factors, such as patients’ clinical conditions with a low certainty of evidence. Mayer et al. [42] noted a non-significant reduction in healthcare-related infections with a very low certainty, especially because the absence of information about the patients to grade imprecision.

Regarding process indicators, Haerkens et al. [41] recorded a significant increase in cardiopulmonary resuscitation (CPR) success, while Mayer et al. [42] observed a significant reduction in the time to place patients on ECMO, but with a low and very low certainty, respectively.

Discussion

This study aimed to evalute the effectiveness of implementing non-technical skills training programs with ICU professionals. In the dimension of reaction to the training, participants perceived the intervention as valuable for professional practice, even with the utilization of diverse teaching strategies. The learning dimensions remain underexplored and yield inconclusive results.

The findings of this systematic review indicate a relationship between the adoption of NTS training programs among ICUs professionals and the improvement of behaviors including teamwork, situational awareness, leadership and communication, but with varying levels of evidence certainty.

Improvements in organizational outcomes linked to the intervention were associated with process indicators such as CPR success, time to place patients on ECMO, reduction in complication rates for patients, and increases in patient safety culture and staff satisfaction with work. However, length of stay and mortality rate did not exhibit significant changes. It is noteworthy that the evidence from these studies carries substantial methodological limitations.

A predominance of studies in ICUs for neonatal/pediatric patients was identified. This could be explained by the higher incidence of safety incidents in these care contexts compared to adult ICUs and the increased vulnerability of neonatal/pediatric patients, intensifying the consequences of errors [5, 49].

However, only two study [42, 46] reported outcomes related to neonatal/pediatric patients, utilizing secondary data with limited information about the sample composition. The absence of evaluating the training’s impact on outcomes for neonatal/pediatric patients constrains the interpretation of the intervention’s benefits for this population.

The duration of training programs was generally short, a common characteristic in this intervention type, particularly in the workplace [50]. Short training programs without a schedule of reinforcement may not produce the desired results, considering that knowledge and behaviors decline over time [51]. Deliberate practice with repetitive sessions are fundamental to produce better learning outcomes on long-term level [50, 52, 53].

Regarding the content of the training programs, teamwork was the most targeted NTS skill, followed by leadership and communication. Intensive care demands processes executed by professionals from various levels and areas in an interdependent manner [54]. Given the volatile nature of team formation due to shift work in patient care, training these skills becomes particularly important at an individual level to prepare professionals for effective collaboration within any team [1, 55].

Among the cognitive NTS, situational awareness was most frequently addressed. In ICU environments, professionals cope with a high volume of information daily, whether through communication with colleagues, family members, and patients, or from equipment data, procedure records, and exams [56]. Situational awareness is critical, involving the mental modeling of perception and interpretation of surrounding information to anticipate future situations [57]. Failures in situational awareness can lead to cognitive lapses, impacting decisions in tasks with high risks for the patient, underscoring its importance in this context [58].

Some studies integrated elements such as procedures and protocols into NTS training program content. While uncommon, the integration of technical and non-technical skills is a desirable, as the literature suggest a positive relationship between these skills, directly influencing professional performance [16, 17, 21, 59, 60].

Simulation training stands out as a prominent teaching strategy in most studies. Salas and Cannon-Bowers [61] argue that simulation-based training is the most effective for teaching NTS, promoting authentic use of communication, leadership, and teamwork techniques.

Nonetheless, many training programs employed more traditional teaching strategies, such as lectures, case discussions and videos combined with simulation training. Miller [62], propose a hierarchical path for skills acquisition, where the participant initially acquires knowledge (knows) followed by competence (knows how to do it), performance (shows how to do it) and action (does it). Thus, it is possible to understand that training programs for NTS in ICUs adopt multicomponent teaching strategies to facilitate knowledge acquisition in both cognitive and psychomotor learning domains [53, 63].

Regarding intervention development, most initiatives to implement training programs were internal, conducted by the hospital staff, with ICU professionals as facilitators/instructors. Internal initiatives, involving professionals and stakeholders, enhance the likelihood of success by leveraging knowledge of the unit’s context, barriers, and enablers [64, 65]. However, studies indicate the importance of the facilitator’s/instructor’s experience in learning, suggesting that professionals with little experience may not be as effective, particularly in practical contexts like simulations [65, 66]. This aspect needs further exploration in the context of NTS training in ICUs.

Regarding study outcomes, in the reaction dimension, there was a positive perception about the training, consistent with existing literature [15, 18, 67, 68]. However, none of the studies evaluated participants’ reactions to the teaching strategies used, which would be essential to identify the most attractive strategies for professionals and suggest potential changes in the training program structure [69].

The learning dimension, directly linked to the effectiveness of teaching strategies, has been minimally explored in studies, presenting varied results for different NTS. This limitation in evaluating the learning dimension aligns with review studies on NTS training programs in other care settings, potentially contributing to the indication of limited effects for some interventions due to the uncertainty about participants’ actual learning [20, 67].

The positive learning improvement results found by Sawyer et al. [45], for instance, may be attributed to the adoption of simulation in the curriculum, in contrast to Kemper et al. [48], classroom-based approach, which reported nonsignificant results.

The studies primarily focused on evaluating the impact of training programs on professionals’ behavior. The results on behavior were mostly based on self-reported research instruments, which pose a higher risk of bias compared to objective hetero-evaluation. This bias may be influenced by the Dunning-Kruger effect, a cognitive bias that causes individuals to overestimate their abilities or knowledge, or leads those who are skilled in a particular area to underestimate their abilities [70,71,72].

Nevertheless, most studies reported enhanced performance in teamwork, situational awareness, leadership, and communication, both self-reported and through direct observation. Despite methodological variability, training in these skills appears associated with improved performance by ICU professionals [14, 15, 18, 67].

In the organizational results dimension, effects on patient safety culture varied among studies. As safety culture is a collective and structural product, contextual factors inherent to each service studied may influence this mediation, explaining diverse results [73, 74].

Concerning patient-related outcome indicators, studies did not find significant differences in general mortality and length of stay in the ICU, wich is consistent with the literature [15, 68, 75, 76]. The crude mortality rate at the hospital level might not be the most suitable measure for evaluating the effect of interventions like NTS training, given its dependence on both care and the patient’s clinical conditions [77].

Overall, the indicators used to measure organizational results of NTS training programs in ICUs may have not capture all proposed improvements. From the perspective that training would enhance care performance and safety, process indicators like adherence to safety procedures and outcome indicators such as the incidence of adverse events or near misses would be more appropriate [48, 75].

Concerning methodological quality, a substantial number of before-and-after studies compromise the interpretation of cause-effect relationships in interventions. Despite this limitation, we included this design due to its frequent adoption in evaluating training programs for healthcare professionals, providing insights into evidence produced in a real work context [78,79,80].

However, an important methodological issue was the low adoption of interrupted time series evaluations, a valid alternative for mitigating time-related confounding and enhancing findings’ robustness [78, 79].

Loss to follow-up was also a common issue in the analysis of outcomes. Many studies used extended follow-up times for some outcomes, and voluntary responses may lead to selective data loss, compromising internal validity. Participants who were more committed to the intervention might be more motivated to participate in follow-up [81].

Therefore, we believed that cluster randomised clinical trials may help to draw more reliable conclusions about the effectivness of training interventions, providing a higher level of certainty to the evidence. However, this will required a much larger sample size [82].

Behavior assessment, for example, may be feasible and still providing evidence with a good level of certainty, using a controlled enviroment, such as simulation training, which allows for video recording and independent, blinded assessment, as demonstrated in some studies [36, 47].

Limitations

This review encountered significant heterogeneity between included studies, including differences in study designs, intervention designs, and outcome measurement, preventing the execution of a meta-analysis. This heterogeneity also complicates the comparison of study results in identifying interventions with superior performance. Conducting a meta-analysis under these condtions could mislead the reader about the effect of an intervention. Moreover, our decision to exclude other forms of publication, such as pilot studies and conference abstracts, may have led to the omission of smaller intervention studies.

Nevertheless, this study offers an overview of the literature on NTS training programs for ICU professionals, a context insufficiently explored by systematic reviews. The exploration of training programs encompassing a set of NTS present in curricula distinguishes this study from others focusing on developing specific skills [50, 83, 84]. The results presented provide valuable insights for health professionals, managers, and researchers seeking to implement similar interventions.

Conclusion

The implementation of NTS training programs may enhanced performance of ICU professionals in teamwork, situational awareness, leadership, and communication. While professionals find these training programs beneficial for their practice, their impact on learning outcomes remains limited. The positive effects on organizational outcomes are related to process indicators, reduced complications for patients and patient safety culture indicators. However, length of stay and general mortality rates did not exhibit significant changes.

In the ICU setting, training professionals to enhance NTS appears to be a valid component of a comprehensive training curriculum for healthcare professionals aiming to improve patient safety. However, the need for experimental studies with improved methodological quality is evident to establish, with greater certainty and scientific precision, the causal relationship between the intervention and the observed outcomes.

Future research could explore the impact of training on indicators of process and results related to patient safety and professional performance, such as incidence of falls, success in complex procedures as in intubation and PCR response, quality of handoffs, and incidence of nearmisses. Additionally, studies incorporating cost indicators, including return on investment, are essential, particularly in contexts where training is outsourced or requires investments in professional trainer education or the establishment of simulation facilities.

Data availability

The datasets supporting the conclusions of this article are included within the article and its additional files.

Abbreviations

NTS:

Non-technical Skills

ICU:

Intensive care unit

PROSPERO:

International prospective register of systematic reviews

PRISMA:

Preferred reporting items for systematic reviews and meta-analyses

MeSH:

Medical subject headings

EMTREE:

Embase subject headings

ANTS:

Anaesthetists' non-technical skills system

RCT:

Randomized clinical trialsTrial

EPOC:

Cochrane effective practice and organization of care

RoB 2:

Cochrane risk of bias 2

ROBINS:

I-risk of bias in non-randomized studies of interventions

ECMO:

Extracorporeal membrane oxygenation

AHRQ:

Agency for healthcare research and quality

NICE:

National intensive care evaluation

CPR:

Cardiopulmonary resuscitation

References

Flin R, O’Connor P, Crichton M. Safety at the Sharp end: A guide to Non-Technical skills. 1st ed. London: CRC; 2017.

Scott J, Revera Morales D, Mcritchie A, et al. Non-technical skills and health care provision in low- and middle-income countries: A systematic review. Med Educ. 2016;50:441–55. https://doi.org/10.1111/medu.12939

Flin R. Identifying and training non-technical skills for teams in acute medicine. Qual Saf Health Care. 2004;13:i80–4. https://doi.org/10.1136/qshc.2004.009993

Crichton MT, Flin R. Identifying and training non-technical skills of nuclear emergency response teams. Ann Nucl Energy. 2004;31:1317–30. https://doi.org/10.1016/j.anucene.2004.03.011

Agarwal S, Classen D, Larsen G, et al. Prevalence of adverse events in pediatric intensive care units in the united States. Pediatr Crit Care Med. 2010;11:568–78. https://doi.org/10.1097/PCC.0b013e3181d8e405

Amaral ACKB, McDonald A, Coburn NG, et al. Expanding the scope of critical care rapid response teams: A feasible approach to identify adverse events. A prospective observational cohort. BMJ Qual Saf. 2015;24:764–8. https://doi.org/10.1136/bmjqs-2014-003833

Forster AJ, Kyeremanteng K, Hooper J, et al. The impact of adverse events in the intensive care unit on hospital mortality and length of stay. BMC Health Serv Res. 2008;8:1–8. https://doi.org/10.1186/1472-6963-8-259

Merino P, Álvarez J, Martín MC, et al. Adverse events in Spanish intensive care units: the SYREC study. Int J Qual Health Care. 2012;24:105–13. https://doi.org/10.1093/intqhc/mzr083

Panagioti M, Khan K, Keers RN, et al. Prevalence, severity, and nature of preventable patient harm across medical care settings: systematic review and meta-analysis. BMJ. 2019;366:l4185. https://doi.org/10.1136/bmj.l4185

Corwin GS, Mills PD, Shanawani H, Hemphill RR. Root cause analysis of ICU adverse events in the veterans health administration. Jt Comm J Qual Patient Saf. 2017;43:580–90. https://doi.org/10.1016/j.jcjq.2017.04.009

Pronovost PJ, Thompson DA, Holzmueller CG, et al. Toward learning from patient safety reporting systems. J Crit Care. 2006;21:305–15. https://doi.org/10.1016/j.jcrc.2006.07.001

Reader T, Flin R, Lauche K, Cuthbertson BH. Non-technical skills in the intensive care unit. Br J Anaesth. 2006;96:551–9. https://doi.org/10.1093/bja/ael067

Schmutz J, Manser T. Do team processes really have an effect on clinical performance? A systematic literature review. Br J Anaesth. 2013;110:529–44. https://doi.org/10.1093/bja/aes513

Weaver SJ, Dy SM, Rosen MA. Team-training in healthcare: a narrative synthesis of the literature. https://doi.org/10.1136/bmjqs

O’Dea A, O’Connor P, Keogh I. A Meta-Analysis of the effectiveness of crew resource management training in acute care domains. Postgrad Med J. 2014;90:699–708. https://doi.org/10.1136/postgradmedj-2014-132800

McCulloch P, Mishra A, Handa A, et al. The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Care. 2009;18:109–15. https://doi.org/10.1136/qshc.2008.032045

Riem N, Boet S, Bould MD, et al. Do technical skills correlate with non-technical skills in crisis resource management: A simulation study. Br J Anaesth. 2012;109:723–8. https://doi.org/10.1093/bja/aes256

Gjeraa K, MØller TP, Østergaard D. Efficacy of simulation-based trauma team training of non-technical skills. A systematic review. Acta Anaesthesiol Scand. 2014;58:775–87. https://doi.org/10.1111/aas.12336

Gordon M, Darbyshire D, Baker P. Non-technical skills training to enhance patient safety: A systematic review. Med Educ. 2012;46:1042–54. https://doi.org/10.1111/j.1365-2923.2012.04343.x

Gross B, Rusin L, Kiesewetter J, et al. Crew resource management training in healthcare: a systematic review of intervention design, training conditions and evaluation. BMJ Open. 2019;9:e025247. https://doi.org/10.1136/bmjopen-2018-025247

Hull L, Arora S, Aggarwal R, et al. The impact of nontechnical skills on technical performance in surgery: A systematic review. J Am Coll Surg. 2012;214:214–30. https://doi.org/10.1016/j.jamcollsurg.2011.10.016

Santiago Fernandes Pimenta ID, de Sousa Mata ÁN, Machado Bezerra IN, et al. Nontechnical skills training in intensive care units: protocol for a systematic review and meta-analysis. PLoS ONE. 2023;18. https://doi.org/10.1371/journal.pone.0280132

Page MJ, McKenzie JE, Bossuyt PM et al. (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ n71. https://doi.org/10.1136/bmj.n71

Higgins JP, Thomas J, Chandler J, et al. Cochrane handbook for systematic reviews of interventions data. Ltd: Wiley; 2019.

Kirkpatrick DL, Kirkpatrick JD. Implementing the four levels: A practical guide for effective evaluation of training programs. San Francisco, California: Berrett-Koehler; 2007.

Fletcher G, Flin R, McGeorge M, et al. Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90:580–8. https://doi.org/10.1093/bja/aeg112

EPOC. (2013) Data Collection Form. Oslo. https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/Strom_SSU-protocol_Modifieddatacollectionform180916.docx

Flemyng E, Moore TH, Boutron I, et al. Using risk of bias 2 to assess results from randomised controlled trials: guidance from Cochrane. BMJ Evidence-Based Med. 2023;28(4):260–6.

Sterne JAC, Savović J, Page MJ, et al. RoB 2: A revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:1–8. https://doi.org/10.1136/bmj.l4898

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Online). 2016;355:4–10. https://doi.org/10.1136/bmj.i4919

NIH. (2014) Study Quality Assessment Tools. NHLBI, NIH. In: National Heart, Lung, and Blood Institute. https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools. Accessed 21 Dec 2021.

Campbell M, McKenzie JE, Sowden A, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368:1–6. https://doi.org/10.1136/bmj.l6890

Andrews JC, Schünemann HJ, Oxman AD, et al. GRADE guidelines: 15. Going from evidence to recommendation—determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013;66:726–35. https://doi.org/10.1016/j.jclinepi.2013.02.003

Murad MH, Mustafa RA, Schünemann HJ, et al. Rating the certainty in evidence in the absence of a single estimate of effect. Evid Based Med. 2017;22:85–7. https://doi.org/10.1136/ebmed-2017-110668

Allan CK, Thiagarajan RR, Beke D, et al. Simulation-based training delivered directly to the pediatric cardiac intensive care unit engenders preparedness, comfort, and decreased anxiety among multidisciplinary resuscitation teams. J Thorac Cardiovasc Surg. 2010;140:646–52. https://doi.org/10.1016/j.jtcvs.2010.04.027

Anderson JDM, Murphy AA, Boyle KB, et al. Simulating extracorporeal membrane oxygenation emergencies to improve human performance. Part II: assessment of technical and behavioral skills. Simul Healthc. 2006;1:228–32. https://doi.org/10.1097/01.SIH.0000243551.01521.74

van den Bos-Boon A, Hekman S, Houmes RJ, et al. Effectiveness of simulation training and assessment of PICU nurses’ resuscitation skills: A mixed methods study from the Netherlands. J Pediatr Nurs. 2021;59:e52–60. https://doi.org/10.1016/j.pedn.2021.01.029

Brodsky D, Gupta M, Quinn M, et al. Building collaborative teams in neonatal intensive care. BMJ Qual Saf. 2013;22:374–82. https://doi.org/10.1136/bmjqs-2012-000909

Colman N, Figueroa J, McCracken C, Hebbar KB. Can simulation Based-Team training impact bedside teamwork in a pediatric intensive care unit?? J Pediatr Intensive Care. 2019;08:195–203. https://doi.org/10.1055/s-0039-1688928

Figueroa MI, Sepanski R, Goldberg SP, Shah S. Improving teamwork, confidence, and collaboration among members of a pediatric cardiovascular intensive care unit multidisciplinary team using simulation-based team training. Pediatr Cardiol. 2013;34:612–9. https://doi.org/10.1007/s00246-012-0506-2

Haerkens MHTMTM, Kox M, Lemson J, et al. Crew resource management in the intensive care unit: A prospective 3-year cohort study. Acta Anaesthesiol Scand. 2015;59:1319–29. https://doi.org/10.1111/aas.12573

Mayer CM, Cluff L, Lin W-T, et al. Evaluating efforts to optimize teamstepps implementation in surgical and pediatric intensive care units. Joint Comm J Qual Patient Saf. 2011;37:365–AP3. https://doi.org/10.1016/S1553-7250(11)37047-X

Meurling L, Hedman L, Sandahl C, et al. Systematic simulation-based team training in a Swedish intensive care unit: A diverse response among critical care professions. BMJ Qual Saf. 2013;22:485–94. https://doi.org/10.1136/bmjqs-2012-000994

Murphy T, Laptook A, Bender J. Sustained improvement in neonatal intensive care unit safety attitudes after teamwork training. J Patient Saf. 2015;14:174–80. https://doi.org/10.1097/PTS.0000000000000191

Sawyer T, Laubach VA, Hudak J, et al. Improvements in teamwork during neonatal resuscitation after interprofessional teamstepps training. Neonatal Netw. 2013;32:26–33. https://doi.org/10.1891/0730-0832.32.1.26

Mileder LP, Baik-Schneditz N, Pansy J, et al. Impact of in situ simulation training on quality of postnatal stabilization and resuscitation—a before-and-after, non-controlled quality improvement study. Eur J Pediatr. 2024;183(11):4981–90. https://doi.org/10.1007/s00431-024-05781-3

Jonsson K, Brulin C, Härgestam M, et al. Do team and task performance improve after training situation awareness? A randomized controlled study of interprofessional intensive care teams. Scand J Trauma Resusc Emerg Med. 2021;29:1–11. https://doi.org/10.1186/s13049-021-00878-2

Kemper PF, De Bruijne M, Van Dyck C, et al. Crew resource management training in the intensive care unit. A multisite controlled before-after study. BMJ Qual Saf. 2016;25:577–87. https://doi.org/10.1136/bmjqs-2015-003994

Stockwell DC, Landrigan CP, Toomey SL, et al. Adverse events in hospitalized pediatric patients. Pediatrics. 2018. https://doi.org/10.1542/peds.2017-3360

de Sousa Mata ÁN, de Azevedo KPM, Braga LP, et al. Training in communication skills for self-efficacy of health professionals: a systematic review. Hum Resour Health. 2021;19:1–9. https://doi.org/10.1186/s12960-021-00574-3

Bluestone J, Johnson P, Fullerton J, et al. Effective in-service training design and delivery: evidence from an integrative literature review. Hum Resour Health. 2013. https://doi.org/10.1186/1478-4491-11-51

Berkhof M, van Rijssen HJ, Schellart AJM, et al. Effective training strategies for teaching communication skills to physicians: an overview of systematic reviews. Patient Educ Couns. 2011;84:152–62. https://doi.org/10.1016/j.pec.2010.06.010

Gysels M, Richardson A, Higginson IJ. Communication training for health professionals who care for patients with cancer: A systematic review of training methods. Support Care Cancer. 2005;13:356–66. https://doi.org/10.1007/s00520-004-0732-0

Ervin JN, Kahn JM, Cohen TR, Weingart LR. Teamwork in the intensive care unit. Am Psychol. 2018;73:468–77. https://doi.org/10.1037/amp0000247

Di Muzio M, Dionisi S, Di Simone E, et al. Can nurses’ shift work jeopardize the patient safety? A systematic review. Eur Rev Med Pharmacol Sci. 2019;23:4507–19. https://doi.org/10.26355/eurrev_201905_17963

Koch SH, Weir C, Haar M, et al. Intensive care unit nurses’ information needs and recommendations for integrated displays to improve nurses’ situation awareness. J Am Med Inform Assoc. 2012;19:583–90. https://doi.org/10.1136/amiajnl-2011-000678

Endsley MR. (2021) SITUATION AWARENESS. Handbook of Human Factors and Ergonomics 434–455. https://doi.org/10.1002/9781119636113.CH17

D’Esmond LK. What was I thinking? Decision making and its impact on outcomes, quality improvement, and research. Intensive Crit Care Nurs. 2018;45:1–2. https://doi.org/10.1016/j.iccn.2018.01.002

Coggins A, Desai M, Nguyen K, Moore N. Early acquisition of non-technical skills using a blended approach to simulation-based medical education. Adv Simul. 2017;2:1–7. https://doi.org/10.1186/s41077-017-0045-2

Hull L, Arora S, Symons NRA, et al. Training faculty in nontechnical skill assessment: National guidelines on program requirements. Ann Surg. 2013;258:370–5. https://doi.org/10.1097/SLA.0b013e318279560b

Salas E, Cannon-Bowers JA. (2004) Methods, tools, and strategies for team training. Training for a rapidly changing workplace: Applications of psychological research 249–279. https://doi.org/10.1037/10260-010

Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl). https://doi.org/10.1097/00001888-199009000-00045. S63-7.

Riley W, Davis S, Miller K, et al. Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Jt Comm J Qual Patient Saf. 2011;37:357–64. https://doi.org/10.1016/S1553-7250(11)37046-8

Shah V, Warre R, Lee SK. Quality improvement initiatives in neonatal intensive care unit networks: achievements and challenges. Acad Pediatr. 2013;13:S75–83. https://doi.org/10.1016/j.acap.2013.04.014

Wisshak S, Hochholdinger S. Trainers’ knowledge and skills from the perspective of trainers, trainees and human resource development practitioners. Int J Train Res. 2018;16:218–31. https://doi.org/10.1080/14480220.2018.1576327

Lehtinen A, Viiri J. Guidance provided by teacher and simulation for Inquiry-Based learning: a case study. J Sci Educ Technol. 2017;26:193–206. https://doi.org/10.1007/s10956-016-9672-y

Hitchins CR, Metzner M, Edworthy J, Ward C. Non-technical skills and Gastrointestinal endoscopy: A review of the literature. Frontline Gastroenterol. 2018;9:129–34. https://doi.org/10.1136/flgastro-2016-100800

Gjeraa K, Spanager L, Konge L, et al. Non-technical skills in minimally invasive surgery teams: a systematic review. Surg Endosc. 2016;30:5185–99. https://doi.org/10.1007/s00464-016-4890-1

Reio TG, Rocco TS, Smith DH, Chang E. A critique of Kirkpatrick’s evaluation model. New Horizons Adult Educ Hum Resource Dev. 2017;29:35–53. https://doi.org/10.1002/nha3.20178

Arora S, Miskovic D, Hull L, et al. Self vs expert assessment of technical and non-technical skills in high fidelity simulation. Am J Surg. 2011;202:500–6. https://doi.org/10.1016/j.amjsurg.2011.01.024

Rizan C, Ansell J, Tilston TW, et al. Are general surgeons able to accurately self-assess their level of technical skills? Ann R Coll Surg Engl. 2015;97:549–55. https://doi.org/10.1308/rcsann.2015.0024.

Nayar SK, Musto L, Baruah G, et al. Self-Assessment of surgical skills: A systematic review. J Surg Educ. 2020;77:348–61. https://doi.org/10.1016/j.jsurg.2019.09.016

Meyer GS, Denham CR, Battles J et al. (2010) National Quality Forum (NQF). Safe Practices for Better Healthcare– 2010 Update.

Zeltser MV, Nash DB. Approaching the evidence basis for Aviation-Derived teamwork training in medicine. Am J Med Qual. 2019;34:455–64. https://doi.org/10.1177/1062860619873215

Leuschner S, Leuschner M, Kropf S, Niederbichler AD. Non-technical skills training in the operating theatre: A meta-analysis of patient outcomes. Surgeon. 2019;17:233–43. https://doi.org/10.1016/j.surge.2018.07.001

Merriel A, Ficquet J, Barnard K, et al. The effects of interactive training of healthcare providers on the management of life-threatening emergencies in hospital. Cochrane Database Syst Reviews. 2019;2019. https://doi.org/10.1002/14651858.CD012177.pub2

Goodacre S, Campbell M, Carter A. What do hospital mortality rates tell Us about quality of care? Emerg Med J. 2015;32:244–7. https://doi.org/10.1136/emermed-2013-203022

Kontopantelis E, Doran T, Springate DA, et al. Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ (Online). 2015;350:1–4. https://doi.org/10.1136/bmj.h2750

Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013;13:S38–44. https://doi.org/10.1016/j.acap.2013.08.002

Saturno-Hernandez PJ. Métodos y Herramientas Para La Realización de Ciclos de Mejora de La Calidad En servicios de Salud. 1st ed. México: Instituto Nacional de Salud Pública, Cuernavaca; 2015.

Groves RM, Peytcheva E. The impact of nonresponse rates on nonresponse bias: A Meta-Analysis. Public Opin Q. 2008;72:167–89. https://doi.org/10.1093/poq/nfn011

Puffer S, Torgerson DJ, Watson J. Cluster randomized controlled trials. J Eval Clin Pract. 2005;5479–83. https://doi.org/10.1111/j.1365-2753.2005.00568.x

McEwan D, Ruissen GR, Eys MA, et al. The effectiveness of teamwork training on teamwork behaviors and team performance: A systematic review and Meta-Analysis of controlled interventions. PLoS ONE. 2017;12:e0169604. https://doi.org/10.1371/journal.pone.0169604

Straus SE, Soobiah C, Levinson W. The impact of leadership training programs on physicians in academic medical centers: A systematic review. Acad Med. 2013;88:710–23. https://doi.org/10.1097/ACM.0b013e31828af493