Interrater reliability and agreement of the NEUMOBACT checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios

Abstract

Objective

To analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible.

Methods

A validation study with a cross-sectional design, compliant with the GRRAS checklist, among ICU nurses attending a SIMULAZERO course with an Objective Structured Clinical Evaluation simulation format, to verify transfer from theory to clinical practice of knowledge and skills in ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB) prevention. A minimum sample size of 111 pairs of nurse raters was calculated. Interrater agreement was analysed using Gwet’s AC1 for each item and as a total for each of the three checklists in the NEUMOBACT instrument.

Results

A total of 95 pairs of valid NEUMOBACT checklists were completed by 190 raters with a median age of 29 [25–35] years, 93.7% were female. At the Central Venous Catheter insertion station, Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). Only 2 of the 17 items scored below 0.9. At the Endotracheal Suctioning station, Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items that made up this station, 16 had an agreement percentage above 0.9, a further 9 were between 0.821 and 0.884, and item 13 had a value of 0.789. At the Patient Care station, Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items, 17 showed an agreement percentage above 0.9 and 4 were between 0.810 and 0.894.

Conclusions

The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses.

Relevance to clinical practice

The NEUMOBACT checklist can be useful for assessing skills before and after training in VAP and CRB prevention measures and during debriefing (post-simulation feedback), to reinforce the scientific evidence behind actions and decisions for items that have been performed incorrectly, thus consolidating training already received.

Introduction

Intensive care units (ICUs) underwent structural and organisational changes as a result of the Covid-19 pandemic, and these changes lead to a two- to threefold increase in the incidence rates of ventilator-associated pneumonia (VAP) and catheter-related bacteraemia (CRB), together with increased length of lCU stays and intra-ICU mortality [1]. Measures to address these problems included a proposal by the Ministry of Health’s board of experts to “promote Zero Project training among existing, newly recruited and temporary ICU healthcare providers”. These projects include Bacteraemia Zero and Pneumonia Zero, which are training and epidemiological surveillance programmes designed to reduce the incidence of CRB and VAP, respectively, that are implemented across Spain and are supported by experts from the Spanish Society for Intensive and Critical Care Medicine and Coronary Units (SEMICYUC) and the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC).

The SEEIUC Simulation working group runs a training course called SIMULAZERO, which applies teaching through simulation to assess ICU nurses’ knowledge and skills in VAP and CRB prevention [2]. The SIMULAZERO course is based on an Objective Structured Clinical Evaluation (OSCE) composed of two training scenarios for technical skills (endotracheal suctioning and central venous catheter insertion, respectively) and a training scenario for non-technical skills [3], which, in this case, concerns decision-making, resource management, safe practice and reduction of adverse events related to the care of critically ill patients.

In simulation, OSCEs are considered the gold standard for competency-based assessments. In OSCEs, candidates rotate through a series of time-limited stations in a circuit designed to assess professional performance in a simulated environment. These assessments facilitate the objective evaluation of a complex range of skills, knowledge, and attitudes through direct observation [4], while providing an adequate method for engaging and integrating theory with practice [5]. These evaluations can be training or assessment based, or both [6]. Various checklists are used in the ICU to guide evidence-based clinical practice, with a compliance rate ranging from 73% to 91%, improving care outcomes [7], but no checklist was found in the literature to assess ICU nurses’ performance during an OSCE, and therefore the NEUMOBACT checklist was created, designed and face validity and content [8]. The aim of this study was to analyse the interrater reliability of the NEUMOBACT checklist and verify whether consistent results are reproducible.

Materials and methods

A descriptive study of a metric nature, reported using the GRRAS checklist (S1 Checklist) for reporting of studies of reliability and agreement [9] and was conducted among ICU nurses attending the SIMULAZERO course at the 47th National Congress of SEEIUC. Recruitment was conducted by convenience from 13 to 15 June 2022. The nurses worked at various ICUs across Spain, permitting the identification of potential heterogeneity among those ICUs. Nurses were included if they had participated in the course as raters, completed all the NEUMOBACT checklist items, and agreed to participate in the study.

Sample calculation

According to the COSMIN Study Design checklist for patient-reported outcome measurement instruments [10], a minimum of 100 pairs of assessments by at least two unbiased evaluators is required to analyze inter-observer reliability. Two independent observers should be used to assess the simulation stations under similar environmental conditions, having received the same instructions. To account for expected losses, a further 10% was added, resulting in a sample of 111 pairs of nurses to evaluate the NEUMOBACT checklist during the simulation.

Variables and information sources

The data were collected during the SIMULAZERO course, based on a multi-station OSCE-format simulation (specifically 3), whereby participants rotated around a circuit of sequential stations. Each station presented a clinical case, requiring students to perform a specific set of tasks by applying a range of knowledge, skills and attitudes, resulting in the simultaneous assessment of several competence-based components. Each participant started their test at a different station, but at the same time as each other (about 10 minutes). The scenarios were of a similar length so that participants would leave one station and start the next in synchronisation. The resources used included standard materials commonly available in intensive care units, as well as basic CPR manikins, avoiding any material limitations for future replication in other ICUs. Two simultaneous rounds were conducted, with each station requiring one technician, two assessors, and two OSCE coordinators.

Raters received specific training in their roles as raters to provide them with detailed assessment criteria and the meaning of each item. The raters were paired up and given the NEUMOBACT checklist, which had been previously developed by experts using the Delphi technique, to use for each station assessment, with only two response options for each item: correct/incorrect (Table 1). They were given 10 minutes to familiarise themselves with the checklist and were reminded of the importance of completing all the items when assessing each station. They were also instructed not to share the information with each other and for this purpose they were positioned one on each side of the simulator. Since the three NEUMOBACT checklists are independent of each other, each rater pair started the assessment at a different station and followed the circuit described above. The rater pairs were also asked to respond to sociodemographic questions on their age, gender, nursing experience, ICU nursing experience and whether they had postgraduate training.

[Figure omitted. See PDF.]

S1 File describes the learning objective, participants’ roles and actors for each of the three stations in the circuit.

Data analysis

Qualitative variables were expressed as frequency and percentage and quantitative variables as mean and standard deviation or median and interquartile range (P25-P75), depending on the data distribution. To analyse the reliability of the instrument, interrater agreement was calculated using Gwet’s AC1 for each NEUMOBACT checklist item, and for each station as a total value. Gwet’s AC1 coefficient method was used because it is more stable and less affected by prevalence and marginal probability than Cohen’s Kappa [11, 12]. Gwet’s AC1 values range from -1 to 1, where a negative value indicates significant disagreement, zero indicates what would be expected by chance, values ≤0.3 indicate slight agreement, 0.31–0.6 moderate agreement, and ≥0.7 substantial agreement [11].

Ethical aspects

The project was approved by the Bioethics Commission of the University of Barcelona (code number: IRB00003099). The investigators undertook to comply with Organic Law 3/2018 of 5 December on personal data protection and the guarantee of digital rights. The participants were informed about the study’s purpose and invited to participate. Consent was given verbally and confirmed again verbally when participants chose to respond to the checklist. In order to enhance data protection, only the principal investigator recorded and had access to the socio-demographic and academic data of each rater.

Results

Demographic characteristics

95 pairs of valid NEUMOBACT checklists were collected. The raters (n = 190) had a median age (median [P25-P75]) of 29 [25–35] years and 93.7% were female. They had 5 [2–8] years of nursing experience and 3 [2–7] years of ICU-specific experience. A total of 54.7% reported postgraduate-level education.

Reliability

At the Central Venous Catheter (CVC) insertion station, the total value for Gwet’s AC1 was 0.934 (95% CI [0.919–0.949]). All items had an agreement percentage above 0.9 except item 10 (“Use maximum barrier measures (mask, cap, eye protection and gown, sterile drapes, sheets and gloves) for CVC insertion”) and item 17 (“Place needle-free connectors only at sites where boluses are to be administered”) (Table 2).

[Figure omitted. See PDF.]

At the Endotracheal Suctioning (ETS) station, the total value for Gwet’s AC1 was 0.869 (95% CI [0.851–0.886]). Of the 26 items assessed at this station, 16 showed agreement percentages of >0.9, a further 9 items (specifically, items 4, 9, 10.1, 10.3, 10.5, 11.3, 11.4, 11.8 and 12) had percentages between 0.821 and 0.884, and item 13 had a value of 0.789 (Table 2).

At the Patient Care (PC) station, the total value for Gwet’s AC1 was 0.911 (95% CI [0.896–0.927]). Of the 21 items assessed at this station, 17 showed agreement percentages of >0.9, and the other 4 items (specifically, items 1e, 7, 14 and 17) had percentages between 0.810 and 0.894 (Table 2).

To improve the reliability of items with agreement percentages below 0.9, the investigators, through a focus group discussion, reviewed the checklist and proposed the following changes to make it easier for raters to understand the items. For item 10 of the CVC insertion station, the word "ALL" was added to clarify that the item should be marked as correct when all the measures are applied and as incorrect when only some of them are applied. The content of item 17 was expanded, adding "intermittent medication and/or bolus". For the ETS station, the word "ALL" was added to item 4 for the same reason as for the CVC station item. For the PC station, it was proposed to make the word “and” uppercase and bold in items 13 and 14 to highlight that both points in the item have to be performed (Table 3).

[Figure omitted. See PDF.]

Discussion

Very good interrater reliability was achieved on all items of the NEUMOBACT checklist for use in the assessment of ICU nurses’ knowledge and skills in VAP and CRB prevention using simulation-based training. The checklist can be useful to assess skills before and after training in the Pneumonia Zero and Bacteraemia Zero project, to verify theory-to-practice transfer, which is a controversial aspect of training [13, 14] and, during debriefing (post-simulation feedback), to reinforce the scientific evidence behind actions and decisions for items that have been performed incorrectly [15–18]. Although this program has been conducted with ICU nursing professionals, it could be implemented in undergraduate studies, as various authors highlight the benefits of using simulation methodology during the pre-professional stage [19, 20].

Mogyoródi et al. [21] analysed before-and-after compliance with VAP prevention measures in clinical practice with a PowerPoint training programme and found a significant improvement in compliance three months after training, but a decrease to baseline at 12 months. Jansson et al. [22] found no impact on skills following simulation training on adherence to oral care recommendations. Gerolemou et al. [23] evaluated simulation-based training for CVC insertion and demonstrated an 85% reduction in the incidence of CRB infections in a critical care unit after the intervention.

Various VAP and CRB prevention checklists and care bundles have been published in the literature. These instruments were either created to assess compliance with pneumonia prevention measures in clinical practice [24–26] or designed to be applied during simulation of CVC-related skills [23, 27], but as far as we are aware, our NEUMOBACT checklist is the only one that contains all the necessary items for VAP and CRB prevention, created and validated by infection prevention and simulation experts [9].

Furthermore, unlike other instruments, the NEUMOBACT checklist requires no prior training or preparation for use during the simulation [28, 29]. This study, with a large sample of 190 raters, shows that if clear instructions are giving on using the checklist, (i.e., mark each item as correct/incorrect when those actions/decisions are observed during the simulation), it is feasible to achieve good interrater agreement for checklist completion.

Limitations

In the sample analysed, half of the nurses had postgraduate education. This level of education may have enhanced the nurses’ understanding of the checklist items compared to other nurses with undergraduate training only, because they are more familiar with complex ICU procedures, the theory behind those procedures, and simulation scenarios in general, as it is a widely used teaching method in postgraduate training [30]. To compensate for this possible bias, the raters were informed that the checklists used during the SIMULAZERO course were for research to assess interrater reliability, and that at no time would the checklists be analysed to assess participant performance of actions/decisions during the simulation.

It is essential to have validated rubrics in Spain to evaluate simulation-based activities [31]. Since developing an instrument is both costly and time-consuming, adapting existing instruments into another language offers several advantages. On one hand, it reduces research costs while preserving the psychometric properties of the original instrument. On the other hand, it allows for comparison of equally valid and reliable results with those from other national and international studies that have used the same instrument [32].

Conclusions

The interrater reliability of the NEUMOBACT checklist shows substantial agreement between pairs of raters and is therefore validated in this large sample of ICU nurses. The checklist requires no prior training and is validated and reliable for assessing nurses’ performance during simulation-based training in VAP and CRB preventive measures.

Supporting information

S1 Checklist. Checklist for reporting of studies of reliability and agreement (GRRAS checklist).

https://doi.org/10.1371/journal.pone.0313175.s001

S1 File. Description of the simulation stations used in the OSCE-format SIMULAZERO course.

https://doi.org/10.1371/journal.pone.0313175.s002

(DOCX)

Acknowledgments

The authors are grateful to the nurses who participated in the workshops at the Seville congress and to Maria Pérez Riart and Purificación Pérez Teran from the Hospital del Mar for their teaching and methodological support.

References

1. 1. Sociedad Española de Enfermería Intensiva y Unidades Coronarias [Internet]. [cited 2023 Nov 24]. Available from: https://seeiuc.org/

2. 2. Raurell-Torredà M, Zaragoza-García I, Aliberch-Raurell AM, Sánchez-Chillón J, Torralba-Melero M, Arrogante O, et al. SIMULAZERO: taller de simulación para actualizar conocimientos y habilidades en la prevención de la neumonía asociada a ventilación mecánica y bacteriemia relacionada con catéter (Proyectos Zero). Enferm Intensiva. 2022;33:S45–55.

* View Article

* Google Scholar

3. 3. Lioce L, editor. Healthcare Simulation Dictionary [Internet]. 2nd ed. Agency for Healthcare Research and Quality; 2020 [cited 2023 Jun 30]. Available from: https://www.ahrq.gov/patient-safety/resources/simulation/terms.html

4. 4. Goh HS, Zhang H, Lee CN, Wu XV, Wang W. Value of Nursing Objective Structured Clinical Examinations: A Scoping Review. Nurse Educ. 2019;44(5):E1–6. pmid:30371544

* View Article

* PubMed/NCBI

* Google Scholar

5. 5. Taylor I, Bing-Jonsson PC, Johansen E, Levy-Malmberg R, Fagerström L. The Objective Structured Clinical Examination in evolving nurse practitioner education: A study of students’ and examiners’ experiences. Nurse Educ Pract. 2019;37:115–23. pmid:31136916

* View Article

* PubMed/NCBI

* Google Scholar

6. 6. Holzhausen Y, Maaz A, März M, Sehy V, Peters H. Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination. BMC Med Educ. 2019;19(1):319. pmid:31438938

* View Article

* PubMed/NCBI

* Google Scholar

7. 7. Erikson EJ, Edelman DA, Brewster FM, Marshall SD, Turner MC, Sarode VV, et al. The use of checklists in the intensive care unit: a scoping review. Crit Care. 2023;27(1):468. pmid:38037056

* View Article

* PubMed/NCBI

* Google Scholar

8. 8. Raurell-Torredà M, Arrogante O, Aliberch-Raurell AM, Sánchez-Chillón FJ, Torralba-Melero M, Rojo-Rojo A, et al. Design and content validation of a checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios. J Clin Nurs. 2024;33(8):3188–98. pmid:38348543

* View Article

* PubMed/NCBI

* Google Scholar

9. 9. Kuiper JH. GRRAS checklist for reporting of studies of reliability and agreement. Available in:https://www.equator-network.org/wp-content/uploads/2012/12/GRRAS-checklist-for-reporting-of-studies-of-reliability-and-agreement.pdf

* View Article

* Google Scholar

10. 10. Mokkink LB, Prinsen CA, Patrick DL, Alonso J, Bouter LM, de Vet HC, et al. COSMIN Study Design checklist for Patient-reported outcome measurement instruments. [access september 5, 2024]. Available in: https://www.cosmin.nl/wp-content/uploads/COSMIN-study-designing-checklist_final.pdf#.

* View Article

* Google Scholar

11. 11. Gwet KL. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters: Vol 2: Analysis of quantitative ratings. Advanced Analytics; 2021. 223–246 p.

* View Article

* Google Scholar

12. 12. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1):61. pmid:23627889

* View Article

* PubMed/NCBI

* Google Scholar

13. 13. Blot S, Conoscenti E. Why to monitor adherence to oral care protocols in intensive care units?. Aust Crit Care. 2023;36(2):169–170. pmid:34980521

* View Article

* PubMed/NCBI

* Google Scholar

14. 14. Lombardo L, Ferguson C, George A, Villarosa AR, Villarosa BJ, Kong Ariana C, et al. Interventions to promote oral care regimen adherence in the critical care setting: A systematic review. Aust Crit Care. 2022;35(5):583–594. pmid:34764003

* View Article

* PubMed/NCBI

* Google Scholar

15. 15. O’Grady NP. Prevention of Central Line–Associated Bloodstream Infections. Taichman DB, editor. N Engl J Med. 2023;389(12):1121–1131. pmid:37733310

* View Article

* PubMed/NCBI

* Google Scholar

16. 16. Blakeman TC, Scott JB, Yoder MA, Capellari E, Strickland SL. AARC Clinical Practice Guidelines: Artificial Airway Suctioning. Respir Care. 2022;67(2):258–271. pmid:35078900

* View Article

* PubMed/NCBI

* Google Scholar

17. 17. Garegnani LI, Giménez ML, Escobar Liquitay CM, Franco JVA. Oral hygiene interventions to prevent ventilator‐associated pneumonia: A network meta‐analysis. Nurs Crit Care. 2023;28(5):744–55.

* View Article

* Google Scholar

18. 18. Unahalekhaka A, Butpan P, Wongsaen R, Phunpae P, Preechasuth K. Contamination of antimicrobial-resistant bacteria on toothbrushes used with mechanically ventilated patients: A cross sectional study. Intensive Crit Care Nurs. 2022;68:103120. pmid:34391630

* View Article

* PubMed/NCBI

* Google Scholar

19. 19. Díaz Agea JL, Ramos-Morcillo AJ, Amo Setien FJ, Ruzafa-Martínez M, Hueso-Montoro C, Leal-Costa C. Perceptions about the self-learning methodology in simulated environments in nursing students: a mixed study. Int J Environ Res Public Health. 2019 Nov 22;16(23):4646. pmid:31766613

* View Article

* PubMed/NCBI

* Google Scholar

20. 20. Romero-Castillo R, Garrido-Bueno M, Fernández-León P. Nursing students’ perceptions and satisfaction with a self-learning methodology in simulated environments: A mixed-methods study. Nurse Educ Pract. 2024 Sep 12;80:104141. pmid:39293166

* View Article

* PubMed/NCBI

* Google Scholar

21. 21. Mogyoródi B, Skultéti D, Mezőcsáti M, Dunai E, Magyar P, Hermann C, et al. Effect of an educational intervention on compliance with care bundle items to prevent ventilator-associated pneumonia. Intensive Crit Care Nurs. 2023;75:103342. pmid:36464606

* View Article

* PubMed/NCBI

* Google Scholar

22. 22. Jansson MM, Syrjälä HP, Ohtonen PP, Meriläinen MH, Kyngäs HA, Ala‐Kokko TI. Effects of simulation education on oral care practices–a randomized controlled trial. Nurs Crit Care. 2017;22(3):161–168. pmid:28093837

* View Article

* PubMed/NCBI

* Google Scholar

23. 23. Gerolemou L, Fidellaga A, Rose K, Cooper S, Venturanza M, Aqeel A, et al. Simulation-Based Training for Nurses in Sterile Techniques During Central Vein Catheterization. Am J Crit Care. 2014;23(1):40–48. pmid:24382616

* View Article

* PubMed/NCBI

* Google Scholar

24. 24. Jam Gatell MR, Santé Roig M, Hernández Vian Ó, Carrillo Santín E, Turégano Duaso C, Fernández Moreno I, et al. Assessment of a training programme for the prevention of ventilator‐associated pneumonia. Nurs Crit Care. 2012;17(6):285–92. pmid:23061618

* View Article

* PubMed/NCBI

* Google Scholar

25. 25. Behzadi F, Khanjari S, Haghani H. Impact of an education program on the performance of nurses in providing oral care for mechanically ventilated children. Aust Crit Care. 2019;32(4):307–313. pmid:30126677

* View Article

* PubMed/NCBI

* Google Scholar

26. 26. Jansson MM, Ala-Kokko TI, Syrjälä HP, Kyngäs HA. Development and psychometric testing of ventilator bundle questionnaire and observation schedule. Am J Infect Control. 2014;42(4):381–384. pmid:24679564

* View Article

* PubMed/NCBI

* Google Scholar

27. 27. Barsuk JH, Cohen ER, Mikolajczak A, Seburn S, Slade M, Wayne DB. Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses. J Nurs Adm. 2015;45(10):511–517. pmid:26425976

* View Article

* PubMed/NCBI

* Google Scholar

28. 28. Rodríguez E, Chabrera C, Farrés M, Castillo J, Virumbrales M, Raurell M. Adaptation and validation of a Spanish version of the KidSIM Team Performance Scale. J Prof Nurs. 2021;37(5):795–803. pmid:34742507

* View Article

* PubMed/NCBI

* Google Scholar

29. 29. Roldán-Merino J, Farrés-Tarafa M, Estrada-Masllorens JM, Hurtado-Pardos B, Miguel-Ruiz D, Nebot-Bergua C, et al. Reliability and validity study of the Spanish adaptation of the “Creighton Simulation Evaluation Instrument (C-SEI)”. Nurse Educ Pract. 2019;35:14–20. pmid:30640046

* View Article

* PubMed/NCBI

* Google Scholar

30. 30. El Hussein MT, Harvey G, Bell N. The Influence of Nursing Simulation on Patient Outcomes and Patient Safety: A Scoping Review. Clin Simul Nurs. 2022;70:37–46.

* View Article

* Google Scholar

31. 31. Unver V, Basak T, Watts P, Gaioso V, Moss J, Tastan S, et al. The reliability and validity of three questionnaires: the Student Satisfaction and Self-Confidence in Learning Scale, Simulation Design Scale, and Educational Practices Questionnaire. Contemp Nurse. 2017;53(1):60–74. pmid:28084900

* View Article

* PubMed/NCBI

* Google Scholar

32. 32. Escobar Bravo MÁ. Transcultural adaptation of health-related measurement instruments. Enferm Clin. 2004;14(2):102–6.

* View Article

* Google Scholar

Citation: Raurell-Torredà M, Zaragoza-García I, Arrogante O, Aliberch-Raurell AM, Sánchez-Chillón FJ, Torralba-Melero M, et al. (2024) Interrater reliability and agreement of the NEUMOBACT checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios. PLoS ONE 19(12): e0313175. https://doi.org/10.1371/journal.pone.0313175

About the Authors:

Marta Raurell-Torredà

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

Affiliations: Department of Fundamental and Clinical Care Nursing, Hospitalet del Llobregat, Universitat de Barcelona, Campus de Bellvitge, Barcelona, Spain, Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Mental Health, Psychosocial and Complex Nursing Care Research Group—2021 SGR 01083, Spain

Ignacio Zaragoza-García

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations: Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Department of Nursing, Faculty of Nursing, Physiotherapy and Podology, Universidad Complutense Madrid, Madrid, Spain, Care Research Group (Invecuid), 12 de Octubre Hospital Institute of Health Research (imas12), Madrid, Spain

ORICD: https://orcid.org/0000-0002-8606-1415

Oscar Arrogante

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Formal analysis, Investigation, Validation, Writing – original draft, Writing – review & editing

Affiliations: Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Department of Nursing, Faculty of Nursing, Physiotherapy and Podology, Universidad Complutense Madrid, Madrid, Spain, Research Nursing Group of Instituto de Investigación Sanitaria Gregorio Marañón (IiSGM), Madrid, Spain

ORICD: https://orcid.org/0000-0001-8304-9093

Anna María Aliberch-Raurell

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Investigation, Methodology, Project administration, Writing – review & editing

Affiliations: Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Department of Critical Care, Hospital Clínic de Barcelona, Barcelona, Spain

Francisco Javier Sánchez-Chillón

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Writing – review & editing

Affiliations: Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Simulation Centre, Hospital 12 de Octubre, Madrid, Spain

Martín Torralba-Melero

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Writing – review & editing

Affiliations: Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Department of Critical Care, Hospital General Universitario de Albacete, Albacete, Spain

ORICD: https://orcid.org/0000-0001-5769-4486

Andrés Rojo-Rojo

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Funding acquisition, Investigation, Methodology, Writing – review & editing

Affiliations: Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain, Department of Nursing, Universidad Católica de Murcia, Campus de los Jerónimos, Guadalupe, Murcia, Spain

ORICD: https://orcid.org/0000-0001-8409-1744

Alfonso Muriel-García

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Writing – review & editing

Affiliation: Biostatistics Unit, Department of Nursing and Physiotherapy, Hospital Universitario Ramón y Cajal, IRYCIS, CIBERESP, Universidad de Alcalá, Madrid, Spain

Ana Carolina Amaya Arias

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Formal analysis, Methodology, Writing – review & editing

Affiliation: Division of Non-Communicable Diseases, Ministry of Health and Social Protection, Bogotá, Colombia

Juan Roldán-Merino

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Formal analysis, Methodology, Writing – review & editing

Affiliations: Mental Health, Psychosocial and Complex Nursing Care Research Group—2021 SGR 01083, Spain, Department of Nursing, Sant Joan de Déu Teaching Campus, Universitat de Barcelona, Sant Boi de Llobregat, Barcelona, Spain

Mariona Farrés-Tarafa

Contributed equally to this work with: Marta Raurell-Torredà, Ignacio Zaragoza-García, Oscar Arrogante, Anna María Aliberch-Raurell, Francisco Javier Sánchez-Chillón, Martín Torralba-Melero, Andrés Rojo-Rojo, Alfonso Muriel-García, Ana Carolina Amaya Arias, Juan Roldán-Merino, Mariona Farrés-Tarafa

Roles: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing

Affiliations: Department of Fundamental and Clinical Care Nursing, Hospitalet del Llobregat, Universitat de Barcelona, Campus de Bellvitge, Barcelona, Spain, Simulation Group of the Spanish Society for Intensive Care and Coronary Unit Nursing (SEEIUC), Madrid, Spain

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

References

1. Sociedad Española de Enfermería Intensiva y Unidades Coronarias [Internet]. [cited 2023 Nov 24]. Available from: https://seeiuc.org/

2. Raurell-Torredà M, Zaragoza-García I, Aliberch-Raurell AM, Sánchez-Chillón J, Torralba-Melero M, Arrogante O, et al. SIMULAZERO: taller de simulación para actualizar conocimientos y habilidades en la prevención de la neumonía asociada a ventilación mecánica y bacteriemia relacionada con catéter (Proyectos Zero). Enferm Intensiva. 2022;33:S45–55.

3. Lioce L, editor. Healthcare Simulation Dictionary [Internet]. 2nd ed. Agency for Healthcare Research and Quality; 2020 [cited 2023 Jun 30]. Available from: https://www.ahrq.gov/patient-safety/resources/simulation/terms.html

4. Goh HS, Zhang H, Lee CN, Wu XV, Wang W. Value of Nursing Objective Structured Clinical Examinations: A Scoping Review. Nurse Educ. 2019;44(5):E1–6. pmid:30371544

5. Taylor I, Bing-Jonsson PC, Johansen E, Levy-Malmberg R, Fagerström L. The Objective Structured Clinical Examination in evolving nurse practitioner education: A study of students’ and examiners’ experiences. Nurse Educ Pract. 2019;37:115–23. pmid:31136916

6. Holzhausen Y, Maaz A, März M, Sehy V, Peters H. Exploring the introduction of entrustment rating scales in an existing objective structured clinical examination. BMC Med Educ. 2019;19(1):319. pmid:31438938

7. Erikson EJ, Edelman DA, Brewster FM, Marshall SD, Turner MC, Sarode VV, et al. The use of checklists in the intensive care unit: a scoping review. Crit Care. 2023;27(1):468. pmid:38037056

8. Raurell-Torredà M, Arrogante O, Aliberch-Raurell AM, Sánchez-Chillón FJ, Torralba-Melero M, Rojo-Rojo A, et al. Design and content validation of a checklist about infection-prevention performance of intensive care nurses in simulation-based scenarios. J Clin Nurs. 2024;33(8):3188–98. pmid:38348543

9. Kuiper JH. GRRAS checklist for reporting of studies of reliability and agreement. Available in:https://www.equator-network.org/wp-content/uploads/2012/12/GRRAS-checklist-for-reporting-of-studies-of-reliability-and-agreement.pdf

10. Mokkink LB, Prinsen CA, Patrick DL, Alonso J, Bouter LM, de Vet HC, et al. COSMIN Study Design checklist for Patient-reported outcome measurement instruments. [access september 5, 2024]. Available in: https://www.cosmin.nl/wp-content/uploads/COSMIN-study-designing-checklist_final.pdf#.

11. Gwet KL. Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters: Vol 2: Analysis of quantitative ratings. Advanced Analytics; 2021. 223–246 p.

12. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med Res Methodol. 2013;13(1):61. pmid:23627889

13. Blot S, Conoscenti E. Why to monitor adherence to oral care protocols in intensive care units?. Aust Crit Care. 2023;36(2):169–170. pmid:34980521

14. Lombardo L, Ferguson C, George A, Villarosa AR, Villarosa BJ, Kong Ariana C, et al. Interventions to promote oral care regimen adherence in the critical care setting: A systematic review. Aust Crit Care. 2022;35(5):583–594. pmid:34764003

15. O’Grady NP. Prevention of Central Line–Associated Bloodstream Infections. Taichman DB, editor. N Engl J Med. 2023;389(12):1121–1131. pmid:37733310

16. Blakeman TC, Scott JB, Yoder MA, Capellari E, Strickland SL. AARC Clinical Practice Guidelines: Artificial Airway Suctioning. Respir Care. 2022;67(2):258–271. pmid:35078900

17. Garegnani LI, Giménez ML, Escobar Liquitay CM, Franco JVA. Oral hygiene interventions to prevent ventilator‐associated pneumonia: A network meta‐analysis. Nurs Crit Care. 2023;28(5):744–55.

18. Unahalekhaka A, Butpan P, Wongsaen R, Phunpae P, Preechasuth K. Contamination of antimicrobial-resistant bacteria on toothbrushes used with mechanically ventilated patients: A cross sectional study. Intensive Crit Care Nurs. 2022;68:103120. pmid:34391630

19. Díaz Agea JL, Ramos-Morcillo AJ, Amo Setien FJ, Ruzafa-Martínez M, Hueso-Montoro C, Leal-Costa C. Perceptions about the self-learning methodology in simulated environments in nursing students: a mixed study. Int J Environ Res Public Health. 2019 Nov 22;16(23):4646. pmid:31766613

20. Romero-Castillo R, Garrido-Bueno M, Fernández-León P. Nursing students’ perceptions and satisfaction with a self-learning methodology in simulated environments: A mixed-methods study. Nurse Educ Pract. 2024 Sep 12;80:104141. pmid:39293166

21. Mogyoródi B, Skultéti D, Mezőcsáti M, Dunai E, Magyar P, Hermann C, et al. Effect of an educational intervention on compliance with care bundle items to prevent ventilator-associated pneumonia. Intensive Crit Care Nurs. 2023;75:103342. pmid:36464606

22. Jansson MM, Syrjälä HP, Ohtonen PP, Meriläinen MH, Kyngäs HA, Ala‐Kokko TI. Effects of simulation education on oral care practices–a randomized controlled trial. Nurs Crit Care. 2017;22(3):161–168. pmid:28093837

23. Gerolemou L, Fidellaga A, Rose K, Cooper S, Venturanza M, Aqeel A, et al. Simulation-Based Training for Nurses in Sterile Techniques During Central Vein Catheterization. Am J Crit Care. 2014;23(1):40–48. pmid:24382616

24. Jam Gatell MR, Santé Roig M, Hernández Vian Ó, Carrillo Santín E, Turégano Duaso C, Fernández Moreno I, et al. Assessment of a training programme for the prevention of ventilator‐associated pneumonia. Nurs Crit Care. 2012;17(6):285–92. pmid:23061618

25. Behzadi F, Khanjari S, Haghani H. Impact of an education program on the performance of nurses in providing oral care for mechanically ventilated children. Aust Crit Care. 2019;32(4):307–313. pmid:30126677

26. Jansson MM, Ala-Kokko TI, Syrjälä HP, Kyngäs HA. Development and psychometric testing of ventilator bundle questionnaire and observation schedule. Am J Infect Control. 2014;42(4):381–384. pmid:24679564

27. Barsuk JH, Cohen ER, Mikolajczak A, Seburn S, Slade M, Wayne DB. Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses. J Nurs Adm. 2015;45(10):511–517. pmid:26425976

28. Rodríguez E, Chabrera C, Farrés M, Castillo J, Virumbrales M, Raurell M. Adaptation and validation of a Spanish version of the KidSIM Team Performance Scale. J Prof Nurs. 2021;37(5):795–803. pmid:34742507

29. Roldán-Merino J, Farrés-Tarafa M, Estrada-Masllorens JM, Hurtado-Pardos B, Miguel-Ruiz D, Nebot-Bergua C, et al. Reliability and validity study of the Spanish adaptation of the “Creighton Simulation Evaluation Instrument (C-SEI)”. Nurse Educ Pract. 2019;35:14–20. pmid:30640046

30. El Hussein MT, Harvey G, Bell N. The Influence of Nursing Simulation on Patient Outcomes and Patient Safety: A Scoping Review. Clin Simul Nurs. 2022;70:37–46.

31. Unver V, Basak T, Watts P, Gaioso V, Moss J, Tastan S, et al. The reliability and validity of three questionnaires: the Student Satisfaction and Self-Confidence in Learning Scale, Simulation Design Scale, and Educational Practices Questionnaire. Contemp Nurse. 2017;53(1):60–74. pmid:28084900

32. Escobar Bravo MÁ. Transcultural adaptation of health-related measurement instruments. Enferm Clin. 2004;14(2):102–6.