Developing medical educators - a mixed method evaluation of a teaching education program

Abstract

Background: It is well accepted that medical faculty teaching staff require an understanding of educational theory and pedagogical methods for effective medical teaching. The purpose of this study was to evaluate the effectiveness of a 5-day teaching education program.

Methods: An open prospective interventional study using quantitative and qualitative instruments was performed, covering all four levels of the Kirkpatrick model: Evaluation of 1) 'Reaction' on a professional and emotional level using standardized questionnaires; 2) 'Learning' applying a multiple choice test; 3) 'Behavior' by self-, peer-, and expert assessment of teaching sessions with semistructured interviews; and 4) 'Results' from student evaluations.

Results: Our data indicate the success of the educational intervention at all observed levels. 1) Reaction: The participants showed a high acceptance of the instructional content. 2) Learning: There was a significant increase in knowledge (P<0.001) as deduced from a pre-post multiple-choice questionnaire, which was retained at 6 months (P<0.001). 3) Behavior: Peer-, self-, and expert-assessment indicated a transfer of learning into teaching performance. Semistructured interviews reflected a higher level of professionalism in medical teaching by the participants. 4) Results: Teaching performance ratings improved in students' evaluations.

Conclusions: Our results demonstrate the success of a 5-day education program in embedding knowledge and skills to improve performance of medical educators. This multimethodological approach, using both qualitative and quantitative measures, may serve as a model to evaluate effectiveness of comparable interventions in other settings.

RESEARCH ARTICLE

Developing medical educators - a mixed method evaluation of a teaching education program

Marco Roos1*, Martina Kadmon2, Michael Kirschfink3, Eginhard Koch4, Jana Jünger5, Veronika Strittmatter-Haubold6 and Thorsten Steiner7

1Institute of General Practice, University of Erlangen-Nuremberg, Erlangen, Germany; 2Department of Surgery, Heidelberg University Hospital, Heidelberg, Germany; 3Institute of Immunology, Heidelberg University Hospital, Heidelberg, Germany; 4Department of Child and Adolescent Psychiatry, Heidelberg University Hospital, Heidelberg, Germany; 5Department of General Internal Medicine and Psychosomatic, Heidelberg University Hospital, Heidelberg, Germany; 6Academy of Adult Education, Heidelberg University of Education, Heidelberg, Germany; 7Department of Neurology, Klinikum Frankfurt Höchst, Frankfurt, Germany

Abstract

Background : It is well accepted that medical faculty teaching staff require an understanding of educational theory and pedagogical methods for effective medical teaching. The purpose of this study was to evaluate the effectiveness of a 5-day teaching education program.

Methods : An open prospective interventional study using quantitative and qualitative instruments was performed, covering all four levels of the Kirkpatrick model: Evaluation of 1) 'Reaction' on a professional and emotional level using standardized questionnaires; 2) 'Learning' applying a multiple choice test; 3) 'Behavior' by self-, peer-, and expert assessment of teaching sessions with semistructured interviews; and 4) 'Results' from student evaluations.

Results : Our data indicate the success of the educational intervention at all observed levels. 1) Reaction: The participants showed a high acceptance of the instructional content. 2) Learning: There was a significant increase in knowledge (P <0.001) as deduced from a pre-post multiple-choice questionnaire, which was retained at 6 months (P <0.001). 3) Behavior: Peer-, self-, and expert-assessment indicated a transfer of learning into teaching performance. Semistructured interviews reflected a higher level of professionalism in medical teaching by the participants. 4) Results: Teaching performance ratings improved in students' evaluations.

Conclusions : Our results demonstrate the success of a 5-day education program in embedding knowledge and skills to improve performance of medical educators. This multimethodological approach, using both qualitative and quantitative measures, may serve as a model to evaluate effectiveness of comparable interventions in other settings.

Keywords: faculty development; medical education; medical teacher; educational theory; collaborative feedback

*Correspondence to: Marco Roos, Institute of General Practice, University of Erlangen-Nuremberg, Krankenhausstr. 12, DE-91054 Erlangen, Germany, Email: [email protected]

Received: 21 January 2014; Revised: 22 February 2014; Accepted: 7 March 2014; Published: 27 March 2014

Medical Education Online 2014. © 2014 Marco Roos et al. This is an Open Access article distributed under the terms of the Creative Commons CC-BY 4.0 License (http://creativecommons.org/licenses/by/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material for any purpose, even commercially, provided the original work is properly cited and states its license.

Citation: Med Educ Online 2014, 19: 23868 - http://dx.doi.org/10.3402/meo.v19.23868

The drive for continuous improvement in medical education is propelled by both advancements in educational theory and research evidence, which is subsequently changing the traditional requirements of a medical educator (1-4). Hence, many medical faculties have endorsed development programs to improve teaching skills of their staff (5-7). A variety of different approaches have been surveyed, however, establishing the effectiveness of new faculty development programs and their impact on student education remains a challenge (8-10). In a systematic review by Steinert et al., the effects of faculty development interventions on knowledge, attitudes, and skills of educators on quality of education delivered, and on the institutions in which they worked, was reported (10). This review identified that repetitive interventions over time, using a deliberate adoption of theory of learning and educational principles, and the support of reflection and learning among participants, was effective. It was recommended that such interventions should be accompanied by process and outcome-oriented research, using multiple methods (quantitative as well as qualitative) in a performance-based way to assess changes on the basis of a conceptual framework (10). Kirkpatrick's model developed for measuring training effectiveness is one such useful framework (11). It measures effectiveness on four outcome levels: 1) the participants' affective responses to training content and environment, 2) the impact of the training itself (level of learning), 3) the long-term outcome in job-related performance (level of behavior), and ultimately 4) institutional changes (level of results) (4, 5, 12-14).

The Heidelberg Medical Faculty implemented a 5-day education program in 2001 to support the implementation of its new medical curriculum HeiCuMed (Heidelberger Curriculum Medicinale) (15-17). The program prepares members of the medical faculty for their role as educators. Prior to 2001 within the old curriculum, the role of faculty members that were involved in teaching medical students was that of a directive instructor rather than that of a facilitator to support the learning process of students, as it is within the new curriculum. The 5-day education program targeted the entire teaching staff of the faculty (including those involved in both core sciences and clinical teaching). To date, over 1,000 faculty members have passed through this education program. It is delivered by qualified faculty members holding a Master degree in Medical Education or a Federal Certificate in Higher Education (in collaboration with external experts for adult education from University of Education, Heidelberg).

The content of the education program covers five essential objectives: 1) learning theory and educational principles, the 'sandwich-architecture,' and 'constructive alignment' as a general framework for teaching sessions, 2) simulation and skills labs as teaching methods and environments, 3) problem-based learning (PBL), 4) modern teaching assessment methods, and 5) reflection on the role of a medical educator. Each day of the course was concluded with a session of peer-coaching on an individual teaching session (facilitator-learner feedback as well as learner-learner feedback). The intent was to actively support participants to improve the ways in which they structured their teaching sessions as well as to enhance their teaching skills. These sessions were also designed to incorporate aspects of constructive alignment (18). The program content of each day is summarized in Table 1.

[Table omitted -see PDF.]

Table shows categories and samples of 149 text units reported in the semistructured interview.

Conclusions

The primary goal of this study was to identify several factors that influence the developmental process of participants in the 5-day education program to become better medical educators. It found a high acceptance of the theoretical content, which was reflected in its approval on a professional and emotional level. This was an important prerequisite for achieving new knowledge and its transfer to on-the-job performance (in clinical settings), which was confirmed in the results of MCQ and qualitative measurements. Finally, it found a high concordance of self-, peer-, and expert-assessment in following clinical settings. According to the results of our study, the 5-day education program fulfills criteria from Kirkpatrick's framework on all explored levels measuring effectiveness of an education intervention.

As a precondition of achieving knowledge and of changing behavior a high satisfaction with the education structure and the content is needed (11). Our results confirmed satisfaction as shown in daily evaluations during the training, using items on professional aspects of the training content and on emotional reactions of our participants (Kirkpatrick's level of reaction). At the same time, education specific knowledge increased significantly (with high levels of effect size) during the training period and was maintained at the 6-month follow-up, as shown by the MCQ at three time points (p <0.001) (Kirkpatrick's level of learning). These findings are in line with previously published reviews and confirm the importance of an integrative theory of education motivation (2, 4, 12, 22). They also support the fact that reactions to education interventions have a fundamental impact on the engagement with theoretical content (knowledge and skill acquisition) and the transfer of new knowledge to job-performance (22).

A supportive organizational environment is indispensable for the sustainability of educational interventions to improve the transfer of new knowledge to job-performance and to stimulate changes in behavior as a medical educator (5, 23-25). This study identified three main supportive elements in the environment of our 5-day training program. First was establishing peer-coaching structures by implementing small group work of 4-5 peer tandems (learner with learner) with the aim of transferring the theoretical content into participants' teaching portfolio (on-the-job performance) and to enhance development by collaborative working behaviors. Furthermore, a non-judgmental and peer-coaching environment with a constructive, formative feedback promotes a supportive learning environment and keeps participants motivated (26, 27). As a result, participants used their new knowledge for peer observation in the post-intervention period by evaluating and discussing job-related performance in peer-assessment, including suggestions for improvement (formative feedback). Supporting a feedback culture among the participants was one of the major training aims to encourage collegiality and collaboration within and across disciplines. Supporting collaborative formative feedback seemed to be the most valuable factor in learning processes, promoting effectiveness of training intervention (28).

Second, the fact that we found high agreement in observations between peer- and expert-assessment implies that our participants acquired the ability to apply theoretical content. They were supported in the transfer process into their teaching practice with peer- and own-performance reflections, as an important stage in their professional development (2, 14). The participants gave feedback to each other on content and quality of teaching performance. This individual feedback was comparable with expert feedback in quantity and quality, although the evaluation form only provides headings to guide and standardize feedback.

Third, the delivery of the intervention by professional facilitators with medical and pedagogical backgrounds was an important prerequisite for the effectiveness of this education intervention. This seems to be an essential structural element to avoid barriers between educators and clinicians (5, 10, 14).

Although the results of the study met the goals of our study, several limitations still remain. The needs of our institution did not allow for a control group. Therefore, we cannot be sure that some of the outcomes we attribute to the 5-day program are not actually due to a selection bias. Participation in the 5-day program is a prerequisite for the postdoctoral qualification for some participants. Furthermore, the 5-day program is compulsory for all faculty teaching staff within the faculty development program.

A second limitation is the small size of our study group (the whole faculty has approximately 1,200 members). However, for the intervention group, we reached 85% participation for quantitative (Level 1 and 2 of Kirkpatrick's framework) and 20% for qualitative measures (Level 3 of Kirkpatrick's framework). This might be a contrast to other findings in literature (29).

Third, long-term outcomes were measured at a 6-month follow-up, and some may regard this as too short an evaluation interval. However, we could demonstrate that the immediate and 6-month outcomes were still significantly higher than pre-intervention scores.

In conclusion, our findings indicate that the participants of our 5-days education intervention achieved a higher level of educational proficiency. Our participants achieved the required cognitive development. They felt familiar with learning theory and expressed their intention to apply the learned educational principles. Indeed, results of self- and peer-assessment revealed a direct impact of theoretical content on their job-related performance. Furthermore, peer-assessment motivated the participants to collaboratively work on the improvement of their teaching performance. Finally, the participants identified more with the role as a medical educator. We are convinced that this education intervention supported self-reflection of medical educators in their professional environment, promoted collegiality and collaboration within and across traditional discipline boundaries, and exerted an important impact on an effective faculty development (5, 10, 14, 30, 31).

More than 10 years after the initial implementation of the 5-day program, implemented to develop teaching staff in line with the new medical curriculum HeiCuMed, the success of this continuous quality improvement is confirmed by various studies on student evaluations and satisfaction (32, 33).

Acknowledgements

We thank all participants and the Medical Faculty of University of Heidelberg for their contribution to this study. We thank Jelena Zwingmann who provided support in the study design. We thank Jobst-Hendrik Schultz and Andreas Möltner who provided support in some statistical analysis. Finally, we thank Sarah Berger for linguistic improvement.

Authors' contributions

MR designed and carried out the study protocol, performed the statistical analysis, and drafted the manuscript. TS participated in the design and coordination of the study protocol, participated in the statistical analysis, and helped to draft the manuscript. MKa participated in the design of the study and helped to draft the manuscript. MKi, EK, JJ, and VS helped to draft the manuscript.

Conflict of interest and funding

The authors declare no conflict of interest and funding.

References

1. Harden RM, Crosby J. AMEE guide no 20: the good teacher is more than a lecturer - the twelve roles of the teacher. Med Teach 2000; 22: 334-47. Publisher Full Text

2. Higgs J, McAllister L. Educating clinical educators: using a model of the experience of being a clinical educator. Med Teach 2007; 29: e51-7. PubMed Abstract | Publisher Full Text

3. Stone S, Ellers B, Holmes D, Orgren R, Qualters D, Thompson J. Identifying oneself as a teacher: the perceptions of preceptors. Med Educ 2002; 36: 180-5. PubMed Abstract | Publisher Full Text

4. Arthur W, Jr., Bennett W, Jr., Edens PS, Bell ST. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol 2003; 88: 234-45. PubMed Abstract | Publisher Full Text

5. McLean M, Cilliers F, Van Wyk JM. Faculty development: yesterday, today and tomorrow. Med Teach 2008; 30: 555-84. PubMed Abstract | Publisher Full Text

6. Gruppen LD, Simpson D, Searle NS, Robins L, Irby DM, Mullan PB. Educational fellowship programs: common themes and overarching issues. Acad Med 2006; 81: 990-4. PubMed Abstract | Publisher Full Text

7. Searle NS, Hatem CJ, Perkowski L, Wilkerson L. Why invest in an educational fellowship program? Acad Med 2006; 81: 936-40. PubMed Abstract | Publisher Full Text

8. Irby DM, Hekelman FP. Future directions for research on faculty development. Fam Med 1997; 29: 287-9. PubMed Abstract

9. Reid A, Stritter FT, Arndt JE. Assessment of faculty development program outcomes. Fam Med 1997; 29: 242-7. PubMed Abstract

10. Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME guide no. 8. Med Teach 2006; 28: 497-526. PubMed Abstract | Publisher Full Text

11. Kirkpatrick DL. Evaluating training programs: the four levels, 2nd ed. San Francisco, CA: Berrett-Koehler; 1998.

12. Tannenbaum SI, Mathieu JE, Salas E, Cannon-Bowers JA. Meeting trainees' expectations: the influence of training fulfillment on the development of commitment, self-efficacy, and motivation. J Appl Psychol 1991; 76: 759-69. Publisher Full Text

13. Alliger G, Tannenbaum SI, Bennett W, Traver H, Shotland A. A meta-analysis of the relations among training criteria. Person Psychol 1997; 50: 341-58. Publisher Full Text

14. Mann KV, Sutton E, Frank B. Twelve tips for preparing residents as teachers. Med Teach 2007; 29: 301-6. PubMed Abstract | Publisher Full Text

15. Huwendiek S, Kadmon M, Jünger J, Kirschfink M, Bosse H, Resch F, et al. Umsetzung der deutschen Approbationsordnung 2002 im modularen Reformstudiengang Heidelberger Curriculum Medicinale (HeiCuMed). ZFHE 2008; 3: 17-27.

16. Seller H. Studienreform an der Medizinischen Fakultät Heidelberg: Einführung des neuen klinischen Curriculums Heicumed. GMS Z Med Ausbild 2003; 20: 39-42.

17. Steiner T, Jünger J, Schmidt J, Bardenheur H, Kirschfink M, Kadmon M, et al. HEICUMED: Heidelberger Curriculum Medicinale - Ein modularer Reformstudiengang zur Umsetzung der neuen Approbationsordnung. GMS Z Med Ausbild 2003; 20: 87-91.

18. Biggs JB, Tang C. Teaching for quality learning at university: what the student does. Maidenhead: Open University Press; 2007.

19. Sackman H. Delphi-assessment: expert opinion, forecasting and group process. Santa Monica, CA: RAND; 1974.

20. Strauss AL, Corbin J. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage; 1990.

21. Mayring P. Qualitative content analysis. Weinheim: Psychology Verlags Union; 2000.

22. Colquitt JA, LePine JA, Noe RA. Toward an integrative theory of training motivation: a meta-analytic path analysis of 20 years of research. J Appl Psychol 2000; 85: 678-707. PubMed Abstract | Publisher Full Text

23. Bligh J. Faculty development. Med Educ 2005; 39: 120-1. PubMed Abstract | Publisher Full Text

24. Clark JM, Houston TK, Kolodner K, Branch WT, Levine RB, Kern DE. Teaching the teachers. National survey of faculty development in departments of medicine of U.S. teaching hospitals. J Gen Intern Med 2004; 19: 205-14. PubMed Abstract | PubMed Central Full Text | Publisher Full Text

25. Steinert Y, Cruess S, Cruess R, Snell L. Faculty development for teaching and evaluating professionalism: from programme design to curriculum change. Med Educ 2005; 39: 127-36. PubMed Abstract | Publisher Full Text

26. Dahlgren LO, Eriksson BE, Gyllenhammar H, Korkeila M, Saaf-Rothoff A, Wernerson A, et al. To be and to have a critical friend in medical teaching. Med Educ 2006; 40: 72-8. PubMed Abstract | Publisher Full Text

27. MacDougall J, Drummond MJ. The development of medical teachers: an enquiry into the learning histories of 10 experienced medical teachers. Med Educ 2005; 39: 1213-20. PubMed Abstract | Publisher Full Text

28. Hattie J. Visible learning. A synthesis of over 800 meta-analyses relating to achievement. London: Routledge Chapman & Hall; 2008.

29. Marks MB, Wood TJ, Nuth J, Touchie C, O'Brien H, Dugan A. Assessing change in clinical teaching skills: are we up for the challenge? Teach Learn Med 2008; 20: 288-94. PubMed Abstract | Publisher Full Text

30. Knight AM, Carrese JA, Wright SM. Qualitative assessment of the long-term impact of a faculty development programme in teaching skills. Med Educ 2007; 41: 592-600. PubMed Abstract | Publisher Full Text

31. McLeod PJ, Steinert Y. Peer coaching as an approach to faculty development. Med Teach 2009; 31: 1043-4. PubMed Abstract | Publisher Full Text

32. Kadmon G, Schmidt J, De Cono N, Kadmon M. Integrative vs. traditional learning from the student perspective. GMS Z Med Ausbild 2011; 28: Doc28.

33. Kadmon M, Reimann K, Porsche M, Ganschow P. Nachhaltigkeit der verbesserten studentischen Evaluation im operativen Fachgebiet des Reformstudienganges HeiCuMed. GMS Z Med Ausbild 2007; 24: Doc32.