Designing Direct Observation of Procedural Skills (DOPS) Test for Selective Skills of Orthopedic Residents and Evaluating Its Effects from Their Points of View

Headnote

Abstract

Introduction: Direct Observation of Procedural Skills (DOPS) is a valuable method to evaluate procedural skills. The aim of this study was to develop a DOPS test for assessment of first- and second-year orthopedic residents and evaluate its effects on their learning.

Methods: Seven residents and nine faculty members of the orthopedic department of Tabriz University of Medical Sciences participated in this study. A questionnaire containing twelve closed and four open questions was used for assessment. The acquired data were analyzed by descriptive statistics (frequency, percent, mean and standard deviation).

Results: The results showed that residents' performances were almost good (mean of good performances = 50.6%), the participants' performances increased in the second stage (from 50.6% to 59.4%) but this increasing performance decreased in the third stage (from 59.4% to 39%). Most faculty members and residents believed that DOPS tests can have an effective role in facilitating students' learning and also can help them to succeed in their final test. Some of the residents believed that DOPS tests decrease their stress in the final exam. Promoting students' procedural skills and independent learning are the pros and a stressful test experience is the con of this study.

Conclusion: According to results of this study, DOPS tests had an effective role in facilitating students' learning and skills. Test repetition for the second time is sufficient and useful for evaluating residents' clinical and instrumental skills.

Article info

Article Type:

Original Research

Article History:

Received: 16 Feb 2015

Accepted: 14 Dec 2015

ePublished: 1 Feb 2016

Keywords:

Assessment methods, Direct Observation of Procedural Skills (DOPs), Clinical skills, Orthopedic resident, Tabriz University of Medical Sciences

Introduction

Clinical education and assessment are very important in medical education and provide an exceptional opportunity for teaching required medical skills and promoting procedural skills in interactions with patients.1 Today in medical education, two important features are considered more than in the past: an emphasis on evaluation as a tool for ensuring the quality of education programs and moving toward outcome-based education in which the outcomes of learning are defined and content is developed according to the outcomes.2,3 Effective and appropriate assessment of students' clinical competences and skills have been one of the most important and difficult duties of faculty members, because usually students' clinical assessments are done by investigating their thinking capabilities or based on a teacher's subjective understanding of a student's activity.4-6 Regarding the necessity of assessment, it is required to use methods and tools of formative assessment for clinical skills in order to help students to learn and be prepared for final tests. One of the common methods for assessing procedural and instrumental skills is Direct Observation of Procedural Skills (DOPS), which involves the direct observation of students using clinical skills. In this method, the evaluator's observations are registered based on a checklist and real findings will be reflected to students. Depending on the main required skills for learning, the number of tests differs and can be up to eight tests in one period.7-8 DOPS is an opportunity to provide students with constructive feedback, concentration and focus for using required skills, because evaluation, with the purpose of promoting performance, needs on-time and specific feedback.3 Beard et al. in 20059 and Wiles et al. in 200710 confirmed the results of DOPS.11 Also, based on the results of a study in Imperial College London, the reliability and validity of this method are appropriate in evaluating clinical processes.12

Regarding the importance of applying appropriate and effective methods of assessment and the lack of studies in this field in Iran, the present study was designed and performed with the purpose of developing DOPS checklists for selective skills of first- and second-year orthopedic residents and evaluating its effects on their learning at Tabriz University of Medical Sciences.

Materials and Methods

This is a mixed (quantitative and qualitative) study that was designed and performed in Shohada Hospital of Tabriz University of Medical Sciences in 2014. We used concuss method. Our inclusion critera study is all first- and secondyear orthopedic residents.

In order to design and perform DOPS tests, seven essential skills were selected by academic staff:

* preparing patient on the bed of operating room for an elective orthopedic surgery

* managing deep ulcer of limb

* treating dislocations and fractures

* all kinds of plaster cast of limbs

* hand washing in operation room

* debridement and stitching

* using pneumatic tourniquet

Evaluation checklists were prepared for each skill by taking into consideration the related literature and orthopedic faculty members' ideas. The checklists were used after determining validity. Every test was repeated three times. The content validity of the researcher-designed questionnaire was determined by experts and nine orthopedic faculty members. The Content Validity Index (CVI) was 0.95 and the test-retest method was (ICC=0.85). The reliability of the questionnaire was confirmed by Cronbach's Alpha, which was α=0.8. Faculty members' and residents' opinions were evaluated about the effects of the test on facilitating learning and skills and also helping them for preparation and success in the final Objective Structured Clinical Examination (OSCE). This questionnaire consisted of twelve closed questions and four open questions.

The data were analyzed by descriptive methods of statistics (frequency, percent, mean and standard deviation).

In this research, content analysis was applied manually to analyze qualitative data. Content analysis is a qualitative approach used to recognize content in data. The themes are related to backgrounds and patterns that have the features of repeatability and coding.

Ethical considerations

The research Ethics Committee of Tabriz University of Medical Sciences approved the study protocol, all the participants provided informed consent and Participation was voluntary.

Results

The mean age of faculty members was 43±5.21 Four individuals were assistant faculty members, four of them were associate professors and one was a full professor. The mean years they had been working was 7.3±8.10. The mean age of residents was 28.7±1.2.

According to the DOPS test results, the overall performance of residents was almost good. Also, the second testing was useful for residents with good performance, but repetition of DOPS for the third time was only useful for the residents with weak performance. We suggest the DOPS test to be repeated for the third time only for residents with weak performance results the second time. (Figure 1)

In this study, we asked nine orthopedic faculty members and seven first- and second-year orthopedic residents of Tabriz University of Medical Sciences about the effects of DOPS in facilitating students' learning. Results are shown in Table 1.

As Table 1 shows, most participants believed that DOPS tests can have an effective role in facilitating practiced learning and also helping them to be prepared for the final exam. Also, most of them believed that DOPS tests are effective in increasing students' procedural skills. DOPS tests provide opportunity for reflection and self-assessment, and some participants believed that DOPS tests decrease students' stress in final exams. The results of the qualitative analysis of the open questionnaire form about DOPS tests are presented in Table 2.

Discussion

*The results of a study by Bazrafkan et al. in Shiraz showed that 87.6% of students had an acceptable performance in DOPS tests, which are in concordance with the results of this study. Also, the researchers in that study showed that DOPS tests can be used as an effective and useful method for evaluating students.13 The results of a study by Bagheri et al., who investigated the effect of evaluation on emergency medical students' learning clinical skills by DOPS in Mashhad, showed that students had appropriately good performance and evaluation by DOPS had a significant effect on students' learning.14 One of the probable reasons for students' good scores in DOPS tests can be motivation that is caused by the DOPS test. The results of evaluations in different stages in this study showed that, generally, there is an increasing trend in residents' performance from the first stage to the second stage, but a decreasing trend from the second stage to the third stage. In some residents' opinions, the third test was not necessary. In a study by Akbari and Mahavelati Shamsabadi, who investigated the advantages of DOPS tests in students of restorative dentistry, 86% of students believed that two stages of tests in each period were sufficient.15 In a study by Khoshrang et al., who investigated residents' viewpoints about evaluating procedural skills by DOPS in Gilan University of Medical Sciences, every resident was evaluated four to six times in a year by DOPS and more than half of them were not satisfied with the number of times the test was held.16 In a study by Wiles et al. in the neurology ward of Imperial College London showed almost similar results.10 According to Amin, all the evaluated trends are defined based on required cases of competency in each period and tests be repeated up to eight times in each period.6 It seems that more repetition of exams makes evaluators tired, which could discourage them from participating. However, the number of tests should be appropriated by the content of the course and test. It seems that more tests can be used in areas with complicated and widespread content.

*Most faculty members and residents believe that appropriate use of DOPS tests and feedback to students can have a great effect on promoting students' skill and competence. Also, the results of different studies showed that DOPS tests have a great role in promoting students' learning and competence. In their study on medical students in the gynecology ward of Kurdistan University of Medical Sciences, Shahgheibi et al. showed that DOPS tests are very capable in promoting students' learning and clinical skills.17 In a study on surgery residents of Malaysia University, Shahid Hassan also considered DOPS tests as highly capable in promoting students' performance and learning.18 Also, Tsui et al., in their study in Taiwan, declared that these types of tests have a great role in promoting medical students' skill and competence.19 Chen et al. showed that DOPS tests in senior students of medicine promoted self-reporting, skill and self-confidence.20 Thus, it seems that DOPS tests can be applied as an appropriate method of evaluation, and this type of evaluation can be used as an educational tool in teaching and improving students' competence.

*In this study, one of the advantages of DOPS tests, according to both faculty members and residents, is students' freedom and independence during evaluation. In a study by Akbari et al., 71% of students stated that DOPS tests have a positive effect on students' learning and independence and because of that they requested continuing evaluation by DOPS tests.15 Also, a study by Sahebalzamani et al. showed that nurses were highly satisfied with the independence and feedback during DOPS tests.11

One of the weak points of DOPS tests in the participants' points of view in this study was the unequal holding of tests for all the participants or, in other words, a lack of justice among different participants. The reason for this can be the differences between patients and different evaluators. This subject makes clear the necessity of paying attention to the reliability and validity of these types of tests. Wilkinson et al. evaluated the validity of DOPS tests and it was good.21 In addition, Moorthy et al. confirmed the validity of these types of tests in surgery.22 Bould et al. evaluated the validity of these tests in anesthesia and it was high as well.23 Regarding the reliability of these tests, Weller et al. got high reliability for these tests.24 Other studies noted acceptable reliability too.25,26

*The above problem can be reduced by teaching residents and evaluators, repeating tests and justifying learners.

*Another weak point of the present study is lack of a comparative group.

We are suggesting future investigators to study more students and all procedural skills of residents.

Conclusion

Regarding the importance of clinical education, effective methods are needed in order to evaluate students' practical learning. The results of evaluating orthopedic residents in this study showed that they had appropriate performance. According to the results of this study, repeating the tests for the second time is suitable. The strong points of DOPS tests were evaluating and promoting students' procedural skills and independence during the test, and the weak points of these tests from participants' points of view were different interactions with students and stressful tests. Considering the positive effect of DOPS in decreasing stress and increasing the results of final exams, we can modify the exam environment for improvement. Some of the most important suggestions of participants for promoting the results of DOPS tests were video recording instead of faculty members' observation, giving continuous tests, teaching the principles and methods of DOPS and informing the subject of tests before giving them. Developing and using DOPS for all clinical students is recommended and can help them to learn better and be prepared for final exams.

Study limitations

The number of residents was limited and a repetition of the study is needed for other residents or other orthopedic departments.

Competing interests

The authors declare that there is no conflict of interest.

Acknowledgements

Special thanks go to all the orthopedic faculty members, residents and other members of the group of Tabriz University of Medical Sciences and also to all the individuals who helped us in carrying out this study.

Sidebar

Citation: Amini A, Shirzad F, Mohseni MA, Sadeghpour A, Elmi A. Designing Direct Observation of Procedural Skills (DOPS) Test for Selective Skills of Orthopedic Residents and Evaluating Its Effects from Their Points of View. Res Dev Med Educ 2015;4(2):147-152. doi:10.15171/rdme.2015.026

References

References

1. Hoseini BL, Jafarnejad F, Mazlom SR, Foroghi pour M, karimi Mouneghi H. Midwifery Students' Satisfaction with Logbook as a Clinical Assessment Means in Mashhad University of Medical Sciences, 2010. Iranian Journal of Medical Education 2012;11(8): 933-941. [In Persian].

2. rank JR, editor. The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better care. Ottawa: The Royal College of Physicians and Surgeons of Canada; 2005. Available from: http://rcpsc. medical.org/canmeds/CanMEDS20705/index.php.

3. Shumway JM, Harden RM; Association for Medical Education in Europe.. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach 2003;25(6):569-584. doi:10.1080/0142159032000151907

4. Ghojazadeh M, Aghaei MH, Naghavi-Behzad M, Piri R, Hazrati H, Azami-Aghdas S. Using Concept Maps for Nursing Education in Iran: A Systematic Review. Res Dev Med Educ 2014;3(1):67-72. doi:10.5681/rdme.2014.014

5. Sahebalzamani M, Farahani H, Jahantigh M. Validity and Reliability of Direct Observation of Procedural Skills in Evaluating the Clinical Skills of Nursing Students of Zahedan Nursing and Midwifery School. Zahedan Journal of Research in Medical Science 2012;14(2):76-81. [In Persian].

6. Amin Z, Chong Y, Khoo H. Direct observation of procedural skills(chapter16). In: Amin Z, Chong Y, Khoo H. Practical guid to medical student. 1st ed. Singapore: World Scientific Printers CO; 2006. 71-4.

7. Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the united Kingdom. Med Educ. 2008 Apr;42(4):364-73. doi:10.1111/j.1365- 2923.2008.03010.x

8. Norcini JJ, Mackinley DW. Assessment methods in medical education. Teach Teacher Educ 2007; 23(3):239-250. doi:10.1016/j.tate.2006.12.021

9. Beard JD, Jolly BC, Newble DI, Thomas WE, Donnelly J, Southgate LJ. Assessing the technical skills of surgical trainees. Br J Surg 2005;92(6):778-782. doi:10.1002/ bjs.4951

10. Wiles CM, Dawson K, Hughes TA, Llewelyn JG, Morris HR, Pickersgill TP, Robertson NP, Smith PE. Clinical skills evaluation of trainees in a neurology department. Clin Med (Lond) 2007;7(4):365-369. doi:10.7861/ clinmedicine.7-4-365

11. Kuhpayehzade J, Hemati A , Baradaran HR, Mir Hosseni F, Akbari H, Sarvieh M. Validity and reliability of direct observation of procedural skills in evaluating clinical skills of midwifery students of Kashan Nursing and Midwifery School. Journal of Sabzevar University of Medical Sciences 2014;21( 1):145-154.

12. Wilkinson J, Benjamin A, Wade W. Assessing the performance of doctors in training. BMJ 2003;327(7416):s91-2. doi:10.1136/bmj.327.7416.s91

13. Bazrafkan L, Shokrpour N, Torabi K. Comparison of the Assessment of Dental Students. Journal of Medical Education 2009;13(1,2):16-23.

14. Bagher M, Sadeghnezhad M, Sayyadee T, Hajiabadi F. The Effect of Direct Observation of Procedural Skills (DOPS) Evaluation Method on Learning Clinical Skills among Emergency Medicine Students. Iranian Journal of Medical Education 2014;13(12):1073-1081. [In Persian].

15. Akbari M, Mahavelati Shamsabadi R. Direct Observation of Procedural Skills (DOPS) in Restorative Dentistry: Advantages and Disadvantages in Student's Point of View. Iranian Journal of Medical Education 2013;13(3):212-220. [In Persian].

16. khoshrang H. Assistants specialized perspectives on evaluation method of practical skills observations (DOPS) in Guilan University of Medical Sciences. Journal of Research in Medical Education 2011;2(2):40-44. [In Persian].

17. Shahgheibi S, Pooladi A, BahramRezaie M, Farhadifar F, Khatibi R. Evaluation of the Effects of Direct Observation of Procedural Skills (DOPS) on Clinical Externship Students' Learning Level in Obstetrics Ward of Kurdistan University of Medical Sciences. Journal of Medicine Eduction 2009;13(1,2):29-33.

18. Hassan S. Faculty Development: DOPS as Workplace- Based Assessment. Education in Medical Journal 2011;3(1):32-43. doi:10.5959/eimj.v3i1.94

19. Tsui KH, Liu CY, Lui JM, Lee ST, Tan RP, Chang PL. Direct observation of procedural skills to improve validity of students' measurement of prostate volume in predicting treatment outcomes. Urological Science 2013;24(3):84-88. doi:10.1016/j.urols.2012.09.001

20. Chen W, Liao SC, Tsai CH, Huang CC, Lin CC, Tsai CH. Clinical Skills in Final-year Medical Students: The Relationship between Self-reported Confidence and Direct Observation by Faculty or Residents. Ann Acad Med Singapore 2008;37(1):3-8.

21. Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implemnting workplace-based assessment across the medical specialities in the united Kingdon. Med Educ 2008;42(4):364-73. doi:10.1111/j.1365- 2923.2008.03010.x

22. Moorthy K, Munz Y, Sarker S, Darzi A. Objective assessment of technical skills in surgery. BMJ. 2003;327(7422):1032-7. doi:10.1136/bmj.327.7422.1032

23. Bould MD, Crabtree NA, Naik VN. Assessment of procedural skills in anaesthesia. Br J Anaesth 2009;103(4):472-83. doi:10.1093/bja/aep241

24. Weller JM, Jolly B, Misur MP, Merry AF, Jones A, Crossley JG, Pedersen K, Smith K. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth 2009;102(5):633-41. doi:10.1093/bja/aep055

25. Al-Naami MY. Reliability, validity, and feasibility of the Objective Structured Clinical Examination in assessing clinical skills of final year surgical clerkship. Saudi Med J 2008;29(12):1802-1807.

26. Tudiver F1, Rose D, Banks B, Pfortmiller D. Reliability and validity testing of an evidence-based medicine OSCE station. Fam Med 2009;41(2):89-91.