Balancing workload with sensitivity to efficiently identify randomised controlled trials in an education systematic review

Abstract

There is increasing interest in improving the efficiency of systematic review production, yet there is limited literature considering its application within the education field. This article analyses the study identification process adopted in a systematic review on effective teacher professional development, which identified 121 randomised controlled trials. It considers both human and technological inputs that aided production. It draws on project notes, an analysis of database sources and terminology used to identify randomised controlled trials, a retrospective evaluation of useful search terms and an analysis of using machine learning to reduce human workload during eligibility screening of citation records. Study identification was aided by four team processes (relating to ways of working and understanding the review context), the choice of information sources spanning education, psychology and economics research, and a variety of search terms for randomised controlled trials. The search resulted in 5,527 records identified from the main searches, and a further 3,614 records from forward and backward citation searching from the 121 included randomised controlled trials. Machine learning reduced screening workload, but implementation challenges included decisions on when to cease manual screening. In conclusion, carefully planned literature searches combined with machine learning to support eligibility screening can provide workload savings for sensitive study identification of randomised controlled trials in education. Improved reporting of randomised controlled trial design within research would aid these processes. Tools could also be developed to aid implementation of machine learning.