Validity of using a semi-automated screening tool in a systematic review assessing non-specific effects of respiratory vaccines

Abstract

Background

The abstract screening process of systematic reviews can take thousands of hours by two researchers. We aim to determine the reliability and validity of Research Screener, a semi-automated abstract screening tool within a systematic review on non-specific and broader effects of respiratory vaccines on acute lower respiratory infection hospitalisations and antimicrobial prescribing patterns in young children.

Methods

We searched online databases for Medline, Embase, CINAHL, Scopus and ClinicalTrials.gov from inception until 24th January 2024. We included human studies involving non-specific and broader effects of respiratory vaccines and excluded studies investigating live-attenuated vaccines. The RS trial compared relevant abstracts flagged by RS to manual screening. RS ranks abstracts by relevance based on seed articles used to validate the search strategy. Abstracts are re-ranked following reviewers’ feedback. Two reviewers screened RS independently with a third reviewer resolving conflicts; three reviewers screened manually with a fourth reviewer resolving conflicts.

Results

After removal of duplicates, 9,727 articles were identified for abstract screening. Of those, 3,000 were randomly selected for screening in RS, with 18% (540) screened in RS and 100% manually. In RS, 99 relevant articles were identified. After comparing RS to manual screening and completing full-text review on 26 articles not captured by RS, 4 articles were missed by RS (2 due to human error, 2 not yet screened). Hence, RS captured articles accurately whilst reducing the screening load.

Conclusions

RS is a valid and reliable tool that reduces the amount of time spent screening articles for large-scale systematic reviews. RS is a useful tool that should be considered for streamlining the process of systematic reviews.