Common issues of systematic reviews in the sports and exercise medicine field

Abstract

Correspondence to Aamir Raoof Memon; [email protected] Systematic reviews that include meta-analysis, and in particular meta-analysis of randomised controlled trials, with the exception of umbrella and rapid reviews,1 are at the top of hierarchy of evidence, as these studies apply the scientific method to identify high-quality evidence, such as randomised controlled trials, and synthesise this evidence via meta-analytical methods that afford greater statistical power than any one study alone.2 3 Subsequently, systematic reviews underscore meta-analyses that are commonly relied upon during the development of policy and clinical practice guidelines. Duplicate publications in systematic reviews The presence of multiple reports from one study impacts evidence synthesis and effect estimates.17 This often happens when resulting reports from the same study do not cite prior reports and/or clinical trial registry identifiers.18 19 The Cochrane Handbook provides guidance on this in the section 5.2.1 and suggests ways in which authors can identify when multiple reports emanate from one study, such as trial registration numbers, authors’ names, numbers of participants and baseline data.7 A related problem is the inclusion of participants from the same intervention (or control) group based on data from multiple reports of one study. Potential solutions to address this issue are available in the Cochrane Handbook in the section 23.3.4 and include combining groups to create a single pairwise comparison and selecting one pair of interventions and excluding the others.7 Once again, we contend considering these methods a priori and ensuring these are integrated into the synthesis component of any systematic review. [...]reviews should be prospectively registered (eg, PROSPERO) and should report the risk of bias assessment.20 However, many systematic reviews do not adhere to these reporting guidelines, despite commonly including a completed checklist as a supplement accompanying the final publication and/or review process.20 21 Specifically from our experience in the sports and exercise medicine field, common examples of PRISMA not being followed include, but are not restricted to: (a) providing list of excluded studies at full text with reasons; (b) providing access to the statistical code, raw tabulated data and extraction sheet such as on a data-sharing platform (eg, osf.io); (c) transparently reporting deviations to the a priori study protocol; (d) reporting the search strategy in full22 and (e) structuring the abstract in accordance with the relevant extensions of PRISMA statement (http://prisma-statement.org/Extensions/Abstracts).6 Including these elements not only facilitates the peer-review process insofar that reviewers are able to provide higher-level feedback and editors are able to reach a decision in a more timely manner, yet also conveys the methods and results of the systematic review in a more transparent and interpretable format to expected end users (eg, researchers, clinicians and policymakers). [...]we recommend authors: (a) familiarise themselves with not only the PRISMA checklist, but also the supporting documentation regarding its implementation,23 (b) design and draft the systematic review through the lens of the PRISMA checklist, rather than applying it post hoc, and (c) engage with the relevant extensions, such as that relevant to abstracts (http://prisma-statement.org/Extensions/Abstracts).6 Grey literature The term ‘grey literature’ is used to describe reports published outside of traditional commercial publishing, such as dissertations, preprints, conference abstracts and reports.7 This is an important step as part of secondary searches which helps reduce the risk of publication bias.

Correspondence to Aamir Raoof Memon; [email protected]

Systematic reviews that include meta-analysis, and in particular meta-analysis of randomised controlled trials, with the exception of umbrella and rapid reviews,¹ are at the top of hierarchy of evidence, as these studies apply the scientific method to identify high-quality evidence, such as randomised controlled trials, and synthesise this evidence via meta-analytical methods that afford greater statistical power than any one study alone.^{2 3} Subsequently, systematic reviews underscore meta-analyses that are commonly relied upon during the development of policy and clinical practice guidelines. Therefore, it is important that systematic reviews are ethically and methodologically robust. While methodological guideline for conducting systematic reviews may be obtained from various sources such as Cochrane Collaboration, Campbell Collaboration and the Joanna Briggs Institute,^4–7 potential oversight by authors might sometimes compromise the quality of a systematic review. Anecdotally, we have frequently observed such issues in our experience as reviewers and editors in the sports and exercise medicine field. Hence, this editorial discusses some issues that might compromise the robustness of systematic reviews. We also provide suggestions for researchers to avoid such issues in the future. We hope this editorial will serve as an educational guide and a step towards improving the quality of systematic reviews in the sports and exercise medicine field.

Reports from predatory journals in systematic reviews

The presence of reports from potential predatory or questionable journals in systematic reviews is a major methodological concern because these data may bias results. Deciding whether a journal is legitimate or potentially predatory is a complex decision and often subjective, especially considering the grey zone created by legitimate, although comparatively low-quality, content-specific regional journals.⁸ It might be argued that papers published in predatory journals may be of sufficient quality. For example, this might hold value in case of hijacked journals.⁹ However, publishing a paper in predatory journals reflects lack of knowledge and careful consideration of target journal by the authors. Experiments performed previously have shown that predatory journals are likely to accept anything irrespective of the quality of the paper.¹⁰ It should also be noted that some authors deliberately publish in predatory journals making it rather a symbiotic relationship.¹¹ Subsequently, discussion specifically on the quality of papers published in such journals is complex and beyond the scope of this editorial.

There are several approaches that may aid identifying these predatory or questionable journals, including, yet not limited to, Think-Check-Submit (https://thinkchecksubmit.org) Initiative and Open Access Journal Quality Indicators (https://www.gvsu.edu/library/sc/open-access-journal-quality-indicators-5.htm).^{8 12 13} Once identified, the ethical quandary of how to handle these reports is similarly surrounded by layers of complexity. For instance, some authors (1) might exclude records from journals in the grey zone (poor-quality regional journals) whereby potential records might be missed, or (2) might include records from a journal they consider legitimate rather than predatory and ultimately end up including such studies. Notably, forward and backward citation tracking is recommended when conducting a systematic review, yet one common method of using Google Scholar increases the potential of finding publications from predatory journals given the breadth of indexing.¹⁴ For instance, one recent review reported that reports from predatory journals are often cited in systematic reviews in health sciences.¹⁵ Given that existing systematic review guidelines, such as Cochrane Collaboration, Campbell Collaboration and the Joanna Briggs Institute, have no clear guidance on how to deal with reports from potentially predatory journals, we suggest authors establish the following prior to conducting a systematic review:

• Criteria to define a journal as ‘potentially predatory’, which can be achieved in light of many resources describing their characteristics.^{8 13 16}

• Whether reports identified as ‘potentially predatory’ will be included in primary syntheses.

• Whether sensitivity analyses will be employed to evaluate the impact of including ‘potentially predatory’ reports.

Similar to how search strategies and intended methods of synthesising effect estimates are registered a priori, we contend this should also apply to methods for identifying and handling ‘potentially predatory’ reports.

Duplicate publications in systematic reviews

The presence of multiple reports from one study impacts evidence synthesis and effect estimates.¹⁷ This often happens when resulting reports from the same study do not cite prior reports and/or clinical trial registry identifiers.^{18 19} The Cochrane Handbook provides guidance on this in the section 5.2.1 and suggests ways in which authors can identify when multiple reports emanate from one study, such as trial registration numbers, authors’ names, numbers of participants and baseline data.⁷ A related problem is the inclusion of participants from the same intervention (or control) group based on data from multiple reports of one study. Potential solutions to address this issue are available in the Cochrane Handbook in the section 23.3.4 and include combining groups to create a single pairwise comparison and selecting one pair of interventions and excluding the others.⁷ Once again, we contend considering these methods a priori and ensuring these are integrated into the synthesis component of any systematic review.

Reporting recommendations for systematic reviews

Journals suggest authors to use Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) checklist,⁶ which ultimately improves the structure and transparency of reporting. For example, reviews should be prospectively registered (eg, PROSPERO) and should report the risk of bias assessment.²⁰ However, many systematic reviews do not adhere to these reporting guidelines, despite commonly including a completed checklist as a supplement accompanying the final publication and/or review process.^{20 21}

Specifically from our experience in the sports and exercise medicine field, common examples of PRISMA not being followed include, but are not restricted to: (a) providing list of excluded studies at full text with reasons; (b) providing access to the statistical code, raw tabulated data and extraction sheet such as on a data-sharing platform (eg, osf.io); (c) transparently reporting deviations to the a priori study protocol; (d) reporting the search strategy in full²² and (e) structuring the abstract in accordance with the relevant extensions of PRISMA statement (http://prisma-statement.org/Extensions/Abstracts).⁶ Including these elements not only facilitates the peer-review process insofar that reviewers are able to provide higher-level feedback and editors are able to reach a decision in a more timely manner, yet also conveys the methods and results of the systematic review in a more transparent and interpretable format to expected end users (eg, researchers, clinicians and policymakers). Therefore, we recommend authors: (a) familiarise themselves with not only the PRISMA checklist, but also the supporting documentation regarding its implementation,²³ (b) design and draft the systematic review through the lens of the PRISMA checklist, rather than applying it post hoc, and (c) engage with the relevant extensions, such as that relevant to abstracts (http://prisma-statement.org/Extensions/Abstracts).⁶

Grey literature

The term ‘grey literature’ is used to describe reports published outside of traditional commercial publishing, such as dissertations, preprints, conference abstracts and reports.⁷ This is an important step as part of secondary searches which helps reduce the risk of publication bias. This step is crucial in cases where literature is expected to be small (eg, qualitative studies). However, these studies may be unrepresentative sample of all unpublished studies. The Cochrane Handbook provides guidance on this in the sections 3.4, 4.3.5 and 21.7 and provides guidance on how to deal with grey literature.⁷

In conclusion, authors of systematic reviews in the field of sports and exercise medicine are encouraged to check the methodological guideline for conducting systematic reviews from resources such as Cochrane Collaboration, Campbell Collaboration and the Joanna Briggs Institute Reports. Authors of systematic reviews should also consider issues related to predatory journals in systematic reviews, duplicate publications in systematic reviews and reporting recommendations for systematic reviews in light of this editorial.

Ethics statements

Patient consent for publication

Not required.

Ethics approval

Not applicable.

Footnote

Twitter @DptAamir, @PatrickOwenPhD, @Sportmednews, @evertverhagen, @NiamhMundell, @belavyprof

Contributors ARM, PJO and DLB—conception and conceptualisation. All authors contributed to the write-up and revision of the manuscript.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests All authors, except NLM, are on the editorial board of BMJ Open Sport and Exercise Medicine.

Provenance and peer review Commissioned; externally peer reviewed.

References

¹ Munn Z, Pollock D, Barker TH, et al. The dark side of rapid reviews: a retreat from systematic approaches and the need for clear expectations and reporting. Ann Intern Med 2023; 176: 266–7. doi:10.7326/M22-2603

² Fusar-Poli P, Radua J. Ten simple rules for conducting umbrella reviews. Evid Based Ment Health 2018; 21: 95–100. doi:10.1136/ebmental-2018-300014

³ Murad MH, Asi N, Alsawas M, et al. New evidence pyramid. Evid Based Med 2016; 21: 125–7. doi:10.1136/ebmed-2016-110401

⁴ Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs 2014; 114: 53–8. doi:10.1097/01.NAJ.0000444496.24228.2c

⁵ Cumpston MS, McKenzie JE, Welch VA, et al. Strengthening systematic reviews in public health: guidance in the cochrane handbook for systematic reviews of interventions. J Public Health (Oxf) 2022; 44: e588–92. doi:10.1093/pubmed/fdac036

⁶ Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021; 372. doi:10.1136/bmj.n71

⁷ Higgins JPT, Thomas J. Cochrane Handbook for Systematic Reviews of Interventions version 6.4. The Cochrane Collaboration, 2023.

⁸ Memon AR. Revisiting the term predatory open access publishing. J Korean Med Sci 2019; 34: e99. doi:10.3346/jkms.2019.34.e99

⁹ Memon AR. Hijacked journals: a challenge unaddressed to the developing world. J Pak Med Assoc 2019; 69: 1413–5.

¹⁰ Bohannon J. Who’s afraid of peer review Science 2013; 342: 60–5. doi:10.1126/science.2013.342.6154.342_60

¹¹ Eriksson S, Helgesson G. Time to stop talking about ‘predatory journals. Learned Publishing 2018; 31: 181–3. doi:10.1002/leap.1135

¹² Memon AR. Research publications and education in Pakistani medical universities: avoiding predatory journals and improving the quality of research. J Pak Med Assoc 2017; 67: 830–3.

¹³ Memon AR. Predatory journals spamming for publications: what should researchers do Sci Eng Ethics 2018; 24: 1617–39. doi:10.1007/s11948-017-9955-6

¹⁴ Munn Z, Barker T, Stern C, et al. Should I include studies from ‘predatory’ journals in a systematic review? Interim guidance for systematic reviewers. JBI Evid Synth 2021; 19: 1915–23. doi:10.11124/JBIES-21-00138

¹⁵ Ross-White A, Godfrey CM, Sears KA, et al. Predatory publications in evidence syntheses. J Med Libr Assoc 2019; 107: 57–61. doi:10.5195/jmla.2019.491

¹⁶ Cukier S, Helal L, Rice DB, et al. Checklists to detect potential predatory biomedical journals: a systematic review. BMC Med 2020; 18: 104. doi:10.1186/s12916-020-01566-1

¹⁷ Wager E, Wiffen PJ. Ethical issues in preparing and publishing systematic reviews. J Evid Based Med 2011; 4: 130–4. doi:10.1111/j.1756-5391.2011.01122.x

¹⁸ Choi W-S, Song S-W, Ock S-M, et al. Duplicate publication of articles used in meta-analysis in Korea. Springerplus 2014; 3: 182. doi:10.1186/2193-1801-3-182

¹⁹ von Elm E, Poglia G, Walder B, et al. Different patterns of duplicate publication: an analysis of articles used in systematic reviews. JAMA 2004; 291: 974–80. doi:10.1001/jama.291.8.974

²⁰ Memon AR, Stanton R, To Q, et al. Sedentary behaviour research in adults: a scoping review of systematic reviews and meta-analyses. J Sports Sci 2021; 39: 2219–31. doi:10.1080/02640414.2021.1928382

²¹ Page MJ, Shamseer L, Altman DG, et al. Epidemiology and reporting characteristics of systematic reviews of BIOMEDICAL research: a cross-sectional study. PLoS Med 2016; 13. doi:10.1371/journal.pmed.1002028

²² Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Syst Rev 2021; 10: 39. doi:10.1186/s13643-020-01542-z

²³ Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ 2021; 372. doi:10.1136/bmj.n160