Abstract

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77–94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

The CONSORT-AI extension was developed to provide specific guidance for randomised controlled trials involving Artificial Intelligence (AI) interventions. Here, the authors show that since publication of CONSORT-AI, several AI-specific considerations remain systematically underreported.

Details

Title
Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines
Author
Martindale, Alexander P. L. 1   VIAFID ORCID Logo  ; Llewellyn, Carrie D. 2 ; de Visser, Richard O. 2 ; Ng, Benjamin 3   VIAFID ORCID Logo  ; Ngai, Victoria 4   VIAFID ORCID Logo  ; Kale, Aditya U. 5 ; di Ruffano, Lavinia Ferrante 6   VIAFID ORCID Logo  ; Golub, Robert M. 7 ; Collins, Gary S. 8   VIAFID ORCID Logo  ; Moher, David 9   VIAFID ORCID Logo  ; McCradden, Melissa D. 10   VIAFID ORCID Logo  ; Oakden-Rayner, Lauren 11 ; Rivera, Samantha Cruz 12 ; Calvert, Melanie 13   VIAFID ORCID Logo  ; Kelly, Christopher J. 14 ; Lee, Cecilia S. 15   VIAFID ORCID Logo  ; Yau, Christopher 16 ; Chan, An-Wen 17 ; Keane, Pearse A. 18   VIAFID ORCID Logo  ; Beam, Andrew L. 19   VIAFID ORCID Logo  ; Denniston, Alastair K. 20   VIAFID ORCID Logo  ; Liu, Xiaoxuan 21   VIAFID ORCID Logo 

1  Brighton and Sussex Medical School, Brighton, UK (GRID:grid.414601.6) (ISNI:0000 0000 8853 076X) 
2  Brighton and Sussex Medical School, Department of Primary Care and Public Health, Brighton, UK (GRID:grid.414601.6) (ISNI:0000 0000 8853 076X) 
3  Sandwell and West Birmingham NHS Trust, Birmingham and Midland Eye Centre, Birmingham, UK (GRID:grid.414513.6) (ISNI:0000 0004 0399 8996); University of Oxford, Christ Church, Oxford, UK (GRID:grid.4991.5) (ISNI:0000 0004 1936 8948) 
4  University College London Medical School, London, UK (GRID:grid.83440.3b) (ISNI:0000 0001 2190 1201) 
5  University of Birmingham, Institute of Inflammation and Ageing, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK (GRID:grid.412563.7) (ISNI:0000 0004 0376 6589); University of Birmingham, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486) 
6  University of York, York Health Economics Consortium, Birmingham, UK (GRID:grid.5685.e) (ISNI:0000 0004 1936 9668) 
7  Northwestern University Feinberg School of Medicine, Chicago, USA (GRID:grid.16753.36) (ISNI:0000 0001 2299 3507) 
8  University of Oxford, Centre for Statistics in Medicine//UK EQUATOR Centre, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, Oxford, UK (GRID:grid.4991.5) (ISNI:0000 0004 1936 8948) 
9  Ottawa Hospital Research Institute, Centre for Journalology, Clinical Epidemiology Program, Ottowa, Canada (GRID:grid.4991.5) (ISNI:0000 0004 0500 0659) 
10  The Hospital for Sick Children, Department of Bioethics, Toronto, Canada (GRID:grid.42327.30) (ISNI:0000 0004 0473 9646); Peter Gilgan Centre for Research & Learning, Genetics & Genome Biology Research Program, Toronto, Canada (GRID:grid.42327.30); Dalla Lana School of Public Health, Division of Clinical and Public Health, Toronto, Canada (GRID:grid.17063.33) (ISNI:0000 0001 2157 2938) 
11  University of Adelaide, Australian Institute for Machine Learning, Adelaide, Australia (GRID:grid.1010.0) (ISNI:0000 0004 1936 7304) 
12  University of Birmingham, Birmingham Health Partners Centre for Regulatory Science and Innovation, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University of Birmingham, Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486) 
13  University of Birmingham, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University of Birmingham, Birmingham Health Partners Centre for Regulatory Science and Innovation, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University of Birmingham, Centre for Patient Reported Outcomes Research (CPROR), Institute of Applied Health Research, College of Medical and Dental Sciences, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University of Birmingham, NIHR Applied Research Collaboration (ARC) West Midlands, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University of Birmingham, NIHR Blood and Transplant Research Unit (BTRU) in Precision Transplant and Cellular Therapeutics, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486) 
14  Google Health, London, UK (GRID:grid.498398.7) (ISNI:0000 0004 6043 9189) 
15  University of Washington, Seattle, USA (GRID:grid.34477.33) (ISNI:0000 0001 2298 6657) 
16  University of Oxford, Nuffield Department of Women’s and Reproductive Health, Oxford, UK (GRID:grid.4991.5) (ISNI:0000 0004 1936 8948); Health Data Research UK, London, UK (GRID:grid.507332.0) (ISNI:0000 0004 9548 940X) 
17  Women’s College Hospital. University of Toronto, Department of Medicine, Toronto, Canada (GRID:grid.17063.33) (ISNI:0000 0001 2157 2938) 
18  Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, NIHR Biomedical Research Centre at Moorfields, London, UK (GRID:grid.436474.6) (ISNI:0000 0000 9168 0080) 
19  Harvard. T.H. Chan School of Public Health, Department of Epidemiology, Boston, USA (GRID:grid.38142.3c) (ISNI:000000041936754X); Harvard Medical School, Department of Biomedical Informatics, Boston, USA (GRID:grid.38142.3c) (ISNI:000000041936754X) 
20  University of Birmingham, Institute of Inflammation and Ageing, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK (GRID:grid.412563.7) (ISNI:0000 0004 0376 6589); University of Birmingham, National Institute for Health and Care Research (NIHR) Birmingham Biomedical Research Centre, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University of Birmingham, Birmingham Health Partners Centre for Regulatory Science and Innovation, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology, NIHR Biomedical Research Centre at Moorfields, London, UK (GRID:grid.436474.6) (ISNI:0000 0000 9168 0080) 
21  University of Birmingham, Institute of Inflammation and Ageing, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486); University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK (GRID:grid.412563.7) (ISNI:0000 0004 0376 6589); University of Birmingham, Birmingham Health Partners Centre for Regulatory Science and Innovation, Birmingham, UK (GRID:grid.6572.6) (ISNI:0000 0004 1936 7486) 
Pages
1619
Publication year
2024
Publication date
2024
Publisher
Nature Publishing Group
e-ISSN
20411723
Source type
Scholarly Journal
Language of publication
English
DOI
https://doi.org/10.1038/s41467-024-45355-3
ProQuest document ID
2930208824
Copyright
© The Author(s) 2024. corrected publication 2024. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.