Correspondence to Edith Ginika Otalike; [email protected]
STRENGTHS AND LIMITATIONS OF THIS STUDY
The use of a systematic methodology review to identify potential domains and items for the checklist.
The e-Delphi survey eliminates the potential for peer-group dominance influence, enables global representation and allows for a large sample size and diverse participants.
The use of a virtual format for both Delphi rounds and the consensus meeting may contribute to low response rates or participant attrition.
Restricting the survey to English could affect global representation.
Introduction
Individual participant data meta-analysis (IPD-MA) involves sourcing data for each individual in an eligible research study for use in a more robust synthesis and complex analysis than is possible with traditional meta-analysis of aggregate data from publications.1 Although aggregate data meta-analysis is a powerful tool for summarising estimates, it has drawbacks. These include the inability to conduct a consistent analysis across relevant subgroups.2 Other limitations are inconsistent definitions of outcomes, application of different statistical models across studies and variations in the reported effect estimates (eg, HR, OR).3 IPD-MA overcomes some of these challenges by using participant-level data to perform a harmonised and standardised analysis.1 IPD-MA offers a more direct approach to consolidating evidence to support healthcare decision-making,4 because it models participant-specific effects, enabling tailored treatment strategies5 6 and study-specific treatment-covariate interactions to be analysed in a single model.7 IPD-MA can help minimise research waste by improving information size and promoting data sharing, which enables independent scrutiny of trial data.8 There are examples of where IPD-MA have dramatically reformed the design, execution and interpretation of trials.9 Given the numerous advantages and the robustness of evidence it produces, IPD-MA is regarded as the gold standard in evidence synthesis of intervention studies1 3 9 and has gained increasing interest across healthcare research.
Despite the advantages of IPD-MA, it is not a remedy for all the limitations of traditional meta-analyses. For example, inconsistencies or poor quality in the underlying research can introduce bias in the results and, hence, the interpretation of findings.10 IPD-MA is susceptible to biases such as selection, publication and data unavailability bias,11 with data inaccessibility being particularly problematic as it has left some IPD-MA projects incomplete or abandoned.12 Also, IPD-MA requires significant resources and time compared with aggregate-data meta-analysis. Recent evidence from a systematic review of more than 320 IPD-MAs, which used a checklist that combined elements from multiple checklists and recommendations from published articles13–16 for assessing the quality of the IPD-MA, revealed that the studies performed poorly in conventional and IPD-MA-specific methodological items.17 For instance, over 50% of the studies failed to pre-specify the analytical method for effect estimation, and a few others did not account for the clustering of participants in their studies.17 Tools are available for reporting and guiding the conduct of IPD-MA in randomised trials.14 18–20 These tools have provided a framework for some completed assessments of methods used for IPD-MA17 and ongoing research in this field, such as the development of a data quality and integrity checklist.21 However, there is no consensus-based critical appraisal tool that specifically addresses IPD-MA-specific challenges. Furthermore, there is an increasing number of IPD-MAs being conducted, including many that do not use randomised trials; Wang et al17 identified 1038 IPD-MAs based on observational studies (flow chart diagram). However, recommendations on assessing the methodology for IPD-MA of observational studies have not been given much attention. Considering the lack of a tool to conduct a reproducible assessment of the methodological quality of IPD-MA, there is a need for an objective tool to identify high-quality IPD-MA based on randomised and/or observational studies in keeping with the ongoing development of data processing and integrity tools to accommodate randomised and observational studies.21
Experts in IPD-MA research are likely to have various opinions on what should constitute an item, domain and critical quality domain for a quality assessment tool for IPD-MA. We will seek consensus using an e-Delphi survey from individuals with diverse experiences and opinions. The e-Delphi survey is preferred for several reasons. For instance, it ensures more involvement of experts from diverse geographical locations and research experiences, thereby mitigating recruitment bias.22 It reduces potential subjective bias due to the influence or dominance of peer groups,23 it is logistically feasible for the recruitment of a large number of participants, and it can be beneficial when subjective judgments are needed and can guide opinions to a final agreement.24
Therefore, we will use a multi-round e-Delphi survey to develop a consensus-based critical appraisal checklist for IPD-MA based on randomised and/or observational studies. This will then be validated by external investigators.
Methodology
Our study will have three phases, as shown below (figure 1). (1) A systematic methodology review to identify items that will inform the e-Delphi survey, (2) At least two rounds of an e-Delphi survey among participants, a consensus meeting among the steering committee and pilot testing to refine the checklist and explanatory text and (3) Tool validation. A steering committee with expertise in evidence synthesis, IPD-MA methodology, consensus methods, statisticians, healthcare providers and representatives of the patient and public partners will coordinate the project through the outlined phases.
Phase 1: a systematic methodology review to identify and collate potential quality items
Following Cochrane guidance for systematic reviews,25 we will conduct a systematic methodology review to identify quality items and domains based on IPD-MA guidance and recommendations. A protocol for this review covering all the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P)26 can be found in the Open Science Framework (https://osf.io/v5n9r/).
In summary, we will search multiple databases and sources, including Medline, Embase, Web of Science, Scopus, CINAHL/PsycINFO, Cochrane Library, Journal of Research Synthesis Methods and Journal of Royal Statistical Society Series A, B, C. The full search strategy is provided in online supplemental file 1. We will identify and document available guidance on methodology and statistical recommendations for IPD-MA of randomised and/or observational studies from the included articles. Inclusion criteria are preprint and peer-reviewed articles published in English that address methodological issues, including IPD-MA of simulations and empirical studies. All studies without a focus on methodological guidance for IPD-MA will be excluded, especially if the focus is to answer a specific research question. All identified references from the database search will be imported into the Covidence software,27 and duplicates will be removed. Two independent reviewers will screen titles and abstracts based on the inclusion and exclusion criteria. Similarly, at the second level, two reviewers will screen full-text articles independently against the eligibility criteria and justify any article removed. Likewise, appropriate risk of bias tools28–31 will be used to assess the quality of the included studies. A pre-designed data extraction form will be piloted on a random subset of 5% of the included studies and revised according to feedback from reviewers. All eligible citations will be extracted using the calibrated data extraction form to capture the author’s conclusions and relevant items on the proposed domains table 1. The proposed domains are derived from the relevant existing checklists.13 18 19 Any disagreement will be resolved by discussion until a consensus is reached. Where consensus cannot be reached, a third reviewer will be invited to reconcile disagreements during the study selection and data extraction process. Descriptive statistics will be used to summarise general information about the included studies. Recurring themes beyond proposed domains will be identified and synthesised as domains, and signalling questions will be synthesised as items for each domain. The findings of the systematic review will be reported following the PRISMA 2020 guidance32 and disseminated through a peer-reviewed publication.
Table 1Proposed domains for the checklist
| Sections | Systematic review framework | IPD collection and processing | Synthesis |
| Proposed domains | Protocol registration | Identification of eligible participants | Meta-analytic approach |
| Research question | Data collection | Combining aggregate data and IPD | |
| Eligibility criteria | Data processing and management | Missing data | |
| Initial risk of bias assessment | Data integrity assessment | Sub-group analysis | |
| Identification of eligible study | Data quality checks | Heterogeneity | |
| Risk of bias assessment of RCTs for IPD | Special consideration for observational studies | ||
| Interpretation of research findings | |||
| Reporting of study |
IPD, individual participant data; RCTs, randomised controlled trials.
Finally, the expertise and experience of the steering committee will be leveraged to pilot test the proposed e-Delphi survey questionnaire in two rounds to refine domains and signalling questions for inclusion in the questionnaire that will be used for the e-Delphi survey33 using Qualtrics metrics.34 For each item, the Steering Committee will independently vote on a 5-point Likert scale: (1=‘strongly agree’; 2=‘agree’; 3=‘neutral’; 4=‘disagree’ and 5=‘strongly disagree’). Items with 75% or more votes for agree or strongly agree will be included, and items for which 75% or more vote strongly disagree or disagree will be excluded. The team will discuss items which do not meet either of these criteria before the final draft is sent to the e-Delphi participants.
Phase 2: two rounds of an e-Delphi survey among invited experts and a consensus meeting
This phase aims to collate and analyse agreement on the checklist items from IPD-MA experts globally to form a consensus on the checklist for assessing the methodological quality of an IPD-MA. After the design of the questionnaire in phase 1, at least two rounds of an e-Delphi survey will be conducted following established guidelines,33 35 among methodologists, statisticians, healthcare guidelines and consensus experts, patients and public partners to achieve consensus on the methodological quality items for consideration when conducting an IPD-MA. We will use quasi-anonymity, because the response will be anonymous to all but the project lead, who will link the response to participants through a unique identifier to enable controlled feedback. Other hallmarks of the Delphi process, including iteration, expert panels and quantitative aggregation of response,36 will be maintained. We will have at least two rounds and ensure that the checklist is as concise as possible to prevent survey fatigue and a low response rate.37
Steering committee members
This core research team will guide the conduct of the consensus process throughout, from the systematic methodology review to the dissemination of the final critical appraisal tool. The members are multi-disciplinary researchers with vast experience conducting IPD-MA, statisticians, methodologists, healthcare providers, patients and public partners. In addition to voting on the candidate items, the steering committee will determine the profile of participants and sample size for the e-Delphi survey. They will decide on the survey approach to administer the questionnaire. The members of the steering committee will not take part in the e-Delphi survey but will make decisions on the acceptance of the items that achieved consensus, confirm the exclusion of items that were not agreed on and consider recommendations from participants.33
e-Delphi participants
Although there is no consensus on the appropriate sample size for an e-Delphi survey,23 the attrition rate, the complexity of the topic, and the level of diversity required among experts and logistics should guide the sample size determination.36 We intend to include as many potential participants as possible (at least n=250) to mitigate attrition bias. Furthermore, to ensure international representation and uptake of the checklist, panellists will have a diverse geographical representation with relevant experience and from relevant professional networks and organisations. Researchers with expertise in IPD-MA, methodologists, statisticians, journal editors, health policymakers, healthcare providers, pharmaceutical representatives, experts in consensus development methods, evidence synthesis, knowledge translation and patient and public partners will be invited.36 The participants will be recruited through correspondence with the authors of the studies included in the systematic methodology review, the Cochrane Methods Group for IPD-MA, researchers in the networks of the steering committee, snowball sampling where participants can refer others from within their networks, and the publication of this protocol.
Patient and public engagement
Patient/public engagement in research is gaining traction as a standard for review by reputable organisations (eg, Strategy for Patient-Oriented Research (SPOR) Evidence Alliance, Cochrane Consumer Network). Their involvement is crucial as the quality of evidence from IPD-MA directly impacts patient care through evidence-based individual and population healthcare decision-making. Patients and the relevant stakeholders can help promote the use of the tool, thereby promoting rigorous and high-quality research, addressing patient-centred outcomes, enhancing transparency, supporting knowledge translation and informing decision-making for policy and practice. We will convene a workshop for the patient/public partners in collaboration with a SPOR representative to provide training on the methods of IPD-MA conduct, the intention of the research, the process and the expected outcome. We will also be available to answer questions and concerns throughout the process. We will stratify the responses from the survey into those from patient/public partners and the research experts. Patient/public partner’s involvement will be reported following the GRIPP2 reporting checklist.38
The e-Delphi process
Participants will be invited through email and provided with details of the consensus process, including the likely number of rounds, their role and the expected timeline for completion. They will be asked to provide consent by signing and returning an attached consent form.23 Each round of the survey will be open for 3 weeks. Reminder emails will be sent 2–3 times to non-responders. All potential participants will be provided with ample information, including links to the consensus protocol, systematic methodology review, informed consent statements and the online survey platform. Each participant will be assigned an anonymous identifier and asked to vote using a 5-point Likert scale (1=‘strongly disagree’; 2=‘disagree’; 3=‘neutral’; 4=‘agree’ and 5=‘strongly agree’).39 The platform will allow participants to suggest new items and toprovide comments on the items listed. Remainder emails will be sent 2 weeks after launching the survey, and the last reminder email will be sent 48 hours before the deadline for the first round. There will be a 2 week interval between the first and the second rounds, during which feedback from all first-round responders will be summarised and provided to the participants. All items from the first round, additional comments and items from the participants will be considered for inclusion in the checklist for the second round. Retaining all items, including those with sufficient votes in the first round, allows for the assessment of the stability of the agreement.40 Round 2 of the e-Delphi process will include experts regardless of their participation in round 1, to mitigate bias due to attrition and potential false consensus and ensure that the checklist is representative of panellist opinions.41
Synthesis of response
Descriptive statistics will be used to provide the demographic characteristics of the participants to show the level of representativeness, including gender, area of expertise, country of residence and level of experience.42 A mixed-method synthesis will be used to analyse the data from the process. There will be a qualitative synthesis of the free text provided by the participants into recurring themes, which will be determined by the steering committee for each round of the e-Delphi, and a quantitative report of the frequencies, proportion, median and IQR for each domain and item across panellists.36 The response stability between the rounds will be estimated using the matched pairs Wilcoxon signed rank test. Stability is achieved when no statistically significant change is observed between rounds for each item.43 There is no standard consensus definition for e-Delphi surveys; we will use a threshold of ≥75% rating for strongly agree or agree and a 50% response rate to include an item in the final tool, and ≥75% rating for strongly disagree or disagree for items that did not achieve consensus. Anything else indicates uncertainties about the importance of an item, therefore no consensus and these items will be discussed in the consensus meeting (see below).22 36 44
Analysis
Summary statistics of participants’ demographic information will be presented as n (%)42 for
Gender.
Age.
Geographic location.
Background.
Qualifications.
Qualitative synthesis
Participants will be able to provide free text responses ranging from brief comments to several40 sentences. The qualitative synthesis will focus on identifying key themes and patterns across responses, allowing us to capture both concise insights and more detailed narratives.
Syntax analysis
Labelling will be added for each comment as a whole sentence, phrase or catchy word. A high proportion of sentences indicates high-level engagement (a quality measure).45
Content analysis
This analysis describes and quantifies the comments. It aims to condense a broad description of words. This will be more deductive, where comments are grouped into predefined domains, and inductive, which aims to code and categorise the response into domains not defined a priori. The frequency of all comments will be graphically presented using bar charts and a frequency table with a heat map.
Quantitative
A normality test using the Kolmogorov-Smirnov test will be applied to determine whether the parametric or non-parametric test should be used.
Strength of consensus
This will be estimated by determining the response rate, response stability and proportion of43 agreements (consensus) as outlined below.
Response rate per item based on the number of participants who responded to the survey (proportion).
Stability will be estimated using the Wilcoxon matched-pairs signed rank test. It uses paired data of the same group of participants to test response stability between each e-Delphi round. A participant’s response is considered stable if no statistically significant change is observed between rounds for each item.
The measure of agreement will be the proportion of response for each item. It will be presented graphically. A median and IQR will be used to estimate the measure of central tendency and dispersion, respectively. An IQR of less than one (more than 50% of opinions fall within a particular point on a scale) will be considered appropriate for this 5-point Likert scale.
Post-hoc test to assess non-response bias: the Mann-Whitney U-test will be used to determine the difference between early and late responders for each item, assuming that the relationship between delay in response and outcome frequency can predict non-response.46 Late responders are participants who responded after the last reminder; otherwise, they are early responders.
Consensus meeting
After completing the synthesis of responses from the two rounds, the steering committee will hold a virtual consensus meeting to decide on the items to include in the final checklist and accompanying explanatory text. The focus of this meeting will be on items for which consensus could not be reached.42 After discussion, the steering committee members will anonymously vote for those items to be included or not. The final checklist will include items that ≥75% of those voting agree to include.
Pilot testing and refinement of the developed critical appraisal tool
The steering committee members and a convenience sample of IPD-MA reviewers will be invited to apply the developed checklist to their most recently conducted IPD-MA and to provide practical feedback. This feedback will be used to refine the checklist and the explanatory text. The resulting consensus statement will be reported following the Accurate Consensus Reporting Document guidelines36 and disseminated in peer-reviewed publications, at conferences and through the networks of the research team.
Phase 3: tool validation
Content validity study participants
The assessment of the tool’s content validity will follow COSMIN standards.47 This phase of the study will include 30 participants. This will be a subset of the e-Delphi participants with varying backgrounds, including healthcare providers with research backgrounds, IPD-MA reviewers, methodologists and individuals from the Cochrane IPD-MA methods group, to assess the content validity.
Response options
This panel will be asked to rate each item based on the criteria in table 2 on a 4-point scale (1=strongly disagree, 2=disagree, 3=agree and 4=strongly agree). For each item, a response average of 3 for all the listed criteria is considered adequate agreement on its content validity.
Table 2Criteria for assessing content validity
| S/N | Signalling question |
| Relevance | |
| 1 | Is the included item relevant to the construct of interest |
| 2 | Are the included items relevant for the context of the use of interest? |
| 3 | Are the response options appropriate? |
| Comprehensiveness | |
| 4 | Are key concepts missing |
| Comprehensibility | |
| 5 | Are instructions understood? |
| 6 | Are the tool item response options understood? |
| 7 | Are items adequately worded? |
| 8 | Does the response option match the questions? |
Participants in reliability testing
COSMIN guidelines48 will be followed to assess the tool’s reliability. We will conduct a database search to identify 45 completed IPD-MA in three research areas (cancer, cardiovascular diseases and respiratory infection), with 15 from each area published after 1 January 2020. Three pairs of graduate students will be recruited and trained to apply the developed tool to evaluate the quality of the identified articles. For quality measures, three steering committee members will each rate the quality of each study alongside the pairs of raters.
Response options
For each IPD-MA, each item will be scored based on whether it is of high (4), moderate (3), low (2) or critically low (1) quality. The inter-rater reliability of each item for each pair will be estimated using intraclass correlation coefficients.
Data management
All data will be stored in the Western University OneDrive. The existing Microsoft agreement protects it, provided it meets the Western data classification standard. Access to the data is restricted to the project lead, who uses assigned Western login credentials and multi-factor authentication.
Ethics and dissemination
Ethics approval has been obtained from the Western University Health Science Research Ethics Board, with a reference number 2024-125162-94267. The validated checklist will be published in a peer-reviewed open-access journal and disseminated among the networks of the steering committee members, Cochrane IPD-MA methods group, the institutions’ social media platforms, international and national conferences, at scientific meetings and engage public and other stakeholders and webinars with accessible summaries on our institution’s websites.
Ethics statements
Patient consent for publication
Not applicable.
X @eotalike
Contributors Conceptualisation: EGO and JG. Methodology: EGO, MC, AAV, ACT, DM, BS, AD-K, N-BK and JG. Writing initial draft: EGO. Writing review and editing: EGO, MC, AAV, ACT, DM, BS, AD-K, N-BK and JG. Guarantor: JG.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests The authors of this manuscript have the following competing interests: EO receives internal funding from Western University and the Dean's Research Scholarship award. ACT declared funding from the tier 1 Canada research chair in knowledge synthesis for knowledge users. All other authors have no competing interest to declare.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
1 Debray TPA, Moons KGM, van Valkenhoef G, et al. Get real in individual participant data (IPD) meta-analysis: a review of the methodology. Res Synth Methods 2015; 6: 293–309. doi:10.1002/jrsm.1160 26287812
2 Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med 2002; 21: 1559–73. doi:10.1002/sim.1187
3 Riley RD, Debray TPA, Morris TP, et al. The two-stage approach to ipd meta-analysis: a handbook for healthcare research. In: R RDR, JF T, LA S, eds. Individual participant data meta-analysis:A handbook for healthcare research. 1st edn. Wiley, 2021: 89–127.
4 Vale CL, Rydzewska LHM, Rovers MM, et al. Uptake of systematic reviews and meta-analyses based on individual participant data in clinical practice guidelines: descriptive study. BMJ 2015; 350: h1088. doi:10.1136/bmj.h1088
5 Clarke M, Stewart L, Pignon JP, et al. Individual patient data meta-analysis in cancer. Br J Cancer 1998; 77: 2036–44. doi:10.1038/bjc.1998.339
6 Veroniki AA, Seitidis G, Tsivgoulis G, et al. An Introduction to Individual Participant Data Meta-analysis. Neurology (ECronicon) 2023; 100: 1102–10. doi:10.1212/WNL.0000000000207078
7 Tierney JF, Fisher DJ, Burdett S, et al. Comparison of aggregate and individual participant data approaches to meta-analysis of randomised trials: An observational study. PLoS Med 2020; 17: e1003019. doi:10.1371/journal.pmed.1003019
8 Richard RD, Jayne TF, Lesley SA. Individual participant data meta‐analysis. Wiley, 2021.
9 Tierney JF, Pignon J-P, Gueffyier F, et al. How individual participant data meta-analyses have influenced trial design, conduct, and analysis. J Clin Epidemiol 2015; 68: 1325–35. doi:10.1016/j.jclinepi.2015.05.024
10 Burdett S, Stewart LA. A comparison of the results of checked versus unchecked individual patient data meta-analyses. Int J Technol Assess Health Care 2002; 18: 619–24.
11 Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ 2012; 344. doi:10.1136/bmj.d7762
12 Jaspers GJ, Degraeuwe PLJ. A failed attempt to conduct an individual patient data meta-analysis. Syst Rev 2014; 3: 97. doi:10.1186/2046-4053-3-97
13 Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017; 358: j4008. doi:10.1136/bmj.j4008
14 Tierney JF, Vale C, Riley R, et al. Individual Participant Data (IPD) Meta-analyses of Randomised Controlled Trials: Guidance on Their Use. PLoS Med 2015; 12: e1001855. doi:10.1371/journal.pmed.1001855
15 Fisher DJ, Copas AJ, Tierney JF, et al. A critical review of methods for the assessment of patient-level interactions in individual participant data meta-analysis of randomized trials, and guidance for practitioners. J Clin Epidemiol 2011; 64: 949–67. doi:10.1016/j.jclinepi.2010.11.016
16 Riley RD, Simmonds MC, Look MP. Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol 2007; 60: 431–9. doi:10.1016/j.jclinepi.2006.09.009
17 Wang H, Chen Y, Lin Y, et al. The methodological quality of individual participant data meta-analysis on intervention effects: systematic review. BMJ 2021; 373: n736. doi:10.1136/bmj.n736
18 Stewart LA, Clarke M, Rovers M, et al. Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. JAMA 2015; 313: 1657–65. doi:10.1001/jama.2015.3656
19 Tierney JF, Stewart LA, Vale CL, et al. A tool for the critical appraisal of ipd meta-analysis projects (checkmap). In: Individual participant data meta-analysis:a handbook for healthcare research. Wiley, 2021: 271–83.
20 Tierney J, Stewart L, Clarke M, et al. Individual participant data: cochrane handbook for systematic reviews of interventions. In: Higgins J, Thomas J, Chandler J, et al, eds. Cochrane handbook for systematic reviews of interventions. 2nd edn. John Wiley & Sons, 2023.
21 Hunter KE, Webster AC, Clarke M, et al. Development of a checklist of standard items for processing individual participant data from randomised trials for meta-analyses: Protocol for a modified e-Delphi study. PLoS ONE 2022; 17: e0275893. doi:10.1371/journal.pone.0275893
22 Keeney S, Hasson F, McKenna H. The delphi technique in nursing and health research. Wiley, 2011.
23 Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol 2021; 11: 116–29. doi:10.5662/wjm.v11.i4.116
24 McKenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs 1994; 19: 1221–5. doi:10.1111/j.1365-2648.1994.tb01207.x
25 Clarke M. Guide to the contents of a cochrane methodology protocol and review. In: Clarke M, ed. Guide to the contents of Cochrane Methodology protocol and review. 2020.
26 Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015; 4: 1. doi:10.1186/2046-4053-4-1
27 Covidence systematic review software, veritas health innovation. Covidence. 2024 Available: https://www.covidence.org/
28 Gagnier JJ, Moher D, Boon H, et al. Investigating clinical heterogeneity in systematic reviews: a methodologic review of guidance in the literature. BMC Med Res Methodol 2012; 12: 111. doi:10.1186/1471-2288-12-111
29 Baethge C, Goldbeck-Wood S, Mertens S. SANRA-a scale for the quality assessment of narrative review articles. Res Integr Peer Rev 2019; 4: 5. doi:10.1186/s41073-019-0064-8
30 Whiting P, Savović J, Higgins JPT, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016; 69: 225–34. doi:10.1016/j.jclinepi.2015.06.005
31 Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Stat Med 2019; 38: 2074–102. doi:10.1002/sim.8086
32 Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS Med 2021; 18: e1003583. doi:10.1371/journal.pmed.1003583
33 Gattrell WT, Hungin AP, Price A, et al. ACCORD guideline for reporting consensus-based methods in biomedical research and clinical practice: a study protocol. Res Integr Peer Rev 2022; 7: 3. doi:10.1186/s41073-022-00122-0
34 Qualtrics version 2024. Qualtrics; 2024. Available: https://uwo.eu.qualtrics.com
35 Moher D, Schulz KF, Simera I, et al. Guidance for developers of health research reporting guidelines. PLoS Med 2010; 7: e1000217. doi:10.1371/journal.pmed.1000217
36 Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med 2024; 21: e1004326. doi:10.1371/journal.pmed.1004326
37 de Koning R, Egiz A, Kotecha J, et al. Survey Fatigue During the COVID-19 Pandemic: An Analysis of Neurosurgery Survey Response Rates. Front Surg 2021; 8: 690680. doi:10.3389/fsurg.2021.690680
38 Staniszewska S, Brett J, Simera I, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 2017; 358: j3453. doi:10.1136/bmj.j3453
39 William JT, Mark J, Wierman U, et al. RANKING ORDINAL SCALES USING THE CONSENSUS MEASURE. Issues In Information Systems 2005; 6.
40 Beiderbeck D, Frevel N, von der Gracht HA, et al. Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. MethodsX 2021; 8: 101401. doi:10.1016/j.mex.2021.101401
41 Boel A, Navarro-Compán V, Landewé R, et al. Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome. J Clin Epidemiol 2021; 129: 31–9. doi:10.1016/j.jclinepi.2020.09.034
42 Logullo P, van Zuuren EJ, Winchester CC, et al. ACcurate COnsensus Reporting Document (ACCORD) explanation and elaboration: Guidance and examples to support reporting consensus methods. PLoS Med 2024; 21: e1004390. doi:10.1371/journal.pmed.1004390
43 Heiko A. Consensus measurement in Delphi studies Review and implications for future quality assurance. Technol Forecast Soc Change 2012; 79: 1525–36. doi:10.1016/j.techfore.2012.04.013
44 Srikandarajah N, Noble A, Clark S, et al. Cauda Equina Syndrome Core Outcome Set (CESCOS): An international patient and healthcare professional consensus for research studies. PLoS ONE 2020; 15: e0225907. doi:10.1371/journal.pone.0225907
45 Roßmann B, Canzaniello A, von der Gracht H, et al. The future and social impact of Big Data Analytics in Supply Chain Management: Results from a Delphi study. Technol Forecast Soc Change 2018; 130: 135–49. doi:10.1016/j.techfore.2017.10.005
46 Sheikh K, Mattingly S. Investigating non-response bias in mail surveys. J Epidemiol Community Health 1981; 35: 293–6. doi:10.1136/jech.35.4.293
47 Terwee CB, Prinsen CAC, Chiarotto A, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res 2018; 27: 1159–70. doi:10.1007/s11136-018-1829-0
48 Prinsen CAC, Mokkink LB, Bouter LM, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res 2018; 27: 1147–57. doi:10.1007/s11136-018-1798-3
© 2025 Author(s) (or their employer(s)) 2025. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ Group. http://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ . Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Clarke, Mike 2
; Areti, Angeliki Veroniki 3
; Tricco, Andrea C 4
; Moher, David 5 ; Shea, Beverley 6 ; Doherty-Kirby, Amanda 7 ; Kandala, Ngianga-Bakwin 1 ; Gagnier, Joel 8 1 Epidemiology and Biostatistics, University of Western Ontario, London, Ontario, Canada
2 Northern Ireland Methodology Hub, Queen’s University Belfast, Belfast, UK
3 IHPME, University of Toronto Institute of Health Policy Management and Evaluation, Toronto, Ontario, Canada; Knowledge Translation Program, St. Michael’s Hospital, Toronto, Ontario, Canada
4 Knowledge Translation Program, University of Toronto Institute of Health Policy Management and Evaluation, Toronto, Ontario, Canada
5 University of Toronto, Toronto, Ontario, Canada; University of Ottawa, Ottawa, Ontario, Canada
6 University of Ottawa, Ottawa, Ontario, Canada
7 St. Michael’s Hospital, Toronto, Ontario, Canada
8 Western University Schulich School of Medicine & Dentistry, London, Ontario, Canada