Exploration of registration and the risk of bias in acupuncture randomised controlled trials: a systematic review protocol

Abstract

Background

Randomised controlled trials (RCTs) are the predominant type in acupuncture clinical research, and the publications have increased rapidly in recent years, but there is a prevalence of the high risk of bias and poor methodological design in acupuncture RCTs. Clinical trial registration can improve the transparency and credibility of studies by disclosing key information in advance. However, the registration in acupuncture RCTs is not satisfactory, as there is widespread of the under-registration, inconsistency with published studies and insufficient disclosure of key methodological information. Whether registration can reduce the risk of bias in acupuncture RCTs and improve data transparency has not been fully explored. Therefore, we constructed this study to investigate the association between registration and risk of bias and data sharing level in acupuncture RCTs.

Methods

Seven databases including MEDLINE, EMBASE, CENTRAL, CBM, CNKI, Wanfang and VIP databases will be systematically searched between 1 January 2014 and 31 March 2024, for acupuncture RCTs. Two reviewers will independently extract data using a predefined standardised format and perform secondary validation. The characteristics and data sharing level of the included studies will be summarised. The risk of bias of included RCTs will be assessed by the revised Cochrane risk-of-bias tool for randomised trials. The risk of bias and registration in acupuncture RCTs will be analysed by logistic or quantile regression analyses (depending on the number of minimum events). The data sharing level and registration will be analysed by quantile regression analyses.

Ethics and dissemination

As the systematic review aims to consolidate info from published sources, ethical approval is not necessary for this study. The study’s findings will be submitted to a peer-reviewed academic journal and disseminated via conference presentations. This protocol has been registered in Open Science Framework Registries.

YD, YD and BT are joint first authors.

STRENGTHS AND LIMITATIONS OF THIS STUDY

• The study will systematically collect the registration status and data sharing level of acupuncture randomised controlled trials (RCTs) published from 2014 to 2024 without language restriction.

• The study will use the revised Cochrane risk-of-bias tool for randomised trials to evaluate the risk of bias of acupuncture RCTs.

• Regression analyses will be used to explore the registration with risk of bias and data sharing level.

• We will conduct three rounds of pretests to guarantee consistency among researchers.

• Due to the time limitation, we limit our inclusion to representative study designs, studies published in peer-reviewed journals with two groups and parallel RCTs.

Introduction

Randomised controlled trials (RCTs) are the gold standard for exploring the study cause relationships between interventions and outcomes as randomisation and standardised intervention procedures eliminate much of the bias inherent in study designs.¹ RCTs provide extensive research evidence for clinical practice and represent the cornerstone of Evidence-Based Medicine.² However, any bias in RCTs can affect the credibility of the study results³ and may even mislead clinical practice and harm patients.⁴ The revised Cochrane risk-of-bias tool (RoB 2) is the most common assessment tool for evaluating the risk-of-bias of RCTs,⁵ it can help researchers judge the methodological quality of studies and determine the strength of evidence.⁶

RCTs are the most dominant type in acupuncture clinical research, and the publications have increased rapidly in recent years, but there is a prevalence of high risk of bias in acupuncture RCTs.⁷ Reducing the risk of bias is an urgent issue for acupuncture RCTs to improve the transparency and credibility of study results.^{8 9} Clinical trial registration can systematically disclose key information with time-stamped about a clinical trial before it begins.^{10 11} Adequate trial registration is a safeguard to limit bias such as modifying study endpoints and intervention protocols to increase the transparency of the study and the credibility of the results.^{12 13}

Although clinical trial registration has become the rule, the registration of published acupuncture RCTs is significantly low, with the proportion of 19.3%¹⁴ and 41.9%.¹⁵ Moreover, 39.1% of published acupuncture RCTs have discrepancies with the registration information,¹⁵ and 25.1% of acupuncture registration did not disclose the allocation methods, 57.1% did not report blinding implementation.^{16 17} Registration of acupuncture RCTs has not received sufficient attention, and the quality of acupuncture registrations is not satisfactory.Therefore, it is an issue to clarify the role of registration in acupuncture RCTs to strengthen acupuncture registration.

Data sharing is a scientific paradigm proposed by the International Committee of Medical Journal Editors to ensure reproducibility and credibility of research, enhance research transparency and reduce research waste.^{18 19} Data sharing has been widely recognised by publishers, funders and the scientific community nowadays.^{20 21} However, the data sharing level in RCTs is unsatisfactory due to researcher concerns about publication rights, scientific sensitivity and unauthorised commercial use.²² Although registration requires an articulation of data sharing, the relationship between registration and data sharing has not been investigated to data.

Therefore, we designed this study to analyse the current status of registration of acupuncture RCTs in the last decade and to explore the relationship between registration and the risk of bias and level of data sharing in acupuncture RCTs. We will use regression analyses to analyse the risk of bias and data sharing levels with registration status in acupuncture RCTs and further treat funding, publication language and publication year as confounders to reduce bias. We hope our study can foster the enhanced quality of acupuncture RCTs.

Methods and analysis

This protocol will be conducted in the following sections: (1) search for eligible RCTs: (2) extract characteristics, data sharing level and the risk of bias assessment results from included RCTs and (3) the use logistic regression analysis to compare between the registered and unregistered RCTs. This protocol has been registered and is publicly available via the Open Science Framework²³ (Registration DOI: https://doi.org/10.17605/OSF.IO/GZA9V). This study will adhere strictly to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols checklist in order to ensure a rigorous and systematic approach to the systematic review protocol.²⁴

Inclusion and exclusion criteria

We will include the following RCTs: (1) acupuncture therapy (acupuncture alone or in combination) only used in the intervention group, (2) acupuncture therapy should based on traditional Chinese medicine theory and the needle should penetrate the skin; (3) the control group contains sham acupuncture, no intervention, western medicine treatment or other interventions (such as routine nursing, rehabilitation and psychotherapy); (4) the study design based on two parallel study groups applied to humans; (5) full-text studies without language restrictions and (6)studies published in peer-reviewed journals.

To improve the quality of the included literature, we will exclude the following articles: (1) sample size less than or equal to 10 in each group; (2) less than three or fewer authors and (3) duplicated studies.

Databases and search strategy

We will search for acupuncture RCTs in seven databases including three English databases (MEDLINE, Excerpta Medica Database (EMBASE), Cochrane Central Register of Controlled Trials (CENTRAL)) through OVID and four Chinese databases (Chinese Biomedical Literature Service System (CBM), China National Knowledge Infrastructure (CNKI), Wanfang Data and VIP Chinese Medical Journal Database) from 1 January 2014 to 31 March 2024, with no language restrictions. The detailed search strategy is shown in online supplemental material 1.

Literature screening

We will use EndNote (V.X7.1) and Rayyan (https://www.rayyan.ai/) to screen and exclude included literature. Two researchers (ZX and PZ) will independently screen the titles and abstracts of the included acupuncture RCTs based on inclusion and exclusion criteria. Then two researchers (ZX and PZ) will independently screen the full texts and exclude those that do not meet the eligible criteria. Discrepancies will be discussed by two senior investigators (YDuan and LY).

Patient and public involvement

Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Training and pretest

All reviewers will undergo training about the definition, classification and response of RoB 2 based on the Cochrane website.²⁵ To ensure the consistency and accuracy of RoB 2 evaluation, we will conduct three rounds of pretests before the formal evaluation, only the kappa coefficients >0.75 will start the formal extraction and evaluation.

Data extraction

A standard information extraction table with a Microsoft Excel spreadsheet (Microsoft Excel 2019) will be created for data extraction. Then, the eligible studies will undergo a coding process and will be randomised into two equal segments. Each of the two reviewers (ML and XY) will assume responsibility for one segment and extract data. Following this, the extracted data will undergo cross-verification, with a specific allocation as follows: ML will review XY’s data, XY will review ML’s data. This cross-verification is aimed at ensuring the accuracy of data extraction. In case of any discrepancies, two senior investigators (YDuan and LY) will be involved to resolve the discrepancies.

The characteristics of included RCTs will be extracted: (1) title; (2) author; (3) publication language; (4) publication year; (5) registration status; (6) single-centre or multicentre; (7) acupuncture modality (head acupuncture, body acupuncture, electroacupuncture, ear acupuncture, ankle acupuncture and mixed acupuncture) and (8) control group type (fake acupuncture, therapy, blank control). Researchers will independently compare the differences between publications and registrations (if any) and the following information will be collected: (1) outcomes (outcome indicators or measure time change); (2) participants (any difference, for example, ageing, gender, disease change); (3) concealment (techniques or methods); (4) blinding (techniques or methods) and (5) randomisation (techniques or methods).

We will classify the trial registration into prospectively registered, retrospectively registered and unregistered. Prospectively registered is defined as trial registration before the study starts. The following conditions should be met for prospectively registered and retrospectively registered: (1) studies provide a registration number in the abstract, main text or online supplemental file; (2) the registration number should be provided by a professional website such as ClinicalTrials.gov, Cochrane Controlled Trials Register; (3) or the study did not register on a professional website but published a research protocol in professional journals; (4) if studies claiming they have registered or developed a protocol, but failed to provide any details and can not be obtained by contacting the corresponding author will not be treated as registered and (5) if studies provide protocol in the attachments, but without a registry number or published DOI will not be treated as registered.

The data sharing level will be assessed based on our previous study²⁶ by open science practices, including the following items: (1) whether registration details are available; (2) whether the data are accessible? (3) whether specific data are shared? (4) whether additional documentation are accessible? (5) whether provide data sharing dates? (6) whether designated contact person exists? (7) provide type of analysis? (8) whether code/algorithms/software be provided? (9) whether there exists a direct linkage to the data? In general, we will score the assessment based on nine items, we will define ‘reported’ as ‘0’, ‘not reported’ as ‘1’ and the total score will be the data sharing level (scores of 0–9). The higher the score, the worse of the data sharing level.

RoB 2 assessment

The RoB 2 tool will be used to assess the risk of bias of included RCTs. The eligible studies will be coded and will be randomised into two equal portions. Each of two researchers (YDeng and BT) will be responsible for one section and independently complete formal evaluation using the RoB 2 assessment form. Then, the extracted data will be cross-validated to ensure the accuracy of RoB 2 Assessment, with specific assigned as follows: YDeng will review BT’s data, BT will review YDeng’s data. When disagreement happens, the decision will be made by two senior investigators (PZ and YDuan).

A total of 28 items in 5 domains of RoB 2 will be evaluated. Each item has five possibilities: ‘Y’ for ‘yes’, ‘PY’ for ‘probably yes’, ‘PN’ for ‘probably no’, ‘N’ for ‘No’, ‘NI’ for ‘no information’. The risk of bias for each domain will be judged after each domain response, and the overall risk of bias will be judged after all domain responses. We will define ‘low risk of bias’ as ‘0’, ‘some concerns’ as ‘1’ and ‘high risk of bias’ as ‘2’.

Statistical analysis

The primary outcome of this study is the association between the registration and the risk of bias of acupuncture RCTs. The secondary outcomes are the registration percentage of acupuncture, the risk of bias of acupuncture RCTs and the association between the registration and data sharing levels of acupuncture RCTs.

All analyses will be conducted with the Stata software (Version Stata MP V.17). Statistical significance will be determined with two-sided 95% CIs, p<0.05. Descriptive (mean and 95% CI) statistics will be used to present the characteristics and the risk of bias of the included RCTs. Categorical data will be presented as numbers (n) and percentages (%).

Confounding will be evaluated using prior knowledge by directed acyclic graphs (DAGitty V.3.1).²⁷ The following covariables will be considered for inclusion in the final model: funding, publication language and publication year (see online supplemental material 2). According to Peduzzi et al,²⁸ the number of events per variable (EPV) in binary logistic regression analysis should be 10 or greater to reduce bias. This model will have one independent variable and three covariables, requiring a minimum of 40 EPV. If each number of events (overall risk of bias for acupuncture RCTs evaluated as low risk of bias or others) is greater or equal to 40. We will use binary logistic regression analyses, we will use ORs and 95% CIs of the overall RoB 2 and registration.

If not, we will use the five-domain summed risk of bias score (summed RoB 2 scores) as the dependent variable (scores of 0–10, the higher the score, the higher risk of bias). We will not aim to explore the mean expected value of summed RoB 2 scores, but the full distribution of summed RoB 2 scores in the regression model. Quantile regression is suitable for describing the relationship between outcomes and the conditional quantile of the dependent variable, rather than the median of the dependent variable.²⁹ We will use quantile regression to explore the summed RoB 2 score to registration over the nine (10th, 20th, 30th, 40th, 50th, 60th, 70th, 80th and 90th) percentiles. The data sharing level and registration will be analysed by quantile regression too (the quantile range of 10th, 20th, 30th, 40th, 50th, 60th, 70th, 80th and 90th).

Ethics and dissemination

This review analyses existing studies and involves no patients or members of the public so ethics committee approval was not sought. The included studies must adhere to the Declaration of Helsinki and current ethical norms. The findings of this study will be disseminated through publication in a peer-reviewed journal or pertinent conference.

The proportion of registrations can only be effectively increased by fully clarifying the role that registrations play in acupuncture RCTs. In this study, we will use quantile regression to explore the full distribution of conditional quantile of RoB 2 summed score in acupuncture RCTs. It will help to elucidate the extent of the role of registration in acupuncture RCTs. These data will provide reliable quantitative evidence for improving the quality of acupuncture RCTs and for promoting the improvement of registration in acupuncture RCTs.

Ethics statements

Patient consent for publication

Not applicable.

Footnote

Contributors YDuan and PZ conceptualised and supervised the study. YDuan, ZX and PZ developed the search strategies and made the assessment standards. YDuan and YDeng drafted the manuscript. PZ and LY revised the manuscript. BT, ZX, XY, ML, SL, WZ and LX provided critical comments and substantially improved the quality of the manuscript. YDuan and LY provided funding for the manuscript. All authors provided detailed comments on earlier drafts and approved the final manuscript. The corresponding authors attest that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding This work was supported by the Guangzhou Health Science and technology project (grant number 20242A011005), Guangzhou Science and Technology Fund (grant number 2024A03J0791), Young Science and Technology Talents Fund of The Affiliated TCM Hospital of Guangzhou Medical University (grant number 2022RC07) and the Guangzhou Key Lab of Acupuncture-Moxibustion (grant number 2023A03J0776).

Disclaimer None of the fundings will play a role in any process of the study (ie, study design, analysis, interpretation of data, writing of the report, and decision to submit the article for publication). All of the authors will have full access to all the data in the study and accept the responsibility to submit it for publication.

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Sibbald B, Roland M. Understanding controlled trials. Why are randomised controlled trials important?.' BMJ. BMJ 1998; 316: 201. doi:10.1136/bmj.316.7126.201

² Oxford Centre for Evidence based Medicine. Oxford centre for evidence-based medicine: levels of evidence, Available: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/oxford-centre-for-evidence-based-medicine-levels-of-evidence-march-2009 [Accessed 20 Oct 2023 ].

³ Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001; 323: 42–6. doi:10.1136/bmj.323.7303.42

⁴ Pronk AJM, Roelofs A, Flum DR, et al. Two decades of surgical randomized controlled trials: worldwide trends in volume and methodological quality. Br J Surg 2023; 110: 1300–8. doi:10.1093/bjs/znad160

⁵ Sterne JAC, Savović J, Page MJ, et al. Rob 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019; 366: l4898. doi:10.1136/bmj.l4898

⁶ Barcot O, Boric M, Poklepovic Pericic T, et al. Risk of bias judgments for random sequence generation in cochrane systematic reviews were frequently not in line with Cochrane Handbook. BMC Med Res Methodol 2019; 19: 170. doi:10.1186/s12874-019-0804-y

⁷ Gang W-J, Fei Y-T, Liu J-P, et al. Improving Acupuncture research: progress, guidance, and future directions. Zhongguo Zhen Jiu 2023; 43: 3–7. doi:10.13703/j.0255-2930.20221205-0001

⁸ Cai Y, Shen J, Zhong D, et al. Status quo, issues, and challenges for acupuncture research evidence: an overview of clinical and fundamental studies. J Evid Based Med 2012; 5: 12–24. doi:10.1111/j.1756-5391.2012.01163.x

⁹ Li XH, Li JL, Yang JG. Thoughts on improving the design quality of acupuncture clinical randomized controlled trials. Zhen Ci Yan Jiu 2023; 48: 519–24. doi:10.13702/j.1000-0607.20220547

¹⁰ Update on trials registration: is this clinical trial fully registered?: A statement from the International Committee of Medical Journal Editors, Available: https://www.icmje.org/news-and-editorials/update_2005.html [Accessed 20 Oct 2023 ].

¹¹ TARG Meta-Research Group & Collaborators. Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses. BMJ Open 2023; 13: e076264. doi:10.1136/bmjopen-2023-076264

¹² Mathieu S, Boutron I, Moher D, et al. Comparison of registered and published primary outcomes in randomized controlled trials [published correction appears in JAMA. JAMA 2009; 302: 977–84. doi:10.1001/jama.2009.1242

¹³ Gray R, Bressington D, Thompson DR, et al. Why pre-registration of research must be taken more seriously. Nurs Rep 2023; 13: 679–81. doi:10.3390/nursrep13020060

¹⁴ Su C-X, Han M, Ren J, et al. Empirical evidence for outcome reporting bias in randomized clinical trials of acupuncture: comparison of registered records and subsequent publications. Trials 2015; 16: 28. doi:10.1186/s13063-014-0545-5

¹⁵ Won J, Kim S, Bae I, et al. Trial registration as a safeguard against outcome reporting bias and spin? A case study of randomized controlled trials of Acupuncture. PLoS One 2019; 14: e0223305. doi:10.1371/journal.pone.0223305

¹⁶ Zhang X, Tian R, Yang Z, et al. Quality assessment of clinical trial registration with traditional Chinese medicine in WHO registries. BMJ Open 2019; 9: e025218. doi:10.1136/bmjopen-2018-025218

¹⁷ Clinical trial registry, Available: http://ClinicalTrials.gov [Accessed 20 Oct 2023 ].

¹⁸ Taichman DB, Sahni P, Pinborg A, et al. Data sharing statements for clinical trials. BMJ 2017; 357: j2372. doi:10.1136/bmj.j2372

¹⁹ Li R, von Isenburg M, Levenstein M, et al. COVID-19 trials: declarations of data sharing intentions at trial registration and at publication. Trials 2021; 22: 153. doi:10.1186/s13063-021-05104-z

²⁰ Vazquez E, Gouraud H, Naudet F, et al. Characteristics of available studies and dissemination of research using major clinical data sharing platforms. Clin Trials 2021; 18: 657–66. doi:10.1177/17407745211038524

²¹ Gaba JF, Siebert M, Dupuy A, et al. Funders' data-sharing policies in therapeutic research: a survey of commercial and non-commercial Funders. PLoS One 2020; 15: e0237464. doi:10.1371/journal.pone.0237464

²² Tedersoo L, Küngas R, Oras E, et al. Data sharing practices and data availability upon request differ across scientific disciplines. Sci Data 2021; 8: 192. doi:10.1038/s41597-021-00981-0

²³ OSF. Available: https://osf.io/gza9v

²⁴ Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015; 4: 1. doi:10.1186/2046-4053-4-1

²⁵ Risk of bias 2 (rob 2) tool. Available: https://www.cochranelibrary.com/ [Accessed 20 Oct 2023 ].

²⁶ Duan Y, Xu Z, Li X, et al. Reporting and data-sharing level of acupuncture randomised controlled trials: a cross-sectional study protocol. BMJ Open 2023; 13: e070545. doi:10.1136/bmjopen-2022-070545

²⁷ DAGitty v3.1, Available: https://dagitty.net/dags.html [Accessed 20 Oct 2023 ].

²⁸ Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49: 1373–9. doi:10.1016/s0895-4356(96)00236-3

²⁹ Staffa SJ, Kohane DS, Zurakowski D. Quantile regression and its applications: a primer for Anesthesiologists. Anesth Analg 2019; 128: 820–30. doi:10.1213/ANE.0000000000004017