Methodological Reporting of Randomized Trials in Five Leading Chinese Nursing Journals

Abstract

Background

Randomized controlled trials (RCTs) are not always well reported, especially in terms of their methodological descriptions. This study aimed to investigate the adherence of methodological reporting complying with CONSORT and explore associated trial level variables in the Chinese nursing care field.

Methods

In June 2012, we identified RCTs published in five leading Chinese nursing journals and included trials with details of randomized methods. The quality of methodological reporting was measured through the methods section of the CONSORT checklist and the overall CONSORT methodological items score was calculated and expressed as a percentage. Meanwhile, we hypothesized that some general and methodological characteristics were associated with reporting quality and conducted a regression with these data to explore the correlation. The descriptive and regression statistics were calculated via SPSS 13.0.

Results

In total, 680 RCTs were included. The overall CONSORT methodological items score was 6.34±0.97 (Mean ± SD). No RCT reported descriptions and changes in “trial design,” changes in “outcomes” and “implementation,” or descriptions of the similarity of interventions for “blinding.” Poor reporting was found in detailing the “settings of participants” (13.1%), “type of randomization sequence generation” (1.8%), calculation methods of “sample size” (0.4%), explanation of any interim analyses and stopping guidelines for “sample size” (0.3%), “allocation concealment mechanism” (0.3%), additional analyses in “statistical methods” (2.1%), and targeted subjects and methods of “blinding” (5.9%). More than 50% of trials described randomization sequence generation, the eligibility criteria of “participants,” “interventions,” and definitions of the “outcomes” and “statistical methods.” The regression analysis found that publication year and ITT analysis were weakly associated with CONSORT score.

Conclusions

The completeness of methodological reporting of RCTs in the Chinese nursing care field is poor, especially with regard to the reporting of trial design, changes in outcomes, sample size calculation, allocation concealment, blinding, and statistical methods.

About the Authors:

Chunhu Shi

Affiliations Evidence-Based Medicine Center, Lanzhou University, Lanzhou, China, Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China

Jinhui Tian

Affiliations Evidence-Based Medicine Center, Lanzhou University, Lanzhou, China, Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China

Dan Ren

Affiliation: The First Clinical Medicine School, Lanzhou University, Lanzhou, China

Hongli Wei

Affiliation: The Second Clinical Medical School, Lanzhou University, Lanzhou, China

Lihuan Zhang

Affiliation: The Second Clinical Medical School, Lanzhou University, Lanzhou, China

Quan Wang

Affiliations Evidence-Based Medicine Center, Lanzhou University, Lanzhou, China, Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China, The First Clinical Medicine School, Lanzhou University, Lanzhou, China

Kehu Yang

* E-mail: [email protected]

Affiliations Evidence-Based Medicine Center, Lanzhou University, Lanzhou, China, Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China

Introduction

The randomized controlled trial (RCT) is the most appropriate design for evaluating the effectiveness of a nursing intervention among eligible primary study designs for evidence-based practice [1]. Evidence-based nursing practitioners believe that randomization and unbiased allocation of interventions assure methodological quality and the subsequent findings of RCTs [2], [3]. However, well-conducted trials may be reported poorly [4], and failing to report all of the study's methods, details, and findings, especially for RCTs, seriously distorts the evidence base for clinical decision-making [5].

As a set of evidence-based reporting guidelines for RCTs, the Consolidated Standards of Reporting Trials (CONSORT) [6] was developed to improve the transparency of reporting and interpreting RCTs, minimize biased conclusions, ultimately facilitate clinical decision making, and indirectly drive better design of trials. The CONSORT guidelines have been associated with improving the quality of RCT reporting [7]. To date, CONSORT has been accepted by 600 medical journals, including 20 nursing care journals in English [8], and it has been revised twice, in 2001 and 2010 [9].

For RCTs in various disciplines (e.g., diagnostic procedures, interventions, cancer trials, and alternative medicine), the reporting quality has been assessed via CONSORT [5] to ensure their transparency, and found widespread inadequate reporting of methodological issues. However, evidence is still deficient on the reporting quality of RCTs in nursing care, especially in China. Existing reviews have focused on only a specific discipline [10], [11], were not conducted systematically [12], [13], used a modified CONSORT checklist according to the purposes of the study [14], or focused on only a specific methodological issue [15], [16]. This study was thus conducted to investigate (1) the adherence to methodological reporting complying with CONSORT on Chinese nursing RCTs, and (2) the correlation between CONSORT adherence and key indicators regarding characteristics and methodological factors.

Methods

Information Source

According to their total number of citations and impact factors (IFs) in the “Chinese S&T Journal Citation Reports 2012” [17], we selected the following five leading Chinese nursing journals(see Appendix S1): Chinese Journal of Nursing (CJN, 1954–), Chinese Journal of Modern Nursing (CJMN, 1995–), International Journal of Nursing (IJN, 1980–), Chinese Journal of Practical Nursing (CJPN, 1985–) and Journal of Nurses Training (JNT, 1986–). In June 2012, a comprehensive electronic search strategy for the Chinese Bio-Medicine disc (CBM; see Appendix S2) was performed using “randomized controlled trial” or “clinical trial” as keywords, limited to these five targeted journals.

Eligibility Criteria

Inclusion criteria.

1. Citations claiming to be RCTs in the titles and/or abstracts, according to the “Cochrane definitions and criteria for randomized controlled trials and controlled clinical trials” [18].

2. Controlled trials with two comparators.

3. Trials of both nursing interventions (e.g., drug interventions, social and psychological interventions, traditional Chinese medicine interventions, complex interventions) and non-nursing interventions (e.g., nursing education program interventions).

Exclusion criteria.

1. “Quasi-random” trials: when “random” was mentioned with a high-risk randomization method, such as alternation, assignment based on date of birth, case record number, or date of presentation.

2. Irrelevant study reports: reports that are published in a summary form, reviews of randomized trials, and randomized trials on animals, even though they may be reported as “random.”

3. Other study designs, such as cohort study and case series report.

Study Selection and Data Extraction

All citations from the CBM were allocated evenly to three units. By first screening titles and abstracts and then screening fulltexts, three groups consisting of two authors each selected eligible studies from these units of citations independently. When a disagreement arose in any of groups, another group would participate to arrive at an agreement. Then, two authors extracted relevant information and data from the included studies. A well-designed extraction form (Appendix S3) was formulated to assess the reporting quality of RCTs according to CONSORT [6]. The extraction form included (1) general characteristics [5] (i.e., journal name, IF, publication date, number of authors, multicenter trials, types of interventions, sample size, and funding source) and a description of the interventions [19], including length of descriptions of interventions and controls in words and the reproducibility of the interventions, which authors evaluated by assessing whether the intervention was described in enough detail to be reproducible; (2) CONSORT methodological items; and (3) methodological quality measurements [20], including allocation concealment and intention-to-treat (ITT) analysis. The overall CONSORT methodological items score was the total score of 17 items with response options of “yes” (scored as 1) or “no” (scored as 0) on whether the component had been reported as recommended. Items were scored as “no” (i.e., 0) if the item was only partly met by the research report. Disagreements during data extraction were resolved by discussion.

Quality Control

We implemented rigorous quality control throughout the study. Before selecting the eligible studies, investigators were trained to distinguish RCTs from irrelevant study designs using the definitions from the Centre for Evidence-Based Medicine, Oxford, UK (University of Oxford) [21]. Before data extraction, the extraction form (Appendix S3) following CONSORT guidelines was established via three rounds of pre-survey and team discussion.

Data Analysis

Using SPSS 13.0, we performed a descriptive analysis, obtaining frequencies and percentages for categorical variables, and means, medians, standard deviations, and inter-quartile ranges (IQR) for continuous variables. Then, exploratory regression was conducted to examine whether general and methodological characteristics are associated with reporting quality, as measured by the CONSORT methodological items score. Univariate regression was performed for each potentially associated variable, and variables with P values lower than 0.25 were then included in a stepwise backward linear regression. Any P value lower than 0.05 was regarded as statistically significant. In this study, the sample sizes of the trials were included to power the exploratory linear regression.

Results

Study Inclusion

In total, 58,823 citations published in the five leading nursing journals from 1954 to June 2012 were retrieved via the “Journal Search Engine” function of the CBM disc. After screening abstracts and fulltexts, 5519 fulltexts that mentioned “random” were initially identified. Among them, 5502 trials described “randomization” in both abstracts and fulltexts, and 17 described “randomization” in only the fulltexts. Six hundred and eighty trials met the criteria for “details of randomization” and were included in the final analysis. Figure 1 depicts a flow diagram of the inclusion decisions.

[Figure omitted. See PDF.]

Figure 1. Flow diagram of included studies.

https://doi.org/10.1371/journal.pone.0113002.g001

Characteristics of Included Studies

Most of the included trials (99.9%) were published after the 2001 CONSORT (Appendix S4). Furthermore, 27.6% (188/680) of trials were published in CJN and JNT, which had higher IFs than the average (mean ± SD  = 1.20±1.05); 61% (415/680) of trials were reported by no more than three authors (i.e., the median number of authors). Two point five percent (17/680) of trials reported a collaboration between multicenter studies. However, 21.6% of these trials did not specify the number of clinical trial centers. Most trials (83.4%) studied the implementation of non-pharmacological interventions. The median sample size of the included studies was 90 (IQR  = 62.75). Most trials (75.6%) did not mention financial support. By our judgment, 85.4% of trials provided enough details in the description of interventions to facilitate replication; however, only 48.2% of control group protocols were sufficiently described. Nisnety percent of trials from CJPN described the control group protocol as “the control group used standard nursing inventions for this disease.” There were significant differences in the number of words used to describe the interventions and controls (median  = 215.0 and IQR  = 422.75 for interventions, and median  = 32.0 and IQR  = 47.0 for controls, Z = −21.192, P<0.001). For methodological quality measurements, only two trials (0.3%) reported the details of the allocation concealment mechanism; 85.3% of trials had no dropouts, and 5.7% conducted per protocol analysis. Table 1 depicts the characteristics of the included studies.

[Figure omitted. See PDF.]

Table 1. Characteristics of Included Studies.

https://doi.org/10.1371/journal.pone.0113002.t001

Reporting of CONSORT Methodological Items

When screening abstracts rather than fulltexts, 97.9% of RCTs could be identified. Only two trials (0.3%) reported “randomized control trial” in the title. Overall, the mean CONSORT methodological score was 6.34±0.97 out of 17 sub-items. Additionally, in order to explore the influence of the score of “blinding (11a, 11b)”, an analysis excluding data on blinding showed that the mean and SD of the CONSORT score were 6.28 and 0.94 respectively, which were similar to the mean score obtained while including data on blinding. No RCT reported descriptions and changes in “trial design,” changes in “outcomes” and “implementation,” or descriptions of the similarity of interventions for “blinding.” For the description of trial design, 99.7% of the studies used a parallel design while 0.3% used a crossover design. Sub-standard reporting was also found regarding the details of “settings of participants” (13.1%), “type of randomization sequence generation” (1.8%), calculation methods of “sample size” (0.4%), explanation of any interim analyses and stopping guidelines of “sample size” (0.3%), additional analyses of “statistical methods” (2.1%), and targeted subjects and methods of “blinding” (5.9%). In addition, no central randomization that implied allocation concealment was reported. Only two studies conducted allocation concealment through sequentiallynumbered opaque sealed envelopes. More than 50% of trials described randomization sequence generation, the eligibility criteria of participants, “interventions,” and definitions of “outcomes” and “statistical methods.” Table 2 depicts the reporting of individual CONSORT methodological items.

[Figure omitted. See PDF.]

Table 2. Reporting of individual CONSORT methodological items.

https://doi.org/10.1371/journal.pone.0113002.t002

Predictors Associated with CONSORT Methodological Score

We explored the association of predictors, including general characteristics and methodological quality measurements, with CONSORT methodological score with a linear regression and found that publication year and ITT analysis were weakly associated with CONSORT score. Each predictor was associated with a small increase of 0.047 and 0.601 points to the overall CONSORT methodological score. The two predictors associated with CONSORT score (i.e. publication year and ITT analysis) remained after the data on “blinding” were excluded, and their P values were 0.005 and <0.001 respectively. Table 3 depicts the results of our linear regression.

[Figure omitted. See PDF.]

Table 3. Results of linear regression for predictors associated with CONSORT methodological score.

https://doi.org/10.1371/journal.pone.0113002.t003

Discussion

In this review, we investigated 680 RCT fulltexts using CONSORT methodological items and found that RCTs of nursing interventions in Chinese nursing care journals reported “trial design,” “participant settings,” changes in “outcome,” “sample size,” “allocation concealment mechanism,” “implementation,” “blinding,” and additional analyses of “statistical methods” poorly. The overall CONSORT score was low. Weak correlations between CONSORT score and two variables (i.e., publication year and ITT analysis) were found.

Concordant with our findings, Zhang et al. [11] showed that the reporting of methodological items in cancer nursing care literature was also poor in terms of outcomes, sample size, randomization sequence generation, allocation concealment mechanism, implementation, and blinding. Zhu et al. [15] demonstrated that only 3.8% of RCTs of nursing interventions described randomization sequence generation methods; furthermore, in traditional Chinese medicine nursing studies [10], 7.8% of RCTs described randomization sequence generation, 1.4% described the allocation concealment mechanism, and 3.1% described blinding. Evidence from RCTs of nursing interventions in English [13], [14], [16] also supported our findings. Samaan et al. [5] reviewed systematic reviews of adherence to reporting guidelines across different clinical areas and found a prevailing lack of reporting quality in most disciplines. Additionally, Samaan et al. [5] showed that 80% of the studies that did not adhere to CONSORT guidelines had negative conclusions, such as “some improvements are needed, reporting inadequate/poor/medium/suboptimal, etc.” Our findings are consistent with other systematic reviews in other specializations [19], [22]. Given these findings, we have provided important evidence to alert Chinese nursing researchers of the urgent need for improving the reporting of RCTs on nursing interventions. This is especially true for randomization, as we found that 87.7% (4839/5519) of RCTs that mentioned “random” in abstracts and/or fulltexts did not actually include details of the randomization or used incorrect randomized methods, such as alternate allocation and allocation by record number. Similar results were reported by Wu et al. [24].

We found poorer reporting of blinding in 83.4% of non-pharmacological interventions, but no correlation was established via the regression analysis. Due to some methodological challenges, such as the complexity of the intervention [23], expertise of the care provider [23], and difficulties with blinding [16], it is believed that non-pharmacological interventions always limit the blinding of participants and caregivers; however, the blinding of data collectors is often achievable. Regardless of whether blinding is possible, authors can and should state who was blinded. Therefore, although it seemed unnecessary in some included studies, blinding was judged as impracticable because the authors did not report its details. Our regression analysis indicated that publication year had a weak correlation with reporting quality. This demonstrates that reporting quality is improving yearly. We also examined the relationship between reporting quality and methodological quality of RCTs by regression of allocation concealment and ITT analysis. The regression indicated only weak correlations with ITT analysis, and no correlations with the other variable. In order to explore the influence of the score of “blinding (11a, 11b)” on regression analysis, data on “blinding” were deleted, and the results showed that the two same predictors (i.e. publication year and ITT analysis) were associated with CONSORT score analysis.

The main strengths of this study were the comprehensive search strategy, determination of associations of CONSORT score with various predictors, and the well-conducted quality control assessment. However, this review had some limitations. First, nursing interventions are often far more complicated than drug interventions are. We did not investigate the reporting of items related to external validity. Second, as our eligible criteria was not restricted to any specific nursing intervention such as drug interventions, social and psychological interventions, traditional Chinese medicine interventions, or complex interventions, we did not use CONSORT extensions for social and psychological interventions, non-pharmacological interventions, herbal interventions, and CReDECI criteria [25]–[28]. In addition, we did not find a pragmatic trial among the included studies, so the CONSORT extension for pragmatic trials [29] was also not used. Third, according to our eligibility criteria, no dissertation in nursing care was included. A search on the Internet in the Doctoral Dissertations and Masters' Theses sub-set of the China Knowledge Resource Integrated Database yielded 2312 citations dating up to June 2013 that mentioned “random.” Finally, because none of the studied nursing journals endorsed CONSORT as a requirement for a manuscript submission in China, we did not investigate differences in the reporting between each item by comparing CONSORT-endorsing journals with CONSORT non-endorsing journals.

Conclusions

In conclusion, the completeness of methodological reporting of RCTs in Chinese nursing care journals was very poor, especially in terms of trial design, outcome changes, sample size calculation, allocation concealment, blinding, and statistical methods. Authors are highly recommended to use the CONSORT criteria when submitting reports of RCTs. Editors should also present their peer-reviewed manuscripts under the guidance of CONSORT to ensure that journal articles contain the key clinical and methodological details that readers need. Initiatives recommended by Altman [30] should be accepted; these suggest that authors should sign a publication transparency declaration as part of every journal submission and journals should update their instructions to authors.

Supporting Information

[Figure omitted. See PDF.]

Appendix S1.

The total cites and impact factors in five Chinese nursing journals.

https://doi.org/10.1371/journal.pone.0113002.s001

(DOC)

Appendix S2.

Comprehensive electronic search strategy for CBM disc.

https://doi.org/10.1371/journal.pone.0113002.s002

(DOC)

Appendix S3.

Data extraction form.

https://doi.org/10.1371/journal.pone.0113002.s003

(DOC)

Appendix S4.

Number of RCTs in each publication year.

https://doi.org/10.1371/journal.pone.0113002.s004

(TIF)

Checklist S1.

PRISMA Checklist.

https://doi.org/10.1371/journal.pone.0113002.s005

(DOC)

Acknowledgments

Thanks to Daniel James Green and Editage for providing important comments and English editing. We paid for the services of Editage and there were no conflicts of interest.

Author Contributions

Conceived and designed the experiments: CS JT LZ. Performed the experiments: CS JT LZ HW DR QW KY. Analyzed the data: CS LZ KY. Contributed reagents/materials/analysis tools: KY HW. Wrote the paper: CS DR QW KY. Addressed disagreements: KY.

Citation: Shi C, Tian J, Ren D, Wei H, Zhang L, Wang Q, et al. (2014) Methodological Reporting of Randomized Trials in Five Leading Chinese Nursing Journals. PLoS ONE 9(11): e113002. https://doi.org/10.1371/journal.pone.0113002

References

1. DiCenso A, Cullum N, Ciliska D (1998) Implementing evidence-based nursing: some misconceptions. Evidence-Based Nursing 1: 38–40.

2. Cullum N (2000) Evaluation of studies of treatment or prevention interventions. Evidence Based Nursing 3: 100–102.

3. Borglin G, Richards DA (2010) Bias in experimental nursing research: Strategies to improve the quality and explanatory power of nursing science. Int J Nurs Stud 47: 123–128.

4. Huwiler-Müntener K, Jüni P, Junker C, Egger M (2002) Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 287: 2801–2804.

5. Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, et al. (2013) A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc 6: 169–188.

6. Schulz KF, Altman DG, Moher D; CONSORT Group (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 152: 726–732.

7. Turner L, Shamseer L, Altman DG, Weeks L, Peters J, et al. (2012) Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 11: MR000030.

8. CONSORT Group. CONSORT Endorsers - Journals. Available: http://www.consort-statement.org/about-consort/consort-endorsement/consort-endorsers-journals/. Accessed: 2013 Dec 26.

9. Altman DG, Moher D, Schulz KF (2012) Improving the reporting of randomised trials: the CONSORT Statement and beyond. Stat Med 31: 2985–2997.

10. Mei LJ, Zheng GH, Chen QY, Lin R, Yan Y, et al. (2012) Methodological Evaluation on Domestic Clinical Trials on Traditional Chinese Medicine Nursing from 2006 to 2011. Chin J Evid-based Med 12: 735–739.

11. Zhang XX, Yin WR, Liu YQ (2009) Reporting quality assessment on randomized controlled trials of cancer nursing care: based on CONSORT statement. Chinese Journal of Practical Nursing 25: 72–73.

12. Li XL, Tian YM, Zen JH, Liu L (2002) Evaluation on the randomized controlled trials papers published in Journal of Nurses Training. Journal of Nurses Training 17: 16–18.

13. Parent N, Hanley JA (2009) Assessing quality of reports on randomized clinical trials in nursing journals. Can J Cardiovasc Nurs 19: 25–39.

14. Smith BA, Lee HJ, Lee JH, Choi M, Jones DE, et al. (2008) Quality of reporting randomized controlled trials (RCTs) in the nursing literature: application of the consolidated standards of reporting trials (CONSORT). Nurs Outlook 56: 31–37.

15. Zhu D, Huang WX, Li XL, Jiang XL, Wang SP (2003) Investigation on randomized controlled trials of nursing care field in China. Journal of Practical Nursing 19: 60–61.

16. Polit DF, Gillespie BM, Griffin R (2011) Deliberate ignorance: a systematic review of blinding in nursing clinical trials. Nurs Res 60: 9–16.

17. Institute of Scientific and Technical Information of China (2012) Chinese S&T journal citation reports. Beijing: Scientific and Technical Documents Publishing House 350 p.

18. Higgins JPT, Green S (2011) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available: www.cochrane-handbook.org. Accessed: 2013 Dec 26.

19. Adie S, Harris IA, Naylor JM, Mittal R (2013) CONSORT compliance in surgical randomized trials: are we there yet? A systematic review. Ann Surg 258: 872–878.

20. Boutron I, Tubach F, Giraudeau B, Ravaud P (2003) Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. JAMA 290: 1062–1070.

21. University of Oxford (2010) Study Designs. The Centre for Evidence Based Medicine, University of Oxford, Available: http://www.cebm.net/index.aspx?o=1039. Accessed: 2012 Jul 1.

22. Xu L, Li J, Zhang M, Ai C, Wang L (2008) Chinese authors do need CONSORT: reporting quality assessment for five leading Chinese medical journals. Contemp Clin Trials 29: 727–731.

23. Stephenson J, Imrie J (1998) Why do we need randomised controlled trials to assess behavioural interventions? BMJ 316: 611–613.

24. Wu T, Li Y, Bian Z, Liu G, Moher D (2009) Randomized trials published in some Chinese journals: how many are randomized? Trials 10: 46.

25. Mayo-Wilson E, Montgomery P, Hopewell S, Macdonald G, Moher D, et al. (2013) Developing a reporting guideline for social and psychological intervention trials. Br J Psychiatry 203: 250–254.

26. Möhler R, Bartoszek G, Köpke S, Meyer G (2012) Proposed criteria for reporting the development and evaluation of complex interventions in healthcare (CReDECI): guideline development. Int J Nurs Stud 49: 40–46.

27. Gagnier JJ, Boon H, Rochon P, Moher D, Barnes J, et al. (2006) Reporting randomized, controlled trials of herbal interventions: an elaborated CONSORT statement. Ann Intern Med 144: 364–367.

28. Boutron I, Moher D, Altman DG, Schulz K, Ravaud P, for the CONSORT group (2008) Methods and Processes of the CONSORT Group: Example of an Extension for Trials Assessing Nonpharmacologic Treatments. Ann Intern Med 2008: W60–W67.

29. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, et al. (2008) Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 337: a2390.

30. Altman DG, Moher D (2013) Declaration of transparency for each research article. BMJ 347: f4796.