Adherence to ARRIVE Guidelines in Chinese Journal Reports on Neoplasms in Animals

Abstract

Background

The Animals in Research: Reporting In Vivo Experiments (ARRIVE) guidelines were published in 2010 with the aim of improving the quality of studies involving animals. However, how well Chinese studies involving animal neoplasms adhere to these guidelines has not been assessed.

Objective

To evaluate the reporting quality of such experiments published between 2010 and 2012 in Chinese journals with support from the National Natural Science Foundation of China.

Methods

We searched the Chinese Science Citation and Chinese Journal Full-Text Databases for articles published between 2010 and 2012 involving neoplasms in animals. The data were extracted into pre-prepared forms. Reporting quality was assessed using the ARRIVE checklist—39 items plus information on blinding.

Results

Three hundred and ninety-six animal studies were included in the analysis: 127 studies published in 2010, 140 studies published in 2011, and 129 studies published in 2012. The range of ARRIVE score is from 12 to 27 with a maximum possible score of 40. Studies published in 2012 (P = 0.012), 2011 (P = 0.015), 2010, July~Dec (P<0.017) had a significantly larger ARRIVE checklist score than those published in Jan.~June, 2010, respectively.

Conclusions

Experiments involving neoplasms in animals published in Chinese journals generally have not comprehensively reported the information recommended by the ARRIVE guidelines. We strongly recommend that researchers conducting such studies report this information.

About the Authors:

Yali Liu

* E-mail: [email protected]

Affiliations Evidence Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China, Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou, China, Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Dongzhimen, Beijing, China

Xingxing Zhao

Affiliation: Peking University Health Science Center, Beijing, China

Yuefen Mai

Affiliations Evidence Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China, The Second Clinical Medical College of Lanzhou University, Lanzhou, China

Xinxin Li

Affiliation: The First Clinical Medical College of Lanzhou University, Lanzhou, China

Jin Wang

Affiliation: The Second Clinical Medical College of Lanzhou University, Lanzhou, China

Lili Chen

Affiliation: The First Clinical Medical College of Lanzhou University, Lanzhou, China

Jing Mu

Affiliation: The Second Clinical Medical College of Lanzhou University, Lanzhou, China

Gengxue Jin

Affiliation: The Second Clinical Medical College of Lanzhou University, Lanzhou, China

Hongping Gou

Affiliation: The First Clinical Medical College of Lanzhou University, Lanzhou, China

Wanting Sun

Affiliations Evidence Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China, The First Clinical Medical College of Lanzhou University, Lanzhou, China

Yuchen Feng

Affiliation: The Second Clinical Medical College of Lanzhou University, Lanzhou, China

Introduction

Animals are often used during biomedical research studies, but such studies are often controversial because there are many differences between animals and humans. The aims of preclinical animal experiments are to perform preliminary safety and efficacy validations of the (new) intervention under study, with the results determining whether the intervention should be further assessed in clinical studies and trials. Every year, many animal experiments, including those supported by foundations and other organizations, are published in a variety of journals. In China, the National Natural Science Foundation of China (NSFC) is one of the major funding sources for basic science research.

Although random allocation and blinding are common in clinical trials, they are not common in animal experiments that precede clinical trials. One study[1] assessed 290 published animal experiments and reported that 32.41% (94/290) used randomization and 9.66% (28/290) used blinding. Non-randomized, non-blinded animal studies are more likely to report a difference between study groups than animal studies that use these methods[1]. Experimental design, statistical analysis, and reporting issues have also been found for animal studies [2,3].

The Animals in Research: Reporting In Vivo Experiments (ARRIVE) guidelines published in 2010 were developed to improve the transparency and accuracy of bioscience research reporting. The guidelines detail the minimum information that should be reported when using animals in a research study and include a 20-item checklist[4]. In 2011, the ARRIVE guidelines were introduced in China[5]. To date, no study has assessed whether animal experiments of neoplasms published in Chinese journals adhere to these guidelines.

The aim of this study was to determine the reporting quality of experiments, specifically those involving neoplasms in animals, which were published in Chinese journals between 2010 and 2012 and were supported by the NSFC.

Methods

Inclusion/exclusion criteria

The targeted studies were those involving neoplasms in living rodents, including rats, mice, nude mice, and guinea pigs (Mammalia: Rodentia), published between 2010 and 2012 in Chinese journals, and supported by the NSFC.

We excluded studies on animals that were dead before the start of the experiment, when the studies were invitro, when no intervention was performed and no control group was included, when the studies focused on precancerosis, or when the studies were part of an academic dissertation or review article.

Search strategy

We comprehensively and systematically searched the Chinese Science Citation Databases (CSCD) and the Chinese Journal Full-Text Databases (CJFD) on July 13, 2012 for all entries submitted from January 2010 to June 2012, and on June 19, 2013 for all entries submitted from July 2012 to December 2012. The main search terms were “neoplasm,” “animal experiment, in vivo experiment, basic research,” and “National Natural Science Foundation.” The search strategy is presented in S1 File.

Screening

The titles and abstracts were independently screened by at least two reviewers (Xingxing Zhao, Yuefeng Mai, Xinxin Li, and/or Wanting Sun). Then, the full texts of potentially suitable articles were retrieved based on the inclusion and exclusion criteria. Disagreements concerning the suitability of an article were resolved by group discussions.

Data extraction

Extraction into a pilot-tested standardized data form based on the ARRIVE guidelines was performed independently by at least two reviewers (Xingxing Zhao, Yuefeng Mai, Xinxin Li, Jin Wang, Lili Chen, Jing Mu, Gengxue Jin, and/or Hongping Gou). Inconsistencies were subsequently resolved by discussion among two or more reviewers, or the principal investigator Yali Liu made the final decision. The form consisted of two sections: (1) general characteristics (publication time, role of first author, the condition of interest, the allocation method of the included animals, and the number of funding organizations), (2) the ARRIVE information (39 items), and information about blinding that was separated from the checklist as item 6f and written as “Describe the information: If done, describe who was blinded (for example, outcome assessors) and how”, i.e., for a total of 40 scored items. Each item was assessed as “yes” (the item was described in the study) or “no” (the item was not described in the study).

Data analysis

The data were summarized using Microsoft Excel (Version 2007; http://office.microsoft.com/zh-cn) and SPSS software (Version 21.0; http://www.spss.com). For continuous variables (for example the ARRIVE checklist scores), we expressed results as median and quartile (P₂₅, P₄₅) and comprised them for Kruskal-Wallis H Test. We use the traditional 0.05 definition of significance. The Bonferroni method was used for correcting for multiple comparisons. We mainly focus on a contrastive analysis of animal experiments before and after the publication of ARRIVE guidelines. We made 3 comparisons, then the new threshold is 0.017 (0.05/3).For categorical variables (for example the reported rate of ARRIVE checklist items), we expressed result as frequently and comprised them for the chi-square test.

Results

The literature search

We initially identified 2846 studies. Of these, 1889 were excluded given their title or abstract, and a further 520 were excluded after an assessment of their full texts because they did not meet the inclusion criteria. The full texts of an additional 41 articles were not available. The remaining 396 studies met the inclusion criteria and were included in the analysis (Fig 1 and S2 File).

[Figure omitted. See PDF.]

Fig 1. Flow chart of articles identified, included and exclude.

https://doi.org/10.1371/journal.pone.0154657.g001

General Characteristics of the Included Studies

The characteristics of the included studies are shown in Table 1. Of these studies, 126 (31.82%), 141 (35.61%), and 129 (32.58%) were published in 2010, 2011, and 2012, respectively. There were 77 studies which were published the release of the before ARRIVE guideline. For slightly more than half of the studies (219/396; 55.30%), the first author was affiliated with a hospital. The most commonly studied conditions were liver and lung cancers. The majority of the studies randomly assigned animals to the experimental or control group, but only 26.77% (106/396) described the randomization method in detail. Eleven studies used a random number table to perform randomization. Other randomization methods are described in Table 2.A total of 159 studies (40.15%) were supported by only the NSFC. The other 237 studies (59.85%) were supported by up to five sources.

[Figure omitted. See PDF.]

Table 1. Characteristics of included studies.

https://doi.org/10.1371/journal.pone.0154657.t001

[Figure omitted. See PDF.]

Table 2. Reporting of checklists for ARRIVE Guidelines.

https://doi.org/10.1371/journal.pone.0154657.t002

Conformity with ARRIVE Guidelines

The ARRIVE checklist scores are shown in Table 2. The range of ARRIVE score is from 12 to 27 with a maximum possible score of 40.The value for each of the median (P25, P45) ARRIVE checklist scores for studies published during January and June2010, July and December 2010, 2011, and 2012 was18.50(17.00,20.00), 19.00 (18.00,21.00), 19.00 (18.00,21.00) and 20.00(18.00,22.00), respectively. Studies published in 2012(P = 0.012), 2011 (P = 0.015), 2010, July~Dec (P<0.017) had a significantly larger ARRIVE checklist score than those published in Jan.~June,2010,respectively.

The least frequently reported items (reported in ≤30% of the studies) were items 3b, 5, 6d, 6f, 7c, 7d, 9b, 9c, 10b, 10c, 11a, 13b, 13c, 14, 15a, 15b, 17a, 17b, 18b, 18c, and 19. These items correspond to information on the scientific background, ethical statements, experimental procedures, sample size, statistical methods, baseline data, numbers of animals analyzed, adverse events, and interpretation/scientific implications. No study provided a time-line diagram or flow chart, information on sample size, information on how the experimental protocol reduced adverse events, or experimental methods or findings for the replacement, refinement, or reduction (the 3Rs) of the use of animals in research. The frequency of reporting the ARRIVE checklist items 7b, 7c, 9a, 9c, 10a, 16, 18b, and 19 differed according to the publication year, but the reporting frequencies for all other individual ARRIVE checklist items were similar for the three years.

Discussion

Over the last decade, numerous studies have examined the quality of healthcare reporting by assessing the compliance of randomized controlled trials[6],observational studies[7], and systematic reviews[8] with various assessment instruments. The CONSORT statement published in 1996[9], the ARRIVE guidelines[4], and the Gold Standard Publication Checklist (GSPC)[10] published in 2010 represent substantial improvements in methods used for animal studies. Before these guidelines, insufficient reporting occurred in many animal experiments, e.g., a lack of randomization and/or blinding[11–13]. The reporting quality of animal experiments in periodontology, e.g., implant dentistry, published after 2010 has been assessed using the ARRIVE and modified ARRIVE guidelines[2,3,14–17].

In the present study, we focused on 396 reports involving neoplasms in rodents published in Chinese journals between 2010 and 2012. To our knowledge, this is the first assessment of this type of study published in Chinese journals and supported by the NSFC. Although we did not perform an intervention systematic review, we tried to conform to the preferred reporting items for the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA statement) checklist[8]. In our study, we found that the completeness of these studies regarding the ARRIVE guidelines was suboptimal. The mean ARRIVE checklist score was only 19.48 out of 40, and many studies failed to report important information. Furthermore, half the items in the ARRIVE checklist were reported by <50% of studies. Items 3b, 5, 6d, 6f, 7c, 7d, 9b, 9c, 10b, 10c, 11a, 13b, 13c, 14, 15a, 15b, 17a, 17b, 18b, 18c, and 19 were particularly poorly reported.

All studies included in this analysis were supported by at least one funding organization. Only two studies reported that no conflict of interest existed, and the other studies did not provide a declaration of interests. Accurate reporting of conflicts of interest is very important to enable readers to judge the risk of publication bias.

We found that few studies (1.77%) provided a statement of ethics or provided information on animal welfare. Researchers need to pay close attention to study design, data collection, reporting, and the welfare of the animals to take effective measures to alleviate animal suffering. The ARRIVE guidelines recommend providing a time-line diagram or flow chart of the study design, but neither was part of any study. Similarly, no study described how the sample size was calculated, and therefore it was unclear if the sample sizes were adequate. To minimize the effect of random error on the results, studies should include independent repetitions of each experiment. Although many studies reported the number of repeated measurements, many failed to describe the number of independent replications for each experiment.

As with clinical trials, animal experiments should report if the subjects were randomized into groups and also describe the randomization method in detail to help readers assess the risk of selective bias. Furthermore, it is important to blind outcome assessors and data analysts to the group assignment, thereby reducing the risk of measurement bias for some subjective outcomes. However, blinding was used in only one study.

Although all studies reported the experimental outcomes, many did not define primary and secondary outcomes. Information concerning statistical methods was often incomplete, e.g., the unit of analysis for each dataset was not specified, or the rationale for selecting the statistical approach was not provided.

Only one studies (0.25%) provided the relevant characteristics and health status of the animals, e.g., the strain, sex, age, and weight of the experimental and control animals, which are important factors that allow the reader to judge if the groups were or were not balanced at baseline.

Insufficient information was often provided on the number of animals studied, and no descriptions or explanations were provided concerning animal death or loss to follow-up or incomplete outcome data, making it difficult for readers to evaluate the risk of attrition bias. In addition, although all studies reported the efficacy of the treatment, only 41 (10.35%) reported adverse events, and none described the approach(s) used to reduce adverse events.

Ideally, the discussion section of such studies should address how the results translate to other systems or species and the feasibility of testing the treatment in clinical trials. When animals are used in research, the researcher has the responsibility to work toward the replacement, refinement, and reduction of animal use, i.e., to improve the use value of the animals, reduce the number of animals used, and increase the use of alternative methods. Unfortunately, none of the studies reported information about these issues.

All experimental research has limitations; however, many of the studies did not address this point.

Analysis reveals that the reporting quality of animal experiments reports shows a rising tendency after the first publication of the ARRIVE in June, 2010.However, the quality of reporting was still poor, and future studies should be more transparent and accurately reported. We believe that reporting guidelines, including those of ARRIVE and GSPC, have not been widely used by Chinese researchers who use animals, as our results indicate that neither the ARRIVE nor the GSPC guidelines were used by such researchers from 2010 July to 2012. The ARRIVE and GSPC guidelines outline the minimum information that should be provided when reporting an animal experiment(s). These guidelines should be used when designing, performing, reporting, reviewing, and publishing an animal study. They will also be of use to funding organizations, especially large funding bodies, which have a responsibility to provide financial assistance and to strictly evaluate the quality of the research that is funded[18]. Good reporting allows readers to fully understand the methods and processes involved and to assess the reliability and validity of the findings.

There are several limitations to our study. First, our analyses were limited to studies on rodents, and studies on other animals were excluded. Second, we only included animal experiments involving neoplasms that were published in Chinese journals between 2010 and 2012. Third, our scoring criteria (yes or no) did not allow for partial information. Fourth, we used unweighted scores for the ARRIVE checklist items, although this may not be a valid approach. Finally, in general, the reporting quality of experiments involving neoplasms in animals in Chinese journals may be worse than what we found because all the studies included in our analyses were supported by the NSFC.

Supporting Information

[Figure omitted. See PDF.]

S1 File. The Chinese databases search strategy.

https://doi.org/10.1371/journal.pone.0154657.s001

(DOC)

S2 File. Three hundred and ninety-six animal studies.

https://doi.org/10.1371/journal.pone.0154657.s002

(DOC)

Acknowledgments

We thank Xiaohuan Feng, Lili Zhang, Xiaokang Zhang, Xin Tian, Yimin Lu, and Qingshan Guo (Lanzhou University) for previous contributions to this study. We thank BiomEditor and American Journal Experts (AJE) for providing assistance with the final revisions of the manuscript.

Author Contributions

Conceived and designed the experiments: YLL. Analyzed the data: YFM YCF YLL. Contributed reagents/materials/analysis tools: YCF YLL. Wrote the paper: YLL. Searching: YLL XXZ. Screening: XXZ YFM XXL WTS. Data extracting: XXZ YFM XXL JW LLC JM GXJ HPG YLL.

Citation: Liu Y, Zhao X, Mai Y, Li X, Wang J, Chen L, et al. (2016) Adherence to ARRIVE Guidelines in Chinese Journal Reports on Neoplasms in Animals. PLoS ONE 11(5): e0154657. https://doi.org/10.1371/journal.pone.0154657

References

1. Bebarta V, Luyten D, Heard K. Emergency medicine animal research: does use of randomization and blinding affect the results? Academic Emergency Medicine. 2003;10(6):684–7. pmid:12782533

2. Thoma DS, Martin IS, Mühlemann S, Jung RE. Systematic review of pre-clinical models assessing implant integration in locally compromised sites and/or systemically compromised animals. Journal of Clinical Periodontology. 2012;39(s12):37–62.

3. Schwarz F, Iglhaut G, Becker J. Quality assessment of reporting of animal studies on pathogenesis and treatment of peri-implant mucositis and peri-implantitis. A systematic review using the ARRIVE guidelines. Journal of Clinical Periodontology. 2012;39(s12):63–72.

4. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS biology. 2010;8(6):e1000412. pmid:20613859

5. Yali L, Peng Z, Wanbin H, Haoyu Z, Kehu Y. Improving the reporting quality of animal experiments, enhancing the develepment of the systematic review of animal studies. Chinese Journal of Evidence Based Pediatrics. 2011;6(3):233–6.

6. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux P, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. pmid:20332511

7. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Preventive medicine. 2007;45(4):247–51. pmid:17950122

8. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine. 2009;151(4):264–9. pmid:19622511

9. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. JAMA. 1996;276(8):637–9. pmid:8773637

10. Hooijmans CR, Leenaars M Fau—Ritskes-Hoitinga M, Ritskes-Hoitinga M. A gold standard publication checklist to improve the quality of animal studies. Altern Lab Anim. 2010;38(2):167–82. pmid:20507187

11. Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One. 2009;4(11):e7824. pmid:19956596

12. McCANCE I. Assessment of statistical procedures used in papers in the Australian Veterinary Journal. Australian veterinary journal. 1995;72(9):322–9. pmid:8585846

13. Smith JA, Birke L, Sadler D. Reporting animal use in scientific papers. Laboratory animals. 1997;31(4):312–7. pmid:9350702

14. Berglundh T, Stavropoulos A. Preclinical in vivo research in implant dentistry. Consensus of the eighth European workshop on periodontology. Journal of Clinical Periodontology. 2012;39(s12):1–5.

15. Vignoletti F, Abrahamsson I. Quality of reporting of experimental research in implant dentistry. Critical aspects in design, outcome assessment and model validation. Journal of Clinical Periodontology. 2012;39(s12):6–27.

16. Stadlinger B, Pourmand P, Locher MC, Schulz MC. Systematic review of animal models for the study of implant integration, assessing the influence of material, surface and design. Journal of Clinical Periodontology. 2012;39(s12):28–36.

17. Chiyu J, Feng B, Yunchuan W. Analysis and evaluation of the articles concerning animal experiments published in Chinese Journal of Burns during 2000 to 2004. Chines Journal of Burns. 2006;22(4):285–7.

18. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76 pmid:24411647