Trustworthiness of randomized trials in endocrinology—A systematic survey

Abstract

Background

Trustworthy (i.e. low risk of bias) randomized clinical trials (RCTs) play an important role in evidence-based decision making. We aimed to systematically assess the risk of bias of trials published in high-impact endocrinology journals.

Methods

We searched the MEDLINE/PubMed database between 2014 and 2016 for phase 2–4 RCTs evaluating endocrine-related therapies. Reviewers working independently and in duplicate used the Cochrane Risk of Bias Tool (CCRBT) to determine the extent to which the methods reported protected the results of each RCT from bias.

Results

We assessed 292 eligible RCTs, of which 40% (116) were judged to be at low risk, 43% (126) at moderate, and 17% (50) at high risk of bias. Blinding of outcome assessment was the least common domain reported 43% (125), while selective reporting of outcomes was the most common 97% (282). In multivariable analysis, RCTs with a parallel design (OR 2.4; 95% CI; 1.2–4.6) and funded by for-profit sources (OR 2.2; 95% CI; 1.3–3.6) were more likely to be at low risk of bias.

Conclusions

Trustworthy evidence should ultimately shape care to improve the likelihood of desirable patient outcomes. Six out-of 10 RCTs published in top endocrine journals are at moderate/high-risk of bias. Improving this should be a priority in endocrine research.

About the Authors:

José Gerardo González-González

Contributed equally to this work with: José Gerardo González-González, Edgar Gerardo Dorsey-Treviño

Roles Conceptualization, Investigation, Methodology, Supervision, Writing – review & editing

Affiliations Endocrinology Division, Department of Internal Medicine, University Hospital “Dr. José E. González”, Universidad Autónoma de Nuevo León, Monterrey, México, Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México, Research Unit, University Hospital “Dr. José E. González”, Universidad Autónoma de Nuevo León, Monterrey, México

Edgar Gerardo Dorsey-Treviño

Contributed equally to this work with: José Gerardo González-González, Edgar Gerardo Dorsey-Treviño

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft

Affiliations Endocrinology Division, Department of Internal Medicine, University Hospital “Dr. José E. González”, Universidad Autónoma de Nuevo León, Monterrey, México, Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México

ORCID logo http://orcid.org/0000-0001-6082-0631

Neri Alvarez-Villalobos

Roles Conceptualization, Data curation, Investigation, Methodology

Affiliations Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México, Research Unit, University Hospital “Dr. José E. González”, Universidad Autónoma de Nuevo León, Monterrey, México, Knowledge and Evaluation Research Unit in Endocrinology, Mayo Clinic, Rochester, MN, United States of America

Francisco Jesús Barrera-Flores

Roles Data curation

Affiliation: Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México

Alejandro Díaz González-Colmenero

Roles Conceptualization, Data curation, Methodology

Affiliation: Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México

Carolina Quintanilla-Sánchez

Roles Data curation

Affiliation: Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México

Victor M. Montori

Roles Conceptualization, Writing – review & editing

Affiliations Knowledge and Evaluation Research Unit in Endocrinology, Mayo Clinic, Rochester, MN, United States of America, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic, Rochester, MN, United States of America

Rene Rodriguez-Gutierrez

Roles Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Endocrinology Division, Department of Internal Medicine, University Hospital “Dr. José E. González”, Universidad Autónoma de Nuevo León, Monterrey, México, Plataforma INVEST Medicina UANL—KER Unit Mayo Clinic (KER Unit México), Universidad Autónoma de Nuevo León, Monterrey, México, Knowledge and Evaluation Research Unit in Endocrinology, Mayo Clinic, Rochester, MN, United States of America, Division of Endocrinology, Diabetes, Metabolism, and Nutrition, Department of Medicine, Mayo Clinic, Rochester, MN, United States of America

ORCID logo http://orcid.org/0000-0002-3022-5302

Introduction

Well conducted randomized clinical trials (RCTs), should help clinicians, patients, and policymakers make more confident decisions about care. Attention to this so-called core principle of evidence-based medicine (EBM),[1] has supported the critical appraisal of the methods used in RCTs, and has contributed to improve health care.[2–5] Such critical appraisal focuses on recognizing that RCTs often lack sufficient protection against bias that consequently reduces the confidence in their estimates.[6, 7] This confidence creates the trustworthiness for clinicians to apply evidence into patient care.

Trustworthiness in RCTs can be drawn by assuring the transparency of trial’s methods.[7] To do so, several strategies have been adopted to guide researchers in their reporting of methods.[8–11] However, despite these guidance, low quality methodological reporting seems to prevail among several fields of medicine.[12–15] These untrustworthy studies, which reliability is at most questionable, are frequently used by policymakers to develop clinical guidelines, promote an intervention, or generate recommendations often labeled as strong.[16, 17] If patient care should be stemmed from research evidence that mainly draws its recommendations from solid evidence aimed at discover, uncover, or invent treatments that improve their lives, relying in trials in which confidence appear to be obscure by untrustworthy methods opposes the true essence of EBM.[18] This incongruency of developing guidelines and recommendations based on quivery evidence, is prone to over- or underestimate the true effect of an intervention, and may ultimately cause harm to the patient or end up being research waste.[19–21] In this instance, conducting low quality clinical research translates into low quality of evidence that ultimately causes low quality of care for patients.

The extent to which the results of important RCTs of treatments for endocrine conditions are protected against bias and thus are trustworthy, however, remains uncertain. Consequently, we aimed to systematically evaluate the overall risk of bias of endocrine RCTs published in high-impact journals between 2014–2016.

Material and methods

This systematic review adheres to the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) (S1 Appendix).[22]

Study eligibility criteria

Eligible articles were phase 2 to 4 RCTs enrolling patients with an endocrinopathy (e.g., diabetes, thyroid, obesity, bone metabolism, cardiovascular (lipids)/metabolism, and pituitary-gonadal-adrenal axis) to estimate treatment efficacy, regardless of language of publication or number of participants included in the trial. As our intention was to evaluate potential bias of RCTs, we decided to only include the first report of the trial and exclude all follow-ups or any other observational designs (i.e. extensions from an RCTs) aimed at evaluating RCT population.

Search strategy

With the help of an experienced librarian (N.A-V.), we comprehensively searched MEDLINE/PubMed using the Cochrane Highly Sensitive Search Strategy[23] for RCTs published between January 2014 to September 2016—the time frame was chosen arbitrarily—in; a) top five impact-factor medical journals (New England Journal of Medicine, The Journal of the American Medical Association [JAMA], Lancet, Annals of Internal Medicine, and the British Medical Journal [BMJ]), b) two top diabetes journals (Diabetologia and Diabetes Care), c) two top general endocrinology journals (Lancet Diabetes & Endocrinology and Journal of Clinical Endocrinology and Metabolism [JCEM]), d) two top cardiovascular journals (Journal of the American College of Cardiology and Circulation), and e) the five top impact-factor journals of thyroid, pituitary, bone, and obesity journals (Thyroid, Pituitary, Journal of Bone and Mineral Research, International Journal of Obesity). All journals were selected based on the 2015 Journal Citation Reports (JCR) [24]. The complete search strategy is provided in the S2 Appendix.

Selection of studies

Two pairs of reviewers working independently and in duplicate reviewed all potentially eligible articles. In order to standardize the reviewers’ judgments based on the aforementioned inclusion and exclusion criteria, a pilot study reviewing 20 articles was performed with discussion until the pairs achieved optimal chance-adjusted inter-reviewer agreement (kappa ≥ 0.8). Disagreements between reviewers were initially resolved by consensus and, when needed, by adjudication by an endocrinologist and methodologist (R.R-G. or V.M.M.).

Data collection

Using a standardized web-based form (Online Microsoft Excel 2016, Microsoft, Redmont, WA, USA), reviewers working independently and in duplicate used the Cochrane Risk of Bias Tool (CCRBT) to assess the protection against bias afforded by random sequence, allocation concealment, blinding of personnel and participants, blinding of outcome assessment, incomplete outcome data, selective reporting and use of the intention-to-treat analysis.[9] Additionally, we extracted data regarding year of publication, branch of endocrinology, funding, number of centers, type of outcomes (patient-important outcomes or surrogate or laboratory outcomes), analysis of data (intention-to-treat or per protocol), and type of journal, intervention, and design.

Risk of bias classification

Each of these seven domains was classified as indicative of high, moderate, or low-risk of bias based on specific criteria. For instance, we classified random sequence generation as placing a study at low risk of bias if the method of allocation was explicitly stated in the article (e.g., a computer-based program was used to randomly allocate patients); when the allocation was reported only as random, we classified the level of protection against bias as unclear. RCTs were also considered at low risk of bias when the investigator gathering the data or processing the data (e.g., trial statistician) were reportedly blind to trial allocation (blinded outcome assessor), when outcomes in trials showed no apparent sign of omission or reporting only positive outcomes (selective reporting), when loss to follow-up was <20% (incomplete outcome data) and when analyses adhered to the intention-to-treat principle. A full and detailed description of each domain is provided in the S1 Table.

When adequate protection was present across all seven domains or if only one domain was unprotected, we classified the study as at low risk of bias. When >3 domains were classified as having poor or unclear protections against bias, we classified the study as at high risk of bias. All other RCTs were classified as at being at moderate risk of bias.

Missing data

When data were missing, unclear or incomplete, we searched for this information in the registration record of the trial in clinicaltrials.gov, the Australian New Zealand Clinical Trial Registry (ANZCTR), or the University Hospital Medical Information Network Clinical Trial Registry (UMIN-CTR). If the study was not registered or data was still unavailable, we contacted the corresponding author. After a lapse of 10 days, if no response was received, we excluded the article. Every contact was documented and reported. Additionally, we foresighted that some RCTs would fail to report the study phase. Whenever this happened, we evaluated each RCT and judged whether it had phase 1 properties, and if so, it was excluded, otherwise the study was included and labeled as “Not Reported”.

Statistical analysis

We used a descriptive analysis to report categorical variables with frequencies and percentages. We used multivariable analysis using a logistic regression model to assess the probability of a study being at low risk of bias (yes/no). Predictors were selected based on previous evidence and included type of intervention (pharmacological vs. nonpharmacological), trial design (parallel vs. other), type of outcome (patient-important outcomes vs. surrogate or laboratory outcomes), funding (non-profit sources vs. for-profit sources), and number of centers (single vs. multicenter).[16, 25, 26] Adjustments were made on these same variables and goodness-of-fit was determined by the c-statistic and Hosmer-Lemeshow test. Additionally, an univariate analysis was performed to assess the impact of adjusted variables. Trials in which data was missing, were excluded from the multivariable analysis. All variables were inputted in a stepwise backward manner and then excluded until a model that best fitted our data was identified. We took a p value < .05 as statistically significant; associations were described using odds ratios (ORs) and their associated 95% confidence interval (CI). We used SPSS version 22 (IBM Corp, Armonk, NY, USA) for all statistical analyses.

Results

We found 292 eligible RCTs (Fig 1). Most RCTs were industry funded (51%), single-center trials (44%) testing a drug vs. placebo (38%) in patients with diabetes (67%) and measuring its impact on surrogate or laboratory outcomes (70%) (Table 1). A complete list of included articles is provided as a supplementary file (S2 Table)

[Figure omitted. See PDF.]

Fig 1. Flow diagram of included studies.

https://doi.org/10.1371/journal.pone.0212360.g001

[Figure omitted. See PDF.]

Table 1. Risk of bias of the 292 included randomized trials.

https://doi.org/10.1371/journal.pone.0212360.t001

Chance-adjusted inter-observer agreement for risk of bias judgments ranged from moderate to acceptable (kappa = 0.5 to 0.8). Of the 292 trials, 116 (40%; 95% CI; 34–45) were judged to be at low risk, 126 (43%; 95% CI; 37–48) at moderate, and 50 (17%; 95% CI; 13–21%) at high risk of bias. Blinding of outcomes assessment was the least used protection (43%; 95% CI; 37–49%) and complete reporting the most commonly used protection against bias (97%; 95% CI; 94–99%) (Table 2). In the multivariable analysis, parallel design (OR 2.4; 95% CI; 1.2–4.6) and funded by profitable sources (OR 2.1; 95% CI; 1.2–3.6) were associated with better protection against bias. (Table 3). However, this association was found to be substantially small (R2 = 0.1) and the goodness-of-fit test was moderately inadequate (c-statistic = 0.6 and Hosmer and Lemeshow = 0.7).

[Figure omitted. See PDF.]

Table 2. Low risk assessment of the seven domains from the Cochrane Risk of Bias Tool.

https://doi.org/10.1371/journal.pone.0212360.t002

[Figure omitted. See PDF.]

Table 3. Predictor variables for trustworthy RCTs.

https://doi.org/10.1371/journal.pone.0212360.t003

Discussion

Summary of findings

About 4 in 10 endocrine RCTs in top medical journals are adequately protected against bias and thus warrant high confidence in their estimates. Industry-funded parallel design RCTs evaluating drugs exhibited the most methodological protections against bias. Fewer than a third of the reviewed RCTs assessed patient-important outcomes, and almost half of these produced results judged to be at low-risk of bias.

Strengths and limitations

To our knowledge, this is the most contemporary analysis evaluating the overall trustworthiness of RCTs in endocrinology. Although a newer version of the CCRBT has recently been published, perhaps our results may vary with these new rationales for assessing bias in RCTs.[27] However, because we reviewed only top journals, these results may represent a best-case scenario, assuming that the peer-review process enriches the published record with more trustworthy trials. Conversely, our results may underestimate the protection against bias to the extent that RCT reports fail to report methods that the investigators did implement.[28] Our protocol-driven methods and our reliance on multiple, independent, and reproducible judgments to select trials into the review and to analyze each RCT’s risk of bias using a standardized tool should warrant confidence in our findings.

Comparison with previous studies

Several fields of medicine have sought to evaluate the quality of reporting methods in a variety of fields of medicine.[29–32] Regarding endocrinology, it appears that it has passed more than 20 years and the reporting of methods appears to have barely improve. For instance, in 1996, McIver et. al. assessed the quality of methods in 79 trials enrolling patients with type 2 diabetes and found that the methods used for randomization and blinding were reported in only 15% and 46%, respectively.[33] Ten years later, Montori et al. evaluated 199 diabetes trials and judged 53% of these to be at high risk of bias.[26] Although they did not use the CCRBT, that study also found trials funded by industry to exhibit more protection against bias.[26] More recently, in 2008, Rios et al. broaden the scope and analyzed 89 trials published in three top endocrinology journals (e.g. Journal of Clinical Endocrinology and Metabolism [JCEM], European Journal of Endocrinology, and Clinical Endocrinology) and found that the reporting of allocation concealment (10%), blinding (20%), and intention-to-treat analysis (16%) was suboptimal.[34] Although we found an improvement in the reporting of methods, when analyzing these domains as a whole to determine the overall trustworthiness rendered by trials, we found that endocrine trials warrant untrustworthy results due to insufficient protection against bias. This lack of confidence in endocrine trials resonates with what stated by Brito et. al. in their systematic review, showing that most clinical guidelines recommendations in endocrinology were supported by evidence warranting low confidence.[35]

Implications for clinical practice and research

For healthcare, to assure that patients receive optimal care that ultimately seeks to improve their lives, that care must be supported by trustworthy evidence. Clinical recommendations are a way in which the healthcare enterprise assures that clinicians can make confident decisions regarding patients’ care.[36, 37] Thus, this so called evidence-based practice should be based on steadfast evidence that warrants confidence and aims at improving patients’ needs and preferences. Nonetheless, it appears, based on our results, that RCTs are predominantly being conducted without any proper protection against bias, which consequently causes clinical guidelines to draw recommendations based on estimates warranting low confidence and directed at improving intangible surrogate markers that render little or no benefit for patients.[38, 39] These incongruencies in clinical research and practice seems to obviate the main purpose of evidence-based medicine, assuring patient’s wellbeing. In light of this situation, clinicians should be more judicious in the confidence inputted in studies or recommendations they use to provide care.

Furthermore, our findings demonstrate that most investigator initiated RCTs, particularly those funded by federal agencies and foundations are associated with features that place their results at high risk of bias. There is evidence that industry funded trials, although more protected from bias, they are more likely to be affected by spin features—the distortion introduced by subtle features related to the trial question (e.g., selection of patients, interventions, outcomes, and methods) and to the presentation of its results—that mislead readers and are harder to ascertain.[40] Some of these concerns are being addressed by a series of efforts that include initiatives to promote better standards,[41] prospectively register trials,[42, 43] publish all the results from all trials,[44] and report trial results in adherence to the Consolidated Standards of Reporting Trials (CONSORT) reporting statement.[45, 46]. The extent to which these initiatives are improving the evidence base for endocrine practice appears limited at this point, and over the last decade. Even efforts to determine the questions that require better evidence derived from systematic guideline development appear futile: a systematic survey found that only 25% of the research gaps identified by The Endocrine Society guideline recommendations as based on evidence warranting low or very low confidence were being tested in ongoing RCTs.[47]

Conclusion

Most of the RCTs in endocrinology published in top medical journals seem insufficiently protected against bias. Improving the methodological quality of RCTs should be a top priority in endocrine research.

Supporting information

[Figure omitted. See PDF.]

S1 Appendix. PRISMA checklist.

https://doi.org/10.1371/journal.pone.0212360.s001

(DOC)

S2 Appendix. Full search strategy.

https://doi.org/10.1371/journal.pone.0212360.s002

(DOCX)

S1 Table. Detailed description of the Cochrane Risk of Bias Tool domains.

https://doi.org/10.1371/journal.pone.0212360.s003

(DOCX)

S2 Table. List of included articles.

https://doi.org/10.1371/journal.pone.0212360.s004

(DOCX)

Acknowledgments

We thank Dr. Sergio Lozano for the revision of the manuscript’s language.

Citation: González-González JG, Dorsey-Treviño EG, Alvarez-Villalobos N, Barrera-Flores FJ, Díaz González-Colmenero A, Quintanilla-Sánchez C, et al. (2019) Trustworthiness of randomized trials in endocrinology—A systematic survey. PLoS ONE 14(2): e0212360. https://doi.org/10.1371/journal.pone.0212360

References

1. Montori VM. Evidence-based endocrinology. Treat Endocrinol. 2004;3(1):1–10. pmid:15743108

2. Cohen ND, Shaw JE. Diabetes: advances in treatment. Intern Med J. 2007;37(6):383–8. pmid:17535382

3. Kelly MP, Capewell S. Relative contributions of changes in risk factors and treatment to the reduction in coronary heart disease mortality. Health Development Agency. 2004.

4. Chu EK, Drazen JM. Asthma: one hundred years of treatment and onward. Am J Respir Crit Care Med. 2005;171(11):1202–8. Epub 2005/03/22. pmid:15778490.

5. Lau BD, Haut ER. Practices to prevent venous thromboembolism: a brief review. BMJ quality & safety. 2014;23(3):187–95. Epub 2013/05/28. pmid:23708438; PubMed Central PMCID: PMCPMC3932749.

6. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evidence Based Medicine. 2016. pmid:27339128

7. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276(8):637–9. Epub 1996/08/28. pmid:8773637.

8. Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The validity of peer review in a general medicine journal. PLoS One. 2011;6(7):e22475. Epub 2011/07/30. pmid:21799867; PubMed Central PMCID: PMCPMC3143147.

9. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343. pmid:22008217

10. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12. Epub 1996/02/01. pmid:8721797.

11. Lim SM, Shin ES, Lee SH, Seo KH, Jung YM, Jang JE. Tools for assessing quality and risk of bias by levels of evidence. J Korean Med Assoc. 2011;54(4):419–29.

12. Clark L, Schmidt U, Tharmanathan P, Adamson J, Hewitt C, Torgerson D. Poor reporting quality of key Randomization and Allocation Concealment details is still prevalent among published RCTs in 2011: a review. J Eval Clin Pract. 2013;19(4):703–7. Epub 2013/05/08. pmid:23648066.

13. Groves T. What makes a high quality clinical research paper? Oral Dis. 2010;16(4):313–5. Epub 2010/03/18. pmid:20233312.

14. Mills E, Wu P, Gagnier J, Heels-Ansdell D, Montori VM. An analysis of general medical and specialist journals that endorse CONSORT found that reporting was not enforced consistently. J Clin Epidemiol. 2005;58(7):662–7. Epub 2005/06/09. pmid:15939216.

15. Uetani K, Nakayama T, Ikai H, Yonemoto N, Moher D. Quality of reports on randomized controlled trials conducted in Japan: evaluation of adherence to the CONSORT statement. Intern Med. 2009;48(5):307–13. Epub 2009/03/03. pmid:19252352.

16. Gandhi GY, Murad MH, Fujiyoshi A, Mullan RJ, Flynn DN, Elamin MB, et al. Patient-important outcomes in registered diabetes trials. JAMA. 2008;299(21):2543–9. Epub 2008/06/05. pmid:18523223.

17. Gartlehner G, Thieda P, Hansen RA, Morgan LC, Shumate JA, Nissman DB. Inadequate reporting of trials compromises the applicability of systematic reviews. Int J Technol Assess Health Care. 2009;25(3):323–30. pmid:19619351

18. Montori VM, Guyatt GH. Progress in evidence-based medicine. JAMA. 2008;300(15):1814–6. pmid:18854545

19. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. The Lancet. 374(9683):86–9.

20. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med. 2001;135(11):982–9. Epub 2001/12/04. pmid:11730399.

21. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352(9128):609–13. Epub 1998/09/24. pmid:9746022.

22. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. Epub 2009/07/23. pmid:19622551; PubMed Central PMCID: PMCPMC2714657.

23. edited by Julian PTH, Sally G. Cochrane handbook for systematic reviews of interventions: Chichester, West Sussex; Hoboken NJ: John Wiley & Sons, [2008] 2008; 2008.

24. InCites Journal Citation Reports. Available from: http://jcr.incites.thomsonreuters.com.

25. Bhandari M, Busse JW, Jackowski D, Montori VM, Schunemann H, Sprague S, et al. Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ. 2004;170(4):477–80. Epub 2004/02/19. pmid:14970094; PubMed Central PMCID: PMCPMC332713.

26. Montori VM, Wang YG, Alonso-Coello P, Bhagra S. Systematic Evaluation of the Quality of Randomized Controlled Trials in Diabetes. Diabetes Care. 2006;29(8):1833–8. pmid:16873788

27. Higgins JPT SJ, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials. Cochrane Database Syst Rev. 2016;(10).

28. Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM, Schunemann HJ, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004;57(12):1232–6. Epub 2004/12/25. pmid:15617948.

29. Bridoux V, Moutel G, Roman H, Kianifard B, Michot F, Herve C, et al. Methodological and Ethical Quality of Randomized Controlled Clinical Trials in Gastrointestinal Surgery. J Gastrointest Surg. 2012;16(9):1758–67. pmid:22777055

30. Chen X, Zhai X, Wang X, Su J, Li M. Methodological reporting quality of randomized controlled trials in three spine journals from 2010 to 2012. Eur Spine J. 2014;23(8):1606–11. Epub 2014/04/22. pmid:24748442.

31. Lee JW, Chung JH, Jo JK, Lee SW. Assessing the quality of randomized controlled trials published in neurourology and urodynamics from 1993 to 2012. Neurourol Urodyn. 2014;33(5):472–4. Epub 2013/07/23. pmid:23868818.

32. Jo JK, Chung JH, Kim KS, Song SH, Lee SW. Reporting of randomized controlled trials in andrology journals: a quality assessment. J Sex Med. 2015;12(2):350–7. Epub 2014/12/30. pmid:25537865.

33. McIver B DS. An overview of randomized controlled trials in non-insulin dependent diabetes mellitus. Diabetologia. 1996;39(A193).

34. Rios LP, Odueyungbo A, Moitri MO, Rahman MO, Thabane L. Quality of reporting of randomized controlled trials in general endocrinology literature. J Clin Endocrinol Metab. 2008;93(10):3810–6. Epub 2008/06/28. pmid:18583463.

35. Brito JP, Domecq JP, Murad MH, Guyatt GH, Montori VM. The Endocrine Society guidelines: when the confidence cart goes before the evidence horse. J Clin Endocrinol Metab. 2013;98(8):3246–52. Epub 2013/06/21. pmid:23783104.

36. Grimshaw JM, Russell IT. Achieving health gain through clinical guidelines II: Ensuring guidelines change medical practice. Qual Health Care. 1994;3(1):45. pmid:10136260

37. Montori VM, Brito J, Murad M. The optimal practice of evidence-based medicine: Incorporating patient preferences in practice guidelines. JAMA. 2013;310(23):2503–4. pmid:24165826

38. Alexander PE, Brito JP, Neumann I, Gionfriddo MR, Bero L, Djulbegovic B, et al. World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance. J Clin Epidemiol. 2016;72:98–106. Epub 2015/01/27. pmid:25618534.

39. Alexander PE, Bero L, Montori VM, Brito JP, Stoltzfus R, Djulbegovic B, et al. World Health Organization recommendations are often strong based on low confidence in effect estimates. J Clin Epidemiol. 2014;67(6):629–34. Epub 2014/01/07. pmid:24388966.

40. Mahtani KR. 'Spin' in reports of clinical research. Evid Based Med. 2016;21(6):201–2. Epub 2016/10/16. pmid:27737894.

41. Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards [cited 2018 August, 13]. Available from: https://www.pcori.org/research-results/about-our-research/research-methodology/pcori-methodology-standards.

42. Scott A, Rucklidge JJ, Mulder RT. Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration. PLoS One. 2015;10(8). pmid:26287998; PubMed Central PMCID: PMCPMC4546113.

43. Fredrickson MJ, Ilfeld BM. Prospective trial registration for clinical research: what is it, what is it good for, and why do I care? Reg Anesth Pain Med. 2011;36(6):619–24. Epub 2011/10/13. pmid:21989152.

44. Zarin DA, Tse T, Williams RJ, Carr S. Trial Reporting in ClinicalTrials.gov—The Final Rule. N Engl J Med. 2016;375(20):1998–2004. pmid:27635471.

45. Consort:Transparent Reporting of Trials. Consort—Welcome to the CONSORT. Welcome to the CONSORT. [February 22, 2018]. Available from: http://www.consort-statement.org/.

46. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006;185(5):263–7. Epub 2006/09/05. pmid:16948622.

47. Singh Ospina N, Rodriguez-Gutierrez R, Brito JP, Young WF Jr., Montori VM. Is the endocrine research pipeline broken? A systematic evaluation of the Endocrine Society clinical practice guidelines and trial registration. BMC Med. 2015;13:187. Epub 2015/08/13. pmid:26265226; PubMed Central PMCID: PMCPMC4533940.