Study Design and "Evidence" in Patient-oriented Research

Abstract

Individual studies in patient-oriented research, whether described as "comparative effectiveness" or using other terms, are based on underlying methodological designs. A simple taxonomy of study designs includes randomized controlled trials on the one hand, and observational studies (such as case series, cohort studies, and case-control studies) on the other. A rigid hierarchy of these design types is a fairly recent phenomenon, promoted as a tenet of "evidence-based medicine," with randomized controlled trials receiving gold-standard status in terms of producing valid results. Although randomized trials have many strengths, and contribute substantially to the evidence base in clinical care, making presumptions about the quality of a study based solely on category of research design is unscientific. Both the limitations of randomized trials as well as the strengths of observational studies tend to be overlooked when a priori assumptions are made. This essay presents an argument in support of a more balanced approach to evaluating evidence, and discusses representative examples from the general medical as well as pulmonary and critical care literature. The simultaneous consideration of validity (whether results are correct "internally") and generalizability (how well results apply to "external" populations) is warranted in assessing whether a study's results are accurate for patients likely to receive the intervention-examining the intersection of clinical and methodological issues in what can be called a medicine-based evidence approach. Examination of cause-effect associations in patient-oriented research should recognize both the strengths and limitations of randomized trials as well as observational studies.

Headnote

Individual studies in patient-oriented research, whether described as "comparative effectiveness" or using other terms, are based on underlying methodological designs. A simple taxonomy of study designs includes randomized controlled trials on the one hand, and observational studies (such as case series, cohort studies, and case-control studies) on the other. A rigid hierarchy of these designtypes is a fairly recentphenomenon, promotedas a tenet of "evidence-based medicine," with randomized controlled trials receiving gold-standard status in terms of producing valid results. Although randomized trials have many strengths, and contribute substantially to the evidence base in clinical care, making presumptionsaboutthe quality of a study basedsolelyoncategory of research design is unscientific. Both the limitations of randomized trials as well as the strengths of observational studies tend to be overlooked when a priori assumptions are made. This essay presents an argument in support of a more balanced approach to evaluating evidence, and discusses representative examples from the general medical as well as pulmonary and critical care literature. The simultaneous consideration of validity (whether results are correct "internally") and generalizability (how well results apply to "external" populations) is warranted in assessing whether a study's results are accurate for patients likely to receive the intervention- examining the intersection of clinical and methodological issues in what can be called a medicine-based evidence approach. Examination of cause-effect associations in patient-oriented research should recognize both the strengths and limitations of randomized trials as well as observational studies.

Keywords: epidemiologic research design; randomized controlled trials; cohort studies; case-control studies; evidence-based medicine

The spectrum of medical research includes studies in patientoriented (nonlaboratory) research that focus on an intact person or patient as the unit of observation. Patient-oriented research relies on the basic science of clinical epidemiology, and individual studies are often described using terms such as outcomes research or health services research. Comparative effectiveness research is a recent addition to this list, bolstered by $1.1 billion of funding from the American Recovery and Reinvestment Act of 2009. Several types of study design (research architecture) are commonly employed in patient-oriented research, including randomized controlled trials (RCTs) as well as observational studies, such as case reports, case series, cohort studies, and case- control studies. The RCT (a.k.a., randomized trial) achieved great popularity during the latter-half of the 20th century, and few would argue with the assertion that RCTs have provided immeasurable benefits to health and health care.

The RCT was already established as a mainstay of patientoriented research when the paradigm of evidence-based medicine (EBM) emerged at the end of the 20th century. By promoting the retrieval and critical appraisal of patient-oriented research, EBM has the laudable goal of applying the "current best evidence in making decisions about the care of individual patients." (1). Perhaps because of the success of RCTs, the tenets of EBM include a formal hierarchy that considers the RCT (and collections of RCTs as systematic reviews or meta-analyses) as the highest form of evidence (2) (see Figure 1). Yet, if science is the search for knowledge, then claims of a hierarchy of research design should themselves be tested-rather than simply being accepted as dogma. The main objective of this essay is to emphasize how the limitations of RCTs and the strengths of observational studies are underappreciated. Such an examination is urgently needed, in light of problems arising from the axiomatic approach of EBM.

METHODOLOGICAL CONTROVERSY

One of the assertions of EBM-in part due to problems arising from historical, controlled trials (3)-is the inherent inferiority of observational studies compared with RCTs, because of confounding (susceptibility bias). Although this dogma is now firmly established, various questions have been raised regarding its legitimacy (4-18). A representative criticism is that "two familiar justifications offered for the EBM hierarchy of evidence-that the hierarchy provides special access to causes, and that evidence derived from research methods ranked higher on the hierarchy is less biased than evidence ranked lower-both fail" [emphasis added], and that we should not use the EBM hierarchy as a guide to medical research or practice (13).

Given the special status afforded to RCTs by EBM, evidence contradicting a "design hierarchy" is central to evaluating the controversy. In contrast, claims have not been made that observational studies are inherently free of bias. Rather, criticisms of EBM include arguments that bias should be addressed rigorously, rather than feared as if it were a miasma that causes harm in mysterious ways (14). In particular, "unknown confounders" are not a threat if providers are unaware and unaffected by them when selecting treatments (19).

A comprehensive evaluation of research design should consider strengths as well as limitations of both randomized and observational studies (see Table 1). Given the extensive professional acceptance of EBM and its doctrines, however, an emphasis on the limitations of RCTs and the strengths of observational studies is warranted. Six issues are presented in this essay, organized as three limitations of RCTs and three strengths of observational studies, along with representative publications, including examples from the pulmonary or critical care literature. These issues are then combined as an argument in support of what can be called a medicine-based evidence approach to evaluating studies in patient-oriented research.

LIMITATIONS OF RCTs

Limitations of RCTs include: RCTs on the same topic are often contradictory; meta-analyses and large RCTs often disagree; and RCTs can have limited generalizability (i.e., applicability to external populations).

RCTs on the Same Topic Are Often Contradictory

A representative publication (20) from more than 25 years ago reviewed over 200 RCTs addressing 36 topics in cardiology and gastroenterology. Trials on the same topic often disagreed in terms of cause-effect inference (i.e., supportive, equivocal, nonsupportive). Although reasons for discordance included technical issues that are emphasized by EBM, such as concealment of allocation, results also differed due to "eligibility criteria and the selection of study groups, baseline differences in the available population, [and] variability in indications for the principal and concomitant therapies." (20) Thus, details at the interface of clinical medicine (e.g., patient characteristics) and research methods (e.g., study design) are highly relevant in interpreting evidence.

As an example from the pulmonary literature, some RCTs evaluating bacillus Calmette-Guérin vaccine in the prevention of tuberculosis suggested protective efficacy, whereas other trials did not (21). One reason cited (22) for the contradictory results involved patterns of using routine clinical care or specific screening efforts to diagnose tuberculosis in the study populations. Whether called detection bias or clinical insight (or both), this topic reflects how clinicomethodological issues are important in evaluating patient-oriented research.

Meta-Analyses and Large RCTs Often Disagree

Whereas small RCTs might be expected to yield discordant results, large RCTs and meta-analyses are both considered gold-standard evidence in EBM. Yet, one investigation (23) that identified 12 large RCTs and 19 meta-analyses addressing a total of 40 primary or secondary outcomes concluded that "the outcomes of the 12 large RCTs that we studied were not predicted accurately 35 percent of the time by the meta-analyses published previously on the same topics." (23). Thus, the agreement between these highly regarded designs was only fair-and certainly not worthy of the exalted status afforded to them in hierarchies of research design.

One example from the pulmonary literature is a meta-analysis of 15 RCTs reporting that noninvasive ventilation was very effective in treating acute cardiogenic pulmonary edema (24). A subsequent large RCT, however, concluded that the same intervention for the same condition did not improve outcomes (25). Discussion later ensued about the proportion of eligible patients enrolled and participants subsequently intubated (26), once again highlighting the interface of clinical and methodological issues (this example also underscores the challenges of using intention-to-treat analysis [27] in RCTs).

RCTs Can Have Limited Generalizability

Perhaps the most frequently mentioned flaw of RCTs is limited generalizability. For example, major RCTs on cardiovascular disease were found to frequently exclude patients with renal disease or not report on the range of renal function among participants (28), precluding application of results to patients with impaired kidney function. Similar examples of limited generalizability from other specialties are plentiful, including studies of special populations, such as veterans (29).

From the critical care literature, an RCT compared low-dose vasopressin and norepinephrine to norepinephrine alone in treating septic shock (30). This trial was intended to show that combination therapy was superior, but no statistically significant difference in mortality was found. Among several reasons possibly explaining the results was that the observed mortality (less than 40%) was substantially lower than both reported values (50-60%) in the literature (31) and the projected value (60%) for the study. Thus, questions were raised as to whether enrolling a more "real-world" population, with higher mortality, might have yielded different results (31).

Of note, large, simple randomized trials can address this problemby enrolling a broader patient population, but only to a limited extent in that participants and interventions are still study specific. For example, after prior studies found an increased cardiovascular risk of tiotropium as a treatment for chronic obstructive pulmonary disease, a large (nz6,000) RCT found the drug to be "safe" (32). Importantly, the interpretation of results from the trial should emphasize that high-risk patients had been excluded-instead, an overall trust was placed in the data from "the largest and longest randomized trial of tiotropium to date" (33).

STRENGTHS OF OBSERVATIONAL STUDIES

Three strengths of observational studies can be emphasized: the validity of observational studies can be enhanced with appropriate methodological strategies; observational studies and RCTs with the same focus have been found to agree when compared head-to-head; and numerous treatments evaluated in nonrandomized studies have been found to be safe and effective.

Methodological Strategies Can Strengthen Observational Studies

A publication (34) in 1990 established that, when eligibility criteria are restricted and when a "zero-time" for interventions is identified, as with randomized trials, then observational studies- using a cohort design, in this example-can obtain results similar to those of RCTs. In contrast, more liberal inclusion and exclusion criteria, along with varying timing of treatments, can provide a more generalizable (and still valid) result.

From the pulmonary literature-and using a case-control design-a representative study (35) focused on b2-agonist bronchodilators and the risk of death from asthma. This study paid particular attention to methodological issues as they related to the clinical characteristics of participants, including detailed data on dose-response and several drug-outcome models, and found an increased risk of death associated with regular use of certain bronchodilators. In following up the original study (35), a specific question was asked about whether the results were affected by confounding-by-severity of asthma (i.e., sicker patients used more medication) (36), and the answer was "no." This topic illustrates how observational studies can account for the clinical attributes of exposure-outcome relationships. Although some controversy on this topic persists, relying predominantly on RCTs to seek "truth" is arguably unscientific.

Observational Studies and RCTs Agree Head-to-Head

Using different research strategies and evaluating a variety of clinical topics and corresponding publications, summary results from well designed observational studies were determined to be concordant with corresponding summary results from randomized trials. Representative examples include conclusions that: "treatment effects obtained from randomised and non-randomised studies may differ, but one method does not give a consistently greater effect than the other" (17); "little evidence [was found] that estimates of treatment effects in observational studies reported after 1984 are either consistently larger than or qualitatively different from those obtained in randomized, controlled trials" (18); and "the results of well-designed observational studies (with either a cohort or case-control design) do not systematically overestimate the magnitude of the effects of treatment as compared with those in randomized, controlled trials" (7). Conversely, clinical topics with discordant results from RCTs and observational studies can be cited, and the findings from selected RCTs might be judged to be valid-but a key question is whether the clinicomethodological aspects of the studies were evaluated systematically. From an overarching perspective, neither research design can be said to be definitive in producing trustworthy results.

Again considering the topic of bacillus Calmette-Guérin vaccination for tuberculosis, and as one of a set of several formal comparisons with the same take-home message (7), 13 RCTs provided a summary relative risk of 0.49 (95% confidence interval = 0.34-0.70) for vaccine efficacy, and 10 case-control studies provided a summary odds ratio of 0.50 (95% confidence interval = 0.39-0.65). This example demonstrates how results of individual RCTs or observational studies can vary, yet evidence from the entirety of each category can still agree.

Treatments Evaluated in Nonrandomized Studies Are Safe and Effective

A number of medical therapies evaluated in nonrandomized studies have been found to be safe and effective, and are still used clinically. One publication (37) identified 31 oncology drugs that had been approved by the FDA without an RCT, and 30 of these drugs are still fully approved. The authors concluded that drugs approved with data from observational studies "have a reassuring record of long-term safety and efficacy" (37). Thus, nonrandomized studies can provide trustworthy results-and methods of observational research are improving continuously.

From the critical care literature, extracorporeal membrane oxygenation (ECMO) was established as an effective form of life support in neonatal respiratory failure without evidence from an RCT (38). Although randomized trials were conducted subsequently, the need for such studies was questioned at the time (38). A tension existed between a patient-based perspective that conducting an RCT was unnecessary, and an opposing EBMtype perspective that the science supporting ECMO was insufficient until such a trial had been performed. In this situation, rigid opinions about study design did not advance understanding of the clinicomethodological issues involved.

NEED FOR MEDICINE-BASED EVIDENCE

The topic of hormone replacement therapy is a commonly cited example in support of a hierarchy of research design. Briefly, observational studies had suggested a protective effect of hormone replacement therapy on cardiovascular outcomes, whereas subsequent RCTs found that women started on hormone therapy had increased cardiovascular events. A rigorous evaluation of the evidence, discussed elsewhere (19), shows convincingly that observational and randomized studies on this topic are actually another example of agreement, rather than conflict. As an overview: (1) RCT and observational evidence on hormone replacement and health outcomes are concordant on virtually all topics except cardiovascular disease; (2) observational studies of cardiovascular disease that accounted for participant characteristics (especially socioeconomic status) replicated the results of RCTs; and (3) new initiation of hormones in women enrolled in RCTs led to early cardiovascular events, whereas such events had occurred already when observational studies assessed late outcomes among women taking hormones for years-and both results are "true" clinically.

The premise of medicine-based evidence (15) is that systematic consideration of characteristics of the participants, interventions, and outcomes will improve the quality and utility of patient-oriented research. A specific recommendation is made to simultaneously evaluate a study's validity (are the results correct internally?) and generalizability (how well do the results apply externally?). This approach looks beyond whether a study has a randomized or observational design to consider its accuracy (i.e., whether the results are "true" for patients who will receive the intervention) (9). Returning to the topic of hormone replacement therapy and cardiovascular outcomes in women, the controversy could have been resolved in a timelier manner if a medicine-based evidence perspective had been adopted (see Table 2; of note, medicine-based evidence was also recently applied [39] to the topic of screening for prostate cancer).

Other issues are relevant to the conduct of RCTs and observational studies. For example, although RCTs will continue to be the mainstay of evaluating therapy, an estimate of $500 million to $2 billion per drug developed with RCTs (40) suggests that not all questions can be answered using this approach. Another concern is that the concept of "clinical equipoise" is used to justify giving a treatment assigned by randomization, but concerns exist regarding how equipoise is determined and implemented, including exactly how therapeutic uncertainty is established (41). (The previously cited topic of ECMO therapy is relevant to this ethical dilemma-one report described a death that was attributed to the perceived need to conduct an RCT after suitable evidence was already available [42].) Regarding observational studies, an important issue is whether data come from primary (e.g., medical record review) or secondary (e.g., healthcare database) sources; the latter category is generally viewed as more challenging to deal with (43), but specific methodological strategies have been shown to enhance validity (44). Importantly, observational studies need not find a "substitute" for randomization by using complex analytic approaches, such as propensity scores, instrumental variables, and inverse probability weighting. These techniques can be helpful-but when used in addition to, and not in lieu of, considerations of accuracy.

Specific examples further justify the need for a medicine-based evidence approach. The Randomized Aldactone Evaluation Study (45) was a well-conducted RCT showing that sprinoloactone was effective in reducing mortality among patients with congestive heart failure. In real-world practice, however, publication of the RCT led to an increase in hospitalizations and death due to hyperkalemia attributed to increased use of spironolactone (46). Although clinicians might have been more careful in monitoring potassium levels, perhaps the larger issue is that the results of RCTs do not necessarily translate into good clinical practice.

The topic of evaluating new treatments for sepsis or septic shock further illustrates the pitfalls of placing RCTs on a hierarchical pedestal. In this context, RCTs known by acronyms such as PROWESS, ADDRESS, and PROWESS-SHOCK-as described elsewhere (47)-evaluated the impact of drotrecogin alfa (activated) on mortality and other outcomes. This topic highlights a variety of methodological issues, including (but not limited to) interim analyses and premature closure of RCTs (48), post hoc analyses of subgroups in RCTs (49), and moral dilemmas when promoting or refuting results of RCTs (47). From a more general perspective, one author even opined that "we should abandon randomized controlled trials in the intensive care unit" (50). At the very least, and as was the case with studies of vitamin E and cardiovascular outcomes (19), claims of RCTs correcting observational studies are illogical when RCTs themselves disagree. The "right" answer will not emerge from simply denigrating or dismissing observational studies.

Finally, a trial of gastric banding (51) highlights the current infatuation with RCTs. In brief, rather than conduct a case series to document short-term remission of diabetes mellitus accompanying weight loss after surgery, a randomized trial was conducted that included a comparison arm of medical therapy- with an unrealistic 15% projected remission (not just adequate control) of diabetes for "at least 1 year without active pharmacological therapy" (51). Of course, no magical disappearance of diabetes was observed, but to avoid having no outcomes in one arm of the trial (and therefore calculations involving zero), the study included an "assumption that remission had occurred in the 2 [out of 20] patients in the medical-therapy group who dropped out" (51)-in other words, 10% of participants in one arm of the trial were assumed to have been cured of their diabetes because they had withdrawn! This publication reflects the unquestioned prestige we give to RCTs. Perhaps the investigators felt pressured to conduct an RCT or recognized that the evidence would be valued more highly coming from an RCT-but perhaps clinicians should ask "why wouldn't the evidence be convincing enough if produced by another research design?"

The broader topics of clinical guidelines and clinical judgment are beyond the scope of this essay, but several issues are pertinent. For example, the preoccupation with RCTs has been described as causing RCT-myopia when clinical actions are justified only by RCTs, or as causing evidence-based paralysis if actions are not taken without "incontrovertible proof" from RCTs (52). In addition, the statement, "The results of clinical research, pathophysiologic reasoning, and clinical experience represent different kinds of medical knowledge crucial for effective clinical decision making" (53), emphasizes the imperative of using professional (clinical) judgment. From an overarching perspective, one author stated: "it is in fact very difficult to see any cogent reason for thinking as highly of RCTs as the medical community does" (16). Another writer worried that the "brilliant success of the RCT has now become a form of intellectual tyranny" (54).

CONCLUSIONS

A balanced view regarding study design in patient-oriented research is represented by statements such as: "The importance lies not in arguing about which methodology is better than the other, but what can be learned about disease activity and therapy from each type of study" (10), and "Decision makers need to assess and appraise all the available evidence irrespective of whether it has been derived from randomised controlled trials or observational studies; and the strengths and weaknesses of each need to be understood" (12). These cogent recommendations can even be applied retrospectively to data from the 1940s, when the landmark RCT conducted by the United Kingdom Medical Research Council (55) and the observational study conducted by the U.S. Department of Veterans Affairs (56) both showed that streptomycin was effective in treating tuberculosis. Pulmonary medicine therefore has the distinction of perhaps having the earliest medical evidence refuting a hierarchy of research design.

Author disclosures are available with the text of this article at www.atsjournals.org.

References

References

1. Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching the practice of medicine. JAMA 1992;268: 2420-2425.

2. Guyatt GH, Sackett DL, Sinclair JC, Hayward R, Cook DJ, Cook RJ. Users' guides to the medical literature. IX. A method for grading health care recommendations. Evidence-Based Medicine Working Group. JAMA 1995;274:1800-1804. [Published erratum appears in JAMA 275:1232.]

3. Sacks H, Chalmers TC, Smith HJ Jr. Randomized versus historical controls for clinical trials. Am J Med 1982;72:233-240.

4. Horwitz RI. The dark side of evidence-based medicine. Cleve Clin J Med 1996;63:320-323.

5. Feinstein AR, Horwitz RI. Problems in the "evidence" of "evidencebased medicine". Am J Med 1997;103:529-535.

6. Swales JD. Evidence-based medicine and hypertension. J Hypertens 1999;17:1511-1516.

7. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl JMed 2000;342: 1887-1892.

8. Sehon SR, Stanley DE. A philosophical analysis of the evidence-based medicine debate. BMC Health Serv Res 2003;3:14.

9. Concato J, Horwitz RI. Beyond randomised versus observational studies. Lancet 2004;363:1660-1661.

10. Chakravarty EF, Fries JF. Science as experiment; science as observation. Nat Clin Pract Rheumatol 2006;2:286-287.

11. Ligthelm RJ, Borzi V, Gumprecht J, Kawamori R, Wenying Y, Valensi P. Importance of observational studies in clinical practice. Clin Ther 2007;29:1284-1292.

12. Rawlins M. De testimonio: on the evidence for decisions about the use of therapeutic interventions. Lancet 2008;372:2152-2161.

13. Borgerson K. Valuing evidence: bias and the evidence hierarchy of evidence-based medicine. Perspect Biol Med 2009;52:218-233.

14. Concato J.When to randomize, or 'evidence-based medicine needs medicinebased evidence'. Pharmacoepidemiol Drug Saf 2012;21:6-12.

15. Concato J. Is it time for medicine-based evidence? JAMA 2012;307: 1641-1643.

16. Worrall J. Why there's no cause to randomize. Br J Philos Sci 2007;58: 451-488.

17. McKee M, Britton A, Black N, McPherson K, Sanderson C, Bain C. Methods in health services research. Interpreting the evidence: choosing between randomised and non-randomised studies. BMJ 1999;319:312-315.

18. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med 2000;342:1878-1886.

19. Concato J, Lawler EV, Lew RA, Gaziano JM, Aslan M, Huang GD. Observational methods in comparative effectiveness research. Am J Med 2010;123(Suppl 1)e16-e23.

20. Horwitz RI. Complexity and contradiction in clinical trial research. Am J Med 1987;82:498-510.

21. Colditz GA, Brewer TF, Berkey CS, Wilson ME, Burdick E, Fineberg HV, Mosteller F. Efficacy of BCG vaccine in the prevention of tuberculosis: meta-analysis of the published literature. JAMA 1994; 271:698-702.

22. Clemens JD, Chuong JJ, FeinsteinAR. The BCG controversy: a methodological and statistical reappraisal. JAMA 1983;249:2362-2369.

23. LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl JMed 1997;337:536-542.

24. Masip J, Roque M, Sánchez B, Fernández R, Subirana M, Expósito JA. Noninvasive ventilation in acute cardiogenic pulmonary edema: systematic review and meta-analysis. JAMA 2005;294:3124-3130.

25. Gray A, Goodacre S, Newby DE, Masson M, Sampson F, Nicholl J; 3CPO Trialists. Noninvasive ventilation in acute cardiogenic pulmonary edema. N Engl JMed 2008;359:142-151.

26. McDermid RC, Bagshaw SM. Noninvasive ventilation in acute cardiogenic pulmonary edema. N Engl J Med 2008;359:2068, author reply:2069.

27. Wright CC, Sim J. Intention-to-treat approach to data from randomized controlled trials: a sensitivity analysis. J Clin Epidemiol 2003;56:833- 842.

28. Coca SG, Krumholz HM, Garg AX, Parikh CR. Underrepresentation of renal disease in randomized controlled trials of cardiovascular disease. JAMA 2006;296:1377-1384.

29. Chao HH, Mayer T, Concato J, Rose MG, Uchio E, Kelly WK. Prostate cancer, comorbidity, and participation in randomized controlled trials of therapy. J Investig Med 2010;58:566-568.

30. Russell JA, Walley KR, Singer J, Gordon AC, Hébert PC, Cooper DJ, Holmes CL, Mehta S, Granton JT, Storms MM, et al.; VASST Investigators. Vasopressin versus norepinephrine infusion in patients with septic shock. N Engl J Med 2008;358:877-887.

31. Parrillo JE. Septic shock-vasopressin, norepinephrine, and urgency. N Engl J Med 2008;358:954-956.

32. Tashkin DP, Celli B, Senn S, Burkhart D, Kesten S, Menjoge S, Decramer M; UPLIFT Study Investigators. A 4-year trial of tiotropium in chronic obstructive pulmonary disease. N Engl J Med 2008;359:1543- 1554.

33. Michele TM, Pinheiro S, Iyasu S. The safety of tiotropium-the FDA's conclusions. N Engl J Med 2010;363:1097-1099.

34. Horwitz RI, Viscoli CM, Clemens JD, Sadock RT. Developing improved observational methods for evaluating therapeutic effectiveness. Am J Med 1990;89:630-638.

35. Spitzer WO, Suissa S, Ernst P, Horwitz RI, Habbick B, CockcroftD, Boivin JF, McNutt M, Buist AS, Rebuck AS. The use of b-agonists and the risk of death and near death from asthma. N Engl J Med 1992; 326:501-506.

36. Ernst P, Habbick B, Suissa S, Hemmelgarn B, CockcroftD, Buist AS, Horwitz RI, McNutt M, SpitzerWO. Is the association between inhaled beta-agonist use and life-threatening asthma because of confounding by severity? Am Rev Respir Dis 1993;148:75-79.

37. Tsimberidou AM, Braiteh F, Stewart DJ, Kurzrock R. Ultimate fate of oncology drugs approved by the US Food and Drug Administration without a randomized trial. J Clin Oncol 2009;27:6243- 6250.

38. Wolfson PJ. The development and use of extracorporeal membrane oxygenation in neonates. Ann Thorac Surg 2003;76:S2224-S2229.

39. Concato J. What will the emperor say? Screening for prostate cancer as of 2008. Cancer J 2009;15:7-12.

40. Adams CP, Brantner VV. Estimating the cost of new drug development: is it really 802 million dollars? Health Aff(Millwood) 2006;25:420- 428.

41. Miller FG, Joffe S. Equipoise and the dilemma of randomized clinical trials. N Engl J Med 2011;364:476-480.

42. Royall RM, Bartlett RH, Cornell RG, Byar DP, Dupont WD, Levine RJ, Lindley F, Simes RJ, Zelen M. Ethics and statistics in randomized clinical trials. Stat Sci 1991;6:52-88.

43. Sarrazin MS, Rosenthal GE. Finding pure and simple truths with administrative data. JAMA 2012;307:1433-1435.

44. Tannen RL, Weiner MG, Xie D. Use of primary care electronic medical record database in drug efficacy research on cardiovascular outcomes: comparison of database and randomised controlled trial findings. BMJ 2009;338:b81.

45. Pitt B, Zannad F, Remme WJ, Cody R, Castaigne A, Perez A, Palensky J, Wittes J; Randomized Aldactone Evaluation Study Investigators. The effect of spironolactone on morbidity and mortality in patients with severe heart failure. N Engl J Med 1999;341:709-717.

46. Juurlink DN, Mamdani MM, Lee DS, Kopp A, Austin PC, Laupacis A, Redelmeier DA. Rates of hyperkalemia after publication of the Randomized Aldactone Evaluation Study. N Engl J Med 2004;351: 543-551.

47. Angus DC. Drotrecogin alfa (activated).a sad final fizzle to a rollercoaster party. Crit Care 2012;16:107.

48. Goodman SN. Stopping at nothing? Some dilemmas of data monitoring in clinical trials. Ann Intern Med 2007;146:882-887.

49. Lagakos SW. The challenge of subgroup analyses-reporting without distorting. N Engl J Med 2006;354:1667-1669.

50. Vincent JL. We should abandon randomized controlled trials in the intensive care unit. Crit Care Med 2010;38(10, Suppl):S534- S538.

51. Mingrone G, Panunzi S, De Gaetano A, Guidone C, Iaconelli A, Leccesi L, Nanni G, PompA, CastagnetoM, GhirlandaG, et al. Bariatric surgery versus conventional medical therapy for type 2 diabetes. N Engl J Med 2012;366:1577-1585.

52. Ziemer DC. The dogma of "tight control": beyond the limits of evidence [letter]. Arch Intern Med 2006;166:1672.

53. Tonelli MR, Curtis JR, Guntupalli KK, Rubenfeld GD, Arroliga AC, Brochard L, Douglas IS, Gutterman DD, Hall JR, Kavanagh BP, et al.; ACCP/ATS/SCCM Working Group. An official multi-society statement: the role of clinical research results in the practice of critical care medicine. Am J Respir Crit Care Med 2012;185:1117-1124.

54. Fletcher JC, Branson R, Freireich EJ. Ethical considerations in clinical trials: invited remarks. Clin Pharmacol Ther 1979;25:742- 746.

55. Medical Research Council. Streptomycin treatment of pulmonary tuberculosis. BMJ 1948;2:769-782.

56. Barnwell JB. Veterans Administration Tuberculosis Division, 1945-1947; progress report. Am Rev Tuberc 1948;58:64-76.