Content area
Full Text
The advent of evidence based medicine has generated considerable interest in developing and applying methods that can improve the appraisal and synthesis of data from diverse studies. Some methods have become an integral part of systematic reviews and meta-analyses, with reviewers, editors, instructional handbooks, and guidelines encouraging their routine inclusion. However, the evidence for using these methods is sometimes lacking, as the reliance on funnel plots shows.
What is a funnel plot?
The funnel plot is a scatter plot of the component studies in a meta-analysis, with the treatment effect on the horizontal axis and some measure of weight, such as the inverse variance, the standard error, or the sample size, on the vertical axis. Light and Pillemer proposed in 1984: "If all studies come from a single underlying population, this graph should look like a funnel, with the effect sizes homing in on the true underlying value as n increases. [If there is publication bias] there should be a bite out of the funnel." 1 Many meta-analyses show funnel plots or perform various tests that examine whether there is asymmetry in the funnel plot and directly interpret the results as showing evidence for or against the presence of publication bias.
The plot's wide popularity followed an article published in the BMJ in 1997. 2 That pivotal article has already received over 800 citations (as of December 2005) in the Web of Science. With two exceptions, this is more citations than for any other paper published by the BMJ in the past decade. The authors were careful to state many reasons why funnel plot asymmetry may not necessarily reflect publication bias. However, apparently many readers did not go beyond the title of "Bias in meta-analysis detected by a simple, graphical test."
The influential Cochrane Handbook adopts a relatively conservative view and acknowledges that there are problems with the concept. 3 Yet it devotes more than four pages to this subject, far more than for any other test of bias and heterogeneity in meta-analysis. Whereas the widely accepted quality of reporting of meta-analysis (QUOROM) statement simply requires in its proposed checklist a description of "any assessment for publication bias," 4 its equally accepted counterpart for meta-analyses of observational studies in epidemiology (MOOSE) states that "methods...