Content area
Full Text
Published online: 18 March 2014
© Psychonomic Society, Inc. 2014
Abstract Recent controversies have questioned the quality of scientific practice in the field of psychology, but these concerns are often based on anecdotes and seemingly isolated cases. To gain a broader perspective, this article applies an objective test for excess success to a large set of articles published in the journal Psychological Science between 2009 and 2012. When empirical studies succeed at a rate much higher than is appropriate for the estimated effects and sample sizes, readers should suspect that unsuccessful findings have been suppressed, the experiments or analyses were improper, or the theory does not properly account for the data. In total, problems appeared for 82 % (36 out of 44) of the articles in Psychological Science that had four or more experiments and could be analyzed.
Keywords Statistical inference . Statistics . Probabilistic reasoning
It is widely recognized that a bias exists across articles in the field of psychology (Fanelli, 2010; Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995). These studies have noted that approximately 90 % of published experiments are report- ed to be successful, which suggests that many unsuccessful experiments remained unpublished. However, it is not clear what such a bias means with regard to believing a specific reported experimental finding or theory. Bias across articles may merely reflect a desire among authors and journals to publish about topics that tend to reject the null hypothesis with typical experimental designs and such a bias does not neces- sarily cast doubt on the findings or theories within any specific article. When judging the quality of scientific work, a finding of bias within an article is more important than bias across articles, because the presence of bias within an article under- mines that article's theoretical conclusions. Recent investiga- tions (Bakker, van Dijk, & Wicherts, 2012;Francis,2012a, 2012b, 2012c, 2012d, 2012e, 2013a, 2013b; Renkewitz, Fuchs, & Fiedler, 2011; Schimmack, 2012) have used an objective bias analysis to indicate that some articles (or closely related sets of articles) in the field of psychological science appear to be biased. However, such analyses of individual studies do not indicate whether the appearance of bias within an article is rare or common in psychology.
Partly to estimate the within-article...