Content area
Full Text
The ease with which data can be collected and analyzed via personal computer makes it potentially attractive to "peek" at the data before a target sample size is achieved. This tactic might seem appealing because data collection could be stopped early, which would save valuable resources, if a peek revealed a significant effect. Unfortunately, such data snooping comes with a cost. When the null hypothesis is true, the Type I error rate is inflated, sometimes quite substantially. If the null hypothesis is false, premature significance testing leads to inflated estimates of power and effect size. This program provides simulation results for a wide variety of premature and repeated null hypothesis testing scenarios. It gives researchers the ability to know in advance the consequences of data peeking so that appropriate corrective action can be taken.
Imagine the following scenario. A researcher, heeding the advice to take statistical power seriously (see, e.g., Cohen, 1988,1992,1994; Wilkinson& Task Force, 1999), estimates in advance of his research study a sample size that will produce a .80 probability of correctly rejecting a false null hypothesis at the .05 level of significance. He is a bit disheartened at the size of the projected sample, perhaps because most published research is underpowered (see, e.g., dark-Carter, 1997; Cohen, 1962; Dar, Serlin, & Omer, 1994; Finch, Gumming, & Thomason, 2001; Sedlmeier & Gigerenzer, 1989), and so suggests that fewer subjects are usually needed to find significant results. The situation is perhaps all the more frustrating given scarce subject pool resources and the time and money it will take to conduct the larger-than-expected study. But he sees a possible solution. Data collection is automated via computer, allowing each participant's responses to be quickly appended to the growing data file. Moreover, a simple statistical analysis is just a mouse-click away. Why not test the key hypothesis as the data are collected rather than waiting until the target sample is achieved? If a significant result emerges before the target sample size is reached, the study can be concluded early, saving valuable resources.
On the face of it, this plan seems sensible. Resources usually are quite scarce, and any savings on one study can be devoted to other research. Furthermore, the power analysis may have been based...