Content area
Full Text
Establishing reporting guidelines for studies of any kind is an important step in the direction of improving research. The Standards Committee is to be commended for taking on this difficult and time-consuming task for experimental designs (see Gerber et al. 2014). We have no doubt that this is a positive development, something good for science as a whole, as well as for our particular discipline.
Nonetheless, in the spirit of making something that is already quite good even better, we would like to highlight some of the problems with the standards as currently written. This is not to detract from the committee's accomplishment, but is offered in the spirit of constructive suggestions for revision.
We discuss three aspects of the recommendations. The first concerns manipulation checks, a practice of great importance in experimental methodology that is not addressed by the Standards. The second is a more minor point concerning the reporting of response rates, and how studies become classified as surveys under their suggested framework. The third issue concerns the recommendations for detecting errors in randomization. Our critique and suggestions for improvement on this front require substantial background, so we save this more involved issue for last.
MANIPULATION CHECKS
First, we recommend that manipulation checks be added to the JEPS checklist of desirable components of experiments. As is the case with many other items on the checklist, this requirement will not be relevant to every experiment, but it will be applicable to a large number of them and, most importantly, it will improve what can be learned from their results.
Manipulation checks establish that the treatment has had an effect on the theoretically relevant causal construct. In other words, manipulation checks are "a way of ensuring that an experiment actually has been conducted (i.e., that the IV has been effectively manipulated)" (Sansone et al. 2008). The majority of experiments in political science do not report manipulation checks, despite their prominence in other social science disciplines. Many social science disciplines have deemed them basic enough to be required in all but a limited number of cases. As a sociology volume on experimentation argues, "It is an essential part of an experiment to include manipulation checks. . . . It is equally important to report the results...