The high-hurdle technique put to the test: Failure to find evidence that increasing loading times enhances data quality in Web-based studies

Headnote

Two Web-based experiments examined the usefulness of artificially delaying the loading of the first page of the study. The idea pursued in this technique is to filter out less-motivated respondents through a higher respondent burden in the form of waiting time. Participants who remain in the study despite having had to wait for the first page of the study to appear on the screen are expected to be more highly motivated, and thus to produce data of higher quality. In both experiments, as expected, the longer the loading time, the lower the likelihood of people responding to the study. However, contrary to expectation, the dropout rate and quality of data were independent of the loading time. Therefore, artificially delaying the loading of the first page of the study is counterproductive.

Web-based data collection bears many advantages, which have rendered it popular-for instance, through potential access to respondents from all over the world and lower costs (Birnbaum, 2004; Göritz & Schumacher, 2000; Skitka & Sargis, 2006). However, there are also disadvantages to this method in comparison with more conventional methods of data collection. There have been attempts to develop techniques and standards to prevent or at least to alleviate these disadvantages (Reips, 2002b).

One of the challenges of Web-based relative to more traditional methods of data collection is increased dropout (O'Neil, Penrod, & Bornstein, 2003). The term dropout denotes participants' premature abandonment of a study before the end of the questionnaire or interview is reached. Reips (2002b) has suggested dealing with dropout in two ways: (1) reduce dropout itself or (2) reduce the undesired consequences of dropout. With regard to reducing the dropout rate, material incentives have been shown to be effective (Göritz, 2004, 2005, 2006), as well as a study's layout and design (Reips, 2000,2002a; Wenzel, 2001) and prudent usage of technologies such as JavaScript, Flash, or Java (Schmidt, 2000; Schwarz & Reips, 2001). To alleviate the undesired consequences of dropout on data quality, Reips (2002b) suggested methods such as the warm-up technique and the high-hurdle technique.

Participants are more likely to drop out at the beginning than near the end of a questionnaire (Reips, 2002c). The reason for participants dropping out at the beginning is thought to be their decision that they...