Content area
Full text
Focal Article
The authors gratefully acknowledge the following individuals for their helpful feedback on earlier drafts: David Fisher, Fred Oswald, Mitch Rothstein, Paul Sackett, Piers Steel, and Frances Wen. No endorsement of this work, in whole or in part, is implied.
There is little doubt that meta-analysis has greatly benefited the science of work behavior. The major take-home message from the earliest applications in industrial and organizational (I-O) psychology (Schmidt, Gast-Rosenberg, & Hunter, 1980; Schmidt & Hunter, 1977, 1978, 1984; Schmidt, Hunter, & Caplan, 1981; Schmidt, Hunter, & Pearlman, 1981; Schmidt, Hunter, Pearlman, & Shane, 1979; Schmidt, Ocasio, Hillery, & Hunter, 1985) was that, of all the variability observed across studies in validity estimates for cognitive ability tests, the lion's share is due to sampling error and other artifacts. Thus, the doctrine of situational specificity gave way to one of validity generalization, which diminished the need (and value) of local validation and strengthened the credibility of research findings through aggregation. Given the celebrated gains of validity generalization in the realm of ability testing, it is understandable that mean validity is the most cited meta-analytic output (Carlson & Ji, 2011).
Past debates on generalizability inferences in meta-analysis (e.g., James, Demaree, & Mulaik, 1986; Sackett, Tenopyr, Schmitt, Kehoe, & Zedeck, 1985; Schmidt, Hunter, Pearlman, & Rothstein-Hirsh, 1985) tended to resolve toward rejection of situational specificity (in support of generalizability), albeit with unsettled questions. More recent perspectives (e.g., James & McIntyre, 2010; Murphy, 2003; Steel & Kammeyer-Mueller, 2008) echo the challenges of generalizing meta-analytic findings. Sackett (2003) states:
Validity generalization is still wrongly viewed by many, not as a theory about the process of drawing inferences from cumulative data, but as a general statement that the bulk of the variability in research findings is due to statistical artifacts. (p. 111)
Review of meta-analytic findings published over the past 35 years confirms the need to revisit this issue as, in many cases, substantial variance in validity estimates remains after accounting for artifacts, rendering the strong focus on mean effect sizes potentially tenuous if not misguided. It is our hope that renewed discussion of generalizability will promote the value of meta-analysis as a knowledge-generating framework.
Our specific aims are to (a) raise awareness of the potential confusion arising from...