Content area
Abstract
The present study is based on two earlier studies of rational validity estimation. These studies compared the accuracy of rational estimates of validity coefficients to empirical studies. The first study employed experts in the field of personnel psychology (Schmidt, Hunter, Croll, & McKenzie, 1983). The second study employed less experienced psychologists (Hirsh, Schmidt, & Hunter, 1986). The experts were more accurate than novices.
The first portion of the present study addressed the issue of expert accuracy. Fourteen participants from the earlier studies were interviewed over the telephone about the process they used to estimate validity coefficients. No significant differences were found between experts and novices. Experts were variable in the strategies they used to make their estimates and their strategies overlapped with those used by novices. It was hypothesized that differences in accuracy between experts and novices were caused by knowledge of validity coefficients, statistical estimation, or both.
The second portion of the present study compared the accuracy of rational validity estimates to statistical modesl derived from validity generalization data. Six models were derived based on varying levels of test-job combinations. Three job levels were defined using the first three digits of the DOT code. Two levels of test similarity were defined. First, tests were defined by the construct measured, and then constructs were grouped into measures of general intelligence or specific knowledge. For each level of test-job combination, both the levels and patterns of validity coefficients were compared to the true validities. The level of expert estimates were slightly better than validity generalization. In addition, validity generalization was comparable to the accuracy of novice estimates. The pattern analysis revealed that validity generalization is better than rational estimates in predicting the pattern of true validities. It was concluded that rational estimates provide unque information in the estimation of levels of employment test validities, especially when the judges are selectively chosen. In addition, validity generalization estimates were extremely accurate in estimating relative validities. The results suggest that validity generalization is applicable in more situations than originally thought.
