Development and Evaluation of a Confidence-Weighting Computerized Adaptive Testing

Abstract

The purpose of this study was to examine whether the efficiency, precision, and validity of computerized adaptive testing (CAT) could be improved by assessing confidence differences in knowledge that examinees possessed. We proposed a novel polytomous CAT model called the confidence-weighting computerized adaptive testing (CWCAT), which combined a confidence-weighting scoring scheme with the graded response model (GRM). The CWCAT provided a more interactive testing environment by focusing on the examinees’ confidence in their responses. An experiment was conducted to evaluate the comparison between the CWCAT and conventional CAT in terms of efficiency, precision, and validity. As expected, the polytomous method provided better discrimination among individual differences in the confidence in knowledge and required fewer items per examinee. Results also showed that CWCAT yielded ability estimates that were higher and better correlated to examinees’ performance in English learning. Furthermore, the ability measured by CWCAT was not as likely to be affected by guessing as on conventional CAT, and, therefore, was more consisted with examinees’ true ability.