Exploring Validity of Computer-Based Test Scores with Examinees' Response Behaviors and Response Times

Abstract

Examining the testing processes, as well as the scores, is needed for a complete understanding of validity and fairness of computer-based assessments. Examinees’ rapid-guessing and insufficient familiarity with computers have been found to be major issues that weaken the validity arguments of scores. This study has three goals: (a) improving methods to set and evaluate threshold values for detecting rapid-guessing, (b) comparing the response behaviors of examinees with high and low computer familiarity performed, and © understanding the potential relationships between computer familiarity, response processes, and scores.

Data were drawn from the Programme for International Student Assessment (PISA) 2012. Specifically, the analysis of included Australian examinees’ responses to ten released computer-based mathematics items, log data associated with these questions, and seven questions in the information and communications technology (ICT) questionnaire on using computers in schools for mathematical tasks.

Response time based methods suggested by Kong, Wise, and Bhola (2007) were used for setting thresholds for identifying rapid-guessing. Enhanced methods were constructed by incorporating response time and number of response behaviors. Both response time based methods and enhanced methods were evaluated using Wise and Kong’s (2005) criteria as well as number and type of response behaviors. To explore computer familiarity, response behaviors of examinees with low-computer familiarity and high-computer familiarity were compared, and correlations were calculated between computer-familiarity, response behaviors, and response times.

Findings indicated that enhanced methods performed superior than the response time based methods on some of the evaluation criteria; examinees with low computer familiarity displayed some response behaviors more frequently than the examinees with high computer familiarity; and no association between computer familiarity, scores, number of response behaviors, and response time were found. This study concluded that using response times and response behaviors were useful for improving identifying rapid-guessing and exploring the potential role of low computer familiarity on examinees’ testing processes and scores.