High-Stakes Testing and Second Chances: From Data to Models

Abstract

High-stakes tests are high pressure, high consequence exams that are used to inform decisions of significant impact, such as educational opportunities, job placement, and career advancement. This dissertation advances the understanding of high-stakes aptitude assessments through empirical examination of the Aviation Selection Test Battery (ASTB), focusing on its role in selecting candidates for military aviation training. This research utilizes a rich dataset spanning 2013 to 2020, which includes detailed sociodemographic, educational, and aviation-related data used to evaluate the impact of prior experiences, including retesting, on selection test scores and aviation training performance. Findings reveal complexities in distinguishing innate aptitudes from prior educational and test-related experiences, which are also confounded with sociodemographic variables. Furthermore, retesting provides individuals with limited test-relevant experiences additional opportunities to improve selection test scores. However, large practice effects complicate the interpretation of retest scores, which also exhibit differential prediction; over-predicting training performance outcomes for retesters. These findings underscore the challenges with aptitude assessment within high-stakes settings where candidates are highly motivated to seek out test practice and those recently acquired skills are associated with test scores. This can be problematic when prior experiences, like flight instruction, may be financially prohibitive for otherwise capable individuals. Proposed interventions are discussed and include providing official test preparation to minimize disparities due to unequal access, as well as reassessing the role of certain subtests and retest scores in the selection process. This dissertation contributes to the broader discourse on high-stakes testing and has implications for military policy aimed at the pursuit of a highly capable and representative military aviation community. Through comprehensive statistical analyses and computational modeling, this dissertation provides a nuanced understanding of the ASTB's utility and limitations, suggesting pathways for enhancing equity and predictive validity in military aptitude testing.