Independent Evaluation of Lone Star AP CSP: Final Report

Abstract

This report describes the evaluation of Lone Star AP CSP, a program developed by the National Mathematics + Science Initiative and funded by a 2019 Education Innovation and Research grant. The three-year program aimed to increase student access to and success in the Advanced Placement (AP) Computer Science Principles (CSP) course in 37 Texas high schools through teacher professional development, supports for students, and supports for schools. The American Institutes for Research conducted an independent evaluation of the program, including studies of implementation, impact on teachers' instruction, and impact on students' participation in and performance on the AP CSP exam. The implementation study examined whether the program was implemented with fidelity in each of its three years and the extent to which participants made use of the program and found it useful. Key findings were that there was no year in which all three components were implemented with fidelity and that teacher and student uptake levels were low but increased over time. The study of impact on teachers' instruction examined the effect of the program on three instructional practice outcomes that were derived from a survey of Lone Star AP CSP teachers along with a volunteer sample of other AP CSP teachers. Separately in each of the three years of the program, the instructional practices impact study used a quasi-experimental matched comparison group design and an analysis-of-covariance (regression) estimation model to examine the effect of Lone Star AP CSP on each of the three survey-based outcomes. The evaluation found that the program had positive effects on teachers' instructional practices in the first and second years of implementation but not in the third. Finally, the student impact study examined school-level outcomes in the analysis of the impact of Lone Star AP CSP on student AP CSP performance and participation. Separately in each of the three outcome years, the study used a quasi-experimental matched comparison group design and an analysis-of-covariance (regression) estimation model to examine the effect of the program on each outcome, followed by meta-analysis to combine the estimates from two samples of schools: those that had AP CSP exam takers in the baseline year and those that did not. Key findings were that the program did not have a significant positive impact on the percentage of AP CSP exam takers that received a qualifying score on the exam in any of the three years, nor did the program have a significant positive impact on schools' percentages of students in grades 9-12 that took the AP CSP exam in any of the three years. In fact, in Years 2 and 3 the program's effect on student participation in the AP CSP exam was significantly negative. The evaluators note that relatively low levels of program implementation and uptake probably help explain why the program did not achieve its hypothesized positive impacts on students. They also acknowledge that the comparison schools included in the analyses could have been implementing other interventions pertaining to AP CSP. [This is a report to the National Math + Science Initiative Regarding Grant U411C190037.]