Data Collection and Monitoring in an Educational RCT of a Postsecondary Access Program: Assessing Internal and External Validity

Abstract

The objective of this article is to discuss the advantages of effective educational monitoring in the context of a longitudinal RCT. Intentional data collection and monitoring enables the important assessment of issues of both internal and external validity. We discuss how we used mixed methods data collection to reveal important changing contextual factors in an evaluation of a postsecondary access program in the U.S. state of Texas. Specifically, we employed quantitative analysis of the RCT to compare the college enrollment rates of high schools that were randomly assigned a college adviser with schools that were not assigned a college adviser. We employed survey data collection, qualitative interviews, and site visits to monitor the fidelity of treatment implementation and compliance to treatment assignment over time. In the absence of monitoring treatment fidelity and compliance over time in both treatment and control schools, we would have missed critical changes that explain the observed attenuation of treatment effect estimates. We also discuss how monitoring can inform defenses of the stable unit treatment value assumption and suggest how effective the program will be when applied more widely or to other contexts.