The Impact of Course Placement in STEM Sequences on Students' Short-Term and Longer-Term University Success Outcomes

Abstract

Research has frequently demonstrated negative effects of placing students into developmental education, but very little inquiry has considered the impact of placing students into different levels of non-developmental coursework. The present study explored this issue within sequenced pairs of STEM gateway courses using doubly-robust propensity score analyses and a total sample of 11,532 undergraduates. The results indicate that starting in a lower-level course frequently predicted favorable short-term academic outcomes, but course level was unrelated to retention, and starting in lower-level coursework was often inversely associated with the likelihood of graduating from the university or receiving a STEM bachelor's degree.

Every college or university makes decisions about how to place students into different levels of coursework, but these choices may happen with varying levels of intentionality or forethought. Course placement may be guided by one or more of a variety of factors: a placement test score; a standardized admissions test overall score or subscore; an Advanced Placement (AP), International Baccalaureate (IB), or A-level examination score; the successful completion of relevant prior coursework in high school or college; and/or the discretion and judgments of students, academic advisors, or faculty (Ganga & Mazzariello, 2019; Hughes & Scott-Clayton, 2011; Melguizo et al., 2014). These placement decisions must carefully balance concerns about students taking courses for which they are underprepared and therefore are unlikely to be successful versus courses that are redundant with prior knowledge and therefore lead to students having slower progress and potentially becoming frustrated and disengaged.

The literature on course placement has often examined the role of placement into developmental education, which was formerly known as remedial education. This research frequently demonstrates the negative short-term and long-term effects of placing students into developmental education among students who are close to the cutoff score (see Jaggars & Stacey, 2014; Valentine et al., 2017). Other work has shown that the outcomes associated with developmental education can be bolstered through appropriate reforms to this coursework and its sequencing within the curriculum (Boatman, 2021; Jaggars & Bickerstaff, 2018; Logue et al., 2016, 2019). However, the impact of course placement within non-developmental courses has rarely been explored; such inquiry is necessary for institutions to make effective decisions about their course placement policies and practices.

Science, technology, engineering, and mathematics (STEM) fields may be a particularly important area to examine course placement decisions, since early STEM experiences shape students' persistence and completion of STEM coursework (Bahr et al., 2017, 2022). As discussed in more detail below, course placement can affect the early academic momentum of undergraduate students, which has implications for STEM participation and success (Wang, 2017). Therefore, the present study fills this substantial and important gap in the literature by examining the following research question: To what extent does starting in a lower-level STEM gateway course predict college grades, academic good standing, credits earned, retention, and graduation? These analyses focus on three pairs of courses in different subjects (i.e., chemistry, computer science, and mathematics) in which the higher-level course is required for certain majors, but students may be placed into and/or choose to start within the lower-level course.

Literature Review

Course placement can be conceptualized as an administrative function that fulfills a key academic and institutional responsibility. In seeking to maximize students' learning and future success, colleges seek to refer students to courses that are well-suited for their academic skills, needs, and interests. This process happens through determination of course placement and referral criteria, decisions about which assessments and measures to use for making these determinations, and the structure of academic advising. Placement tests are commonly-used tools, and their proliferation may be due to their administrative advantages: they can be less time-consuming and less resource-intensive to administer than interviews or reviews of individual files and transcripts, and placement tests are available in computerized formats that can enable colleges to assess many students and provide course placement results more quickly (Burdman, 2012; Hughes & Scott-Clayton, 2011; Melguizo et al., 2014).

The Implications of Course Placement

Making a correct initial placement decision is important, because students' starting point in their academic trajectory has tremendous consequences. Community college students who begin in lower-level courses are less likely to persist in and complete college-level courses (Bailey et al., 2010; Fong et al., 2015). Research using causal methods has also shown that developmental math and English lead to worse outcomes for students placed in these courses compared to those placed into college-level courses (Melguizo et al., 2016). A meta-analytic review of studies at two- and four-year colleges that used regression discontinuity designs found negative effects of developmental education placement on the successful completion of college-level courses, credit accrual, and degree attainment (Valentine et al., 2017).

Researchers have also examined how placement in developmental courses affects students. Using data from a large community college system, Scott-Clayton and Rodriguez (2015) investigated three different hypotheses regarding the effect of placement into lower-level courses. Lower-level courses could result in development of student skills, thereby preparing them for success in upper-level coursework. Alternatively, lower-level courses could result in discouragement, such that students would fail to persist and complete additional courses after developmental education. Finally, lower-level courses could be a diversion if students continued to enroll in courses and complete them, but they then shifted away from degree-bearing pathways and instead remained in non-credit courses. The study revealed that development education's primary function was to divert students away from credit-bearing pathways.

Others have described additional ways that course placement shapes students' early college experiences. Ngo and Velasquez (2020) used longitudinal student records spanning high school and community college enrollment to identify and define various patterns in math course-taking after the initial course placement decision. While some students were able to progress in their math course-taking and had "math mobility" (moving forward in the math sequence), others experienced "math repetition" (attempting the same or lower level of math) and "math traps" (enrolling in a lower level of math in college than one did in high school, and never surpassing the highest level completed in high school). Examining subsequent outcomes, Bicak et al. (2023) observed that math course repetition was associated with lower grade point average (GPA) and increased time to degree among community college transfer students. Course placement is also a concern because of the potential for curricular misalignment, when the initial college course to which a student is referred does not follow typical curricular sequences. A study of math misalignment between community colleges and feeder high schools revealed substantial rates of math misalignment among first-time college students, along with significant differences by race/ethnicity (Melguizo & Ngo, 2020). Math misalignment was also found to be associated with reduced likelihood of STEM participation (Park et al., 2021) and racial/ethnic inequities in persistence and credit completion (Ngo & Melguizo, 2021).

Given these potential problems with course placement, researchers and practitioners have been interested in assessing the usefulness and accuracy of the placement tools often used to make these decisions. For example, research examining two large community college systems estimated that the use of placement tests resulted in nearly one-third of all incoming students being misassigned to developmental English courses and one-quarter of students being misassigned to developmental math courses (Scott-Clayton et al., 2014). Most of these placement errors occurred for students who were predicted to succeed in college-level courses but were instead placed into developmental courses. Similar research has examined the accuracy of state-developed assessments (Leeds & Mokher, 2022), the impact of changing placement practices (Kolesnikov et al., 2020; Ngo & Kwon, 2015; Ngo & Melguizo, 2016), and the institutional processes and decision-making that shape the development and implementation of course placement policies (Melguizo et al., 2014). Changes to placement practices, such as the use of multiple measures and predictive analytics to place students in courses, have been shown to improve student enrollment in and completion of college-level courses (Barnett et al., 2018). Allowing students to self-place into math courses improved completion outcomes but also created equity issues as some student groups, such as Black students and women, were more likely to underestimate their abilities and place themselves into lower-level courses (Kosiewicz & Ngo, 2020). These changes to placement practices in developmental education are part of a larger wave of reform efforts that are changing the nature of assessment, placement, instruction, and support services in community colleges (Jaggars et al., 2014; Rutschow et al., 2019).

Research Evidence by Institutional and Disciplinary Context

While much of this evidence base stems from studies in two-year colleges and open-access settings, similar studies have been conducted at four-year institutions in which students typically have to meet some admissions criteria. Analyses using nationally representative datasets have observed that four-year college students who enrolled in remedial courses were significantly less likely to earn a bachelor's degree (Attewell et al., 2006; Shields & O'Dwyer, 2017). Indeed, the meta-analysis described above showed the negative effects of developmental education were actually larger for university students than community college students (Valentine et al., 2017). Other research suggests that these effects vary by level of academic preparation, with some positive effects of remediation observed for incoming students with lower levels of preparation (Boatman & Long, 2018). These effects may also vary by students' academic characteristics, such as degree aspirations and major (Daugherty et al., 2021). Students who began in developmental math at one four-year institution were actually less likely to drop out than those who enrolled directly in college-level math (Lesik, 2007). Likewise, evidence from a university in Colombia where students are tracked into lower- and higher-ability classes within majors revealed that marginal students placed higher were less likely to complete their first-year courses and, in the long run, less likely to earn a degree (de Roux & Riehl, 2022).

Much of the scholarship examines course placement in developmental math and English courses. Less attention has been paid to course placement in other fields, such as STEM fields that also have sequences of coursework. This dearth of evidence is especially problematic, because about one-third of students in the Beginning Postsecondary Students Longitudinal Study (BPS:04/09) intending to major in STEM fields eventually left these fields, and the available evidence points towards intensity of first-year STEM coursework and their performance in STEM courses as playing significant roles in the decision to switch majors (Chen, 2015). A large-scale study of California community college students in STEM found that the courses through which students entered STEM curricula (e.g., math, chemistry, and physics) were significantly predictive of how far students would advance in those STEM curricula (Bahr et al., 2017). Grades in first year STEM courses are also predictive of STEM persistence (Thompson, 2021), suggesting initial STEM course placement can have significant long-term implications.

Using AP credits to inform decisions about course placement is one strategy in STEM fields, and research suggests that applying AP credits to skip introductory STEM courses can be beneficial (Fischer et al., 2023). Nevertheless, inequities in access to AP courses means these benefits are more likely to accrue to more advantaged students (Kolluri, 2018). Given the implications of introductory STEM coursework on subsequent STEM participation (Bahr et al., 2017, 2022; Mervis, 2010), it is important to examine course placement in STEM pathways and how these decisions shape subsequent student outcomes.

Theoretical Framework and Present Study

This study is situated within the holistic theoretical model of momentum by Wang (2017) for community college student success. Although this theory refers to one type of institutional context, we believe that it is highly relevant for understanding student experiences and outcomes at four-year institutions. Wang argues that momentum toward achievement of students' educational goals is critical for the realization of those goals. This model focuses on three primary domains of momentum: curricular (which includes a proper course/program pathway, enrollment intensity, and enrollment continuity); teaching and learning (which consists of cognitive and metacognitive dimensions); and motivation (which includes aspirations, growth mindset, perseverance, and connection of institutional agents). These collegiate domains are influenced by carry-over momentum from students' pre-entry academic, social, and motivational factors. Students may also face counter-momentum friction from financial barriers, lack of clear pathways, inadequate or lack of advising, and lack of practitioners' professional development.

Course placement is relevant to multiple domains and subdomains within this theory, which can be used to make divergent predictions about the impact of early course placement. An argument for having students start in lower-level courses is that they need a greater foundation in relevant knowledge and skills before moving to more advanced courses. This approach may build cognitive momentum and perhaps also metacognitive momentum (through learning appropriate study strategies) with less advanced content before moving onto more challenging content. These students may also develop growth mindset and perseverance in the context of more manageable academic challenge.

Conversely, others may argue for the importance of starting in a higher-level course. Because degree completion is inherently a product of accumulating appropriate credits, a more advanced initial course will allow students to move into upper-level coursework more quickly and therefore gain curricular momentum early in their undergraduate enrollment. By essentially moving one semester ahead within a required course sequence, students may be able to maintain enrollment continuity while also having a more manageable enrollment intensity by taking fewer credits in future semesters. These students may also be able to engage more quickly in activities such as undergraduate research, which has been shown to promote retention within STEM fields and beyond (National Academies of Sciences, Engineering, and Mathematics, 2017).

The impact of course placement may also differ considerably in relation to determining the appropriate college-level course (as the present study examined) versus determining whether students should be placed into a developmental course (as the overwhelming majority of previous studies have considered). In terms of motivational momentum, starting in developmental coursework could have an immediate, detrimental effect on students' growth mindset, perseverance, and/or aspirations, since they may perceive that the institution does not consider them to be "college-ready." Placing students into a developmental course also constitutes an even greater setback for curricular momentum, since students may need to successfully complete at least one developmental course before they can even enroll in a desired introductory STEM gateway course. The present study adds to the literature by examining placements into lower and higher levels of college-level courses, and therefore also extends the study of momentum beyond two-year colleges and developmental courses.

Method

Participants and Setting

Participants were undergraduates who started attending a large, Midwestern public university from Fall 2012 to Fall 2019. As a result of divergent grading practices during the COVID-19 pandemic, those semesters were not included in the analyses, but longer-term student outcomes that occurred during this time were assessed (e.g., for someone who started in Fall 2017 and graduated in Spring 2021). Students were included in the sample if their first course in chemistry, computer science, or mathematics was either the lower- or higher-level gateway course in a sequential pair. The mathematics course sequence examined here was designed for students majoring in fields within the biological sciences; overall, about three-quarters of incoming STEM majors were required to take at least one of the three courses examined in this study.

The full analytic sample consisted of 11,532 undergraduates; descriptive statistics appear in Table 1. The percentage of students starting in the lower-level course varies across STEM subjects from 40% (computer science) to 61%

(mathematics), and the mean gateway course grade corresponds to roughly a B-. Slightly over half (53%) of students graduated from the university during the time period of the study, and about half of those students (26% total) received their bachelor's in a STEM field of study. In terms of sample demographics, 59.2% were female, 77.5% were White, 8.3% were Latinx, 6.1% were Asian, 3.8% were multiracial, 2.3% were Black, 1.9% were another or unknown race, and 25.1% were first-generation students (i.e., no parent had any postsecondary degree). This representation is reasonably close to the population of all U.S. undergraduates in terms of students' sex (58% female; U.S. Department of Education, 2022) and first-generation status (estimates vary from 24%, per the Postsecondary National Policy Institute, 2022, to 33% for the Center for First-Generation Student Success, 2018). However, the present sample has a notably higher percentage of White students than all undergraduates nationally (51%; U.S. Department of Education, 2022). That said, this analytic sample consists of students who enrolled in a relevant STEM gateway course, so these national figures are not directly comparable.

Students could place into the higher-level course in each pair through several different criteria that involved prior coursework and/or test scores; some criteria were recommendations rather than requirements needed to enter the course. Students' course-taking decisions were facilitated by a meeting with their academic advisor, who often provided guidance based on a variety of factors. In general, students who had very low placement test scores were unlikely to take the higher-level course, even if they were technically allowed to do so based on another criterion. Therefore, to facilitate the propensity score analyses discussed below, students whose placement test scores were well below the threshold for the higher-level course were excluded from the relevant analysis. Specifically, students whose ALEKS score was more than 15 points below the cutoff (for examining mathematics and computer science) or whose Chemistry Diagnostic Test score was at least five points below the recommended value (for examining chemistry) were excluded from the analytic sample. These exclusion criteria reflected scores that were about one standard deviation below the cutoff within the full dataset; students at each excluded placement test score below this threshold generally had less than a 1 in 3 chance of taking the higher-level course (and often considerably lower than this proportion).

For other STEM gateway courses at this university, too few students placed into the lower-level physics course to conduct comparisons with appropriate statistical power, and biology course placement was entirely dictated by prior coursework, so it was not ideal for the propensity score analyses that are better suited for more complex placement decisions. This university also had different math course sequences for engineering and for various majors, but the vast majority of students started in the relevant lower-level course, so there was not sufficient sample size to conduct these analyses. That said, the use of different course pairs in chemistry, computer science, and mathematics facilitated comparisons across STEM departments and gateway sequences, which is important because the impact of course placement may vary as a result of contextual factors (e.g., the level of the particular cutoff score, the difficulty of each course, the extent to which the early course content is necessary for subsequent courses).

Measures

All data in this study were obtained through the university's registrar office. For outcome variables, grade within the introductory course, first-year GPA, and final overall GPA were measured on a 4.0 scale. Several binary outcomes relevant to academic momentum were used (all coded 0 = no, 1 = yes): DFWI grade within the introductory course (i.e., D+, D, D-, F, withdrawal, or incomplete), ever placed on academic probation, retention to fall of the second year, graduation from the university with any major, and graduation with a STEM major.

The treatment variable indicated whether the student started within the lower-level course of the pair (0 = higher-level course, 1 = lower-level course). A variety of relevant covariates and predictors were included. Academic variables were high school GPA, the number of years of high school math credit above Algebra II, and placement test score. The analyses examining computer science and mathematics courses used the ALEKS placement test (with score cutoffs of 75 and 70, respectively), whereas the analyses examining chemistry used a chemistry diagnostic test (CDT; recommended score of 16). Students' incoming major was indicated by their initial primary broad field of study in the larger-sample analyses (i.e., social sciences, arts/humanities, non-STEM vocational, and STEM) and by their two-digit CIP code in the restricted-sample analyses. Relevant CIP codes from students' secondary or tertiary major or program of study were also incorporated here, since students' primary major was sometimes unrelated to their need to take one of these courses. Demographic variables were first-generation college student (0 = no, 1 = yes), sex (0 = male, 1 = female), and race (separate dummy-coded variables for Asian, White, and multiracial or unknown racial identity, with Underrepresented Racial Minority [Black, Latinx, Native, or Pacific Islander] as the referent group). This use of a single URM category was necessary to ensure that appropriate balance could be achieved given the small sample sizes in some analyses, especially for computer science coursework.

Analyses

As mentioned above, students can be placed into higher-level courses within each pair through meeting one of several possible criteria of prior coursework and/or test scores, and multiple different tests could be used in this process. Students were also able to repeat the ALEKS and CDT to achieve a higher score, and they sometimes had the option of selecting into the more advanced course even when they did not meet these criteria. Given these various potential paths for course entry, falling just on one side or the other of a particular placement test "cutoff" was not strongly related to the course in which students ultimately enrolled. As a result, even fuzzy regression discontinuity analyses—which are often employed in research for placement into developmental education coursework (Valentine et al., 2017)—would not be an effective approach for examining these data. Preliminary analyses confirmed that fuzzy regression discontinuity analyses could not be used here, since the weak association between the placement test cutoff and course placement led to very large standard errors for estimating the link between course placement and the student success outcomes.

Instead, this study used propensity score analyses to determine the potential impact of course placement (for overviews of this technique, see Bai & Clark, 2019; Guo & Fraser, 2015; Holmes, 2014). Propensity score analyses seek to compare groups of participants who are virtually identical in their proclivity to participate in the treatment, so any differences in outcomes between the two conditions may be attributable to the treatment itself. These analyses use covariates that occur before the treatment and are believed to affect participants' selection into the treatment and/or their subsequent outcomes; this process assigns each participant with a score that indicates how likely they were to participate in the treatment.

To create comparable treatment and control groups, the present study employed doubly-robust propensity score analyses with augmented inverse probability weighting (AIPW), which use covariates both to create the propensity score and to predict the outcome. As a result, this approach has the benefit of yielding valid causal inferences about average treatment effects, even if one of the two stages of the analysis is misspecified (for more information including relevant formulas, see Bang & Robins, 2005; Glynn & Quinn, 2010; Imbens & Rubin, 2015). We chose to implement this AIPW technique, because it has the advantage of providing these doubly-robust estimates, and it examines a larger sample than other propensity score techniques that frequently drop students from the analyses.

The covariates used in propensity score analyses should predict selection into the treatment and/or the level of the outcome (e.g., Brookhart et al., 2006; Patrick et al., 2011). Importantly, the quality of selected covariates matters far more than the mere number of covariates in the model (e.g., Steiner et al., 2010). The covariates in this study included placement test scores, relevant prior coursework, and incoming undergraduate major, all of which directly affect students' course placement. High school GPA is a strong predictor of college grades and retention (Credé & Niehorster, 2012; Robbins et al., 2004; Schneider & Preckel, 2017), so this indicator of precollege academic preparation was also used. In terms of Wang's (2017) momentum framework, placement tests, prior coursework, and high school GPA are all indicators of carry-over momentum upon entering college, and undergraduate major shapes students' curricular momentum. In addition, the analyses incorporated key demographic variables that are notably related to college student success (i.e., race/ethnicity, sex, and first-generation status), which may also pertain to various forms of counter-momentum friction via structural and institutional barriers that students must overcome.

Separate propensity score analyses were conducted for each outcome and each course pair, and binary outcomes were examined with logistic models. Moreover, each of these analyses was conducted using two different samples: (1) all students who started within one of the relevant courses, and (2) students who started in a relevant course and were enrolled in an incoming undergraduate major or program of study that required the higher-level course. Students were also included in this latter group if they had declared a program of study that required the higher-level course in the course plan, even if it was not a formal undergraduate major (e.g., Pre-Medicine, Public Health Interest). These restricted-sample analyses focused on students who needed to make a meaningful decision about which course to take (as opposed to those who could fulfill a requirement solely via the lower-level course), thereby potentially reducing any selection bias that may remain even after conducting the doubly-robust analyses.¹

We also present results of ordinary least squares regression analyses using the same set of covariates for comparisons with the AIPW approach. Within the dataset, complete data were available for all collegiate measures, and missing data on precollege variables was minimal (95% of observations had complete data on all measures), so cases with missing data were deleted listwise within each analysis.

Limitations

Some limitations should be noted. First, this sample was drawn from a single university, so it is unclear to what extent the findings might generalize to other institutional contexts. Second, the three course pairings all occurred within STEM disciplines, so the results may or may not extend beyond those contexts. Third, the presence of various ways in which students can enter lower- or higher-level coursework prevented us from using a regression discontinuity design, which is common within research on placement into developmental education (Valentine et al., 2017). A benefit of the present approach is that regression discontinuity is less useful for drawing inferences further away from the cutpoint (Cattaneo et al., 2020), whereas propensity score analyses can compare all students in the treatment and control conditions. Finally, in a related issue, propensity score analyses only provide accurate estimates of causal relationships to the extent that relevant covariates have been included (Rosenbaum & Rubin, 1983). The implementation of doubly-robust AIPW analyses and the incorporation of relevant academic and demographic variables constitute notable strengths of this study, but it is possible that additional pertinent covariates were not included. Although we believe that this analytic approach likely removed the vast majority of selection bias, the current findings may not accurately indicate causal relationships, so we avoid using causal language in our discussion of these findings.

Results

Selection into Lower-versus Higher-Level Coursework and Propensity Score Weighting

The differences in group means between each covariate in the treatment (lower-level coursework) and control (higher-level coursework) conditions are provided in Table 2. These figures are shown for groups before and after propensity score weighting was implemented, as well as for the full-sample and restricted-sample analyses. The unweighted differences illustrate disparities in the characteristics of students that initially enroll in lower-versus higher-level courses. These standardized coefficients also have the benefit of allowing readers to compare the relative magnitude of differences across covariates. In the unweighted full sample, the strongest relationships with initial coursework occurred for the continuous variables of placement test scores, advanced high school math coursework, and high school GPA (in that order), such that each of these variables was inversely related to starting in the lower-level course. In addition, students were more likely to start in the lower-level STEM course if they were female, URM, or White (relative to Asian), and first-generation college students.

As shown in Table 2, the overwhelming majority of the 62 mean differences for covariate balance were below or near .10 SDs when propensity score weighting was implemented.² In fact, within the full sample, only one of the weighted mean differences was greater than an absolute value of .07 SDs. The largest mean differences occurred for first-generation status and high school GPA within the restricted-sample analyses for computer science coursework (around .20 SDs) and for the ALEKS placement test for the mathematics course in both the full sample and restricted sample (around .25 SDs). That said, these weighted group differences for the ALEKS test constituted a massive

reduction relative to the unweighted sample, as the analyses succeeded in drastically reducing the often substantial between-group differences in high school GPA, advanced high school math coursework, placement test scores, and major or program of study.

Although these balance statistics would be acceptable by some widely-used criteria, we took multiple additional steps to ensure the robustness of our conclusions. First, we examined different specifications of the model (e.g., removing one or two covariates, adding interactions among covariates), and we found that these choices did not affect the substantive conclusions of the study. Second, we discuss below (with caution) the few instances in which non-significant results were close to being statistically significant and/or had reasonably large effect sizes, especially for the analyses that had one or two covariates that exhibited somewhat larger between-group differences. It is worth noting that the results from this handful of analyses are consistent with the broader pattern of statistically significant findings.

In addition to these covariate balance statistics, the results for the covariates simultaneously predicting participation in the treatment (which are used to calculate the propensity score) appear in the Appendix. The most consistent predictors of starting in the lower-level course were placement test scores, advanced high school math coursework, and high school GPA, all of which exhibited inverse relationships; female students were also sometimes more likely to start in the lower-level course. Significant relationships were more often observed for chemistry and mathematics than for computer science.

Findings from Regression Analyses

The results for regression analyses for the full sample appear within the left-hand numerical columns of Tables 3-5. Relative to students who started in the higher-level chemistry course, students who started in the lower-level chemistry course had significantly higher course grades, first-year and final GPAs, and chemistry credits earned, and they were less likely to receive a DFWI grade in the course, be placed on academic probation, or receive a STEM degree. For computer science, students who started in the lower-level course were less likely to be placed on academic probation than those in the higher-level course, but no other significant differences were observed. In addition, students starting in the lower-level mathematics received higher course grades, first-year GPAs, overall GPAs, and mathematics credits earned, and they were also more likely to graduate with a STEM degree. These students in the lower-level mathematics course were also less likely to receive a DFWI grade in that class, be placed on probation, or graduate from the university with any degree.

Findings from Propensity Score Analyses

The AIPW results for placement into the lower-versus higher-level chemistry course predicting success outcomes are provided in Table 3. In the samples of all students and of students with selected majors, taking a lower-level course led to higher grades within this first gateway course and more total chemistry credits earned, but it was also associated with a lower likelihood of graduating with a STEM degree. Several favorable findings were statistically significant within the full-sample but not the restricted sample propensity score analyses: Starting in the lower-level course predicted a higher first-year and overall university GPA, as well as a reduced likelihood of receiving a DFWI grade in the course or being placed on academic probation. In contrast, the restricted-sample analyses found that starting in the lower-level course was associated a lower likelihood of graduating from the university, whereas this pattern was not significant within the full sample.

Table 4 displays the findings of propensity score analyses for computer science coursework. Within both samples, starting in the lower-level computer science class predicted a lower likelihood of receiving a DFWI course grade. Starting in the lower-level course was associated with a lower likelihood of being placed on academic probation in the full sample, whereas it predicted a lower likelihood of receive a STEM degree in the restricted sample. In addition, the restricted-sample analyses found a marginally significant (p < .09) result, such that students who started in the lower-level computer science course were less likely to graduate from the university than those who started in the higher-level course. These findings indicated 7-12 percentage-point differences on each binary outcome. Finally, within this small restricted sample (N = 389), starting in the lower-level course also predicted a nonsignificant trend toward greater retention to the second year (four percentage points, p = .16).

The results for propensity score analyses of the gateway mathematics coursework designed for biological sciences majors are shown in Table 5. Within both samples, students who started in the lower-level course accrued more mathematics credits than those who started in the higher-level course. In the full sample, various other positive results for lower-level coursework were observed: higher grade within the course, lower likelihood of receiving a DFWI grade within the course, higher first-year and overall university GPAs, and greater likelihood of graduating with a STEM degree. However, also within the full sample, students starting in the lower-level course were less likely to graduate at all from the university. No other results were statistically significant in the restricted sample analyses, but starting in the lower-level mathematics coursework had a non-significant negative effect size of nine percentage points that was consistent with the significant finding in the full sample.

Discussion

This study indicates that starting with a less advanced STEM gateway course may often be beneficial—and certainly not detrimental—for college students' short-term academic outcomes. Although the positive findings are more pronounced within the full-sample propensity score analyses, both the full-sample and restricted-sample analyses often observed that students who started in the lower-level course received significantly higher grades within that course, had a lower likelihood of receiving a DFWI grade in the course, and accrued a larger number of credits within that subject. For multiple STEM subjects, the full-sample analyses also found that students starting in the lower-level course had higher first-year GPAs and were less likely to be placed on academic probation than those starting in the corresponding higher-level course.

These results diverge notably from research on developmental education, which has frequently found negative effects of being placed into developmental courses in math and English instead of non-developmental courses (Valentine et al., 2017). However, the dynamics for placement into or out of developmental education differ from placement exclusively within non-developmental coursework in multiple ways. First, students may feel stigmatized if they are placed into developmental coursework, since they are receiving a very explicit message that they are not academically prepared for higher education. Second, many developmental education courses do not provide college credit, which means that students who place into developmental education may spend considerable time engaging in coursework that does not actually lead to satisfying their academic requirements toward a postsecondary degree or certificate (Scott-Clayton & Rodriguez, 2015). Through the lens of momentum (Wang, 2017), both of these dynamics would suggest that placement into developmental education may have a notably more adverse impact on motivational momentum than placement within non-developmental courses. Moreover, developmental education inherently creates greater challenges for curricular momentum, since students would have to take at least one course before they enrolled in the lower-level courses examined in this study. The courses examined here were generally taken as gateways into STEM majors than as general education requirements; this relevance to students' intended field of study may also bolster the motivational momentum for these students.

In contrast with the often favorable findings for short-term academic outcomes in the present study, several results from the propensity score analyses indicate that students who start in a lower-level STEM gateway course in chemistry, computer science, or mathematics were less likely to receive a bachelor's degree from the university (7-10 percentage points). Starting in lower-level coursework was also negatively related to receiving a STEM bachelor's degree for chemistry and computer science (11-12 percentage points in three different analyses), whereas it was positively related this outcome for mathematics in the full-sample analysis (seven percentage points). Moreover, when examining a short-term enrollment outcome, students' initial course level did not predict retention to the second year in any of the propensity score analyses. The frequently negative findings for graduation and non-significant findings for retention seem incongruous with the sometimes positive results for shorter-term academic achievement, which is strongly related to retention and graduation outcomes in prior research (see Mayhew et al., 2016; Pascarella & Terenzini, 2005). The momentum framework also suggests that early STEM momentum is predictive of subsequent educational outcomes in STEM (Chan & Wang, 2018; Wang, 2015, 2016).

Why, then, does success in a lower-level introductory STEM course not translate to longer-term attainment in STEM or overall? One possibility is that students who start in a lower-level course may receive higher grades on average, but they simultaneously may perceive that they are behind peers within their major who started in a higher-level course, which means that they are further away from moving into upper-level requirements for this intended major. This challenge with curricular momentum could adversely affect students' motivational momentum and professional identification as a scientist, thereby reducing their likelihood of persisting until graduation (Graham et al., 2013). The disconnect between the significant first-year coursework and GPA outcomes versus the lack of significant findings for retention to the second year is especially relevant, since this combination seems to imply that some other force(s) are counteracting the potential benefits of early academic achievement on continued college enrollment. Indeed, institutional conditions, the quality of STEM instruction, and supportive learning environments are all important for STEM persistence (Xu, 2018); it is possible that these lower-level and higher-level STEM courses create different learning opportunities and experiences that have implications for longer-term persistence and attainment.

Although the full-sample and restricted-sample analyses broadly supported some of the same conclusions, the full-sample analyses were far more likely to observe significant positive findings for lower-level coursework when predicting short-term and academic achievement outcomes, especially for chemistry and mathematics. This divergence begs the question of whether one set of results is more valid than the other. On the one hand, in the restricted-sample analyses, the selection of only students whose initial majors require the higher-level course led to substantial sample size reductions for the computer science and math analyses (59% and 74%, respectively), which considerably reduced the statistical power to detect significant findings. The full sample analyses also include some students who may ultimately switch into a relevant STEM major, since the restricted sample is based only on students' incoming major or program of study. On the other hand, the restricted-sample approach of examining only students who needed to eventually take the higher-level course has the notable advantage of focusing on students who must make this decision with the intent to progress within relevant coursework. As a result, these models may better reflect the potential tradeoffs between curricular momentum (by starting at different points within a specific, relevant STEM major pathway) versus motivational and cognitive momentum (by having different experiences within the initial STEM course and therefore different motivation to continue within the major or at the university). Providing additional evidence, preliminary analyses examined a less restrictive sample reduction of students whose majors were well-represented within the relevant gateway course pair; these analyses yielded results that were somewhat "between" the two sets of analyses described here (in terms of the frequency of identifying significant results for the short-term and academic achievement outcomes).

It is also worth noting some variation across the three STEM subject areas. Significant results were far less prevalent overall for the computer science pairing, which may be at least partially attributable to the much smaller sample sizes for these analyses. Moreover, the lone positive significant finding for lower-level coursework predicting a STEM bachelor's degree was surprising, and it is not clear how to reconcile this with the various negative findings for graduation overall and with a STEM degree. Each of the three course pairings differ from one another in important ways, which can contribute to this variation. For instance, idiosyncratic differences in the instructors and grading practices of these courses, the role of the content of this coursework within students' intended majors, and other factors may all shape these outcomes. As one key attribute that is observable within the present dataset, mathematics was the only STEM subject for which the DFWI rate was notably higher in the upper-level course than the corresponding lower-level course. This and other divergent factors across subjects may inform how course placement influences the teaching/learning, curricular, and motivational aspects of students' momentum within their degree programs.

Conclusion and Implications

This article provided one of the few studies of the impact of non-developmental course placement on college student success. The examination of a large sample size, consideration of several course pairs and various outcomes, use of doubly-robust propensity score analyses, and exploration of different samples for each analysis all constitute notable strengths of this study. These analyses also focused on gateway course sequences in key STEM fields and therefore add to the literature on STEM momentum and attainment. Overall, starting in a lower-level course (as opposed to a corresponding higher-level course) frequently led to higher academic achievement and more credits earned within the subject. However, this lower-level course placement was unrelated to retention and often negatively related to graduation.

These analyses were conducted with the clear intent of making recommendations for institutional policy and practice, but the mixed findings across outcomes unfortunately leads to challenges for doing so. The good majority of statistically significant results from this study were favorable for starting in a lower-level course, which would seem to imply the benefits of policies (e.g., via requirements for higher-level course enrollment) and practices (e.g., conversations with academic advisors) that lead students to start in less challenging coursework. However, because college graduation may be considered the ultimate success indicator, the adverse results for this very important outcome gives us pause. To what extent should administrators and practitioners care about positive short-term academic outcomes when these are not accompanied by greater retention and graduation rates? If anything, we recommend that institutions consider increasing the prevalence of students placing into more advanced early STEM courses, particularly when those courses do not have high DFW rates and are therefore less likely to create barriers for students' curricular and motivational momentum. The findings should also focus practitioners' attention on cultivating STEM persistence among those students who do begin in lower-level STEM courses so that short-term outcomes translate to long-term achievement.

While the present study provides useful insights into potential causal effects, we strongly encourage colleges and universities to conduct careful analyses of their own coursework when making these decisions. Institutions should consider each criterion that they use for course placement, the extent to which these criteria accurately distinguish between students who are more or less likely to be successful in a relatively advanced course, and the usefulness of any particular cutoff score and metric (e.g., for placement tests, admissions tests, AP or IB examinations). Specifically, on what basis were the determinations for these criteria made, and what does recent institutional evidence suggest about their utility? Even if a useful criterion is employed (i.e., it is a strong indicator of students' preparation for relevant material), an institution may be using a cutoff score that inadvertently places many underprepared students into advanced coursework or places many students into redundant lower-level coursework (Leeds & Mokher, 2022; Ngo, 2020; Scott-Clayton et al., 2014). An in-depth consideration of the decisions and processes pertaining to course placement would be useful for informing not only institutional assessment, but also research that may generalize to various institutions.

Colleges and universities should also attend to equity in their course placement decisions. For instance, students who hold minoritized identities are often adversely affected by stereotype threat in high-stakes testing situations (Nguyen & Ryan, 2008). Thus, relying on test scores can result in underestimating the preparation of negatively stereotyped students (Walton & Spencer, 2009), which then leads to systematic disparities in course placement. Indeed, the underrepresentation of minoritized students within college classrooms itself can increase grade disparities between identity groups (Bailey et al., 2020; Bowman et al., 2022, 2023; Oliver, 2023), thereby further contributing to the importance of these decisions for shaping equitable experiences and outcomes. One potential step toward a solution would be to provide several different criteria through which students can demonstrate their preparedness for higher-level coursework (i.e., using multiple measures; see Barnett et al., 2018).

The divergent results across outcomes in this study also broach a broader issue about assessing the potential impact of institutional practices and policies. Many studies focus on short-term academic outcomes for a variety of reasons: they are often easier to assess, they can be measured more quickly after an intervention or change is made, and they may be more strongly associated with the intervention (and therefore more likely to provide significant results). In contrast, longer-term, general outcomes are shaped by a wide variety of factors that sometimes extend beyond the control of individual colleges and universities (Mayhew et al., 2016; Roksa et al., 2022). The present study would have reached very different conclusions if it had not included graduation within this set of outcomes, so we strongly urge that future research and assessment employ long-term outcomes for examining the effects of relevant policies and practices.

Future research should more directly consider aspects of the holistic theoretical model of momentum (Wang, 2017) to provide a more in-depth understanding of relevant mechanisms. Specifically, this work could explore why early curricular momentum in STEM gateway courses does not necessarily translate into long-term outcomes, what sorts of counter-momentum frictions decelerate students' progress in STEM course sequences, and what can be done to support students to maintain momentum after success in the gateway STEM course. This subsequent inquiry could take a variety of forms, such as interviews with students and academic advisors or systematic mapping of students' course pathways and grades in those courses, which would provide specific, practical guidance for practitioners and administrators.

Footnotes

Acknowledgment

This research was sponsored by the University of Iowa P3 Program in Support of Strategic Priorities.

1. We also conducted supplementary analyses to examine these dynamics within student subgroups by race, sex, and first-generation status. In some instances, these analyses yielded small sample sizes, and consistently obtaining appropriate covariate balance between treatment and control conditions for all subgroups was also a problem. Given these challenges and the fact that supplementary analyses did not find consistent patterns of results across any demographic group, we do not present detailed results by student subgroup here. Additional supplementary analyses showed that the substantive findings did not vary across different choices for limiting the sample based on placement test scores.

2. There is no consensus in the literature on how small the differences within the propensity score weighted sample must be to determine whether the treatment and control conditions are sufficiently balanced. The available recommendations include mean differences between groups of no more than .25 standard deviations (SDs; Rubin, 2001; Stuart, 2010) or .20 SDs (Holmes, 2014), whereas others have suggested that a mean difference of no more than .10 SDs is negligible (Normand et al., 2001; Zhang et al., 2019). Additional publications refer to the existence of guidelines for .10 or .25 SDs while noting the presence of divergent opinions and not endorsing the strict use of a particular value (Austin, 2011; Harder et al., 2010; Pattanayak, 2015), and one highly-cited book on propensity score analyses provides no specific effect size criterion for making this determination (Guo & Fraser, 2015).

References

Attewell, P., Lavin, D., Domina, T., & Levey, T. (2006). New evidence on college remediation. The Journal of Higher Education, 77(5), 886-924.

Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399-424.

Bahr, P. R., Jackson, G., McNaughtan, J., Oster, M., & Gross, J. (2017). Unrealized potential: Community college pathways to STEM baccalaureate degrees. The Journal of Higher Education, 88(3), 430-478.

Bahr, P. R., McNaughtan, J., & Jackson, G. R. (2022). Reducing the loss of community college students who demonstrate potential in STEM. Research in Higher Education, 64(5), 675-704.

Bai, H., & Clark, M. H. (2019). Propensity score methods and applications. SAGE.

Bailey, E. G., Greenail, R. F., Baek, D. M., Morris, C., Nelson, N., Quirante, T. M., Rice, N.S., Rose, S., & Williams, K. R. (2020). Female in-class participation and performance increase with more female peers and/or a female instructor in life sciences courses. CBE—Life Sciences Education, 19(ar30), 1-14.

Bailey, T., Jeong, D. W., & Cho, S.-W. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29(2), 255-270.

Bang, H., & Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4), 962-973.

Barnett, E., Bergman, P., Kopko, E. M., Reddy, V. T., Belfield, C., & Roy, S. (2018). Multiple measures placement using data analytics: An implementation and early impacts report. Community College Research Center, Teachers College, Columbia University.

Bicak, I., Schudde, L., & Flores, K. (2023). Predictors and consequences of math course repetition: The role of horizontal and vertical repetition in success among community college transfer students. Research in Higher Education, 64(2), 260-299.

Boatman, A. (2021). Accelerating college remediation: Examining the effects of math course redesign on student academic success. Journal of Higher Education, 92(6), 927-960.

Boatman, A. & Long, B. T. (2018). Does remediation work for all students? How the effects of postsecondary remedial and developmental courses vary by level of academic preparation. Educational Evaluation and Policy Analysis. 40(1), 29-58.

Bowman, N. A., Logel, C., LaCosse, J., Jarratt, L., Canning, E. A., Emerson, K. T. U., & Murphy, M. C. (2022). Gender representation and academic achievement among STEM-interested students in college STEM courses. Journal of Research in Science Teaching, 59(10), 1876-1900.

Bowman, N. A., Logel, C., LaCosse, J., Canning, E. A., Emerson, K. T. U., & Murphy, M. C. (2023). The role of minoritized student representation in promoting achievement and equity within college STEM courses. AERA Open, 9(1), 1-16.

Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006). Variable selection for propensity score models. American Journal of Epidemiology, 163(12), 1149-1156.

Burdman, P. (2012). Where to begin? The evolving role of placement exams for students starting college. Jobs for the Future.

Cattaneo, M. D., Idrobo, N., & Titiunik, R. (2020). A practical introduction to regression discontinuity designs: Foundations. Cambridge University Press.

Center for First-Generation Student Success. (2018). First-generation student success: A landscape analysis of programs and services at four-year institutions. Center for First-Generation Student Success, NASPA-Student Affairs Administrators in Higher Education and Entangled Solutions.

Chan, H. Y., & Wang, X. (2018). Momentum through course-completion patterns among 2-year college students beginning in STEM: Variations and contributing factors. Research in Higher Education, 59(6), 704-743.

Chen, X. (2015). STEM attrition among high-performing college students: Scope and potential causes. Journal of Technology and Science Education, 5(1), 41-59.

Credé, M., & Niehorster, S. (2012). Adjustment to college as measured by the Student Adaptation to College Questionnaire: A quantitative review of its structure and relationships with correlates and consequences. Educational Psychology Review, 24(1), 133-165.

Daugherty, L., Gerber, R., Martorell, F., Miller, T., & Weisburst, E. (2021). Heterogeneity in the effects of college course placement. Research in Higher Education, 62(7), 1086-1111.

de Roux, N., & Riehl, E. (2022). Do college students benefit from placement into higher-achieving classes? Journal of Public Economics, 210, 104669.

Fischer, C., Witherspoon, E., Nguyen, H., Feng, Y., Fiorini, S., Vincent-Ruz, P., Mead, C., Bork Rodriguez, W. N., Matz, R. L., & Schunn, C. (2023). Advanced placement course credit and undergraduate student success in gateway science courses. Journal of Research in Science Teaching, 60(2), 304-329.

Fong, K., Melguizo, T., & Prather, G. (2015). Increasing success rates in developmental math: The complementary role of individual and institutional characteristics. Research in Higher Education, 56(7), 719-745.

Ganga, E., & Mazzariello, A. (2019). Modernizing college course placement by using multiple measures. Education Commission of the States.

Glynn, A. N., & Quinn, K. M. (2010). An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18(1), 36-56.

Graham, M. J., Frederick, J., Byars-Winston, A., Hunter, A. B., & Handelsman, J. (2013). Increasing persistence of college students in STEM. Science, 341(6153), 1455-1456.

Guo, S., & Fraser, M. W. (2015). Propensity score analysis: Statistical methods and applications (2^nd ed.). SAGE.

Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15(3), 234-249.

Holmes, W. M. (2014). Using propensity scores in quasi-experimental designs. SAGE.

Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community colleges. Community College Review, 39(4), 327-351.

Imbens, G. W., & Rubin, D. B. (2015). Causal inference for statistics, social, and biomedical sciences: An introduction. Cambridge University Press.

Jaggars, S. S., & Bickerstaff, S. (2018). Developmental education: The evolution of research and reform. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research (Vol. 33, pp. 469-503). Springer.

Jaggars, S. S., & Stacey, G. W. (2014). What we know about developmental education outcomes. Community College Research Center, Teachers College, Columbia University.

Kolesnikov, A., Wang, X., Bonaduce, M., Cunningham, M., Fontinell, L., Halliwell, T., & Twillman, M. (2020). Imagine no remediation: Evaluation of a placement policy change. Research in Higher Education, 61(7), 849-870.

Kolluri, S. (2018). Advanced Placement: The dual challenge of equal access and effectiveness. Review of Educational Research, 88(5), 671-711.

Kosiewicz, H., & Ngo, F. (2020). Giving community college students choice: The impact of self-placement in math courses. American Educational Research Journal, 57(3), 1358-1391.

Leeds, D. M., & Mokher, C. G. (2022). Unintended short- and longer-term consequences of assignment to college readiness courses: Evidence from Florida. Educational Policy, 36(6), 1315-1347.

Lesik, S. A. (2007). Do developmental mathematics programs have a causal impact on student retention? An application of discrete-time survival and regression-discontinuity analysis. Research in Higher Education, 48(5), 583-608.

Logue, A. W., Douglas, D., & Watanabe-Rose, M. (2019). Corequisite mathematics remediation: Results over time and in different contexts. Educational Evaluation and Policy Analysis, 41(3), 294-315.

Logue, A. W., Watanabe-Rose, M., & Douglas, D. (2016). Should students assessed as needing remedial mathematics take college-level quantitative courses instead? A randomized controlled trial. Educational Evaluation and Policy Analysis, 38(3), 578-598.

Mayhew, M. J., Rockenbach, A. N., Bowman, N. A., Seifert, T. A., & Wolniak, G. C., with Pascarella, E. T., & Terenzini, E. T. (2016). How college affects students (Vol. 3): 21st century evidence that higher education works. Jossey-Bass.

Melguizo, T. & Ngo, F. (2020). Mis/alignment between high school and community college standards. Educational Researcher, 49(2), 130-133.

Melguizo, T., Bos, J. M., Ngo, F., Mills, N., & Prather, G. (2016). Using a regression discontinuity design to estimate the impact of placement decisions in developmental math. Research in Higher Education, 57, 123-151.

Melguizo, T., Kosiewicz, H., Prather, G., & Bos, J. (2014). How are community college students assessed and placed in developmental math? Grounding our understanding in reality. The Journal of Higher Education, 85(5), 691-722.

Mervis, J. (2010). Undergraduate science. Better intro courses seen as key to reducing attrition of STEM majors. Science, 330(6002), 306.

National Academies of Sciences, Engineering, and Medicine. (2017). Undergraduate research experiences for STEM students: Successes, challenges, and opportunities. The National Academies Press.

Ngo, F. (2020). High school all over again: The problem of redundant college mathematics. The Journal of Higher Education, 91(2), 222-248.

Ngo, F., & Kwon, W. W. (2015). Using multiple measures to make math placement decisions: Implications for access and success in community colleges. Research in Higher Education, 56, 442-470.

Ngo, F., & Melguizo, T. (2016). How can placement policy improve math remediation outcomes? Evidence from experimental in community colleges. Educational Evaluation and Policy Analysis, 38(1), 171-196.

Ngo, F., & Melguizo, T. (2021). The equity cost of inter-sector math misalignment: Racial and gender disparities in community college student outcomes. The Journal of Higher Education, 92(3), 410-434.

Ngo., F. & Velasquez, D. (2020). Inside the math trap: Chronic math tracking from high school to community college. Urban Education.

Nguyen, H. H., & Ryan, A. M. (2008). Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence. Journal of Applied Psychology, 93(6), 1314-1334.

Normand, S. L. T., Landrum, M. B., Guadagnoli, E., Ayanian, J. Z., Ryan, T. J., Cleary, P. D., & McNeil, B. J. (2001). Validating recommendations for coronary angiography following acute myocardial infarction in the elderly: A matched analysis using propensity scores. Journal of Clinical Epidemiology, 54(4), 387-398.

Oliver, D. M. (2023). Underrepresented minority students in college: The role of classmates. . Educational Evaluation and Policy Analysis.

Park, E., Ngo, F., & Melguizo, T. (2021). The role of math misalignment in the community college STEM pathway. Research in Higher Education, 62, 403-447.

Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2): A third decade of evidence. Jossey-Bass.

Patrick, A. R., Schneeweiss, S., Brookhart, M. A., Glynn, R. J., Rothman, K. J., Avorn, J., & Stürmer, T. (2011). The implications of propensity score variable selection strategies in pharmacoepidemiology – An empirical illustration. Pharmacoepidemiology and Drug Safety, 20(6), 551-559.

Pattanayak, C. W. (2015). Evaluating covariate balance. In W. Pan & H. Bai (Eds.), propensity score analysis: Fundamentals and developments (pp. 89-112). Guilford.

Postsecondary National Policy Institute. (2022). First generation students in higher education. Author.

Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130(2), 261-288.

Roksa, J., Silver, B. R., & Wang, Y. (2022). Inequality in higher education: Sociological understandings of student success. In: N. A. Bowman (Ed.), How college students succeed: Making meaning across disciplinary perspectives. Stylus.

Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41-55.

Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services and Outcomes Research Methodology, 2(3), 169-188.

Rutschow, E. Z., Cormier, M. S., Dukes, D., & Cruz Zamora, D. E. (2019). The changing landscape of developmental education practices: Findings from a national survey and interviews with postsecondary institutions. Center for the Analysis of Postsecondary Readiness, Teachers College, Columbia University.

Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143(6), 565-600.

Scott-Clayton, J., Crosta, P. M., & Belfield, C. R. (2014). Improving the targeting of treatment: Evidence from college remediation. Educational Evaluation and Policy Analysis, 36(3), 371-393.

Scott-Clayton, J., & Rodriguez, O. (2015). Development, discouragement, or diversion? New evidence on the effects of college remediation policy. Education Finance and Policy, 10(1), 4-45.

Shields, K. A., & O'Dwyer, L. M. (2017). Remedial education and completing college: Exploring differences by credential and institutional level. The Journal of Higher Education, 88(1), 85-109.

Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010). Importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250-267.

Stuart, E. A. (2010). Matching methods for causal inference: A review and look forward. Statistical Science, 25(1), 1-21.

Thompson, M. E. (2021). Grade expectations: The role of first-year grades in predicting the pursuit of STEM majors for first-and continuing-generation students. The Journal of Higher Education, 92(6), 961-985.

U.S. Department of Education. (2022). Undergraduate enrollment. Condition of Education. Institute of Education Sciences, U.S. Department of Education.

Valentine, J. C., Konstantopoulos, S., & Goldrick-Rab, S. (2017). What happens to students placed into developmental education? A meta-analysis of regression discontinuity studies. Review of Educational Research, 87(4), 806-833.

Walton, G. M., & Spencer, S. J. (2009). Latent ability: Grades and test scores systematically underestimate the intellectual ability of negatively stereotyped students. Psychological Science, 20(9), 1132-1139.

Wang, X. (2015). Pathway to a baccalaureate in STEM fields: Are community colleges a viable route and does early STEM momentum matter? Educational Evaluation and Policy Analysis, 37(3), 376-393.

Wang, X. (2016). Course-taking patterns of community college students beginning in STEM: Using data mining techniques to reveal viable STEM transfer pathways. Research in Higher Education, 57(5), 544-569.

Wang, X. (2017). Toward a holistic theoretical model of momentum for community college student success. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research (Vol. 32, pp. 259-308). Springer.

Xu, Y. J. (2018). The experience and persistence of college students in STEM majors. Journal of College Student Retention: Research, Theory & Practice, 19(4), 413-432.

Zhang, Z., Kim, H. J., Lonjon, G., & Zhu, Y. (2019). Balance diagnostics after propensity score matching. Annals of Translational Medicine, 7(1), 16.