Statistical Profiling of Academic Oral English Proficiency Based on an ITA Screening Test

Abstract

At the University of California, Los Angeles, the Test of Oral Proficiency (TOP), an internally developed oral proficiency test, is administered to international teaching assistant (ITA) candidates to ensure an appropriate level of academic oral English proficiency. Test taker performances are rated live by two raters according to four subscales. While the subscale scores have potential as valuable feedback to major stakeholders, only a weighted average of the four subscale scores are currently reported and used. This study presents a way of extracting valuable information from the TOP subscale scores. In particular, it investigates an approach to obtaining oral English proficiency profiles based on the subscale score patterns of 960 TOP test takers. This study utilized item response theory and finite mixture modeling to investigate profiles of academic oral English proficiency in terms of the TOP subscales. A higher-order generalization of the graded response model was formulated to estimate subscale scores that accounted for structural dependencies and rater effects in the observed TOP scores. The estimated scores were clustered using a multivariate normal mixture model to yield subscale score profiles. The mixture model suggested seven profile groups and classified the TOP test takers into the seven groups. The profile groups were then interpreted and labeled based on characteristic score patterns and linguistic background shared by group members. To achieve a thorough understanding of the resulting profiles, discourse features of test taker performances sampled from different profile groups were closely examined. A small corpus was constructed based on the sampled test taker performances and compared to a reference corpus to explore the overall pattern of TOP test takers' language use. The comparison showed that the TOP test takers tended to use relatively fewer function words than speakers in the reference corpus. Characteristic features of each profile group's discourse were investigated through an identification and examination of discourse organizing lexical bundles. The results suggested that the use of metadiscourse and textual reference bundles with an explicit past reference point might be related to test takers' academic oral English proficiency. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone [phone]. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.]