Development and validation of an instrument for assessing secondary students’ transdisciplinary STEM practices

Abstract

Background

Given that limited well-developed instruments are available to assess students’ transdisciplinary STEM practices (T-STEMP), this study aims to develop and validate an instrument for examining secondary students’ T-STEMP competence. According to the T-STEMP assessment framework proposed in our previous study, we developed items and validated the items using Bayesian structural equation modeling (BSEM) and item response theory (IRT).

Results

First, the structure of T-STEMP has been justified, that the four design phases are related to a common third-order factor: knowledge-based reasoning and each design phase is composed of three key design challenges. The goodness of fit indices indicated that a two-parameter logistic (2PL) graded response model could fit the data well. The discrimination parameters indicated that most of the items for different grades performed well in distinguishing between students with various ability levels. The Cronbach’s alpha also indicated that the instrument had good reliability in terms of internal consistency.

Conclusions

The model of one third-order factor with four second-order factors and 12 first-order factors was justified as the fitted model representing the structure of the T-STEMP instrument. It means that the four design phases are related to knowledge-based reasoning and each design phase is composed of three key design challenges. The IRT findings show that Taiwanese students may have basic T-STEMP competence, but the breadth and depth of their responses to the key design challenges need to be advanced. Implications for further studies and suggestions for STEM teaching are also provided.