A Mixed-Methods Evaluation to Measure 4-H STEM Program Quality

Abstract

The 4-H Science: Building a 4-H Career Pathway Initiative was a 3-year collaboration among National 4-H Council, Lockheed Martin, and state 4-H grantees to help more than 50,000 youth in 13 states develop STEM and workforce skills for STEM professions. A mixed-methods design used observations and interviews to assess program quality. Researchers observed 4-H STEM programming and conducted individual and focus group interviews with youth, parents, community volunteers, corporate volunteers, and professionals. Observations were conducted using a validated observational tool, the Out-of-School Time (OST) Observation Instrument with STEM Plug-In. This instrument measured youth relationship building, youth participation, staff relationship building, staff instructional strategies, activity content and structure, and STEM instruction. Findings from the observations and interviews were combined to assess program quality. Sites scoring highest on the OST Observation Instrument reported using quality STEM curriculum, especially National 4-H Youth Science Day lessons. The 4-H STEM programs demonstrated highly evident and consistent youth relationship building (e.g., being friendly and collaborative) and youth participation (e.g.., contributing ideas and taking leadership). Yet, in many cases, STEM youth skill development (e.g., drawing connections to real-world concepts) and STEM staff instructional practices (e.g., discussing how youth could pursue STEM content through their education and/or career) were inconsistent and rarely evident. Recommendations include substantive professional and volunteer development for both STEM competencies and enhanced youth development. The OST Observation Instrument with STEM Plug-In provided a comprehensive tool to evaluate program quality, and it is recommended for use in evaluating other 4-H STEM programs.