Content area
Purpose
The objective of this study is to explore the structural relationships among context (C), input (I) and process (P) and product (P) (CIPP) components of teacher preparation programs based on students' perceptions.
Design/methodology/approach
In this study, data were collected using a 17-item scale. The study sample consisted of 213 pre-service teachers enrolled in the Postgraduate Professional Diploma in Teaching (PPDT). Quantitative research methodology with multivariate structural equation modeling (SEM) was utilized to examine the two suggested models.
Findings
The results of this study show that the CIPP model can be projected into preservice teachers' perceptions of the CIPP components. These preservice teachers' perceptions of preparation programs from the three components (CIP) can predict preservice teachers' perceptions of teachers’ preparation program products or outcomes (i.e. the fourth CIPP component). This result indicates that the relationships between the CIPP components and the pre-service teachers' perceptions of the Diploma in Teaching program are direct.
Originality/value
Two suggested models were tested to explore the structural relationships between CIPP components. The first model represents the original CIPP model with indirect relationships between the four components: CIPP. The second model suggests direct relationships between the first three components (CIP) and the objectives or products (P).
Introduction
Educational programs need a system that can assess their performance and effectiveness regardless of their type. Stufflebeam (1971) stated that evaluation is the process of defining, gaining and providing helpful information for judging decision alternatives. Umam and Saripah (2018) add that it is an activity of collecting, analyzing and reporting information about an object, the results of which can be used in decision-making. Kuo et al. (2012) believe that evaluation is an activity that aims to understand how things happen.
Although program evaluation in education starts with gathering information (Brown, 1989), it achieves different purposes such as decision-making (Nunan, 1991; Aziz et al., 2018), assessing the program quality to deliver useful information to the program stakeholders (Lynch, 1990), gaining feedback to improve the program and prepare it for accountability (Peacock, 2009), and determining whether a program is effective or not (Tyler, 2013).
Many approaches in the literature can be used to evaluate educational programs. One of these approaches is the CIPP model. This approach was developed by Stufflebeam (1971), and it is used for formative or summative purposes by linking the four stages of the program: context (C), input (I), process (P) and product (P) stages of the program. The CIPP model has been a valuable tool for evaluating programs in higher education (Kitivo and Kavulya, 2024). It is a bidirectional model with a proactive approach in directing needs assessment, aims setting, planning, application, quality assurance and the focus on continuous improvement. In addition, it has a reactionary orientation by assessing the accountability and value of programs and provides effective participation of program stakeholders throughout the evaluation process (Stufflebeam and Zhang, 2017).
Robinson (2002) reported that this model was invented to link evaluation with program decision-making by providing information and choices to enhance the quality of decisions made by program administration (Fitzpatrick et al., 2011; Sankaran and Saad (2022). Nikijuluw (2020) added that CIPP can be used to help form a customized program that benefits participants of these programs.
This approach consists of four stages of evaluation: context, input, process and products. A stage selection relies on the program’s aims (Sankaran and Saad, 2022). According to Stufflebeam et al. (2000), the primary goals of a context assessment are to provide a comprehensive overview of the environment in which a particular service is to be implemented and analyze the requirements to uncover any potential challenges that prevent arriving at the goals. Additionally, the context assessment aims to define the available resources and financial possibilities that can be used to address these specific needs, all while evaluating the suitability and relevance of program products.
The context evaluation is conducted to answer the following questions: (1) What program parts have not met needs? (2) What development objectives are related to meeting needs? (3) What are the easily attainable objectives? In short, context evaluation is a judgment of the circumstances surrounding the program (Alvianita et al., 2022).
The input stage involves monitoring resource utilization for the implementation of the program. It aims to ease the implementation of the program as designed in the context stage and focuses on the main resources such as human resources, supporting facilities and equipment, funds or budget, procedures and rules, policies, strategies, limitations and hurdles of the education system (Saif, 2019).
Stufflebeam (1985) introduced the concept of “process evaluation for decision support,” a form of evaluation that is designed to assist in the implementation of decisions. The process evaluation stage highlights three key targets: firstly, it aims to identify or anticipate the program’s design or execution procedures at the implementation stage; secondly, it provides insights into the program’s outcomes; and finally, it ensures the documentation and record-keeping of the implemented procedures (Worthen and Sanders, 1987). Saif (2019) added that process evaluation encompasses the collection of data that has been planned and put into action during program execution. Its primary objective is to gauge the advancement and pinpoint specific areas that require attention. It involves identifying or estimating performance issues that may arise during program implementation and evaluates its effectiveness. This process also delves into how the educational program impacts the learners.
Product evaluations aim to assess a program’s outcomes to determine its success or failure through measuring, interpreting and evaluating the results (Sankaran and Saad, 2022). Zhang et al. (2011) added that the target of product assessment is to attain the outcomes achieved through implementing a system or program and to plan for subsequent actions or steps to be taken after the system or specific programs have been implemented. Furthermore, Stufflebeam (1985) added that product evaluation serves recycling decisions.
A significant target of CIPP evaluation is to enhance the performance of a program. The CIPP model is a cyclical process (Stufflebeam and Zhang, 2017) that prioritizes the assessment of the process over the final product, and this emphasis on process improvement is an essential aspect of evaluating the process stage. Its purpose is to refine and enhance the programs. Research has indicated that the CIPP model considers all levels of reviewing programs comprehensively. It furnishes the needed information required for constructive enhancements in educational programs and for making well-informed decisions. This model not only addresses specific questions but also places importance on systematically and comprehensively assessing the capabilities of a program (Toosi et al., 2021). Moreover, the CIPP model “is not exclusive of other defensible models to assessment or the wide range of specific inquiry techniques. Instead, it embraces and provides room for selectively and appropriately incorporating a complete pharmacopeia of sound, qualitative, and quantitative inquiry methods and tools” (Stufflebeam and Zhang, 2017, p. 9).
The CIPP evaluation model has been extensively implemented (Granger et al., 1965; Guba and Stufflebeam, 1968; Nevo, 1974; Stufflebeam et al., 1995, 2002; Candoli et al., 1997; Gally, 1984). Examples from empirical studies which evaluated different programs in different fields are many; to mention a few, Sankaran and Saad (2022) evaluated the Bachelor of Education at Malaysian University; Elmer (2023) employed CIPP to judge the relevance of the teacher education courses of one state university in Samar Philippines to the national licensure examination for teachers; Alhoushan (2020) evaluated distance learning during COVID-19 pandemic for students at the undergraduate level in Saudi universities, and Utsman and Kisworo (2020) evaluated the quality of Early Childhood Education Program Services in Indonesia. In an English education context, Erdogan and Mede (2021) evaluated the English Preparatory School Language Program in a Turkish state university, alongside Agustina and Mukhtaruddin (2019) who used Stufflebeam’s CIPP evaluation model qualitatively to evaluate an Integrated English Learning (IEL) program in Yogyakarta, Indonesia, from the students’ experience and perspective. Dehkordi and Talebinezhad (2018) also evaluated the effectiveness of high school grammar programs from the perspectives of instructors and students using the CIPP evaluation model.
Several research studies have used the CIPP model to evaluate educational programs. In these studies, the model was moved from a management-oriented evaluation model to a consumer-oriented evaluation model. In these studies, students were the source of information about the four CIPP components. Çolakoğlu et al. (2020) studied the dynamic relationships between the dimensions of the CIPP model in the “English Course Teaching Program.” The model that was tested is shown in Figure 1. The research results showed a direct linear relationship between the CIPP components. This supported model has no direct relationship between context and process or between the input and product components. The results of Çolakoğlu et al. (2020) showed that program input and process mediated the relationship between program context and product. On the other hand, Utsman and Kisworo's (2020) study evaluated the relationships between context, input evaluation and the Evaluation of Early Childhood Education Program products. The model that was tested is shown in Figure 2. The results showed that the CIP evaluations have positive and significant relationships with the program’s products. It can be seen from Figures 1 and 2 that this first one, which is supported by the study of Çolakoğlu et al. (2020), has four layers where every model component has affected the next component and is affected by the previous one. On the other hand, the second model, supported by the study of Utsman and Kisworo (2020), has only two layers. In this model, the three model components (CIP) have direct relationships with the fourth component (product). These two different structural relationships between CIPP components from program participants might raise a question about the stronger model that represents the relationships between CIPP components, and this is the main question and aim of the current study.
Research rationale
The main objective of this study is to test the relationships among CIPP components of teachers’ preparation programs based on students' perceptions. The focus is on a new construct, “student perceptions,” to determine the extent to which student perceptions could impact the relationships between the CIPP components of the PPDT program and which component (s), in particular, can predict the effectiveness of the educational program. This could contribute to the existing literature by providing a different understanding of the predictors of education program effectiveness from the early stages of evaluation development using the consumer CIPP-based model. The literature review shows two suggested models, as shown in Figures 1 and 2. The first model represents the original CIPP model, which is a linear model where making the right decisions in the previous stage enhances and develops the next stage, leading the program to achieve its objectives and products. The second model suggests that the CIP components lead to achieving the program objectives and products (P). Several research studies were conducted to evaluate educational programs from the participant’s point of view using surveys constructed based on the CIPP model using the second model (Utsman and Kisworo, 2020; Danju, 2017; Alvianita et al., 2022). The structural relationships among the CIPP components in the second model differ from those in the original CIPP model. Therefore, the main goal of this study is to test the validity of the two models.
Research question
The question that this study is trying to answer is: Are the relationships among CIPP components of teacher preparation programs based on students' perceptions direct or indirect and are they statistically significant? The models that are going to be tested are shown in Figures 1 and 2.
Program description
The “Postgraduate Professional Diploma in Teaching” (PPDT) program is designed for teachers and educators who already hold a bachelor’s degree and seek a professional qualification in teaching. The PPDT study plan requirements are 24 credit hours; 18 compulsory credit hours including the Practicum module (field internship) and 6 credit hours for 2 elective modules. The program was first accredited by the Commission for Academic Accreditation (CAA) in the United Arab Emirates (UAE) in 2006 and by the Council for the Accreditation of Educator Preparation (CAEP) in 2022.
Every year, almost 800 students graduate from this program. This program has two tracks. It is offered in both English and Arabic medium of instruction. In addition, it is multinational as it houses more than 20 nationalities among students and instructors.
Methodology
Instrument
Data were collected in this study using 17 five-point Likert scale items. The items were originally selected from Erdogan and Mede’s (2021) study and later adapted to suit the current study. The items are five-point Likert scale items (Strongly Agree (5) to Strongly Disagree (1)) distributed to four components that represent the CIPP model. Erdogan and Mede (2021) adapted these items from Stufflebeam’s (1971) CIPP model and provided evidence of the instrument’s psychometric properties. Moreover, the reliability of the instrument was assessed using the Cronbach’s alpha reliability coefficient based on the current dataset collected for the purpose of this study. The Cronbach’s alpha of the four components ranged 0.836–0.944. Table 1 shows the reliability coefficients of the four CIPP subscales of the survey.
Sample
Data were collected from students of the PPDT at Al-Ain University who voluntarily participated in this study. The voluntary sampling consists of 213 pre-service teachers, of which, 173 were females. They varied in their majors as follows: 89 in IT and Computer Science, 75 in Science Education and 49 in English Language Education. Data were collected by emailing the Google Form link to all students who are registered for the PPDT at Al-Ain University. The authors contacted the current module instructors in this program to motivate students to respond to survey. Although voluntary response sampling does not allow for generalizability due to the possible volunteer bias, it was time-effective, quick, flexible, easy and inexpensive to implement online on a large number of diverse respondents quickly and efficiently (Pineau and Slotwiner, 2003). The potential bias that the representative sample may imbed does not affect the findings of the study as the research objective is to generalize findings only to populations that share characteristics with the sample.
Data analysis
The research question in the current study requires comparing two models. Every model has four components and each component is measured by several items. The statistical analysis that can test the relationships between items and their components, and among the components is the multivariate statistical analysis technique. Structural equation modeling (SEM) is a multivariate statistical analysis utilized to analyze structural relationships. SEM is the combination of factor analysis and multiple regression, and it is used to analyze the structural relationship between survey items and the components they measure (Schumacker and Lomax, 2004). Thus, SEM combines two important models into one analysis. The measurement model summarizes how each latent variable (CIPP components) is operationalized according to the manifest indicators (survey items). The structural equation model is the second model, which illustrates the hypothesized relationships (which should be supported by the literature) between a set of independent and dependent variables or constructs.
In this study, SEM was used to test these two models. The sem option provided by the lavaan package has two parts: The first one is called the measurement model, which is a confirmatory factor analysis (CFA), which provides evidence of the construct validity of the used survey by testing the relationships between the items and the CIPP components that they measure. The second model is called the structural model. This model tests the relationships between the CIPP components. The efficiency of the model is assessed using efficiency indices or model fit measurement (as shown in Table 2); only if the model fit measurement indicate a good fit, the relationships between the observed variables and the latent variables can be assessed.
Results
To answer the research question, SEM under lavaan package was used to test the two models. The sem function under the lavaan package has two parts: The first one is called the measurement model, which is a confirmatory factor analysis (CFA), which provides evidence of the construct validity of the used survey by testing the relationships between the items and the CIPP components measured by these items.
Several global fit indices (model fit measurement) are used to evaluate the models, and there is no agreement about interpreting and assessing the cutoff criteria for these indices. Therefore, we follow the recommendation that suggests using more than one of the global indices (Devlieger et al., 2019). Hu and Bentler (1999) suggested using root mean square error of approximation (RSMEA) of 0.060 or less and standardized root mean square residual (SRMR) of 0.090 or more as a two-index cut-off score to assess model fit. Therefore, the four common global fit indices for the two models were calculated utilizing the sem function under the lavaan package, and the results are shown in Table 2.
The RMSEA is an indicator of the error of the approximate fit and when the suggested model is not misspecified, the value within the square root has the expected value of zero, this is why an RMSEA close to zero indicates a “best” fit. The SRMR is also a “badness of fit” indicator as it measures the averaged squared differences between each bivariate empirical correlation and the respective model-implied counterpart. The best value is zero means a perfect reproduction of the empirical correlation matrix, while higher SRMR values reflect a poorer model fit (Goretzko et al., 2024). Based on CFI, a model comparison between the suggested model and a baseline model is conducted. A value of one indicates that the suggested model provides the biggest improvement over the baseline model and can fit the data perfectly, while a value of zero means that the suggested model has no explanatory value (Goretzko et al., 2024).
The structural relationships tested in the first model represent the original CIPP model which is a managerial-oriented model that collects data about the program to identify strengths and limitations in content or delivery, to improve program effectiveness or to plan for the future of the program and the focus is on continuous improvement by concentrating on the four CIPP stages of the program. The data collected in this study represent the perceptions of preservice teachers about the CIPP components of the PPDT. The results of testing this model, as shown in Table 2, indicate the collected data does not fit that model.
Regarding the results of testing the second model, most of the global fit indices reached the cutoff criteria. Based on the mentioned criterion suggested by Hu and Bentler (1999) of RSMEA of 0.060 or less and SRMR of 0.090 or more, the second tested model has a model fit. This model indicates that preservice teachers' perceptions of preparation programs for three components (CIP) can predict preservice teachers' perceptions of teachers’ preparation program products (P) or objectives. This means that the perceptions of preservice teachers about the CIPP components of the PPDT fit this model. The significance of the relationships between the components of this model was investigated. The results are shown in Tables 3 and 4 and Figure 3. The results are shown in two tables for simplifications only. Table 3 represents the measurement model which is CFA that provides information about item loadings on its component, while Table 4 shows the structural model which represents the relationships between the CIPP components or latent variables.
Table 3 shows the loadings of each item in the survey on its factor or preservice teachers' perceptions of teachers’ preparation program CIPP components and its standardized parameter regression (path) coefficients (PE.est), standard errors of the parameter (PE.se), and p-values (PE.p-value) and standardized parameter path (regression) coefficients (PE.est). Path coefficients are standardized because they are estimated from correlations (a path regression coefficient is unstandardized). A positive coefficient means that a unit increase in the variable of one structure leads to a direct increase in the variable of structures it projects to, proportional to the size of the coefficient. All of these coefficients presented in Table 3 are statistically significant. As stated earlier, these CFAs add evidence that the survey or data collection tool used to collect the data has good construct validity. The content of Table 3 indicates that the items of every CIPP component have the ability to measure that component, and this is the reason behind naming the model a measurement model.
The path coefficients shown in Table 4 show the direction and the strength of the relationships between the perceptions of preservice teachers about the product (P) component of the Diploma in Teaching (PPDT) program and the other three components (CIP). All of these coefficients are positive and statistically significant. This part of the model called structural equation model shows structural relationships between the CIPP components. In the terms of this study, the perceptions of preservice teachers about the three components (CIP) components of the Diploma in Teaching program (PPDT) can predict or explain the perceptions of preservice teachers about the product (P) component of the PPDT program. It is worth noticing that the highest path coefficient is between Input (I) and the Products (P). This indicates that from students’ perceptions, the input of the program contributes more than other components to achieve the program objectives or products. This finding resonates with what Guerra- López (2008) and Robinson (2002) suggest about the need for evaluators to refrain from waiting until the completion of the evaluation process, bearing in mind that the original CIPP model features a holistic interactive relationship among its components.
The study also yields a significant similar finding to that of Pada et al. (2024) who examined Indonesian students’ perceptions of the implementation of E-Learning USK using the CIPP components and yielded a positive relationship between students' perceptions and the effectiveness of implementing a program. However, the current study contributes to the current literature in that the perceptions of students on context, input and process can be indicators of their perceptions of the fourth component which is “outcome”.
Summary and discussion
The results of this study show that the CIPP model can be projected into preservice teachers' perceptions about the CIPP components of the Diploma in Teaching program (PPDT). From preservice teachers’ perspectives, CIP have direct positive relationships with program products or outcomes. This result suggests that evaluation involves systematic collection and analysis of information related to a program’s design, implementation and outcomes from students’ perspectives to monitor and improve the effectiveness of the educational program activities, characteristics and products.
CIPP components contrary to the original indirect relationships between CIPP components, this result indicates that the relationships between the CIPP components from pre-service teachers' perceptions regarding the Diploma in Teaching program (PPDT) program are direct relationships with a two-layers’ structure. Stufflebeam’s original work on program evaluation focused on the role of evaluation in decision-making. Accordingly, this model should generate information for decision-making, accountability and improvement of the education program (Stufflebeam and Shinkfield, 2007).
Therefore, knowledge of the programs' products is essential to stakeholders such as educational administrators, boards and policymakers who are responsible for funding and organizational effectiveness. Evidence of program products is also essential for developing programs that better meet the needs of the targeted audience or students. Scriven (1991) stated that evaluation refers to defining the merit, worth or value of something or the product of that process. The results of this study are in line with Scriven’s (1991) definition of evaluation as the results indicate the students could be a reliable and important source of data that could be used to understand better the relationships between three components of the Stufflebeam model (CIP) and the products of that educational program, and at the end determining the merit, worth or value of an educational program.
As the focus of the study is on preservice teacher perceptions of their training program, one of the study’s points of strength, it provides evidence that students' perceptions of CIP model components can help program administration assess the program’s ability to achieve its goals and objectives (products). Product evaluation aims to gauge and interpret attainments during and after the program. It is the decision-maker who uses the collected information to determine the program’s worthiness. This approach in evaluation is called the consumer-oriented model, from the fact that it is an evaluation approach that, in the context of this study, takes students' points of view into account to develop the educational programs, which can also be a significant factor in the predictions of the effectiveness of the “product” and the overall educational programs. The CIPP model is mainly a formative evaluation model while the consumer-oriented model is summative. The results of this study could indicate the ability to connect the two models to evaluate an educational program, yielding another point of strength, which is the integration of formative and summative evaluation approaches. The finding of this study could be an impetus for further evidence-based research on other educational programs at different levels (pre-service or in-service teachers) in other culture-specific contexts to compare and contrast results based on the student perceptions factor. Acknowledging the sample in this study (n = 213) and the statistical analysis used in this study, replicating this study on a larger sample size of preservice teachers is recommended to increase the external validity of the sampling and allow for generalizability.
It should be taken into account that the results of this study are based on students' points of view regarding the CIPP model components of one educational program, which is the teacher education program. Therefore, it is highly recommended that future research investigates the structural relationships between CIPP components from the faculty members of the Diploma in Teaching program (PPDT). Evaluation facilitates understanding educational programs' unforeseen dynamics (Patton, 2011), and Riech (1983) stated that program evaluation aims to identify the “knowledge gap” between the program stakeholders. Therefore, having evaluation studies similar to this study but from faculty members of teachers' education programs helps define this gap and develop the program.
Moreover, it is highly recommended that future research utilize data from students' points of view from different colleges and levels other than the College of Education. Another suggestion for future research is to develop a reliable and valid instrument tool to measure CIPP components from program participants' perspectives, as the tool used in this study is designed to measure preservice teachers' perspectives of their training program.
Figure 1
The first model
[Figure omitted. See PDF]
Figure 2
The second model
[Figure omitted. See PDF]
Figure 3
Final supported model
[Figure omitted. See PDF]
Table 1
The psychometric properties of students' perceptions of CIPP model components
| CIPP component | No of items | Cronbach Alpha |
|---|---|---|
| Context | 4 | 0.93 |
| Input | 5 | 0.87 |
| Process | 4 | 0.91 |
| Product | 4 | 0.87 |
Source(s): Authors’ work
Table 2
Global fit indices of the two suggested models
| Fit index | Value: Model 1 | Value: Model 2 | Cut-off criteria |
|---|---|---|---|
| 3.71 | 3.59 | <5 | |
| Root Mean Square Error of Approximation (RMSEA) | 0.112 | 0.102 | <0.06 |
| comparative fit index (CFI) | 0.894 | 0.914 | >0.90 |
| Standardized root mean square residual (SRMR) | 0.150 | 0.062 | <0.08 |
Source(s): Authors’ work
Table 3
Measurement model (CFA) statistics
| CIPP | Item no. | PE.est | PE.se | PE.z | PE.p-value |
|---|---|---|---|---|---|
| Context | Q1 | 1.000 | 0.000 | ||
| Q2 | 1.049 | 0.053 | 19.842 | 0.000 | |
| Q3 | 1.019 | 0.050 | 20.369 | 0.000 | |
| Q4 | 0.970 | 0.053 | 18.193 | 0.000 | |
| Input | Q5 | 1.000 | 0.000 | ||
| Q6 | 1.119 | 0.062 | 18.017 | 0.000 | |
| Q7 | 1.099 | 0.063 | 17.365 | 0.000 | |
| Q8 | 0.953 | 0.068 | 14.027 | 0.000 | |
| Q9 | 0.733 | 0.105 | 6.982 | 0.000 | |
| Process | Q10 | 1.000 | 0.000 | ||
| Q11 | 0.953 | 0.050 | 18.955 | 0.000 | |
| Q12 | 0.842 | 0.053 | 15.982 | 0.000 | |
| Q13 | 0.839 | 0.061 | 13.691 | 0.000 | |
| Product | Q14 | 1.000 | 0.00 | ||
| Q15 | 1.250 | 0.105 | 11.928 | 0.000 | |
| Q16 | 1.031 | 0.086 | 11.936 | 0.000 | |
| Q17 | 1.121 | 0.103 | 10.924 | 0.000 |
Source(s): Authors’ work
Table 4
Structural model statistics
| Predicted | CIPP- predictor | PE.est | PE.se | PE.z | PE.p-value |
|---|---|---|---|---|---|
| Product | Process | 0.166 | 0.043 | 3.849 | 0.000 |
| Product | Input | 0.520 | 0.126 | 4.124 | 0.000 |
| Product | Context | 0.097 | 0.104 | 0.842 | 0.040 |
Source(s): Authors’ work
© Emerald Publishing Limited.