Content area
Background
Student evaluations of teaching (SETs) provide educators feedback to improve the quality of instruction; however, response rates often are low. Incentives may be used, but there is limited data on their effect on SETs response rates and course means.
Method
This study used a descriptive study design.
Results
A total of 250 courses were included in the data review. SETs response rates were significantly higher in those courses that offered an incentive, yet course means were higher in courses that did not offer an incentive. SETs response rates were higher for required courses; course means were higher for non-clinical and elective courses.
Conclusion
This study adds to the body of evidence regarding the use of incentives for completing SETs and other factors that may influence response rates and course means. This study found that incentives increased response rates on SETs without affecting course ratings.
Full text
In higher education, student evaluations of teaching (SETs) serve a critical role in providing feedback on course content, instructional methods, and overall student learning experiences. The purpose of SETs is to help educators improve the quality of instruction and identify areas for refinement. Additionally, SETs provide institutions with data for making informed decisions about course improvements and faculty performance. However, it is important to recognize that SETs are not without limitations. SETs can be prone to biases, such as gender, race, or personality biases, which may skew results (Stroebe, 2020). Furthermore, students may not always take SETs seriously, leading to concerns about the validity and reliability of the feedback collected (Sullivan et al., 2024).
Ratings on SETs are not always related to the quality of teaching. Students, especially those earlier in their educational programs, acknowledge that they are more likely to give positive ratings for courses with easier or fewer examinations (Almakadma et al., 2023). There is evidence that more challenging courses and courses with a higher workload receive lower ratings on SETs and that students in health care programs tend to rate clinical and elective courses more positively (Constantinou & Wijnen-Meijer, 2022). In recent years, the effects of the coronavirus disease 2019 (COVID-19) pandemic caused changes in the learning experience, some of which have been sustained and have been correlated with changed student ratings on SETs (Rayner & Papakonstantinou, 2024). Rayner and Papakonstantinou (2024) found that students' perceptions of teaching excellence were significantly lower in 2022 (during the pandemic) compared with 2017; however, mean SETs were significantly higher in 2022 compared with 2017, indicating a discordance between SETs and students' perceptions of teaching excellence.
One of the main problems with using SETs is the consistently low response rates, which have been well-documented in the literature (Lipsey & Shepperd, 2021). Low response rates can lead to incomplete or biased data, making it hard to reach meaningful conclusions from the feedback. When only a small portion of students complete SETs, the results may overly represent the views of those with extremely positive or negative experiences, distorting the overall picture. This lack of engagement can hinder the usefulness of SETs as a tool for improving course quality. Addressing the root causes of low response rates is a persistent challenge for educators.
To combat low response rates, various strategies have been employed to incentivize students to complete SETs (Lipsey & Shepperd, 2021). Some institutions offer extra credit, early access to grades, or even monetary rewards to encourage participation. Although these incentives can lead to higher response rates, they raise questions about the quality of feedback provided. Students might rush through SETs to receive the incentive, offering less thoughtful or reflective responses. Additionally, there are concerns that such strategies may undermine the integrity of the evaluation process, as students may not provide authentic feedback if their primary motivation is the incentive itself rather than a genuine desire to improve the learning experience. This has led to an ongoing debate about the most effective and ethical ways to increase participation in SETs (Sullivan et al, 2024).
Local Problem
Within a prelicensure nursing program at a small private university, there was variation in how—or if—incentives were used for SET. Between Fall 2017 and Summer 2021, faculty could choose if and what incentives were offered for their courses. Due to student dissatisfaction with variation in incentives offered, the prelicensure nursing program faculty voted to standardize incentives. Faculty could still choose whether or not they wanted to offer an incentive; however, if they did, each faculty was required to offer the same incentive of 1% bonus added to the student's final grade if >90% of the students completed both course and faculty SETs; this change went into effect in Fall 2021. However, due to perceived unprofessional student behavior (i.e., peer pressure and coercion of classmates to complete SETs), the faculty voted in December 2023 to prohibit faculty from offering bonus points as incentives for SETs completion; as such, no point-based incentives were provided for Spring 2024 courses. With this change, the response rate decreased drastically. Between Fall 2021 and Fall 2023, the average response rate was 88%; for Spring 2024, the response rate was only 32.8%.
From the literature, grade incentives have been shown to increase SETs response rates (Gordon et al, 2018). However, it is unclear if higher response rates are also associated with different course means. To better understand how response rates may affect course means, a group of faculty sought to review our local institutional data for the pre-licensure nursing program. The purpose of this study was to determine how providing incentives for completion of SETs affect response rates and course means. Secondary outcomes included analyzing the differences in response rates and course means between the following variables: (1) type of course: clinical versus non-clinical and required versus elective courses; (2) semester which the course was offered; and (3) courses offered prior to (Fall 2017 to Fall 2019) or after (Spring 2020 to Spring 2024) the COVID-19 pandemic.
Method
Design, Setting, and Sample
A retrospective descriptive study design was used. The setting was a small, private university in the southeastern United States that admits approximately 70 prelicensure nursing students twice a year (fall and spring entry). The prelicensure program was an accelerated bachelor of science in nursing (ABSN) program for students with a previous degree(s). The program is completed in approximately 16 months over four semesters.
Analysis
Response rate and course mean data were analyzed from Fall 2017 through Spring 2024. Response rates were calculated as number of SETs received divided by the total number of students enrolled in the course. Within the SETs survey, students were asked to rate five questions using a 4-point Likert scale ranging from 1 = strongly disagree to 4 = strongly agree. Overall course means were calculated by totaling responses for each question and dividing by the total number of responses. Response rate and course mean data were provided in aggregate form by the institution's Office of Academic Assessment and Evaluation.
Additionally, data on whether courses offered incentives were collected. If offered, the type of incentive was collected (e.g., points added to an assignment, percentage added to the overall grade, or raffles for prizes) along with the stipulations guiding whether incentives were provided (e.g., >90% of students completed SETs). These data were collected by reviewing the syllabi for each course, as it is standard for this institution to include information about SETs incentives within the syllabus. If this information was not provided in the syllabus, the course faculty was contacted.
Independent sample t tests were used to determine whether there were significant differences in response rate and course means between the stated variables. An analysis of variance (ANOVA) test was used to determine differences in response rates and course means between semesters. Data were analyzed using Excel®, and the level of significance was set at p < .05. A run chart of the data also was developed to visually see changes in response rates and course means over time. This study was approved by the university's Institutional Review Board and deemed to not be human subjects' research.
Nursing Faculty Contributions
Three nursing faculty members (SSR, LL, VK) designed the study, with one faculty member (SSR) analyzing and interpreting the data. The school of nursing's evaluation program manager provided the historic data needed to complete the analyses. All four members (SSR, LL, VK, AB) of the research team (three faculty [SSR, LL, VK] and one evaluation program manager [AB]) prepared the article.
Results
The nursing program offer classes in spring, summer, and fall semesters. A total of 20 semesters of data and 250 courses were included in the analysis. Average course means and response rates are provided in Figure 1.
Incentives Offered
Of the 250 courses with SETs between Fall 2017 and Spring 2024, a total of 84 courses offered incentives. Most of the courses (n = 75 [89.2%]) offered point incentives (e.g., one bonus point if there was >90% participation on course and faculty evaluations). Less than half (n = 104 [41.6%]) of the course faculty did not offer incentives. Many faculty did not offer incentives for various reasons, including (1) the belief that completing SETs is part of a student's professional obligation; (2) a concern that incentives may influence students' ability to pass the course, rather than reflect their actual performance; and (3) a concern that students may be motivated to complete SETs solely for the incentive, leading to less meaningful feedback. For the remaining 62 courses, information regarding incentives included (a) unable to locate the syllabus (n = 2); (b) incentives were not mentioned on the syllabus and faculty could not recall if they had offered incentives (n = 34); or (c) information was not mentioned on the syllabus and faculty could not be reached (n = 26). As such, analysis regarding incentives was only conducted on the 188 courses for which data were available.
Not surprisingly, the response rate was significantly higher in those courses that offered an incentive (M = 89.1% [SD = 11.5]) versus those that did not (M = 58.2% [SD = 19.7], p = .0001). Interestingly, the course mean was significantly higher in the courses that did not offer an incentive (M = 3.54 [SD = 0.1]) compared with those courses that did offer an incentive (M = 3.41 [SD = 0.28], p = .002), although this difference in means likely was not clinically significant, as both means were above the institutional benchmark of 3.0.
Data also were analyzed between semesters in which there was no standardized incentive (Fall 2017 to Summer 2021) and those semesters in which the optional standardized policy for incentives was in place (Fall 2021 to Fall 2023); Spring 2024 data were not included, as incentives were not allowed based on the new policy. Response rates during semesters in which there was a standardized policy were significantly higher (n = 89, M = 88% [SD = 12.87]) than the previous semesters (n = 149, M = 64.72% [SD = 18.88], p = .0001). However, the course means were similar (pre: M = 3.46 [SD = 0.34] versus during policy (M = 3.44 [SD = 0.28], p = .56). As previously mentioned, the response rate for Spring 2024 when incentives were not allowed was only 32.8%. The average course mean for Spring 2024 was 3.40, which was similar with previous semesters. These findings indicate that incentives can increase response rates without significantly affecting the course means.
Clinical Versus Nonclinical Courses
The response rate for clinical courses (n = 140, M = 72.8% [SD = 21.4]) was similar to nonclinical courses (n = 110, M = 69.7% [SD = 22.17], p = .27). However, the course mean was significantly higher for nonclinical courses (mean = 3.55 [SD = 0.31]) compared with clinical courses (M = 3.37 [SD = 0.32], p = .0001), which could be due to students' perception that nonclinical courses are less rigorous.
Required Versus Elective Course
The response rate for required courses (n = 201, M = 73.15% [SD = 20.96]) was significantly higher than elective courses (n = 49, M = 64.38% [SD = 23.66], p = .02). However, the course mean was significantly higher for elective courses (mean = 3.61 [SD = 0.32]) compared with required courses (mean = 3.41 [SD = 0.32], p = .0002).
Semester
An ANOVA was conducted to evaluate differences in response rates among semesters. Data were coded for each of the four semesters, with an additional code for those courses open to both third- and fourth-semester students. There was a significant difference in response rate means, with the highest response rate from third-semester courses (n = 62, M = 79.8% [SD = 16.92]); the response rate was lowest in those courses offered to both third- and fourth-semester students (n = 25, M = 61.1% [SD = 22.99], p = .0005). Conversely, the course mean was highest in those courses offered to both third- and fourth-semester students (M = 3.69 [SD = 0.39]); the lowest course mean was identified in the fourth semester (M = 3.42 [SD = 0.27], p = .008).
Before and After COVID-19
The response rate was significantly higher in courses after COVID-19 (Spring 2020 to Spring 2024, n = 91, M = 77% [SD = 20.63]) compared with before COVID-19 (Fall 2017 to Fall 2019, n = 159, M = 61.75% [SD = 20.31], p = .0001). There was no difference in course means before COVID-19 (M = 3.43 [SD = 0.36]) and after COVID-19 (M = 3.46 [SD = 0.31], p = .41).
Discussion
As expected, the response rate for SETs was increased significantly when an incentive was provided. Interestingly, course means were significantly higher for courses that did not provide an incentive for completion of SETs. This differs from much of the literature. For example, Lipsey and Shepperd (2021) found that faculty who incentivized students for completing evaluations saw an improvement in the response rate but no significant difference in means from courses without incentives. Uttl (2021) described many “teaching effectiveness irrelevant factors” that correlated to higher SETs ratings to include incentives, such as offering cookies or chocolate at the time of administering the SETs.
This study found that course means were higher for nonclinical courses and elective courses. Higher ratings for elective courses are reported in the literature, but the evidence about clinical courses contradicts the findings of the current study. In the nursing curriculum, clinical courses tend to be more challenging and pose a heavier workload, which are factors associated with lower ratings on SETs, but the literature also reports that students in health care programs tend to rate clinical courses higher (Constantinou & Wijnen-Meijer, 2022). The findings of the current study could be explained by the fact that the accelerated nursing program only includes nursing courses, with liberal arts and general education courses being prerequisite to the program.
Although this study found a significantly higher SETs response rate for third semester students, there was no literature found on SETs response rates by semester or year in a nursing program for comparison. During their third semester, students take a total of five courses (two clinical, two nonclinical, and one elective). The explanation for this difference in SETs response rate by semester is unclear; however, one potential explanation for the higher response rate during the third semester may be due to the possibility that students are more familiar with the routines of nursing school and may be less overwhelmed by the competing demands. Additionally, the content and intensity of these courses, which often involve more advanced material, also might encourage students to engage more actively in the evaluation process. The highest course means were from the courses taken by both third- and fourth-semester students, which were elective courses; this is consistent with prior evidence. The lowest course mean was for fourth-semester students. The literature has reported that students early in their program of study were more likely to rate courses more positively if the workload and rigor was lower (Almakadma et al., 2023), but the findings of the current study are the first to describe a difference in ratings from students at the end of their academic program.
The COVID-19 pandemic had a huge effect on academia, prompting changes and innovations in teaching and learning. Many of these changes and innovations have been continued after the pandemic, such as hybrid or remote learning strategies. Although students recognize these changes, their responses on SETs have not changed significantly (Rayner & Papakonstantinou, 2024). The current study supported this finding, with no significant change in course means between pre- and post-COVID-19 SETs.
There are many persuasive and impassioned arguments in the literature that SETs are not effective or accurate measures of teaching excellence (Uttl, 2021). Considering the myriad factors that are not relevant to teaching effectiveness but influence student ratings on SETs, there is an understandable call for deemphasizing the importance of SETs, particularly in faculty tenure and promotion decisions (Cook et al., 2022). The value of SETs may not be to measure teaching excellence, but may offer educators insight into the student's learning experience and the alignment or lack thereof between the course and the student's expectations. SETs often are the only measure of teaching quality or efficacy, outside of grades, and therefore are valuable for course planning and improvement.
SETs are an established part of academic evaluation and as such, it is essential to collect the best quality of data possible. By increasing the response rate, we can gather more robust and reliable quantitative data to identify areas for improvement. Therefore, it is imperative to implement strategies to boost student participation in SETs.
Limitations and Recommendations for Future Research
Although the SETs from 250 courses held over 20 semesters was a robust data pool, this study was limited by being conducted at a single institution and considering data from one academic program. In addition, the researchers were limited by not having information about incentives for SETs completion on a portion of the courses. Future research should compare SETs response rate and course ratings data from multiple institutions and across multiple programs. Additionally, it would be valuable to analyze the data received from the SETs in terms of component scores and qualitative comments. Lastly, many factors may influence SETs response rate and course means outside of the variables that were analyzed in this study.
Conclusion
This study analyzed response rate and mean ratings of SETs from 20 semesters of an ABSN program. Offering students incentives for the completion of SETs resulted in significantly higher response rates, without significantly altering mean ratings. As the most commonly available view into the student perspective of learning, SETs have value despite their limitations. Increasing response rates on SETs offers the opportunity to collect more and representative data on the student experience of learning. This study demonstrates that incentives increase response rate on SETs without affecting course ratings, supporting the use of incentives for this purpose.
Almakadma, A. S., Fawzy, N. A., Baqal, O. J., & Kamada, S. (2023). Perceptions and attitudes of medical students towards student evaluation of teaching:Across-sectional study. Medical Education Online, 28(1), 2220175. 10.1080/10872981.2023.2220175 PMID: 37270796
Constantinou, C., & Wijnen-Meijer, M. (2022). Student evaluations of teaching and the development ofacomprehensive measure of teaching effectiveness for medical schools. BMC Medical Education, 22(1), 113. 10.1186/s12909-022-03148-6 PMID: 35183151
Cook, C., Jones, J., & Al-Twal, A. (2022). Validity and fairness of utilising student evaluation of teaching(SET) asaprimary performance measure. Journal of Further and Higher Education, 46(2), 172–184. 10.1080/0309877X.2021.1895093
Gordon, H., Stevenson, E., Brookhart, A., & Oermann, M. H. (2018). Grade incentive to boost course evaluation response rates. International Journal of Nursing Education Scholarship, 15(1), 20180031. 10.1515/ijnes-2018-0031 PMID: 29920182
Lipsey, N., & Shepperd, J. (2021). Examining strategies to increase student evaluation of teaching completion rates. Assessment & Evaluation in Higher Education, 46(3), 424–437. 10.1080/02602938.2020.1782343
Rayner, G., & Papakonstantinou, T. (2024). COVID-19 reshaped students' perceptions of teaching excellence but not their evaluations of teaching. Journal of University Teaching and Learning Practice, 21(9). 10.53761/j5rhea97
Stroebe, W. (2020). Student evaluations of teaching encourages poor teaching and contributes to grade inflation:Atheoretical and empirical analysis. Basic and Applied Social Psychology, 42(4), 276–294. 10.1080/01973533.2020.1756817
Sullivan, D., Lakeman, R., Massey, D., Nasrawi, D., Tower, M., & Lee, M. (2024). Student motivations, perceptions and opinions of participating in student evaluation of teaching surveys:Ascoping review. Assessment & Evaluation in Higher Education, 49(2), 178–189. 10.1080/02602938.2023.2199486
Uttl, B. (2021). Lessons learned from research on student evaluation of teaching in higher education. In Rollett W., Bijlsma H., & Rohl S.(Eds.), Student feedback on teaching in schools: Using student perceptions for the development of teaching and teachers(pp. 237–256). Springer. 10.1007/978-3-030-75150-0_15
The authors thank Ann Brookhart (Duke University School of Nursing) for assistance with this project.
From Duke University, School of Nursing, Durham, North Carolina.
Disclosure: SSR is the Editor-in-Chief of the Journal of Nursing Care Quality . The remaining authors have disclosed no potential conflicts of interest, financial or otherwise.
Copyright 2025, SLACK Incorporated