Content area
The key challenges in the delivery of distance learning programs include maintaining student engagement and ensuring that students achieve intended learning outcomes. While video content has the advantage over lecture notes in being able to hold a student's attention more effectively and improve comprehension, the preparation of high-quality pre-recorded content can involve a large initial time investment, while infrastructure constraints such as availability of devices and network connectivity may also hold, especially in developing countries. This research studies the transition of two course modules offered in the first year of a distance learning program in Sri Lanka from a 100% notes-based delivery style to the incorporation of pre-recorded videos for each of the topics in the modules. Specifically, the study investigates the impact of the inclusion of video content into the course content on learning behavior measured by interactions with the LMS and learning outcomes measured by continuous assessment and written examination marks. The study uses data from 12 cohorts of students and two modules, where the first five cohorts did not have access to any videos, while the remaining cohorts did. The impact of the availability of video content is assessed using a repeated-cohort multiple regression model where changes to the examiner of the module can be controlled for. Results of this study show that marks for non-formative quizzes and summative continuous assessments are positively and significantly associated with the minutes of video content in both modules. However, the impact on end-semester examinations and submission of formative assignments varies across modules and is even negative for one subject. These findings suggest that the incorporation of video content perhaps has a more substantial short-term effect on learning outcomes, helping with immediate retention of content, though the evidence about longer-term effects, as observed in final examinations, is mixed. While further work is needed to incorporate additional course modules and perhaps use detailed topic-wise marks from final examinations, the findings of this research provide a case study for the use of data-driven strategies for content design and development.
Abstract: The key challenges in the delivery of distance learning programs include maintaining student engagement and ensuring that students achieve intended learning outcomes. While video content has the advantage over lecture notes in being able to hold a student's attention more effectively and improve comprehension, the preparation of high-quality prerecorded content can involve a large initial time investment, while infrastructure constraints such as availability of devices and network connectivity may also hold, especially in developing countries. This research studies the transition of two course modules offered in the first year of a distance learning program in Sri Lanka from a 100% notes-based delivery style to the incorporation of pre-recorded videos for each of the topics in the modules. Specifically, the study investigates the impact of the inclusion of video content into the course content on learning behavior measured by interactions with the LMS and learning outcomes measured by continuous assessment and written examination marks. The study uses data from 12 cohorts of students and two modules, where the first five cohorts did not have access to any videos, while the remaining cohorts did. The impact of the availability of video content is assessed using a repeated-cohort multiple regression model where changes to the examiner of the module can be controlled for. Results of this study show that marks for non-formative quizzes and summative continuous assessments are positively and significantly associated with the minutes of video content in both modules. However, the impact on end-semester examinations and submission of formative assignments varies across modules and is even negative for one subject. These findings suggest that the incorporation of video content perhaps has a more substantial short-term effect on learning outcomes, helping with immediate retention of content, though the evidence about longer-term effects, as observed in final examinations, is mixed. While further work is needed to incorporate additional course modules and perhaps use detailed topic-wise marks from final examinations, the findings of this research provide a case study for the use of data-driven strategies for content design and development.
Keywords: Course Design, Video Lectures, Student Interactions, Online Courses, Learning Materials
(ProQuest: ... denotes formulae omited.)
1. Introduction
Online and distance learning programs are now a crucial means of expanding access to higher education and fostering lifelong learning. While such programs allow students the flexibility to access educational materials at their own pace remotely, they come with their own set of challenges. Such challenges include uneven access to technology, low student engagement, limited feedback from both students to teachers and teachers to students, and a lack of motivation among students (Ucar and Saritepeci, 2022; Culduz, 2024). These problems are especially pronounced in situations where there are differences in digital literacy, device availability, and internet reliability (Ndibalema, 2022). However, given that the objective of these programs is to instill skills and knowledge among the learners, the biggest challenge in such settings is ensuring that students are engaged and achieving the desired outcomes (Gaskell and Mill, 2017).
Many strategies are adopted to enhance student engagement in online and distance learning, such as providing feedback on time, creating a sense of community among students, and improving the effectiveness of learning materials through interactivity and gamification, etc. (Chakraborty and Nafukho, 2014; Hew, 2014; Pakpour et al., 2021; Young and Bruce, 2011). The use of video-based learning to enhance interaction and engagement in online settings is a commonly employed strategy to address the challenge of developing effective learning materials, although evidence supporting its effectiveness is mixed. On the one hand, video content has been found to considerably boost student engagement and understanding by making difficult lessons simpler and increasing student interest (Anie and Faustino, 2024; Simi, 2025; Zhu et al. 2022). Several studies focusing on student perceptions about learning materials also note that video materials improve the student experience and improve the grasp of difficult subjects (Albalawi, 2024; MacDonald and Thompson, 2005; Yang, 2017). While increased engagement and interest are known to improve academic performance, there is not as much work directly evaluating the impact of video content on final learning outcomes, as this requires a longer time frame of data to compare outcomes of students with and without access to the video content. Given that considerable costs are involved in developing high-quality video content, as well as creating content that is engaging for learners, evaluations of this nature would be very useful for educational planners and content developers.
The Centre for Open and Distance Learning (CODL) of the University of Moratuwa offers a three-year, fully online Bachelor of Information Technology (BIT) external degree program. The program is delivered through Moodle, a popular learning management system (LMS) for online teaching and learning management. All subjects in the program include continuous assessments (CA), which take the form of quizzes and assignments, and a written examination (WE), conducted physically at the university at the end of the semester. Eligibility for sitting for the WE is based on passing the CA component.
At its inception in 2006/07, the main components of the course materials were text-based content, including lecture slides and PDF notes. However, in 2018, along with a major curriculum revision, video recordings of lectures were incorporated into the learning materials as a way to improve student interaction, understanding, and the quality of lesson delivery. This was rolled out gradually over several years for several courses. At the time of writing, most of the modules offer video lectures alongside the other learning materials. However, no empirical research has been conducted yet to assess its effectiveness in improving student engagement and achieving learning outcomes.
Therefore, in order to determine whether the addition of video lectures has improved student engagement with the course materials, this study evaluates LMS logs and students' grades on continuous assessments and semester-end exams from two modules in the BIT program for which video content was introduced. By using data on 12 cohorts, some of whom were exposed to the video learning materials and some of whom were not, this study's primary goals are to:
* Compare the academic performance of students in a fully video-based, partially video-based, and nonvideo- based module.
* Assess the effect of video lectures on student engagement by examining Moodle LMS logs.
* Determine whether video-based content increases student grades in comparison to traditional textbased materials.
Given the limited empirical evidence on this topic in the Sri Lankan context, this study provides some insights on how e-learning and teaching can be adapted to the local context. Moreover, while this study focuses on selected modules within a particular degree program, the research provides a case study for the use of data-driven strategies for designing curricula and course content for online and distance learning program.
2. Data and Methods
This study investigates the impact of incorporating video lectures in two first-year modules of the University of Moratuwa's Bachelor of Information Technology (BIT) external degree program. Using data from twelve student intakes from 2018 to 2023 (the program enrolls two intakes per year), we examine the potential effects of this design change on academic performance and student engagement with the LMS.
2.1 Selected Courses
This study uses data on two modules: Visual Application Programming (ITE1112) and Mathematics for IT (ITE1812), both of which are offered during the first year of the BIT degree program. While the BIT program's curriculum revision took place in 2018, the shiftfrom text-based content to video-based content occurred from 2020 onwards. Accordingly, we study 12 cohorts (from Intake 2 of 2018 to Intake 2 of 2023), covering cohorts with and without access to these video materials. It should be noted that the introduction of video content was a single intervention - there were no changes in module content or structure of formative or summative assessments at the same time. Accordingly, Moodle course page structures are consistent before and after the introduction of videos, allowing for the comparison of student behavior and results over time.
2.2 Modelling the Introduction of Video Content
Initially, the course materials in both modules were limited to text-based PDFs and PowerPoint decks, with video lectures added starting with Intake 2 of 2020. We calculate the total number of minutes of video content uploaded to the Moodle page for each topic for each intake for each module (this is zero for all cohorts up until Intake 2 of 2020). It should be noted that for ITE 1112 the video content was introduced gradually with videos introduced for half the topics in the course for Intake 2 of 2020 and the remaining half included for Intake 1 of 2021 onwards. For ITE 1812, video content for all topics was introduced in a single semester, Intake 2 of 2020.
We also create a dummy variable to indicate the change in the course instructor. This allows us to control for possible changes in final results caused by changes in lecturing staffwho set the summative assessments.
2.3 Student Interactions and Outcomes
The Moodle logs for every cohort were used to gather information on student interactions with the LMS. Specifically, we do the following: For ITE 1112, we take the number of student submissions for each of 16 ungraded learning activities and the average number of clicks per student on each of the 16 course learning activities of ITE 111 (given weekly for all of the 12 cohorts) to measure engagement. To enable aggregate analysis across students, submissions were transformed into a binary variable: 1 for submitted, 0 for no submission. Given that the grades of these activities were not included in the final grade, we do not have the marks for these activities; hence, the use of submission as the outcome variable. For ITE 1812, the formative assessments were quizzes rather than assignment submissions. Accordingly, for this subject, we use the marks from 19 formative assessment quizzes given over 16 weeks to study students' engagement with the course content and Moodle page.
Both written exam (WE) scores and continuous assessment (CA) marks, which are scaled to percentages, were used to capture academic achievement. These scores were averaged across all students and recorded for each cohort (intake). The reason for this is that while the video content was prepared per topic, the only information we have on topic-wise engagement and learning outcomes are the interactions with formative assessments. So cross-topic variation in video content can only be exploited against formative assessment completion. For the summative assessments (WE and CA) it is the total count of video minutes that matter, which means that the only source of variation there is across the cohorts.
Descriptive statistics for the variables used in the study are provided in Table 1.
2.4 Statistical analysis
To examine the association between the introduction of video content and student performance and engagement, we used a regression-based methodology. Specifically, multiple linear regression models were also estimated in order to assess the impact of video lectures. These models looked at the link between the amount of video minutes and the following outcomes:
* Average clicks per learning activity and average submission rate (as a proportion) for ITE 1112
* Marks for formative assessment quizzes in ITE 1812
* Aggregate CA marks and WE marks for both modules
A dummy variable was incorporated into each regression to account for change in examiners that took place during the time period considered. This enables the separation of the instructor's influence from the impact of the educational format.
The regression models are estimated separately for each module at two different levels: when CA or WE marks are the dependent variable, the regressions are estimated at the student level, and video minutes are fixed for the pre- and post-video cohorts. The specification is as follows:
...
where the grades for student i in intake t are represented by yijt , and the error term is represented by eijt.
When interactions with learning activities (for ITE 1112) or formative assessment marks (for ITE 1812) are used, we estimate the regressions at the activity or topic level with the video minutes varying across both pre- and post-video cohorts and topic. The specification is as follows:
...
where the grades or interaction metric for topic j in intake t is represented by yijt.
3. Results
Regression analyses were performed to assess the effect of video lecture content on students' academic performance and engagement with the Moodle learning management system (LMS) across two core modules: ITE1112 (Visual Application Programming) and ITE1812 (Mathematics for IT). The empirical results are discussed in this section. Quiz marks (out of 10), Written Exam (WE) and Continuous Assessment (CA) scores (out of 100), and LMS engagement metrics (clicks and submissions) are evaluated by the models, to examine the impact of "video minutes" as the main explanatory variable.
3.1 Academic Performance in ITE1112
Figure 1 plots the average CA and WE marks before and after the introduction of video minutes for ITE 1112. As the figure shows, there is a reduction in both CA and WE marks after the introduction of video content. However, this figure does not control for the effect of the change in the lecturer, which could drive differences in overall grades.
Table 1 shows the regression estimates of the impact of video minutes on student outcomes and engagement after controlling for changes in the lecturer. Note that the outcomes regressions (CA and WE marks) are estimated at cohort-student level whereas the engagement with learning activities regressions are estimated at the cohort-learning activity level explaining the difference in sample size across the models.
Contrary to Figure 1, the regression results show a statistically significant positive relationship between continuous assessment grades of ITE1112 and video material duration (β = 0.0124, p < 0.001) after controlling for variations in average marks caused by the change to the examiner. In particular, the provision of a minutelong addition of video material to the Moodle course is related, on average, to an increase of about 0.012 percentage points in continuous assessment marks. The lecturer change effect is also large: the dummy variable for the lecturer had a large negative coefficient (β = -16.48), indicating a significant drop in continuous assessment grades after a change in the course examiner. Similarly, the video minutes variable shows a statistically significant and positive association with WE marks (β = 0.0125, p < 0.001), consistent with the findings based on the CA model. These results support the assumption that instructional videos increase students' understanding of course content, especially in technical courses that require visual interpretation.
Interestingly, however, we do not find a positive effect on the LMS interactions with learning activities or submission of learning activities - instead, the estimated coefficients are significantly negative, suggesting that fewer clicks and fewer submissions are made for subjects with more video minutes. However, given that these learning activity assignments are not graded or counted for the final grade, this may not be particularly concerning as students may be attempting the activity themselves even if they do not do the submission.
3.2 Academic Performance in ITE 1812
Figure 2 summarizes the CA and WE marks for ITE 1812 before and after the introduction of video lectures. Unlike for ITE 1112, we see an increase in the average CA marks after the introduction of video content, even without controlling for the change in examiner. However, the average final examination shows a reduction following the change like in ITE 1112.
Table 3 provides the regression results. Consistent with Figure 2, we see a positive and significant effect of video minutes on CA scores (beta = 0.0099, p < 0.001) again affirming the claim that video-based delivery aids learning outcomes. The value of coefficient is a little smaller in magnitude than in ITE1112 but not significantly so. However, WE marks show a negative association with video minutes (beta = -0.0057, p = 0.008), and is statistically significant, even after accounting for the change in the examiner for the module. This finding appears unexpected, especially given the positive associations observed with both CA marks and formative quizzes (0.0095, p < 0.001), which rule out the possibility of insufficient practice or interaction. However, it is possible that the WE tests different learning outcomes or skills that are not assessed in the continuous or formative assessments provided during the semester.
4. Discussion and Conclusion
This study examines the impact of the introduction of video lectures on two first-year modules offered in an online, distance learning program offered by a Sri Lankan university. Using data from 12 cohorts which cover populations with and without access to video content, we estimate the impact of videos (measured by the duration of video content) on student learning outcomes as well as interactions with the LMS.
Considering the results from both modules together, we see some interesting features. It is possible that the negative impact of video content on LMS interactions and submissions observed for ITE 1112 are due to the assignment nature of the learning activities. When the formative assessment takes the form of a quiz (as in ITE 1812), where there is feedback to the student and the feedback is obtained immediately, it is likely that the students have more motivation to complete them, with the video content positively affecting the learning outcomes. This then reflects in the overall CA marks, suggesting that the learning taking place during the semester is positively influenced by the video minutes. The negative impact on clicks could also be explained by limitations with using conventional LMS log indicators, as also put forward by Macfadyen and Dawson (2010), where clicks and log-ins alone are not really an accurate measure of the real time involvement or understanding of students.
However, when it comes to the final examination, which takes place as a physical, proctored examination, the estimated effect of the video content is mixed. In ITE 1112, the written examination marks also correlate positively, whereas in ITE 1812, the estimated effect is significantly negative. While there could be many explanations for this result, one could be that the final examination and continuous assessments test different learning outcomes. If, for instance, the quizzes and CA test more basic learning outcomes whereas the written examination tests higher order thinking skills, it may be students are preparing insufficiently for these higher level questions. This distinction suggests that video by itself is not a perfect solution, but a supplemental instrument, the usefulness of which is amplified when combined with proper instructional design and assessment strategies.
The analysis in this study could be extended in different directions to improve on the findings. One obvious direction is to increase the number of modules included in the study - this will help with the generalizability of the findings as well as highlight any difference in the impact of videos across disciplines (e.g. mathematics, statistics, programming etc.). Another would be to further drill down into the CA and WE marks - if marks breakdown per question are available, we could map the marks against the topics in the course to obtain a more detailed picture of which topics ultimately benefit from the provision of video content. Accordingly, while this study provides a case study for the data-driven strategies for informing course design and delivery, the need for continuous self-monitoring, evaluation and re-design are key for improving maintaining student engagement and learning, particularly in distance learning programs.
Ethics Declaration
This research was conducted in accordance with ethical guidelines and approved by the University of Moratuwa Ethics Review Committee, with approval number EDN/2023/008. All data utilized in this research were deidentified before they were analyzed and kept within secure systems. Individual student identifiers like names, registration numbers, or contact information were not accessed or utilized at any stage of the research.
AI Declaration
AI software was not used in the design of the research, preparation of data analysis, or writing the manuscript. All scholarly findings, interpretations, and conclusions contained in this paper are the property of the authors.
References
Albalawi, A., 2024. The Impact of Adopting OER Videos as Supplementary Resources on Student Scores and Engagement (Doctoral dissertation, The Claremont Graduate University).
Chakraborty, M. and Muyia Nafukho, F., 2014. Strengthening student engagement: what do students want in online courses?. European Journal of Training and Development, 38(9), pp.782-802. https://doi.org/10.1108/EJTD-11-2013- 0123
Culduz, M., 2024. Benefits and challenges of e-learning, online education, and distance learning. In Incorporating the human element in online teaching and learning (pp. 1-27). IGI Global. DOI: 10.4018/979-8-3693-4131-5.ch001
Faustino, M.A.L., 2024. Effectiveness of Video Lessons in Enhancing Student Engagement and Comprehension During the COVID-19 Pandemic. International Journal of Research and Innovation in Social Science, 8(11), pp.1910-1913. https://doi.org/https://dx.doi.org/10.47772/IJRISS.2024.8110149.
Gaskell, A. and Mills, R. (2014) 'The quality and reputation of open, distance and e-learning: what are the challenges?', Open Learning: The Journal of Open, Distance and e-Learning, 29(3), pp. 190-205. doi: 10.1080/02680513.2014.993603.
Hew, K.F., 2016. Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), pp.320-341. https://doi.org/10.1111/bjet.12235
MacDonald, C.J. and Thompson, T.L., 2005. Structure, content, delivery, service, and outcomes: Quality e-learning in higher education. International Review of Research in Open and Distributed Learning, 6(2), pp.1-25. https://doi.org/10.19173/irrodl.v6i2.237.
Ndibalema, P., 2022. Constraints of transition to online distance learning in Higher Education Institutions during COVID-19 in developing countries: A systematic review. E-learning and Digital Media, 19(6), pp.595-618, https://doi.org/10.1177/20427530221107510
Simi (2025) The Impact of Video in eLearning: How Video Content is Transforming Online Education, Cincopa.com. https://doi.org/10.5281/zenodo.8341475
Uçar, A. and Sarıtepeci, M., 2022. How can student engagement be improved in Massive Open Online Courses?. Instructional Technology and Lifelong Learning, 3(2), pp.176-206. https://doi.org/10.52911/itall.1194260
Yang, D., 2017. Instructional strategies and course design for teaching statistics online: perspectives from online students. International Journal of STEM education, 4, pp.1-15. https://doi.org/10.1186/s40594-017-0096-x.
Young, S. and Bruce, M.A., 2011. Classroom community and student engagement in online courses. Journal of Online Learning and Teaching, 7(2), pp.219-230.
Zhu, J., Yuan, H., Zhang, Q., Huang, P.H., Wang, Y., Duan, S., Lei, M., Lim, E.G. and Song, P., 2022. The impact of short videos on student performance in an online-flipped college engineering course. Humanities and Social Sciences Communications, 9(1), pp.1-10. https://doi.org/10.1057/s41599-022-01355-6
Copyright Academic Conferences International Limited 2025