Learning analytics in higher education: an analysis of case studies

Abstract

Purpose

The purpose of this paper is to present a systematic review of the mounting research work on learning analytics.

Design/methodology/approach

This study collects and summarizes information on the use of learning analytics. It identifies how learning analytics has been used in the higher education sector, and the expected benefits for higher education institutions. Empirical research and case studies on learning analytics were collected, and the details of the studies were categorized, including their objectives, approaches, and major outcomes.

Findings

The results show the benefits of learning analytics, which help institutions to utilize available data effectively in decision making. Learning analytics can facilitate evaluation of the effectiveness of pedagogies and instructional designs for improvement, and help to monitor closely students’ learning and persistence, predict students’ performance, detect undesirable learning behaviours and emotional states, and identify students at risk, for taking prompt follow-up action and providing proper assistance to students. It can also provide students with insightful data about their learning characteristics and patterns, which can make their learning experiences more personal and engaging, and promote their reflection and improvement.

Originality/value

Despite being increasingly adopted in higher education, the existing literature on learning analytics has focussed mainly on conventional face-to-face institutions, and has yet to adequately address the context of open and distance education. The findings of this study enable educational organizations and academics, especially those in open and distance institutions, to keep abreast of this emerging field and have a foundation for further exploration of this area.

Introduction

Learning analytics (LA) refers to the process of collecting, evaluating, analysing, and reporting organizational data for decision making (Campbell and Oblinger, 2007). It involves the use of big data analysis for understanding and improving the performance of educational institutions in educational delivery. Open and distance learning (ODL) institutions present an ideal context for the use of LA as, with their large student numbers and the increasing use of the internet and mobile technologies, they already have a very substantial amount of data available for analysis with analytics.

Despite LA being increasingly applied in a wide range of educational organizations, the literature in this area has usually focussed on conventional face-to-face institutions. In the ODL setting, there is yet to be a systematic review summarizing existing work on the potential benefits of LA to open and distance institutions (Firat and Yuzer, 2016; Prinsloo and Slade, 2014), and relevant research findings potentially applicable to these institutions (Rienties et al., 2016).

This paper gives a systematic review of the mounting research work on LA that has been published in recent years to provide an overview of this emerging field and serves as a foundation for further exploration. It addresses the potential problems of ODL institutions that could be solved by using LA, and the benefits that could be obtained according to the existing case studies. It also presents a meta-analysis of relevant empirical studies which shows the effect of intervention for at-risk students based on the use of LA.

Related studies

LA involves the use of a broad range of data and techniques for analysis – covering, for example, statistical tests, explanatory and predictive models, and data visualization (Arroway et al., 2016). Various stakeholders, such as administrators, teaching staff, and students, can then act on the data-driven analysis. Without a standardized methodology, LA has been implemented using diverse approaches for various objectives. Gašević et al. (2016) summarized three major themes in LA implementation, namely, the development of predicators and indicators for various factors (e.g. academic performance, student engagement, and self-regulated learning skills); the use of visualizations to explore and interpret data and to prompt remedial actions; and the derivation of interventions to shape the learning environment. The diversity in LA implementation poses a challenge for education institutions which plan to be involved in it, leading to a commonly voiced question – “How do we start the process for the adoption of institutional learning analytics?” (Gašević et al., 2016, p. 4).

As an emerging field of study, an increasing number of case studies relevant to the implementation of LA in higher education have been published. However, only a small number of reviews summarize these individual case studies. Among them, Dyckhoff (2011) reviewed the research questions and methods of these studies. The findings showed that existing studies have focussed on six types of research questions: qualitative evaluation; quantitative measures of use and attendance; differentiation between groups of students; differentiation between learning offerings; data consolidation; and effectiveness. The research methods used include online surveys, log files, observations, group interviews, students’ class attendance, eye tracking, and the analysis of examination grades. Based on the results, suggestions were given on LA indicators for improving teaching.

Papamitsiou and Economides (2014) focussed on the impacts of LA and educational data mining on adaptive learning. They reviewed the experimental case studies between 2008 and 2013, and identified four distinct categories, namely, pedagogy-oriented issues, contextualization of learning, networked learning, and the handling of educational resources.

Also, Nunn et al. (2016) discussed LA’s methods, benefits, and challenges. It was found that the methods used included visual data analysis, social network analysis, semantic analysis, and educational data mining. The benefits of LA were seen to revolve around targeted course offerings; curriculum development; student learning outcomes; behaviours and processes; personalized learning; improvements in instructor performance; post-educational employment opportunities; and enhancement of educational research. The challenges included the tracking, collection, evaluation and analysis of data, as well as a lack of connection to learning science, the need for learning environment optimization, and issues concerning ethics and privacy.

Focussing on computer science courses, Ihantola et al. (2015) surveyed LA case studies in terms of their goals, approaches, contexts, subjects, tasks, data and collection, and methods of analysis. The goals were related to students, programming, and the learning environment. The approaches included case studies, constructive research, experimental studies, and survey research. They also found that most of the research work was undertaken in a course context, with the number of subjects ranging from 10 to 265,000, with 64 per cent of the studies having 500 or fewer subjects. In most of the studies, students were required to complete multiple programming tasks. Over 60 per cent of the studies used automated data collection that logged students’ actions, and a variety of data analysis methods such as descriptive and inferential statistics.

The existing reviews of LA case studies provide a basic descriptive summary. However, as a new area in education, there remain many uncertainties for ODL institutions about involving themselves in it. To make an informed decision on whether or not to implement LA, a key question is: “What are the expected benefits for the institution?” This paper addresses this issue by surveying the outcomes of LA implementation for institutions.

Methodology

This study aims to investigate how LA has been used in higher education institutions and the outcomes obtained. Relevant case studies were collected from Scopus, using the key terms “academic analytics” and “learning analytics” for the period from 2007 to 2016. The studies were selected based on the following criteria:

1. the study reported one or more empirical cases of the use of LA in a higher education institution;

2. the institution in question was accredited by the government or government-related bodies;

3. the institution had 1,000 or more students; and

4. the source information contained the aims of using LA, a description of the analytics, its implementation and the outcomes.

An initial search returned 1,492 results. After screening, a total of 43 cases which fulfilled the criteria for inclusion were selected for further analysis. They were analysed in terms of their objectives, approaches, and major outcomes.

A meta-analysis was also conducted to synthesize the empirical findings reported in the case studies. Studies which included relevant quantitative data analysis were chosen, resulting in six studies on student support and analysis of learning behaviours, with the effect of LA intervention validated and reported.

Results

Benefits for institutions, staff, and students

A summary of the objectives and approaches of the use of LA in the institutions chosen is presented in Table AI. The benefits of LA for the institutions, staff and students revolve around the following aspects.

Improving student retention

Table I presents the use of LA which improved student retention. By closely monitoring students’ learning and persistence, undesirable learning behaviours and emotional states can be detected, and students who are at risk can be identified early. Factors leading to student dropout or retention can be identified and prediction models developed. Staff can take prompt follow-up action and provide proper assistance to students who need extra support, such as counselling, suggesting learning resources, and formulating individual learning plans. Students’ level of achievement, as well as their retention, can be enhanced.

Supporting informed decision making

Table II shows the use of LA which supported informed decision making. Institutions are provided with information and analyses generated from a massive amount of data for informed decision making. For example, planning can be carried out on course development and resources allocation on the basis of information about the popularity of courses, and types and frequency of materials reviewed by students.

Increasing cost-effectiveness

Table III presents cases of LA use which increased cost-effectiveness. LA can be integrated with other platforms such as the learning management system. Instructors can then access various kinds of information online for providing feedback and support to students. Analyses and feedback on students’ study progress can be delivered to staff, students, or parents in an automatic and cost-effective manner.

Understanding students’ learning behaviours

Table IV presents the use of LA for understanding students’ learning behaviours. By analysing diverse sources of data (e.g. learning management systems and social networks), institutions and academic staff can understand the relationships among students’ utilization of resources, learning behaviours and characteristics, and learning outcomes, which helps them to evaluate the effectiveness of pedagogies and instructional designs for improvement. For instance, the use of LA helps to capture the students’ behaviours in watching course videos by highlighting the patterns of their preferences and behaviours as well as showing the parts of videos which were watched most and least frequently. Curriculum and learning materials can thus be better designed to address students’ preferences and needs.

Providing personalized assistance for students

Table V illustrates the use of LA for providing students with insightful data about their learning characteristics and patterns, which can make their learning experiences more personal and engaging, and facilitate their reflections and improvements while a course is still in progress. Early alerts can be automatically generated and sent to students if their academic performance is below a certain standard. Students can also be encouraged to engage more in the personalized learning activities which are conducive to success in their studies.

Timely feedback and intervention

Table VI presents the use of LA for timely feedback and intervention. Instructors can obtain up-to-date and holistic information about students’ study progress, so that timely feedback can be given and individualized interventions made. Students develop a sense of belonging to the learner community through personalized feedback given to them. For example, the use of social network analytics allows instructors to understand the development of the learner community and identify students who are performing poorly or are isolated from the main discussion, and then provide intervention during discussion in real time. This is especially important for ODL institutions, where students may be using different study modes and social media is a major communication channel.

Meta-analysis of the effect of interventions on student success

An important function of LA is to predict at-risk students and deliver early alerts and interventions to them, in order to improve their academic attainment, and their retention and graduation rate. This section provides a meta-analysis of the various prediction models utilized in LA systems, and the effect of the intervention solutions on enhancing students’ success.

Among the case studies examined, only six which provided quantitative analysis results were selected and the results are synthesized in this section. The effect sizes for each analysis were calculated where the data required for the calculation were available, and a descriptive comparison of the effect sizes across the studies was made. Table VII presents a summary of the predictive models and intervention solutions employed in the six case studies; and Table VIII summarizes the results of quantitative analyses for the intervention solutions and the effect sizes for each study.

To summarize, a common approach utilized in the cases of intervention for student success was to collect and analyse data from students’ learning activities and employ a specific computational model to predict and prioritize those students who were at-risk of dropping out or getting poor academic results. Based on the findings of the predictive modelling, subsequent measures can be taken for intervention. A common practice was to get academic staff to contact the at-risk students and provide personalized learning support to them. Such an approach to prediction and intervention was found to effectively enhance students’ success, as measured by various indicators such as GPA, study progress, the retention rate, and the graduation rate.

According to the meta-analysis of the quantitative results, all the institutions found improvement in the students’ success in the intervention group compared to the control group, although the effect size varied across different types of indicators for success and different institutions. For instance, the intervention groups in the case of Marist College showed a 6 per cent improvement in the students’ final grades compared to the non-intervention control groups (Sclater et al., 2016), while the effect size was in the range of small to medium based on Cohen’s (1988) convention. For the retention rate examined in Mattingly et al. (2012) for the Course Signal System of Purdue University, the intervention groups showed a nearly 50 per cent performance improvement compared to the control groups. In spite of the small sample size, the meta-analysis showed an encouraging result for the benefits of LA in aiding institutions to make effective informed decisions to improve students’ learning performance and success.

Discussion and conclusion

This study shows that positive outcomes have been widely reported in relevant case studies. The results suggest great potential for ODL institutions to utilize LA for analysing existing data, which is expected to benefit their operations in areas such as quality assurance and student support. This study also reviewed various predictive models for student success which were developed and validated to identify and prioritize students who may be in need of support. The quantitative analyses confirmed that the learning performance of these students improved after they had been approached for LA-based interventions. The findings of this study thus provide various stakeholders – institutions, staff, and students – with the benefits they may gain from LA.

In particular, the results related to student learning suggest that, to change students’ behaviours, it may suffice to simply make them aware of their learning engagement through LA tools in relation to other students or indicate that they are at risk (Jayaprakash et al., 2014; Sclater and Mullan, 2017). Complex data visualizations or dashboards may not be necessary. What is more important, as recommended in Gašević et al. (2016), is to help students to interpret correctly the information from visualizations or dashboards.

The meta-analysis revealed that only a few case studies related to LA implementation provided quantitative analyses data – a limitation which may be caused by the relatively new development of LA. Therefore, empirical investigations and validation of many new models and new theories in this area remain to be carried out. While an increase in the quantity of empirical and quantitative research can be expected in future, it is also important to develop and test innovative solutions supported by LA. Present LA-based interventions, as reviewed in this paper, were mostly based on the interaction and discussion between students and instructors. Although such interventions were shown to be effective in general, their effectiveness may vary among different groups of students in different contexts.

A challenge in measuring the effectiveness of LA implementation lies in the difficulty of identifying the extent to which any change after the LA implementation is attributed to the LA itself. As discussed in Sclater and Mullan (2017), it may not be feasible to isolate the influence of LA when it is part of a wider initiative to develop data-informed approaches in an institution. The case studies published and reviewed in this paper would thus be biased to the institutions which only deployed LA without other measures in their data-informed approaches.

In the ODL context, work on LA remains at an initial stage. Features of ODL, such as open admission which allows a broad range of students to study the same course with very limited face-to-face interaction, are yet to be studied in relation to LA implementation. It is therefore suggested that future research can involve more fine-grained validation studies to identify the effect of the various factors involved the implementation of LA. In particular, investigation on those factors related to ODL institutions, staff and students, as well as the plausible constraints on their use of LA, would shed light on how they can benefit more from involvement in LA.

The work described in this paper was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (UGC/IDS16/15).

Table I

Use of LA which improved student retention

Institution	Major outcomes	Source
Bowie State University	More student activities and communication were initiated through the system	Chacon et al. (2012)
Edith Cowan University	The student retention rate for those who got support was higher than the university’s average rate	Atif et al. (2013)
Harvard University	The results demonstrate the potential for natural language processing to contribute to predicting student success in MOOCs and other forms of open online learning	Robinson et al. (2016)
New York Institute of Technology	An at-risk model of high predictive power was developed	Sclater et al. (2016)
Northern Arizona University	Student-instructor interaction was increased and personal interventions were given; and students showed better academic performance, retention and graduation rates	Star and Collette (2010)
Paul Smith’s College	Students devoting more efforts in their studies resulted in a higher chance of success, and better persistence and graduation rates	McAleese and Taylor (2012)
Rio Salado Community College	A 40% decrease in drop-out rate was obtained for students who received welcome e-mails compared with those who did not	Smith et al. (2012)
The Open University (UK)	A vast majority of students showed continuous engagementStudent retention was at an average to good levelStudents demonstrated higher satisfaction	Rienties et al. (2016)
University of New England	The student attrition dropped from 18 to 12%Students demonstrated an increase in their sense of belonging to the learner community and learning motivation	Sclater et al. (2016)

Table II

Use of LA which supported informed decision making

Institution	Major outcomes	Source
Grand Rapids College	Better decisions can be made about course delivery to help to ensure student success through a LA tool which is easy for end user analysis	Fritz and Kunnen (2010)
The Open University (UK)	Elements tacitly implicated in pedagogical decisions during course design were unpicked	Toetenel and Rienties (2016)
University of Adelaide	Educators were provided with guidelines to design collaborative learning activities	Tarmazdi et al. (2015)
University of Edinburgh	Through identification of socially engaged students, the instructional team can identify suitable teaching assistants	Kovanović et al. (2016)
University of North Bengal	Counsellors and faculty members were provided with useful inputs to advise learners on the best possible completion options	Yasmine (2013)
University of Salamanca	Visual analytics was shown to help to lead to better understanding of what is happening in a student. Informed decisions can be made that help students to succeed	Conde et al. (2015)
The Technical University of Madrid	Information was provided by the LA system which helped to prevent problems, carry out corrective measures and make informed decisions to improve students’ learning	Fidalgo-Blanco et al. (2015)

Table III

Use of LA which increased cost-effectiveness

Institution	Major outcomes	Source
Bridgewater College	Notifications were automatically generated and sent to students and their parents to recognize students’ good performance	Sclater et al. (2016)
Drexel University	Faculty, programme developers, and programme administrators were able to analyse the connections between a specific programme outcome and data related to that outcome	Harvey (2013)
Georgia Institute of Technology and Carnegie Mellon University	High reliability was achieved for analysing students’ online discussion data	Wang et al. (2016)
Harvard University	A machine learning prediction model was shown to be effective for predicting students who would complete an online course	Robinson et al. (2016)
Lancaster University	Tutors could efficiently access various kinds of data for providing students with timely support	Sclater et al. (2016)
New York Institute of Technology	A dashboard simple and easy to use by staff was developed	Sclater et al. (2016)
Open University of Catalonia	Information could be updated and maintained automatically	Guitart et al. (2015)
Portland State University	Operation efficiency was increased, e.g. faster generation of reportsThe system could easily be modified to fit the needs of other institutions	Blanton (2012)
Purdue University	Students who had engaged with the LA system sought more help and resources than other students	Arnold and Pistilli (2012)
Rio Salado College	The likelihood of successful course completion was accurately assessed	Smith et al. (2012)
The Hong Kong Institute of Education	There was greater interaction between teachers and students	Wong and Li (2016)
University of Adelaide	Lecturers were allowed to assess and monitor students’ collaboration in an online environment, without having to traverse a large discussion forum	Tarmazdi et al. (2015)
University of Michigan	The system demonstrated high scalability and extensibility	Mattingly et al. (2012)
University of Salamanca	The system allowed the provision of learning support to students in an automatic manner	Cruz-Benito et al. (2014)
University of the South Pacific	The utilization of open source resources could be modified and adapted by anyone to meet specific user needs	Prasad et al. (2016)
University of Sydney	LA features such as instant feedback and auto-grading are especially useful for instructors teaching subjects in computer science education	Gramoli et al. (2016)

Table IV

Use of LA which helped in understanding students’ learning behaviours

Institution	Major outcomes	Source
Ball State University	Data analyses showed the consistent predictive power of the LA system on students’ academic performance, persistence, retention and graduation	Jones and Woosley (2011)
Georgia Institute of Technology and Carnegie Mellon University	Students who displayed more higher-order thinking behaviours learnt more through deeper engagement with course materials displayed by their discussion behavioursThese students in turn also learnt more than students who were constantly off topic in the forumsSocial-oriented topics triggered richer discussion compared with biopsychology oriented topics, and higher-order thinking behaviours tended to appear together within threads in the forums	Wang et al. (2016)
McGill University	It provides an unprecedented opportunity to use data from real learners in authentic learning situations to better understand learning processesThe study demonstrated how to detect learner misconceptionsPrediction precision and weighted relative accuracy were significantly increased	Poitras et al. (2016)
Oxford Brookes University	Problems were identified with ethnic minority students in particular courses	Sclater et al. (2016)
The Hong Kong Institute of Education	Potential indicators were found for predicting student performance, such as the contribution of in-depth contents in online discussion	Wong and Li (2016)
The Open University (UK)	Common pedagogical patterns were identified from learning designs, showing the relationship between learning activities and students’ learning outcomes	Toetenel and Rienties (2016)
The Technical University of Madrid	Relationship between student interaction and individual performance was identified	Fidalgo-Blanco et al. (2015)
The University of Melbourne	Relationships among students’ motivation, participation and performance in MOOCs were found	Barba et al. (2016)
The University of Melbourne	Learners’ learning progress could be visualized showing their development from novice to expert	Milligan (2015)
University of Adelaide	Lecturers could track the evolution of team roles across each study group and identify various sentiments within each group	Tarmazdi et al. (2015)
University of Edinburgh	Patterns of students’ engagement in MOOC learning activities were found, showing differences in their learning behaviours between enrolments in the same courses	Kovanović et al. (2016)
University of North Bengal	Factors leading to students’ dropout were identified, such as pregnancy and the remoteness of residence locations	Yasmine (2013)
University of Rijeka	Student activities on the learning management system (e.g. assignment uploads and course views) were shown as predictors of academic success	Sisovic et al. (2015)
University of Santiago de Compostela	Teachers could understand more clearly how students behave during a course that facilitated the evaluation process	Gewerc et al. (2014)

Table V

Use of LA for providing personalized assistance to students

Institution	Major outcomes	Source
Albany Technical College	Based on analysis of students’ study results, demographics and social data, at-risk students were identified for providing individual counselling	Karkhanis and Dumbre (2015)
Bridgewater College	Tutors were provided with detailed information to discuss with students on their progress against targets and suggested actions	Sclater et al. (2016)
Open Universities Australia	Students obtained from the system recommended content and activities and a personalized learning environment	Atif et al. (2013)
The Technical University of Madrid	The LA system provided information for preventing problems, carrying out corrective measures and improving students’ learning	Fidalgo-Blanco et al. (2015)
University of Michigan	Customized recommendations were provided, including suggestions on study habits, assignment practice, feedback on progress and encouragement	Mattingly et al. (2012)

Table VI

Use of LA for timely feedback and intervention

Institution	Major outcomes	Source
Edith Cowan University	Students likely to need support were automatically identified and support staff could efficiently reach them for interventions	Sclater et al. (2016)
Marist College	Interventions resulted in a 6% improvement in final grades for the treatment group compared to the control group	Jayaprakash et al. (2014)
Northern Arizona University	Instructors’ feedback was available to individual students and to university personnel, facilitating a comprehensive support network for all students	Star and Collette (2010)
Purdue University	Interventions were provided to at-risk students, and a higher student retention rate was achieved	Arnold and Pistilli (2012)
San Diego State University	Interventions through e-mails were shown to be the best treatment within constraints, while having an impact on student achievement	Dodge et al. (2015)
University of Adelaide	The LA system allowed instructors to be aware when particular students are behaving differently from the others for making appropriate and timely interventions	Tarmazdi et al. (2015)
University of Edinburgh	Instant feedback was shown to be a useful LA feature for students in courses on computer programming	Kovanović et al. (2016)
University of Michigan	Students were provided with feedback (e.g. grade prediction) for self-reflection	Mattingly et al. (2012)
University of Wollongong	Students who are isolated from the main discussion could be identified, and interventions could be provided during discussion in real time	Mat et al. (2013)

Table VII

Summary of predictive model and intervention solution for selected case studies

Institution	Learning analytics system (s)	Predictive model	Intervention solution
Georgia Institute of Technology and Carnegie Mellon University (Wang et al., 2016)	Interactive-Constructive-Active-Passive (ICAP) framework	It was predicted that engaging in higher-order thinking behaviours results in better learning outcomes than paying general or focussed attention to course materials	Students’ online discussion behaviours were categorized into three types: Higher-order – the student has contributed at least one constructive or interactive post during a course Paying-attention – the student has contributed at least one active post during the course but has not displayed any constructive or interactive posts No contribution to any on-topic discussion during the courseTogether with the students’ other persistent characteristics, treatment and control groups were formed to investigate differences in their learning outcomes
Hong Kong Institute of Education (Wong and Li, 2016)	KeyGraph algorithm and Polaris (a software tool)	A test-mining analytical tool was used to predict students’ academic performance. The tool visualizes the hidden patterns and linkages among students’ learning activities. The findings of the study showed that this approach can provide insights into predicting students’ performance, and students with a higher grade tended to contribute more in-depth contents in an online learning environment	Students’ posts in an online learning forum were extracted and analysed – how the students presented concepts, specifically whether they can make linkage among various concepts. Such a pattern was correlated with the grades they obtained. The findings can be used to guide interventions on students’ learning process, and inform ways to give feedback to improve teaching and learning
Marist College (Jayaprakash et al., 2014)	Open Academic Analytics Initiative	A machine learning algorithm and logistic regression were used to predict whether students are at risk based on their demographic details, aptitude data, and various aspects of their usage of the virtual learning environment obtained from the LA system	An online academic support environment was developed containing study skills materials and community support for specialists and student mentors. At-risk students identified by the predictive model were directed to the support environment
Nottingham Trent University (Sclater et al., 2016)	NTU Student Dashboard	Students’ engagement was assessed using indicators, such as door swipes into academic buildings, visits to the virtual learning environment, the submission of assignments, and the frequency of borrowing library resources. Each student received one of five engagement ratings: high, good, partial, low and not fully enroled	Tutors are prompted to contact students to give assistance when the students’ engagement drops off. Students can view their own engagement scores on the dashboard so that they will be self-motivated
Paul Smith’s College (McAleese and Taylor, 2012)	Rapid Insight’s Veera, Starfish EARLY ALERT, and CONNECT	Rapid Insight’s Veera combines different file types and uses automatic analyses and predictive modelling to identify at-risk students prior to their enrolment. Starfish EARLY ALERT automates data collection and uses analytics to increase the identification of at-risk students	The Starfish EARLY ALERT and CONNECT automatically prioritize students who are identified as at-risk and facilitate intervention and outreach
Purdue University (Arnold and Pistilli, 2012)	Course Signal System	The Course Signal System predicted students’ performance relying on a series of variables, including students’ demographic characteristics, academic performance, past academic history, and students’ efforts devoted to study	Instructors provided real-time personalized feedback to each student based on the outcomes generated from LA, in which the student is informed about how he/she is doing

Table VIII

Summary of quantitative analysis results for selected case studies

Institution	Independent variable	Dependent variable	Statistical method	Description of result	Effect size type	Effect size [95% CI]	Interpretation of effect size
Georgia Institute of Technology and Carnegie Mellon University	Higher-order thinking behaviours	Test score	Regression	The average posttest score of the treatment group (with higher-order thinking behaviour) was significantly higher than that of the control group (without higher-order thinking behaviour)	Hedge’s g	0.237 [0.018, 0.492]	Small-to-medium effect size
Hong Kong Institute of Education	“Contribution” and “innovation” from students’ postings in discussion forum	Final grade	χ2 test of independence	Students who obtained better grades usually contributed more in-depth contents in their posts which linked to other concepts compared to those with lower grades who tended to provide isolated facts with little or no connection or transition from one concept to another	Odds ratio (OR)	0.634 [0.504, 0.798]	The students who contributed more in-depth contents were 63.4% more likely to get a higher grade than those contributing isolated facts
Marist College	Intervention	Final grade	One-way ANOVA	Groups receiving intervention obtained significantly higher final grade than groups receiving no intervention	Hedge’s g	0.373 [0.176, 0.571]	Small-to-medium effect size
Nottingham Trent University	Level of engagement rating	Progression status	Descriptive categorical data analysisa	A much larger proportion of students with satisfactory to high engagement ratings obtained progression status than those with low engagement ratings	–	–	–
Paul Smith’s College	Intervention	Grade, suspension or probation rate, graduation rate	Descriptive categorical data analysisa	Student groups receiving intervention were less likely to get a grade D or below, to end a semester with probation or suspension, and more likely to get good standing by GPA and to graduate on time	–	–	–
Purdue University	Intervention	Retention rate	χ2 test of independence	Student groups receiving intervention had a higher retention rate than those receiving no intervention	Odds ratio (OR)	0.455b [0.427, 0.485]	The intervention group was 45.5% less likely to dropout than the non-intervention group

Notes: ^aThe results presented in the case studies of these two institutions did not involve any statistical tests and complete information for the data – that is, sample size for each category was not provided. Therefore, no effect size could be calculated from the available data; ^bthe effect size was computed by combining the data for the second-year retention rate for three cohorts (2007, 2008, 2009) from the original tables in Mattingly et al. (2012)

Table AI

Summary of the objectives and approaches of higher education institutions in the use of learning analytics

Institution	Approaches	Objectives	Source
1. Albany Technical College	Monitoring, intervention	Identify at-risk students and provide them with counselling	Karkhanis and Dumbre (2015)
2. Ball State University	Monitoring, intervention	Identify at-risk students and provide them with counsellingIncrease effectiveness by reducing the time required to diagnose problems and targeting specific issuesHelp the institution to make informed decisions about student success programmes and retention servicesAllow students to become aware of the gaps between their behaviours and expected outcomes, to understand elements of their academic success, and to utilize on-campus resources to solve their problems	Jones and Woosley (2011)
3. Bowie State University	Monitoring, intervention	Support student retentionTrack students’ progress towards graduation to facilitate decision makingProvide early alerts for staff to intervene to prevent dropout	Chacon et al. (2012)
4. Bridgewater College	Monitoring, intervention	Track students’ attainment levelSupport students to do better than the national average	Sclater et al. (2016)
5. California State University	Monitoring	Analyse how students use the learning management system	Allen et al. (2012)
6. Drexel University	Updating data and curriculum	Measure the effectiveness of specific course components through maintaining data records aligned with the curriculum, courses and syllabi, course learning objectives and assessment strategiesManage student learning outcomes and performance criteria	Harvey (2013)
7. Edith Cowan University	Monitoring, intervention	Identify students who need supportEstablish a system to contact a large number of students and manage interventionsImprove student retentionImprove graduation rates	Sclater et al. (2016)
8. Georgia Institute of Technology and Carnegie Mellon University	Monitoring, analysis	Better scaffolded online discussion to improve learning in a MOOC contextExplore effects of higher-order thinking behaviours in learningIdentify kinds of discussion behaviours associated with learningInvestigate types of learning materials which trigger richer discussion	Wang et al. (2016)
9. Harvard University	Monitoring, prediction	Analyse the extent to which students’ responses about motivation and utility value can predict persistence and completion of study	Robinson et al. (2016)
10. Lancaster University	Monitoring, intervention, feedback	Allow tutors to access the transcripts of their studentsAllow early interventionEnsure student work is graded and feedback given to students in a timely manner	Sclater et al. (2016)
11. Loughborough University	Feedback	Provide academics with a better and more holistic picture of student engagementProvide staff with actionable insights into student learning experienceProvide students with their own educational data in a meaningful way	Sclater et al. (2016)
12. Manchester Metropolitan University	Monitoring, curriculum design	Improve student experience as reflected in the National Student SurveyProvide data for improving the undergraduate curriculum	Sclater et al. (2016)
13. Marist College	Prediction, intervention	Predict academic successProvide interventions	Jayaprakash et al. (2014)
14. McGill University	Monitoring, analysis	Identify misconceptions of medical students as reflected in their interactions in the online learning environment	Poitras et al. (2016)
15. New York Institute of Technology	Prediction, intervention	Create an at-risk model to identify students in need of supportImprove student retention in their first year of studyProvide information that could support counsellor in their work	Sclater et al. (2016)
16. Northern Arizona University	Feedback	Facilitate online interaction between students and instructorsAllow students to receive direct feedback on issues such as academic concerns and grades	Star and Collette (2010)
17. Nottingham Trent University	Intervention	Enhance retention and improve attainmentIncrease students’ sense of belonging within the course community, particularly with tutors	Sclater et al. (2016)
18. Open Universities Australia	Intervention	Identify at-risk studentsSuggest alternative modules to students which are more appropriate for their needs	Atif et al. (2013)
19. Open University of Catalonia	Information collection and management	Identify automatically pieces of knowledge taught in each subjectGather students’ informationKeep information updated	Guitart et al. (2015)
20. Oxford Brookes University	Monitoring	Improve student experienceSupport progress evaluation of modules and programmes, and the identification of priorities at an institutional level	Sclater et al. (2016)
21. Paul Smith’s College	Monitoring, intervention	Identify at-risk students and prioritize outreach for themProvide more efficient and effective interventions for student success	McAleese and Taylor (2012)
22. Portland State University	Information management	Make information more accessible and easier to use	Blanton (2012)
23. Purdue University	Monitoring, intervention	Give students early and frequent performance notificationsHelp faculty members to steer students towards additional campus resources as needed	Arnold and Pistilli (2012)
24. Rio Salado College	Prediction	Identify factors having a significant statistical correlations with final course outcomes	Grush (2011)
25. San Diego State University	Intervention	Identify methods and interventions that would alleviate students’ failureDiscover approaches that could be applied with minimal support and are scalable to a large number of courses	Dodge et al. (2015)
26. The Hong Kong Institute of Education	Monitoring, feedback	Provide insights into predicting students’ performanceDevelop measures to assess students’ online learningBoost teachers’ and students’ interactionAllow students to realize their knowledge discoveryFacilitate teachers to assess students’ performance	Wong and Li (2016)
27. The Open University (UK)	Monitoring, intervention, personalization	Identify learners at risk and needing supportImprove learning designDeliver personalized intervention for studentsAchieve cost-effectiveness	Rienties et al. (2016)
	Identifying patterns	Identify common patterns in course designFind out pedagogical implications for various patterns and learning designs	Toetenel and Rienties (2016)
28. The Technical University of Madrid	Monitoring, evaluation	Support teachers’ monitoring and evaluation of individual students’ progress within a team	Fidalgo-Blanco et al. (2015)
29. The University of Adelaide	Monitoring, feedback	Analyse students’ online discussion data, such as team mood, role distribution and emotional climateDevelop students’ soft skills necessary for collaborative work	Tarmazdi et al. (2015)
30. The University of East London	Monitoring, feedback	Monitor student attendance and learning activitiesCollect student data, such as demographic information, library activities, coursework, and download of free booksSend automated e-mails to students showing their attendance, and warnings to students without satisfactory attendance	Sclater et al. (2016)
31. The University of Melbourne	Monitoring, analysis	Investigate how motivation and participation influence students’ performance in a MOOC	Barba et al. (2016)
		Analyse how MOOC participants use online forums to support learning	Milligan (2015)
		Investigate how students interpret feedback delivered via learning analytics dashboard and the relevant influence on their learning strategies and motivation	Corrin and Barba (2015)
32. Universidad a Distancia de Madrid	Monitoring, analysis	Find predictors of teamwork and commitment as cross-curricular competences	Iglesias-Pradas et al. (2015)
33. University of Edinburgh	Analysis, prediction	Examine MOOC data about students who enroled in the same course at least twiceIdentify changes in their behaviours between the two enrolments to the same course	Kovanović et al. (2016)
34. University of Maryland, Baltimore County	Monitoring, feedback, reflection	Reduce student barriersCreate a community of learnersImprove students’ self-awareness by providing feedbackProvide early alerts to students if their GPA falls below a level	Mattingly et al. (2012)
35. University of Michigan	Monitoring, personalization, reflection	Identify at-risk studentsProvide personalized feedback to students	Mattingly et al. (2012)
36. University of New England	Monitoring, intervention	Foster a sense of community among students studying part-time, at a distance as well as on-campusIdentify students who are struggling in order to provide timely supportDevelop a dynamic, systematic and automated process to capture the learning well-being status of studentsEncourage peer-to-peer student networkingDisseminate information and connect support staff with the students	Sclater et al. (2016)
37. University of North Bengal	Prediction	Examine the predictive relationship between learners’ pre-entry demographic information and their dropout behaviours	Yasmine (2013)
38. University of Rijeka	Data mining, analysis	Find out factors leading to student success in studyIdentify problems timely and increase the course pass rate	Sisovic et al. (2015)
39. University of Salamanca	Information extraction, analysis	Extract information useful for teaching/administrative staff, such as interaction of students with peers, teachers, the system, and course contentsProvide teachers with tools to facilitate managerial tasks	Conde et al. (2015)
		Support practical learning in a 3D virtual environment, analyse the problems that arisen, and report relevant data to students and teachers	Cruz-Benito et al. (2014)
40. University of Santiago de Compostela	Analysis, evaluation	Generate automatically reports of learners’ activities that take place in a virtual learning environmentImprove the efficiency of the evaluation process	Gewerc et al. (2014)
41. University of Sydney	Analysis, observation	Identify the relationship among student performance, choices of programming languages for study, and times at which a student starts and stops working on an assignment	Gramoli et al. (2016)
42. University of the South Pacific	Monitoring	Track individual learners’ online and offline interactions with open learning resources	Prasad et al. (2016)
43. University of Wollongong	Analysis, intervention, reflection	Visualize patterns of student interactions on discussion forumsAllow instructors to identify at-risk students and potentially high and low performing students for planning interventions, and the extent to which a learner community is developing in a class	Mat et al. (2013)