The Effects of WhatsApp and Telegram on Student Engagement: An Analysis from the Mixed-Methods Approach

Abstract

One of the most studied variables in virtual online education is engagement because it contributes to retention and academic performance. Several studies show that the didactic design of the virtual course, the role of the teacher, interactivity, interaction, time spent in the virtual environment, and the use of digital social networks increase engagement. However, there is a lack of research that compares which of the two digital social networks, WhatsApp or Telegram, promotes higher levels of engagement. This study’s objective is to analyze the effect of the educational use of Telegram on student engagement. An experiment (pretest and posttest with a control group) is designed using a mixed-methods approach based on a convergent or concurrent triangulation design. The study populations (coincides with the sample) are those enrolled (n = 229) in the six editions of a virtual postgraduate course. The techniques used for data analysis were scatter plots, content analysis of teachers’ narratives, and statistical methods. The triangulation of the quantitative and qualitative results confirms that the educational use of digital social networks promotes engagement in all experimental groups (use of the virtual learning environment and Telegram) obtaining greater significant differences than in the control groups. It is concluded that the use of the following Telegram functionalities, chat groups, peer assessments, support for various types of online interaction, the exchange of digital media, and the design of surveys contributed, under the guidance of teachers, is to increase the student’s engagement.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

1.1. Engagement in Virtual Courses

Postgraduate education has diversified the offers for professional improvement, increasingly using virtual courses. In Ibero-American countries, there is a negative trend in the number of university graduates who continue their professional training through these courses [1] due, among other factors, to lack of motivation, free access to virtual courses, time available, technological availability, and engagement [2]. Of these factors, engagement is one of the most studies as it is related to academic performance and retention.

By trend, engagement refers to the degree of active participation of students in a learning activity, helping to develop the success indicators of the effectiveness of the learning process. Student engagement is a multifunctional and interdisciplinary construct that traditionally comprises three dimensions [2]: (1) behavioral, related to students participation; (2) emotional, expressed in students emotions toward the faculty, the course, the educational process, and the school; and (3) cognitive, related to academic expectations and achievements before and during the teaching-learning process.

These dimensions allow the assessment of student engagement. However, in virtual online education, they include specific indicators. Several researchers [3] of virtual online education add the social dimension associated with interaction and interactivity in virtual courses and digital social networks (DSN). In this scenario, engagement studies promote commitment, motivation, and positive reactions in virtual online education. Students with adequate engagement are characterized by positive emotions, proactive behaviors, and cognitive performance according to the didactic objectives [4, 5].

1.2. The Assessment of Engagement in Virtual Courses: Scales and Questionnaires

Several studies propose scales and corresponding questionnaires to measure engagement in virtual courses [6]. Recent research [2, 3] refined these scales and questionnaires based on their strengths and weaknesses.

In this research, the authors assumed the scale validated by Deng et al. [3] given that it improves the scales proposed by previous studies. This author establishes the following dimensions:

(i) Emotional dimension or emotional engagement. This refers to the emotional connections (attachment, belonging, curiosity, enthusiasm, enjoyment, pleasure, and fun) of the students with the virtual course. Fundamentally, it is measured through the emotional commitment and emotional reaction (positive or negative) of students to technology-mediated learning and their sense of belonging.

(ii) Social dimension or social engagement. It focuses on student-teacher and student-student interactions during the online course. Some researchers conceive it as a subdimension of behavioral engagement [7] while others as a dimension [8]. It is measured primarily through the interaction and interactivity in the academic activities carried out in the virtual course, the quality of interactions, and the willingness and desire to promote the creation and maintenance of relationships during the study.

(iii) Cognitive dimension or cognitive engagement. It focuses on the self-regulation of student learning in their interaction with digital educational resources. This involves the systematization of learning, interactivity, and the fulfillment of objectives; that is, it is not satisfied with the e-qualification obtained but works on its improvement [9].

(iv) Behavioral dimension or behavioral engagement. This refers to the observable actions of students and their participation and involvement in educational activities. Fundamentally, it is measured by the quality of interactivity, the correct execution of activities, and the progress patterns, using the analytics and statistics of learning platforms.

In several empirical studies, the scales proposed in the literature [6] were applied although, standout those proposed by Deng et al. [3] and Rueda et al. [2].

Measuring engagement is not a simple process given that it involves external variables that often cannot be controlled by teachers, such as demographics, economic, social, migratory, and family status. Empirical studies tend to focus on how to promote engagement within the virtual course and not so much on how to enhance it “as a whole” with interaction and interactivity activities in the DSNs [3, 10].

1.3. The Use of DSN to Promote Engagement in Online Courses

It is known that the educational use of DSNs allows diversifying the interaction paths and interactivity between students, learning content, and digital educational resources.

There are several theoretical studies to demonstrate the potential of WhatsApp in education, mainly through group chat, collaborative activities, calls, and video calls [11]. Empirical studies allow asserting that its educational use allowed improving digital skills and engagement in students [11–13]. These studies also highlight latent concerns related to chat control (time and content), effective communication achievement, informational overload, and information loss in large groups [14].

Similar results are observed in the educational use of Telegram [15, 16] allowing the promotion of interaction and interactivity and, consequently, student engagement. The main functionalities employed were chatbots, chat groups, channels, and video calls [10]. Although both DSNs are used for educational purposes, the use of Telegram allows teamwork to be enhanced to a greater extent [17].

Therefore, the educational use of these DSNs to promote student engagement is well known. In this sense, as stated by Deng et al. [3], it is still necessary to systematize educational experiences of the educational use of DSNs through interactive and collaborative activities.

This research is carried out in this direction. In University of informatics Science (Universidad de las Ciencias Informáticas, Cuba), the national project entitled “ICTs that support educational processes and knowledge management in higher education (ELINF)” is implemented in collaboration with Chinese universities and the Inter-university Network for International Cooperation, aimed at strengthening postgraduate training through virtual online education and the use of educational technology. To this end, one of the virtual courses designed and implemented is “Evaluation of usability in computer systems” offered at the University of Informatics Science. The course traditionally (first two editions) was designed in the e-learning modality and supported by interactive and collaborative activities on the virtual learning environment and WhatsApp. The research question is as follows: Does the redesign of the virtual course based on activities in the virtual learning environment and Telegram promote higher levels of engagement in the enrolled students? The main objective is to analyze the effect of the educational use of Telegram on student engagement.

Therefore, it is expected to obtain higher levels of engagement in students who use Telegram. The previous studies [15] helped us to propose the following statistical hypotheses (quantitative dimension of the research):

(i) Null hypothesis (H0): in the experimental group, there are no changes between the mean score obtained in the pre- and posttest, at a significance level of 0.05.

(ii) Alternative hypothesis (H1): in the experimental group, there are changes between the mean score obtained in the pre- and posttest, at a significance level of 0.05.

2. Methods

2.1. Participant

The study population consisted of those enrolled in the six editions of the virtual postgraduate course: Evaluation of usability in computer systems. The sample (Table 1) included (coincides with the population) 229 graduate students in at the postgraduate school of University of Informatics Science. The analysis is carried out because of the six editions of the virtual graduate course offered at the University of Informatics Science. The sample size is small but adequate and sufficient according to the mixed nature of the research [18].

Table 1

Distribution of the sample.

Edition	Period	Control group (CG)	Experimental group (EG)	Media	Sex
First	February–April 2019	15	15	37 years (dt = 4.31)	40% women
					60% men
Second	November–January 2020	15	15	38 years (dt = 4.38)	40% women
					60% men
Third	July–September 2020	15	15	35 years (dt = 4.36)	43.33% women
					56.67% men
Fourth	November–January 2021	23	24	36 years old (dt = 4.54)	38.29% women
					61.71% men
Quinta	February–April 2021	32	33	33 years (dt = 4.29)	44.61% women
					55.39% men
Sixth	May–July 2021	13	14	35 years (dt = 4.37)	51.85% women
					48.15% men

Finally, the students in the sample are professionals from different areas: first edition: Computer Engineers (16), Graduates in Computer Science (5), and Graduates in Computer Education (9). Second edition: Computer Engineers (24) and Graduates in Computer Science (6). Third edition: Computer Engineers (16), Graduates in Computer Science (5), and Graduates in Computer Education (9). Fourth edition: Computer Engineers (22), Graduates in Computer Science (15), and Graduates in Computer Education (10). Fifth edition: Computer Engineers (34), Graduates in Computer Science (15), and Graduates in Computer Education (18). Finally, sixth edition: Computer Engineers (14), Computer Science Graduates (7), Journalism Graduates (1), and Computer Education Graduates (5).

2.2. Design and Measurements

An experimental design was adopted (pretest and posttest with control and experimental group) using a mixed-methods approach [18].

There are various criteria in experimental designs. The authors assume Tejedor-Tejedor [19] to “guarantee and reduce” the errors of internal and external validity according to the theoretical aspects that conform to our research (Figure 1).

[figure(s) omitted; refer to PDF]

The independent variable is the design of the virtual course that included interactive and collaborative activities in the Telegram. The dependent variable is student engagement.

Without transgressing the limits of the quantitative and qualitative paradigms, the mixed method is used through the principle of integration. In this method, the triangulation technique allows to corroborate and comprehensively analyze the quantitative and qualitative results. Therefore, the results will be presented methodologically according to the methods used, that is, separately (quantitative and qualitative results), and then, the results obtain from the triangulation.

As a classification of this methodology, a convergent or concurrent triangulation design was followed, collecting quantitative and qualitative data simultaneously (Figure 2) and relating the interpretation of the results [20]. The results finally obtained are not divergent. Therefore, it does not diminish the epistemological barrier that may or may not characterize this type of study. The data analysis was carried out according to the basic fusion analysis and use peer review as an integrating mechanism [20]. We decided to use the mixed methods to understand engagement from an integrative perspective since the empirical studies analyzed generally focus on only one type of methodology (quantitative or qualitative).

[figure(s) omitted; refer to PDF]

To measure student engagement the following instruments are used:

(i) Ad hoc scale and its questionnaire. We used the scale validated by Deng et al. [3], which was adapted to our research. It was necessary to adapt the scale proposed by Deng et al. [3] since, in our research, we included the use of WhatsApp and Telegram DSNs. To evaluate the questionnaire, a Likert-type scale of five values is used (1 = not at all, 2 = little, 3 = somewhat, 4 = quite a bit, and 5 = much) composed of four dimensions and 34 items (final questionnaire: https://doi.org/10.6084/m9.figshare.15153522.v1). The expert technique (n = 35) was used to determine content validity. The experts are doctors in education and are from Spain, China, Ecuador, and Argentina. The global questionnaire (internal consistency) has a Cronbach’s alpha (α) value equal to .91, expressing excellent results [21]. The reliability analysis (https://doi.org/10.6084/m9.figshare.16531077.v1) of each dimension is as follows: social α = .75; emotional α = .68; cognitive α = .73; and behavioral α = .88. To the scale, a questionnaire [18]was designed to apply to the students and find out their perceptions. For the validity of understanding the questionnaire, a pilot study was carried out with 15 teaching leaders from the institution itself and 21 students. In total, 458 questionnaires were registered (pre- and posttest). The Kaiser–Meyer–Olkin test was adequate (KMO = .83) and, the Bartlett sphericity test showed appropriate figures (χ2 = 3064.63; $p<.001$).

(ii) Direct observations. In relation to the scale, an observation guide was designed [22] to record student behavior in the virtual course, WhatsApp and Telegram. A total of 826 observations (observation guides) were recorded. Content analysis was applied to the observation guides.

(iii) Discussion group [22]. It was used in the control group (virtual course supported by interactive and collaborative activities in the virtual learning environment and WhatsApp) and experimental group (virtual course supported by interactive and collaborative activities in the virtual learning environment and Telegram). Independently of the questionnaire applied, we intended (student-student and student-teacher interaction) to know the individual and group perception about the engagement towards the virtual course and the activities in the DSNs. This technique is not to reach consensus but to reach dissent, thus enriching the data obtained.

(iv) Trace analysis (Moodle). To analyze student behavior patterns, with emphasis on interaction and time spent studying.

(v) Finally, to achieve an integrative analysis and assess the levels of engagement, the triangulation of the quantitative and qualitative results was applied [18].

(vi) The list of instruments applied according to the dimensions was as follows:

(1) Social dimension and emotional dimension. Instruments: direct observation, questionnaire, and discussion group.

(2) Cognitive dimension and behavioral dimension. Instruments: direct observation, questionnaire, discussion group, and analysis of the traces in the virtual course in Moodle.

2.3. Data Analysis

The qualitative approach allowed understanding the student engagement process without manipulating the educational scenarios to establish the following conditions: (1) compare engagement levels achieved through the educational use of WhatsApp and Telegram; (2) understand the meaning of data such as ideas, feelings, and behaviors before being quantified; and (3) describe and understand the process that occurred and not as a product, as is the case of the engagement levels. The study involved content analysis, keyword identification, and grouping of terms using word clouds [22]. The study was carried out from a deductive and inductive approach based on the teacher’s narrative (analysis of recorded observations) and triangulation during the creation and analysis of the words. ATLAS. ti software is used for content analysis.

The quantitative approach allowed us to understand and statistically interpret the results obtained in the levels of engagement. Statistical tests and techniques were applied (standard deviation, mean, Kolmogorov–Smirnov test, Levene’s test, Student’s t-test, Cohen’s d effect size, and biserial correlation r). We decided to use Student’s t-test and not the Z-test or Pearson’s correlations because all samples are smaller than 35 subjects [23]. Statistical analysis is performed using the SPSS version 25.0. The interaction time is show by scatter plots.

Following the mixed design, we employed triangulation to analyze the data from an integrative perspective.

2.4. Procedure

The phases of data collection, the instruments, and the techniques used are shown in Figure 2. First, the consent of the students was requested. Randomization was guaranteed in both control and experimental groups using a statistical program. At the same time, they were not informed whether they belonged to the control or experimental group; for this reason, the results are not due to the Hawthorne effect [24], which manifests itself when the participants in the experimental group obtain good results because they know to which group they belong. The objective of Phase II is to record student behavior in the virtual learning environment and on DSNs. The control group received the traditionally delivered virtual course that includes interactive and collaborative activities on WhatsApp. The experimental group received the redesigned virtual course that includes interactive and collaborative activities on Telegram.

It is worth noting that in the observation guide was conducted a content analysis of the virtual course activities (quality and quantity of messages sent/received in chats, forums, and wiki), WhatsApp activities (quality and quantity of messages sent/received in chat and group video calls), and Telegram activities (use of chatbots and quality and quantity of messages sent/received in group chats and group video calls).

The duration of each edition of the course is 10 weeks in the e-learning modality using the Moodle platform (https://aulacened.uci.cu/). The average minimum time dedicated by each participant according to the records was eight hours per week. The topics or didactic units of the virtual course are the following: Topic I: Usability: definition and general fundamentals; and Topic II: General concepts of usability evaluation: models, methods, and standards.

2.5. Ethical Procedure

The voluntary consent of the participants was requested. The confidentiality and anonymity of the data were guaranteed. None of the participants obtained any benefit/reward at the end of the study.

3. Results

The results obtained answer the research question: Does the redesign of the virtual course based on Telegram activities promote higher levels of engagement?

3.1. Quantitative Results

The research is based on the analysis carried out in the six editions of the virtual course. For this purpose, the values of all the factors of the questionnaire were measured. In all cases, a normal distribution is evident according to the $p$-value of the Kolmogorov–Smirnov normality test (greater than 0.05). To study the difference between the means, the Student’s t-test was applied and the Cohen’s d effect size and the estimation of the effect size by means of r were also analyzed. The measurements obtained are shown in Tables 2 and 3. In all cases, Levene’s test showed that the significance level was greater than 0.05 (α = 0.05), thus guaranteeing the assumption of homogeneity of variances.

Table 2

Pre- and posttest statistics (first three editions).

			Mean	DE	$p$	Comparison of Student’s t means
						Mean	t	Sig.	d	r
Edition 1	Social dimension	CG_pretest	2.93	0.22	0.24	0.38	−4.06	0.000	1.47	−0.59
		CG_posttest	3.31	0.29	0.57
		EG_pretest	2.7	0.28	0.78	1.13	−2.08	0.003	1.92	−0.69
		EG_posttest	3.33	0.37	0.16
	Emotional dimension	CG_pretest	3	0.26	0.15	0.35	−3.40	0.002	1.27	−0.53
		CG_posttest	3.35	0.29	0.15
		EG_pretest	2.76	0.38	0.48	0.65	−4.77	0.000	1.75	−0.65
		EG_posttest	3.41	0.36	0.64
	Cognitive dimension	CG_pretest	3	0.26	0.15	0.43	−4.37	0.000	1.62	−0.62
		CG_posttest	3.43	0.27	0.25
		EG_pretest	2.68	0.34	0.79	0.87	−6.56	0.000	1.48	−0.59
		EG_posttest	3.55	0.4	0.25
	Behavioral dimension	CG_pretest	3.03	0.26	0.26	0.6	−4.94	0.000	1.02	−0.45
		CG_posttest	3.63	0.38	0.79
		EG_pretest	2.93	0.31	0.47	0.68	−4.71	0.000	1.17	−0.5
		EG_posttest	3.61	0.46	0.67
Edition 2	Social dimension	CG_pretest	2.93	0.22	0.24	0.48	−4.65	0.000	1.71	−0.65
		CG_posttest	3.41	0.33	0.57
		EG_pretest	2.66	0.29	0.31	0.82	−5.72	0.000	2.13	−0.72
		EG_posttest	3.48	0.46	0.71
	Emotional dimension	CG_pretest	2.93	0.29	0.06	0.47	−4.35	0.000	1.62	−0.62
		CG_posttest	3.4	0.29	0.13
		EG_pretest	2.8	0.33	0.55	0.76	−5.86	0.000	1.21	−0.52
		EG_posttest	3.56	0.38	0.69
	Cognitive dimension	CG_pretest	2.95	0.36	0.42	0.58	−4.59	0.000	1.08	−0.47
		CG_posttest	3.53	0.32	0.63
		EG_pretest	2.63	0.37	0.72	1.1	−7.28	0.000	2.7	−0.8
		EG_posttest	3.73	0.44	0.97
	Behavioral dimension	CG_pretest	3.03	0.26	0.26	0.6	−4.94	0.000	1.84	−0.67
		CG_posttest	3.63	0.38	0.79
		EG_pretest	2.83	0.3	0.36	0.98	−7.34	0.000	2.72	−0.8
		EG_posttest	3.81	0.41	0.56
Edition 3	Social dimension	CG_pretest	2.9	0.28	0.4	0.51	−4.57	0.000	1.66	−0.64
		CG_posttest	3.41	0.33	0.57
		EG_pretest	2.6	0.32	0.45	1.11	−8.64	0.000	2.92	−0.82
		EG_posttest	3.71	0.43	0.8
	Emotional dimension	CG_pretest	2.91	0.3	0.12	0.49	−4.37	0.000	1.66	−0.63
		CG_posttest	3.4	0.29	0.13
		EG_pretest	2.91	0.43	0.43	0.77	−4.67	0.000	1.74	−0.65
		EG_posttest	3.68	0.45	0.34
	Cognitive dimension	CG_pretest	2.95	0.36	0.42	0.58	−4.59	0.000	1.7	−0.64
		CG_posttest	3.53	0.32	0.63
		EG_pretest	2.55	0.42	0.15	1.38	−8.12	0.000	2.98	−0.83
		EG_posttest	3.93	0.5	0.87
	Behavioral dimension	CG_pretest	3.03	0.26	0.26	0.6	−4.94	0.000	1.84	−0.67
		CG_posttest	3.63	0.38	0.79
		EG_pretest	2.73	0.29	0.45	1.37	−2.14	0.000	4.48	−0.91
		EG_posttest	4.1	0.32	0.84
		EG_posttest	3.03	0.61	0.13

Cg: control group; EG: experimental group.

Table 3

Pre- and posttest statistics (first last editions).

			Media	DE	$p$	Comparison of Student’s t means
						Media	t	Sig.	d	r
Edition 4	Social dimension	CG_pretest	2.9	0.29	0.13	0.33	−3.15	0.003	0.92	−0.42
		CG_posttest	3.23	0.41	0.66
		EG_pretest	2.88	0.3	0.21	0.56	−5.2	0.000	1.48	−0.59
		EG_posttest	3.44	0.44	0.76
	Emotional dimension	CG_pretest	2.84	0.28	0.13	0.44	−4.23	0.000	1.27	−0.53
		CG_posttest	3.28	0.4	0.57
		EG_pretest	2.84	0.28	0.24	0.63	−6.16	0.000	1.82	−0.67
		EG_posttest	3.47	0.4	0.53
	Cognitive dimension	CG_pretest	2.85	0.28	0.16	0.43	−3.86	0.000	1.11	−0.48
		CG_posttest	3.28	0.47	0.61
		EG_pretest	2.81	0.33	0.41	0.74	−5.8	0.000	1.72	−0.65
		EG_posttest	3.55	0.51	0.35
	Behavioral dimension	CG_pretest	2.89	0.32	0.07	0.37	−3.06	0.003	1	−0.44
		CG_posttest	3.26	0.41	0.66
		EG_pretest	2.85	0.33	0.17	0.54	−3.59	0.001	1.11	−0.48
		EG_posttest	3.39	0.6	0.25
Edition 5	Social dimension	CG_pretest	2.85	0.32	0.16	0.36	−3.83	0.000	0.94	−0.42
		CG_posttest	3.21	0.43	0.25
		EG_pretest	2.9	0.32	0.054	0.53	−5.29	0.000	1.31	−0.55
		EG_posttest	3.43	0.47	0.41
	Emotional dimension	CG_pretest	2.82	0.28	0.39	0.43	4.66	0.000	1.22	−0.52
		CG_posttest	3.25	0.41	0.07
		EG_pretest	2.81	0.28	0.2	0.69	7.72	0.000	1.93	−0.69
		EG_posttest	3.5	0.42	0.32
	Cognitive dimension	CG_pretest	2.83	0.27	0.27	0.4	3.57	0.001	1.37	−0.56
		CG_posttest	3.23	0.49	0.31
		EG_pretest	2.78	0.35	0.14	0.83	6.5	0.000	1.77	−0.66
		EG_posttest	3.61	0.56	0.08
	Behavioral dimension	CG_pretest	2.88	0.34	0.03	0.33	−2.76	0.004	0.85	−0.39
		CG_posttest	3.21	0.43	0.25
		EG_pretest	2.88	0.34	0.78	0.4	−2.97	0.004	0.85	−0.39
		EG_posttest	3.28	0.57	0.18
Edition 6	Social dimension	CG_pretest	2.88	0.29	0.39	0.5	−3.5	0.001	1.58	−0.62
		CG_posttest	3.38	0.34	0.3
		EG_pretest	2.91	0.36	0.41	0.48	−2.05	0.03	0.97	−0.43
		EG_posttest	3.39	0.6	0.38
	Emotional dimension	CG_pretest	2.9	0.29	0.52	0.48	−3.61	0.001	1.51	−0.6
		CG_posttest	3.38	0.34	0.63
		EG_pretest	2.73	0.28	0.3	0.81	−4.71	0.000	1.88	−0.68
		EG_posttest	3.54	0.54	0.52
	Cognitive dimension	CG_pretest	2.88	0.29	0.39	0.54	−4.06	0.000	1.31	−0.54
		CG_posttest	3.42	0.41	0.43
		EG_pretest	2.71	0.36	0.89	0.98	−4.55	0.000	1.82	−0.67
		EG_posttest	3.69	0.67	0.73
	Behavioral dimension	CG_pretest	2.88	0.29	0.39	0.5	−4.19	0.000	1.58	−0.62
		CG_posttest	3.38	0.34	0.72
		EG_pretest	2.91	0.36	0.41	0.35	−0.66	0.003	0.69	−0.32
		EG_posttest	3.26	0.62	0.13

Cg: control group; EG: experimental group.

In both the control and experimental groups, it is possible to increase the engagement of the enrolled students, since significant differences are evident at a confidence level of 95% (α = 0.05). However, in the experimental groups, the significant differences are greater than those of the control groups.

These results are ratified [25] by the values obtained in the effect size (d) and its estimation (r) since the lowest value obtained (d = 0.07) indicates that 52.8% of the experimental group will be above the mean of the control group; that is, there is a 52% probability that a person in this group will have a higher score than a person chosen at random from the control group.

The statistical results allow us to reject (probability terms) the null hypothesis and accept the alternative. Triangulation was also applied to the information coming from the quantitative Moodle traces and to the record of observations made by the teachers to the interactive and collaborative activities (virtual classroom and DSNs).

The data referring to the time dedicated by students to the virtual environment and interaction in WhatsApp and Telegram were quantified and represented in a scatter plot. In the first three editions, a greater use of the virtual environment and Telegram was observed (Figure 3). In turn, in the last three editions, there is also a significant difference in the use of Telegram. It is worth noting that in all cases there is greater participation in the interactive and collaborative activities in the DSNs than in the activities designed in the virtual environment.

[figure(s) omitted; refer to PDF]

3.2. Qualitative Results

The following is a synthesis of the results obtained (826 records of direct observations, six focus groups, and 229 ratings in open-ended questions of the questionnaire). The results are presented according to the dimensions of engagement.

3.2.1. Social Dimension

In the records of observations of student behavior (OSB), it is highlighted that:

(i) Control group.

  Shares on WhatsApp videos downloaded from the Internet and, in some cases, edits it (OSB-1).

  Interact in the chat group (WhatsApp), forums, and Moodle chat rooms. Shares the URL of learning materials external to the course as a way of deepening the content and curiosity of learning (OSB-11).

(ii) Experimental group.

  Create channels and share information of interest (OSB-21).

  Shares various groups associated with software quality (OSB-27).

  Shows constant interaction with teachers and students through the activities in Moodle and the chat group in Telegram (OSB-31).

In the self-perceptions of the students (SPS), opinions are as follows:

(i) Control group.

  Sometimes some students talked about topics that did not interest me or were external to the objective of the course (SPS-15).

  Thanks to the collaboration (multimedia resources and clarifications of doubts) of some colleagues in the course I was able to understand some tasks; also, using WhatsApp was very helpful (SPS-27).

  The forums in the course allowed me to exchange with professors and students and thus clarify my doubts (SPS-16).

(ii) Experimental group.

  The Moodle and Telegram forums and chat rooms served me to solve my concerns (SPS-19).

3.2.2. Emotional Dimension

The records of the observations highlight that:

(i) Control group.

  Expresses in the video call interest and excitement to perform the learning activities (OSB-41).

  Does not always expose ideas related to the learning content (WhatsApp) but does in the Moodle forums (OSB-35).

(ii) Experimental group.

  He is assertive and consistent in his opinions expressed through the chat group on Telegram (OSB-61).

  Sometimes it shared other groups or channels, but without explaining what it consisted of. It only broadcasts that “it is interesting this channel” (OSB-39).

In the students’ self-perceptions, opinions are as follows:

(i) Control group.

  I felt good interacting via WhatsApp, but not so much in the Moodle Chat room as it is very cold (SPS-46).

  The teachers were always responsive to my questions, which helped me to successfully complete the course. I was happy with the course (SPS-59).

(ii) Experimental group.

  I loved Telegram; I had not used it before. You can share a lot of information and its functionalities are very diverse. I think it made the course much more enjoyable and interactive (SPS-53).

3.2.3. Cognitive Dimension

The records of the observations highlight that:

(i) Control group.

  In the solution of the e-task, he manages to go deeper into the content. Therefore, he not only reflects what he learned according to what was exposed in the course, but also consulted other materials (OSB-50).

  Interacts on several occasions with digital educational resources (OSB-57).

(ii) Experimental group.

  Share learning materials through Telegram, mainly infographics and channels (OSB-26).

  Repeat the e-self-assessments until the maximum grade is achieved.

  Commits to the teacher to obtain better grades in the next evaluative activities (OSB-44).

  In the students’ self-perceptions, opinions are as follows:

(i) Control group.

  At times, I was disappointed with the grade on my work, but the professor encouraged me and I did not drop the course. I made a commitment to study more (SPS-53).

(ii) Experimental group.

  The self-assessment activities served as training for me. Until I got the maximum grade (three attempts), I did not stop doing it (SPS-45).

3.2.4. Behavioral Dimension

The records of the observations highlight that:

(i) Control group.

  Interacts on several occasions with digital educational resources (OSB-16). Spends time interacting (OSB-5).

(ii) Experimental group.

(iii) There is a time lapse of at least three days between the interaction with the learning materials and the completion of the evaluative activities (OSB-3).

(iv) Requests through the Telegram Chat group for help or summaries made by colleagues (OSB-33).

In the students’ self-perceptions, opinions are as follows:

(i) Control group.

  Sometimes we would share notes and summaries of what we had learned, but sometimes you would get lost in conversation with other students (SPS-19).

(ii) Experimental group.

  The exchange through personalized surveys via Telegram motivated me to learn more and dedicate more time to learning (SPS-11).

In the content analysis of the narratives expressed by the teachers in the observation guides, a word cloud was elaborated of which eight words were the most recurrent. As a result, they were quantified (percentage) in correspondence to the behavior of each student. It can be observed that there is a trend toward greater interaction and interactivity in the DSNs than in the virtual classroom (Figures 4 and 5).

[figure(s) omitted; refer to PDF]

The educational use of the DSNs contributed to raise positive emotions in the students and, consequently, a higher engagement. Greater interactivity, interaction, and positive emotions were observed in the experimental groups using the Telegram DSN.

3.3. Results of the Mixed Analysis

In the triangulation of the qualitative and quantitative data obtained (Figure 6), the dimensions of engagement with the highest levels of evidence in the students were as follows:

(i) First edition. The control group (cognitive and behavioral dimensions) and the experimental group (social and behavioral dimensions).

(ii) Second and third editions. The control group (cognitive and behavioral dimensions) and the experimental group (social, emotional, and behavioral dimensions).

(iii) Fourth and fifth editions. The control group (social, emotional, and behavioral dimensions) and the experimental group (social, emotional, and behavioral dimensions).

(iv) Sixth edition. The control group (social, emotional, and behavioral dimensions) and the experimental group (cognitive, emotional, and behavioral dimensions).

[figure(s) omitted; refer to PDF]

The analysis of quantitative and qualitative data allows us to affirm that the educational use of the virtual learning environment in combination with Telegram promoted higher levels of engagement in students. All the experimental groups show higher levels of engagement than the control groups. Even so, also in the control groups, there are satisfactory results in student engagement.

The following aspects maybe summarize the main findings in the triangulation of the data:

(i) Relationship between the quality of the discussion tasks in the virtual course and the levels obtained in cognitive engagement.

(ii) Relationship between the quantity and quality of the messages sent/received (Moodle chat room, forums, and chat on WhatsApp or Telegram) and the levels of social engagement.

(iii) Relationship between student-teacher interaction and participation in learning activities in DSNs, and levels of emotional engagement.

(iv) Relationship between the use of infographics or other learning materials shared by other students in the chat group created on the digital social networks WhatsApp or Telegram and the levels of behavioral engagement.

4. Discussion and Conclusions

The quantitative results assert that the use of Telegram functionalities with collaborative activities in Moodle promoted greater engagement in the student body. These results coincide with similar investigations by Conde et al. [17] and Aladsani [10]. Collaborative interactions through chat groups and peer evaluations increased engagement and motivation. Feedback on Telegram was done in student-centered and a process-centered dialogue. For this reason, it was necessary to train the teacher and the student in the didactic use of this social network [26].

Three essential aspects are reflected in the qualitative results: (1) greater engagement in the use of digital social networks; (2) the value of interactivity in Moodle and Telegram forums and chat rooms, the need to achieve from the video call; and (3) the interest and enthusiasm to carry out the learning activities. These results do not coincide with the observations made by Vilches and Reche [14] since collaborative activities were achieved through WhatsApp, which coincides with the investigations of Escobar-Mamani and Gómez-Arteta [27].

The triangulation of the qualitative and quantitative results shows a direct relationship between the dimensions of student engagement and the collaborative activities carried out with Moodle and digital social networks. Only joint activities allowed developing student engagement (learning platform and social network); these results coincide with those obtained by Estrada-Molina and Fuentes-Cancell [26]. Consequently, from digital social networks, the collaborative influence of fellow students exerted a direct influence on the behavior of students and teachers; this result coinciding with those obtained by Yu et al. [28].

Numerous studies reaffirm that the educational use of DSNs contributes to increase engagement [13, 27]. Our study reaffirms the importance of diversifying learning scenarios using WhatsApp and Telegram. Despite the discrete number of professionals enrolled in each edition of the course, the mixed method allowed us to verify the usefulness of the DSNs to promote engagement and confirm that the educational use of Telegram was more effective than WhatsApp.

Postgraduate higher education promotes virtual online education through the design of virtual courses. In this context, generating adequate engagement is vital to achieving high retention rates. We agree with other similar research types [4, 11] that the educational use of WhatsApp influences the social, emotional, and behavioral dimensions of engagement, while Telegram fundamentally influences the cognitive, emotional, and behavioral dimensions [26].

The possibilities offered by Telegram related to chat groups, peer assessments, support for various types of online interaction, exchange of digital media, and the design of surveys made it possible, under the guidance of teachers, to promote engagement. Similar studies [10] obtained similar results, and although they were based on virtual courses supported by Blackboard, they also reflect that students prefer to interact through the DSNs and not in interactive activities (forums and chat rooms) designed in the learning management platform.

The results confirm the importance of autonomy, satisfaction, and collaborative and interactive learning in a virtual course, that is, through forums, chat rooms, and collaborative learning tasks in the virtual learning environment [29]. Even though there is a lack of empirical studies comparing the effects of the educational use of Telegram and WhatsApp and the engagement in the context of virtual online education, Telegram evidences more educational opportunities due to the diversity of its functionalities [17].

The possibility of carrying out collaborative activities in the DSNs allowed generating positive emotions in students even when there are different limitations mainly focused on communication control [14]. The results show the engagement evidenced in the control and experimental groups, although these communicative shortcomings persist. In this sense, this study has practical implications; the results suggest the pedagogical training of teachers for the educational and communicative use of DSNs. It is worth noting that the results show that students interacted more in the DSN than in the forums and chat rooms designed in Moodle, which would be interesting to delve deeper into their causes and how they relate to engagement.

In the didactic design carried out, two actions gave satisfactory results. Identifying outstanding students in their academic performance and with high levels of engagement allowed us to assign them leadership roles in the chat sessions on WhatsApp or Telegram (depending on the group to which they belong). These students shared their emotions and learning experiences during the course and moderated some key chat sessions. Second, encouraging the student to express her views and ensuring educational feedback contributed to “creating or maintaining” positive emotions in the virtual course.

The mixed-methods approach allowed us to analyze the participation of students in the learning platform and the DSNs from different perspectives, this being one of the main shortcomings of previous studies [30]. Finally, it is confirmed that time spent participating in virtual activities on a learning management platform (Moodle) is also related to engagement. This effect was also latent in the time dedicated by students to the DSNs. The students who spent the most time in the e-learning environment were also those who spent the most time on the DSNs. We appreciate that applying content analysis and thematic analysis to messages received/sent by students allowed us to obtain more information about their participation in the DSNs.

In general, the results are satisfactory, suggesting that the design of a virtual course that includes activities in the WhatsApp and Telegram DSNs contributes to promoting adequate levels of engagement. Of course, it is not only the didactic design, but also to apply educative communication and the “adequate educational control” in the DSNs. The main result of this research is to reflect the influence of the use of DSNs (WhatsApp and Telegram) to promote engagement from virtual education and, in addition, to highlight the statistical advantages and the behavior of students who performed collaborative and interactive activities in Telegram in relation to those of the control group related to the use of WhatsApp.

4.1. Limitations and Future Studies

The main limitation is the number of students who participated in each edition of the virtual course. From a quantitative perspective, it is insufficient to generalize the results. Therefore, a mixed method was designed to complement the quantitative data with information from qualitative methods. For this reason, we consider continuing to analyze these results in future editions.

Another possible line of research is to use the new scale proposed to measure engagement [5], but we would have to adapt it to our context, as it is still focused on the design of the virtual course without the use of the DSNs. In addition, we will study the correlation between the time spent by students in the e-learning environment and the DSNs and their relationship with engagement and academic performance.

Acknowledgments

The author would like to thank for institutional support from the University of Information Sciences of Cuba.

References

[1] N. Lázaro Alvarez, Z. Callejas, D. Griol, "Factores que inciden en la deserción estudiantil en carreras de perfil ingeniería informática," Revista Fuentes, vol. 1 no. 22, pp. 105-126, DOI: 10.12795/revistafuentes.2020.v22.i1.09, 2020.

[2] P. O. Rueda, Ó. M. Torelló, F. S. Mut, "Educational disengagement profiles: a multidimensional contribution within basic vocational education and training," Revista de Educación, vol. 389, pp. 69-94, 2020.

[3] R. Deng, P. Benckendorff, D. Gannaway, "Learner engagement in MOOCs: scale development and validation," British Journal of Educational Technology, vol. 51 no. 1, pp. 245-262, DOI: 10.1111/bjet.12810, 2020.

[4] O. Estrada Molina, D. R. Fuentes Cancell, A. García Hernández, "Elengagementen la educación virtual:experiencias durante la pandemia COVID-19," Texto Livre: Linguagem e Tecnologia, vol. 14 no. 2,DOI: 10.35699/1983-3652.2021.33936, 2021.

[5] H. Benito Mundet, E. Llop Escorihuela, M. Verdaguer Planas, J. Comas Matas, A. Lleonart Sitjar, M. Orts Alis, A. Amadó Codony, C. Rostan Sánchez, "Multidimensional research on university engagement using a mixed method approach," Educación XX1, vol. 24 no. 2, pp. 65-96, DOI: 10.5944/educxx1.28561, 2021.

[6] F. H. Veiga, "Assessing student engagement in school: development and validation of a four-dimensional scale," Procedia—Social and Behavioral Sciences, vol. 217, pp. 813-819, DOI: 10.1016/j.sbspro.2016.02.153, 2016.

[7] J. Maroco, A. L. Maroco, J. A. D. B. Campos, J. A. Fredricks, "University student’s engagement: development of the university student engagement inventory (USEI)," Psicologia: Reflexão e Crítica, vol. 29 no. 1,DOI: 10.1186/s41155-016-0042-8, 2016.

[8] M. T. Wang, J. A. Fredricks, F. Ye, T. L. Hofkens, J. S. Linn, "The math and science engagement scales: scale development, validation, and psychometric properties," Learning and Instruction, vol. 43, pp. 16-26, DOI: 10.1016/j.learninstruc.2016.01.008, 2016.

[9] J. Wong, M. Baars, D. Davis, T. Van Der Zee, G. J. Houben, F. Paas, "Supporting self-regulated learning in online learning environments and MOOCs: a systematic review," International Journal of Human-Computer Interaction, vol. 35 no. 4-5, pp. 356-373, DOI: 10.1080/10447318.2018.1543084, 2019.

[10] H. K. Aladsani, "University students’ use and perceptions of telegram to promote effective educational interactions: a qualitative study," International Journal of Emerging Technologies in Learning (iJET), vol. 16 no. 9,DOI: 10.3991/ijet.v16i09.19281, 2021.

[11] C. Costa-Sánchez, M. Guerrero-Pico, "What is whatsapp for? Developing transmedia skills and informal learning strategies through the use of whatsapp—a case study with teenagers from Spain," Social Media+ Society, vol. 6 no. 3,DOI: 10.1177/2056305120942886, 2020.

[12] B. Afful, R. Akrong, "Whatsapp and academic performance among undergraduate students in Ghana: evidence from the University of Cape Coast," The Journal of Education for Business, vol. 95 no. 5, pp. 288-296, DOI: 10.1080/08832323.2019.1644276, 2020.

[13] M. Á. Conde, F. J. Rodríguez-Sedano, C. Fernández, A. Gutiérrez-Fernández, L. Fernández-Robles, M. Castejón Limas, "A learning analytics tool for the analysis of students’ telegram messages in the context of teamwork virtual activities," Proceedings of the 8th International Conference on Technological Ecosystems for Enhancing Multiculturality,DOI: 10.1145/3434780.3436601, .

[14] Vilela M. J. Vilches, Urbano E. Reche, "Limitaciones de whatsapp para la realización de actividades colaborativas en la universidad," RIED. Revista Iberoamericana de Educación a Distancia, vol. 22 no. 2,DOI: 10.5944/ried.22.2.23741, 2019.

[15] M. Alahmad, "The effectiveness of telegram app in learning english," Budapest International Research and Critics in Linguistics and Education (BirLE) Journal, vol. 3 no. 3, pp. 1274-1280, DOI: 10.33258/birle.v3i3.1165, 2020.

[16] M. Á. Conde, F. J. Rodríguez-Sedano, F. J. Rodríguez-Lera, A. Gutiérrez-Fernández, Á. M. Guerrero-Higueras, "Assessing the individual acquisition of teamwork competence by exploring students’ instant messaging tools use: the whatsapp case study," Universal Access in the Information Society, vol. 20 no. 3, pp. 441-450, DOI: 10.1007/s10209-020-00772-1, 2020.

[17] M. Conde, F. J. Rodríguez-Sedano, F. J. R. Rodríguez Lera, A. Gutiérrez-Fernández, Á. M. Guerrero-Higueras, "Whatsapp or telegram. Which is the best instant messaging tool for the interaction in teamwork?," Learning and Collaboration Technologies: New Challenges and Learning Experiences, pp. 239-249, DOI: 10.1007/978-3-030-77889-7_16, 2021.

[18] R. B. Johnson, A. J. Onwuegbuzie, "Mixed methods research: a research paradigm whose time has come," Educational Researcher, vol. 33 no. 7, pp. 14-26, DOI: 10.3102/0013189x033007014, 2004.

[19] F. Tejedor-Tejedor, "Validez interna y externa en los diseños experimentales," Revista Espanola de Pedagogia, vol. 39 no. 151, pp. 15-40, 1981.

[20] S. Bagur Pons, M. R. Rosselló Ramon, B. Paz Lourido, S. Verger, "El enfoque integrador de la metodología mixta en la investigación educativa," RELIEVE—Revista Electrónica de Investigación y Evaluación Educativa, vol. 27 no. 1,DOI: 10.30827/relieve.v27i1.21053, 2021.

[21] H. C. Oviedo, A. Campo-Arias, "Aproximación al uso del coeficiente alfa de cronbach," Revista Colombiana de Psiquiatria, vol. 34 no. 4, pp. 572-580, 2005.

[22] J. I. Ruiz-Olabuénaga, Metodología de la Investigación Cualitativa, 2012.

[23] D. Altman, D. Machin, T. Bryant, M. Gardner, Statistics with Confidence Confidence Intervals and Statistical Guidelines, 2000.

[24] C. M. Hart, "Los experimentos de Hawthorne," Revista Cubana de Salud Pública, vol. 38 no. 1, pp. 156-167, DOI: 10.1590/s0864-34662012000100015, 2012.

[25] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 1988.

[26] O. Estrada-Molina, D. R. Fuentes-Cancell, "Engagement and desertion in MOOCs: systematic review," Comunicar, vol. 30 no. 70, pp. 111-124, DOI: 10.3916/c70-2022-09, 2022.

[27] F. Escobar-Mamani, I. Gómez-Arteta, "Whatsapp for the development of oral and written communication skills in Peruvian adolescents," Comunicar, vol. 28 no. 65, pp. 111-120, DOI: 10.3916/c65-2020-10, 2020.

[28] Z. Yu, W. Xu, P. Sukjairungwattana, "Examining student behavioral intention of superstar learning system by extending its technology acceptance model," International Journal of Web-Based Learning and Teaching Technologies, vol. 17 no. 1,DOI: 10.4018/ijwltt.305804, 2022.

[29] Y. Sun, Y. Guo, Y. Zhao, "Understanding the determinants of learner engagement in MOOCs: an adaptive structuration perspective," Computers and Education, vol. 157,DOI: 10.1016/j.compedu.2020.103963, 2020.

[30] L. M. Nkomo, B. K. Daniel, R. J. Butson, "Synthesis of student engagement with digital technologies: a systematic review of the literature," International Journal of Educational Technology in Higher Education, vol. 18 no. 1,DOI: 10.1186/s41239-021-00270-1, 2021.