Content area
Smart learning environments (SLEs) utilize technological advancements to facilitate effective, engaging, and personalized learning experiences. They depend on sensors and advanced connectivity to gather information and make informed decisions. Multisensory Environments (MSEs) naturally align with and enhance the capabilities of SLEs offering new opportunities to enhance learning effectively, and engage children with stimulating educational experiences leveraging different interaction modalities. Investigating how children interact with these new systems is important to design educational technologies. However, limited research has been conducted to evaluate the role of interaction modalities in moderating the relationship between students’ experience and their learning outcomes in a MSE. We, therefore, tracked 175 students’ (aged 6-10) correctness rate to questions and their states through motion, heart rate, and electrodermal activity, obtaining their levels of fatigue, stress, engagement, emotional regulation, and anxiety. We then analysed the moderating role of five different interaction modalities ("card", "feet", "hands", "voice", "wand") on the relationship between correctness rate and states. The results of this in-situ study show that the relationship between student states and their performance is moderated by the interaction modalities, offering important design and theoretical implications on the role of the interaction modalities in the learning experience of students with an MSE. The contributions of this research benefit all stakeholders involved, including students who receive appropriate learning experiences, and practitioners who can make informed decisions on what interaction modalities to use to support the learning experience.
Introduction
Creating effective digital educational experiences that address students’ needs and preferences is a complex task, requiring careful consideration of various factors. Traditional SLEs rely on mainstream data, such as computer logs from learners and educators, to support richer representations (e.g., correctness and response times) (Tabuenca et al., 2021). Recently, there has been a surge in curiosity about exploring various methods of utilising and engaging with digital technology, with a particular focus on multisensory interaction (Dourish, 2001) for learning. Multisensory environments (MSEs) are digital spaces that include several devices and sensors designed to stimulate human senses, making them an ideal platform for promoting natural and meaningful play and learning for students (Gelsomini & Leonardi, 2020; Lee-Cultura, Sharma, Cosentino, Papavlasopoulou, and Giannakos, 2021). As a result of their recent affordability and widespread use, these devices are now easily usable with students, encouraging meaningful and organic play and learning (Cosentino & Giannakos, 2023).
MSEs have the capacity to collect large amounts of Learning Analytics (LA) from mainstream log-based analytics that represent students’ mastery and response times to more sophisticated Multimodal Learning Analytics (MMLA) that account for students’ learning states such as arousal, engagement, and fatigue (Chaouachi & Frasson, 2012; Bower and Carroll, 2017; Arguedas, Daradoumis, and Xhafa Xhafa, 2016). MSEs affordances alongside the respective insights provided via MMLA have been successful in providing a more comprehensive understanding of learners’ experiences (Blikstein and Worsley, 2016; Chejara et al., 2019; Shakroum et al., 2018; Cosentino & Giannakos, 2023). Despite these advantages, limited research has attempted to explore the effect of MSEs on affording students to use different interaction modalities (Chen et al., 2017; Anthony et al., 2012; Jevtić et al., 2015; Donker and Reitsma, 2007; Hourcade et al., 2018; Spitale et al., 2019). More research is needed to investigate how different interaction modalities influence the relationship between students’ learning states and their mastery within MSEs. Addressing this gap is the central objective of our study.
This paper presents findings from an in-situ study with 175 students (ages 6–10) in an MSE setting, combining performance data (correctness) with sensor-based MMLA to provide a holistic view of learning. We specifically examine how students’ physiological states relate to their mastery, and how five interaction modalities–“card,” “feet,” “hands,” “voice,” and “wand”–moderate this relationship. Thus, the following research question (RQ) is attempted to be addressed in this work:
“How do the interaction modalities moderate the relationship between students’ physiological processes (as inferred from sensor-based MMLA) and their mastery level (as inferred from their correctness) in MSEs?”.
Our study aims to identify the role of interaction modalities in supporting learning strategies within MSEs. Key contributions include actionable insights for designing adaptive learning experiences, practical guidance for educators on modality use, and theoretical implications on how interaction design can align students’ abilities with learning challenges.
Related work
In the evolving landscape of education, two innovative approaches are making significant strides toward transforming how we understand and facilitate learning, namely MSEs and MMLA. MSEs refer to spaces or settings designed to stimulate multiple senses simultaneously. This could involve anything from visual and auditory stimuli to tactile or olfactory inputs. The core idea is that by engaging more than one sense at a time, learners can have a richer, more engaging experience. On the other hand, MMLA takes a more tech-savvy approach as it gathers and analyzes data from different modes of learning and interaction to gain a comprehensive understanding of the learning process, identifying how cognitive, emotional and social factors contribute to educational outcomes.
MSEs in education
MSEs have increasingly become a focal point in discussions around enhancing therapeutic, rehabilitative, and educational practices (Almjally et al., 2020; Gelsomini & Leonardi, 2020). By integrating stimuli that simultaneously engage multiple senses, these environments offer a unique and effective approach to learning and development. This concept, while not novel, is rooted in the understanding that learners benefit significantly from experiences that draw upon their sensory perceptions–sight, sound, touch, and even smell. The effectiveness of multisensory learning in increasing memory recall and information retention has been well documented in various studies (Lee-Cultura et al., 2021; Kosmas et al., 2018; Malinverni et al., 2019). Research in the field has explored the application of MSEs across different educational contexts, highlighting their potential to support a wide range of learning outcomes. The benefits of such environments are manifold, including improved memory recall and retention, heightened student engagement, and increased motivation. These advantages are not confined to any single subject area; evidence suggests that MSEs can support learning in mathematics (Chejara et al., 2019; Howison, Trninic, Reinholz, and Abrahamson, 2011; Abrahamson et al., 2020), science (Ping & Goldin-Meadow, 2010), music (Robb, 2018), language acquisition (Glenberg et al., 2011), and health training (Di Mitri et al., 2020). Importantly, MSEs have shown particular promise for learners with special needs, such as those with dyslexia (Wang et al., 2018), underscoring their versatility and potential for inclusivity in education. Several studies used multisensory technologies to employ spatial, haptic, and auditory interactions with objects to support active learning strategies for children with sensorimotor diversity (Rühmann et al., 2016).
The number of MSEs’ sensors that identify the actions and behaviours of the user, including gesture and motion recognition, led researchers to develop new forms of interaction modalities (Karray et al., 2008). Providing a framework for a certain level of embodiment, as seen in physical user interfaces, motion interfaces, and gestural interfaces, to improve the naturalness and significance of user-machine connection (Kourakli et al., 2017; Castelhano & Roque, 2017; Malinverni et al., 2017). When integrated, the employed interaction modalities are highly important for learning, surpassing representationalist modes of cognition and favouring enactivist reasoning (Abrahamson et al., 2021; Di Mitri et al., 2017). The significance of engaging students through multisensory interactions aligns with several established learning theories. These theories underscore the importance of maintaining student involvement in activities that are goal-oriented, suggesting that this engagement is crucial for effective learning. This principle is especially relevant in the context of digital learning tools, where the design and implementation of MSEs can play a pivotal role in improving educational outcomes (Kolb and Kolb, 2005; Nielsen-Englyst, 2003; So, 1964; Webster et al., 1993; Csikszentmihalyi, 2014). The use of MSEs into educational settings represents a significant step forward in the development of teaching methods that are both effective and inclusive. By leveraging the power of multisensory interactions, educators can create more engaging, memorable, and accessible learning experiences for all students.
The emerging advantages of MMLA
Recent advancements in sensor and data analysis technologies have enabled researchers to investigate the relationship between different sensory modalities and learning outcomes (Zou et al., 2017; Giannakos, Horn, Read, and Markopoulos, 2020; Gelsomini et al., 2019) deepening a branch defined as Multimodal Learning Analytics (MMLA). MMLA leverages diverse data sources, including verbal, visual, and gestural interactions, to understand and enhance the learning process (Spitale et al., 2019; Poon et al., 2019; Ouhaichi et al., 2023; Bartoli et al., 2014; Bhattacharya et al., 2015; Garzotto, Gelsomini, Gianotti, and Riccardi, 2019). This approach integrates various signals and interaction modalities, analyzing them to provide insights into learners’ cognitive, emotional, and social dimensions. MMLA aims to create a comprehensive picture of the learning environment, enabling educators and researchers to tailor educational experiences more effectively to individual learner needs and preferences, and to improve educational outcomes by addressing the multifaceted nature of learning.
A significant body of research has explored the use of sensor-based MMLA. Among others, sensor-based MMLA is used to predict the following users’ states (Chaouachi & Frasson, 2012; Bower and Carroll, 2017; Arguedas et al., 2016).
Engagement: the key predictor of learning success. Sensor-based MMLA utilizes a combination of physiological sensors (e.g., eye-tracking to monitor attention focus, wearable devices measuring heart rate variability for arousal levels) and environmental sensors (e.g., audio analysis for classroom noise levels, motion sensors to detect physical activity) to gauge student engagement. Machine learning models analyze these data to identify patterns of engaged behavior, allowing educators to adjust instructional strategies in real-time to recapture or enhance student engagement (Mangaroska et al., 2020; Worsley, 2018).
Mental Workload: understanding a student’s mental workload is crucial for optimizing learning outcomes. Overloading students can lead to cognitive fatigue, while underloading may result in boredom. MMLA employs sensors like EEG (electroencephalography) to measure brain activity patterns associated with cognitive workload. By correlating these patterns with task difficulty and performance, MMLA can predict when students are likely to experience cognitive overload or underload, guiding the pacing of instruction and the complexity of learning materials (Ding et al., 2020).
Affective States: Affective states such as frustration, boredom, anxiety, and excitement significantly impact learning (D’Mello et al., 2012). MMLA leverages facial expression analysis, skin conductance sensors, and posture detection to infer students’ emotional states during learning activities. These affective analytics enable educators to identify when students might need additional support or a change in learning activities to maintain motivation and reduce negative emotions that hinder learning (Sharma & Giannakos, 2020).
Learning Outcomes: By integrating data from diverse sensors, MMLA can provide early predictions of learning outcomes. This integration includes analysis of academic performance data (e.g., quiz scores, assignment completion rates) with sensor-based data on engagement levels, emotional states, and environmental factors. Predictive analytics models use this comprehensive dataset to forecast individual and class-wide learning achievements, allowing for interventions to support students at risk of underperforming (Sharma & Giannakos, 2020).
The integration of MMLA and MSEs
In the evolving landscape of educational technology, the integration of MSEs and MMLA presents a promising frontier for enhancing learning experiences (Cosentino & Giannakos, 2023). MSEs, with their rich history in therapy, rehabilitation, and education, leverage the power of multisensory stimuli to deepen engagement and improve the retention of information. This approach draws on a robust body of research and learning theories that underscore the importance of engaging students in goal-driven activities. On the other hand, the advent of advanced sensor technologies and data analysis techniques has given rise to MMLA, a field dedicated to exploring the synergies among various sensory modalities and their impact on learning outcomes (Abrahamson et al., 2021). By harnessing data from verbal, visual, and gestural interactions, MMLA seeks to offer a holistic view of the learning process, considering cognitive, emotional, and social dimensions. This integrated approach not only aims to tailor educational experiences to individual needs and preferences, but also strives to enhance overall educational outcomes by addressing the complex and multifaceted nature of learning. Together, MSE and MMLA embody a synergistic relationship, where the enrichment of sensory experiences in learning environments is meticulously analyzed and optimized through data-driven insights, setting a new standard for personalized and effective education (Sharma & Giannakos, 2020; Blikstein and Worsley, 2016).
However, the relationship between different learning states and students’ learning outcomes is complex, and more research is needed to investigate the role of specific interaction modalities in supporting learning in MSEs. Recent advances in machine learning and artificial intelligence have enabled researchers to develop models that can analyze the data collected from MSEs and provide insights into how students interact with multisensory educational interventions. In particular, deep learning techniques have shown promising results in modeling learners’ interactions and predicting their performance (Blanchard & Gerstner, 2019; Choi & Lee, 2018). For instance, (Blanchard & Gerstner, 2019) proposed a deep learning model that can predict learners’ engagement and cognitive load levels in real-time using multimodal data collected from sensors and logs. Similarly, (Choi & Lee, 2018) developed a deep learning-based algorithm to predict learners’ emotional states from facial expressions and physiological signals collected from sensors.
Despite these promising developments, several challenges still need to be addressed to fully realize the potential of MSEs in education. For example, through personalized and immersive learning experiences that incorporate multiple sensory modalities, MSEs can enhance education. The collective use of log-based and sensor-based analytics can provide a more comprehensive understanding of learners’ interactive experiences and inform the design of more effective educational interventions. As a relatively new field, the investigation of how the sensory modalities employed in MSEs impact the students’ learning experience including their affective states still presents many challenges related to data collection, analysis, and interpretation.
Hypothesis development
The relationship between students’ states has been found to have an important role in their learning experience, and their mastery level (Lee-Cultura, Sharma, and Giannakos, 2023; Cosentino & Giannakos, 2023; Sharma & Giannakos, 2020). Learning involves the interaction of cognitive, emotional, and physiological elements. In this context, learner affective variables, such as concentration, fatigue, difficulty, time pressure, and motivation are considered a very important part of the knowledge acquisition process (Hourcade et al., 2018). Previous research has shown that when students have high levels of stress (when it is not under control), are more likely to provide a wrong response (Harada, 2002; Herborn et al., 2015). Stress might make challenging for students to effectively retrieve information, it might also interfere with attentional processes and concentration, leading to errors, omissions, or misinterpretations of information, resulting in lower correctness in the learning tasks (Sharma et al., 2019). We therefore hypothesize that:
H1: Students with higher stress levels have significantly lower correctness.
When students experience high anxiety, they might find difficult to focus and reasoning on the tasks they engage with. In the context of assessments, anxiety can increase distractibility, difficulty in organizing thoughts, and even blanking out on previously learned material, all of which contribute to lower correctness rates (Rodrigo et al., 2009; Di Leo et al., 2019). Anxiety can also increase self-doubt and lower confidence, which can further affect learning outcomes negatively and may have implications for students’ academic trajectory (Lee-Cultura et al., 2020). We therefore hypothesize that:
H2: Students with higher anxiety levels have significantly lower correctness.
Previous studies demonstrated that fatigue negatively impacts cognitive processes, making it harder for students to retain new information accurately (Lee-Cultura, Sharma, Cosentino, Papavlasopoulou, and Giannakos, 2021). This may be because fatigue negatively affects motivation and cognitive load during the learning process. In the context of learning technologies, student fatigue can lead to reduced motivation by decreasing the fulfillment of these basic psychological needs (Ryan & Deci, 2020; Chen & Qin, 2023). We have therefore formulated the following hypothesis:
H3: Students with higher fatigue levels have significantly lower correctness.
High engagement levels during the student learning exprience may foster motivation, attention and active participation (Soland, Jensen, Keys, Bi, and Wolk, 2019); Students that are actively engaged in the learning process are able to interact with the task thoroughly, clarify misunderstandings, and reinforce their comprehension, which eventually results in higher correctness (Lindgren, Tscholl, Wang, and Johnson, 2016). We have therefore formulated the following hypothesis:
H4: Students with higher engagement levels have significantly higher correctness.
When arousal levels are high, students might experience an overload of the working memory capacity. While moderate levels of arousal can enhance cognitive performance by increasing alertness and motivation (Malmberg et al., 2022; Pijeira-Díaz et al., 2018), high arousal levels can undermine correctness in learning by impairing attentional control, decision-making, information processing, and increasing error rates (Dawson et al., 2007; Pizzie & Kraemer, 2021). These negative emotions can further impair cognitive functioning and interfere with learning correctness by disrupting attention and information processing. Hence, it is hypothesized that:
H5: Students with higher arousal levels have significantly lower correctness.
Lastly emotional regulation might helps students to manage distractions and maintain focus on learning tasks. By effectively regulating their emotions, students can minimize the impact of negative emotions that might otherwise interfere with their concentration and cognitive processes, allowing them to achieve higher correctness in learning. When students are able to regulate their emotions, they are better equipped to engage in critical thinking and problem-solving, leading to a more accurate understanding (van den Hoogen et al., 2008). Emotional regulation could also affects academic performance by enhancing self-awareness and self-regulation which contribute to higher correctness (Yu et al., 2022); We therefore hypothesized that:
H6: Students with higher emotional regulation levels have significantly higher correctness.
In the context of this study, we were interested in investigating how different interaction modalities given by a MSE moderate the relationship between students’ states and their mastery. To investigate this objective, based on five main interaction modalities explained in section "Interaction modalities" ("card", "feet", "hands", "voice", and "wand"), we have formulated the following hypotheses:
H7a: Interaction modalities have a significant moderating effect on the relationship between students’ stress and mastery.
H7b: Interaction modalities have a significant moderating effect on the relationship between students’ anxiety and mastery.
H7c: Interaction modalities have a significant moderating effect on the relationship between students’ fatigue and mastery.
H7d: Interaction modalities have a significant moderating effect on the relationship between students’ engagement and mastery.
H7e: Interaction modalities have a significant moderating effect on the relationship between students’ arousal and mastery.
H7f: Interaction modalities have a significant moderating effect on the relationship between students’ emotional regulation and mastery. The research hypotheses are summarized in Fig 1.
[See PDF for image]
Fig. 1
Path diagram of the research hypotheses. The black lines depict direct effects and the dotted lines the moderation effects
The main reason for using moderator analysis is threefold. First, moderator analysis helps us to understand the relationship between MMLA and MSE conditionally based on which type or subgroup of interaction is used. Second, Moderator analysis enables us to achieve better prediction and interpretation of models. Finally, third, moderation analysis provides enhanced practical guidelines for educational technology designers, students, and educators.
The system
MSEs are typically "built-in" to a specific space (e.g., a room or a portion of a room) and incur significant installation expenses. (Agosta et al., 2015; Gelsomini et al., 2018; Gelsomini & Leonardi, 2020). In the context of this study, a portable MSE-enabling technology was developed, called "MOVES", that overcomes the MSEs’ hardwired nature. MOVES (https://dl.acm.org/doi/abs/10.1145/3585088.3594493) has a solidly and flexibly built structure. To adapt to various situations, the base is equipped with wheels that can be locked and that make it simple to move and reposition. The platform includes a mini-PC that reads motion-sensing (RGB and depth) video and audio streams (Orbecc Astra Pro) and sends them to two Ultra Short Throw LED projectors (projecting the interacting area on the floor and on the wall). Wi-Fi routers provide wired connectivity to PCs, creating a Wi-Fi network on which a controlling device (smartphone, tablet, remote controller) can connect to control the experience. The MOVES platform is equipped with SENSEi software (Gelsomini, 2023). SENSEi is a suite of software modules that enables the PC to manage several input and output devices. In a lower level, these devices communicate with the PC via traditional drivers. A higher level of access is provided by simple, homogeneous, and intuitive APIs for novices and professionals alike. SENSEi thus enables this simplification by accessing device providers’ Software Development Kits (SDK) and translating them into a standardized documented form. As a set of modules, the software communicates with sensing and actuation devices and displays contents through a final viewable layer.
Interaction modalities
MOVES offered five types of interaction modalities, including hand, feet, voice, card, and wand as shown in Fig.2).
[See PDF for image]
Fig. 2
The five interaction modalities of MOVES
Card interaction (Fig. 2A) The player is positioned in proximity of MOVES with the wall screen in front of them and the floor screen projected beneath their feet. The MOVES platform is equipped with an RFID reader. Both projections show the activity, with the front projection (i.e., wall) showing the main task and its solutions. As soon as the task begins and the assignment is displayed, users are prompted to select an RFID card that corresponds to a letter located near the answer content. Once the user selects the card and places it in the reading zone, the content is automatically selected, and the answer is deemed as given.
Feet interaction (Fig. 2B) The player stands 1 to 3 m from MOVES, facing the wall screen ahead and the floor screen projected beneath their feet. The activity is displayed on both projections, with the primary task being presented on the front projection and answers being displayed on the floor projection. The movement of both user’s feet is tracked, and an average is calculated to show a circular shape beneath the user’s body, following their movements. When the shape is positioned over an answer, the corresponding content scales up to 120% of its original size. To confirm the selection of the content, the user must hover over it for 3 s while a progress wheel is displayed.
Hands interaction (Fig. 2C) (hereinafter hand) The player is positioned in front of MOVES, facing the wall screen ahead and the floor screen projected beneath their feet at a distance of 1 to 3 m. The activity is depicted on both the front and floor projections, with the primary task and answers displayed on the front projection. Both user’s hands are tracked and displayed on the front projection in the form of a hand cursor. To select content displayed on the front projection, the user must move their hand to match the selected content. Upon hovering the hand cursor over an answer, the corresponding content scales up to 120
Voice interaction (Fig. 2D) The player is positioned in front of MOVES, facing the wall screen ahead and the floor screen projected below their feet. The activity is displayed on both projections, with the primary task and answers depicted on the front projection. Once the activity commences, users may speak out the content in full form (e.g., "I want the tiger" or "tiger is the answer"), straight form (e.g., "tiger"), or ID form (e.g., "letter B" corresponding to the tiger). The system recognizes the answer in less than a second and selects the referenced content.
Wand interaction (Fig. 2E) The player is positioned in front of MOVES, ideally with the wall screen ahead and the floor screen projected under their feet. The activity is displayed on both projections, with the main assignment and answers depicted on the front projection. When the activity starts and the assignment is shown, the user holds an air mouse (referred to as the magic wand) with their preferred hand and moves a cursor displayed on the front screen. When the wand cursor goes over one of the answers, the content scales up to 120%. To select the content, the user presses the air mouse main button, and the answer is considered given.
While all five modalities were designed to achieve the same task, selecting an answer, they could differ in terms of physical demands, familiarity, and complexity, which may have influenced students’ performance and affective responses. These differences were intentionally preserved to explore a range of interaction styles within MSEs; however, they also highlight how interaction modality design can shape learning experiences through varying levels of intuitiveness, effort, and engagement. This diversity was key to our analysis of how each modality moderated the relationship between students’ affective states and mastery.
Methods
Participants
WThe participants in our study comprised 175 students from primary school (103 M and 72 F) from 6 to 10 years old (mean age = 8.12 years, SD = 1.48 years). There was no prior experience with MSEs among the students. The guardians of the students signed informed consent forms prior to their participation. The students were also rewarded with stickers representing characters from the game as an additional reward. The national human research ethics organization approved all ethical procedures. Students’ participation and data collection were conducted after approval from the national Data Protection Official for Research [IRB information anonymized], following all the regulations and recommendations for research with students. In particular all collected data were anonymized at the point of collection by assigning randomized participant IDs, with no personally identifiable information stored alongside the physiological or performance data. The mapping between participant names and IDs was stored in a separate, encrypted file, accessible only to the core research team for organizational and consent-tracking purposes. To ensure data security, we followed institutional data protection protocols: all data were stored on secure, access-restricted servers, with encrypted backups. No raw physiological data were shared externally; only aggregated and anonymized metrics were used for analysis and publication.
Procedure
We carried out the study in summer 2022, in ten public primary schools located in [omitted] over a period of two months (June-July). To ensure a fair selection, educators were instructed to randomly group five students and supervise each group as they moved back and forth from the study area. The facilitator was responsible for welcoming the students, introducing them to the study, initiating the experiment, and responding to their queries. Additionally, the facilitator was expected to be transparent and honest with the students, explaining the research’s purpose and how the sensors functioned. During the study, an observer was present, carefully noting the students’ behaviors, difficulties, and concerns without disrupting the study. The study area was a dedicated classroom within each school to minimize external distractions. We conducted a between-subjects experiment to examine how students experienced five different interaction modalities, and each student tried one condition. After receiving brief information about the five modalities, the students selected the interaction modality (self-selection of the condition) with which they wished to engage. The study session consisted of the following stages:
Introduction: The main character (called SENSEi) avatar was introduced to students to aid in learning new subjects. SENSEi described an imaginary world consisting of five islands, with each island representing one of the human senses.
Registration: At the entrance, the main character (SENSEi) asked each student to introduce themselves and scan an RFID tag. The observer recorded the students’ names and matched them with the scanned tags.
Selection: Each student was individually called to play while the others waited outside the playground. SENSEi assigned an inhabitant of an island to each student who offered students a choice of topics (geography, math, science, language, music, and history), graphic representation (cartoon or realistic), and interaction modality. Before choosing an interaction modality, the students experimented with them in a tutorial with dummy material (see Fig. 2).
Activity: Student choices generated the activity, and mechanics were chosen at random from four categories. Each consisted of three questions of varying complexity to ensure internal validity. The learning subjects and questions were developed with educators using the government education curricula to avoid overly simple or difficult content for the students.
Quiz: A question with three alternative responses is displayed. Participants must choose the correct response.
Sequence: A number of items are displayed. Participants must place the items in the correct sequence.
Match: A series of items and labeled boxes are given. Participants have to match the items with the corresponding boxes.
Memory: A series of covered items are shown. Participants have to find a couple of matching items.
Conclusion: After all five students completed the activity, the characters from the story greeted them, and the students exited the room.
Data collection
Students’ activity sessions were recorded using a Logitech video camera and two additional sensor devices: Empatica E4 wristbands and the Astra Pro camera. Event data and activity analytics were also collected from system logs.
Activity Logs: Log files were automatically collected to store participant information (e.g., identification, initial preferences, learning resources, responses, and correctness).
Skeleton Logs: Nuitrack software (3DiVi, 2023) was used to collect these data using the Orbbec Astra motion-sensing device at a sampling rate of 10Hz. These represented the 3D position of 20 joints: head, neck, spine, hip-center, and left and right hands, wrists, elbows, shoulders, feet, ankles, knees, and hips.
Physiological data: Empatica E4 wristbands were used to capture physiological data from students, including Heart Rate Variability—HRV (1Hz), Electrodermal Activity—EDA (64Hz), Skin Temperature—T (4Hz), and Blood Volume Pulse—BVP (4Hz).
Measurements
We used the following sensor-based and log-based measurements in this contribution. Most of these measurements have been used to capture and analyse learners’ experiences and processes (Di Lascio et al., 2018; Sharma & Giannakos, 2020; Lee-Cultura et al., 2023):
Correctness Proportion of correct responses and the total responses for each student. This was used as a proxy of students’ mastery.
Stress (sensor-based) Computed as HRV’s increasing slope. The more positive the slope of the HRV is in a given time window, the higher the stress is (Taelman et al., 2009). The HRV has been used to measure stress in educational (Sharma et al., 2019) and problem-solving (Mirjafari et al., 2019) contexts. In several recent contributions (independent studies, meta-analysis and systematic literature reviews), HRV-based measurements have been showed to be appropriate measures (Seipäjärvi et al., 2022; Bravi et al., 2013; Antoun, Edwards, Sweeting, and Ding, 2017; van Loon et al., 2022). Anxiety (sensor-based) Computed using the slope of the High-Frequency Component of HRV. The lower slope of this component in a given time window is, the lower anxiety is (Chalmers et al., 2014; Gorman & Sloan, 2000).
Emotional Regulation (ER) (sensor-based) Computed as the rate of arrival of HRV peaks as suggested by (Berntson and Cacioppo, 2004; Williams et al., 2015). The lower the arrival rate of HRV peaks is, the higher is the emotional regulation.
Arousal (sensor-based) Computed by the increasing slope of EDA. The more positive the slope of the EDA in a given time window is, the higher the arousal is (Hedman et al., 2012; Lee-Cultura, Sharma, Papavlasopoulou, Retalis, and Giannakos, 2020).
Engagement (sensor-based) a linear combination of EDA’s increasing slope and the arrival rate of EDA peaks. The more positive slope of the EDA and the higher the rate of arrival of peaks in a given time window is, the higher the engagement is (Hasson et al., 2008; Leiner et al., 2012).
Fatigue (sensor-based) Proportional to the Jerk in the movement. Jerk is computed as the time derivative of the acceleration of the Joint’s movements (alias fourth derivative of displacement), and represents the average jerk of all the joints. It is the inverse of energy spent (Guigon et al., 2007).
Data analysis
In this contribution, we address the following question: “How do the interaction modalities moderate the relationship between students’ learning states (as inferred from sensor-based MMLA) and their mastery level (as inferred from their correctness) in MSEs?”. Figure 1 shows the relation between the constructs, measurements, and variables used in this study.
To find how the interaction modality affects the relation between students’ learning states and their mastery level, we chose to conduct moderation analysis (Lance, 1988). Moderation describes a situation in which the relationship between two constructs (e.g., students’ learning states and their mastery) is not constant but depends on the values of a third variable (e.g., interaction modality), referred to as a moderator variable (Hair Jr et al., 2021). In terms of an Analysis of Covariance (ANCOVA), this variable is added as a covariate that might or might not have a direct effect on the dependent variable, but when combined with the main independent variable, shows a significant interaction effect.
In the present analyses, we use sensor-based measurements as the independent variables; mastery measurements as the dependent variable; and interaction modality as the potential moderator variables. Therefore, to respond to the RQ and the hypotheses of this paper, we created an ANCOVA model for each of the sensor-based measurements (i.e., stress, anxiety, fatigue, engagement, and ER) as the independent variables and the mastery measurement (i.e., correctness) as the dependent variable; along with the interaction modality (i.e., hand, feet, card, voice, and wand) as the covariate.
To confirm a significant moderation effect, we would inspect the ANCOVA models for two effects: (1) a significant direct effect between the dependent and the independent variables; and (2) a significant interaction effect of the covariate and independent variables on the dependent variable. The moderator selected in this analysis has more than two categories, therefore once we establish an overall moderation effect, we conduct post-hoc pairwise moderation effect analysis to gain a deeper understanding of the role of interaction modality as a moderator. Since there were multiple comparisons, we applied Bonferroni corrections. For these corrections, we divide the acceptable p-value by the number of hypotheses tested. Therefore, the acceptable p-value for this contribution is 0.003.
Results
This section presents the results of the moderation analysis, focusing on the MMLA measurements:
Stress
From table 1, we can observe that there is a negative relationship between stress and correctness (H1 is verified); we also observe that there is a significant moderation effect of the interaction modalities on the relationship between stress and correctness. (H7a is verified). Further, as a post-hoc moderation effect analysis, we observe (table 2) that the relationship between stress and correctness is significantly more negative for feet, card, and wand than those for hand and voice (Fig. 3 and Table 2). That is when the students are interacting via feet, card, or wand their stress has a significantly stronger negative relationship with students’ mastery of responses than in the cases when the students are interacting via hand and voice.
Table 1. Moderator effect calculation for stress
Estimate | Std. Error | t val | p val | |
|---|---|---|---|---|
stress | 0.7350 | 0.2194 | 3.35 | 0.001 |
IM–feet | 0.8959 | 0.0458 | 19.57 | 0.0001 |
IM–hand | 0.7285 | 0.0617 | 11.82 | 0.0001 |
IM–card | 0.8371 | 0.0426 | 19.66 | 0.0001 |
IM–voice | 0.7330 | 0.0690 | 10.62 | 0.0001 |
IM–wand | 0.9142 | 0.0328 | 27.89 | 0.0001 |
interaction (hand) | 0.5423 | 0.2690 | 2.02 | 0.04 |
interaction (card) | 0.2979 | 0.3101 | 0.96 | 0.33 |
interaction (voice) | 0.6288 | 0.2858 | 2.20 | 0.02 |
interaction (wand) | 0.0732 | 0.2767 | 0.26 | 0.79 |
Table 2. Pair-wise moderator effect for the different interaction modalities. The table shows t-values for stress
feet | hand | card | voice | |
|---|---|---|---|---|
hand | 2.7 * | – | – | – |
card | 1.2 | 2.2* | – | – |
voice | 2.8* | 0.2 | 2.0 | – |
wand | 0.4 | 2.3* | 0.5 | 3.6* |
Anxiety
From table 3, we can observe a significant negative relationship between anxiety and correctness (H2 is verified); we also observe that there is an interaction effect of modalities and anxiety on correctness. Therefore, we can conclude that there is a significant moderation effect of the interaction modalities on the relationship between anxiety and correctness (H7b is verified). In particular, the relation between anxiety and correctness is significantly less negative for feet, card, and wand than those for hand and voice (Fig. 3 and Table 4). That is when the students are interacting via feet, card, or wand their anxiety has a weaker relationship with the incorrectness of responses than in the cases when the students are interacting via hand and voice.
Table 3. Moderator effect calculation for anxiety
Estimate | Std. Error | t val | p val | |
|---|---|---|---|---|
anxiety | 0.3857 | 0.1568 | 2.46 | 0.0150 |
IM–feet | 0.8996 | 0.0582 | 15.45 | 0.0001 |
IM–hand | 1.0264 | 0.0520 | 19.76 | 0.0001 |
IM–card | 0.9385 | 0.0599 | 15.68 | 0.0001 |
IM-voice | 0.9550 | 0.0506 | 18.87 | 0.0001 |
IM-wand | 0.9847 | 0.0472 | 20.86 | 0.0001 |
interaction (hand) | 0.5460 | 0.1998 | 2.73 | 0.007 |
interaction (card) | 0.0879 | 0.2206 | 0.40 | 0.69 |
interaction (voice) | 0.4430 | 0.2160 | 2.05 | 0.04 |
interaction (wand) | 0.1444 | 0.1976 | 0.73 | 0.46 |
Table 4. Pair-wise moderator effect for the different interaction modalities. The table shows t-values for anxiety
feet | hand | card | voice | |
|---|---|---|---|---|
hand | 2.4 * | – | – | – |
card | 0.4 | 2.0 | – | – |
voice | 2.9* | 0.5 | 2.0 | – |
wand | 0.8 | 2.1* | 0.3 | 2.2* |
Fatigue
There is no direct relation between fatigue and correctness, therefore there is no moderator effect (Fig. 6 in the Appendix and table 5). Thus, H3 and H7c can not be tested.
Table 5. Moderator effect calculation for fatigue
Estimate | Std. Error | t val | p val | |
|---|---|---|---|---|
fatigue | 0.0044 | 0.0086 | 0.51 | 0.61 |
IM–feet | 0.7352 | 0.0647 | 11.36 | 0.0001 |
IM–hand | 0.6502 | 0.0668 | 9.73 | 0.0001 |
IM–card | 0.7668 | 0.0446 | 17.19 | 0.0001 |
IM–voice | 0.6644 | 0.0559 | 11.88 | 0.0001 |
IM–wand | 0.8325 | 0.0285 | 29.22 | 0.0001 |
interaction (hand) | 0.0028 | 0.0181 | 0.16 | 0.87 |
interaction (rid) | 0.0045 | 0.0094 | 0.48 | 0.63 |
interaction (voice) | 0.0015 | 0.0127 | 0.12 | 0.90 |
interaction (wand) | 0.0125 | 0.0093 | 1.36 | 0.17 |
Engagement
From table 6, we can observe that there is a positive direct effect between engagement and correctness (H4 is verified). However, there is no moderation effect because there is no moderation effect of the interaction modalities on the relationship between engagement and correctnesss (Fig. 6 in the Appendix and table 6). Therefore, H7d is rejected.
Table 6. Moderator effect calculation for engagement
Estimate | Std. Error | t val | p val | |
|---|---|---|---|---|
engagement | 3.4724 | 1.0151 | 3.42 | 0.0008 |
IM–feet | 0.0374 | 0.2140 | 0.17 | 0.86 |
IM–hand | 0.2646 | 0.0709 | 3.73 | 0.0003 |
IM–card | 0.3156 | 0.2144 | 1.47 | 0.14 |
IM–voice | 0.4250 | 0.1086 | 3.91 | 0.0001 |
IM–wand | 0.1351 | 0.1348 | 1.00 | 0.31 |
interaction (hand) | 0.3192 | 1.2123 | 0.26 | 0.79 |
interaction (card) | 1.4213 | 1.4053 | 1.01 | 0.31 |
interaction (voice) | 0.9549 | 1.4188 | 0.67 | 0.50 |
interaction (wand) | 0.5370 | 1.1782 | 0.46 | 0.64 |
Arousal
There is no direct relation between arousal and correctness, therefore there is no moderator effect (Fig. 6 in the Appendix and Table 7). Thus, H5 and H7e can not be tested.
Table 7. Moderator effect calculation for arousal
Estimate | Std. Error | t val | p val | |
|---|---|---|---|---|
arousal | 0.0114 | 0.1963 | 0.06 | 0.95 |
IM–feet | 0.7717 | 0.1125 | 6.86 | 0.0001 |
IM–hand | 0.6265 | 0.0742 | 8.45 | 0.0001 |
IM–card | 0.7705 | 0.0787 | 9.79 | 0.0001 |
IM–voice | 0.7942 | 0.0873 | 9.10 | 0.0001 |
IM–wand | 0.8234 | 0.0723 | 11.38 | 0.0001 |
Interaction (hand) | 0.0703 | 0.2409 | 0.29 | 0.77 |
Interaction (card) | 0.0022 | 0.2460 | 0.01 | 0.99 |
Interaction (voice) | 0.1790 | 0.2535 | 0.71 | 0.48 |
Interaction (wand) | 0.0500 | 0.2344 | 0.21 | 0.83 |
Emotional regulation
From table 8, we observe a positive effect between ER and correctness (H6 is verified); we also observe that there is an interaction effect of ER and interaction modalities on correctness (H7f is verified. Therefore, we can conclude that there is a moderator effect of interaction modalities on the relationship between ER and correctness. The relation between ER and correctness is significantly less positive for feet, card, and wand than those for hand and voice (Fig. 3, table 9). That is when the students are interacting via feet, card, or wand their ER has a weaker relationship with the correctness of responses than in the cases when the students are interacting via hand and voice.
Table 8. Moderator effect calculation for emotional regulation
Estimate | Std. Error | t value | p value | |
|---|---|---|---|---|
emotional regulation | 0.1659 | 0.1787 | 0.93 | 0.35 |
IM–feat | 0.6956 | 0.0784 | 8.88 | 0.0001 |
IM–hand | 0.3328 | 0.0507 | 6.57 | 0.0001 |
IM–card | 0.6970 | 0.0707 | 9.85 | 0.0001 |
IM–voice | 0.4694 | 0.0565 | 8.31 | 0.0001 |
IM–wand | 0.6821 | 0.0511 | 13.36 | 0.0001 |
Interaction (hand) | 0.7574 | 0.2237 | 3.38 | 0.0009 |
Interaction (card) | 0.0213 | 0.2558 | 0.08 | 0.93 |
Interaction (voice) | 0.4786 | 0.2324 | 2.06 | 0.04 |
Interaction (wand) | 0.1329 | 0.2215 | 0.60 | 0.54 |
Table 9. Pair-wise moderator effect for the different interaction modalities. The table shows t-values for emotional regulation
feet | hand | card | voice | |
|---|---|---|---|---|
hand | 3.1 * | – | - | – |
card | 0.1 | 2.7* | – | – |
voice | 2.6* | 1.3 | 2.1* | – |
wand | 0.6 | 3.1* | 0.5 | 2.1* |
[See PDF for image]
Fig. 3
The moderating effect of interaction modalities on the relationship between students’ states (stress,anxiety and ER) and mastery
Discussion
From the analysis, we identified that different interaction modalities moderate the relationship between students’ affective states and their performance (mastery level). Depending on the modality used, stress, anxiety, and emotional regulation (ER), as captured by sensor-based MMLA, show different relationship with students’ correctness. This opens up several important considerations for how interaction modalities can support students’ experiences in immersive educational systems like MSEs. In this section, we discuss how these findings inform the design of educational MSEs and their implications for practice and theory. Interaction modalities and interface metaphors have long been recognized in both learning technology and HCI communities as essential to facilitating learning (Neale & Carroll, 1997; Carroll and Mack, 1985). They do so by leveraging learners’ existing mental models and prior knowledge, enabling faster understanding and more intuitive engagement (Bruner, 1960; Streitz, 1988).
Our study revealed that the interaction modalities used in our MSE fall into two distinct groups (see Fig. 4). The first group consists of voice and hands, while the second includes feet, cards, and wand. Voice and hand interactions are more familiar in students’ everyday digital experiences, such as through gaming systems like the Xbox video game console (Kandroudi and Bratitsis, 2012) or smart assistants like Siri, Alexa, and Google (Garg et al., 2022; Jung, Kim, So, Kim, and Oh, 2019). This familiarity may explain why, when using these modalities, students’ stress levels were less negatively correlated with performance.
However, the same group showed a stronger negative correlation between anxiety and correctness, suggesting that while students may understand these modalities well, they might also feel more pressure to perform accurately. Interestingly, a more positive correlation was found between the correctness of their answers and controlling highly emotional reactions such as calming emotions, focusing on tasks, refocusing attention on a new task, and controlling impulses (all of which are emotional regulation processes).
In contrast, Group 2 (feet, card, wand) includes less common modalities with which students generally have less experience. These modalities may require newer skills or knowledge, which can be challenging to acquire, they may also perceive these modalities less intuitive and more difficult to use at first, which can increase frustration and stress (Velloso, Schmidt, Alexander, Gellersen, and Bulling, 2015; Jordan, 2020; Shih, 2011). Our results confirmed that when students interacted with these modalities their stress level is more closely related to their wrong replies and their emotional regulation affect their correctness less positively compared to Group 1.
[See PDF for image]
Fig. 4
Summary of the moderator effect the interaction modalities have on the relation between sensor-based MMLA and students’ correctness
Implications for design and practice
The two groups we identified in our results offer clear guidance for adapting learning experiences based on students’ affective states. Designing MSEs that can dynamically respond not just to what students are learning, but how they are interacting, opens opportunities for more personalized and supportive educational environments. By combining user modeling with content-behavioral and interactional data, learning systems could provide adaptive interventions. For instance, when the kids are highly stressed and their answers are wrong, it could be advantageous to offer not only content-based feedback (e.g. tips, decrease the difficulty) (Csikszentmihalyi et al., 2018) but also interactions that affect less negatively the relation between stress and their answers. This study encourages practitioners and scholars to focus on the interactions’ design for young students in digital learning experiences such as MSE. Children’s interaction capabilities are influenced by age-related factors like motor skills and cognitive load, which differ significantly from those of adults (Markopoulos et al., 2008; Sherwin & Nielsen, 2019). These must be reflected in the design of learning activities–not only in terms of content and interface layout, but also in the physical and cognitive effort required for interaction.
Interaction reaction time is another crucial factor, the length of time a posture is held to select content or the type of feedback they get might influence the students’ frustration leading to bad performance (Van Duijvenvoorde, Zanolie, Rombouts, Raijmakers, and Crone, 2008). Moreover, students may be confused by the different representations used to support different interaction modalities (e.g., hand, full body, leg) since those representations have a high impact on how the interaction modality operates (e.g., synchrony, congruency) and thus can hinder the learning experience or even lead to low performance (e.g., fail to make the selection the student wanted) (Lee-Cultura, et al., 2020).
To help contextualize our findings, we present four example scenarios (see Fig. 5) based on the measurements we utilized in the study to assist readers comprehend how our findings could contribute to the moderating role of various interaction modes in MSE context. When reading Fig. 5, it’s important to note that the consequences of ’low’ and ’high’ levels for each MMLA measurement may vary. For example, for measurements such as stress and anxiety, ’high’ levels are generally hindering learning, while for ER, ’high’ levels support learning engagement (and learning in general).
[See PDF for image]
Fig. 5
Scenarios on the connection between sensor-based MMLA and students’ mastery. For MMLA measures like stress and anxiety, ’high’ levels are generally considered unfavourable conditions, while for measures of ER, ’high’ levels are seen as favourable conditions
The first scenario where performance is low, stress or anxiety is low, or ER is high makes students bored and lose motivation. According to recent findings from individual learning situations, students’ boredom may be undesirable to both academic and task-based performance (Dowd et al., 2015; Baker et al., 2010). In other circumstances, students might get distracted from the task, increasing feelings of boredom and not participating in active problem solving (Sharma et al., 2022). Scenario number two indicates the case where students are frustrated or confused because they are experiencing low performance, high stress or anxiety, or low ER. Previous studies indicate that the feeling of frustration or confusion was negatively correlated with the learning outcome, and most of the time are also associated when students are interacting/engaging with learning contents that are complex (Rodrigo et al., 2009; Liu et al., 2013; Di Leo et al., 2019). When the performance is high, the stress or the anxiety low, or the ER high we referred to group number three, in this case, the students feel relaxed, and in control of the situation. This is generally a positive state if lasts the right amount of time because it can easily become apathy and boredom if the students do not keep their motivation high. Indeed scenario number four brings back the students to a high level of stress or anxiety, or ER without affecting their performance, this is the optimal state where the students feel focused and happy, influencing the learning outcome in a positive way (van den Hoogen, IJsselsteijn, and de Kort, 2008). Understanding when and how an interaction modality is moderating the relation between students’ affective state and their performance, we could take a step further from the behavioral or content interventions (e.g., content-based hints, decreasing difficulty of the task, reinforcement feedback (Csikszentmihalyi, 1997; Lee-Cultura et al., 2022; Sharma et al., 2021)). Indeed, providing students with interactional cues based on the scenario they are in (Fig. 5), might help them better understand the affordances of the interaction and prevent bad performance.
The volume of data that MSEs can collect offers strong empirical support for designing personalized learning experiences. This goes beyond simply adapting content and difficulty levels–it also involves integrating interaction modality design as a core component of effective learning activities.
In addition to supporting students, sensor-based MMLA can provide meaningful feedback to teachers. The rich data streams available through MSEs can help educators monitor students’ learning progress in real time and intervene when needed to keep learners in a productive state of flow (Verbert et al., 2013; Rienties et al., 2018). Learning analytics dashboards serve as a valuable tool for visualizing this data, enabling teachers to make informed, data-driven decisions by offering both formative and summative insights. However, to ensure their usefulness, these dashboards must be carefully designed with educators’ needs in mind (Dourado et al., 2021; Kaliisa et al., 2021; Sawyer, 2014), including the consideration of interaction modalities as part of the learning intervention space.
Beyond individual feedback, our findings suggest group- and class-level intervention strategies. For instance, in a collaborative setting, participants using hand and voice interactions might require lighter stress-aware feedback, while those using wand, feet, or card modalities might benefit from more supportive scaffolding. These distinctions can inform the design of group-level feedback mechanisms, ensuring that interaction modality is considered alongside content and behavior.
Moreover, while designing a tool that supports a class-wide behaviour, such as a learning analytics dashboard for the teacher, the findings for individual cases can be generalised here as well. The main idea is to first inform the teacher about the moderator effect. This is necessary to provide a proper onboarding for the teacher (Lee-Cultura et al., 2023). Once the teacher has understood the role of interaction mode as the moderator then by displaying the interaction modality (or the group, e.g., group 1 with hand and voice) for each participant one can easily support the scaffolding provided by the teacher whenever necessary.
Limitations and ethical implications
First, regardless of the relatively large sample size, we must keep in mind that long-term use and different populations (age, expertise) may affect the results (i.e., a threat to generalizability). Second, even though in our study, we used measurements based on literature and tested the most common interaction modalities (except for touch interaction), we need to highlight that these methodological decisions entail certain limitations (e.g., additional interaction modalities, measurements to account for students’ states, and factors we did not consider) for the outcomes of this study. Third, we only focused on the overall aggregated measurements. This limits one of the great potentials of MMLA which is to consider how students’ states and mastery evolves over time. Therefore, in the future, we should consider collecting data over time (longitudinal study) and use temporal analysis to understand how students’ states and interaction behavior evolve (e.g., what happens when students fully master a certain interaction or a topic).
As the use of sensor-based technologies becomes more prevalent in educational settings, especially with young learners, it is essential to consider the ethical implications surrounding privacy, consent, and data sensitivity. Collecting detailed physiological data through wearable devices introduces potential risks related to data security, misuse, and informed participation. In our study, we followed ethical protocols by securing informed consent from students’ guardians and receiving institutional approval. However, we also recognize the importance of going beyond procedural compliance. Following recommendations from recent literature on the ethics of Multimodal Learning Analytics (MMLA) (Alwahaby et al., 2022; Sharma & Giannakos, 2021), we invested time in clearly communicating the purpose, benefits, and potential risks of the study to children, parents, and educators. Furthermore, we adopted data minimization principles, ensuring that only the information necessary for analysis was collected and that all data was stored securely. These efforts are critical to maintaining trust, ensuring transparency, and protecting students’ rights in sensor-rich learning environments. Future work should continue to explore child-centered approaches to ethical design, including how to give young learners a voice in how their data is collected, interpreted, and used.
Conclusion
In this study, we explore how different interaction modalities moderate the relationship between students’ affective states and their mastery level by employing sensing technologies of a MSE. We highlight the importance of designing interactive digital tools in promoting learning for students. By incorporating elements of interactivity into digital learning tools, educators and developers can help students stay motivated and engaged, leading to better learning outcomes. This is consistent with the concept of personalized learning, which has gained attention in recent years as a way to cater to the unique needs and preferences of each learner. By using data-driven approaches in immersive learning experiences, educators and developers can help students reach their full potential and maximize their learning outcomes. By collecting data on students’ emotional states as they engage with digital learning tools, it is possible to identify what intervention is needed to support them and maximize their learning outcomes. In our research, we found that interaction modalities play an important role in the relationship between the two. This contribution added to previous research considering the way in which students interact with the learning content as part of the learning dimensions, designing a go-with-the-flow interaction modality can help students manage negative affective states (high stress or anxiety or low ER) and perform better. This has important implications for the design of digital learning tools such as MSEs and the role of technology in education more broadly. Our findings can also inform the development of assessment tools that measure students’ emotional states during learning activities. This can be useful for both formative and summative assessment purposes, as it provides insights into how students are feeling and how this is related to their performance. Additionally, by incorporating measures of emotional states into assessment tools, educators can provide more comprehensive feedback to students, helping them to better understand their strengths and areas for improvement.
Acknowledgements
We extend our heartfelt gratitude to the participants of this study, without whom this research would not have been possible. Their willingness to engage with the technology, share their experiences, and provide invaluable feedback was fundamental to the success of our investigation into the moderating role of interaction modalities in education. A special acknowledgment is due to the educators and administrative staff at the schools involved. Their support in coordinating the study sessions and facilitating our access to the necessary environments played a crucial role in the seamless execution of our research activities. We are also deeply appreciative of the parents and guardians of our participants. Their trust and encouragement were crucial in enabling the participation of the students in this innovative exploration of educational technology. Our study’s achievements are a direct result of the time, energy, and enthusiasm generously contributed by each of these individuals. Their roles in this research have not only enriched our findings but have also paved the way for future advancements in the field of educational technology.
Authors’ contributions
Cosentino led the paper writing process and was the primary author, led the research design, data collection and analysis of children's interaction with the modalities. Gelsomini contributed to the technical development of the MOVES platform and data collection. Sharma supported the statistical analysis and contributed to the interpretation of physiological and motion data. Giannakos oversaw the study design, ensured alignment with research goals, and contributed to the writing and synthesis of findings. Everyone contributed to writing.
Data availability
The data that support the findings of this study are available from the Norwegian Agency for Shared Services in Education and Research (Sikt), but restrictions apply to the availability of these data, which were used under licence for the current study and so are not publicly available. The data are, however, available from the authors upon reasonable request and with the permission of Norwegian Agency for Shared Services in Education and Research (Sikt).
Declarations
Ethical approval
Participation was voluntarily, and all the data collected anonymously. Appropriate permissions and ethical approval were requested and approved.
Conflict of interest
None.
Prior research
Prior research related to this paper included a full paper article stating both the educational and technology design https://dl.acm.org/doi/abs/10.1145/3585088.3589385. This contribution is unique relative to the previous one, including an analysis about the moderating role of interaction modalities on the relationship between correctness rate and states, offering design and theoretical implications on the role of the interaction modalities in the learning experience with the MSE.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
Abrahamson, D; Nathan, MJ; Williams-Pierce, C; Walkington, C; Ottmar, ER; Soto, H; Alibali, MW. The future of embodied design for mathematics teaching and learning. Frontiers in education; 2020; 5, 147.
Abrahamson, D., Worsley, M., Pardos, Z., & Ou, L. (2021). Learning analytics of embodied design: Enhancing synergy. Elsevier.
Agosta, G., Borghese, L., Brandolese, C., Clasadonte, F., Fornaciari, W., & Gelsomini, M. et al. (2015). Playful supervised smart spaces (p3s)–a framework for designing, implementing and deploying multisensory play experiences for children with special needs. In 2015 Euromicro Conference on Digital System Design (pp. 158–164).
Almjally, A., Howland, K., & Good, J. (2020). Investigating children’s spontaneous gestures when programming using tuis and guis. In Proceedings of the Interaction Design and Children Conference (pp. 36–48).
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2022). The evidence of impact and ethical considerations of multimodal learning analytics: A systematic literature review. The Multimodal Learning Analytics Handbook, 289–325.
Anthony, L., Brown, Q., Nias, J., Tate, B., & Mohan, S. (2012). Interaction and recognition challenges in interpreting children’s touch and gesture input on mobile devices. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces (pp. 225–234).
Antoun, M; Edwards, KM; Sweeting, J; Ding, D. The acute physiological stress response to driving: A systematic review. PLoS one; 2017; 12,
Arguedas, M; Daradoumis, A; Xhafa Xhafa, F. Analyzing how emotion awareness influences students’ motivation, engagement, self-regulation and learning outcome. Educational technology and society; 2016; 19,
Baker, R.S., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Educational Data Mining.
Baker, RS; D’Mello, SK; Rodrigo, M; Graesser, AC. Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies; 2010; 68,
Bartoli, L., Gelsomini, M., Oliveto, L., & Valoriani, M. (2014). Designing and evaluating touchless playful interaction for asd children. In Proceedings of the 2014 Conference on Interaction Design and Children (pp. 17–26).
Berntson, GG; Cacioppo, JT. Heart rate variability: Stress and psychiatric conditions. Dynamic Electrocardiography; 2004; 41,
Bhattacharya, A., Gelsomini, M., Pérez-Fuster, P., Abowd, G.D., & Rozga, A. (2015). Designing motion-based activities to engage students with autism in classroom settings. In Proceedings of the 14th International Conference on Interaction Design and Children (pp. 69–78).
Blanchard, T.C., & Gerstner, W. (2019). Deep reinforcement learning and its neuroscientific implications. Advances in Neural Information Processing Systems (pp. 12108–12118).
Blikstein, P; Worsley, M. Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics; 2016; 3,
Bower, JM; Carroll, A. Capturing real-time emotional states and triggers for teachers through the teacher wellbeing web-based application t*: A pilot study. Teaching and Teacher Education; 2017; 65, pp. 183-191.
Bravi, A; Green, G; Herry, C; Wright, HE; Longtin, A; Kenny, GP; Seely, AJ. Do physiological and pathological stresses produce different changes in heart rate variability?. Frontiers in Physiology; 2013; 4, 197.
Bruner, JS. On learning mathematics1. The Mathematics Teacher; 1960; 53,
Carroll, JM; Mack, RL. Metaphor, computing systems, and active learning. International Journal of Man-Machine Studies; 1985; 22,
Castelhano, N., & Roque, L. (2017). Lessons from designing a game to support playfulness in multisensory stimulation environments. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (pp. 57–68).
Chalmers, JA; Quintana, DS; Abbott, MJ-A; Kemp, AH. Anxiety disorders are associated with reduced heart rate variability: A meta-analysis. Frontiers in Psychiatry; 2014; 5, 80.
Chaouachi, M., & Frasson, C. (2012). Mental workload, engagement and emotions: an exploratory study for intelligent tutoring systems. In Intelligent Tutoring Systems: 11th International Conference, its 2012, chania, crete, greece, june 14-18, 2012. Proceedings 11 (pp. 65–71).
Chejara, P., Prieto, L.P., Ruiz-Calleja, A., Rodríguez-Triana, M.J., & Shankar, S.K. (2019). Exploring the triangulation of dimensionality reduction when interpreting multimodal learning data from authentic settings. In: Transforming Learning with Meaningful Technologies: 14th European Conference on Technology Enhanced Learning, ec-tel 2019, delft, The Netherlands, September 16–19, 2019, Proceedings 14 (pp. 664–667).
Chen, Y., & Qin, X. (2023). Student fatigue and its impact on teaching effectiveness based on online teaching. Education and Information Technologies, 1–24.
Chen, Z., Li, J., Hua, Y., Shen, R., & Basu, A. (2017). Multimodal interaction in augmented reality. In 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 206–209).
Choi, J., & Lee, D. (2018). Deep learning for multi-modal sentiment analysis in internet of things. In 2018 IEEE International Conference on Consumer Electronics (ICCE) (pp. 1–6).
Cosentino, G; Giannakos, M. Multisensory interaction and analytics to enhance smart learning environments: A systematic literature review. IEEE Transactions on Learning Technologies; 2023; 16,
Csikszentmihalyi, M., Montijo, M. N., & Mouton, A. R. (2018). Flow theory: Optimizing elite performance in the creative realm.
Csikszentmihalyi, M. Flow and the psychology of discovery and invention. HarperPerennial, New York; 1997; 39, pp. 1-16.
Csikszentmihalyi, M. Applications of flow in human development and education; 2014; Springer:
Dawson, ME; Schell, AM; Filion, DL et al. The electrodermal system. Handbook of psychophysiology; 2007; 2, pp. 200-223.
Di Lascio, E; Gashi, S; Santini, S. Unobtrusive assessment of students’ emotional engagement during lectures using electrodermal activity sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies; 2018; 2,
Di Leo, I; Muis, KR; Singh, CA; Psaradellis, C. Curiosity.. confusion? frustration! the role and sequencing of emotions during mathematics problem solving. Contemporary Educational Psychology; 2019; 58, pp. 121-137.
Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., & Specht, M. (2017). Learning pulse: A machine learning approach for predicting performance in self-regulated learning using multimodal data. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 188–197).
Di Mitri, D., Schneider, J., Trebing, K., Sopka, S., Specht, M., & Drachsler, H. (2020). Real-time multimodal feedback with the cpr tutor. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part i 21 (pp. 141–152).
Ding, Y; Cao, Y; Duffy, VG; Wang, Y; Zhang, X. Measurement and identification of mental workload during simulated computer tasks with multimodal methods and machine learning. Ergonomics; 2020; 63,
D’Mello, S; Dale, R; Graesser, A. Disequilibrium in the mind, disharmony in the body. Cognition & Emotion; 2012; 26,
Donker, A; Reitsma, P. Aiming and clicking in young children’s use of the computer mouse. Computers in Human Behavior; 2007; 23,
Dourado, R.A., Rodrigues, R.L., Ferreira, N., Mello, R.F., Gomes, A.S., Verbert, K. (2021). A teacher-facing learning analytics dashboard for process-oriented feedback in online learning. In Lak21: 11th International Learning Analytics and Knowledge Conference (pp. 482–489).
Dourish, P. Where the action is; 2001; Cambridge, MIT press Cambridge:
Dowd, JE; Araujo, I; Mazur, E. Making sense of confusion: Relating performance, confidence, and self-efficacy to expressions of confusion in an introductory physics class. Physical Review Special Topics-Physics Education Research; 2015; 11,
Garg, R., Cui, H., Seligson, S., Zhang, B., Porcheron, M., Clark, L., & Beneteau, E. (2022). The last decade of hci research on children and voice-based conversational agents. In Chi Conference on Human Factors in Computing Systems (pp. 1–19).
Garzotto, F., Gelsomini, M., Gianotti, M., & Riccardi, F. (2019). Engaging children with neurodevelopmental disorder through multisensory interactive experiences in a smart space. Social Internet of Things (pp. 167–184). Springer.
Gelsomini, M. (2023). Sensei - multisensory and multimodal research framework. https://sensei.space/www.sensei.space
Gelsomini, M., & Leonardi, G. (2020). Embodied learning in immersive smart spaces. InProceedings of the 2020 Chi Conference on Human Factors in Computing Systems (pp. 1–14).
Gelsomini, M., Cosentino, G., Spitale, M., Gianotti, M., Fisicaro, D., Leonardi, G., & Bordegoni, M. (2019). Magika, a multisensory environment for play, education and inclusion. In Extended Abstracts of the 2019 Chi Conference on Human Factors in Computing Systems (pp. 1–6).
Gelsomini, M., Rotondaro, A., Cosentino, G., Gianotti, M., & Riccardi, F. (2018). On the effects of a nomadic multisensory solution for children’s playful learning. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces (pp. 189–201).
Giannakos, M.N., Horn, M.S., Read, J.C., & Markopoulos, P. (2020). Movement forward: The continued growth of child-computer interaction research. Elsevier.
Glenberg, AM; Goldberg, AB; Zhu, X. Improving early reading comprehension using embodied CAI. Instructional Science; 2011; 39,
Gorman, JM; Sloan, RP. Heart rate variability in depressive and anxiety disorders. American Heart Journal; 2000; 140,
Guigon, E; Baraduc, P; Desmurget, M. Computational motor control: Redundancy and invariance. Journal of Neurophysiology; 2007; 97,
Hair, JF, Jr; Hult, G; Ringle, CM; Sarstedt, M; Danks, NP; Ray, S et al. Moderation analysis; 2021; Partial Least Squares Structural Equation Modeling (PLS-SEM) Using R, A Workbook: pp. 155-172.
Harada, N. Cold-stress tests involving finger skin temperature measurement for evaluation of vascular disorders in hand-armn vibration syndrome: review of the literature. International Archives of Occupational and Environmental Health; 2002; 75, pp. 14-19.
Hasson, U; Furman, O; Clark, D; Dudai, Y; Davachi, L. Enhanced intersubject correlations during movie viewing correlate with successful episodic encoding. Neuron; 2008; 57,
Hedman, E., Miller, L., Schoen, S., Nielsen, D., Goodwin, M., & Picard, R. (2012). Measuring autonomic arousal during therapy. In Proceedings of Design and Emotion (pp. 11–14).
Herborn, KA; Graves, JL; Jerem, P; Evans, NP; Nager, R; McCafferty, DJ; McKeegan, DE. Skin temperature reveals the intensity of acute stress. Physiology & Behavior; 2015; 152, pp. 225-230.
Hourcade, J.P., Zeising, A., Iversen, O.S., Skov, M.B., Antle, A.N., Anthony, L. & Walsh, G. (2018). Child-computer interaction sig: Ubiquity and big data–a changing technology landscape for children. In Extended Abstracts of the 2018 Chi Conference on Human Factors in Computing Systems (pp. 1–4).
Howison, M., Trninic, D., Reinholz, D., & Abrahamson, D. (2011). The mathematical imagery trainer: from embodied interaction to conceptual learning. In Proceedings of the Sigchi Conference on Human Factors in Computing Systems (pp. 1989–1998).
Jevtić, A; Doisy, G; Parmet, Y; Edan, Y. Comparison of interaction modalities for mobile indoor robot guidance: Direct physical interaction, person following, and pointing control. IEEE Transactions on Human-Machine Systems; 2015; 45,
Jordan, PW. An introduction to usability; 2020; CRC Press:
Jung, H., Kim, H.J., So, S., Kim, J., & Oh, C. (2019). Turtletalk: an educational programming game for children with voice user interface. In Extended Abstracts of the 2019 Chi Conference on Human Factors in Computing Systems (pp. 1–6).
Kaliisa, R., Gillespie, A., Herodotou, C., Kluge, A., & Rienties, B. (2021). Teachers’ perspectives on the promises, needs and challenges of learning analytics dashboards: Insights from institutions offering blended and distance learning. Visualizations and Dashboards for Learning Analytics (pp. 351–370). Springer.
Kandroudi, M; Bratitsis, T. Exploring the educational perspectives of xbox kinect based video games. Proceedings ECGBL; 2012; 2012, pp. 219-227.
Karray, F; Alemzadeh, M; Abou Saleh, J; Arab, MN. Human-computer interaction: Overview on state of the art. International Journal on Smart Sensing and Intelligent Systems; 2008; 1,
Kolb, AY; Kolb, DA. Learning styles and learning spaces: Enhancing experiential learning in higher education. Academy of Management Learning & Education; 2005; 4,
Kosmas, P; Ioannou, A; Retalis, S. Moving bodies to moving minds: A study of the use of motion-based games in special education. TechTrends; 2018; 62,
Kourakli, M; Altanis, I; Retalis, S; Boloudakis, M; Zbainos, D; Antonopoulou, K. Towards the improvement of the cognitive, motoric and academic skills of students with special educational needs using kinect learning games. International Journal of Child-Computer Interaction; 2017; 11, pp. 28-39.
Lance, CE. Residual centering, exploratory and confirmatory moderator analysis, and decomposition of effects in path models containing interactions. Applied Psychological Measurement; 1988; 12,
Lee-Cultura, S., Sharma, K., & Giannakos, M. N. (2023). Multimodal teacher dashboards: Challenges and opportunities of enhancing teacher insights through a case study. IEEE Transactions on Learning Technologies,1–17. https://doi.org/10.1109/TLT.2023.3276848
Lee-Cultura, S., Sharma, K., Cosentino, G., Papavlasopoulou, S., & Giannakos, M. (2021). Children’s play and problem solving in motion-based educational games: synergies between human annotations and multi-modal data. Interaction Design and Children (pp. 408–420).
Lee-Cultura, S., Sharma, K., Papavlasopoulou, S., Retalis, S., & Giannakos, M. (2020). Using sensing technologies to explain children’s self-representation in motion-based educational games. In Proceedings of the Interaction Design and Children Conference (pp. 541–555).
Lee-Cultura, S; Sharma, K; Giannakos, M. Children’s play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach. International Journal of Child-Computer Interaction; 2022; 31, 100355.
Leiner, D; Fahr, A; Früh, H. Eda positive change: A simple algorithm for electrodermal activity to measure general audience arousal during media exposure. Communication Methods and Measures; 2012; 6,
Lindgren, R; Tscholl, M; Wang, S; Johnson, E. Enhancing learning and engagement through embodied interaction within a mixed reality simulation. Computers & Education; 2016; 95, pp. 174-187.
Liu, Z., Pataranutaporn, V., Ocumpaugh, J., & Baker, R. (2013). Sequences of frustration and confusion, and learning. Educational Data Mining 2013.
Malinverni, L; Mora-Guiard, J; Padillo, V; Valero, L; Hervás, A; Pares, N. An inclusive design approach for developing video games for children with autism spectrum disorder. Computers in Human Behavior; 2017; 71, pp. 535-549.
Malinverni, L; Schaper, M-M; Pares, N. Multimodal methodological approach for participatory design of full-body interaction learning environments. Qualitative Research; 2019; 19,
Malmberg, J; Haataja, E; Järvelä, S. Exploring the connection between task difficulty, task perceptions, physiological arousal and learning outcomes in collaborative learning situations. Metacognition and Learning; 2022; 17,
Mangaroska, K; Sharma, K; Gaševic, D; Giannakos, M. Multimodal learning analytics to inform learning design: Lessons learned from computing education. Journal of Learning Analytics; 2020; 7,
Markopoulos, P; Read, JC; MacFarlane, S; Hoysniemi, J. Evaluating children’s interactive products: Principles and practices for interaction designers; 2008; Elsevier:
Mirjafari, S; Masaba, K; Grover, T; Wang, W; Audia, P; Campbell, AT et al. Differentiating higher and lower job performers in the workplace using mobile sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies; 2019; 3,
Neale, D.C., & Carroll, J.M. (1997). The role of metaphors in user interface design. Handbook of Human-Computer Interaction (pp. 441–462). Elsevier.
Nielsen-Englyst, L. (2003). Game design for imaginative conceptualisation. In Proceedings of the International Workshop on Experimental Interactive Learning in Industrial Management (pp. 149–164).
Ouhaichi, H; Spikol, D; Vogel, B. Research trends in multimodal learning analytics: A systematic mapping study. Computers and Education: Artificial Intelligence; 2023; 4, 100136.
Pijeira-Díaz, HJ; Drachsler, H; Kirschner, PA; Järvelä, S. Profiling sympathetic arousal in a physics course: How active are students?. Journal of Computer Assisted Learning; 2018; 34,
Ping, R; Goldin-Meadow, S. Gesturing saves cognitive resources when talking about nonpresent objects. Cognitive Science; 2010; 34,
Pizzie, RG; Kraemer, DJ. The association between emotion regulation, physiological arousal, and performance in math anxiety. Frontiers in Psychology; 2021; 12, 639448.
Poon, A., Quan, J., Cheng, J., & Blikstein, P. (2019). Multimodal learning analytics in virtual reality for hands-on activities. In 2019 IEEE Conference on Virtual Reality and 3d User Interfaces (vr) (pp. 191–198).
Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. International Review of Research in Open and Distributed Learning, 19(5).
Robb, H. (2018). Music and embodied cognition: Listening, moving, feeling, and thinking. by arnie cox. Oxford University Press.
Rodrigo, M.M.T., Baker, R.S., Jadud, M.C., Amarra, A.C.M., Dy, T., Espejo-Lahoz, M.B.V. & Tabanao, E.S. (2009). Affective and behavioral predictors of novice programmer achievement. In Proceedings of the 14th Annual ACM Sigcse Conference on Innovation and Technology in Computer Science Education, (pp. 156–160).
Rühmann, L.M., Otero, N., & Oakley, I. (2016). A tangible tool for visual impaired users to learn geometry. In Proceedings of the tei’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction (pp. 577–583).
Ryan, RM; Deci, EL. Intrinsic and extrinsic motivation from a self-determination theory perspective: Definitions, theory, practices, and future directions. Contemporary Educational Psychology; 2020; 61, 101860.
Sawyer, R.K. (2014). The future of learning: Grounding educational innovation in the learning sciences. The Cambridge Handbook of the Learning Sciences, 726–746.
Seipäjärvi, SM; Tuomola, A; Juurakko, J; Rottensteiner, M; Rissanen, A-PE; Kurkela, JL; Wikgren, J. Measuring psychosocial stress with heart rate variability-based methods in different health and age groups. Physiological Measurement; 2022; 43,
Shakroum, M; Wong, KW; Fung, CC. The influence of gesture-based learning system (GBLS) on learning outcomes. Computers & Education; 2018; 117, pp. 75-101.
Sharma, K., Lee-Cultura, S., Giannakos, M. (2021). Keep calm and do not carry-forward: Toward sensor-data driven ai agent to enhance human learning. Frontiers in Artificial Intelligence, 4.
Sharma, K; Giannakos, M. Multimodal data capabilities for learning: What can multimodal data tell us about learning?. British Journal of Educational Technology; 2020; 51,
Sharma, K; Giannakos, M. Sensing technologies and child-computer interaction: Opportunities, challenges and ethical considerations. International Journal of Child-Computer Interaction; 2021; 30, 100331.
Sharma, K; Papamitsiou, Z; Giannakos, M. Building pipelines for educational data using ai and multimodal analytics: A “grey-box” approach. British Journal of Educational Technology; 2019; 50,
Sharma, K; Papavlasopoulou, S; Giannakos, M. Children’s facial expressions during collaborative coding: Objective versus subjective performances. International Journal of Child-Computer Interaction; 2022; 34, 100536.
Sherwin, K., & Nielsen, J. (2019). Children’s ux: Usability issues in designing for young people. Nielsen Norman Group, 13.
Shih, C-H. Assisting people with attention deficit hyperactivity disorder by actively reducing limb hyperactive behavior with a gyration air mouse through a controlled environmental stimulation. Research in Developmental Disabilities; 2011; 32,
So, I. Cognitive development in children: Piaget development and learning. Journal of Research in Science Teaching; 1964; 2, pp. 176-186.
Soland, J; Jensen, N; Keys, TD; Bi, SZ; Wolk, E. Are test and academic disengagement related? Implications for measurement and practice. Educational Assessment; 2019; 24,
Spitale, M., Gelsomini, M., Beccaluva, E., & Viola, L. (2019). Meeting the needs of people with neuro-developmental disorder through a phygital approach. In Proceedings of the 13th Biannual Conference of the Italian Sigchi Chapter: Designing the Next Interaction (pp. 1–10).
Streitz, N. A. (1988). Mental models and metaphors: Implications for the design of adaptive user-system interfaces. Springer.
Tabuenca, B., Serrano-Iglesias, S., Carruana-Martin, A., Villa-Torrano, C., Dimitriadis, Y. A., Asensio-Perez, J. I., & Kloos, C. D. (2021). A systematic literature review. In IEEE Transactions on Learning Technologies: Affordances and Core Functions of Smart Learning Environments.
Taelman, J., Vandeput, S., Spaepen, A., & Huffel, S.V. (2009). Influence of mental stress on heart rate and heart rate variability. In 4th European Conference of the International Federation for Medical and Biological Engineering (pp. 1366–1369).
Van Duijvenvoorde, AC; Zanolie, K; Rombouts, SA; Raijmakers, ME; Crone, EA. Evaluating the negative or valuing the positive? neural mechanisms supporting feedback-based learning across development. Journal of Neuroscience; 2008; 28,
van Loon, AW; Creemers, HE; Okorn, A; Vogelaar, S; Miers, AC; Saab, N; Asscher, JJ. The effects of school-based interventions on physiological stress in adolescents: A meta-analysis. Stress and Health; 2022; 38,
van den Hoogen, W.M., IJsselsteijn, W.A., & de Kort, Y.A. (2008). Exploring behavioral expressions of player experience in digital games. In Proceedings of the Workshop on Facial and Bodily Expression for Control and Adaptation of Games ECAG (Vol. 2008, pp. 11–19).
Velloso, E; Schmidt, D; Alexander, J; Gellersen, H; Bulling, A. The feet in human-computer interaction: A survey of foot-based interaction. ACM Computing Surveys (CSUR); 2015; 48,
Verbert, K; Duval, E; Klerkx, J; Govaerts, S; Santos, JL. Learning analytics dashboard applications. American Behavioral Scientist; 2013; 57,
Wang, J; Dawson, K; Saunders, K; Ritzhaupt, AD; Antonenko, P; Lombardino, L et al. Investigating the effects of modality and multimedia on the learning performance of college students with dyslexia. Journal of Special Education Technology; 2018; 33,
Webster, J; Trevino, LK; Ryan, L. The dimensionality and correlates of flow in human-computer interactions. Computers in Human Behavior; 1993; 9,
Williams, DP; Cash, C; Rankin, C; Bernardi, A; Koenig, J; Thayer, JF. Resting heart rate variability predicts self-reported difficulties in emotion regulation: A focus on different facets of emotion regulation. Frontiers in Psychology; 2015; 6, 261.
Worsley, M. (2018). (dis) engagement matters: Identifying efficacious learning practices with multimodal learning analytics. In Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 365–369).
Yu, J; Huang, C; He, T; Wang, X; Zhang, L. Investigating students’ emotional self-efficacy profiles and their relations to self-regulation, motivation, and academic performance in online learning contexts: A person-centered approach. Education and Information Technologies; 2022; 27,
Zou, L., Tal, I., Covaci, A., Ibarrola, E.,Ghinea, G., & Muntean, G-M. (2017). Can multisensorial media improve learner experience? Proceedings of the 8th ACM on Multimedia Systems Conference (pp. 315–320).
3DiVi. (2023). Nuitrack full body skeletal tracking software. Nuitrack. https://nuitrack.com/
© The Author(s) 2025. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.