Content area
Finding ways to support learners in becoming effective collaborators is a key challenge in higher education. Educational technologies can help to achieve this goal. However, the effectiveness of pedagogical design principles underlying these technologies needs to be tested empirically to inform evidence-based teaching in higher education. In the present study, we examine the effect of a technologically-supported collaborative reflection activity on learners’ knowledge gain about effective collaboration and about the quality of their interaction. To this end, we compare the results of a field study that was conducted in a course for civil engineering students (n = 66), with results of a laboratory study with n = 57 university students. Both field and laboratory study consisted of two collaborative problem-solving phases, in which students worked in small groups to solve information pooling problems. Multilevel modeling indicates that the technology-supported collaborative reflection activity between the two collaborative problem-solving phases increased explicit knowledge about effective collaboration. The quality of collaboration during subsequent collaboration, however, was not affected. Further, we found that groups’ self-assessments were in line with expert ratings of their collaboration quality. We discuss these findings in terms of the learning mechanisms behind technology-supported collaborative reflection and the extent to which these forms of support foster collaboration skills. Thus, our study adds insights on how to join educational technologies with pedagogical design principles to support collaborative learning.
Introduction
Global challenges, such as the COVID-19 pandemic, regional conflicts, or the climate crisis, but also projects such as the treatment of complicated medical cases and the planning and execution of large construction projects like an airport, cannot be tackled by an individual alone. In these cases, highly skilled experts from different professional fields need to collaborate to find appropriate solutions. Thus, the capacity to learn, work, and solve problems collaboratively has been acknowledged as a vital skill (van Laar et al., 2017; Vauras et al., 2019). Consequently, students need to be prepared for collaboration in complex environments (Asia Society & OECD, 2017). While collaborating to learn is widely used as a method in teaching domain-specific knowledge, less attention has been paid to the acquisition of collaboration skills, learning to collaborate.
This paper investigates the effectiveness of an instructional approach that aims to foster collaboration skills in a computer-supported collaborative learning (CSCL) arrangement in higher education. Building on the findings of a laboratory study (Strauß et al., 2025), we tested the effectiveness of a technology-supported collaborative reflection activity in a course for civil engineering students. Our study had the primary goal to test the ecological validity of our instructional design and of previous findings on learning outcomes (Bracht & Glass, 1968; Fahmie et al., 2023).
We argue that our study represents a test case for a form of collaboration support which acknowledges learners’ prior knowledge about effective collaboration and their agency over their collaboration, to foster the acquisition of collaboration skills. Further, our study illustrates how an instructional design can be derived from theoretical considerations, subsequently tested under controlled conditions, and then further evaluated in an authentic learning setting. Thus, our study informs evidence-based teaching in higher education.
Theoretical background
Acquiring the skills for effective collaboration
To teach and assess students’ ability to engage in fruitful collaboration, it is necessary to identify the characteristics of productive collaboration and define observable interaction patterns. Collaboration related competency models come from programs for educational monitoring (e.g., collaborative problem solving in PISA 2015, see Graesser et al., 2018, or the so-called 21st century skills, see van Laar et al., 2017), but also from research on learning and instruction (e.g., Meier et al., 2007; Schürmann et al., 2025). A more general model of fruitful collaboration underlies rating schema for assessing the quality of computer-supported collaboration processes proposed by Meier et al. (2007) was developed using a literature review (top-down) and collaboration data (bottom-up) in order to determine aspects of collaboration that are relevant for collaboration success (Meier et al., 2007; Rummel et al., 2011).
The authors distinguish between five aspects that play a role for beneficial collaboration: communication, coordination, joint information processing, relationship management, and motivation. These five aspects are further subdivided into nine dimensions, for example: sustaining mutual understanding, dialogue management, and information pooling. While the original goal of the rating schema was to assess the quality of the collaboration in a group, it also hints at skills that are necessary to achieve a high level of collaboration quality. For example, the rating handbook specifies that episodes of sustaining mutual understanding are of high quality when “speakers make their contributions understandable for their collaboration partner, for instance by avoiding or explaining technical terms from their domain of expertise […], rather than reading them aloud to their partner” (Meier et al., 2007, pp. 81–82). Thus, group members should become aware that tailoring their messages to their learning partners’ prior knowledge is relevant for effective collaboration and learn how to achieve this. In our laboratory experiment (Authors) and the present study, we used this rating schema as our model of fruitful collaboration.
In the CSCL community, there is an ongoing debate on how to best use educational technologies to scaffold collaboration processes so that groups achieve fruitful collaboration which is associated with learning and successful problem solving (see Wise & Schwarz, 2017). In this regard, the ‘conciliator’ in Wise and Schwarz (2017) calls for the investigation of forms of collaboration support that offer knowledge about collaboration while granting learners freedom over their collaboration, thus retaining their autonomy (also see autonomy-supportive teaching, Boud, 1988; Reeve & Cheon, 2021). One such way is to elicit collaborative reflection. In the present study, we focus on collaborative reflection to foster collaboration skills by leveraging learners’ prior knowledge about collaboration and using educational technology to help them engage in joint reflection.
Reflection about the collaboration
Groups gain autonomy over their collaboration when they are required to regulate their collaboration and receive opportunities to do so. Following Zimmerman (2000), regulation encompasses a preparation (forethought) phase, a performance phase, and a reflection phase. During the reflection phase, learners retrospectively evaluate which strategies were advantageous and which require modification in future learning situations because they did not lead to the desired outcome. Reflection is therefore a metacognitive process to make sense of past experiences (Boud et al., 1985; Yang & Choi, 2023; Yukawa, 2006). In collaboration settings, a ‘reflective group’ (Gabelica et al., 2014) is characterized as a group that gathers feedback about their collaboration, evaluates the group’s performance, discusses the group’s performance, collects potential remedial actions to adapt their future collaboration, and eventually implements new strategies.
During collaborative reflection, we expect that groups gather information on their past collaboration (i.e., feed-up), discuss their performance against the background of a desired goal-state (i.e., feed-back), diagnose the need for regulation, and if necessary, derive plans on how to adapt the collaboration to reach the desired goals (i.e., feed-forward, Hattie & Timperley, 2007). Collaborative reflection processes interrupt automated behavior, which makes deliberate information processing more likely (see Mamede & Schmidt, 2017). Following Renner et al.’s (2016) and Yang and Choi’s (2023) argumentation that group settings can promote reflection on levels that one individual alone could not achieve, we assume that situations where group members reflect collaboratively provide them with the opportunity to share knowledge and strategies which may not have been available for the individual alone. Thus, the group members can collaboratively co-construct and internalize knowledge about effective collaboration (e.g., Chi & Wylie, 2014; Weinberger et al., 2007). The resulting knowledge can include knowledge about desired goal states, conditional strategy knowledge, or knowledge about effective strategies and how they are performed. When groups implement plans derived from reflection, groups may improve the quality of their collaborations, given they identified ineffective collaboration patterns and implement effective strategies (Sobocinski et al., 2020).
The effect of reflection on collaboration
We conjecture that the process of deliberate reflection leads to an interactive process of knowledge sharing (Weinberger et al., 2007) and co-construction (e.g., Chi & Wylie, 2014; Weinberger et al., 2007) where learners construct and internalize knowledge about effective collaboration. To promote reflection, scaffolds can be employed that help groups become aware of impasses in their collaboration and subsequently stimulate discussions about potential remedial actions (Heitzmann et al., 2023). Previous studies have investigated a number of approaches to eliciting reflection, such as journals or prompts (for overviews see Fessl et al., 2017; Guo, 2022, and Yang & Choi, 2023). In the field of CSCL, group awareness tools represent an approach that provides groups with opportunities for collaborative reflection (GATs, Bodemer et al., 2018; Jeong & Hmelo-Silver, 2016; Strauß & Rummel, 2023). These tools collect data about the group and visualize it for the group (Janssen & Bodemer, 2013). Groups can then take up and engage with this feedback during reflection (Jermann & Dillenbourg, 2008; Schnaubert & Vogel, 2022; Strauß & Rummel, 2023). Research in this area suggest, that some groups may struggle to utilize the feedback that is provided by GAT for the regulation of their collaboration (Dehler et al., 2009; Strauß & Rummel, 2021, 2023).
Previously, the focus of research lay on the ways that groups use feedback and GATs to adapt their collaboration and how this affects their acquisition of domain-specific knowledge (e.g., Radović et al., 2023), or the interaction processes and group performance (see e.g., Janssen et al., 2007; Janssen et al., 2011; Lin & Tsai, 2016; Strauß & Rummel, 2021). The link between reflection and the acquisition of collaboration skills, however, has received comparatively little attention.
One study that focused on the link between collaborative reflection and the acquisition of collaboration skills comes from Schürmann et al. (2025). The authors investigated the effectiveness of a debriefing activity on students’ perceived collaborative interaction and perceived knowledge gain (both, domain-specific knowledge and collaboration skills). Their results suggest that a collaborative reflection activity after the first half of the group project led to an increase in (perceived) coordination, monitoring and reflection, while there was no effect of groups’ performance or perceived knowledge gain.
Other studies that investigated collaborative reflection focused on the effects of reflection on regulation or group performance. For instance, Phielix et al. (2011) investigated the suggestions by Hattie and Timperley (2007) to design a co-reflection activity that prompted groups to clarify their goals of the current activity (feed-up), decide whether progress is being made towards this goal (feed-back), and eventually decide which activities are needed to progress towards the goal (feed-forward). The findings of the study indicated that the co-reflection activity prompted students to formulate plans concerning aspects of the collaboration that were associated with the dimensions visualized in the GAT (i.e., regulating the coordination and communication to improve on the dimension “collaboration”).
With respect to the role of reflection for collaboration processes and problem-solving performance, in their study with a co-located, complex collaborative task, Gabelica et al. (2014) found that, while reflection-in-action neither improved or reduced performance, reflecting between two tasks (i.e., reflection-on-action) led to higher performance in a subsequent task. Further, Eshuis et al. (2019) showed that groups which first received instruction on effective collaboration and subsequently reflected on the collaboration, were able to improve their communication quality, while groups that received neither direct instruction nor a reflection tool did not.
In sum, previous research has explored the role of team-feedback and collaborative reflection for changes in collaboration processes, regulation and group performance. Yet, only limited evidence exists about the effects of collaborative reflection on the acquisition of collaboration skills and the quality of the collaboration processes. The present study addresses these gaps.
A laboratory study as the starting point for a field trial
In a recent study, we (Strauß et al., 2025) conducted a laboratory experiment to investigate how collaborative reflection affects learners’ knowledge about effective collaboration and the groups’ collaboration. To address the challenge that groups may not take up and process feedback about the collaboration (e.g., Dehler et al., 2009; Strauß & Rummel, 2021, 2023), we designed a reflection scaffold to help groups take up, deliberately process, and leverage the information from a GAT to adapt their interaction in a subsequent collaboration. The design acknowledged findings from research on instructional feedback (Hattie & Timperley, 2007; Phielix et al., 2011) and learning analytics (Wise, 2014; Wise & Vytasek, 2017). Specifically, we scaffolded the process of feed-up, feed-back and feed-forward and provided students with a frame of reference for interpreting the analytics (a GAT) and guided the group through the collaborative reflection process while providing space for individual and collaborative meaning making and goal setting.
In our laboratory study with 150 German university students from different study programs (e.g., applied computer science, German language studies, educational research, social sciences), we found that an external collaboration script and a collaborative reflection activity increased participants’ knowledge about effective collaboration, while neither type of support led to a significant increase in collaboration quality (Strauß et al., 2025).
To inform educational practice, especially considering the discourse on the importance of evidence-based practice in teaching (e.g., Slavin, 2002), it is crucial that pedagogical approaches are tested using controlled experiments, and replication studies (see Connolly et al., 2018). At the same time, because the context of teaching and the group of students can differ greatly, it is also necessary to investigate whether effects that were found in a controlled laboratory setting generalize to real-world teaching settings (i.e., effectiveness trials, Kim, 2019) where students’ motivation, their learning goals, and other characteristics or circumstances of the learning setting represent confounding variables that affect the effectiveness of an instructional approach (i.e., ecological validity, Bracht & Glass, 1968; Fahmie et al., 2023). To learn more about the effectiveness of collaborative reflection on the acquisition of knowledge about effective collaboration, we conducted a field study in the context of a lecture for civil engineering students. We posit that this setting is more ecologically valid than the laboratory due to three reasons. First, students in the field can be assumed to have a different motivation to participate in the activities of the study compared to students who voluntarily participated in our laboratory study, as participation in the field study was associated with a study project that the students had to fulfil to earn their course credits. Second, students in the field formed the collaborative groups themselves (natural groups) and previous relations among the group members likely played a role for interactions during the study. These relations were also personally relevant to students after the study, in contrast to the laboratory setting with unknown group members that the participants will likely never meet again. The groups in the field setting consisted of four to five students, which is a reasonable group size for authentic educational settings, given that examiners need to grade fewer group projects if the groups are larger. In contrast, the groups in the laboratory study consisted of three students. Finally, the collaboration tasks were adapted to a civil engineering context and students’ domain-specific knowledge taught in the course.
Civil engineering education as a context for teaching collaboration skills
The demands for graduates in civil engineering include the capability for technology-supported collaborative work in interdisciplinary teams, especially in the context of Building Information Modeling (BIM) (van Treeck et al., 2017). Consequently, a course on BIM was selected as the context for the field study. BIM enables the digital modelling of all relevant steps in the lifecycle of a civil engineering project, starting with the design of the construction, through planning and execution, and management, to the conversion or demolition of a building (Borrmann et al., 2015). BIM combines complex information about a civil engineering project in a digital model, such as the geometry of all components and non-geometric information (e.g., schedules, costs). During the early phases of the BIM process, it is especially crucial that all involved partners generate error-free and consistent digital models of the construction project (Adamu et al., 2015). A lack of coordination results in increasing costs and delays of the construction project. Thus, training for interdisciplinary collaboration can be understood as a crucial component of civil engineering education. While the technical aspects of the BIM method are increasingly implemented into curricula of civil engineering programs, training the necessary collaboration skills has so far been neglected. Thus, implementing training for interdisciplinary collaboration into a BIM course for civil engineering students can potentially improve civil engineering education, and represents an adequate context for testing our laboratory findings under authentic conditions.
Research questions
Collaborative reflection about the collaboration in a group has already been explored as a means to support collaborative learning and previous studies suggest that reflection may promote the acquisition of collaboration skills (e.g., Eshuis et al., 2019; Authors). For the present study, we implemented the instructional design of the previous laboratory study (Strauß et al., 2025) into a course for civil engineering students. This aimed at evaluating the effectiveness of a collaborative reflection activity under authentic conditions, thus gathering additional data to further corroborate our findings. Specifically, our study addressed the following research questions:
RQ (1)
How did the groups utilize the collaborative reflection scaffold?
RQ (2)
How did learners perform in a test on collaboration knowledge and how did their knowledge change after the first collaboration and the collaborative reflection phase?
RQ (3)
How did the groups perform in terms of collaboration quality in the collaboration phase before and after the collaborative reflection activity respectively?
To contextualize the data obtained in the field, we compare the results of the field study with the findings from the laboratory study (Strauß et al., 2025). This comparison is guided by the following research question:
RQ (4)
How do the samples in the laboratory study and the field study compare in terms of demographics, performance in the collaboration knowledge test, and collaboration quality?
Methods
Sample
The field study was conducted in a regular course for B. Sc. students in civil engineering at a large German university. The course was part of a module that focused on Building Information Modeling (BIM). As part of the BIM module, students were required to work on a graded group project. The study was introduced before the group project started as an opportunity to become familiar with the other group members. Participation in the study was voluntary but rewarded with bonus points to the final grade.
Out of the 125 students (25 groups), n = 66 students between the age of 20 and 29 years (M = 22.44; SD = 1.97) volunteered to participate in the field study. These students autonomously formed n = 14 groups of four or five students. All participants gave their informed consent to take part in the study and nine out of the 14 groups agreed to also have their collaboration process during the intervention recorded (audio, video/screen). Thus, analyses that are concerned with collaboration quality were conducted with n = 9 group.
The sample in the laboratory study consisted of 150 German university students from various subjects who collaborated in n = 50 groups of three participants and received 35€ as compensation (see Authors). To answer RQ 4, we report data from a subsample of n = 57 participants between 18 and 43 years (M = 23.42; SD = 4.5), in n = 17 groups that performed the collaborative reflection activity.
Design & procedure
The procedure and the instructional design were identical in the field study and in the laboratory study (see Strauß et al., 2025 for further details). Both studies were conducted online, and participation was possible only via videoconference due to local COVID regulations. After logging in, participants completed a questionnaire that included questions about demographics and prior knowledge regarding effective collaboration. Afterwards, the participants viewed a short video with general tips for successful collaboration. Subsequently, all participants read the task materials and collaborated to solve a collaborative task. The groups had a maximum of 30 min to solve the task. Following the first collaboration phase (learning phase), groups filled in the questionnaire for the GAT and performed the collaborative reflection activity. Afterwards, the participants again answered the items for their explicit knowledge about effective collaboration. After a short break, the groups worked on the second task for another 30 min (testing phase).
The design and procedure in the field study differed from the laboratory study in the following aspects: While in both studies the solution to the problem task was only to be found through pooling and evaluating the information distributed between all roles, the difficulty of the tasks in the field study was adapted to the professional and technical competence of the civil engineering students. For example, participants in the field study worked with a 3D model in a typical BIM software instead of a 2D floor plan. The task descriptions contained more detailed information about the construction and called for more domain expertise in civil engineering. Also, the groups consisted of four to five students in the field study instead of three in the laboratory study. Accordingly, the role material for the field study contained two additional roles. Another difference was that the two studies were conducted using two different video conferencing systems: While Zoom was used for the laboratory study and students received links to online-questionnaires throughout the study, a specific digital learning environment was available for the field study that allowed for communication via the open-source videoconferencing software Jitsi Meet (https://jitsi.org/jitsi-meet/) and included questionnaires to avoid switching environments. The reflection phase was 8 min long in the field study and therefore shorter than in the laboratory study (12 min). This difference was a result of the changes that were made while implementing the collaboration and reflection into the field. For instance, these changes encompassed revising the wording of some questions in the self-assessment questionnaire to increase clarity, eliminating redundant questions, and adding two questions to evoke reflection of the usage of the BIM software that was used during the collaboration. Another change concerns the scale in the self-assessment questionnaire. In the field study, we implemented the questionnaire with a 5-point Likert scale, instead of a 7-point scale.
Material
A detailed overview of the differences between the studies and the study materials (experimenter manual, collaborative tasks, role material, GAT questionnaire), and our instruments can be found in the supplementary material on OSF:https://osf.io/ne265/.
The collaboration environment is shown in Fig. 1. The architectural model was visible in the center of the screen. At the top edge of the screen the name of the current phase and a timer were displayed. The tiles containing the webcam videos of the group members were arranged to the right-hand side of the screen.
Introduction video
An introductory video briefly characterized typical pitfalls of collaboration and strategies based on the process dimensions described by Meier et al. (2007). The strategies from the video corresponded to the strategies addressed by the GAT. Participants were asked to keep in mind these challenges during their collaboration.
Collaboration tasks
The two collaboration tasks consisted of civil engineering problems that required finding ways to integrate new requirements into a 3D model of a kindergarten construction project (see Fig. 1). The assignment presented groups with information-pooling problems with a hidden-profile (e.g., Stasser & Titus, 1985). In both tasks, the groups had to find a way to rearrange the 3D model to incorporate a new room or special requirements into an existing building.
Each group member was assigned to one expert role. Three roles were already present in the laboratory study, namely an architect, a daycare manager, and a fire and health protection officer. Each of these roles held domain-specific expertise, for instance about safety guidelines for daycare buildings. In the field study, additional roles that were based on authentic roles in BIM were introduced. For example, there was a BIM Manager and a BIM Coordinator who had access to a 3D model of a construction project. The BIM coordinator shared it with the rest of the group using the “Share screen” function, allowing this role to read important information and model changes directly within it.
[See PDF for image]
Fig. 1
Screenshot of the digital learning environment during collaboration
The role material was presented to the participants as short texts in the collaboration environment and contained information about the task and their specific role: shared information (i.e., information all group members received), unshared information (i.e., information representing expert knowledge) as well as different requirements and one proposed expert-specific solution each. To reach a joint solution, the group members had to pool their unshared information (i.e., hidden profile, Brodbeck et al., 2007). There was no single best solution to the problems posed by the two collaborative tasks, so the groups had to evaluate different solutions and decide between alternatives.
Reflection scaffold
The reflection scaffold consisted of a self-assessment questionnaire that served as the basis for a phase of collaborative reflection. The goal of the reflection scaffold was to assist groups to discuss how they perceived the quality of their collaboration, whether they identified shortcomings or challenges in their collaboration, and derive remedial actions that they could implement in the subsequent collaboration (i.e., testing phase). The reflection scaffold was integrated into a digital learning environment that was developed for the field study. The application was based on a client-server architecture in which the client and server communicate with each other via a web socket connection using JSON messages. On the server side, we used the Spring Framework and Google’s GSON, while the Angular Framework was used on the side of the server.
The reflection scaffold was modeled after the activity described by Phielix et al. (2011) and consisted of three phases: feed-up, feed-back and feed-forward (see Hattie & Timperley, 2007). For the design of the reflection scaffold, we also followed the suggestions by Wise (2014) and Wise and Vytasek (2017) for learning analytics interventions. Specifically, we described the pedagogical goal of the learning activity and how the displayed data about the collaboration help the group engage in effective collaboration, provide learners with the opportunity to set goals individually and assess whether they were able to attain them.
The reflection scaffold consisted of two phases. In the first phase, the members of the group individually and anonymously filled out an online questionnaire in which they assessed the quality of the collaboration of their group during the learning phase (feed-up). This contained 20 reflection questions that targeted the dimensions of effective collaboration as described by Meier et al. (2007) (i.e. sustaining mutual understanding, dialogue management, information pooling, reaching consensus, task division, time management, technical coordination, reciprocal interaction, and individual task orientation). For example, group members evaluated their ‘information pooling’ strategies with reflection questions such as “During the collaboration we exchanged all relevant information with each other” on a 5-point-Likert scale (1 “completely disagree”, 5 “completely agree”). The questionnaire was presented to each group member individually and the group members were informed that the results will be displayed anonymously.
After all group members had entered their ratings, the program averaged the individual ratings for each dimension and visualized the data as spider plots together with a prompt for collaborative reflection (i.e., the GAT) to the whole group (see Fig. 2). The spider-plots showed the results for each dimension of on a scale from 1 (do not agree) to 5 (fully agree).
[See PDF for image]
Fig. 2
View during the reflection phase. Center: Spider-plot with group average for the reflection questions
During this second phase (8 min), groups were prompted to reflect on their self-assessment (feed-back) and discuss how they would like to improve their collaboration in the second collaboration phase (feed-forward). Specifically, groups were instructed to “[…] please discuss what you would like to do differently or in the same way in the second collaboration phase and again record your findings in writing in the shared text editor […]”.
After the reflection, groups worked on a second collaborative task. This provided them with the opportunity to implement the changes they had planned during the reflection phase.
Measures
Explicit knowledge about effective collaboration
As in our laboratory study, participants’ explicit knowledge about effective collaboration was assessed based on Rummel and Spada’s (2005) approach and consisted of two questions that participants answered in an open-ended format: (1) Imagine you are asked to find a solution to problems in a construction project together with two cooperative partners. Describe in keywords how you imagine the individual phases of cooperation in solving a problem in a group. How should the cooperation partners proceed step by step in their cooperation? (2) What should be taken into account in good collaboration and communication in general?
We coded participants’ responses based on the rating schema for productive computer-based collaborative problem-solving that was developed by Meier et al. (2007). Their rating schema includes five aspects and nine dimensions (i.e., sustaining mutual understanding, dialogue management, information pooling, reaching consensus, task division, time management, technical coordination, reciprocal interaction, and individual task orientation). Table 1 provides examples for our coding schema.
We first broke down the participants’ responses into units, based on numbering, bullet points, paragraphs, or punctuation, and each unit was assigned to one code. Two coders independently analyzed the answers using the coding manual from the laboratory study. Interrater reliability was sufficient in the laboratory (Cohen’s kappa κ = 0.74) and the field study (Cohen’s kappa κ = 0.93). One point was awarded for each aspect of a participant’s response that matched the codes or sub-codes. To obtain the overall score, we calculated the sum of the points from both questions.
Table 1. Excerpt of the coding schema for the knowledge tests
Aspect | Code: Dimension | Sub-Code | Anchor Examples |
|---|---|---|---|
Categories of interactions that are relevant for successful learning and problem-solving in CSCL. | Sub-categories of interactions | Brief description of desirable characteristics of the interaction. | Quotes from the data used to illustrate the indicators. |
Communication | Sustaining Mutual Understanding | Agreement on a common goal. | “Agreeing on what exactly we envision as the goal of the group’s work” |
Establishing common understanding of the problem. | “First talk generally about the problem and work out what exactly the problem is” | ||
… | … |
Collaboration quality
The collaboration quality during the learning and the testing phase was assessed using an adapted version of the rating scheme by Meier et al. (2007), as in the laboratory study. First, each utterance in the video-audio-data of the collaboration processes was assigned one of the nine codes based on the rating schema by Meier and colleagues, using the software MAXQDA. This process was performed by three coders who reached an agreement of 74.51% on 20% of the data and resolved disagreements by discussion. In a second step, the coded videos were divided into segments of three minutes. For each segment, we rated the quality regarding each individual dimension, using a three-point-scale (0;1;2) using a coding manual, where 0 represented very poor quality and 2 very high quality. Table 2 shows an excerpt of the rating schema.
Table 2. Excerpt of the rating schema for the collaboration quality
0 | 1 | 2 |
|---|---|---|
Sustaining Mutual Understanding | ||
Misunderstandings are revealed in interaction without the active creation of understanding by those involved. | Comprehension questions are asked without making misunderstandings or lack of understanding explicit. | Participants actively seek to establish shared understanding to prevent misunderstanding. They provide efficient feedback on each other’s contributions. |
Information Pooling | ||
Superficial, concise, or difficult-to-understand sharing of information predominates. | Comprehensible and comprehensive sharing of information prevails. | Comprehensible and comprehensive sharing of information prevails. In addition, information is also related to previously shared information. |
On average, the ratings of the collaboration quality for the different dimensions of collaboration quality reached satisfactory interrater agreement in the laboratory and the field (see Table 3). We calculated the mean of the quality scores within each dimension to obtain the quality of the groups’ collaboration on each dimension. To obtain a value for the overall collaboration quality during the learning and testing phase, we calculated a grand mean across all dimensions for during each phase.
Table 3. Interrater reliability (Cohen’s κ) for the coding of the collaboration quality
Dimension | Laboratory study | Field study |
|---|---|---|
Sustaining mutual understanding | 0.89 | 0.83 |
Dialogue management | 0.90 | 0.84 |
Information pooling | 0.81 | 0.82 |
Reaching consensus | 0.80 | 0.92 |
Individual task orientation | 0.75 | 0.88 |
Reciprocal interaction | 0.87 | 0.86 |
Time management | 1.00 | 0.79 |
Technical coordination | 0.91 | 0.73 |
Task division | 0.89 | 0.74 |
Perceived collaboration quality
Perceived collaboration quality was extracted from the GAT. As described above, the questions in the GAT were based on the process dimensions specified in the rating schema by Meier et al. (2007). For each group, we first calculated the mean for each process dimension and then calculated the grand-mean over all process dimensions. Based on our experience with the self-assessment scale in the laboratory study, we adjusted the wording of the questionnaire, eliminated duplicates, and added two questions that covered handling the BIM software during collaboration. To be able to compare the two samples of the field and the laboratory study, we only included the questions that were used in both studies. To ensure comparability, we normalized the data that was collected in the laboratory study with new maximum of 5, using the formula: .
Targets of the plans
Using the same process that we used to assess participants’ explicit knowledge about collaboration, we counted the number of individual plans that groups created during the collaborative reflection phase and coded the targets of these plans. For an example reflection note that contains multiple plans see Fig. 3 below. Two coders independently coded 100% of the reflection notes and assigned each collaboration plan up to two of the nine process dimensions using the coding handbook for the knowledge tests. To obtain the final data set for analysis, differences were resolved by discussion.
Statistical analyses
We present the findings to RQ 1, RQ 2 and RQ 3 with descriptive statistics. For the comparisons between the samples of the field study and the laboratory study (RQ 4), we applied two different strategies. First, for comparisons regarding variables that were collected on the level of the individual participants (i.e., knowledge about effective collaboration), we used hierarchical linear modeling (HLM). This allowed us to account for the nested structure of our data, namely participants (level 1) collaborating in small groups (level 2). Neglecting the nested structure would underestimate effects and violate the assumption of statistical independence of observations (Cress, 2008; Janssen et al., 2013). For our analysis, we constructed a random-intercept model that predicted knowledge after the learning phase based on the study from which a participant originated, that is, the field or the laboratory study (between subjects). The model further accounted for the interaction between participants’ origin (field or laboratory study) and the change in knowledge over the course of the learning phase (pre and post; within-subjects We specified the HLM as follows:
knowledge ~ study + pre-post gain [as repeated measure] + study * pre-post gain + (1| group).
We compared the fit of this random-intercept model with an intercept-only model (i.e., “null-model”) which allowed us to determine the intraclass correlation (ICC), that is, the degree to which being a member in a specific small group explains variance in participants’ knowledge about collaboration after the learning phase. To assess which multilevel model described our data the best, we performed deviance tests as suggested by (Field et al., 2012). The multilevel analyses were conducted in R-Studio (Version: 3.4.1, R Core Team 2022) using the “lme4” (Bates et al., 2015) and “sjPlot” (Lüdecke, 2024) packages. In all frequentist analyses we used 𛼠≤ 0.05 to determine statistical significance.
Second, to compare the samples regarding variables that were measured on the level of the group (i.e., collaboration quality, perceived collaboration quality, number of plans), we employed a Bayesian approach (see Wagenmakers et al., 2018 for an overview). We classified the strength of the evidence for (or against) hypotheses using the Bayes factor (BF) (see Kruschke & Liddell, 2018; van Doorn et al., 2021). The Bayes factor (BF) reflects how well a hypothesis (e.g., H0: no difference exists vs. H1: difference exists) predicts the data that has been collected. For example, a BF10 = 3.0 indicates that the data collected are three times more likely to occur under H1 than under H0 (van Doorn et al., 2021). Bayesian analyses can be tailored so that we can interpret them either in favor of the null hypothesis (BF01, note that the subscript is 0 followed by 1) or in favor of the alternative hypothesis (BF10, note that the subscript is 1 followed by 0). For an intuitive interpretation of the results, we report the Bayes factor that is larger. In contrast to the frequentist approach using fixed thresholds for the interpretation of the Bayes factor is incompatible with the framework of Bayesian inference. Nevertheless, a rule of thumb suggests the following levels of evidence: BF between 1 and 3: weak evidence, BF between 3 and 10: moderate evidence, BF greater than 10: strong evidence (see van Doorn et al., 2021 and Kruschke & Liddell, 2018). The Bayesian analyses were performed in JASP (Version 0.18.3, JASP Team, 2024). Whenever variances were not homogeneously distributed, we conducted Bayesian Mann-Whitney tests, in all other cases, we performed Bayesian Student t-tests for independent samples. For our analysis, we used an uninformed prior, that is, the JASP-default Cauchy prior distribution with r = 0.707.
Effect sizes were interpreted following Cohen’s (1992) suggestions, as d = 0.2 (small), 0.5 (medium) and 0.8 (large) effects.
Results
The dataset and the syntax for the statistical analyses are available in the supplementary material on OSF:https://osf.io/ne265/.
RQ 1)
How did the groups utilize the collaboration support?
After the learning phase, all participants in the field study assessed the collaboration quality in their group by rating the group performance with 20 items on a five-point scale. On average, the groups in the field study perceived their collaboration quality as close to the maximum on all process dimensions of effective collaboration (M = 4.03; SD = 0.46). The results of the self-assessment questions were reported back to the groups in six spider-plot diagrams grouped by dimension. During the reflection, groups in the field study created on average M = 2.71 (SD = 2.4) plans ranging from 0 to 7 plans, whereas groups in the laboratory study created between 0 and 9 plans, with the average group creating M = 3.87 (SD = 2.7) plans.
Figure 3 shows an example collection of plans that a group created. Usually, groups wrote down their plans in the form of lists that contained plans for improvements and sometimes notes on what went well (and thus does not require changes in interaction). The note presented in the figure, for instance, includes multiple plans regarding information pooling (exchanging information about each other’s roles and expertise).
[See PDF for image]
Fig. 3
Example reflection note with plans that a group created during collaborative reflection
A Bayesian Mann-Whitney tests suggest more data is needed to determine whether the samples did not differ in the number of plans that were created during the reflection BF01 = 2.05 (95% CI: [-0.97, 0.39]).
Inspecting the actions that were planned by the groups during the collaborative reflection phase, it became obvious that groups in the two studies emphasized different dimensions of their collaboration. Most plans from groups in the field study targeted ‘reaching consensus’ (9 plans in total) and ‘information pooling’ (8 plans in total). Groups in the laboratory study created plans that addressed ‘sustaining mutual understanding’ (29 plans in total) and ‘reaching consensus’ (10 plans in total). In both studies, the two least covered dimensions of collaboration quality were ‘dialogue management’ (field study: 1 plan in total; laboratory study: 2 plans in total) and ‘technical coordination (field study: 1 plan in total; laboratory study: 0 plans).
RQ 2)
Did learners acquire knowledge about effective collaboration?
To answer the question whether participants in the field study acquired explicit knowledge about effective collaboration, we analyzed participants’ knowledge before and after the learning phase. Participants in the field study already held prior knowledge about effective collaboration and also gained new knowledge through the study however, descriptively, their knowledge gain was smaller than in the field study (see Table 4).
Table 4. Knowledge about effective collaboration in the field study and the laboratory study
Study | ||
|---|---|---|
Laboratory | Field | |
M (SD) | M (SD) | |
Pretest | 4.95 (1.86) | 4.11 (2.08) |
Posttest | 6.94 (2.57) | 4.98 (2.42) |
Knowledge gain | 1.86 (2.08) | 0.89 (2.28) |
Overall, participants in the field study reported knowledge on all nine dimensions of effective collaboration processes. In the pretest, strategies that fell into the dimension ‘reciprocal interaction’ were named most frequently. The most notable differences between the pretest and the posttest were the increased naming of strategies that belong to sustaining mutual understanding, information pooling, and dialogue management, while participants mentioned reciprocal interaction and individual task orientation less frequently in the posttest (see Table 5).
Table 5. Mean number of times of naming strategies on the nine dimensions according to Meier et al. (2007) at both measurement points for knowledge in the field study
Field study | ||
|---|---|---|
Pretest (M; SD) | Posttest (M; SD) | |
Sustaining mutual understanding | 0.50 (0.50) | 0.70 (0.46) |
Information Pooling | 0.52 (0.50) | 0.67 (0.48) |
Reaching consensus | 0.62 (0.49) | 0.62 (0.49) |
Time management | 0.02 (0.12) | 0.06 (0.25) |
Reciprocal interaction | 0.73 (0.45) | 0.62 (0.49) |
Dialogue management | 0.36 (0.48) | 0.49 (0.50) |
Individual task orientation | 0.16 (0.37) | 0.05 (0.21) |
Task division | 0.19 (0.39) | 0.13 (0.34) |
Technical coordination | 0.00 (0.00) | 0.10 (0.30) |
RQ 3)
How did the groups perform in terms of collaboration quality before and after collaborative reflection?
Our next research question was concerned with the collaboration quality that groups exhibited during the learning phase and the testing phase. As illustrated in Table 6, the overall collaboration quality of groups in the laboratory was slightly higher in both phases compared to the groups in the field study.
Table 6. Collaboration quality during the learning phase and the testing phase
Study | ||
|---|---|---|
Laboratory (n = 16) | Field (n = 9) | |
Phase | M (SD) | M (SD) |
Learning Phase | 1.11 (0.29) | 1.04 (0.17) |
Testing Phase | 1.23 (0.15) | 1.16 (0.15) |
We further inspected the collaboration quality in more detail. Table 7 summarizes the collaboration quality on the different dimensions for both studies during the learning and the testing phase.
Table 7. Comparison of collaboration quality on the different dimensions
Dimension | Learning phase | Testing phase | ||
|---|---|---|---|---|
Laboratory (M; SD) | Field (M; SD) | Laboratory (M; SD) | Field (M; SD) | |
Joint information processing | 1.11 (0.40) | 1.29 (0.30) | 1.23 (0.30) | 1.42 (0.21) |
Sustaining mutual understanding | 1.05 (0.57) | 1.27 (0.48) | 1.38 (0.38) | 1.36 (0.17) |
Dialogue management | 1.05 (0.27) | 0.81 (0.18) | 1.12 (0.27( | 0.88 (0.21) |
Information pooling | 1.04 (0.45) | 1.29 (0.32) | 1.13 (0.38) | 1.42 (0.22) |
Reaching consensus | 1.19 (0.39) | 1.28 (0.30) | 1.33 (0.34) | 1.42 (0.30) |
Task division | 1.32 (0.24) | 1.20 (0.19) | 1.24 (0.28) | 1.18 (0.16) |
Time management | 0.81 (0.40) | 0.36 (0.45) | 1.19 (0.40) | 1.06 (0.47) |
Technical coordination | 0.90 (0.60) | 0.96 (0.37) | 1.00 (0.48) | 0.98 (0.50) |
Reciprocal interaction | 1.12 (0.24) | 1.11 (0.23) | 1.28 (0.24) | 0.99 (0.25) |
Individual task orientation | 1.39 (0.49) | 1.05 (0.38) | 1.39 (0.43) | 0.87 (0.42) |
In the learning phase, groups in the field study performed best on the dimensions ‘individual task orientation’, ‘sustaining mutual understanding’ and ‘reaching consensus’, which are important aspects of interdisciplinary collaboration. Groups in the field study improved their performance in the testing phase regarding ‘information pooling’, ‘reaching consensus’ and sustaining mutual understanding’.
In contrast, groups in the laboratory study achieved the highest collaboration quality during the learning phase on the dimensions ‘individual task orientation’, ‘task division’ and ‘reaching consensus’. These groups also improved their collaboration on several dimensions, most notably regarding ‘sustaining mutual understanding’, ‘reaching consensus’, and ‘time management’.
RQ 4)
How do the samples in the laboratory study and the field study compare in terms of performance in the knowledge test and collaboration quality?
Regarding their age, the participants from the laboratory study (M = 23.42; SD = 4.50) and the field study (M = 22.44; SD = 1.97) did not differ significantly (BF01 = 4.94; 95%-CI: [-0.33, 0.36]).
Knowledge about effective collaboration
Further, we compared the samples from the two studies in terms of their knowledge about effective collaboration before the collaboration and after performing the collaborative reflection using hierarchical linear modeling (HLM). A descriptive overview of prior knowledge, knowledge after the learning phase and the knowledge-gain can be found in Table 4.
The intercept-only model (null-model) showed that the intraclass correlation (ICC) for the level-2 variable (the small groups) was ICC = 0.23, indicating that approx. 23% of the variance in participants’ knowledge about productive collaboration could be attributed to the small-groups. To compare the samples, we calculated a random-intercept model. This model predicted participants’ knowledge based on the study type in which they participated (laboratory versus field), while allowing the small groups within the studies to have random intercepts. The models are shown in Table 8
Table 8. Overview of MLMs comparing the development of knowledge in both studies
Null-Model | Random Intercept Model | |||||
|---|---|---|---|---|---|---|
predictors | Estimates | 95%-CI | p | Estimates | 95%-CI | p |
Intercept | 5.24 | 4.75–5.73 | < 0.05 | 4.95 | 4.26–5.63 | < 0.05 |
Study | - | - | - | -0.83 | -1.81–0.16 | 0.10 |
Time (Pre vs. Post) | - | - | - | 1.94 | 1.16–2.72 | < 0.05 |
Study * Time | - | - | - | -1.08 | -2.13 – -0.03 | < 0.05 |
Note. n = 239 participants in N = 33 groups
Coding of the studies for the model: Laboratory = 1, Field = 2
The random-intercept model showed that the average knowledge of participants in the laboratory study before the learning phase (pretest) of α = 4.95 was significantly different from 0. The prior knowledge of participants in the field study was γ = -0.83 points lower, however, this difference between the conditions was not significant (p > 0.05). Participants in the laboratory study increased their knowledge about effective collaboration by b = 1.94 (p < 0.05). The significant interaction between study and measuring time shows that participants in both studies acquired different amounts of new knowledge over the course of the learning phase. Specifically, participants in the field study acquired b = -1.08 fewer knowledge elements compared to their peers in the laboratory study.
A deviance test showed that the random-intercept model fits the data significantly better than the intercept-only (null) model, χ²(3) = 36.48, p < 0.05, meaning that the small groups differed significantly from each other.
Collaboration quality
Next, we compared the samples regarding the collaboration quality during the learning phase and the testing phase (see Table 7 for an overview). A Bayesian repeated measures ANOVA with study type (laboratory or field study) as the between-subjects factor, phase (learning phase, testing phase) as within-subjects factor, and collaboration quality as dependent variable indicates weak evidence (BF10 = 1.71) for a main effect of phase on collaboration quality, weak evidence (BF01 = 1.90) that there was no significant main effect of study type on collaboration quality, and weak evidence (BF01 = 2.50) that there was no significant interaction between phase and study type. These findings suggest that groups in both studies improved their collaboration quality, while the collaboration quality did not differ between the studies. However, the small Bayes factors indicate that additional data is needed to further corroborate these findings.
Perceived collaboration quality
After the learning phase, all group members assessed the collaboration quality in their group. On average, the groups in the laboratory study (M = 4.45; SD = 0.47) and in the field study (M = 4.03; SD = 0.46) perceived their collaboration quality as close to the maximum across all dimensions of effective collaboration. A Bayesian t-test for independent samples with study (field vs. laboratory) as grouping variable and the grand-mean over all GAT-dimensions as the dependent variable revealed that that our data provide moderate evidence of BF10 = 4.46 that groups in the two studies differed in their self-assessment of collaboration quality, with groups in the laboratory story perceiving their collaboration quality as higher than groups in the field study (large effect d = 0.90, 95%-CI: [0.1, 1.6]).
Correlation between rated and perceived collaboration quality
Next, we explored to which degree groups’ self-assessment of their collaboration quality was correlated with our fine-grained rating of the collaboration. A bi-variate correlation analysis (Kendall’s ðœB) suggests that, overall, groups’ self-assessment correlated positively with the rating of the collaboration quality (moderate correlation, ðœB = 0. 48, CI: [0.17, 0.67], BF10 = 59.82). Figure 4 illustrates this correlation. As suggested by the correlation coefficient, groups’ perception of the quality of their collaboration is associated with our rating of the recorded interaction. In the graph it becomes visible that groups from both studies perceived their interaction quality as rather high, while values below 3 did not occur. Therefore, the regression line can only be drawn for this part of the scale.
[See PDF for image]
Fig. 4
Correlation between self- and expert raring of collaboration quality (learning phase)
We further explored this correlation for each study separately. For the laboratory study, we found weak evidence for a positive and moderate correlation between groups’ self-assessment and expert-rating (ðœB = 0.38 (CI: [0.03, 0.62]; BF10 = 2.34). In the field study, we found moderate evidence for a positive and high correlation, ðœB = 0.63 (CI: [0.05, 0.81]; BF10 = 4.54). This suggests that participants in the field study perceived the quality of their collaboration more accurately than their peers in the laboratory study.
Discussion
Educational technologies can support learners’ during collaborative problem solving and help them acquire the skills necessary to become proficient collaborators. Before implementing technological and pedagogical innovations into teaching, however, it is important to test their effectiveness not only in the laboratory but also under realistic conditions in the field. In the present study, we compared the effects of a collaborative reflection scaffold in a field context with the effects under laboratory conditions. Our results showed that groups of civil engineering students followed the instructions of the collaborative reflection and derived plans on how to improve their collaboration during a subsequent collaborative task (RQ 1). This collaborative reflection promoted the acquisition of explicit knowledge about effective collaboration (RQ 2). However, the quality of the collaboration in a subsequent collaboration only increased slightly in the field setting (RQ 3). Comparing the results of the field study with the results of the laboratory study helped put the findings of the field study into perspective (RQ 4). Specifically, the civil engineering students entered the field study with less prior knowledge about effective collaboration and showed a smaller knowledge gain than the general sample of higher education students in the laboratory study. Furthermore, civil engineering students differed from the groups in the laboratory study in terms of the reflection process, for instance the number of plans they created, and the resulting collaboration quality. With the field study, we could increase the population and ecological validity of our previous data collection, and therefore of the results regarding the implementation of collaborative reflection in collaborative learning situations (Bracht & Glass, 1968; Fahmie et al., 2023). We discuss these findings in more detail in the following.
The process of collaborative reflection
Our data suggests that the groups in the field study perceived their collaboration quality as very high regarding all dimensions of effective collaboration. Nevertheless, the groups created plans on how to adapt the collaboration during the subsequent collaboration. These plans predominantly targeted the dimensions that are required for effective interdisciplinary collaboration (Meier et al., 2007), such as ‘sustaining mutual understanding’, ‘information pooling’, and ‘reaching consensus’. This finding is in line with the results reported by Schürmann et al. (2025) who found that groups indeed utilized the opportunity for debriefing, reflected on their previous collaboration and developed plans, suggesting that groups of university students are capable of monitoring and adaptively regulating their collaboration (also see Sobocinski et al., 2020).
Our analysis also revealed that groups in the field were able to assess the quality of their collaboration with some degree of accuracy. This is in line with the findings reported by Sobocinski et al. (2020) and our the laboratory study (Authors), in which we found a moderate correlation between the our rating of the collaboration quality and groups’ perception. Thus, the field study corroborated our findings from the laboratory.
The ability to accurately monitor the collaboration (Borge et al., 2018) by observing the collaboration process and comparing the current state to a desired goal-state (Järvelä et al., 2018) is crucial for regulation of the collaboration. The importance of the accuracy of this self-assessment should not be underestimated, given that regulatory processes are dependent on recognizing a need for regulation. Winne and Hadwin (1998) established this relation for self-regulated learning as IF-THEN-ELSE contingency whereby monitoring uncovers a discrepancy between established standards and current products, which then points to the need to regulate. If this discrepancy is not detected by groups, they may fail to adequately regulate their collaborative processes. A striking finding in our study was that participants generally perceived the collaboration as high, while our ratings often ranged in the midfield of the scale. While our findings suggest that groups are capable to assess the collaboration quality, this discrepancy suggests that groups may lack a sense of the possible range of collaboration quality. Still, we acknowledge that the correlation between the groups’ perception of the collaboration quality and our rating of the interaction was only moderate. Therefore, it can be assumed that groups still need to improve their monitoring accuracy. This finding therefore presents an intriguing avenue for future research.
Effect on the acquisition of knowledge about effective collaboration
All participants in the field and in the laboratory held knowledge about effective collaboration upon entering the studies. In a direct comparison, we found that participants acquired more knowledge about collaboration under laboratory conditions. This is a relevant finding given that Schürmann et al. (2025) did not find an effect of a collaborative reflection activity on knowledge when using a scale for perceived knowledge gain.
One potential explanation for this finding may be the advantage of participants in the laboratory study in terms of their prior knowledge. While we only found a small difference in prior knowledge between the participants from the two studies (weak evidence), the predictive power of prior knowledge for learning is well established (Simonsmeier et al., 2022). It can be assumed that participants, who entered the study with more prior knowledge about effective collaboration, could extract and assimilate more additional knowledge elements from the GAT, and the discussion during the collaborative reflection, respectively. This conjecture of an aptitude-treatment effect should be investigated in future studies.
Further explanations for the difference in knowledge acquisition are rooted in differences in the design of the collaboration setting. First, groups in the field study had less time for their reflection compared to groups in the laboratory study. This difference in time for reflection (4 min) was due to changes we made when implementing the reflection activity into the field (e.g., adapting questions in the reflection questionnaire). If groups spent less time discussing and developing plans the group’s members had less exposure to elaboration processes which helped them construct and integrate new insights on fruitful collaboration. Against this background, it would be worthwhile for future studies to investigate the relationship between characteristics of the reflection process and the acquisition of knowledge about effective collaboration in more detail.
A third explanation for the difference could be due to the different group sizes in the field and laboratory study. Larger groups, as in the field study, make it more difficult for individual group members to participate in discussions which leads to lower participation (e.g., Johnson & Johnson, 2009; Shaw, 2013). Therefore, individual group members receive fewer opportunities to repeat and co-construct knowledge about effective collaboration which may reflect in lower scores in the knowledge test.
Finally, differences in the acquisition of knowledge about collaboration may be affected by differences in participants’ motivation. While our data does not allow to this test this conjecture, it can be expected that differences in participants’ achievement goals intrinsic motivation, situational interest during the collaboration, may have led to different levels in cognitive engagement during knowledge co-construction, as well as strategy use (Kempler Rogat et al., 2013; Rienties et al., 2009; Schoor & Bannert, 2011).
Nevertheless, the field study adds to research on the role of reflection during collaboration that collaboration not only improves problem solving performance (Gabelica et al., 2014), and interaction (Eshuis et al., 2019), but also promotes learning to collaborate. To increase this effect further, it may be helpful to add a collaborative planning phase before the collaboration as described by Zheng et al. (2019). During such a phase, group members may receive an additional opportunity to share ideas about effective collaboration which they then can revisit during reflection.
Effect on collaboration quality
With respect to the effect of a collaborative reflection activity on the collaboration, while our data did not provide enough evidence that supports a strong interpretation, the results are promising in the sense that a collaboration reflection activity may help groups improve the quality of their collaboration. Based on theoretical considerations (Strauß & Rummel, 2023; Gabelica et al., 2014; Heitzmann et al., 2023; Mamede & Schmidt, 2017; Dillenbourg & Jermann, 2008), we expected that groups leverage the collaborative reflection to review their previous collaboration, identify potential aspects of the collaboration that require improvement, agree on plans that help achieve these improvements and subsequently implement respective strategies. However, in both studies the increase in collaboration quality was only small. This finding mirrors the mixed results reported by Schürmann et al. (2025) who did not find an overall effect of a debriefing activity on participants’ perceived collaboration processes, but on coordination, monitoring and reflection. This absence of an effect on subsequent collaboration observed in groups that had engaged in collaborative reflection, was not anticipated. In the following, we propose two possible explanations for this finding. First, this result may be explained by the short time of the training activity but also needs to be discussed against the background of our rating schema for collaboration quality (see Limitations). Research on the development of mastery (e.g., Ericsson, 2003) and internal collaboration scripts (e.g., Fischer et al., 2013) suggests that learners need multiple learning opportunities to develop, apply and automate new configurations of their internal collaboration scripts. This conjecture may be tested in future studies that employ a longitudinal approach to understand when explicit knowledge about effective collaboration reliably translates into higher degrees of collaboration quality, and how these two factors are related to problem solving performance of the group.
A second potential explanation concerns groups’ monitoring and control during collaboration. Finding only a moderate positive correlation between our rating of the collaboration quality and groups’ perception implies that not all groups were equally competent to monitor their collaboration accurately. In this case, groups may have overlooked planning and subsequently implementing adequate remedial actions. If groups either did not monitor their collaboration accurately or did not adapt their collaboration effectively, their collaboration was unlikely to improve (Sobocinski et al., 2020; Strauß & Rummel, 2023).
Limitations
When interpreting these results, some limitations must be considered. First, given the available sample size additional data collected in the field is required to gain more confidence in the findings, as indicated by Bayes factors below 10. Small samples are a typical challenge for CSCL research (as evidenced in meta analyses, e.g., Radkowitsch et al., 2020 and reviews in the field, e.g., Schürmann et al., 2024), especially when the group is the level of analysis (Cress, 2008). Sampling groups is challenging due to organizational constraints, such as scheduling data collections and making sure that nobody drops out. Thus, conforming to the assumptions of HLMs is a challenge for studies such as the present one. Nevertheless, the fact that the findings from the field study are mostly in line with the findings from the laboratory study, increased our confidence in the positive effects of the reflection scaffold.
Second, it was necessary to rescale the GAT-questionnaire from the laboratory study (7-point Likert scale) to match it to the scale that was employed in the field study (5-point Likert scale). Normalizing the scale from a 7-point scale to a 5-point scale reduced the variance available for statistical analyses. However, in comparison to the opposite approach, this “downscaling” does not artificially create variance, and thus, can be assumed to be more conservative. As exemplified by our analyses, the resulting data still allowed for relevant insights.
Another limitation concerns the rating schema for collaboration quality. The original rating schema by Meier et al. (2007) employs a 5-point scale to rate collaboration quality, whereas we utilized a 3-point scale. This decision was made to address the inconsistent and sometimes dissatisfactory interrater reliability reported in previous studies (Kahrimanis et al., 2012; Meier et al., 2007; Rummel et al., 2009). Consequently, our analysis may have been less sensitive to small differences between groups while allowing us to achieve high, and consistent interrater reliability.
A final limitation is the lack of a control condition in the field. This decision was based on two considerations. First withholding an intervention that has been shown to be effective in the laboratory study for a part of the sample (i.e., control condition) would have led to inappropriate disadvantage regarding the graded final examination in the course. Second, including a control condition in the field would have reduced the data available to test the effect of the instructional support. Thus, we used the data from the laboratory study for comparison. We acknowledge that this approach does not afford a strict comparison. Still, finding comparable results in the field, supports the generalizability of our findings. Future studies that can recruit a larger sample or implement a different research design, such as longitudinal studies or studies that use a within-subjects design, are desirable to further corroborate our findings.
The discussion of the limitations of our field study has made it clear that despite the insights that we have gained, there are further interesting lines of work to be explored in future studies.
Practical implications
With respect to educational practice, our findings have several implications. First, we found that university students are aware of several aspects of productive collaboration, however, they do not hold comprehensive explicit knowledge. Similarly, groups of students in artificial and natural groups, can show mediocre collaboration quality. Thus, university students may benefit from further training that enhances their collaboration skills. Our findings suggest that a reflection scaffold as presented in our study can be implemented into university teaching. Participants in the laboratory but also students in a regular course were willing to follow the instructions and benefitted in terms of knowledge about effective collaboration, which is among the most relevant domain-general skills (Asia Society & OECD, 2018; van Laar et al., 2017). With respect to the discussion about the degree of coercion of instructional support, the work presented in this paper makes us confident that providing groups of learners with the opportunity to reflect on their collaboration and implement actions that they planned is a viable strategy to promote collaborative learning in higher education settings.
Outlook and conclusion
The results we obtained in two studies suggest that a guided collaborative reflection activity represents an autonomy-supportive way to help learners acquire new knowledge about effective collaboration. Our studies add to the discourse in the field of CSCL that providing instructional support that allows for more learner agency and acknowledge learners’ prior experience is indeed valuable to pursue (Wise & Schwarz, 2017).
Future studies that focus on collaborative reflection processes and how they affect social regulation may consider additional contextual variables such as students’ intrinsic motivation for the course and for participating in a group project, their situational interest, or the perceived relevance of the course for their studies. Similarly, population characteristics such as collaboration related self-efficacy, or participants’ goal orientation may shed further light on learning gains, since these characteristics affect cognitive and behavioral engagement during collaboration (see e.g., Rogat et al. 2013). Furthermore, future studies may complement the analysis of data about groups’ perception of their collaboration and video recordings, with other behavioral traces or sensor data (e.g., Järvenoja et al., 2020; Martinez-Maldonado et al., 2021; Wise et al., 2021). While previous studies have explored the use of collaboration analytics for a diverse set of outcomes (see e.g., Martinez-Maldonado et al., 2021; Olsen et al., 2020; Praharaj et al., 2021; Schneider et al., 2021), predicting relevant outcomes of collaborative learning and problem solving has yielded mixed results (Schneider et al., 2021). One core challenge for effectively using multimodal learning analytics for measuring is to safeguard the movement “from clicks to constructs” (Buckingham Shum & Deakin Crick, 2016), that is, using low-level metrics to develop reliable and valid operationalizations of constructs. This process is far from trivial and requires expertise in theories of learning and collaboration (Martinez-Maldonado et al., 2021; Wise et al., 2021; Wise & Shaffer, 2015), psychometrics (for a demonstration see Landers et al., 2022; Drachsler & Goldhammer, 2020), and computer and data science. One study that carefully developed operationalizations from low-level indicators can be found in Zhou et al. (2024). Against this background, we posit that further data sources may complement analysis such as those presented here, if constructs are carefully defined and measured.
Taken together, our work represents an example of how we can derive pedagogical and technological innovations from theories of learning and collaboration and test them in different contexts. Empirically testing the effects of instructional support in both controlled laboratory settings and externally valid field contexts increased our confidence that guided collaborative reflection can be employed as an effective teaching approach to promote collaboration skills that is needed to tackle the complex challenges of our future.
References
Adamu, Z; Emmit, S; Soetanto, R. Social BIM: Co-creation with shared situational awareness. ITcon Technology Strategies for Collaborative Working; 2015; 20, pp. 230-252.
Asia, Society, & OECD. (2018). & Teaching for Global Competence in a Rapidly Changing World. Advance online publication. https://doi.org/10.1787/9789264289024-en
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear Mixed-Effects models using lme4. Journal of Statistical Software, 67(1). https://doi.org/10.18637/jss.v067.i01
Bodemer, D., Janssen, J., & Schnaubert, L. (2018). Group Awareness Tools for Computer-Supported Collaborative Learning. In F. Fischer, C. E. Hmelo-Silver, S. R. Goldman, & P. Reimann (Eds.), International Handbook of the Learning Sciences (pp. 351–358). Routledge. https://doi.org/10.4324/9781315617572-34
Borge, M; Ong, YS; Rosé, CP. Learning to monitor and regulate collective thinking processes. International Journal of Computer-Supported Collaborative Learning; 2018; 13,
Borrmann, A., König, M., Koch, C., & Beetz, J. (2015). Building information modeling. Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-05606-3
Boud, D. (Ed.). (1988). Developing student autonomy in learning (2. ed., repr). Kogan Page.
Boud, D., Keogh, R., & Walker, D. (1985). Promoting reflection in learning: A model. In R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning (pp. 18–40). Taylor and Francis.
Bracht, GH; Glass, GV. The external validity of experiments. American Educational Research Journal; 1968; 5,
Brodbeck, F; Kerschreiter, R; Mojzisch, A; Schulz-Hardt, S. Group decision making under conditions of distributed knowledge: The information asymmetry model. Academy of Management Review; 2007; 32,
Buckingham Shum, S; Deakin Crick, R. Learning analytics for 21st century competencies. Journal of Learning Analytics; 2016; 3,
Chi, MTH; Wylie, R. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist; 2014; 49,
Cohen, J. (1992). A power primer. In A. E. Kazdin (Ed.), Methodological issues and strategies in clinical research (pp. 279–284). American Psychological Association. https://doi.org/10.1037/14805-018
Connolly, P; Keenan, C; Urbanska, K. The trials of evidence-based practice in education: A systematic review of randomised controlled trials in education research 1980–2016. Educational Research; 2018; 60,
Cress, U. The need for considering multilevel analysis in CSCL research—An appeal for the use of more advanced statistical methods. International Journal of Computer-Supported Collaborative Learning; 2008; 3,
Dehler, J; Bodemer, D; Buder, J; Hesse, FW. Providing group knowledge awareness in computer-supported collaborative learning: Insights into learning mechanisms. Research and Practice in Technology Enhanced Learning; 2009; 04,
Drachsler, H., & Goldhammer, F. (2020). Learning Analytics and eAssessment—Towards Computational Psychometrics by Combining Psychometrics with Learning Analytics. In D. Burgos (Ed.), Lecture Notes in Educational Technology. Radical Solutions and Learning Analytics: Personalised Learning and Teaching Through Big Data (1st ed. 2020, pp. 67–80). Springer Singapore; Imprint Springer. https://doi.org/10.1007/978-981-15-4526-9_5
Ericsson, K. A. (2003). The acquisition of expert performance as problem solving: Construction and modification of mediating mechanisms through deliberate practice. In J. E. Davidson & R. J. Sternberg (Eds.), EBSCOhost eBook Collection. The psychology of problem solving (Vol. 2003, pp. 31–83). Cambridge University Press.
Eshuis, EH; Vrugte, J; Anjewierden, A; Bollen, L; Sikken, J; de Jong, T [Ton]. Improving the quality of vocational students’ collaboration and knowledge acquisition through instruction and joint reflection. International Journal of Computer-Supported Collaborative Learning; 2019; 14,
Fahmie, TA; Rodriguez, NM; Luczynski, KC; Rahaman, JA; Charles, BM; Zangrillo, AN. Toward an explicit technology of ecological validity. Journal of Applied Behavior Analysis; 2023; 56,
Fessl, A; Blunk, O; Prilla, M; Pammer, V. The known universe of reflection guidance: A literature review. International Journal of Technology Enhanced Learning; 2017; 9,
Field, A., Miles, J., & Field, Z. (2012). Discovering statistics using R. Sage.
Fischer, F; Kollar, I; Stegmann, K; Wecker, C. Toward a script theory of guidance in Computer-Supported collaborative learning. Educational Psychologist; 2013; 48,
Gabelica, C; van den Bossche, P; Segers, M; Gijselaers, W. Dynamics of team reflexivity after feedback. Frontline Learning Research; 2014; 2,
Graesser, A., Foltz, P. W., Rosen, Y., Shaffer, D. W., Forsyth, C., & Germany, M. L. (2018). Challenges of Assessing Collaborative Problem Solving. In E. Care, P. Griffin, & M. Wilson (Eds.), Educational Assessment in an Information Age. Assessment and Teaching of 21st Century Skills (pp. 75–91). Springer International Publishing. https://doi.org/10.1007/978-3-319-65368-6_5
Guo, L. How should reflection be supported in higher education? — A meta-analysis of reflection interventions. Reflective Practice; 2022; 23,
Hattie, J; Timperley, H. The power of feedback. Review of Educational Research; 2007; 77,
Heitzmann, N; Stadler, M; Richters, C; Radkowitsch, A; Schmidmaier, R; Weidenbusch, M; Fischer, MR. Learners’ adjustment strategies following impasses in simulations - Effects of prior knowledge. Learning and Instruction; 2023; 83, 101632. [DOI: https://dx.doi.org/10.1016/j.learninstruc.2022.101632]
Janssen, J; Bodemer, D. Coordinated Computer-Supported collaborative learning: awareness and awareness tools. Educational Psychologist; 2013; 48,
Janssen, J; Erkens, G; Kanselaar, G. Visualization of agreement and discussion processes during computer-supported collaborative learning. Computers in Human Behavior; 2007; 23,
Janssen, J; Erkens, G; Kirschner, PA. Group awareness tools: It’s what you do with it that matters. Computers in Human Behavior; 2011; 27,
Janssen, J., Cress, U., Erkens, G., & Kirschner, P. A. (2013). Multilevel analysis for the analysis of collaborative learning. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan, & A. M. O’Donnell (Eds.), Educational psychology handbook. The international handbook.of collaborative learning. Taylor and Francis. https://doi.org/10.4324/9780203837290.ch6
Järvelä, S., Hadwin, A. F., Malmberg, J., & Miller, M. (2018). Contemporary perspectives of regulated learning in collaboration. In F. Fischer (Ed.), International handbook of the learning sciences (pp. 127–136). Routledge.
Järvenoja, H; Malmberg, J; Törmänen, T; Mänty, K; Haataja, E; Ahola, S; Järvelä, S. A collaborative learning design for promoting and analyzing adaptive motivation and emotion regulation in the science classroom. Frontiers in Education; 2020; 5, 111. [DOI: https://dx.doi.org/10.3389/feduc.2020.00111]
JASP Team (2024). JASP (Version 0.18.3) [Computer software]. https://jasp-stats.org/
Jeong, H; Hmelo-Silver, CE. Seven affordances of Computer-Supported collaborative learning: How to support collaborative learning?? How can technologies help??. Educational Psychologist; 2016; 51,
Jermann, P; Dillenbourg, P. Group mirrors to support interaction regulation in collaborative problem solving. Computers & Education; 2008; 51,
Johnson, DW; Johnson, RT. An educational psychology success story: Social interdependence theory and cooperative learning. Educational Researcher; 2009; 38,
Kahrimanis, G., [Georgios], Chounta, I. A., & Avouris, N. (2012). Validating empirically a rating approach for quantifying the quality of collaboration. In T. Daradoumis, S. N. Demetriadis, & F. Xhafa (Eds.), Studies in computational intelligence. Intelligent adaptation and personalization techniques in Computer-Supported collaborative learning (Vol. 408, pp. 295–310). Springer. https://doi.org/10.1007/978-3-642-28586-8_13
Kempler Rogat, T., Linnenbrink-Garcia, L., & DiDonato, N. (2013). Motivation in collaborative groups. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan, & A. M. O’Donnell (Eds.), Educational psychology handbook. The international handbook.of collaborative learning (pp. 250–267). Taylor and Francis.
Kim, JS. Making every study count: Learning from replication failure to improve intervention research. Educational Researcher; 2019; 48,
Kruschke, JK; Liddell, TM. Bayesian data analysis for newcomers. Psychonomic Bulletin & Review; 2018; 25,
Landers, RN; Auer, EM; Mersy, G; Marin, S; Blaik, J. You are what you click: Using machine learning to model trace data for psychometric measurement. International Journal of Testing; 2022; 22,
Lin, JW; Tsai, CW. The impact of an online project-based learning environment with group awareness support on students with different self-regulation levels: An extended-period experiment. Computers & Education; 2016; 99, pp. 28-38. [DOI: https://dx.doi.org/10.1016/j.compedu.2016.04.005]
Lüdecke, D. (2024). sjPlot: Data Visualization for Statistics in Social Science (Version R package version 2.8.15) [Computer software]. https://CRAN.R-project.org/package=sjPlot
Mamede, S; Schmidt, HG. Reflection in medical diagnosis: A literature review. Health Professions Education; 2017; 3,
Martinez-Maldonado, R; Gašević, D; Echeverria, V; Fernandez Nieto, G; Swiecki, Z; Shum, B. What do you mean by collaboration analytics?? A conceptual model. Journal of Learning Analytics; 2021; 8,
Meier, A; Spada, H; Rummel, N. A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning; 2007; 2,
Olsen, JK; Sharma, K; Rummel, N; Aleven, V. Temporal analysis of multimodal data to predict collaborative learning outcomes. British Journal of Educational Technology; 2020; 51,
Phielix, C; Prins, FJ; Kirschner, PA; Erkens, G; Jaspers, J. Group awareness of social and cognitive performance in a CSCL environment: Effects of a peer feedback and reflection tool. Computers in Human Behavior; 2011; 27,
Praharaj, S; Scheffel, M; Drachsler, H; Specht, M. Literature review on Co-Located collaboration modeling using multimodal learning Analytics—Can we go the whole nine yards??. IEEE Transactions on Learning Technologies; 2021; 14,
Radkowitsch, A; Vogel, F; Fischer, F. Good for learning, bad for motivation? A meta-analysis on the effects of computer-supported collaboration scripts. International Journal of Computer-Supported Collaborative Learning; 2020; 15,
Radović, S; Firssova, O; Hummel, HGK; Vermeulen, M. The case of socially constructed knowledge through online collaborative reflection. Studies in Continuing Education; 2023; 45,
Reeve, J; Cheon, SH. Autonomy-supportive teaching: Its malleability, benefits, and potential to improve educational practice. Educational Psychologist; 2021; 56,
Renner, B; Prilla, M; Cress, U; Kimmerle, J. Effects of prompting in reflective learning tools: Findings from experimental field, lab, and online studies. Frontiers in Pychology; 2016; 7, 820. [DOI: https://dx.doi.org/10.3389/fpsyg.2016.00820]
Rienties, B; Tempelaar, D; van den Bossche, P; Gijselaers, W; Segers, M. The role of academic motivation in Computer-Supported collaborative learning. Computers in Human Behavior; 2009; 25,
Rummel, N; Spada, H. Learning to collaborate: An instructional approach to promoting collaborative problem solving in Computer-Mediated settings. Journal of the Learning Sciences; 2005; 14,
Rummel, N; Spada, H; Hauser, S. Learning to collaborate while being scripted or by observing a model. International Journal of Computer-Supported Collaborative Learning; 2009; 4,
Rummel, N., Deiglmayr, A., Spada, H., Kahrimanis, G., [George], & Avouris, N. (2011). Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme. In S. Puntambekar, G. Erkens, & C. Hmelo-Silver (Eds.), Analyzing Interactions in CSCL (pp. 367–390). Springer US. https://doi.org/10.1007/978-1-4419-7710-6_17
Schnaubert, L; Vogel, F. Integrating collaboration scripts, group awareness, and self-regulation in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning; 2022; 17,
Schneider, B., Sung, G., Chng, E., & Yang, S. (2021). How can High-Frequency sensors capture collaboration?? A review of the empirical links between multimodal metrics and collaborative constructs. Sensors (Basel, Switzerland), 21(24). https://doi.org/10.3390/s21248185
Schoor, C; Bannert, M. Motivation in a computer-supported collaborative learning scenario and its impact on learning activities and knowledge acquisition. Learning and Instruction; 2011; 21,
Schürmann, V; Marquardt, N; Bodemer, D. Conceptualization and measurement of peer collaboration in higher education: A systematic review. Small Group Research; 2024; 55,
Schürmann, V., Bodemer, D., & Marquardt, N. (2025). Applying debriefings in the context of higher education: How joint reflection fosters students’ collaborative problem solving. Social Psychology of Education, 28(1). https://doi.org/10.1007/s11218-024-09991-3
Shaw, RS. The relationships among group size, participation, and performance of programming Language learning supported with online forums. Computers & Education; 2013; 62, pp. 196-207. [DOI: https://dx.doi.org/10.1016/j.compedu.2012.11.001]
Simonsmeier, BA; Flaig, M; Deiglmayr, A; Schalk, L; Schneider, M. Domain-specific prior knowledge and learning: A meta-analysis. Educational Psychologist; 2022; 57,
Slavin, RE. Evidence-Based education policies: Transforming educational practice and research. Educational Researcher; 2002; 31,
Sobocinski, M; Järvelä, S; Malmberg, J; Dindar, M; Isosalo, A; Noponen, K. How does monitoring set the stage for adaptive regulation or maladaptive behavior in collaborative learning?. Metacognition and Learning; 2020; 15,
Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478. https://doi.org/10.1037//0022-3514.48.6.1467
Strauß, S; Rummel, N. Promoting regulation of equal participation in online collaboration by combining a group awareness tool and adaptive prompts. But does it even matter?. International Journal of Computer-Supported Collaborative Learning; 2021; 16,
Strauß, S., & Rummel, N. (2023). Feed-Back About the Collaboration Process from a Group Awareness Tool. Potential Boundary Conditions for Effective Regulation. In O. Noroozi & B. de Wever (Eds.), Social Interaction in Learning and Development. The Power of Peer Learning (pp. 183–213). Springer International Publishing. https://doi.org/10.1007/978-3-031-29411-2_9
Strauß, S., Tunnigkeit, I., Eberle, J., Avdullahu, A., & Rummel, N. (2025). Comparing the effects of a collaboration script and collaborative relection on promoting knowledge about good collaboration and effective interaction. International Journal of Computer-Supported Collaborative Learning, 20, 121–159. https://doi.org/10.1007/s11412-024-09430-7
van Doorn, J; van den Bergh, D; Böhm, U; Dablander, F; Derks, K; Draws, T; Etz, A; Evans, NJ; Gronau, QF; Haaf, JM; Hinne, M; Kucharský, Š; Ly, A; Marsman, M; Matzke, D; Gupta, ARKN; Sarafoglou, A; Stefan, A; Voelkel, JG; Wagenmakers, EJ. The JASP guidelines for conducting and reporting a bayesian analysis. Psychonomic Bulletin & Review; 2021; 28,
van Laar, E; van Deursen, AJ; van Dijk, JA; de Haan, J. The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior; 2017; 72, pp. 577-588. [DOI: https://dx.doi.org/10.1016/j.chb.2017.03.010]
van Treeck, C., Fischer, E., & Zander, J. (2017). BIM (Building Information Modeling)– Nutzen für Sachverständige? In M. Oswald & M. Zöller (Eds.), Aachener Bausachverständigentage 2017 (pp. 166–171). Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-18370-7_19
Vauras, M., Volet, S., & Bobbitt Nolen, S. (2019). Supporting Motivation in Collaborative Learning: Challenges in the Face of an Uncertain Future. In E. Gonida & M. Lemos (Eds.), Advances in motivation and achievement: Vol. 20. Motivation in education at a time of global change: Theory, research, and implications for practice (Vol. 20, pp. 187–203). Emerald publishing. https://doi.org/10.1108/S0749-742320190000020012
Wagenmakers, EJ; Love, J; Marsman, M; Jamil, T; Ly, A; Verhagen, J; Selker, R; Gronau, QF; Dropmann, D; Boutin, B; Meerhoff, F; Knight, P; Raj, A; van Kesteren, EJ; van Doorn, J; Šmíra, M; Epskamp, S; Etz, A; Matzke, D; Morey, RD. Bayesian inference for psychology. Part II: Example applications with JASP. Psychonomic Bulletin & Review; 2018; 25,
Weinberger, A; Stegmann, K; Fischer, F. Knowledge convergence in collaborative learning: Concepts and assessment. Learning and Instruction; 2007; 17,
Winne, P. H., & Hadwin, A. F. (1998). Studying as Self-Regulated Learning. In Metacognition in Educational Theory and Practice (0th ed., pp. 291–318). Routledge. https://doi.org/10.4324/9781410602350-19
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In M. Pistilli, J. Willis, D. Koch, K. Arnold, S. Teasley, & A. Pardo (Eds.), Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 203–211). ACM. https://doi.org/10.1145/2567574.2567588
Wise, AF; Schwarz, BB. Visions of CSCL: Eight provocations for the future of the field. International Journal of Computer-Supported Collaborative Learning; 2017; 12,
Wise, AF; Shaffer, DW. Why theory matters more than ever in the age of big data. Journal of Learning Analytics; 2015; 2,
Wise, A. F., & Vytasek, J. (2017). Learning Analytics Implementation Design. In C. Lang, G. Siemens, A. Wise, & D. Gasevic (Eds.), Handbook of Learning Analytics (pp. 151–160). Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17.013
Wise, A. F., Knight, S., & Shum, S. B. (2021). Collaborative Learning Analytics. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning (pp. 425–443). Springer International Publishing. https://doi.org/10.1007/978-3-030-65291-3_23
Yang, T; Choi, I. Reflection as A social phenomenon: A conceptual framework toward group reflection research. Educational Technology Research and Development; 2023; 71,
Yukawa, J. Co-reflection in online learning: Collaborative critical thinking as narrative. International Journal of Computer-Supported Collaborative Learning; 2006; 1,
Zheng, L; Li, X; Zhang, X; Sun, W. The effects of group metacognitive scaffolding on group metacognitive behaviors, group performance, and cognitive load in computer-supported collaborative learning. The Internet and Higher Education; 2019; 42, pp. 13-24. [DOI: https://dx.doi.org/10.1016/j.iheduc.2019.03.002]
Zhou, Q; Suraworachet, W; Cukurova, M. Detecting non-verbal speech and gaze behaviours with multimodal data and computer vision to interpret effective collaborative learning interactions. Education and Information Technologies; 2024; 29,
Zimmerman, B. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, Zeidner, & Moshe (Eds.), Handbook of self-regulation (pp. 13–39). Academic.
© The Author(s) 2025. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.