Combining cognitive theory and data driven approaches to examine students’ search behaviors in simulated digital environments

Abstract

Background

Digital Information Literacy (DIL) refers to the ability to obtain, understand, evaluate, and use information in digital contexts. To accurately capture various dimensions of DIL, assessment designers have increasingly looked toward complex, interactive simulation-based environments that afford more authentic learner performances. These rich assessment environments can capture process data produced by students’ goal driven interactions with digital sources but linking this data to inferences about the target constructs introduces significant measurement challenges which cognitive theory can help us address.

Methods

In this paper, we analyzed data generated from a simulated web search tool embedded within a theoretically-grounded virtual world assessment of multiple-source inquiry skills. We describe a multi-step clustering approach to identify patterns in student’s search processes by bringing together theory-informed process data indicators and sequence clustering methods.

Results

We identified four distinct search behaviors captured in students’ process data. We found that these search behaviors differed both in their contribution to the web search tool subscores as well as correlations with task level multiple-source inquiry subconstructs such as locating, evaluating, and synthesizing information. We argue that the search behaviors reflect differences in how students generate and update their task goals.

Conclusion

The data-driven approach we describe affords a qualitative understanding of student strategy use in a complex, dynamic simulation- and scenario-based environment. We discuss some of the strengths and challenges of using a theoretical understanding of multiple-source inquiry to inform how we processed, analyzed, and interpreted the data produced from this assessment tool and the implications of this approach for future research and development.