Comparing Information Retrieval Effectiveness of Learning Object Recommendation Strategies for Course Developers

Abstract

The demand for online education among students at U.S. universities has grown rapidly in recent years, creating a need to develop new online courses faster and more cost-effectively. One barrier to doing so is the difficulty of finding reusable instructional content modules known as learning objects in large repositories. This quantitative quasi-experimental study compared the effectiveness of alternative strategies for automatically recommending learning objects to course developers. Several competing recommendation strategies have been proposed but to date no systematic comparative evaluation of these strategies has been performed. In this study, three learning object recommendation strategies were compared to each other and to a simplistic control strategy. A sample of 46 course developers at a single large U.S. university rated the relevance of learning object recommendations generated for them using each of the strategies. For each strategy, precision (a measure of the strategy's ability to select relevant objects) and recall (a measure of the strategy's ability to filter out nonrelevant objects) were computed. Application of the Friedman test indicated statistically significant differences in both precision (χ²(3) = 46.47, p < .001) and recall (χ²(3) = 84.55, p < .001) among the strategies. Pairwise comparisons using the Bonferroni-Dunn post-hoc test showed the content-based and syllabus-based strategies performed significantly better than did the control strategy in precision (z = 4.52, p < .001 for content-based, and z = 4.60, p < .001 for syllabus-based) and recall (z = 4.20, p < .001 for content-based, and z = 4.24, p < .001 for syllabus-based). For the collaborative filtering strategy, precision was not significantly different from that of the control strategy (z = 0.08, p = 1.000) and recall was significantly worse than that of the control strategy ( z = -2.95, p = .019). On the basis of these findings, content-based and syllabus-based strategies appear promising as methods for recommending learning objects to course developers, whereas collaborative filtering methods appear ineffective in this application. Further research is recommended to confirm these results with other learning object repositories and in other institutional settings.