1. Introduction
Modern technologies and multimedia have been employed as effective solutions or as complements to inquiry-based educational processes for the last four decades [1]. In the field of inquiry-based history education, several solutions are available, such as Historiana [2], DocsTeach [3,4] or Virtual Historian [5] and related studies have been published for over two decades, i.e., [6,7,8,9,10]. However, none of these tools unite the development using systematic evidence from the field with a complex ambition at not merely supplemental material, but a profound teaching and learning tool for history education designed to a general curriculum jurisdiction. In this study, we present the HistoryLab (
The HistoryLab concept was validated by a pilot run conducted from 2016 to 2020, and primarily by the usage of the system during the recent COVID-19 lockdown in Czechia, which prompted the unprecedented implementation of distance learning for secondary and upper secondary school students. The unique challenges posed by COVID-19 had a significant impact on several aspects of the field of education. In the situation prior to the pandemic, the employment of modern multimedia in teaching was rather a complementary element to the classical transmissive style of teaching and was used by teachers naturally inclined to employ modern teaching methods and technology [11]. In the COVID-19 pandemic situation, the necessity to use distance and technology-aided learning methods expanded practically to the whole educational community. This involves a number of significant challenges and, rather surprisingly, feedback from the HistoryLab application, an application that clearly departs from the transmissive approach to teaching, was overwhelmingly positive. For the positive feedback we obtained during the pilot run, we suggest that the HistoryLab concept and experience is relevant to researchers and teachers in the field of large-scale distance learning and education of inquiry-based learning in history. In this study, we present some quantitative data from the project, as well as an examination of several lessons learned.
The contributions of the paper can be summarized as follows:
An explanation of an innovative concept for modern history distance learning and the method of how the system was designed from a didactic and pedagogical viewpoint,
presentation of selected technical details of the system for educational community members who are designing or implementing systems with similar goals,
presentation of experience data from the system operation during COVID-19 lockdown period in Czechia during 2020, and,
formulation of lessons learned from the project that might be utilized in similar projects in the future. These lessons result from the pilot run of the project as well as from the application of this concept for distance learning during the COVID-19 lockdown in Czechia.
The paper is organized as follows. Section 2 discusses related work and policies in the fields that are the subject of this study. Section 3 introduces the concept of the HistoryLab environment and its principles. More technical and implementational details then follow in Section 4. Section 5 describes system testing and verification. Section 6 summarizes the experience with distance learning in the 2020 COVID-19 period in Czechia, from the general experience to the details related to the operation of the HistoryLab system and lessons learned from this. Section 7 discusses possible threats to validity. Section 8 concludes the paper.
2. Related Work and Educational Policies
In this section, we summarize major educational policies related to the field of distance learning and the employment of communication technology in modern history education. Then, we analyse and discuss previous reports on the ways in which the COVID-19 situation has impacted the educational process. The third part of this section is dedicated to a summary of e-learning systems in the field of modern history.
The importance of the development of digital learning has been recently highlighted in various international education strategies. The United Nations Sustainable Development Goals (SDGs) contain 17 global goals towards a sustainable future, aimed to be fulfilled by 2030. The fourth goal [12] aims for the global expansion of the number of scholarships, in underdeveloped countries, in developed countries and other developing countries in the area of information and communications technology, technical, engineering and scientific programmes. The programmes and projects initiated by the UN are often focused on the support of language literacy in poor areas, as well as computer literacy. Several projects are also targeted at sharing indigenous knowledge of local communities in their mother tongues. The goal of these activities is to foster international cultural and natural diversity and a worldwide intercultural dialogue. A global perspective of e-learning and digital learning in general was also articulated in Future of Education and Skills 2030: OECD Learning Compass 2030 [13]. This strategy emphasizes the need to use new information and communication technology devices as well as critical thinking skills and creative skills.
Reports on the impact of COVID-19 on education mostly revolve around the issues of preparedness, early findings and reviews of the potential of e-learning and digital learning in general when face-to-face education is not possible. The article “Learning remotely when schools close: How well are students and schools prepared? Insights from PISA” of the Policy Responses to Coronavirus generally stated that “…the COVID-19 crisis struck at a point when most of the education systems covered by the OECD’s 2018 round of the Programme for International Student Assessment [14] (pp. 16–17), were not ready for the world of digital learning opportunities.” This has particularly been the case for Czech teachers, partly due to the low usage of ICT tools in classes prior to the COVID-19 situation [14] (p. 13). Dhawan stipulates that the level of preparedness is key to successful adaptation to a large-scale event with severe implications [15] (pp. 18–19). The low level of technology usage amongst Czech teachers is in contrast to the relatively widespread penetration of technology and internet access among households and students [15] (pp. 18–19). The findings of early studies of the Czech educational environment found significant, unparalleled involvement of parents in online learning (and education in general) and an imbalance between the flood of assigned tasks for students and feedback from the teachers lagging behind [16], the latter being confirmed by the Czech School Inspectorate (CSI) [17]. In the lower secondary schools, teaching methods consisted of predominantly frontal, passive or largely unsupported assignments [18]. Early attempts at summarizing the experience of COVID-19 caused a shift from survival mode to a more professional approach of coping with an array of online learning technology tools and features that include a cautious mix of synchronous and asynchronous technologies, employing teamwork etc. [19]. Other studies have stressed the changes in demand for assessment and the corresponding need to adjust the assessment to these new learning requirements [20]. The Czech school inspectorate concluded that for the Czech primary and secondary schools, the implementation of cognitive activation strategies using online tools remains the key pedagogical and organizational challenge [21] (p. 46). The field of tools for inquiry-based learning in history education had not been unpopulated before HistoryLab had entered it. The programmes of the Stanford History Education group have the longest track record dating back to the early 1990s [22]. The closest of the programs is Beyond the Bubble [23], a series of History Assessment of Thinking activities. Though the teaching materials are online, no interactive solution is provided. Despite the fact that Beyond the Bubble does not provide a platform for online learning, it is included in a comparison in the HistoryLab’s Concept section of this article due to its strong theoretical and methodological background [24]. The main freely available interactive online tools are Historiana (
3. HistoryLab Concept
The HistoryLab project aims to support hybrid and/or distance learning of modern history for secondary schools with a set of interactive multimedia-based activities which encourage active learning. Activities consist of posing inquiry questions and a series of tasks that require direct engagement with primary sources in steps that allow for scaffolded analysis and interpretation. The scaffolded learning experience is ensured by applying the key principle of constructivist pedagogy- the zone of proximal development [25]. This principle underlies the pedagogical, as well as the design and technological solutions to ensure each step is attainable by the majority of students. Skills of historical literacy are also fostered through encouraging discussion of the problem at hand.
The target audience of the Learning Management System (LMS) is secondary education history teachers. The teachers can organize their HistoryLab-powered courses in LMS—create classes, add students to classes, enrol in activities from a growing catalogue and assign them to their students. The students can work on their assignments individually or in a group. There is also a third assignment mode, called a class assignment, where the teacher demonstrates the process of working with HistoryLab on a selected activity. The LMS also provides a feedback mechanism for the teachers to grade the completed assignments. Communication between both parties is achieved using email.
Students open an activity in a web browser and, following the instructions, they complete a task encouraging objective discussion of a particular topic from modern history, searching for more information and sharing their ideas or opinions with their teacher or classmates.
Activities can be done individually by students as homework or during a distance learning session. They can also be completed using student teamwork or in cooperation with a teacher during a collective discussion about the problem.
An example of such an activity is given in Figure 1. The activity is called “Why were they displaced”.
In this activity, an inquiry question is used as the name of the activity. This question leads the student’s focus, while this principle of naming the activity after the inquiry question applies to the majority of activities. The educational aim is set in the recommended procedure (e.g., [26]) for particular activity. This procedure is available in a supporting document for the teacher to effectively guide the students through the activity and beyond using a standardized structure. The projected time to complete this eight-step activity is 40 min, making it one of the longest in the catalogue. The activity includes tasks involving two historical photographs, two snippets of historical documents and a modern short text written for the activity by a historian. The set of tools to analyse and interpret these pieces of evidence consists of a magnifying glass for inspecting the two photographs, Scalable Vector Graphics (SVG) marks and text to analytically engage with the photographs, SVG text balloons, a highlighting text editor to mark key passages of the text and a text field to answer the final summarizing questions. The complexity, projected duration, suitable methods of work, suitability of tools used as well as the efficacy of the tools themselves have all been aspects tested during the proof-of-concept runs.
The educational aim is set as the following: “In working with the photographs, students engage with the perspectives of period actors. With the help of sources, they understand the different political contexts of the displacements in 1938 and 1945.”. The phrasing of the educational aim generally uses non-academic language, since one of the findings of early testing led to simplifying the language and abandoning sophisticated pedagogical phrasing since it did not resonate well with teachers.
Currently, there are 80 activities available publicly in the system, covering various topics from across the world, including European and Czech modern history from the 19th century to recent events. Additionally, there is a set of 11 activities covering selected topics of older, pre-19th century history. The last testing phase (winter 2020/2021) consisted of a further 28 activities that were made public in March 2021. The activities use at least one, but often several of these types of media: photograph, newspaper reprints, maps, audio files, videos, mass media online articles, snippets of scholarly texts, postal stamps, caricatures, local chronicles, book covers, official and personal documents.
Aligning the available activities with a particular curriculum can easily be done with a special catalogue of the activities, allowing the user to filter them by their metadata and compose a suitable sequence of the activities, supporting a particular plan of lessons. An example of this catalogue is given in Figure 2.
The catalogue is based on a system of controlled vocabulary (indexing language) that includes a number of mostly closed-list values for variables to concisely describe each activity. These variables include projected duration, the level of complexity or the number of tools used as shown in the example above. Moreover, activities can be ordered according to first-order historical concepts (such as war), second-order didactical ones (e.g., continuity and change) or general national curriculum topics, which are included so that the teacher is able to navigate to the most suitable activity based on their teaching plans and preferences. Having departed from the Big Six Concepts of Historical Thinking [27], an adapted version of 4 + 1 big concepts [28] were the result of iterative development. Finally, both the literature [29] and the results of the pilot run [30] point towards a gap between classroom practice and the second-order concepts of historical thinking. That is why an alternative set of historical reasoning subskills more comprehensible to teachers, recently called for by Drie and Boxtel [31] (p. 167), has been developed specifically within Historylab. Thus, an original list of subskills appears at the forefront of the catalogue and includes: formulating and checking a hypothesis, uncovering sources’ hidden meaning, looking for key details, guessing the meaning, comparing sources, forming (creating), putting together substantiated answers, discussing and looking back at what we have learned.
The catalogue itself was created by an iterative development. In its final iteration, we created so-called distributed methodology [28] that can be summed up as replacing a single, detached document on the methodology of the app with a complex net of low-threshold ways of communicating the methodology (catalogue, recommended procedure, tools within activities and others).
Activities and LMS are localized in Czech and English languages, with an option to add more languages when needed.
An important part of the development of the HistoryLab webapp is a systematic design approach closely integrated with didactic principles. Early integration of designing the user interface (UI) and user experience (UX) into the development process was fundamental for the resulting shape. The didactic principles (such as inquiry-based learning, focus on the work with primary sources, historical and media literacy) work as a metric system for proposed design solutions. Such a metric system helps to make better design decisions which lead to better UX, an easy-to-use application and transformative learning environment: two goals which are often in contradiction. The systematic approach to design is reflected in design principles [32,33] which empower creators with a decision framework beyond the sole didactic principles. It also guides the team in creating new features and activities: how to design them and how to evaluate the proposed designs. The principles are invisible design, simplicity but not simplification, educational potential, one slide and one operation and habit forming experience [34]. The invisible design aims to reduce unnecessary visual and interactive clutter which could distract students from accomplishing a given operation. It emphasizes content over the user interface. This principle is guided by “attention politics”. Simplicity not banality communicates a goal of not making the application too “digital”, too smart, and eventually too banal, which would work against the cognitive activation potential. On the contrary, it suggests simple solutions which leave students enough space and time to immerse themselves into the topic, and at the same time create clear limits which bring creativity to the process. This principle also helps to bring the complexity of the world to the front, not hiding it behind. It asks for transparency. Educational potential asks creators to design any new feature in a way that enriches the learning situation, not the other way around. Despite the temptation to make very easy-to-use applications, it is not worth the price of losing the educational potential. Striving to maximize the educational potential leads to the introduction of friction, which is understood as a feature, not an obstacle, of the app. With friction at hand, designers keep the workflow in line with the inquiry-based learning pedagogy. One slide and one operation is closely related to the scaffolding principle. It enables creators to work with dramaturgy within activities. Its main goal is to prevent cognitive overload of the users. Emphasis on habit forming experience aims for long-term use of the application and on long-term results. The application should empower teachers in such ways that lead to regular and continuous use of the application in their teaching. More importantly, the design of the application should support students in using acquired skills and competencies in the repeated use of the application as well as outside of it.
To summarize, the HistoryLab project contains several innovative approaches to teaching modern history online compared to the solutions that are available in the field of history education worldwide. These innovative features can be inspiring for educational community members when designing e-learning systems or distance learning methods in the modern history field. Though these are described primarily in comparison to the other online educational tools due to the number of analogies, we have included the aforementioned History Assessment of Thinking set of non-digital activities called Beyond the Bubble developed by the Stanford History Education Group into the comparative sample as well.
-
The dense and thorough series of controlled pilot runs in the field ensures evidence-based tools and all their components (activities and their components: individual sources, instructions to work with them, set digital analytical and interpretive tools, recommended procedures, annotations, the LMS and the catalogue). Neither DocsTeach’s nor Historiana’s system and content are based on such thorough proofs of concepts, while HAT has carried out robust pilots with a sample of its content [9].
-
A specific, tailor-made theory underlying both the method employed in HistoryLab and its technological solution has been developed and adjusted iteratively, hand-in-hand with the UX. The result is not only an adapted version of the Big Six historical concepts [27], but also an interconnected set of skills, operations and associated tools defined within the project. This “distributed” theory reflected in each component of the system as well as the app as a whole is communicated primarily through the app itself, rather than through associated methodical texts. That is in contrast with both Historiana and HAT. Historiana singles out multiperspectivity as the guiding principle and a broad historical thinking discourse as its theoretical basis [35]. HAT’s theoretical background is firm, building on a consistent line of Reading like a Historian programme from early works by Samuel Wineburg [22] to its current, adapted version [9].
-
A pedagogically safe environment for students to learn, formed by (1) instructions, activities and questions that presuppose personal involvement of students and expose the interpretative nature of historical inquiry, (2) immersion into the activity made smoother thanks to the “one slide and one operation” policy, which also prevents cognitive overload of the user, (3) the non-restrictive nature of the application which allows users to go back to change their previous answers and (4) the tight scaffolding allowed by the interconnectedness of a step at hand with the analytical work done in previous steps of an activity thanks to the through export across slides, or as a split-screen for further questions and tasks. Even though the e-activities Historiana and DocsTeach mostly follow these principles as well, the tight scaffolding is missing.
-
A unique consortium that includes accomplished historians, didactic researchers, memory studies experts, sociologists, UX design experts and software developers involved in the development ensure evidence-based curation on the level of sources as well as activities. Where HistoryLab offers a closed system of dense curation, Historiana is based on a collaborative, semi-open project developed and curated by a large consortium of history educators. Historiana’s approach offers less control, but more options for experienced users, whereas HistoryLab guarantees assured quality at the expense of the options of the user to adapt.
-
Suitability for K-12 education, pre-service teacher education and teacher training in Czechia is assured by the compatibility with the Czech general curricula and dense piloting of all these forms with the respective population since 2016. Since Historiana is a European project, it naturally could not become compatible with a single general curricula jurisdiction with a single European country or other entity. This applies similarly to DocsTeach and HAT due to their primary appeal to the history educators in the US, which has a decentralized curricular system.
4. Technical Aspects
At the beginning of the HistoryLab project, an intensive discussion was dedicated to whether to implement it as a standalone proprietary application or if to use an existing framework or tool that allows creating multimedia applications to implement the students’ activities. As an example of such tools, H5P (
Complete flexibility of technical options to implement the activities. A group of educational specialists designs the activities in the project team, and their details are thoroughly considered to support the educational process in the best way. Details might play a role in this point, and our intention was not to limit the design of activities by certain technical constraints that might be imposed by using a third-party framework or tool.
Potential specific requirements of the Czech Ministry of Education, which is now using the HistoryLab platform as one of its official e-learning tools. These requirements relate to the accessibility of the activities to the target group as well as specific didactic requirements. High flexibility in the implementation of activities is necessary to satisfy these requirements.
Necessity to report progress status and details from the activity by a specific protocol. Various details of the activity have to be stored in the backed part of the system to be further analysed to improve the activities. Existing solutions might bring certain limits to this flexibility.
The LMS part of the system is implemented as a standard client–server application, using HTML, CSS, JavaScript library React and NodeJS. At the start of the project, we carried out an extensive survey of available open-source LMS in order to find a suitable one to reuse; however, during the development, it turned out that designing and implementing our own LMS specially tailored for our purpose was the most viable option from a technical and economic viewpoint. The LMS server stores the data in the MySQL database.
The activities, which make up the client-side part of the system, are implemented as single-page applications developed in HTML, CSS and JavaScript. It communicates with the LMS server via Representational State Transfer (REST) API with messages in JavaScript Object Notation (JSON) format. To increase communication security, HTTPS protocol is supported for exchange between activities and LMS.
The applications run on the server in individual Docker (
Email notifications are sent by HistoryLab in various instances, for example when a teacher sets an assignment for students, when a student or students finish an assignment, when a teacher reviews and evaluates the assignment or when a teacher forgets the password to log into LMS and asks for it to be reset.
The overall system architecture allows for scaling of the system to high numbers of concurrent users. Other architectural features of the system also support this flexibility—NodeJS technology and the architecture of the LMS part of the system, as well as the possibility to distribute the data storage or readiness of the whole system to be deployed in a cloud infrastructure.
To minimize the possible overhead caused by account management and distribution of access rights in the class or within the system, a lightweight solution was selected which proved to be the ideal option during the pilot run. No explicit registration is needed from the students.
In this process, the teacher will choose an activity from the catalogue and a class from their list of classes. Currently, 59 activities are publicly available in the catalogue and in addition to this set, 28 more activities are currently being tested in a pilot run. The class contains a list of students with their names and email addresses. Classes can be created in advance, or the teacher can collect the email addresses of the students in their classes and add them on the fly. An assignment email is sent to every student with an URL directing them to the selected HistoryLab activity page. The link contains additional information such as the enrolment code and email address of the student. These data bind the assigned activity, the student and his/her enrolment in this activity.
The communication protocol used between the activity and LMS is open and based on JSON format, which allows for easy integration of created activities with other alternative LMS.
5. System Testing and Verification
In this section of the study, we report on the data and experience of public usage of HistoryLab during the COVID-19 lockdown. However, before HistoryLab was employed as a public tool, a series of tests and a pilot run was conducted. To provide a broader context, we briefly introduce these efforts in this section and explain the user experience theory behind the app.
Starting from the system development process, developer tests, as well as independent manual technical tests, were conducted after each increment or release of the system, to prevent functional defects or major usability suboptimalities in the system. For this regression testing, selected parts of these tests were automated.
The system was tested using end-to-end (E2E) tests that simulate the behaviour of real users. For this purpose, we have used Cypress.io (
After these technical tests, several rounds of pilot runs with selected secondary schools in Czechia were conducted to assess the viability of the concept and collect feedback for improvements of the system.
The pilot runs were designed in a proof-of-concept style to iteratively test various aspects of the system including its UX and pedagogical theory and the pedagogical content of the activities. The system aspects included the design of activities, analytical tools used in the activities, the LMS etc. A total of 7 runs of the testing have been conducted since 2016, with 68 unique teachers and 108 “teacher-tests” which correspond to approximately 67,000 “student-tests”. Altogether, the test resulted in approximately 8936 completed activities and 89 different activities tested. The design of each test included structured field reports (protocols) by teachers covering most classes (n = 960), final profiling questionnaires (n = 116), 10 focus groups, helpdesk communications and 25 structured protocols of observation in the classroom. The findings thereof were used to improve the iterative development, while key early findings were presented in a chapter [30] of a summary report [38].
Feedback from several rounds of pilot runs as described above was being continuously evaluated and reflected in the improvements and updates of the system, in order to tailor its functionality and usability to fit the needs of the educational community in the best way.
6. Experience Data and Feedback
In this section, we discuss the specifics of lockdown and its impact on the educational system, followed by a discussion of the feedback from the HistoryLab system run and the lessons learned.
6.1. COVID-19 Lockdown Specifics
In the periods of February to June 2020 and October 2020 to February 2021 the COVID-19 pandemic heavily affected life in Czechia, and its educational system was no exception. Approximately half a year after the first weeks of lockdown, a time when society was unprepared for the necessity of distance teaching, teachers and students were getting used to this new format of education, and the effectiveness of the educational process was improving. By this point, we have gained valuable experience of the practicalities of distance learning and can share the lessons learned so far to help improve the educational process during this challenging situation.
The challenges to address can be categorized as two principal areas which we denote as (1) social aspects and (2) technical aspects, which may mutually interact and overlap.
In this section, we describe these individual challenges, then we discuss how these challenges can be addressed and mitigated by a general e-learning system, and how they are addressed in the HistoryLab project.
6.1.1. Social Aspects
The COVID-19 pandemic engendered a novel set of challenges that have not been experienced in the Western world in recent decades, generating uncertainty and a certain level of chaos that caused stress to teachers and students. Compared to certain implicit, organized, positive stress, which is a natural part of the educational system and leads to better performance of students and generally improved effectiveness of the educational process, stress caused by lockdown had a rather negative impact on the effectiveness of teaching and learning. In this aspect, the stress caused by uncertainty was exacerbated by the fact that a majority of students and teachers had not previously encountered distance learning. In a relatively very short time, practically everybody in the educational community, often with minimal experience with effective methods and best practices of distance learning, had to abandon the face-to-face classroom and teach online. This naturally caused several suboptimalities and sources of ineffectiveness in the educational process, especially at the beginning of the lockdown situation.
Another problem adding to these difficulties was the economic situation of certain low-income families, resulting in a lack of equipment needed for e-learning, including obsolete or absent hardware or an absence of good quality network connection.
Students’ reduced focus and the various distractions in their home environment during lockdown is another issue relatively difficult to handle. During distance learning in the form of homework or organized video calls, more liberty is left to students to spend the learning time concurrently using various forms of digital entertainment, like watching films or using social media. Due to the nature of distance learning, it is difficult for the teacher to effectively prevent this from happening, or even to recognize its occurrence. Low bandwidth networks combined with network overload might hinder the practical possibility of students streaming their cameras during a call. Even with the students sharing their videos, it is often difficult to discern if the student is really focusing on the lesson, or on another application opened in parallel on their computer or device.
It is possible that some supervision to mitigate this could be provided by parents, however, parents’ capacity to supervise might be limited due to their work duties overlapping with teaching times.
Additionally, class management in distance teaching might be challenging for teachers inexperienced in this type of education; possible technical difficulties amplify the scale of the problem.
The list presented might not be exhaustive and other issues can be identified; in this overview, we attempt to cover the most significant ones experienced by most teachers. With regards to the social aspects, some technical aspects might add to these problems, as suggested in the analysis above.
6.1.2. Technical Aspects
The lockdown situation significantly increased demands on network connectivity and bandwidth, which complicated the situation for participants of the educational process who have access only to low-quality data networks. This problem is composed of two possible factors. Firstly, a high-quality network connection might not be available in rural areas and certain urban areas with lower citizen density. Moreover, only lower bandwidth connections are available to certain low-income segments of the population. Secondly, the necessity to work from home during the lockdown and to conduct many more everyday operations online naturally lead to network overload, adding to the previously mentioned issues. Unfortunately, a naive approach to the construction of web and communication applications in the last two decades led to a significant increase in demands of these applications on network bandwidth. These unnecessary requirements cause possible bottlenecks in situations when these systems are being used intensively by an unexpected number of users concurrently.
Additionally, the broader community is using various hardware, including obsolete devices, which increases demand on the interoperability and compatibility of software applications used in the educational process. Combined with the possibility that the participant is in a lower-income segment, the default expectation for everyone to have the latest devices is neither realistic nor reasonable.
6.1.3. Possible Mitigation of Discussed Problems
The aforementioned problems and challenges can be mitigated by the design of a suitable e-learning system, accompanying methodological support as well as other supportive actions such as short video tutorials or ongoing teacher training offers. To give the research community more detailed insight, we discuss this situation in Table 1. Each of the difficulties outlined in Section 6.1.1 and Section 6.1.2 are accompanied by general suggestions for mitigating the problems and a description of how the HistoryLab project supports these solutions.
6.2. Data from System Run
In this section, we report on the public release of the HistoryLab system during the COVID-19 lockdown period in Czechia. The data collected from the system run are recorded separately from the previous data collected during the system testing and previously mentioned rounds of pilot runs.
The number of new user registrations in the period March 2020 to December 2020 is presented in Figure 4. The total number of new registrations completed during that period was 326. In the graph, two main waves are visible: from mid-March until the end of May and from mid-September to December. These waves of increased traffic correspond with the schedule of the school year, which is unified in Czechia, as well as with two major COVID-19 outbreak waves, and thus school closures happening in these periods. Clearly, both of these factors contributed to the trend of increased registrations.
The second trend depicted in Figure 5 displays the number of completed activities during this period. In total, 19085 activities were completed. Regarding the distribution of the data over time, the trend correlates with/runs parallel to the number of new registrations on the system.
Of the completed activities, 10,028 were started directly via public links from the catalogue of activities (
6.3. User Feedback
There are numerous channels from which one can analyse the feedback of users:
Large data sets from the system run including saved and completed activities and data from LMS. The activities include a final open text field for messages from students.
Communication of teachers with the helpdesk.
Field notes from the lecturers and participants of the “Introduction to Working with HistoryLab” online training series.
Data gathered specifically for COVID-19 through a pilot run.
Firstly, the nature of each set of these data sets is described. Secondly, the analysis of these data sets is shortly outlined and the results are presented. The interpretation and application of these results are included in the following section on the lessons learned.
The field notes from the lecturers and participants of the “Introduction to Working with HistoryLab” consist of written communication between the lecturers and course participants and individual notes of the lecturers. In the six courses organized between April 2020 and May 2021, 10 groups of teachers were formed. Each individual course consisted of a synchronous introductory online seminar, at least two rounds piloting HistoryLab in the classroom, and a final, synchronous online seminar. The focus on the quality of feedback given to the students was supported by further interim consultations with HistoryLab’s didactic staff to offer one-to-one “meta-feedback” on the style and principles of giving students feedback and also assessing their creative work in the app. The dropout rate oscillated between one quarter and one half.
The helpdesk operation was streamlined using an online form for the specification of requests and comments from March 2020. Moreover, questions and requests were posed by phone and contact email. The resulting written records of the helpdesk amount to 150 entries with 82 ones initiated via the online form.
The pilot run from November 2020 to January 2021 included 28 HistoryLab activities and was carried out with 17 lower and upper secondary school history teachers that made a total of 138 pilot lessons with their students. As the pilot ran mostly during the lockdown period, both the final questionnaire and two focus groups aimed at mapping the experience with the way HistoryLab performed in distance learning.
An explorative probe into the system run data reveals that the general boom in the use of HistoryLab documented above by the system run data on the number of registered users and filled-in activities has been followed by a relative decrease in the rate of use of LMS. When compared with the pre-COVID-19 period, the rate of the use of LMS among all users decreased from over 60% of completed activities done through the LMS to little less than 50%. However, this relative decline has been followed by a notable increase in the use of feedback options within HistoryLab’s LMS. While in the pre-COVID-19 period, the share of the total number of completed activities that were given feedback “directly” through the tailored option within the LMS amounted only to 17%, the COVID-19 period share accounted for 25%. The change indicates a shift from the use of HistoryLab as a fringe or supplementary activity to a more integral part of learning activities.
A qualitative analysis of the helpdesk communication reveals three types of largely technical issues reported by the users of HistoryLab. The first group consists of reports on general technical issues with the app, such as system downs, bugs and typos in recommended procedures. These reports have been particularly useful in fine-tuning the app. The second group consists of problems regarding user credentials and other issues regarding log-ins. Finally, the third group of queries were addressing problems with the mailing system and generally, the delivery completed, i.e., filled-in by students. Mostly, these problems stem from a situation when activities were assigned to students through LMS but were received from the non-LMS system due to students not correctly following instructions. Apart from these technical problems, helpdesk has been approached relatively frequently.
The pilot run’s final questionnaire generally revealed a high level of satisfaction of teachers with the way the app performs in distance learning situations during lockdowns. This is visible from the fact that the user satisfaction measured by the question “Would you recommend the app to your colleagues?” fared more conclusively compared to the previous periods before lockdown with 95% answering “certainly yes” and 5% “probably yes”.
The teachers of the pilot run predominantly expressed the lack of social control while working with HistoryLab distantly compared to face-to-face teaching. However, as the focus groups revealed, this was rather a general sentiment about the lockdown situation. This sentiment regarding the limited attention span and opportunities of synchronous learning becoming a scarce resource, that was already mentioned in Seciton Section 6.1.1, was not associated with HistoryLab in particular, quite the opposite. As one of the participants of the focus group expressed for the rest: “The online teaching has some benefits. While working individually, it is [much easier with HistoryLab] to see whether I have assigned the activity in a proper way and whether the activity is suitable. The students are welcomed to work at their own pace.” This has been overwhelmingly supported by references to HistoryLab as being successful to bridge the synchronous and asynchronous elements of distance learning.
Some of the basic principles of the HistoryLab concept as described above were underlined and commended vis-a-vis the distance learning situation. Keeping the “zone of proximal development” of the pupils mentioned in Section 3 in employing the activities and selecting them properly for the class in the first place resurfaced as fundamental. Ensuring the feeling of success due to attainability under conditions less prone to constant formative feedback seemed to be of increased importance. “The students have marked these activities as simple. They felt they are successful”, as one of the participants of the focus group summed that up.
The most important results of the qualitative analysis of field notes of “Introduction to Working with HistoryLab” intense online training sessions are threefold. They are associated with feedback given to students and the choice of appropriate activities, which points to the transformative nature of HistoryLab in the Czech history education sector.
There is a predominant interest in the methodical approach towards feedback and assessment of HistoryLab activities, that by their nature include space for students to express creativity. It was overwhelmingly appreciated that the course included experiential learning elements. This was accomplished by putting the teachers into the shoes of students in the first phase, during the first online meeting. As the teachers experienced the creative work in the role of end-users, they were much better able to both offer support and provide formative assessment. Another successful principle.
Another experiential learning principle highly appreciated by the participants was the task to choose two HistoryLab activities from a limited set of the ones determined as suitable for the course. According to most participants, the demand to substantiate this selection followed by dense feedback from the lecturer and natural feedback from its application in the class became a good starting point for thinking about the successful integration of HistoryLab into the curriculum. Finally, the turn from summative assessment of rather passive learning to the formative assessment of active learning revealed the fact that a more fine-grained approach to the performance of students is needed. Most of the practical questions regarding the feedback was dealing with “fragmentary”, “shallow” “surface-scratching only” or ”too brief”. More subtle, refined regard to the indices on the thought process of the students was best identified in a group-work and was called an eye-opener by many participants.
6.4. Lessons Learned
We gained several insights into the practical workings of the HistoryLab project during the project pilot runs and in the recent lockdown period that might be useful for the educational community implementing a similar project. Hence, in this section, we present selected highlights of lessons learned collected during the project.
The controlled pilot run differed from the live application of the system during the lockdown. Several factors played a part in this. The primary issue is absence of leverage allowed by the knowledge of profile users. In the lockdown system operation, in contrast to the pilot project run, we did not gather detailed information about the types of users, meaning that we did not know if any of the users were actually teachers. To mitigate such a situation in the future, it will be necessary to contact some of the users with a brief questionnaire to gather such information. Still, the detailed knowledge of the profile of users allowed by the testing phase is no longer attainable.
Practically every user in the pilot run and almost every user in the public project run before the COVID-19 lockdown received some initial training and methodological information about the system. However, this was not the case for the public run of the system in the lockdown period. Urgent priorities related to the lockdown rendered this situation impractical. The potential result was twofold.
Firstly, some users might not have used the system in an optimal way and did not fully utilize its potential. This was well-illustrated by the decrease of the ratio/number of users utilizing the available LMS by 10%.
Secondly, data acquired about the system run were less accurate. To give an example, the options offered for methods of work—a closed list that consists of individual, group or in class—could not be used as intended due to a general lack of understanding of these options. This resulted in less valid results of meta-data from the LMS for the filled-in activities. To mitigate this effect, the system shall be more intensively supported by an online methodological support system.
Various options can be employed to address this matter. In the project, we conducted four online webinars to support the users of the system. From March 2020, we answered calls for a more intense form of online teacher training and we have developed the aforementioned an ‘Introduction to Working with HistoryLab’ seminar.
High demand for these online seminars led to 6 runs with groups of 15 teachers (the optimal size of the group with 2 lecturers) while the data on the use of feedback and assessment features in LMS already suggest that over half of the results from alumni of these seminars use the “teacher note” (direct feedback through the HistoryLab LMS) sent to the student. This contrasts with only 20% use in the group without training.
The next option is to organize a more comprehensive helpdesk system to support particular users’ problems and provide guidance for users who are less experienced with distance learning and the usage of web applications. The operation of the helpdesk has once again confirmed the uneven level of pedagogical understanding of the active learning methods and inquiry-based learning. Though the demand for continuous pedagogical support through helpdesk is increasing, it is unfeasible to provide fully personalized advice in such a general pedagogical issue.
Still, these interventions do not seem to be sufficient, as the capacity of the online teacher training seminars is limited. We have launched a structured set of video manuals that include a four-part basic instructional clips [39].
Several lessons have also been learned from a technical viewpoint. First, the compatibility of exercises, catalogue and LMS with a wide spectrum of browsers has to be maintained. Considering the wide usage of the system and different income classes of secondary school students, the usage of the latest devices updated to the most recent versions of the browsers and operating systems cannot be assumed.
This question arises when considering compatibility of the exercises with touchscreens, and here two problems impact the overall usability: (1) the difference between touch screen and classical computer devices and (2) display size. The takeaway from this project is that the adaptation of the exercises for touchscreen tablets is easier because the only aspect that must be changed is interface controls. In contrast, making the exercises compatible with mobile phones, due to the size of the display and its orientation, often involves the creation of a separate version of the exercise/application. Due to the typical usage of the system, this variant was not explicitly supported in the project.
Another technical takeaway is that when the user traffic in the system rises, a special system analytical tool is needed to evaluate the usage of the system. We implemented this support as a standalone application accessing the LMS database and visualizing the data from the system run. For instance, the graphs presented in Figure 4 and Figure 5 were created via this analytical system.
Another technical takeaway related to a sudden increase in traffic is the email service. The application uses email as the main channel of communication between the system and its users: teachers and students. Emails are not used just for ordinary notifications (e.g., forgotten passwords) but also for critical notifications, e.g., a new assigned activity (which triggers an email sent to the student) or a new completed activity (which triggers emails sent to both the student who received the assignment and the teacher who assigned it). It might be complicated for students to see their completed activities without those emails as they have no user account in the application. We have experienced hitting email service limits during the peaks in spring as well as in autumn of 2020. On top of that, there is another unexpected issue related to user experience. Students do not like and often do not even know how to use emails. There are also many typos made when filling in email addresses, so many emails simply cannot be delivered due to non-existent addresses. School email domains also have quite strict anti-spam policies. The former problem of reaching email service limits, purely a technical issue, could be solved through better optimization of our email handling, which has already been partially done, however, such optimizations have their limits and a new external API-based email service should be considered. The latter, a more design-related issue, may be solved by providing better support for teachers, providing guides and recommendations of how to use the application alongside an improved user interface and form validation. Even though the absence of students’ accounts is considered as an intentional feature of the application (no threshold related to the registration process for students), it should be reconsidered in light of the issues mentioned above.
Large user traffic also requires a proper server data backup procedure and continuous server monitoring; however, this is de facto standard for larger web applications and our system was not an exception in this point.
7. Threats to Validity
As for the user data from both the LMS and the completed activities, there are three key limitations that should be taken into account. The first two are associated with the current functionality of HistoryLab, while the third rests upon strategic decisions on the possibilities of different future uses of HistoryLab beyond formal education.
Some of the trained users have used the testing server outside of the testing period for their teaching purposes, which decreases the reliability of the information on the number of regular users and completed activities on the public server. The figure does not capture all of the regular users and uses. This skew is relevant for inferences about the behaviour of testers as trained regular users, but as the increase of traffic in the COVID-affected period is in two orders, this does not affect the general findings.
The ongoing development of the project also had an impact on the data on the choice of activities in two ways: (1) due to the design of the catalogue, the default order of activities may have influenced user choices more than other factors such as educational aims or media used, and (2) the range of offered activities has been expanded during the testing period in the COVID crisis.
This could affect choices in two ways: by promotion of new activities through webinars and thanks to the “new release” label in the catalogue, more attention could be directed towards these. On the other hand, the shorter availability of these activities in the offer could have reduced their potential use.
Finally, as the system is open, the use of LMS, the completion of a single activity and registration in LMS are not contingent upon any thorough identity check of the user beyond their email. This means that HistoryLab might not be used only by teachers and students of secondary schools, and the extent to which we can presume that our data is derived from school users is limited. The open character of HistoryLab is desirable since the fostering of historical literacy should not be limited to formal education. Initial probes into user data indicate that the majority of activities are completed by users in formal education as a teacher email is included, and the number of other users is negligible. However, if the uses of Historylab extend beyond formal education systems in the future, options for discriminating the data based on types of users should be explored.
8. Conclusions
In this study, we have introduced and presented the HistoryLab project, which aims to interactively support modern history education and encourage an engagement in historical thinking amongst students, employing various types of multimedia and communication technology. During the recent and ongoing COVID-19 lockdown in Czechia, this platform served as a complementary tool for secondary school teachers of modern history. The feedback from this run showed excellent viability of the concept, hence we have described the HistoryLab concept and reported on this experience in order to share the lessons learned during this period with the wider educational community.
The lockdown situation brings specific challenges for distance learning. The most significant of these challenges is the necessity to conduct this type of education on a massive scale and the technological obstacles arising from this situation, such as limited network bandwidth as well as the variety of devices used by the students, some of which are obsolete. Though these constraints may affect the quality of delivery of the service, there are some households with no access to an online connection at all. Though the digital divide is growing in significance, especially in low and middle-level income countries, sometimes to the extent that effective distance learning is disabled [40], the growing digital gap in education is a key challenge accentuated by COVID-19 in high-income countries as well [41]. This problem can be minimized by the extension of the current system to support an offline content mode and by offering wider support for various devices. However, considering the high effort needed to maintain system interoperability within a wide spectrum of various and obsolete devices, this is one of the most significant challenges in system production.
Additionally, class management in mixed mode, when some of the students are connected remotely whilst some are present in a classroom, poses a further challenge in this sense. According to reports from educational practitioners in the recent period, this configuration is even more challenging than pure distance learning class management.
We hope that the presentation of this innovative concept together with proof of its viability will be beneficial for other researchers and practitioners in the field of history education. Lessons learned from the project shared in this paper shall add to this contribution as well.
Conceptualization, M.B.; Investigation, J.M.; Methodology, V.R. and K.B.; Project administration, K.F.; Software, M.B., K.F. and J.M.; Supervision, K.C.; Validation, K.B., K.F., J.M. and K.C.; Writing—original draft, M.B., V.R., K.B. and K.F.; Writing—review and editing M.B., V.R., K.F. and K.B.; Funding acquisition, V.R. and K.C. All authors have read and agreed to the published version of the manuscript.
This research was funded by the Technology Agency of the Czech Republic, grant number TL01000046 Historylab: using technology to foster historical literacy.
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Enlarge this image.Figure 1. An example of HistoryLab interactive activity filled in by students (original Czech text filled in by students using a text tool has been translated into English).
Enlarge this image.Figure 4. Number of new registrations per day to HistoryLab public portal from 25 March to 31 December 2020 (data between 11 and 25 March 2020 are invalid).
Enlarge this image.Figure 5. Number of completed activities per day submitted in the HistoryLab public portal from 11 March to 31 December 2020.
The problems discussed, their possible solutions and support for these solutions in HistoryLab.
| Discussed Problem | Possible Mitigation | Mitigation Support in HistoryLab |
|---|---|---|
| Social aspects | ||
| The stress caused by lockdown for teachers and students | (1) Design of the e-learning system to be straightforward and intuitive, (2) Attractiveness of the e-learning content, (3) Methodological support for new participants in distance learning | (1) Several rounds of usability testing including internal team and pilot users, an independent usability audit by an external agency, (2) Activities designed by a dedicated methodological and pedagogical team to be attractive and appealing for students, (3) Part of the project team dedicated to providing methodological support for the users, helpdesk available for the users |
| Students and teachers with little to no previous experience of distance learning | (1) High usability and intuitive user interface of the e-learning system, (2) Methodological support from the distance learning team to the users | (1) Aforementioned rounds of usability testing and independent usability audit, (2) Methodological helpdesk available for the users, methodology available also in document form |
| Possible limited equipment needed for e-learning | (1) Compatibility of the e-learning system with a variety of devices, including obsolete devices and older versions of operation systems, (2) Prevention of excessive requirements of the e-learning system on hardware performance and network bandwidth | (1) Compatibility testing of activities and LMS on various devices and continuous improvements of the system to improve this compatibility, several rounds of pilot runs of the project, with participants using a variety of hardware, (2) Continuous optimization of the system and design to be economical with the hardware requirements and bandwidth limitations, lightweight communication protocol between activities and LMS |
| Reduced focus of students and various distractions in the home environment | (1) Interactivity of the e-learning system and the attractiveness of distance learning content, (2) Methodological support for how to conduct distance classes | (1) Design of the activities by a dedicated methodological and pedagogical team to ensure that they are attractive and appealing for students, (2) Methodological support available for users and flexibility in how HistoryLab can be used: direct support for distance learning conducted via teleconference, homework, or student collaboration in distributed teams, communicating via teleconference or social networks, for instance |
| Parents’ limited capacity to supervise | Measures given in the previous line apply here as well | Measures given in the previous line apply here as well |
| Class management in distance teaching might be challenging | (1) Providing teachers with more experience of distance learning, (2) Possibility to react to various unexpected situations in the class and possibly of adapting the plan of the lesson or teaching style if needed | (1) Methodological support provided by the project helpdesk, lessons learned from the pilot runs of the project being available to the community, (2) The aforementioned flexibility of HistoryLab to be used in various styles of distance teaching |
| Technical aspects | ||
| Quality network connection might not be available | (1) The e-learning system designed to have economical demands on bandwidth if possible, (2) Necessity of special tests of system functionality during limited network connectivity | (1) HistoryLab design minimizing bandwidth needs, including lightweight JSON-based communication protocol between activities and LMS, optimization of the size of multimedia used in the activities, (2) Special technical tests exercising the system under limited network connection, verifying form transactionality of data transfer and user experience |
| Data network overload | Measures given in the previous line apply here as well | Measures given in the previous line apply here as well |
| Variety in hardware, including obsolete devices | (1) Design of the e-learning system to not require specific elements available only on specific hardware or platforms, (2) compatibility tests of the system | (1) Architecture and technology used in HistoryLab does not include specific elements that are restricted to particular platforms; classical standard HTML, CSS and JavaScript is used in the activities, LMS uses standard web-developer stack as well, (2) The aforementioned compatibility tests, including several rounds of pilot runs with various devices |
References
1. Nicholson, P. A history of e-learning. Computers and Education; Springer: Berlin/Heidelberg, Germany, 2007; pp. 1-11.
2. Euroclio. Historiana. Library Catalog. Available online: https://historiana.eu/ (accessed on 10 March 2022).
3. Archive, N. DocsTeach. Available online: https://www.docsteach.org/ (accessed on 10 March 2022).
4. Hussey, M. Teaching with online primary sources: Documents from the national archives: DocsTeach.gov from the national archives. Teaching History: A Journal of Methods; Emporia State University: Emporia, KS, USA, 2011; Volume 36, pp. 34-39.
5. Levesque, S. The Virtual Historian. L’historien Virtuel. Available online: https://www.virtualhistorian.ca/ (accessed on 10 March 2022).
6. Lee, J.K.; Friedman, A.M. Research on Technology in Social Studies Education; Information Age Publishing: Charlotte, SC, USA, 2009.
7. Haydn, T.; Counsell, C. History, ICT and Learning in the Secondary School; Psychology Press: Hove, UK, 2003.
8. Poitras, E.; Lajoie, S.; Hong, Y.-J. The design of technology-rich learning environments as metacognitive tools in history education. Instr. Sci.; 2012; 40, pp. 1033-1061. [DOI: https://dx.doi.org/10.1007/s11251-011-9194-1]
9. Breakstone, J. Try, Try, Try Again: The Process of Designing New History Assessments. Theory & Research in Social Education; Routledge: Abingdon-on-Thames, UK, 2014; Volume 42, pp. 453-485. [DOI: https://dx.doi.org/10.1080/00933104.2014.965860]
10. Sleeter, N.; Schrum, K.; Swan, A.; Broubalow, J. “Reflective of my best work”: Promoting inquiry-based learning in a hybrid graduate history course. Arts and Humanities in Higher Education; SAGE Publications: Thousand Oaks, CA, USA, 2020; Volume 19, pp. 285-303. [DOI: https://dx.doi.org/10.1177/1474022219833662]
11. Mynaříková, L.; Novotný, L. Knowledge Society Failure? Barriers in the Use of ICTs and Further Teacher Education in the Czech Republic. Sustainability; 2020; 12, 6933. [DOI: https://dx.doi.org/10.3390/su12176933]
12. European Commission: Sustainable Development Goals. Available online: https://ec.europa.eu/sustainable-development/goal4_en (accessed on 14 March 2022).
13. Future of Education and Skills 2030: OECD Learning Compass 2030. Available online: https://www.oecd.org/education/2030-project/ (accessed on 14 March 2022).
14. Schleicher, A. The Impact of COVID-19 on Education Insights from Education at a Glance. 2020; Available online: https://www.oecd.org/education/the-impact-of-covid-19-on-education-insights-education-at-a-glance-2020.pdf (accessed on 10 March 2022).
15. Dhawan, S. Online Learning: A Panacea in the Time of COVID-19 Crisis. J. Educ. Technol. Syst.; 2020; 49, pp. 5-22. [DOI: https://dx.doi.org/10.1177/0047239520934018]
16. Brom, C.; Lukavský, J.; Greger, D.; Hannemann, T.; Straková, J.; Švaříček, R. Mandatory Home Education During the COVID-19 Lockdown in the Czech Republic: A Rapid Survey of 1st–9th Graders’ Parents. Front. Educ.; 2020; 5, 103. [DOI: https://dx.doi.org/10.3389/feduc.2020.00103]
17. Pavlas, T.; Pražáková, D.; Zatloukal, T.; Andrys, O.; Novosák, J.; Folwarczný, R.; Borkovcová, I.; Modráček, Z.; Chovancová, K. Distance Education in Primary and Secondary Schools; Česka ŠkolnÍ Inspekce: Prague, Czech Republic, 2020.
18. Česká školní inspekce. Zkušenosti žáků a učitelů ZŠ s distanční výukou ve 2. pololetí 2019/2020. 2020; Available online: https://www.zkola.cz/zkusenosti-zaku-a-ucitelu-zs-s-distancni-vyukou-ve-2-pololeti-2019-2020/ (accessed on 10 March 2022).
19. Bonk, C.J. Pandemic ponderings, 30 years to today: Synchronous signals, saviors, or survivors?. Distance Educ.; 2020; 41, pp. 589-599. [DOI: https://dx.doi.org/10.1080/01587919.2020.1821610]
20. Rapanta, C.; Botturi, L.; Goodyear, P.; Guàrdia, L.; Koole, M. Online University Teaching During and After the Covid-19 Crisis: Refocusing Teacher Presence and Learning Activity. Postdigital Sci. Educ.; 2020; 2, pp. 923-945. [DOI: https://dx.doi.org/10.1007/s42438-020-00155-y]
21. Česká školní inspekce. Tematická zpráva-Distanční vzdělávání v základních a středních školách; Tematická zpráva, Česká školní inspekce, Praha. 2021; Available online: https://www.csicr.cz/cz/Aktuality/Tematicka-zprava-Distancni-vzdelavani-v-zakladnich (accessed on 10 March 2022).
22. Wineburg, S.S. On the Reading of Historical Texts: Notes on the Breach Between School and Academy. Am. Educ. Res. J.; 1991; 28, pp. 495-519. [DOI: https://dx.doi.org/10.3102/00028312028003495]
23. Group, S.H.E. Beoynd the Bubble- History Assessments. Available online: https://online.stanford.edu/courses/gse-xsheg0009-beyond-bubble-assessing-historical-thinking (accessed on 10 March 2022).
24. Blankenship, W.G. Teaching and Assessing Historical Thinking: Reading Like a Historian and Beyond the Bubble. Theory Res. Soc. Educ.; 2015; 43, pp. 568-576. [DOI: https://dx.doi.org/10.1080/00933104.2016.1099596]
25. Roth, W.M.; Radford, L. Re/thinking the Zone of Proximal Development (Symmetrically). Mind, Cult. Act.; 2010; 17, pp. 299-307. [DOI: https://dx.doi.org/10.1080/10749031003775038]
26. HistoryLab Project: An Example of the Recommended Procedure for an Activity. Available online: https://historylab.cz/cviceni-aktualni/assets/pdf/doporuceny-postup/why-were-they-displaced.pdf (accessed on 14 March 2022).
27. Seixas, P.; Morton, T.; Colyer, J.; Fornazzari, S. The Big Six: Historical Thinking Concepts; Nelson Education: Toronto, ON, Canada, 2013.
28. Činátl, K.; Najbert, J.; Ripka, V. Historical Literacy in the Historylab Application: A Realistic Approach to Mastering the Didactic Theory of History. Hist. Issues Probl.; 2021; 2021, pp. 50-70.
29. Van Boxtel, C.; van Drie, J.; Stoel, G. Improving teachers’ proficiency in teaching historical thinking. The Palgrave Handbook of History and Social Studies Education; Springer: Berlin/Heidelberg, Germany, 2020; pp. 97-117.
30. Ripka, V.; Hoření, K. Průběh a výstupy testování aplikace HistoryLab.cz. Promýšlet Dějepis v 21. Století: Digitální Aplikace pro Práci s Prameny HistoryLab.cz; Najbert, J. Ústav pro Studium Totalitních Režimů: Praha, Czech Republic, 2017.
31. Boxtel, C.V.; Drie, J.V. Historical Reasoning. The Wiley International Handbook of History Teaching and Learning; John Wiley & Sons: Hoboken, NJ, USA, 2018; pp. 149-176. [DOI: https://dx.doi.org/10.1002/9781119100812.ch6]
32. Badashov, A. Design Principles behind Great Products. 2017; Available online: https://medium.muz.li/design-principles-behind-great-products-6ef13cd74ccf?gi=f2ae3f7743c1 (accessed on 10 March 2022).
33. Hess, W. Design Principles: The Philosophy of UX. Available online: https://whitneyhess.com/blog/2011/05/04/design-principles-the-philosophy-of-ux/ (accessed on 10 March 2022).
34. Eyal, N. Hooked: How to Build Habit-Forming Products; Illustrated ed. Hoover, R. Portfolio: New York, NY, USA, 2014.
35. Tribukait, M.; Stegers, S. Historiana. An online resource designed to promote multi-perspective and transnational history teaching. History Education and Digital Practices; Carretero, M.; Carassou, M.C. Routledge: New York, NY, USA, 2021.
36. Bures, M. PCTgen: Automated Generation of Test Cases for Application Workflows. New Contributions in Information Systems and Technologies; Springer: Cham, Switzerland, 2015; Volume 353, pp. 789-794. [DOI: https://dx.doi.org/10.1007/978-3-319-16486-1_78]
37. Bures, M.; Cerny, T.; Klima, M. Prioritized Process Test: More Efficiency in Testing of Business Processes and Workflows. International Conference on Information Science and Applications (ICISA) 2017; Lecture Notes in Electrical Engineering Springer: Singapore, 2017; Volume 424, pp. 585-593. [DOI: https://dx.doi.org/10.1007/978-981-10-4154-9_67]
38. Najbert, J. Promýšlet Dějepis v 21. Století Digitální Aplikace pro Práci s Prameny HistoryLab.cz; Ústav pro Studium Totalitních Režimů: Praha, Czech Republic, 2017.
39. Oddělení vzdělávání, ÚSTR. Jak na Historylab-YouTube Playlist. 2021; Available online: https://www.youtube.com/watch?v=iikRR6Xdm9Q (accessed on 10 March 2022).
40. Dreesen, T.; Akseer, S.; Brossard, M.; Dewan, P.; Giraldo, J.P.; Kamei, A.; Mizunoya, S.; Ortiz, J.S. Promising Practices for Equitable Remote Learning: Emerging Lessons from COVID-19 Education Responses in 127 Countries; United Nations: New York, NY, USA, 2020; [DOI: https://dx.doi.org/10.18356/3ec22dd9-en]
41. Kang, B. How the COVID-19 Pandemic Is Reshaping the Education Service. The Future of Service Post-COVID-19 Pandemic; Springer: Berlin/Heidelberg, Germany, 2021; Volume 1, pp. 15-36. [DOI: https://dx.doi.org/10.1007/978-981-33-4126-5_2]
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Ripka, Vojtech 2
; Buresova, Karolina 3 ; Frajtak, Karel 2 ; Maha, Jiri 2
; Cinatl, Kamil 2
1 Department of Computer Science, Faculty of Electrical Engineering, Czech Technical University, 121 35 Prague, Czech Republic
2 Institute for the Study of Totalitarian Regimes, 130 00 Prague, Czech Republic;
3 Department of Education, Faculty of Arts, Charles University, 116 38 Prague, Czech Republic;