A Novel Gamification Application for High School Student Examination and Assessment to Assist Student Engagement and to Stimulate Interest

Abstract

Formal education in high school focuses primarily on knowledge acquisition via traditional classroom teaching. Younger generations of students tend to lose interest and to disengage from the process. Gamification, the use of gaming elements in the training process to stimulate interest, has been used lately to battle this phenomenon. The use of an interactive environment and the employment of tools familiar to today’s students aim to bring the student closer to the learning process. Even though there have been several attempts to integrate gaming elements in the teaching process, few applications in the student assessment procedure have been reported so far. In this article, a new approach to student assessment is implemented using a gamified quiz as opposed to standard exam formats, where students are asked to answer questions on the material already taught, using various gaming elements (leaderboards, rewards at different levels, etc.). The results show that students are much more interested in this interactive process and would like to see this kind of performance assessment more often in their everyday activity in school. The participants are also motivated to learn more about the subject of the course and are generally satisfied with this novel approach compared to standard forms of exams.

1. Introduction

Gamification has become a major tool for providing knowledge to various audiences using gaming elements to stimulate interest since 2000. Gamification emerged as a concept when various researchers realized that most people spend a considerable amount of their free time in gaming activities, which provide them with satisfaction and relaxation [1,2]. Gamification emerged as a concept to use this human behavior in combination with work in various areas such as customer engagement, employee engagement, and various other business activities [3].

There are two main definitions of gamification widely accepted by the academic and professional communities. In their well-known paper [4], Deterding et al. defined gamification as the “use of game design elements in nongaming context”. Another was given by Huotari and Hamari [5], focusing on the interaction between customers and companies, as “a process of providing affordances for gameful experiences which support the customers’ overall value creation”. The main idea is to provide the user with motivation and to increase their engagement in various areas of everyday human activities. Gamification incorporates gaming elements such as points, badges awarded for specific achievements, challenges that motivate the user to learn or search for the answer, leaderboards that show the progress of the members of the group and enhance competition, and various incentives to engage the user in the process. A comprehensive review of the theory, the game mechanics used in various applications, and the objectives was given by Gupta and Gomathi [6].

The results have been particularly encouraging from the very start and this has contributed to a strong increase in ongoing research in many fields. The use of computer technology and mobile devices in the developed solutions makes the implementation of such methods particularly attractive to younger generations, thus providing an extensive area of improvement in teaching effectiveness. One of the main advantages of such methods is providing students the ability to interact with the educational material, thus making the process more interesting and entertaining. Furthermore, gamification helps students in improving their thinking and allows for dynamic learning [7].

Motivation and engagement are usually considered prerequisites for the completion of a task or encouragement of a specific behavior. In education, especially in formal high school education, there are growing problems concerning the involvement of students in the everyday educational process. These problems include, but are not limited to, a lack of engagement and a growing distance between the traditional methods of teaching used and the students’ extensive experience with technology. These problems illustrate a lack of motivation, resulting in various forms of students distancing themselves from the process [8,9,10], which not only is the subject of scientific research, but is being discussed in governmental bodies [11] or even in large mainstream media (see for example, [12]). Similar challenges are reported in other educational programs, such as employee training [13,14].

Several studies have focused on the use of gamification in the learning process; however, very few studies focus on the use of gamification in the assessment process. The assessment process is important since it is the main tool for evaluating student performance. The nature of the assessment process may be a huge burden for students (see, for example, [15]), as they tend to feel that they have to memorize things and produce adequate answers within a limited amount of time, writing on paper in a way that is not required of them in their everyday activities. One important aspect of disengagement is the rigorous format of the examination process: students increasingly tend to think of it as a cause of their failure. In this paper, we present an application used for student assessment in the context of teaching a particular course. It has the form of a quiz and may be used either for intermediate assessment or in the place of a final examination. The student may achieve a better grade by completing specific tasks, such as an excellent performance in a section of the quiz, a series of correct answers, etc. This format may provide students with extra motivation and make them feel that they operate in a familiar environment, thus enhancing their performance.

2. Gamification in Education: Basic Principles and Background

In today’s era, the concept of gamification is increasingly prevalent. It has been observed that its use has helped in various fields and subjects, such as product development [16], the development and learning of human resources in businesses [17,18], enhancing productivity in businesses, and developing employee skills [19]. It has also contributed to improving marketing performance [20,21], tourism [22,23], healthcare [24,25], and has gradually been integrated into education [26,27,28].

Traditional school education is considered effective but not as engaging for students. Educators are constantly seeking innovative teaching approaches to motivate students to participate in learning and increase the level of engagement [29]. The methods used by instructors are important in this process and different teachers employ different approaches. A concise review of such approaches and the various factors involved is presented in [30]. These methods take advantage of today’s technological lifestyle and of the digital culture familiar to the young generations of students; therefore, the use of these methods in education may significantly contribute to learning and students’ development. In this context, the use of digital games as learning tools is an innovative approach which may be used to stimulate cognitive processes [31]. Of course, creating an attractive and comprehensive educational game is a time-consuming and costly process [32], as it aims to target specific learning objectives based on the game designer’s intentions. Moreover, the effective implementation of a game in the classroom requires technical infrastructure and appropriate training for educators and students who wish to use it. The educational organization also needs to accurately identify their students’ characteristics and background to design an effective gamified system [33].

The application of gamification in the past decade has seen a significant increase and has produced positive results. Thus, many fields use it as a teaching method, especially when combined with new technologies. The results indicate that students are motivated to engage in the development of the course and try harder, since they see a clear goal reflected in their performance. This is a very important issue, since a considerable percentage of students in the Greek secondary education system find the standard classroom methods of teaching and examinations boring, theoretical, and not very engaging. In particular, the standard examination method, which basically consists of typical written exams, is considered a huge burden and discourages students.

The emergence of such obstacles in student engagement has prompted research aimed at developing new methods and novel tools to motivate students during the learning process. As mentioned earlier, very few studies focus on gamifying the assessment process. Wang [34] used Kahoot!, a commercial quiz software where students have limited time to answer simple questions. Points are awarded for each correct answer and a leaderboard is presented to the students after each question. The student is rewarded if they can achieve a series of correct answers. The interaction and the competitive nature of the process were appreciated by the students. However, there are no gaming elements used in this approach for interaction from the student. Moccozet et al. [35] found that gamification may be implemented to enhance collaboration between members of a group by rewarding their interactions with the system in an online shared workspace platform. Cheong et al. [36] used a gamified multiple-choice quiz, implemented as a software tool under the name Quick Quiz. The students were engaged so that they wanted to complete the quiz and they were happy playing it. A recent work [37] suggested a method to relate user profiles to gamified elements to be used in software applications, using a generic gamification model, GamiProM. The approach tries to determine which gaming elements are the best fit for use with specific audiences. However, this approach may not be applicable in a school environment, where students are not a homogeneous group.

The effect on student engagement was evaluated as positive by the students involved, mainly due to the interactive nature of the applications [38]. Although behavioral responses are in general positive, it is still unclear whether this approach leads to an improvement in learning by the students involved. A systematic approach for the construction of a framework for student engagement was presented in [39], which may be used for the selection of specific gamified elements according to the desired student experience. Still, research involves interventions stimulating executive functions mediated by technologies [31].

It is essential to recognize that there still exist limitations and drawbacks. For instance, Kwon and Ozpolat [40] reported that gamification in student assessment has a negative impact on content knowledge and student perceptions. In [41], perceived anxiety caused by gamification was reported to have a negative effect on knowledge retention. In a concise systematic review of digital games in learning [42], it was claimed that the increased use of gamified elements may result in reduced effects on learning outcomes. Zheng et al. [43] conclude that higher-level knowledge activities may negatively affect learning for specific categories of students, if they do not possess the foundations required.

On the other hand, the wide spectrum of different educational environments makes it difficult to generalize the positive results. In [44], it was found that the effect of gamification on academic achievement differed significantly according to the course type, a result that is in line with the conclusions in [45,46]. Small sample size and homogeneity are recognized as drawbacks [47], while [46] also points to the need for longer periods of experimentation to produce more robust results.

Finally, it is always a challenge for researchers to identify whether gamification results in an improvement in intrinsic motivation from the participants. Ibanez et al. [48] found that students seeking credit recovery achieve better performance with gamified exams but are likely to quit the effort once they complete the mandatory parts of the assignments. Mekler et al. [49] came to a similar conclusion, identifying a lack of effect on intrinsic motivation from points–badges–leaderboards (PBL). While the positive impact of gamified applications on performance metrics is a common observation among almost all such studies, the effects on the attitude of the participants towards knowledge remain mixed and unclear.

Although these limitations and constraints are important, gamification applications may give teachers very useful tools if they are used with appropriate preparation and customization of the material and the exam content.

In this study, an attempt is made to incorporate new gamified elements in student assessment. The student is required to provide the correct answers, as is typically the case in any examination. However, in this study, additional resources are available to the students and their performance is affected by the use of these additional resources. Some of the resources may be spent if the student does not know a specific answer or if the student wants to eliminate some of the available answers. This novel type of reward may provide additional motivation to complete the assessment process and to enhance learning. The aim of this research is to measure student performance and student engagement in this new learning environment. Specifically, the main research questions to be answered are stated below:

• Is the gamification of an exam process acceptable and efficient?

• Does the creation of an entertaining and more playful environment provide motivation for more engagement of the students?

3. Methodology

It is very important to emphasize that the Greek educational system has traditionally been reluctant to adopt novel methods in teaching and exams. Furthermore, the legal framework is strict, and instructors may get in trouble for deviating from the standard guidelines, even in minor aspects of the process. Therefore, very few students nationwide have had similar experiences. Taking these facts into consideration, the research was designed in a way that would be accepted by the school and would not create problems that might jeopardize its focus.

The current research was conducted in a 4th-year high school class taught by the first author in the city of Arta, Epirus. An application was submitted to the school administration, while at the same time, the students of the class were asked if they wished to participate. A total of 36 students expressed their consent. The subject of the course was “Informatics Applications”, and the quiz was used as a midterm exam on the subsection “Principles and practice of HTML”. The students were informed that anonymity was assured, that the process was part of ongoing scientific research to explore various effects of gamification in the education process at different levels, that no personal data were to be used in the research, and that there were no other risks associated with it. The director of the school was also informed in detail of the above aspects of the research and formal approval was given.

The specific course unit dealing with the HTML protocol and its applications was first taught and the appropriate course materials were delivered. Part of the material was the relevant subsection in the official textbook used for the course. The subsection of the book and the other relevant materials were covered in the span of four weeks for a total of sixteen class hours. As is typically the case in Greek high school education, the students attended lectures by the teacher, turned in assignments, and practiced in a computer lab. Students were asked to answer the quiz questions and then fill out the online questionnaire mentioned in the previous section. The students were also encouraged to engage in an informal discussion with the instructor and to provide their comments regarding the quality of the process and their experience. Comments that were considered important are mentioned in the Research Results section.

The quiz application was given to the selected student group under supervision of the instructor. Prior to the start of the process, the participants were informed of the rules of the game and particularly of the role of the gaming elements. Upon completion of the quiz, the students were invited to fill out a questionnaire via Google Forms. The questionnaire consisted of a total of five sections: demographic characteristics, an evaluation of the game in terms of its content, an evaluation of the application, the organization of the research, and a more general evaluation of the inclusion of gamification in the educational process. All questions of the questionnaire were the closed type, either multiple choice or scaled (Not at all–Fairly–Very much). A few sample questions are shown below:

• -. Were the quiz questions relevant to the material taught?

  • Yes

  • No

• -. Were the questions simple and understandable?

  • Yes

  • No

• -. Does the application make you feel interested?

  • Yes

  • No

• -. Changes you would make to the application (pick only one)

  • Increase of time limit for each question

  • Increase of the number of questions

  • Increase in aids (e.g., lives, cherries, light bulbs)

  • Easier questions

• -. Give your own suggestionDo you find the organization of the quiz satisfactory?

  • Yes

  • No

• -. Do you believe the implementation of such quizzes in learning would make teaching more interesting?

  • 1: Not at all

  • 2: Quite a bit

  • 3: Very much

• -. Would you like similar methods to be incorporated into the teaching of courses during the school year?

  • Yes

  • No

4. A Novel Application for Student Assessment

The development of an application based on gamification for educational purposes should encompass certain characteristics. Key features that need to be considered include determining the target group of students, setting the learning objectives, creating the educational environment, and incorporating elements into the game. By studying these criteria, the use of gamification in learning can improve children’s skills by 40% while also motivating students to participate in teaching more enjoyably and efficiently [50]. Gamification influences students’ behavior and provides incentives for them to improve both their knowledge and their skills [10].

To determine whether a gamified exam process would be helpful in student engagement and the stimulation of interest, a gamified quiz was developed. Its goal was to allow students to assess their acquired knowledge while introducing them to a new way of learning.

The quiz consisted of 45 multiple-choice questions. The questions were organized in three sections, with each section representing a higher level of difficulty. A time limit of 39 s was allocated for each question in the first section and the limit was shortened by 6 s for each subsequent section. After completing the quiz, a questionnaire was answered by the students to evaluate the effect of the procedure on the students’ motivation and engagement.

4.1. Application Architecture

The functionality of the application is illustrated in Figure 1 in the form of a flow diagram. Upon launching the application, a check is performed to establish the user’s connection with the central server, which is essential for seamlessly recording their activity during the game. Within the options menu, three choices are offered: connecting to the server, registering as a new user (if necessary), and a review of the game rules. Once the connection is established, an administrator can opt for the quiz management menu, while this option is deactivated otherwise. The application may host several different quizzes and the user selects any one of these quizzes.

The user is assigned a custom number of Hearts, Halves, and Lights, as defined by the administrator. Obviously, the values of these variables are at the discretion of the instructor depending on the difficulty of the material, the level of the students, and any other factor that may be identified as relevant. A Heart is a “life” (a very common gaming element), and it corresponds to a wrong answer given to a particular question. In practice, the initial value of the parameter is the maximum count of wrong answers allowed. When the maximum number is exceeded, the game ends. A Half is an aid allowing the elimination of two out of the three wrong answers for a particular question. This is a very common element of knowledge games and is intended to help the player decide between fewer options. Finally, a Light is a “hint”, providing the correct answer when the player is not sure of the correct answer to a particular question. In practice, the initial value of the parameter is the maximum count of unanswered questions allowed. The player is not awarded any points for unanswered questions.

At the start of the game, a random question is presented to the player along with four possible answers. If the player does not utilize any of the available aids (light bulb, cherry), then, as indicated by the flowchart, only the correctness of the answer is evaluated. Alternatively, if aids are used, additional checks and parameter calculations occur. This process repeats until there are no more “lives” left or until the player has responded in some definite way to all the questions.

4.2. Software and Tools

The main application was implemented using the object-oriented programming language Java. In summary, it has the following features:

• Connection to a database for data storage;

• User registration;

• The creation of new quizzes by administrators;

• Entry, modification, and deletion of questions by administrators.

A website was implemented on a central server for displaying user results, with an administrative environment and additional comparative data representations for gamification purposes. The application installer is available for Windows 7 and above and can be downloaded freely from the server website (https://www.dit.uoi.gr/quiz/setupQuiz.exe, accessed on 31 July 2023). For the application to run, the Java Runtime Environment (JRE) software should be installed, as it includes the necessary libraries and components used by the application. In summary, the commercial products used for the development of the application were Java, MySQL, PHP, JavaScript, and AJAX.

4.3. Graphical Game Environment

The two main screens of the preliminary phase are shown in Figure 2. The player needs to register to be able to play the quiz. Then, the available options are presented to the player and the appropriate quiz is selected from a dropbox. The option “Question manager” is only available to users with administrator privileges to provide access to the material and structure of the quizzes.

A sample screen of a question is shown in Figure 3. The reader may notice the various game elements used (cherry, heart, light bulb). As is typically the case with such applications, the icons used convey a positive message, while implying their actual use: a light bulb provides a hint, while a heart indicates a life extension, allowing for more incorrect answers. The question in the screenshot is stated in Greek, as the screenshot shows what the students actually see. The question asked is “Which HTML element defines the title of a document?”.

4.4. Game Rules

The rules are defined by the instructor; obviously, these are customized according to the student needs, the level of the course, and various other factors. In this game, the rules are defined as follows:

• Response time contributes to the game’s scoring, allowing for a reward for players who answer more quickly. Ideally, the player should not pause at all, if possible.

• If a player is not sure about an answer, a light bulb may be spent to provide the correct answer.

• A player may randomly reduce the available answers by half by spending a cherry.

• As the score increases, the player is rewarded by earning lives (heart icons), ready answers (bulb icons), and hints (cherries).

• During the game, the player is informed of the score needed to achieve the best score for that specific quiz.

• After a certain number of answers, the difficulty level increases and an appropriate graphic appears to inform the player.

• For each correct answer, the player earns 200 points and additional points based on their response speed.

• If a game is not completed, the player’s attempt is recorded as incomplete.

• At the end of the game, the browser automatically opens for the player to view their ranking for the quiz.

4.5. Sample Questions

To illustrate the nature of the quiz and the form of the questions, sample questions for each of the three sections are provided here:

First section

What does HTML mean/

• Home Tool Markup Language

• Hyper Tool Markup Language

• Hyperlinks Tool Markup Language

• HyperText Markup Language

Second section

The <TITLE> label has to be placed:

• Inside the <body> label

• Inside the <head> label

• Outside the <htm1> label

• There is no constraint

Third section

What is the output of the command: <h1 style=“background-color: red;”>Hello World!</h1>

• Hello World! On a red background

• Hello World! In red letters

• Hello World! in Large font size

• Hello World!

The first section is obviously the easiest, partly serving as a warmup process. Questions like this are supposed to give some credit to the student while at the same time introducing them to the environment and the nature of the quiz. The entire section is intended to encourage the student to keep on trying.

The second section intends to check the student’s knowledge on standard mechanics of the language. The presence of the wrong choices helps students decide which one of the available options best fits the question.

The third section intends to check the student’s knowledge on “programming” commands where the format and actual appearance of the output are implemented. Apparently, this is the more complex task that a student must be familiar with.

5. Research Results

As an initial remark, it should be stressed that the students involved participated with enthusiasm and were fully engaged with the process, which is not typical when they take part in standard forms of examinations. All participants answered all the survey questions after the completion of the exam. The main results are summarized below:

• The participants were approximately equally divided by gender (52.8% female, 47.2% male) and were around the age of 15.

• Regarding the questions about the content of the quiz, the students’ responses were unanimous in that they found the quiz questions easy and understandable. Furthermore, most students (80.6%) found the time given for the questions to be sufficient, while the use of aids in the quiz, such as ready-made answers (light bulbs) and discarding half of the choices available (cherries), greatly helped them in answering the questions (97.2%). This is an important finding since it points to a possible indirect method of learning via an appropriate choice of the available answers.

• Given the enthusiasm and focus of the students, it came as no surprise at all that there were unanimous positive responses to the suggestion that similar quizzes be used in other courses and to the suggestion that similar methods be employed in teaching. This conclusion appears to be in line with several similar studies (for instance, see [38,51]). Along the same lines, the overall level of satisfaction from the process was the highest possible for almost 9 out of 10 students (Figure 4). In comments turned in by some of the participants, it was stated that the process as a whole is by far more interesting than the standard methods of examination.

• 4.. The overwhelming majority of students stated that the quiz improved their knowledge of the subject, apparently due to the existence of alternative answers and to the use of the nonstandard helping tools (bulbs and cherries) (Figure 5). A total of 11.1% reported an improvement in skills. Very few reported an improvement in the stress felt. This is an interesting finding, which requires further attention and analysis, since this indirect acquisition of knowledge may be very useful in raising the level of the not-so-strong and/or not committed students.

• 5.. When asked to characterize the learning procedure via the quiz as “easier”, “amusing”, or “more interesting”, a large majority found it amusing, while no student said it was easier (Figure 6). This finding may provide a better insight into the reasons behind student disengagement, as students do not focus on an “easier” exam but rather on the satisfaction they obtain from the process. It points to the motivation created for the students by their experience with gaming tools and mechanics in other aspects of everyday life.

• 6.. When asked whether they believe that the use of such exams would make teaching more interesting, all but one student picked the highest rating (Figure 7). This result shows that such novelties could create a substantial boost in student involvement, provided that an appropriate use is made by the teacher and the educational organization in general, as already suggested in several sources (see e.g., [33]).

• 7.. Finally, the students made extensive use of the various help items (light bulbs, cherries) provided by the quiz, as indicated in the diagram below (Figure 8). After the quiz, some students stated to the instructor that the use of the help items provided them with some insight into the nature of the question, leading them to better understand the issue the question dealt with. This is an important observation, as this feature may attract students who are less knowledgeable or less committed.

Overall, the results of the survey are very encouraging and show that there is ample room for the incorporation of gamified tools in the education process, providing the educational system with strong tools to battle student disengagement from traditional learning methods.

6. Conclusions and Future Research

Based on the current research, it was found that students desire to incorporate similar methods of practice in school teaching, as learning becomes more enjoyable and interesting for them. A desire to repeat the quiz multiple times was observed, as they believe that repeated use of the quiz will enhance their knowledge and make the educational process more entertaining. There was extensive use of the game elements, indicating that familiarity with games may be further explored. Finally, students showed interest in comparing their performance to that of their peers, indicating that the competition between them may prove to be a good motivating factor to improve performance. Overall, the results are encouraging and show that the inclusion of gamified elements in the educational process is very welcomed by the student body. Obviously, most educational systems are slow in adapting to novel approaches and this is a major challenge, particularly in Greece. We believe that this study and other studies in the area will contribute to a shift in educational policies, which will make the educational process more effective and engaging.

Based on the above, it is evident that students desire the integration of new elements into the educational process, and gamification represents an innovative approach that will effectively aid learning and encourage students to study further. Due to the positive response from students to retake the quiz, it is apparent that it would be highly interesting to offer the quiz to other classes and schools. As outlined in Section 2, the course type and subject may have varied effects on the students; therefore, this is an interesting challenge for future research. In this direction, we intend to run similar quizzes in classes with different characteristics and compare the performance of the respective student groups as well as their responses to the review questionnaire. The size of the sample was also identified as a critical factor for the validity of the results [46,47]; therefore, a larger audience should be found for the next stages of this experiment. Furthermore, the inclusion of similar methods of evaluation in the overall evaluation system, possibly adding a variable bonus to the final grade, would be an interesting direction for future research and could provide stronger conclusions regarding the actual effect on student performance. Another promising extension is to run a comparative study between different subjects in the same class, trying to determine whether the effects presented here differ when applied in different knowledge areas (e.g., Math, History, and Technology). Naturally, student performance may be affected to a higher degree in technical subjects, such as Computer Programming, Chemistry, or Mathematics, rather than in more theoretical subjects, such as History or Sociology, due to the very nature of each course. A future study may focus on this question.

Author Contributions

Conceptualization, A.M.G. and N.A.; methodology, A.M.G. and N.A.; software, A.M.G.; validation, N.A.; formal analysis, A.M.G. and N.A.; investigation, A.M.G. and N.A.; resources, A.M.G. and N.A.; data curation, A.M.G.; writing—original draft preparation, A.M.G.; writing—review and editing, A.M.G. and N.A.; visualization, A.M.G. and N.A.; supervision, N.A.; project administration, N.A.; funding acquisition, nonapplicable. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data are not publicly available due to issues related to school regulations regarding students’ assessment.

Acknowledgments

The authors wish to thank Vasileios Charilogis for his insightful remarks and suggestions on the generation of the code used for the quiz and on issues relevant to the administration of the application. They also wish to thank Arta Elpida Tsifetaki for her support in getting the research approved and acquiring the necessary equipment. We acknowledge the support of this work by the project “Immersive Virtual, Augmented and Mixed Reality Center of Epirus” (MIS:5047221), which is implemented under the action “Reinforcement of the Research and Innovation Infrastructure”, funded by the Operational Programme “Competitiveness, Entrepreneurship and Innovation” (NSRF 2014–2020) and co-financed by Greece and the European Union (European Regional Development Fund).

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures

Figure 1. The functionality of the application in the form of a flow diagram.

Figure 2. Registration information for user and main menu of the application.

Figure 3. Sample screen of a typical quiz question.

Figure 4. Level of satisfaction with the quiz.

Figure 5. Factors of improvement for students.

Figure 6. Characterization of learning procedure in comparison to standard methods.

Figure 7. Degree of interest in similar exams in the future.

Figure 8. Extent of use of various help items.

References

1. Aulia, V.R.; Subriadi, A.P.; Nadlifatin, R. Gamification: A Comprehensive Review of Literature. Proceedings of the 2022 1st International Conference on Information System & Information Technology (ICISIT); Yogyakarta, Indonesia, 27–28 July 2022; pp. 277-282. [DOI: https://dx.doi.org/10.1109/ICISIT54091.2022.9872830]

2. Hamari, J. Transforming Homo Economicus into Homo Ludens: A Field Experiment on Gamification in a Utilitarian Peer-to-Peer Trading Service. Electron. Commer. Res. Appl.; 2013; 12, pp. 236-245. [DOI: https://dx.doi.org/10.1016/j.elerap.2013.01.004]

3. Zhao, F.; Guo, D. Rewards in Gamification. HCI in Games. Lecture Notes in Computer Science; Fang, X. Springer: Cham, Switzerland, 2019; Volume 11595, [DOI: https://dx.doi.org/10.1007/978-3-030-22602-2_34]

4. Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L. From game design elements to gamefulness: Defining Gamification. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments; Tampere, Finland, 28–30 September 2011; pp. 9-11. [DOI: https://dx.doi.org/10.1145/2181037.2181040]

5. Huotari, K.; Hamari, J. Defining Gamification—A Service Marketing Perspective. Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments; Tampere, Finland, 28–30 September 2011; pp. 17-22. [DOI: https://dx.doi.org/10.1145/2393132.2393137]

6. Gupta, A.; Gomathi, S. A Review on Gamification and its Potential to Motivate and Engage Employees and Customers: Employee Engagement through Gamification. Int. J. Sociotechnol. Knowl. Dev.; 2017; 9, pp. 42-52. [DOI: https://dx.doi.org/10.4018/IJSKD.2017010103]

7. Man, K.; Chung, G.; Yui, H.; Chan, N. Applying Gamification in Vocational and Professional and Education and Training (VPET) Classroom to Engage Students’ Learning. Proceedings of the Asian Conference on Education; Kobe, Japan, 20–23 October 2016.

8. Caraway, K.; Tucker, C.M.; Reinke, W.M.; Hall, C. Self-efficacy, goal orientation, and fear of failure as predictors of school engagement in high school students. Psychol. Sch.; 2003; 40, pp. 417-427. [DOI: https://dx.doi.org/10.1002/pits.10092]

9. Munna, A.S.; Kalam, M.A. Impact of Active Learning Strategy on the Student Engagement. GNOSI Interdiscip. J. Hum. Theory Prax.; 2021; 4, pp. 96-114.

10. Huang, W.H.-Y.; Soman, D. A Practitioner’s Guide to Gamification of Education. Research Report Series Behavioural Economics in Action; Rotman School of Management, University of Toronto: Toronto, ON, Canada, 2013.

11. Education and Employment-Eighth Report; Education and Employment Committee Publications House of Commons: London, UK, 1999; Available online: https://publications.parliament.uk/pa/cm199899/cmselect/cmeduemp/57/5702.htm (accessed on 17 July 2023).

12. Richtel, M. Growing Up Digital, Wired for Distraction. The New York Times; 21 November 2010; Available online: https://www.nytimes.com/2010/11/21/technology/21brain.html (accessed on 14 July 2023).

13. Lloyd, K. Evaluating the Effectiveness of an Employee-Training Program. For Dummies; Available online: http://www.dummies.com/how-to/content/evalu-ating-the-effectiveness-of-an-employeetrainin.html (accessed on 14 July 2023).

14. Silberman, J. How to Measure Employee Training Effectiveness?. Training Station; Available online: http://trainingstation.walkme.com/how-to-meas-ure-employee-training-effectiveness/ (accessed on 1 May 2013).

15. Kirkpatrick, R.; Zang, Y. The Negative Influences of Exam-Oriented Education on Chinese High School Students: Backwash from Classroom to Child. Lang. Test. Asia; 2011; 1, 36. [DOI: https://dx.doi.org/10.1186/2229-0443-1-3-36]

16. Aazami, M.; Nazari, K.; Akbari, P. Explanation of gamification technology in businesses development. J. Bus. Adm. Res.; 2023; Accepted for publication

17. Nibu, J.T.; Baral, R.; Crocco, O.S. Gamification for HRD: Systematic Review and Future Research Directions. Hum. Resour. Dev. Rev.; 2022; 21, pp. 198-224. [DOI: https://dx.doi.org/10.1177/15344843221074859]

18. Brull, S.; Finlayson, S.; Kostelec, T.; MacDonald, R.; Krenzischeck, D. Using gamification to improve productivity and increase knowledge retention during orientation. JONA J. Nurs. Adm.; 2017; 47, pp. 448-453. [DOI: https://dx.doi.org/10.1097/NNA.0000000000000512]

19. Bahra, W.; Mavrogenis, V.; Sweeney, E. Gamification of warehousing: Exploring perspectives of warehouse managers in the UK. Int. J. Logist. Res. Appl.; 2022; 25, pp. 247-259. [DOI: https://dx.doi.org/10.1080/13675567.2021.1892042]

20. Behl, A.; Pereira, V.; Jayawardena, N.; Nigam, A.; Mangla, S. Gamification as an innovation: A tool to improve organizational marketing performance and sustainability of international firms. Int. Mark. Rev.; 2023; Accepted for publication [DOI: https://dx.doi.org/10.1108/IMR-05-2022-0113]

21. Milanesi, M.; Guercini, S.; Runfola, A. Let’s play! Gamification as a marketing tool to deliver a digital luxury experience. Electron. Commer. Res.; 2022; [DOI: https://dx.doi.org/10.1007/s10660-021-09529-1]

22. Roinioti, E.; Pandia, E.; Konstantakis, M.; Skarpelos, Y. Gamification in Tourism: A Design Framework for the TRIPMENTOR Project. Digital; 2022; 2, pp. 191-205. [DOI: https://dx.doi.org/10.3390/digital2020012]

23. Rallis, I.; Kopsiaftis, G.; Kalisperakis, I.; Stentoumis, C.; Koutsomitsos, D.; Riga, V. Cultural-Oriented Mobile Game Application and the Gamification of Tourism. Trandisciplinary Multispectral Modelling and Cooperation for the Preservation of Cultural Heritage; Moropoulou, A.; Georgopoulos, A.; Doulamis, A.; Ioannides, M.; Ronchi, A. Communications in Computer and Information Science; Springer: Cham, Switzerland, 2022; Volume 1574, [DOI: https://dx.doi.org/10.1007/978-3-031-20253-7_20]

24. Damaševičius, R.; Maskeliūnas, R.; Blažauskas, T. Serious Games and Gamification in Healthcare: A Meta-Review. Information; 2023; 14, 105. [DOI: https://dx.doi.org/10.3390/info14020105]

25. Al-Rayes, S.; Al Yaqoub, F.A.; Alfayez, A.; Alsalman, D.; Alanezi, F.; Alyousef, S.; AlNujaidi, H.; Al-Saif, A.K.; Attar, R.; Aljabri, D. et al. Gaming elements, applications, and challenges of gamification in healthcare. Inform. Med. Unlocked; 2022; 31, 100974. [DOI: https://dx.doi.org/10.1016/j.imu.2022.100974]

26. Oliveira, W.; Hamari, J.; Shi, L.; Toda, A.M.; Rodrigues, L.; Palomino, P.T.; Isotani, S. Tailored gamification in education: A literature review and future agenda. Educ. Inf. Technol.; 2023; 28, pp. 373-406. [DOI: https://dx.doi.org/10.1007/s10639-022-11122-4]

27. Surendeleg, G.; Murwa, V.; Yun, H.-K.; Kim, Y.S. The Role of Gamification in Education—A Literature Review. Contemp. Eng. Sci.; 2014; 7, pp. 1609-1616. [DOI: https://dx.doi.org/10.12988/ces.2014.411217]

28. Kiryakova, G.; Angelova, N.; Yordanova, L. Gamification in Education: Breakthroughs in Research and Practice; IGI Global Publisher of Timely Knowledge: Hershey, PA, USA, 2014; Chapter 32 pp. 635-660. [DOI: https://dx.doi.org/10.4018/978-1-5225-5198-0]

29. Lee, J.; Hammer, J. Gamification in Education: What, How, Why Bother?. Acad. Exch. Q.; 2011; 12, pp. 1-5.

30. Cents-Boonstraa, M.; Lichtwarck-Aschoff, A.; Denessen, E.; Aelterman, N.; Haerens, L. Fostering student engagement with motivating teaching: An observation study of teacher and student behaviours. Res. Pap. Educ.; 2020; 36, pp. 754-779. [DOI: https://dx.doi.org/10.1080/02671522.2020.1767184]

31. Moron, V.B.; Barbosa, D.N.F.; Sanfelice, G.R.; Barbosa, J.L.V.; Leithardt, D.R.F.; Leithardt, V.R.Q. Executive Functions, Motor Development, and Digital Games Applied to Elementary School Children: A Systematic Mapping Study. Educ. Sci.; 2022; 12, 164. [DOI: https://dx.doi.org/10.3390/educsci12030164]

32. Kapp, K.M. The Gamification of Learning and Instruction: Game-Based Methods and Strategies for Training and Education; Pfeiffer: San Francisco, CA, USA, 2012.

33. Ofosu-Abong, K. The Shift to Gamification in Education: A Review on Dominant Issues. J. Educ. Technol. Syst.; 2020; 49, pp. 113-137. [DOI: https://dx.doi.org/10.1177/0047239520917629]

34. Wang, A.I. The wear out effect of a game-based student response system. Comput. Educ.; 2015; 82, pp. 217-227. [DOI: https://dx.doi.org/10.1016/j.compedu.2014.11.004]

35. Moccozet, L.; Tardy, C.; Opprecht, W.; Léonard, M. Gamification-based assessment of group work. Proceedings of the 2013 International Conference on Interactive Collaborative Learning; Kazan, Russia, 25–27 September 2013; [DOI: https://dx.doi.org/10.1109/ICL.2013.6644565]

36. Cheong, C.; Cheong, F.; Filippou, J. Quick Quiz: A Gamified Approach for Enhancing Learning. Proceedings of the Pacific Asia Conference on Information Systems (PACIS); Jeju Island, Republic of Korea, 18–22 June 2013; Volume 206, Available online: https://aisel.aisnet.org/pacis2013/206 (accessed on 18 July 2023).

37. Dalmina, L.; Damasceno Vianna, H.; Pfeiffer SalomaoDias, L.; Lazarotto Schroeder, G.; Francisco, R.; Luis Victoria Barbosa, J. GamiProM: A Generic Gamification Model Based on User Profiles. Int. J. Hum.-Comput. Interact.; 2022; pp. 1-17. [DOI: https://dx.doi.org/10.1080/10447318.2022.2159769]

38. Handoko, W.; Mizkat, E.; Nasution, A.; Hambali,; Eska, J. Gamification in Learning using Quizizz Application as Assessment Tools. Proceedings of the Annual Conference on Science and Technology Research (ACOSTER); Medan, Indonesia, 20–21 June 2020; Volume 1783.

39. Rivera, E.S.; Palmer, C.L. Gamification for student engagement: A framework. J. Furth. High. Educ.; 2021; 45, pp. 999-1012. [DOI: https://dx.doi.org/10.1080/0309877X.2021.1875201]

40. Kwon, H.Y.; Özpolat, K. The Dark Side of Narrow Gamification: Negative Impact of Assessment Gamification on Student Perceptions and Content Knowledge. INFORMS Trans. Educ.; 2021; 21, pp. 67-81. [DOI: https://dx.doi.org/10.1287/ited.2019.0227]

41. Ge, Z.-G. The impact of a forfeit-or-prize gamified teaching on e-learners’ learning performance. Comput. Educ.; 2018; 126, pp. 143-152. [DOI: https://dx.doi.org/10.1016/j.compedu.2018.07.009]

42. Clark, D.B.; Tanner-Smith, E.E.; Killingsworth, S.S. Digital games, design, and learning: A systematic review and meta-analysis. Rev. Educ. Res.; 2016; 86, pp. 79-122. [DOI: https://dx.doi.org/10.3102/0034654315582065]

43. Zheng, B.; Lin, C.H.; Kwon, J.B. The impact of learner-, instructor-, and course-level factors on online learning. Comput. Educ.; 2020; 150, 103851. [DOI: https://dx.doi.org/10.1016/j.compedu.2020.103851]

44. Dikmen, M. Does gamification affect academic achievement? A meta-analysis of studies conducted in Turkey. Int. J. Curric. Instr.; 2021; 13, pp. 3001-3020.

45. Metwally, A.H.S.; Chang, M.; Wang, Y.; Yousef, A.M.F. Does Gamifying Homework Influence Performance and Perceived Gameful Experience?. Sustainability; 2021; 13, 4829. [DOI: https://dx.doi.org/10.3390/su13094829]

46. Ortiz, M.; Chiluiza, K.; Valcke, M. Gamification and learning performance: A systematic review of the literature. Proceedings of the 11th European Conference on Games Based Learning; Graz, Austria, 5–6 October 2017.

47. Nieto-Escamez, F.A.; Roldán-Tapia, M.D. Gamification as Online Teaching Strategy During COVID-19: A Mini-Review. Front. Psychol.; 2021; 12, 648552. [DOI: https://dx.doi.org/10.3389/fpsyg.2021.648552]

48. Ibanez, M.-B.; Di-Serio, A.; Delgado-Kloos, C. Gamification for Engaging Computer Science Students in Learning Activities: A Case Study. IEEE Trans. Learn. Technol.; 2014; 7, pp. 291-301. [DOI: https://dx.doi.org/10.1109/TLT.2014.2329293]

49. Mekler, E.D.; Brühlmann, F.; Tuch, A.N.; Opwis, K. Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Comput. Hum. Behav.; 2017; 71, pp. 525-534. [DOI: https://dx.doi.org/10.1016/j.chb.2015.08.048]

50. Giang, V. Gamification Techniques Increase Your Employees’ Ability To Learn By 40%. Business Insider; 18 September 2013; Available online: https://www.businessinsider.com/gamification-techniques-increase-your-employees-ability-to-learn-by-40-2013-9 (accessed on 22 July 2023).

51. Kotsifakos, D.; Karali, M.G.; Katzola, E.; Kravvari, A.; Douligeris, C. A Web Gamification Platform Inspired by the Maritime Tradition Museum in Perama, Greece, for Students in the Sector of Maritime Professions in Vocational Education and Training. Eur. J. Eng. Technol. Res.; 2022; 1, pp. 49-52. [DOI: https://dx.doi.org/10.24018/ejeng.2022.1.CIE.2963]