1. Introduction
The 21st-century information technology revolution has affected the economic, social, and educational spheres of society. In the education sector, the most distinguishable change was the integration of computers and information and communication technologies (ICT) in the teaching and learning processes to help active education and achieve goals [1].
The proliferation of enabling learning technologies worldwide and the boom in information has caused e-learning to become more accessible, being pointed out as a trend in the educational field [2,3,4]. Recently, due to the social isolation caused by the COVID-19 pandemic, educators have been being forced to find alternatives for teaching by adopting and disseminating new technologies aimed at online education, which reinforces this trend.
New pedagogies, practices, and theories are emerging in the educational technology, such as gamification [5,6,7]. Gamification refers to applying game design aspects in non-gaming contexts to increase engagement, motivation and improve task performance [8,9,10,11,12,13]. It is used as an innovative way to engage citizens in sustainability practices directed at climate change, water scarcity, or extreme weather events, etc. in their local communities [14].
Game elements allow the players to participate in a pseudoconflict to encourage experimental learning in an interactive way and a safe and risk-free environment [15,16,17,18,19,20]. It is produced to educate the player by using attractive and pleasurable educational scenarios [21,22,23,24].
Furthermore, games are present in young learners’ lives [25]. That is why the new generation, known as the “generation of players”, “digital natives”, “network generation”, or “generation Z”, widely accept games [10,26,27,28,29]. This new generation learns differently from their predecessors and has effective “e-lives”, connecting and sharing information instantly through blogs, online shops, social networks, online games, or webchats [30].
Although it is recent, the concept of gamification is widely adopted by researchers from different domains and research areas [31] such as mathematics, computing, social sciences, engineering, health, environment, sustainability, and others [32,33,34,35]. In the educational field, the benefits of gamification have become more apparent due to the need to provide the student with a unique and unusual experience [36]. This experience is essential, especially in courses that strive to emphasize the multidisciplinary nature of disciplines such as sustainability and address issues that raise awareness of sustainable processes in industries [37].
Research focused on game design began in 2003. However, the application of gamification in the educational sector and especially in the e-learning environment increased exponentially from 2014 [27,38,39]. Many researchers strongly believe in gamification to drive learning, interaction, collaboration, skill acquisition, understanding content, and changing attitude or behavior. Compared to traditional practices, they support effectiveness in engagement, motivation, stimulating curiosity, autonomy, and satisfaction [5,6,34,38,40,41,42,43,44,45,46,47,48].
On the other hand, some researchers believe that gamification should be used as a complementary practice despite being a valuable teaching and learning strategy and not replace the current educational methods [13,49,50,51,52]. Moreover, scholars point out that gamification depends on the context and, even more, on users [3,10,53,54,55,56].
In this scenario, the following research question arises: How can the game elements be organized logically to support the gamified system planning to effectively promote more sustainable behavior and guarantee the quality of the learning process?
A systematic literature review was carried out with a content analysis of 130 articles indexed in the Web of Science database to answer this question. Based on the research findings, a framework called 7GOALS (Gamification-Oriented Active Learning Steps) were established relating behavioral attitudes and gamification elements to the PDCA cycle stages (Plan, Do, Check, Act) to provide continuous improvement of the learning process by gamification and expand the application of themes such as sustainability.
To the best of the authors’ knowledge, this is the first study that brings together the main game elements, the PDCA cycle, and behavioral attitudes to promote learning. As a result, researchers on this topic, teachers, and education professionals will have a summarized and structured portrait of the relationship between the main elements and behavioral attitudes leading to improvement by learning gamification.
2. Materials and Methods
This study is a systematic literature review based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol [57,58,59]. The search occurred on the Web of Science database, which is widely recognized by the academic community. Several preliminary searches were performed in the topic field, analyzing titles, abstracts, authors’ keywords, and KeyWords Plus. The investigation broadly included the following query: game* AND e-learn* AND (education OR teaching). There is a gap in the academic literature that includes articles on gamification in e-learning to induce sustainable behaviors.
Therefore, the research method adopted here consisted of two steps: the first established the systematic review of the literature and the second stage involved the content analysis of articles [60]. The search retrieved 578 documents published until 2019, and it covered all research areas. After filtering only the documents classified as Article and Review, 166 articles remained. This study was also limited to documents with full text available and published in English or Spanish, resulting in an additional 10 and 4 documents discarded, respectively. After a full reading of the articles, 130 papers were assessed as eligible to meet the proposed objectives of this study (Figure 1).
After conducting a content analysis, the selected articles’ data were clustered to analyze how behavioral attitudes and gamification elements are related in the literature to provide changes for more sustainable behavior. In this stage, the data were computed quantitatively through meta-analyses and in an inclusive way, allowing an article to encompass various elements and/or attitudes. The tables organized the data and conditional formatting to identify possible relations. The Venn diagram, commonly used in mathematics, was also used to symbolize the associations.
Based on this analysis, the gamification elements that effectively promote behavioral attitudes, sustainable environments and guarantee the quality of the learning process were structured in a sequence of steps and classified as 7GOALS (Gamification-Oriented Active Learning Steps). This study phase was based on adapted concepts of the WCM methodology (World Class Manufacturing) and the PDCA cycle.
The WCM methodology presents a set of management principles, procedures, and techniques to reduce losses and waste and improve production processes [61,62]. Although developed and consolidated in manufacturing environments, this methodology can also be applied in other areas. The WCM methodology uses seven measures (“Seven Problem-Solving Steps”), which allow its evolution step by step [63].
In the “Seven Problem-Solving Steps” approach, the first step identifies the problem phenomenon; the second step studies the system and classifies the problems according to their impact factor; the third defines the objectives and ways to find a solution to it; the fourth analyzes the root cause; the fifth step aims to apply countermeasures with the implementation of actions to overcome the root causes; the sixth step checks the results; and the seventh expands these results so that there is a continuous improvement [63].
The PDCA cycle was also used as a complementary tool, and it corresponds to a method commonly employed for continuous improvement by covering four stages: Plan, Do, Check, Act [64]. These steps are an approach widely used in distinct fields, including e-learning [65,66]. It is one of the most popular quality management system models worldwide. This study was adopted as a method to support action planning to achieve the learning objectives proposed [67]. It is a management tool applied to ensure company’s products and processes continuous improvement [68]. It leads to systematic actions that speed up obtaining better results and guaranteeing organizations’ growth.
Thus, phase P (Plan) defines what you wish to do, establishes objectives and activities to achieve it. Phase D (Do) is characterized by executing what was planned to implement the necessary actions. Phase C (Check) involves monitoring the process and verifying that the objectives have been achieved. Finally, phase A (Act) refers to seeking continuous improvement until reaching the excellent standard or the expected results [69].
A framework is presented associating 7GOALS with the PDCA cycle to ensure continuous improvement of the learning process and generate changes for more sustainable behavior.
3. Results
In response to the research question, this section presents a framework called 7GOALS (Gamification-Oriented Active Learning Steps) with the main game elements organized in a sequence of steps to positively affect users’ emotional feelings. This structure is multidisciplinary and can be used in any knowledge field, such as in sustainability, and it is associated with the PDCA for continuous improvement of the learning process.
3.1. Mapping: Behavioral Attitudes and Elements of Gamification
3.1.1. Behavioral Attitudes Associated with Gamification
According to the literature, gamification induces three behavioral attitudes in students: motivation, engagement, and satisfaction (Table 1).
Motivation, combined with human affection and emotions, is one of the most relevant behavioral attitudes in learning, especially in online courses, as cited by 45 articles. It can be extrinsic or intrinsic. Extrinsic motivation refers to doing something in search of a result, such as getting a good grade. However, intrinsic motivation refers to doing something because it is appealing or pleasant, and it is related to three aspects: autonomy, relationship, and competence [8,33,89,101].
Some factors encourage motivation: competition, challenge, imagination, collaboration, goals, cooperation, reward, and recognition, many of which are present in games [12,28,39,87,91,97,100].
The game also stimulates motivation because of its impact on cognitive development, affective skills, emotional and social states of the players [33,76,93,96]. According to [16,90], add that games allow slightly reducing the theory’s abstraction by encouraging a greater motivation. In addition to motivation, educational games improve interest and concentration, improving students’ learning [94].
Engagement is another behavioral attitude identified as a valuable indicator of academic performance and positively correlates with student learning outcomes [103,106], as seen in 26 articles. Engagement happens when someone’s attention is completely focused on a specific activity. Thus, the virtual games industry indicates that engagement is a tool to keep the player’s attention on the game [110]. Engagement can also be explained as the time and effort that students put into educational activities and the prerequisite for effective educational practices. It is made up of three elements: affection, cognition, and behavior [103].
Behavioral engagement consists of the degree of involvement, imagination, attention, enthusiasm, and willingness to learn students demonstrate during the learning process. Cognitive engagement can be portrayed as the psychological or mental investment and the student’s effort to understand, learn, or acquire accurate knowledge and skills [9].
Strength of educational games derives from their potential to induce student motivation and engagement during the activities performance [30,74,92,104,105]. In [108], it is also highlighted that games can vary their difficulty level, keeping the participant at a high level of engagement. In turn, in [75], it is declared that the games’ learning anxiety might increase the students’ attention as they devote more time to the learning process and task fulfillment, thus obtaining more knowledge retention. The interaction of users with social factors also influences engagement [11,79].
Another crucial behavioral attitude is student satisfaction, which positively affects learning and brings a sense of pleasure [75,80], according to 21 articles. Satisfaction is related to student performance and is an indicator of educational quality as well as an important factor in measuring success [9,10,11,12,13,72,78] or the effectiveness of the e-learning process [98,115].
Studies show that the perception of ease of use and, mainly, playfulness (or usefulness) directly affects the satisfaction and continuous use of a gamified system [80,83]. Social influences generate a sense of competition and improve results, thus yielding various emotional skills such as self-satisfaction, self-esteem, self-efficacy, and pride [79,83,88,113,116]. Still, activities that satisfy the motivational needs for capacity, autonomy, and skills positively impact users’ satisfaction [33].
3.1.2. Relationship between the Behavioral Attitudes Associated with Gamification
Although the behavioral attitudes presented in the previous section are said to be the main ones, there are no studies in the academic literature that associate these attitudes. This work contributes to mitigating the gap.
Therefore, given what is presented in the literature individually, it is possible to analyze the relationships between the three attitudes pairwise. From the Venn diagram, it is possible to verify that of the 63 articles that mention motivation, engagement, and satisfaction as the main behavioral attitudes to promote the gamified learning process, five simultaneously attribute these attitudes while 39 articles, that is, 61.9% of the total, mention them separately. Still, 24 articles indicate at least two of these attitudes, corresponding to 38% of the total. One more attitude, satisfaction, is little mentioned in isolation (9.5%), which shows that students need to be engaged and motivated to be satisfied with the learning process (Figure 2).
Finally, it is possible to conclude that motivation, engagement, and satisfaction improve student interest in the learning process and performance results and should be highly encouraged in gamified systems.
3.1.3. The Main Elements Used to Stimulate the Learning Cycle
Activities based on active learning and gamification can be an alternative to influence more positive attitudes among students and create a more friendly environment for teaching [117]. Besides, the active learning provided by gamification is among the most effective learning strategies as it provides the students with better understanding, retention of information, and greater use of the classes [12,96,98,111].
It is believed that identification and use of game elements in this process can boost learning in several segments, domains, and research areas [31]. The main elements identified in the systematic review are presented by steps to follow (Table 2).
Step 1:. User-centricity and personalized experiences
It is necessary to analyze the psychological profile of the student. Theory and practice should be centered on the individual and not on technology, guided by the learner’s cognitive and socioemotional profile and how it impacts the learning technology. In personalization, user-centered design is of increasing importance and requires clear recognition. The user-centered design allows learning experiences to be psychologically accessible and not just physically. It also reflects the student’s cognitive knowledge and socioemotional profile besides how unique psychological attributes relate to the environment and the pedagogical framework. Still, it allows considering that each student has different needs that must be acknowledged [12,27,54,75,105,119,120,121,123,124].
Personalized or adaptive learning experiences for each student are an essential characteristic of e-learning systems. When referring to personal information, systems can provide customized content for each student or direct them to a specific path [122]. Besides, it must focus on a target audience and be age-oriented. To be successful, correctly getting the age of future players is crucial [34,118].
Step 2:. Challenging and clear missions
Games need to be challenging, involving an appropriate amount of difficulty and complexity, having multiple objectives, and some amount of randomness [30,33]. Players must overcome challenges along the way of the game to achieve specific and clear goals [103].
The challenge refers to the level of difficulty of the activities. It is an element present in many case studies, which states that increasing difficulty levels for students help them develop knowledge progressively [5,30,76,79,99,125,127]. The challenges must be adequately engaging so that students are motivated to solve them. However, it is vital to ensure that the challenges are not too complicated nor generate boredom or anxiety. The challenge should be considered when defining the game’s goals and rules to guarantee that they are both feasible and encouraging [55]. Moreover, it must start at a lower level before increasing the difficulty to keep the interest and the flow going. If there is a moderate difficulty due to in-game mechanics variations, these mechanics should be manageable [118].
Furthermore, the tasks to be performed must be clearly explained by involving players in an active learning process that helps them master the game’s mechanics and overcome challenges [82,93,103,118,126].
Step 3:. Narrative and fantasy
The game’s context, which is expressed by its narrative, graphics, and music, can stimulate the players’ interest if adequately used [93]. The staging and storyline provide scenarios that help the players’ decision-making [53]. A strong narrative with multiple arcs is a sequence of events based on storytelling, exciting characters that students can interact with, and a realistic scenario supported by challenging resources and activities can also influence learning outcomes [103].
The narrative consists of some essential aspects such as event, character, configuration, arrangement, and time. The results show that with these elements, students can convince themselves with the game’s stories through fantasy, which leads to higher levels of learning and motivation to complete tasks [11,39,72,74,79,81,84,106,110]. However, the emphasized fantasy must be carefully chosen to attract the target audience and create intrinsically motivating environments.
It also allows for a fantastic metaphor and a dramatization dynamic underlying a problem-based learning strategy [95,128]. Still, it makes it possible to create more authentic and realistic environments that contribute to better identifying behaviors and greater involvement of participants [102].
Step 4:. Repetitive Loops and Freedom to Fail
The repetitive loops of trying, failing, tuning, improving, and trying again in a game are called productive negativity and contribute to game design. A user tries a challenge with their current knowledge and may fail (negative). They need then to reformulate their understanding (productive) to achieve the game’s goal and get a result [71].
In the form of resentment, disappointment, and irritation, this productive negativity exposes students to something new. It allows them to reach a different perspective of awareness through experience, which promotes learning. Disappointment, failure, and frustration can play a prominent role in motivating and learning in the game. Failure is necessary for this experience, so much so that winning without previous losses creates a feeling of discontent with the victory. In general, games cause negativity while encouraging task-solving, thus ensuring the advancement of knowledge and skills through an engaging environment [71].
Although some degree of anxiety caused by failure is required, it is not interesting when it turns into disappointment. Therefore, task sequences must be carefully designed to suit the players’ knowledge and profile, including low penalties for failures to allow multiple attempts and the willingness to get it right [27,55,93].
Still, the literature suggests that several repetition practices are part of the learning process and are necessary to achieve the desired quality [39,71,87,93,98,109,129].
Step 5:. Competition and social engagement
Competition is a crucial element of games and is constantly used to improve academic performance [9,13]. It must be considered in the game, either against yourself or other users [30,55]. The desire to win the game results in competition and ambition, making the students think and learn quickly to achieve success, providing extra motivation that makes it possible to achieve better results [30,131]. Many scholars believe that the use of competition can improve the robustness of the game’s design [32,55,85,95,98,104,110,116,126,130,131].
The competition allows students to share ideas for help or shift the blame in failure cases. Besides, students like to learn from the unexpected and excitement, waiting for what will come next asking what the topic will be and who will win. That is, challenges and visibility of tasks by other users can boost competition [30,79,92].
Social influence can also stimulate the competition to reach a higher status on the leaderboard and enable numerous emotional skills such as self-satisfaction, self-esteem, and pride [9,79]. Leaderboards, which usually contain names, nicknames, and/or avatars, serve as sources of motivation because students want their effort to be recognized publicly. It helps to compare their progress with other colleagues [10,93].
However, too much emphasis on competition can also hurt some users as they may become obsessed with trying to improve their points and, therefore, lose focus on learning [39,55].
-
Step 6:. Feedback and reflection
Providing feedback and allowing knowledge acquisition through game experience are also essential factors in a positive learning experience [55,75,99,124,134,136]. Feedback prevents students from getting lost or confused about what to do and how to do a task or mission. Furthermore, it helps in tracking progress, performance, improving understanding, and encouraging engagement [70,77,92,106,111,112,133,137,138]. Feedback can also help students experiment and learn from the game [3,135].
In complex games, it is essential to provide specific and individual feedback provided by a facilitator or a tutor for the evolution of tasks [55]. Although the teacher does not act synchronously during the game, he/she plays the role of the mentor by clarifying the rules, training, sending information, updating the content, and providing feedback on the game’s progress [13].
Many students consider immediate feedback as a gamified environment [50,75,138]. According to the learner’s pace and time, direct, individualized, detailed, and continuous feedback is considered to be of great importance to increase engagement and improve understanding [30,70,73,82].
Feedback can also increase through levels by determining specific points and allowing content to be unlocked progressively through levels. The same logic works for tasks as a task answered correctly allows the user to proceed to the next step. This visual progression of content allows feedback and increases focus on the learning process [10,132]. Furthermore, dialog boxes clarifying the appropriate responses, explaining why a response is correct, or simply indicating that the choice was not the most suitable in an educational and nonpunitive way are important sources of feedback [103].
Step 7:. Reward and social credibility
Games try to activate the students’ feelings with reward systems that immediately recognize the success of players [93]. The reward-scoring systems, plotting with animations, badges, trophies, and leaderboards are often employed in educational games to stimulate the player and encouragingly present the progress of activities. When a reward is involved, the players are generally more willing to win, and thus their motivation is increased and eventually encourages participation. Furthermore, it provides teachers with an incentive to introduce technologies into the classroom and offer learning feedback [5,33,56,71,72,75,79,93].
The rewards can be through virtual or real prizes based on the time spent in the system after completing tasks or on other elements. Commitment, motivation, engagement, or satisfaction are the main benefits of this element. The awards/badges and honors are directly related to the visual status and social credibility of players, and that is why they are considered highly motivating, fun, and encouraging [10].
The use of badges (medals) is a typical method of reward. They are visual displays of the users’ progress that can function as an extrinsic reward, leading people to perform specific activities to receive them. They can also stimulate social comparison, that is, compare their achievements with the accomplishments of others and serve as a conductor of behavior [8,85]. The possibility of posting the progress on the social media timeline to share with friends can increase motivation, user interaction with the game and facilitate self-directed learning [83].
However, care must be taken to balance intrinsic and extrinsic motivation in reward systems. Learning can be most effective when one is intrinsically motivated, for example, by one’s curiosity or desire to solve a challenge for one’s satisfaction; or extrinsically, to obtain a grade or other earnings [8,33,55,93].
The elements listed here are essential to promote the learning process. The logic of the proposed learning cycle will not change if other elements are added following the steps shown in Table 2. This addition can make the process even more user-friendly.
3.1.4. Relationship between Gamification Elements
Although game elements are the basic building blocks of gamification, there is no consensus on their use. Academic literature on the topic is dispersed, isolated, and fragmented, as shown in Table 3. Within this scenario, this article focuses on the dispersed elements, that is, to understand the relationship between the elements of the main game to boost the students’ behavioral attitudes and then organize them in a didactic way to contribute to a better learning process.
Table 3 shows the main game elements identified in this work and compares how the studies mention these elements together. It can be seen that the authors make little association of the “User-centricity and personalized experiences” element with the other elements. The “Challenging and clear missions” element and “Repetitive loops and freedom to fail” appear with representation in all the other elements, confirming their importance in gamified systems. In addition, “Narrative and fantasy” is more related to “Competition and social engagement” since systems that allow for greater immersion in the game’s narrative positively impact the competition and the desire to obtain better results to share in social networks. Furthermore, although “Feedback and reflection” in quantitative terms is the most representative element, it still appears in isolation, with few citations compared to other elements. Finally, it is possible to note that the “Reward and social credibility” element is related to “Challenging and clear missions” and “Competition and social engagement”, which is expected since for students to receive rewards, they need to have clear game rules to stimulate competition.
Therefore, this work aimed to reduce the gap in the literature regarding the association of these elements and present how the aspects of gamification can be organized in a sequence of steps to promote behavioral attitudes and ensure the quality of the learning process.
3.2. A Theoretical Framework
The step-by-step idea based on the “Seven Problem-Solving Steps” approach of the WCM methodology was used and adapted to present, in a sequential manner, the evolution of concepts and the system planning process to achieve the learning allowed by gamification in e-learning by increasing motivation, engagement, and satisfaction of students. These elements were classified here using the acronym 7GOALS—Gamification-Oriented Active Learning Steps.
The 7GOALS were identified in the literature in a dispersed and isolated way and here they were gathered and structured to facilitate the design of gamified environments in e-learning. With the support of the PDCA, the elements were combined in phases of planning, execution, control, and action to cluster the systematic activities necessary to create such systems, ensure that all stages are considered and meet the fundamental requirements for the perception of value in students’ interactions with the gamified learning process.
The association of the PDCA with the 7GOALS steps is shown by the framework presented in Figure 3. Thus, through increasing cycles of evolution aimed at student learning, it is possible to promote the improvement and expansion of gamification in this process.
Step 1 (user-centered and personalized experiences), Step 2 (challenging and clear missions), and Step 3 (narrative and fantasy) are part of the system’s planning process (P—Plan). The aim was to define the student’s profile, cognitive skills, target audience, age group, level of difficulty of the activities and establish the challenges by clearly explaining their objectives and drawing them through narratives with a fantasy metaphor to attract the initial audience. This step is vital to planning a gamified system, that is, to get to know the public for whom it is intended, define the challenges that these students expect, and understand the necessary narrative for the immersion to be complete.
Then, Step 4 (repetitive loops and freedom to fail) and Step 5 (competition and social engagement) are associated with the execution of tasks by students (Phase D—Do). Thus, repeating activities several times with the permission to fail prepares students for proficiency in skills/knowledge and competition through social influence and leaderboards. It is here that the student will be able to perform the tasks, repeat them as many times as necessary, not be severely punished for failures, in addition to competing with other students collaboratively or individually with the possibility of sharing the results through social networks.
Then, Step 6 (feedback and reflection) allows you to monitor the learning process, verify that the objectives have been achieved, and reflect on the necessary improvements (Phase C—Check). At this time, students will be able to have signs of the progress of tasks and direction regarding what needs to be done to complete them. As seen in the systematic literature review, several studies consider this element as one of the main elements to learning effectively.
Finally, Step 7 (reward and social credibility) corresponds to the search for the expansion of knowledge caused by the stimulus of rewards and the social credibility that points, medals, and trophies, among others, represent (Phase A (Act)). It is a way of celebrating learning, correct the flaws found in the previous step, and maintaining that constant cycle.
4. Discussion
Although recent, gamification in e-learning has been widely discussed in the literature in several areas, such as mathematics, social sciences, engineering, health, environment, sustainability, and others [32,33,34,35]. The focus of gamification in the learning process has always been on increasing motivation, engagement, and satisfaction to promote behavioral change and increase interest in learning [10,12,18,27,93], whether to stimulate sustainable environments or in the context of general education [53,87].
For example, it is possible to observe that gamification has already been applied in business (to engage employees and increase customers’ loyalty), education (to foster the engagement of students), and environmental sustainability (to inspire and provoke engagement in environmentally positive behaviors) [79]. In the context of education and sustainable environments, scholars have conducted studies to confirm further the advantages of the learner-centered scenario design and game-based learning approach in the knowledge transfer in the topic of sustainability [53]. Some studies integrate agribusiness with environmental and social issues to boost sustainability through gamification applied to producers/professionals/pupils/students on a journey of discovery about agribusiness in all its complexity [128]. Others highlight the players’ quest to achieve the success, whether by fulfilling the game’s mission or developing a sustainable society [97]. Another one mentions the importance of using technology throughout the school working together with companies to produce sustainable, durable, affordable, and safe devices for students in the classroom [5]. In this way, game-based learning and adaptive learning are reasonable solutions to motivate the student, address the knowledge facing a large and heterogeneous public, reduce the costs of teaching collaborating with the financial aspects of education, and share resources and infrastructure for a sustainable society [87].
Learning based on game elements is one of the main trends currently in e-learning [3,55,56]. This learning format can be designed to collaboratively solve complex, real-world problems, make the learning process more efficient, get predictive modeling and real-time visualization, and retain more knowledge than the traditional methods [81,125]. In [108], games were used in virtual-world scenarios to take risks in a safe place for real-world activities. In [93], the students who completed the gamified experiment obtained better grades for practical tasks and overall scores; the results and effectiveness of learning compared to the traditional models is also presented in [102]. Learning to deal with complex and constantly evolving environments is also a result of this learning format [32].
Thus, gamified systems that encourage these behaviors and mix the lightness of playfulness with serious content can be used to disseminate changes towards more sustainable behaviors, considering psychological and social aspects that influence students in the learning process.
In addition, they can be powerful vehicles for assessing skills and behaviors such as persistence, flexibility, creativity, critical thinking, and cognitive development, positively impacting the students’ academic performance [9,33,75,80,93,103,106]. In this way, if the game is focused only on teaching, forgetting the playfulness part, you can create a tool that, despite being motivational, does not succeed in the results. On the other hand, if it is focused only on playfulness without paying attention to the motivational part, it is possible to create a product that does not serve as an awareness tool and does not fulfill its objective. Therefore, games must be developed in a format that should be as fun as possible and as a motivational tool that addresses the teaching theme without the player feeling that he is studying [34].
These duality, playfulness, and seriousness of the teaching process need to be contemplated in gamified systems. The 7GOALS is proposed to help with this, bringing together the main building blocks of gamification. The correct game elements to leverage the learning process and behavioral perspectives should be addressed. Thus, 7GOALS merge the playfulness and seriousness of teaching and support the planning of gamified systems to stimulate sustainable education in both knowledge and behavioral attitudes. First, it establishes elements that seek to know the user centrality and rules required to achieve the learning objectives and then include elements of fantasy, narrative, freedom of failure, feedback, and recognition to provide fun and awareness of learning simultaneously.
In this way, the leading game elements that encourage behavioral attitudes are organized in a logical sequence to guarantee the quality of the learning process. The steps are as follows: user centrality and experience personalization through challenges and clear missions, use of narratives and fantasies for better immersion in the game’s history, allowing repetitive loops with the possibility of failures, encouragement by competition and social engagement to achieve success, providing feedback and reflection to progress and performance, and, finally, rewards and social credibility to celebrate achievements and increase motivation. The PDCA cycle was used to expand the learning process through gamification in e-learning, with its four stages (planning, execution, monitoring, and improvement actions). These steps were associated with 7GOALS and the students’ behavioral attitudes with the development of a framework.
This will help to enable greater applicability of gamified systems for teaching sustainability, explore the finite nature of natural resources through the immersion in games, and demonstrate the importance of collective action at the community, national, and global levels.
The encouragement of more sustainable behaviors for this purpose through the game elements presented here and attitudes such as satisfaction, engagement, and motivation can contribute to students’ awareness and proximity to the real world that requires care and attention. In addition, the proposed format can be playfully used in sustainable education, approaching the new generations that are adept at this format and at the same time bringing a discussion that needs to be exhaustively contemplated in educational environments for greater environmental awareness and correspondence to the sustainable goals of the millennium (United Nations Sustainable Development Goals [139]). It is also possible to be used in the three objectives that make up the sustainability tripod, that is society [140], economy [141] and environment [142]. Furthermore, it can be an excellent resource to bring up sustainability issues in business environments that are not sustainable and help to intraduce the topic to those who have not had access to it yet.
5. Conclusions
This work aimed to present a framework proposal that relates behavioral attitudes and gamification elements to the stages of the PDCA (Plan, Do, Check, Act) to continuously improve the learning process over time. Although gamification is possibly the methodological tool that has stood out the most for implementation in the educational field and disciplines such as sustainability, the literature still presents the game’s elements in an isolated, dispersed, and fragmented way. Therefore, this work aimed to reduce the gap in the literature since to the best of the authors’ knowledge, this is the first study that brings together the main game elements, the PDCA cycle, and behavioral attitudes to stimulate change for more sustainable behavioral attitudes and ensure the quality of the learning process. This work can be helpful for researchers, teachers, and scholars who intend to use gamified environments in e-learning, such as for sustainability across several fields.
Conceptualization, R.P.O., C.G.d.S., A.d.C.R. and W.M.d.S.; Methodology, R.P.O., C.G.d.S., A.d.C.R. and W.M.d.S.; Investigation, R.P.O., C.G.d.S., A.d.C.R. and W.M.d.S.; Writing—original draft preparation, R.P.O., C.G.d.S., A.d.C.R. and W.M.d.S.; Validation, R.P.O., C.G.d.S., A.d.C.R. and W.M.d.S.; Writing—review and editing, R.P.O., C.G.d.S., A.d.C.R. and W.M.d.S. All authors have read and agreed to the published version of the manuscript.
This research received no external funding.
Not applicable.
Not applicable.
Not applicable.
The authors declare no conflict of interest.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Behavioral attitudes addressed in the literature.
| Attitude | Total | References |
|---|---|---|
| Motivation | 45 articles | Garcia-Cabot et al., 2019 [ |
| Engagement | 26 articles | Garcia-Cabot et al., 2019 [ |
| Satisfaction | 21 articles | Chan, Chan, and Fong, 2019 [ |
The main gamification elements used to stimulate the learning cycle.
| Step | Description | Total | References |
|---|---|---|---|
| 1 | User-centricity and personalized experiences | 13 articles | Tan and Zary, 2019 [ |
| 2 | Challenging and clear missions | 14 articles | Doney, 2019 [ |
| 3 | Narrative and fantasy | 15 articles | Chan, Chan, and Fong, 2019 [ |
| 4 | Repetitive loops and freedom to fail | 9 articles | Doney, 2019 [ |
| 5 | Competition and social engagement | 19 articles | Garcia-Cabot et al., 2019 [ |
| 6 | Feedback and reflection | 25 articles | Garcia-Cabot et al., 2019 [ |
| 7 | Reward and social credibility | 13 articles | Garcia-Cabot et al., 2019 [ |
Pairwise relationship between gamification elements.
| Elements of Gamification | User-Centricity and Personalized Experiences | Challenging and Clear missions | Narrative and Fantasy | Repetitive Loops and Freedom to Fail | Competition and Social Engagement | Feedback and Reflection | Reward and Social Credibility |
|---|---|---|---|---|---|---|---|
| User-centricity and personalized experiences | 13 | 1 | 0 | 1 | 0 | 2 | 1 |
| 100% | 8% | 0% | 8% | 0% | 15% | 8% | |
| Challenging and clear missions | 1 | 14 | 3 | 2 | 5 | 5 | 5 |
| 7% | 100% | 21% | 14% | 36% | 36% | 36% | |
| Narrative and fantasy | 0 | 3 | 15 | 2 | 5 | 2 | 3 |
| 0% | 20% | 100% | 13% | 33% | 13% | 20% | |
| Repetitive loops and freedom to fail | 1 | 2 | 2 | 9 | 4 | 1 | 2 |
| 11% | 22% | 22% | 100% | 44% | 11% | 22% | |
| Competition and social engagement | 0 | 5 | 5 | 4 | 19 | 5 | 5 |
| 0% | 26% | 26% | 21% | 100% | 26% | 26% | |
| Feedback and reflection | 2 | 5 | 2 | 1 | 5 | 25 | 3 |
| 8% | 20% | 8% | 4% | 20% | 100% | 12% | |
| Reward and social credibility | 1 | 5 | 3 | 2 | 5 | 3 | 13 |
| 8% | 38% | 23% | 15% | 38% | 23% | 100% | |
| Color legend | Weak | [Image omitted. Please see PDF.] | [Image omitted. Please see PDF.] | Strong |
References
1. Arouri, Y.M.; Odat, B. The Extent of Using E-Learning Among Teachers of Higher Basic Stage in Jordan. Int. J. Inf. Commun. Technol. Educ.; 2020; 16, pp. 57-69. [DOI: https://dx.doi.org/10.4018/IJICTE.2020070105]
2. Liu, G.-Z.; Wu, N.-W.; Chen, Y.-W. Identifying emerging trends for implementing learning technology in special education: A state-of-the-art review of selected articles published in 2008–2012. Res. Dev. Disabil.; 2013; 34, pp. 3618-3628. [DOI: https://dx.doi.org/10.1016/j.ridd.2013.07.007]
3. Sood, S.K.; Singh, K.D. An Optical-Fog assisted EEG-based virtual reality framework for enhancing E-learning through educational games. Comput. Appl. Eng. Educ.; 2018; 26, pp. 1565-1576. [DOI: https://dx.doi.org/10.1002/cae.21965]
4. Moubayed, A.; Injadat, M.; Shami, A.; Lutfiyya, H. Student Engagement Level in an e-Learning Environment: Clustering Using K-means. Am. J. Distance Educ.; 2020; 34, pp. 137-156. [DOI: https://dx.doi.org/10.1080/08923647.2020.1696140]
5. Chew, S.W.; Cheng, I.-L.; Chen, N.-S. Exploring challenges faced by different stakeholders while implementing educational technology in classrooms through expert interviews. J. Comput. Educ.; 2018; 5, pp. 175-197. [DOI: https://dx.doi.org/10.1007/s40692-018-0102-4]
6. Silva, F.B. Implicações da Gamificação no Projeto de Plataforma de Educação Online: Um Estudo de Caso [Tese (Doutorado)]; Universidade Federal de Minas Gerais: Belo Horizonte, Brazil, 2018.
7. Ramos, D.K.; Da Silva, G.A. Formação de professores a distância para o uso de jogos digitais na escola: Mudanças na prática pedagógica?. RIED; 2019; 22, pp. 265-285. [DOI: https://dx.doi.org/10.5944/ried.22.2.22685]
8. Kyewski, E.; Krämer, N.C. To gamify or not to gamify? An experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Comput. Educ.; 2018; 118, pp. 25-37. [DOI: https://dx.doi.org/10.1016/j.compedu.2017.11.006]
9. Bouchrika, I.; Harrati, N.; Wanick, V.; Wills, G. Exploring the impact of gamification on student engagement and involvement with e-learning systems. Interact. Learn. Environ.; 2019; pp. 1-14. [DOI: https://dx.doi.org/10.1080/10494820.2019.1623267]
10. Garcia--Cabot, A.; Garcia--Lopez, E.; Caro--Alvaro, S.; Gutierrez--Martinez, J.; De--Marcos, L. Measuring the effects on learning performance and engagement with a gamified social platform in an MSc program. Comput. Appl. Eng. Educ.; 2019; 28, pp. 207-223. [DOI: https://dx.doi.org/10.1002/cae.22186]
11. Romero-Rodriguez, L.M.; Ramirez-Montoya, M.-S.; Gonzalez, J.R.V. Gamification in MOOCs: Engagement Application Test in Energy Sustainability Courses. IEEE Access; 2019; 7, pp. 32093-32101. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2903230]
12. Terras, M.M.; Boyle, E.A. Integrating games as a means to develop e--learning: Insights from a psychological perspective. Br. J. Educ. Technol.; 2019; 50, pp. 1049-1059. [DOI: https://dx.doi.org/10.1111/bjet.12784]
13. Lorenzo--Alvarez, R.; Rudolphi-Solero, T.; Ruiz-Gómez, M.J.; Sendra--Portero, F. Game--Based Learning in Virtual Worlds: A Multiuser Online Game for Medical Undergraduate Radiology Education within Second Life. Anat. Sci. Educ.; 2019; 13, pp. 602-617. [DOI: https://dx.doi.org/10.1002/ase.1927]
14. Koroleva, K.; Novak, J. How to Engage with Sustainability Issues We Rarely Experience? A Gamification Model for Collective Awareness Platforms in Water-Related Sustainability. Sustainability; 2020; 12, 712. [DOI: https://dx.doi.org/10.3390/su12020712]
15. Rubin-Vaughan, A.; Pepler, D.; Brown, S.; Craig, W. Quest for the Golden Rule: An effective social skills promotion and bullying prevention program. Comput. Educ.; 2011; 56, pp. 166-175. [DOI: https://dx.doi.org/10.1016/j.compedu.2010.08.009]
16. Callaghan, M.J.; McCusker, K.; Losada, J.L.; Harkin, J.; Wilson, S. Using Game-Based Learning in Virtual Worlds to Teach Electronic and Electrical Engineering. IEEE Trans. Ind. Inform.; 2013; 9, pp. 575-584. [DOI: https://dx.doi.org/10.1109/TII.2012.2221133]
17. Cura-González, I.; López-Rodríguez, J.A.; Sanz-Cuesta, T.; Rodríguez-Barrientos, R.; Martín-Fernández, J.; Ariza-Cardiel, G.; Polentinos-Castro, E.; Román-Crespo, B.; Escortell-Mayor, E.; Rico-Blázquez, M. et al. Effectiveness of a strategy that uses educational games to implement clinical practice guidelines among Spanish residents of family and community medicine (e-EDUCAGUIA project): A clinical trial by clusters. Implement Sci.; 2015; 11, 71. [DOI: https://dx.doi.org/10.1186/s13012-016-0425-3] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27189180]
18. Lemcke, S.; Haedge, K.; Zender, R.; Lucke, U. RouteMe: A multilevel pervasive game on mobile ad hoc routing. Pers. Ubiquitous Comput.; 2015; 19, pp. 537-549. [DOI: https://dx.doi.org/10.1007/s00779-015-0843-2]
19. Easterday, M.W.; Aleven, V.; Scheines, R.; Carver, S.M. Using Tutors to Improve Educational Games: A Cognitive Game for Policy Argument. J. Learn. Sci.; 2016; 26, pp. 226-276. [DOI: https://dx.doi.org/10.1080/10508406.2016.1269287]
20. Hernández-Lara, A.B.; Serradell-López, E.; Fitó-Bertran, À. Students’ perception of the impact of competences on learning: An analysis with business simulations. Comput. Hum. Behav.; 2019; 101, pp. 311-319. [DOI: https://dx.doi.org/10.1016/j.chb.2019.07.023]
21. Kenigsberg, P.-A.; Aquino, J.-P.; Bérard, A.; Gzil, F.; Andrieu, S.; Banerjee, S.; Brémond, F.; Buee, L.; Cohen-Mansfield, J.; Mangialasche, F. et al. Dementia beyond 2025: Knowledge and uncertainties. Dementia; 2015; 15, pp. 6-21. [DOI: https://dx.doi.org/10.1177/1471301215574785]
22. Newbery-Jones, C. Ethical experiments with the D-pad: Exploring the potential of video games as a phenomenological tool for experiential legal education. Law Teach.; 2016; 50, pp. 61-81. [DOI: https://dx.doi.org/10.1080/03069400.2016.1146465]
23. Cárdenas-Robledo, L.A.; Peña-Ayala, A. Ubiquitous learning: A systematic review. Telemat. Inform.; 2018; 35, pp. 1097-1132. [DOI: https://dx.doi.org/10.1016/j.tele.2018.01.009]
24. Wisittanawat, P.; Gresalfi, M.S. The “tricky business” of genre blending: Tensions between frames of school mathematics and video game play. J. Learn. Sci.; 2020; 30, pp. 240-278. [DOI: https://dx.doi.org/10.1080/10508406.2020.1817747]
25. Clark, D.; Tanner-Smith, E.; Hostetler, A.; Fradkin, A.; Polikov, V. Substantial Integration of Typical Educational Games Into Extended Curricula. J. Learn. Sci.; 2017; 27, pp. 265-318. [DOI: https://dx.doi.org/10.1080/10508406.2017.1333431]
26. Bekebrede, G.; Warmelink, H.; Mayer, I. Reviewing the need for gaming in education to accommodate the net generation. Comput. Educ.; 2011; 57, pp. 1521-1529. [DOI: https://dx.doi.org/10.1016/j.compedu.2011.02.010]
27. Simões, J.; Redondo, R.D.; Vilas, A.F. A social gamification framework for a K-6 learning platform. Comput. Hum. Behav.; 2013; 29, pp. 345-353. [DOI: https://dx.doi.org/10.1016/j.chb.2012.06.007]
28. Martín-SanJosé, J.-F.; Juan, M.-C.; Gil-Gómez, J.-A.; Rando, N. Flexible learning itinerary vs. linear learning itinerary. Sci. Comput. Program.; 2014; 88, pp. 3-21. [DOI: https://dx.doi.org/10.1016/j.scico.2013.12.009]
29. Nyembezi, N.; Cishe, N.; Mantlana, D. Teachers’ Adoption of Information and Communication Technology in Foundation Phase Teaching: A Study of Selected Schools in the Eastern Cape in South Africa. Int. J. Educ. Sci.; 2016; 12, pp. 57-65. [DOI: https://dx.doi.org/10.1080/09751122.2016.11890413]
30. Karaaslan, H.; Kilic, N.; Yalçin, G.G.; Gullu, A. Students’ Reflections on Vocabulary Learning through Synchronous and Asynchronous Games and Activities. Turk. Online J. Distance Educ.; 2018; 19, pp. 53-70. [DOI: https://dx.doi.org/10.17718/tojde.444640]
31. Alshammari, M.T. Design and Learning Effectiveness Evaluation of Gamification in e-Learning Systems. Int. J. Adv. Comput. Sci. Appl.; 2019; 10, [DOI: https://dx.doi.org/10.14569/IJACSA.2019.0100926]
32. Wautelet, Y.; Kolp, M. e-SPM: An online software project management game. Int. J. Eng. Educ.; 2012; 28, 11.
33. De-Marcos, L.; García-Cabo, A.; López, E.G. Towards the social gamification of e-learning: A practical experiment. Int. J. Eng. Educ.; 2017; 33, pp. 66-73.
34. Romero-Hernandez, A.; Riojo, M.G.; Diaz-Faes-Perez, C.; Manero-Iglesias, B. The Courtesy of Spain: Theater for the New Generations. IEEE Rev. Iberoam. Tecnol. Aprendiz.; 2018; 13, pp. 102-110. [DOI: https://dx.doi.org/10.1109/RITA.2018.2862738]
35. Manzano-León, A.; Camacho-Lazarraga, P.; Guerrero, M.; Guerrero-Puerta, L.; Aguilar-Parra, J.; Trigueros, R.; Alias, A. Between Level Up and Game Over: A Systematic Literature Review of Gamification in Education. Sustainability; 2021; 13, 2247. [DOI: https://dx.doi.org/10.3390/su13042247]
36. Klock, A.C.T.; Gasparini, I.; Pimenta, M.S. User-Centered Gamification for E-Learning Systems: A Quantitative and Qualitative Analysis of its Application. Interact. Comput.; 2019; 31, pp. 425-445. [DOI: https://dx.doi.org/10.1093/iwc/iwz028]
37. Paravizo, E.; Chaim, O.C.; Braatz, D.; Muschard, B.; Rozenfeld, H. Exploring gamification to support manufacturing education on industry 4.0 as an enabler for innovation and sustainability. Procedia Manuf.; 2018; 21, pp. 438-445. [DOI: https://dx.doi.org/10.1016/j.promfg.2018.02.142]
38. Torres-Toukoumidis, A.; Ramírez-Montoya, M.S.; Romero-Rodríguez, L.M. Assessment and evaluation of games-based learning (gbl) in e-learning contexts. Educ. Knowl. Soc.; 2019; 19, pp. 109-128. [DOI: https://dx.doi.org/10.14201/eks2018194109128]
39. Ebner, M.; Holzinger, A. Successful implementation of user-centered game based learning in higher education: An example from civil engineering. Comput. Educ.; 2007; 49, pp. 873-890. [DOI: https://dx.doi.org/10.1016/j.compedu.2005.11.026]
40. Bieliková, M.; Divéky, M.; Jurnečka, P.; Kajan, R.; Omelina, L. Automatic generation of adaptive, educational and multimedia computer games. Signal Image Video Process.; 2008; 2, pp. 371-384. [DOI: https://dx.doi.org/10.1007/s11760-008-0086-z]
41. Wang, T.-H. Web-based quiz-game-like formative assessment: Development and evaluation. Comput. Educ.; 2008; 51, pp. 1247-1263. [DOI: https://dx.doi.org/10.1016/j.compedu.2007.11.011]
42. Miyagawa, T.; Yamagishi, Y.; Mizuno, S. Walk-rally support system using two-dimensional codes and mobile phones. Interact. Learn. Environ.; 2013; 21, pp. 199-210. [DOI: https://dx.doi.org/10.1080/10494820.2012.705857]
43. DelSignore, L.A.; Wolbrink, T.A.; Zurakowski, D.; Burns, J.P.; Hoy, M.; Larsen, D.P.; Johnson, H.; Getchius, T. Test-Enhanced E-Learning Strategies in Postgraduate Medical Education: A Randomized Cohort Study. J. Med. Internet Res.; 2016; 18, e299. [DOI: https://dx.doi.org/10.2196/jmir.6199] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27872034]
44. Khenissi, M.A.; Essalmi, F.; Jemni, M.; Chang, T.-W.; Chen, N.-S. Unobtrusive monitoring of learners’ interactions with educational games for measuring their working memory capacity. Br. J. Educ. Technol.; 2017; 48, pp. 224-245. [DOI: https://dx.doi.org/10.1111/bjet.12445]
45. Stevenson, C.D.; Porter, K.; Stevenson, K.T. Effects of a Game-Based e-Learning Module on Undergraduate Food Science Students’ Planned Behaviors Concerning Good Manufacturing Practices. J. Food Sci. Educ.; 2018; 17, pp. 111-117. [DOI: https://dx.doi.org/10.1111/1541-4329.12148]
46. Tolks, D.; Lampert, C.; Dadaczynski, K.; Maslon, E.; Paulus, P.; Sailer, M. Spielerische Ansätze in Prävention und Gesundheitsförderung: Serious Games und Gamification. Bundesgesundheitsbl Gesundh. Gesundh.; 2020; 63, pp. 698-707. [DOI: https://dx.doi.org/10.1007/s00103-020-03156-1] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32451596]
47. Alshammari, M.T. Evaluation of Gamification in E-Learning Systems for Elementary School Students. TEM J.; 2020; 9, pp. 806-813. [DOI: https://dx.doi.org/10.18421/TEM92-51]
48. Santos-Villalba, M.J.; Leiva Olivencia, J.J.; Navas-Parejo, M.R.; Benítez-Márquez, M.D. Higher Education Students’ Assessments towards Gamification and Sustainability: A Case Study. Sustainability; 2020; 12, 8513. [DOI: https://dx.doi.org/10.3390/su12208513]
49. Zhong, R.Y.; Huang, G.Q. RFID-enabled learning supply chain: A smart pedagogical environment for TELD. Int. J. Eng. Educ.; 2014; 30, pp. 471-482.
50. Boada, I.; Benitez, A.R.; Garcia-Gonzalez, J.M.; Olivet, J.; Carreras, V.; Sbert, M. Using a serious game to complement CPR instruction in a nurse faculty. Comput. Methods Programs Biomed.; 2015; 122, pp. 282-291. [DOI: https://dx.doi.org/10.1016/j.cmpb.2015.08.006]
51. Akaslan, D.; Law, E.L.-C. A model for flipping electrical engineering with e-learning using a multidimensional approach. Turk. J. Electr. Eng. Comput. Sci.; 2016; 24, pp. 3419-3431. [DOI: https://dx.doi.org/10.3906/elk-1411-144]
52. Johnsen, H.M.; Fossum, M.; Vivekananda-Schmidt, P.; Fruhling, A.; Slettebø, A. Nursing students’ perceptions of a video-based serious game’s educational value: A pilot study. Nurse Educ. Today; 2018; 62, pp. 62-68. [DOI: https://dx.doi.org/10.1016/j.nedt.2017.12.022] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29306100]
53. Chan, C.-S.; Chan, Y.-H.; Fong, T.H.A. Game-based e-learning for urban tourism education through an online scenario game. Int. Res. Geogr. Environ. Educ.; 2019; 29, pp. 283-300. [DOI: https://dx.doi.org/10.1080/10382046.2019.1698834]
54. Balogun, W.G. Using Electronic Tools and Resources to Meet the Challenges of Anatomy Education in Sub-Saharan Africa. Anat. Sci. Educ.; 2018; 12, pp. 97-104. [DOI: https://dx.doi.org/10.1002/ase.1831] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30255559]
55. Doney, I. Research into effective gamification features to inform e-learning design. Res. Learn. Technol.; 2019; 27, [DOI: https://dx.doi.org/10.25304/rlt.v27.2093]
56. Bernik, A.; VusiÄ, D.; Milkovic, M. Evaluation of Gender Differences Based on Knowledge Adaptation in the Field of Gamification and Computer Science. IJET; 2019; 14, pp. 220-228. [DOI: https://dx.doi.org/10.3991/ijet.v14i08.9847]
57. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G. The PRISMA Group Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med.; 2009; 6, e1000097. [DOI: https://dx.doi.org/10.1371/journal.pmed.1000097]
58. Vieira, P.S. Supply Chain Risk Management in Healthcare: Identification, Assessment and Mitigation; Programa de Pós-Graduação em Engenharia de Produção e Sistemas, Federal Center for Technological Education of Rio de Janeiro: Rio de Janeiro, Brazil, 2019.
59. Santos, A.C.S.G.; da Cunha Reis, A.; de Souza, C.G.; dos Santos, I.L.; Ferreira, L.A.F. The first evidence about conceptual vs. analytical lean healthcare research studies. J. Health Organ. Manag.; 2020; 34, pp. 789-806. [DOI: https://dx.doi.org/10.1108/JHOM-01-2020-0021]
60. Da Cunha Reis, A.; Oliveira, R.P.; de Carvalho Castro, A. Logística hospitalar: Uma síntese do estado da arte. 2018, 15, 205–231. Revista Gestão Desenvolvimento; 2018; 15, pp. 205-231. [DOI: https://dx.doi.org/10.25112/rgd.v15i1.1276]
61. Silva, M.M.; De Gusmão, A.P.; De Andrade, C.T.A.; Silva, W. The integration of VFT and FITradeoff multicriteria method for the selection of WCM projects. Proceedings of the 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC); Bari, Italy, 6–9 October 2019; pp. 1513-1517. [DOI: https://dx.doi.org/10.1109/smc.2019.8914452]
62. Arsovski, S.; Đokić, I.; Pešić-Đokić, S. Quality in world class manufacturing. Int. J. Qual. Res.; 2011; 5, pp. 309-316.
63. Svrzić, U.; Danon, G. Solving problems in wood parquet production by using ‘7 steps’ of World Class Manufacturing Methodology (WCM). J. Appl. Eng. Sci.; 2014; 12, pp. 113-120. [DOI: https://dx.doi.org/10.5937/jaes12-5532]
64. Novicka, A.; Papcun, P.; Zolotova, I. Mapping of machine faults using tools of World Class Manufacturing. Proceedings of the 2016 IEEE 14th International Symposium on Applied Machine Intelligence and Informatics (SAMI); Herlany, Slovakia, 21–23 January 2016; pp. 223-227. [DOI: https://dx.doi.org/10.1109/sami.2016.7423011]
65. Vyt, A. Smartly using PDCA in quality of distance teaching. The Envisioning Report for Empowering Universities; 3rd ed. European Association of Distance Teaching Universities: Maastricht, The Netherlands, 2019.
66. Hasan, Z.; Mallik, A.; Tsou, J.-C. Learning method design for engineering students to be prepared for Industry 4.0: A Kaizen approach. High. Educ. Ski. Work. Learn.; 2020; 11, pp. 182-198. [DOI: https://dx.doi.org/10.1108/HESWBL-07-2019-0098]
67. Pastrana, R.; Manabat, A. An Outcomes-Based Education (Obe) Approach & Typology-Based Quality Assurance (Qa) System: A Proposed Framework And Transition Strategy For Philippine Higher Education Institution’s (HEI) Shift Towards International Standards. Balk. Reg. Conf. Eng. Bus. Educ.; 2014; 1, pp. 639-644. [DOI: https://dx.doi.org/10.2478/cplbu-2014-0115]
68. Chen, Y.; Zheng, J.; Wu, D.; Zhang, Y.; Lin, Y. Application of the PDCA cycle for standardized nursing management in a COVID-19 intensive care unit. Ann. Palliat. Med.; 2020; 9, pp. 1198-1205. [DOI: https://dx.doi.org/10.21037/apm-20-1084]
69. Pacheco, A.P.R.; Salles, B.W.; Garcia, M.A.; Possamai, A. O Ciclo PDCA na Gestão do Conhecimento: Uma Abordagem Sistêmica; Universidade Federal de Santa Catarina: Florianópolis, Brazil, 2012; Volume 2.
70. Martinengo, L.; Yeo, N.J.Y.; Tang, Z.Q.; Markandran, K.D.; Kyaw, B.M.; Car, L.T. Digital Education for the Management of Chronic Wounds in Health Care Professionals: Protocol for a Systematic Review by the Digital Health Education Collaboration. JMIR Res. Protoc.; 2019; 8, e12488. [DOI: https://dx.doi.org/10.2196/12488] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30907743]
71. Gauthier, A.; Jenkinson, J. Designing productively negative experiences with serious game mechanics: Qualitative analysis of game-play and game design in a randomized trial. Comput. Educ.; 2018; 127, pp. 66-89. [DOI: https://dx.doi.org/10.1016/j.compedu.2018.08.017]
72. Pérez-Manzano, A.; Almela-Baeza, J. Gamification and transmedia for scientific promotion and for encouraging scientific careers in adolescents. Comunicar; 2018; 26, pp. 93-103. [DOI: https://dx.doi.org/10.3916/C55-2018-09]
73. Olszewski, A.E.; Daniel, D.A.; Stein, D.R.; McCulloch, M.; Su, S.W.; Hames, D.L.; Wolbrink, T.A. Teaching Pediatric Peritoneal Dialysis Globally through Virtual Simulation. Clin. J. Am. Soc. Nephrol.; 2018; 13, pp. 900-906. [DOI: https://dx.doi.org/10.2215/CJN.10460917] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29720505]
74. Jackson, L.C.; O’Mara, J.; Moss, J.; Jackson, A.C. A Critical Review of the Effectiveness of Narrative-Driven Digital Educational Games. Int. J. Game-Based Learn.; 2018; 8, pp. 32-49. [DOI: https://dx.doi.org/10.4018/IJGBL.2018100103]
75. Ge, Z.-G. The impact of a forfeit-or-prize gamified teaching on e-learners’ learning performance. Comput. Educ.; 2018; 126, pp. 143-152. [DOI: https://dx.doi.org/10.1016/j.compedu.2018.07.009]
76. Ortiz Benavides, F.L.; Piña Lopez, C.E. Estrategia tecno-didáctica para la solución de problemas de genética en estudiantes de educación a distancia. REurEDC; 2018; 15, pp. 1-19. [DOI: https://dx.doi.org/10.25267/Rev_Eureka_ensen_divulg_cienc.2018.v15.i2.2301]
77. Kooiman, B.J.; Sheehan, D.P.; Wesolek, M.; Retegui, E. Moving online physical education from oxymoron to efficacy. Sport Educ. Soc.; 2017; 22, pp. 230-246. [DOI: https://dx.doi.org/10.1080/13573322.2015.1015978]
78. Torres-Toukoumidis, A.; Romero-Rodríguez, L.M.; Perez-Rodriguez, A.; Björk, S. Modelo Teórico Integrado de Gamificación en Ambientes E-Learning (E-MIGA). 2017, 29, 129–145. Revista Complutense Educación; 2017; 29, pp. 129-145. [DOI: https://dx.doi.org/10.5209/RCED.52117]
79. Sardi, L.; Idri, A.; Fernández-Alemán, J.L. A systematic review of gamification in e-Health. J. Biomed. Inform.; 2017; 71, pp. 31-48. [DOI: https://dx.doi.org/10.1016/j.jbi.2017.05.011] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28536062]
80. Lin, H.-C.; Chiu, Y.-H.; Chen, Y.J.; Wuang, Y.-P.; Chen, C.-P.; Wang, C.-C.; Huang, C.-L.; Wu, T.-M.; Ho, W.-H. Continued use of an interactive computer game-based visual perception learning system in children with developmental delay. Int. J. Med. Inform.; 2017; 107, pp. 76-87. [DOI: https://dx.doi.org/10.1016/j.ijmedinf.2017.09.003] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29029695]
81. De-Marcos, L.; García-López, E.; García-Cabot, A.; Medina-Merodio, J.-A.; Domínguez, A.; Martínez-Herráiz, J.-J.; Diez-Folledo, T. Social network analysis of a gamified e-learning course: Small-world phenomenon and network metrics as predictors of academic performance. Comput. Hum. Behav.; 2016; 60, pp. 312-321. [DOI: https://dx.doi.org/10.1016/j.chb.2016.02.052]
82. Fernández, C.; Esteban, G.; Conde, M.A.; García, F. Improving motivation in a haptic teaching/learning framework. Int. J. Eng. Educ.; 2016; 32, pp. 553-562.
83. Li, T.M.H.; Chau, M.; Sung, W.-K.; Lee, A.J.; Wong, P.W.C.; Yip, P.S.F. Design and evaluation of a Facebook game for self-directed e-learning. Knowl. Manag. E-Learn. Int. J.; 2016; 8, pp. 464-480. [DOI: https://dx.doi.org/10.34105/j.kmel.2016.08.029]
84. Stubbé, H.; Badri, A.; Telford, R.; Van Der Hulst, A.; Van Joolingen, W. E-Learning Sudan, Formal Learning for Out-of-School Children. EJEL; 2016; 14, pp. 136-149.
85. Tenório, T.; Silva, A.R.; Tenório, A. A influência da gamificação na Educação a Distância com base nas percepções de pesquisadores brasileiros. Rev. EDaPECI; 2016; 16, pp. 320-335. [DOI: https://dx.doi.org/10.29276/redapeci.2016.16.24554.320-335]
86. Wang, C.-S.; Li, Y.-C.; Tzeng, Y.-R. How to replicate the cognitive process in computer game-based learning units. Inf. Technol. People; 2015; 28, pp. 327-343. [DOI: https://dx.doi.org/10.1108/ITP-03-2013-0053]
87. Luthon, F.; Larroque, B. LaboREM—A Remote Laboratory for Game-Like Training in Electronics. IEEE Trans. Learn. Technol.; 2014; 8, pp. 311-321. [DOI: https://dx.doi.org/10.1109/TLT.2014.2386337]
88. Labus, A.; Despotović-Zrakić, M.; Radenkovic, M.; Bogdanović, Z. Enhancing formal e-learning with edutainment on social networks. J. Comput. Assist. Learn.; 2015; 31, pp. 592-605. [DOI: https://dx.doi.org/10.1111/jcal.12108]
89. Olsson, M.; Mozelius, P.; Collin, J. Visualisation and Gamification of e-Learning and Programming Education. EJEL; 2015; 13, pp. 441-454.
90. Faíña, A.; López-Rodríguez, J.; Varela-Candamio, L. Using game theory in computer engineering education though case stady methodology: Kodak vs. Polariod in the market for instant cameras. Int. J. Eng. Educ.; 2014; 30, pp. 252-262.
91. Lin, C.F.; Yeh, Y.-C.; Hung, Y.H.; Chang, R.-I. Data mining for providing a personalized learning path in creativity: An application of decision trees. Comput. Educ.; 2013; 68, pp. 199-210. [DOI: https://dx.doi.org/10.1016/j.compedu.2013.05.009]
92. Jackson, G.T.; McNamara, D.S. Motivation and performance in a game-based intelligent tutoring system. J. Educ. Psychol.; 2013; 105, pp. 1036-1049. [DOI: https://dx.doi.org/10.1037/a0032580]
93. Domínguez, A.; Saenz-De-Navarrete, J.; De-Marcos, L.; Fernández-Sanz, L.; Pagés, C.; Martínez-Herráiz, J.-J. Gamifying learning experiences: Practical implications and outcomes. Comput. Educ.; 2013; 63, pp. 380-392. [DOI: https://dx.doi.org/10.1016/j.compedu.2012.12.020]
94. Chen, H.R.; Huang, J.W. Learning effects of RFID-based game-guided learning in libraries. Int. J. Mob. Commun.; 2012; 10, pp. 351-365. [DOI: https://dx.doi.org/10.1504/IJMC.2012.048135]
95. Sancho, P.; Torrente, J.; Fernández-Manjón, B. MareMonstrum: A Contribution to Empirical Research about How the Use of MUVEs May Improve Students’ Motivation. J. Univ. Comp. Sci.; 2012; 18, pp. 2576-2598.
96. Kanthan, R.; Senger, J.-L. The impact of specially designed digital games-based learning in undergraduate pathology and medical education. Arch. Pathol. Lab. Med.; 2011; 135, pp. 135-142. [DOI: https://dx.doi.org/10.5858/2009-0698-OAR1.1]
97. Brom, C.; Šisler, V.; Slavík, R. Implementing digital game-based learning in schools: Augmented learning environment of ‘Europe 2045’. Multimed. Syst.; 2010; 16, pp. 23-41. [DOI: https://dx.doi.org/10.1007/s00530-009-0174-0]
98. Regueras, L.M.; Verdu, E.; Munoz, M.F.; Perez, M.A.; De Castro, J.P.; Verdu, M.J. Effects of Competitive E-Learning Tools on Higher Education Students: A Case Study. IEEE Trans. Educ.; 2009; 52, pp. 279-285. [DOI: https://dx.doi.org/10.1109/TE.2008.928198]
99. Vassileva, J. Toward Social Learning Environments. IEEE Trans. Learn. Technol.; 2008; 1, pp. 199-214. [DOI: https://dx.doi.org/10.1109/TLT.2009.4]
100. Munz, U.; Schumm, P.; Wiesebrock, A.; Allgöwer, F. Motivation and Learning Progress Through Educational Games. IEEE Trans. Ind. Electron.; 2007; 54, pp. 3141-3144. [DOI: https://dx.doi.org/10.1109/TIE.2007.907030]
101. Martens, R.; Gulikers, J.; Bastiaens, T. The impact of intrinsic motivation on e-learning in authentic computer tasks. J. Comput. Assist. Learn.; 2004; 20, pp. 368-376. [DOI: https://dx.doi.org/10.1111/j.1365-2729.2004.00096.x]
102. Mason, J.; Loader, K. Using a Serious Game to Train Violence Risk Assessment and Management Skills. Simul. Gaming; 2019; 50, pp. 124-135. [DOI: https://dx.doi.org/10.1177/1046878119837314]
103. North-Samardzic, A.; De Witt, M. Designing a Human Resource Management Simulation to Engage Students. J. Manag. Educ.; 2019; 43, pp. 359-395. [DOI: https://dx.doi.org/10.1177/1052562919841411]
104. Bond, S.E.; Crowther, S.P.; Adhikari, S.; Chubaty, A.J.; Yu, P.; Borchard, J.P.; Boutlis, C.S.; Yeo, W.W.; Miyakis, S.; Castro-Sánchez, E. et al. Design and Implementation of a Novel Web-Based E-Learning Tool for Education of Health Professionals on the Antibiotic Vancomycin. J. Med. Internet Res.; 2017; 19, e93. [DOI: https://dx.doi.org/10.2196/jmir.6971]
105. Callaghan, N. Investigating the role of Minecraft in educational learning environments. Educ. Media Int.; 2016; 53, pp. 244-260. [DOI: https://dx.doi.org/10.1080/09523987.2016.1254877]
106. Ibáñez, M.-B.; Di-Serio, A.; Delgado-Kloos, C. Gamification for Engaging Computer Science Students in Learning Activities: A Case Study. IEEE Trans. Learn. Technol.; 2014; 7, pp. 291-301. [DOI: https://dx.doi.org/10.1109/TLT.2014.2329293]
107. Creutzfeldt, J.; Hedman, L.; Felländer-Tsai, L. Effects of pre-training using serious game technology on CPR performance—An exploratory quasi-experimental transfer study. Scand. J. Trauma Resusc. Emerg. Med.; 2012; 20, 79. [DOI: https://dx.doi.org/10.1186/1757-7241-20-79]
108. Creutzfeldt, J.; Hedman, L.; Heinrichs, L.; Youngblood, P.; Felländer-Tsai, L. Cardiopulmonary Resuscitation Training in High School Using Avatars in Virtual Worlds: An International Feasibility Study. J. Med. Internet Res.; 2013; 15, e9. [DOI: https://dx.doi.org/10.2196/jmir.1715] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23318253]
109. Moreno-Ger, P.; Torrente, J.; Bustamante, J.; Fernández-Galaz, C.; Fernández-Manjón, B.; Comas-Rengifo, M.D. Application of a low-cost web-based simulation to improve students’ practical skills in medical education. Int. J. Med. Inform.; 2010; 79, pp. 459-467. [DOI: https://dx.doi.org/10.1016/j.ijmedinf.2010.01.017] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/20347383]
110. Franceschi, K.; Lee, R.M.; Zanakis, S.H.; Hinds, D. Engaging Group E-Learning in Virtual Worlds. J. Manag. Inf. Syst.; 2009; 26, pp. 73-100. [DOI: https://dx.doi.org/10.2753/MIS0742-1222260104]
111. Choudhury, B.; Gouldsborough, I.; Gabriel, S. Use of interactive sessions and e-learning in teaching anatomy to first-year optometry students. Anat. Sci. Educ.; 2009; 3, pp. 39-45. [DOI: https://dx.doi.org/10.1002/ase.123]
112. Pistoljevic, N.; Hulusic, V. Educational e-book for children with and without developmental disorders. J. Comput. Educ.; 2019; 6, pp. 117-141. [DOI: https://dx.doi.org/10.1007/s40692-018-0126-9]
113. Khenissi, M.A.; Essalmi, F.; Jemni, M.; Graf, S.; Chen, N.-S. Relationship between learning styles and genres of games. Comput. Educ.; 2016; 101, pp. 1-14. [DOI: https://dx.doi.org/10.1016/j.compedu.2016.05.005]
114. Fitó-Bertran, À.; Hernández-Lara, A.B.; Serradell-Lopez, E. The effect of competences on learning results an educational experience with a business simulator. Comput. Hum. Behav.; 2015; 51, pp. 910-914. [DOI: https://dx.doi.org/10.1016/j.chb.2014.11.003]
115. Zafar, S.; Safdar, S.; Zafar, A.N. Evaluation of use of e-Learning in undergraduate radiology education: A review. Eur. J. Radiol.; 2014; 83, pp. 2277-2287. [DOI: https://dx.doi.org/10.1016/j.ejrad.2014.08.017] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25242658]
116. Martín-SanJosé, J.F.M.S.; Juan, M.-C.; Seguí, I.; García-García, I. The effects of computer-based games and collaboration in large groups vs. collaboration in pairs or traditional methods. Comput. Educ.; 2015; 87, pp. 42-54. [DOI: https://dx.doi.org/10.1016/j.compedu.2015.03.018]
117. Zepeda-Hernandez, S.; Abascal-Mena, R.; Lopez-Ornelas, E. Integration of gamification and active learning in the classroom. REVISTA RA XIMHAI; 2016; 12, pp. 315-325.
118. Tan, J.W.; Zary, N. Diagnostic Markers of User Experience, Play, and Learning for Digital Serious Games: A Conceptual Framework Study. JMIR Serious Games; 2019; 7, e14620. [DOI: https://dx.doi.org/10.2196/14620]
119. Hernández-Lara, A.B.; Serradell-Lopez, E.; Fitó Bertran, M.À. Do business games foster skills? A cross-cultural study from learners’ views. Intang. Cap.; 2018; 14, pp. 315-331. [DOI: https://dx.doi.org/10.3926/ic.1066]
120. Marchetti, E.; Valente, A. Interactivity and multimodality in language learning: The untapped potential of audiobooks. Univers. Access Inf. Soc.; 2018; 17, pp. 257-274. [DOI: https://dx.doi.org/10.1007/s10209-017-0549-5]
121. Slootmaker, A. Developing scenario-based serious games for complex cognitive skills acquisition: Design, development and evaluation of the EMERGO platform. J. Univ. Comp. Sci.; 2014; 20, pp. 561-582.
122. Hwang, G.J.; Sung, H.Y.; Hung, C.M.; Huang, I. A Learning Style Perspective to Investigate the Necessity of Developing Adaptive Learning Systems. Educ. Tech. Soc.; 2013; 16, pp. 188-197.
123. Siddiqui, A.; Khan, M.; Akhtar, S. Supply chain simulator: A scenario-based educational tool to enhance student learning. Comput. Educ.; 2008; 51, pp. 252-261. [DOI: https://dx.doi.org/10.1016/j.compedu.2007.05.008]
124. Wall, J.; Ahmed, V. Use of a simulation game in delivering blended lifelong learning in the construction industry—Opportunities and Challenges. Comput. Educ.; 2008; 50, pp. 1383-1393. [DOI: https://dx.doi.org/10.1016/j.compedu.2006.12.012]
125. Slimani, A.; Elouaai, F.; Elaachak, L.; Yedri, O.B.; Bouhorma, M.; Sbert, M. Learning Analytics Through Serious Games: Data Mining Algorithms for Performance Measurement and Improvement Purposes. Int. J. Emerg. Technol. Learn.; 2018; 13, pp. 46-64. [DOI: https://dx.doi.org/10.3991/ijet.v13i01.7518]
126. Fu, F.-L.; Su, R.-C.; Yu, S.-C. EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Comput. Educ.; 2009; 52, pp. 101-112. [DOI: https://dx.doi.org/10.1016/j.compedu.2008.07.004]
127. Retalis, S. Creating Adaptive e-Learning Board Games for School Settings Using the ELG Environment. J. Univ. Comp. Sci.; 2008; 14, pp. 2897-2908.
128. Bryceson, K.P. The development of VAG—A 3D virtual agribusiness environment and strategy game. Electron. Commer. Res.; 2009; 9, pp. 27-47. [DOI: https://dx.doi.org/10.1007/s10660-009-9026-4]
129. Schmidt, M.W.; Friedrich, M.; Kowalewski, K.-F.; De La Garza, J.; Bruckner, T.; Müller-Stich, B.-P.; Nickel, F. Learning from the surgeon’s real perspective—First-person view versus laparoscopic view in e-learning for training of surgical skills? Study protocol for a randomized controlled trial. Int. J. Surg. Protoc.; 2017; 3, pp. 7-13. [DOI: https://dx.doi.org/10.1016/j.isjp.2017.01.001]
130. Atanasijević-Kunc, M.; Logar, V.; Karba, R.; Papic, M.; Kos, A. Remote Multivariable Control Design Using a Competition Game. IEEE Trans. Educ.; 2011; 54, pp. 97-103. [DOI: https://dx.doi.org/10.1109/TE.2010.2046489]
131. Fernandez, J.; Marin, R.; Wirz, R. Online Competitions: An Open Space to Improve the Learning Process. IEEE Trans. Ind. Electron.; 2007; 54, pp. 3086-3093. [DOI: https://dx.doi.org/10.1109/TIE.2007.907013]
132. Prochazkova, K.; Novotný, P.; Hancarova, M.; Prchalova, D.; Sedlacek, Z. Teaching a difficult topic using a problem-based concept resembling a computer game: Development and evaluation of an e-learning application for medical molecular genetics. BMC Med. Educ.; 2019; 19, pp. 1-8. [DOI: https://dx.doi.org/10.1186/s12909-019-1817-2]
133. Schwade, F.; Schubert, P. The ERP Challenge. Int. J. Hum. Cap. Inf. Technol. Prof.; 2018; 9, pp. 53-69. [DOI: https://dx.doi.org/10.4018/IJHCITP.2018010104]
134. Spiridonova, L.I. Characteristics and advantages of using an e-learning system for training of pre-school and primary school teachers. J. e-Learn. Knowl. Soc.; 2016; 12, [DOI: https://dx.doi.org/10.20368/1971-8829/1028]
135. Rasila, A.; Malinen, J.; Tiitu, H. On automatic assessment and conceptual understanding. Teach. Math. Appl. Int. J. IMA; 2015; 34, pp. 149-159. [DOI: https://dx.doi.org/10.1093/teamat/hrv013]
136. Duque, G.; Demontiero, O.; Whereat, S.; Gunawardene, P.; Leung, O.; Webster, P.; Sardinha, L.; Boersma, D.; Sharma, A. Evaluation of a blended learning model in geriatric medicine: A successful learning experience for medical students. Australas. J. Ageing; 2013; 32, pp. 103-109. [DOI: https://dx.doi.org/10.1111/j.1741-6612.2012.00620.x]
137. Magro, A.; Swarz, J.; Ousley, A. CancerSPACE: An Interactive E-learning Tool Aimed to Improve Cancer Screening Rates. J. Comput. Commun.; 2010; 15, pp. 482-499. [DOI: https://dx.doi.org/10.1111/j.1083-6101.2010.01512.x]
138. Foss, B.A.; Eikaas, T.I. Game Play in Engineering EducationÐ Concept and Experimental Results. Int. J. Eng. Educ.; 2006; 22, 10.
139. Mabkhot, M.M.; Ferreira, P.; Maffei, A.; Podržaj, P.; Mądziel, M.; Antonelli, D.; Lanzetta, M.; Barata, J.; Boffa, E.; Finžgar, M. et al. Mapping Industry 4.0 Enabling Technologies into United Nations Sustainability Development Goals. Sustainability; 2021; 13, 2560. [DOI: https://dx.doi.org/10.3390/su13052560]
140. Senna, P.; Pinha, D.; Ahluwalia, R.; Guimarães, J.C.; Severo, E.; Reis, A.D.C. A three-stage stochastic optimization model for the Brazilian biodiesel supply chain. Production; 2016; 26, pp. 501-515. [DOI: https://dx.doi.org/10.1590/0103-6513.200015]
141. Dos Santos, A.L.T.; Reis, A.D.C. Tendências Teóricas do Gerenciamento de Riscos em Compras Públicas Sustentáveis: Uma Análise Bibliométrica A Partir das Bases Scopus e Web of Science. Rev. Gestão Soc. Ambient.; 2021; 15, e2733. [DOI: https://dx.doi.org/10.24857/rgsa.v15i2.2733]
142. Acioli, C.; Scavarda, A.; Reis, A. Applying Industry 4.0 Technologies in the COVID-19 Sustainable Chains. Int. J. Product. Perform. Manag.; 2021; Available online: https://www.emerald.com/insight/1741-0401.html (accessed on 14 October 2021).
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Gomes de Souza, Cristina
; Augusto da Cunha Reis; Wallice Medeiros de Souza