Exploring the effectiveness of technology-based learning on the educational outcomes of undergraduate healthcare students: an overview of systematic reviews protocol

Abstract

Introduction

Rapid technology development due to the introduction of Industrial Revolution 4.0 and Internet of Things has created a demand and gradual transition from traditional teaching and learning to technology-based learning in higher education, including healthcare education. The COVID-19 pandemic has accelerated this process, with educators now required to quickly adapt to and adopt such changes. The abundance of available systematic reviews has made the effectiveness of such approaches ambiguous especially in healthcare education. Therefore, a protocol of the overview of systematic reviews (OoSR) is planned to extrapolate the effectiveness of technology-based learning in undergraduate healthcare education.

Methods and analysis

Scopus, CINAHL, Academic Search Complete, Cochrane Library, MEDLINE and Psychology and Behavioral Sciences Collection databases were selected. Screening was conducted independently by at least two authors and the decision for inclusion was done through discussion or involvement of an arbiter against a predetermined criteria. Included articles will be evaluated for quality using A MeaSurement Tool to Assess systematic Reviews and Risk of Bias in Systematic Review tools, while primary systematic review articles will be cross-checked and reported for any overlapping using the ‘corrected covered area’ method. Only narrative synthesis will be employed according to the predefined themes into two major dimensions—theory and knowledge generation (focusing on cognitive taxonomy due to its ability to be generalised across disciplines), and clinical-based competence (focusing on psychomotor and affective taxonomies due to discipline-specific influence). The type of technology used will be identified and extracted.

Ethics and dissemination

The OoSR involves analysis of secondary data from published literature, thus ethical approval is not required. The findings will provide a valuable insight for policymakers, stakeholders, and researchers in terms of technology-based learning implementation and gaps identification. The findings will be published in several reports due to the extensiveness of the topic and will be disseminated through peer-reviewed publications and conferences.

PROSPERO registration number

CRD4202017974.

Correspondence to Dr Hafizah Abdul Hamid; [email protected]

Strengths and limitations of this study

• In light of the global COVID-19 pandemic, it is important to assess the effectiveness of technology-based learning in comparison with traditional teaching among undergraduate healthcare students.

• To the best of our knowledge, this is the first overview of systematic reviews that elucidate the effectiveness of technology-based learning across a wide range of healthcare programs.

• The overview of systematic reviews protocol describes a robust and rigorous methodology that could become a good point of reference for other researchers to adopt.

• One of the limitations of the overview of systematic reviews is the lack of granularity of information provided by the authors.

• Heterogeneity is expected from this overview of systematic reviews; therefore, grand meta-analysis is not being considered.

Introduction

Technology has evolved since the modern day of human civilisation from mechanical advancements to the electrical age, to the digital age and now, internet networking. Currently, Industrial Revolution 4.0 (IR 4.0) and Internet of Things (IoT) emphasise on internet and automation technology becoming a norm in daily human life.^1–3 Many traditional and manual technological approaches have become obsolete and archaic and are no longer relevant. As technology has become an integral part of human life, its significance is even more so apparent in the education sector. The traditional approach of teaching, such as manual face-to-face lectures, rote learning, class attending, and manual practice has transitioned towards more dynamic and interactive learning with the use of technology to facilitate learning activities.^4–7 Similarly, technology is now being incorporated into medical and nursing education for knowledge and/or clinical teaching.⁸ Taking the learning dynamics and the preferences of net-generation learners (digital native learners) into account, technology can indeed be used in teaching and learning to benefit the students and educators.^{4 9–11}

Although technology application in the education sector of developing and underdeveloped countries is still a luxury and is mainly dominated by traditional teaching approaches,^12–14 its requirement and relevance has never been more apparent than during this current COVID-19 pandemic. COVID-19 has become a global pandemic. It has interrupted daily human activities at an unimaginable scale; causing changes in behaviour, social aspects and overall lifestyle.^{15 16} With no medication and/or vaccine for COVID-19, the best approach is to rely on preventive measures such as by practising social distancing and good hygiene.^17–20 On a larger scale, some countries have implemented movement restrictions on their citizens in an effort to break the chain of infection.^20–22 Some professional representative bodies have suggested using technology-based learning (TBL) to observe social distancing to minimise such contact, especially among students.^{23 24} This measure has directly impacted the higher education teaching and learning ecosystem in universities and colleges. In turn, a new trend is created where technology-based globalised, borderless, and seamless teaching and learning may become ‘the new normal’.^25–33

TBL can provide a safe and secure environment for learning, besides reducing the burden on educators, and enabling the catering to mass groups irrespective of time zone or geographical location. However, it is important to note that the acceptance and effectiveness of TBL is still contented in comparison with traditional didactic lectures. For example, this method cannot provide a real-life context for student survival after graduation. It also requires the educators to put in more effort for material preparation. Besides, the technology might be unaffordable. Other drawbacks include incompetent educators, unstable, unreliable technology and data providers, and the digital divide.^{13 14 34–38} Additionally, medical and healthcare practitioners are now giving precedence to practice based on the best evidence, including teaching and learning aspects, known as BEME.³⁹ However, there is a dearth in studies evaluating the implementation of TBL and whether it is effective, feasible, acceptable, or potentially reasonable to complement, replace or substitute traditional teaching.

To the best of our knowledge, no overview of systematic reviews (OoSR) has been conducted or registered in the International Prospective Register of Systematic Reviews (PROSPERO) similar to this project. Two overviews were identified related to TBL,^{40 41} however there are gaps that were not addressed. First, both overviews solely focused on nursing education with findings that may be less applicable to other healthcare professional education, as nursing education emphasises on discipline-specific skill acquisition. Second, an overview by Rouleau et al ⁴⁰ focused on post-professional nursing education but pedagogy and andragogy concepts for pre-professional and post-professional education are different.⁴² Mature students are more independent, self-reliant, and have more initiative compared with younger students who require class attendance and require frequent contacts with peers and teachers. Third, the overview by Cant and Cooper⁴¹ focused only on clinical-based outcomes and not technology-based education. Fourth, the overviews were specific to certain types of technology; thus, these studies are unable to provide a comprehensive picture or cover a wide range of educational technology applications. Additionally, a scoping systematic review was found in the literature but it is only limited to the use of handheld devices among healthcare professionals.⁴³ Therefore, another overview of systematic reviews is required to comprehensively synthesise the best available evidence to determine whether or not technology-based teaching and learning is effective. That is, whether it can substitute or complement traditional teaching and learning approaches in healthcare education from a variety of aspects, such as, but not limited to, improving knowledge, critical thinking, engagement, clinical skills, and affective and mindful learning.

Methods and analysis

The OoSR is also known as an umbrella review or a systematic review of systematic reviews. It is a methodology that can be used to assist readers to work through the outcomes of related systematic reviews by collecting, disseminating, synthesising and harmonising its conclusions to guide practice.⁴⁴ One of the purpose of OoSR is to catalogue available research syntheses pertaining to the topic of technology-based education in healthcare and harmonising the evidence outcome for recommendation of practice and future research. Therefore, an OoSR protocol is required to cater to upcoming reports.

To meet the above objective, we followed the existing framework given by Cooper and Koenka.⁴⁴ The framework outlines seven steps: (1) formulating the problem, (2) searching the literature, (3) gathering information from syntheses, (4) evaluating the quality of evidence, (5) analysing and integrating the outcomes of synthesis, (6) interpreting the evidence and (7) presenting the result.

Formulating the problem

Technology-based teaching and learning is vast and complex to define. There are no standard criteria to capture the overall interpretation of technology-based approaches. TBL is defined as learning with the means of electronic technology that encompasses the internet, the intranet, satellite broadcasts, audio and video conferencing, bulletin boards, chat rooms, webcasts and compact disc read-only memory (CD-ROM).^{5 45–47} Other related terms associated with TBL are e-learning, online learning, digital learning, and web-based learning, among others.

There are numerous TBL approaches.^{8 48} TBL in healthcare education covers simple to sophisticated technology to intangible and tangible technology. For instance, simple TBL uses CD-ROM or PowerPoint presentations, while examples of sophisticated technology include virtual synchronous online classes and 3D holography for anatomy training. Furthermore, TBL can also take the form of intangible technology, such as augmented or virtual reality, while tangible technology could be presented as a simulated high-fidelity manikin. The availability of these technologies has been beneficial in advancing medical education and enhanced learners’ understanding due to its interactive nature, while at the same time presenting new challenges to educators as there is an abundance of available technologies that make it difficult to choose the best possible technology to implement in teaching and learning.^{49 50}

Therefore, it is paramount that educators are well informed on the available technologies and the proper application of specific technology in targeting specific skills in learning. Referring to a systematic review is one of the best approaches because a systematic review pools similar technologies and investigates its effectiveness. However, systematic reviews have limitations, such as being homogeneous and focusing on a specific objective. Therefore, educators must search other systematic reviews to supplement remaining objectives.

A preliminary search for this OoSR project using the keywords “systematic review”, “e-learning” and “technology” yielded thousands of articles. This result indicates that there are presumably heaps of systematic reviews available that are related to TBL. Searching and reading separate and multiple literature works would be time consuming and require a lot of effort. Educators are also busy with other responsibilities, so with the many literature works available, all the studies may not be read comprehensively (ie, teachers may only be able to read the abstract).⁵¹ This may lead to a lack of comprehensive understanding on the efficacy of the technology for education purposes. In most case scenarios, educators tend to apply certain technology based on prejudices, hunches, opinions and guesses, which assume that the technology is beneficial for teaching and learning.⁵² Such an assumption may be detrimental because the technology might be costly, effort or time consuming, not fully accepted in practice, and could have a high chance of being abandoned.^53–55 Hence, it is important to apply the concept of BEME even in TBL.⁵² Therefore, conducting an OoSR for TBL may benefit educators, as it can serve as a one-stop point of reference.

To ensure that this OoSR project comprehensively encapsulates all possible angles and aspects of TBL, a research objective was developed in accordance to the Population, Intervention/Exposure, Comparison and Outcome (PICO/PECO) concept.⁵⁶ Table 1 details out the criteria of each PICO. Developing a detailed OoSR objective is important to ensure the methodology is properly developed and explicit, and the overview is focused, manageable and systematic.⁵⁷ Therefore, the objective of this OoSR is ‘to explore the effectiveness of TBL in improving the educational outcomes of undergraduate healthcare students’.

Table 1

Research objective developed using PICO

Population	Undergraduate healthcare students
Intervention	Technology-based learning
Comparison	Non-technology-based or conventional/traditional technology
Outcome	Educational outcome—effectiveness in cognitive, psychomotor or affective domains

Searching the literature

This OoSR was carried out by conducting a search on several electronic databases subscribed by Universiti Putra Malaysia. The electronic databases were selected by discussing among the research team and finding a commonality in the electronic databases used in the topic. First, several published systematic reviews on the topic of TBL in healthcare education were explored. Then, the research team generated some keywords. The relevant keywords were initially identified from the researcher’s knowledge, selected published systematic reviews and by searching the synonyms of the terminologies using Google. Second, the pooled keywords were scrutinised and selected from the discussion among the research team. Third, the selected keywords were tested in the search engines and further customised, expanded or removed until an optimal, manageable number of articles were retrieved. Boolean operators, wildcards, exact, truncation and other commands were used whenever appropriate. Relevant Medical Subject Headings terms identified during the search were included. Table 2 lists the final keywords and search strategy used in this OoSR. If manual search was required, the reference list of the included articles would first be reviewed and any relevant citation to be extracted and screened is then identified, including any relevant article in the possession of, known, or encountered by the researchers during the project duration. The initial electronic databases searching was conducted on 20 March 2020 on six electronic databases. A contingency effort will be taken to regularly update the searching until the acceptance of our OoSR.

Table 2

Search strategy keywords, modified as needed for the electronic databases

(Search engine) Database	Keywords
(Scopus) Scopus	(TITLE-ABS-KEY (“medic*” OR “health” OR “health science*” OR “nurs*” OR “biomedic*” OR “pharma*” OR “nutrition” OR “dietetic*” OR “dental” OR “dentist*” OR “allied health” OR “occupational health” OR “environmental health” OR “occupational therap*” OR “physiotherap*” OR “phsyical therap*” OR “speech therap*” OR “speech language phatolog*” OR “occupational safety” OR psycholog* OR “audiolog*” OR forensic* OR “radiotherap*”) AND TITLE-ABS-KEY (student* OR “undergrad*” OR “universit*” OR “college” “higher education*” OR “tertiary education*”) AND TITLE-ABS-KEY (“e-Learning” OR “e-learning” OR “online” OR “web based” OR “blended” OR “internet” OR “computer*” OR “mobile” OR “simulation*” OR game* OR gamification* OR “MOOC*” OR “e-content” OR “digital” OR “virtual” OR “electronic” OR “computer-based learning” OR “online” OR “technology enhanced learning” OR “augment*” OR technolog* OR “hybrid learning” OR multimedia OR “massive open online course” OR “m-learning” OR moodle OR “flexible” OR “e-pedagogy” OR “social media” OR iot OR “Internet of Things”) AND TITLE-ABS-KEY (“systematic review”))
(EBSCOHost) CINAHL Academic Search Complete Cochrane Database of Systematic Reviews MEDLINE Psychology and Behavioral Sciences Collection	AB (“medic*” OR “health” OR “health science*” OR “nurs*” OR “biomedic*” OR “pharma*” OR “nutrition” OR “dietetic*” OR “dental” OR “dentist*” OR “allied health” OR “occupational health” OR “environmental health” OR “occupational therap*” OR “physiotherap*” OR “phsyical therap*” OR “speech therap*” OR “speech language phatolog*” OR “occupational safety” OR psycholog* OR “audiolog*” OR forensic* OR “radiotherap*”) AND AB (student* OR “undergrad*” OR “universit*” OR “college” “higher education*” OR “tertiary education*”) AND TI (“e-Learning” OR “e-learning” OR “online” OR “web based” OR “blended” OR “internet” OR “computer*” OR “mobile” OR “simulation*” OR game* OR gamification* OR “MOOC*” OR “e-content” OR “digital” OR “virtual” OR “electronic” OR “computer-based learning” OR “online” OR “technology enhanced learning” OR “augment*” OR technolog* OR “hybrid learning” OR multimedia OR “massive open online course” OR “m-learning” OR Moodle OR “flexible” OR “e-pedagogy” OR “social media” OR IoT OR “Internet of Things”) AND AB (education* OR learn* OR teach* OR study* OR train*) AND “systematic review”

Gathering information from syntheses

The third step involves a screening process of the retrieved articles from the systematic searching. The screening process of the articles will be based on predefined inclusion and exclusion criteria. The criteria developed will be categorised into two: standard and tailored. The standard criteria will be developed by consulting previous reviews, while the tailored criteria will be purposely defined for this project. All the criteria developed will be agreed on by the research team. The predefined criteria must be aligned with the OoSR objective to minimise the mismatch scope of the included systematic review with this overview.⁵⁸

The following section lists the operational definition of the inclusion criteria in more detail:

• Undergraduate healthcare students: only field of study under the mainstream medical and health sciences education will be considered, such as medicine, biomedical sciences, pharmacy, nursing, dentistry, nutrition and dietetics, occupational and environmental health, and forensic, etc. Other non-mainstream allied health field of study, such as complementary and traditional medicine (ie, chiropractic, acupuncture, homeopathy) will not be considered. In addition, undergraduates in this OoSR are referred to as entry-level or pre-registration education programme, which abide with the minimum requirement for practice. This group includes Bachelor’s degree and in some countries entry-level Master’s or Doctoral programmes (eg, Doctor of Occupational Therapy, Doctor of Physiotherapy), as the differences in skill and knowledge are minimal.^{59 60} Any postgraduate education programme (eg, Master of Surgery) or post-professional programme (eg, post-basic training, continuous medical education (CME)) will not be considered. The reason for postgraduate or post-professional training not being included as this group could have prior fundamental knowledge pertaining to the subject that may make it easier for them to grasp new knowledge,^{61 62} while experience equips the participants with better clinical reasoning skills.⁶³ Hence, such a situation may pose an advantage to post-professionals when it comes to acquiring understanding as compared with undergraduates who have minimal information on the topic. Therefore, biases could result where the outcome may be compromised, and the improvements to the educational outcomes may not be solely due to intervention. Therefore, this group will be considered only when at least more than half (≥50 %) of the primary studies included in the investigated systematic reviews involve undergraduate or pre-professional training.

• TBL: technology in this OoSR should abide with the concept of IR 4.0 or IoT.^64–67 Therefore, archaic, low or common technology, such as PowerPoint presentation-only lectures, manuals or mechanical equipment, such as low-fidelity manikins and gamification concept teaching not involving sophisticated technology, such as social and cooperative games, will not be considered eligible. Simulation-based studies that do not use technology, such as real patients, standardised patients or healthy individuals, will not be considered. Also, the use of technology as a medium for assessment and evaluation only, such as for objective structured clinical examination and final examination, will not be considered eligible.

• Effectiveness: effectiveness in this OoSR refers to the teaching effectiveness using technology-based approaches. Therefore, teaching effectiveness is defined as the capability to produce gains in student achievement, taking account of a baseline measure of student prior attainment and other characteristics of student intake. The achievement can take the form of improvement, among others, but is not limited to assessment score, knowledge, understanding, critical thinking skills, personal attributes, performances, teamwork and collaboration, or communication skills.^{68 69} Effectiveness will also take into account the cognitive, psychomotor and affective taxonomies in terms of knowledge generation and clinical-based competencies.

• Systematic review: the usage of the term ‘systematic review’ either in the title or the full text is insufficient to be classified as a systematic review.⁷⁰ A systematic review is defined as a type of review that conducts systematic searching and applies a screening process of the literature reviewed against predefined criteria, where each included study (either in tabular or narrative format), and the quality analysis of the included studies are conducted and reported.

The exclusion criteria use standard criteria found in review studies, such as no full text available after the research team has exhausted all possible options to retrieve the full text, such as downloading from the library, searching in online deposits (ie, ResearchGate, Academia), contacting the author and the inability to purchase the article; available in grey literature format (ie, theses, books, conference abstract, reports) as such articles have not undergone rigorous peer-review process; other types of reviews (ie, scoping review, umbrella review, literature review), as these reviews do not abide with the systematic review methodology; and non-English articles, as the research team has limited capacity to understand languages other than English and have limited resources to subsidise translation services. The decision to not include non-English articles will have minimal impact on the outcome of the OoSR.⁷¹

The screening process will be recorded using Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA).⁷² Example of the PRISMA flowchart is shown in figure 1. Screening will be conducted independently by at least two researchers in each stage of the title and abstract, and full-text screening. The screening process will be recorded using a Microsoft Excel document according to the predefined inclusion and exclusion criteria. EndNote V.X8 software (Thompson Reuters) will be used to manage the citations.

Figure 1. PRISMA flowchart for screening process. PRISMA, preferred reporting items for systematic review and meta-analysis.

Pre-consensus agreement will be obtained based on two points; (1) after the abstract screening and before proceeding to full-text screening, and (2) after full-text screening and before synthesising the included articles. The type of agreement analysis depends on the availability of screening reviewers at each stage. Fleiss kappa will be used if more than two reviewers screened the same articles, while Cohen kappa will be used if the screening only involves two reviewers.⁷³ The analysis will be conducted on the whole list of articles; not only on the accepted pre-consensus. The interpretation of the kappa agreement is based on Landis and Koch,⁷⁴ where a kappa value of less than 0 is considered poor, whereas a slight agreement is indicated by a value between 0.01 and 0.20, followed by fair is (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80) and almost perfect (0.81–1.00). Percentage agreement will be used to supplement the agreement value as the kappa value is known to fluctuate and is sensitive to the characteristics and sequences of the ratings.^{75 76} Percentage agreement is calculated based on pre-consensus agreed and accepted articles divided by the total articles reviewed during a particular stage. Discussion will be conducted on disagreed articles between the involved reviewers and any unresolved disputes will be resolved by a third reviewer acting as an arbiter.

A matrix table using the Garrad’s Matrix Method⁷⁷ will be implemented to extract the data from the included articles. Commonly, the pieces of information extracted are citation (ie, authors and year), objective of study and review conclusion. Other pieces of information will be extracted included characteristics of the participants, databases used in the review, number of included studies and quality assessment used. At least one researcher that is not involved during the screening process will be responsible for extracting the information. The extracted information will be cross-checked and verified by researchers involved in the screening process. Subsequently, the researchers will meet and discuss the comprehensiveness and explicitness of the extracted table. Primary studies in the included reviews will be extracted and compared to determine any overlapping evidence, as suggested by Pieper et al ⁷⁸. Considering overlapping studies is important to meticulously develop the conclusion and not to overestimate the overview.

Evaluating the quality of evidence

A MeaSurement Tool to Assess systematic Reviews (AMSTAR) will be used in this OoSR project. The AMSTAR tool is considered the ‘gold standard’ instrument for assessing the quality of systematic reviews and is among the most widely used assessment tools.⁷⁹ AMSTAR has 11 items scored dichotomously as ‘Yes’, ‘No’, ‘Can’t Answer’ or ‘Not Applicable’. The total score is calculated by adding the YES responses. The total score for AMSTAR ranges from 0 to 11, where a higher score indicates a better quality review. An article is considered of good quality if the score is between 9 and 11, followed by moderate quality (5–8) and low quality (0–4).⁸⁰ AMSTAR has established adequate validity and reliability.^{81 82}

Risk of Bias in Systematic Review (ROBIS) aims to evaluate the level of bias present within a systematic review.⁸³ ROBIS consists of four domains, namely (1) study eligibility criteria, (2) identification and selection of studies, (3) data collection and study appraisal, and (4) synthesis and finding. ROBIS is rated based on the judgement of the assessor either as low, high or unclear on each domain based on the signalling questions provided. The final judgement on the overall level of risk of bias will be made considering the results of the four domains. ROBIS has an acceptable level of validity and reliability.⁸³

Prior to the AMSTAR and ROBIS assessment, a pretest study will be carried out to evaluate the validity and reliability of the assessment tools. An inter-rater reliability is planned; where one researcher with experience in publishing overview, experience in using AMSTAR and experience providing training in conducting an overview study is considered as a gold-rater, while at least two inexperienced researchers in overview study will be recruited as comparison-raters. Articles that are accepted during the screening process will be sampled and randomly selected using a computer-generated application (https://www.randomizer.org/). At least 10% of the total included articles or 10 articles, whichever higher, will be selected for inter-rater reliability testing. The sample articles will be independently rated on AMSTAR by the gold-rater and the comparison-raters. The rating will then be compared and calculated for agreement in the percentage format and based on Cohen’s kappa analysis. It is hypothesised that the rating should achieve an overall average of 80%. If the cut-off value is achieved, then, the rating of the remaining articles can be done independently by one rater. If not achieved, then, a discussion will be held between the raters to resolve any confusion and to reach a consensus, and additional training would also be provided. At that juncture, the remaining articles will be rated independently by one rater and verified thereafter. A concurrent validity will be conducted between these two tools (ie, AMSTAR and ROBIS) as none of the research team members are experienced in using ROBIS. All included articles will be selected to be tested for the concurrent validity. These two tools were previously used in unison and compared for reliability and construct validity.⁸³

Analysing and integrating the outcomes of synthesis

According to Cooper and Koenka,⁴⁴ there are several methods available to assist in integrating the outcomes of synthesis. The methods include from the most robust to the least robust are as follows: (1) grand meta-analysis, (2) pool and synthesise primary studies, (3) accept latest publication conclusion, (4) accept the highest quality review conclusion, (5) count the evidence, and (6) simply ignore and report the findings. It is also possible to combine several methods for synthesis purposes. Grand meta-analysis or pool and synthesise primary studies are not suitable for this OoSR as the topic investigated is expected to have a high level of heterogeneity because it involves various types of technology available. Besides, various outcomes will be observed and innumerable amounts of studies are anticipated. Accepting the latest publication is less suitable because it may not be representative for every technology investigated. Therefore, this OoSR will consider to employ the methods of selecting reviews with high quality (with moderate and high quality with AMSTAR and ROBIS) and calculating the available evidences in supporting or refutation of the use of technology.

Interpreting the evidence

Themes will be generated to focus on organising the synthesis into a cohesive narrative and meaningful summary. The themes will be based on the outcomes and grouped per Bloom’s learning taxonomy model: (1) cognitive, (2) psychomotor and (3) affective.⁸⁴ The cognitive domain defines knowledge classification and understanding of theories and information. Thus, the cognitive domain is liberal, and its application transcends a wide spectrum of population. Therefore, the findings on cognitive domains can be generalised across disciplines. The psychomotor domain focuses on physical skills or tasks development, so it is more discipline-specific and its exclusive application in one discipline may not be applicable to others. Meanwhile, the affective domain is based on perspective, attitude and values, which are much more subjective in measurement and tailored to individual personality; also, it is difficult to be generalised across disciplines. Although the three domains are interconnected,⁸⁵ its separation during the analysis for reporting is required, as the approach for each domain will differ. Therefore, the synthesis of the summary will be divided into two major dimensions: theory and knowledge generation, and clinical-based competencies.^{84 86} Moreover, the TBL technologies used will be identified and extracted from the included systematic reviews. The technology will be grouped and listed into two categories, such as (1) internet or application-based, and (2) hardware technology that are either commercially available or purposely developed technology.

As this OoSR will provide a holistic and comprehensive scenario on the effectiveness of TBL in healthcare education, it will also provide the best platform to find overarching gap(s), that is, an absence of evidence from the current available literature. This is in lieu with the suggestion by Paton et al ⁸⁷ which requested researchers to dig deeper and find a unique gap available to conduct further distinctive research and contribute towards richer knowledge development. Therefore, this OoSR will strive to search for that originality and comprehensiveness.

Presenting the result

There is currently no gold standard guideline available in reporting an OoSR; such a guideline is still under development.^{58 88} Therefore, the research team will abide with Cooper and Koenka’s⁴⁴ methodology as a foundation for conducting the OoSR and will be enriched with other literature deemed valuable for producing the highest reporting quality. The abstract of the OoSR will be written according to the Preferred Reporting Items for Overview of Systematic Reviews abstracts outlined by Bougioukas et al ⁸⁹ Ensuring a robust yet explicit abstract is critical, as the abstract is an important source of information that practitioners and researchers will initially read and rely on.⁵¹ The full text will be presented as outlined in this protocol. The outcome of this OoSR project is aimed for several journal publications and conference presentations.

Discussion

An inestimable number of systematic reviews are available; over 40 000 systematic reviews are registered in the PROSPERO alone since its inception and more than a dozen systematic reviews are published daily.^90–92 This situation results in overwhelming numbers of available systematic reviews. Hence, OoSR can collect and synthesise the evidence as a whole to create concrete findings.^{91 93} A similar situation is also observed with TBL. The rapid development of technology has contributed towards abundant technology development for education purposes and in turn has resulted in many systematic reviews produced pertaining to this topic. However, systematic reviews have limitations where some studies are too homogeneous or focused on a single aspect of either one type of population or a specific technology or outcome. Another drawback is that a certain level of skill is required to retrieve suitable and appropriate articles. Therefore, OoSR could facilitate educators to be quickly informed and to become on par with the current available evidences on a particular topic.

Nonetheless, OoSR is not without limitations. OoSR is still considered a new and emerging methodology⁹³; therefore, the available guideline is yet to mature. Integrating several systematic reviews may result in the combination of too many primary studies that are not related. Therefore, the findings may become too generic and diluted.⁹³ Another huge challenge is the reporting of overlapping studies; where the same studies could be included in multiple systematic reviews and thus overestimating its effect.^{58 78 91} However, the overestimation in this OoSR will have little impact and harm due to the nature of the topic and because it is controlled by calculating the overlapping value.⁷⁸ Systematic review alone is a secondary research and requires between 6 months and 2 years to be published; therefore, the evidence may become outdated.⁹¹ Then again, OoSR may also suffer greater deficiency in recentness, as it pools systematic reviews.⁵⁸ However, our protocol will implement regular updates on the searching to overcome this issue. The latest and most recent systematic reviews found will be included in this OoSR, thus ensuring that the abreast gap is kept to a minimal. Additionally, we will also continuously keep track of the list of original publications pertaining to the topic. Any critical and important original study will be synthesised in the body of introduction and discussion sections in our OoSR.

Expected significance of the study

This OoSR is expected to provide valuable information regarding the status of TBL either as a main education medium or maintain its use as a complementary approach alongside conventional/traditional educational teaching and learning among healthcare students. The comprehensive and immediate information will benefit stakeholders, such as future undergraduates, educators in higher education, higher education industries and policymakers. Undergraduate students currently in higher education consist mainly of Generation Z and in a couple of years, Generation Alpha will make up the bulk of students. These groups of learners are considered as technology natives and technology savvy.^94–96 They heavily rely on the use of technology in daily activities. Generation Z, although still requiring face-to-face lectures or meetings with the lecturer physically for guidance, however, prefer more TBL to be incorporated, to have greater ownership and flexibility in learning.⁹⁷ Generation Alpha, on the other hand, are expected to be totally connected digitally and are very comfortable with the highly innovative and free lifestyle. They tend to challenge conventional views in every aspect and as such consider the current model of education as archaic and impractical.^{98 99} Therefore, the higher education sector should prepare for a revolution in teaching beyond the border of institution locality and brick-and-mortar setting. Educators will need to consider the implications of the needs of these generations as they develop curricula and plan for meaningful lesson experiences.

The findings of this OoSR would prove to afford valuable insights that will inform policymakers and the higher education sector of the need of a paradigm shift to cater to such changes and to look at the effectiveness of TBL in future education. This initiative is important because the higher education sector will not only compete among higher education providers to attract pupils (e.g., prospective students), but also between industries such as employment sectors, and the self-employed that rely on the online business model, working without restrictions in terms of working hours and office space.¹⁰⁰ For educators, this OoSR will equip them with available TBL approaches that help reduce the generational gap to make the teaching and learning activities more meaningful.^{94 96} In addition, this OoSR may provide a comprehensive view on TBL and identifying gap(s) that will inform future research in this field.

The goal of ensuring best evidence practice in medical and health sciences education is to promote learning, enhancing clinical practice and improving client outcome.³⁹ This action will indirectly benefit the clients (ie, patients) as the implementation of best education approaches will produce highly competent practitioners. Learning through technology facilitates towards a more dynamic and supportive learning environment that helps enhance understanding and enculturates continuous learning on the current theories and best practices.⁸ In addition, the application of technology in healthcare education may ensure both client and practitioner safety, as students can practice in a safer and controlled environment with reduced contact, and the students can repeat any action without worry or risking the patients’ life.^{8 39} Therefore, this OoSR may contribute towards the above objective.

Ethics and dissemination

This research does not require ethical approval, as the study is a systematic review of published literature. Any changes to the current protocol will be considered protocol amendment, and this will be communicated to the journal, along with a motivation and justification for the protocol amendment. We aim to submit the results of this OoSR to several peer-reviewed journals.

Footnote

Contributors All authors contributed to the conception of the protocol. MSC and HAH initiated the original idea. MHR is the consultant for the project that planned the concept, and MHR and MSC wrote the initial draft of the protocol. MHR, MSC, HAH, MZM and NFMDH critically revised the initial draft for important intellectual content. All authors gave the final approval of the manuscript submitted. All authors agreed to be accountable for all aspects related to this work.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Patient and public involvement This research was done without patient involvement. Patients were not invited to comment on the study design and were not consulted to develop patient relevant outcomes or interpret the results. Patients were not invited to contribute to the writing or editing of this document for readability or accuracy.

Patient consent for publication Not required.

Provenance and peer review Not commissioned; externally peer reviewed.

References

¹ Paul A, Jeyaraj R. Internet of things: a primer. Hum Behav Emerg Technol 2019; 1: 37–47. doi:10.1002/hbe2.133

² Shahroom AA, Hussin N. Industrial revolution 4.0 and education. Int J Acad Res Bus Soc Sci 2018; 8: 314–9. doi:10.6007/IJARBSS/v8-i9/4593

³ Ramani S. The internet and education in the developing world - hopes and reality. Smart Learn Environ 2015; 2: 8. doi:10.1186/s40561-015-0015-x

⁴ Hampton D, Welsh D, Wiggins AT. Learning preferences and engagement level of generation Z nursing students. Nurse Educ 2020; 45: 160–4. doi:10.1097/NNE.0000000000000710 http://www.ncbi.nlm.nih.gov/pubmed/31219957

⁵ Luo J, Boland R, Chan CH. How to use technology in educational innovation. In: Roberts LW, ed. Roberts academic medicine handbook. Cham: Springer, 2020: 141–7.

⁶ Reinhold F, Hoch S, Werner B, et al. Learning fractions with and without educational technology: what matters for high-achieving and low-achieving students? Learn Instr 2020; 65: 101264. doi:10.1016/j.learninstruc.2019.101264

⁷ Bond M, Zawacki-Richter O, Nichols M. Revisiting five decades of educational technology research: a content and authorship analysis of the British Journal of educational technology. Br J Educ Technol 2019; 50: 12–63. doi:10.1111/bjet.12730

⁸ Guze PA. Using technology to meet the challenges of medical education. Trans Am Clin Climatol Assoc 2015; 126: 260–70. http://www.ncbi.nlm.nih.gov/pubmed/26330687

⁹ Prater MR. Teaching millennials and generation Z. In: Gotian R, Kang Y, Safdieh J, eds. Handbook of research on the efficacy of training programs and systems in medical education. Pennsylvania: IGI Global, 2020: 72–91.

¹⁰ Yu E, Canton S. Student-inspired optimal design of online learning for generation Z. J Educ Online 2020; 17: 1–11.

¹¹ Echenique EG. An integrative review of literature on learners in the digital era. Stud Paedagog 2014; 19: 161–84. doi:10.5817/SP2014-4-8

¹² Martinerie L, Rasoaherinomenjanahary F, Ronot M, et al. Health care simulation in developing countries and low-resource situations. J Contin Educ Health Prof 2018; 38: 205–12. doi:10.1097/CEH.0000000000000211 http://www.ncbi.nlm.nih.gov/pubmed/30157154

¹³ Delgaty L, Fisher J, Thomson R. The ‘dark side’ of technology in medical education. MedEdPublish 2017; 6: 19. doi:10.15694/mep.2017.000081

¹⁴ Aung TN, Khaing SS. Challenges of implementing e-Learning in developing countries: a review. In: Zin TT, JC-W L, Pan J-S, et al, eds. Genetic and evolutionary computing. Cham: Springer, 2016: 405–11.

¹⁵ Di Gennaro F, Pizzol D, Marotta C, et al. Coronavirus diseases (COVID-19) current status and future perspectives: a narrative review. Int J Environ Res Public Health 2020; 17: 2690. doi:10.3390/ijerph17082690 http://www.ncbi.nlm.nih.gov/pubmed/32295188

¹⁶ Docea AO, Tsatsakis A, Albulescu D, et al. A new threat from an old enemy: re–emergence of coronavirus (review). Int J Mol Med 2020; 45: 1631–43. doi:10.3892/ijmm.2020.4555 http://www.ncbi.nlm.nih.gov/pubmed/32236624

¹⁷ Adhikari SP, Meng S, Wu Y-J, et al. Epidemiology, causes, clinical manifestation and diagnosis, prevention and control of coronavirus disease (COVID-19) during the early outbreak period: a scoping review. Infect Dis Poverty 2020; 9: 29. doi:10.1186/s40249-020-00646-x http://www.ncbi.nlm.nih.gov/pubmed/32183901

¹⁸ Bai Y, Yao L, Wei T, et al. Presumed asymptomatic carrier transmission of COVID-19. JAMA 2020; 323: 1406–7. doi:10.1001/jama.2020.2565 http://www.ncbi.nlm.nih.gov/pubmed/32083643

¹⁹ Watkins J. Preventing a covid-19 pandemic. BMJ 2020; 368: m810. doi:10.1136/bmj.m810 http://www.ncbi.nlm.nih.gov/pubmed/32111649

²⁰ World Health Organization. Coronavirus disease 2019 (COVID-19) situation report-72, 2020. Available: www.who.int/epi-win

²¹ Chinazzi M, Davis JT, Ajelli M, et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (COVID-19) outbreak. Science 2020; 368: 395–400. doi:10.1126/science.aba9757

²² Ebrahim SH, Ahmed QA, Gozzer E, et al. Covid-19 and community mitigation strategies in a pandemic. BMJ 2020; 368: m1066. doi:10.1136/bmj.m1066 http://www.ncbi.nlm.nih.gov/pubmed/32184233

²³ Foronda C. COVID-19: SSH/INACSL position statement on use of virtual simulation during the pandemic, 2020. Available: https://www.ssih.org/COVID-19-Updates/ID/2237/COVID-19-SSHINACSL-Position-Statement-on-Use-of-Virtual-Simulation-during-the-Pandemic

²⁴ World Federation of Occupational Therapies. COVID-19 and WFOT minimum education standards statement, 2020. Available: https://www.wfot.org/news/2020/covid-19-and-wfot-minimum-education-standards-statement

²⁵ Almarzooq ZI, Lopes M, Kochar A. Virtual Learning During the COVID-19 Pandemic. J Am Coll Cardiol 2020; 75: 2635–8. doi:10.1016/j.jacc.2020.04.015

²⁶ Daniel SJ. Education and the COVID-19 pandemic. Prospects 2020; 49: 91–6. doi:10.1007/s11125-020-09464-3

²⁷ Huang RH, Liu DJ, Tlili A. Handbook on facilitating flexible learning during educational disruption: the Chinese experience in maintaining undisrupted learning in COVID-19 outbreak. Beijing: Smart Learning Institute of Beijing Normal University, 2020. https://iite.unesco.org/news/handbook-on-facilitating-flexible-learning-during-educational-disruption/

²⁸ Lall S, Singh N. Covid-19: unmasking the new face of education. Int J Res Pharm Sci 2020; 11: 48–53. doi:10.26452/ijrps.v11iSPL1.2122

²⁹ Marshall AL, Wolanskyj-Spinner A. COVID-19: challenges and opportunities for educators and generation Z learners. Mayo Clin Proc 2020; 95: 1135–7. doi:10.1016/j.mayocp.2020.04.015

³⁰ Reich J, Buttimer CJ, Fang A. Remote learning guidance from state education agencies during the COVID-19 pandemic: a first look, 2020. Available: https://edarxiv.org/437e2/

³¹ Stambough JB, Curtin BM, Gililland JM, et al. The past, present, and future of orthopedic education: lessons learned from the COVID-19 pandemic. J Arthroplasty 2020; 35: S60–4. doi:10.1016/j.arth.2020.04.032 http://www.ncbi.nlm.nih.gov/pubmed/32345564

³² Zhang W, Wang Y, Yang L, et al. Suspending classes without stopping learning: China’s education emergency management policy in the COVID-19 outbreak. J Risk Financ Manag 2020; 13: 55. doi:10.3390/jrfm13030055

³³ Zhou L, Li F, Wu S, et al. “School’s out, but class’s on”, the largest online education in the world today: taking China’s practical exploration during the COVID-19 epidemic prevention and control as an example. Best Evid Chin Edu 2020; 4: 501–19. doi:10.15354/bece.20.ar023

³⁴ Joseph GV, Thomas KA. Volatility of digital technology enabled learning through social media: educators’ apprehensions. TEST Eng Manag 2020; 82: 5832–9.

³⁵ Wilson NC. New barriers to technology integration and digital education equity. In: Grassetti MT, Jeannetti JZ, eds. Next generation digital tools and applications for teaching and learning enhancement. Pennsylvania: IGI Global, 2020: 122–36.

³⁶ Babu N, Reddy BS. Challenges and opportunity of e-learning in developed and developing countries-a review. Int J Emerg Res Manag Technol 2015; 4: 259–62.

³⁷ Nawaz A, Qureshi QA. Eteaching/epedagogy threats & opportunities for teachers in heis. Glob J Manag Bus Res 2010; 10: 23–31.

³⁸ Richardson D. Don't dump the didactic lecture; fix it. Adv Physiol Educ 2008; 32: 23–4. doi:10.1152/advan.00048.2007 http://www.ncbi.nlm.nih.gov/pubmed/18334564

³⁹ Thistlethwaite J, Hammick M. The best evidence medical education (BEME) collaboration: into the next decade. Med Teach 2010; 32: 880–2. doi:10.3109/0142159X.2010.519068 http://www.ncbi.nlm.nih.gov/pubmed/21039096

⁴⁰ Rouleau G, Gagnon M-P, Côté J, et al. Effects of e-learning in a continuing education context on nursing care: systematic review of systematic qualitative, quantitative, and mixed-studies reviews. J Med Internet Res 2019; 21: e15118. doi:10.2196/15118 http://www.ncbi.nlm.nih.gov/pubmed/31579016

⁴¹ Cant RP, Cooper SJ. Use of simulation-based learning in undergraduate nurse education: an umbrella systematic review. Nurse Educ Today 2017; 49: 63–71. doi:10.1016/j.nedt.2016.11.015 http://www.ncbi.nlm.nih.gov/pubmed/27902949

⁴² Yoshimoto K, Inenaga Y, Yamada H. Pedagogy and andragogy in higher education? A comparison between Germany, the UK and Japan. Eur J Education 2007; 42: 75–98. doi:10.1111/j.1465-3435.2007.00289.x

⁴³ Mickan S, Tilson JK, Atherton H, et al. Evidence of effectiveness of health care professionals using handheld computers: a scoping review of systematic reviews. J Med Internet Res 2013; 15: e212. doi:10.2196/jmir.2530 http://www.ncbi.nlm.nih.gov/pubmed/24165786

⁴⁴ Cooper H, Koenka AC. The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. Am Psychol 2012; 67: 446–62. doi:10.1037/a0027119 http://www.ncbi.nlm.nih.gov/pubmed/22352742

⁴⁵ Mandinach EB, Cline HF. Classroom dynamics: implementing a technology-based learning environment. New York: Routledge, 2013.

⁴⁶ Kearsley G, Shneiderman B. Engagement theory: a framework for technology-based teaching and learning. Educ Technol 1998; 38: 20–3.

⁴⁷ Marquardt M, Kearsley G. Technology-based learning: maximizing human performance and corporate success. New York: CRC Press, 1998. https://books.google.com.my/books?hl=en&lr=&id=3AdJg2527tEC&oi=fnd&pg=PR11&dq=Technology-based+learning:+Maximizing+human+performance+and+corporate+success&ots=LH58v2wMGN&sig=OYhsQVU1Jehxzj_Palswied-vHA (accessed 29 May 2020)

⁴⁸ Moran J, Briscoe G, Peglow S. Current technology in advancing medical education: perspectives for learning and providing care. Acad Psychiatry 2018; 42: 796–9. doi:10.1007/s40596-018-0946-y http://www.ncbi.nlm.nih.gov/pubmed/29949053

⁴⁹ Kodikara KG, Karunaratne WCD, Chandratilake MN. High fidelity simulation in undergraduate medical curricula: experience of fourth year medical students. SE Asian Jnl Med Educ 2020; 13: 25–31. doi:10.4038/seajme.v13i2.207

⁵⁰ Patel SA. Education methods. In: Carstens PK, Paulman P, Paulman A, eds. Comprehensive healthcare simulation: mobile medical simulation. Cham: Springer, 2020: 223–33.

⁵¹ Marcelo A, Gavino A, Isip-Tan IT, et al. A comparison of the accuracy of clinical decisions based on full-text articles and on journal abstracts alone: a study among residents in a tertiary care hospital. Evid Based Med 2013; 18: 48–53. doi:10.1136/eb-2012-100537 http://www.ncbi.nlm.nih.gov/pubmed/22782923

⁵² Sullivan GM. Deconstructing quality in education research. J Grad Med Educ 2011; 3: 121–4. doi:10.4300/JGME-D-11-00083.1 http://www.ncbi.nlm.nih.gov/pubmed/22655130

⁵³ Arkorful V, Abaidoo N. The role of e-learning, advantages and disadvantages of its adoption in higher education. Int J Instr Technol Distance Learn 2015; 12: 29–42.

⁵⁴ Talebian S, Mohammadi HM, Rezvanfar A. Information and communication technology (ICT) in higher education: advantages, disadvantages, conveniences and limitations of applying e-learning to agricultural students in Iran. Procedia Soc Behav Sci 2014; 152: 300–5. doi:10.1016/j.sbspro.2014.09.199

⁵⁵ James G. Advantages and disadvantages of online learning, 2006. Available: www.allencomm.com/events

⁵⁶ Prasad K. Formulating a focused clinical question. In: Prasad K, ed. Fundamentals of evidence based medicine. New Delhi: Springer, 2014: 19–25.

⁵⁷ Hunt H, Pollock A, Campbell P, et al. An introduction to overviews of reviews: planning a relevant research question and objective for an overview. Syst Rev 2018; 7: 39. doi:10.1186/s13643-018-0695-8 http://www.ncbi.nlm.nih.gov/pubmed/29490699

⁵⁸ Ballard M, Montgomery P. Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist. Res Synth Methods 2017; 8: 92–108. doi:10.1002/jrsm.1229 http://www.ncbi.nlm.nih.gov/pubmed/28074553

⁵⁹ Brown T, Crabtree JL, Mu K, et al. The entry-level occupational therapy clinical doctorate: advantages, challenges, and international issues to consider. Occup Ther Health Care 2015; 29: 240–51. doi:10.3109/07380577.2015.1013597 http://www.ncbi.nlm.nih.gov/pubmed/25821889

⁶⁰ Mitchell AW, Xu YJ. Critical reasoning scores of entering bachelor’s and master’s students in an occupational therapy program. Am J Occup Ther 2011; 65: e86–94. doi:10.5014/ajot.2011.001511

⁶¹ Artino AR, Stephens JM. Academic motivation and self-regulation: a comparative analysis of undergraduate and graduate students learning online. Internet High Educ 2009; 12: 146–51. doi:10.1016/j.iheduc.2009.02.001

⁶² Chapell MS, Blanding ZB, Silverstein ME, et al. Test anxiety and academic performance in undergraduate and graduate students. J Educ Psychol 2005; 97: 268–74. doi:10.1037/0022-0663.97.2.268

⁶³ Unsworth CA. The clinical Reasoning of novice and expert occupational therapists. Scand J Occup Ther 2001; 8: 163–73. doi:10.1080/110381201317166522

⁶⁴ Afrianto A. Being a professional teacher in the era of industrial revolution 4.0: opportunities, challenges and strategies for innovative classroom practices. English Lang Teach Res 2018; 2: 1–13.

⁶⁵ Aziz Hussin A, Hussin AA. Education 4.0 made simple: ideas for teaching. Int J Educ Lit Stud 2018; 6: 92. doi:10.7575/aiac.ijels.v.6n.3p.92

⁶⁶ Tang J, Sun D, Liu S, et al. Enabling deep learning on IoT devices. Computer 2017; 50: 92–6. doi:10.1109/MC.2017.3641648

⁶⁷ Charmonman S, Mongkhonvanit P, Dieu N, et al. Applications of Internet of things in e-learning. Int J Comput Internet Manag 2015; 23: 1–4.

⁶⁸ Berk RA. Survey of 12 strategies to measure teaching effectiveness. Int J Teach Learn High Educ 2005; 17: 48–62.

⁶⁹ Arthur Jr W, Tubre T, Paul DS, et al. Teaching effectiveness: the relationship between reaction and learning evaluation criteria. Educ Psychol 2003; 23: 275–85. doi:10.1080/0144341032000060110

⁷⁰ Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J 2009; 26: 91–108. doi:10.1111/j.1471-1842.2009.00848.x http://www.ncbi.nlm.nih.gov/pubmed/19490148

⁷¹ Nussbaumer-Streit B, Klerings I, Dobrescu AI, et al. Excluding non-English publications from evidence-syntheses did not change conclusions: a meta-epidemiological study. J Clin Epidemiol 2020; 118: 42–54. doi:10.1016/j.jclinepi.2019.10.011 http://www.ncbi.nlm.nih.gov/pubmed/31698064

⁷² Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6: e1000097. doi:10.1371/journal.pmed.1000097 http://www.ncbi.nlm.nih.gov/pubmed/19621072

⁷³ McHugh ML. Interrater reliability: the kappa statistic. Biochem Med 2012; 22: 276–82. doi:10.11613/BM.2012.031

⁷⁴ Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74. doi:10.2307/2529310 http://www.ncbi.nlm.nih.gov/pubmed/843571

⁷⁵ Björk J, Rittner R, Cromley E. Exploring inter-rater reliability and measurement properties of environmental ratings using kappa and colocation quotients. Environ Health 2014; 13: 86. doi:10.1186/1476-069X-13-86

⁷⁶ Graham M, Milanowski A, Miller J, et al. Measuring and promoting inter-rater agreement of teacher and principal performance ratings, 2012. Available: https://eric.ed.gov/?id=ED532068

⁷⁷ Garrard J. Health sciences literature review made easy: the matrix method. USA: Jones & Bartlett Learning, 2014.

⁷⁸ Pieper D, Antoine S-L, Mathes T, et al. Systematic review finds overlapping reviews were not mentioned in every other overview. J Clin Epidemiol 2014; 67: 368–75. doi:10.1016/j.jclinepi.2013.11.007 http://www.ncbi.nlm.nih.gov/pubmed/24581293

⁷⁹ Gates A, Gates M, Duarte G, et al. Evaluation of the reliability, usability, and applicability of AMSTAR, AMSTAR 2, and ROBIS: protocol for a descriptive analytic study. Syst Rev 2018; 7: 85. doi:10.1186/s13643-018-0746-1 http://www.ncbi.nlm.nih.gov/pubmed/29898777

⁸⁰ Pieper D, Koensgen N, Breuing J, et al. How is AMSTAR applied by authors - a call for better reporting. BMC Med Res Methodol 2018; 18: 56. doi:10.1186/s12874-018-0520-z http://www.ncbi.nlm.nih.gov/pubmed/29914386

⁸¹ Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007; 7: 10. doi:10.1186/1471-2288-7-10 http://www.ncbi.nlm.nih.gov/pubmed/17302989

⁸² Shea BJ, Hamel C, Wells GA, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009; 62: 1013–20. doi:10.1016/j.jclinepi.2008.10.009 http://www.ncbi.nlm.nih.gov/pubmed/19230606

⁸³ Banzi R, Cinquini M, Gonzalez-Lorenzo M, et al. Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability. J Clin Epidemiol 2018; 99: 24–32. doi:10.1016/j.jclinepi.2018.02.024 http://www.ncbi.nlm.nih.gov/pubmed/29526556

⁸⁴ Bloom BS, Engelhart MD, Furst EJ, et al. Taxonomy of educational objectives: the classification of educational goals: handbook 1 cognitive domain. New York: David McKay Company, Longmans, 1956.

⁸⁵ Sönmez V. Association of cognitive, affective, psychomotor and intuitive domains in education, Sönmez model. Ujer 2017; 5: 347–56. doi:10.13189/ujer.2017.050307

⁸⁶ Anderson LW, Krathwohl DR. A taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives. USA: Pearson Education, 2001.

⁸⁷ Paton M, Kuper A, Paradis E, et al. Tackling the void: the importance of addressing absences in the field of health professions education research. Adv Health Sci Educ Theory Pract 2020. doi: doi:10.1007/s10459-020-09966-x. [Epub ahead of print: 06 Mar 2020 ]. http://www.ncbi.nlm.nih.gov/pubmed/32144528

⁸⁸ Pollock M, Fernandes RM, Pieper D, et al. Preferred reporting items for Overviews of reviews (prior): a protocol for development of a reporting guideline for overviews of reviews of healthcare interventions. Syst Rev 2019; 8: 335. doi:10.1186/s13643-019-1252-9 http://www.ncbi.nlm.nih.gov/pubmed/31870434

⁸⁹ Bougioukas KI, Bouras E, Apostolidou-Kiouti F, et al. Reporting guidelines on how to write a complete and transparent abstract for overviews of systematic reviews of health care interventions. J Clin Epidemiol 2019; 106: 70–9. doi:10.1016/j.jclinepi.2018.10.005 http://www.ncbi.nlm.nih.gov/pubmed/30336211

⁹⁰ Page MJ, Shamseer L, Tricco AC. Registration of systematic reviews in PROSPERO: 30,000 records and counting. Syst Rev 2018; 7: 32. doi:10.1186/s13643-018-0699-4 http://www.ncbi.nlm.nih.gov/pubmed/29463298

⁹¹ Holly C. Opportunities and challenges in a world of data abundance. JBI Database Syst RevImplement Reports 2017; 15: 2597–8. doi:10.11124/JBISRIR-2017-003629

⁹² Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med 2010; 7: e1000326. doi:10.1371/journal.pmed.1000326 http://www.ncbi.nlm.nih.gov/pubmed/20877712

⁹³ McKenzie JE, Brennan SE. Overviews of systematic reviews: great promise, greater challenge. Syst Rev 2017; 6: 185. doi:10.1186/s13643-017-0582-8 http://www.ncbi.nlm.nih.gov/pubmed/28886726

⁹⁴ Voronin DM, Saienko VG, Tolchieva HV. Digital transformation of pedagogical education at the university. Adv Soc Sci Educ Humanit Res 2020; 437: 757–63. doi:10.2991/assehr.k.200509.135

⁹⁵ Schwieger D, Ladwig C. Reaching and retaining the next generation: adapting to the expectations of gen Z in the classroom. Inf Syst Educ J 2018; 16: 45–54.

⁹⁶ Mohr KAJ, Mohr ES. Understanding generation Z students to promote a contemporary learning environment. JETE 2017; 1: 83–94. doi:10.15142/T3M05T

⁹⁷ Cilliers EJ. The challenge of teaching generation Z. People Int J Soc Sci 2017; 3: 188–98. doi:10.20319/pijss.2017.31.188198

⁹⁸ Nagy N, Kölcsey A. Generation alpha: marketing or science. Acta Technol Dubnicae 2017; 7: 107–15. doi:10.1515/atd-2017-0007

⁹⁹ Ramadlani AK, Wibisono M. Visual literacy and character education for alpha generation. In: proceedings of the International seminar on language, education and culture. Malang: Faculty of Letters, Universitas Negeri Malang, 2017: 1–7.

¹⁰⁰ dos Reis TA. Study on the alpha generation and the reflections of its behavior in the organizational environment. Quest Journals J Res Humanit Soc Sci 2018; 6: 9–19.