Which Electronic Health Record System Should We Use? A Systematic Review

Abstract

The UK government had intended to introduce a comprehensive Electronic Health Record (EHR) system in England by 2020. These EHRs would run across primary, secondary, and social care, linking data in a single digital platform. The objectives of this systematic review were to identify studies that compare EHR in terms of direct comparison between systems and to evaluate them using System and Software Quality Requirements and Evaluation (SQuaRE) ISO/IEC 25010. A systematic review was performed by searching Embase and Ovid MEDLINE databases between 1974 and April 2021. All original studies that appraised EHR systems and their providers were included. The main outcome measures were EHR system comparison and the eight characteristics of SQuaRE: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. A total of 724 studies were identified using the search criteria. After a review of titles and abstracts, this was filtered down to 40 studies as per the exclusion and inclusion criteria set out in our study. Seven studies compared more than one EHR. The following number of studies looked at the various aspects of the SQuaRE, respectively – 19 studies: functional suitability, performance efficiency: 18 studies, compatibility: 12 studies, usability: 25 studies, reliability: 6 studies, security: 2 studies, maintainability: 16 studies, portability: 13 studies. Epic was the most studied EHR system and one of the most implemented systems in the US market and one of the top ten in the UK. It is difficult to assess which is the most advantageous EHR system when they are assessed by SQuaRE’s 8 characteristics for software evaluation.

Highlights of the Study

Comparison of Electronic Health Record (EHR) systems using System and Software Quality Requirements and Evaluation.

A deeper understanding of the lag period associated with the implementation of EHR and productivity.

Identifies gender discrepancy in user satisfaction by using an EHR.

Improves knowledge of how to use EHR in research and development.

Introduction

An Electronic Health Record (EHR) is an electronic version of a patient’s medical history maintained by the provider over time. It may include all of the key administrative and clinical data relevant to that person’s care [1]. New computer technology developed in the 1960s and 1970s laid the foundation for the development of the EHR [2]. The development of EHR can be divided into two major periods. Early efforts began in the 1960s and 1970s when academic medical centres developed their systems. Beginning in the 1980s, leaders saw the benefits of industry-wide standards and began forming organizations to tackle the broader issues that would facilitate the widespread use of electronic medical information [3].

Studies have shown the advantages of EHR systems to include improving the quality and safety of care, reducing costs of research, accelerating new medical exploration, giving quick and easy access to a large amount of information in a short time, filtering information according to need, preventing and mitigating medical errors, providing information that is easy to read without handwriting problems, and sharing information among different healthcare providers [4]. Despite these advantages, there are associated drawbacks to EHR systems. These include financial cost implications, changes in workflow, temporary loss of productivity associated with the adoption of EHR, privacy and security concerns [5].

The UK previously tried to adopt the world’s first national EHR system for the entire UK via the National Programme for Information Technology (IT) [6]. This was an ambitious programme that aimed to bring the use of IT by the National Health Service (NHS) into the 21st century, through the introduction of integrated EHR systems and reforming the way that the NHS uses information, hence improving services and the quality of patient care [7]. However, after spending a vast amount of money and extensive delays, it was officially dismantled in 2011 [8]. There are several challenges when implementing EHR systems across the NHS. These include interoperability of IT systems, system installation, staff training, opportunities for patient access, consequences for the doctor-patient relationship, and data security and privacy [9]. Several sectors of the UK's health system use EHR to a varying degree. These include NHS hospitals, primary care (general practice), and social care services. In the UK, the 5-year forward plan of the NHS outlined its desire to cut paper records and go fully digital. In 2015, the NHS intended to connect EHR across primary, secondary, and social care by 2020, although this target has been delayed due to the COVID-19 pandemic [10]. The long-term plan of the NHS published in 2019 looks to improve all aspects of the NHS to prove its success for the next 10 years. It states that the NHS cannot fully embrace the opportunity offered by new technologies if many hospitals and services remain largely paper-based. Digitalizing the health systems can provide clinicians with more timely access to accurate information and support changes to help improve health for all. The digitalization part of the long-term plan of the NHS expects that all services should meet a level of core digitalization by 2024. The move to EHR underpins most of this [11].

West Hertfordshire Teaching Hospitals NHS Trust is currently undergoing redevelopment, which includes the implementation of an EHR. We performed this systematic review to identify studies that directly compare EHR systems to identify the advantages and disadvantages of different systems.

Materials and Methods

This study was performed following guidelines from the PRISMA [12], an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses [12]. The PRISMA checklist was used as a template for this systematic review.

Data Sources and Searches

Studies to be included in the review were identified by searching the following databases: (1) Embase (1974 to April 2021) and (2) Ovid MEDLINE (1974 to April 2021). There was no date limit set to the search, and the entire database was searched (Embase and Ovid MEDLINE’s archive start from 1974). All the databases were employed with the search string outlined in online supplementary Table 1 (see www.karger.com/doi/10.1159/000525135 for all online suppl. material).

System and Software Quality Requirements and Evaluation (ISO/IEC 25010)

System and Software Quality Requirements and Evaluation (SQuaRE) (ISO/IEC 25010) provides the leading models for assessing the software products [13]. The product quality model defined in ISO/IEC 25010 comprises 8 quality characteristics: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. Each characteristic is further divided into sub-characteristics as shown in Figure 1. This model was chosen to assess and compare EHR systems after the peer-review process.

Illustration

Fig. 1.

[Image omitted. See PDF.]

The eight quality characteristics of SQuaRE (ISO/IEC 25010).

Study Selection

Inclusion and exclusion criteria were decided by focus group discussions with all the authors so that the selected studies were closely aligned with the aims of this systematic review. The inclusion criteria were (i) all original studies that compared more than one EHR system and (ii) studies that examined an EHR system as per the SQuaRE’s (ISO/IEC 25010) eight characteristics mentioned above. The exclusion criteria were (i) studies that mentioned EHR or any of the search criteria as a tool for data collection only, (ii) abstract-only studies, and (iii) studies that did not specify the EHR vendor system type. Two authors (M.A. and V.P.) independently reviewed the titles and abstracts of the retrieved articles and selected publications to be included in this review. The two authors reviewed the full texts of these publications and then selected the relevant papers for the study. Cohen’s kappa was used to measure inter-rater reliability.

Data Extraction and Quality Assessment

Two authors (M.A. and V.P.) independently extracted data from the full text, including the source of the article, study design, study period, type of EHR system, functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. Study design and study quality were determined using the Oxford Centre for Evidence-Based Medicine (EBM) study design and evidence classification levels (which classifies studies into randomized control trials, cohort studies, case control, case series, or expert opinion), respectively [14, 15].

Results

Study Selection

Out of 724 potentially relevant articles retrieved, 625 were excluded following title and abstract review. Of the remaining 99 articles, 52 studies were excluded because they were abstracts only. Of the remaining 47 studies, one paper was excluded because the full paper was not in the English language. We excluded 5 papers that did not specify EHR. Also excluded from our study was a paper that did not meet our inclusion criteria and did not address any of the study selection parameters. The remaining 40 studies were included (Fig. 2; online suppl. Table 2). Agreement on the inclusion of the studies between the authors was satisfactory (κ = 0.89, SE = 0.046, 95% CI = 0.80–0.98).

Illustration

Fig. 2.

[Image omitted. See PDF.]

Selection of articles for the systematic review.

Study Characteristics

According to the Oxford Centre for EBM, the level of evidence ranged from 1 to 4 (online suppl. Table 2) [14]. The bulk of the studies were performed in the USA (n = 37), one study was a collaboration between Brazil and the Netherlands (n = 1), and the other was a collaboration between Denmark and Norway (n = 1). There was also a UK-based study (n = 1). The studies were published from 2012 until 2021, with the majority published in the last 3 years (i.e., between 2017 and 2019, n = 30). Online supplementary Table 2 shows an overall breakdown of all 40 studies included in the systematic review. It highlights which aspect of the SQuaRE’s eight characteristics was included in each study, as well as study year, study design, level of evidence, and a summary of the important findings of each study.

System Comparison

Seven studies compared more than one EHR system [16-23]. One study compared the following: Epic, Cerner, MEDITECH, McKesson, MEDHOST, Healthland, Allscripts, and others [16]. Another study compared the following: Epic, MDoffice, NextGen, Allscripts, Medflow, GE Healthcare, Cerner, ManagementPlus, MDIntelleSys, Nextech, Compulink, ManagementPlus, SRSoft, Practice Fusion, EyeMD, Integrity Digital, ModernizingMedicine, iMedicWare, IO Practiceware, and Partners HealthCare [17].

Other comparative studies included a comparison between Epic and MDIntelleSys by Palestine et al. [18], while Epic, Cerner, and Allscripts were also compared by Kamil et al. [20]. Ratwani et al. [21] compared Epic and Cerner. Crawford et al. [22] compared Cerner and Sparrow. Beauvais et al. [23] study looked at Cerner, Epic, and MEDITECH. Based on Holmgren et al. [16], Epic was associated with significantly better performance on five out of six criteria. These criteria included the following: medication orders entered using computerized provider order entry, patient’s ability to view/download/transmit their health information, view/download/transmit used, medication reconciliation, an SCR provided, and SCR sent electronically. In comparison, Cerner ranked second, as it was significantly positively associated with only three of these criteria. Beauvais et al. [23] showed how complex it was to clearly answer which system performs the best. The overriding factor is the research gap pertaining to the study and transparent disclosure of comparative studies of major EHR vendors. Their study showed that none of the EHR systems were associated with a statistically significant financial relationship. Epic was positively associated with Total Performance Score (TPS) outcomes (p = 0.04) and higher patient perceptions of quality (p = 0.003) but was negatively associated with patient safety quality scores (p = 0.03). Cerner and Epic were positively associated with improved efficiency (p = 0.01; Epic: p < 0.001). Finally, all 3 vendors were associated with positive performance in the clinical care domain (Epic: p = 0.002; Cerner: p = 0.02; MEDITECH p = 0.03).

Ratwani et al. [21] compared Cerner with Epic looking at variability in usability and safety from the two vendors across four healthcare systems (2 Epic and 2 Cerner). Although the findings were anonymized, the results of the study revealed wide variability in task duration, clicks, and accuracy when completing basic EHR functions across EHR products from the same vendor and between products from different vendors. The error rates varied widely, reaching 50% for taper orders. They also raised an important question on the usability and safety testing of EHR, given the variability in time, clicks, and accuracy.

Functional Suitability

Nineteen studies investigated functional suitability and its subdivisions (functional completeness, functional correctness, and functional appropriateness) [16, 18-20, 24-38]. There was a consensus that the implementation of EHR systems was challenging. It required intense planning and support before and during the implementation phase. Productivity also dipped in the early phase of implementation, but several studies showed that productivity improved quickly to return to baseline or to make further gains. McDowell et al. [24] showed that operating room efficiency, as measured by patients’ turnover time, decreased in the first 5 months post-implementation returning to baseline at 6 months. Wu et al. [29] showed a significant increase in the first person in operating room delay during the first-month post-implementation. This difference abated by the second month and was statistically indistinguishable from baseline by the fifth month. Read-Brown et al. [35] showed a similar dip in performance for a similar time duration. They reported that intraoperative nursing documentation time significantly increased but returned to near baseline levels within 3 months. Johnson et al. [39] reported that it took 30 months of planning to go live on an Epic platform where 17,000 employees were affected by the switch. A study that looked at the experience of the UK and Denmark in implementation reported that it took Cambridge University Hospitals NHS Foundation Trust 18 months from signing its contract with Epic to going live, while in Denmark where the contract with Epic was signed in 2013, the first implementation began in 2016 with the last of the hospitals making the switch in 2017 [25].

Performance Efficiency

Eighteen studies investigated performance efficiency, measured using the time taken to complete tasks, the number of clicks to complete tasks, and theatre utilization in different studies [17-22, 24-26, 29-33, 35, 38, 40-42]. Long-term use and familiarity with the EHR system were shown to improve efficiency. However, the overall review was poor in performance efficiency, with the most cited issues relating to the time taken to complete tasks. This was particularly challenging for those new to EHR and required extra time outside of normal working hours to complete various tasks. Kamil et al. [20] showed that most responders (n = 84, 66.1%) reported significantly or mildly decreased efficiency.

Compatibility

Twelve studies investigated compatibility [19, 25, 31, 43-51]. Hill-Kayser et al. [44] looked at the feasibility of integrating an electronic interface between Internet-based Survivorship Care Plans and Electronic Medical Records Epic. Although this required a substantial investment of time and money, it was possible to securely transfer data back and forth between the two interfaces, showcasing the interoperability of Epic. Milinovich et al. [45] examined the ability of Epic to coexist and work with Unified Medical Language System to extract and utilize health data stored by Epic for research purposes; they managed to do this successfully, resulting in substantial research support and publications.

Usability

Twenty-six studies investigated usability, and this was the most studied aspect of EHR systems [19-23, 25-27, 29-36, 38-43, 52-54]. Each 5-year decrease in the number of years since graduation was found to be associated with a 16% increase in the odds of agreeing that EHR was easy to use, a 15% increase in the odds of agreeing with the EHR was easy to learn, a 25% increase in the odds of agreeing the EHR improved clinical workflow, a 27% increase in the odds of agreeing that they like to discover new ways to use the EHR, and a 12% increase in the odds of agreeing that EHR helps them be more thorough [32].

There were mixed conclusions when it came to user satisfaction. Issues with efficiency and performance were the driving force behind user dissatisfaction with EHR usage overall. Kamil et al. [20] showed that only 37.8% (n = 48) of respondents reported they were somewhat or very satisfied with the EHR, while 40.9% (n = 52) of respondents reported that they were somewhat or very dissatisfied with the EHR. It is worth noting that this study by Kamil et al. [20] relates to satisfaction with an overall EHR amongst highly specialist clinicians and therefore may not have general applicability.

EHR documentation had a positive impact, especially with basic tasks. It improved productivity when tailored to specific speciality requirements and allowed a more streamlined experience. It also allowed for errors to be picked up and addressed more effectively. Finn et al. [30] showed that Epic Beacon implementation for documentation allowed for identifying over 500% more chemotherapy order errors (41 total errors identified pre-Beacon vs. 250 total errors identified post-Beacon). As Beacon is Epic’s oncology system, documentation may not have general applicability to hospital-wide EHR. Most users stated that the EHR had a positive impact on routine tasks, such as prescription refills (94%). In contrast, fewer agreed when it came to complex tasks, such as delivering guideline-concordant care for chronic illnesses (51%) [32]. This study also reported that providers expressed concerns about the impact of HER on in-person communication with patients [34]. Nearly two-thirds (62%) agreed that the EHR made it difficult to maintain eye contact with their patients, and 40% thought it interfered with provider-patient communication during visits.

An interesting gender difference was noted when it came to EHR user satisfaction. Female physicians demonstrated higher efficiency in completion time (difference 7.1 min; p = 0.207) and mouse clicks (difference 0.54; p = 0.13) [33]. Overall, men reported significantly higher perceived EHR workload stress than women (difference 17.5; p < 0.001). Men reported significantly higher frustration levels with the EHR compared with women (difference 33.15; p < 0.001). Women reported significantly higher satisfaction with the ease of use of the EHR interface than men (difference 0.66; p = 0.03). Women’s perceived overall usability of the EHR is marginally higher than that of men (difference 10.31; p = 0.06).

Reliability

Six studies investigated reliability [19, 23, 25, 31, 46, 54]. Hertzum et al. [25] looked at the implementation of Epic and compared preparations in Norway and the UK. They highlighted a significant reliability issue, whereby a 4-h period of unplanned downtime caused the diversion of all ambulances to other hospitals. Bornstein highlighted teething errors with availability and synching of data across remote sites [31]. Also realizing they had only paper downtime procedures which were later scanned into the electronic record, the availability of paper for downtimes was critical as this was the only source of documentation. This ultimately led to the decision to have completely redundant Citrix servers for scheduled or unscheduled outages.

Security

Two studies investigated security which was the least studied aspect of EHR systems [26, 44]. Hill-Kayser et al. [44] addressed an aspect of data encryption across an EHR platform (Epic) and an Internet-based platform. They managed to find a system of encryption that was tested and verified in a nonclinical environment. Part of this process involved encoding data as extensible markup language and transferring via Web services to the OncoLink platform. This was done using an existing custom web application called MedView, developed by the University of Pennsylvania. MedView is accessed by providers from within Epic via a single integrated sign-on.

Maintainability

Sixteen studies investigated maintainability [19, 20, 25, 28, 29, 31, 37, 44, 46-51, 53, 55]. Several studies were able to show an EHR’s ability to integrate successfully unique adjuncts to support their specific sub-speciality. Devoe et al. [51] showed the ability of Epic to integrate a Silent Best Practice Alert notification system for patient recruitment in clinical research. They found that the recruitment process allowed them to identify potential participants in real time and find more potential participants who meet basic eligibility criteria. The study concluded that Silent Best Practice Alert screening is a considerably faster method that allows for more efficient use of resources. Reagor showed the ability to integrate a dedicated perfusion electronic medical recorder with Epic [46]. As EHR’s uptake was increasing, there was less uptake of perfusion departments to adopted true perfusion EMRs. The study contends that they managed to successfully integrate Epic and the Spectrum Medical VIPER Perfusion EMR. This will allow them to achieve the ideal perfusion data acquisition model, make available in real time the appropriate variables, alert clinicians to a variation in optimal parameters, populate a database for quality monitoring and research, and have the capability of passing de-identified data to national registries.

Portability

Thirteen studies investigated portability [20, 28-31, 37, 43, 44, 47, 48, 50, 51, 55]. The unique selling point of EHR is the ability to replace multiple systems into one system across all departments/environments. Several studies looked at the challenges of introducing EHR systems in various environments and the pre- and post-implementation efforts required. Johnson et al. [28] evaluated how Vanderbilt University Medical Centre was able to replace its homegrown culture and internally developed systems with processes that were more in keeping with industry standards. They switched to the Epic system for all their core clinical information system needs, including ambulatory, inpatient, billing, and provider order entry. They were only at the optimization phase but had already started to see the benefits of an enterprise system.

Discussion

Several systematic reviews in the literature compare various aspects of EHR systems such as implementation challenges; however, these systematic reviews do not address or specify the EHR vendor and therefore their findings apply generally. This systematic review sets out, as stated in the inclusion criteria, to review only those papers that specify the vendor, and therefore attempts to make a direct comparison.

What becomes evident is there are limited studies conducted outside the USA on the subject, where Epic holds the current leading market share with 29% [56]. According to KLAS, a research and insights firm based in Salt Lake City, USA, which reports market share among the top EHR companies serving 5,457 acute care hospitals [56], Epic was the most studied EHR system; studies on Epic have examined most aspects of the system from implementation to research and development. Switching to an EHR system is no easy venture, with substantial planning and resources. An overall dip in performance is expected to occur during the transition. Intensive support across the board, especially from the IT teams, is crucial. Documentation with EHR improves efficiency; however, becoming proficient at using EHR has its drawbacks and limitations. The use of EHR in research has much potential, and the development of software to enable patient-centred care is on the horizon with improvements in technology.

A gold standard EHR system would be challenging to achieve, as evidenced by our systematic review. There are several different user groups (doctors, nurses, ward clerks, administrators, etc.) in different settings (accident and emergency, speciality wards, operating theatres, outpatient clinics, etc.) using the same system. EHR systems should have maximum IT support before and during their implementation phase. Continued training should be offered along with a feedback platform for improvements [57]. Training should start early in medical school or nursing school and be implemented into the syllabus. This will allow for a more effective transition into working life for healthcare professionals [58]. Routine tasks such as requesting scans should not be taxing or complicated with multiple steps. Flexibility in the layout and templates to allow for a diversity of use is essential across the different settings [59]. An EHR should also be efficient in its functionality to allow maximum productivity with minimal input. One drawback of EHR systems is that on average 4,000 clicks were required in a busy accident and emergency 10-h shift [60]. It concluded that Emergency Department physicians spend significantly more time entering data into electronic medical records than on any other activity, including direct patient care. Wolfe et al. [61] also reported this in their EHR review in clinical use. They stated that “Clinicians were initially confronted with decreased efficiency, increased burnout, and high turnover.” Early on, physicians using computerized order entry and electronic documentation were 30% more likely to report burnout after controlling for other variables. Automatic populating of data should be offered to help expedite tasks such as discharge summaries or templates to be used when alerting against potential adverse events. It is worth mentioning Estonia’s e-health system, called the Estonian Nationwide Health Information System. This has been operational since the end of 2008, which is arguably the forerunner in a nationwide fully integrated system. The main success factors for the e-health system in Estonia are clear governance, legal clarity, a mature ecosystem, agreement about access rights, and standardization of medical data and data exchange rules [62].

In terms of the strengths of the present report, most of the studies were conducted in well-respected centres in the USA. They are also current, and most were published in the last 3 years. Most of the results of this study came from one EHR system Epic, which can be seen as one of the limitations of this study. In addition, a meta-analysis could not be performed due to the heterogeneity of the studies. The search strategy was limited because we used system brands in the keyword search, and we did not include the grey literature. This study highlights the lack of comparison between EHR systems and the larger research profile that Epic holds across the USA; the reason for this can be attributed to the fact that Epic tends to be used by large health centres that can afford the cost. These centres also have a strong academic presence conducting large volumes of research. Thus, research and Epic are more of a correlation than causation. This is despite the market distribution of EHR use in US hospitals of which 29% is held by EPIC, 26% by Cerner, and 17% by MEDITECH [56]. While in the UK, Cerner is used in 51 inpatient hospitals and 24 specialist hospitals and EPIC is in 8th place with 10 inpatient hospitals and 4 specialist hospitals [63]. Bearing this in mind, it might not be possible to generalize the findings of this study to the UK.

There were limited data concerning cost implications, which is important in today’s NHS. A systematic review by Kruse et al. [64] showed that cost is repeatedly a primary barrier to the adoption of an EHR. It is also disappointing, considering a significant driver for conversion to EHR is the ease of automated coding and billing. Therefore, more studies should address cost implications from implementation to future cost savings and rewards in specific EHR systems. The question of cost implication becomes even more paramount in developing countries. Syzdykova et al. [65] compared five major open-source EHR systems (GNU Health, OpenEMR, FreeMED, OpenMRS, and Bahmni) following a multidimensional methodology that can provide informed recommendations to other implementers, developers, and healthcare professionals. Other relevant aspects, such as medicolegal implications, were scarcely touched on by one study in its survey findings [20].

While effectiveness in clinical governance was not examined by any of the studies, another important feature that has not been addressed by any of the studies is remote access. The ability to access patient records “off-site” to allow for remote working and consultations has become a necessity. It is now a vital feature in the modern EHR system and the current climate. Finally, more studies on UK-based hospitals are needed on systems commonly used in the UK.

Conclusions

Epic was the most studied EHR system and is one of the most implemented vendors in the USA market and one of the top ten in the UK. There are limited comparative data between EHR systems, and therefore, it is difficult to identify the most advantageous system based on SQuaRE’s eight characteristics for software evaluation. EHR systems are very important as they have an impact on the environmental and economic sustainability of medical services. The transition from paper to a fully electronic system has several implications; they include matters such as a learning curve with decreased productivity, financial implications, and an IT service overhaul with a prolonged period of support.

A transition to an EHR system will place immense pressure on institutions that run a 24-h service and have long-term implications. Therefore, it is important to have a better understanding of how these EHR vendors compare to one another to make a more informed selection of the EHR vendor.

Statement of Ethics

Not applicable because this study is based exclusively on published literature.

Conflict of Interest Statement

The authors have no conflicts of interest to disclose.

Funding Sources

None.

Author Contributions

Mohammed Al Ani: literature search, study design, data collection, data analysis, and data interpretation; George Garas: data analysis, data interpretation, and critical revision; James Hollingshead: study concept and critical revision; Drostan Cheetham: study concept and critical revision; Thanos Athanasiou: study design and critical revision; Vanash Patel: study design, data analysis, data interpretation, and critical revision. All the authors take responsibility for the integrity of the data and the accuracy of the data analysis. All the authors approved the final version of the manuscript.

Data Availability Statement

All data generated or analyzed during this study are included in this article. Further enquiries can be directed to the corresponding author.

References

1 Centers for Medicare and Medicaid Services. Electronic health records. Available from: https://www.cms.gov/Medicare/E-Health/EHealthRecords.

2 Evans RS. Electronic health records: then, now, and in the future. Yearb Med Inform. 2016;Suppl1: S48–61. http://dx.doi.org/10.15265/IYS-2016-s006.

3 Atherton J. Development of the electronic health record. Virtual Mentor. 2011;13(3): 186–9. http://dx.doi.org/10.1001/virtualmentor.2011.13.3.mhst1-1103.

4 Sadoughi F, Khodaveisi T, Ahmadi H. The used theories for the adoption of electronic health record:asystematic literature review. Health Technol. 2019;9(4): 383–400. http://dx.doi.org/10.1007/s12553-018-0277-8.

5 Menachemi N, Collum TH. Benefits and drawbacks of electronic health record systems. Risk Manag Healthc Policy. 2011;4:47–55. http://dx.doi.org/10.2147/RMHP.S12985.

6 Stone CP.Aglimpse at EHR implementation around the world: the lessons the US can learn 2014. Available from: https://www.e-healthpolicy.org/sites/e-healthpolicy.org/files/A_Glimpse_at_EHR_Implementation_Around_the_World1_ChrisStone.pdf.

7 Justinia T. The UK’s national programme for I.T: why was it dismantled?. Health Serv Manage Res. 2017;30(1):2– 9.

8 Robertson A, Bates DW, Sheikh A. The rise and fall of England’s National Programme for IT.JR Soc Med. 2011;104(11): 434–5. http://dx.doi.org/10.1258/jrsm.2011.11k039.

9 Parliamentary Office of Science and Technology. Electronic Health Records. 2016. Available from. https://researchbriefings.files.parliament.uk/documents/POST-PN-0519/POST-PN-0519.pdf.

10 NHS England. Cutting reliance on paper will make patients safer, says NHS England. Available from: https://www.england.nhs.uk/2015/09/cutting-reliance-on-paper.

11 Electronic health records. 2022. Available from: https://www.csp.org.uk/professional-clinical/digital-physiotherapy/electronic-health-records.

12 Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP,. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700.

13 Estdale J, Georgiadou E. Applying the ISO/IEC 25010 quality models to software product. Commun Computer Inf Sci. 2018:492–503. http://dx.doi.org/10.1007/978-3-319-97925-0_42.

14 OCEBM Levels of Evidence Working Group. The Oxford 2011 levels of evidence. http://www.cebm.net/index.aspx?o=5653:OCEBM Table of Evidence Working Group=Jeremy Howick, Iain Chalmers(James Lind Library), Paul Glasziou, Trish Greenhalgh, Carl Heneghan, Alessandro Liberati, Ivan Moschetti, Bob Phillips, Hazel Thornton, Olive Goddard and Mary Hodgkinson. 2011. Available from: https://www.cebm.net/wp-content/uploads/2014/06/CEBM-Levels-of-Evidence-2.1.pdf.

15 OCEBM Levels of Evidence Working Group. The Oxford 2011 levels of evidence. Oxford Centre for evidence-based medicine;2011. Available from: http://www.cebm.net/index.aspx?o=5653.

16 Holmgren AJ, Adler-Milstein J, McCullough J. Are all certified EHR created equal? Assessing the relationship between EHR vendor and hospital meaningful use performance.JAm Med Inform Assoc. 2018;25(6): 654–60.

17 Boland MV, Hwang TS, Lim MC, Peterson JL, Lum F, Lee AY. Medicare incentive payments to U.S. ophthalmologists for use of electronic health records: 2011–2016. Ophthalmology. 2019;126(7): 928–34.

18 Palestine AG, Merrill PT, Saleem SM, Jabs DA, Thorne JE. Assessing the precision of ICD-10 codes for uveitis in2electronic health record systems. JAMA Ophthalmol. 2018;136(10): 1186–90. http://dx.doi.org/10.1001/jamaophthalmol.2018.3001.

19 Koppel R, Lehmann CU. Implications of an emerging EHR monoculture for hospitals and healthcare systems.JAm Med Inform Assoc. 2015;22(2): 465–71. http://dx.doi.org/10.1136/amiajnl-2014-003023.

20 Kamil RJ, Giddings N, Hoffer M, Ying YM, Kwartler J, Brookler K,. Electronic health record use among American Neurotology Society members. Otol Neurotol. 2018;39(9): e876–82.

21 Ratwani RM, Savage E, Will A, Arnold R, Khairat S, Miller K,.Ausability and safety analysis of electronic health records:amulti-center study.JAm Med Inform Assoc. 2018;25(9): 1197–201.

22 Crawford S, Kushner I, Wells R, Monks S. Electronic health record documentation times among emergency medicine trainees. Perspect Health Inf Manag. 2019 Jan1; 16(Winter): 1f.

23 Beauvais B, Kruse CS, Fulton L, Shanmugam R, Ramamonjiarivelo Z, Brooks M. Association of electronic health record vendors with hospital financial and quality performance: retrospective data analysis.JMed Internet Res. 2021;23(4): e23961. http://dx.doi.org/10.2196/23961.

24 McDowell J, Wu A, Ehrenfeld JM, Urman RD. Effect of the implementation ofanew electronic health record system on surgical case turnover time.JMed Syst. 2017;41(3): 42. http://dx.doi.org/10.1007/s10916-017-0690-y.

25 Hertzum M, Ellingsen G. The implementation of an electronic health record: comparing preparations for Epic in Norway with experiences from the UK and Denmark. IntJMed Inform. 2019;129:312–7. http://dx.doi.org/10.1016/j.ijmedinf.2019.06.026.

26 Kirkendall ES, Goldenhar LM, Simon JL, Wheeler DS, Andrew Spooner S. Transitioning fromacomputerized provider order entry and paper documentation system to an electronic health record: expectations and experiences of hospital staff. IntJMed Inform. 2013;82(11): 1037–45. http://dx.doi.org/10.1016/j.ijmedinf.2013.08.005.

27 Whalen K, Lynch E, Moawad I, John T, Lozowski D, Cummings BM. Transition toanew electronic health record and pediatric medication safety: lessons learned in pediatrics withinalarge academic health system.JAm Med Inform Assoc. 2018;25(7): 848–54. http://dx.doi.org/10.1093/jamia/ocy034.

28 Johnson KB, Ehrenfeld JM. An EPIC switch: preparing for an electronic health record transition at Vanderbilt University Medical Center.JMed Syst. 2017;42(1): 6. http://dx.doi.org/10.1007/s10916-017-0865-6.

29 Wu A, Kodali BS, Flanagan HL Jr, Urman RD. Introduction ofanew electronic medical record system has mixed effects on first surgical case efficiency metrics.JClin Monit Comput. 2017;31(5): 1073–9. http://dx.doi.org/10.1007/s10877-016-9933-6.

30 Finn A, Bondarenka C, Edwards K, Hartwell R, Letton C, Perez A. Evaluation of electronic health record implementation on pharmacist interventions related to oral chemotherapy management.JOncol Pharm Pract. 2017;23(8): 563–74. http://dx.doi.org/10.1177/1078155216665247.

31 Bornstein S. An integrated EHR at Northern California Kaiser Permanente: pitfalls, challenges, and benefits experienced in transitioning. Appl Clin Inform. 2012;3(3): 318–25. http://dx.doi.org/10.4338/ACI-2012-03-RA-0006.

32 Makam AN, Lanham HJ, Batchelor K, Moran B, Howell-Stampley T, Kirk L,. The good, the bad and the early adopters: providers' attitudes aboutacommon, commercial EHR.JEval Clin Pract. 2014;20(1): 36–42.

33 Khairat S, Coleman C, Ottmar P, Bice T, Koppel R, Carson SS. Physicians’ gender and their use of electronic health records: findings fromamixed-methods usability study.JAm Med Inform Assoc. 2019;26(12): 1505–14. http://dx.doi.org/10.1093/jamia/ocz126.

34 Makam AN, Lanham HJ, Batchelor K, Samal L, Moran B, Howell-Stampley T,. Use and satisfaction with key functions ofacommon commercial electronic health record:asurvey of primary care providers. BMC Med Inform Decis Mak. 2013;13:86.

35 Read-Brown S, Sanders DS, Brown AS, Yackel TR, Choi D, Tu DC,. Time-motion analysis of clinical nursing documentation during implementation of an electronic operating room management system for ophthalmic surgery. AMIA Annu Symp Proc. 2013;2013:1195–204.

36 Redd T, Read-Brown S, Kitzmiller A, Yackel T, Choi D, Chiang M. Implementation of electronic health record systems in ophthalmology: impact on clinical volume compared to other medical fields. Investig Ophthalmol Vis Sci. 2013;54(15): 4426.

37 Bernstein SL, Rosner J, DeWitt M, Tetrault J, Hsiao AL, Dziura J,. Design and implementation of decision support for tobacco dependence treatment in an inpatient electronic medical record:arandomized trial. Transl Behav Med. 2017;7(2): 185–95.

38 Bush RA, Kuelbs C, Ryu J, Jiang W, Chiang G. Structured data entry in the electronic medical record: perspectives of pediatric specialty physicians and surgeons.JMed Syst. 2017;41(5): 75. http://dx.doi.org/10.1007/s10916-017-0716-5.

39 Johnson KB, Sternberg P, Dubree M. An EPIC switch: observations and opportunities after go-live.JMed Syst. 2018;42(9): 174–3. http://dx.doi.org/10.1007/s10916-018-1023-5.

40 Arndt BG, Beasley JW, Watkinson MD, Temte JL, Tuan WJ, Sinsky CA,. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations. Ann Fam Med. 2017;15(5): 419–26.

41 Chen L, Guo U, Illipparambil LC, Netherton MD, Sheshadri B, Karu E,. Racing against the clock: internal medicine residents' time spent on electronic health records.JGrad Med Educ. 2016;8(1): 39–44.

42 Cox ML, Farjat AE, Risoli TJ, Peskoe S, Goldstein BA, Turner DA,. Documenting or operating: where is time spent in general surgery residency?.JSurg Educ. 2018;75(6): e97–106.

43 Queiroz VNF, Oliveira A, Chaves RCF, Moura LAB, Cesar DS, Takaoka F,. Methodological description of clinical research data collection through electronic medical records inacenter participating in an international multicenter study. Einstein. 2019;17(4): eAE4791.

44 Hill-Kayser CE, Jacobs LA, Gabriel P, Palmer SC, Hampshire MK, Vachani C,. Feasibility study of an electronic interface between internet-based survivorship care plans and electronic medical records.JOncol Pract. 2016;12(4): e380–7.

45 Milinovich A, Kattan MW. Extracting and utilizing electronic health data from Epic for research. Ann Transl Med. 2018;6(3): 42.

46 Reagor JA.Adedicated perfusion electronic medical record with discrete epic integration.JExtra Corpor Technol. 2017;49:291–8.

47 Carter ZA, Goldman S, Anderson K, Li X, Hynan LS, Chong BF,. Creation of an internal teledermatology store-and-forward system in an existing electronic health record:apilot study inasafety-net public health and hospital system. JAMA Dermatol. 2017;153(7): 644–50.

48 Gabel E, Hofer IS, Satou N, Grogan T, Shemin R, Mahajan A,. Creation and validation of an automated algorithm to determine postoperative ventilator requirements after cardiac surgery. Anesth Analg. 2017;124(5): 1423–30.

49 Bain PA. Collaboration saves time. WMJ. 2008;107(8): 380–1.

50 Gold HT, Karia RJ, Link A, Lebwohl R, Zuckerman JD, Errico TJ,. Implementation and early adaptation of patient-reported outcome measures into an electronic health record:atechnical report. Health Informatics J. 2020 Mar;26(1): 129–40.

51 Devoe C, Gabbidon H, Schussler N, Cortese L, Caplan E, Gorman C,. Use of electronic health records to develop and implementasilent best practice alert notification system for patient recruitment in clinical research: quality improvement initiative. JMIR Med Inform. 2019;7(2): e10020.

52 Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. PLoS One. 2019;14(2): e0205379. http://dx.doi.org/10.1371/journal.pone.0205379.

53 Hayek S, Nieva R, Corrigan F, Zhou A, Mudaliar U, Mays D,. End-of-life care planning: improving documentation of advance directives in the outpatient clinic using electronic medical records.JPalliat Med. 2014;17(12): 1348–52.

54 Poulos J, Zhu L, Shah AD. Data gaps in electronic health record(EHR) systems: an audit of problem list completeness during the COVID-19 pandemic. IntJMed Inform. 2021;150:104452. http://dx.doi.org/10.1016/j.ijmedinf.2021.104452.

55 Rumball-Smith J, Fromkin J, Rosenthal B, Shane D, Skrbin J, Bimber T,. Implementation of routine electronic health record-based child abuse screening in General Emergency Departments. Child Abuse Negl. 2018;85:58–67.

56 Epic dominates EMR market share wars; cerner loses ground. Available from: https://www.healthleadersmedia.com/innovation/epic-dominates-emr-market-share-wars-cerner-loses-ground.

57 Robinson KE, Kersey JA. Novel electronic health record(EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout. Medicine. 2018;97(38): e12319. http://dx.doi.org/10.1097/MD.0000000000012319.

58 Training medical students in EHR usage. Available from: www.healthmanagement.org/c/healthmanagement/issuearticle/training-medical-students-in-ehr-usage.

59 Stanhope V, Matthews EB. Delivering person-centered care with an electronic health record. BMC Med Inform Decis Mak. 2019;19(1): 168. http://dx.doi.org/10.1186/s12911-019-0897-6.

60 Hill RG, Sears LM, Melanson SW. 4000 Clicks:aproductivity analysis of electronic medical records inacommunity hospital ED. AmJEmerg Med. 2013;31(11): 1591–4.

61 Wolfe L, Chisolm MS, Bohsali F. Clinically excellent use of the electronic health record: review. JMIR Hum Factors. 2018;5(4): e10426. http://dx.doi.org/10.2196/10426.

62 Metsallik J, Ross P, Draheim D, Piho G, editors. Ten years of the e-health system in Estonia[online]. 2019. Available from: https://zbook.org/read/e1562_ten-years-of-the-e-health-system-in-estonia.html.

63 KLAS Research. Global(non-US) EMR Market share 2020 regional vendor energy rising: KLAS report. 2020.

64 Kruse CS, Kothman K, Anerobi K, Abanaka L. Adoption factors of the electronic health record:asystematic review. JMIR Med Inform. 2016;4(2): e19. http://dx.doi.org/10.2196/medinform.5525.

65 Syzdykova A, Malta A, Zolfo M, Diro E, Oliveira JL. Open-source electronic health record systems for low-resource settings: systematic review. JMIR Med Inform. 2017;5(4): e44. http://dx.doi.org/10.2196/medinform.8131.