Health technology assessment frameworks for eHealth: A systematic review

Abstract

Objectives

Traditionally, health technology assessment (HTA) focuses on assessing the impact of pharmaceutical technologies on health and care. Resources are scarce and policy makers aim to achieve effective, accessible health care. eHealth innovations are increasingly more integrated in all healthcare domains. However, how eHealth is assessed prior to its implementation in care practices is unclear. To support evidence-informed policy making, this study aimed to identify frameworks and methods for assessing eHealth's impact on health care.

Methods

The scientific literature in five bibliographical databases was systematically reviewed. Articles were included if the study was conducted in a clinical setting, used an HTA framework and assessed an eHealth service. A systematic qualitative narrative approach was applied for analysis and reporting.

Results

Twenty-one HTA frameworks were identified in twenty-three articles. All frameworks addressed outcomes related to the technical performance and functionalities of eHealth service under assessment. The majority also addressed costs (n = 19), clinical outcomes (n = 14), organizational (n = 15) and system level aspects (n = 13). Most frameworks can be classified as dimensional (n = 13), followed by staged (n = 3), hybrid (n = 3), and business modeling frameworks (n = 2). Six frameworks specified assessment outcomes and methods.

Conclusions

HTA frameworks are available for a-priori impact assessment of eHealth services. The frameworks vary in assessment outcomes, methods, and specificity. Demonstrated applicability in practice is limited. Recommendations include standardization of: (i) reporting characteristics of eHealth services, and (ii) specifying assessment outcomes and methods following a stepped-approach tailored to the functional characteristics of eHealth services. Standardization might improve the quality and comparability of eHTA assessments.

Health care faces serious challenges when it comes to the economic sustainability of the system. Decisions are made on how to achieve and maintain effective and accessible health care. Thorough impact assessment of existing and new healthcare practices on relevant outcomes is warranted to ensure delivery of the right care at the right time, to the right person. Impact can be understood as the (un)intended consequences that healthcare practices have in different health and care outcomes, including efficacy, safety, effectiveness, costs, and other care provision-related aspects. During the 1980s, health technology assessment (HTA) grew into a discipline, producing and summarizing evidence about efficacy and efficiency of predominantly pharmaceutical innovations (1–4). Over the years, HTA extended the conceptual and methodological assessment of clinical outcomes and cost-effectiveness of pharmaceuticals to include patient perspectives, organizational dimensions, and other societal aspects (5;6). In parallel, the role of technology in health care gradually expanded from a biomedical orientation toward a more holistic approach that includes medical devices such as imaging tools, robotics, and digital healthcare solutions.

Currently, novel interventions are commonly assessed through formal HTA, evaluating mostly clinical outcomes in order to realize evidence-informed decision and policy making. In general, HTA apply frameworks that involves mostly quantitative intervention properties (7). Such frameworks specify methods for assessing the qualities of the intervention under study, including comparative effectiveness research (CER), systematic, and meta-analytic reviews on the clinical effectiveness of an intervention expressed in numbers-needed-to-treat or cost-effectiveness estimates such as incremental cost-effectiveness ratios (7;8).

Various national and international organizations are engaged in HTA and involved in creating or guiding the development of standards for the evidence required including for digital technologies. For example, National Institute for Health and Care Excellence in England (NICE) recently developed an Evidence Standards Framework for Digital Health Technologies (9). This framework provides technology developers with standards of the evidence demonstrating health technologies' (cost)-effectiveness in the UK health and care system. Zorginstituut Nederland (ZIN, the Dutch National Health Care Institute) has statutory assignments to the systematic assessment if healthcare interventions are being deployed in a patent-oriented, clinically effective and cost-effective manner. Such analyses are part of its flagship program “Zinnige Zorg” (appropriate care), specifically designed to identify ineffective and unnecessary care across a number of themes, including patient-centeredness, shared decision making, and approaches to stepped care (10;11). ZIN currently hosts the secretariat of EUnetHTA, a collaborative network aimed to produce and contribute to HTA in Europe. Following the trends of professionalization and widening the focus, HTA practice evolved to structural assessments of the impact of a variety of health innovations across a multitude of health and care organization-related outcomes.

eHealth refers to the organization, delivery, and innovation of health services and information using the Internet and related digital technologies (12;13). Being a container concept, it includes a broad spectrum of digital technologies that purport to assess, improve, maintain, promote or modify health, functioning, or health conditions through diagnostic, preventive, and treatment interventions in somatic and mental health care. eHealth is expected to improve accessibility, affordability and quality of health care (14). eHealth is debated to diverge from traditional health care due to its technological properties, speed of development, and complexity of implementation in existing routine care, as eHealth often impacts interlocking levels of organizational, staff, and client behaviors, beliefs, and norms (15;16). Meanwhile investments in eHealth are rising, the urgency of healthcare transformations is pressing and the use of a wide array of both proven and unproven eHealth applications is growing.

Discussions on the necessity of eHealth specific HTA frameworks are ongoing. In general, technology evolves quickly and impact assessment has often been perceived as a hindering factor. A certain level of maturity of the technology is required for which evidence has been provided to inform decision making. Studies have investigated the use of Rapid Relative Effectiveness Assessments (REAs) to address these concerns and reduce the required time for obtaining evidence and conducting the assessments (17). However, these approaches primarily focus on effectiveness (under routine care conditions) and safety and leave out topics deemed more context dependent such as costs and organizational aspects to improve comparability across healthcare settings.

Nevertheless, the necessity of incorporating eHealth in traditional care practices, and the inherent complexity of achieving sustainable change, requires a multi-perspective and multi-method approach for assessing their impact on relevant outcomes. Ultimately, eHealth is about health and care of and for real people. There should be no exceptions in conducting rigorous assessment of the impact of eHealth services on health and care. It is unclear how eHealth services can be assessed systematically, to inform and support decision making in policy and practice prior to their implementation (16). Contrary to traditional HTA, assessments of eHealth services (eHTA) are less forged in structured approaches. Reasons may include the diverse and rapidly evolving nature of the technological properties and the amalgamative nature of eHealth, referring to the interconnectedness of such services with behavioral, cultural, and organizational aspects of healthcare delivery (11;15;18;19).

We conducted a systematic review of the scientific literature to answer the following question: which frameworks are available to assess the impact of eHealth services on health and care provisions? We regard assessment frameworks as providing a conceptual structure to the assessment but not necessarily specifications of the methods and evaluation instruments that should be used to conduct the actual assessments. A conceptual structure includes for example assessing the effectiveness, safety, and the required technical infrastructure of an eHealth service. An example of the methods and instruments used such concepts include conducting randomized-controlled trials to measure change in blood levels as an indication of symptom reduction, or perform data transfer speed tests to establish the required band-width. Because of this nuance, we also identified the specific assessment methods and instruments proposed by the frameworks.

Methods

A broad search strategy was applied with high sensitivity to two key terms: “eHealth” and “health technology assessment.” A total of 155 synonyms were formulated, resulting in a fine-grained search string for use in five online bibliographical databases: PubMed (Medline), Cochrane (Wiley), PsychINFO (EBSCO), Embase, and Web of Science (Thomson Reuters/Clarivate Analytics). A trained librarian guided the development of the search strings (see Supplementary file 1).

All identified papers were examined for eligibility by two researchers (CV and LB) independently. Disagreements were solved by discussion. The following inclusion criteria were applied: (a) the assessment was conducted in a clinical setting, (b) used an explicit HTA framework, and (c) the clinical intervention included a digital technological component. Articles were excluded if:

• The primary aim was to establish clinical efficacy, effectiveness, cost-effectiveness, feasibility, piloting, usability testing, or design of eHealth services.

• Services included in the assessment can be categorized as medical devices (assistive technology, imaging, and surgical devices), information systems (electronic health records, decision making tools, scheduling systems, information resources, training, and education), or pharmacological interventions.

• They contain implied viewpoints, commentaries, editorials, study protocols, proposals, presentations, posters, HTA-use reviews, and/or development proposals.

• The full text article was not available or not in English.

A standardized data extraction form was developed to extract relevant data from the remaining articles, including:

• General study characteristics.

• eHealth service assessed (participants, inclusion criteria, intervention aim and working principles, target disorder, technological principles, outcomes, and setting).

• HTA framework used (purpose, structure, advantages, methods, evaluation dimensions and criteria, working principles, and intended users).

• Methods and instruments applied (type, aims, purpose, instruments, data collection methods, and link to framework).

A systematic qualitative narrative approach was applied for analysis (20–22). Commonalities across the frameworks were examined to structure the analysis in terms of what, when, and how specific properties of eHealth services can be assessed.

Results

The searches, which were performed in March 2018, resulted in 3,915 articles. After removing duplicates, 2,068 titles and abstracts were screened for eligibility. Figure 1 provides an overview of the identification and selection of studies in different phases of the screening.

Fig. 1.

PRISMA flowchart illustrating study identification and selection process.

[Figure omitted. See PDF]

eHTA Frameworks

The twenty-three articles included in this review report in total twenty-one distinctive eHTA frameworks. Four themes classifying the frameworks emerged from the analysis of the included literature: (i) staged (n = 3, references: 23;8;24), (ii) dimensional (n = 13, references: 6;25–38), (iii) hybrid (n = 3, described in references: 39–41), and (iv) business modeling (n = 2, references 42;43). Table 1 provides an overview of the frameworks identified according to their classification. All frameworks included in this review are summarized in Table 2.

Table 1. Classification of the frameworks identified in the review

Framework classification	Technical	Clinical	Economic	Organizational	Ethical and legal	Care system	Information quality	Needs
Staged frameworks (n = 3)	3	1	3	3	–	1	–	1
Dimensional frameworks (n = 13)	13	9	11	9	3	8	1	–
Hybrid frameworks (n = 3)	3	3	3	2	3	2	–	1
Business modeling frameworks (n = 2)	2	1	2	1	–	2	–	–
Total (N = 21)	21	14	19	15	6	13	1	2

n and N refer to the number of unique frameworks found in this review.

Columns 2–10 indicate the assessment dimensions:

• Technical: aspects related to the technical characteristics of the service.

• Clinical: aspects of clinical outcomes including effectiveness, well-being, and safety.

• Economic: aspects of cost and cost-effectiveness.

• Organizational: aspects of an organizational nature such as training, resources, procedures, etc.

• Ethical and legal: ethical and legal aspects such as data protection.

• Information quality: the quality of the information included in the assessment.

• Needs: aspects defining the needs of stakeholders regarding the service under assessment.

Table 2. Main characteristics of the eHTA frameworks identified in the systematic review

Name	Type and assessment dimensions	Purpose and general approach to the assessment
[Device Selection Matrix] (23)	Type: staged	Purpose: to evaluate and compare candidate ICT devices on the basis of established criteria
	Assessment: Technical, economic, organizational	Provides an unweighted decision list of parameters including and aligned with needs identified in the first phase. Parameters include usability, robustness, size of the unit, ease of setup, cost, weight, and availability
[Staged approach to evaluation of telemedicine] (8)	Type: staged	Purpose: to systematically evaluate telemedicine to inform and foster the (further) development of telemedicine technologies
	Assessment: Technology, clinical, economic, care system	The framework discerns four stages: (i) technical efficacy (accuracy and reliability in ability to transmit accurate information), (ii) specific system objectives (to determine feasibility; single end points in domains of access, quality, or cost), (iii) system analysis related to the global impact on access, quality, and costs for system, and (iv) external validity concerning the impact on access, quality, and costs in a different system
[Stepped evaluation of eHealth services] (24)	Type: staged	Purpose: to identify specific properties in similar eHealth services and compare them with one another to help decision-makers to select and prioritize eHealth services prior to purchase, development, or implementation in today's health and social care organizations
	Assessment: technical, economic, organizational	The framework distinguishes six steps: (i) identify and sort goals and effects, (ii) determine requirements, needs, and preconditions (technical, operational, and financial), (iii) economic impact (i.e., cost estimation in direct, adjustable, and introduction costs), (iv) implementation strategies, (v) rewards and incentives, (vi) get a comprehensive picture of the service, that is, summary of steps i–v in terms of objectives, effects, technical, operational, financial requirements, costs, dissemination, and reward model
[Three-dimensional model for telemedicine evaluation] (6)	Type: Dimensional	Purpose: to produce objective and credible evidence regarding the merits and problems of telemedicine
	Assessment: Technical, care system	Assessment of the application (public health, education, and clinical), perspectives (client, provider, and society), technological characteristics (synchronous online systems, asynchronous store-and-forward systems, modalities of transmission, bandwidth, peripheral devices for diagnosis, and treatment)
[Multi-method telemedicine application evaluation] (25)	Type: Dimensional	Purpose: to assess a telemedicine application's utilization, clinical and organizational impact, technical functioning, and cost-effectiveness
	Assessment: Technical, clinical, economic, organizational	Assessment of utilization (nature and frequency), clinical impact on clinical decision making and clinical care, organizational context, technical performance, costs and cost-effectiveness, triangulation (i.e., the ways in which dimensions i–v may have influenced and been influenced by the others)
Technology, Economic, Market, Political, Evaluation, Social and Transformation (TEMPEST) (26)	Type: Dimensional	Purpose: to compare and contrast benefits and barriers in health technology adoption and diffusion
	Assessment: Technical, economic, care system	Assessment of access and usage (enabling/emerging technologies, interoperability of eHealth, and eHealth service delivery model), economic factors (healthcare funding, performance and population, and labor market segmentation), market (market-driven health care, consumer-driven health care, IT market capabilities and skills), political priorities and barriers (eHealth policy, education and training, and institutional structure), health technology evaluation (i.e., eHealth policy governance, regulation and compliance, eHealth adoption/user engagement, performance measurement, and benefits realization), social (social inclusion/access, patient-centered health care, and demographics), and transformational aspects (i.e., education, training, reform agenda, strategy, and implementation)
Unified eValuation using Ontology (UVON) (27)	Type: Dimensional	Purpose: to evaluate one or more health information systems by organizing, unifying and aggregating the quality attributes extracted from those systems
	Assessment: Technical, clinical, economic, organizational, ethical and legal, care system	Assessment of accessibility, adherence, affordability, authenticity, availability, efficiency, effectiveness, empowerment, safety, and ability to trust
Health Services Research framework (HSR) (28)	Type: dimensional	Purpose: to assess the impact of telemedicine on access, quality, and costs of care
	Assessment: Technical, Clinical, Economic, Organizational	Assessment of accessibility and timeliness of care, costs of care, quality of care in terms of (a) structure (speed and technical quality of transmission, adequacy of equipment, skills, costs, and accessibility), (b) process (sensitivity and specificity of diagnosis, and evidence-base of treatment plan), and (c) outcome (short-term: clinical outcomes; intermediate: adherence, acceptability, and satisfaction; long-term: quality of life, health, or functional status)
[Telehealth Evaluation Framework] (29)	Type: Dimensional	Purpose: to develop a body of knowledge around telehealth evaluation
	Assessment: Technical, clinical, economic, organizational, care system	Assessment of individual structure (i.e., patient access to services, patient acceptability, provider training, and provider change in practice), organizational structure (scheduling, equipment location suitability, culture, costs, and equipment effectiveness), care process (satisfaction, effectiveness of interaction, and management of care process), individual outcomes (satisfaction, quality of life, functional status, number of re-admissions, and adverse effects), and organizational outcomes (use of resources, cost effectiveness, and utilization)
Commonwealth Scientific and Industrial Research Organisation framework (CSIRO) (30)	Type: Dimensional	Purpose: to design delivery, implementation and evaluation of telehealth services
	Assessment: Technical, economic, organizational, care system	Assessment and classification of health domain (i.e., application area), health services (consultation, diagnosis, monitoring, triage, mentoring, training/education, and treatment), technology (postal mail, telephone; email, fax; store and forward, real-time, hybrid; integrated video and data; assistive and censoring; and interactive telepresence), communication technology (telephone lines, television lines, fiber optics, wireless, and satellite dish), environmental settings (people involved, locations, communication modes, and devices used), and socioeconomics (costs, benefits, barriers to uptake, outcomes such as early diagnosis, information flow, reduced delays, safety feasibility, and improved patient care)
[Assessment of telemedicine applications] (31)	Type: Dimensional	Purpose: to provide a broad description of telemedicine for decision-makers, covering technical, clinical, economic, ethical, legal, and organizational issues
	Assessment: Technical, clinical, economic, information quality	Assessment of technical aspects (technical quality of image/voice, reliability, validity, and other characteristics), effectiveness (diagnostic quality, quality of life, clinical, management process, know-how, and non-health patient outcomes), user assessment (quality, usability, and satisfaction with the technology), costs (investment, user charge for equipment, communication lines, staff wages, and education costs), other relevant costs (housing, administration, etc.), patient costs, patient lost working hours/leisure time, health-related intangible costs), study design (randomization, before/after comparison, and control groups), economic evaluation methods (costs, cost-effectiveness, cost–benefit ratio, and cost-utility analysis), and sensitivity analysis
CHEATS (32;33)	Type: Dimensional	Purpose: to provide a comprehensive framework from which aspects can be drawn and parts utilized to evaluate any kind of ICT system
	Assessment: Technical, clinical, economic, organizational	Assessment of clinical aspects (quality of care, diagnostic reliability, impact and continuity of care, acceptance, changes in work practices and resources, acceptance and efficacy, cultural differences, interviewing techniques, and effectiveness of referral), human and organizational aspects (interfaces between care providers), educational (recruitment and retention of staff, training provisions, acceptability, and continuity), administrative (access to care, change in interaction styles, and cost-effectiveness), technical (appropriateness, video/sound quality, differences in techniques, ease of use, technology-specific training, and reliability), and social interactions
[Eight-dimension sociotechnical model of safe and effective IT use] (34)	Type: Dimensional	Purpose: to evaluate the design, development, implementation, use and monitoring of health IT within complex healthcare systems
	Assessment: Technical, organizational, ethical and legal, care system	Assessment of hardware and software in relation to safe and effective use of ICT, clinical content (data, information, and knowledge), human–computer user interface, personnel involved in design, development, implementation, use, and management of IT-enabled health care, workflow and communication, organizational policy, procedure, culture, environment, external rules and regulations, and measurement and monitoring
[Telemedicine evaluation plan] (35)	Type: Dimensional	Purpose: to assess and continue to develop and assess telemedicine applications in the context of Greenland
	Assessment: Technical, clinical, economic, organizational	Assessment of expectations and reactions of professional users, reactions of clients, logistics, organization and technology, medical gain, waiting time for patients, patient travel, economy (in addition, also of staff as well as direct costs and admission costs), transferring competence, and recruiting/retaining staff
Clinical Value Compass (36)	Type: Dimensional	Purpose: to evaluate changes in clinical processes
	Assessment: Technical, clinical, economic	Assessment of health-related quality of life (physical and emotional aspects), patients' satisfaction (about modality and technological aspects), costs (i.e., ER admissions and hospitalizations), and clinical status (disability, relapse, and severity of symptoms)
Telemedicine Quality Control system (TM-QC) (37;38)	Type: Dimensional	Purpose: to monitor and measure the characteristics of telemedicine products and services
	Assessment: Technical, clinical, economic, ethical and legal	The framework describes two assessment phases, each addressing different outcomes. Phase I concerns a preliminary evaluation of features, classification (area of application and users), preliminary evaluation, and areas for improvement. Phase II includes a technical file (TF) detailing (i) performance and functionalities (general description of the product/system, system design, requirements, security standards, and product-building methods, correspondence between requirements and documents in the TF, risk analysis, clinical evaluation, documentation, maintenance), and (ii) a quality assessment checklist (product requirements related to (a) patient safety, data privacy and integrity, transmission security, system requirements, technical service of the product, software certification, standards used, documentation, medico-legal implications and legal validity, efficacy of the telemedical health service, and communication with users, (b) product design, manufacturing, and testing, and (c) evaluation of economic and social aspects)
Khoja–Durrani–Scott Evaluation Framework (KDS) (39)	Type: Hybrid	Purpose: to provide a comprehensive platform for developing an evaluation tool for eHealth programs
	Assessment: Technical, clinical, economic, organizational, ethical and legal, care system	The framework describes seven themes assessed in four stages of the eHealth life cycle: development, implementation, integration, and sustained operation. The seven themes include: health services outcomes (health status and quality of life), technology outcomes (appropriateness, relevance, use, safety, and effectiveness), economic outcomes (affordability relative to willingness-to-pay), behavioral and sociotechnical outcomes (intended/unintended social consequences, planned interventions, and social change processes), ethical outcomes (arising from clinical practice, research, resource allocation, use, and access to technology), readiness and change outcomes, policy outcomes (set of statements, directives, regulations, laws and judicial interpretations required to facilitate structured and consistent eHealth practice)
Model for Assessment of Telemedicine applications (MAST) (40;41)	Type: Hybrid	Purpose: to describe effectiveness and contribution of telemedicine applications to quality of care and to produce a basis for decision making by using a multidisciplinary process which summarizes and evaluates information about medical, social, economic, and ethical issues related to the use of telemedicine in a systematic, unbiased, robust manner
	Assessment: Technical, clinical, economic, organizational, ethical and legal, care system, needs	The framework distinguishes between three phases, each addressing different outcomes. The first phase concerns preceding considerations including legislation, reimbursement, maturity of technology, and number of patients. The second phase concerns a multidisciplinary assessment addressing health problem and characteristics of the application, safety, clinical effectiveness (mortality, morbidity, quality of life, behavioral outcomes, and usage of health services), patient perspectives (satisfaction and acceptance, understanding of information, confidence in the treatment, ability to use the application, access and accessibility, empowerment, and self-efficacy), economic aspects (resources and costs, related changes in use of health care, clinical effectiveness, and business case), organizational aspects (process, structure, culture, and management), and socio-cultural, ethical and legal aspects. The third phase concerns transferability issues, that is, potential for expansion to other disorders and/or systems
[Comprehensive telemedicine evaluation model] (42)	Type: Business modeling	Purpose: to examine the extent to which changes in transactions result in an improvement or deterioration in the health of the population, associated costs, and access to that system
	Assessment: Technical, clinical, economic, care system	The framework discerns three different areas of attention: (1) analysis level (individual, community, or society), (2) focus of analysis (i.e., driving forces of healthcare-related to quality, accessibility, cost, and acceptability), (3) different uses (administrative, educational, intensive care, midlevel care and home care)
Innovating in Healthcare Framework (43)	Type: Business modeling	Purpose: to assess business models to innovate health care
	Assessment: Technical, economic, organizational, care system	The assessment framework has three parts: (i) the three distinctly different types of healthcare innovation (i.e., consumer, technology, and integrator-based ventures), (ii) six-factor alignment, that is, the idea viable (in terms of structure, financing, public policy, technology, consumers, and accountability), and (iii) business model elements, that is, how to make it happen (in terms of strategic market assessment, financial viability, valuation analysis, sustainability, managerial assessment, societal impact, and risk assessment)

[]: Square brackets indicate that the name of the framework is not explicitly mentioned by the authors but is derived by analyzing the manuscript.

Staged: means that the framework concerned predominantly applies a staged approach to eHealth Technology Assessment.

Dimensional: indicates that the framework concerned predominantly applies a dimensional approach to eHealth Technology Assessment.

Hybrid: indicates that the framework concerned can be categorized as combining a staged with a dimensional approach to eHealth Technology Assessment.

Business modeling: indicates that the framework concerned focuses solely on business modeling and economic aspects of eHealth Technology Assessment.

Staged Frameworks

Staged frameworks apply a sequential phased approach for assessing outcomes relevant to the developmental phase of the eHealth service. Three frameworks advocate a staged approach, two of which focus on providing guidance on comparing and selecting eHealth devices, and one on informing (further) development of eHealth services. To highlight these different approaches, two frameworks are explained in more depth. For example, Casper and Kenron (23) take a socio-technical systems approach, aligning users' values with functional elements of the eHealth service. The framework has four phases: (i) development of project needs, (ii) survey of potential eHealth services, (iii) evaluation of candidate services, and (iv) selection of the service. Important considerations in the selection process include technological aspects, user-, and environmental factors. For phase three, an unweighted decision list is based on the needs identified in phase one addressing usability, robustness, size and weight, ease of setup, costs, and availability of the eHealth solution.

DeChant et al. (8) argue that besides the clinical and technological performance of eHealth services, the overall impact on health care should also be assessed. This is because eHealth services, when introduced in routine care, potentially affect a range of aspects of care delivery including care pathways and access to care (8). Consequently, their framework consists of four successive stages addressing (i) technical efficacy, (ii) healthcare system objectives, (iii) analysis, and (iv) external validity. Each stage focuses on the impact of eHealth services on the quality, accessibility, or costs of care. For each stage, the assessment is tailored to the maturity of the eHealth service.

Dimensional Frameworks

Dimensional frameworks prescribe sets of assessment outcomes grouped in dimensions. The dimensions are categorized in accordance with expected impacts of eHealth services, irrespective of their developmental stage. All thirteen identified frameworks focus on assessment of technical features, clinical, and economic outcomes, with few also addressing organizational, ethical, and system level outcomes. Full details are provided in Table 2. Two frameworks are detailed below due to their distinctive features: the Technology, Economic, Market, Political, Evaluation, Social and Transformation framework (TEMPEST) (26) because of its comprehensiveness, and the Telehealth Evaluation Framework (29) which explicates the importance of different perspectives in assessing eHealth services.

The TEMPEST framework specifies seven domains for comparing benefits of and barriers to eHealth service adoption (26). The domains concern outcomes of (i) access, usage, emerging technologies, interoperability, and the delivery model, (ii) economy, including funding, performance, and human resources, (iii) engagement and use in terms of the healthcare market, (iv) policy-related matters, (v) governance, regulation, and compliance, (vi) sociological aspects such as care access, and (vii) care reformation, strategies, and implementation. These seven domains are divided into twenty-one sub-themes with eighty-four quantitative outcomes covering different perspectives, including a market, political, commercial, stakeholders, and individuals.

Hebert (29) places the perspective from which an assessment is conducted central in defining the dimensions for evaluation. Building on Donabedian's work (44), the outcomes relate to individual and organizational assessment perspectives. The underlying premise is a relationship between structure–process–outcome. The framework identifies five domains: (i) individual structure including patients' access and acceptability, and providers' training, and changes in practice, (ii) organizational structure related to scheduling, infrastructure, culture, costs, and equipment effectiveness, (iii) care process concerning satisfaction, effectiveness, and care management, (iv) individual level outcomes such as patient satisfaction, quality of life, functional status, and adverse effects, and (v) organizational level outcomes including resource use, costs, and service utilization.

Hybrid Frameworks

Hybrid frameworks combine a phased perspective on service development with an assessment of the current impact of an innovation using varying sets of criteria to be assessed in a particular order. Three frameworks can be classified as hybrid.

The TM-QC framework (37;38) brings a technological and quality assurance perspective to eHealth assessments and applies two phases. The first phase comprises seven dimensions for classifying the service. For the second phase, a technical dossier is compiled detailing system design, requirements, security standards, risk analysis, clinical evaluation, and maintenance aspects. In addition, a quality assessment is performed on three dimensions: (i) product requirements regarding patient safety and privacy, (ii) product design, manufacturing, and testing, and (iii) economic evaluation and social aspects. Following the quality assessment, a score is calculated and compared to a predefined threshold.

The Khoja–Durrani–Scott Evaluation Framework (KDS) (39) aligns its evaluation themes to the eHealth services' life-cycle phases comprising: (i) development, (ii) implementation, (iii) integration, and (iv) sustained operation. Across these phases, seven groups of outcomes are assessed: (i) health outcomes, (ii) technology including appropriateness, relevance, use, safety, and effectiveness of the service, (iii) economy related to affordability and willingness-to-pay, (iv) behavioral and sociotechnical outcomes covering (un)intended social consequences and social change processes, (v) ethical aspects, (vi) readiness and change outcomes, and (vii) policy outcomes concerning the facilitation of consistent eHealth service delivery. For each of the assessment outcomes specific evaluation methods are included (KDS tools) (45).

The Model for Assessment of Telemedicine applications (MAST) (40;41) defines a three-phased assessment of: (i) preceding considerations to determine the relevance of an assessment, (ii) a broad range of outcomes structured in seven domains, and (iii) transferability to understand the potential for scaling-up or -out. For phase one, issues regarding relevant regulatory aspects (financial, maturity, and potential use) are assessed, addressing questions about the purpose, alternatives, required level of assessment (international, national, regional, and local), and the maturity of the eHealth service. Phase two is based on the EUnetHTA Core Model (46;47) covering seven domains: (i) the health problem targeted, (ii) clinical and technical safety, (iii) clinical effectiveness, (iv) patient perspectives, including satisfaction, acceptance, usability, literacy, access, empowerment, and self-efficacy, (v) economic evaluation addressing costs, related changes in use of health care, and a business case, (vi) organizational aspects including procedures structure, culture, and management aspects, and (vii) further socio-cultural, ethical, and legal issues. The third phase focuses on assessing the potential to effectively transfer the eHealth service to other healthcare systems and its scalability in terms of throughput and costs.

Business Modeling Frameworks

Where the previous frameworks provide a list of outcomes to consider in assessing eHealth services, business modeling frameworks in this context focus on the economic viability and business models for eHealth services.

Alfonzo et al. (42) places transactional distances in providing and receiving care central in their framework. Transactional distance refers to any factor having an impact on an interaction that creates distance between the parties, such as education, culture, ethnicity, gender, health status, geography, etc. (42). The purpose of the assessment is to examine the extent to which changes in the transactions result in an improvement or deterioration in health, the associated costs, and access to care.

The Innovating in Healthcare Framework proposed by Grustam et al. (43) takes a business modeling approach to the assessment of eHealth services. The framework consists of three parts: (i) types of healthcare innovation, (ii) a six-dimension assessment, and (iii) business model elements (43). The assessment includes evaluation of the (i) structure of the system, (ii) financing mechanisms, (iii) regulatory public policies, (iv) technological, developmental, and competitive aspects, (v) consumer empowerment, and (vi) accountability. Business modeling includes assessing the (i) market, (ii) financial viability, (iii) valuations regarding cash flows and rates of return, (iv) financial sustainability, (v) managerial skills and requirements, (vi) societal impact, and (vii) technological risks.

Assessment Methods and Instruments

Specific instruments for outcome assessment and data collection methods were reported for six frameworks. Four frameworks specified only the methods for data collection and performing the assessments. Another four frameworks include a list of specific instruments that can be used to assess the outcomes, but not the methods for collecting and evaluating the data. The frameworks, their methods, and instruments are listed in Table 3.

Table 3. Instruments, methods, and outcomes defined by each framework

Name	Purpose and specification	Outcomes only	Methods only	Methods + outcomes
[Three-dimensional model for telemedicine evaluation] (6)	The purpose of the suggested methods is to assess quality dimensions in a controlled environment. Methods include controlled studies, for example, randomized-controlled trial, addressing two types of research questions that are appropriate for telemedicine evaluation: safety, effectiveness, access, quality, and costs. The framework does not define concrete instruments to be used		X
[Multi-method telemedicine application evaluation] (25)	The framework suggests using a multi-method technique to evaluate various aspects of the telemedicine application. Methods are of a descriptive nature; qualitative and quantitative data and analyzed using various techniques; each component employed different methods of evaluation (although the clinical and organizational components employed multiple methods of enquiry, the results from which were analyzed in relation to one another)		X
[Device Selection Matrix] (23)	To assess the usability of a device in a specific setting using the decision matrix. Methods applied include Likert scale ratings and sum scores			X
Technology, Economic, Market, Political, Evaluation, Social, and Transformation (TEMPEST) (26)	The TEMPEST framework provides eighty-four outcomes to assess the evaluation criteria. The outcomes cover the whole range of evaluation criteria	X
[Staged approach to evaluation of telemedicine] (8)	The framework proposes assessing quality, costs, and accessibility of telemedicine applications at different levels of development. Various methods are suggested ranging from experimental to observational methods depending on stage of assessment		X
Unified eValuation using Ontology (UVON) (27)	Unified eValuation using Ontology for assessing the quality criteria of a specific system for different stakeholders. The assessment instruments involve a questionnaire based on the ten quality criteria formulated by the framework. Two versions of the questionnaire are included: one for the patients and one for the health professionals, each expressing the same concept using a different vocabulary			X
Telemedicine Quality Control system (TM-QC) (37;38)	Informative Questionnaire (InQu): preliminary analysis; to collect from the suppliers structured information on their telemedicine applications with emphasis on the architecture; Classification Form (CF): to delimit application areas of product; Technical File (TF): technical dossier of the product, a document that describes all phases of product manufacturing, from design to post-release assistance; Quality Assessment Check List (QACL): assessing the telemedicine product or service quality. The instruments include a questionnaire and check lists			X
Health Services Research framework (HSR) (28)	This framework suggests three different methods or designs for assessment: (i) true experiments: before-and-after measurements provide a valid basis for causal inference, namely that observed differences in access, costs, or quality of care between the two groups can be appropriately attributed to the experimental manipulation, rather than pre-existing differences between the groups; (ii) quasi-experimental design: provides estimates of the probability of a given outcome given exposure or non-exposure to the intended intervention; and (3) non-experimental methods: the potential for yielding useful insights into the subject matter. True experiments might follow a randomized-controlled trial design. Quasi-experiments could follow case-control study designs, and non-experimental designs could include case studies, case series, and correlational designs		X
Khoja–Durrani–Scott Evaluation Framework (KDS) (39;45)	KDS Evaluation tool to evaluate e-health programs according to pre-specified themes. Surveys, coding, and scoring schemes are available			X
[Assessment of telemedicine applications] (31)	Assessment of evaluation criteria, no specific instruments are proposed by the framework; that is, they vary across evaluation criteria, but are not further specified	X
CHEATS (32;33)	Outcomes are designed for assessing the evaluation criteria of CHEATS for evaluating a specific ICT system. The outcomes suggested relate to qualitative and quantitative instruments: semi-structured interviews with key participants, and questionnaire and existing data about service use and clinical effectiveness	X
[Eight-dimension sociotechnical model of safe and effective IT use] (34)	Specific instruments to assess the predefined evaluation criteria. The framework includes a set of evaluation items determined to be most relevant to the safety and effectiveness of the device	X
[Telemedicine evaluation plan] (35)	Eight different logs and six questionnaires were developed with common types of questions, one of which was copied/based on an existing questionnaire. The framework suggests logging each telemedicine consultation. Questionnaires were designed to be used during structured interviews (repeated) with staff involved			X
Clinical Value Compass (36)	Health-related quality of life to measure patients' subjective health assessments; patients' satisfaction to measure satisfaction with treatment modality; costs to assess medical costs associated with emergency room admissions, hospitalizations, and visits to outpatient clinics; and clinical status to assess clinical outcomes of treatment. A questionnaire is used and phone interviews after 6 months of telecare implementation; cost-data analysis; telephone interview using a structured questionnaire (pre/post), and medical patient file analysis			X

Frameworks that specify a set of methods for data collection and evaluation do so in a broad way. For example, the HSR framework suggests a number of possible broadly defined methodological strategies for assessing outcomes ranging from randomized-controlled trials to quasi-experiments using case-control studies and non-experimental designs such as case studies and correlational research (28).

Four frameworks include operationalized sets of outcomes (i.e., instruments) related to the assessment domains. The most comprehensive set of instruments is provided by the TEMPEST framework (26). In total, eighty-four specific outcomes are defined, including technology penetration and use to assess emerging technological trends, health expenditures to inform economic assessment, and population level epidemiological outcomes for assessing health policies.

Six frameworks specify concrete instruments and methods for collecting data and evaluating outcomes. For example, KDS includes four separate evaluation tools in accordance with the four eHealth life-cycle phases specified by the framework (39). Each of these tools include a set of questions for three types of users (managers, providers, and clients), covering seven outcome themes included in the framework (39;40).

For seven of the twenty-one frameworks, no specific assessment instruments or methods were reported. The main reason for not specifying methods or instruments for collecting data is that the operationalization of outcomes depends on the purpose, the technology, the patient group, and the context in which the eHealth service is to be implemented. In addition, the choice of outcomes and methods for data collection within each assessment domain must follow current state-of-the-art research methods within the domains to produce valid and reliable assessments (41).

eHealth Services Assessed

The eHealth services reported in the articles were described broadly, serving mainly illustrative purposes. The clinical purposes that were reported varied between diagnostic (n = 1), monitoring (n = 5), and treatment (n = 2), and areas of application such as chronic diseases such as multiple sclerosis, chronic heart failure, AIDS, chronic obstructive pulmonary disease (COPD), diabetes, or diabetic foot ulcers. However, patient profiles were not specified. Examples of settings in which an eHealth service was to be implemented included clinical and non-clinical settings or a combination of both. Clinical settings and disciplines were, for example, emergency rooms, tele-diagnosis and telemonitoring in pathology, psychiatry, ophthalmology, wound care, obstetrics, pediatrics, dermatology, and cardiology. Non-clinical settings and disciplines related to, for example, patients' homes, including tele-management systems, tele-rehabilitation, home-based self-monitoring, tele-education, and tele-consultation. The third category of eHealth services aimed to enable collaborative and integrated care between care providers in order to, for example, facilitate knowledge transfer for diagnostic and monitoring purposes.

A limited number of the included studies reported on actual application of the proposed framework. For instance, one study assessed the benefits of short-term implementation of a physical tele-rehabilitation system compared to usual care for Hebrew-speaking patients aged over 18 years old with a relapsing-remitting type of multiple sclerosis (36). The assessment used the Clinical Value Compass framework, defining the outcomes in four domains, including clinical symptom severity, health-related quality of life, patient satisfaction, and medical costs (36). Another example of a study using an experimental approach to verify the utility of a specific eHealth service assessment framework is a study reported by Ekeland et al. (40). This study aims to examine the utility of the MAST framework in the context of twenty-one implementation pilots of telemedicine services for diabetes, COPD, and cardiovascular disease.

Discussion

Main Findings

This study revealed twenty-one frameworks for eHTA that can be classified into four categories: staged, dimensional, hybrid, and business modeling focused. Two frameworks were reported in multiple articles: MAST (40;41) and CHEATS (32;33). The majority of the frameworks constitutes a set of assessment domains related to technical requirements and functionalities, clinical, economic, organizational, ethical, and legal characteristics, as well as characteristics of the healthcare system and stakeholders' needs. There is considerable variation in the guidelines provided by these frameworks regarding the operationalization of outcomes, instruments, and concrete methods for assessing the impact of eHealth services. The eHealth services presented in the articles served mostly illustrative purposes, rendering the proven applicability and real-world usefulness of the frameworks unclear.

The MAST framework provides the most comprehensive approach by including all domains identified in this review. The importance of such comprehensiveness is also found in a review of evaluation criteria for non-invasive telemonitoring for patients with heart failure (48). However, it could be that not all domains are relevant to a particular eHealth service. Not only because of the development phase, but also because of specific aims and properties of the eHealth service and the context in which it is to be implemented.

The challenge in providing comprehensive standards for assessing all eHealth services versus the relevance of assessment outcomes for a specific eHealth service is underlined by the fact that most authors of the included frameworks have a shared view on the importance of a generic assessment framework for eHealth services and do not include concrete assessment methods. However, specificity is sacrificed and applicability of the frameworks for assessing certain eHealth services is reduced. With the evidence standards framework for digital health technologies, NICE introduces a way to stratify eHealth services into evidence standards in proportion to the potential risk to end-users (9). A pragmatic functional classification scheme is proposed differentiating the main functions of the eHealth service. For each of these functions, varying levels and types of assessment data are required. Nevertheless, this approach (still) focuses solely on clinical effectiveness of the eHealth service and its economic impact. It does not provide standards for assessing technical requirements and functionalities, organizational, ethical, and legal aspects, nor does it address impacts on the healthcare system as a whole. Although eHealth services deserve no exception in assessing their impact on health and care, an extended NICE Framework might offer practical guidance that steers toward the assessment of impacts beyond clinical effectiveness and cost-effectiveness, as well as harmonization and reduced proliferation of loaded frameworks.

Limitations

This study is limited due to its focus on the scientific literature. As HTA often has an applied emphasis in informing policy, assessment methods and outcomes do not necessarily need to be disseminated through scientific literature. For this study, gray literature was left out due to issues with accessibility, selecting relevant sources, and its heterogeneous, unstructured, and often incomplete nature (49). However, methods and guidelines in using these resources in systematic reviews are maturing (50). In addition, practice-oriented expert feedback using qualitative and quantitative methods such as Delphi or Concept Mapping techniques (51) might enrich the results of this study and could prove useful in harmonizing and standardizing frameworks for eHTA.

Recommendations

Considering the variation in domains assessed, and level of detail concerning outcome definitions and instruments in the eHTA frameworks found, assessment frameworks would gain considerably in transparency, comparability, and applicability if: (i) the eHealth service being assessed is defined in a structured manner in terms of the clinical aim, target group, and working mechanism, and what, who and how the service is to be provided, and (ii) the assessment methods are standardized in terms of measurable outcomes and instruments. Moreover, the applicability and usefulness of eHTA frameworks should be reflected upon systematically to validate their utility and applicability in real world assessment practices.

The Template for Intervention Description and Replication (TIDieR) checklist could be a useful resource for reporting eHealth services for eHTA purposes (52). This checklist is designed to be generically applicable for reporting complex interventions and includes thirteen topics describing the rationale for the eHealth service, materials, and procedures, and provision details, as well as measures for ensuring adherence and fidelity.

In improving the standardization of eHTA, a systematic whole systems dialog with decision makers, policy makers, and HTA experts using a Delphi or Concept Mapping approach, might facilitate consensus-based prioritization and selection of assessment methods and instruments. The outcomes defined by the TEMPEST or CHEATS frameworks, the ontological methods proposed by UVON, and the comprehensive structure of the MAST framework can serve as a starting point. The work laid down by EUnetHTA (JA3/WP6), NICE, and the opinion report on assessing the impact of digital transformation of health services by the European Expert Panel on effective ways of investing in Health (EXPH) (53) should be considered.

A stepped-approach to what and how eHTA should be applied, tailored to the functional characteristics of the eHealth service, can be fruitful in balancing completeness and relevance of assessment outcomes and methods. For example, a first phase could entail the assessment of safety and (negative) effects on clinical outcomes in combination with an economic assessment of costs-effectiveness. In a second phase, the impact on organizational aspects such as required skills, changes in tasks and roles, and required interoperability with relevant eHealth systems could be assessed. Such a stepped-approach should improve adequacy and efficiency of assessments matching the Appropriate Care program of the National Health Care Institute (10) motto of “no more than needed and no less than necessary.”

Conclusions

HTA frameworks specifically designed for eHealth services are available. Besides technical performance and functionalities, costs and clinical aspects are described most frequently. Standardized sets of concrete evaluation methods and instruments are mostly lacking and evidence demonstrating the applicability of eHTA frameworks is limited.

Considering the purpose of eHTA to inform healthcare policy making and achieve evidence-based health care, the field would benefit from (i) standardizing the way in which eHealth services are reported and (ii) developing a standardized set of assessment tools by incorporating a stepped-approach tailored to the functional characteristics of the eHealth service. Standardized sets of evaluation methods for each of the domains as well as guidelines scoping the eHealth services might improve transparency, comparability, and efficiency of assessments, as well as facilitating collaboration of eHTA practices across healthcare systems in decision and policy making in digital health care.

Supplementary Material

The supplementary material for this article can be found at https://doi.org/10.1017/S026646232000015X.

Acknowledgments

The authors would like to thank Dr C. Planting of the Amsterdam UMC for providing expert advice and help in operationalizing the search strategy for this systematic review. This study was funded by the Dutch National Health Care Institute as it is relevant to their appropriate care program.

Conflict of Interest

Mr. Vis, Ms. Buhrmann, and Prof. Dr. Riper report that their institute (Vrije Universiteit Amsterdam) received a financial contribution from the National Health Care Institute (Zorginstituut Nederland), during the conduct and for the purpose of this study. The research was conducted independently by the researchers. Dr. Ossebaard is employed by the Dutch National Health Care Institute which funded this study.

References

1. Draborg E, Gyrd-Hansen D, Poulsen PB, Horder M. International comparison of the definition and the practical application of health technology assessment. Int J Technol Assess Health Care. 2005; 21: 89–95.

2. May C, Mort M, Williams T, Mair FS, Gask L. Health technology assessment in its local contexts: Studies of telehealthcare. Soc Sci Med. (1982) 2003; 57: 697–710.

3. Williams T, May CR, Mair FS, Mort M, Gask L. Normative models of health technology assessment and the social production of evidence about telehealth care. Health Policy. 2003; 64: 39–54.

4. Woolf SH, Henshall C. Health technology assessment in the United Kingdom. Int J Technol Assess Health Care. 2000; 16: 591–625.

5. Banta D, Jonsson E. History of HTA: Introduction. Int J Technol Assess Health Care. 2009; 25: 1–6.

6. Bashshur R, Shannon G, Sapci H. Telemedicine evaluation. Telemed J E Health. 2005; 11: 296–316.

7. Goodman CS, Ahn R. Methodological approaches of health technology assessment. Int J Med Inf. 1999; 56: 97–105.

8. DeChant HK, Tohme WG, Mun SK, Hayes WS, Schulman KA. Health systems evaluation of telemedicine: A staged approach. Telemed J. 1996; 2: 303 –12.

9. National Institute for Health and Care Excellence (NICE). Evidence Standards Framework for Digital Health Technologies. 2019. p. 1–35. Available from: https://www.nice.org.uk/Media/Default/About/what-we-do/our-programmes/evidence-standards-framework/digital-evidence-standards-framework.pdf.

10. National Health Care Institute. Zinnige Zorg–systematisch doorlichten van het basispakket. 2018. Available from: https://www.zorginstituutnederland.nl/over-ons/werkwijzen-en-procedures/zinnige-zorg---doorlichting-van-het-basispakket.

11. National Health Care Institute. Working method for the Zinnige Zorg (appropriate care) Programme. 2018. p. 1–10.

12. Eysenbach G. What is e-health? J Med Internet Res. 2001; 3: e20.

13. Pagliari C, Sloan D, Gregor P, Sullivan F, Detmer D, Kahan JP, What is eHealth (4): A scoping exercise to map the field. J Med Internet Res. 2005; 7: e9.

14. Ahern DK, Kreslake JM, Phalen JM, Bock B. What is eHealth (6): Perspectives on the evolution of eHealth research. J Med Internet Res. 2006; 8: e54.

15. Stelk WJ. Implementing health-care innovations: In search of a theoretical foundation for a science of implementation. Int J Ment Health. 2006; 35: 35–49.

16. The Lancet. Is digital medicine different? The Lancet. 2018; 392: 95.

17. Doupi P. Evolving health IT systems evaluation: The convergence of health informatics and HTA. In: Ammenwerth E, Rigby M, editors. Evidence-based health informatics. Amsterdam: IOS Press; 2016. pp. 220–236.

18. Blackwood B. Methodological issues in evaluating complex healthcare interventions. J Adv Nurs. 2006; 54: 612 –22.

19. May CR, Finch TL, Mair FS, Ballini L, Dowrick CF, Eccles MP, Understanding the implementation of complex interventions in health care: The normalization process model. BMC Health Serv Res. 2007; 7: 148.

20. Arai L, Britten N, Popay J, Roberts H, Pitticrew M, Rodgers M, Testing methodological developments in the conduct of narrative synthesis: A demonstration review of research on the implementation of smoke alarm interventions. Evid Policy: J Res, Debate Pract. 2007; 3: 361 –83(23).

21. Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005; 10: 6–20.

22. Popay J, Roberts H, Snowden A, Petticrew M, Arai L, Rodgers M, Guidance on the conduct of narrative synthesis in systematic reviews. Southampton: National Centre for Research Methods; 2006.

23. Casper GR, Kenron DA. A framework for technology assessment: Approaches for the selection of a home technology device. Clin Nurse Spec. 2005; 19: 170 –74.

24. Vimarlund V, Davoody N, Koch S. Steps to consider for effective decision making when selecting and prioritizing eHealth services. Stud Health Technol Inform. 2013; 192: 239 –43.

25. Brear M. Evaluating telemedicine: Lessons and challenges. Health Inf Manag J. 2006; 35: 23–31.

26. Currie WL. TEMPEST: An integrative model for health technology assessment. Health Policy Technol. 2012; 1 (1): 35–49.

27. Eivazzadeh S, Anderberg P, Larsson TC, Fricker SA, Berglund J. Evaluating health information systems using ontologies. JMIR Med Inform. 2016; 4: e20.

28. Grigsby J, Brega AG, Devore PA. The evaluation of telemedicine and health services research. Telemed J E Health. 2005; 11: 317 –28.

29. Hebert M. Telehealth success: Evaluation framework development. Stud Health Technol Inform. 2001; 84: 1145 –49.

30. Nepal S, Li J, Jang-Jaccard J, Alem L. A framework for telehealth program evaluation. Telemed J E Health. 2014; 20: 393–404.

31. Ohinmaa A, Hailey D, Roine R. Elements for assessment of telemedicine applications. Int J Technol Assess Health Care. 2001; 17: 190–202.

32. Shaw NT. “CHEATS”: A generic information communication technology (ICT) evaluation framework. Comput Biol Med. 2002; 32: 209 –20.

33. Brown M, Shaw NT. Evaluation practices of a major Canadian telehealth provider: Lessons and future directions for the field. Telemed J E Health. 2008; 14: 769 –74.

34. Sittig DF, Kahol K, Singh H. Sociotechnical evaluation of the safety and effectiveness of point-of-care mobile computing devices: A case study conducted in India. Stud Health Technol Inform. 2013; 192: 515 –19.

35. Stensgaard T, Sørensen T. Telemedicine in Greenland—the creation of an evaluation plan. J Telemed Telecare. 2001; 7: 37–38.

36. Zissman K, Lejbkowicz I, Miller A. Telemedicine for multiple sclerosis patients: Assessment using health value compass. Mult Scler. 2012; 18: 472 –80.

37. Giansanti D, Morelli S, Macellari V. Telemedicine technology assessment part I: Setup and validation of a quality control system. Telemed J E Health. 2007; 13: 118 –29.

38. Giansanti D, Morelli S, Macellari V. Telemedicine technology assessment part II: Tools for a quality control system. Telemed J E Health. 2007; 13: 130 –40.

39. Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed J E Health. 2013; 19: 48–53.

40. Ekeland AG, Grottland A. Assessment of MAST in European patient-centered telemedicine pilots. Int J Technol Assess Health Care. 2016; 31: 304 –11.

41. Kidholm K, Ekeland AG, Jensen LK, Rasmussen J, Pedersen CD, Bowes A, A model for assessment of telemedicine applications: MAST. Int J Technol Assess Health Care. 2012; 28: 44–51.

42. Alfonzo A, Huerta MK, Wong S, Passariello G, Díaz M, La Cruz A, Design of a methodology for assessing an electrocardiographic telemonitoring system. IEEE Eng Med Biol–Proc. 2007; 2007: 3729 –32.

43. Grustam AS, Vrijhoef HJM, Koymans R, Hukal P, Severens JL. Assessment of a Business-to-Consumer (B2C) model for telemonitoring patients with Chronic Heart Failure (CHF). BMC Med Inform Decis Mak. 2017; 17: 145.

44. Donabedian A. Explorations in quality assessment and monitoring: Volume I: The definition of quality and approaches to its assessment, vol. 1. Ann Arbor, Michigan: Health Administration Press; 1980.

45. Khoja S, Durrani H, Scott RE, Sajwani A, Piryani U. KDS Evaluation Tool. 2019. Available from: http://panacea-evaluation.yolasite.com/evaluation-tools.php.

46. Chamova J, EUnetHTA JA. The HTA Core Model^® Guiding principles on use. 2016. p. 1–3. Available from: http:// www.eunetha.eu.

47. Lampe K, Mäkelä M, Garrido MV, Anttila H, Autti-Rämö I, Hicks NJ, The HTA core model: A novel method for producing and reporting health technology assessments. Int J Technol Assess Health Care. 2009; 25: 9–20.

48. Farnia T, Jaulent MC, Steichen O. Evaluation criteria of noninvasive telemonitoring for patients with heart failure: Systematic review. J Med Internet Res. 2018; 20: e16 –9.

49. Benzies KM, Premji S, Hayden KA, Serrett K. State-of-the-evidence reviews: Advantages and challenges of including grey literature. Worldviews Evid-Based Nurs. 2006; 3: 55–61.

50. Mahood Q, van Eerd D, Irvin E. Searching for grey literature for systematic reviews: Challenges and benefits. Res Synth Methods. 2014; 5: 221 –34.

51. Trochim WM, McLinden D. Introduction to a special issue on concept mapping. Eval Program Plann. 2017; 60: 166 –75.

52. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. 2014. p. g1687. Presented at the BMJ (Clinical Research Ed.).

53. EXPH (EXpert Panel on effective ways of investing in Health). Assessing the impact of digital transformation of health services European Commission, DG Health & Food Safety, Directorate B: Health systems, medical products and innovation. 2019.