Introduction
Background
Electronic mental health interventions (eMental health or eMH) for mood disorders such as depression can increase reach and accessibility of mental health services while maintaining comparable clinical outcomes as face-to-face interventions and superior outcomes compared with waiting lists [1-3]. eMH encompasses the use of digital technologies and new media for the delivery of screening, health promotion, prevention, early intervention, treatment, or relapse prevention, as well as for improvement of health care delivery (eg, electronic patient files), professional education (e-learning), and Web-based research in the field of mental health [4]. Research on the translation of the results of these studies into routine care is scarce. Translational research can have two dimensions: dissemination and implementation of an innovation in clinical practice. Dissemination concerns the passive and active spread of information about eMH to relevant stakeholders, including consumers, clinical care providers, and decision- and policy makers. Implementation refers to the process of embedding and integrating new practices into actual care settings [5,6]. It seems that eMH interventions are reasonably well disseminated to clinical practice given that a number of preconditions are fulfilled, such as the availability of technical infrastructures and proper reimbursement of these services [7]. Nevertheless, the actual use of eMH in routine mental health care lags behind expectations. It is unclear why implementation of eMH remains difficult.
A logical approach in addressing this implementation challenge is to identify the factors that might promote or inhibit implementation of eMH in routine practice [8]. On the basis of these determinants, implementation-enhancing interventions might be designed and applied with the aim to improve implementation processes and upscaling of eMH care. Many determinants of different care practices have been identified for a variety of clinical interventions. For example, Krause and colleagues [9] identified over 600 context-specific determinants thought to be relevant in implementing evidence-based interventions for patients with chronic health conditions, including depression in the elderly, chronic obstructive pulmonary disease, and obesity. Examples of these determinants are status and quality of evidence and clinical recommendations, characteristics of the innovation, delivery modalities, reimbursement modalities, implementation leadership, and organizational readiness [10-12]. Similarly, examples of implementation barriers for eMH include the perceived importance of computer literacy skills, knowledge and awareness of existing eMH services, as well as credibility of these services [13]. In turn, many of these determinants have been clustered and framed, currently resulting in more than 60 frameworks used to study and understand implementation processes [14,15]. Although such determinants and frameworks are valuable and comprehensive, they lack specificity to any category of intervention and therefore, provide little practical detail to prioritize determinants and guidance for action to improve the implementation of eMH interventions.
The reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) framework provides a heuristic tool for bridging interventions’ internal validity established in well-controlled conditions and their external validity in real-world conditions [16,17]. It is designed to evaluate the public health impact of health promoting interventions, and it is widely used in implementation research [18]. The framework covers five intervention-related areas of impact: (1) reach as the ability to address those in need of an intervention, (2) effectiveness in terms of the impact of interventions on health outcomes, (3) adoption as a decision to proceed with implementing the clinical intervention, (4) implementation as the process of embedding and integration of the intervention in routine practice and its consistency of delivery and costs, and (5) maintenance as the institutionalization of the intervention in routine care [16,18-20]. Considering the current evidence-base for eMH and the increasing emphasis on comparative effectiveness research in testing clinical and cost-effectiveness of eMH [21], the RE-AIM framework might be a valuable tool to structure determinants of practices that are specific to eMH.
Research Question
Given the absence of a comprehensive overview of determinants of practices, we systematically reviewed the literature to develop a taxonomy relevant to the implementation of eMH. Knowledge on these determinants can inform the study of interventions that aim to improve the implementation of eMH in routine practice. The following research question guided the research: “What determinants of practice are identified as relevant to implementing eMH interventions for mood disorders in routine practice?” A broad view on eMH and care practice settings, including clinical and community practices, was adopted to provide a comprehensive taxonomy of determinants of mental health practice relevant to implementing eMH.
Methods
Study Design
A systematic review of scientific literature was conducted. RE-AIM was used to structure the review. Various implementation studies in the area of mental health care using RE-AIM substantiate the utility of this framework, including evaluations of the implementation of behavior mental health assessment tools [22]; smoking cessation interventions in people with mental illnesses [23]; mental health, substance abuse, and health behavior interventions into specific primary care behavior health programs [24]; tele-mental health consultation program in pediatric primary care in rural settings [25]; and assessing a therapist’s role in eMH for patients with depressive disorders [26].
Search Strategy
Due to the novelty of the topics concerned (ie, eMH and implementation), a broad search strategy was developed with high sensitivity to four key terms (as opposed to a focused strategy with higher specificity [27]): “implementation,” “mental health care practice,” “mood disorder,” and “eMental-health.” No time frame was applied. On the basis of literature, benchmark definitions for these concepts were developed, and a total of 408 synonyms were formulated for the search strings. A trained librarian guided the formulation of the search strings. The benchmark definitions and search strings are included in Multimedia Appendix 1. The search was conducted in July 2015 in the three main bibliographical databases (PubMed, PsycINFO, and EMBASE). All identified papers were examined for eligibility by two researchers (CV and MM) independently. Disagreements were solved by discussion and, where necessary, moderated by a third researcher (AK) to reach consensus.
Inclusion and Exclusion Criteria
The inclusion and exclusion criteria are shown in Textboxes 1 and 2.
Data Extraction
A systematic qualitative narrative approach was applied for the data extraction, analyses, and synthesis of the results [28-30]. A field guide was developed to extract relevant data from the retained articles. Items included the study aim, methods, the psychotherapeutic intervention, eMH technology applied, type of mood disorder, implementation intervention (eg, training of professionals, or a focused marketing campaign to raise awareness of eMH among patients), settings, sample(s), recruitment procedures, results, and findings in terms of determinants of practice. The data were tabulated and categorized in accordance with four of the five RE-AIM dimensions: reach, adoption, implementation, and maintenance. Table 1 presents definitions and adaptations to the RE-AIM dimensions that we applied for the purpose of this study. Effectiveness was not addressed in this review as ample reviews on the clinical effectiveness of eMH for mood disorders are available [1-3]. The implementation dimension was broadened to also include the purposive implementation interventions that might have been employed to achieve better implementation outcomes.
Analyses and Synthesis
Thematic analysis was applied to identify the recurrent and most important determinants to implementing eMH in routine practice (ie, themes) arising in the included literature. Thematic analysis is a common method for identifying, grouping, and summarizing findings from included studies in narrative review [29]. The (groups of) determinants were developed inductively (ie, without a priori defined topics guiding the analysis). We did not apply a threshold for recurrence of certain themes in the data. Data were extracted by three researchers (CV, MM, and LB) independently. Data files were merged and discrepancies solved by discussion to reach consensus. Freely available reference management software (Mendely, Elsevier), a spreadsheet (Microsoft Excel, Microsoft Corporation), and qualitative analysis software (ATLAS.ti, Scientific Software Development GmbH) were used to organize and conduct the selection, data extraction, and data analysis.
Inclusion criteria.
Exclusion criteria.
Table 1. Dimensions of reach, effectiveness, adoption, implementation, and maintenance (RE-AIM); their definitions; and its focus.
| Dimension | Definitions [16] | Comment |
| Reach | Participation ratio of patients and their characteristics | |
| Effectiveness | Impact of the (clinical) intervention on patients’ health, quality of life, and economic outcomes | Not addressed in this study |
| Adoption | Proportion and representativeness of staff and organizations delivering the services | |
| Implementation | (Clinical) interventions’ fidelity and (implementation) costs | Added: deliberate and purposive actions to implement eMHa [31] |
| Maintenance | Extent to which the intervention is and remains to be part of routine care practice |
aeMH: electronic mental health interventions, or eMental health.
Results
Study Selection
The searches resulted in 16,718 records. After removing the duplicates, 13,417 unique titles remained and were screened for eligibility against the inclusion and exclusion criteria. In total, 13,159 articles were excluded on the basis of the information in titles and abstracts. A total of 258 articles were retained, and after examining the full-text articles, a total of 48 studies were included in the analysis. Figure 1 provides an overview of the inclusion and exclusion of studies in the different phases of the systematic review.
General Study Characteristics
Table 2 provides an overview of the main characteristics of the studies, including the RE-AIM dimension(s) addressed, target disorder, therapeutic principles, technology applied, guidance modalities, and study design.
Most studies investigated reach (n=33), followed by adoption (n=19), implementation (n=6), and maintenance (n=4). The specific type of the target disorder was often described in broad terms such as common mental disorders or mood disorders (n=20), or in exemplary disorders such as depression or anxiety (n=17). Most studies (n=39) did not explicitly report the therapeutic principles of the clinical intervention that was implemented. More than half of the studies investigated the provision of mental health services for mood disorders through videoconferencing technologies (n=26), most often by using videoconferencing for support and consultations. The remainder of the studies focused on Internet-based interventions (n=20). Three studies looked at purely self-help interventions (through Web and mobile technologies), and 10 studies did report on a specific eMH intervention but did not report the guidance modality. Eighteen studies specified the eMH intervention and described the guidance modality. The majority (n=44) of the studies were of an observational, that is, descriptive nature. Most of these (n=20) applied mixed-methods (eg, a survey and semistructured interviews), followed by a large proportion (n=16) of studies that applied qualitative methods such as ethnography or consensus-seeking methods using focus-group discussions. Five studies were of an experimental design, applying either quantitative or mixed-methods. More information about the specific studies’ aims, designs, settings, participants, and clinical and implementation-related interventions are reported in Multimedia Appendix 2.
Determinants of Practice
In total, 37 specific determinants of practices relevant to implementing eMH in routine care were identified. The 37 determinants were clustered resulting in a taxonomy of six groups: (1) acceptance of eMH by patients and service delivery staff, (2) appropriateness or clinical relevance of eMH, (3) engagement of participants in implementing and delivering eMH, (4) resources for implementing and delivering eMH, (5) work processes in delivering eMH, and (6) leadership in implementing and delivering eMH. Group definitions are provided in Table 3. The spider diagram in Figure 2 shows that the majority of studies reported determinants in the domain reach that were related to acceptance (n=34) and appropriateness (n=23). When categorized under RE-AIM, reach and the domain adoption were studied most often, addressing determinants related to acceptance (n=17), appropriateness (n=11), and engagement (n=10). Least investigated were the domains of implementation and maintenance.
A detailed list of the determinants is included in Table 4, including their definitions, main perspective, RE-AIM dimensions, and references to the source articles. The following subsections detail the determinants for each of the four RE-AIM domains. The perspective from which become apparent are included, differentiating between (1) patients, (2) staff (individuals and groups) involved in delivering mental health services, (3) organizations as the functional and administrative structures aimed to deliver mental health care, and (4) the system perspective as the human and material resources and organizational arrangements on a community level aimed at to preserve, protect, and restore peoples’ health [32]. More detailed information, including the related excerpts of texts retrieved from the articles, are in Multimedia Appendix 2.
Enlarge this image.Figure 1. Information flow through the different phases of the systematic review.
Reach
The domain reach includes determinants of practices that are related to patients’ participation in eMH and their characteristics. Of the 33 studies that were categorized under reach, most investigated patients’ and mental HCPs perceptions and attitudes of patients and professionals (n=20), or the actual use (n=9) of eMH in a routine care setting. Most studies were of an observational nature (n=31). Two studies used an experimental design for testing interventions aimed at increasing access and use of eMH.
From the perspective of patients, two main groups of factors appeared to be relevant in implementing eMH in routine care: acceptance and appropriateness. Determinants grouped under acceptance concern the perceived and actual feasibility of interacting with eMH. For example, knowledge about the existence of eMH ( awareness, n=13) and technological aspects of the treatment (eg, usability and stability, n=10) were most often reported in the included literature.
Determinants categorized under appropriateness refers to the patients’ perceived fit, relevance, or compatibility of eMH in addressing his or her mental disorder. Within this group, the professional-patient relationship was reported most often by both care providers and patients to be an important aspect that requires consideration when implementing eMH. For example, the perceived importance of interaction and verbal communication was highlighted by van der Vaart, et al [58], showing that the lack in nonverbal communication in Web-based treatments can pose limits to discussing more difficult issues with patients.
From the perspective of staff, engagement emerged as a group of factors next to the determinants grouped under acceptance and appropriateness. Engagement relates to the sustained and effective involvement of staff in implementing and delivering eMH for mood disorders in routine care. Most notably, engagement seem to be related to the organizing structures, policies, and procedures within an organization (n=4), as well as the availability and stability of the required information and communication technology (ICT; n=4). For example, in a qualitative study on expectations of both patients and health professionals in commencing in Internet-based psychotherapy, Montero-Marin et al [48] noted the importance of standardizing Web-based interventions in an integrated service delivery model.
From the perspective of mental health service providing organizations, resources in terms of available and stability of facilitating infrastructure was mentioned (n=2) as an important determinant. In addition, the modus operandi in service delivery both in terms of primary care processes (eg, referral pathways, n=2) as well as facilitating processes (eg, administrative and ICT support and billing processes, n=1) require consideration when implementing eMH in routine practice. Additionally, leadership in terms of existing cultures, strategies, and priorities emerged from the included articles as a determinant of practice (n=1). Regarding the primary care processes, Buist et al [43] showed that considering eMH as a valid service option can influence actual application. Differences in actual use might be caused by differing levels of interest and experience in the eMH service of the service managers.
At health care system level, there were three aspects reported to be of importance, namely policy-making processes (n=2), the availability of appropriate resources including qualified staff (n=2), and collaboration and cooperation within the system and across disciplines (n=1).
Table 2. Overview of studies categorized per reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) domain; technology applied; target disorder; therapeutic principles; and study design.
| Characteristic | Reach (n=33) | Adoption (n=19) | Implementation (n=6) | Maintenance (n=4) | na | |
| Target disorder | ||||||
| Depressive disorder | 8 | 3 | 2 | —b | 10 | |
| Mood disordersc | 16 | 9 | — | 2 | 20 | |
| Not specifiedd | 8 | 7 | 4 | 2 | 17 | |
| Therapeutic principlese | ||||||
| Cognitive behavior therapy | 5 | 3 | 2 | — | 8 | |
| Other (eg, mindfulness) | 1 | — | — | — | 1 | |
| General psychotherapy | 27 | 16 | 4 | 4 | 39 | |
| Technology applied | ||||||
| Internet-based (unguided) | 2 | — | — | — | 2 | |
| Internet-based (guidedf) | 3 | 3 | 1 | — | 5 | |
| Internet-based (minimal guidance) | 1 | — | — | — | 1 | |
| Internet-based (therapist guided) | 1 | — | — | — | 1 | |
| Internet-based (blended) | 1 | 1 | — | — | 1 | |
| Internet-based (not specifiedg) | 8 | 2 | 1 | — | 10 | |
| Computer-based | 1 | 1 | — | — | 1 | |
| mobile health (unguided) | 1 | — | — | — | 1 | |
| Videoconferencing | 15 | 12 | 4 | 4 | 26 | |
| Study design | ||||||
| Experimental—quantitative methods | 2 | — | — | — | 2 | |
| Experimental—mixed-methods | — | 2 | 1 | — | 3 | |
| Observational—qualitative methods | 10 | 9 | 2 | 1 | 15 | |
| Observational—quantitative methods | 6 | 1 | — | 1 | 8 | |
| Observational—mixed-methods | 15 | 7 | 2 | 2 | 20 | |
aThe n in this column are unique references. Some studies were categorized under more than one RE-AIM dimension.
bRefers to no studies categorized under that condition.
cMood disorders including depressive disorder and/or in combination with other mental health disorders.
dRefers to the studies that described the target disorder in exemplary wordings without becoming specific. The generic wordings related to mood disorders.
eNot all studies specifically discussed the target disorder or psychotherapeutic principles of the service as studies focused, for example, on perceptions of the delivery method relevant to implementation and not on the specific treatment itself.
fSome form of guidance; guidance modality and intensity was not specified.
gNot specified if it was a guided intervention or self-help.
Table 3. Identified groups of determinants of practice and their definitions.
| Group | Definition | Determinants |
| Acceptance | The perception among patients, providers, organizations, and systems that eMHa is agreeable, congenial, or satisfactory. | Access to treatment; expectations and preferences; observability and experience; evidence base; convenience; technology; awareness; skills and competences; privacy; clinical cultures; education; costs; policy; health care system structures |
| Appropriateness | The perceived fit, relevance, or compatibility of eMH for the patient in addressing his or her mental disorder. | Professional-patient interaction; effectiveness; personal need; flexibility; negative effects; safety; patient characteristics |
| Engagement | Continuing implementing, delivering, and receiving eMH and remain doing so in the context of concrete treatment plans. | Organizational structures and procedures; leadership; staffing and roles; access and reliability of ICTb; time; collaboration |
| Resources | The availability and appropriateness of resources required in implementing and delivering eMH, including human resources, equipment, funding, and other infrastructural aspects. | Personnel; funds; infrastructure |
| Work processes | The course of action (modus of operandi) in service delivery and all other tasks and responsibilities mental health care service organizations have. | Primary process; facilitating processes |
| Leadership | Directing and controlling the working processes and organizing activities that enable implementation and delivery of eMH. | Culture; communication; management; strategies and priorities; external relations |
aeMH: electronic mental health interventions, or eMental health.
bICT: information and communication technology.
Enlarge this image.Figure 2. Spider diagram of the spread of the number of studies (n=48) categorized under the RE-AIM dimensions and the six main groups of determinants we identified in literature: acceptance, appropriateness, engagement, resources, work processes, and leadership. RE-AIM: reach, effectiveness, adoption, implementation, and maintenance.
Table 4. Determinants of practice identified in the literature mapped on each reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) dimension, including their proposed definitions, main perspective, and references. Indented are determinants grouped within a group of determinants.
| Cluster/Determinant | Perspective | RE-AIMa | n | References | ||
| Acceptance: the perception among patients and providers that using eMHb is agreeable, congenial, or satisfactory | ||||||
| Access to treatment: the state of accessibility and the act of accessing mental health services. | Patient | R, A | 9 | [33-41] | ||
| Expectations and preferences: individual and collective attitudes, expectations, and preexisting preferences about receiving and providing mental health care in general and eMH specifically. | Patient | R, A, I | 12 | [34,37,41-50] | ||
| Expectations and preferences: individual and collective attitudes, expectations, and preexisting preferences about receiving and providing mental health care in general and eMH specifically. | Staff | R, A, I, M | 13 | [43,48,51-61] | ||
| Observability and experience: the possibility and actual of observations in use (seeing or hearing about the treatment) and experiences of staff in the process of accepting eMH as a valid treatment option. | Staff | R, A, I | 7 | [43,51-53,59,62,63] | ||
| Evidence-base: the scientific evidence of the feasibility and effectiveness of eMH. | Staff | R, A, I | 3 | [46,52,61] | ||
| Convenience: the comfort experienced by patients in accessing and receiving mental health care, including overcoming geographical distances, time constraints, and availability of treatment materials. | Patient | R, A, I, M | 14 | [33,34,39-42,47,59,60,62,64-67] | ||
| Technology: the technical aspects of eMH, including availability of and familiarity with ICT, complexity, usability, and working procedures. | Patient | R, A, M | 11 | [34,35,37,42,48,49,51,54,55,66,68] | ||
| Convenience: the comfort experienced by patients in accessing and receiving mental health care, including overcoming geographical distances, time constraints, and availability of treatment materials. | Staff | R, A, I, M | 8 | [43,51-57] | ||
| Technology: the technical aspects of eMH, including availability of and familiarity with ICT, complexity, usability, and working procedures. | Patient | R, A, M | 14 | [34,37,44-46,48-51,59,60,69-71] | ||
| Technology: the technical aspects of eMH, including availability of and familiarity with ICT, complexity, usability, and working procedures. | Staff | R, A, I, M | 8 | [43,46,51,53,62,63,71,72] | ||
| Skills and competences: specific personal capacities and means required for receiving (patients) or providing (staff) eMH. | Patient | R, A | 7 | [33,39,48,51,54,59,73] | ||
| Skills and competences: specific personal capacities and means required for receiving (patients) or providing (staff) eMH. | Staff | R, A, I, M | 5 | [48,54,55,61,66] | ||
| Privacy: respecting patients’ and providers’ freedom from unauthorized intrusion, including discretion and confidentiality. | Patient | R, A | 4 | [35,48,49,73] | ||
| Privacy: respecting patients’ and providers’ freedom from unauthorized intrusion, including discretion and confidentiality. | Staff | R, A | 1 | [48] | ||
| Clinical culture: socially defined and agreed “ways of doing,” including norms, habits, and roles. | Staff | R, A, I, M | 6 | [43,53,60,61,65,67] | ||
| Education: training of staff in providing eMH in routine care, including technical and therapeutic training, formal education, credentialing, peer-group learning, and supervision. | Staff | R, A, I | 13 | [43,46,51-53,58,61-63,67,71,72,74] | ||
| Costs: the expenditures made to receive or provide eMH. | Patient | R, A, M | 3 | [40,66,67] | ||
| Appropriateness: the perceived fit, relevance, or compatibility of eMH for the patient in addressing his or her mental disorder | ||||||
| Professional-patient relationship: the professional interaction between (mental) health care provider and patient, including the aspects such as trust, comfort, and therapeutic interaction. | Patient | R, A, I | 18 | [33,35,39,40,42,46,48,50,54,55,59, 68-70,73,75-77] | ||
| Professional-patient relationship: the professional interaction between (mental) health care provider and patient, including the aspects such as trust, comfort, and therapeutic interaction. | Staff | R, A, I | 10 | [46,52,54,55,57-59,61,71,77] | ||
| Effectiveness: patients’ mental health care needs, including information needs and specific (mental) health conditions. | Patient | R | 3 | [33,35,40] | ||
| Personal need: individual mental health care needs, including information needs and specific (mental) health conditions. | Patients | R, A, M | 8 | [33,35,42,58,59,65,69,75] | ||
| Flexibility: the extent to which care providers can alter or adapt the eMH to the (perceived) needs of the patient or care provider. | Staff | R, A, I, M | 6 | [46,58,61,67,69,72] | ||
| Negative effects: the perceived and actual negative (clinical) outcomes of receiving eMH. | Patient | R, A | 3 | [33,46,78] | ||
| Safety: the physical and mental safety of patients receiving eMH. | Patient | R | 3 | [35,55,78] | ||
| Safety: the physical and mental safety of patients receiving eMH. | Staff | R, A | 3 | [52,55,59,69] | ||
| Patient characteristics: individual patient characteristics, including age, gender, clinical history, social economic status, and clinical symptoms relevant to eMH. | Patient | R, A | 7 | [37,48,69,70,73,78,79] | ||
| Patient characteristics: individual patient characteristics, including age, gender, clinical history, social economic status, and clinical symptoms relevant to eMH. | Staff | R, A, I | 4 | [43,52,59,61] | ||
| Engagement: continuing implementing, delivering, and receiving eMH and remain doing so in the context of concrete treatment plans | ||||||
| Organizational structures and procedures: the organizing structures, policies, and procedures for delivery of eMH, including standards and clinical guidelines, administrative support, technical support, and other facilitating services. | Staff | R, A, I | 8 | [43,48,52,55,59,61,62,72] | ||
| Leadership: the managerial capacity and operationalization of an organization, including leadership, goal setting, strategies, and supportive measures | Staff | R, A, I | 4 | [55,58,62,72] | ||
| Staffing and roles: the availability of staff necessary in delivering eMH, including qualifications, roles, and responsibilities | Staff | R, A, I, M | 7 | [35,48,53,59,60,62,72] | ||
| Access and reliability of ICTc: the availability, stability, and reliability of required technology, including interoperability with other existing technology (eg, electronic patient record). | Staff | R, A, I | 10 | [43,48,52,53,56,59,62,63,71,72] | ||
| Time: the time constraints in providing mental health care in general and eMH specifically. | Staff | I | 1 | [61] | ||
| Collaboration: the possibility and actual act of parties involved in delivery of eMH willingly work together, including sharing of information and expertise. | Staff | R, A, I | 3 | [61,72,77] | ||
| Resources: the availability and appropriateness of resources required in implementing and delivering eMH, including human resources, equipment, funding, and other infrastructural aspects | ||||||
| Personnel: the availability, capacity, and capabilities of persons necessary in the delivering eMH. | Organization | A, I | 2 | [62,80] | ||
| Funds: the availability and sources of pecuniary resources necessary for delivering eMH and its impact on existing (care) budgets | Organization | A, I, M | 3 | [66,67,72,80] | ||
| Infrastructure: availability, quality, and stability of facilitating structures required for delivering eMH, including offices and equipment. | Organization | R, A, I, M | 7 | [43,52,53,60,62,67,72] | ||
| Processes: the course of action (modus of operandi) in service delivery and all other tasks and responsibilities mental health care service organizations have | ||||||
| Primary process: a series of actions conducing to the primary objectives of a mental health care organization such as referral processes, establishing diagnosis, and providing treatment. | Organization | R, A, I, M | 7 | [43,48,53,60,62,67,80] | ||
| Facilitating processes: the facilitating activities required for primary processes to deliver mental health care services. Facilitating processes do not directly add value to service delivery but are necessary to provide the services. | Organization | R, A, I, M | 7 | [43,52,60,62,67,72,80] | ||
| Leadership: directing and controlling the working processes and organizing activities that enable implementation and delivery of eMH | ||||||
| Culture: socially defined and agreed “ways of doing,” including norms, habits, and roles relevant to delivering eMH. | Organization | R, A, I, M | 2 | [43,67] | ||
| Communication: the mechanisms, means, and contents of disseminating information across the mental health care organization. | Organization | A, I | 1 | [62] | ||
| Management: the managerial capacity and operationalization of an organization delivering eMH, including leadership, goal setting, strategies, and supportive measures. | Organization | A, I, M | 3 | [60,62,80] | ||
| Strategies and priorities: the operationalization of and operationalized objectives into feasible working plans, including vision, mission, priorities, and work plans. | Organization | R, A, I, M | 2 | [43,67] | ||
| External relations: cooperation and collaboration of various external parties involved and/or affected by delivery of eMH, including sharing knowledge. | Organization | A, I, M | 3 | [65,67] | ||
| Health care system: the organization of people, institutions, and resources that deliver mental health care services to meet the health needs of target populations | ||||||
| Policy: the plans or courses of actions intended to influence and determine decisions and actions relevant to delivery of eMH. | Setting | R, A, I, M | 2 | [43,60] | ||
| Resources: the availability and appropriateness of resources required in delivering eMH, including HCPsd, ICT and standardization, funding, and other infrastructural aspects. | Setting | R, M | 4 | [60,65,70,71] | ||
| Community acceptance: the shared perception among the community that eMH is agreeable, palatable, or satisfactory. | Setting | M | 2 | [65,66] | ||
| Collaboration: cooperation and collaboration of various parties involved in delivery of eMH, including knowledge sharing. | Setting | R, A, I | 1 | [43] | ||
| Structure: the organizing and organized plan of health services in a given (geographical) context and relevant to the implementation and delivery of eMH. | Setting | M | 1 | [60] | ||
aRE-AIM: reach, effectiveness, adoption, implementation, and maintenance. Please refer to Table 1 for the specific definitions of the RE-AIM framework. The following abbreviations are used in this column: R: reach; A: adoption; I: implementation; and M: maintenance.
beMH: electronic mental health interventions. or eMental health.
cICT: information and communication technology.
dHCPs: health care professionals.
Adoption
Adoption mirrors the decision of staff and organizations involved in delivering the eMH services and the extent to which they actually use and deploy the services to their patients. Of the 19 studies that were characterized under adoption, 16 studies investigated adoption-related perceptions and attitudes toward eMH (n=9), or actual use (n=7) of eMH in routine care settings showing adoption. Three studies investigated and tested an adoption-enhancing intervention aimed at increasing the number of staff involved in the delivery of eMH.
Seen from the perspective of staff delivering the services, a frequently mentioned determinant grouped under acceptance was patients’ awareness and knowledge of the existence of eMH (n=5). Similarly, the awareness of eMH as a viable treatment option among staff was also identified as a relevant determinant in staff adopting eMH (n=6). Adoption can be facilitated by allowing clinicians to gain experience with eMH and the observability of eMH (n=7). In terms of appropriateness of eMH, the studies indicated that patient-professional relationship is an important determinant to consider when designing interventions aimed at improving adoption rates (n=7). To illustrate, May et al [54] reported on the use of videoconferencing technology in delivering psychotherapy, indicating that the therapist-patient relation should include strategies that appropriately addresses the disorder for which verbal interaction might be essential. Furthermore, the availability and stability of the technical aspects, including infrastructure and interoperability of related ICT (n=8), can be an influential factor in facilitating the engagement of professionals in continuing to offer and apply eMH to their patients.
From the organizations’ perspective, the determinants addressing adoption related mostly to the availability of infrastructural resources (n=5) and the primary care process (n=5). Infrastructural resources included the availability, quality, and stability of facilitating structures such as office rooms and ICT equipment. Determinants related to the primary care processes included issues with referral procedures, diagnostic procedures, and therapy guidelines and manuals. For instance, Jameson et al [53] highlighted that clinical policies and procedures for initiating a referral and coordinating between the various partners involved in service delivery are necessary for successful and sustainable use of eMH.
One article reported determinants from a health care system perspective. Buist et al [43] reported on the importance of mechanisms that enable collaboration, sharing of information, and policies supporting better use of these mechanisms.
Implementation
Determinants categorized under implementation relate to the extent to which eMH is used in real-world settings as intended (ie, fidelity of use), implementation costs, or deliberate and purposive actions to implement eMH. Of the 6 studies identified under implementation, 2 investigated an implementation-related intervention focusing on training mental health providers to use eMH in daily practice. The other 4 studies performed a process evaluation (n=1) and investigated use and utilization of eMH (n=3).
The most frequently reported determinants from the perspective of staff were related to acceptance. These concerned raising staffs’ awareness about the existence of eMH (n=3) and providing education to staff (n-4) in applying eMH in routine care. Specific determinants included references to technical and therapeutic training, formal education and credentialing, and peer-group learning and supervision. For example, Willhelmsen et al [61] showed the importance of training of general practitioners (GPs) in increasing patients’ acceptance of eMH, which might strengthen the perceived credibility of eMH.
Furthermore, from the perspective of staff, engagement was found to be influenced by the availability of support and facilitating services (n=4). For example, Avey et al [72] reported in a qualitative study on implementation processes that coordination and collaboration between the various persons involved in the service delivery should be facilitated effectively and that a dedicated program coordinator was valued highly among the participating hospitals.
From the viewpoint of an organization, the availability of resources such as staffing (n=2), funding (n=2), and infrastructural facilities (n=2) were reported as relevant determinants. In addition, various factors emerged from the literature related to the primary modes operandi (n=3). For example, Reifels et al [80] discussed that successful implementation might depend on the existence or establishment of effective primary processes in the service delivery structures. Similarly, implementation outcomes can be determined by factors facilitating and supporting the primary processes in delivering mental health care services (n=4). Examples include issues with office space, availability of equipment, and administrative support as Adler et al [62] highlighted. Besides the organizational structures and processes, leadership and management (n=3) need to be considered when implementing eMH. This includes scheduling problems, lack of a clear goals, and managerial support to address issues with existing clinical demands.
From the perspective of health care systems, less rich information was found in the included studies. However, Buist et al [43] did report on determinants of practices relating to the availability of policy measures (n=1) and possibilities to collaborate and share knowledge within and across disciplines and settings (n=1).
Maintenance
Under maintenance, determinants were categorized that relate to keeping the eMH as a normal part of routine care practices. All 4 maintenance studies were of a descriptive nature aiming to establish usage and utility figures of videoconferencing- delivered mental health services (n=2), capture end-user perceptions (n=1), or describe potential success factors (n=1) of programs that remained in practice after their implementation phase.
From the patients’ viewpoint, the convenience of eMH was seen as an important determinant in maintaining the service in practice (n=4). In an evaluation of patients’ perceptions of a routine tele-psychiatry service in central Alberta, Simpson et al [66] highlighted the importance of reducing waiting times and travel time and that this in the long term might outweigh preferences for face-to-face consultations.
From the perspective of mental health staff, the clinical culture in terms of socially defined and agreed ways of doing (n=2), including norms, habits, and roles, are considered to be important in maintaining the services in routine practice. Hailey et al [65] showed that traditional patterns might keep staff from changing their practice, even if the service is in operation for a considerable time.
At the organizational level, various determinants were reported, including availability of funds (n=2) and infrastructure (n=2), the primary modes of operation (n=2), supporting structures and activities (n=2), and leadership and management (n=3). Regarding the latter, Whitten et al [67] showed in a study comparing tele-psychiatry programs that are in routine care for some time that the different business approaches these programs took might have contributed to their success.
From the perspective of the health care system, besides the importance of policy (n=1), community acceptance (n=2), and organizing and organized plans of health services (ie, structure; n=1), the availability and appropriateness of resources required in maintaining eMH in practice were mentioned (n=2).
Discussion
Principal Findings
We developed a taxonomy of 37 determinants of mental health care practices known in the literature as relevant to successfully implement eMH for mood disorders. The determinants of practices clustered in six groups are expressed at (a combination of) patient, staff, organization, and setting levels and address one or more RE-AIM dimensions (see Table 3). Three determinants were reported most frequently: (1) acceptance of eMH in terms of the expectations and preferences of patients and professionals; (2) appropriateness of eMH in addressing the mental health disorder, and specifically, the therapeutic interactions mediated by eMH; and (3) the availability, stability, and reliability of required technologies, including successful interoperability with other existing technologies.
Strengths and Limitations
The search strategy in this review aimed to capture as much relevant scientific literature as possible. For this reason, broadly defined search terms were used. By applying a standardized integrative approach (RE-AIM in combination with qualitative thematic analysis), we were able to search for commonalities in the concepts and underlying study characteristics while preserving the heterogeneous nature of the data retrieved from the studies. However, and although we searched three important bibliographic databases, it is likely that important work from social scientist generalist databases was excluded.
The evidence supporting the determinants identified in this study is mostly of a descriptive nature obtained from observational studies. Due to the limited empirical evidence verifying causality of specific determinants of practices and implementation successes, the findings of this work should be interpreted with care. In an attempt to substantiate this, we conducted a quality appraisal analysis. We included a wide variety of studies ranging from observational case studies using qualitative ethnographic methods to randomized controlled trials quantitatively testing specific implementation interventions. However, because of the heterogeneity of these studies and the absence of validated instruments to assess quality, it proved impossible to come to sensible conclusions about the quality of the evidence. An elaborate approach as done by Greenhalgh et al [81,82], meta-narrative approach in developing a model of diffusion of innovations by including the research traditions from which the included studies emerged might be a fruitful approach but was beyond the scope of this review.
Comparison With Other Work
Drozd et al [83] conducted a scoping review of 164 publications (including gray literature). The investigators applied the Active Implementation Framework (AIF) to identify implementation-related factors [84]. The AIF describes the components of an implementation practice, including aspects of staff and patient selection, training, supervision, performance assessment, decision support, administrative support, system intervention, and leadership. Drozd and colleagues found in their review factors similar to those that emerged from our analysis of the literature, including certain competences of patients and professionals and organizational drivers. Regarding the latter, the authors did not find empirical support for determinants such as leadership. The authors conclude that not finding empirical evidence for organizational drivers merely indicates a gap in the implementation-related research. Despite the low numbers (n=4), our study shows that leadership indeed is found in empirical research to be a relevant determinant in implementing eMH. This difference can perhaps be explained by the methodological choices that were made for reviewing the literature. Where Drozd and colleagues choose to follow a top-down approach (the AIF), our review followed a quantitative inductive process in identifying the topics related to implementing eMH that emerged from the included articles. Furthermore, the search strategy and data sources in light of their quality and comparability most likely influenced the results.
Similarly, Ross et al [85] updated a systematic review (of reviews, n=44) and looked at qualitative accounts of factors that influence implementation of eHealth interventions in a broader context, including somatic care. Factors identified by these researchers are comparable with the ones presented here, including complexity factors and adaptability, adding to the users’ perception of the acceptability of eHealth interventions. However, it should be noted that the concept of eHealth used by the authors included a variety of ICT-mediated health care services in four main categories: management systems, communication systems, clinical decision support systems, and information systems. In this respect, the authors did not address eHealth to contain purposed intrinsic therapeutic content aimed at improving health conditions as we did. This raises the question of whether generic eHealth both in terms of care setting (health care in general vs mental health care for mood disorders) and purpose (information sharing, support systems vs therapeutic interventions focusing on care and cure) give rise to (partial) different taxonomies of determinants of practice.
Recommendations for Implementation Practice
Implementation practitioners might benefit in implementing eMH in routine care practices by taking into account the barriers and facilitators that are identified in this systematic review. Specific implementation activities can be designed and applied on the basis of these factors to achieve better implementation outcomes.
One of the most frequently mentioned barriers emerging from the literature concerns the expectations and preferences of patients and professionals about eMH services. Negative individual and collective attitudes, expectations, and existing preferences can prohibit successful implementation of eMH. Ebert et al [45] showed that providing information to patients can enhance their acceptance of eMH. In addressing expectations and preferences of mental health care staff, it is advisable to include service delivery staff in the early stages of decision making and strategy development to increase acceptance and inform concrete implementation activities aimed at the concerns of the end users.
A second important determinant of practice is related to the appropriateness of the eMH intervention in addressing the mental disorder. Within this cluster, the nature and quality of the interactions between the professional and the patient is thought to be highly influential in obtaining favorable clinical outcomes. This includes aspects such as building trust, comfort, and the quality of the therapeutic interactions. eMH interventions delivered through ICT are thought to influence these interactions negatively. Hadjistavropoulos et al [74,86] showed that specific training can change knowledge about, attitudes toward and confidence in delivering eMH. Careful development of training programs and (continuous) guidance of HCPs in applying the eMH intervention might lower barriers with perceived patient-professional interaction through eMH. In addition, innovative models of for integrating therapist support in eMH services might address issues with engagement and the patient-professional relationship [87].
Third, the availability and reliability of required technologies is considered an important determinant for mental HCPs to remain engaged in providing eMH to their patients in routine care. This includes the interoperability with other existing technologies such as electronic health records. It seems important to ensure that the user perspective, including that of the service delivery staff, is taken into account and that the eMH service seamlessly fits within existing technologies and work processes. Here, single-sign on technology and intelligent portal designs might be fruitful avenues to explore.
Future Research
To increase impact and added value of future research on implementation of eMH for mood disorders in routine practice, the following two topics should be taken into account: (1) identifying organization and system-level determinants and (2) empirical evidence on the effects of implementation strategies in addressing specific barriers and exploiting facilitating factors.
Until now, most implementation research was focused on practitioner and patient-level determinants. Service delivery takes place in a social context at micro (individuals, teams), mesa (organizations), and macro (systems) level. Knowledge about how these different contexts influence implementation efforts can facilitate further scaling up of eMH. Research on systems level might focus on the possible policy measures that enhance implementation of eMH at service deliverer level. For example, what resources at organization or health care system-level are required to deliver eMH? This can include processes of task shifting, curricula and certification of mental health staff, ICT and standardization, funding, and other infrastructural aspects. Or, what role does community acceptance have in implementing eMH in routine practice, and how can the shared perception of community as a whole be changed? Detailed knowledge of organization and setting level factors might be more likely to come from a combination of clinical psychology, social sciences, organizational psychology, and policy research. Here, the MasterMind project [88] might provide inspiration for further research on determinants of practices of eMH.
Furthermore, the field would benefit from well-performed experiments designed to test implementation interventions addressing specific determinants of practices. As shown in this review, there is limited evidence on the causal relationship between determinants and implementation outcomes. Well-designed experiments studying the effects of to the local context–tailored implementation strategies might contribute to the understanding of mechanisms of implementation processes. Do, for example, educational meetings (and in what formats) contribute in raising awareness among GPs about which patient might benefit most from which eMH intervention? Or can championing an Internet-based cognitive behavioral therapy service increase the adoption of other therapists in mental health care team while maintaining the flexibility therapists need to adapt parts of the treatments to the patients’ needs? Fusing implementation practices and research into natural implementation laboratories might be a valuable approach to engage in comparative effectiveness studies of implementation interventions. In these types of studies, experimental implementation interventions can be compared with usual implementation activities for their effects on the degree of normalization of a clinical intervention in real-world service delivery settings. The ImpleMentAll project (project position paper and study protocol forthcoming) might be a good example of this approach. This type of future research might lead to a shift from practice-based and evidence-informed to evidence-based implementation of clinically effective and relevant eMH interventions.
Conclusions
This study systematically reviewed scientific literature and developed an evidence-informed taxonomy of six clusters of 37 determinants of practices we found in literature: (1) acceptance of eMH interventions among patients, providers, organizations, and health care settings; (2) appropriateness of eMH interventions in addressing the disorder; (3) engagement in implementing, delivering, and receiving eMH interventions and remain doing so; (4) the availability and appropriateness of resources for implementing and delivering eMH interventions; (5) processes relating to the modus of operandi in delivering eMH interventions; and (6) leadership directing and controlling processes and organizing activities enabling implementation and delivery of eMH interventions. On the basis of these determinants of practices, implementation-enhancing interventions can be designed, tested, and applied to achieve better implementation outcomes. Suggestions for implementation practice are discussed, such as in-depth training of professionals, careful selection, and continuous development of the eMH technology used. In addition, focal points for future research are provided, including implementation-related factors on organization and system level, as well as (quasi) experimental research to test the effectiveness of specific implementation interventions in attaining better implementation outcomes for eMH service provision.
Acknowledgments
The authors would like to thank Dr C. Planting of the VU Medical Centre/GGZ InGeest for providing expert advice and help in operationalizing the search strategy for this systematic review. Furthermore, the authors would like to express gratitude to the MasterMind and the ImpleMentAll project for supporting this work. The MasterMind project is partially funded under the ICT Policy Support Programme (ICT PSP) as part of the Competitiveness and Innovation Framework Programme (CIP) by the European Community (Grant Agreement number: 621000). The ImpleMentAll project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 733025. These funding bodies had no influence in the design, execution, analysis, or interpretation of the results of this study.
Authors' Contributions
HR, JS, and CV originated the idea for conducting this systematic review. CV, MM, AK, and HR designed the study protocol. CV, MM, and LB executed the search strategy and extracted the data. CV authored the study. All authors provided feedback and suggestions for this manuscript and read and approved the final manuscript.
Conflicts of Interest
None declared.
Multimedia Appendix 1
Benchmark definitions, definitions of RE-AIM, and the actual search strings that were applied in the search and analysis strategy.
Lindefors N, Andersson G. In: Lindefors N, Andersson G, editors. Guided Internet-Based Treatments in Psychiatry. Cham: Springer; 2016.
Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry 2014 Oct;13(3):288-295
Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012 Jun;32(4):329-342.
Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res 2010;12(5):e74
May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci 2016 Dec 19;11(1):141
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health 2009 Jan;36(1):24-34
Topooco N, Riper H, Araya R, Berking M, Brunn M, Chevreul K, et al. Attitudes towards digital treatment for depression: a European stakeholder survey. Internet Interv 2017 Jun;8:1-9.
Wensing M, Oxman A, Baker R, Godycki-Cwirko M, Flottorp S, Szecsenyi J, et al. Tailored Implementation For Chronic Diseases (TICD): a project protocol. Implement Sci 2011 Sep 07;6:103
Krause J, Agarwal S, Bodicoat DH, Ring A, Shepherd D, Rogers S, et al. Evaluation of a tailored intervention to improve management of overweight and obesity in primary care: study protocol of a cluster randomised controlled trial. Trials 2014 Mar 19;15:82
Aarons GA, Sommerfeld DH. Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. J Am Acad Child Adolesc Psychiatry 2012 Apr;51(4):423-431
Fleuren MA, Paulussen TG, Van Dommelen DP, Van Buuren BS. Towards a measurement instrument for determinants of innovations. Int J Qual Health Care 2014 Oct;26(5):501-510
Wensing M, Huntink E, van Lieshout LJ, Godycki-Cwirko M, Kowalczyk A, Jäger C, et al. Tailored implementation of evidence-based practice for patients with chronic diseases. PLoS One 2014;9(7):e101981
Wozney L, Newton AS, Gehring ND, Bennett K, Huguet A, Hartling L, et al. Implementation of eMental Health care: viewpoints from key informants from organizations and agencies with eHealth mandates. BMC Med Inform Decis Mak 2017 Jun 2;17(1):1.
Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med 2012 Sep;43(3):337-350
Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci 2015;10:53
Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327.
Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ Res 2006 Oct;21(5):688-694
Gaglio B, Shoup JA, Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health 2013 Jun;103(6):e38-e46.
Belza B, Toobert DJ, Glasgow RE. RE-AIM for Program Planning: Overview and Applications. In: National Council on Aging. Washington: National Council on Aging; 2014:1-14.
Jilcott S, Ammerman A, Sommers J, Glasgow RE. Applying the RE-AIM framework to assess the public health impact of policy change. Ann Behav Med 2007 Oct;34(2):105-114.
Kleiboer A, Smit J, Bosmans J, Ruwaard J, Andersson G, Topooco N, et al. European COMPARative Effectiveness research on blended Depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials 2016 Dec 03;17(1):387
Krist AH, Phillips SM, Sabo RT, Balasubramanian BA, Heurtin-Roberts S, Ory MG, MOHR Study Group. Adoption, reach, implementation, and maintenance of a behavioral and mental health assessment in primary care. Ann Fam Med 2014;12(6):525-533
Saw A, Kim J, Lim J, Powell C, Tong EK. Smoking cessation counseling for Asian immigrants with serious mental illness: using RE-AIM to understand challenges and lessons learned in primary care-behavioral health integration. Health Promot Pract 2013 Sep;14(5 Suppl):70S-79S
Young J, Gilwee J, Holman M, Messier R, Kelly M, Kessler R. Mental health, substance abuse, and health behavior intervention as part of the patient-centered medical home: a case study. Behav Med Pract Policy Res 2012 Jun 20;2(3):345-354.
Arora PG, Connors EH, Blizzard A, Coble K, Gloff N, Pruitt D. Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example. Eval Program Plann 2017 Dec;60:56-63.
Mol M, Dozeman E, van Schaik DJ, Vis CP, Riper H, Smit JH. The therapist's role in the implementation of internet-based cognitive behavioural therapy for patients with depression: study protocol. BMC Psychiatry 2016 Sep 30;16(1):338
Green S, Higgins J. Preparing a Cochrane review. In: Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons, Ltd; Sep 26, 2008:11-30.
Arai L, Britten N, Popay J, Roberts H, Petticrew M, Rodgers M, et al. Testing methodological developments in the conduct of narrative synthesis: a demonstration review of research on the implementation of smoke alarm interventions. Evid Policy J Res Debate Pract 2007 Aug 01;3(3):361-383.
Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy 2005 Jul;10 Suppl 1:6-20.
Popay J, Roberts H, Snowden A, Petticrew M, Arai L, Rodgers M, et al. Lancaster.ac.uk. 2006 Apr. Guidance on the Conduct of Narrative Synthesis in Systematic Reviews: A Product from the ESRC Methods Programme URL: http://www.lancaster.ac.uk/shm/research/nssr/research/dissemination/publications/NS_Synthesis_Guidance_v1.pdf [accessed 2018-03-09] [WebCite Cache]
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76
Porta M. A Dictionary of Epidemiology, Sixth edition. New York: Oxford University Press; 2014.
Beattie A, Shaw A, Kaur S, Kessler D. Primary-care patients' expectations and experiences of online cognitive behavioural therapy for depression: a qualitative study. Health Expect 2009 Mar;12(1):45-59 Included in the review
De Las Cuevas C, Artiles J, De La Fuente J, Serrano P. Telepsychiatry in the Canary Islands: user acceptance and satisfaction. J Telemed Telecare 2003;9(4):221-224 Included in the review.
Gibson KL, Coulson H, Miles R, Kakekakekung C, Daniels E, O'Donnell S. Conversations on telemental health: listening to remote and rural First Nations communities. Rural Remote Health 2011;11(2):1656 Included in the review
Graham C, Franses A, Kenwright M, Marks I. Psychotherapy by computer: a postal survey of responders to a teletext article. Psychiatr bull 2018 Jan 2;24(09):331-332 Included in the review.
Handley TE, Kay-Lambkin FJ, Inder KJ, Attia JR, Lewin TJ, Kelly BJ. Feasibility of internet-delivered mental health treatments for rural populations. Soc Psychiatry Psychiatr Epidemiol 2014 Feb;49(2):275-282 Included in the review.
Hilty DM, Nesbitt TS, Kuenneth CA, Cruz GM, Hales RE. Rural versus suburban primary care needs, utilization, and satisfaction with telepsychiatric consultation. J Rural Health 2007;23(2):163-165 Included in the review.
Mucic D. International telepsychiatry: a study of patient acceptability. J Telemed Telecare 2008 Jul;14(5):241-243 Included in the review.
Musiat P, Goldstone P, Tarrier N. Understanding the acceptability of e-mental health--attitudes and expectations towards computerised self-help treatments for mental health problems. BMC Psychiatry 2014 Apr 11;14(1):109 Included in the review
Saurman E, Lyle D, Perkins D, Roberts R. Successful provision of emergency mental health care to rural and remote New South Wales: an evaluation of the Mental Health Emergency Care-Rural Access Program. Aust Health Rev 2014 Feb;38(1):58-64 Included in the review.
Boggs JM, Beck A, Felder JN, Dimidjian S, Metcalf CA, Segal ZV. Web-based intervention in mindfulness meditation for reducing residual depressive symptoms and relapse prophylaxis: a qualitative study. J Med Internet Res 2014 Mar 24;16(3):e87 Included in the review
Buist A, Coman G, Silvas A, Burrows G. An evaluation of the telepsychiatry programme in Victoria, Australia. J Telemed Telecare 2000;6(4):216-221 Included in the review.
Casey LM, Joy A, Clough BA. The impact of information on attitudes toward e-mental health services. Cyberpsychol Behav Soc Netw 2013 Aug;16(8):593-598 Included in the review.
Ebert DD, Berking M, Cuijpers P, Lehr D, Pörtner M, Baumeister H. Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord 2015 May 01;176:9-17 Included in the review.
Jones RB, Ashurst EJ. Online anonymous discussion between service users and health professionals to ascertain stakeholder concerns in using e-health services in mental health. Health Informatics J 2013 Dec;19(4):281-299 Included in the review.
Mohr DC, Siddique J, Ho J, Duffecy J, Jin L, Fokuo JK. Interest in behavioral and psychological treatments delivered face-to-face, by telephone, and by internet. Ann Behav Med 2010 Aug;40(1):89-98 Included in the review
Montero-Marín J, Prado-Abril J, Botella C, Mayoral-Cleries F, Baños R, Herrera-Mercadal P, et al. Expectations among patients and health professionals regarding Web-based interventions for depression in primary care: a qualitative study. J Med Internet Res 2015 Mar 10;17(3):e67 Included in the review
Proudfoot J, Parker G, Hadzi PD, Manicavasagar V, Adler E, Whitton A. Community attitudes to the appropriation of mobile phones for monitoring and managing depression, anxiety, and stress. J Med Internet Res 2010 Dec 19;12(5):e64 Included in the review
Werner P. Willingness to use telemedicine for psychiatric care. Telemed J E Health 2004;10(3):286-293 Included in the review.
Carper MM, McHugh RK, Barlow DH. The dissemination of computer-based psychological treatment: a preliminary analysis of patient and clinician perceptions. Adm Policy Ment Health 2011 Oct 15;40(2):87-95 Included in the review.
Gibson K, O'Donnell S, Coulson H, Kakepetum-Schultz T. Mental health professionals' perspectives of telemental health with remote and rural First Nations communities. J Telemed Telecare 2011;17(5):263-267 Included in the review.
Jameson JP, Farmer MS, Head KJ, Fortney J, Teal CR. VA community mental health service providers' utilization of and attitudes toward telemental health care: the gatekeeper's perspective. J Rural Health 2011;27(4):425-432 Included in the review.
May CR, Ellis NT, Atkinson T, Gask L, Mair F, Smith C. Psychiatry by videophone: a trial service in north west England. Stud Health Technol Inform 1999;68:207-210 Included in the review.
May C, Gask L, Atkinson T, Ellis N, Mair F, Esmail A. Resisting and promoting new technologies in clinical practice: the case of telepsychiatry. Soc Sci Med 2001 Jun;52(12):1889-1901 Included in the review.
Monthuy-Blanc J, Bouchard S, Maïano C, Séguin M. Factors influencing mental health providers' intention to use telepsychotherapy in First Nations communities. Transcult Psychiatry 2013 Apr;50(2):323-343 Included in the review.
Sinclair C, Holloway K, Riley G, Auret K. Online mental health resources in rural Australia: clinician perceptions of acceptability. J Med Internet Res 2013 Sep 05;15(9):e193 Included in the review
van der Vaart R, Witting M, Riper H, Kooistra L, Bohlmeijer ET, van Gemert-Pijnen LJ. Blending online therapy into regular face-to-face therapy for depression: content, ratio and preconditions according to patients and therapists using a Delphi study. BMC Psychiatry 2014 Dec 14;14(1):355 Included in the review
Whitten P, Kuwahara E. A multi-phase telepsychiatry programme in Michigan: organizational factors affecting utilization and user perceptions. J Telemed Telecare 2004;10(5):254-261 Included in the review.
Ohinmaa A, Roine R, Hailey D, Kuusimäki ML, Winblad I. The use of videoconferencing for mental health services in Finland. J Telemed Telecare 2008;14(5):266-270 Included in the review.
Wilhelmsen M, Høifødt RS, Kolstrup N, Waterloo K, Eisemann M, Chenhall R, et al. Norwegian general practitioners' perspectives on implementation of a guided web-based cognitive behavioral therapy for depression: a qualitative study. J Med Internet Res 2014 Sep 10;16(9):e208 Included in the review
Adler G, Pritchett LR, Kauth MR, Nadorff D. A pilot project to improve access to telepsychotherapy at rural clinics. Telemed J E Health 2014 Jan;20(1):83-85 Included in the review
Austen S, McGrath M. Telemental health technology in deaf and general mental-health services: access and use. Am Ann Deaf 2006;151(3):311-317 Included in the review.
Casey LM, Wright MA, Clough BA. Comparison of perceived barriers and treatment preferences associated with internet-based and face-to-face psychological treatment of depression. Int J Cyber Behav Psychol Learn 2014 Oct;4(4):16-22 Included in the review.
Hailey D, Ohinmaa A, Roine R, Bulger T. Uptake of telemental health services in Alberta: a success, but not in all regions. J Telemed Telecare 2016 Dec 02;13(3_suppl):42-44 Included in the review.
Simpson J, Doze S, Urness D, Hailey D, Jacobs P. Telepsychiatry as a routine service--the perspective of the patient. J Telemed Telecare 2001;7(3):155-160 Included in the review.
Whitten P, Do C, Kingsley C. An analysis of telepsychiatry programs from an organizational perspective. Cyberpsychol Behav 2000 Dec;3(6):911-916 Included in the review.
Greenwood J, Chamberlain C, Parker G. Evaluation of a rural telepsychiatry service. Australas Psychiatry 2004 Sep;12(3):268-272 Included in the review.
Morrison C, Walker G, Ruggeri K, Hughes JM. An implementation pilot of the MindBalance web-based intervention for depression in three IAPT services. tCBT 2014 Nov 14;7:e15 Included in the review.
Rowe N, Gibson S, Morley S, Krupinski EA. Ten-year experience of a private nonprofit telepsychiatry service. Telemed J E Health 2008 Dec;14(10):1078-1086 Included in the review.
Simpson J, Doze S, Urness D, Hailey D, Jacobs P. Evaluation of a routine telepsychiatry service. J Telemed Telecare 2001;7(2):90-98 Included in the review.
Avey JP, Hobbs RL. Dial in: fostering the use of telebehavioral health services in frontier Alaska. Psychol Serv 2013 Aug;10(3):289-297 Included in the review.
Rohland BM, Saleh SS, Rohrer JE, Romitti PA. Acceptability of telepsychiatry to a rural population. Psychiatr Serv 2000 May;51(5):672-674 Included in the review.
Hadjistavropoulos HD, Thompson MJ, Klein B, Austin DW. Dissemination of therapist-assisted internet cognitive behaviour therapy: development and open pilot study of a workshop. Cogn Behav Ther 2012;41(3):230-240 Included in the review.
Callahan EJ, Hilty DM, Nesbitt TS. Patient satisfaction with telemedicine consultation in primary care: comparison of ratings of medical and mental health applications. Telemed J 1998;4(4):363-369 Included in the review.
Fortney JC, Pyne JM, Mouden SB, Mittal D, Hudson TJ, Schroeder GW, et al. Practice-based versus telemedicine-based collaborative care for depression in rural federally qualified health centers: a pragmatic randomized comparative effectiveness trial. Am J Psychiatry 2013 Apr;170(4):414-425 Included in the review
Swinton JJ, Robinson WD, Bischoff RJ. Telehealth and rural depression: physician and patient perspectives. Fam Syst Health 2009 Jun;27(2):172-182 Included in the review.
DeAndrea DC. Testing the proclaimed affordances of online support groups in a nationally representative sample of adults seeking mental health assistance. J Health Commun 2015;20(2):147-156 Included in the review.
Titov N, Andrews G, Kemp A, Robinson E. Characteristics of adults with anxiety or depression treated at an internet clinic: comparison with a national survey and an outpatient clinic. PLoS One 2010 May 28;5(5):e10885 Included in the review
Reifels L, Bassilios B, King KE, Fletcher JR, Blashki G, Pirkis JE. Innovations in primary mental healthcare. Aust Health Rev 2013 Jun;37(3):312-317 Included in the review.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82(4):581-629
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: a meta-narrative approach to systematic review. Soc Sci Med 2005 Jul;61(2):417-430.
Drozd F, Vaskinn L, Bergsund HB, Haga SM, Slinning K, Bjørkli CA. The implementation of internet interventions for depression: a scoping review. J Med Internet Res 2016 Sep 08;18(9):e236
Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation Research: A Synthesis of the Literature. Tampa, Florida, USA: National Implementation Research Network, FMHI Publication #231; 2005:1-119.
Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci 2016 Dec 26;11(1):146
Hadjistavropoulos HD, Pugh NE, Nugent MM, Hesser H, Andersson G, Ivanov M, et al. Therapist-assisted internet-delivered cognitive behavior therapy for depression and anxiety: translating evidence into clinical practice. J Anxiety Disord 2014 Dec;28(8):884-893
Schueller SM, Tomasino KN, Mohr DC. Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin Psychol Sci Pract 2016 Nov 17;24(1):27-45.
Vis C, Kleiboer A, Prior R, Bønes E, Cavallo M, Clark SA, et al. Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interv 2015 Nov;2(4):399-409.
© 2018. This work is licensed under https://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Mol, Mayke
; Kleiboer, Annet
; Bührmann, Leah
; Finch, Tracy
; Smit, Jan
; Riper, Heleen