Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER): Development of a Conceptual Framework

Abstract

Background: Conversational agents (CAs), also known as chatbots, are computer programs that simulate human conversations by using predetermined rule-based responses or artificial intelligence algorithms. They are increasingly used in health care, particularly via smartphones. There is, at present, no conceptual framework guiding the development of smartphone-based, rule-based CAs in health care. To fill this gap, we propose structured and tailored guidance for their design, development, evaluation, and implementation.

Objective: The aim of this study was to develop a conceptual framework for the design, evaluation, and implementation of smartphone-delivered, rule-based, goal-oriented, and text-based CAs for health care.

Methods: We followed the approach by Jabareen, which was based on the grounded theory method, to develop this conceptual framework. We performed 2 literature reviews focusing on health care CAs and conceptual frameworks for the development of mobile health interventions. We identified, named, categorized, integrated, and synthesized the information retrieved from the literature reviews to develop the conceptual framework. We then applied this framework by developing a CA and testing it in a feasibility study.

Results: The Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER) conceptual framework includes 8 iterative steps grouped into 3 stages, as follows: design, comprising defining the goal, creating an identity, assembling the team, and selecting the delivery interface; development, including developing the content and building the conversation flow; and the evaluation and implementation of the CA. They were complemented by 2 cross-cutting considerations—user-centered design and privacy and security—that were relevant at all stages. This conceptual framework was successfully applied in the development of a CA to support lifestyle changes and prevent type 2 diabetes.

Conclusions: Drawing on published evidence, the DISCOVER conceptual framework provides a step-by-step guide for developing rule-based, smartphone-delivered CAs. Further evaluation of this framework in diverse health care areas and settings and for a variety of users is needed to demonstrate its validity. Future research should aim to explore the use of CAs to deliver health care interventions, including behavior change and potential privacy and safety concerns.

Introduction

Background

Conversational agents (CAs) are computer programs that use text, speech, and other input modalities to enable communication with users [1]. They can be accessed through a variety of ways, such as social media platforms (eg, Facebook Messenger), websites, and smartphone apps, or deployed using stand-alone digital devices (eg, Alexa, Google Assistant, and Siri). The interactive nature of CAs makes them acceptable to a diverse group of users [2-4] and a preferred tool in a number of disciplines, including customer service, retail, and e-commerce [5-7].

In health care, CAs are increasingly used to assist in various tasks, such as patient education, self-management of chronic conditions, and routine task automation (eg, appointment booking), and support health professionals’ decision-making for diagnosis and triage [3,8-10]. More recently, CAs have seen large-scale implementation with the introduction of Babylon’s artificial intelligence (AI)–based symptom checker CA to the UK National Health Service and to Rwanda’s National Health Insurance Scheme [11]. CAs have the potential to support health care delivery, improve access to health care services, and automate tasks [12], and they may also reduce health professionals’ workload [13].

CAs vary in complexity and capability. There are 3 design dimensions used to classify CAs: purpose, communication channels, and response generation architecture [6]. According to purpose, CAs can be classified into task- or goal-oriented CAs, which respond to a limited number of tasks within a prespecified domain, or non–task- or non–goal-oriented CAs, which are potentially able to respond to an unrestricted variety of user requests [6]. Communication channels can commonly be divided into 2 main types: text-based or voice-based CAs. Response generation architecture can be broadly classified into 3 groups: rule-based and retrieval-based CAs, which produce a response by selecting it from a pool of predetermined responses either following simple rules to match phrases or identifying specific keywords in the text [6,14,15], and generative-based CAs, which use AI algorithms to develop a contextual response informed by the system’s previous and ongoing learning [6,14-16]. Although all 3 groups may involve the use of AI algorithms [6], rule-based CAs allow developers greater control over the conversation content and flow, which is a useful feature when developing CAs for health care. By contrast, AI algorithms, particularly neural networks, may develop decisions that are not explainable or understood by the end user, a phenomenon referred to as the black box [17]. In health care settings, the black box effect may lead to biased or erroneous decision-making and patient harm [18], which may limit the use of AI. A new field of explainable AI is currently emerging that aims to provide justification for algorithm predictions and increase system transparency, although the validity of results for individual patients should be carefully considered [19].

CAs can be deployed using a variety of digital devices, including smartphones. The widespread availability of smartphones in high-income countries and increasingly in low- and middle-income countries [20] makes them an ideal interface to deliver CA interventions. Smartphones offer users the possibility of continuous and dynamic monitoring of health conditions in a private space and at the time of their convenience [21] not only of subjective, self-reported data but also of objective, sensor-based data. Furthermore, smartphones allow for the delivery of interventions according to user needs [22]. CA interventions are complex and often require lengthy, costly design and development processes led by multidisciplinary teams of health care professionals, computer scientists, and app developers, which may limit the number of teams able to engage in CA development, particularly in low- and middle-income countries. However, mobile health (mHealth) interventions, particularly SMS text messages delivered using mobile phones, are effective in delivering health care interventions, especially in low-resource settings [23,24].

Several frameworks for the design and development of mHealth interventions currently exist, offering guidance at every step of the cycle, from the conceptualization of user needs [25,26] to the development of the digital health intervention [25-27]. These frameworks focus on generic, app-based interventions without a conversational interface. However, Zhang et al [28] described a framework for the development of AI-based CAs to deliver behavior change interventions that may require significant deployment of resources, including a large, multidisciplinary team, and close supervision of the AI algorithms to prevent unintended and potentially harmful effects on the users. However, to date, no conceptual framework for the design, development, and evaluation of rule-based CAs has been published despite a growing interest in the use of CAs in health care settings.

Objectives

CAs constitute a specific type of digital intervention characterized by the use of a conversational interface, often led by an agent with a distinct personality as evidenced by its tone of speech, method of interaction, and visual representation, which is often associated with higher levels of engagement with the user. These features and the ubiquity of smartphones support the need for a framework that is accessible to large as well as smaller research teams with limited resources to guide CA development, including the distinct design and development challenges of CAs such as the creation of dialogs and the look and personality of the agent, grounded in current best evidence. Therefore, this research aimed to develop a conceptual framework for the design, development, evaluation, and implementation of smartphone-delivered, rule-based, goal-oriented, and text-based CAs for health care.

Methods

We developed the Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER) conceptual framework according to the methodology described by Jabareen [29], consisting of the iterative, qualitative analysis of multidisciplinary data based on the grounded theory method. It comprises 8 interlinked steps aimed at integrating and analyzing the data and developing and validating the conceptual framework [29] (Figure 1).

Figure 1. The 8 phases of the methodology by Jabareen [29] for conceptual framework development.

Step 1

We conducted 2 literature reviews. The first review aimed to summarize the current literature on conceptual frameworks for the design, development, and evaluation of mHealth interventions, and the second review focused on smartphone-delivered, rule-based CAs. A description of these literature reviews can be found in Multimedia Appendix 1 [5,30-62] and Multimedia Appendix 2 [5,30-62]. Multimedia Appendix 3 presents the search strategy used to retrieve the studies for the review of CAs.

Step 2 and Step 3

The screening of retrieved citations was performed in 2 stages, independently and in parallel, by DD and LM. The same 2 reviewers extracted data from all the included studies independently and in parallel. At all stages of screening and data extraction, the results were compared, and discrepancies were resolved by consensus between the reviewers.

Step 4

The data analysis followed qualitative meta-synthesis to systematically summarize the findings across all the included studies. This step involved grouping the concepts extracted from both literature reviews into overarching domains.

Step 5 and Step 6

The next 2 steps involved linking the overarching domains and developing the first iteration of the conceptual framework.

Step 7 and Step 8

The conceptual framework was further amended based on discussions among the research team members and feedback from colleagues collected in a seminar. We subsequently applied the conceptual framework to develop a rule-based, text-based, smartphone-delivered CA prototype (Precilla) designed to support healthy lifestyle changes and educate participants about diabetes. The development, feasibility, and acceptability of Precilla have been reported elsewhere [63,64].

The feedback received from team members and colleagues and the lessons learned during the application study led to the refinement of concepts and domain labels, definitions, order, and grouping that were derived in the current version of DISCOVER presented in this paper.

Ethical Considerations

This study was approved by the Nanyang Technological University Institutional Review Board (IRB-2018-11-032).

Results

A Framework for Guiding the Design, Development, Evaluation, and Implementation of Smartphone-Delivered, Rule-Based CAs in Health Care: Overview

The conceptual framework development was informed by the 2 literature reviews and iterative consultations within the research team. Further refinements were also informed by the development of our CA prototype (Precilla) [63,64] as well as by presentations at clinical seminars and conferences. Multimedia Appendix 4 outlines the methodology applied in the development of the DISCOVER framework according to each step described by Jabareen [29]. Multimedia Appendix 5 [63,64] presents the steps to develop CA Precilla mapped to the steps of the current version of the conceptual framework.

The 2 literature searches retrieved a total of 55 studies, of which 41 (75%) described conceptual frameworks for the design, development, and evaluation of mHealth interventions and 14 (25%) were clinical trials evaluating smartphone- and rule-based CAs. The findings from these reviews are presented in Multimedia Appendices 1 and 2. The “Characteristics of included studies” tables are presented in Multimedia Appendix 6 [47-58], Multimedia Appendix 7 [5,32,65-67], and Multimedia Appendix 8 [3,30,31,33,34,68-80].

The initial framework contained 8 steps. They were subsequently condensed into 5 steps augmented by 2 overarching themes relevant to all phases of the development process. Further refinements led to the framework presented in this paper consisting of an iterative process of design, development, evaluation, and implementation steps, each comprising several components, as presented in Figure 2 and described in the following sections.

Figure 2. The DISCOVER conceptual framework for the design, development, and evaluation of rule-based, smartphone-based conversational agents in health care.

Step 1: Design

The first stage comprised 4 interlinked steps encapsulating the initial conceptual work of identifying the health care focus of the CA, target users, multidisciplinary team members, and the CA delivery interface.

Defining the Goal

Overview

A clearly defined goal is the first step in the design process and the foundation that will guide the development and evaluation of the CA. This step consists of 3 interlinked areas of evaluation—completing a thorough needs assessment, defining the aim, and characterizing the end user and objectives—which, in turn, determine the parameters to be tested and reported. The CA goal was described in 64% (35/55) of the papers in our reviews [3,5,25-28,65,68-76,81-99].

Needs Assessment

The design process should commence with an in-depth needs assessment to understand existing gaps that may be filled by the CA. These may be informed by a literature review [83,90,91,96,100] to assess potential research areas and the needs and challenges of the target population, including not only patients but also caregivers, health care providers (HCPs), and other experts [25,26,87,89,95,98]. Researchers should also involve end users in this initial phase by using surveys and a variety of qualitative methods [70,87] such as in-depth interviews and focus group discussions to gather their views.

The Aim

Aligned with the needs assessment, the design team should formulate clear, attainable, and relevant objectives to drive the CA design and development process. It is important to consider the CA temporal profile, which characterizes 4 types of CAs according to the type and frequency of CA-user dialogs [101]. The CA temporal profile will also determine the type of objectives included, broadly classified as short term or long term [101]. A short-term objective refers to an outcome to be completed as soon as the interaction with the CA ends, such as medication reminders [30]. A long-term goal would involve several CA-user interactions being completed over a period, as in mental health interventions to promote mental well-being in the general population [3] or young people with cancer [70]. Complex CA interventions may include short- and long-term goals, such as CA Vik [30] providing medication reminders (short-term goal) and health education (long-term goal) to patients with breast cancer. Furthermore, Kowatsch et al [73] used prompts and reminder SMS text messages to enhance children’s discipline and routine, which are essential for the self-management of asthma.

Determining the End User

The next important design consideration is to determine the target population. An initial assessment should establish whether the CA will be offered to healthy users or individuals with a specific medical condition, caregivers, or HCPs. It is important to generate a detailed and accurate portrayal of the target user, including gender, age group, cultural beliefs and socioeconomic concerns, digital and health literacy, access to digital devices, and smartphone penetration rate. If the intervention is educational, a knowledge test should be implemented [73]. The acceptability of CAs by the target population and the perceived risk of using a CA for health care matters should be evaluated, particularly for severe or highly stigmatizing conditions [102] such as mental health disorders [103,104].

Creating the CA Identity

This step involves determining the CA’s name, appearance, tone of communication, language, and other characteristics that define its identity. This step was discussed in 25% (14/55) of the papers in our reviews [5,31,32,66,69,70,72-78].

CA Personality

User interaction with CAs appears to be enhanced when the CA displays a well-defined, positive, and empathic personality [105,106]. In general, giving a name and profile picture to the CA may enhance its social presence and user acceptance [107], although its effect appears to be small [106]. In health care settings, using a human-like avatar rendering realistic features, including medical attire, may increase user satisfaction [105], although avatars displaying highly realistic features may upset users and decrease engagement, an experience referred to as the “uncanny valley” [108].

Studies have consistently shown that CAs displaying empathy, relational behavior, and self-disclosure enhance the user experience [78,105] and increase the working alliance [109]. Conversely, users would notice if the CA did not convey empathy [69].

Acceptability may be further enhanced if the CA design acknowledges the specific cultural or demographic traits of the target population [73] or offers options to personalize the interface (eg, offering a male and female persona) [31,73]. Alternatively, CAs may explicitly disclose their identity [70] to reduce user expectations about their capabilities. Finally, CA personality should align with its intended function. For example, health care CAs often display one of two personality types: a more approachable, empathic coach-like personality, particularly if delivering behavior change interventions [77] supporting self-management of chronic disorders [73,78] and mental health conditions [3], or a health care professional persona to emphasize the legitimacy of the CA and its content [1].

Tone and Language

The language recommended for text-based interventions should be encouraging, positive, friendly, polite, and light-hearted and may include light humor while at the same time being formal [110]. To maintain the flow of the conversation, it may be advisable to use visual cues such as successive moving dots signaling that the CA is “typing” the next message.

The text should be written in clear, short sentences using simple language and avoid scientific jargon. The National Institutes of Health recommends that patient education materials be written at or below the sixth-grade reading level (ages of 11 and 12 years) to reach a diverse range of individuals with varying levels of literacy [111]. The readability of the text can be assessed using a scale such as the Flesch-Kincaid grade level to determine its suitability [112]. Furthermore, the CA should use the target population’s native language in its communications [75] and, if needed, the conversational content may be translated to one or more languages, particularly if the CA will be deployed in multiethnic, multilanguage societies.

With regard to the tone of the conversation, despite the text-based nature of the CA, it may be advisable to simulate more casual, verbal speech while avoiding the use of “textese” [113], a form of abbreviated written or typed language characterized by unconventional spelling and grammar (eg, “tonite” instead of “tonight”) and abbreviations and contractions (eg, “pls” instead of “please” or “wanna” for “want to”) [114]. Furthermore, words written in full capital letters should be avoided as they equate to shouting [110].

Emojis may be used to articulate emotions or other expressions more efficiently than text [70]. However, emojis are vulnerable to varied interpretations across cultures and contexts and should be used mindfully. Fadhil et al [115] noted a context-specific nature of emojis whereby they increased efficacy in a mental health intervention but did not help in promoting physical well-being.

CAs designed to address sensitive topics such as HIV and AIDS, sexually transmitted infections, or mental health disorders may emphasize the confidential nature of the messages or include code words to protect users’ privacy. This is particularly relevant in low- and middle-income settings, where family members may share a single smartphone [116].

Selecting the Delivery Interface

Human Involvement

Conditional to the CA’s aim, the design and development team may consider a “hybrid” intervention where the interaction with the CA would be complemented by regular interactions with HCPs offering timely feedback on a self-management technique or regular support and motivation [33,72,73]. Alternatively, as presented in the study by Stasinaki et al [79], the CA may be fitted with multiple channels, where the user can converse with the CA in one channel and directly with an HCP in another.

Peer support is recognized to play an important role in promoting adherence to self-management interventions [117] and a further point of human involvement to be considered. The CA intervention may include an additional communication channel for users to interact, share experiences, and receive peer support. For example, Wang et al [75] developed a WeChat intervention to support smoking cessation where the CA not only responded to individual users but also acted as a group moderator.

Delivery Channel

CAs may be delivered through a variety of channels, such as stand-alone apps [3,73] and existing messaging platforms [68,71,75] such as Facebook Messenger, Telegram, WeChat, and WhatsApp, or embedded in a website [69]. Each channel possesses its own set of complexities, and the decision regarding the delivery channel should be based on the target population needs and the expertise of the CA development team [118]. If the research team does not include app developers or computer scientists, the CA may be embedded in a messaging platform or may be developed using a CA development platform that offers templates or other design solutions for individuals with no previous programming knowledge [5,118], such as Chatfuel, ManyChat, and others. CAs are generally web-based, and some of these platforms are free of charge. Alternatively, if the team expertise or project budget allows, the CA may be delivered through a stand-alone app. This approach offers design flexibility, such as a variety of data collection sources including smartphone sensors, health programming interfaces, connected medical devices, and patient self-reported data [119]. The combination of subjective patient reports with objective, real-time data may reduce users’ responsibility to update their progress and at the same time receive relevant, dynamic coaching based on the current data [120], which in turn may increase adherence to the intervention.

In addition, factors associated with the target population may also affect the selection of the most suitable delivery channel and operating system (eg, Android or Apple’s iOS). For example, Kamita et al [71] implemented their CA on the messaging platform “LINE” as it was the most popular social network service in Japan, and Wang et al [75] selected WeChat, the most common messaging app in Hong Kong.

Communication Modalities

Aligned with the framework focus, text would be the CA’s main input and output modality. Messages should be brief, fit the mobile screen without scrolling [69], and be of an adequate font size to allow for comfortable reading. Moreover, if the CA targets populations for whom reading might be challenging, such as older adults or visually impaired individuals, text-to-speech assistive technology may be incorporated into the app.

Visual aids such as images or videos are useful to adapt content to audiences with lower educational attainment [121], deliver personal narratives relevant to the end users (eg, young people with cancer), or decrease the amount of textual information [76]. When using multimedia content, it is important to use high-resolution files to avoid pixelated or blurred images. Furthermore, if pictures are obtained from the web, developers should abide by copyright regulations and either source the pictures from free stock photo repositories, acquire the image rights, or produce the images in-house.

Assembling a Multidisciplinary Team

The composition of the design and development team would be based on the objectives of the intervention. In addition to the inclusion of health professionals with the relevant expertise, it is recommended to include end users as well [69,70]. For example, a CA to support a lifestyle intervention in overweight adolescents was developed by a multidisciplinary team including computer scientists, physicians, a psychotherapist, and diet and sports experts [72]. End-user involvement in the intervention design is critical to ensure that it aligns with user needs. User involvement was reported in a large number of studies in our review (36/55, 65%); for example, young people with cancer participated in focus groups to refine the content of a CA aimed at delivering positive psychology to enhance well-being [70], and young patients with asthma and their parents were part of a multidisciplinary team of experts who developed a CA to improve cognitive and behavioral skills [73]. In general, studies that mentioned the composition of their multidisciplinary teams often reported computer scientists and physicians as key members [72-75], although other health professionals such as physiotherapists [78], psychologists [3], and music therapists [76] may be included as well.

Step 2: Development

Developing the Content

Content development may involve determining the sources of information, adapting content to the target audience, defining the behavior change theories and techniques guiding the intervention [28,94], and establishing error management and safety-netting strategies [26-28, 30, 67-70, 73, 75, 77, 79, 80, 82, 83, 85, 87, 90, 92-97, 99, 100].

Evidence-Based Information

All health-related information included in the CA should be derived from reputable sources and adequately referenced. Sources of evidence-based information include comprehensive literature reviews; clinical practice guidelines; Cochrane systematic reviews; and reputable organization websites such as the World Health Organization, MEDLINE Plus, and the Centers for Disease Control and Prevention in the United States or the National Health Service Health A to Z in the United Kingdom [65]. For example, Kowatsch et al [73] used evidence from multiple sources such as published literature on the improvement of asthma management in children [122], technology acceptance research [123], and user-CA working alliances [124] to inform their intervention for asthma management.

Managing Errors

Another important aspect of content development is to ensure an adequate understanding of user requests, particularly for potentially serious or life-threatening health conditions. Safeguards to be implemented within the dialog include the request for clarification if the CA receives an unfamiliar input or directing the user to contact an HCP or a human administrator [125,126]. These strategies were included in TensioBot, an intervention to facilitate self-measurement of blood pressure where, after obtaining confirmation of a blood pressure measurement value outside the normal range, the CA alerted the attending physician [68]. Important strategies to manage unintended errors include using validated data entry fields; limiting the data input to predetermined number ranges, words, or characters; or including predefined options for the user to select.

Safety Netting

In general, health care CAs should include a disclaimer clearly stating that the intervention “does not replace healthcare provider’s advice.” Furthermore, in the case of health conditions associated with rapid deterioration of patient status leading to medical emergencies, such as cardiovascular conditions, diabetes, chronic pulmonary disorders, or mental health conditions that increase the risk of suicide, information should be included to assist users in managing an emergency situation, such as the provision of emergency services or crisis helpline telephone numbers [127], links to contact their primary physician, or clear advice on first aid treatments such as offering a sugary drink to manage a hypoglycemic event in a person with diabetes [128].

Types of Messages

The content and style of the messages should be aligned with the health condition and CA aim. Broadly, the messages may be educational [30,78] or motivational [34,77,79] or deliver reminders to perform a self-management task [68], input data [77], comply with preset tasks [73], take a medication, or attend an HCP appointment [68]. For CAs tasked with engaging with the user during clinic visits, it may be useful to include a status report or summary of the consultation [126].

CAs assuming a coach-like persona might emphasize sympathy, empathy, and participants’ achievements [78]. Interventions attempting to modify users’ behavior may deliver messages with higher emotional content, as reported in the study by Carfora et al [80], where only emotional messages led users to reduce red meat consumption. In addition, the Wang et al [75] CA used 4 types of messages to deliver a smoking cessation intervention: group announcements, health-related information, reminders to share positive results and progress, and fixed answers to frequently asked questions or requests.

Behavior Change Theories

CAs are increasingly used to promote behavior change [1,129]. Behavior change interventions are complex [130] and often comprise one or more behavior change techniques (BCTs) to induce change. In our assessment, 4% (2/55) of the studies used a behavior change theory to guide the intervention design, including the Health Action Process Approach [78] and the technology acceptance model [71]. In addition, 13% (7/55) of the studies [31,72-75,77,80] reported the use of specific BCTs such as goal setting, self-monitoring, tracking and feedback, social support, use of rewards, and anticipated regret.

For example, a study described a multicomponent behavior change intervention incorporating several BCTs, such as goal setting, self-monitoring, stimulus control, and behavioral contract, to support a healthy lifestyle for adolescents with obesity [34,79]. Furthermore, including group chats where peers or HCPs offer relevant information and emotional support may also assist in promoting positive behavior change, such as using a CA-led WeChat peer group to promote smoking cessation [75].

Optional Add-ons

Depending on the purpose of the CA, it may be appropriate to integrate data from external devices such as glucometers [131] or activity trackers [119]. Alternatively, access to smartphone sensor data [132] may facilitate passive monitoring of the user’s activity [79] or determine novel digital biomarkers to assess the user’s mood [133] or disease status [134]. The use of smartphone sensors for passive monitoring may further allow for real-time information sharing with HCPs, caregivers, or peers, a feature that may be particularly useful to monitor older people living alone, who may be at higher risk of falling, or individuals with severe chronic illnesses and multiple hospital admissions.

Building the Conversation Flow

A good CA is eloquent and knowledgeable and, thus, requires a meticulously crafted script. Conversation flow building was discussed in 35% (19/55) of the papers in our literature search [3,27,28,30-32,65,73,78,79,82,85,87,92-96,99].

Providing Suitable Answer Options

For a good conversation flow, the predefined answer options should be sufficient and appropriate to align with the user intent, defined as the user goals or intentions in each conversation turn. Constructing a mind map outlining the different facets associated with a topic (eg, medication adherence) and the likely influencing factors (lifestyle components or emotional state) would help predict the most relevant answer options to provide to the user [135].

Selecting a Mapping Tool

A mind map is a diagram representing concepts, ideas, or tasks generated from a key concept, which is generally represented in the center of the graph [136]. Mind maps are an effective method of brainstorming [137] that can be applied to building the conversation flows. Several web-based programs and platforms are available to organize the conversation flow, including tools specifically designed to build the CA conversation, such as SAP Conversational AI [138] or MobileCoach [35]. Conversation flows may also be built using nonspecific mind mapping software such as Xmind [139]. Mind mapping is useful to assist in recording the flow of conversations between different topics or different user interactions. A well-constructed conversation flow leads the conversation, guides the user, and can address all relevant questions about its purpose. Furthermore, interactivity, personalization, and consistent messaging have been noted as valued qualities [140].

Personalizing Content and Delivery

Interventions should be tailored to individual participant needs [110]. When compared with generic CAs, context, situational, or individually aware agents promote a more positive user experience [132]. Personalized interventions include addressing the user by their name or nickname [141]; delivering notifications and reminders tailored to individual needs [110], such as medication or appointment reminders; and notifications for missed activities or unread messages [30,78]. For example, an intervention promoting self-management of chronic pain offered personalized content based on the user’s type and duration of pain and personal interests [78].

An important caveat involves the design of interventions offering personalized advice based on user measurements, such as suggesting a treatment based on individually reported data (eg, blood glucose levels or blood pressure readings), as these interventions may require regulatory oversight and be considered a “mobile medical application” [142].

Selecting Appropriate Message Timing and Frequency

The timing and frequency of messages are important components when planning the intervention and may be determined by the intervention scope as well as user preference. Earlier studies on SMS text messaging interventions have suggested a preference for weekly messaging [113]. However, different intervention types may require a more adaptive message delivery system, such as smoking cessation programs that often require an increased volume of messages close to the desired quit date [143] or high-risk behavior prevention programs targeting binge drinking or inappropriate sexual behaviors timing their messages to when the risky behavior is expected to occur, for example, on a Friday night [116,141]. Therefore, strategies for message delivery and frequency could be adapted to suit the CA intervention.

Just-in-time adaptive interventions (JITAIs) leverage smartphone sensor data to “provide the right type (or amount) of support at the right time” [22]. Smartphone sensor data would determine and even predict “states of vulnerability” (susceptibility to negative health outcomes) [144] and “states of receptivity” (the capacity to receive, process, and use the intervention) [120] in the user when the intervention may be required and more useful. This novel approach may be particularly useful for behavior change interventions supporting a healthy lifestyle, such as increasing physical activity or adhering to a healthier diet, or supporting substance use remission [22,120]. Nevertheless, researchers considering this approach should take into account human and economic resources as JITAI design may require a larger development team that includes computer scientists and app developers.

Using Engagement Strategies

Strategies to keep the users engaged for the intended duration of the intervention are particularly important in health care settings. These aspects were discussed in 11% (6/55) of the studies in our reviews [3,30,31,73,78,79]. Reported strategies included notifications, weekly summaries, reminders, motivational statements, persuasive techniques, a high frequency of messages to promote habit formation, and daily encouragement. In addition, CA-specific engagement strategies included building rapport and attachment with the user [72,73] or adding gamified components to incentivize CA use for rewards and points [73,79].

Step 3: Evaluation and Implementation

Evaluation

The evaluation of digital interventions, including CAs, starts early in the development process and comprises several iterative steps. To ensure the validity of the results, the process must use a robust methodology that is adequate for the intervention design [15]. In digital health interventions, a commonly used evaluation methodology is the multiphase optimization strategy by Collins et al [145,146].

The CA evaluation follows 3 distinct stages representing the intervention development process. The initial iterations of the CA may be evaluated using one or more usability testing methods [147] aiming to produce a minimum viable prototype. Once this working prototype is ready, pilot and randomized trials may ensue to assess the effectiveness of the CA [148]. Several aspects of CA evaluation were discussed in 36% (20/55) of the studies in our reviews [25, 26,28, 33, 71,73, 83, 85, 86, 88, 89, 91-95, 97-100].

The evaluation design may include one or more aspects of the CA functionalities, including clinical or technical attributes and user experience. The outcomes should be clearly defined and include widely used and validated outcome measurement tools whenever possible to improve the comparability and reproducibility of the research results. Examples of outcome measurement tools include the Patient Health Questionnaire-9 [149] to screen for depression, the Flourishing Scale [150] to assess psychological well-being, the Brief Pain Inventory [151] to assess pain intensity and its interference in activities of daily living, and the Working Alliance Inventory-Short Revised [152] to evaluate the CA-user working alliance.

Usability Testing

The evaluation of the CA should start early in the development cycle [153]. In the initial stages, formative evaluation aims to assess the viability of the digital tool by assessing its usability, usefulness, and user experience [154] using one or more qualitative or quantitative research designs. Qualitative methods include surveys, interviews, focus group discussions, and “think aloud” protocols [147] in which users express their opinions about the product as they use it. Quantitative methods include closed-ended questionnaires, task completion assessments, and A/B testing [147,155]. An A/B test, split test, or controlled experiment compares two or more versions of a product to evaluate the intervention components that perform better or are preferred by the user [155]. This stage relates to the screening and confirming stages in the multiphase optimization strategy [145,146], which use a fractional factorial design to assess which components should be included in the digital intervention and the best dosages to use in a more cost-effective fashion. Finally, microrandomized trials are another novel methodology that is particularly useful for assessing and optimizing the delivery of JITAIs [156]. Microrandomized trials allow the randomization of multiple components to occur at multiple times triggered by predefined decision points [156] and have been used to evaluate CA interventions, as reported by Kramer et al [119,157].

Efficacy and Effectiveness of the CA Intervention

Once initial evaluations have determined the components that should be included in the intervention and the frequency of administration, a traditional randomized trial design should be implemented to assess the effectiveness of the CA intervention compared with current best practices [145,146,148]. Given the complexities and cost that a full-powered randomized controlled trial often entails, researchers may consider conducting a pilot study to refine the study methodology or assess the feasibility of the study design and participant recruitment strategies, among other aspects [158]. For example, Casas et al [77] conducted a pilot study to preliminarily assess a CA aimed at coaching participants to make healthier food choices, whereas Greer et al [70] evaluated a CA delivering a positive psychological intervention to young people with cancer.

User Engagement and Acceptability

Overview

Digital health interventions often report high rates of participant attrition, which may limit the validity of research findings and, more importantly, the effectiveness of the intervention. Therefore, the assessment of the CA-led intervention should be complemented by regular evaluations of end-user adherence to as well as engagement with and acceptability of the intervention. Several assessment methods are commonly used, including quantitative, data-driven analyses and qualitative assessments of users’ opinions.

Data-Driven Analyses

The definition of adherence to digital health interventions refers to the extent to which a user has interacted with the intervention [159]. This term may be used to define the degree to which a user interacts with the CA (greater adherence equals more time engaging with the intervention) or the degree to which the user-CA interaction complies with the prescribed recommendation (intended use of the intervention) [159]. In health care interventions, the concept of “intended use” is preferred, and it should be clearly defined during the CA design and development stage for the subsequent adherence measurements to be meaningful. Increased adherence to an intervention may be related to its increased effectiveness [75,160], although the data are not conclusive [3,161,162].

User engagement with the CA may be evaluated using data metrics such as the times the user opened the app, time spent interacting with the CA, the extent of the dialog, or the number of screens opened if the CA also includes other functions [70]. Chaix et al [30] measured use duration, interest in various educational contents, and level of interactivity as indicators of engagement. Nevertheless, researchers should consider the challenges of defining engagement with digital interventions, which may include other user-related variables such as the severity or stage of the disease as well as the long-term engagement with the CA [163].

Other aspects of CA use, such as underused or missing topics or CA functionalities not working as intended, may also be assessed. CA use analytics are often embedded in host platforms. Commercial platforms such as ManyChat [164] may offer a variety of built-in analytics tools such as the number of times the CA is accessed. Some of these platforms offer free-of-charge services. For health care CAs, the open-source MobileCoach platform [35] offers flexible, customizable use analytics.

Qualitative Evaluation

Acceptability refers to the “affective attitudes towards a new digital health intervention” [165]. It is a dynamic concept comprising the intention to engage with the novel CA, the actual interaction with the CA, and the postengagement satisfaction [165].

Acceptability is a subjective term that is generally assessed using questionnaires or other qualitative methods such as focus groups or interviews. For example, Kowatsch et al [73] evaluated the acceptance of a CA to support asthma self-management using a 7-point Likert scale (strongly agree-strongly disagree) for perceived usefulness, ease of use, enjoyment, and use intention, and Echeazarra et al [68] used a survey with questions on ease of use, preference for the CA over existing methods, CA usefulness for its intended purpose, and whether the user had stopped using it as measures of acceptability and satisfaction. Furthermore, Gabrielli et al [69] facilitated a participatory design workshop where suggestions for improvement were provided via open-ended questions, and Ly et al [3] conducted semistructured interviews on the benefits, opportunities, and challenges associated with the CA for mental health. Yan et al [166] described a very involved process of evaluation of an mHealth intervention to promote physical activity. A focus group discussion was organized whereby each SMS text message was displayed and participants were required to respond either with “Yes, I like it” or “No, let’s change it to make it better.” This voting was then followed by a discussion in which suboptimal messages were improved and the strengths of effective messages were noted. Finally, participants may also be questioned about their willingness to recommend the conversation to others, which is a good indicator of satisfaction and acceptability [70].

Several aspects of user engagement and acceptability may be measured using one of several app quality rating tools, of which the most commonly used one is the Mobile App Rating Scale [167]. The use of standardized, validated rating scales may improve the reproducibility of this research area and facilitate the reporting of trial results, although they are not specific for CAs.

Economic Evaluation

The economic evaluation includes not only the affordability of the project but also the cost-benefits associated with developing the CA. These analyses should consider the end-user perspective as well as the potential benefits for the health care system in general [168,169]. Digital health interventions appear to be cost-effective [170], although reports often present varying, inconclusive results [171]. Although it is often mentioned that one of the potential advantages of digital health interventions, particularly in the long term, may be a significant decrease in health care costs [172], the upfront expenses of developing the digital intervention might be substantial. For example, Kowatsch et al [73] reported upfront expenses of approximately US $250,000 to develop a CA to support asthma self-management in young patients. The development costs will vary conditional to the type and functionalities of the CA, the use of a messaging platform or development as a stand-alone app, and the number of team members, among other aspects. Despite the increasing importance of conducting economic evaluations of digital health care interventions, only 2% (1/55) of the studies included in our reviews reported economic evaluation data [73]. Recent documents from the World Health Organization [168] and the International Training and Education Center for Health [169] at the University of Washington in the United States, as well as a recent review [171], present a practical overview of how to perform economic evaluations.

Implementation

Once the effectiveness of the CA intervention has been determined in rigorous clinical trials, the research team should consider implementing the intervention in the broader population. Implementation research aims to integrate research and practice [173] and understand the users and context in which an intervention would be implemented. The research methods, including pragmatic trials, participatory action research, and mixed methods studies, aim to assess the intervention “acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, coverage, and sustainability” [174-177]. Important considerations include the need to upgrade the systems to adapt to higher traffic, personnel to provide long-term system maintenance and updates, and the costs these changes may incur [25,26]. Furthermore, the team should consider CA intervention commercialization strategies, including engaging HCPs, health insurers, or governmental organizations if aligned with the health care focus of the intervention [26].

Finally, the team should be aware of and comply with the current regulatory frameworks for digital health interventions. Increasingly, countries are developing national policy frameworks to regulate the evaluation, use, and commercialization of digital health interventions [178], particularly if the intervention is considered a digital therapeutic [179]. Digital therapeutics refer to “evidence-based therapeutic interventions that are driven by high-quality software programs to prevent, manage, or treat a medical disorder or disease” [179], may require a provider’s prescription to be accessed [179], and often require approval from official regulatory bodies such as the Food and Drug Administration in the United States [180] and the Conformité Européenne mark in the European Union (EU) [181].

Cross-Cutting Considerations

The themes described in this section are relevant throughout all the design stages referred to in the previous sections.

User-Centered Design and Co-design

User-centered design refers to design practices that include the end users’ views to guide the process, either in a passive, consultive manner or as active participants in the design process (co-design) [182]. Several approaches to user-centered design have been described. They share the general principles of involving users during the design process, although the steps involved in the process and the type and extent of end-user involvement may differ. They include but may not be limited to human-centered design [183,184] and design thinking [185] (often considered synonyms), user-centered design [186], co-design [182], and participatory action research [187].

End users include patients, caregivers, HCPs, or other relevant stakeholders. There are several benefits of including end users as part of the CA development team, such as a better understanding of users’ and communities’ needs, development of culturally sensitive products, and improved communication between the different stakeholders [188,189]. This, in turn, may increase compliance with the intervention and improve health-related outcomes [190]. For example, to develop a CA to promote positivity and well-being in young people after cancer treatment, Greer et al [70] conducted interviews and focus groups with young adults treated for cancer to refine the informational content.

During the evaluation stage, thinking-out-loud usability testing is another example of a user-centered design methodology in the design of digital health interventions, including CAs [191].

The role of user-centered design in the development of digital health interventions has been repeatedly emphasized by several frameworks included in our review (36/55, 65%) [25,26,36-39,41-46,62,69,70,73,76,82-85,87-100,110].

Privacy and Security

Overview

Safeguarding the privacy and security of CA users’ data is essential and should be a part of the entire design and development cycle. Health information is considered personal, sensitive information that should be protected at all times. The level of data protection should align with the data collected by the CA, if any. Therefore, the functionalities of the CA will determine the type of sensitive data to be collected and guide the inclusion of data protection software such as firewalls and encryption.

In general, developers should minimize the amount of personal and sensitive information collected from users by asking specific questions to avoid oversharing or simply providing predetermined responses instead of using free text. Furthermore, all CAs should include a privacy policy that is brief and written in clear language outlining the data collected and the uses of these data. All data must be encrypted during transit (when the message is being sent) and at rest (when the message has been delivered) [192]. The platform on which the CA will be deployed may also vary according to the CA functionalities. For example, a CA collecting users’ personal data should not be deployed on proprietary or messaging platforms as the platform data management policies may not be clearly reported [65] or data sharing with third parties may occur without informing the user [193]. This might create an ever-increasing digital footprint, potentially allowing for user identification from data aggregation rather than actually identifiable information [194].

A 2020 framework for governing the responsible use of CAs in health care highlighted the importance of safeguarding data privacy, including user health data, history of interactions, and disclosure of user data even if unintended [195]. In addition, the framework highlighted the user’s right to access their personally identifiable information, the requirement of user consent before recording or saving health-related data, and the preclusion of using the stored data as a means of surveillance or to discriminate users against health care privileges or coverage [195].

Compliance With Data Privacy Laws

Health care CAs that collect users’ sensitive data must comply with country-relevant data privacy laws, such as the Health Insurance Portability and Accountability Act in the United States [192] or the General Data Protection Regulation (GDPR) in the EU [196]. These laws’ jurisdiction is generally limited to the issuing country; however, the GDPR applies to any EU citizen within or outside the EU. The GDPR, which went into effect in 2018, is an overarching law that aims to enhance the rights of individuals over their personal data, defined as any data that may allow for the identification of a person on their own or combined with other data, including pseudonymized data [196]. Alternatively, the Health Insurance Portability and Accountability Act is industry-specific and applies only to health-related data [197]. Other countries have adopted their own data protection laws and regulations. In Singapore, the Personal Data Protection Act is a baseline regulatory framework informing the collection, distribution, and use of personal data [198].

In addition to the aforementioned GDPR, children’s data are generally more stringently safeguarded. For example, in the United States, the Children’s Online Privacy Protection Act [199] requires that verifiable parental consent be obtained by all digital operators (not restricted to health care) collecting data from children (aged < 13 years). Similar considerations are included within the GDPR and the Singapore Personal Data Protection Act, with the caveat that, in some European countries, parental consent is required for children and adolescents aged < 16 years.

Discussion

Principal Findings

We present a new conceptual framework for the design, development, evaluation, and implementation of smartphone-delivered, rule-based, and text-based CAs. The DISCOVER conceptual framework includes 8 iterative steps arranged in three main groups: (1) design, which includes defining the goal, creating an identity, assembling the team, and selecting the delivery interface; (2) development, which comprises developing the content and building the conversation flow; and (3) evaluation and implementation. User-centered design and privacy and security were included as cross-cutting considerations, which are relevant at every stage of the framework.

This framework was based on the comprehensive analysis of 36 mHealth frameworks, 5 CA taxonomies, and 14 primary studies reporting on the design and development of rule-based health care CAs. The framework was applied in a web-based pilot study using a CA deployed on Facebook Messenger. The existing mHealth frameworks provided general guidelines to develop mHealth interventions for health care, from the characterization of the target population to evaluation, with emphasis on the application of user-centered design techniques in all stages of development. Concurrently, the CA taxonomies provided focused on several aspects of CA design and evaluation as well as the impact of design features on CA-user interactions.

Considering the multifaceted nature of embodied CAs, we decided to focus on CAs that are nonembodied.

Comparisons With Prior Work

The existing frameworks for the design and development of mHealth interventions provide detailed guidance in all steps of the intervention development, starting with an understanding of the needs and the profile of the end users through a review of existing literature or formative research [67], and they emphasize the need for patient and public involvement to make the intervention as relevant to the target population as possible [90,98]. These frameworks also described the importance of conducting iterative evaluations to identify limitations before testing the mHealth intervention in a larger-scale trial [28,98,99]. However, the literature on the design and development of CAs was restricted to the development of taxonomies that were not limited to health care describing CA design platforms [5], classification of CAs according to the approach to conversation design [67], characteristics of embodied agents [66], or the impact of CA characteristics on user interactions [32]. Moreover, the taxonomy by Denecke et al [65] referred to health care CAs, but they focused exclusively on CA evaluation. Therefore, a conceptual framework guiding the development of health care CAs was needed to expand previous mHealth frameworks with elements particularly relevant to CAs, such as personality development, converting evidence-based content into conversations, and using novel research designs for evaluation. Furthermore, our framework focused particularly on the development of the CA, including personality, display of empathy, and disclosure of its identity as a computer-generated entity without human involvement, and on the development of dialogs guided by up-to-date evidence-based information sources.

This framework described the development of rule-based CAs, allowing the research team total control of the conversation and dialog flow. There are several reasons for this. First, our framework presents easy-to-follow steps that could be applied by smaller research teams that do not include computer science or AI specialists or that undertake the CA development project under restricted financial resources. Second, we aimed to provide guidance for the development of goal-oriented CAs aimed at delivering health education content or simple interventions aimed at improving healthy lifestyle choices or self-management behavior and, therefore, prioritize control over the conversation content using a rule-based paradigm, albeit less engaging, over AI algorithms that have yet to become truly explainable.

Implications for Future Research

Future research should apply the DISCOVER conceptual framework to the development of CAs offering behavior change interventions aimed at different specialties, settings (hospital or outpatient), target groups, and cultures. Moreover, although the use of theories in the design of behavior change interventions is favored and may increase the effectiveness of the intervention [77,200], it is still unclear which behavior change theories or techniques are better suited for CA-led interventions. Alternatively, because of the interactive nature of CAs, it would be appropriate to assess whether behavior change interventions previously proved effective in traditional face-to-face settings are equally effective when led by a CA.

Although the concepts of identity creation, conversational flow, and delivery are important, their relative relevance to varying target populations is still unknown. In addition, more research on the assessment of health care chatbot interventions can help inform the ideal health-related outcome measures and digital data sets required for a comprehensive evaluation. Finally, although this framework is comprehensive and many components may apply to AI CAs, a separate framework is needed to describe specific aspects relevant to AI CAs, such as dialog development using machine learning or natural language processing techniques, voice versus text parsing, and many others.

Strengths

This is, to the best of our knowledge, the first conceptual framework outlining the steps required to develop a smartphone-delivered, rule-based health care CA offering clear yet comprehensive guidelines to accommodate health care researchers with varying computer science expertise.

The DISCOVER framework builds on an analysis of existing mHealth frameworks and a stringent analysis of rule-based CA literature complemented by the team’s demonstration of its applicability in the development of a rule-based CA to support lifestyle changes in people at risk of developing diabetes.

Limitations

Much of the information provided is anecdotal or derived from research conducted on SMS text messaging and other mHealth interventions because of the scarcity of research on the evidence-based development of rule-based CAs for health care. Therefore, this framework provides an overview of the main steps required to develop a rule-based CA.

The descriptions and examples presented in the conceptual framework focused on CA interventions for end users to support either a healthy lifestyle or the management of a chronic condition, as derived from the literature reviews and our experience developing a CA. Nevertheless, the design and development principles discussed in this study could apply to other relevant user groups such as caregivers and health care professionals.

Furthermore, this framework is focused on rule-based CAs and, although it may guide researchers in the development of particular aspects of AI CAs, it does not provide guidance on the development of AI-based conversations. In addition, the economic, social, and behavioral characteristics of different populations may limit its generalizability.

Conclusions

The interest in and potential for CAs in health care are growing, but guidelines to design, develop, evaluate, and implement these interventions are currently lacking. Drawing on published evidence, the DISCOVER conceptual framework provides the first attempt to fill this void. The process was divided into 8 iterative steps arranged in 3 overarching groups and complemented by 2 cross-cutting considerations. Future research should explore aspects of CA development such as the use of behavior change theories and privacy and safety concerns. Further evaluation of this framework in diverse health care areas and settings and for a variety of users is needed to demonstrate its validity.

Acknowledgments

This research was supported by the Ageing Research Institute for Society and Education (ARISE), Nanyang Technological University, Singapore. This study was also supported by the Singapore Ministry of Education under the Singapore Ministry of Education Academic Research Fund Tier 1 (RG36/20). This research was conducted as part of the Future Health Technologies program, which was established collaboratively between ETH Zürich and the National Research Foundation, Singapore. This research was supported by the National Research Foundation, Prime Minister’s Office, Singapore, under its Campus for Research Excellence and Technological Enterprise program.

Authors' Contributions

DAD designed the study, extracted the data, conducted the analysis, and wrote the manuscript. LM conducted the analysis and wrote the manuscript. M-HRH, SJ, TK, and RA provided a critical review of the manuscript. LTC conceptualized and designed the study, provided a critical review of the manuscript, and provided supervision at all steps of the research. All authors approved the final version of the manuscript, and they take accountability for all aspects of this work.

Conflicts of Interest

TK is affiliated with the Centre for Digital Health Interventions, a joint initiative of the Institute for Implementation Science in Health Care at the University of Zurich; the Department of Management, Technology and Economics at ETH Zurich; the Future Health Technologies Programme at the Singapore-ETH Centre; and the School of Medicine and Institute of Technology Management at the University of St. Gallen. Centre for Digital Health Interventions is funded in part by CSS, a Swiss health insurer. TK is also a cofounder of Pathmate Technologies, a university spin-off company that creates and delivers digital clinical pathways. However, neither CSS nor Pathmate Technologies were involved in this study. SJ is also affiliated with Salesforce Research. However, Salesforce Research was not involved in this study. The other authors declare that they have no conflicts of interest.

Multimedia Appendix 1

Literature review of conceptual frameworks for the design, development, and evaluation of mobile health interventions.

References

Tudor Car L, Dhinagaran DA, Kyaw BM, Kowatsch T, Joty S, Theng YL, et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res 2020 Aug 07;22(8):e17158

Abd-Alrazaq AA, Alajlani M, Ali N, Denecke K, Bewick BM, Househ M. Perceptions and opinions of patients about mental health chatbots: scoping review. J Med Internet Res 2021 Jan 13;23(1):e17828

Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv 2017 Dec;10:39-46

Milne-Ives M, de Cock C, Lim E, Shehadeh MH, de Pennington N, Mole G, et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res 2020 Oct 22;22(10):e20346

Diederich S, Brendel AB, Kolbe LM. Towards a taxonomy of platforms for conversational agent design. In: Proceedings of the 14th International Conference on Wirtschaftsinformatik. 2019 Presented at: WI '19; February 24-27, 2019; Siegen, Germany p. 1100-1114 URL: https:/​/web.​archive.org/​web/​20200619062226id_/​https:/​/aisel.​aisnet.org/​cgi/​viewcontent.​cgi?article=1247 & context=wi2019

Mohamad Suhaili S, Salim N, Jambli MN. Service chatbots: a systematic review. Expert Syst Appl 2021 Dec;184:115461.

Bavaresco R, Silveira D, Reis E, Barbosa J, Righi R, Costa C, et al. Conversational agents in business: a systematic literature review and future research directions. Comput Sci Rev 2020 May;36:100239.

Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017 Jun 06;4(2):e19

Gaffney H, Mansell W, Edwards R, Wright J. Manage Your Life Online (MYLO): a pilot trial of a conversational computer-based intervention for problem solving in a student sample. Behav Cogn Psychother 2014 Nov;42(6):731-746.

Ireland D, Liddle J, McBride S, Ding H, Knuepffer C. Chat-bots for people with Parkinson's disease: science fiction or reality? Stud Health Technol Inform 2015;214:128-133.

Jack A. Rwanda venture tests digital health potential in developing world. Financial Times. 2021 Jan 18. URL: https://www.ft.com/content/4fe33c92-cbd5-459a-8df6-20d0d1f57ec8 [accessed 2022-01-19]

Palanica A, Flaschner P, Thommandram A, Li M, Fossat Y. Physicians' perceptions of chatbots in health care: cross-sectional Web-based survey. J Med Internet Res 2019 Apr 05;21(4):e12887

Joshi I. Waiting for deep medicine. Lancet 2019 Mar 23;393(10177):1193-1194.

Ramesh K, Ravishankaran S, Joshi A, Chandrasekaran K. A survey of design techniques for conversational agents. In: Proceedings of the 2nd International Conference on Information, Communication and Computing Technology. 2017 Presented at: ICICCT '17; May 13, 2017; New Delhi, India p. 336-350.

Laranjo L, Dunn AG, Tong HL, Kocaballi AB, Chen J, Bashir R, et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc 2018 Sep 01;25(9):1248-1258

Gao J, Galley M, Li L. Neural Approaches to Conversational AI: Question Answering, Task-Oriented Dialogues and Social Chatbots. Delft, The Netherlands: Now Publishers; 2019.

Wadden JJ. Defining the undefinable: the black box problem in healthcare artificial intelligence. J Med Ethics (forthcoming) 2021 Jul 21:medethics-2021-107529-2021-107529.

Quinn TP, Jacobs S, Senadeera M, Le V, Coghlan S. The three ghosts of medical AI: can the black-box present deliver? Artif Intell Med 2022 Feb;124:102158.

Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. Lancet Digit Health 2021 Nov;3(11):e745-e750

Silver L. Smartphone ownership is growing rapidly around the world, but not always equally. Pew Research Center. 2019 Feb 5. URL: https:/​/www.​pewresearch.org/​global/​2019/​02/​05/​smartphone-ownership-is-growing-rapidly-around-the-world -but-not-always-equally/​ [accessed 2022-03-28]

Dogan E, Sander C, Wagner X, Hegerl U, Kohls E. Smartphone-based monitoring of objective and subjective data in affective disorders: where are we and where are we going? Systematic review. J Med Internet Res 2017 Jul 24;19(7):e262

Nahum-Shani I, Smith SN, Spring BJ, Collins LM, Witkiewitz K, Tewari A, et al. Just-in-Time Adaptive Interventions (JITAIs) in mobile health: key components and design principles for ongoing health behavior support. Ann Behav Med 2018 May 18;52(6):446-462

Marcolino MS, Oliveira JA, D'Agostino M, Ribeiro AL, Alkmim MB, Novillo-Ortiz D. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018 Jan 17;6(1):e23

Eze P, Lawani LO, Acharya Y. Short message service (SMS) reminders for childhood immunisation in low-income and middle-income countries: a systematic review and meta-analysis. BMJ Glob Health 2021 Jul;6(7):e005035

Whittaker R, Merry S, Dorey E, Maddison R. A development and evaluation process for mHealth interventions: examples from New Zealand. J Health Commun 2012;17 Suppl 1:11-21.

Mummah SA, Robinson TN, King AC, Gardner CD, Sutton S. IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J Med Internet Res 2016 Dec 16;18(12):e317

Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res 2014 Jun 05;16(6):e146

Zhang J, Oh YJ, Lange P, Yu Z, Fukuoka Y. Artificial intelligence chatbot behavior change model for designing artificial intelligence chatbots to promote physical activity and a healthy diet: viewpoint. J Med Internet Res 2020 Sep 30;22(9):e22845

Jabareen Y. Building a conceptual framework: philosophy, definitions, and procedure. Int J Qual Methods 2009 Dec 01;8(4):49-62.

Chaix B, Bibault JE, Pienkowski A, Delamon G, Guillemassé A, Nectoux P, et al. When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR Cancer 2019 May 02;5(1):e12856

Heldt K, Büchter D, Brogle B, Chen-Hsuan IS, Rüegger D, Filler A, et al. Telemedicine therapy for overweight adolescents: first results of a novel smartphone app intervention using a behavioural health platform. Obes Facts 2018 May 26;11(Suppl 1):214-215

Zierau N, Elshan E, Visini C, Janson A. A review of the empirical literature on conversational agents and future research directions. In: Proceedings of the 2020 International Conference on Information Systems. 2020 Presented at: ICIS '20; December 13-16, 2020; Hyderabad, India p. 1414.

l'Allemand D, Shih CH, Heldt K, Büchter D, Brogle B, Rüegger D, et al. Design and interim evaluation of a smartphone app for overweight adolescents using a behavioural health intervention platform. Obes Rev 2018 Dec 03;19(Suppl 1):102.

Stasinaki A, Brogle B, Buchter D, Shih CH, Heldt K, White C, et al. A novel digital health intervention improves physical performance in obese youth. In: Joint Annual Meeting Swiss Society of Paediatrics, Swiss Society of Paediatric Surgery and Swiss Society of Child and Adolescent Psychiatry and Psychotherapy. 2018 Presented at: SMW '18; May 24-25, 2018; Lausanne, Switzerland p. 10S.

MobileCoach. URL: https://www.mobile-coach.eu/ [accessed 2022-03-02]

Bauerle Bass S, Jessop A, Gashat M, Maurer L, Alhajji M, Forry J. Take Charge, Get Cured: the development and user testing of a culturally targeted mHealth decision tool on HCV treatment initiation for methadone patients. Patient Educ Couns 2018 Nov;101(11):1995-2004.

Chen Y, Wu F, Wu Y, Li J, Yue P, Deng Y, et al. Development of interventions for an intelligent and individualized mobile health care system to promote healthy diet and physical activity: using an intervention mapping framework. BMC Public Health 2019 Oct 17;19(1):1311

Curtis KE, Lahiri S, Brown KE. Targeting parents for childhood weight management: development of a theory-driven and user-centered healthy eating app. JMIR Mhealth Uhealth 2015 Jun 18;3(2):e69

Farao J, Malila B, Conrad N, Mutsvangwa T, Rangaka MX, Douglas TS. A user-centred design framework for mHealth. PLoS One 2020 Aug 19;15(8):e0237910

Jindal D, Gupta P, Jha D, Ajay VS, Goenka S, Jacob P, et al. Development of mWellcare: an mHealth intervention for integrated management of hypertension and diabetes in low-resource settings. Glob Health Action 2018;11(1):1517930

Kazemi DM, Borsari B, Levine MJ, Lamberson KA, Dooley B. REMIT: development of a mHealth theory-based intervention to decrease heavy episodic drinking among college students. Addict Res Theory 2018;26(5):377-385

McBride B, Nguyen LT, Wiljer D, Vu NC, Nguyen CK, O'Neil J. Development of a maternal, newborn and child mHealth intervention in Thai Nguyen Province, Vietnam: protocol for the mMom project. JMIR Res Protoc 2018 Jan 11;7(1):e6

Patel S, Arya M. The BUS framework: a comprehensive tool in creating an mHealth app utilizing behavior change theories, user-centered design, and social marketing. J Mob Technol Med 2017 Apr;6(1):39-45

Sporrel K, De Boer RD, Wang S, Nibbeling N, Simons M, Deutekom M, et al. The design and development of a personalized leisure time physical activity application based on behavior change theories, end-user perceptions, and principles from empirical data mining. Front Public Health 2020 Feb 2;8:528472

Sun CJ, Anderson KM, Kuhn T, Mayer L, Klein CH. A sexual health promotion app for transgender women (trans women connected): development and usability study. JMIR Mhealth Uhealth 2020 May 12;8(5):e15888

Verbiest ME, Corrigan C, Dalhousie S, Firestone R, Funaki T, Goodwin D, et al. Using codesign to develop a culturally tailored, behavior change mHealth intervention for indigenous and other priority communities: a case study in New Zealand. Transl Behav Med 2019 Jul 16;9(4):720-736.

Bartholomew LK, Parcel GS, Kok G. Intervention mapping: a process for developing theory- and evidence-based health education programs. Health Educ Behav 1998 Oct;25(5):545-563.

Norman DA. User Centered System Design: New Perspectives on Human-Computer Interaction. Boca Raton, FL, USA: CRC Press; 1986.

Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011 Apr 23;6:42

Hevner AR, March ST, Park J, Ram S. Design science in information systems research. MIS Q 2004 Mar;28(1):75-105.

Get Started with Design Thinking. Hasso Plattner Institute of Design at Stanford University. 2022. URL: https://dschool.stanford.edu/resources/getting-started-with-design-thinking [accessed 2022-02-03]

Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A, "Psychological Theory" Group. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care 2005 Feb;14(1):26-33

Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ 2021 Sep 30;374:n2061

Dijkstra A, De Vries H. The development of computer-generated tailored interventions. Patient Educ Couns 1999 Feb;36(2):193-203.

Oinas-Kukkonen H, Harjumaa M. Persuasive systems design: key issues, process model, and system features. Commun Assoc Inf Syst 2009 May 1;24:485-500.

Michie S, Hyder N, Walia A, West R. Development of a taxonomy of behaviour change techniques used in individual behavioural support for smoking cessation. Addict Behav 2011 Apr;36(4):315-319.

Bratteteig T, Bødker K, Dittrich Y, Holst Mogensen P, Simonsen J. Methods: organising principles and general guidelines for participatory design projects. In: Simonsen J, Robertson T, editors. Routledge International Handbook of Participatory Design. London, UK: Routledge; 2012:117-144.

Wagner EH, Austin BT, Davis C, Hindmarsh M, Schaefer J, Bonomi A. Improving chronic illness care: translating evidence into action. Health Aff (Millwood) 2001;20(6):64-78.

Provoost S, Lau HM, Ruwaard J, Riper H. Embodied conversational agents in clinical psychology: a scoping review. J Med Internet Res 2017 May 09;19(5):e151

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101.

mHealth: new horizons for health through mobile technologies: second global survey on eHealth. World Health Organization. 2011. URL: https://apps.who.int/iris/bitstream/handle/10665/44607/9789241564250_eng.pdf?sequence=1 & isAllowed=y [accessed 2022-09-21]

Modi D, Gopalan R, Shah S, Venkatraman S, Desai G, Desai S, et al. Development and formative evaluation of an innovative mHealth intervention for improving coverage of community-based maternal, newborn and child health services in rural areas of India. Glob Health Action 2015 Feb 16;8:26769

Dhinagaran DA, Sathish T, Kowatsch T, Griva K, Best JD, Tudor Car L. Public perceptions of diabetes, healthy living, and conversational agents in Singapore: needs assessment. JMIR Form Res 2021 Nov 11;5(11):e30435

Dhinagaran DA, Sathish T, Soong A, Theng Y, Best J, Tudor Car L. Conversational agent for healthy lifestyle behavior change: Web-based feasibility study. JMIR Form Res 2021 Dec 03;5(12):e27956

Denecke K, Warren J. How to evaluate health applications with conversational user interface? Stud Health Technol Inform 2020 Jun 16;270:976-980.

ter Stal S, Kramer LL, Tabak M, op den Akker H, Hermens H. Design features of embodied conversational agents in eHealth: a literature review. Int J Human Comput Stud 2020 Jun;138:102409.

Hussain S, Sianaki OA, Ababneh N. A survey on conversational agents/chatbots classification and design techniques. In: Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications. 2019 Presented at: WAINA '19; March 27-29, 2019; Matsue, Japan p. 946-956.

Echeazarra L, Pereira J, Saracho R. TensioBot: a chatbot assistant for self-managed in-house blood pressure checking. J Med Syst 2021 Mar 15;45(4):54.

Gabrielli S, Rizzi S, Carbone S, Donisi V. A chatbot-based coaching intervention for adolescents to promote life skills: pilot study. JMIR Hum Factors 2020 Feb 14;7(1):e16762

Greer S, Ramo D, Chang YJ, Fu M, Moskowitz J, Haritatos J. Use of the chatbot "Vivibot" to deliver positive psychology skills and promote well-being among young people after cancer treatment: randomized controlled feasibility trial. JMIR Mhealth Uhealth 2019 Oct 31;7(10):e15018

Kamita T, Ito T, Matsumoto A, Munakata T, Inoue T. A chatbot system for mental healthcare based on SAT counseling method. Mob Inf Syst 2019 Mar 03;2019:1-11.

Kowatsch T, Nißen M, Shih CH, Rüegger D, Volland D, Filler A, et al. Text-based healthcare chatbots supporting patient and health professional teams: preliminary results of a randomized controlled trial on childhood obesity. In: Persuasive Embodied Agents for Behavior Change. 2017 Presented at: PEACH '17; August 27, 2017; Stockholm, Sweden.

Kowatsch T, Schachner T, Harperink S, Barata F, Dittler U, Xiao G, et al. Conversational agents as mediating social actors in chronic disease management involving health care professionals, patients, and family members: multisite single-arm feasibility study. J Med Internet Res 2021 Feb 17;23(2):e25060

Kowatsch T, Volland D, Shih I, Rüegger D, Künzler F, Barata F, et al. Design and evaluation of a mobile chat app for the open source behavioral health intervention platform MobileCoach. In: Proceedings of the 12th International Conference on Designing the Digital Transformation. 2017 Presented at: DESRIST '17; May 30-June 1, 2017; Karlsruhe, Germany p. 485-489.

Wang H, Zhang Q, Ip M, Fai Lau JT. Social media–based conversational agents for health management and interventions. Computer 2018 Aug;51(8):26-33.

Denecke K, Hochreutener SL, Pöpel A, May R. Self-anamnesis with a conversational user interface: concept and usability study. Methods Inf Med 2018 Nov;57(05-06):243-252.

Casas J, Mugellini E, Khaled O. Food diary coaching chatbot. In: Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 2018 Presented at: UbiComp '18; October 8-12, 2018; Singapore, Singapore p. 1676-1680.

Hauser-Ulrich S, Künzli H, Meier-Peterhans D, Kowatsch T. A smartphone-based health care chatbot to promote self-management of chronic pain (SELMA): pilot randomized controlled trial. JMIR Mhealth Uhealth 2020 Apr 03;8(4):e15806

Stasinaki A, Büchter D, Shih CH, Heldt K, Güsewell S, Brogle B, et al. Effects of a novel mobile health intervention compared to a multi-component behaviour changing program on body mass index, physical capacities and stress parameters in adolescents with obesity: a randomized controlled trial. BMC Pediatr 2021 Jul 09;21(1):308

Carfora V, Bertolotti M, Catellani P. Informational and emotional daily messages to reduce red and processed meat consumption. Appetite 2019 Oct 01;141:104331.

Depp C, Torous J, Thompson W. Technology-based early warning systems for bipolar disorder: a conceptual framework. JMIR Ment Health 2016 Sep 07;3(3):e42

Direito A, Walsh D, Hinbarji M, Albatal R, Tooley M, Whittaker R, et al. Using the intervention mapping and behavioral intervention technology frameworks: development of an mHealth intervention for physical activity and sedentary behavior change. Health Educ Behav 2018 Jun;45(3):331-348.

Duff O, Walsh D, Malone S, McDermott L, Furlong B, O'Connor N, et al. MedFit app, a behavior-changing, theoretically informed mobile app for patient self-management of cardiovascular disease: user-centered development. JMIR Form Res 2018 Apr 27;2(1):e8

Fedele DA, McConville A, Moon J, Thomas JG. Topical review: design considerations when creating pediatric mobile health interventions: applying the IDEAS framework. J Pediatr Psychol 2019 Apr 01;44(3):343-348

Good A, Omisade O. Linking activity theory with user centred design: a human computer interaction framework for the design and evaluation of mHealth interventions. Stud Health Technol Inform 2019 Jul 30;263:49-63.

Horvath KJ, Ecklund AM, Hunt SL, Nelson TF, Toomey TL. Developing Internet-based health interventions: a guide for public health researchers and practitioners. J Med Internet Res 2015 Jan 23;17(1):e28

Jennings HM, Morrison J, Akter K, Kuddus A, Ahmed N, Kumer Shaha S, et al. Developing a theory-driven contextually relevant mHealth intervention. Glob Health Action 2019;12(1):1550736

Pham Q, Graham G, Lalloo C, Morita PP, Seto E, Stinson JN, et al. An analytics platform to evaluate effective engagement with pediatric mobile health apps: design, development, and formative evaluation. JMIR Mhealth Uhealth 2018 Dec 21;6(12):e11447

Ravn Jakobsen P, Hermann AP, Søndergaard J, Wiil UK, Clemensen J. Development of an mHealth application for women newly diagnosed with osteoporosis without preceding fractures: a participatory design approach. Int J Environ Res Public Health 2018 Feb 13;15(2):330

Skrabal Ross X, Gunn KM, Patterson P, Olver I. Development of a smartphone program to support adherence to oral chemotherapy in people with cancer. Patient Prefer Adherence 2019 Dec 20;13:2207-2215

Schellong J, Lorenz P, Weidner K. Proposing a standardized, step-by-step model for creating post-traumatic stress disorder (PTSD) related mobile mental health apps in a framework based on technical and medical norms. Eur J Psychotraumatol 2019 May 15;10(1):1611090

Schnall R, Rojas M, Bakken S, Brown W, Carballo-Dieguez A, Carry M, et al. A user-centered model for designing consumer mobile health (mHealth) applications (apps). J Biomed Inform 2016 Apr;60:243-251

Tay I, Garland S, Gorelik A, Wark JD. Development and testing of a mobile phone app for self-monitoring of calcium intake in young women. JMIR Mhealth Uhealth 2017 Mar 07;5(3):e27

van Agteren JE, Lawn S, Bonevski B, Smith BJ. Kick.it: the development of an evidence-based smoking cessation smartphone app. Transl Behav Med 2018 Mar 01;8(2):243-267.

Vilardaga R, Rizo J, Zeng E, Kientz JA, Ries R, Otis C, et al. User-centered design of learn to quit, a smoking cessation smartphone app for people with serious mental illness. JMIR Serious Games 2018 Jan 16;6(1):e2

Wilhide Iii CC, Peeples MM, Anthony Kouyaté RC. Evidence-based mHealth chronic disease mobile app intervention design: development of a framework. JMIR Res Protoc 2016 Feb 16;5(1):e25

Wittenberg E, Xu J, Goldsmith J, Mendoza Y. Caregiver communication about cancer: development of a mHealth resource to support family caregiver communication burden. Psychooncology 2019 Feb;28(2):365-371

Woods L, Cummings E, Duff J, Walker K. Design thinking for mHealth application co-design to support heart failure self-management. Stud Health Technol Inform 2017;241:97-102.

Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res 2015 Jan 30;17(1):e30

Fjeldsoe BS, Miller YD, O'Brien JL, Marshall AL. Iterative development of MobileMums: a physical activity intervention for women with young children. Int J Behav Nutr Phys Act 2012 Dec 20;9:151

Nißen M, Selimi D, Janssen A, Cardona DR, Breitner MH, Kowatsch T, et al. See you soon again, chatbot? A design taxonomy to characterize user-chatbot relationships with different time horizons. Comput Human Behav 2022 Feb;127:107043.

Burkill S, Copas A, Couper MP, Clifton S, Prah P, Datta J, et al. Using the Web to collect data on sensitive behaviours: a study looking at mode effects on the British national survey of sexual attitudes and lifestyles. PLoS One 2016 Feb 11;11(2):e0147983

Nadarzynski T, Miles O, Cowie A, Ridge D. Acceptability of artificial intelligence (AI)-led chatbot services in healthcare: a mixed-methods study. Digit Health 2019 Aug 21;5:2055207619871808

Boucher EM, Harake NR, Ward HE, Stoeckl SE, Vargas J, Minkel J, et al. Artificially intelligent chatbots in digital mental health interventions: a review. Expert Rev Med Devices 2021 Dec;18(sup1):37-49.

Curtis RG, Bartel B, Ferguson T, Blake HT, Northcott C, Virgara R, et al. Improving user experience of virtual health assistants: scoping review. J Med Internet Res 2021 Dec 21;23(12):e31737

De Cicco R, Silva SC, Alparone FR. Millennials' attitude toward chatbots: an experimental study in a social relationship perspective. Int J Retail Distrib Manag 2020 Jul 08;48(11):1213-1233.

Etemad-Sajadi R. The impact of online real-time interactivity on patronage intention: the use of avatars. Comput Human Behav 2016 Aug;61:227-232.

Thaler M, Schlögl S, Groth A. Agent vs. avatar: comparing embodied conversational agents concerning characteristics of the uncanny valley. In: Proceedings of the 2020 IEEE International Conference on Human-Machine Systems. 2020 Presented at: ICHMS '20; September 7-9, 2020; Rome, Italy p. 1-6.

Bickmore T, Gruber A, Picard R. Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Educ Couns 2005 Oct;59(1):21-30.

Morton K, Sutton S, Hardeman W, Troughton J, Yates T, Griffin S, et al. A text-messaging and pedometer program to promote physical activity in people at high risk of type 2 diabetes: the development of the PROPELS follow-on support program. JMIR Mhealth Uhealth 2015 Dec 15;3(4):e105

Mac OA, Thayre A, Tan S, Dodd RH. Web-based health information following the renewal of the cervical screening program in Australia: evaluation of readability, understandability, and credibility. J Med Internet Res 2020 Jun 26;22(6):e16701

Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for navy enlisted personnel. Institute for Simulation and Training, University of Central Florida. 1975. URL: https://stars.library.ucf.edu/istlibrary/56/ [accessed 2022-01-19]

Bobrow K, Farmer A, Cishe N, Nwagi N, Namane M, Brennan TP, et al. Using the Medical Research Council framework for development and evaluation of complex interventions in a low resource setting to develop a theory-based treatment support intervention delivered via SMS text message to improve blood pressure control. BMC Health Serv Res 2018 Jan 23;18(1):33

van Dijk CN, van Witteloostuijn M, Vasić N, Avrutin S, Blom E. The influence of texting language on grammar and executive functions in primary school children. PLoS One 2016 Mar 31;11(3):e0152409

Fadhil A, Schiavo G, Wang Y, Yilma BA. The effect of emojis when interacting with conversational interface assisted health coaching system. In: Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare. 2018 Presented at: PervasiveHealth '18; May 21-24, 2018; New York, NY, USA p. 378-383.

Menacho LA, Blas MM, Alva IE, Roberto Orellana E. Short text messages to motivate HIV testing among men who have sex with men: a qualitative study in Lima, Peru. Open AIDS J 2013 Apr 5;7:1-6

Bock BC, Rosen RK, Barnett NP, Thind H, Walaska K, Foster R, et al. Translating behavioral interventions onto mHealth platforms: developing text message interventions for smoking and alcohol. JMIR Mhealth Uhealth 2015 Feb 24;3(1):e22

Adamopoulou E, Moussiades L. An overview of chatbot technology. In: Proceedings of the 16th International Conference on Artificial Intelligence Applications and Innovations. 2020 Presented at: AIAI '20; June 5–7, 2020; Neos Marmaras, Greece p. 373-383.

Kramer JN, Künzler F, Mishra V, Smith SN, Kotz D, Scholz U, et al. Which components of a smartphone walking app help users to reach personalized step goals? Results from an optimization trial. Ann Behav Med 2020 Jun 12;54(7):518-528

Künzler F, Mishra V, Kramer JN, Kotz D, Fleisch E, Kowatsch T. Exploring the state-of-receptivity for mHealth interventions. Proc ACM Interact Mob Wearable Ubiquitous Technol 2019 Dec 11;3(4):1-27.

Meppelink CS, van Weert JC, Haven CJ, Smit EG. The effectiveness of health animations in audiences with different health literacy levels: an experimental study. J Med Internet Res 2015 Jan 13;17(1):e11

Klok T, Kaptein AA, Brand PL. Non-adherence in children with asthma reviewed: the need for improvement of asthma care and medical education. Pediatr Allergy Immunol 2015 May;26(3):197-205.

Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q 2003 Sep;27(3):425-478.

Bickmore TW, Picard RW. Establishing and maintaining long-term human-computer relationships. ACM Trans Comput Hum Interact 2005 Jun 01;12(2):293-327.

Fadhil A. Can a chatbot determine my diet?: addressing challenges of chatbot application for meal recommendation. arXiv 2022 Preprint posted online on February 25, 2018.

Radziwill NM, Benton MC. Evaluating quality of chatbots and intelligent conversational agents. arXiv 2022 Preprint posted online on April 15, 2017.

Martinengo L, Van Galen L, Lum E, Kowalski M, Subramaniam M, Car J. Suicide prevention and depression apps' suicide risk assessment and management: a systematic assessment of adherence to clinical guidelines. BMC Med 2019 Dec 19;17(1):231

Lum E, Jimenez G, Huang Z, Thai L, Car J. Appropriateness of action prompts for hypoglycaemia and hyperglycaemia in type 2 diabetes self-management apps. Diabetes Metab Res Rev 2020 Feb;36(2):e3235.

Pereira J, Díaz Ó. Using health chatbots for behavior change: a mapping study. J Med Syst 2019 Apr 04;43(5):135.

Michie S, Wood CE, Johnston M, Abraham C, Francis JJ, Hardeman W. Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data). Health Technol Assess 2015 Nov;19(99):1-188

Krishnakumar A, Verma R, Chawla R, Sosale A, Saboo B, Joshi S, et al. Evaluating glycemic control in patients of South Asian origin with type 2 diabetes using a digital therapeutic platform: analysis of real-world data. J Med Internet Res 2021 Mar 25;23(3):e17908

Chatterjee S, Price A. Healthy living with persuasive technologies: framework, issues, and challenges. J Am Med Inform Assoc 2009;16(2):171-178

Rykov Y, Thach TQ, Bojic I, Christopoulos G, Car J. Digital biomarkers for depression screening with wearable devices: cross-sectional study with machine learning modeling. JMIR Mhealth Uhealth 2021 Oct 25;9(10):e24872

Tinschert P, Rassouli F, Barata F, Steurer-Stey C, Fleisch E, Puhan MA, et al. Nocturnal cough and sleep quality to assess asthma control and predict attacks. J Asthma Allergy 2020 Dec 14;13:669-678

Fadhil A. A conversational interface to improve medication adherence: towards AI support in patient's treatment. arXiv 2022 Preprint posted online on March 3, 2018.

Burgess-Allen J, Owen-Smith V. Using mind mapping techniques for rapid qualitative data analysis in public participation processes. Health Expect 2010 Dec;13(4):406-415

Davies M. Concept mapping, mind mapping and argument mapping: what are the differences and do they matter? High Educ 2011;62(3):279-301.

Low code chatbot building platform. SAP Conversational AI. URL: https://cai.tools.sap/ [accessed 2022-03-02]

XMind. URL: https://www.xmind.net/ [accessed 2022-01-19]

Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health 2015 Mar 18;36:393-415

Coomes CM, Lewis MA, Uhrig JD, Furberg RD, Harris JL, Bann CM. Beyond reminders: a conceptual framework for using short message service to promote prevention and improve healthcare quality and clinical outcomes for people living with HIV. AIDS Care 2012;24(3):348-357.

Policy for Device Software Functions and Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff. U.S. Food & Drug Administration. 2019 Sep 27. URL: https://www.fda.gov/media/80958/download [accessed 2022-03-03]

Mbuagbaw L, Mursleen S, Lytvyn L, Smieja M, Dolovich L, Thabane L. Mobile phone text messaging interventions for HIV and other chronic diseases: an overview of systematic reviews and framework for evidence transfer. BMC Health Serv Res 2015 Jan 22;15:33

Mishra V, Künzler F, Kramer JN, Fleisch E, Kowatsch T, Kotz D. Detecting receptivity for mHealth interventions in the natural environment. Proc ACM Interact Mob Wearable Ubiquitous Technol 2021 Jun;5(2):74

Collins LM, Murphy SA, Nair VN, Strecher VJ. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med 2005 Aug;30(1):65-73.

Collins LM, Kugler KC. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: Advanced Topics. Cham, Switzerland: Springer; 2018.

Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019 Jun;126:95-104.

Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med 2016 Nov;51(5):843-851

Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613

Diener E, Wirtz D, Tov W, Kim-Prieto C, Choi D, Oishi S, et al. New well-being measures: short scales to assess flourishing and positive and negative feelings. Soc Indic Res 2009 May 28;97(2):143-156.

Cleeland CS, Ryan KM. Pain assessment: global use of the Brief Pain Inventory. Ann Acad Med Singap 1994 Mar;23(2):129-138.

Munder T, Wilmers F, Leonhart R, Linster HW, Barth J. Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients. Clin Psychol Psychother 2010;17(3):231-239.

Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions-a call for innovative evidence generation approaches. NPJ Digit Med 2020 Aug 27;3:110

Blandford A, Gibbs J, Newhouse N, Perski O, Singh A, Murray E. Seven lessons for interdisciplinary research on interactive digital health interventions. Digit Health 2018 May 3;4:2055207618770325

Kohavi R, Longbotham R. Online controlled experiments and A/B testing. In: Sammut C, Webb GI, editors. Encyclopedia of Machine Learning and Data Mining. Boston, MA, USA: Springer; 2017:922-929.

Klasnja P, Hekler EB, Shiffman S, Boruvka A, Almirall D, Tewari A, et al. Microrandomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol 2015 Dec;34S:1220-1228

Kramer JN, Künzler F, Mishra V, Presset B, Kotz D, Smith S, et al. Investigating intervention components and exploring states of receptivity for a smartphone app to promote physical activity: protocol of a microrandomized trial. JMIR Res Protoc 2019 Jan 31;8(1):e11540

Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 2010 Jan 06;10:1

Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the concept of adherence to eHealth technology: systematic review on when usage becomes adherence. J Med Internet Res 2017 Dec 06;19(12):e402

Cao W, Milks MW, Liu X, Gregory ME, Addison D, Zhang P, et al. mHealth interventions for self-management of hypertension: framework and systematic review on engagement, interactivity, and tailoring. JMIR Mhealth Uhealth 2022 Mar 02;10(3):e29415

Paz Castro R, Haug S, Filler A, Kowatsch T, Schaub MP. Engagement within a mobile phone-based smoking cessation intervention for adolescents and its association with participant characteristics and outcomes. J Med Internet Res 2017 Nov 01;19(11):e356

Achilles MR, Anderson M, Li SH, Subotic-Kerry M, Parker B, O'Dea B. Adherence to e-mental health among youth: considerations for intervention development and research design. Digit Health 2020 May 21;6:2055207620926064

Torous J, Michalak EE, O'Brien HL. Digital health and engagement-looking behind the measures and methods. JAMA Netw Open 2020 Jul 01;3(7):e2010918

ManyChat. URL: https://manychat.com/ [accessed 2022-03-04]

Perski O, Short CE. Acceptability of digital health interventions: embracing the complexity. Transl Behav Med 2021 Jul 29;11(7):1473-1480

Yan AF, Stevens P, Wang Y, Weinhardt L, Holt CL, O'Connor C, et al. mHealth text messaging for physical activity promotion in college students: a formative participatory approach. Am J Health Behav 2015 May;39(3):395-408.

Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27

Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. World Health Organization. Geneva, Switzerland: World Health Organization; 2016. URL: https://apps.who.int/iris/bitstream/handle/10665/252183/9789241511766-eng.pdf [accessed 2022-03-29]

Babigumira JB, Dolan S, Shade S, Puttkammer N, Bale J, Tolentino H, et al. Applied Economic Evaluation of Digital Health Interventions. Centers for Disease Control and Prevention. 2021 Jan. URL: https://www.go2itech.org/wp-content/uploads/2021/02/I-TECH_HIS_Economic_Evaluation.pdf [accessed 2022-03-29]

Jiang X, Ming WK, You JH. The cost-effectiveness of digital health interventions on the management of cardiovascular diseases: systematic review. J Med Internet Res 2019 Jun 17;21(6):e13166

Gomes M, Murray E, Raftery J. Economic evaluation of digital health interventions: methodological issues and recommendations for practice. Pharmacoeconomics 2022 Apr;40(4):367-378

Topol E. The Topol Review: Preparing the healthcare workforce to deliver the digital future: an independent report on behalf of the Secretary of State for Health and Social Care. National Health Service. 2019 Feb. URL: https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf [accessed 2022-03-17]

Theobald S, Brandes N, Gyapong M, El-Saharty S, Proctor E, Diaz T, et al. Implementation research: new imperatives and opportunities in global health. Lancet 2018 Nov 17;392(10160):2214-2228.

Peters DH, Adam T, Alonge O, Agyepong IA, Tran N. Implementation research: what it is and how to do it. BMJ 2013 Nov 20;347:f6753.

Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide. Alliance for Health Policy and Systems Research & World Health Organization. 2013. URL: https://apps.who.int/iris/handle/10665/91758 [accessed 2022-04-11]

Bauer MS, Damschroder L, Hagedorn H, Smith J, Kilbourne AM. An introduction to implementation science for the non-specialist. BMC Psychol 2015 Sep 16;3(1):32

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4:50

Essén A, Stern AD, Haase CB, Car J, Greaves F, Paparova D, et al. Health app policy: international comparison of nine countries' approaches. NPJ Digit Med 2022 Mar 18;5(1):31

Understanding DTx. Digital Therapeutics Alliance. URL: https://dtxalliance.org/understanding-dtx/ [accessed 2022-04-11]

Device Software Functions Including Mobile Medical Applications. U.S. Food & Drug Administration. 2019. URL: https://tinyurl.com/5n896nhk [accessed 2022-05-31]

Clever Compliance. 2020 Mar 17. URL: https://blog.clevercompliance.io/medical-product-compliance/ce-marking-medical -device/ [accessed 2022-05-31]

Sanders EB, Stappers PJ. Co-creation and the new landscapes of design. CoDesign 2008 Mar;4(1):5-18.

Matheson GO, Pacione C, Shultz RK, Klügl M. Leveraging human-centered design in chronic disease prevention. Am J Prev Med 2015 Apr;48(4):472-479.

Melles M, Albayrak A, Goossens R. Innovating health care: key characteristics of human-centered design. Int J Qual Health Care 2021 Jan 12;33(Supplement_1):37-44

Plattner H, Meinel C, Leifer L. Design Thinking : Understand - Improve - Apply. Berlin, Germany: Springer; 2011.

Still B, Crane K. Fundamentals of User-Centered Design: A Practical Approach. Boca Raton, FL, USA: CRC Press; 2017.

Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health 2006 Oct;60(10):854-857

Steen M, Manschot M, De Koning N. Benefits of co-design in service design projects. Int J Des 2011;5(2):53-60

Pirinen A. The barriers and enablers of co-design for services. Int J Des 2016;10(3):27-42

Georgsson M, Staggers N. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016 Feb;59:115-129

Martinengo L, Spinazze P, Car J. Mobile messaging with patients. BMJ 2020 Mar 16;368:m884.

Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices of smartphone apps for depression and smoking cessation. JAMA Netw Open 2019 Apr 05;2(4):e192542

Grande D, Luna Marti X, Feuerstein-Simon R, Merchant RM, Asch DA, Lewson A, et al. Health policy and privacy challenges associated with digital technology. JAMA Netw Open 2020 Jul 01;3(7):e208285

Sundareswaran V, Sarkar A. Chatbots RESET: a framework for governing responsible use of conversational AI in healthcare. World Economic Forum. 2020 Dec. URL: http://www3.weforum.org/docs/WEF_Governance_of_Chatbots_in_Healthcare_ 2020.pdf [accessed 2022-03-17]

Rumbold JM, Pierscionek B. The effect of the general data protection regulation on medical research. J Med Internet Res 2017 Feb 24;19(2):e47

Summary of the HIPAA Privacy Rule. U.S. Department of Health and Human Services. 2013. URL: https://www.hhs.gov/hipaa/for-professionals/privacy/laws-regulations/index.html [accessed 2022-03-17]

Wong BY. Data privacy law in Singapore: the personal data protection act 2012. Int Data Priv Law 2017 Nov;7(4):287-302.

Children's Online Privacy Protection Rule ("COPPA"). Federal Trade Commission. URL: https://www.ftc.gov/legal-library/browse/rules/childrens-online-privacy-protection-rule-coppa [accessed 2022-05-31]

Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010 Feb 17;12(1):e4

Dalgetty R, Miller CB, Dombrowski SU. Examining the theory-effectiveness hypothesis: a systematic review of systematic reviews. Br J Health Psychol 2019 May;24(2):334-356.