Introduction
Depression and anxiety are common in adolescents. [1,2]. If left untreated, these common illnesses can result in significant sequelae including school dropout, substance abuse, and suicide [3]. At any one time, 18% of adolescents report depressive symptoms [1,2], and about 3% are diagnosed with a depressive disorder [2]. Depression and anxiety often co-occur; 1 in 3 teens will experience clinically significant anxiety symptoms [4]. These numbers have risen in recent years due to the isolating effects of quarantine and the stress of virtual learning during the COVID-19 pandemic [5,6]. Even before the pandemic, suicide was the second highest cause of death in this age group [7-9], but in 2020, emergency department (ED) visits for suspected suicide attempts increased by 30% compared to 2019 [10]. The effects of the pandemic on mental health in general has been unprecedented; prepandemic mental health workforce shortages have become more dire. Thus, there is an urgent need to provide prompt, effective treatments that can be readily integrated into the day-to-day lives of adolescents and disseminated remotely during periods of quarantine and ongoing workforce issues.
Evidence-based treatments for pediatric depression and anxiety include selective serotonin and serotonin-norepinephrine reuptake inhibitor (SSRI, SNRI) medications and cognitive behavioral therapy (CBT), used separately or in combination [11-13]. Yet, few teens receive treatment with these therapies [13]. CBT is typically provided through a series of face-to-face interactions with a trained therapist over several months and is often preferred by families over medication [3,14]. However, CBT is seldom used due to pervasive and persistent problems with access, cost, and stigma [15-18]. Internet and mobile health (mHealth) interventions reduce these barriers to utilization and are acceptable and effective alternatives to live CBT for the treatment of depression and anxiety in adults [19]. Conversational agents or “chatbots” that deliver CBT via a text-based, semiautomated algorithm are known to reduce mild-to-moderate depressive symptoms in nonclinical populations [17,20-23]. However, few have been evaluated in clinical populations, with scarce rigorous study in youth [21,24,25].
About 95% of teens in the United States use smartphones [26] or other web-enabled mobile devices, and nearly half report being online almost constantly. Over half of US teens use their devices for accessing social media platforms, and this number may be higher in adolescents with known mental health diagnoses [27]. Although internet use—social media use in particular—has known serious negative effects on adolescent mental health [28], mobile technology can also promote social support and even assistance to young people dealing with emotional challenges [29]. Adolescents are increasingly using their devices to search the internet for information about mental health [18]. Many teens want to be self-reliant when coping with emotional distress and prefer to manage difficult situations virtually rather than through in-person interactions [18]. However, few digital therapeutic interventions have been specifically developed for children and adolescents [26]. Using mHealth to deliver CBT to teens merits further evaluation, as it is inexpensive, easy to access and administer, and scalable, which are important attributes for widespread dissemination, adoption, and public health impact [30].
Because the management of adolescents with depression and anxiety is often done by primary care providers (PCPs), the goal of this study was to investigate the feasibility of implementing an existing mHealth intervention [20,31] as part of the treatment plan for adolescents diagnosed with depression and anxiety by their PCP. The intervention, Woebot for Adolesent Depression, or W-GenZ, is an mHealth application that uses a relational conversational agent known as Woebot to deliver CBT to adolescents presenting with mood and anxiety concerns. W-GenZ is one among a suite of emotional support Woebot-based applications that have been previously studied. Previous research demonstrated Woebot’s feasiblity and acceptability as well as efficacy to reduce (1) depression and anxiety among young adults [20], (2) depression among postpartum women [32,33], and (3) substance abuse among adults [34,35]. In addition, data indicate that Woebot establishes a therapeutic working alliance with users [33]. The W-GenZ app was developed specifically for use in adolescents, and while preliminary effectiveness data among this population are promising [36], it has not yet been rigorously tested for effectiveness. Moreover, the application has not yet been implemented in real-world clinical settings, where such interventions would ideally track and report progress to PCPs and parents, which is critical for maintaining safety and treatment adherence.
In this open-label, randomized pilot study, our objectives were 2-fold: (1) to establish that the app is usable, acceptable, and likely to provide benefit for adolescents, and (2) to determine the optimal way to integrate mHealth interventions such as the app into primary care management of adolescent depression. This study focused on adolescents newly diagnosed with depression and anxiety, as we anticipated immediate benefit from using the app while awaiting a therapeutic response from usual care, including antidepressant medication and/or referral for psychotherapy. We hypothesized that the app would be acceptable and easy to use and that use of the app [16] would lead to greater improvement in symptoms for adolescents newly diagnosed with depression and anxiety compared to the wait list control group.
Methods
Methodology
Study activities were conducted within the Washington University Pediatric and Adolescent Ambulatory Research Consortium (WU PAARC), part of the WU Institute for Clinical and Translational Science (ICTS). The practice network includes over 100 pediatric care providers from approximately 60 practices in the greater metropolitan St Louis area and serves approximately 175,000 patients. Across the consortium, 25% (approximately 44,000) of the patients are publicly insured (eg, Medicaid) and 28% (approximately 50,000) are Black. Participation was restricted to patients and PCPs from 11 practices from the practice-based research network (PBRN) who had completed a 12-month quality improvement (QI) initiative targeted at improving care for adolescents with depression [37], specifically by establishing systems of care that follow the current national guidelines [12,13]. Study activities included a small pilot randomized trial to gather data from teens and their parent and semistructured interviews with PCPs. The study was conducted from October 2020 to April 2021, encompassing the second wave of COVID-19 infections in the region, which limited or at times completely shut down in-person clinical care delivery during the study period.
Ethics Approval
The study was approved by the Washington University Institutional Review Board (202005103). Parents or legal guardians provided informed consent, and adolescents provided informed assent to participate in the study. The consent/assent process was initiated by phone due to pandemic-related barriers to in-person participation. Formal consent/assent was conducted using a Research Electronic Data Capture (REDCap)–based electronic consent form. Following a prescreening phone call, the parents or legal guardians of potential adolescent participants received an email with a unique link to review the informed consent form online. After the research team explained the study and answered any questions, parents or legal guardians and potential participants clicked an “agree” button, which was accompanied by text indicating that by clicking the button, they were providing consent/assent to participate. They were then asked to type their full name, which served as their electronic signature confirming consent/assent. Upon completion of the form, participants were presented with the option to download an electronic copy of the executed form. A digital copy was also emailed to each participant’s parent or legal guardian. Participating parents or guardians and adolescents each received up to US $40 as reimbursement for their time to complete study assessments ($5 to $10 per survey).
Randomized Pilot Study
Participants were randomized 1:1 for 12 weeks either to a wait list control or to the app intervention. Time points for clinical assessments were based on clinical care guidelines (baseline, 4, 8, and 12 weeks) [13], with the primary analysis at 4 weeks. The primary outcome was depression symptom severity on the 9-item Patient Health Questionnaire (PHQ-9). Secondary outcomes included anxiety symptom severity assessed with the 7-item Generalized Anxiety Disorder questionnaire (GAD-7), which uses scoring similar to the PHQ-9 to indicate the frequency and severity of anxiety symptoms, and mental health self-efficacy assessed with the Mental Health Self-Efficacy Scale (MHSES). All patients in the intervention and wait list control groups received usual care for their depression and anxiety from their PCP. Participants in the control group were offered use of the app at the end of the 12-week study period.
Intervention
W-GenZ delivers evidence-based therapeutic elements via brief “conversations” with a fully automated, relational conversational agent named Woebot. Powered by natural language processing and machine learning techniques, the brief, self-guided intervention draws from CBT, interpersonal psychotherapy for adolescents (IPT-A), and some elements of dialectical behavior therapy (DBT), tailoring the conversation to the present situation to help the adolescent develop emotion regulation skills in the context of their everyday life for the problem at hand. The user experience is centered around mood tracking and goal-oriented, tailored conversations. Woebot checks in with the user, and depending on the user’s reported mood or desire to work on a problem or learn something new, Woebot will offer and guide the user through CBT-based psychoeducation and tools, tailored to the reported need in that moment. Daily push notifications prompt users to check in. Using proprietary neurolinguistic programing and artificial intelligence, the platform designs a personalized program to meet the user’s needs in real time. The app conforms with safety recommendations from the American Psychiatric Association (APA) and American Medical Association (AMA) [38,39]. As part of the app’s onboarding process, users are provided with the privacy policy, theoretical underpinning, targeted difficulties, expected results, research findings, and other information requested by potential young users [18]. Safety features include informing the user they are talking to a robot and not a real person, reminding them that the app is not a crisis intervention for suicidal ideation, encouraging them to seek additional support if they are feeling unwell, and providing helplines for assistance. The safety net protocol detects concerning language, confirms it with the user, and provides a local suicide crisis hotline number as needed.
Study Population
Adolescents were eligible for inclusion if they were 13 to 17 years old and had a new diagnosis of depression and anxiety in the past 3 months, as reported by their parent or legal guardian. Adolescents were excluded if their parent reported any lifetime history of severe depression, substance use disorder, psychotic illness, obsessive compulsive disorder, posttraumatic stress disorder, panic disorder, or specific phobias, as this could indicate that the current reported episode was a more complicated or severe presentation of depression or that the current episode was recurrent. Those who reported a psychiatric hospitalization in the previous month, were not accompanied by a guardian to the diagnostic visit, did not have access to a mobile device (cell phone/tablet) for regular use, and were unable to read and write English were excluded.
Recruitment Procedures
Five of the 11 eligible practices referred patients to the randomized controlled trial (RCT). Potential participants were informed about the study via a flyer distributed by their PCP at the diagnostic visit or within the first 3 months after diagnosis. For those interested in participation, the parent completed the online eligibility screen. If the adolescent was eligible, the parent and adolescent reviewed the electronic consent document and had a telephone conversation with a member of the study team to review participation requirements and answer questions.
If the family decided to participate, the adolescent and 1 parent signed the electronic consent and completed baseline web-based surveys via REDCap. Subsequently, the adolescent was randomized using REDCap [40]. Those randomized to the intervention received a link and a unique password to download the intervention app on their mobile device, proceed through the enrollment screens, and initiate the program. All participants were followed for 12 weeks with assessments, as indicated in Figure 1. After data collection was completed, participants’ medical charts were reviewed to confirm any additional mental health diagnoses and record their initial treatment plan.
Enlarge this image.Figure 1. Flowchart indicating the study procedures and schedule of events. AIM: Acceptability of Intervention Measure; FIM: Feasibility of Intervention Measure; GAD-7: 7-item Generalized Anxiety Disorder questionnaire; MHSES: Mental Health Self-Efficacy Scale; PHQ-9: 9-item Patient Health Questionnaire; SUS: System Usability Scale.
Measurement
In both study groups, study surveys were administered to adolescents and parents through a secure digital link to a REDCap survey. For each survey, participants were directed to a URL via email or text, with up to 2 push notification reminders. Time points for the primary (4 weeks) and safety (2 and 4 weeks) outcomes were selected based on the practice guidelines for PCPs managing adolescent depression [13]. If the measurement was not completed within 1 week, participants received a phone call from a study team member to assess study engagement. The surveys varied in duration and took 5 to 20 minutes to complete. Gift cards were provided to compensate participants for their time to complete data collection and were distributed immediately upon survey completion (up to US $40). Study design and assessments are indicated in Figure 1.
Depression symptoms were assessed using the PHQ-9 [41] modified for use in adolescents (PHQ-A) [42]. The PHQ-A also consists of 9 questions, but with wording changes to accommodate an adolescent reading level, and was used to screen for adolescent depression according to Diagnostic and Statistical Manual of Mental Disorders fifth edition (DSM-5) criteria. Although the unmodified version has good validity and reliability in adolescent populations, the PHQ-A has modified the wording of the questions so that it can be entirely self-administered in individuals with a grade 6 reading level. The overall PHQ-9 score ranges from 0 to 27, with individual item symptom severity scores of 0, 1, 2, and 3, which correspond to “not at all,” “several days,” “more than half the days,” and “nearly every day,” respectively. Overall cumulative scores of 5, 10, 15, and 20 represent cut points for mild, moderate, moderately severe, and severe depression, respectively.
Anxiety symptoms were assessed using GAD-7 [43]. This is a reliable, valid, and responsive screening instrument commonly used in primary care settings [44] and has been extensively studied and validated in diverse adolescent populations, including for screening in primary care settings in Finland [45], China [46], Korea [47], Ghana [48], and Canada [49]. GAD-7 has also been validated in the United States in a clinical population of adolescents with generalized anxiety disorder [50]. Respondents use the same item scoring system as the PHQ-9 to indicate frequency and severity of anxiety symptoms, and a cut point of > 10 is used to identify clinical cases of anxiety.
Confidence in self-management of mental health issues was measured using the MHSES [51], which has 6 items that address confidence in managing stress, depression, and anxiety and are scored on a 10-point scale (1: not at all confident, 10: very confident). Ratings are summed for an overall measure of self-efficacy (range 6 to 60), with higher scores indicating more self-efficacy.
Safety was assessed at 2, 4, 8, and 12 weeks. Parents were asked to report any hospitalizations or ED visits made by their child for depression/anxiety-related problems in the preceding 2 weeks up to 1 month.
Utilization, Acceptability, and Feasibility
Deidentified aggregate app usage data were obtained from the app developer. Metrics included aggregate descriptive summaries of engagement metrics, including total number of check-ins, days in app, and messages sent during the 4-week intervention period. Each conversation with the app is tailored to the users’ reported needs at the moment and thus necessarily varies in length and composition. Nonetheless, these metrics are provided as examples to demonstrate various ways that users engaged with the app’s offerings.
The acceptability and feasibility of using the intervention was assessed after 2 weeks from the perspective of the adolescents (n=10, 56%) and their parent or adult legal guardian (n=10, 44%) randomized to the intervention group of the RCT. Parents completed the 4-item Acceptability of Intervention Measure (AIM) and the 4-item Feasibility of Intervention Measure (FIM) developed by Weiner et al [52]. To aid understanding by the 13- to 17-year-old adolescent participants, only 1 item from each of these measures was used.
Adolescents assessed the usability of the app using 5 items from the 10-item System Usability Scale (SUS) [53] that includes statements about the effectiveness, efficiency, and satisfaction with use (score range 5 to 25). For the AIM, FIM, and SUS, respondents used a 5-item response scale (1: completely disagree, 5: completely agree) to indicate their agreement with item statements, and a summary score was created. For the AIM and FIM, higher scores indicated greater acceptability and feasibility [52,54].
Acceptability and feasibility for PCPs were assessed by semistructured interviews (described in the subsequent section).
Safety
For all participants, an additional layer of safety monitoring was provided by the study team through surveillance of the digital PHQ-9 assessment at baseline, 2, 4, 6, 8, and 12 weeks. Parents provided consent for communication between the study team and the PCP for this purpose. When the PHQ-9 flagged suicidal ideation, an email alert was sent to the study team, and the PI then contacted the participant’s PCP to share information and discuss next steps regarding assessment of suicide risk. The principal investigator (PI; author JG) summarized the relevant information and alerted the psychiatrist. Management options included (1) the PCP electing to alert the family to the situation and complete additional suicide risk assessment themselves or (2) the PCP requesting further assessment be completed by the pediatric study psychiatrist, who then conducted additional assessment with patients and their legal guardians over the phone and provided a brief written assessment and recommendations back to the PCP. The PCP also had the option of further discussing the case with the study psychiatrist by phone. The PCP’s preferred management approach was prioritized, and the psychiatrist provided further assessment by phone and offered treatment recommendations to the PCP when requested using an evidence-based telephone consultation model [55].
Statistical Analyses
Summary statistics are reported as percentages for categorical variables and mean and SD for continuous variables. We evaluated the Cohen d effect size and 95% CIs on the mean PHQ-9, GAD-7, and MHSES scores achieved by treatment group at the primary end point of 4 weeks. All statistical analyses were completed using SAS software version 9 (SAS Institute Inc).
PCP Interviews
We completed semistructured interviews with PCPs to better understand current attitudes toward and practices related to recommending CBT to pediatric patients as well as toward the use of mobile health applications for delivering behavioral mental health interventions to youth.
PCPs who participated in the PBRN quality improvement (QI) initiative were eligible to participate in the interviews, and all received an email invitation. Interviews were conducted concurrently with the pilot RCT portion of the study between October 2020 and April 2021. A total of 13 PCPs from the 11 participating clinics completed a 30-minute virtual video interview with the PI, who was also the PBRN director. She used a semistructured interview guide to ask PCP about their use of apps as part of a treatment plan to improve physical and mental health, how they found out about apps to recommend, and positive and negative features of the app considered when selecting an app to recommend. After these general questions, participating PCPs were provided with some promotional material about the intervention app that included a picture of the chatbot (without its name) and information about how it interacts with the user. They were asked about their first impressions, how likely they were to recommend it, and for which patients and when in the treatment course it would likely be most useful. Interviews were continued until thematic saturation was achieved. All interviews were digitally recorded and transcribed verbatim. Participants received a US $50 gift card for participation.
Qualitative Analysis of PCP Interviews
Transcripts were analyzed using an inductive coding approach based on pragmatic-variant grounded theory [56]. Authors JG and GN independently reviewed interview transcripts for emergent themes in PCP perspectives on the use of mobile applications to augment physical and mental health care.
Results
Pilot RCT
Figure 2 shows participant flow through the pilot RCT. Of the 40 families that completed the eligibility screening for the RCT, 17 (43%) were ineligible, 2 (5%) declined to participate, 3 (8%) were lost to follow-up, and 18 (45%) were randomized to the intervention (10/40, 25%) or wait list control (8/40, 2%). Chart review revealed the following diagnoses: depression (8/18, 44%), depressive symptoms (2/18, 11%), anxiety (16/18, 89%), stress-related headaches (1/18, 6%), and functional abdominal pain and adjustment disorder (1/18, 6%). A review of the notes and baseline PHQ-9 and GAD-7 scores resulted in a sample of 17 for the analyses, with 10 (59%) that had both depression and anxiety 7 (41%) that had anxiety alone. One participant originally randomized to the wait list control group was excluded from analyses following chart review based on the chart notation “episodic tension headaches due to stress” and baseline measures (PHQ-9=5, GAD-7=6), as they were deemed to be ineligible (no diagnosis of depression or anxiety in their chart and no evidence of moderate disease).
Participant Characteristics
The participating parent was typically the mother (16/17, 94%). Participating teens are described in Table 1 and did not differ by treatment group. They were predominantly female (15/17, 88%), White (15/17, 88%), and had family work-related health insurance (15/17, 88%). Initial treatment plans did not differ between study groups and included CBT (5/17, 29%), unspecified counseling, 15/17, 88%), and SSRI medication (11/17, 65%). By self-report, 7 (41%) teens had previously received at least 1 counselling session for depression or anxiety.
Table 1. Randomized Controlled Trial (RCT) participant baseline characteristics.
| Characteristic | Total (N=17) | Intervention (N=10) | Wait list control (N=7) | |
| Sex, n (%) | ||||
| Male | 1 (5.9) | 1 (10) | 0 (0) | |
| Female | 15 (88.2) | 9 (90) | 6 (85.7) | |
| Missing | 1 (5.9) | 0 (0) | 1 (14.3) | |
| Age in years, mean (SD) | 14.7 (1.7) | 14.7 (1.7) | 14.8 (1.7) | |
| Race, n (%) | ||||
| White | 15 (88.2) | 8 (80) | 7 (100) | |
| Mixed | 2 (11.8) | 2 (20) | 0 (0) | |
| Ethnicity, n (%) | ||||
| Non-Hispanic/Latino | 17 (100) | 10 (100) | 7 (100) | |
| Health Insurance, n (%) | ||||
| Private insurance | 15 (88.2) | 9 (90) | 6 (85.7) | |
| Medicaid | 2 (11.8) | 1 (10) | 1 (14.3) | |
| Living situation, n (%) | ||||
| Two parents | 12 (70.6) | 5 (50) | 7 (100) | |
| Diagnosis from chart review, n (%) | ||||
| Anxiety disorder | 17 (100) | 10 (100) | 7 (100) | |
| Depressive disorder | 10 (58.8) | 5 (50) | 5 (71.4) | |
| Initial treatment plan from chart review, n (%) | ||||
| SSRIa | 11 (64.7) | 6 (60) | 5 (71.4) | |
| Other medication(s) | 0 (0) | 0 (0) | 0 (0) | |
| CBTb | 5 (29.4) | 3 (30) | 2 (28.6) | |
| Unspecified counseling | 15 (88.2) | 8 (80) | 6 (85.7) | |
| Referral to psychiatry | 2 (11.8) | 1 (10) | 1 (14.3) | |
| Measures at Baseline, mean (SD) | ||||
| PHQ-9c | 12.1 (5.3) | 10.1 (3.9) | 14.9 (6) | |
| GAD-7d | 12.8 (4.9) | 11 (5) | 15.3 (3.5) | |
| MHSESe | 30.9 (10.2) | 32.4 (10.1) | 28.9 (10.9) | |
aSSRI: selective serotonin reuptake inhibitor.
bCBT: cognitive behavioral therapy.
cPHQ-9: 9-item Patient Health Questionnaire.
dGAD-7: 7-item Generalized Anxiety Disorder questionnaire.
eMHSES: Mental Health Self-Efficacy Scale.
Pilot RCT Outcomes
Table 2 and Figure 3 show the overall study results. The small sample size precluded meaningful comparisons between study groups. However, large effect sizes were observed between groups on depression symptom severity score at 4 weeks, with the intervention group showing greater improvement on each scale than the wait list control group. Specifically, mean PHQ-9 scores at 4 weeks decreased by 3.3 units in the intervention group (representing a transition from moderate to mild symptom severity categories) and 2 units in the wait list control group (no shift in symptom severity category). In subgroup analyses among the 10 participants diagnosed with depression, PHQ-9 scores similarly decreased by 3 units in the intervention group (11.8 to 8.8; from moderate to mild severity categories) versus 1 unit in the wait list control group (15 to 14; no shift in symptom severity category).
Table 2. Primary and secondary study outcomes at 4 weeks.
| Outcome, mean (SD) | Intervention group | Wait list control group | 95% CI | Cohen d |
| PHQ-9a | 6.8 (6.9) | 12.9 (5.4) | −12.9 to 0.7 | .98 |
| GAD-7b | 7.4 (6.5) | 11.7 (4.8) | −10.6 to 2 | .75 |
| MHSESc | 38.7 (8.4) | 31.7 (11) | −3.4 to 17.3 | .71 |
aPHQ-9: 9-item Patient Health Questionnaire.
bGAD-7: 7-item Generalized Anxiety Disorder questionnaire.
cMHSES: Mental Health Self-Efficacy Scale.
Enlarge this image.Figure 3. Change in mean PHQ-9 and GAD-7 scores by treatment group over 12 weeks. GAD-7: 7-item Generalized Anxiety Disorder questionnaire; PHQ-9: 9-item Patient Health Questionnaire.
We also compared remission in depressive symptoms (PHQ-9 < 5) at 4 and 12 weeks in patients with a baseline PHQ-9 > 9 [57] in the subgroup diagnosed with depression. The percentage of participants achieving remission at both time points seemed to favor the active intervention, at 67% (2/3) and 0% (0/5) at 4 weeks and 50% (1/2) and 20% (1/5) at 12 weeks, respectively.
Safety and Satisfaction With Care
When completing the PHQ-9 on one of the 6 adolescent surveys, 10 (59%) of participants, of which 4 (40%) were in the intervention group and 6 (35%) were in the wait list control group, triggered at least 1 alarm to assess for suicidal ideation. During study participation, 4 (24%) of the participants had 1 alert, 4 (24%) had 3, and 2 (12%) had 6. These patients were managed by their PCP with support from a pediatric research psychiatrist, who interviewed all 4 (24%) participants by phone, discussed management with the legal guardian (who was the mother in all cases), and facilitated referral of 2 (12%) participants to a child psychiatrist for ongoing clinical management. One parent from the intervention group reported at week 12 that their teen was seen in an ED and discharged to home. Parental satisfaction with care did not differ between study groups.
Acceptability, Feasibility, and Usability for Adolescents and Parents
Adolescents found the app to be (1) acceptable, with a total of 8 (80%) agreeing or completely agreeing with the statement “I like using the app”; (2) feasible, with 7 (70%) agreeing with the statement “using the app in the treatment of depression seems possible”; and (3) usable (mean usability score 21.4, SD 1.7, possible range 5 to 25). The mean parental scores for acceptability and feasibility were 16.6 (SD 1.8) and 17 (SD 1.5), respectively (possible range 1 to 20).
Acceptability and Feasibility for Providers
Potential advantages for learning CBT skills via an app compared with face-to-face training identified by provider participants included ease/immediacy of access to evidence-based behavioral therapy, increased sense of anonymity or reduction in perception of stigma against mental health concerns, reduction in school absences to attend appointments, no need for transportation to and from a therapist’s office, and reduction of out-of-pocket costs. Although providers acknowledged that an app might reduce common barriers to engagement in therapy, they voiced several concerns about using an app to augment clinical recommendations. Concerns commonly involved efficacy of the app, the potential safety risks of missing reports of suicidal ideation, overreliance on technology and reduced self-efficacy for adaptive help-seeking, increased screen time, and privacy and confidentiality of sensitive information.
Providers generally felt that an app to deliver CBT and positive psychology could be useful for teens with anxiety and mild-to-moderate depression, particularly early in the course of illness, and saw the app as being able to bridge the gap between diagnosis and obtaining an appointment with a therapist or the onset of therapeutic effects of prescribed antidepressant medications—both of which can take several weeks. Due to acute safety concerns, they recommended against using an app in teens with active suicidality or self-harm. They felt these patients needed closer clinical monitoring and were unlikely to use the app because of their low motivation to adopt new behaviors.
PCP participants suggested that the best time to introduce a CBT app to patients and families as part of the treatment plan would be at the time of diagnosis. Several suggested that usage might be improved if the pediatrician introduced the app during an appointment and showed them how to download and use it (eg, by saying to the patient, “Here’s what I’d like you to start doing…”). In summary, a CBT app was acceptable to these pediatric PCPs, and they suggested that it would be feasible to use in teens with mild/moderate depression or anxiety.
App Utilization
Participant app use during the intervention period (collected via the app) included days of app use, number of check-ins, and number of messages sent, lesson and tools use rates, lesson acceptability ratings indicated on a binary scale (ie, a thumbs up or thumbs down emoticon), and mood impact after tool utilization (ie, feeling same, better, or worse after completion). In-the-moment emotional state was reported through emoji selection with a default menu of 19 total moods, including options for negative (angry, sad, and anxious), positive (happy and content), and average mood (okay), with an additional ability to type in free text emotion words and self-selected emoji expressions.
Over the course of the intervention from baseline to 4 weeks, participants’ app use averaged a mean of 6 (SD 6.9) days, with 55 (SD 7.14) mood check-ins, and 313.17 (SD 447.30) sent messages, with the use of 13.63 (SD 4.5) psychoeducational lessons and 9 (SD 14.91) tools. All completed psychoeducational lessons were rated positively (thumbs up). Over two-thirds (7/10, 69%) of participants reported feeling better after using the app, with 25% (3/10) and 1% (1/10) reporting feeling the same or worse, respectively.
Discussion
In this pragmatically designed, randomized pilot and feasibility study, we demonstrated the preliminary acceptability and feasibility of augmenting initial depression treatment in adolescents with CBT delivered via an evidence-based chatbot intervention. Although the sample size needed to detect between-group differences was not achieved, we observed a trend toward improvement in depressive symptoms with the app over a wait list control condition. Moreover, we observed that the detection and management of suicidal ideation in depressed adolescents was safely enhanced by use of the app in concert with PCP-based clinical care and telephonic consultation with a child psychiatrist. Finally, we found that using this app to address depression and anxiety as part of an overall treatment plan developed in collaboration with the treating PCP was acceptable to the adolescents, their parents, and their PCPs. These findings suggest that this app should be further evaluated for use in the primary care management of adolescents with depression and anxiety.
Adolescents visit their PCP 2 to 3 times per year on average and commonly report feeling most comfortable disclosing mental health concerns and obtaining mental health care in the primary care setting [14]. A recent review to assess the efficacy and acceptability of cell phone apps to support mental health management in adolescents found high levels of adherence, but few evaluative studies employed quantitative assessment of these process measures, which are key for guiding real-world implementation. This small pilot study provided additional lessons regarding real-world clinical implementation of mobile health interventions in pediatric primary care. For example, the frequency of suicidal ideation reporting (as assessed by question 9 of the PHQ-9) decreased over the course of study participation, even when the severity of other depressive symptoms remained in a clinically significant range. Although this study was not designed to detect differences between groups on suicidality, we observed a reduction in self-reported suicidal ideation that appeared to favor the active treatment group. This is particularly relevant in the delivery of mental health care to adolescent populations, who may feel more comfortable sharing such information anonymously through an app than with a trusted adult or health care provider.
Our results are subject to important limitations. First, despite using a fully remote study design and a well-established PBRN during a period of high mental health need, recruitment was challenged by the circumstances of the COVID-19 pandemic and the reduced access to and utilization of primary care resources, especially early on in the pandemic. Although the hybrid study design is important for piloting the intervention in the patient population of interest, the results must be interpreted with caution. Few conclusions can be drawn about intervention effectiveness due to the small sample size and the racial and geographic homogeneity and relative affluence of the enrolled participants, who were mostly female, White, and privately insured. The ability to detect differences between treatment groups may have been further limited by the concurrent use of antidepressant medications in some participants. The results regarding implementation effectiveness are subject to additional limitations, as the PBRN and PBRN providers in this study are not representative of the broader population of pediatric primary care clinics or providers in the United States [16]. Moreover, our safety approach of providing case review and rapid, individualized review and treatment planning with a child psychiatrist may also have contributed to the lack of separation between treatment groups, as everyone in the study received high-quality clinical care and close follow-up for suicidality reported on the PHQ-9.
The fully remote nature of the trial, which was conducted during critical periods in the COVID-19 pandemic, when child and adolescent mental health needs were most critical, was both a limitation and a strength. Additionally, this is the first study we are aware of that employed digital assessment and remote methods for managing suicidality in adolescents. The concern that suicidality will be exacerbated or missed with digital interventions has been a major barrier to offering these treatments to youth. The method we used to ensure safety (phone consultation with a child psychiatrist within 24 hours of reporting clinically significant suicidal ideation) could not have disguised symptom severity in the wait list control group, since we were responsive to all alerts. Another major strength of this pilot study was evaluating the feasibility of implementation in a real-world primary care setting. Although this study does not provide sufficient evidence to support the use of apps in primary care settings for teens with mental health problems [21], the results presented in this paper provide promising evidence that preliminarily support the feasibility of using a CBT-based chatbot technology to supplement mental health treatment of adolescents with mild-to-moderate depression cared for in the primary care setting. Additional studies are needed to further evaluate the effectiveness of both the intervention and implementation in real-world primary care settings and in patient populations that include underrepresented populations and prioritize socioeconomically disadvantaged and rural communities, where access to high quality medical and mental health care are extremely limited.
Acknowledgments
Research reported in this publication was made possible through an academic partnership with the leadership of WoeBot, Allison Darcy, PhD (founder and president), Athena Robinson, PhD (chief clinical officer), and Sarah Parajito, MA (research scientist at WoeBot). This work was funded by the Washington University School of Medicine (WUSM) Clinical and Translational Funding Program (CTRFP) and the Institute of Clinical and Translational Sciences (ICTS) grant (UL1TR002345) from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH) and NIH/NCATS UL1 TR000445 (REDCap). The work reported in this manuscript was supported by the WUSM Pediatric and Adolescent Ambulatory Research Consortium (WU PAARC) and the WUSM Mobile Health Research Core (mHRC), both research cores of the WU ICTS. The content of this manuscript is solely the responsibility of the authors and does not necessarily represent the official view of the NIH. Finally, we would like to thank the patients, families, and WU PAARC physicians for their participation, without which this study would not have been possible.
Conflicts of Interest
GN has received research support from Usona Institute (drug only), has served as principal or co-investigator on studies sponsored by Alkermes, Otsuka and LB Pharmaceuticals, and has received consulting fees from Alkermes, IngenioRx, Novartis and Sunovion. RW, SG, SD and JG have no conflicts of interest to disclose. AD, AR and SP are employees of WoeBot Health and were not involved in obtaining grant funding or IRB approval for this study, nor did they participate in data analyses. WoeBot Health provided app utilization data as reported in this manuscript.
------------
Abbreviations
| AIM: Acceptability of Intervention Measure |
| AMA: American Medical Association |
| APA: American Psychiatric Association |
| CBT: cognitive behavioral therapy |
| DBT: dialectical behavior therapy |
| DSM-5: Diagnostic and Statistical Manual of Mental Disorders fifth edition |
| ED: emergency department |
| FIM: Feasibility of Intervention Measure |
| GAD-7: 7-item Generalized Anxiety Disorder questionnaire |
| ICTS: Institute for Clinical and Translational Science |
| IPT-A: interpersonal psychotherapy for adolescents |
| mHealth: mobile health |
| MHSES: Mental Health Self-Efficacy Scale |
| PBRN: practice-based research network |
| PCP: primary care provider |
| PHQ-9: 9-item Patient Health Questionnaire |
| PHQ-A: 9-item Patient Health Questionnaire modified for Adolescents |
| PI: principal investigator |
| QI: quality improvement |
| RCT: randomized controlled trial |
| REDCap: Research Electronic Data Capture |
| SNRI: serotonin-norepinephrine reuptake inhibitor |
| SSRI: selective serotonin reuptake inhibitor |
| SUS: System Usability Scale |
| WU PAARC: Washington University Pediatric and Adolescent Ambulatory Research Consortium |
------------
Edited by A Mavragani; submitted 12.06.22; peer-reviewed by R Benson, A Ecker; comments to author 14.09.22; revised version received 10.10.22; accepted 25.10.22; published 22.11.22
Saluja G, Iachan R, Scheidt PC, Overpeck MD, Sun W, Giedd JN. Prevalence of and risk factors for depressive symptoms among young adolescents. Arch Pediatr Adolesc Med 2004 Aug;158(8):760-765.
Perou R, Bitsko RH, Blumberg SJ, Pastor P, Ghandour RM, Gfroerer JC, Centers for Disease Control and Prevention (CDC). Mental health surveillance among children--United States, 2005-2011. MMWR Suppl 2013 May 17;62(2):1-35.
Asarnow JR, Jaycox LH, Duan N, LaBorde AP, Rea MM, Murray P, et al. Effectiveness of a quality improvement intervention for adolescent depression in primary care clinics: a randomized controlled trial. JAMA 2005 Jan 19;293(3):311-319.
McCarthy C. Anxiety in teens is rising: What's going on? American Academy of Pediatrics. 2019. URL: https://www.healthychildren.org/English/health-issues/conditions/emotional-problems/Pages/Anxiety-Disorders.aspx [accessed 2020-09-10]
Leeb RT, Bitsko RH, Radhakrishnan L, Martinez P, Njai R, Holland KM. MMWR Morb Mortal Wkly Rep 2020 Nov 13;69(45):1675-1680
Racine N, McArthur BA, Cooke JE, Eirich R, Zhu J, Madigan S. Global prevalence of depressive and anxiety symptoms in children and adolescents during COVID-19: a meta-analysis. JAMA Pediatr 2021 Nov 01;175(11):1142-1150
Heron M. Deaths: leading causes for 2017. Natl Vital Stat Rep 2019 Jun;68(6):1-77
Miron O, Yu K, Wilf-Miron R, Kohane IS. Suicide rates among adolescents and young adults in the United States, 2000-2017. JAMA 2019 Jun 18;321(23):2362-2364
Curtin SC. State suicide rates among adolescents and young adults aged 10-24: United States, 2000-2018. Natl Vital Stat Rep 2020 Sep;69(11):1-10.
Yard E, Radhakrishnan L, Ballesteros MF, Sheppard M, Gates A, Stein Z, et al. Emergency department visits for suspected suicide attempts among persons aged 12-25 years before and during the COVID-19 pandemic - United States, January 2019-May 2021. MMWR Morb Mortal Wkly Rep 2021 Jun 18;70(24):888-894
March J, Silva S, Petrycki S, Curry J, Wells K, Fairbank J, et al. Fluoxetine, cognitive-behavioral therapy, and their combination for adolescents with depression: Treatment for Adolescents With Depression Study (TADS) randomized controlled trial. JAMA 2004 Aug 18;292(7):807-820.
Cheung AH, Zuckerbrot RA, Jensen PS, Laraque D, Stein REK, GLAD-PC Steering Group. Guidelines for Adolescent Depression in Primary Care (GLAD-PC): Part II. Treatment and ongoing management. Pediatrics 2018 Mar;141(3)
Zuckerbrot RA, Cheung A, Jensen PS, Stein REK, Laraque D, GLAD-PC Steering Group. Guidelines for Adolescent Depression in Primary Care (GLAD-PC): Part I. Practice preparation, identification, assessment, and initial management. Pediatrics 2018 Dec;141(3)
Stein REK, Zitner LE, Jensen PS. Interventions for adolescent depression in primary care. Pediatrics 2006 Aug;118(2):669-682.
Shafran R, Clark DM, Fairburn CG, Arntz A, Barlow DH, Ehlers A, et al. Mind the gap: Improving the dissemination of CBT. Behav Res Ther 2009 Nov;47(11):902-909.
Garbutt J, Sterkel R, Ruecker K, Dodd S, Smith E, Plax K. Ready for the challenge of depression care in the medical home. Clin Pediatr (Phila) 2019 Jun;58(7):816-818
Fulmer R, Joerin A, Gentile B, Lakerink L, Rauws M. Using psychological artificial intelligence (tess) to relieve symptoms of depression and anxiety: randomized controlled trial. JMIR Ment Health 2018 Dec 13;5(4):e64
Kretzschmar K, Tyroll H, Pavarini G, Manzini A, Singh I, NeurOx Young People’s Advisory Group. Can your phone be your therapist? Young people's ethical perspectives on the use of fully automated conversational agents (chatbots) in mental health support. Biomed Inform Insights 2019;11:1178222619829083
Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. J Anxiety Disord 2018 Apr;55:70-78
Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017 Jun 06;4(2):e19
Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MFG. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. BMJ 2012 Apr 18;344:e2598
Ly KH, Ly A, Andersson G. A fully automated conversational agent for promoting mental well-being: A pilot RCT using mixed methods. Internet Interv 2017 Dec;10:39-46
Hoermann S, McCabe KL, Milne DN, Calvo RA. Application of synchronous text-based dialogue systems in mental health interventions: systematic review. J Med Internet Res 2017 Jul 21;19(8):e267
Ebert DD, Zarski A, Christensen H, Stikkelbroek Y, Cuijpers P, Berking M, et al. Internet and computer-based cognitive behavioral therapy for anxiety and depression in youth: a meta-analysis of randomized controlled outcome trials. PLoS One 2015;10(3):e0119895
Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009;38(4):196-205.
Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res 2017 May 25;19(5):e176
Aschbrenner KA, Naslund JA, Tomlinson EF, Kinney A, Pratt SI, Brunette MF. Adolescents' use of digital technologies and preferences for mobile health coaching in public mental health settings. Front Public Health 2019 Jul 2;7:178
Nick EA, Kilic Z, Nesi J, Telzer EH, Lindquist KA, Prinstein MJ. Adolescent digital stress: frequencies, correlates, and longitudinal association with depressive symptoms. J Adolesc Health 2022 Feb;70(2):336-339.
Arias-de la Torre J, Puigdomenech E, García X, Valderas JM, Eiroa-Orosa FJ, Fernández-Villa T, et al. Relationship between depression and the use of mobile technologies and social media among adolescents: umbrella review. J Med Internet Res 2020 Aug 26;22(8):e16388
Glasgow RE, Fisher L, Strycker LA, Hessler D, Toobert DJ, King DK, et al. Minimal intervention needed for change: definition, use, and value for improving health and health research. Transl Behav Med 2014 Mar;4(1):26-33
Li LSE, Wong LL, Yap KY. Quality evaluation of stress, anxiety and depression apps for COVID-19. J Affect Disord Rep 2021 Dec;6:100255
Ramachandran M, Suharwardy S, Leonard S, Gunaseelan A, Robinson A, Darcy A, et al. Acceptability of postnatal mood management through a smartphone-based automated conversational agent. Am J Obstet Gynecol 2020 Jan;222(1):S62
Suharwardy S, Ramachandran M, Leonard S, Gunaseelan A, Robinson A, Darcy A, et al. Effect of an automated conversational agent on postpartum mental health: A randomized, controlled trial. Am J Obstet Gynecol 2020 Jan;222(1):S91
Prochaska JJ, Vogel EA, Chieng A, Kendra M, Baiocchi M, Pajarito S, et al. A therapeutic relational agent for reducing problematic substance Use (Woebot): development and usability study. J Med Internet Res 2021 Mar 23;23(3):e24850
Prochaska JJ, Vogel EA, Chieng A, Baiocchi M, Maglalang DD, Pajarito S, et al. A randomized controlled trial of a therapeutic relational agent for reducing substance misuse during the COVID-19 pandemic. Drug Alcohol Depend 2021 Oct 01;227:108986
Darcy AP, Eaneff S, Parajito S, Robinson A. Feasibility, acceptability, and effectiveness of a digital therapeutic application vs psychoeducation content for mild-to-moderate depression among adolescents. 2020 Presented at: Association for Behavioral and Cognitive Therapies; November 19-22; Virtual p. 2022.
Garbutt J, Dodd S, Rook S, Graham S, Wang R, Sterkel R, et al. Improving follow-up for adolescents with depression in primary care. Pediatrics 2022 Jun 01;149(6):2021.
Torous JB, Chan SR, Gipson SYT, Kim JW, Nguyen T, Luo J, et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv 2018 Feb 15:appips201700423.
O'Reilly K. Proposed guidelines aim for safe, effective mobile health apps. American Medical Association. 2018. URL: https://www.ama-assn.org/practice-management/digital/proposed-guidelines-aim-safe-effective-mobile-health-apps [accessed 2022-09-10]
Kianersi S, Luetke M, Ludema C, Valenzuela A, Rosenberg M. Use of research electronic data capture (REDCap) in a COVID-19 randomized controlled trial: a practical example. BMC Med Res Methodol 2021 Aug 21;21(1):175
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613
Johnson JG, Harris ES, Spitzer RL, Williams JBW. The patient health questionnaire for adolescents: validation of an instrument for the assessment of mental disorders among adolescent primary care patients. J Adolesc Health 2002 Mar;30(3):196-204.
Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097.
Dear BF, Titov N, Sunderland M, McMillan D, Anderson T, Lorian C, et al. Psychometric comparison of the generalized anxiety disorder scale-7 and the Penn State Worry Questionnaire for measuring response during treatment of generalised anxiety disorder. Cogn Behav Ther 2011;40(3):216-227.
Tiirikainen K, Haravuori H, Ranta K, Kaltiala-Heino R, Marttunen M. Psychometric properties of the 7-item Generalized Anxiety Disorder Scale (GAD-7) in a large representative sample of Finnish adolescents. Psychiatry Res 2019 Feb;272:30-35.
Sun J, Liang K, Chi X, Chen S. Psychometric properties of the Generalized Anxiety Disorder Scale-7 Item (GAD-7) in a large sample of Chinese adolescents. Healthcare (Basel) 2021 Dec 09;9(12):1709
Ryu HJ, Lee S, Chung JH. Association between asthma and generalized anxiety disorder-7 (GAD-7) in Korean adolescents. J Asthma 2022 Feb 02:1-7.
Adjorlolo S. Generalised anxiety disorder in adolescents in Ghana: Examination of the psychometric properties of the Generalised Anxiety Disorder-7 scale. AJOPA 2019 Jul 18;1.
Romano I, Ferro M, Patte K, Leatherdale S. Measurement invariance of the GAD-7 and CESD-R-10 among adolescents in Canada. J Pediatr Psychol 2022 May 13;47(5):585-594
Mossman SA, Luft MJ, Schroeder HK, Varney ST, Fleck DE, Barzman DH, et al. The Generalized Anxiety Disorder 7-item scale in adolescents with generalized anxiety disorder: Signal detection and validation. Ann Clin Psychiatry 2017 Nov;29(4):227-234A
Clarke J, Proudfoot J, Birch M, Whitton AE, Parker G, Manicavasagar V, et al. Effects of mental health self-efficacy on outcomes of a mobile phone and web intervention for mild-to-moderate depression, anxiety and stress: secondary analysis of a randomised controlled trial. BMC Psychiatry 2014;14:272
Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017 Aug 29;12(1):108
Lewis JR. The System Usability Scale: past, present, and future. Int J Hum-Comput Interact 2018 Mar 30;34(7):577-590
Brooke J. SUS: a "quick and dirty" usability scale. In: Usability Evaluation in Industry. London, UK: Taylor and Francis; 1996:189-194.
Ramtekkar U, Maras M, Ell W, Nicol G, Young-Walker L. Academic-community partnership to improve pediatric mental health access: Missouri Child Psychiatry Access Project. Psychiatr Serv 2022 May;73(5):588-591.
Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101.
Schueller SM, Kwasny MJ, Dear BF, Titov N, Mohr DC. Cut points on the Patient Health Questionnaire (PHQ-9) that predict response to cognitive-behavioral treatments for depression. Gen Hosp Psychiatry 2015;37(5):470-475
© 2022. This work is licensed under https://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Wang, Ruoyun
; Graham, Sharon
; Dodd, Sherry
; Garbutt, Jane