Mobile Therapeutic Attention for Treatment-Resistant Schizophrenia (m-RESIST) Solution for Improving Clinical and Functional Outcomes in Treatment-Resistant Schizophrenia: Prospective, Multicenter Efficacy Study

Abstract

Background:Treatment-resistant schizophrenia (TRS) is a severe form of schizophrenia associated with low adherence to treatment and poor outcomes. Mobile health (mHealth) interventions may be effective in preventing relapses, increasing treatment adherence, and managing some of the symptoms of schizophrenia. Mobile therapeutic attention for treatment-resistant schizophrenia (m-RESIST) is an innovative mHealth developed specifically for TRS.

Objective:We aim to evaluate the effects of m-RESIST on the clinical and functional outcomes and on the perceived quality of life in people with TRS.

Methods:A feasibility study without a control group was performed to test the m-RESIST solution on patients with TRS. Participants were recruited from Spain, Israel, and Hungary. This study’s population (N=31) followed 3 months of intervention. The m-RESIST was configured by an app, a wearable, and a web-based platform. The severity of symptoms was evaluated by using the Positive and Negative Syndrome Scale (PANSS) and the Clinical Global Impression-Schizophrenia (CGI-SCH) scale. Functionality was assessed by the Global Assessment of Functioning and perceived quality of life was evaluated by the EuroQol visual analogue scale (EQ-VAS).

Results:Significant reductions were found in symptoms from pretrial to posttrial on the PANSS total (mean difference −7.2, 95% CI −11.1 to −3.4; P=.001), the PANSS positive (mean difference −1.36, 95% CI −2.6 to −0.1; P=.04), the PANSS negative (mean difference −2.1, 95% CI −3.1 to −1.1; P<.001), and the PANSS general symptoms (mean difference −3.8, 95% CI −6.8 to −0.8; P=.02). In almost one-fifth of the participants (6/31), the overall score for the PANSS decreased by more than 20%, which may be considered a clinically significant change. On the CGI-SCH scale, the sum of total severity of illness decreased significantly (P=.03). A decrease in the sum of positive and negative symptoms of the CGI-SCH score was also found (P=.04 and P=.03, respectively). The sum of depressive or cognitive symptoms did not change. The functionality of participants increased significantly on the Global Assessment of Functioning (P≤.001). The perceived quality of life on the EQ-VAS also improved (mean difference 6.7, 95% CI 0.5 to 12.9; P=.04).

Conclusions:To our best knowledge, this was the first study to address the efficacy of the mHealth app m-RESIST on the symptoms and functional capacity and on the quality of life for people with TRS. Our preliminary findings showed that implementing the m-RESIST solution decreased the symptoms and severity of disease, and improved the functionality and perceived quality of life among those with TRS. The change of symptoms on the PANSS total may be clinically significant. Modern technologies such as mHealth interventions may be useful in treating symptoms and functionality even in TRS, which is a major clinical challenge, with usually poor outcomes. These results should be corroborated by performing a controlled trial.

Trial Registration:ClinicalTrials.gov NCT03064776; https://clinicaltrials.gov/study/NCT03064776

International Registered Report Identifier (IRRID):RR2-10.1136/bmjopen-2017-021346

Introduction

Background

Schizophrenia is a severe mental disorder and has a prevalence of 1% of the population worldwide [1]. A meta-analysis showed that only 13.5% of the patients were in recovery, which included clinical remission and good social functioning outcome [2]. Despite the common use of antipsychotic drugs, 30%‐50% of patients with schizophrenia still obtained a low benefit from conventional treatments [3,4]. This inadequate response to pharmacological treatment is known as treatment-resistant schizophrenia (TRS). TRS implicates an important burden at three levels, as follows: (1) clinical (negative attitude to medication, drug abuse, and nutritional or physical health problems); (2) economic (hospitalizations and polytherapy); and (3) humanistic (depression and social isolation) [5]. As a result, dealing with TRS involves a high emotional burden for patients and their caregivers, affecting their quality of life. The current health care and social support systems are unable to provide adequate and effective solutions to these patients.

Results from earlier studies on psychosis [6-8] supported that information and communication technology (ICT) tools in the treatment of patients with TRS would be useful. Smartphone ownership among people with schizophrenia is also relatively high and increasing [9-11]. Patients seemed capable of using mobile phones to monitor their symptoms, participate in therapeutic interventions, and increase physical exercise [12]. Previous feasibility studies have shown that interventions based on mobile health (mHealth) could promote the empowerment of patients with schizophrenia [9,13]. Some mHealth interventions may also be effective for increasing treatment adherence and decreasing relapses [13,14].

Several digital solutions also specifically for schizophrenia exist [15,16]. Apps such as FOCUS [6,17-19], SMARTapp [20], MACS [21], weCope [22], My Journey 3 [23,24], and App4Independence [25] offer disease management strategies in general. Others are more focused on symptom monitoring, namely Crosscheck [26], Actissist [27], ClinTouch [28], ExPRESS [29], Ginger.io [30], ReMindCare [31], and MindFrame [32]. Other mobile apps are supported in cognitive behavioral therapy (SlowMo) [33]; TechCareApp [34]; movisensXS [35]; PRIME [36], or work as an agenda (MONEO) [37], or target specific domains such as social skills (MASS) [38], sleep (ExpiWell) [39], medication adherence (MedActive) [40], or delivers positive psychology content (+ Connect) [41]. Digitally delivered psychological therapies, the use of social media, virtual reality, and chatbots are also digital solutions that may have a role in the treatment of those with schizophrenia [42,43]. Digital phenotyping and machine learning, virtual reality interventions, and language models, with ChatGPT, as a well-known example, have developed in recent years [42,43].

A systematic review and meta-analysis by Vitger et al [44] examined digital shared decision-making interventions in mental health care, analyzing 16 randomized controlled trials (RCTs), of which only 3 evaluated apps. The findings suggest that digital shared decision-making tools may positively impact patient activation, decisional conflict, working alliance, and symptom severity. Specific digital interventions include a tablet app for patients with schizophrenia, designed to assess and record satisfaction with life and treatment domains, facilitating discussions between patients and nurses [45]. Another tablet app addresses concerns raised by patients with psychosis, providing methods for clinicians and patients to collaboratively explore and manage issues [46]. Additionally, the smartphone app “Power Up” aims to promote patient activation among children and adolescents facing mental health challenges [47].

Patient-reported outcomes collect data directly from patients. These data are used in clinical practice, helping decision-making. A systematic mapping review analyzed the characteristics and outcomes of mHealth interventions in psychosis based on 29 publications reporting on 23 studies [48]. Objective measures included data passively obtained from phone usage or embedded sensors. Subjective measurement instruments include either observer-rated or self-report questionnaires that can be written or administered digitally, whether regarding their health status or experience using the mHealth intervention [48]. Outcomes were grouped into 8 categories, namely severity of psychosis-related symptoms, functioning, well-being, medication adherence, adverse events, user experience, technical, and all other outcomes reported [48].

Real-time and in-context patient-generated symptom data, obtained through remote-monitoring platform technology, have the potential to timely warn clinicians about the need for intervention, improve treatment decisions by providing a clearer picture of changing patterns of symptoms, and support scheduling of health care contacts based on need [42].

Systematic comparison of apps with other digital interventions may help us to better understand their roles in mental health care. The main use of smartphone data (active and passive) is tracking mood and lifestyle in people with major depression, bipolar disorder, and psychosis. They also have future potential in machine learning toward individualized risk prediction and delivery of targeted “just in time” intervention. Key issues are a lack of validation across studies and establishing trust around data usage. Priority actions include data standards for interoperability and validation, and industry-academic partnerships around access [42].

Digitally delivered psychological therapies are used in the self-management of symptoms of depression and anxiety. They have the potential for precision interventions and preventive treatments. Lack of engagement, saturated consumer marketplace, and claims outpacing clinical evidence are major issues. Establishing an evidence base for use in people with diagnosed mental disorders is warranted [42].

Population level monitoring of mood and anxiety is possible with the help of social media. The future potential of social media lies on its potential for real time monitoring of mental health state and for access to peer support. Key issues are sampling bias, access to data from social media companies, and also privacy. Priority actions are industry-academic partnerships and privacy standards [42].

The main use of virtual reality is exposure therapies, and it may have higher engagement and potentially higher efficacy than apps. Increased accessibility, low-cost headsets, and expanded clinical targets are important [42]. Chatbots have future potential to increase access to care but they have limited range of appropriate response. Establishing evidence base for use in people with diagnosed mental disorders is needed [42].

The 5 principles of the Collaborative Care Model are evidence-based care, measurement-based treatment to target, population health management, accountable care, and team-based care [49]. Practices with integrated mental health clinicians can maximize the impact of digital mental health tools by integrating digital information and interventions into the course of clinical care. This allows for the practice of hybrid care, a combination of synchronous and asynchronous health care, which can enable increased access to care. Hybrid care can address the known challenge of low retention in digital health tools, given that a digital navigator or clinician can promote engagement, guide the patient to relevant content, and, depending on the tool, use data from the patient’s use in further guiding face-to-face care [49].

The findings from the pilot study suggest that smartphone apps may become a useful tool for psychiatric rehabilitation, addressing both psychiatric and co-occurring medical problems. Individualizing functions to each patient and facilitating connection with a certified peer specialist may be an important feature of useful apps [50].

Apps can be successfully integrated into real-world care [43]. These models of care involve both synchronous (eg, a therapy visit) and asynchronous telehealth (ie, app use between visits) and can provide therapeutic synergy between in-person and remote care [51]. The ReMindCare app for early psychosis has been integrated into clinics since 2018, and implementation models for integrating apps in youth services for schizophrenia, and coordinated specialty, have already been developed [31,52-54]. While the actual dissemination of these models is impossible to ascertain, they can already support the use of apps to improve vocational training, educational opportunities, medication adherence, cognitive training, symptom tracking, psychoeducation, and more. Apps for schizophrenia have also led to the development of human support roles as part of mental health services to facilitate their implementation in clinical practice, such as the FOCUS mHealth support specialist [55] and the mindLAMP digital navigator [53].

A renewed focus on privacy and data protection is essential for the success of mental health apps, as users must trust these apps for widespread adoption. Many health-related apps collect sensitive data, including demographic, medical, and lifestyle information, sometimes even unrelated to the app’s intended purpose [42,43,56,57]. Despite the importance of privacy, a significant portion of mental health apps (31%‐49%) lack a privacy policy, and collected data can be shared with third parties or used for advertising [42,43,56,57]. Vulnerable populations, such as individuals with schizophrenia, require additional research protections to ensure their privacy and security [43].

A cross-cultural psychological assessment focuses on understanding how a client’s cultural background influences their behavior, emotional state, and psychological needs. A recent review of 2248 studies found that only 57 specifically targeted migrants, refugees, or cultural minorities [58]. Of these, half used digital health technologies developed for the general population without cultural adaptations, while the other half tailored the interventions to meet the specific needs of these populations [58]. However, only a small number of studies involved a high level of participation from these groups in the development and testing of the digital tools [58].

Despite rapid progress, there is room for improvement. Presently, there are fewer than 10 apps whose design was specifically tailored to people with schizophrenia and that are accessible for immediate public download on the Apple or Android (Open Handset Alliance [led by Google]) marketplaces. Moreover, many of these 10 apps appear neglected by developers after their launch, with an average time since the last update of 1121 days [43,59]. The number of studies on the efficacy of mHealth apps on the symptoms and functional capacity, and on the quality of life in schizophrenia and related psychotic disorders is limited according to recent systematic reviews [15,56,60,61].

The systematic review by Miralles et al [56] found 3 studies evaluating efficacy of mHealth solutions in schizophrenia spectrum and in other psychotic disorders. Two of the three papers found decrease in symptoms measured by the Positive and Negative Syndrome Scale (PANSS) [6,27]. The third one reported no significant change in the symptoms on the PANSS, or no positive changes in functioning and in quality of life [36]. .

Additionally, a cross-sectional study on WeChat use and endorsement of WeChat-based mHealth among people living with schizophrenia (N=400) in China was carried out. Compared to nonusers, WeChat use was associated with improved health outcomes, including lower psychiatric symptoms, lower depression, higher functioning, better recovery, and higher quality of life [62].

A recent study of patients with a diagnosis of schizophrenia, randomized to receive either SlowMo, a blended digital therapy targeting reasoning, or usual care, found no significant difference between groups on the primary outcomes related to paranoia at 24 weeks (the primary end‐point) [33].

The Mobile Therapeutic Attention for Treatment-Resistant Schizophrenia Project

The project of mobile therapeutic attention for treatment-resistant schizophrenia (m-RESIST), funded by the Horizon 2020 Programme of the European Union, designed and developed a mobile therapeutic attention for TRS. This digital solution consisted of a new and innovative mHealth solution based on novel technology and offering highly modular and flexible functioning. This solution was developed specifically for patients with TRS and their caregivers based on their needs [63-65].

The positive acceptance of the m-RESIST solution was related to its usefulness in meeting user needs, its capacity to empower them, and its possibility of maintaining human contact [63].

The only mHealth study performed in those with TRS indicated that the m-RESIST solution is feasible and is an acceptable, satisfactory, and potentially useful tool for a population with TRS [65].

Aims

In this study, our aim was to evaluate the effects of the m-RESIST solution on the clinical and functional outcomes and the perceived quality of life for those with TRS. To our best knowledge, there are no previous studies which have addressed these issues.

Methods

Study Design and Setting

This was a prospective, 3-month long, multicenter follow-up study without a control group that was carried out with patients with a diagnosis of TRS between March and November 2017. Blinding was not considered due to the absence of a comparator group. No sample size calculation was done.

In this study, the intervention involved patients and their caregivers, and the main actors involved in the deployment of the m-RESIST solution were psychiatrists, psychologists, and case managers. The key aim of the m-RESIST solution was to engage patients with TRS, together with their caregivers, in therapeutic processes, and empower them to enable their self-management.

Participants were recruited from 3 clinical sites: Sant Pau Hospital, Semmelweis University, Sheba Medical Center, and Gertner Institute of Epidemiology and Health Policy Research. These sites provided full-spectrum mental health services to adult people from their catchment area and have extensive experience in participating in large-scale clinical trials in patients with schizophrenia.

Patients underwent a 3-month long modular intervention designed for patients with TRS. The intervention was based on the recognition of early warning signs of psychosis to improve positive symptoms, treatment adherence, and a healthy lifestyle.

The m-RESIST Solution

The intervention was supported by 3 mHealth tools: a wearable (smartwatch), a mobile app, and a web-based platform, which are presented in the figure of the protocol article. Data for usage was captured automatically by the m-RESIST software [64].

The functionality of the smartphone was based on the m-RESIST app. Through this app, patients had access to educational content about TRS condition and related issues; could track their early warning signs symptoms and biological variables (eg, sleep and steps counter); could ask for help with questionnaires or the “alarm button”; receive and practice helpful CBT-based coping strategies and exchange messages with their caregiver or health care provider [64].

The wearable was a smartwatch that collected data from patients and sent it wirelessly to the smartphone. Sensor data was recorded through automatic passive upload. The variables collected were the level of activity, heart rate, sleeping pattern, and steps counter [64].

The web-based platform was the tool that the health care providers (case manager, psychiatrist, and psychologist) used to collect assessment data, to monitor patients’ state and review data collected by sensors, to communicate by texting with patients, caregivers, and other professionals, and to consult recommendations (based on guidelines and experts’ opinion) [64]. Mental health care professionals helped patients with TRS in using the intervention [64]. The m-RESIST intervention was provided in 5 languages: Spanish, Catalan, Hungarian, Hebrew, and English.

The trial was registered in Clinical Trials (NCT03064776), and the study protocol was published earlier [64].

Staff or Participant Training Protocols

Three user guides were prepared to ensure that all m-RESIST users (patients, caregivers, and clinicians) will be able to understand and follow the instructions of the devices or system used in home or health care facilities. The dashboard user guide explained quickly and simply the applications and functionalities that clinicians can perform when logging in. The mobile application installation guide was addressed to clinicians in case they need to install the m-RESIST solution in patients’ devices. This guide details the steps to download the m-RESIST application and install it on the smartphone and the smartwatch. The mobile application user guide was addressed to patients, caregivers, and clinicians and described the functionalities of the m-RESIST app working together with the smartwatch, once the app was installed on the Android smartphone provided. This user guide also specifies briefly the operations and the use of the smartwatch given to the patient at the beginning of this study. Finally, a short instruction sheet was designed. This sheet contained the main steps to charge smartphone and smartwatch devices and the main considerations to take into account to ensure devices were connected and worked properly.

In case of the PANSS, raters at each site had to perform a training to ensure liability between sites.

Language Adaptation Process

To ensure the language adaptation process, the clinical research team from each site was responsible for adapting the m-RESIST solution from English to Spanish, Catalan, Hungarian, and Hebrew.

Technical Support Procedures

To solve and improve the issues of the m-RESIST solution during the pilots, incidences were collected by the clinicians and communicated by email to the technical team. Thus, they could fix them.

Ethical Considerations

This study protocol received approval from the ethics committees of the 3 participating institutions: Hospital Santa Creu i Sant Committee (IIBSP-RES-2016‐51), Egészségügyi Tudományos Tanács Tukeb, Semmelweis Ethical Committee (54920-4/2016/EKU), and Sheba Medical Center Ethical Committee (3472‐16 SMC). Participants were fully informed of the purpose and procedures of this study, and signed informed consent forms were obtained before any study-related activities were performed. The database generated by this study did not contain any identification of the participants but kept only a numerical code in a separate list; thus, the patients’ identities were protected. The information collected in this study was always treated as grouped data and never as individual or personal data, thereby maintaining anonymity and confidentiality. No compensation was provided to the participants.

Study Participants

Patients with TRS were recruited for this study. The eligibility criteria were (1) patients aged 18‐45 years with a diagnosis of schizophrenia according to Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition criteria; (2) duration of disease less than 15 years to meet the criteria for TRS [66,67]; (3) familiarity with ICT tools; and (4) physical capability to use them; and presence of a caregiver. Exclusion criteria were (1) remission according to the Remission of Schizophrenia Working Group [68]; (2) presence of delusions mainly related to their therapists or with new technologies; (3) presence of vision, hearing, or motor impairment, interfering with operating a smartphone; (4) presence of a caregiver or informal carer who is not used to ICT tools or has physical incapability to use them; and (5) presence of intellectual developmental disability. The caregivers were also recruited as part of the m-RESIST project.

A total of 81 patients with TRS, with their caregivers, were selected for invitation to participate by researchers. Almost half of them (n=39) were willing to enroll, and nearly 80% (n=31) of those started this study and followed the whole 3 months of intervention. The dropout rate was 18% (n=8).

Measures in This Study

All the outcomes of this study’s protocol are shown in Table 1.

Table 1.Schedule of study protocol periods and assessments.

	Recruitment	Preintervention	Intervention						Follow-up
Activity or assessment		V0	V1	V2	V3	V4	V5	V6	V7
Eligibility screen	✓
Informed consent	✓
Delivery and training of devices	✓
Sociodemographic data		✓
Clinical characteristics		✓
Relapse signature and treatment plan		✓
Willingness to enroll	✓
PANSSa		✓							✓
CGI-SCHb		✓	✓	✓	✓	✓	✓	✓	✓
GAFc		✓							✓
EQ-VASd		✓							✓

^aPANSS: Positive and Negative Syndrome Scale.

^bCGI-SCH: Clinical Global Impression Schizophrenia.

^cGAF: Global Assessment of Functioning.

^dEQ-VAS: EuroQol visual analogue scale.

In this study, the severity of symptoms was assessed using the PANSS and the Clinical Global Impression-Schizophrenia (CGI-SCH) scale [69,70].

The PANSS was used to assess positive and negative symptoms of schizophrenia. This scale consists of 30 items rated from 1 to 7 according to increasing levels of psychopathology, where: 1=absent, 7=extreme. Four scores are obtained: positive, negative, general symptoms, and total. The PANSS was completed on 2 occasions, at the start (VO) and the end (V7) of the intervention.

The CGI-SCH scale measures global clinical impression and consists of 5 different ratings (positive, negative, depressive, cognitive, and global) that are rated from 1 to 7, where: 1=normal, 7=among the most extremely ill patients. the CGI-SCH scale was evaluated in all protocol visits (V0-V7).

Functional and Perceived Quality of Life Outcomes

Functionality was assessed by using the Global Assessment of Functioning (GAF) scale [71] and perceived quality of life by the EuroQol visual analogue scale (EQ-VAS) [72].

The GAF scale ranges from 1 (maximum level of disease) to 100 (the healthiest). It was performed on 2 occasions, at VO and at V7.

The EQ-VAS is scored on a 20 cm visual analogue scale with end points labeled “the best health you can imagine” and “the worst health you can imagine.” It was administered on 2 occasions, at V0 and V7.

Adverse Events

Clinician asked for adverse events at each pilot visit, and they were reported. Safety measures were collected throughout this study. The presence of serious and nonserious adverse events, defined as any clinical change or illness reported during this study, was monitored in every clinical visit. The adverse events observed when carrying out this study, either by the clinician or by the patient themselves and regardless of the causality relationship ascribed, were recorded in the clinical records and on the patient`s dashboard.

Statistical Analysis

The sociodemographic characteristics of the participants with TRS were presented using frequency distributions for categorical variables and mean with SD for age. The outcomes in baseline and follow-up were presented using mean with SD for PANSS scores and EQ-5D, and medians with IQR for CGI-SCH and GAF scores. The change from the baseline to the follow-up was analyzed using the paired t test (PANSS scores and EQ-5D) or the Wilcoxon test (CGI-SCH and GAF scores). Statistical analyses were performed using the SPSS Statistics (version 29.0; IBM Corp).

Results

The study participation flowchart based on the CONSORT (Consolidated Standards of Reporting Trials) 2010 statement recommendations [73] is presented in Figure 1.

Figure 1. m-RESIST study participation flowchart. m-RESIST: mobile therapeutic attention for treatment-resistant schizophrenia.

Background Variables

The sociodemographic characteristics of this study’s population (n=31) are presented in Table 2. No statistically significant differences in sociodemographic characteristics were found at baseline between the participants in the 3 sites. The percentage of men and women was almost equal (n=16, 51.6% and n=15, 48.4%, respectively). Most participants were singles (n=25, 80.6%) and lived with parents or relatives (n=23, 74.2%). Almost half of the study population had a full-time or a half-time job (n=15, 48.4%). The most common education level was primary studies (n=15, 48.4%). Almost two of the third lives with parents or relatives (n=20, 64.5%). Nearly all study participants had a caregiver (n=28, 90.3%; Table 2). Those patients without a caregiver had support in their immediate environment.

Nearly all participants were currently on 1 or 2 antipsychotics (30/31, 96%), and clozapine was the most used drug (11/31, 35.5%). Olanzapine, clozapine, and risperidone were previously the 3 most frequently taken antipsychotics (13/31, 41.9%; 9/31, 29%; and 9/31, 29%, respectively).

Table 2.The sociodemographic variables of this study’s population.

	Values
Age, mean (SD)	33.9 (7.6)
Gender, n (%)
Male	16 (51.6)
Female	15 (48.4)
Marital status, n (%)
Single	25 (80.6)
Married or with a stable partner	4 (12.9)
Divorced or separated	2 (6.5)
Cohabitation, n (%)
Lives alone	4 (12.9)
Lives with parents or relatives	23 (74.2)
Lives with a stable partner	3 (9.7)
Lives with roommates	1 (3.2)
Occupation, n (%)
Full-time job	7 (22.6)
Part-time job	8 (25.8)
Unemployed	9 (29)
Pension	6 (19.4)
Protected job	1 (3.2)
Educational level, n (%)
Primary studies	15 (48.4)
Secondary studies	10 (32.3)
University	6 (19.4)
Lives with parents or relativesa, n (%)
No	8 (25.8)
Yes	20 (64.5)
Caregiver availabilitya, n (%)
No	2 (6.5)
Yes	28 (90.3)

^aMissing n=1.

Clinical Outcomes

Significant reductions were found in symptoms from pretrial to posttrial on the PANSS total (mean 73.7, SD 14.8 versus mean 66.5, SD 12.8; mean difference −7.2, 95% CI −11.1 to −3.4; P=.001), PANSS positive (mean 15.2, SD 2.8 versus 13.8, SD 3.2; mean difference −1.36, 95% CI −2.6 to −0.1; P=.04), PANSS negative (mean 21.2, SD 4.9 versus 19.1, SD 4.3; mean difference −2.1, 95% CI −3.1 to −1.1; P < .001), and PANSS general symptoms (mean 37.3, SD 9.3 versus 33.6, SD 8.4; mean difference 3.8, 95% CI −6.8 to −0.8; P=.02) (Table 3).

In almost one-fifth of the participants (6/31), the overall score for the PANSS decreased by more than 20%.

The sum of global severity of illness also decreased significantly on the CGI-SCH scale (median 4, IQR 3-4 vs median 3, IQR 3-4; median difference 0, IQR −1 to 0; P=.03). A decrease in the sum of positive (median 3, IQR 2.75-4 vs median 3, IQR 2-3.25; median difference 0, IQR −1 to 0; P=.04) and negative (median 4, IQR 3-4 vs median 3, IQR 3-4; median difference 0, IQR −1 to 0; P=.03) symptoms was also found. The sum of depressive or cognitive symptoms did not change (Table 3).

Table 3.Change of symptoms between baseline and follow-up.

		Baseline	Follow-up	Difference	P value
PANSSa,b
	Total score	73.7 (14.8)	66.5 (12.8)	−7.2 (−11.1 to −3.4)	.001
	Positive symptoms score	15.2 (2.8)	13.8 (3.2)	−1.36 (−2.6 to −0.1)	.04
	Negative symptoms score	21.2 (4.9)	19.1 (4.3)	−2.1 (−3.1 to −1.1)	< .001
	Generalized symptoms score	37.3 (9.3)	33.6 (8.4)	−3.8 (−6.8 to −0.8)	.02
CGI-SCHc,d
	Sum of positive symptoms	3 (2.75 to 4)	3 (2 to 3.25)	0 (−1 to 0)	.03
	Sum of negative symptoms	4 (3 to 4)	3 (3 to 4)	0 (−1 to 0)	.03
	Sum of depressive symptoms	3 (2 to 3.25)	3 (2 to 3)	0 (−1 to 0.5)	.22
	Sum of cognitive symptoms	3 (2.75 to 4)	3 (2 to 3)	0 (−1 to 0)	.23
	Sum of global severity	4 (3 to 4)	3 (3 to 4)	0 (−1 to 0)	.03
GAFd,e		55 (45 to 55)	60 (50 to 65)	5 (0 to 10)	< .001
EQ-VASb,f		66.1 (18)	72.8 (16.5)	6.7 (0.5 to 12.9)	.04

^aPANSS: Positive and Negative Syndrome Scale.

^bMean (SD) for the baseline and follow-up, and mean (95% CI) with paired samples t-test for the difference.

^cCGI-SCH: Clinical Global Impression-Schizophrenia.

^dMedian (IQR) for the baseline and follow-up, and median (IQR) with Wilcoxon test for the difference.

^eGAF: Global Assessment of Functioning.

^fEQ-VAS: EuroQol visual analogue scale.

Functional and Perceived Quality of Life Outcomes

The functionality of participants increased significantly from pretrial to posttrial on the GAF scale (median 55, IQR 45-55 vs median 60, IQR 50-65; median difference 5, IQR 0-10; P < .001), but it was not considered clinically significant due to participants remaining in the same interval (moderate symptoms).

The perceived quality of life evaluated by the EQ-VAS also improved (mean 66.1, SD 18 vs mean 72.8, SD 16.5; difference 6.7, 95% CI 0.5-12.9; P=.04) (Table 3).

Adverse Events

The symptoms of TRS got worse in 1 participant before the intervention and in 2 participants during it, which was considered not to be related to the m-RESIST solution.

Discussion

Principal Results

To our best knowledge, this was the first study collecting data on the symptoms, psychosocial functioning, and quality of life for people with TRS by using the mHealth platform (m-RESIST solution). Our novel preliminary findings showed that the total, positive, negative, and general symptoms according to the PANSS were significantly decreased after intervention compared with the starting point. In almost one-fifth of the participants (6/31), the overall score for the PANSS decreased by more than 20%, which may be considered a clinically significant change [74].

According to Suzuki et al [75], the partial response of clinical trials in schizophrenia could be defined ≥10% to < 20% decrease on the PANSS. In our study, the change in positive score was 8.9%. Interestingly, the change in the negative score was 10.2%, which may mean a partial response according to Suzuki`s definition. The decrease in the generalized symptoms scores and the total score was 10.2% versus 9.8%. Therefore, the mean differences of 1‐2 points on the positive and negative subscale scores may be indicatively clinically meaningful to a small extent at the most. Nevertheless, one should be careful in interpretations regarding the significance of the results. The clinical implications of PANSS scores may not always be clear [76].

The sum of global severity, and the sum of positive and negative symptoms on the CGI-SCH scale were lower at the end of the study compared to the starting point, but the sum of depressive or cognitive symptoms did not change. The functionality measured by the GAF scale was also significantly better at the end of this study as compared to its level at the beginning. The perceived quality of life also increased during this study.

Comparisons With Prior Work

The finding of a significant decrease in all PANSS scores and global severity, positive and negative symptoms on the CGI-SCH scale at the study end compared to the starting point is partly in line with former data, as previous studies have shown efficacy of mobile interventions for schizophrenia [6,15,27,36,43,56,61,62]. To the best of our knowledge, this is the first study to show the efficacy of an mHealth platform (m-RESIST solution) in TRS.

A study by Ben-Zeev et al [6] demonstrated the feasibility, acceptability, and efficacy of a smartphone intervention for schizophrenia, followed by those with schizophrenia or schizoaffective disorder (N=33) for 1 month without a control group. Its results showed a significant decrease in the total, positive, and general scores but not in the negative score in the PANSS. A noncontrolled study by Kidd et al [25] with patients with primary psychosis (N=38) reported a decrease in certain items of the Brief Symptom Inventory (eg, psychotism, paranoid ideation, and depression) when followed for 1 month.

Bucci et al [27] carried out an RCT among those with early psychosis (N=36). The results showed a decrease in the total, negative, and general scores of the PANSS after a follow-up of 12 weeks.

WeChat is the most widely and extensively used mobile social networking app in China. A cross-sectional study on WeChat use and endorsement of WeChat-based mHealth among people living with schizophrenia (n=400) in China was carried out. Compared to nonusers in this study’s population, WeChat use was associated with improved health outcomes, including lower psychiatric symptoms (the Brief Psychiatric Rating Scale), lower depression (the Patient Health Questionnaire-9), higher functioning (GAF), better recovery (the Recovery Assessment Scale), and higher quality of life (the first 2 general questions from the World Health Organization Quality of Life Brief Scale) [62].

However, another RCT with participants with recent-onset schizophrenia spectrum disorders (N=43) showed no changes in their positive or negative score of the PANSS or their functioning measured by the Role Functioning Scale. The quality of life measured by the interview-based Quality of Life Scale Abbreviated did not improve [36]. A recent study with patients with a diagnosis of schizophrenia (N=361), randomized to receive either SlowMo, a blended digital therapy targeting reasoning, or usual care, found also no significant difference between groups on the primary outcomes related to paranoia (the Green et al [77] Paranoid Thoughts Scale) at the primary end point at 24 weeks [33].

Decrease in Symptoms

It should be taken into account that the decrease in symptoms in patients with TRS, characterized by the persistent presence of symptoms, would not be expected in such a short term. Following the considerations of this study by Ben-Zeev et al [6], the improvement in the PANSS scores obtained in this study of the m-RESIST solution should be evaluated prudently. In the case of patients with schizophrenia, in the framework of a study, a placebo effect of the offered intervention has been observed. One factor that would influence the improvement of symptoms would be all the care received for participating in a study. It could be considered that participation in a structured intervention program would promote a state of security, such that it would improve aspects such as the symptoms of the disorder itself [78,79].

However, the overall score for the PANSS decreased by more than 20% in almost one-fifth of the participants which may be considered a clinically significant change [74]. Furthermore, the negative symptoms on the PANSS decreased during a rather short period of 3 months. It may imply that somehow the m-RESIST solution may manage to activate patients.

Perceived Quality of Life and Functioning

The perceived quality of life and functioning improved in our study, which is partly in line with the earlier studies, as a study by Schlosser et al [36] did not find positive effects on functioning and quality of life. Instead, the findings of Yu et al [62] showed improvement both in functioning and quality of life. Our findings on the improvement of quality of life may need to be treated with caution as the findings of the EQ-5D-5L scale have shown weak correlations with the objective measures among patients with schizophrenia [80].

Study Timeline

The data collection period was in 2017. Although progress has been made, there is still room for improvement in mHealth apps for people with schizophrenia, with fewer than 10 apps specifically designed for this population available for public download [43,59]. The m-RESIST solution is a new, innovative mHealth tool developed for patients with TRS and their caregivers, offering modular and flexible functions. This study aimed to evaluate the effects of m-RESIST on clinical outcomes, functional outcomes, and perceived quality of life in individuals with TRS. To the best of our knowledge no previous studies have addressed these specific issues.

For the above reasons, our research results provide significant new information on the use of mobile applications in the treatment of TRS.

Strengths

This study has several strengths. To our best knowledge, this was the first study collecting data on the symptoms, functioning, and perceived quality of life among patients with TRS by an mHealth intervention. This was also a multicenter study covering different cultural environments. The m-RESIST solution is an innovative mHealth solution based on novel technology, offering highly modular and flexible functioning and planned intervention. It has been developed specifically for patients with TRS [63-65]. The outcomes of this study will also help to perform a cost-effectiveness RCT in the future.

Limitations

Some limitations of this study need to be discussed. First, there was no control group, so it is not possible to determine whether the clinical improvements were related to the use of the m-RESIST solution. Second, this study’s design is a limitation, as, due to the small sample size and the short length of the intervention and follow-up period, the differences found in outcomes should be treated with caution. Therefore, the generalization and maintenance of the results are uncertain. Third, the so-called Hawthorne effect may also have implications for the generalizability of results because the behavior of research participants is influenced by observation and measurement. In general, results in experimental settings may be more positive compared to routine clinical applications [78,79].

The moderately high mean baseline score for the PANSS (73.7) demonstrates that our patients with TRS have quite severe psychotic symptoms. Even so, they responded to intervention markedly.

Potential Selection Bias Regarding Digital Literacy

Digital literacy can be defined as the varying ability of both children and adults to use digital technologies and understand their risks. Patients who do not know how to use digital health tools, do not see the importance of those tools, or cannot access them in their preferred language, ultimately will not use them [81]. This may cause potential selection bias because, for the reasons above, those people may not participate in the studies [81].

Technical Implementation Challenges

The digital divide remains a barrier, with people with schizophrenia eager and able to obtain smartphones and learn new skills, but resources and training are often not provided. Expanding train-the-trainer and literacy programs such as the Digital Health Navigator initiative, which was specially designed to support people with psychosis, offers 1 tangible next step [42,43,53] However, a lagging infrastructure and skill base have limited the application of digital solutions in mental health care [82]. Additionally, evidence is needed on the effectiveness and appropriateness of digital health tools for patients who are marginalized and may lack access to digital health interventions [83].

The digital divide remains a barrier, with people with schizophrenia eager and able to obtain smartphones and learn new skills, but resources and training are often not provided [51]. Expanding train-the-trainer and literacy programs such as the Digital Health Navigator initiative, which was specially designed to support people with psychosis, offers 1 tangible next step [53,84].

Cost Implications for Real-World Implementation

Despite modest effect sizes, the relatively low cost and high scalability of most of the interventions tested support their public health relevance [85]. A recent review on this topic found only 37 relevant studies published in middle- and lower-income countries between 2016 and 2020, with the majority reporting feasibility and accessibility outcomes instead of efficacy, cost-effectiveness, or implementation [86]. However, the costs of implications for real-world implementation may be a concern, and that is why it has been asked if mHealth tools can decrease hospital stays and encourage patients to use mobile apps in favor of reducing costs [61]. Digital health tools still hold a unique potential to extend access to care in middle- and lower-income countries, where there is less investment or infrastructure around mental health [86]. Smartphone use is common and rapidly expanding in those countries, and thus represents a promising tool to reduce the mental health gap [42].

Conclusions

Our preliminary findings showed that implementing the m-RESIST solution may decrease the symptoms and severity of illness and improve the functionality and perceived quality of life among people with TRS. The change of symptoms on the PANSS scores may be indicatively clinically meaningful to a small extent at the most. Nevertheless, one should be careful in interpretations regarding the significance of the results.

Implementing the m-RESIST solution may be a promising tool to improve the treatment even in TRS, and to offer a new tool for its present treatment options, already in the near future. If successful, this would be a very important step forward as TRS is a major clinical challenge, with usually poor outcomes. Before being implemented in daily practice, these results should be corroborated by performing controlled clinical trials with people with TRS, comparing treatment as usual with the m-RESIST solution.

Acknowledgments

This work has been supported by the Horizon 2020 Framework Programme of the European Union (grant 643552) and partly funded by the CERCA (Centres de Recerca de Catalunya) Programme/Generalitat de Catalunya. We thank all the patients who participated in this study for their essential contribution to our field of research. We are also thankful to Ona Corporation (IIBSP-AMA2016-99). We are grateful to all the members of the mobile therapeutic attention for treatment-resistant schizophrenia (m-RESIST) project, who are also the authors of this paper under the name of the m-RESIST Group.

The m-RESIST Group is composed of the following members: Francisco Alcalde, Caritat Almazán, Anna Alonso-Solís, Jesús Berdún, István Bitter, Walter Baccinelli, Maria Bulgheroni, Caro Mendivelso, Asaf Caspi, Tanguy Coenen, Xavier Constant, Iluminada Corripio, Enrico d’Amico, Ilaria De Vita, Marisol Escobar, Kinga Farkas, Kata Fazekas, Shenja van der Graaf, Eva Grasa, Margarita Hospedales, Elena Huerta-Ramos, Matti Isohanni, Erika Jääskeläinen, Charlotte Jewel, Teija Juola, Timo Jämsä, Rachelle Kaye, Panagiotis Kokkinakis, Hannu J Koponen, Silvia Marcó, Gregoris Mentzas, Jouko Miettunen, Jani Moilanen, Susana Ochoa, Ilias Papas, Fotis Paraskevopoulos, Elisabeth Reixach, Katya Rubinstein, Elena Rubio-Abadal,Garifalia Sebú, Annika Seppälä, Jussi Seppälä, Valentina Simonetti, Matthias Stevens, Anna Triantafillou, Zsolt Szabolcs Unoka, Judith Usall, Vincenzo Vella, and David Vermeir.

Data Availability

The datasets generated and analyzed during this study are available from the corresponding author on reasonable request.

Authors' Contributions

ARB, AAS, EG, IC, JM, CA, ER, JB, JU, SO, SA, JCM, MH, JS, and EJ planned the investigation. MH was responsible for statistical analysis. KR, AC, ZU, KF, EG, and IC were responsible for carrying out this clinical study. JS, EG, IC, EJ, MI, MH, and JM wrote this paper. All authors revised and edited this paper critically and had final approval of the version to be published.

Conflicts of Interest

None declared.

------------

Abbreviations

CGI-SCH: Clinical Global Impression-Schizophrenia

CONSORT: Consolidated Standards of Reporting Trials

EQ-VAS: EuroQol visual analogue scale

GAF: Global Assessment of Functioning

ICT: information and communication technology

m-RESIST: mobile therapeutic attention for treatment-resistant schizophrenia

mHealth: mobile health

PANSS: Positive and Negative Syndrome Scale

RCT: randomized controlled trial

TRS: treatment-resistant schizophrenia

------------

Edited by Andre Kushniruk; submitted 17.10.24; peer-reviewed by Jimmy Lee, Pierfelice Cutrufelli, Vjekoslav Peitl; final revised version received 26.03.25; accepted 27.03.25; published 15.05.25.

References

Leucht S, Busch R, Kissling W, Kane JM. Early prediction of antipsychotic nonresponse among patients with schizophrenia. J Clin Psychiatry. Mar 2007;68(3):352-360.

Jääskeläinen E, Juola P, Hirvonen N, et al. A systematic review and meta-analysis of recovery in schizophrenia. Schizophr Bull. Nov 2013;39(6):1296-1306.

Kennedy JL, Altar CA, Taylor DL, Degtiar I, Hornberger JC. The social and economic burden of treatment-resistant schizophrenia: a systematic literature review. Int Clin Psychopharmacol. Mar 2014;29(2):63-76.

Mortimer AM, Singh P, Shepherd CJ, Puthiryackal J. Clozapine for treatment-resistant schizophrenia: National Institute of Clinical Excellence (NICE) guidance in the real world. Clin Schizophr Relat Psychoses. Apr 2010;4(1):49-55.

Millier A, Schmidt U, Angermeyer MC, et al. Humanistic burden in schizophrenia: a literature review. J Psychiatr Res. Jul 2014;54:85-93.

Ben-Zeev D, Brenner CJ, Begale M, Duffecy J, Mohr DC, Mueser KT. Feasibility, acceptability, and preliminary efficacy of a smartphone intervention for schizophrenia. Schizophr Bull. Nov 2014;40(6):1244-1253.

Granholm E, Ben-Zeev D, Link PC, Bradshaw KR, Holden JL. Mobile Assessment and Treatment for Schizophrenia (MATS): a pilot trial of an interactive text-messaging intervention for medication adherence, socialization, and auditory hallucinations. Schizophr Bull. May 2012;38(3):414-425.

Palmier-Claus JE, Ainsworth J, Machin M, et al. The feasibility and validity of ambulatory self-report of psychotic symptoms using a smartphone software application. BMC Psychiatry. Oct 17, 2012;12:172.

Firth J, Cotter J, Torous J, Bucci S, Firth JA, Yung AR. Mobile phone ownership and endorsement of “mHealth” among people with psychosis: a meta-analysis of cross-sectional studies. Schizophr Bull. Mar 2016;42(2):448-455.

Brunette MF, Achtyes E, Pratt S, et al. Use of smartphones, computers and social media among people with SMI: opportunity for intervention. Community Ment Health J. Aug 2019;55(6):973-978.

Young AS, Cohen AN, Niv N, et al. Mobile phone and smartphone use by people with serious mental illness. Psychiatr Serv. Mar 1, 2020;71(3):280-283.

Firth J, Torous J. Smartphone apps for schizophrenia: a systematic review. JMIR mHealth uHealth. Nov 6, 2015;3(4):e102.

Bonet L, Izquierdo C, Escartí MJ, et al. Use of mobile technologies in patients with psychosis: a systematic review. Rev Psiquiatr Salud Ment (Engl Ed). Jul 2017;10(3):168-178.

Buck B, Scherer E, Brian R, et al. Relationships between smartphone social behavior and relapse in schizophrenia: a preliminary report. Schizophr Res. Jun 2019;208:167-172.

Clarke S, Hanna D, Mulholland C, Shannon C, Urquhart C. A systematic review and meta-analysis of digital health technologies effects on psychotic symptoms in adults with psychosis. Psychosis. Oct 2, 2019;11(4):362-373.

de Almeida RS, Marques A. User engagement in mobile apps for people with schizophrenia: a scoping review. Front Digit Health. 2022;4:1023592.

Achtyes ED, Ben-Zeev D, Luo Z, et al. Off-hours use of a smartphone intervention to extend support for individuals with schizophrenia spectrum disorders recently discharged from a psychiatric hospital. Schizophr Res. Apr 2019;206:200-208.

Ben-Zeev D, Brian RM, Jonathan G, et al. Mobile health (mHealth) versus clinic-based group intervention for people with serious mental illness: a randomized controlled trial. Psychiatr Serv. Sep 1, 2018;69(9):978-985.

Ben-Zeev D, Brian RM, Aschbrenner KA, Jonathan G, Steingard S. Video-based mobile health interventions for people with schizophrenia: bringing the “pocket therapist” to life. Psychiatr Rehabil J. Mar 2018;41(1):39-45.

Hanssen E, Balvert S, Oorschot M, et al. An ecological momentary intervention incorporating personalised feedback to improve symptoms and social functioning in schizophrenia spectrum disorders. Psychiatry Res. Feb 2020;284:112695.

Moitra E, Park HS, Gaudiano BA. Development and initial testing of an mHealth transitions of care intervention for adults with schizophrenia-spectrum disorders immediately following a psychiatric hospitalization. Psychiatr Q. Mar 2021;92(1):259-272.

Almeida RFSD, Sousa TJ, Couto AS, Marques AJ, Queirós CM, Martins CL. Development of weCope, a mobile app for illness self-management in schizophrenia. Arch Clin Psychiatry (São Paulo). 2019;46(1):1-4.

Steare T, O’Hanlon P, Eskinazi M, et al. Smartphone-delivered self-management for first-episode psychosis: the ARIES feasibility randomised controlled trial. BMJ Open. Aug 2020;10(8):e034927.

Steare T, Giorgalli M, Free K, et al. A qualitative study of stakeholder views on the use of a digital app for supported self-management in early intervention services for psychosis. BMC Psychiatry. Jun 19, 2021;21(1):311.

Kidd SA, Feldcamp L, Adler A, et al. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: App4Independence (A4i). PLoS ONE. 2019;14(7):e0219491.

Ben-Zeev D, Brian R, Wang R, et al. CrossCheck: integrating self-report, behavioral sensing, and smartphone use to identify digital indicators of psychotic relapse. Psychiatr Rehabil J. Sep 2017;40(3):266-275.

Bucci S, Barrowclough C, Ainsworth J, et al. Actissist: proof-of-concept trial of a theory-driven digital intervention for psychosis. Schizophr Bull. Aug 20, 2018;44(5):1070-1080.

Lewis S, Ainsworth J, Sanders C, et al. Smartphone-enhanced symptom management in psychosis: open, randomized controlled trial. J Med Internet Res. Aug 13, 2020;22(8):e17019.

Eisner E, Bucci S, Berry N, Emsley R, Barrowclough C, Drake RJ. Feasibility of using a smartphone app to assess early signs, basic symptoms and psychotic symptoms over six months: a preliminary report. Schizophr Res. Jun 2019;208:105-113.

Niendam TA, Tully LM, Iosif AM, et al. Enhancing early psychosis treatment using smartphone technology: a longitudinal feasibility and validity study. J Psychiatr Res. Jan 2018;96:239-246.

Bonet L, Torous J, Arce D, Blanquer I, Sanjuan J. ReMindCare app for early psychosis: pragmatic real world intervention and usability study. JMIR mHealth uHealth. Nov 6, 2020;8(11):e22997.

Terp M, Jørgensen R, Laursen BS, Mainz J, Bjørnes CD. A smartphone app to foster power in the everyday management of living with schizophrenia: qualitative analysis of young adults’ perspectives. JMIR Ment Health. Oct 1, 2018;5(4):e10157.

Garety P, Ward T, Emsley R, et al. Effects of SlowMo, a blended digital therapy targeting reasoning, on paranoia among people with psychosis: a randomized clinical trial. JAMA Psychiatry. Jul 1, 2021;78(7):714-725.

Gire N, Caton N, McKeown M, et al. ’Care co-ordinator in my pocket’: a feasibility study of mobile assessment and therapy for psychosis (TechCare). BMJ Open. Nov 16, 2021;11(11):e046755.

Moore E, Williams A, Bell I, Thomas N. Client experiences of blending a coping-focused therapy for auditory verbal hallucinations with smartphone-based ecological momentary assessment and intervention. Internet Interv. Mar 2020;19:100299.

Schlosser DA, Campellone TR, Truong B, et al. Efficacy of PRIME, a mobile app intervention designed to improve motivation in young people with schizophrenia. Schizophr Bull. Aug 20, 2018;44(5):1010-1020.

Krzystanek M, Borkowski M, Skałacka K, Krysta K. A telemedicine platform to improve clinical parameters in paranoid schizophrenia patients: results of a one-year randomized study. Schizophr Res. Feb 2019;204:389-396.

Fulford D, Mote J, Gard DE, et al. Development of the Motivation and Skills Support (MASS) social goal attainment smartphone app for (and with) people with schizophrenia. J Behav Cogn Ther. Apr 2020;30(1):23-32.

Taylor KM, Bradley J, Cella M. A novel smartphone-based intervention targeting sleep difficulties in individuals experiencing psychosis: a feasibility and acceptability evaluation. Psychol Psychother. Sep 2022;95(3):717-737.

Kreyenbuhl J, Record EJ, Himelhoch S, et al. Development and feasibility testing of a smartphone intervention to improve adherence to antipsychotic medications. Clin Schizophr Relat Psychoses. 2019;12(4):152-167.

Lim MH, Gleeson JFM, Rodebaugh TL, et al. A pilot digital intervention targeting loneliness in young people with psychosis. Soc Psychiatry Psychiatr Epidemiol. Jul 2020;55(7):877-889.

Torous J, Bucci S, Bell IH, et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. Oct 2021;20(3):318-335.

Torous J, Smith KA, Hardy A, et al. Digital health interventions for schizophrenia: setting standards for mental health. Schizophr Res. May 2024;267:392-395.

Vitger T, Korsbek L, Austin SF, Petersen L, Nordentoft M, Hjorthøj C. Digital shared decision-making interventions in mental healthcare: a systematic review and meta-analysis. Front Psychiatry. 2021;12(691):691251.

MacInnes D, Kinane C, Parrott J, et al. A pilot cluster randomised trial to assess the effect of a structured communication approach on quality of life in secure mental health settings: the Comquol study. BMC Psychiatry. Sep 29, 2016;16(1):335.

Priebe S, Kelley L, Golden E, et al. Effectiveness of structured patient-clinician communication with a solution focused approach (DIALOG+) in community treatment of patients with psychosis--a cluster randomised controlled trial. BMC Psychiatry. Jun 26, 2013;13(1):173.

Edbrooke-Childs J, Edridge C, Averill P, et al. A feasibility trial of power up: smartphone app to support patient activation and shared decision making for mental health in young people. JMIR mHealth uHealth. Jun 4, 2019;7(6):e11677.

Loh PY, Martinengo L, Heaukulani C, et al. Characteristics and outcomes of mHealth interventions in psychosis: systematic mapping review. J Med Internet Res. Dec 23, 2024;26:e55924.

Lim CT, Fuchs C, Torous J. Integrated digital mental health care: a vision for addressing population mental health needs. Int J Gen Med. 2024;17:359-365.

Mueller NE, Panch T, Macias C, Cohen BM, Ongur D, Baker JT. Using smartphone apps to promote psychiatric rehabilitation in a peer-led community support program: pilot study. JMIR Ment Health. Aug 15, 2018;5(3):e10092.

Smith KA, Blease C, Faurholt-Jepsen M, et al. Digital mental health: challenges and next steps. BMJ Ment Health. Feb 2023;26(1):e300670.

Camacho E, Torous J. Introducing an implementation framework for augmenting care with digital technology for early psychosis patients: theory and motivation. J Ment Health. Dec 2022;31(6):816-824.

Green JB, Rodriguez J, Keshavan M, Lizano P, Torous J. Implementing technologies to enhance coordinated specialty care framework: implementation outcomes from a development and usability study. JMIR Form Res. Oct 3, 2023;7:e46491.

Camacho E, Torous J. Impact of digital literacy training on outcomes for people with serious mental illness in community and inpatient settings. Psychiatr Serv. May 1, 2023;74(5):534-538.

Buck B, Kopelovich SL, Tauscher JS, Chwastiak L, Ben-Zeev D. Developing the workforce of the digital future: leveraging technology to train community-based mobile mental health specialists. J Technol Behav Sci. Aug 3, 2022:1-7.

Miralles I, Granell C, Díaz-Sanahuja L, et al. Smartphone apps for the treatment of mental disorders: systematic review. JMIR mHealth uHealth. Apr 2, 2020;8(4):e14897.

Agarwal S, Jalan M, Wilcox H, Sharma R, Hill R, Pantalone E, et al. Evalution of mental health mobile applications. Agency for Healthcare Research and Quality (US); 2022. URL: https:/​/effectivehealthcare.​ahrq.gov/​sites/​default/​files/​product/​pdf/​mental-health-mobile-apps-tech-brief.​pdf [Accessed 2025-04-24]

Radu I, Scheermesser M, Spiess MR, Schulze C, Händler-Schuster D, Pehlke-Milde J. Digital health for migrants, ethnic and cultural minorities and the role of participatory development: a scoping review. Int J Environ Res Public Health. Oct 23, 2023;20(20):6962.

Kwon S, Firth J, Joshi D, Torous J. Accessibility and availability of smartphone apps for schizophrenia. Schizophrenia (Heidelb). Nov 16, 2022;8(1):98.

Seppälä J, De Vita I, Jämsä T, et al. Mobile phone and wearable sensor-based mHealth approaches for psychiatric disorders and symptoms: systematic review. JMIR Ment Health. Feb 20, 2019;6(2):e9819.

Chivilgina O, Wangmo T, Elger BS, Heinrich T, Jotterand F. mHealth for schizophrenia spectrum disorders management: a systematic review. Int J Soc Psychiatry. Nov 2020;66(7):642-665.

Yu Y, Li Y, Li T, et al. New path to recovery and well-being: cross-sectional study on WeChat use and endorsement of WeChat-based mHealth among people living with schizophrenia in China. J Med Internet Res. Sep 18, 2020;22(9):e18663.

Huerta-Ramos E, Escobar-Villegas MS, Rubinstein K, et al. Measuring users’ receptivity toward an integral intervention model based on mhealth solutions for patients with treatment-resistant schizophrenia (m-RESIST): a qualitative study. JMIR mHealth uHealth. Sep 28, 2016;4(3):e112.

Alonso-Solís A, Rubinstein K, Corripio I, et al. Mobile therapeutic attention for treatment-resistant schizophrenia (m-RESIST): a prospective multicentre feasibility study protocol in patients and their caregivers. BMJ Open. Jul 16, 2018;8(7):e021346.

Grasa E, Seppälä J, Alonso-Solis A, et al. m-RESIST, a mobile therapeutic intervention for treatment-resistant schizophrenia: feasibility, acceptability, and usability study. JMIR Form Res. Jun 30, 2023;7:e46179.

Elkis H, Meltzer HY. Refractory schizophrenia. Braz J Psychiatry. Oct 2007;29 Suppl 2:S41-S47.

Peuskens J. The evolving definition of treatment resistance. J Clin Psychiatry. 1999;60 Suppl 12:4-8.

Andreasen NC, Carpenter WT, Kane JM, Lasser RA, Marder SR, Weinberger DR. Remission in schizophrenia: proposed criteria and rationale for consensus. AJP. Mar 1, 2005;162(3):441-449.

Kay SR, Fiszbein A, Opler LA. The Positive and Negative Syndrome Scale (PANSS) for schizophrenia. Schizophr Bull. 1987;13(2):261-276.

Haro JM, Kamath SA, Ochoa S, et al. The Clinical Global Impression–Schizophrenia scale: a simple instrument to measure the diversity of symptoms present in schizophrenia. Acta Psychiatr Scand. May 2003;107(s416):16-23.

Endicott J, Spitzer RL, Fleiss JL, Cohen J. The global assessment scale. A procedure for measuring overall severity of psychiatric disturbance. Arch Gen Psychiatry. Jun 1976;33(6):766-771.

Brooks R. EuroQol: the current state of play. Health Policy. Jul 1996;37(1):53-72.

Eldridge SM, Chan CL, Campbell MJ, et al. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. Oct 24, 2016;355:i5239.

Hermes EDA, Sokoloff D, Stroup TS, Rosenheck RA. Minimum clinically important difference in the Positive and Negative Syndrome Scale with data from the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE). J Clin Psychiatry. Apr 2012;73(4):526-532.

Suzuki T, Remington G, Mulsant BH, et al. Defining treatment-resistant schizophrenia and response to antipsychotics: a review and recommendation. Psychiatry Res. May 15, 2012;197(1-2):1-6.

Leucht S, Kane JM, Kissling W, Hamann J, Etschel E, Engel RR. What does the PANSS mean? Schizophr Res. Nov 15, 2005;79(2-3):231-238.

Green CEL, Freeman D, Kuipers E, et al. Measuring ideas of persecution and social reference: the Green et al. Paranoid Thought Scales (GPTS). Psychol Med. Jan 2008;38(1):101-111.

Sedgwick P, Greenwood N. Understanding the Hawthorne effect. BMJ. Sep 2015;4:h4672.

McCambridge J, Witton J, Elbourne DR. Systematic review of the Hawthorne effect: new concepts are needed to study research participation effects. J Clin Epidemiol. Mar 2014;67(3):267-277.

König HH, Roick C, Angermeyer MC. Validity of the EQ-5D in assessing and valuing health status in patients with schizophrenic, schizotypal or delusional disorders. Eur Psychiatry. Apr 2007;22(3):177-187.

López MDPA, Ong BA, Frigola XB, et al. Digital literacy as a new determinant of health: a scoping review. PLOS Digit Health. Oct 2023;2(10):e0000279.

Balcombe L, De Leo D. Digital mental health challenges and the horizon ahead for solutions. JMIR Ment Health. Mar 29, 2021;8(3):e26811.

Philippe TJ, Sikder N, Jackson A, et al. Digital health interventions for delivery of mental health care: systematic and comprehensive meta-review. JMIR Ment Health. May 12, 2022;9(5):e35159.

Kozelka EE, Acquilano SC, Al-Abdulmunem M, et al. Digital mental health and its discontents: assumptions about technology that create barriers to equitable access. PS (Wash DC). Mar 1, 2024;75(3):299-302.

Goldberg SB, Lam SU, Simonsson O, Torous J, Sun S. Mobile phone-based interventions for mental health: a systematic meta-review of 14 meta-analyses of randomized controlled trials. PLOS Digit Health. 2022;1(1):e0000002.

Carter H, Araya R, Anjur K, Deng D, Naslund JA. The emergence of digital mental health in low-income and middle-income countries: a review of recent advances and implications for the treatment and prevention of mental disorders. J Psychiatr Res. Jan 2021;133:223-246.