1. Introduction
Electronic health (eHealth) solutions have demonstrated their potential to provide more personalised “citizen-centric” healthcare that is targeted, effective and efficient, facilitating access to care and improving patients’ quality of life [1]. eHealth refers to those healthcare services provided with the support of information and communication technologies (ICT). As part of eHealth, mobile health (mHealth) solutions, which imply the use of portable devices (smartphones, tablets and, even, wearables like smartwatches), make it possible to monitor and intervene remotely whenever and wherever acute and chronic medical conditions occur, as mHealth overcomes many of the traditional barriers of distance, time and cost, offering convenience and opportunities for care in nonclinical environments [2,3,4]. Based on this clinical evidence, organizations and associations globally have been developing recommendations and policy initiatives aimed at fostering and facilitating the uptake and wider development of eHealth and mHealth interventions [1,5,6,7]. As a result, the European mHealth market is valued at 5.40 billion euros in 2020 and is expected to reach 23.58 billion euros by 2025 at a Compound Annual Growth Rate of 34.29% during the forecast period 2020–2025 [8]. However, retention rates are still low for medical apps: 20% on day 1, 7% on day 7 and 3.5% on day 30, as are those related to health and fitness: 20.2% on day 1, 8.5% on day 7 and 4% on day 30 [9]. Clinical evidence, clinician’s proactivity, data privacy and the integration into provider healthcare systems are fundamental to more fully recognizing the value of mHealth apps and, therefore, to increasing user adoption [10,11,12], which may lead to improvement in patient self-management and retention in care [13]. Effective self-management is a central component of chronic disease control to optimize health outcomes. Evidence has been documented that successful self-management improves the self-reported health status and health-related quality of life while decreasing healthcare utilization and costs [14]. Retention in care is key in the maintenance of individuals’ health, especially in those clinical conditions that require long-term therapy, usually with multiple drugs, which have potential side effects and interactions that may influence patients’ adherence to treatment, like tuberculosis or HIV infection [15].
HIV now has a normal life expectancy in individuals who have access to testing, treatment and care, as in the European Union, where it is estimated that 2.5 million of people are living with HIV [16,17]. People living with medically stable HIV are currently seen by an HIV clinician every three–six months in accordance with the European guidelines [18]. The majority of individuals have an undetectable HIV viral load on antiretroviral treatment, and many are now seeking to reduce the impact of HIV on their lives—for example, those related to drug side effects, psychosocial issues, stigma or numerous hospital visits.
Recent interventions have demonstrated that mHealth can have a positive impact on clinical outcomes, as well as improvements in retention in the care for key populations [19] and newly diagnosed HIV-positive people aged less than 30 years [20,21], who have been historically difficult to reach with traditional interventions. The findings from systematic reviews on eHealth and mHealth interventions for HIV show that there has been an increase in the number of self-management mHealth interventions evaluated with people living with HIV (PLWH) and that these interventions have been shown to be effective, feasible and acceptable [22]. However, only a small proportion of the interventions identified were developed using a theoretical framework, and the majority of them offer a small number of limited functions, with not enough user involvement in the design and development phases, which may affect user acceptance. In addition, the reviews identified gaps around the linkage to care, retention in care and initiation of antiretroviral therapy [23]. In one paper [20], the authors reflected on the significant technical and operational challenges of implementing a mobile phone-based intervention in urban South Africa, emphasizing the need to investigate further for mHealth approaches to be effective. Similarly, investigators [22] have identified the need to develop, implement and evaluate more comprehensive mHealth interventions in order to address the self-management needs of PLWH. Specifically, the authors stated the need to focus on those who are stable during treatment, an under-researched and promising area in which previous experiences have demonstrated that web-based telemedicine approaches are feasible and safe tools for the management of stable HIV [24,25].
Within the EmERGE project [26,27,28], we have codesigned, implemented and evaluated a novel mHealth pathway of care for people living with medically stable HIV in five diverse clinic settings. The EmERGE mHealth platform integrates with hospital information systems to provide patients with secure access to their own data and to offer an alternative to face-to-face appointments as part of a person-centered package of care. It also allows clinics to manage their capacity, creating more time for clinicians to focus on those with more complex needs. We report here on the feasibility, usability and acceptability of introducing the EmERGE platform in a digital health pathway of care.
2. Materials and Methods
2.1. Technology Used
The platform consists of a web application that is installed in the clinical facilities and a mobile application, available on Apple and Google Stores. Apart from these two main components, two interfaces were developed for exchanging and managing data. The technologies involved in the development of the EmERGE platform were Ruby 2.3.3p222 (2016-11-21 revision 56859) (x86_64-linux) [29] and Ruby on Rails 4.2.10 [30] for both the web application and the two interfaces and Ionic 6.5.0 for the mobile application [31]. Ruby on Rails is an open-source, mature and well-maintained server-side web application framework written in Ruby. It is a Model–View–Controller framework, providing default structures for databases, web services and web pages. It encourages and facilitates the use of web standards such as JSON or XML for data transfer and HTML, CSS and JavaScript for display and user interfacing. The mobile application was built using a hybrid approach provided by the Ionic Framework, a complete open-source mobile UI toolkit for developing high-quality cross-platform apps for native iOS, Android and the web, using web technologies. Ionic is based in Apache Cordova, an open-source mobile development framework, which allows Ionic to build and deploy as a native app. The core of Cordova applications uses CSS3 and HTML5 for their presentation and JavaScript for the logic. HTML5 provides access to the underlying hardware of the mobile phone. For Android, Ionic supports Android 4.1 and up. For iOS, Ionic supports iOS 7 and up. Heroku is a cloud platform as a service supporting several programming languages. It lets the developer build, run and scale applications. Heroku [32] has been used to host several components of the EmERGE platform and the Heroku Command Line Interface 7.47.6 linux-x64 node-v12.16.2 to create and manage the Heroku apps directly from the terminal.
2.2. EmERGE Platform Development Methodology
The EmERGE platform was developed using Scrum, an agile methodology based on an iterative and incremental approach, delivering the software on a regular basis for feedback. Agile development is recommended for small-to-medium-sized projects [33], and many studies consider the agile method as a natural fit for the mobile app industry, although it is recommended to tailor the development process to better suit mobile apps’ peculiarities [34]. Numerous agile approaches for mobile software development have been proposed. In the literature, Extreme Programming and Scrum are the most common agile methods for mobile application development [33].
The development process followed produced incremental versions of the whole platform considering the feedback coming from the codesign process implemented by the EmERGE Consortium. The codesign process consisted of 3 phases of codesign workshops and semi-structured interviews [35,36]. In phase 1, 14 workshops and 22 interviews were conducted involving 97 PLWH and 63 clinicians. In this phase, participants were asked to reflect on the potentiality and risks associated with mHealth in the context of HIV treatment and care and on the functionalities that could support the self-management of HIV. Phases 2 and 3 aimed at capturing patients’ and clinicians’ experiences in the use of the platform to support continuous improvement. Thirteen workshops and 47 interviews were conducted during phases 2 and 3 involving 87 PLWH and 32 clinicians.
Different deployment environments were created to deal with the iterative development process: development, User Acceptance Testing (UAT) and production environments. A UAT allows final users to test the system and verify that the requirements and specifications are met. Finally, a ticketing system was used to better manage the support and maintenance activities during the formal evaluation and, therefore, guarantee the feasibility of the project.
2.3. EmERGE Platform Evaluation Methodology
A large prospective cohort study, undertaken in five European sites, was designed and performed to assess the impact of the mHealth supported pathway of care using a tailored Health Technology Assessment (HTA) process: the Model for Assessment of Telemedicine Applications (MAST) [37]. MAST summarizes and evaluates information about the medical, social, economic and ethical issues related to the use of telemedicine in a systematic, unbiased and robust manner.
The five clinical sites involved in the study were HIV clinics in Antwerp, Barcelona, Brighton, Lisbon and Zagreb. The general care of PLWH was very similar in the five clinical sites—specifically, in the follow-up of individuals living with stable HIV. PLWH who were participating in the study had visits at baseline and months 6, 12, 18, 24 and 30. At baseline, participants visited the hospital for a face-to-face visit and were introduced to the EmERGE platform. At six months, the individuals attended for bloods tests, as per the standard of care. The results were checked by a clinician and then pushed through to the patient’s mobile phone if the results were unremarkable. Visits at 18 and 30 months were conducted in a similar fashion. At 12 months and 24 months, the patients were seen by their clinical team in standard care.
As part of the formal evaluation, usability was assessed through the System Usability Score (SUS) [38], a scoring system that enables the evaluation of many interventions, including hardware, software, mobile devices, websites and applications. EmERGE participants were asked to complete the SUS after one year and two years of mHealth use. The SUS is a 10-item Likert scale (1–5, strongly disagree to strongly agree) assessment of usability that includes the following questions:
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well-integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
For items 1, 3, 5, 7 and 9, the contribution is the scale position minus 1. For items 2, 4, 6, 8 and 10, the contribution is 5 minus the scale position. The score is calculated as 2.5 × (sum of item contributions). Therefore, the resulting score ranges from 0 to 100, with a higher score indicating greater usability [39]. A score over 68 is considered above average [40].
Acceptability was assessed by means of a Patient-Reported Experience Measure (PREM) completed by participants at month 12 and month 24. This paper reports the results of two measures included in the PREM, as they were the questions related to the use of the EmERGE platform as part of the mHealth supported pathway:
“How would you rate your overall satisfaction with the EmERGE service?” (Excellent/Good/Satisfactory/Poor/Very poor).
“Would you recommend the EmERGE service to a friend?” (Yes/No).
In addition to the acceptability and usability assessment, this article describes the usage data to analyze the use, performance and operation of the EmERGE platform both from the PLWH and the clinical and administrative staff perspectives. It also allows comparisons of the use of the platform in different clinical sites.
3. Results
3.1. EmERGE Platform Specification
The objective of the EmERGE platform is to provide the relevant, timely and secure delivery of healthcare data to people living with stable HIV infection through the use of novel mHealth tools. These healthcare data include medication data (name and information about the drug, dose, frequency and date of commencement); appointments data (date, time and type of the appointment) and blood test results (current and historical data and evolution graphs). The data are sent along with a message from the clinician that provides the patient with the necessary information to interpret and understand the data. The main users of the EmERGE platform are PLWH and the clinical and administrative staffs of the clinical sites involved. The use cases and the specific requirements of the platform were determined through a codesigned work with PLWH and clinicians [35,36]. Table 1 and Table 2 summarize the functional and nonfunctional requirements, respectively, as codesigned with clinicians and the community.
3.2. Architecture
The EmERGE platform consists of four main components: the Adaptor, the Clinical Web Application (CWA), the Messaging Service (MS) and the Mobile Application. This group of software allows clinical data to be gathered from the hospital network, formatted appropriately, sent to and displayed on patient devices. The basic structure of the EmERGE platform and the interaction of the four key elements is shown in Figure 1.
The Adaptor provides a single Application Program Interface (API) for the CWA to extract the data via an interface with each hospital database. The Adaptor ensures that the data are in a single format for onward processing, facilitating the integration with different clinical systems. In the context of this study, each clinical site had a very different way of obtaining data: Brighton and Lisbon both used API-based systems, whereas Zagreb and Antwerp both provided SQL Server databases from which to gather data. The Adaptor integrates these with different demographic and clinical information systems by adjusting the correct set of models for gathering data. The implementation of the Adaptor was made in close collaboration with the IT staff of each hospital. The CWA is hosted in the hospital network and provides clinical users with a web application to register and manage patient users and their mobile devices and to provide executive control over the delivery of relevant information to mobile devices. The MS is a cloud-based API sitting between the CWA and the Mobile Application that stores encrypted data around medication, blood test results and appointments for specific individuals. The MS is responsible for delivering these data to the Mobile Application on demand, so that this information is not stored on the patient mobile device. The Mobile Application receives user specific information from the CWA via the MS and provides app users with access to their own data.
3.3. User Interfaces and Graphical Design
The EmERGE platform offers two main User Interfaces: the CWA for clinicians and the Mobile Application for PLWH. The CWA was built using Bootstrap-Sass 3 for the graphical design. Bootstrap is a popular open-source framework that offers web designers many functions that facilitate development and accelerate site building. This version of Bootstrap uses Sass (Syntactically Awesome Style Sheets) as the preprocessor scripting language that is interpreted or compiled into CSS. The Mobile Application also uses Sass for the graphical design. Figure 2 shows the CWA and the Mobile Application interfaces.
3.4. Security and Privacy
Security is a key factor for our patient group and for their clinicians. The CWA within each site is regarded as a “walled-off data silo”. It exists within the existing IT infrastructure of the clinical site and should not be open for external access. As such, unless connected via a Virtual Private Network, it is not possible to access to the CWA from outside of the hospital network. All clinical and demographic data stored locally within the CWA’s local database are encrypted to a minimum of 256-bit Advanced Encryption Standard (AES). The transmission of data outside of the hospital network, from the CWA to the Mobile Application, is encrypted over Transport Layer Security (TLS) and includes no patient identifiable information. As an additional security measure, the clinical information is not stored on the patient’s mobile device at any point.
Strong authentication procedures were designed in order to guarantee confidentiality and privacy. In this regard, a one-time password was generated by the clinical site personnel during the patient registration procedure and, once communicated to the patient, was permanently deleted. This password was required during the installation and setup of the Mobile Application, as well as an additional passcode defined by the patient, and used to sign into the Mobile Application from that moment onwards.
3.5. Evaluation of the EmERGE platform
3.5.1. Usage Data
The EmERGE platform was successfully deployed in five clinical sites in Europe during 2017: the sites were integrated sequentially between April 2017 and November 2017. The patients (2251) used the platform by downloading and installing the Mobile Application: 309 patients in Zagreb, 565 in Brighton, 549 in Barcelona, 249 in Antwerp and 579 in Lisbon. Enrollment closed in October 2018, with the follow-up continuing for a minimum of 12 months. Of the individuals enrolled in the study, 2048 (91.0%) were male; the median age was 43.0 years (interquartile range (IQR) 37.0 to 51.0 years). Eighty-two (3.6%) withdrew from the study within the 12 months of follow-up, with a further 84 (3.8%) by 30 months (a total of 166 withdrawals). There were no security incidents and only three major technical incidents and bugs identified during the course of the cohort study (April 2017–October 2019): (1) blood test result evolution graphs not being displayed, (2) Antwerp server not available for one week and (3) automatic log out of the Mobile Application. These technical issues were solved with no significant impact on the study, although the automatic log out issue caused the withdrawal of several patients.
The usage of the EmERGE CWA in each clinical site is described in Figure 3 by the number of patients registered on the EmERGE platform (A) and the number of data submissions to the users’ Mobile Application (B) at each clinical site during the course of the cohort study. The number of data submissions was weighted according to the total number of patients registered on the platform in each site to allow comparisons among the different clinical sites.
For the EmERGE Mobile Application, this paper reports the usage data of the iOS users, as the historical usage data from the Android users are not available. This is due to a modification in the statistics reports carried out by Google in July 2019 that impeded the retrieval of consistent data for the analysis. From the App Analytics tool available in Apple Store Connect, we can state that at least 33.8% of EmERGE participants used iOS: 54.8% in Brighton, 33.6% in Lisbon, 32.9% in Antwerp, 27.3% in Barcelona and 11.8% in Zagreb. There were a total of 840 installations and 86 uninstallations during the course of the cohort study. Eighty-seven point one percent of participants maintained the app installed in their devices, ranging from 73.4% in Antwerp to 96.1% in Brighton. Figure 4 shows the monthly distribution of the number of iOS devices with at least one session during the course of the study, i.e., the percentage of iOS users who did and did not use the app each month at each clinical site. It is important to note that the App Analytic tool only reported data from those users that consented to data sharing.
3.5.2. Impact on Acceptability and Usability
All PLWH enrolled in the study were asked to complete the SUS questionnaire and a questionnaire on their experience with the new mHealth-supported pathway at month 12 and month 24. The use of different platforms (Android vs iOS) did not influence the usability results and the experiences reported, as the EmERGE Mobile Application provided identical functionalities and graphical user interfaces thanks to the cross-platform approach followed.
The SUS was considered complete if a SUS score could be calculated (all items were entered). The SUS was completed by 1209 participants at month 12 and by 380 at month 24. Limited data was available for Lisbon for month 24 (seven participants). Figure 5 shows the SUS score by site and EmERGE study time point.
The SUS scores were high at all sites at both month 12 and month 24, with the medians ranging from 77.5 in Barcelona at month 12 and month 24 to 100.0 in Zagreb at month 24. Within the sites, the scores between the time points were very similar. Overall, the median (IQR) SUS score at month 12 was 85.0 (70.0 to 95.5) and 85.0 (71.3 to 95.0) at month 24.
The PREM was collected at month 12 and month 24. Here, we report the results of two measures included in the PREM questionnaire, as they are the questions related to the use of the EmERGE platform as a part of the mHealth-supported pathway. Figure 6 shows that participant satisfaction with the EmERGE service was high at all sites at both time points, with 82.8% and 83.6% of participants rating it either “Good” or “Excellent” overall at month 12 and month 24, respectively. Ninety-four point six percent of the participants would recommend EmERGE to a friend at month 12 and 94.2% at month 24. Overall, this ranged from 89.7% in Antwerp at month 24 to 98.4% in Zagreb at month 12 (excluding the 100% at Lisbon at month 24, as this was based on only seven participants).
4. Discussion
This article provides clinical evidence on the implementation of mHealth interventions, demonstrating that this digital supported pathway of care is considered feasible and acceptable by people living with stable HIV and their healthcare providers. In this regard, a large and high-quality cohort study has been designed and implemented and a structured and comprehensive HTA framework has been used, MAST, which provides a scientific robust approach based on rigorous data and enables evidence-informed decision-making. MAST fosters the involvement of the users in the assessment process enhancing the acceptability and usability of the mHealth applications developed. In the EmERGE project, a strong emphasis has been placed on the codesign phase, which has resulted in a person-centered digital solution. The benefits of people-centered approaches are well-documented, and policymakers have recognised the need to promote them in order to provide health services that are responsive to individuals and communities [1,41,42]. Another important characteristic of the EmERGE platform is the capability of integrating with hospital information systems to provide PLWH with secure access to their own data. All of these characteristics may have contributed to the high perceived value of the platform by end users, as stated by the acceptability and usability assessment results. The SUS scores at all sites at months 12 and 24 showed a high level of usability, with medians ranging from 77.5 in Barcelona at month 12 and month 24 to 100.0 in Zagreb at month 24, where 77.5 could be described as “Excellent” and 100 as the “Best Imaginable” [40]. The satisfaction with the EmERGE service was high at all sites at both month 12 and month 24, according to the PREM results. The differences between clinical sites’ results could be motivated by their previous experience with digital pathways, as well as the benefits perceived by the end users, such as, for example, Barcelona, where there was already an eHealth system with a very complete set of functionalities [24,25], or Zagreb, with no digital background, where the introduction of the EmERGE platform has improved the reporting of blood test results with a more systematic procedure.
The usage data showed that the EmERGE platform has been used uniformly in each clinical site once the patient population was stable (from June/July 2018), although Barcelona and especially Lisbon professional users carried out fewer data submissions, considering the participants recruited in those sites. With regards to the use of the Mobile Application, the participants from Brighton represented the highest number of iOS users and Zagreb the fewest. The participants from Antwerp, Barcelona and Lisbon had similar distributions. All of the iOS users showed similar usages of the Mobile Application from November 2017. The lack of usage data availability for Android users represents a limitation to this study and should be taken into account when interpreting the results. Additional learnings could be extracted from the usage data if further data analytics tools would have been available, such as, for example, data usage patterns and the functionalities more frequently used. However, privacy issues should be considered before including these kinds of tools, as they may influence user acceptance. Despite these limitations, and taking into account the information coming from the usage data and the withdrawal rate, it could be inferred that the app retention was high, as well as the retention in care during the cohort study.
The clinical outcomes from the formal evaluation [26] and the smooth running of the EmERGE platform during the course of the cohort study demonstrated the clinical and technical feasibility of the EmERGE pathway overcoming all technical and operational issues reported by previous experiences [20,21]. The availability of the Mobile Application across common platforms and the use of access points to download the app (Google and Apple Play Stores) facilitated the recruitment of the study participants, as well as the quality of the app, thanks to the verification procedures provided by these platforms. The use of agile methodologies and cross-platform technologies allowed a quick adaptation of the platform to the evolving needs coming from the codesign process and the study implementation. The availability of different deployment environments and the use of a ticketing system were demonstrated to be useful to deal with both the iterative development process and the support and maintenance activities during the course of the study. Moreover, there were no security breaches and few technical incidents, despite the large group of users (2251 participants and more than 20 clinicians and administrative staff) and the long evaluation period (30 months).
The EmERGE platform is in compliance with the EU Digital Single Market strategy, which promotes citizens’ secure access to health data, the full-scale deployment of new care models, the use of HTA and the utilization of interoperable digital solutions for the delivery of efficient and cost-effective care. In this respect, the EmERGE platform is a General Data Protection Regulation (GDPR)-compliant tool [43] that ensures secure access to health data by the exchange of pseudo-anonymous and unreadable data and provides strong and reliable authentication procedures. Additionally, the modular technical infrastructure proposed allows the future uptake of the EmERGE platform by new clinical sites and new health systems by adjusting the Application Programming Interface adapter to the specific new site IT system characteristics. The mHealth platform could also be easily scaled up and transferred to different clinical domains.
5. Conclusions
mHealth technology is being used increasingly and may be a leading tool in future healthcare paradigms in which personalized health and care, and the use of digital tools for citizen empowerment and for person-centered care, in compliance with data protection rules, acquire special relevance [7]. There is a large body of evidence through Randomized Clinical Trials and systematic reviews that mHealth is an effective mechanism for innovative healthcare strategies and a promising solution to provide efficient, cost-effective and patient-centered care, both in terms of prevention and treatment [44]. In the field of HIV, the successful scale-up of antiretroviral therapy globally gives mHealth applications the potential to transform HIV care beyond viral suppression in terms of comorbidity management, health-related quality of life and patient-reported outcomes [45]. However, there are arguments that further rigorous studies are needed to assess the outcomes and guide the implementation of personalized mHealth interventions aimed at enhancing the uptake and acceptability in routine practice that is fundamental for enabling a continuity of care [7]. In this regard, and thanks to the funding received from the European Union’s Horizon 2020 research and innovation programme, this article provides clinical evidence on the implementation of mHealth interventions through the assessment of a novel mHealth pathway of care for people living with medically stable HIV. The EmERGE platform supporting the digital pathway provides person-centered services, complies with data protection regulations and has been shown to be feasible and acceptable by end users. A not-for-profit company has been created with the aim of expanding the EmERGE paradigm of care to other clinical sites in Europe and supporting new research experiences to gain clinical evidence. New clinical sites have already shown their interest in implementing the EmERGE clinical pathway, and the research intervention is being defined in the context of the pre-exposure prophylaxis, a successful intervention to prevent HIV infection. Future works will include, among others, the development of a video conference and a messaging system to provide synchronous and asynchronous communication between patients and healthcare providers, a module for medication collection, the migration of the technology to React Native and the implementation of recommended standards for exchanging Electronic Health Records across the EU (HL7 standard), which will enhance the interoperability capabilities of the EmERGE platform, facilitating digital scale-ups and fostering the large-scale deployment and uptake of the EmERGE platform.
Conceptualization, J.W., F.G., C.C., E.W. and E.J.G.; methodology, J.W., C.I.J., F.G., C.C., P.S.-G., E.W. and E.J.G.; software, F.J.G., P.C. and E.W.; validation, F.J.G., P.C., J.W. and C.I.J.; formal analysis, F.J.G., P.C., C.I.J. and P.S.-G.; investigation, F.J.G., P.C., J.W., C.I.J. and F.G.; resources, F.J.G., P.C., C.I.J., J.W. and F.G..; data curation, F.J.G., P.C. and C.I.J.; writing—original draft preparation, F.J.G. and P.C.; writing—review and editing, F.J.G., P.C., J.W., C.I.J., F.G., C.C., P.S.-G., E.W. and E.J.G.; visualization, F.J.G., P.C. and C.I.J.; supervision, J.W., F.G., C.C., P.S.-G., E.W. and E.J.G.; project administration, P.C., J.W., F.G. and E.J.G. and funding acquisition, J.W., F.G., E.W. and E.J.G. All authors have read and agreed to the published version of the manuscript.
This project received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No. 643736. The dissemination activities within the EmERGE project do not represent the opinion of the European Community and only reflects the opinion of the authors.
The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the sponsor and each institution (The Ethical Committee for Clinical Research of the Hospital Clinic de HC-IDIBAPS, the South East Coast-BSUHT & Sussex Research Ethics Committee 16/LO/2122 10Jan17, the Institutional Review Board of the ITM, the Ethics Committee members of the University Hospital for Infectious Diseases “Fran Mihaljevic”, UHID, Croatia and the Ethics Committee for Health, Centro Hospitalar de Lisboa, Central CPE).
Informed consent was obtained from all subjects involved in the study.
The data are not publicly available due to the conditions of individual patient consent.
The authors thank the people living with HIV and their clinicians for their commitment and willingness to participate in the EmERGE project. The scientific members of the EmERGE consortium are: J. Whetham, D. Fatz and G. Weir, Brighton and Sussex University Hospitals NHS Trust, UK; M. Borges, E. Teofilo, G. Rodrigues and A. Cunha, Centro Hospitalar De Lisboa Central, EPE, Portugal; C. Fisher and S. Beaumont, EmERGE mHealth Ltd., UK; B. West, M. Dutarte, A. I. von Lingen, F. Greenhalgh and K. Block, European Aids Treatment Group, Belgium; F. Garcia, D. Garcia, R. Muñoz Pina, F. Etcheverry, L. Leal, L. Moreno and E. González, Fundacio Privada Clinic Per A La Recerca Biomedica, Spain; L. Apers, L. Mertens and S. Hoornaert, Institute of Tropical Medicine Antwerp, Belgium; J. Begovac, S. Zekan and I. Benkovic, Klinika Za Infektivne Bolesti Fran Mihaljevic, Croatia; J. Wyatt and M. Fraser, Modus Research and Innovation, UK; E. Beck, S. Mandalia and P. Yfantopoulos, NPMS-HHC CIC, UK; E. J. Gomez and P. Chausa, F. J. Gárate UPM, Spain; F. Henwood, M. Darking and B. Marent, University of Brighton, UK and S. Bremner and C. Jones, University of Sussex, UK. The EmERGE advisory board includes: C. Cáceres, R. Harding, A. León and E. Wallitt.
The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses or interpretation of data; in the writing of the manuscript or in the decision to publish the results.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Enlarge this image.Figure 3. (A) Evolution of the number of patients registered for the EmERGE, and (B) data submissions to users’ devices at each clinical site during the course of the cohort study.
Enlarge this image.Figure 4. % of iOS users who did/did not use the app each month at each clinical site.
Enlarge this image.Figure 5. System Usability Score (SUS) score by site and EmERGE study time point. IQR: interquartile range.
Enlarge this image.Figure 6. Distribution of the responses related to satisfaction with the EmERGE service at month 12 and month 24 across the sites.
Summary of the main EmERGE platform functional requirements. PLWH: people living with HIV.
| Functionality | Description | User |
|---|---|---|
| Authentication | Users are required to login into the platform on each use | All users |
| Patient/device registration | Registration of patients and the associated mobile devices | Administrators and clinicians |
| All patient visualization | Search and filtering of all registered patients | Administrators and clinicians |
| Patient data visualization | Overview of each patient user: current medication list (name, dose/frequency and date of commencement); blood test results (graphically and in a table format) and appointment lists | Administrators and clinicians |
| “Virtual clinics” appointment management | Send data to the mobile application; rebook the patient for a future revision; pause the patient in the system | Administrators and clinicians |
| Messaging | Messaging from Hospital clinicians | All users |
| FAQ management | Provision of Frequently Ask Questions | Administrators |
| User management | Creation and elimination of users and assignment of roles | Administrators |
| Medication visualization | Visualization of medication list (name of medication, dose/frequency and date of commencement). Link to external Liverpool HIV drugs interaction webpage | PLWH |
| Blood test results visualization | Visualization of blood test results: current and historical data; evolution graphs | PLWH |
| Appointments | Visualization of next appointments. Integration with patient mobile calendar | PLWH |
| Help desk and information | Support and access to FAQs | PLWH |
| Application Settings | Erase application data, change passcode, change security question, ask for a new password, study withdrawal | PLWH |
Summary of the main EmERGE platform nonfunctional requirements.
| Requirement | Description |
|---|---|
| Usability | Easy to be used and learnt, easy to be remembered, efficient and able to generate satisfaction in the users. Intuitive interfaces, readable and adapted to users, different web browsers and device resolution screens |
| Adaptability | Platform to be introduced in systems of different hospitals. It must be easy to implement and easy to readapt to different characteristics and requirements |
| Scalability | System architecture should be designed to appropriately handle increasing and decreasing workloads and prepared for a rapid expansion in number of clinical sites and number of users |
| Security | Information should be sent in a secure way. Stored data only accessible by authorized health staff. System only accessible to registered and validated users |
| Performance | The response time in operations must be the expected time of any web application |
| Multiplatform | The platform should be accessible through mobile applications (iOS and Android) or any of the existing web browsers depending on the role of the user |
References
1. eHealth Action Plan 2012–2020-Innovative Healthcare for the 21st Century. Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions. Available online: https://ec.europa.eu/digital-single-market/en/news/ehealth-action-plan-2012-2020-innovative-healthcare-21st-century#:~:text=The%20European%20Commission’s%20eHealth%20Action,and%20patient%2Dcentred%20health%20services (accessed on 9 February 2021).
2. Sim, I. Mobile Devices and Health. N. Engl. J. Med.; 2019; 381, pp. 956-968. [DOI: https://dx.doi.org/10.1056/NEJMra1806949]
3. Whitehead, L.; Seaton, P. The Effectiveness of Self-Management Mobile Phone and Tablet Apps in Long-term Condition Management: A Systematic Review. J. Med. Internet Res.; 2016; 18, e97. [DOI: https://dx.doi.org/10.2196/jmir.4883]
4. Lee, J.A.; Choi, M.; Lee, S.A.; Jiang, N. Effective behavioral intervention strategies using mobile health applications for chronic disease management: A systematic review. BMC Med. Inform. Decis. Mak.; 2018; 18, 12. [DOI: https://dx.doi.org/10.1186/s12911-018-0591-0]
5. Siddiqui, J.; Herchline, T.; Kahlon, S.; Moyer, K.; Scott, J.; Wood, B.; Young, J. Infectious Diseases Society of America Position Statement on Telehealth and Telemedicine as Applied to the Practice of Infectious Diseases. Clin. Infect. Dis.; 2017; 64, pp. 237-242. [DOI: https://dx.doi.org/10.1093/cid/ciw773]
6. WHO. Global Diffusion of eHealth: Making Universal Health Coverage Achievable. Report of the Third Global Survey on eHealth; World Health Organization: Geneva, Switzerland, 2016; ISBN 978-92-4-151178-0
7. EU4Health 2021–2027–a Vision for a Healthier European Union. Available online: https://ec.europa.eu/health/funding/eu4health_en (accessed on 9 February 2021).
8. Europe mHealth Market Research Report. Available online: https://www.marketdataforecast.com/market-reports/europe-mobile-health-market (accessed on 9 February 2021).
9. Retention Rate on Day 1 and Day 30 of Mobile App Installs Worldwide as of August 2020, by Category. Available online: https://www.statista.com/statistics/259329/ios-and-android-app-user-retention-rate/ (accessed on 9 February 2021).
10. Patient Adoption of mHealth. Use, Evidence and Remaining Barriers to Mainstream Acceptance; IMS Institute for Healthcare Informatics: Parsippany, NJ, USA, 2015.
11. Leigh, S.; Ashall-Payne, L. The role of health-care providers in mHealth adoption. Lancet Digit. Health; 2019; 1, pp. e58-e59. [DOI: https://dx.doi.org/10.1016/S2589-7500(19)30025-1]
12. Jacob, C.; Sanchez-Vazquez, A.; Ivory, C. Social, Organizational, and Technological Factors Impacting Clinicians’ Adoption of Mobile Health Tools: Systematic Literature Review. JMIR Mhealth Uhealth; 2020; 8, e15935. [DOI: https://dx.doi.org/10.2196/15935]
13. Hall, B.J.; Sou, K.L.; Beanland, R.; Lacky, M.; Tso, L.S.; Ma, Q.; Doherty, M.; Tucker, J.D. Barriers and Facilitators to Interventions Improving Retention in HIV Care: A Qualitative Evidence Meta-Synthesis. AIDS Behav.; 2017; 21, pp. 1755-1767. [DOI: https://dx.doi.org/10.1007/s10461-016-1537-0] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27582088]
14. Ko, D.; Bratzke, L.C.; Roberts, T. Self-management assessment in multiple chronic conditions: A narrative review of literature. Int. J. Nurs. Stud.; 2018; 83, pp. 83-90. [DOI: https://dx.doi.org/10.1016/j.ijnurstu.2018.04.009] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29709734]
15. Bulsara, S.M.; Wainberg, M.L.; Newton-John, T.R.O. Predictors of Adult Retention in HIV Care: A Systematic Review. AIDS Behav.; 2018; 22, pp. 752-764. [DOI: https://dx.doi.org/10.1007/s10461-016-1644-y] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27990582]
16. May, M.; Gompels, M.; Delpech, V.; Porter, K.; Orkin, C.; Kegg, S.; Hay, P.; Johnson, M.; Palfreeman, A.; Gilson, R. et al. Impact on life expectancy of HIV-1 positive individuals of CD4+ cell count and viral load response to antiretroviral therapy: UK cohort study. AIDS; 2014; 28, pp. 1192-1202. [DOI: https://dx.doi.org/10.1097/QAD.0000000000000243] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/24556869]
17. Global HIV & AIDS Statistics—2019 Fact Sheet. UNAIDS. Available online: https://www.unaids.org/en/resources/fact-sheet (accessed on 18 February 2020).
18. EACS Guidelines Version 10.0. November 2019. Available online: https://www.eacsociety.org/files/2019_guidelines-10.0_final.pdf (accessed on 18 February 2020).
19. Dillingham, R.; Ingersoll, K.; Flickinger, T.E.; Waldman, A.L.; Grabowski, M.; Laurence, C.; Wispelwey, E.; Reynolds, G.; Conaway, M.; Cohn, W.F. PositiveLinks: A Mobile Health Intervention for Retention in HIV Care and Clinical Outcomes with 12-Month Follow-Up. AIDS Patient Care STDS; 2018; 32, pp. 241-250. [DOI: https://dx.doi.org/10.1089/apc.2017.0303]
20. Venter, W.; Coleman, J.; Chan, V.L.; Shubber, Z.; Phatsoane, M.; Gorgens, M.; Stewart-Isherwood, L.; Carmona, S.; Fraser-Hurt, N. Improving Linkage to HIV Care Through Mobile Phone Apps: Randomized Controlled Trial. JMIR MHealth UHealth; 2018; 6, e155. [DOI: https://dx.doi.org/10.2196/mhealth.8376]
21. Venter, W.; Fischer, A.; Lalla-Edward, S.T.; Coleman, J.; Lau Chan, V.; Shubber, Z.; Phatsoane, M.; Gorgens, M.; Stewart-Isherwood, L.; Carmona, S. et al. Improving Linkage to and Retention in Care in Newly Diagnosed HIV-Positive Patients Using Smartphones in South Africa: Randomized Controlled Trial. JMIR MHealth UHealth; 2019; 7, e12652. [DOI: https://dx.doi.org/10.2196/12652]
22. Cooper, V.; Clatworthy, J.; Whetham, J.; Consortium, E. mHealth Interventions to Support Self-Management in HIV: A Systematic Review. Open AIDS J.; 2017; 11, pp. 119-132. [DOI: https://dx.doi.org/10.2174/1874613601711010119] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29290888]
23. Muessig, K.E.; Nekkanti, M.; Bauermeister, J.; Bull, S.; Hightow-Weidman, L.B. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr. HIV AIDS Rep.; 2015; 12, pp. 173-190. [DOI: https://dx.doi.org/10.1007/s11904-014-0239-3] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25626718]
24. Caceres, C.; Gomez, E.J.; Garcia, F.; Gatell, J.M.; del Pozo, F. An integral care telemedicine system for HIV/AIDS patients. Int. J. Med. Inform.; 2006; 75, pp. 638-642. [DOI: https://dx.doi.org/10.1016/j.ijmedinf.2006.04.010] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/16759900]
25. León, A.; Cáceres, C.; Fernández, E.; Chausa, P.; Martin, M.; Codina, C.; Rousaud, A.; Blanch, J.; Mallolas, J.; Martinez, E. et al. A New Multidisciplinary Home Care Telemedicine System to Monitor Stable Chronic Human Immunodeficiency Virus-Infected Patients: A Randomized Study. PLoS ONE; 2011; 21, e14515. [DOI: https://dx.doi.org/10.1371/journal.pone.0014515]
26. Whetham, J.; Jones, C.; Bremner, S.; Apers, L.; Begovac, J.; Borges, M.; Leon, A.; Vera, J.; Zekan, S.; West, B. et al. EmERGE: Feasibility and uptake of a co-designed digital health supported pathway of care for people living with medically stable HIV. P111. J. Int. AIDS Soc.; 2020; 23, 95.Available online: https://onlinelibrary.wiley.com/doi/10.1002/jia2.25616 (accessed on 18 March 2021).
27. EmERGE Project Website. Available online: https://www.emergeproject.eu/ (accessed on 9 February 2021).
28. EmERGE Project on the Community Research and Development Information Service Website. Available online: https://cordis.europa.eu/project/id/643736 (accessed on 9 February 2021).
29. Ruby on Rails Website. Available online: https://rubyonrails.org/ (accessed on 9 February 2021).
30. Ruby Website. Available online: https://www.ruby-lang.org/en/ (accessed on 9 February 2021).
31. Ionic website. Available online: https://ionicframework.com/ (accessed on 9 February 2021).
32. Heroku Website. Available online: https://www.heroku.com/ (accessed on 9 February 2021).
33. Santos, A.R.; Kroll, J.; Sales, A.; Fernandes, P.; Wildt, D. Investigating the Adoption of Agile Practices in Mobile Application Development. ICEIS; 2016; [DOI: https://dx.doi.org/10.5220/0005835404900497]
34. Flora, H.K.; Chande, S.V. A review and analysis on mobile application development processes using agile methodologies. Int. J. Res. Comput. Sci.; 2013; 3, pp. 9-18. [DOI: https://dx.doi.org/10.7815/ijorcs.34.2013.068]
35. Marent, B.; Henwood, F.; Darking, M. on behalf of the EmERGE Consortium. Ambivalence in digital health: Co-designing an mHealth platform for HIV care. Soc. Sci. Med.; 2018; 215, pp. 133-241. [DOI: https://dx.doi.org/10.1016/j.socscimed.2018.09.003] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30232053]
36. Marent, B.; Henwood, F.; Darking, M. EmERGE Consortium. Development of an mHealth platform for HIV Care: Gathering User Perspectives Through Co-Design Workshops and Interviews. JMIR Mhealth Uhealth; 2018; 6, e184. [DOI: https://dx.doi.org/10.2196/mhealth.9856] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30339132]
37. Kidholm, K.; Ekeland, A.G.; Jensen, L.K.; Rasmussen, J.; Pedersen, C.D.; Bowes, A.; Flottorp, S.A.; Bech, M. A model for assessment of telemedicine applications: Mast. Int. J. Technol. Assess Health Care; 2012; 28, pp. 44-51. [DOI: https://dx.doi.org/10.1017/S0266462311000638] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22617736]
38. Brooke, J.B. SUS-a quick and dirty usability scale. Usability Evaluation in Industry; Jordan, P.; Thomas, B.; Weerdmeester, B.; McLelland, I. Taylor and Francis: London, UK, 1996.
39. SUS: A ‘quick and dirty’ usability scale. Usability Evaluation in Industry; 1st ed. McClelland, L.; Jordan, W.; Thomas, B.; Weerdmeester, A. Taylor & Francis: London, UK, 1996; pp. 189-194.
40. Bangor, A.; Kortum, P.; Miller, J. Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. J. Usability Stud.; 2009; 4, pp. 114-123.
41. World Health Organization. People-Centred and Integrated Health Services: An Overview of the Evidence. 2015; Available online: https://apps.who.int/iris/bitstream/handle/10665/155004/WHO_HIS_SDS_2015.7_eng.pdf?sequence=1 (accessed on 9 February 2021).
42. WHO Global Strategy on People-Centred and Integrated Health Services. World Health Organization. 2015; Available online: https://apps.who.int/iris/bitstream/handle/10665/155002/WHO_HIS_SDS_2015.6_eng.pdf?sequence=1 (accessed on 9 February 2021).
43. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation). Available online: https://eur-lex.europa.eu/eli/reg/2016/679/oj (accessed on 18 March 2021).
44. Hill, P. mHealth: A cost-effective solution to chronic problems?. J. Med. Therap.; 2018; 2, pp. 2-4. [DOI: https://dx.doi.org/10.15761/JMT.1000130]
45. van den Berk, G.E.L.; Leoni, M.C.; Behrens, G.M.N.; Taljaard, J.; Arends, J.E. Improving HIV-related care through eHealth. Lancet HIV; 2019; 7, pp. e8-e10. [DOI: https://dx.doi.org/10.1016/S2352-3018(19)30348-0]
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Whetham, Jennifer 2 ; Jones, Christopher Iain 3
; García, Felipe 4
; Cáceres, César 5 ; Sánchez-González, Patricia 6
; Wallitt, Edward 7 ; Gómez, Enrique J 6 ; Tchounwou, Paul1 Biomedical Engineering and Telemedicine Centre, ETSI Telecomunicación, Center for Biomedical Technology, Universidad Politécnica de Madrid, 28040 Madrid, Spain;
2 Department of Sexual Health and HIV Medicine, Brighton and Sussex University Hospitals NHS Trust, BN2 5BE Brighton, UK;
3 Brighton and Sussex Medical School, BN1 9PH Falmer, UK;
4 Infectious Diseases Department, Fundacio Privada Clinic per a la Recerca Biomedica—IDIBAPS, University of Barcelona, 08036 Barcelona, Spain;
5 Biomedical Engineering and Telemedicine Centre, ETSI Telecomunicación, Center for Biomedical Technology, Universidad Politécnica de Madrid, 28040 Madrid, Spain;
6 Biomedical Engineering and Telemedicine Centre, ETSI Telecomunicación, Center for Biomedical Technology, Universidad Politécnica de Madrid, 28040 Madrid, Spain;
7 Podmedics, HA6 2UE Northwood, UK;