Measuring Health-Related Quality of Life With Multimodal Data: Viewpoint

Abstract

The ability to objectively measure aspects of performance and behavior is a fundamental pillar of digital health, enabling digital wellness products, decentralized trial concepts, evidence generation, digital therapeutics, and more. Emerging multimodal technologies capable of measuring several modalities simultaneously and efforts to integrate inputs across several sources are further expanding the limits of what digital measures can assess. Experts from the field of digital health were convened as part of a multi-stakeholder workshop to examine the progress of multimodal digital measures in two key areas: detection of disease and the measurement of meaningful aspects of health relevant to the quality of life. Here we present a meeting report, summarizing key discussion points, relevant literature, and finally a vision for the immediate future, including how multimodal measures can provide value to stakeholders across drug development and care delivery, as well as three key areas where headway will need to be made if we are to continue to build on the encouraging progress so far: collaboration and data sharing, removal of barriers to data integration, and alignment around robust modular evaluation of new measurement capabilities.

Introduction

The field of digital health has become a multibillion dollar market, powering a paradigm shift by enabling the continuous capture of multimodal data including activity, sleep, vital signs, and contextual information. Novel machine learning applications are pioneering the conversion of these multimodal data into measures for health-related quality of life (QOL)–relevant symptoms like fatigue [1], stress [2], and depression [3,4]. These insights have the potential to enable improved care delivery [5] and a deeper understanding of patients’ lived experiences and better, more personalized medicines. However, important barriers remain to realize these benefits, both in technical and social aspects of real-world adoption.

On July 27, 2021, as part of the IEEE-EMBS International Conference on Biomedical and Health Informatics jointly organized with the 17th IEEE-EMBS International Conference on Wearable and Implantable Body Sensor Networks [6], a workshop was held on “Measuring Quality of Life with Multimodal Data.” The workshop was divided into two sessions, the first focusing on disease detection and the second focusing on the measurement of well-being. Abstracts from the keynotes and talks are presented in Table 1; this meeting report summarizes key discussion points, relevant literature, and finally a vision for the immediate future, including how multimodal measures can provide value to stakeholders across drug development and care delivery, as well as three key areas where headway will need to be made if we are to continue to build on the encouraging progress so far: collaboration and data sharing, removal of barriers to data integration, and alignment around robust modular evaluation of new measurement capabilities.

Table 1. Talk titles and abstracts for all presented work.

Speaker			Affiliation		Title		Abstract		Citations and further reading
Session 1: disease detection
	Author LF	Evidation Health		PGHDa: a new ally for public health		PGHD from smartphones, wearables, and other sensors have the potential to transform the way health is measured, with broad-ranging applications from clinical research to public health and health care at large. This talk will survey examples of applications of PGHD across therapeutic areas, including post-op monitoring, screening for cognitive impairment, and a particular focus on public health applications for flu and COVID-19 detection and quantification. Finally, I will discuss lessons learned in translating PGHD research into benefits for the individual, with emphasis on the importance of evaluating analytics performance (eg, AUROCb, sensitivity, and specificity) within a specific context of use of a real-world application.		[7,8]
	Author FL	Roche Pharma Research and Early Development, F. Hoffmann-La Roche Ltd		Digital health technology tools and quality of life: examples from current studies in neurological disorders		In recent years, DHTTsc such as smartphones and wearables are becoming an integrated part of clinical research. Augmented by novel often AId-powered signal processing, they enable continuous and precise measurements of disease symptoms. It is therefore becoming important to link these measures to the different aspects of QOLe of patients to make them meaningful tools for drug decision-making. In this talk, I will highlight examples from DHTTs we are developing for neurological disorders such as Parkinson disease, multiple sclerosis, and Huntington disease. Leveraging active testing and patient questionnaires accompanied by passive continuous monitoring in daily life, these tools offer rich sets of data. General signal processing and dedicated machine learning/AI solutions are used to unlock these data sets and relate them back to standard clinical scores of disease severity. I will show how resulting measures relate to patients’ self-perceived health-related quality of life, how DHTTs used during COVID-19–induced lockdowns can offer new insights on QOL perception, and how we envision strengthening the link between novel sensor measurements and patient-relevant symptoms and impacts.		[9,10]
	Author BV	Byteflies		Leveraging multimodal sensor data to assess complex chronic conditions at home		Byteflies’s Sensor Dot platform enables continuous acquisition of physiologic and behavioral data. We leverage this multimodal data to move diagnostic tests typically performed in a specialized environment to the home of the user and to make longitudinal assessments of chronic conditions possible. In both cases, an understanding of the continuous changes in activities of daily living is crucial for safe and accurate clinical interpretation of the data. In this talk, I will discuss EpiCare@Home, a remote epileptic seizure monitoring solution built on top of the Byteflies platform.		[11,12]
	Author GG	Department of Neuroscience, University of Padua, Italy; SENSEDAT Srl, Padua, Italy		Unsupervised wearable and machine learning approach to identify depression, anxiety, and stress physiological phenotypes		Background: Anxiety and depression are defined with clinical interviews in RCTsf, possibly inflating intervention’s/placebo’s effects. We here introduce an algorithm to identify anxiety and depression with wearable-measured physiological biomarkers. Objectives: To validate a machine learning–based algorithm using wearable unsupervised measurements of the autonomic nervous system and physiological parameters to classify clinical anxiety and depression according to validated questionnaires. Methods: Included were physically healthy workers from the general population wearing an arm-band wearable device equipped with photoplethysmogram and electrodermal activity sensors for 24 hours. Participants answered validated self-report questionnaires for mental health, including PSS-10g, GAD-7h, and PHQ-9i. Wearable recordings were subject to artifact removal, signal preprocessing, and split in 30-second blocks for which physiological indexes and related features were extracted. A feature fusion approach was implemented together with the C5.0 machine learning algorithm, which was run on 70% randomly selected preprocessed blocks, and on the remaining 30% for external validation. Coprimary outcomes were anxiety (GAD-7≥10), and depression (PHQ-9≥10). Results: We included 95 participants (yielding 237,778 monitoring blocks), 47.7% females, mean age 37.2 (SD 15.5) years. Overall, 13.7% had anxiety, 12.6% had depression, and 7.4% had both. In the main sample, the wearable machine learning algorithm showed excellent accuracy for coprimary outcomes, namely, AUCj=0.928 for anxiety and AUC=0.959 for depression. Discussion: Limitations of the study include self-report questionnaires to assess primary outcomes and its cross-sectional nature. Potential implications of this work include biomarker-based inclusion criteria in RCT testing interventions for anxiety and depression, as well as screening and monitoring tools of mental health issues in the general population. Further studies should replicate the proposed algorithm against structured interview-based diagnoses with different wearable devices on clinical samples, possibly with a longitudinal design.		[13-15]
Session 2: measuring well-being
	Author AS	Rice University		Multimodal sensor data analysis and modeling for health and well-being		Digital phenotyping and machine learning technologies have shown a potential to measure objective behavioral and physiological markers, provide risk assessment for people who might have a high risk of poor health and well-being, and help make better decisions or behavioral changes to support health and well-being. I will introduce a series of studies, algorithms, and systems we have developed for measuring, predicting, and supporting personalized health and well-being. I will also discuss challenges, learned lessons, and potential future directions in health and well-being research.		[2,16]
	Author BS	UCSDk Department of Bioengineering and the Halicioglu Data Science Institute; Oura		The future of health and wellness discovery is democratic		Engineered solutions for personal data generation (wearable sensors, apps, etc) and analysis are proliferating rapidly, but health services served by these technologies continue to lag behind. Complexity in human diversity stymies algorithm generalizability and hampers successful wide adoption of any specific solution. We propose that efforts at expanding engagement in discovery will achieve two complementary goals: (1) promote mapping of biological diversity beyond demography and genetics into physiology and behavior so that algorithms can be developed on empirically determined subpopulations, and (2) fertilize natural experiments that will reveal communities sharing needs and goals, for whom solutions can then be tailored. Efforts to expand engagement may enable a virtuous cycle where iterative improvement and expansion in precision wellness technologies go from intractable to standard in personal, community, and clinical settings.		[17,18]
	Author FC	Cambridge Cognition; Department of Psychiatry, University of Cambridge		Characterizing fatigue using digital technologies		Fatigue is both common and burdensome across a range of patient groups. The manifestation of fatigue is complex, comprising both subjective and objective changes to cognitive and physical performance, and is determined by a range of factors, including sleep, mood, time of day, competing demands, and environmental context, as well as disease-specific variables. These factors, and consequently the patient’s experience of fatigue, vary with time, meaning that infrequent in-clinic assessments are likely to be of limited sensitivity. Given this complexity, we have been interested in exploring the potential role of digital technologies in capturing and characterizing fatigue, particularly the impact of fatigue on cognitive performance, across a range of clinical conditions. This talk will focus on methods of data collection such as brief active assessments, voice capture, and passive data from wearable technology, and describe insights these data provide us into this complex symptom.		[3,19]
	Author CvH	Connected Health Solutions, imec; OnePlanet Research Center		Nanoelectronics and AI for our (and our planet’s) health		We are faced with global challenges related to health, food, sustainability, and the environment. While these are formidable challenges, they also represent a substantial opportunity to improve people’s lives on a global scale while at the same time creating new economic opportunities. We are convinced nanoelectronics and digital technologies are the key tools for disruptive solutions. With that purpose in mind, the OnePlanet Research Center was created as a multidisciplinary collaboration between imec, Radboud University Medical Center, and Wageningen University & Research. In OnePlanet, we apply nanoelectronics and analytics innovations to solve problems related to personalized health, personalized nutrition, mental well-being, sustainable food production, and reduced environmental impact. The sensors and data innovations are working toward the creation of digital twins for prevention, early detection, or interception of disease.		[20,21]
	Author SF	MIT Media Lab		Monitoring well-being using longitudinal passive data		The boundaries between the consumer and clinical device markets are becoming leaner every year. This trend is driven by a number of factors including consumer demand for ubiquitous and constantly accessible health care; increased presence of chronic conditions (eg, high blood pressure, diabetes, depression, and obesity); and a corresponding need for preventive health care, an increasingly aging global population, availability of cost-effective wearable technology, and remote access to storage and computation resources. This trend enables substantial opportunities for providing health care services to larger populations at lower cost. It will also pave the way to personalized medicine where prevention, diagnosis, and treatment of a disease can be tailored to individuals’ characteristics and behavior. In this presentation, recent developments of wearable technologies at MIT Media Lab and their application to the diagnosis of mental health diseases and overall well-being are discussed.		[22,23]

^aPGHD: person-generated health data.

^bAUROC: area under the receiver operating characteristic curve.

^cDHTT: digital health technology tool.

^dAI: artificial intelligence.

^eQOL: quality of life.

^fRCT: randomized controlled trial.

^gPSS-10: Perceived Stress Scale.

^hGAD-7: Generalized Anxiety Disorder–7

ⁱPHQ-9: Patient Health Questionnaire.

^jAUC: area under the curve.

^kUCSD: University of California, San Diego.

Background: A Shared Lexicon

To begin discussions, participants shared their perspectives on some of the terminology relevant to this emerging area of research. In Table 2, we restate some of the key points raised to orientate readers in the following report.

Table 2. Key terms relevant to the discussion. Participants shared terminology relevant to this emerging area of research.

Term	Definition	References
Multimodal measures	Referencing “Multimodal Deep Learning,” multimodal measures are derived from multiple input modalities (eg, activity, sleep, heart rate, patient-reported outcomes, or contextual data)	[24]
Health-related quality of life	An individual’s or a group’s self-perceived physical and mental health over time	[25]
Digital measure	Sensor-derived objective measures arising from “connected digital products.” Includes active tests captured via a mobile platform and continuous passive data collected from a wearable technology but excludes electronic patient-reported outcomes and other subjective measures collected from mobile platforms. An all-inclusive term, encompassing all stages of maturity, settings, and technologies.	[26,27]
Digital end point	A subset of robustly evaluated digital measures that have successfully pursued acceptance or qualification and can be used as decision-making evidence in clinical trials	[27]
Digital biomarker	Objective quantifiable physiological and behavioral data that are collected and measured by means of digital devices such as portables, wearables, implantables, or digestibles. The data collected are typically used to explain, influence, or predict health-related outcomes.	[27]
Patient-reported outcome	Assessments about how patients feel or function in their daily lives where the information is reported by the patient themselves, without interpretation or modification by someone else. Note that assessments can cover a wide range of relevant categories, some of which are more quantifiable and less subjective (including medication use or symptom presence), and some which are more subjective (including symptom severity and perception of well-being).	[26]

Session 1: Disease Detection

The first session focused on the use of multimodal data and machine learning for disease detection. Detecting deviations from normal behaviors and processes is a key step in triggering further actions, whether that be a follow-up with a health care provider or a direct digital intervention [28-30].

The session started with a keynote from author LF of Evidation Health, who spoke about how person-generated health data (PGHD; adapted from [31]) are transforming public health applications of disease detection. Author LF provided an overview of how PGHD are being used to detect and measure disease progression in a range of indications, the use of PGHD for detecting COVID-19, and the challenges of distinguishing COVID-19 from other influenza-like illnesses and infections [7]. LF underlined that machine learning model performance needs to be evaluated with a specific context of use in mind and that, without such context, model performance is ultimately of little relevance in terms of clinical utility and large-scale adoption. Finally, the keynote closed with a discussion of how what we classically think of as evidence can be a source of value to patients themselves, by helping them manage and understand their own health.

The following talks covered a wide range of indications, including author FL of Roche who discussed the use of smartphone-based apps to monitor neurological conditions including Parkinson disease [9] and multiple sclerosis [10,32]. Author BV shared work from the Byteflies platform showing how the system is being deployed for longitudinal monitoring of sleep disorders and cardiorespiratory and neurodegenerative conditions, and for detecting seizures in epilepsy [11,12]. Finally, author GG presented recent work examining how unsupervised measurements of autonomic nervous system signals, including photoplesmography and electrodermal activity (EDA), are showing value in the detection and staging of mental health conditions like anxiety and depression, and how these measures play a complementary role to traditional biomarkers, becoming a useful tool in enhancing clinical trials and precision psychiatry [13-15].

The session was closed with a short panel discussion featuring all the speakers that focused on questions raised by the attendees. One question addressed the pros and cons of data collection via bring your own device (BYOD; ie, allowing participants to connect their own devices) versus data collection via an app versus provisioned devices. The speakers agreed that there are different advantages to each approach. For example, BYOD enables comparison to a personal baseline and has advantages for device adherence, whereas provisioned devices can enable higher data uniformity and eliminate barriers to participation due to lack of access to appropriate hardware. Overall, the key is to select the right data collection approach for a given setting; where data consistency or a specific data type or density is priority, for example, in a smaller randomized controlled trial, provisioning may be preferred [33]; BYOD may in turn be preferred in settings where scale becomes limiting or where long-term “pervasive” monitoring places an emphasis on measuring ecologically valid natural behavior [34]. It was noted that while progress has been made around BYOD for patient-reported outcomes [35-37], similar progress for digital measures has not been seen and will be a key step in unlocking the value previously outlined. Another question focused on challenges to integrating objective (ie, from wearable devices) and subjective (eg, from surveys of patient-reported outcomes) inputs. The panel pointed out that many disease detection applications combine both objective and subjective inputs, for example, asking participants to confirm a signal or get a follow-up test. They also pointed out that subjective and objective inputs measure different aspects, so we should not expect them to correlate; however, this also means that they may have different relationships to a given concept of interest [38]. Thus applications that combine objective and subjective inputs can have an advantage in signal detection for disease detection. To help clarify this point, consider the following example on general well-being: a range of objective characteristics can be measured that are informative of overall well-being, including social media activity, patterns of sleep and activity, news consumption, patterns of independence, and many other objective data sources; these sources are informative of several aspects of subjective well-being (eg, perception of health), but none have a direct relationship to any specific aspect of subjective well-being, and what relationship there is differs between individuals [39].

There were also questions on the value of specific objective features (eg, EDA in stress), and GG discussed how this is a special case because this objective marker directly measures autonomic nervous system activation and thus gives a very good signal on psychological state. This was contrasted against other objective measures (eg, step counts) that have a more indirect relationship to symptoms like depression and anxiety.

The panel also discussed the impact of covariates within a cohort (eg, comorbidities) and how it influences model performance. Specifically, when trying to derive more “generalizable” models, which perform well across a broad range of unseen individuals, there is a need to incorporate a large number of covariates, and these covariates can have highly varying relevance across individuals. Progress on this topic has been made in other fields [40], but it was noted that such considerations are particularly relevant to multimodal measures.

Session 2: Measuring Well-being

The second session focused on measuring QOL and well-being. This ever-growing field has seen proof of concepts for measures across a range of health-related QOL-relevant symptoms, including fatigue [1,41], depression [3,42,43], stress [2], anxiety [44], and independence [8,45], and significant resource is being invested to understand what “wellness” means for diverse populations [46].

Author AS of Rice University started the session with a keynote on multimodal sensor data analysis and modeling for health and well-being, discussing her vision for how measures can underpin decision support and behavior change interventions. Her examples included schizophrenia [47], mood, and stress [16]. She also discussed challenges in in-the-wild multimodal data modeling such as model personalization and adaptability to new users/patients and missing data.

Author BS of the University of California, San Diego then discussed current limitations in generalizability [48]. He proposed that the growth of personal sensor devices should enable us to augment classification by demographics and genetics, by including time series of physiology and behavior in our understanding of human diversity [17,18,49]. BS suggested developing algorithms that account for these dynamical differences, especially in health and wellness settings. Author FC of Cambridge Cognition then presented her work on the measurement of fatigue, which is increasingly understood to be a highly patient-relevant symptom across a large range of conditions [50-53]. She discussed the heterogeneity of the manifestations of fatigue, and their approach to combining active tests, voice biomarkers, and passive data collection to capture this complex symptom. Author CvH of imec then presented his work on digestible sensors and sensorized toilets for examining gut physiology. Finally, author SF of the Massachusetts Institute of Technology presented work on digital signals from wearables and smartphones, with application for the assessment of depression symptoms [22] and suicidal thoughts [23].

Overall, the panel discussion focused on measuring well-being as a whole versus specific aspects of QOL. While developing measures for specific aspects remains highly relevant for clinical development and treatment—for example, a measure of anxiety severity enables drug development and management of diagnosed individuals—personalized measures of general well-being have substantial application in public health and in engagement with individuals’ prediagnosis. Advancements to date have focused more on the former, as the relevance to the pharmaceutical industry is higher and the validation pathway is simpler [27]. The former also limits the diversity of experience captured and so frames an opportunity for reconceptualizing wellness, health, and QOL derived from broader participation in mapping individuals’ perceived needs. The panel also discussed whether it is possible and valuable to stratify mood predictions (ie, creating semipersonalized models where individuals with similar manifestations, personas, or journey stages are grouped together). Stratification based on objective behavioral data and digital signals can also advance our understanding of a condition by delineating commonalities across patients.

Key Discussion Points

Value of Multimodal Measures

Multimodal digital measures have expanded the number of possibilities for new ways to measure health by capturing an increasing number of proxies for multiple aspects of functions related to health. The panel emphasized that such measures are not a replacement for patient-reported outcomes but additional, complimentary tools to help understand the patients’ lived experience, ideally in a low burden and unobtrusive way. The research priority should therefore focus on measures that matter when defining patients’ health or general wellness. To achieve that, a 4-level sequential framework has been recently proposed by Manta et al [54] to evaluate meaningfulness of digital measures, namely, meaningful aspects of health, defining the aspect of a disease to address; specific and targeted concept of interest; outcome to be measured; and end point, including methodology and analysis plan to estimate patient improvement (eg, due to treatment).

While the majority of the research efforts are focusing on the definition and development of outcome measures, the adoption and investigation of these outcome measures in clinical trials as exploratory assessments is key to the development and validation of end points. The panel highlighted the rapidly expanding range of digital cognitive decline measures as an example and the need for the field to do more comparative studies [3] and patient-centric research [54] to focus efforts around the most meaningful and valuable candidate measures. Personalized or individual health trajectories were highlighted as potentially highly valuable, both to patients and to stakeholders outside of clinical development, for example, payer organizations exploring value-based agreements. Personalized health trajectories will require the possibility to define multiple health measures of interest, as no single measure will be equally relevant across individuals and across individual health journeys [54]. LF pointed to a key enabler being access to “healthy” data via monitoring of individuals prior to key events or diagnoses such that individualized baselines and, subsequently, individualized responses can be observed [8].

In the past decade, and accelerated by the widespread use of smartphones and other connected digital products, the use of digital products and devices in clinical trials has grown substantially, albeit primarily in observational studies and non–industry-funded clinical trials focusing on wellness [55]. The COVID-19 pandemic has by necessity further accelerated the adoption of digital health solutions for clinical research in the context of remote monitoring and telehealth [56].

Examples of the most advanced clinical applications of multimodal digital data are in Parkinson disease [57-60] and multiple sclerosis [60,61] with focus on motor function; cognitive decline in Alzheimer disease [62]; and diagnosis of depression [3], Friedreich’s ataxia [63], chronic obstructive pulmonary disease [60,64], and COVID-19 [7,65,66]. Interest in multimodal digital measures is also growing among early drug discovery researchers, where personalized medicine approaches can be enabled by capturing longitudinal information on patients behaviors and in real-world settings, sometimes referred to as “digital phenotyping” [67-70]. Indeed digital measures are seen as a new component of real-world data [71]; thus to drug discovery stakeholders, multimodal measures can also serve an important role by helping to bridge the gap between evidence generation in clinical development and late-phase studies.

Challenges Remaining

As the number of technologies and sources of digital health data increases, data integration and harmonization remain open challenges. The panelists identified three key obstacles that will need to be overcome to maintain momentum in the field.

First, slow and limited collaborative efforts in prioritizing data sharing will continue to hold back at-scale development and evaluation of novel digital measures and end points. Many companies are starting to realize the value of data sharing internally to their own walls [67], and increasingly, Findable, Accessible, Interoperable, Reusable data principles are becoming a core part of many data strategies [72]. Collaborations like the Innovative Medicines Initiative project RADAR-BASE [73-75] and the subsequent impact on a range of projects and application areas point to a possible path forward and the impact that precompetitive work in this space can have on productivity. Furthermore, multimodal sensor data is currently lacking broadly accepted and adopted common data models [76], which follow the example of other data types such as genomics and electronic health records, and have been a catalyst for progress in those fields; progress is being made [77], but more needs to be done to drive broad adoption [78]. Progress here will facilitate data integration, synchronization, and fusion that are often significant technical challenges at the individual study level when aligning and analyzing a network of connected devices [79]. A consequence of this is that substantial resources must be dedicated to technical challenges, slowing overall progress and innovation. The impact of better alignment on standards can be seen, for example, in the impact the Clinical Data Interchange Standards Consortium (CDISC) [80] has on submission data; thus it is important that collaborative efforts to make CDISC-compliant adaptations for digital health data are making progress [81]. Equally, progress on Fast Healthcare Interoperability Resources specifically on global standardization of data formats for digital health applications is encouraging [82].

Lastly, the rapid evolution of the digital health market and the short life cycle of wearables and connected devices [83,84] are challenges for data integration and reproducibility and generalization of analytical methodologies at the basis of digital measures and end points. Scaling innovation and efficient evaluation of new technologies and updated versions of hardware and software will require adherence to modular evaluation frameworks [85].

Future advances are expected from cross-industry initiatives to develop data platforms such as the Digital Medicine Society sensor integration initiative [78].

Conclusions and Path Forward

With the future of health care in mind, the panelists touched on a broad range of key takeaways. It is critical to incorporate practical, representative, and systematic approaches to involving patients in everyday health decisions [14]. Several examples discussed highlighted the importance of decision support systems or outcomes for clinical development and the value of early engagement with regulators in this space [86]. The panelists also discussed the significance in bridging the gap from measures to medicine: clinician confidence. Multimodal measures and continuous data capture are new concepts and have not been used by many practitioners, but these methods have the ability to contextualize observations and provide a direct connection to patients.

The workshop focused on sharing experiences and perspectives in the expanding use of multimodal data (multiple simultaneously collected objective data modalities, contextual information, and subjective inputs) to detect disease and capture complex outcomes. Across a wide range of examples, from infectious diseases to mental health and well-being, the speakers showcased the progress made and expressed optimism for future advancement and progression in the field.

Acknowledgments

We are grateful to the speakers who submitted their work to the workshop and took part in the panels, and to all attendees who contributed to such lively discussions. We also thank Paolo Bonato and the other conference organizers for their input and the chance to host the workshop.

Authors' Contributions

VDL and IC organized the workshop. All authors contributed to the conception and design of the article, drafted the article, and are responsible for final approval of the version to be published.

Conflicts of Interest

IC is an employee of the Digital Medicine Society. VDL is an employee of Novartis Pharma AG. BV is an employee of Byteflies. BS is a scientific advisor to and has an economic interest in OuraRing Inc. GG is an employee of the Department of Neuroscience, University of Padua Italy; he reports funding from the EU Horizon 2020–PD_Pal Grant 825785, and he owns stock in Sensedat srl. LF is cofounder of Evidation Health Inc. FL is an employee of F. Hoffmann-La Roche Ltd. AS has received travel reimbursement or honorarium payments from Gordon Research Conferences, Pola Chemical Industries, Leuven Mindgate, American Epilepsy Society, and IEEE. AS has also received research support from Microsoft, Sony Corporation, NEC Corporation, and Pola Chemicals and consulting fees from Gideon Health and Suntory Global Innovation Center. AS was paid by the European Science Foundation for a grant review.

References

1. Luo H, Lee PA, Clay I, Jaggi M, De Luca V. Assessment of fatigue using wearable sensors: a pilot study. Digit Biomark 2020;4(Suppl 1):59-72 [

References

Luo H, Lee PA, Clay I, Jaggi M, De Luca V. Assessment of fatigue using wearable sensors: a pilot study. Digit Biomark 2020;4(Suppl 1):59-72

Sano A, Taylor S, McHill AW, Phillips AJ, Barger LK, Klerman E, et al. Identifying objective physiological markers and modifiable behaviors for self-reported stress and mental health status using wearable sensors and mobile phones: observational study. J Med Internet Res 2018 Jun 08;20(6):e210

Sverdlov O, Curcic J, Hannesdottir K, Gou L, De Luca V, Ambrosetti F, et al. A study of novel exploratory tools, digital technologies, and central nervous system biomarkers to characterize unipolar depression. Front Psychiatry 2021;12:640741.

Abbas A, Sauder C, Yadav V, Koesmahargyo V, Aghjayan A, Marecki S, et al. Remote digital measurement of facial and vocal markers of major depressive disorder severity and treatment response: a pilot study. Front Digit Health 2021;3:610006

Vandendriessche B, Godfrey A, Izmailova ES. Multimodal biometric monitoring technologies drive the development of clinical assessments in the home environment. Maturitas 2021 Sep;151:41-47.

IEEE BHI-BSN-2021. URL: https://www.bhi-bsn-2021.org/ [accessed 2021-08-03]

Shapiro A, Marinsek N, Clay I, Bradshaw B, Ramirez E, Min J, et al. Characterizing COVID-19 and influenza illnesses in the real world via person-generated health data. Patterns (N Y) 2021 Jan 08;2(1):100188

Karas M, Marinsek N, Goldhahn J, Foschini L, Ramirez E, Clay I. Predicting subjective recovery from lower limb surgery using consumer wearables. Digit Biomark 2020;4(Suppl 1):73-86

Lipsmeier F, Taylor KI, Kilchenmann T, Wolf D, Scotland A, Schjodt-Eriksen J, et al. Evaluation of smartphone-based testing to generate exploratory outcome measures in a phase 1 Parkinson's disease clinical trial. Mov Disord 2018 Aug;33(8):1287-1297

Midaglia L, Mulero P, Montalban X, Graves J, Hauser SL, Julian L, et al. Adherence and satisfaction of smartphone- and smartwatch-based remote active testing and passive monitoring in people with multiple sclerosis: nonrandomized interventional feasibility study. J Med Internet Res 2019 Aug 30;21(8):e14863

Dan J, Vandendriessche B, Paesschen WV, Weckhuysen D, Bertrand A. Computationally-efficient algorithm for real-time absence seizure detection in wearable electroencephalography. Int J Neural Syst 2020 Nov;30(11):2050035.

Swinnen L, Chatzichristos C, Jansen K, Lagae L, Depondt C, Seynaeve L, et al. Accurate detection of typical absence seizures in adults and children using a two-channel electroencephalographic wearable behind the ears. Epilepsia 2021 Nov;62(11):2741-2752.

Fava M, Evins AE, Dorer D, Schoenfeld D. The problem of the placebo response in clinical trials for psychiatric disorders: culprits, possible remedies, and a novel study design approach. Psychother Psychosom 2003;72(3):115-127.

Fernandes BS, Williams LM, Steiner J, Leboyer M, Carvalho AF, Berk M. The new field of 'precision psychiatry'. BMC Med 2017 Apr 13;15(1):80

Carvalho AF, Solmi M, Sanches M, Machado MO, Stubbs B, Ajnakina O, et al. Evidence-based umbrella review of 162 peripheral biomarkers for major mental disorders. Transl Psychiatry 2020 May 18;10(1):152.

Li B, Sano A. Extraction and interpretation of deep autoencoder-based temporal features from wearables for forecasting personalized mood, health, and stress. Proc ACM Interactive Mobile Wearable Ubiquitous Technologies 2020 Jun 15;4(2):1-26.

Smarr BL, Ishami AL, Schirmer AE. Lower variability in female students than male students at multiple timescales supports the use of sex as a biological variable in human studies. Biol Sex Differ 2021 Apr 22;12(1):32

Smarr BL, Aschbacher K, Fisher SM, Chowdhary A, Dilchert S, Puldon K, et al. Feasibility of continuous fever monitoring using wearable devices. Sci Rep 2020 Dec 14;10(1):21640.

Cormack FK, Barnett JH, Taptiklis N. Acoustic features of voice as a measure of cognitive load during performance of serial subtraction in a remote data collection context. Alzheimer Dementia 2021 Dec 31;17(S11):e056271.

van Kraaij AWJ, Schiavone G, Lutin E, Claes S, Van Hoof C. Relationship between chronic stress and heart rate over time modulated by gender in a cohort of office workers: cross-sectional study using wearable technologies. J Med Internet Res 2020 Sep 09;22(9):e18253

Smets E, Rios Velazquez E, Schiavone G, Chakroun I, D'Hondt E, De Raedt W, et al. Large-scale wearable data reveal digital phenotypes for daily-life stress detection. NPJ Digit Med 2018;1:67.

Ghandeharioun A, Fedor S, Sangermano L, Ionescu D, Alpert J, Dale C, et al. Objective assessment of depressive symptoms with machine learning and wearable sensors data. 2017 Presented at: Seventh International Conference on Affective Computing and Intelligent Interaction; October 23-26, 2017; San Antonio, TX.

Kleiman EM, Turner BJ, Fedor S, Beale EE, Picard RW, Huffman JC, et al. Digital phenotyping of suicidal thoughts. Depress Anxiety 2018 Jul;35(7):601-608.

Ngiam J, Khosla A, Kim M, Nam J, Lee H, Ng AY. Multimodal deep learning. In: Proceedings of the 28th International Conference on International Conference on Machine Learning. 2011 Presented at: ICML'11; June 28-July 2, 2011; Bellevue, WA p. 689-696 URL: https://dl.acm.org/doi/10.5555/3104482.3104569

Health-related quality of life (HRQOL). Centers for Disease Control and Prevention. 2021. URL: https://www.cdc.gov/hrqol/index.htm [accessed 2021-12-17]

Coravos A, Goldsack JC, Karlin D, Nebeker C, Perakslis E, Zimmerman N, et al. Digital medicine: a primer on measurement. Digit Biomark 2019;3(2):31-71

Goldsack JC, Dowling AV, Samuelson D, Patrick-Lake B, Clay I. Evaluation, acceptance, and qualification of digital measures: from proof of concept to endpoint. Digit Biomark 2021;5(1):53-64

Brönneke JB, Debatin JF, Hagen J, Kircher P, Matthies H, Stern AD. DiGA VADEMECUM: How to Launch Digital Health Apps in the German Healthcare System. Berlin: MWV; 2021.

Maricich YA, Bickel WK, Marsch LA, Gatchalian K, Botbyl J, Luderer HF. Safety and efficacy of a prescription digital therapeutic as an adjunct to buprenorphine for treatment of opioid use disorder. Curr Med Res Opin 2021 Feb;37(2):167-173

Tudor Car L, Dhinagaran DA, Kyaw BM, Kowatsch T, Joty S, Theng Y, et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res 2020 Aug 07;22(8):e17158

Determining real-world data’s fitness for use and the role of reliability. Margolis Center for Health Policy. 2019. URL: https://healthpolicy.duke.edu/sites/default/files/2019-11/rwd_reliability.pdf [accessed 2021-08-23]

Montalban X, Graves J, Midaglia L, Mulero P, Julian L, Baker M, et al. A smartphone sensor-based digital outcome assessment of multiple sclerosis. Mult Scler 2022 Apr;28(4):654-664

Mueller A, Hoefling HA, Muaremi A, Praestgaard J, Walsh LC, Bunte O, et al. Continuous digital monitoring of walking speed in frail elderly patients: noninterventional validation study and longitudinal clinical trial. JMIR Mhealth Uhealth 2019 Nov 27;7(11):e15191

Kaye J. Public Policy Aging Rep 2017;27(2):53-61

Gwaltney C, Coons SJ, O'Donohoe P, O'Gorman H, Denomey M, Howry C, et al. "Bring Your Own Device" (BYOD): the future of field-based patient-reported outcome data collection in clinical trials? Ther Innov Regul Sci 2015 Nov;49(6):783-791.

Byrom B, Gwaltney C, Slagle A, Gnanasakthy A, Muehlhausen W. Measurement equivalence of patient-reported outcome measures migrated to electronic formats: a review of evidence and recommendations for clinical trials and bring your own device. Ther Innov Regul Sci 2019 Jul;53(4):426-430.

Byrom B, Doll H, Muehlhausen W, Flood E, Cassedy C, McDowell B, et al. Measurement equivalence of patient-reported outcome measure response scale types collected using bring your own device compared to paper and a provisioned device: results of a randomized equivalence trial. Value Health 2018 May;21(5):581-589

Houts CR, Patrick-Lake B, Clay I, Wirth RJ. The path forward for digital measures: suppressing the desire to compare apples and pineapples. Digit Biomark 2020;4(Suppl 1):3-12

Voukelatou V, Gabrielli L, Miliou I, Cresci S, Sharma R, Tesconi M, et al. Measuring objective and subjective well-being: dimensions and data sources. Int J Data Sci Analytics 2020 Jun 29;11(4):279-309.

de Jong VMT, Moons KGM, Eijkemans MJC, Riley RD, Debray TPA. Developing more generalizable prediction models from pooled studies and large clustered data sets. Stat Med 2021 Jul 10;40(15):3533-3559

IDEA-FAST. URL: https://idea-fast.eu/ [accessed 2021-08-03]

Makhmutova M, Kainkaryam R, Ferreira M, Min J, Jaggi M, Clay I. Prediction of self-reported depression scores using person-generated health data from a virtual 1-year mental health observational study. In: Proceedings of the 2021 Workshop on Future of Digital Biomarkers. 2021 Presented at: DigiBiom '21; June 25, 2001; Virtual p. 4-11.

Jacobson NC, Weingarden H, Wilhelm S. Digital biomarkers of mood disorders and symptom change. NPJ Digit Med 2019;2:3.

Jacobson NC, Summers B, Wilhelm S. Digital biomarkers of social anxiety severity: digital phenotyping using passive smartphone sensors. J Med Internet Res 2020 May 29;22(5):e16875 https://www.jmir.org/2020/5/e16875/.

Bahej I, Clay I, Jaggi M, De Luca V. Prediction of patient-reported physical activity scores from wearable accelerometer data: a feasibility study. In: Masia L, Micera S, Akay M, Pons JL, editors. Converging Clinical and Engineering Research on Neurorehabilitation III: Proceedings of the 4th International Conference on NeuroRehabilitation (ICNR2018), October 16-20, 2018, Pisa, Italy. Cham: Springer; 2019:668-672.

All of Us Research Program. 2020. URL: https://allofus.nih.gov/future-health-begins-all-us [accessed 2021-12-17]

Tseng VW, Sano A, Ben-Zeev D, Brian R, Campbell AT, Hauser M, et al. Using behavioral rhythms and multi-task learning to predict fine-grained symptoms of schizophrenia. Sci Rep 2020 Sep 15;10(1):15100.

Futoma J, Simons M, Panch T, Doshi-Velez F, Celi LA. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit Health 2020 Sep;2(9):e489-e492

Grant AD, Newman M, Kriegsfeld LJ. Ultradian rhythms in heart rate variability and distal body temperature anticipate onset of the luteinizing hormone surge. Sci Rep 2020 Nov 23;10(1):20378.

Elefante E, Tani C, Stagnaro C, Ferro F, Parma A, Carli L, et al. Impact of fatigue on health-related quality of life and illness perception in a monocentric cohort of patients with systemic lupus erythematosus. RMD Open 2020 Feb;6(1):e001133

Rodríguez Antolín A, Martínez-Piñeiro L, Jiménez Romero ME, García Ramos JB, López Bellido D, Muñoz Del Toro J, et al. Prevalence of fatigue and impact on quality of life in castration-resistant prostate cancer patients: the VITAL study. BMC Urol 2019 Oct 16;19(1):92

McCabe RM, Grutsch JF, Braun DP, Nutakki SB. Fatigue as a driver of overall quality of life in cancer patients. PLoS One 2015;10(6):e0130023

Eaton-Fitch N, Johnston SC, Zalewski P, Staines D, Marshall-Gradisnik S. Health-related quality of life in patients with myalgic encephalomyelitis/chronic fatigue syndrome: an Australian cross-sectional study. Qual Life Res 2020 Jun;29(6):1521-1531

Manta C, Patrick-Lake B, Goldsack JC. Digital measures that matter to patients: a framework to guide the selection and development of digital measures of health. Digit Biomark 2020;4(3):69-77

Marra C, Chen JL, Coravos A, Stern AD. Quantifying the use of connected digital products in clinical research. NPJ Digit Med 2020;3:50.

Marra C, Gordon WJ, Stern AD. Use of connected digital products in clinical research following the COVID-19 pandemic: a comprehensive analysis of clinical trials. BMJ Open 2021 Jun 22;11(6):e047341

Viceconti M, Hernandez Penna S, Dartee W, Mazzà C, Caulfield B, Becker C, et al. Toward a regulatory qualification of real-world mobility performance biomarkers in Parkinson's patients using digital mobility outcomes. Sensors (Basel) 2020 Oct 20;20(20):5920

Morgan C, Craddock I, Tonkin E, Kinnunen K, McNaney R, Whitehouse S, et al. Protocol for PD SENSORS: Parkinson's disease symptom evaluation in a naturalistic setting producing outcome measures using SPHERE technology. An observational feasibility study of multi-modal multi-sensor technology to measure symptoms and activities of daily living in Parkinson's disease. BMJ Open 2020 Nov 30;10(11):e041303

Merola A, Sturchio A, Hacker S, Serna S, Vizcarra JA, Marsili L, et al. Technology-based assessment of motor and nonmotor phenomena in Parkinson disease. Expert Rev Neurother 2018 Nov;18(11):825-845.

Rochester L, Mazzà C, Mueller A, Caulfield B, McCarthy M, Becker C, et al. A roadmap to inform development, validation and approval of digital mobility outcomes: the Mobilise-D approach. Digit Biomark 2020;4(Suppl 1):13-27

Woelfle T, Pless S, Wiencierz A, Kappos L, Naegelin Y, Lorscheider J. Practice effects of mobile tests of cognition, dexterity, and mobility on patients with multiple sclerosis: data analysis of a smartphone-based observational study. J Med Internet Res 2021 Nov 18;23(11):e30394

Muurling M, de Boer C, Kozak R, Religa D, Koychev I, Verheij H, RADAR-AD Consortium. Remote monitoring technologies in Alzheimer's disease: design of the RADAR-AD study. Alzheimers Res Ther 2021 Apr 23;13(1):89

Mueller A, Paterson E, McIntosh A, Praestgaard J, Bylo M, Hoefling H, et al. Digital endpoints for self-administered home-based functional assessment in pediatric Friedreich's ataxia. Ann Clin Transl Neurol 2021 Sep;8(9):1845-1856.

Rahman MJ, Nemati E, Rahman M, Vatanparvar K, Nathan V, Kuang J. Toward early severity assessment of obstructive lung disease using multi-modal wearable sensor data fusion during walking. Annu Int Conf IEEE Eng Med Biol Soc 2020 Jul;2020:5935-5938.

Ni X, Ouyang W, Jeong H, Kim J, Tzaveils A, Mirzazadeh A, et al. Automated, multiparametric monitoring of respiratory biomarkers and vital signs in clinical and home settings for COVID-19 patients. Proc Natl Acad Sci U S A 2021 May 11;118(19):e2026610118

Quer G, Radin JM, Gadaleta M, Baca-Motes K, Ariniello L, Ramos E, et al. Wearable sensor data and self-reported symptoms for COVID-19 detection. Nat Med 2021 Jan;27(1):73-77.

Hartl D, De Luca V, Kostikova A, Laramie J, Kennedy S, Ferrero E, et al. Translational precision medicine: an industry perspective. J Transl Med 2021 Jun 05;19(1):245

Insel T. Digital phenotyping: technology for a new science of behavior. JAMA 2017 Oct 03;318(13):1215-1216.

Huckvale K, Venkatesh S, Christensen H. Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit Med 2019;2:88.

Fagherazzi G. Deep digital phenotyping and digital twins for precision health: time to dig deeper. J Med Internet Res 2020 Mar 03;22(3):e16770

Dagenais S, Russo L, Madsen A, Webster J, Becnel L. Use of real-world evidence to drive drug development strategy and inform clinical trial design. Clin Pharmacol Ther 2022 Jan;111(1):77-89.

Wise J, de Barron AG, Splendiani A, Balali-Mood B, Vasant D, Little E, et al. Drug Discov Today 2019 Apr;24(4):933-938

RADAR-base. GitHub. URL: https://github.com/RADAR-base [accessed 2022-02-07]

RADAR-base. 2018. URL: https://radar-base.org/ [accessed 2022-02-07]

Ranjan Y, Rashid Z, Stewart C, Conde P, Begale M, Verbeeck D, Hyve, RADAR-CNS Consortium. RADAR-Base: open source mobile health platform for collecting, monitoring, and analyzing data using sensors, wearables, and mobile devices. JMIR Mhealth Uhealth 2019 Aug 01;7(8):e11734

OMOP Common Data Model. Observational Health Data Sciences and Informatics. URL: https://www.ohdsi.org/data-standardization/the-common-data-model/ [accessed 2021-12-17]

IEEE Standard for Open Mobile Health Data--representation of metadata, sleep, and physical activity measures. IEEE Standards Association. URL: https://standards.ieee.org/standard/1752_1-2021.html [accessed 2021-12-21]

Clay I, Angelopoulos C, Bailey AL, Blocker A, Carini S, Carvajal R, et al. Sensor data integration: a new cross-industry collaboration to articulate value, define needs, and advance a framework for best practices. J Med Internet Res 2021 Nov 09;23(11):e34493

Mazzà C, Alcock L, Aminian K, Becker C, Bertuletti S, Bonci T, et al. Technical validation of real-world monitoring of gait: a multicentric observational study. BMJ Open 2021 Dec 02;11(12):e050785

Facile R, Muhlbradt EE, Gong M, Li Q, Popat V, Pétavy F, et al. Use of Clinical Data Interchange Standards Consortium (CDISC) standards for real-world data: expert perspectives from a qualitative Delphi survey. JMIR Med Inform 2022 Jan 27;10(1):e30363

Hume S, Aerts J, Sarnikar S, Huser V. Current applications and future directions for the CDISC Operational Data Model standard: a methodological review. J Biomed Inform 2016 Apr;60:352-362

Lehne M, Luijten S, Vom Felde Genannt Imbusch P, Thun S. The use of FHIR in digital health - a review of the scientific literature. Stud Health Technol Inform 2019 Sep 03;267:52-58.

Gurova O, Merritt T, Papachristos E, Vaajakari J. Sustainable solutions for wearable technologies: mapping the product development life cycle. Sustainability 2020 Oct 14;12(20):8444.

Brönneke JB, Müller J, Mouratis K, Hagen J, Stern AD. Regulatory, legal, and market aspects of smart wearables for cardiac monitoring. Sensors (Basel) 2021 Jul 20;21(14):4937

Goldsack JC, Coravos A, Bakker J, Bent B, Dowling A, Fitzer-Attas C, et al. Verification, analytical validation, and clinical validation (V3): the foundation of determining fit-for-purpose for Biometric Monitoring Technologies (BioMeTs). NPJ Digit Med 2020;3:55.

Cerreta F, Ritzhaupt A, Metcalfe T, Askin S, Duarte J, Berntgen M, et al. Digital technologies for medicines: shaping a framework for success. Nat Rev Drug Discov 2020 Sep;19(9):573-574.