Promoting mental health and wellbeing among post-secondary students with the JoyPop™ app: study protocol for a randomized controlled trial

Abstract

Background

Technology use may be one strategy to promote mental health and wellbeing among young adults in post-secondary education settings experiencing increasing distress and mental health difficulties. The JoyPop™ app is mobile mental health tool with a growing evidence base. The objectives of this research are to (1) evaluate the effectiveness of the JoyPop™ app in improving emotion regulation skills (primary outcome), as well as mental health, wellbeing, and resilience (secondary outcomes); (2) evaluate sustained app use once users are no longer reminded and determine whether sustained use is associated with maintained improvements in primary and secondary outcomes; (3) determine whether those in the intervention condition have lower mental health service usage and associated costs compared to those in the control condition; and (4) assess users’ perspectives on the quality of the JoyPop™ app.

Methods

A pragmatic, parallel arm randomized controlled trial will be used. Participants will be randomly allocated using stratified block randomization in a 1:1 ratio to the intervention (JoyPop™) or control (no intervention) condition. Participants allocated to the intervention condition will be asked to use the JoyPop™ app at least twice daily for 4 weeks. Participants will complete outcome measures at four assessment time-points (first [baseline], second [after 2 weeks], third [after 4 weeks], fourth [after 8 weeks; follow-up]). Participants in the control condition will be offered access to the app after the fourth assessment time-point.

Discussion

Results will determine the effectiveness of the JoyPop™ app for promoting mental health and wellbeing among post-secondary students. If effective, this may encourage more widespread adoption of the JoyPop™ app by post-secondary institutions as part of their response to student mental health needs.

Trial registration

ClinicalTrials.gov NCT06154369. Registered on November 23, 2023.

Administrative information

Note: the numbers in curly brackets in this protocol refer to SPIRIT checklist item numbers. The order of the items has been modified to group similar items (see http://www.equator-network.org/reporting-guidelines/spirit-2013-statement-defining-standard-protocol-items-for-clinical-trials/).

Title {1}	Promoting mental health and wellbeing among post-secondary students with the JoyPop™ app: study protocol for a randomized controlled trial
Trial registration {2a and 2b}	ClinicalTrials.gov NCT06154369. Registered on November 23, 2023.
Protocol version {3}	August 9, 2024; version 2
Funding {4}	Brain Canada—Future Leaders in Canadian Brain Research Program
Author details {5a}	AM: Department of Psychology, Lakehead University VM: Department of Psychology, Lakehead University ET: Department of Psychology, Lakehead University FS: Department of Psychology, Lakehead University; Children’s Centre Thunder Bay JO: Department of Psychology, University of New Brunswick SHS: Departments of Psychiatry and Psychology & Neuroscience, Dalhousie University AN: Department of Pediatrics, University of Alberta AO: School of Public Health, University of Alberta ARM: Department of Psychology, Lakehead University
Name and contact information for the trial sponsor {5b}	Aislin R. Mushquash Department of Psychology, Lakehead University 955 Oliver Road, Thunder Bay, Ontario, P7B5E1 Email: [email protected]
Role of sponsor {5c}	The funder has no role in the study design; data collection, management, or analysis; or in the writing and publishing of reports.

Introduction

Background and rationale {6a}

Canadian youth and young adults have experienced an increase in mental health difficulties over the past several years [1, 2]. Post-secondary students in this age group are no exception, with the number of students with mental health symptoms and diagnoses significantly increasing between 2013 and 2019 according to population-level data [3]. Further, symptoms of anxiety, depression, and stress rose among Canadian post-secondary students compared to other young adults during the COVID-19 pandemic [4–7], though understanding the longer-term outcomes is still evolving. Service utilization has also increased among youth and young adults, including counseling services and hospital visits [3, 8]. While most Canadian post-secondary institutions offer campus mental health supports to address these concerns [9], specific services vary by institution, and formal assessment, service evaluation, and early identification procedures are sometimes lacking [10, 11]. Many students believe their institution should increase mental health resources [12], while some young adults perceive barriers to using mental health services including perceived time constraints or concerns about privacy and stigma [13, 14].

Mobile apps can increase access to and engagement in mental health-related interventions [15]. Many apps exist, with meta-analyses and systematic reviews suggesting mental health-focused apps improve mental health-related outcomes such as depression, anxiety, stress, and quality of life, with small to large effects [16–18]. Findings also generally suggest that post-secondary students are open to using apps to address their mental health concerns [19]. While promising in their impact, mobile mental health apps are heterogenous in terms of their specific interventions (e.g., mindfulness techniques, stress reduction), and there are many apps that have not been adequately evaluated [20]. For these reasons, it is imperative to build the evidence base for any new app prior to widespread implementation.

The JoyPop™ app was co-designed for youth and young adults by researchers, youth, and service providers [21]. It contains features (see Fig. 1) that promote wellbeing and resilience by targeting emotion regulation [22]. Considered a transdiagnostic risk factor, emotion regulation is linked to many mental health difficulties such as anxiety and depression and is directly related to one’s ability to cope with stress [23], thus rendering it a relevant target for intervention among post-secondary students. Understanding and managing one’s emotions is also linked to better academic performance [24, 25], further emphasizing the importance of emotion regulation for students.

Fig. 1 [Images not available. See PDF.]

The JoyPop app

Initial study of the JoyPop™ app was conducted with first-year university students to understand whether the app would support them in terms of the stressors and challenges associated with transitioning to university [22]. Results demonstrated positive changes in emotion regulation and depressive symptoms with each additional day of app usage over a 4-week period, suggesting those who used the app for more days experienced greater benefit [22]. In qualitative studies examining the acceptability and utility of the app, users expressed that the app had several positive outcomes such as increasing self-monitoring and awareness, improving emotion regulation, and providing helpful distraction [26, 27].

While promising, prior research has lacked a control group of participants who do not receive access to the app to more rigorously evaluate mental health and wellbeing outcomes associated with the JoyPop™ app. Study of whether app benefits are sustained when users are no longer reminded to use the JoyPop™ app is also needed. Additionally, while smaller samples have qualitatively provided feedback on the acceptability and quality of the app, quantitative metrics assessing perceptions of app quality from a larger sample would provide a more robust understanding of the user experience. Lastly, while the JoyPop™ app is theorized to support student mental health needs, formal evaluations of whether the app reduces the need for, and cost associated with, other mental health services have not been conducted. The present study aims to address these gaps via a randomized controlled trial design.

Objectives {7}

The first objective of this research is to evaluate the effectiveness of the JoyPop™ app in improving post-secondary students’ emotion regulation skills (primary outcome), as well as their mental health, wellbeing, and resilience (secondary outcomes). Prior research suggests that the JoyPop™ app will be associated with positive change in emotion regulation and mental health over time as compared to no intervention. We also expect app use to result in improved resilience and wellbeing as the app was designed to support adaptation and perseverance in the face of stress [21, 22, 28]. The second, third, and fourth objectives are more exploratory. The second objective is to evaluate sustained app use once users are no longer reminded and determine whether sustained use is associated with maintained benefit in outcomes. The third objective is to determine whether those using the app have lower mental health service usage and associated costs. The fourth objective is to assess users’ perspectives on the quality of the JoyPop™ app.

Trial design {8}

This study design is a pragmatic, parallel arm randomized controlled superiority trial. Stratified block randomization will be used to randomize participants in a 1:1 ratio to either the intervention (JoyPop™) or control (no intervention) condition. Those in the intervention condition will be asked to use the JoyPop™ app at least twice per day for 4 weeks. Primary (emotion regulation) and secondary (mental health, wellbeing, and resilience) outcomes and mental health service usage (number of visits to various service providers) will be assessed among participants at four assessments time-points (first [baseline], second [after 2 weeks], third [after 4 weeks], fourth [after 8 weeks; follow-up]). Participants in the intervention condition will complete measures of app quality at the third assessment time-point. The Standard Protocol Items: Recommendations for Intervention Trials (SPIRIT) guidelines are being followed. The trial was registered with ClinicalTrials.gov (NCT06154369). All procedures have been reviewed and approved by the Research Ethics Board at the Thunder Bay Regional Health Sciences Centre (acting as the Board of Record for Lakehead University clinical research projects).

Methods: participants, interventions, and outcomes

Study setting {9}

All data will be collected at Lakehead University, a public university with a main campus in Thunder Bay, Ontario, Canada, and a smaller campus in Orillia, Ontario, Canada. Participants will be drawn from the undergraduate student population of approximately 7000 students across both campuses. The first (baseline) assessment time-point will be on-campus in a laboratory setting or virtually via Zoom depending on student preference, whether the student requires use of an iPhone to take part in the study (which would necessitate an on-campus visit to secure the phone), and depending on their location (i.e., those in Orillia will occur via Zoom). Subsequent assessment time-points will be administered online.

Eligibility criteria {10}

Participants are required to be between the ages of 18–25 years, enrolled as an undergraduate student at Lakehead University, and not have participated in any other study that evaluated the JoyPop™ app. Students from the Orillia campus will require access to their own iOS device.

Who will take informed consent? {26a}

Students will attend a study orientation session either in-person or virtually, during which time a research assistant will explain the study and answer questions. Students will also receive an information letter and consent form and will be given time to review and ask any questions of the research assistant before proceeding. Undergraduate research assistants will have undergone training including shadowing the graduate student researcher and being observed to ensure consistency in protocol administration. All research team members will also have completed the Canadian Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans training.

Additional consent provisions for collection and use of participant data and biological specimens {26b}

N/A. No additional consent provisions are needed, and no biological samples will be collected.

Interventions

Explanation for the choice of comparators {6b}

The control condition (no intervention) was chosen to account for natural changes over time and regression to the mean expected in outcomes such as emotion regulation [29]. Comparing use of the JoyPop™ app to receiving no intervention creates the opportunity to test whether those who receive the app experience improved outcomes compared to those who do not receive it.

Intervention description {11a}

Participants in the intervention group will download and use the JoyPop™ app. Participants will be asked to use the app twice daily for 4 weeks, with no other requirements specifying their usage. The JoyPop™ app features are Rate My Mood, SleepEase, Breathing Exercises, SquareMoves, Journaling, Art, Circle of Trust, and Call for Help (see Fig. 1). Participants from the Thunder Bay campus who do not have an iOS device will be given a restricted-access iPhone to borrow for the study, with only the JoyPop™ app enabled.

Individuals in the control condition will not receive any intervention.

Criteria for discontinuing or modifying allocated interventions {11b}

Participants may request to withdraw from the study at any time by contacting the research team via email; otherwise, there are no pre-specified criteria for modifying or discontinuing the intervention.

Strategies to improve adherence to interventions {11c}

Those in the intervention condition will receive a text message each morning and evening (8 am and 8 pm) reminding them to use the app.

Relevant concomitant care permitted or prohibited during the trial {11d}

Participants will not be prohibited from accessing any concomitant care or other interventions throughout the study. The use of other mental health services will be evaluated as part of the third study objective to determine whether those who received the app have lower mental health service usage and associated costs.

Provisions for post-trial care {30}

While no harm associated with study participation is expected, there is a risk participants may feel distressed answering questions related to emotion regulation, mental health, and wellbeing. The information letter will include contact information for mental health support services, including local counseling services and telephone mental health hotlines.

Outcomes {12}

Primary outcomes

The primary outcome is change in emotion regulation over time. Emotion regulation will be measured at each of the four assessment time-points using the Difficulties in Emotion Regulation Scale—Short Form (DERS-SF) [30, 31]. We will examine change in overall emotion regulation via the total DERS-SF score and change in specific domains via the DERS-SF subscales (i.e., strategy use related to emotion regulation, non-acceptance of emotion, impulsivity, focus on goals, awareness of emotion, and clarity of emotion). Evaluating emotion regulation is an important outcome as it is a transdiagnostic risk factor relevant across many mental health difficulties [23] and as it is related to academic success [24, 25].

Secondary outcomes

Secondary outcomes are change in mental health, wellbeing, and resilience over time. Each of these variables will be measured at each of the four assessment time-points. Mental health, consisting of psychological distress, depressive symptoms, anxious symptoms, and stress, will be measured with the Depression, Anxiety, and Stress Scale 21 (DASS-21) [32]. Psychological distress will be assessed by the total score on the DASS-21, while depression, anxiety, and stress symptoms are measured by their respective subscale scores. General mental health will be measured using the total score from the General Health Questionnaire 12 (GHQ-12) [33]. These aspects of mental health are important to assess as many students face mental health difficulties and prior research suggests the JoyPop™ app may impact mental health symptoms [22].

Wellbeing will be measured using the total score from the Warwick Edinburgh Mental Wellbeing Scale [34, 35]. It is important to measure the impact of the JoyPop™ app on wellbeing as it is theorized to represent an element of mental functioning that is partially independent from the presence/absence of mental illness among Canadian students [28, 36]. Resilience will be measured using the Connor-Davidson Resilience Scale-10 (CD-RISC-10) [37]. Resilience is important as the ability to adapt and thrive in the face of stress and adversity [38] is relevant to students dealing with stressors. The JoyPop™ app was designed with resilience theory in mind [21].

Exploratory data

Primary and secondary outcomes will be assessed at the fourth assessment time-point as part of the exploratory objective of evaluating sustained app use and determining whether this use is associated with maintained benefit in primary and secondary outcomes. Self-reported mental health service utilization will be measured at each of the four time-points using a brief measure developed by the research team. Change in both total frequency of health service use and individual service provider visits will be calculated.

App quality will be evaluated during the third assessment time-points among those using the intervention. Quality will be assessed using the User Version of the Mobile Application Rating Scale (UMARS) [39, 40]. Total scores and subscale scores (i.e., engagement, functionality, esthetics, information, and perceived impact) will be calculated. App quality is relevant because a smartphone intervention is effective to the extent that the user is willing to engage with it and continues to use the app over time [41]. Thus, the target population for the JoyPop™ app must experience it being good enough quality to use and benefit from. Using the UMARS will allow us to compare app quality for the JoyPop™ app relative to other similar apps for which UMARS data have been published (e.g., Thabrew et al. [42]).

Participant timeline {13}

See Fig. 2.

Fig. 2 [Images not available. See PDF.]

SPIRIT schedule of enrollment, intervention, and assessments. *Week 0 (enrollment) includes any time prior to starting the study. **Intervention group only

Sample size {14}

Our past research suggests a medium effect for the primary outcome [22]. A sample size of 120 will be required to achieve power to detect a small to medium effect in the primary outcome when using a 2 × 4 (group by assessment time-point) design [43]. We had good retention in our past research, with 79% of participants completing outcome measures at baseline, after 2 weeks of app use, and after 4 weeks of app use [22]. However, our prior research did not involve a control group or a follow-up assessment after 8 weeks, both of which are likely to impact retention. Other app-based trials show retention rates between 65 and 80% [44]. Consistency and long-term engagement with apps outside of a research context is low, with 25% of users abandoning apps after one use [45]. Considering this information, we conservatively estimated 60% retention throughout the study (40% attrition). As such, we will recruit an initial sample of 160.

Recruitment {15}

Potential participants will be recruited through advertisements on social media (e.g., Facebook, Instagram), emails to students sent by course instructors, announcements made in classes, flyers distributed to students on campus, and posters displayed around campus. Advertisements will contain a QR code directing interested participants to an online initial interest survey which collects their basic contact information and eligibility information so that the research team can initiate contact over email or phone for the eligibility screen. Interested students can alternatively email the research team expressing interest in participating. Students enrolled in psychology classes at Lakehead University can obtain bonus points for participating in research studies; as such, we will advertise on the university’s online research sign-up system.

Assignment of interventions: allocation

Sequence generation {16a}

Stratified block randomization will be used to randomly assign participants to either the intervention condition or control condition in a 1:1 ratio.

Concealment mechanism {16b}

Random assignments based on the allocation sequence will be placed into numbered, opaque, sealed envelopes that will be kept in a locked filing cabinet. After a student consents to participate and completes the measures, the research assistant will access the next sealed envelope.

Implementation {16c}

A researcher not involved in the trial will use a computer-generated sequencing tool to create the allocation sequence [46]. During the first assessment time-point, the research assistant will access the allocation envelope from the locked filing cabinet, open it to determine the allocation, inform the participant, and proceed with associated study tasks.

Assignment of interventions: blinding

Who will be blinded {17a}

Trial participants themselves cannot be blinded to their condition due to the nature of the intervention and control conditions (i.e., using an app vs. no intervention). Research assistants who directly interact with participants will also not be blinded. The principal investigator and co-investigators will be blinded to condition assignment.

Procedure for unblinding if needed {17b}

If an adverse event were to occur, the participant’s allocation may be revealed to the principal investigator by the research assistant and/or by the participant.

Data collection and management

Plans for assessment and collection of outcomes {18a}

Assessment data will be collected online through SurveyMonkey [47]. During the orientation session, the research assistant will explain how to complete the assessments and participants can ask questions if they are unsure of how to answer a question. The research assistant will also explain that participants can expect to see similar measures at each subsequent assessment time-point. In addition to data collection through SurveyMonkey, the JoyPop™ app records usage data indicating when and for how long the app was accessed by participants. This will be used to measure sustained app use for the exploratory analysis of whether benefits in primary and secondary outcomes are maintained during the follow-up period.

Demographic information will be collected at the first assessment time-point using five items assessing age, race/ethnicity, gender identity, biological sex, and current year of study in university.

Emotion regulation will be measured with the DERS-SF [30, 31], an 18-item measure that asks participants to rate how much various statements regarding difficulty regulating emotion have applied to them over the last 2 weeks. Items are rated on a scale from 1—“Almost Never” to 5—“Almost Always.” Total scores range from 18 to 90, with higher scores indicating greater difficulty regulating one’s emotions. Subscales measure strategies (e.g., “When I was upset, it took me a long time to feel better”), non-acceptance (e.g., “When I was upset, I felt guilty for feeling that way”), impulse (e.g., “When I was upset, I lost control over my behaviour”), goals (e.g., “When I was upset, I had difficulty focusing on other things”), awareness (e.g., “I cared about what I was feeling”), and clarity (e.g., “I was confused about how I feel”). The DERS-SF is positively correlated with symptoms of emotional disorders and demonstrates high internal consistency [31].

Mental health data will be collected using the DASS-21 [32], a 21-item measure that asks participants to rate how much various statements have been relevant for them in the last week. Items are rated on a scale from 0—“Did not apply to me at all (never)” to 3—“Applied to me very much, or most of the time (almost always).” Total scores range from 0 to 63, with higher scores indicating greater psychological distress. In addition to psychological distress, subscales include depressive symptoms (e.g., “I felt that I had nothing to look forward to”), anxiety symptoms (e.g., “I felt I was close to panic”), and stress (e.g., “I found it difficult to relax”). The DASS-21 is positively correlated with other measures of depression, anxiety, and stress in university students and demonstrates high internal consistency [48]. The GHQ-12 [26] is a 12-item measure that asks participants to rate how much they have recently had various feelings or experiences related to general mental health difficulties (e.g., “Felt constantly under strain”). Items are rated on a scale from 0—“Less than usual” to 3—“Much more than usual.” It is correlated with measures of depression and mental health and demonstrates high internal consistency [49].

Wellbeing data will be collected via the Warwick Edinburgh Mental Wellbeing Scale [34, 35], a 14-item measure that asks individuals to rate how often they have had certain feelings and experiences related to wellbeing over the last 2 weeks. Items (e.g., “I’ve been dealing with problems well”) are rated on a scale from 1—“None of the time” to 5—“All of the time.” Total scores range from 14 to 70 with higher scores indicating greater wellbeing. This measure is positively correlated with other measures of wellbeing and related constructs of happiness and life satisfaction; it also demonstrates high internal consistency [34].

Resilience data will be collected using the CD-RISC-10 [37], a 10-item measure that asks individuals to rate their agreement with statements related to resilience reflecting on the last 2 weeks. Items (e.g., “I am able to adapt when changes occur”) are rated on a scale from 0—“Not at all true” to 4—“True nearly all the time.” Total scores range from 0 to 40 with higher scores indicating greater resilience. The CD-RISC-10 is positively correlated with social support and hardiness, and negatively with stress vulnerability [37] and has high internal consistency [22].

Mental health service usage will be collected using five questions asking the number of visits to various service providers for mental health reasons over the last 2 weeks, including walk-in clinic, family doctor or nurse practitioner, emergency department, mental health counselor, and mental health hotline/phone support. App quality data will be collected via the UMARS [39, 40], a 27-item measure that asks individuals to rate app quality within various domains. Items are rated on a 5-point scale with 5 generally representing the most positive quality rating. Total scores range from 22 to 130, with higher scores indicating higher perceived app quality. Measure subscales assess engagement (e.g., “Is the app interesting to use? Does it present its information in an interesting way compared to other similar apps?”), functionality (e.g., “How easy is it to learn how to use the app; how clear are the menu labels, icons and instructions?”), esthetics (e.g., “Is arrangement and size of buttons, icons, menus and content on the screen appropriate?”), information (e.g., “Is the information within the app comprehensive but concise?”), subjective quality (e.g., “What is your overall (star) rating of the app?”), and perceived impact (e.g., “This app has increased my knowledge/understanding of mental health”). The UMARS demonstrates high internal consistency and acceptable test–retest reliability [39].

Plans to promote participant retention and complete follow-up {18b}

Email reminders will be sent to participants to promote retention and completion of the measures at the four assessment time-points. Additional reminders will be sent each day for up to 2 days following the initial email if the participant has not completed the corresponding assessment. To incentivize completion, participants will receive cash compensation ($20 for first assessment time-point, $25 for second assessment time-point, $30 for third assessment time-point, $25 for forth assessment time-point) or one bonus point per assessment time-point on the university’s online research sign-up system.

Data management {19}

Data will be stored online using SurveyMonkey, which has its server located in the USA. SurveyMonkey’s data centers are SOC-2 accredited and are protected by continuous security monitoring, cameras, visitor logs, and entry limitations. Further information on storage and security is available on their website [50]. Data in transit are encrypted with secure TLS cryptographic protocols. App usage data will be stored within a password-protected Canadian server and encrypted during transmission. Assessment data and app usage data will be downloaded only by research team members ARM or VM onto a password-protected computer. Data will be stored for at least 5 years following data collection as specified by Lakehead University policy.

Confidentiality {27}

Names and contact information for interested and enrolled participants will be gathered through recruitment methods and will be kept in a tracking list accessed only by the research assistants. Those enrolled will be given an ID number and this will be added to the tracking list. Assessment and app usage data will be kept in a separate spreadsheet, organized by ID number. The spreadsheet will not contain any identifying information. Names and contact information will be deleted from the tracking list upon completion of data collection. All procedures pertaining to confidentiality will be conducted in accordance with the Canadian Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans and in accordance with the protocols approved through the Thunder Bay Regional Health Sciences Centre and Lakehead University Research Ethics Boards.

Plans for collection, laboratory evaluation, and storage of biological specimens for genetic or molecular analysis in this trial/future use {33}

N/A. Biological specimens will not be collected.

Statistical methods

Statistical methods for primary and secondary outcomes {20a}

Linear mixed modeling will be conducted to determine whether the change over time in the primary outcome (emotion regulation) and secondary outcomes (mental health, wellbeing, and resilience) is greater for those in the intervention condition. This analytic approach is appropriate for longitudinal data as it effectively models the variation in an outcome attributed to both within-individual factors and between-individual differences (i.e., attributed to group assignment) [51].

Interim analyses {21b}

N/A. There are no interim analyses or stopping guidelines planned.

Methods for additional analyses (e.g., subgroup analyses) {20b}

There are no planned subgroup analyses. Subgroup analyses may be conducted based on evolving research questions of interest and pending adequate power.

Methods in analysis to handle protocol non-adherence and any statistical methods to handle missing data {20c}

Analyses will be conducted using an intention-to-treat approach. An attrition analysis will be conducted to compare participants who provided data at all assessment time-points to those who did not complete measures at one or more assessment time-points. Missing data will be handled using full information maximum likelihood estimation [52, 53].

Plans to give access to the full protocol, participant-level data, and statistical code {31c}

The protocol is registered on clinicaltrials.gov.

Oversight and monitoring

Composition of the coordinating center and trial steering committee {5d}

Principal investigator ARM will provide oversight on all trial matters. The graduate student research assistant (VM) will lead data collection and coordinate participant enrolment and will support and train undergraduate research assistants. Meetings involving the principal investigator and research assistants will occur weekly to discuss trial progress and matters related to recruitment and retention. ARM will also meet with co-investigators (JO, ET, FS, AO, AN, and SHS) on a quarterly basis to consult about ongoing implementation and plan for future trial activities. ARM also meets with a local group of healthcare administrators, frontline staff, youth, and caregivers (functioning as a Stakeholder and Public Involvement Group) quarterly to discuss local implementation and evaluation of the JoyPop™ app across settings.

Composition of the data monitoring committee, its role and reporting structure {21a}

A formal data monitoring committee was not deemed necessary as minimal risk is involved in participation. Determination of minimal risk of harm is supported by prior study results which did not suggest any unfavorable impact on participants or relevant outcomes [22]. Aspects of data collection will be informally monitored by the research team, including recruitment and participants’ completion of assessment time-points.

Adverse event reporting and harms {22}

Participation is associated with minimal risk, particularly considering prior findings which did not indicate any unfavorable changes in outcomes most likely to be impacted by the intervention [22]. Adverse events identified among or reported by participants will be reported by the research assistants to the principal investigator immediately. The principal investigator will also inquire about any adverse events during weekly meetings with the research assistants. If an adverse event were to occur, the principal investigator would report it immediately to the Thunder Bay Regional Health Sciences Centre Research Ethics Board using the Research Ethics Local Serious Adverse Event Reporting Form.

Frequency and plans for auditing trial conduct {23}

There are no formal plans for auditing by an independent party. During weekly research team meetings, the principal investigator will ensure the protocol is being followed during the running of the trial. Research assistants will be trained on the protocol and quality and ethical standard of clinical trials [54]. They will be observed conducting study tasks prior to independent administration to ensure the protocol is followed.

Plans for communicating important protocol amendments to relevant parties (e.g., trial participants, ethical committees) {25}

If needed, modifications to the protocol would be discussed with co-investigators during quarterly meetings or over email. Changes would be submitted to the Research Ethics Board at the Thunder Bay Regional Health Sciences Centre and would be implemented once approved. The protocol registration would also be revised, and participants would be informed by phone or email if needed.

Dissemination plans {31a}

All participants will be asked if they would like to opt in to receive a summary of the research findings following study completion. This information will be kept in a separate document that is not linked to their assessment data, app usage data, or ID numbers. The research summary will be sent to interested participants via email. Results from this trial will be published in peer-reviewed, open-access journals, presented at relevant conferences, and shared with the Lakehead University community. Media interviews may also be used as a form of dissemination.

Discussion

Canadian young adults who are completing post-secondary education are facing increasing mental health concerns [3]. Barriers exist with respect to student mental health needs being fully addressed by institutional resources [11, 16], suggesting novel, accessible, and low-cost approaches to proactively promote mental health may be of benefit to students [12, 19]. This trial will evaluate the effectiveness of the JoyPop™ app as a tool to promote improved emotion regulation, mental health, wellbeing, and resilience. Results may support wider uptake of the app by post-secondary institutions in supporting students, particularly if the app is also associated with lower health service utilization and if users view the app as high-quality and are willing to adopt it.

Trial status

The trial study protocol was approved by the Research Ethics Board at the Thunder Bay Regional Health Sciences Centre (acting as the Board of Record for Lakehead University clinical research projects; #100,251).

Protocol version 2; August 9, 2024.

Trial start date: November 22, 2023.

Anticipated recruitment completion: December 2024.

Acknowledgements

We would like to thank the youth who informed the initial design of the JoyPop™ app and those who continue to inform its evolution through participation in past and current research. We would also like to acknowledge the undergraduate student research assistants involved in data collection for their integral role in this study.

Authors’ contributions {31b}

ARM is the principal investigator and with VM initially conceived the study and led its design. ET, JO, FS, AN, AO, and SHS also provided design contributions. ARM, ET, JO, AO, FS, AN, and SHS sought funding for the study. VM is leading trial coordination and data collection while ARM provides supervision and oversight. AN and SHS provide mentorship to ARM and her team throughout the trial. AM drafted the initial version of the manuscript and ARM revised the manuscript. All authors read and provided feedback on the final manuscript.

Authors’ information

ARM is an Associate Professor in the Department of Psychology at Lakehead University and Clinical Psychologist at Dilico Anishinabek Family Care in Thunder Bay, Ontario. AM and VM are both clinical psychology PhD students in the Department of Psychology at Lakehead University. FS is an Adjunct Professor in the Department of Psychology at Lakehead University. ET is an Adjunct Professor in the Department of Psychology at Lakehead University and Clinical Psychologist at Dilico Anishinabek Family Care. JO is an Associate Professor in the Department of Psychology at the University of New Brunswick. SHS is a Professor in the Departments of Psychiatry and Psychology & Neuroscience at Dalhousie University and a Canada Research Chair (Tier 1) in Addictions and Mental Health and Fellow of the Royal Society of Canada. AN is a Professor in the Department of Pediatrics at the University of Alberta. AO is a Professor in the School of Public Health at the University of Alberta.

Funding {4}

This trial is funded by the Brain Canada—2021 Future Leaders in Canadian Brain Research grant (ARM). We also acknowledge the support provided through a Tier 1 Canada Research Chair in Addictions and Mental Health (SHS).

Availability of data and materials {29}

The final trial dataset will be accessed by the principal investigator ARM as well as graduate student researcher VM.

Declarations

Ethics approval and consent to participate {24}

The Research Ethics Board at the Thunder Bay Regional Health Sciences Centre, which acts as the Board of Record for clinical research projects at Lakehead University, has reviewed and approved all study materials (file #100251). Informed consent will be obtained from all participants; a detailed verbal and written description of the study will be provided and participants will complete a consent form.

Consent for publication {32}

N/A. We have not included any details, images, or videos relating to an individual person. ARM will provide a model consent form upon request.

Competing interests {28}

The majority of the authors have no potential or actual competing interests to declare. The principal investigator (ARM) recently acquired intellectual property ownership rights for the JoyPop™ app (in June 2024). Prior to this time, intellectual property rights were owned by another researcher/institution. ARM did not own any intellectual property rights when applying for funding for this study, when creating the protocol, or when initiating the study. To mitigate risk related to ARM’s new intellectual property ownership rights, ARM will not be involved in collecting or analyzing the data and will consult with co-investigators throughout the study.

Abbreviations

iOS device

IPad, iPod, iPhone

DERS-SF

Difficulties in Emotion Regulation Scale—Short Form

DASS-21

Depression, Anxiety, and Stress Scale 21

GHQ-12

General Health Questionnaire

UMARS

User Version of the Mobile Application Rating Scale

CD-RISC-10

Connor-Davidson Resilience Scale-10

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

1. Chiu, M; Gatov, E; Fung, K; Kurdyak, P; Guttmann, A. Deconstructing the rise in mental health–related ED visits among children and youth in Ontario. Canada Health Aff; 2020; 39, 10 pp. 1728-1736. [DOI: https://dx.doi.org/10.1377/hlthaff.2020.00232]

2. Wiens, K; Bhattarai, A; Pedram, P; Dores, A; Williams, J; Bulloch, A; Patten, S. A growing need for youth mental health services in Canada: examining trends in youth mental health from 2011 to 2018. Epidemiol Psychiatr Sci; 2020; 17, 29 e115. [DOI: https://dx.doi.org/10.1017/S2045796020000281]

3. Linden, B; Boyes, R; Stuart, H. Cross-sectional trend analysis of the NCHA II survey data on Canadian post-secondary student mental health and wellbeing from 2013 to 2019. BMC Public Health; 2021; 21, 590. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33765965][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7992810][DOI: https://dx.doi.org/10.1186/s12889-021-10622-1]

4. Gouin, JP; MacNeil, S; De la Torre-Luque, A; Chartrand, E; Chadi, N; Rouquette, A; Boivin, M; Côté, S; Geoffroy, MC. Depression, anxiety, and suicidal ideation in a population-based cohort of young adults before and during the first 12 months of the COVID-19 pandemic in Canada. Can J Public Health; 2023; 114, 3 pp. 368-377. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37093498][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10124701][DOI: https://dx.doi.org/10.17269/s41997-023-00772-7]

5. King, N; Pickett, W; Rivera, D; Byun, J; Li, M; Cunningham, S; Duffy, A. The impact of the COVID-19 pandemic on the mental health of first-year undergraduate students studying at a major Canadian university: a successive cohort study. Can J Psychiatry; 2023; 68, 7 pp. 499-509. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35450455][DOI: https://dx.doi.org/10.1177/07067437221094549]

6. Sheikhan, NY; Hawke, LD; Ma, C; Courtney, D; Szatmari, P; Cleverley, K; Voineskos, A; Cheung, A; Henderson, J. A longitudinal cohort study of youth mental health and substance use before and during the COVID-19 pandemic in Ontario, Canada: an exploratory analysis. Can J Psychiatry; 2022; 67, 11 pp. 842-854. [DOI: https://dx.doi.org/10.1177/07067437221097906]

7. Smith AC, Filice D, Poole H, Khan A, Whalen K, Smilek D. Indicators of student well-being in Canadian undergraduates before and during the COVID-19 pandemic. Scholarsh Teach Learn Psychol. 2022. https://psycnet.apa.org/doiLanding?doi=10.1037%2Fstl0000338.

8. Gandhi, S; Chiu, M; Lam, K; Cairney, JC; Guttmann, A; Kurdyak, P. Mental health service use among children and youth in Ontario: population-based trends over time. Can J Psychiatry; 2016; 61, 2 pp. 119-124. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27253703][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4784237][DOI: https://dx.doi.org/10.1177/0706743715621254]

9. Jaworska, N; De Somma, E; Fonseka, B; Heck, E; MacQueen, GM. Mental health services for students at postsecondary institutions: a national survey. Can J Psychiatry; 2016; 61, 12 pp. 766-775. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27310230][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5564891][DOI: https://dx.doi.org/10.1177/0706743716640752]

10. Heck, E; Jaworska, N; DeSomma, E; Dhoopar, AS; MacMaster, FP; Dewey, D; MacQueen, G. A survey of mental health services at post-secondary institutions in Alberta. Can J Psychiatry; 2014; 59, 5 pp. 250-258. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25007278][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4079139][DOI: https://dx.doi.org/10.1177/070674371405900504]

11. Read, A; Lutgens, D; Malla, A. A descriptive overview of mental health services offered in post-secondary educational institutions across Canada. Can J Psychiatry; 2023; 68, 2 pp. 101-108. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36200430][DOI: https://dx.doi.org/10.1177/07067437221128168]

12. Moghimi, E; Stephenson, C; Gutierrez, G; Jagayat, J; Layzell, G; Patel, C; McCart, A; Gibney, C; Langstaff, C; Ayonrinde, O; Khalid-Khan, S. Mental health challenges, treatment experiences, and care needs of post-secondary students: a cross-sectional mixed-methods study. BMC Public Health; 2023; 23, 1 655. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37020282][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10076091][DOI: https://dx.doi.org/10.1186/s12889-023-15452-x]

13. Murray, JK; Knudson, S. Mental health treatment and access for emerging adults in Canada: a systematic review. Front Public Health; 2023; 11, 1088999. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37501945][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10370273][DOI: https://dx.doi.org/10.3389/fpubh.2023.1088999]

14. Robinson AM, Jubenville TM, Renny K, Cairns SL. Academic and mental health needs of students on a Canadian campus. Can J Couns Psychother. 2016;50(2):108–23.

15. Price, M; Yuen, EK; Goetter, EM; Herbert, JD; Forman, EM; Acierno, R; Ruggiero, KJ. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother; 2014; 21, 5 pp. 427-436. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23918764][DOI: https://dx.doi.org/10.1002/cpp.1855]

16. Eisenstadt, M; Liverpool, S; Infanti, E; Ciuvat, RM; Carlsson, C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: systematic review and meta-analysis. JMIR Ment Health; 2021; 8, 11 e31170. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34747713][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8663676][DOI: https://dx.doi.org/10.2196/31170]

17. Linardon, J; Cuijpers, P; Carlbring, P; Messer, M; Fuller-Tyszkiewicz, M. The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry; 2019; 18, 3 pp. 325-336. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31496095][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6732686][DOI: https://dx.doi.org/10.1002/wps.20673]

18. Oliveira, C; Pereira, A; Vagos, P; Nóbrega, C; Gonçalves, J; Afonso, B. Effectiveness of mobile app-based psychological interventions for college students: a systematic review of the literature. Front Psychol; 2021; 12, 647606. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34045994][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8144454][DOI: https://dx.doi.org/10.3389/fpsyg.2021.647606]

19. Johnson, KF; Kalkbrenner, MT. The utilization of technological innovations to support college student mental health: mobile health communication. J Technol Hum Serv; 2017; 35, 4 pp. 314-339. [DOI: https://dx.doi.org/10.1080/15228835.2017.1368428]

20. Marshall, JM; Dunstan, DA; Bartik, W. The digital psychiatrist: in search of evidence-based apps for anxiety and depression. Front Psychiatry; 2019; 10, 831 pp. 1-7.[COI: 1:CAS:528:DC%2BC1MXhtVKisbbF]

21. Wekerle C. JoyPop app. Resilience in youth [Internet]. Available from: https://youthresilience.net/joypop-app. Accessed 25 Jan 2024.

22. MacIsaac, A; Mushquash, AR; Mohammed, S; Grassia, E; Smith, S; Wekerle, C. Adverse childhood experiences and building resilience with the JoyPop app: evaluation study. JMIR Mhealth Uhealth; 2021; 9, 1 e25087. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33393908][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7813633][DOI: https://dx.doi.org/10.2196/25087]

23. Compas, BE; Jaser, SS; Bettis, AH; Watson, KH; Gruhn, MA; Dunbar, JP; Williams, E; Thigpen, JC. Coping, emotion regulation, and psychopathology in childhood and adolescence: a meta-analysis and narrative review. Psychol Bull; 2017; 143, 9 939. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28616996][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7310319][DOI: https://dx.doi.org/10.1037/bul0000110]

24. Jarrell, A; Lajoie, SP; Hall, NC; Horrocks, PTM. Antecedents and consequences of emotion regulation in STEM degree programs. Innov High Educ; 2022; 47, pp. 493-514. [DOI: https://dx.doi.org/10.1007/s10755-021-09587-1]

25. MacCann, C; Jiang, Y; Brown, LER; Double, KS; Bucich, M; Minbashian, A. Emotional intelligence predicts academic performance: a meta-analysis. Psychol Bull; 2020; 146, 2 pp. 150-186. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31829667][DOI: https://dx.doi.org/10.1037/bul0000219]

26. Mushquash, AR; Pearson, ES; Waddington, K; MacIsaac, A; Mohammed, S; Grassia, E; Smith, S; Wekerle, C. User perspectives on the resilience-building JoyPop app: qualitative study. JMIR Mhealth Uhealth; 2021; 9, 7 e28677. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34255696][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8299348][DOI: https://dx.doi.org/10.2196/28677]

27. Malik, I; Perez, A; Toombs, E; Schmidt, F; Olthuis, JV; Charlton, J; Grassia, E; Squier, C; Stasiuk, K; Bobinski, T; Mushquash, AR. Female youth and mental health service providers’ perspectives on the JoyPopTM app: a qualitative study. Front Digit Health; 2023; 5, pp. 1-20. [DOI: https://dx.doi.org/10.3389/fdgth.2023.1197362]

28. Durand-Bush N, McNeill K, Harding M, Dobransky J. Investigating stress, psychological well-being, mental health functioning, and self-regulation capacity among university undergraduate students: is this population optimally functioning? Can J Couns Psychother. 2015;49(3):253–74.

29. Barnett, AG; Van Der Pols, JC; Dobson, AJ. Regression to the mean: what it is and how to deal with it. Int J Epidemiol; 2005; 1, pp. 215-220.

30. Gratz, KL; Roemer, L. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assess; 2004; 26, 1 pp. 41-54. [DOI: https://dx.doi.org/10.1023/B:JOBA.0000007455.08539.94]

31. Kaufman, EA; Xia, M; Fosco, G; Yaptangco, M; Skidmore, CR; Crowell, SE. The difficulties in emotion regulation scale short form (DERS-SF): validation and replication in adolescent and adult samples. J Psychopathol Behav Assess; 2016; 38, pp. 443-455. [DOI: https://dx.doi.org/10.1007/s10862-015-9529-3]

32. Lovibond, PF; Lovibond, SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther; 1995; 33, 3 pp. 335-343.[COI: 1:STN:280:DyaK2M3ktl2jsA%3D%3D] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/7726811][DOI: https://dx.doi.org/10.1016/0005-7967(94)00075-U]

33. Goldberg, DP; Williams, P. A user’s guide to the General Health Questionnaire; 1988;

34. Stewart-Brown S, Janmohamed K. Warwick-Edinburgh Mental Well-being Scale (WEMWBS): user guide version 1 (J. Parkinson, Ed.). Scotland: University of Warwick and the University of Edinburgh; 2008.

35. Tennant, R; Hiller, L; Fishwick, R; Platt, S; Joseph, S; Weich, S; Parkinson, J; Secker, J; Stewart-Brown, S. The Warwick-Edinburgh Mental Well-being Scale (WEMWBS): development and UK validation. Health Qual Life Outcomes; 2007; 5, 1 63. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18042300][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2222612][DOI: https://dx.doi.org/10.1186/1477-7525-5-63]

36. Westerhof, GJ; Keyes, CLM. Mental Illness and mental health: the two continua model across the lifespan. J Adult Dev; 2010; 17, 2 pp. 110-119. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/20502508][DOI: https://dx.doi.org/10.1007/s10804-009-9082-y]

37. Campbell-Sills, L; Stein, MB. Psychometric analysis and refinement of the Connor-Davidson resilience scale (CD-RISC): validation of a 10-item measure of resilience. J Trauma Stress; 2007; 20, 6 pp. 1019-1028. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18157881][DOI: https://dx.doi.org/10.1002/jts.20271]

38 Masten, AS; Barnes, AJ. Resilience in children: developmental perspectives. Children.; 2018; 5, 7 98. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30018217][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6069421][DOI: https://dx.doi.org/10.3390/children5070098]

39. Stoyanov, SR; Hides, L; Kavanagh, DJ; Wilson, H. Development and validation of the user version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth; 2016; 4, 2 e5849. [DOI: https://dx.doi.org/10.2196/mhealth.5849]

40. Stoyanov, SR; Hides, L; Kavanagh, DJ; Zelenko, O; Tjondronegoro, D; Mani, M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth; 2015; 3, 1 e3422. [DOI: https://dx.doi.org/10.2196/mhealth.3422]

41. Peng, W; Kanthawala, S; Yuan, S et al. A qualitative study of user perceptions of mobile health apps. BMC Public Health; 2016; 16, 1158. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27842533][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5109835][DOI: https://dx.doi.org/10.1186/s12889-016-3808-0]

42. Thabrew, H; Boggiss, AL; Lim, D; Schache, K; Morunga, E; Cao, N; Cavadino, A; Serlachius, AS. Well-being app to support young people during the COVID-19 pandemic: randomised controlled trial. BMJ Open; 2022; 12, 5 e058144. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35589362][DOI: https://dx.doi.org/10.1136/bmjopen-2021-058144]

43. Faul, F; Erdfelder, E; Lang, AG; Buchner, A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods; 2007; 39, pp. 175-191. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/17695343][DOI: https://dx.doi.org/10.3758/BF03193146]

44. Lui, J; Marcus, D; Barry, C. Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Prof Psychol Res Pract; 2017; 48, pp. 199-210. [DOI: https://dx.doi.org/10.1037/pro0000122]

45. Bauer, M; Glenn, T; Geddes, J; Gitlin, M; Grof, P; Kessing, LV; Whybrow, PC. Smartphones in mental health: a critical review of background issues, current status and future concerns. Int J Bipolar Disord; 2020; 8, pp. 1-20. [DOI: https://dx.doi.org/10.1186/s40345-019-0164-x]

46. Sealed Envelope Ltd. Create a blocked randomisation list. 2022. https://www.sealedenvelope.com/simple-randomiser/v1/list

47. Survey Monkey Inc. Survey Monkey. San Mateo, California, USA. www.surveymonkey.com. Accessed 25 Jan 2024.

48. Osman, A; Wong, JL; Bagge, CL; Freedenthal, S; Gutierrez, PM; Lozano, G. The Depression Anxiety Stress Scales—21 (DASS-21): further examination of dimensions, scale reliability, and correlates. J Clin Psychol; 2012; 68, 12 pp. 1322-1338. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22930477][DOI: https://dx.doi.org/10.1002/jclp.21908]

49. Romppel, M; Braehler, E; Roth, M; Glaesmer, H. What is the General Health Questionnaire-12 assessing?: dimensionality and psychometric properties of the General Health Questionnaire-12 in a large scale German population sample. Compr Psychiatry; 2013; 54, 4 pp. 406-413. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/23206494][DOI: https://dx.doi.org/10.1016/j.comppsych.2012.10.010]

50. Survey Monkey Inc. Security statement. 2023. https://www.surveymonkey.com/mp/legal/security/.

51. Singer JD, Willett JB. Applied longitudinal data analysis: modeling change and event occurrence. New York: Oxford University Press; 2003.

52. Dong, Y; Peng, C-Y. Principled missing data methods for researchers. Springerplus; 2023; 2, 222. [DOI: https://dx.doi.org/10.1186/2193-1801-2-222]

53. Jelicić, H; Phelps, E; Lerner, RM. Use of missing data methods in longitudinal studies: the persistence of bad practices in developmental psychology. Dev Psychol; 2009; 45, 4 pp. 1195-1199. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/19586189][DOI: https://dx.doi.org/10.1037/a0015665]

54. Otte, A; Maier-Lenz, H; Dierckx, RA. Good Clinical Practice: historical background and key aspects. Nucl Med Commun; 2005; 26, pp. 563-574. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/15942475][DOI: https://dx.doi.org/10.1097/01.mnm.0000168408.03133.e3]