Predictors, Outcomes, and Statistical Solutions of Missing Cases in Web-Based Psychotherapy: Methodological Replication and Elaboration Study

Abstract

Background: Missing cases present a challenge to our ability to evaluate the effects of web-based psychotherapy trials. As missing cases are often lost to follow-up, less is known about their characteristics, their likely clinical outcomes, or the likely effect of the treatment being trialed.

Objective: The aim of this study is to explore the characteristics of missing cases, their likely treatment outcomes, and the ability of different statistical models to approximate missing posttreatment data.

Methods: A sample of internet-delivered cognitive behavioral therapy participants in routine care (n=6701, with 36.26% missing cases at posttreatment) was used to identify predictors of dropping out of treatment and predictors that moderated clinical outcomes, such as symptoms of psychological distress, anxiety, and depression. These variables were then incorporated into a range of statistical models that approximated replacement outcomes for missing cases, and the results were compared using sensitivity and cross-validation analyses.

Results: Treatment adherence, as measured by the rate of progress of an individual through the treatment modules, and higher pretreatment symptom scores were identified as the dominant predictors of missing cases probability (Nagelkerke R²=60.8%) and the rate of symptom change. Low treatment adherence, in particular, was associated with increased odds of presenting as missing cases during posttreatment assessment (eg, odds ratio 161.1:1) and, at the same time, attenuated the rate of symptom change across anxiety (up to 28% of the total symptom with 48% reduction effect), depression (up to 41% of the total with 48% symptom reduction effect), and psychological distress symptom outcomes (up to 52% of the total with 37% symptom reduction effect) at the end of the 8-week window. Reflecting this pattern of results, statistical replacement methods that overlooked the features of treatment adherence and baseline severity underestimated missing case symptom outcomes by as much as 39% at posttreatment.

Conclusions: The treatment outcomes of the cases that were missing at posttreatment were distinct from those of the remaining observed sample. Thus, overlooking the features of missing cases is likely to result in an inaccurate estimate of the effect of treatment.

Introduction

Background

The ability to evaluate the effect of psychotherapy often depends on the measurement of outcomes before-and-after an intervention. However, many participants are unable to complete measurement questionnaires and become missing cases, thus threatening the validity of conclusions drawn from trials. Missing cases are frequently reported in psychotherapy trials [1,2] and pose a risk to the validity of the evidence base for some treatments [3,4]. Overlooking the causes and outcomes of missing cases can lead to systematic measurement bias and misrepresentation of treatment outcomes and, therefore, risks compromising the validity of clinical research [5,6]. For this reason, careful analysis of the effect of missing cases is now considered an important part of the process of measuring and reporting clinical evidence [3].

Although the importance of handling missing cases is well understood [3,7], accounting for the outcomes of missing cases is a challenging task, as researchers can never verify whether the replacement values they generate accurately captured patient outcomes. Thus, researchers must rely on statistical approximation and the assumption that any replacement outcomes are suitable [8].

A key requirement for handling missing data is to ensure that the outcomes of missing cases are represented within statistical analyses [8]; typically, this involves using a statistical solution that generates replacement values for missing cases [5,8,9]. Researchers rely on statistical methods that explore the characteristics of missing cases to determine whether a statistical solution is suitable for missing cases and whether these features could also be associated with distinct clinical outcomes. This is typically achieved through analyses that identify variables that predict both the probability that participants will become missing cases and the clinical outcome of such missing cases [4,8,10]. Identifying such variables enables researchers to generate replacement scores that are likely to capture the outcomes of treatment for missing cases [7,10]. For example, if older age is associated with a decreased probability of becoming a missing case and an increased rate of symptom change, a statistical model that can adjust for participants’ age will be considered to create replacement outcomes that are more accurate and representative of the effects of treatment than models that overlook age. In statistical terms, variables that predict both the likelihood of becoming a missing case and the outcome of missing cases are known as mechanisms of nonignorable missing cases [6,10,11].

Although statistical models that incorporate replacement values for missing cases have been in use for decades [7,8,12], relatively few published studies have reported the characteristics of missing cases in psychotherapy trials or research that identified nonignorable mechanisms of noncompletion that might influence the reported outcomes [2,13]. This gap in methodological research may result from (1) the limited knowledge about missing cases and the patient features that may generalize across clinical trials [2] and (2) the scarcity of large and comparable treatment samples that are statistically powered to explore nonignorable mechanisms of noncompletion.

Preliminary evidence from trials of internet-delivered cognitive behavioral therapy (iCBT) suggests that common patient variables, such as treatment completion and baseline depressive symptom severity, were the main predictors of both the likelihood of patients dropping out of treatment and moderating the clinical effect [2,4]. These findings suggested that (1) the symptom outcomes of missing cases were not comparable with the patients that provided their data following treatment and (2) missing cases can be characterized through key features that shape the likelihood of a case to present as missing during posttreatment assessment. In particular, minimal treatment adherence, as measured by the partial progress of an individual through the treatment modules, was associated with increased odds of presenting as a missing case during posttreatment assessment (eg, odds ratio 70.6, 95% CI 34.5 to 145.1) and a lower rate of symptom change (eg, 21% for low treatment adherence vs 49% for high adherence) [4]. Without accounting for these variables, web-based psychotherapy researchers risk overlooking a systematic pattern of worse treatment outcomes for missing cases and generating estimates of treatment effects that are unrealistically optimistic. However, the evidence from this study regarding the effect of missing cases in internet-delivered psychotherapy is limited to a single study that focused on symptoms of depression using data from a highly controlled clinical trial with high participant retention (87%) [4]. Replicating this study in an additional therapeutic context and within additional clinical outcomes is needed before conclusions can be drawn regarding the characteristics and effect of missing cases in internet-delivered psychotherapy and the appropriate statistical methods for handling missing cases.

Objectives

The main aim of this study is to examine the characteristics and possible clinical outcomes of missing cases in a large sample in routine care and compare different statistical methods for estimating those outcomes. This study examined the outcomes of a large sample of patients enrolled in treatment courses provided by an established digital mental health service (DMHS) offering internet psychotherapy based on cognitive behavior therapy (n=6701), in which the patients were administered validated self-report questionnaires to measure symptoms of depression, anxiety, and psychological distress at baseline, at intervals during treatment, and at follow-up. It was hypothesized that (1) lower treatment completion and increased baseline depressive symptoms would predict both increased likelihood of noncompletion and higher symptoms of depression posttreatment and that (2) statistical models that account for these features will result in higher posttreatment symptom replacement scores compared with the statistical models that assume missing cases occur as a random event.

Methods

The Sample

This study examined the outcome of routine care provided by Australian National DMHS, the MindSpot Clinic [14]. All participants provided consent for their deidentified data to be used in evaluation and quality improvement activities. Approval for this research was provided by the Macquarie University Human Research Ethics Committee. Further information about the sample, the course content and delivery protocols, and the outcomes of the iCBT can be found in a study by Titov et al [15]. The standardized nature of clinical engagement and treatment delivery in iCBT reduces the likelihood that differences in outcomes are because of different approaches of therapists.

The 6701 participants who commenced treatment during a 30-month period completed self-report symptom scales and provided other information pretreatment and completed symptom scales midtreatment (surveyed at Week 4), posttreatment (Week 8), and at follow-up (Week 20).

In this study, emphasis was on the prediction of posttreatment symptom outcomes, where posttreatment was considered the main time point for evaluating the effects of treatment [15]. From the participants who initiated treatment, 63.7% (4271/6701) of the sample provided data posttreatment, with 36% (2430/6701) considered to be missing cases as individuals who did not comply with weekly email and telephone prompts to complete a posttreatment evaluation assessment. For cross-replication analysis, the sample was randomly allocated into 5 subgroups, each with more than 1340 participants pretreatment and more than 840 completed measurements posttreatment. Tables 1 and 2 collate the demographic information of the samples, including chi-square values, to confirm adequate randomization.

Table 1. Randomization of cross-validation samples and participant characteristics (N=6701).

Sample		Available sample at pretreatment, n (%)	Available sample at posttreatment, n (%)	Randomization test
				Chi-square (df)	P value
Total sample		6701 (100)	4271 (64)	0.01 (4)	.99
	Replication sample 1	1341 (20.01)	842 (62.79)	N/Aa	N/A
	Replication sample 2	1340 (20.00)	846 (63.13)	N/A	N/A
	Replication sample 3	1340 (20.00)	843 (62.91)	N/A	N/A
	Replication sample 4	1340 (20.00)	846 (63.13)	N/A	N/A
	Replication sample 5	1340 (20.00)	848 (63.28)	N/A	N/A

^aN/A: not applicable (redundant parameter).

Table 2. Sample demographics.

Variable	Value	Randomization test
		Chi-square (df)	P value
Age (years), mean (SD)	37.57 (10.9)	3.8 (1)	.44
Completed 1/5 modules, n (%)	513 (7.66)	7.5 (4)	.96
Completed 2/5 modules, n (%)	715 (10.67)	N/Aa	N/A
Completed 3/5 modules, n (%)	718 (10.71)	N/A	N/A
Completed 4/5 modules, n (%)	653 (9.74)	N/A	N/A
Completed 5/5 modules, n (%)	4102 (61.21)	N/A	N/A
In a relationship, n (%)	4458 (66.53)	0.6 (1)	.97
Employment (employed), n (%)	4908 (73.24)	0.8 (1)	.94
Education (tertiary), n (%)	3239 (48.34)	4.0 (1)	.41
Gender (female), n (%)	4866 (48.34)	6.8 (1)	.15
Comorbidity (GAD-7b ≤8 and PHQ-9c ≤10), n (%)	3437 (51.29)	3.0 (1)	.56

^aN/A: not applicable (redundant parameter).

^bGAD-7: generalized anxiety disorder-7 item scale.

^cPHQ-9: patient health questionnaire-9.

Intervention

The participants enrolled in the Wellbeing Course [15], a 5-lesson course delivered over 8 weeks to patients experiencing depression and anxiety. The lessons covered (1) the cognitive behavioral model and symptom identification, (2) thought monitoring and challenging, (3) de-arousal strategies and pleasant activity scheduling, (4) graduated exposure, and (5)

References

Fernandez E, Salem D, Swift J, Ramtahal N. Meta-analysis of dropout from cognitive behavioral therapy: magnitude, timing, and moderators. J Consult Clin Psychol 2015 Dec;83(6):1108-1122.

Karyotaki E, Kleiboer A, Smit F, Turner DT, Pastor AM, Andersson G, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol. Med 2015 Apr 17;45(13):2717-2726

Little RJ, D'Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med 2012 Oct 04;367(14):1355-1360

Karin E, Dear B, Heller G, Crane M, Titov N. "Wish You Were Here": examining characteristics, outcomes, and statistical solutions for missing cases in web-based psychotherapeutic trials. JMIR Ment Health 2018 Apr 19;5(2):e22

Bell M, Fairclough DL. Practical and statistical issues in missing data for longitudinal patient-reported outcomes. Stat Methods Med Res 2014 Oct;23(5):440-459

RUBIN DB. Inference and missing data. Biometrika 1976;63(3):581-592.

Lang K, Little TD. Principled missing data treatments. Prev Sci 2018 Apr;19(3):284-294.

Mealli F, Rubin D. Clarifying missing at random and related definitions, and implications when coupled with exchangeability: Table 1. Biometrika 2015 Sep 22;102(4):995-1000

Bell M, Fiero M, Horton N, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol 2014 Nov 19;14:118

Little RJ, Rubin DB. Statistical Analysis with Missing Data. In: Bayesian Analysis. Hoboken, NJ: John Wiley & Sons; 2014.

Little RJA. Modeling the drop-out mechanism in repeated-measures studies. J of the Amer Stati Asso 1995 Sep;90(431):1112-1121.

Schafer J, Graham J. Missing data: Our view of the state of the art. Psychological Methods 2002;7(2):147-177

Alfonsson S, Olsson E, Hursti T. Motivation and treatment credibility predicts dropout, treatment adherence, and clinical outcomes in an internet-based cognitive behavioral relaxation program: a randomized controlled trial. J Med Internet Res 2016 Mar 08;18(3):e52

A digital mental health clinic for all Australians. MindSpot Clinic. URL: https://mindspot.org.au/ [accessed 2021-01-06]

Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. The first 30 months of the MindSpot Clinic: Evaluation of a national e-mental health service against project objectives. Aust N Z J Psychiatry 2017 Dec;51(12):1227-1239

Kroenke K, Spitzer R, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613

Staples LG, Dear BF, Gandy M, Fogliati V, Fogliati R, Karin E, et al. Psychometric properties and clinical utility of brief measures of depression, anxiety, and general distress: The PHQ-2, GAD-2, and K-6. Gen Hosp Psychiatry 2019;56:13-18

Spitzer R, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097.

Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SLT, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med 2002 Aug;32(6):959-976

Sieverink F, Kelders S, van Gemert-Pijnen JE. Clarifying the concept of adherence to eHealth technology: systematic review on when usage becomes adherence. J Med Internet Res 2017 Dec 06;19(12):e402

R Core Team. The R Project for Statistical Computing. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2021. URL: https://www.r-project.org/ [accessed 2020-06-06] [WebCite Cache]

Donohue MC, Gamst AC, Edland SD. Package ‘longpower’. Sample Size Calculations for L.: CRAN; 2020 Apr 21. URL: https://pbil.univ-lyon1.fr/CRAN/web/packages/longpower/longpower.pdf [accessed 2021-01-06] [WebCite Cache]

Harrell JF. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. In: Springer Series in Statistics. New York: Springer; 2015.

Nagelkerke NJD. A note on a general definition of the coefficient of determination. Biometrika 1991;78(3):691-692

LIANG K, ZEGER SL. Longitudinal data analysis using generalized linear models. Biometrika 1986;73(1):13-22.

Karin E, Dear B, Heller G, Gandy M, Titov N. Measurement of symptom change following web-based psychotherapy: statistical characteristics and analytical methods for measuring and interpreting change. JMIR Ment Health 2018 Jul 12;5(3):e10200

Cook JA, Hislop J, Altman DG, Fayers P, Briggs AH, Ramsay CR, DELTA group. Specifying the target difference in the primary outcome for a randomised controlled trial: guidance for researchers. Trials 2015 Jan 15;16:12

Greene C, Morland L, Durkalski V, Frueh BC. Noninferiority and equivalence designs: issues and implications for mental health research. J Trauma Stress 2008 Oct;21(5):433-439

Delgadillo J, McMillan D, Lucock M, Leach C, Ali S, Gilbody S. Early changes, attrition, and dose-response in low intensity psychological interventions. Br J Clin Psychol 2014 Mar;53(1):114-130

Robinson L, Delgadillo J, Kellett S. The dose-response effect in routinely delivered psychological therapies: a systematic review. Psychother Res 2020 Jan 20;30(1):79-96

Cornelisz I, Cuijpers P, Donker T, van Klaveren C. Addressing missing data in randomized clinical trials: a causal inference perspective. PLoS One 2020 Jul 6;15(7):e0234349

Levis B, Benedetti A, Ioannidis JPA, Sun Y, Negeri Z, He C, et al. Patient Health Questionnaire-9 scores do not accurately estimate depression prevalence: individual participant data meta-analysis. J Clin Epidemiol 2020 Jun;122(1):115-128

Falkenström F, Josefsson A, Berggren T, Holmqvist R. How much therapy is enough? Comparing dose-effect and good-enough models in two different settings. Psychotherapy (Chic) 2016 Mar;53(1):130-139

DiMatteo M, Lepper H, Croghan TW. Depression is a risk factor for noncompliance with medical treatment: meta-analysis of the effects of anxiety and depression on patient adherence. Arch Intern Med 2000 Jul 24;160(14):2101-2107

Fernández-Álvarez J, Díaz-García A, González-Robles A, Baños R, García-Palacios A, Botella C. Dropping out of a transdiagnostic online intervention: a qualitative analysis of client's experiences. Internet Interv 2017 Dec;10:29-38