Accuracy of the Canadian COVID-19 Mortality Score (CCMS) to predict in-hospital mortality among vaccinated and unvaccinated patients infected with Omicron: a cohort study

Abstract

Objective

The objective is to externally validate and assess the opportunity to update the Canadian COVID-19 Mortality Score (CCMS) to predict in-hospital mortality among consecutive non-palliative COVID-19 patients infected with Omicron subvariants at a time when vaccinations were widespread.

Design

This observational study validated the CCMS in an external cohort at a time when Omicron variants were dominant. We assessed the potential to update the rule and improve its performance by recalibrating and adding vaccination status in a subset of patients from provinces with access to vaccination data and created the adjusted CCMS (CCMS^adj). We followed discharged patients for 30 days after their index emergency department visit or for their entire hospital stay if admitted.

Setting

External validation cohort for CCMS: 36 hospitals participating in the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN). Update cohort for CCMS^adj: 14 hospitals in CCEDRRN in provinces with vaccination data.

Participants

Consecutive non-palliative COVID-19 patients presenting to emergency departments.

Main outcome measures

In-hospital mortality.

Results

Of 39 682 eligible patients, 1654 (4.2%) patients died. The CCMS included age, sex, residence type, arrival mode, chest pain, severe liver disease, respiratory rate and level of respiratory support and predicted in-hospital mortality with an area under the curve (AUC) of 0.88 (95% CI 0.87 to 0.88) in external validation. Updating the rule by recalibrating and adding vaccination status to create the CCMS^adj changed the weights for age, respiratory status and homelessness, but only marginally improved its performance, while vaccination status did not. The CCMS^adj had an AUC of 0.91 (95% CI 0.89 to 0.92) in validation. CCMS^adj scores of <10 categorised patients as low risk with an in-hospital mortality of <1.6%. A score>15 had observed mortality of >56.8%.

Conclusions

The CCMS remained highly accurate in predicting mortality from Omicron and improved marginally through recalibration. Adding vaccination status did not improve the performance. The CCMS can be used to inform patient prognosis, goals of care conversations and guide clinical decision-making for emergency department patients with COVID-19.

JP and RR are joint senior authors.

STRENGTHS AND LIMITATIONS OF THIS STUDY

• A strength of this study is the large, socially and ethnographically diverse Canadian population represented, which spans a geographic region of six time zones over multiple pandemic waves, including a period after vaccines became widely available and public health restrictions were lifted.

• The study cohort leveraged a registry created under a waiver of informed consent and requiring sites to capture data on >99% of COVID-19 presentations to minimise selection bias of patients. In addition, sites had to comply with rigorous data verification and quality checks.

• The model was derived and validated without requiring linkage to administrative data and thus can be applied at the bedside using simple variables documented in electronic medical records at the point of care.

• A limitation is that the Canadian COVID-19 Mortality Score has not been validated in other healthcare systems without universal health coverage.

Introduction

The Canadian COVID-19 Mortality Score (CCMS) was created to enable accurate mortality risk prediction in non-palliative emergency department patients with COVID-19 to inform shared decision-making around patients’ goals of care.¹ The predictor variables that make up the CCMS include age, sex, type of residence, arrival mode, presence of chest pain, moderate/severe liver disease, arrival respiratory rate and emergency department oxygen delivery and can be ascertained within minutes of arrival (online supplemental etable 1). The area under the curve (AUC) for the original CCMS was 0.92 in validation, and the score had excellent calibration and discrimination. However, it was derived and validated when the wild type virus was circulating, before Omicron subvariants became dominant and before widespread natural infections and vaccinations conferred population-level protection from COVID-19.² This simple clinical risk score was intended for use by clinicians at the bedside, and in contrast to other rules, did not require any laboratory or radiographic studies, access to coded comorbidity data or computation in an electronic medical record. The simplicity and ease of use of the CCMS made it an attractive tool to be used at the point of care to inform conversations with patients and their families about their desired level of care and immediate resuscitation decisions in the emergency department.

The objective of this study included externally validating the original CCMS in a new cohort of patients infected with Omicron SARS-CoV-2 variants at a time when widespread natural infections and vaccinations conferred waxing and waning population-level immunity, and evidence-based treatments for severe COVID-19 were standard care.³ A secondary objective was to attempt updating the CCMS by recalibrating the weights of the original variables and adding vaccination status to the model.

Methods

Design

This study is an external validation and update of a clinical decision rule based on observational data, where model development and reporting followed Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis standards.⁴ The Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN, pronounced ‘SEDrin’, www.ccedrrn.com) is a multicentre pan-Canadian registry that enrolled consecutive eligible COVID-19 patients presenting to emergency departments in hospitals located in eight Canadian provinces, including the most populous.⁵ CCEDRRN data have been used to derive and validate clinical decision instruments,^{1 6 7} as well as complete observational studies on diagnostic testing,^{8 9} prognosis and treatments^{3 10} and patient-reported outcomes.^{11 12}

Participants

Participating sites needed to demonstrate>99% compliance in enrolling consecutive eligible patients for their data to be included in this study. We included data from 36 CCEDRRN sites that met this criterion for external validation of the CCMS and updated the rule in 14 sites from British Columbia and Nova Scotia, which were the only provinces that permitted linkage to vaccination registry data (online supplemental etable 2). All study participants were recruited after the original CCMS derivation and validation cohorts.¹

We included patients with COVID-19 who presented to a participating emergency department between 1 February 2021 and 30 September 2022. We defined COVID-19 as patients presenting to the emergency department reporting ongoing COVID-19 symptoms in the context of a positive nucleic acid amplification test (NAAT) for SARS-CoV-2 obtained within 14 days prior to, or after their arrival in the emergency department. This allowed us to capture patients who were diagnosed in the community and subsequently presented to the emergency department, and those with early false negative tests that became positive during the course of hospitalisation.⁸ We included patients presenting with COVID-19 symptoms and diagnosed with ‘confirmed COVID-19’ to capture patients who were transferred into a CCEDRRN hospital whose NAAT at the sending site could not be confirmed in our dataset.

We excluded patients under 18 years of age, those whose goals of care precluded invasive mechanical ventilation and who as a result may have had a different prognosis, and patients transferred to a hospital outside of CCEDRRN, as we would have been unable to ascertain their outcomes (figure 1).¹³ We followed patients by chart review for 30 days after their index emergency department visit if discharged from the emergency department, or until hospital discharge if their admission lasted longer than 30 days.

Figure 1. (A) Patient flow for the CCMS external validation. (B) Patient flow for recalibration and addition of vaccination data to create the CCMS adj . CCMS adj , adjusted Canadian COVID-19 Mortality Score; ED, emergency department.

Predictors

Trained research assistants abstracted data from electronic and paper-based medical records into a central, web-based REDCap database (Vanderbilt University; Nashville, Tennessee, USA) and captured demographics, vital signs, symptoms, comorbidities, COVID-19 exposure risk, diagnostic test results and patient outcomes. The CCMS had been developed on 7420 patients presenting to CCEDRRN sites between 1 March 2020 and 31 January 2021.¹ The predictor variables that made up the CCMS were age, sex, type of residence, arrival mode, presence of chest pain, moderate/severe liver disease, arrival respiratory rate and emergency department oxygen delivery with possible scores ranging from −1 to 19, with an AUC of 0.92 (95% CI 0.90 to 0.93).

We linked the CCEDRRN dataset to population-based administrative databases from the provinces of British Columbia and Nova Scotia to obtain vaccination data. Data included the type, number of doses and date(s) on which COVID-19 vaccines were received. We created a vaccination variable by determining if a patient had received 0–1 or ≥2 doses >14 days prior to their emergency department visit, which is the time frame needed to confer vaccine-mediated immunity.¹⁴

Outcome

The primary outcome was all-cause in-hospital mortality including emergency department mortality. All patients had complete follow-up data at the time of the data cut. We categorised patients who were discharged from the hospital as alive according to their latest hospitalisation.

Sample size

For external validation studies, 100–250 events are recommended to obtain reasonable statistical power.¹⁵ The cohort had 1654 in-hospital deaths, which exceeded the number of events required. For the purposes of updating the CCMS and adding the vaccination variable, the model has 16 degrees of freedom. Conservative rules of thumb suggest 10 events per degree of freedom, and thus 160 events are required for the update cohort.¹⁶

External validation of the CCMS

We created an external validation cohort consisting of patients who were not part of the derivation or internal validation cohorts for the CCMS. We examined missing data and only two variables had a small proportion of missing data: 0.16% for patient residence and 2.66% for an initial respiratory rate at emergency department. Since the proportion of patients having any variable missing was less than 3% of the cohort, we performed single imputation using predictive mean matching methods using all of the other predictor variables from the CCMS.¹⁷ We examined the distribution of the CCMS through plots, including a calibration plot that showed the observed to expected probabilities of outcome and a receiver operating characteristic (ROC) curve. We reported the optimism-corrected AUC receiver operator curve with 95% CIs. We calculated sensitivity, specificity and negative predictive and positive predictive values for different rule in and rule out cut-off values with 95% CIs.

Planned update and validation of the adjusted CCMS (CCMS^adj)

We planned to update the original rule in a subset of patients from provinces that provided access to linked vaccination data, which we call the CCMS^adj cohort. We split the CCMS^adj cohort into an update cohort and a validation cohort by randomly assigning patients to be in either the update or validation cohorts. Using all the original predictors from the CCMS and including a vaccination variable (0–1 doses vs 2+ doses), we fit a logistic regression on the update CCMS^adj cohort and assigned points to predictors to obtain the CCMS^adj score. We assessed the CCMS^adj’s performance in the validation cohort and the combined update and validation cohorts (CCMS^adj cohort) similar to the external validation of the CCMS. We used DeLong’s test to assess for statistically significant differences in performance between the AUCs for the original CCMS and the CCMS^adj in the CCMS^adj cohort.

We performed analyses in R using the rms and pROC packages.^18–20 To ensure patient privacy, a cell size restriction policy prohibited reporting counts of less than five. All p values were two sided.

Patient and public involvement

CCEDRRN has a pan-Canadian patient partner committee with lived and/or caregiving experience with COVID-19. The committee reviewed and provided feedback on data collection fields and the research protocol.

Results

External validation of the CCMS

There were 45 157 patients who presented to participating emergency departments with COVID-19 during the study period (figure 1A), of whom 39 682 (44 253 emergency department visits, 1654 deaths) were eligible for inclusion into the external validation of the CCMS. Patient and emergency department visit characteristics as well as outcomes are described in table 1.

Table 1

Characteristics and outcomes in the CCMS external validation and CCMS^adj update cohort

	CCMS external validation cohort (n=44 253)	CCMSadj cohort (n=16 559)
		Update (n=8176)	Validation (n=8383)
Age in years, mean (SD)	53.3 (19.9)	52.1 (19.3)	52.0 (19.2)
Female (%)	21 989 (49.7)	3857 (47.2)	4053 (48.3)
Province (%)
British Columbia	14 894 (33.7)	7355 (90.0)	7539 (89.9)
Alberta	12 955 (29.3)	0	0
Quebec	7164 (16.2)	0	0
Ontario	5741 (13.0)	0	0
Saskatchewan	1740 (3.9)	0	0
Nova Scotia	1665 (3.8)	821 (10.0)	844 (10.1)
New Brunswick	94 (0.2)	0	0
Arrival from (%)
Home	41 771 (94.4)	7629 (93.3)	7832 (93.4)
Long-term care or other institutional living	1255 (2.8)	153 (1.9)	176 (2.1)
No fixed address	1155 (2.6)	378 (4.6)	358 (4.3)
Missing	72 (0.2)	16 (0.2)	17 (0.2)
Arrival mode (%)
Self	26 976 (61.0)	5280 (64.6)	5424 (64.7)
Ambulance or police	17 277 (39.0)	2896 (35.4)	2959 (35.3)
Arrival heart rate, beats/min, mean (SD)	94.0 (19.7)	94.7 (19.2)	93.9 (18.9)
Arrival systolic blood pressure, mm Hg, mean (SD)	130.8 (22.1)	131.9 (22.2)	132.1 (22.7)
Arrival diastolic blood pressure, mm Hg, mean (SD)	77.4 (13.3)	75.9 (13.2)	75.9 (13.3)
Arrival respiratory rate/min, mean (SD)	20.5 (5.9)	20.1 (5.6)	20.0 (5.7)
Arrival temperature in degrees Celsius, mean (SD)	37.0 (0.9)	37.1 (0.9)	37.1 (0.8)
Presence of respiratory distress (%)	9983 (22.6)	1077 (13.2)	1137 (13.6)
Top 5 COVID-19 symptoms (%)
Cough	23 566 (53.3)	4893 (59.8)	5037 (60.1)
Shortness of breath (dyspnoea)	20 761 (46.9)	3988 (48.8)	4015 (47.9)
Fever	15 972 (36.1)	3421 (41.8)	3582 (42.7)
Fatigue/malaise	13 357 (30.2)	2463 (30.1)	2500 (29.8)
Chest pain (includes discomfort or tightness)	10 961 (24.8)	2379 (29.1)	2300 (27.4)
Top 10 comorbidities (%)
Hypertension	11 706 (26.5)	2207 (27.0)	2164 (25.8)
Diabetes	6457 (14.6)	1382 (16.9)	1373 (16.4)
Dyslipidaemia	6132 (13.9)	1200 (14.7)	1191 (14.2)
Mental health diagnosis	5318 (12.0)	1021 (12.5)	1023 (12.2)
Asthma	3247 (7.3)	689 (8.4)	749 (8.9)
Chronic neurologic disorder (not dementia)	2965 (6.7)	656 (8.0)	679 (8.1)
Coronary artery disease	2959 (6.7)	521 (6.4)	582 (6.9)
Hypothyroidism	2826 (6.4)	504 (6.2)	562 (6.7)
Rheumatologic disorder	2820 (6.4)	579 (7.1)	615 (7.3)
Chronic lung disease (not asthma or pulmonary fibrosis)	2524 (5.7)	528 (6.5)	521 (6.2)
Oxygen required in emergency department (%)	5507 (12.4)	1295 (15.8)	1355 (16.2)
Vaccine status*
0 dose	NA	3736 (45.7)	3853 (46.0)
1 dose	NA	512 (6.3)	469 (5.6)
2 or 2+ doses	NA	3928 (48.0)	4061 (48.4)
Emergency department disposition (%)
Discharged	27 055 (61.1)	5478 (67.0)	5560 (66.3)
Admitted	16 676 (37.7)	2633 (32.2)	2750 (32.8)
Left against medical advice	215 (0.5)	45 (0.6)	48 (0.6)
Died in emergency department	58 (0.1)	7 (0.1)	12 (0.1)
Other†	242 (0.5)	10 (0.1)	9 (0.1)
Missing	7 (<0.1)	3 (<0.1)	4 (<0.1)
Died in emergency department or in hospital (%)	1654 (3.7)	226 (2.8)	238 (2.8)

*Vaccination status was only available for the CCMS^adj update and validation cohorts. There were 95, 25 and 106 deaths in the update cohort for doses 0, 1 and 2 or 2+ groups, respectively. There were 108, 21 and 109 deaths in the validation cohort for doses 0, 1 and 2 or 2+ groups, respectively.

†Included discharge to homeless shelter or COVID-19 isolation facility.

CCMS^adj, adjusted Canadian COVID-19 Mortality Score.

When the CCMS was applied to the external validation cohort, scores ranged from −1 to 18, and the observed and predicted risks were close (figure 2). The AUC was 0.88 (95% CI 0.87 to 0.88), with excellent performance across a range of thresholds to rule in and rule out in-hospital mortality (table 2). A score of 10 or less would categorise patients as being at low risk for in-hospital death, with a negative predictive value of 98.4% (95% CI 98.3% to 98.5%). Patients in the low-risk group had an in-hospital mortality rate of <1.6%. For scores of 15 or higher, the observed in-hospital mortality rate was >54.1% and the CCMS would categorise such patients as being at high risk for in-hospital death, with a specificity of >98.8% (95% CI 99.7% to 99.8%).

Figure 2. Distribution and performance of the Canadian COVID-19 Mortality Score in external validation. (A) distribution of the score, (B) observed in-hospital mortality rates across the range of the score, (C) predicted versus observed risk of in-hospital death (dashed line represents line of no difference between predicted and observed risk) and (D) receiver-operating characteristic curve (ROC) with area under the curve (AUC) and associated 95% CIs.

Table 2

Performance of the CCMS to rule out and rule in in-hospital mortality in external validation cohort

Score	N (%) n=44 253	Sensitivity % (95% CI)	Specificity % (95% CI)	Negative LR	Positive LR	PPV (%)	NPV (%)	In-hospital mortality (%)
Rule out:
≤−1	1049	100 (99.8 to 100)	2.5 (2.3 to 2.6)	0.0	1.0	3.8	100	0.0
≤0	4563	100 (99.8 to 100)	10.7 (10.4 to 11.0)	0.0	1.1	4.2	100	0.0
≤1	8011	99.8 (99.4 to 99.9)	18.8 (18.4 to 19.2)	0.0	1.2	4.6	100	0.0
≤2	10 361	99.4 (98.9 to 99.7)	24.3 (23.9 to 24.7)	0.0	1.3	4.9	99.9	0.1
≤3	12 442	99.2 (98.7 to 99.6)	29.2 (28.7 to 29.6)	0.0	1.4	5.2	99.9	0.1
≤4	14 973	98.7 (98.1 to 99.2)	35.1 (34.6 to 35.6)	0.0	1.5	5.6	99.9	0.1
≤5	18 298	97.5 (96.7 to 98.2)	42.9 (42.4 to 43.3)	0.1	1.7	6.2	99.8	0.2
≤6	22 411	95.8 (94.7 to 96.7)	52.4 (52.0 to 52.9)	0.1	2.0	7.3	99.7	0.3
≤7	26 790	92.9 (91.6 to 94.1)	62.6 (62.2 to 63.1)	0.1	2.5	8.8	99.6	0.4
≤8	31 208	87.0 (85.3 to 88.6)	72.8 (72.3 to 73.2)	0.2	3.2	11.0	99.3	0.7
≤9	35 234	77.0 (74.9 to 79.0)	81.8 (81.4 to 82.2)	0.3	4.2	14.1	98.9	1.1
≤10	38 512	62.6 (60.2 to 64.9)	89.0 (88.7 to 89.2)	0.4	5.7	18.0	98.4	1.6
Rule in:
≥10	9019	77.0 (74.9 to 79.0)	81.8 (81.4 to 82.2)	0.3	4.2	14.1	98.9	14.1
≥11	5741	62.6 (60.2 to 64.9)	89.0 (88.7 to 89.2)	0.4	5.7	18.0	98.4	18.0
≥12	3380	47.6 (45.2 to 50.0)	93.9 (93.7 to 94.1)	0.6	7.8	23.3	97.9	23.3
≥13	1633	29.6 (27.4 to 31.9)	97.3 (97.2 to 97.5)	0.7	11.0	30.0	97.3	30.0
≥14	677	16.3 (14.5 to 18.1)	99.0 (98.9 to 99.1)	0.8	17.0	39.7	96.8	39.7
≥15	231	7.6 (6.3 to 8.9)	99.8 (99.7 to 99.8)	0.9	30.4	54.1	96.5	54.1
≥16	48	1.6 (1.1 to 2.4)	100 (99.9 to 100)	1.0	33.1	56.2	96.3	56.2
≥17	6	0.3 (0.1 to 0.7)	100 (100)	1.0	128.8	83.3	96.3	83.3

CCMS, Canadian COVID-19 Mortality Score; LR, likelihood ratio; NPV, negative predictive value; PPV, positive predictive value.

Update and validation of the CCMS^adj

The update cohort included 7306 patients (8176 emergency department visits, 226 deaths) and the validation cohort 7518 patients (8383 emergency department visits, 238 deaths) for whom vaccination data were available (see, table 3). When all individual predictors from the CCMS along with the vaccination variable were entered into a regression model to update the rule in the update cohort, the estimates suggested recalibrating the points allocated for age, residence type, respiratory status and intubation status and not assigning any points for the vaccination variable to form the CCMS^adj (table 3). The CCMS^adj performed well in both the CCMS^adj validation and combined update and validation cohorts, with AUCs of 0.91 (95% CI 0.89 to 0.92) and 0.90 (95% CI 0.88 to 0.92), respectively (online supplemental efigure 1). A score of 10 or less would categorise patients as being at low risk for in-hospital death, with a negative predictive value of 98.4% (95% CI 98.2% to 98.6%; online supplemental etable 3). Patients in the low-risk group had an in-hospital mortality rate of <1.6%. For scores of 15 or higher, the observed in-hospital mortality rate was 56.8%, and the CCMS^adj would categorise patients as being at high risk for in-hospital death, with a specificity of 99.9% (95% CI 99.8% to 99.9%). We compared the performance of the original CCMS and the updated CCMS^adj in the CCMS^adj cohort to ensure a comparison of performance in the same group of patients. The AUC was 0.88 (95% CI 0.87 to 0.89) for the CCMS compared with 0.90 (95% CI 0.88 to 0.92) for the CCMS^adj. The difference was statistically significant (p<0.01).

Table 3

Recalibrated associations between individual predictors of the original CCMS and vaccination data and in-hospital mortality to create the CCMS^adj

Variable	Estimate (SE)	OR (95% CI)	CCMSadj
Age*	1.44 (0.24)	4.27 (2.66 to 6.83)	<40: 0 40–49: 1 50–59: 2 60–69: 3 70–79: 4 ≥80: 6
Sex
Male	Ref	Ref	1
Female	−0.12 (0.16)	0.89 (0.65 to 1.21)	0
Arrival from
Home or community	Ref	Ref	0
No fixed address	0.61 (0.33)	1.84 (0.97 to 3.50)	1
Institution	0.07 (0.32)	1.06 (0.57 to 1.98)	0
Arrival mode
Self	Ref	Ref	0
Ambulance or police	0.16 (0.17)	1.17 (0.83 to 1.65)	1
Chest pain	−0.57 (0.21)	0.57 (0.38 to 0.86)	−1
Moderate/severe liver disease	1.00 (0.54)	2.72 (0.93 to 7.91)	2
Arrival respiratory rate, breaths/min†	−0.03 (0.18)	0.97 (0.69 to 1.38)	<20: 0 20–29: 1 ≥30: 2
Mode and level of oxygen in ED
No oxygen	Ref	Ref	0
Nasal prongs<6 L/min	1.38 (0.21)	9.08 (5.44 to 15.14)	3
Nasal prongs>6 L/min, facemask or simple rebreather	2.21 (0.26)	9.08 (5.44 to 15.14)	4
BiPAP, CPAP or HFNO	2.55 (0.27)	12.77 (7.38 to 21.58)	5
Intubation	3.93 (3.95)	51.83 (29.82 to 90.10)	7
Vaccination
0/1 dose	Ref	Ref	0
≥2 doses	0.01 (0.16)	1.00 (0.73 to 1.37)	0

*OR was calculated between 51 and 79 years.

†OR was calculated between 18 and 26 breaths/min.

BiPAP, bilevel positive airway pressure; CCMS^adj, adjusted Canadian COVID-19 Mortality Score; CPAP, continuous positive airway pressure; ED, emergency department; HFNO, high flow nasal oxygen.

Discussion

The CCMS, originally developed and validated using pan-Canadian data from patients presenting to emergency departments during the first two pandemic waves, remained highly predictive of in-hospital mortality in a new and large pan-Canadian cohort of patients during a period of Omicron dominance when vaccinations were widespread.¹ Updating the rule by recalibrating the weights of the original variables only changed the weight of a few variables. Adding vaccination status did not add to the rule’s predictive performance. Updating only provided marginal improvements in mortality risk prediction. Given the larger sample size in which we were able to externally validate the original CCMS model, we recommend using the CCMS at the bedside for mortality risk prediction to inform goals of care conversations in critically ill patients presenting to emergency departments. In times of crisis resource management when ventilator and critical care capacity are inadequate, the CCMS could also be used to inform rational and evidence-based resource allocation with accurate mortality prediction.

During the early pandemic, many predictive models were developed and validated to predict COVID-19-related outcomes.^{21 22} However, most were developed for use among admitted patients and made use of laboratory variables at admission.^{23 24} Few were validated or updated for use in Omicron subvariants at a time when natural infection, vaccine-mediated immunity and hybrid immunity were widespread.^25–27 The QCOVID4 risk algorithm was derived and validated using large linked administrative datasets from the UK to predict mortality risk in the general population during the BA.1 dominant Omicron wave (until 31 March 2022).²⁵ Similar to our model, the authors found that the same factors as earlier in the pandemic remained important predictors. However, with over 50 variables, QCOVID4 requires access to linked administrative health data or coded comorbidity data within electronic health records into which the model can be preprogrammed to apply in clinical practice. QCOVID4 is helpful in healthcare planning and in identifying subgroups of the population who should be prioritised for vaccinations and therapeutic interventions. The QCOVID4 was neither intended nor validated for use as a bedside clinical decision-making tool for emergency department patients and their caregivers. A similar study from Israel used linked population-level administrative databases to derive and validate a model to predict severe COVID-19 from the general population in a cohort infected during the early Omicron wave (until 18 January 2022).²⁷ However, similar to QCOVID4, Mizrahi Reuveni et al’s²⁷ model relies on many variables derived from linked administrative records and is useful to identify patients in the general population who will most likely benefit from prophylactic or outpatient therapeutic interventions. Patients included by Mizrahi Reuveni et al were younger and healthier than those presenting to the emergency department with COVID-19 in our study, as evidenced by a positive predictive value of the model of 2.7% in the highest-risk group, compared with a much greater positive predictive value of 54.1% for the highest-risk groups with scores of >15 in the CCMS for emergency department patients. Portuondo-Jiménez et al developed a predictive model including a subgroup of patients presenting to the emergency department, but validated in a cohort including only in a small number of patients infected with Omicron subvariants (recruitment ended in 9 January 2022, which was the beginning of the Omicron wave).²⁶ Similar to other models, this model contains 28 variables derived from International Classification of Diseases-coded comorbidities from administrative data, many of which are not readily available at the bedside at the time a patient requires immediate resuscitation decisions. The Pandemic Respiratory Infection Emergency System Triage (PRIEST) Score was initially derived and validated in the early pandemic²⁸ and subsequently updated and validated in the Omicron wave in low-income and middle-income countries.²⁹ It included the patient’s performance status as a variable, which is not routinely collected on emergency department arrival, yet resulted in inferior performance in both internal and external validation studies.^{26 27} Finally, previous models included patients presenting with ‘do not resuscitate’ orders and other restrictions in their level of care that would have precluded life-sustaining mechanical ventilation and critical care admission and were thus more likely to die.^25–27 This introduces self-fulfilling prophecy bias, whereby the prognostic performance of mortality risk prediction may be inflated.^{13 30} For these reasons, previous models should not be used for mortality risk prediction in emergency department patients whose goals of care are unrestricted at the time of presentation. These factors limit the utility of previously updated models for use in the emergency department.^25–29

Interestingly, we found that adding vaccination status did not yield additional predictive power, beyond the previously included clinical variables in our model. This should not be misconstrued as an absence of evidence of vaccine effectiveness. The effectiveness of vaccines in preventing COVID-19-related emergency department visits and severe disease has been proven conclusively.^{31 32} Our cohort excludes the many patients who would likely have presented to the emergency department if not vaccinated.³³ However, our data indicate that once patients presented to the emergency department with COVID-19 at a time when most Canadians had been vaccinated or recovered from prior natural infections, other demographic and clinical variables were more predictive.³⁴ This simplifies risk prediction, in particular for patients who are sick and unable to provide a history.

Strengths of our work include the large socially and ethnographically diverse and representative Canadian population which spans a geographic region of six time zones, and our ability to ensure the absence of selection bias of patients by requiring sites to capture data on >99% of COVID-19 presentations.⁵ We did not include prior natural infection as a candidate predictor variable in our model as most countries have abandoned routine SARS-CoV-2 testing, precluding reliable and complete ascertainment. The rapid growth of natural infection-mediated immunity during the study period, often through undocumented and asymptomatic infections, underscores this and increases the generalizability of the CCMS to patients who had asymptomatic prior infections or are unable to recount prior infections at the time of arrival.³⁴ In other words, the application of the CCMS does not require knowledge of prior SARS-CoV-2 infections or vaccination status, which increases its generalisability and usability. Finally, our model does not require access to linked administrative data or comorbidities documented in electronic medical records, yet could easily be preprogrammed into existing electronic health records with linkage to triage vital signs and oxygen delivery mode to automate its application to inform clinical decisions.

Limitations

While our study externally validated the CCMS in a new cohort of patients at a time when different variants were circulating, it has not been validated in other health systems without universal health coverage, or in other care settings and should only be used for emergency department patients. Prospective data collection is considered optimal for clinical decision rule derivation and validation. However, infection control precautions during the pandemic precluded prospective data collection and would have introduced volunteer bias, limiting our model’s generalisability by excluding patients unable to provide informed consent.^{22 35} Instead, we validated the inter-rater agreement of retrospective variables as well as vaccination variables in prior studies.^{5 11} Nonetheless, the CCMS should be evaluated prospectively to understand any differences in performance when applied to retrospective data compared with real-time use, and to assess its impact on clinical care. Finally, we were only able to update the CCMS^adj in populations in two provinces that provided access to linked vaccination data due to bureaucratic hurdles in other provinces, potentially limiting the external validity of the CCMS^adj.³⁶ Interestingly, in this smaller cohort, mortality was lower than in the larger external validation cohort, which could reflect differences in population, healthcare access or healthcare delivery between provinces.

In summary, the CCMS remained highly accurate in predicting mortality from Omicron, and only marginal improvement through recalibration. Vaccination status did not improve the performance of the updated rule. The CCMS can be used to inform patient prognosis and guide clinical decision-making for emergency department patients with COVID-19.

We thank the UBC clinical coordinating centre staff, the UBC legal, ethics, privacy and contract staff and the research staff at each of the participating institutions in the network outlined in the attached online supplemental file 1 . The network would not exist today without the dedication of these professionals. Thank you to all of our patient partners who shared their lived experiences and perspectives to ensure that the knowledge we cocreate addresses the concerns of patients and the public. Creating the largest network of collaboration across Canadian emergency departments would not have been feasible without the tireless efforts of emergency department chiefs, and research coordinators and research assistants at participating sites. Finally, our most humble and sincere gratitude to all of our colleagues in medicine, nursing and the allied health professions who have been on the front lines of this pandemic from day one staffing our ambulances, emergency departments, intensive care units and hospitals bravely facing the risks of COVID-19 to look after our fellow citizens and after one another. We dedicate this network to you.

Data availability statement

Data are available on reasonable request. Raw data are available on reasonable request. They can be shared after approval by the Executive Committee through a process outlined on our website (https://www.ccedrrn.com/). Unpublished analyses are available on reasonable request through contact with the corresponding author.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

The University of BC Clinical Research Ethics Board (H20-01015) and the research ethics boards of all participating institutions reviewed and approved the study protocol with a waiver for informed consent, allowing us to capture a complete sample.

Footnote

Collaborators Patrick Fok; Hana Wiemer; Samuel Campbell; Kory Arsenault; Tara Dahn; Kavish Chandra; Joel Turner; Lars Grant; Éric Mercier, Greg Clark; Sébastien Robert; Raoul Daoust; Ivy Cheng; Michelle Welsford; Robert Ohle; Rohit Mohindra; Megan Landes; Tomislav Jelic; Philip Davis; Brian Rowe; Katie Lin; Andrew McRae; Stephanie VandenBerg; Jake Haward; Jaspreet Khangura; Daniel Ting; Maja Stachura; Frank Scheuermeyer; Baljeet Brar; John Taylor; Ian Martin; Sean Wormsbecker; Elizabeth Purssell; Lee Graham

Contributors All authors (CMH, RR, JP, DSY, JY, PMA, SCB and LJM) conceived the study, with input on the design and selection of variables from all other contributors. CMH and PMA obtained funding on behalf of the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN) investigators. CMH and DSY managed data collection along with other members of the CCEDRRN and verify the accuracy of underlying data. CMH, RR, JP and DSY developed the analytic plan. DSY performed the analysis, with assistance from RR and CMH, including accessing and verification of underlying data. All contributors provided input on the interpretation of our findings. CMH, JP, RR and DSY drafted the manuscript. All authors reviewed and approved the final manuscript for publication. All authors agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved the final manuscript for publication. CMH is the guarantor of the study and accepts full responsibility for the work and/or the conduct of the study, had access to the data and controlled the decision to publish.

Funding The Canadian COVID-19 Emergency Department Rapid Response Network is funded by the Canadian Institutes of Health Research (grant 447679), the BC Academic Health Science Network Society (grant N/A), BioTalent Canada (grant N/A), Genome BC (grants COV024 and VAC007), the Ontario Ministry of Colleges and Universities (grant C-655-2129), the Saskatchewan Health Research Foundation (grant 5357), the Fondation du CHU de Québec (grant 4007) and the Public Health Agency of Canada/COVID-19 Immunity Task Force (2122-HQ-000054). PMA holds a Fonds de recherche du Québec—Santé Clinical Scholar Award (grant N/A).

Competing interests None declared.

Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ Hohl CM, Rosychuk RJ, Archambault PM, et al. The CCEDRRN COVID-19 Mortality Score to predict death among nonpalliative patients with COVID-19 presenting to emergency departments: a derivation and validation study. CMAJ Open 2022; 10: E90–9. doi:10.9778/cmajo.20210243

² Bobrovitz N, Ware H, Ma X, et al. Protective effectiveness of previous SARS-CoV-2 infection and hybrid immunity against the omicron variant and severe disease: a systematic review and meta-regression. Lancet Infect Dis 2023; 23: 556–67. doi:10.1016/S1473-3099(22)00801-5

³ Hohl CM, Rosychuk RJ, Hau JP, et al. Treatments, resource utilization, and outcomes of COVID-19 patients presenting to emergency departments across pandemic waves: an observational study by the Canadian COVID-19 Emergency Department Rapid Response Network (CCEDRRN). CJEM 2022; 24: 397–407. doi:10.1007/s43678-022-00275-3

⁴ Moons KGM, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015; 162: W1–73. doi:10.7326/M14-0698

⁵ Hohl CM, Rosychuk RJ, McRae AD, et al. Development of the Canadian COVID-19 Emergency Department Rapid Response Network population-based registry: a methodology study. CMAJ Open 2021; 9: E261–70. doi:10.9778/cmajo.20200290

⁶ Brooks SC, Rosychuk RJ, Perry JJ, et al. Derivation and validation of a clinical decision rule to risk-stratify COVID-19 patients discharged from the emergency department: The CCEDRRN COVID discharge score. J Am Coll Emerg Physicians Open 2022; 3: e12868. doi:10.1002/emp2.12868

⁷ McRae AD, Hohl CM, Rosychuk R, et al. CCEDRRN COVID-19 Infection Score (CCIS): development and validation in a Canadian cohort of a clinical risk score to predict SARS-CoV-2 infection in patients presenting to the emergency department with suspected COVID-19. BMJ Open 2021; 11: e055832. doi:10.1136/bmjopen-2021-055832

⁸ Hohl CM, Hau JP, Vaillancourt S, et al. Sensitivity and Diagnostic Yield of the First SARS-CoV-2 Nucleic Acid Amplification Test Performed for Patients Presenting to the Hospital. JAMA Netw Open 2022; 5: e2236288. doi:10.1001/jamanetworkopen.2022.36288

⁹ Davis P, Rosychuk R, Hau JP, et al. Diagnostic yield of screening for SARS-CoV-2 among patients admitted to hospital for alternate diagnoses: an observational cohort study. BMJ Open 2022; 12: e057852. doi:10.1136/bmjopen-2021-057852

¹⁰ Lin K, Xu K, Daoust R, et al. Prognostic association between d-dimer thresholds and 30-day pulmonary embolism diagnosis among emergency department patients with suspected SARS-CoV-2 infection: a Canadian COVID-19 Emergency Department Rapid Response Network study. CJEM 2023; 25: 134–42. doi:10.1007/s43678-022-00440-8

¹¹ Archambault PM, Rosychuk RJ, Audet M, et al. Accuracy of Self-Reported COVID-19 Vaccination Status Compared With a Public Health Vaccination Registry in Québec: Observational Diagnostic Study. JMIR Public Health Surveill 2023; 9: e44465. doi:10.2196/44465

¹² Bola R, Sutherland J, Murphy RA, et al. Patient-reported health outcomes of SARS-CoV-2-tested patients presenting to emergency departments: a propensity score-matched prospective cohort study. Pub Health (Fairfax) 2023; 215: 1–11. doi:10.1016/j.puhe.2022.11.016

¹³ Turgeon AF, Lauzier F, Simard J-F, et al. Mortality associated with withdrawal of life-sustaining therapy for patients with severe traumatic brain injury: a Canadian multicentre cohort study. CMAJ 2011; 183: 1581–8. doi:10.1503/cmaj.101786

¹⁴ Baden LR, El Sahly HM, Essink B, et al. Efficacy and Safety of the mRNA-1273 SARS-CoV-2 Vaccine. N Engl J Med 2021; 384: 403–16. doi:10.1056/NEJMoa2035389

¹⁵ Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 2016; 35: 214–26. doi:10.1002/sim.6787

¹⁶ Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 1996; 49: 1373–9. doi:10.1016/s0895-4356(96)00236-3

¹⁷ Harrell FE. Regression Modeling Strategies With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. 2nd edn. New York: Springer, 2015.

¹⁸ R Core Team. R: A Language and Environment for Statistical Computing. Vienna,Austria: R Foundation for Statistical Computing, 2021.

¹⁹ Harrell F. Rms: regression modeling strategies. r package version 6.3-0. 2022. Available: https://CRAN.R-project.org/package=rms

²⁰ Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 2011; 12: 77. doi:10.1186/1471-2105-12-77

²¹ Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 2020; 369: m1328. doi:10.1136/bmj.m1328

²² Buttia C, Llanaj E, Raeisi-Dehkordi H, et al. Prognostic models in COVID-19 infection that predict severity: a systematic review. Eur J Epidemiol 2023; 38: 355–72. doi:10.1007/s10654-023-00973-x

²³ Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: development and validation of the 4C Mortality Score. BMJ 2020; 370: m3339. doi:10.1136/bmj.m3339

²⁴ Carr E, Bendayan R, Bean D, et al. Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study. BMC Med 2021; 19: 23. doi:10.1186/s12916-020-01893-3

²⁵ Hippisley-Cox J, Khunti K, Sheikh A, et al. Risk prediction of covid-19 related death or hospital admission in adults testing positive for SARS-CoV-2 infection during the omicron wave in England (QCOVID4): cohort study. BMJ 2023; 381: e072976. doi:10.1136/bmj-2022-072976

²⁶ Portuondo-Jiménez J, Barrio I, España PP, et al. Clinical prediction rules for adverse evolution in patients with COVID-19 by the Omicron variant. Int J Med Inform 2023; 173: 105039. doi:10.1016/j.ijmedinf.2023.105039

²⁷ Mizrahi Reuveni M, Kertes J, Shapiro Ben David S, et al. Risk Stratification Model for Severe COVID-19 Disease: A Retrospective Cohort Study. Biomedicines 2023; 11: 767. doi:10.3390/biomedicines11030767

²⁸ Goodacre S, Thomas B, Sutton L, et al. Derivation and validation of a clinical severity score for acutely ill adults with suspected COVID-19: The PRIEST observational cohort study. PLoS ONE 2021; 16: e0245840. doi:10.1371/journal.pone.0245840

²⁹ Marincowitz C, Hodkinson P, McAlpine D, et al. LMIC-PRIEST: Derivation and validation of a clinical severity score for acutely ill adults with suspected COVID-19 in a middle-income setting. PLoS ONE 2023; 18: e0287091. doi:10.1371/journal.pone.0287091

³⁰ Teixeira FJP, Ahmad B, Gibatova V, et al. Do Neuroprognostic Studies Account for Self-Fulfilling Prophecy Bias in Their Methodology? The SPIN Protocol for a Systematic Review. Crit Care Explor 2023; 5: e0943. doi:10.1097/CCE.0000000000000943

³¹ Thompson MG, Stenehjem E, Grannis S, et al. Effectiveness of Covid-19 Vaccines in Ambulatory and Inpatient Care Settings. N Engl J Med 2021; 385: 1355–71. doi:10.1056/NEJMoa2110362

³² Feikin DR, Higdon MM, Abu-Raddad LJ, et al. Duration of effectiveness of vaccines against SARS-CoV-2 infection and COVID-19 disease: results of a systematic review and meta-regression. The Lancet 2022; 399: 924–44. doi:10.1016/S0140-6736(22)00152-0

³³ McNamara LA, Wiegand RE, Burke RM, et al. Estimating the early impact of the US COVID-19 vaccination programme on COVID-19 cases, emergency department visits, hospital admissions, and deaths among adults aged 65 years and older: an ecological analysis of national surveillance data. The Lancet 2022; 399: 152–60. doi:10.1016/S0140-6736(21)02226-1

³⁴ Murphy TJ, Swail H, Jain J, et al. The evolution of SARS-CoV-2 seroprevalence in Canada: a time-series study, 2020-2023. CMAJ 2023; 195: E1030–7. doi:10.1503/cmaj.230249

³⁵ Stiell IG, Wells GA. Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 1999; 33: 437–47. doi:10.1016/s0196-0644(99)70309-4

³⁶ McRae AD, Archambault P, Fok P, et al. Reducing barriers to accessing administrative data on SARS-CoV-2 vaccination for research. CMAJ 2022; 194: E943–7. doi:10.1503/cmaj.211712