Characterizing medical patients with delirium: A cohort study comparing ICD-10 codes and a validated chart review method

Abstract

Background

Delirium is a major cause of preventable mortality and morbidity in hospitalized adults, but accurately determining rates of delirium remains a challenge.

Objective

To characterize and compare medical inpatients identified as having delirium using two common methods, administrative data and retrospective chart review.

Methods

We conducted a retrospective study of 3881 randomly selected internal medicine hospital admissions from six acute care hospitals in Toronto and Mississauga, Ontario, Canada. Delirium status was determined using ICD-10-CA codes from hospital administrative data and through a previously validated chart review method. Baseline sociodemographic and clinical characteristics, processes of care and outcomes were compared across those without delirium in hospital and those with delirium as determined by administrative data and chart review.

Results

Delirium was identified in 6.3% of admissions by ICD-10-CA codes compared to 25.7% by chart review. Using chart review as the reference standard, ICD-10-CA codes for delirium had sensitivity 24.1% (95%CI: 21.5–26.8%), specificity 99.8% (95%CI: 99.5–99.9%), positive predictive value 97.6% (95%CI: 94.6–98.9%), and negative predictive value 79.2% (95%CI: 78.6–79.7%). Age over 80, male gender, and Charlson comorbidity index greater than 2 were associated with misclassification of delirium. Inpatient mortality and median costs of care were greater in patients determined to have delirium by ICD-10-CA codes (5.8% greater mortality, 95% CI: 2.0–9.5 and $6824 greater cost, 95%CI: 4713–9264) and by chart review (11.9% greater mortality, 95%CI: 9.5–14.2% and $4967 greater cost, 95%CI: 4415–5701), compared to patients without delirium.

Conclusions

Administrative data are specific but highly insensitive, missing most cases of delirium in hospital. Mortality and costs of care were greater for both the delirium cases that were detected and missed by administrative data. Better methods of routinely measuring delirium in hospital are needed.

Introduction

Delirium is a common and treatable neurocognitive disorder, characterized by acute onset of change in mental status, psychomotor disturbance and hallucinations [1]. It affects up to 50% of hospitalized adults over the age of 65, is potentially preventable, and is associated with numerous negative outcomes in hospital including increased mortality, longer stays, and higher health care costs [2,3]. Post-discharge, it is associated with nursing home placement, declines in cognitive function, and incident dementia [4–6]. Given its great clinical and economic burden, improving prevention and management of delirium is increasingly of interest and delirium is now widely recognized as an indicator of healthcare quality [7–11]. However, diagnosing and measuring rates of in-hospital delirium remains a challenge. Tracking the impact of quality improvement initiatives and the conduct of system-wide evaluation are therefore difficult.

Delirium rates published in the literature are obtained through a variety of methods. Administrative data capture the rate of delirium diagnosis, chart review estimates the rate of delirium based on chart documentation, and clinical assessment determines the incidence or prevalence of delirium from signs, symptoms, and collateral information. Given its lower cost and ready accessibility, administrative data has been recommended for evaluating delirium rates according to quality standards in Australia and Canada [9,10]. This method is also used in population-based and health services research [12,13]. However, it is recognized that administrative data substantially underestimate the frequency of delirium compared to case ascertainment by either chart review or clinical assessment [12,14–22]. While some studies have compared administrative coding to clinical assessment, there are limited data comparing administrative codes to a validated chart review method [17–19,23]. Moreover, these studies have focused on post-surgical populations with limited data from general medical inpatients. From a system perspective, understanding who is, or is not, captured by each method may be of more use than understanding the accuracy of each detection method. With this knowledge, those working in healthcare institutions can decide which method of delirium identification is most appropriate for their aims when implementing quality assurance and improvement interventions to prevent delirium. In this study, we aimed to characterize and compare the populations of medical inpatients with delirium, as identified by administrative data and a chart review method.

Method

Design, setting and study population

We conducted a retrospective chart review study, using the CHART-DEL method ²³. We randomly selected 3881 hospital admissions from four healthcare institutions (six hospitals) in Toronto, Ontario. Participating institutions were St. Michael’s Hospital (SMH), University Health Network (UHN: Toronto General Hospital and Toronto Western Hospital), Sunnybrook Health Sciences Centre (SBK), and Trillium Health Partners (THP: Credit Valley Hospital and Mississauga Hospital). Each hospital is part of the GEMINI research network which collects administrative and clinical data on all adult (>18 year of age) patients who are admitted to or discharged from general medicine services [24,25]. These hospitals range in size from 433 to 1325 acute inpatient beds, with general internal medicine (GIM) patients accounting for 24% of hospital bed-days and 39% of admissions through the emergency department during the study period of April 1 2010 –March 20 2015 [24]. Ontario has a publicly funded health care system that insures residents for medically necessary services provided in hospital settings.

Data sources

GEMINI collects and links administrative and clinical data for patients admitted to or discharged from general medicine services [24,26]. The administrative data include information reported by participating hospitals to the Canadian Institute for Health Information (CIHI) Discharge Abstract Database (DAD): demographic characteristics, diagnoses, interventions, discharge destinations, and resource use. For each hospitalization up to 25 diagnoses are recorded at the time of discharge in the DAD.

The CHART-DEL method, which has a sensitivity of 74% and specificity of 83% compared to clinical assessment, was used to identify delirium during hospitalization [23]. Information about and documentation of acute changes in mental status were transcribed verbatim, as well as improvements in mental state.

Chart-based delirium identification

Trained abstractors, who all had clinical experience, used the CHART-DEL method to code for the presence of delirium during admission to hospital [23]. As outlined in the CHART-DEL training guide, all abstractors participated in the recommended training procedure [27]. This included didactic sessions on delirium and the chart review method. Initial chart abstractions were performed jointly and each abstractor then reviewed ten charts independently, which were compared to an expert rater (EH, KAS, DG, TT) to ensure accuracy and reliability. Abstractors reviewed the entire chart, including daily progress notes by physicians and nursing staff, assessments by allied health professionals such as occupational and physical therapy, specialist physician consultations, and admission and discharge notes. They were blinded to administrative data coding of delirium. If abstractors were unable to determine whether delirium was present, the case was labelled as uncertain and then reviewed by expert raters (experienced physicians in internal medicine and consultation-liaison psychiatry with specialized interest in delirium) at each institution to make a final determination. Chart data were accessed between August 6 2016 and December 20 2019. Authors did not have access to information that could identify individual participants during or after data collection.

Administrative data delirium identification

Administrative data were generated by hospital-based chart coders who review the patient chart post-discharge and code diagnoses using the enhanced Canadian version of the 10^th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10-CA). This is a standard process completed for all patients discharged from hospital in Canada. Hospital coders did not have access to study data. Given challenges with delirium nomenclature, we used both narrow (codes that include delirium related and unrelated to substances) and broad definitions (codes that include encephalopathy) of delirium in administrative data [18,28] (see S1 Appendix). In contrast to the United States, there is no billing incentive for encephalopathy compared to delirium and the broad definition only identified three additional cases, which were not analysed further.

Variables

Administrative health data were linked with clinical data extracted from hospital information systems [24]. Demographic variables included age and gender. Clinical variables included the laboratory-based acute physiology score (LAPS), Charlson comorbidity index (CCI), and a pre-admission diagnosis of dementia [29–31]. Processes of care variables included length of stay, admission to an intensive care unit (ICU), and ICU length of stay (LOS). Outcome variables included total cost of hospitalization (direct and indirect costs from the emergency department and inpatient portions of the hospital visit), mortality, 30-day readmission to general internal medicine (GIM) at any participating hospital, new diagnosis of dementia at discharge, and a new discharge destination of a long term care (LTC) facility [32].

In addition to variables extracted from these electronic systems, processes of care variables were also coded during the process of manual chart review. These included specialist physician consultation (geriatric medicine, psychiatry, and geriatric psychiatry), allied health (physiotherapy and occupational therapy) assessment, and completion of neurocognitive screening.

Statistical analysis

The sample size was selected based on the feasibility of chart review, with an estimated 30–60 minutes required per admission. Chart-based identification of delirium was compared with administrative data identification by calculating sensitivity, specificity, positive predictive value, and negative predictive value. The chart-based method was used as the reference standard. Descriptive statistics were used to compare demographic, clinical, service use, and outcome variables between those identified by chart review and those identified through administrative data. Given the sample size, standardized differences (SD) were used to assess statistical significance, with differences of greater than 0.1 considered significant [33]. Baseline factors associated with incorrect identification of delirium by administrative data (false positives and false negatives) were examined using unadjusted and adjusted logistic regression models, including patient demographic and clinical variables. A random subsample of 5% of charts were coded by two abstractors to calculate inter-rater reliability using a kappa score.

Ethics approval

Ethics approval was obtained from the research ethics board of all participating hospitals (St. Michael’s Hospital, University Health Network, Sunnybrook Health Sciences Centre, Trillium Health Partners), with St. Michael’s Hospital as the Board of Record and a waiver of patient consent for this retrospective study using routinely collected health data.

Results

Of the 3881 randomly selected patient records, the final sample included 3859 patient admissions across six hospitals. Twenty-two were excluded because we were unable to locate patient records (Fig 1). Interrater reliability on the double-coded charts (5% sample, N = 193) was high, with kappa = 0.9 for presence or absence of delirium.

[Figure omitted. See PDF.]

As shown in Table 1, the sample was 49.6% male and the median age was 73 years (IQR: 57–84). The mean LAPS was 20.48 (SD: 17.56). Similar proportions of the sample had either low (41.5%, score of 0) or high (42.3%, score >2) levels of comorbidity as measured by the Charlson comorbidity index. Approximately 2% (N = 67) had dementia noted as a pre-admission comorbidity [31]. Delirium was identified in 992 (25.7%) patients by chart review, compared to 245 (6.3%) using ICD-10 codes. In only six cases (0.16%) was delirium identified by ICD-10 codes, but not by chart review (S2 Appendix shows sample characteristics by chart review delirium identification). The use of the chart review method detected 753 more patients than those identified by ICD-10 codes alone.

[Figure omitted. See PDF.]

Tables 1–3 compare baseline, process, and outcome data across four groups: those identified as not having delirium by both methods (no delirium, n = 2861), those identified as having delirium by ICD-10 codes (n = 245), those identified as having delirium by chart review (n = 992), and those identified by chart review but not ICD-10 codes (n = 753). Delirium was associated with older age, greater LAPS, greater Charlson comorbidity index, and having a pre-admission diagnosis of dementia (Table 1). Patients with delirium identified by either method were more likely than patients without delirium to have assessments completed by occupational therapy, physiotherapy, geriatric medicine or psychiatry, and have cognitive testing completed (Table 2). Those with delirium identified by ICD-10 codes were more likely to have these assessments performed than those identified by chart review alone. Hospital length of stay and costs of care were greater for patients identified as having delirium by ICD-10 codes (median 11.45 days, IQR 5.83–22.17; median $11,023.60, IQR 5,125.00–22,223.10) or chart review (median 8.8 days, IQR 4.5–17.4; median $9,180.00, IQR 4,552.60–19,411.00) compared to patients without delirium (median 4.08 days, IQR 2.04–7.62; median $4,199.30, IQR 2,290.45–7,885.79). Similarly, inpatient mortality and ICU admission were greater in patients with delirium identified by ICD-10 codes (mortality 9.4%, ICU 10.2%) or chart review (mortality 15.5%, ICU 15.5%) compared to patients without delirium (mortality 3.6%, ICU 5.6%). A new diagnosis of dementia and new placement in long-term care were also greater in patients with delirium identified by either method compared to patients without delirium (Table 3).

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

Compared to delirium cases identified by chart review, delirium cases identified by ICD-10 codes had greater median hospital length of stay by 2.7 days (95% CI: 0.4, 4.3) whereas they had lower mortality by 6.1% (95%CI: -1.8, -10.5) and lower ICU use by 5.3% (95%CI: -0.9, -9.8). It was uncertain whether median costs of care were greater for delirium cases identified by ICD-10 codes than chart review, with the estimate of difference being $1,844 but confidence intervals including no difference (95%CI: -602, 4019). Using chart review as the standard, ICD-10 codes had specificity of 99.8% (95%CI: 99.5–99.9%), sensitivity of 24.1% (95%CI: 21.5–26.8%), positive predictive value of 97.6% (95%CI: 94.6–98.9%) and negative predictive value of 79.2% (95%CI: 78.6–79.7%). As shown in Table 4, age over 80, male gender, and Charlson Comorbidity Index greater than 2 were associated with misclassification of delirium (i.e. delirium present by chart review, but absent by administrative data (n = 753) or present by administrative data, but absent by chart review (n = 6)).

[Figure omitted. See PDF.]

Discussion

To assess delirium rates and populations for quality improvement and health services research, we need a reliable and scalable method for case identification. In this study, we compared routinely collected administrative data with a validated but resource-intensive chart review method for identifying delirium. We found that, although ICD-10 codes were highly specific, they substantially underreported the prevalence of delirium in hospitalized medical patients and missed approximately three-quarters of those identified by the chart review method. This suggests that studies and quality improvement (QI) projects which rely on administrative data alone for delirium identification likely miss most patients with delirium. Those that use administrative data to estimate costs and resource use associated with delirium would substantially underestimate the true burdens of delirium, and those that report outcomes would underestimate mortality and ICU admission, which is particularly important as delirium is known to be associated with death and critical illness [12,34]. Moreover, while there have been attempts to develop data-driven and machine learning approaches to predict delirium in hospital, these methods often use administrative data to train and test their models, which then replicates these limitations of identification through administrative data [35–38]

Importantly, our study demonstrates that there are differences between those identified with delirium by administrative coding and those identified by chart review. Administrative data was more likely to miss delirium in men, patients older than 80 years, and those with greater comorbidity. While we are not able to determine the mediating factors leading to these differences, previous research has demonstrated that documentation of features of the hyperactive subtype of delirium impacts administrative coding [18]. This discordance may also reflect differences in documentation of symptoms and behaviour across gender and illness severity groups. The observation that delirium is more likely to be misclassified by administrative data in patients over 80 years of age may indicate that coders struggle to distinguish between the acute mental status changes in delirium and chronic age-related cognitive changes, which is unsurprising given that this distinction can be challenging for clinicians.

Individuals with delirium identified by administrative data or chart review were more likely to have allied health assessments (OT or PT), specialist consultations (psychiatry and geriatric medicine), and cognitive testing. However, it appears that a greater proportion of those identified by administrative data had allied health and specialist consultations compared to those identified by chart review. From this study, we are not able to determine the directionality of this relationship. For example, these assessments may reflect recognition of a change in mental status or they may assist in identifying delirium when present, improving identification by both chart review and administrative data coding.

Strengths

This study is novel in terms of its size, use of CHART-DEL which is a validated method of chart review, and its focus on medical inpatients which represent the largest proportion of admissions and hospital bed days at the hospitals studied. While delirium research often focuses on surgical and intensive care populations [18,39], medical ward patients are at high risk of delirium and its negative impacts. Our study includes multiple sites, including both academic and community hospitals with similar delirium rates across sites.

Limitations

Although we did not include comparison to clinical assessment or screening, the proportion of patients identified by chart review as having delirium are similar to studies that have used these other methods [20,40]. Numerous tools for delirium screening have been validated, however, conducting and documenting these assessments is labor-intensive. It also requires significant institutional investment to support education, training, and audit and feedback [41–43]. Although routine screening using the Confusion Assessment Method (CAM) has since been implemented at several of the study sites, this was not conducted regularly during the study period [44]. Of note, despite twice daily screens by nursing staff and extensive staff training and education, delirium rates determined using the CAM at one study site were found to be lower than those determined by both chart review and administrative codes (3% unpublished data, personal communication). This is similar to other studies that have found that routine clinical screening under-reports up to three-quarters of cases compared to clinical assessments for research [41,45–47].

Previous studies comparing chart review to high-quality clinical assessments for research demonstrate that hypoactive delirium is more likely to be missed by chart review [48]. Inouye et al. (2005) concluded that the CHART-DEL method is appropriate to measure the impact of quality improvement initiatives and institutional “report cards” for delirium [23]. However, this method relies on documentation of delirium signs and symptoms in the chart and, when these continue to be under-reported or poorly documented, then delirium may not be identified. The chart review method may also result in false positive identification of delirium in patients with a higher baseline risk of delirium (e.g. those with dementia, severe illness). Performing regular chart reviews or reviewing large numbers of charts to track delirium rates also proves challenging from a human resources perspective, as each one can take a trained individual a substantial amount of time (15–60 minutes per chart) to complete. Nevertheless, the chart review method has been well-validated, is more feasible to implement, and does have some strengths above cross-sectional clinical assessments alone, such as the identification of symptoms that occurred at night. The most accurate method for determination of delirium combines clinical assessment with other sources of information, including collateral from informants and chart review.

Conclusions

Our study adds to the delirium literature by characterizing populations identified through administrative diagnosis codes and a validated chart review method. Understanding who is captured by each identification method is important for several reasons. At both an institutional and health system level, determining rates through administrative data is faster and less costly. However, the risk of under-diagnosis and misclassification of delirium with administrative data is high. Relying on administrative data alone for quality improvement initiatives may risk over- or under-estimating the effectiveness of delirium interventions. Moreover, at a system level, using administrative data alone will also significantly underestimate the burden of delirium in terms of total days in hospital, mortality, ICU admission, and attributable costs.

Improvements in low-cost approaches to identify delirium are urgently needed. Novel approaches, including machine learning, using more aspects of administrative, electronic clinical data, and medical records have the potential to improve accuracy of identification while being faster and less costly than chart review.

Supporting information

S1 Appendix. ICD-10 codes used.

https://doi.org/10.1371/journal.pone.0302888.s001

(DOCX)

S2 Appendix. Baseline and outcome characteristics association with delirium as determined by chart review.

https://doi.org/10.1371/journal.pone.0302888.s002

(DOCX)

S1 File. PLOS ONE clinical studies checklist.

https://doi.org/10.1371/journal.pone.0302888.s003

(DOCX)

S2 File. STROBE statement—checklist of items that should be included in reports of observational studies.

https://doi.org/10.1371/journal.pone.0302888.s004

(DOCX)

Acknowledgments

The authors would like to thank Janooshsheya (Jana) Balasundaram, Samia Amer, Karan Bajwa, Melody Hung, Reyam Nassif, Nino Parunashvili, Nikhil Patil, Mehrdad Rahmanian and Roxana Rahmanian for their work in conducting the delirium chart reviews for this study. This study was also supported by the GEMINI Investigators group.

Citation: Sheehan KA, Shin S, Hall E, Mak DYF, Lapointe-Shaw L, Tang T, et al. (2024) Characterizing medical patients with delirium: A cohort study comparing ICD-10 codes and a validated chart review method. PLoS ONE 19(5): e0302888. https://doi.org/10.1371/journal.pone.0302888

About the Authors:

Kathleen A. Sheehan

Roles: Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations: Department of Psychiatry, University of Toronto, Toronto, ON, Canada, Centre for Mental Health, University Health Network, Toronto, ON, Canada

ORICD: https://orcid.org/0000-0002-8700-0809

Saeha Shin

Roles: Data curation, Formal analysis, Methodology, Writing – review & editing

Affiliation: St. Michael’s Hospital, Unity Health Network, Toronto, ON, Canada

Elise Hall

Roles: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review & editing

Affiliations: Department of Psychiatry, University of Toronto, Toronto, ON, Canada, Department of Psychiatry, Unity Health Network, Toronto, ON, Canada

Denise Y. F. Mak

Roles: Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing – review & editing

Affiliation: St. Michael’s Hospital, Unity Health Network, Toronto, ON, Canada

Lauren Lapointe-Shaw

Roles: Data curation, Project administration, Writing – review & editing

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Department of Medicine, University Health Network, Toronto, ON, Canada

Terence Tang

Roles: Data curation, Project administration, Supervision, Writing – review & editing

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Institute for Better Health, Trillium Health Partners, Mississauga, ON, Canada

ORICD: https://orcid.org/0000-0002-1735-7298

Seema Marwaha

Roles: Data curation, Project administration, Supervision, Writing – review & editing

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Department of Medicine, Unity Health Network, Toronto, ON, Canada

Dov Gandell

Roles: Data curation, Funding acquisition, Project administration, Supervision, Writing – review & editing

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Department of Medicine, Sunnybrook Heatlh Sciences Centre, Toronto, ON, Canada

Shail Rawal

Roles: Data curation, Project administration, Supervision, Writing – review & editing

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Department of Medicine, University Health Network, Toronto, ON, Canada

Sharon Inouye

Roles: Methodology, Supervision, Writing – review & editing

Affiliations: Aging Brain Center, Hebrew Senior Life, Boston, MA, United States of America, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, United States of America

Amol A. Verma

Roles: Conceptualization, Data curation, Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing

¶‡ AAV and FR are joint senior authors.

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Department of Medicine, Unity Health Network, Toronto, ON, Canada

Fahad Razak

Roles: Conceptualization, Data curation, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review & editing

¶‡ AAV and FR are joint senior authors.

Affiliations: Department of Medicine, University of Toronto, Toronto ON, Canada, Department of Medicine, Unity Health Network, Toronto, ON, Canada

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

References

1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM-5®). American Psychiatric Pub; 2013. 991 p.

2. Inouye SK, Westendorp RGJ, Saczynski JS. Delirium in elderly people. Lancet. 2014 Mar 8;383(9920):911–22. pmid:23992774

3. Oh ES, Fong TG, Hshieh TT, Inouye SK. Delirium in older persons: Advances in diagnosis and treatment. JAMA. 2017 Sep 26;318(12):1161–74. pmid:28973626

4. Witlox J, Eurelings LSM, de Jonghe JFM, Kalisvaart KJ, Eikelenboom P, van Gool WA. Delirium in elderly patients and the risk of postdischarge mortality, institutionalization, and dementia: a meta-analysis. JAMA. 2010 Jul 28;304(4):443–51. pmid:20664045

5. MacLullich AMJ, Beaglehole A, Hall RJ, Meagher DJ. Delirium and long-term cognitive impairment. Int Rev Psychiatry. 2009 Feb;21(1):30–42. pmid:19219711

6. Jackson TA, MacLullich AMJ, Gladman JRF, Lord JM, Sheehan B. Undiagnosed long-term cognitive impairment in acutely hospitalised older medical patients with delirium: a prospective cohort study. Age Ageing. 2016 Jul;45(4):493–9. pmid:27076525

7. Wong K, Tsang A, Liu B, Schwartz R. The Ontario Senior Friendly Hospital Strategy: Delirium and Functional Decline Indicators [Internet]. Senior Friendly Hospital Indicators Working Group; 2012. Available from: https://www.rgptoronto.ca/wp-content/uploads/2017/12/SFH_Delirium_and_Functional_Decline_Indicators.pdf.

8. Gage L HDB. CCSMH Guideline Update: The Assessment and Treatment of Delirium [Internet]. Canadian Coalition for Seniors’ Mental Health; 2014. Available from: https://ccsmh.ca/wp-content/uploads/2016/03/2014-ccsmh-Guideline-Update-Delirium.pdf.

9. Australian Commission on Safety and Quality in Health Care. Delirium Clinical Care Standard [Internet]. Sydney: ACSQHC; 2016. Available from: https://www.safetyandquality.gov.au/sites/default/files/migrated/Delirium-Clinical-Care-Standard-Web-PDF.pdf.

10. Ontario Health. Quality Standard: Delirium Care for Adults [Internet]. Toronto: Ontario Health; 2021. Available from: https://hqontario.ca/Portals/0/documents/evidence/quality-standards/qs-delirium-quality-standard-en.pdf.

11. Bush SH, Marchington KL, Agar M, Davis DHJ, Sikora L, Tsang TWY. Quality of clinical practice guidelines in delirium: a systematic appraisal. BMJ Open. 2017 Mar 10;7(3):e013809. pmid:28283488

12. Webber C, Watt CL, Bush SH, Lawlor PG, Talarico R, Tanuseputro P. The occurrence and timing of delirium in acute care hospitalizations in the last year of life: A population-based retrospective cohort study. Palliat Med. 2020 Sep;34(8):1067–77. pmid:32515283

13. Zhong X, Lin J-Y, Li L, Barrett AM, Poeran J, Mazumdar M. Derivation and validation of a novel comorbidity-based delirium risk index to predict postoperative delirium using national administrative healthcare database. Health Serv Res. 2021 Feb;56(1):154–65. pmid:33020939

14. Casey P, Cross W, Mart MW-S, Baldwin C, Riddell K, Dārziņš P. Hospital discharge data under-reports delirium occurrence: results from a point prevalence survey of delirium in a major Australian health service. Intern Med J. 2019 Mar;49(3):338–44. pmid:30091294

15. Ravi B, Pincus D, Choi S, Jenkinson R, Wasserstein DN, Redelmeier DA. Association of Duration of Surgery With Postoperative Delirium Among Patients Receiving Hip Fracture Repair. JAMA Netw Open. 2019 Feb 1;2(2):e190111. pmid:30794305

16. Redelmeier DA, Thiruchelvam D, Daneman N. Delirium after elective surgery among elderly patients taking statins. CMAJ. 2008 Sep 23;179(7):645–52. pmid:18809895

17. Katznelson R, Djaiani G, Tait G, Wasowicz M, Sutherland AM, Styra R, et al. Hospital administrative database underestimates delirium rate after cardiac surgery. Can J Anaesth. 2010 Oct;57(10):898–902. pmid:20645040

18. Bui LN, Pham VP, Shirkey BA, Swan JT. Effect of delirium motoric subtypes on administrative documentation of delirium in the surgical intensive care unit. J Clin Monit Comput. 2017 Jun;31(3):631–40. pmid:27108353

19. Kim DH, Lee J, Kim CA, Huybrechts KF, Bateman BT, Patorno E, et al. Evaluation of algorithms to identify delirium in administrative claims and drug utilization database. Pharmacoepidemiol Drug Saf. 2017 Aug;26(8):945–53. pmid:28485014

20. McCoy TH Jr, Snapper L, Stern TA, Perlis RH. Underreporting of Delirium in Statewide Claims Data: Implications for Clinical Care and Predictive Modeling. Psychosomatics. 2016 Sep;57(5):480–8. pmid:27480944

21. Romano PS, Chan BK, Schembri ME, Rainwater JA. Can administrative data be used to compare postoperative complication rates across hospitals? Med Care. 2002 Oct;40(10):856–67. pmid:12395020

22. Marco J, Méndez M, Cruz-Jentoft AJ, García Klepzig JL, Calvo E, Canora J, et al. Clinical characteristics and prognosis for delirium in Spanish internal medicine departments: An analysis from a large clinical-administrative database. Rev Clin Esp. 2019 Nov;219(8):415–23.

23. Inouye SK, Leo-Summers L, Zhang Y, Bogardus ST Jr, Leslie DL, Agostini JV. A chart-based method for identification of delirium: validation compared with interviewer ratings using the confusion assessment method. J Am Geriatr Soc. 2005 Feb;53(2):312–8. pmid:15673358

24. Verma AA, Guo Y, Kwan JL, Lapointe-Shaw L, Rawal S, Tang T, et al. Patient characteristics, resource use and outcomes associated with general internal medicine hospital care: the General Medicine Inpatient Initiative (GEMINI) retrospective cohort study. CMAJ Open. 2017 Dec 11;5(4):E842–9. pmid:29237706

25. GEMINI [Internet]. GEMINI Study. [cited 2023 Mar 14]. Available from: https://geminimedicine.ca/

26. Verma AA, Pasricha SV, Jung HY, Kushnir V, Mak DYF, Koppula R, et al. Assessing the quality of clinical and administrative data extracted from hospitals: the General Medicine Inpatient Initiative (GEMINI) experience. J Am Med Inform Assoc. 2021 Mar 1;28(3):578–87. pmid:33164061

27. Xu G, Fong TG, Yee J, Inouye SK. Delirium Identification: A training guide to a chart-based delirium instrument [Internet]. Hebrew Rehabilitation Center; 2011. Available from: http://www.hospitalelderlifeprogram.org/uploads/disclaimers/CHART-DEL_Manual_03-27-2015.pdf

28. Slooter AJC, Otte WM, Devlin JW, Arora RC, Bleck TP, Claassen J, et al. Updated nomenclature of delirium and acute encephalopathy: statement of ten Societies. Intensive Care Med. 2020 May;46(5):1020–2. pmid:32055887

29. Quan H, Li B, Couris CM, Fushimi K, Graham P, Hider P, et al. Updating and validating the Charlson comorbidity index and score for risk adjustment in hospital discharge abstracts using data from 6 countries. Am J Epidemiol. 2011 Mar 15;173(6):676–82. pmid:21330339

30. Escobar GJ, Gardner MN, Greene JD, Draper D, Kipnis P. Risk-adjusting hospital mortality using a comprehensive electronic record in an integrated health care delivery system. Med Care. 2013 May;51(5):446–53. pmid:23579354

31. Jaakkimainen RL, Bronskill SE, Tierney MC, Herrmann N, Green D, Young J, et al. Identification of Physician-Diagnosed Alzheimer’s Disease and Related Dementias in Population-Based Administrative Data: A Validation Study Using Family Physicians’ Electronic Medical Records. J Alzheimers Dis. 2016 Aug 10;54(1):337–49. pmid:27567819

32. Canadian patient cost database metadata [Internet]. Canadian Institute for Health Information. [cited 2022 Feb 18]. Available from: https://www.cihi.ca/en/canadian-patient-cost-database-metadata

33. Tritchler D. Interpreting the standardized difference. Biometrics. 1995 Mar;51(1):351–3. pmid:7766789

34. Hughes CG, Hayhurst CJ, Pandharipande PP, Shotwell MS, Feng X, Wilson JE, et al. Association of Delirium during Critical Illness With Mortality: Multicenter Prospective Cohort Study. Anesth Analg. 2021 Nov 1;133(5):1152–61. pmid:33929361

35. Mufti HN, Hirsch GM, Abidi SR, Abidi SSR. Exploiting Machine Learning Algorithms and Methods for the Prediction of Agitated Delirium After Cardiac Surgery: Models Development and Validation Study. JMIR Medical Informatics. 2019 Oct 23;7(4):e14993. pmid:31558433

36. Davoudi A, Ebadi A, Rashidi P, Ozrazgat-Baslanti T, Bihorac A, Bursian AC. Delirium Prediction using Machine Learning Models on Preoperative Electronic Health Records Data. Proc IEEE Int Symp Bioinformatics Bioeng. 2017 Oct;2017:568–73. pmid:30393788

37. Jauk S, Kramer D, Großauer B, Rienmüller S, Avian A, Berghold A, et al. Risk prediction of delirium in hospitalized patients using machine learning: An implementation and prospective evaluation study. J Am Med Inform Assoc. 2020 Jul 1;27(9):1383–92. pmid:32968811

38. Wong A, Young AT, Liang AS, Gonzales R, Douglas VC, Hadley D. Development and Validation of an Electronic Health Record-Based Machine Learning Model to Estimate Delirium Risk in Newly Hospitalized Patients Without Known Cognitive Impairment. JAMA Netw Open. 2018 Aug 3;1(4):e181018. pmid:30646095

39. Krewulak KD, Hiploylee C, Ely EW, Stelfox HT, Inouye SK, Fiest KM. Adaptation and Validation of a Chart-Based Delirium Detection Tool for the ICU (CHART-DEL-ICU). J Am Geriatr Soc [Internet]. 2020 Dec 21; Available from: pmid:33348428

40. Ryan DJ, O’Regan NA, Caoimh RÓ, Clare J, O’Connor M, Leonard M, et al. Delirium in an adult acute hospital population: predictors, prevalence and detection. BMJ Open [Internet]. 2013 Jan 7;3(1). Available from: http://dx.doi.org/10.1136/bmjopen-2012-001772

41. Loftus CA, Wiesenfeld LA. Geriatric Delirium Care: Using Chart Audits to Target Improvement Strategies. Can Geriatr J. 2017 Dec;20(4):246–52. pmid:29296131

42. Aparanji K, Kulkarni S, Metzke M, Schmudde Y, White P, Jaeger C. Quality improvement of delirium status communication and documentation for intensive care unit patients during daily multidisciplinary rounds. BMJ Open Qual. 2018 Jun 29;7(2):e000239. pmid:30019010

43. Bauernfreund Y, Butler M, Ragavan S, Sampson EL. TIME to think about delirium: improving detection and management on the acute medical unit. BMJ Open Qual. 2018 Aug 13;7(3):e000200. pmid:30167472

44. Inouye SK, van Dyck CH, Alessi CA, Balkin S, Siegal AP, Horwitz RI. Clarifying confusion: the confusion assessment method. A new method for detection of delirium. Ann Intern Med. 1990 Dec 15;113(12):941–8. pmid:2240918

45. Solberg LM, Plummer CE, May KN, Mion LC. A quality improvement program to increase nurses’ detection of delirium on an acute medical unit. Geriatr Nurs. 2013 Jan;34(1):75–9.

46. Rice KL, Bennett M, Gomez M, Theall KP, Knight M, Foreman MD. Nurses’ recognition of delirium in the hospitalized older adult. Clin Nurse Spec. 2011 Nov;25(6):299–311. pmid:22016018

47. Lemiengre J, Nelis T, Joosten E, Braes T, Foreman M, Gastmans C, et al. Detection of delirium by bedside nurses using the confusion assessment method. J Am Geriatr Soc. 2006 Apr;54(4):685–9. pmid:16686883

48. Saczynski JS, Kosar CM, Xu G, Puelle MR, Schmitt E, Jones RN, et al. A tale of two methods: chart and interview methods for identifying delirium. J Am Geriatr Soc. 2014 Mar;62(3):518–24. pmid:24512042