Heart rate n-variability (HRnV) measures for prediction of mortality in sepsis patients presenting at the emergency department

Abstract

Sepsis is a potentially life-threatening condition that requires prompt recognition and treatment. Recently, heart rate variability (HRV), a measure of the cardiac autonomic regulation derived from short electrocardiogram tracings, has been found to correlate with sepsis mortality. This paper presents using novel heart rate n-variability (HRnV) measures for sepsis mortality risk prediction and comparing against current mortality prediction scores. This study was a retrospective cohort study on patients presenting to the emergency department of a tertiary hospital in Singapore between September 2014 to April 2017. Patients were included if they were above 21 years old and were suspected of having sepsis by their attending physician. The primary outcome was 30-day in-hospital mortality. Stepwise multivariable logistic regression model was built to predict the outcome, and the results based on 10-fold cross-validation were presented using receiver operating curve analysis. The final predictive model comprised 21 variables, including four vital signs, two HRV parameters, and 15 HRnV parameters. The area under the curve of the model was 0.77 (95% confidence interval 0.70–0.84), outperforming several established clinical scores. The HRnV measures may have the potential to allow for a rapid, objective, and accurate means of patient risk stratification for sepsis severity and mortality. Our exploration of the use of wealthy inherent information obtained from novel HRnV measures could also create a new perspective for data scientists to develop innovative approaches for ECG analysis and risk monitoring.

About the Authors:

Nan Liu

Roles Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Duke-NUS Medical School, National University of Singapore, Singapore, Singapore, Health Services Research Centre, Singapore Health Services, Singapore, Singapore, Institute of Data Science, National University of Singapore, Singapore, Singapore

ORCID logo https://orcid.org/0000-0003-3610-4883

Marcel Lucas Chee

Roles Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

Affiliation: Faculty of Medicine, Nursing and Health Sciences, Monash University, Melbourne, Australia

Mabel Zhi Qi Foo

Roles Investigation, Methodology, Writing – original draft, Writing – review & editing

Affiliation: Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore

Jeremy Zhenwen Pong

Roles Data curation, Investigation, Methodology, Writing – original draft, Writing – review & editing

Affiliation: Duke-NUS Medical School, National University of Singapore, Singapore, Singapore

Dagang Guo

Roles Data curation, Investigation, Methodology, Writing – review & editing

Affiliations Duke-NUS Medical School, National University of Singapore, Singapore, Singapore, SingHealth Duke-NUS Emergency Medicine Academic Clinical Programme, Singapore, Singapore

Zhi Xiong Koh

Roles Data curation, Investigation, Methodology, Project administration, Writing – review & editing

Affiliations Duke-NUS Medical School, National University of Singapore, Singapore, Singapore, Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore

Andrew Fu Wah Ho

Roles Investigation, Methodology, Validation, Writing – review & editing

Affiliations Duke-NUS Medical School, National University of Singapore, Singapore, Singapore, Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore

ORCID logo https://orcid.org/0000-0003-4338-3876

Chenglin Niu

Roles Formal analysis, Investigation, Methodology, Validation, Writing – review & editing

Affiliation: Duke-NUS Medical School, National University of Singapore, Singapore, Singapore

Shu-Ling Chong

Roles Investigation, Methodology, Validation, Writing – review & editing

Affiliations Duke-NUS Medical School, National University of Singapore, Singapore, Singapore, Department of Children’s Emergency, KK Women’s and Children’s Hospital, Singapore, Singapore

ORCID logo https://orcid.org/0000-0003-4647-0019

Marcus Eng Hock Ong

Roles Investigation, Methodology, Validation, Writing – review & editing

Affiliations Duke-NUS Medical School, National University of Singapore, Singapore, Singapore, Department of Emergency Medicine, Singapore General Hospital, Singapore, Singapore

Introduction

Sepsis is a potentially life-threatening condition caused by the body’s dysregulated response to infection [1]. Every year, over 50 million people are affected, resulting in over five million deaths worldwide [2]. Prompt recognition and treatment of sepsis has been shown to impact patient outcomes, and guidelines have been developed for its management [3]. There is, however, a need for a rapid method to grade sepsis severity and prognosticate the risk for mortality in septic patients. A quick and accurate triage tool for risk stratification of septic patients presenting at the emergency department (ED) would be invaluable, allowing for greater confidence in clinical decisions, and in guiding management.

Several common disease severity scoring systems that have been utilised in the ED for the prediction of sepsis mortality including the Mortality in ED Sepsis (MEDS) score [4], quick SOFA (qSOFA) [5], and intensive care unit (ICU)-based scores such as the Sequential Organ Failure Assessment (SOFA) score [6], and the well-established Acute Physiology and Chronic Health Evaluation II (APACHE II) score [7]. Although these scoring systems have shown good predictive value, certain limitations have prevented their widespread adoption [8–11]. In recent years, heart rate variability (HRV) measurements derived from electrocardiogram (ECG) tracings have allowed for an alternative and complementary approach to predict sepsis mortality. HRV analysis measures the beat-to-beat variation between each R-R interval on an ECG tracing and reflects the autonomic regulation of the cardiovascular system [12]. Being a non-invasive tool that can be rapidly obtained even from patients who are unable to give a history, HRV has been shown to be dysregulated in sepsis [13] and correlates well with subsequent mortality [14,15]. Indeed, scoring systems that incorporate HRV parameters among its predictors have outperformed traditional clinical indicators and established disease severity scores in predicting sepsis mortality [16–19]. The use of HRV may thus further enhance our ability to stratify for risk of sepsis mortality.

In our previous work [20], we invented novel heart rate n-variability (HRnV) parameters to provide enhanced prognostic information to complement traditional HRV parameters. The proposed HRnV has two measures—HRnV and HRnVm. HRnV is derived from non-overlapping R-R intervals, while HRnVm is computed from overlapping R-R intervals. For each of the traditional HRV, HRnV, and HRnVm measures, time domain, frequency domain, and nonlinear analysis will yield its respective set of parameters. An application of the novel HRnV variables demonstrated improved predictive ability for major adverse cardiac events among patients with chest pain presenting at the ED [20].

This paper aims to study the prognostic ability of HRnV measures alongside traditional HRV parameters in predicting the outcomes in septic patients presenting at the ED and comparing the HRnV-based model with existing mortality prediction scores.

Methods

Study design and clinical setting

We conducted a retrospective cohort analysis on a convenience sample of patients presenting to Singapore General Hospital (SGH) between September 2014 to April 2017. SGH is the largest hospital in Singapore, with its ED seeing 300 to 500 patients daily. Patients are triaged on presentation at the ED according to a symptom-based Patient Acuity Category Scale (PACS). The PACS system has four levels: PACS 1 patients are critically ill, PACS 2 patients are non-ambulant but stable, PACS 3 patients are ambulant, and PACS 4 patients are non-emergency. This study was approved, and informed consent was waived by the SingHealth Centralized Institutional Review Board (CIRB Ref No.: 2016/2858).

Study population and eligibility

Patients were included in the study if they were aged 21 years and above, triaged to either PACS 1 or 2 at the ED, suspected to have sepsis as determined by their attending physician, and if they met two or more out of four Systemic Inflammatory Response Syndrome (SIRS) criteria [21,22]. The SIRS criteria (temperature >38°C or <36°C, heart rate >90 beats per minute, respiratory rate >20 breaths per minute, and total white blood cell count >12,000/mm3 or <4000/mm3) were used despite recent revisions under the Sepsis-3 consensus that recommend for sepsis screening with qSOFA score [1]. This decision was made primarily to allow for comparability with the existing literature. Additionally, subsequent validation studies have disputed the utility of qSOFA over SIRS for sepsis screening in the ED due to its poor sensitivity for septic patients [23–26]. Patients were excluded if their ECGs had non-sinus rhythm, a high noise level (>30% of the entire recording), or if they had a pacemaker or were on mechanical ventilator support.

Data collection

Five-minute one-lead ECGs were performed on patients who met the inclusion criteria using the ZOLL X Series monitor/defibrillator (ZOLL Medical Corporation, Chelmsford, MA). In addition, patient demographics, vital signs taken at triage, medical history, and laboratory investigations performed in the ED were retrieved from the electronic medical records. We defined the primary outcome as 30-day in-hospital mortality (IHM).

HRnV measure and analysis

We processed the ECGs and detected QRS complex to convert the original ECG signals into R-R interval (RRI) sequences (i.e., intervals of consecutive R peaks in ECGs). Fig 1 illustrates the definitions of RRI and the derived RRnI and RRnIm sequences. Conventional HRV analysis evaluates consecutive single RRIs in ECGs. Novel HRnV measures (HRnV and HRnVm) analyse consecutive combined RRIs (RRnI and RRnIm) [20].

[Figure omitted. See PDF.]

Fig 1. Illustration of the RR intervals (RRIs) and the definitions of RRnI and RRnIm, where 1 ≤ n ≤ 3, 1 ≤ m ≤ 2.

Parameter m indicates the non-overlapping portion between two successive RRnIm sequences.

https://doi.org/10.1371/journal.pone.0249868.g001

To define the HRnV measure, a new type of RRI called RRnI is obtained, where n is an integer between 1 and N and N (the number of conventional RRIs combined to form a new RRnI; for example, RR2I is a combination of 2 consecutive RRIs) is much smaller than (total number of RRIs). With newly generated RRnI sequences, traditional time and frequency domains, and nonlinear analyses [27,28] are applied to calculate HRnV parameters. In addition to conventional HRV parameters, HRnV also evaluates two newly created parameters: NN50n and pNN50n. These two parameters differ from the traditional NN50 and pNN50 parameters in that the threshold is changed from 50 ms to 50×n ms in describing the absolute difference between successive RRnI sequences.

Similarly, HRnVm is a measure derived from RRnIm, where m is the number determining non-overlapping RRIs for each RRnI. When m = n, RRnIm becomes RRnI as there are no overlapping RRIs, resulting in an upper limit of N-1 for m. Fig 1 depicts a scenario when n = 3 and m can be 1 or 2. Utilising all permissible combinations of n and m, N(N+1)/2 sets of traditional HRV, novel HRnV and HRnVm parameters can be generated from a single RRI sequence. Our analysis set the upper limit of N as three due to the relatively short duration of collected ECG samples. As a result, one set of HRV parameters, two sets of HRnV (HR2V and HR3V) parameters, and three sets of HRnVm (HR2V1, HR3V1, and HR3V2) parameters were calculated. The HRnV-Calc software suite (https://github.com/nliulab/HRnV) was used for calculating the HRV and HRnV parameters, in which the functions from PhysioNet Cardiovascular Signal Toolbox [29] were performed for ECG signal processing.

Statistical analysis

Categorical variables were compared between patients who did and did not meet the primary outcome (30-day IHM) using χ2 test or Fisher’s exact test where appropriate. Continuous variables were checked for normality with the Kolmogorov-Smirnov test. Subsequently, normally distributed variables were presented as mean and standard deviation (SD) and were compared with independent two-tailed t test between groups, while non-normally distributed variables were presented as median and interquartile range (IQR; 25th to 75th percentiles) and compared using the Mann-Whitney U test.

Univariable regression analysis was conducted on traditional HRV parameters, novel HRnV parameters and demographic and clinical variables. Each variable was evaluated as an individual predictor of the primary outcome (30-day IHM) using binary logistic regression with odds ratio (OR), 95% confidence interval (CI), and p-value reported. For multivariable regression analysis, we adjusted for age, temperature, systolic blood pressure, heart rate, and Glasgow Coma Scale (GCS) as these variables were either shown to be significant predictors of sepsis mortality in previous literature [15,30–32], or are included in well-established sepsis scoring systems such as the National Early Warning Score (NEWS) [33], Modified Early Warning Score (MEWS) [34], qSOFA, or APACHE II. HRV and HRnV parameters were included in the multivariable analysis if they achieved p<0.2 in the univariable analysis. Included variables were then checked for collinearity using Pearson’s R correlation. For each collinear pair, the variable with the higher p-value on univariate analysis was eliminated until no collinear pairs remained.

The remaining variables were then fed into a backward stepwise multivariable logistic regression model, which used p<0.1 as an endpoint. We took statistical significance at p<0.05. Backward elimination was chosen for our stepwise variable selection because it has the advantage to assess the joint predictive ability of variables, and it removes the least essential variables. However, the eliminated variables cannot re-enter the model [35]. In comparison, all possible subset selection examines every combination of variables, requiring tremendous computing resources yet likely overfitting the model when the number of variables is large [35].

In predictive modelling with the selected variables, we conducted 10-fold cross-validation to avoid overfitting in evaluating models. We split the entire dataset into 10 non-overlapping subsets of equivalent size and then used nine subsets to build a model and validated the model with the remaining one subset. We repeated the above process ten times to ensure that each of the ten subsets could be validated. Subsequently, a receiver operating characteristic (ROC) curve was plotted to assess the predictive ability of the multivariable regression model and compared against other established disease scoring systems on their area under the curve (AUC).

Missing data were addressed by median imputation, in consideration of the low proportion of missing data (<0.3%) for each variable, the nature of variables, and recommendations for missing data in clinical trials [36]. There were three missing observations for which the median value was imputed; one patient had an unknown medical history of cancer, and another patient was missing both initial and worst qSOFA scores.

All statistical analyses were carried out using Python version 3.8.0 (Python Software Foundation, Delaware, USA) using the SciPy library (version 1.3.1). Regression models were built using the StatsModels library (version 0.10.2) and scikit-learn library (version 0.22). All methods were implemented in accordance with relevant guidelines and regulations.

Results

Patient recruitment

Fig 2 presents the patient recruitment flowchart. Of the 659 patients that were initially recruited, 190 patients did not meet the SIRS criteria, and 127 patients had inapplicable ECG readings. Three hundred forty-two patients were included for analysis and classified depending on whether they met the primary outcome of 30-day IHM (n = 66, 19%) or did not meet the primary outcome (n = 276, 81%).

[Figure omitted. See PDF.]

Fig 2. Patient recruitment flowchart.

ECG: Electrocardiogram; ED: Emergency department; IHM: In-hospital mortality; SIRS: Systemic inflammatory response syndrome.

https://doi.org/10.1371/journal.pone.0249868.g002

Baseline characteristics and clinical parameters

Table 1 illustrates baseline characteristics and clinical parameters of patients who met and did not meet with 30-day IHM. Patients who met with 30-day IHM were older and presented with higher respiratory rates but lower temperatures, systolic blood pressures (SBP) and GCS scores, compared to patients who did not meet with 30-day IHM. The worst recorded values of respiratory rate, GCS, and SBP during each patient’s ED stay were also significantly more abnormal in patients that met with 30-day IHM. The difference in disposition from the ED was significant, with a larger proportion of patients who eventually met with 30-day IHM requiring admission to the ICU as compared to patients who did not meet with 30-day IHM (16.7% vs 4.3%, p = 0.001). Additionally, a larger proportion of patients who met with 30-day IHM had a respiratory source of infection (45.5% vs 27.2%, p = 0.006) while a smaller proportion had a source of infection originating from the urinary tract (7.6% vs 25.7%, p = 0.003) when compared to patients who did not meet with 30-day IHM. No significant differences were detected in gender, PACS status, ethnicity, or medical history between both groups.

[Figure omitted. See PDF.]

Table 1. Baseline characteristics and clinical parameters.

https://doi.org/10.1371/journal.pone.0249868.t001

HRV and HRnV parameter description and univariable analysis

Table 2 presents the descriptive analysis of HRV and HRnV parameters. In this study, N was set as 3 and HR2V, HR2V1, HR3V, HR3V1 and HR3V2 parameters were calculated. Among time domain parameters such as mean NN and SDNN, HRnV and HRnVm values are generally directly proportional to n and increase when n increases. HR2V SampEn and HR3V SampEn were considerably larger than SampEn parameters of HRV, HR2V1, HR3V1, and HR3V2. This was because of insufficient data points since our ECG recordings were only five minutes long. HR2V1, HR3V1 and HR3V2 did not encounter this limitation as more data points were available from a calculation using overlapping RRnIm sequences [20].

[Figure omitted. See PDF.]

Table 2. Descriptive analyses of heart rate variability (HRV) and heart rate n-variability (HRnV) parameters.

https://doi.org/10.1371/journal.pone.0249868.t002

Table 3 shows the results of univariable analysis of HRV and HRnV parameters. Of 142 HRV and HRnV parameters, 85 were significantly different between the two outcome groups. Specifically, 14 HRV, 14 HR2V, 16 HR2V1, 11 HR3V, 16 HR3V1, and 14 HR3V2 parameters were statistically significant. In at least four out of six HRnV measures, RMSSD, kurtosis, NN50, pNN50, NN50n, pNN50n, HF power, HF power norm, Poincare SD1, and Poincare SD1/SD2 were significantly higher, while LF power norm and DFA α2 were significantly lower in patients who met the primary outcome compared to those who did not. Additionally, VLF power and DFA α1 were not significant in HRV analysis but were statistically significant in several HRnV measures.

[Figure omitted. See PDF.]

Table 3. Univariable analysis of HRV and HRnV parameters.

https://doi.org/10.1371/journal.pone.0249868.t003

Overall, six baseline characteristics (age and vital signs at triage including temperature, respiratory rate, SpO2, SBP and GCS), 17 HRV parameters, and 96 HRnV parameters had p<0.2 on univariable analysis. After collinearity assessment, the remaining 87 variables were entered into a stepwise-selection regression model.

Multivariable analysis and ROC analysis

Table 4 presents the multivariable analysis of variables found to be significantly different on univariable analysis. A total of 21 out of 87 variables were selected through stepwise selection. Of the 21 variables, 16 showed p<0.05. These include vital signs such as respiratory rate (OR = 1.168; 95% CI 1.085–1.257; p<0.001), SBP (OR = 0.978; 95% CI 0.966–0.990; p = 0.001), SpO2 (OR = 0.892; 95% CI 0.838–0.950; p = <0.001), and GCS (OR = 0.845; 95% CI 0.769–0.929; p = 0.001), and HRnV measures such as HR2V1 NN50 (OR = 0.808; 95% CI 0.682–0.958; p = 0.014), HR2V pNN50 (OR = 0.290; 95% CI 0.115–0.732; p = 0.009), HR2V1 pNN50 (OR = 5.700; 95% CI 1.784–18.213; p = 0.003), HR2V ApEn (OR = 0.106; 95% CI 0.013–0.877; p = 0.037) and several HR3V1 and HR3V2 parameters which demonstrated strong predictive power in assessing the risk for 30-day IHM. The final multivariable predictive model consisted of four vital signs, two traditional HRV parameters, and 15 novel HRnV parameters. Hereafter, we refer to this model as the HRnV model.

[Figure omitted. See PDF.]

Table 4. Multivariable analysis of HRV and HRnV parameters on 30-day in-hospital mortality.

https://doi.org/10.1371/journal.pone.0249868.t004

ROC curves were plotted for assessment of the HRnV model and compared against established disease severity scoring systems to predict 30-day IHM in patients presenting to the ED with sepsis (Fig 3). The AUC of the HRnV model based on 10-fold cross-validation was 0.77 (95% CI: 0.70–0.84), outperforming the AUC of NEWS 0.71 (95% CI: 0.64–0.78), MEWS 0.60 (95% CI: 0.53–0.67), SOFA 0.71 (95% CI: 0.64–0.78), APACHE II 0.74 (95% CI: 0.68–0.80), and the patient’s worst qSOFA value 0.72 (95% CI: 0.65–0.79).

[Figure omitted. See PDF.]

Fig 3. ROC curves for HRnV-based prediction models, alongside other disease severity scoring systems.

APACHE II: Acute Physiology and Chronic Health Evaluation II; AUC: Area under the receiver operating characteristic curve; HRnV: Heart rate n-variability; MEWS: Modified Early Warning Score; NEWS: National Early Warning Score; SOFA: Sequential Organ Failure Assessment; qSOFA(w): Worst quick SOFA score.

https://doi.org/10.1371/journal.pone.0249868.g003

Discussion

In recent years, there has been a surge in research interest in HRV and its ability to prognosticate for adverse patient outcomes across various disease processes [18,19,28]. To improve the predictive power of HRV, several studies have sought to utilise advanced nonlinear techniques to derive novel HRV parameters [29,30]. Indeed, we previously employed the novel HRnV measures to assess the risk of 30-day major adverse cardiac events in patients presenting to the ED with chest pain [20].

This study evaluated the predictive value of novel HRnV measures (HRnV and HRnVm) in estimating the risk of 30-day IHM in patients presenting to the ED with sepsis. In addition to the 22 traditional HRV parameters, we derived an additional 120 HRnV parameters, 71 of which were found statistically significant in their association with the primary outcome. The newly generated HRnV parameters greatly augment the number of candidate predictors and have demonstrated improved predictive ability for sepsis mortality. Although the physiological meaning and clinical interpretation of some HRnV parameters are yet to discover, the rich inherent information obtained from novel HRnV measures could create a new perspective for data scientists and machine learning researchers to investigate innovative approaches for ECG analysis and risk monitoring.

In HRnV measures, the newly added parameters, NN50n and pNN50n, were significantly associated with mortality in the univariate analysis. They characterise the number of times that the absolute difference between two successive RRnI sequences exceeds 50×n ms, by assuming the absolute difference may be magnified when the corresponding RRnI is n times longer than RRI [20]. The composite HRnV model derived from multivariable logistic regression achieved the highest AUC on ROC analysis and outperformed other established disease scoring systems such as NEWS, MEWS, SOFA, and APACHE II for the prediction of 30-day IHM in patients presenting to the ED with sepsis.

In addition to demonstrating the superior predictive ability for sepsis mortality, the HRnV model is made even more relevant in its capacity for rapid and objective prognostication where only vital signs and parameters calculated from five-minute ECG tracings are needed. Many established disease severity scores require invasive tests, which need long turnaround time and resources to obtain or include subjective parameters that involve interrater variability while scoring. Among disease severity scores, the MEDS score, explicitly developed for risk stratification of septic patients in the ED, suffers from some of these limitations and its adoption has thus not been widespread. Consequently, MEDS was not included in our comparison. APACHE II and SOFA scores initially designed for use in the intensive care unit (ICU) setting similarly require invasive investigations to calculate its score. In these aspects, the HRnV model which is derived from vital signs taken on ED presentation, and HRV and HRnV parameters calculated from five-minute ECG tracings, can overcome these limitations and provide a rapid, objective, and accurate risk assessment of the septic patient. A triage tool with these characteristics would be invaluable to the physician and can aid in risk stratification, clinical management, patient disposition, and accurate patient classification for administrative or research purposes. Furthermore, our HRnV analysis and modelling modules can be readily integrated into a monitoring device, making the real-time prediction of sepsis severity a feasible task.

Limitations

First, a majority of the HRnV variables were removed from predictive modelling with traditional logistic regression, which hindered the release of the power of the novel HRnV representation. Moving forward, we endeavour to explore the use of black box or interpretable machine learning algorithms [37–39] for full utilisation of the HRnV parameters. Second, the difficulty in interpreting the HRnV parameters may pose a challenge in their clinical implementation and adoption. However, data scientists and machine learning practitioners may find these parameters valuable in data mining tasks. Third, we were only able to recruit a convenience sample of all suspected sepsis patients at the ED due to resource constraints and the difficulty of continuous ECG measuring in an emergency setting. Moreover, there was a selection bias in patient recruitment as we only included patients from PACS 1 or 2 triage category. To address this issue, we are planning a prospective study to include all ED sepsis patients. Last, sepsis is a seasonal illness that varies throughout the year, but we were unable to examine the impact of seasonality because Singapore’s weather is warm and humid all year round.

Conclusions

The use of novel HRV measures (HRnV and HRnVm) can provide additional power to predictive models in the risk stratification of patients who present to the ED with sepsis. When included in a model with other clinical variables, the HRnV model outperforms traditional risk stratification scoring systems as shown in our preliminary results. Prospective multi-centre cohort studies would be valuable in validating the effectiveness of the HRnV parameters. The use of HRnV may allow for a rapid, objective, and accurate means of patient risk stratification for sepsis severity and mortality.

Citation: Liu N, Chee ML, Foo MZQ, Pong JZ, Guo D, Koh ZX, et al. (2021) Heart rate n-variability (HRnV) measures for prediction of mortality in sepsis patients presenting at the emergency department. PLoS ONE 16(8): e0249868. https://doi.org/10.1371/journal.pone.0249868

References

1. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). Jama. 2016;315(8):801–10. Epub 2016/02/24. pmid:26903338.

2. Fleischmann C, Scherag A, Adhikari NK, Hartog CS, Tsaganos T, Schlattmann P, et al. Assessment of Global Incidence and Mortality of Hospital-treated Sepsis. Current Estimates and Limitations. Am J Respir Crit Care Med. 2016;193(3):259–72. Epub 2015/09/29. pmid:26414292.

3. Rhodes A, Evans LE, Alhazzani W, Levy MM, Antonelli M, Ferrer R, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017;43(3):304–77. Epub 2017/01/20. pmid:28101605.

4. Shapiro NI, Wolfe RE, Moore RB, Smith E, Burdick E, Bates DW. Mortality in Emergency Department Sepsis (MEDS) score: a prospectively derived and validated clinical prediction rule. Crit Care Med. 2003;31(3):670–5. Epub 2003/03/11. pmid:12626967.

5. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):762–74. Epub 2016/02/24. pmid:26903335.

6. Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med. 1996;22(7):707–10. Epub 1996/07/01. pmid:8844239.

7. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: A severity of disease classification system. Crit Care Med. 1985;13(10):818–29. pmid:3928249

8. Macdonald SP, Arendts G, Fatovich DM, Brown SG. Comparison of PIRO, SOFA, and MEDS scores for predicting mortality in emergency department patients with severe sepsis and septic shock. Acad Emerg Med. 2014;21(11):1257–63. Epub 2014/11/08. pmid:25377403.

9. Williams JM, Greenslade JH, Chu K, Brown AF, Lipman J. Severity Scores in Emergency Department Patients With Presumed Infection: A Prospective Validation Study. Crit Care Med. 2016;44(3):539–47. Epub 2016/02/24. pmid:26901543.

10. Hilderink MJ, Roest AA, Hermans M, Keulemans YC, Stehouwer CD, Stassen PM. Predictive accuracy and feasibility of risk stratification scores for 28-day mortality of patients with sepsis in an emergency department. European journal of emergency medicine: official journal of the European Society for Emergency Medicine. 2015;22(5):331–7. Epub 2014/08/22. pmid:25144398.

11. Pong JZ, Koh ZX, Samsudin MI, Fook-Chong S, Liu N, Ong MEH. Validation of the mortality in emergency department sepsis (MEDS) score in a Singaporean cohort. Medicine (Baltimore). 2019;98(34):e16962. pmid:31441900.

12. Malik M, Camm AJ, Bigger JT Jr, Breithardt G, Cerutti S, Cohen RJ, et al. Heart rate variability. Standards of measurement, physiological interpretation, and clinical use. European Heart Journal. 1996;17(3):354–81. pmid:8737210

13. Scheff JD, Griffel B, Corbett SA, Calvano SE, Androulakis IP. On heart rate variability and autonomic activity in homeostasis and in systemic inflammation. Mathematical biosciences. 2014;252:36–44. Epub 2014/04/01. pmid:24680646.

14. Barnaby D, Ferrick K, Kaplan DT, Shah S, Bijur P, Gallagher EJ. Heart rate variability in emergency department patients with sepsis Acad Emerg Med. 2002;9:661–70.

15. Chen WL, Chen JH, Huang CC, Kuo CD, Huang CI, Lee LS. Heart rate variability measures as predictors of in-hospital mortality in ED patients with sepsis. Am J Emerg Med. 2008;26(4):395–401. Epub 2008/04/16. pmid:18410805.

16. Pong JZ, Fook-Chong S, Koh ZX, Samsudin MI, Tagami T, Chiew CJ, et al. Combining Heart Rate Variability with Disease Severity Score Variables for Mortality Risk Stratification in Septic Patients Presenting at the Emergency Department. Int J Environ Res Public Health. 2019;16(10). pmid:31100830.

17. de Castilho FM, Ribeiro ALP, da Silva JLP, Nobre V, de Sousa MR. Heart rate variability as predictor of mortality in sepsis: A prospective cohort study. PLoS One. 2017;12(6):e0180060. Epub 2017/06/28. pmid:28654692.

18. Barnaby DP, Fernando SM, Herry CL, Scales NB, Gallagher EJ, Seely AJE. Heart Rate Variability, Clinical and Laboratory Measures to Predict Future Deterioration in Patients Presenting With Sepsis. Shock. 2019;51(4):416–22. Epub 2018/05/31. pmid:29847498.

19. Arnold R, Green G, Bravi A, Hollenberg S, Seely A. Impaired heart rate variability predicts clinical deterioration and progressive organ failure in emergency department sepsis patients. Crit Care. 2012;16(Suppl 1):P37–P. Epub 2012/03/20.

20. Liu N, Guo D, Koh ZX, Ho AFW, Xie F, Tagami T, et al. Heart rate n-variability (HRnV) and its application to risk stratification of chest pain patients in the emergency department. BMC Cardiovasc Disord. 2020;20(1):168. Epub 2020/04/12. pmid:32276602.

21. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–55. Epub 1992/06/01. pmid:1303622.

22. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250–6. Epub 2003/04/12. pmid:12682500.

23. Simpson SQ. New Sepsis Criteria. Chest. 2016;149(5):1117–8. pmid:26927525

24. Fernando SM, Tran A, Taljaard M, Cheng W, Rochwerg B, Seely AJE, et al. Prognostic Accuracy of the Quick Sequential Organ Failure Assessment for Mortality in Patients With Suspected Infection. Annals of Internal Medicine. 2018;168(4). pmid:29404582

25. Van der Woude SWvD F.F.; Hutten B.A.; Nellen F.J.; Holleman F. Classifying sepsis patients in the emergency department using SIRS, qSOFA or MEWS. Neth J Med. 2018;76:158–66. pmid:29845938

26. Usman OA, Usman AA, Ward MA. Comparison of SIRS, qSOFA, and NEWS for the early identification of sepsis in the Emergency Department. Am J Emerg Med. 2019;37(8):1490–7. Epub 2018/11/25. pmid:30470600.

27. Electrophysiology TFotES. Heart Rate Variability. Circulation. 1996;93(5):1043–65. pmid:8598068

28. Shaffer F, Ginsberg JP. An Overview of Heart Rate Variability Metrics and Norms. Front Public Health. 2017;5:258. Epub 2017/10/17. pmid:29034226.

29. Vest AN, Da Poian G, Li Q, Liu C, Nemati S, Shah AJ, et al. An open source benchmarked toolbox for cardiovascular waveform and interval analysis. Physiol Meas. 2018;39(10):105004. Epub 2018/09/11. pmid:30199376.

30. Lopez-Delgado JC, Brink A, Alsma J, Verdonschot RJCG, Rood PPM, Zietse R, et al. Predicting mortality in patients with suspected sepsis at the Emergency Department; A retrospective cohort study comparing qSOFA, SIRS and National Early Warning Score. Plos One. 2019;14(1). pmid:30682104

31. Samsudin MuI, Liu N, Prabhakar SM, Chong S-L, Kit Lye W, Koh ZX, et al. A novel heart rate variability based risk prediction model for septic patients presenting to the emergency department. Medicine. 2018;97(23). pmid:29879021

32. Sanderson M, Chikhani M, Blyth E, Wood S, Moppett IK, McKeever T, et al. Predicting 30-day mortality in patients with sepsis: An exploratory analysis of process of care and patient characteristics. Journal of the Intensive Care Society. 2018;19(4):299–304. pmid:30515239

33. Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84(4):465–70. Epub 2013/01/09. pmid:23295778.

34. Subbe CP. Validation of a modified Early Warning Score in medical admissions. Qjm. 2001;94(10):521–6. pmid:11588210

35. Chowdhury MZI, Turin TC. Variable selection strategies and its importance in clinical prediction modelling. Fam Med Community Health. 2020;8(1):e000262–e. pmid:32148735.

36. Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. The Prevention and Treatment of Missing Data in Clinical Trials. New England Journal of Medicine. 2012;367(14):1355–60. pmid:23034025

37. Chiew CJ, Liu N, Tagami T, Wong TH, Koh ZX, Ong MEH. Heart rate variability based machine learning models for risk prediction of suspected sepsis patients in the emergency department. Medicine. 2019;98(6):e14197. Epub 2019/02/09. pmid:30732136.

38. Xie F, Chakraborty B, Ong MEH, Goldstein BA, Liu N. AutoScore: A Machine Learning-Based Automatic Clinical Score Generator and Its Application to Mortality Prediction Using Electronic Health Records. JMIR medical informatics. 2020:21798. pmid:33084589

39. Ross EG, Shah NH, Dalman RL, Nead KT, Cooke JP, Leeper NJ. The use of machine learning for the identification of peripheral artery disease and future mortality risk. J Vasc Surg. 2016;64(5):1515–22.e3. Epub 2016/10/26. pmid:27266594.