A comparison of qSOFA, SIRS and NEWS in predicting the accuracy of mortality in patients with suspected sepsis: A meta-analysis

Abstract

Objective

To identify and compare prognostic accuracy of quick Sequential Organ Failure Assessment (qSOFA) score, Systemic Inflammatory Response Syndrome (SIRS) criteria, and National Early Warning Score (NEWS) to predict mortality in patients with suspected sepsis.

Methods

This meta-analysis followed accordance with the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. We searched PubMed, EMBASE, Web of Science, and the Cochrane Library databases from establishment of the database to November 29, 2021. The pooled sensitivity and specificity with 95% CIs were calculated using a bivariate random-effects model (BRM). Hierarchical summary receiver operating characteristic (HSROC) curves were generated to assess the overall prognostic accuracy.

Results

Data of 62338 patients from 26 studies were included in this meta-analysis. qSOFA had the highest specificity and the lowest sensitivity with a specificity of 0.82 (95% CI: 0.76–0.86) and a sensitivity of 0.46 (95% CI: 0.39–0.53). SIRS had the highest sensitivity and the lowest specificity with a sensitivity of 0.82 (95% CI: 0.78–0.85) and a specificity 0.24 (95% CI: 0.19–0.29). NEWS had both an intermediate sensitivity and specificity with a sensitivity of 0.73 (95% CI: 0.63–0.81) and a specificity 0.52 (95% CI: 0.39–0.65). qSOFA showed higher overall prognostic accuracy than SIRS and NEWS by comparing HSROC curves.

Conclusions

Among qSOFA, SIRS and NEWS, qSOFA showed higher overall prognostic accuracy than SIRS and NEWS. However, no scoring system has both high sensitivity and specificity for predicting the accuracy of mortality in patients with suspected sepsis.

About the Authors:

Can Wang

Roles Conceptualization, Data curation, Formal analysis, Methodology, Writing – original draft, Writing – review & editing

Affiliations Department of Pharmacy, The Second Affiliated Hospital of Army Medical University, Chongqing, China, Department of Pharmacy, University Town Hospital Affiliated of Chongqing Medical University, Chongqing, China

ORCID logo https://orcid.org/0000-0002-0996-9569

Rufu Xu

Roles Formal analysis, Methodology

Affiliation: Department of Pharmacy, The Second Affiliated Hospital of Army Medical University, Chongqing, China

Yuerong Zeng

Roles Data curation, Formal analysis

Affiliations Department of Pharmacy, The Second Affiliated Hospital of Army Medical University, Chongqing, China, Department of Pharmacy, University Town Hospital Affiliated of Chongqing Medical University, Chongqing, China

Yu Zhao

Roles Conceptualization

* E-mail: [email protected] (XH); [email protected] (YZ)

Affiliation: Department of Pharmacy, University Town Hospital Affiliated of Chongqing Medical University, Chongqing, China

Xuelian Hu

Roles Conceptualization, Funding acquisition, Writing – original draft

* E-mail: [email protected] (XH); [email protected] (YZ)

Affiliation: Department of Pharmacy, The Second Affiliated Hospital of Army Medical University, Chongqing, China

ORCID logo https://orcid.org/0000-0002-7040-8760

Introduction

In the United States, the sepsis incidence of 5.9% among hospitalized patients, a trend that has been increasing annually [1]. Although the mortality of sepsis has decreased in recent years, it is still the main cause of mortality worldwide [2]. The key strategies for a successful outcome in patients with sepsis are early recognition and timely therapy. However, accurate identification of sepsis is still a problem for clinicians. A reliable method to evaluate sepsis can help clinicians correctly identify sepsis, improve the initial treatment plan of patients, and ultimately improve the survival rate.

Sepsis-1 [3] in 1991 and Sepsis-2 [4] in 2001 suggest that sepsis should be defined as infection with Systemic Inflammatory Response Syndrome (SIRS). The definition of Sepsis-3 [5] was released in 2016 and recommended “Sequential Organ Failure Assessment (SOFA)” or “quick Sequential Organ Failure Assessment (qSOFA)”. The qSOFA score is a simplified score based on the SOFA score, which is said to be more accurate than SOFA in departments outside the intensive care unit (ICU) [6]. In recent years, some studies have tried to evaluate the prognostic accuracy of qSOFA and SIRS. In general, SIRS has high sensitivity but low specificity, and qSOFA has high specificity but low sensitivity in the prognosis of sepsis [7]. In addition, the National Early Warning Score (NEWS) is widely used in the UK as a tool to assess and monitor the clinical status of hospital patients and has the same or higher prognostic accuracy [8]. However, it is unclear which scoring system has higher prognostic accuracy in patients with suspected sepsis.

We included all studies that compared qSOFA, SIRS, and NEWS in suspected sepsis patients and performed a meta-analysis of the available studies to determine the accuracy of these scoring systems in predicting mortality in suspected sepsis patients.

Method

This meta-analysis was conducted in accordance with the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. The protocol for this review was prospectively registered in INPLASY (number INPLASY202140029)

Search strategy and selection criteria

We searched PubMed, EMBASE, Web of Science, and the Cochrane Library databases from establishment of the database to November 29, 2021. The search strategy was as follows: (“quick Sequential Organ Failure Assessment” OR “qSOFA”) OR (“Systemic Inflammatory Response Syndrome” OR “SIRS”) OR (“National Early Warning Score” OR “NEWS”) AND (“sepsis”) AND (“mortality”) AND (“emergency department” OR “ED” OR “outside ICU” OR “outside Intensive Care Unit”). Two investigators (Can Wang and Yuerong Zeng) independently screened and included the eligible studies according to the inclusion and exclusion criteria. In case of any disagreement, the study group should discuss and resolve it. The inclusion criteria were the following: (1) the study population was adult patients with suspected or sepsis outside ICU, (2) the purpose was to evaluate or compare the accuracy of qSOFA, SIRS, and NEWS in predicting mortality, (3) a 2 × 2 contingency table (true positives [TP], false positives [FP], false negatives [FN], and true negatives [TN]) can be obtained directly or indirectly through the information in the literature. The exclusion criteria were as follows: review articles, letters, and conference abstracts.

Data extraction

Two investigators (Can Wang and Yuerong Zeng) independently extracted data from the selected articles. The extracted data were as follows: study characteristics (author, year of publication, country of origin, type of study), patient characteristics (selection criteria of patients, number of patients enrolled, age, sex, setting in which patient was seen), and outcomes (type of measured mortality, cut-off value of qSOFA, SIRS, and NEWS, TP, FP, FN, TN, sensitivity, specificity)

Study quality assessment

Two investigators (Can Wang and Yuerong Zeng) independently evaluated the quality of included studies according to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) [9]. In case of any disagreement, the study group should discuss and resolve it. The evaluation content mainly consists of four parts: patient selection, index test, reference standard, flow and timing. According to the "yes", "no", or "uncertain" answers to the relevant landmark questions included in each part, the risk of bias can be judged as "low", "high", or "uncertain".

Statistical analysis

Statistical analysis was conducted using STATA15 and RevMan5.3. The pooled sensitivity and specificity with 95% CIs were calculated using a bivariate random-effects model (BRM) [10]. Hierarchical summary receiver operating characteristic (HSROC) curves were generated to assess the overall prognostic accuracy. Lambda, Theta, and Beta are the estimated parameters of HSROC. Lambda is the natural logarithm of the diagnostic odds ratio (DOR), Theta is the mean of the log of sensitivity and the log of 1-specificity, Beta is the parameter that defines the shape of the summary curve [11]. The Beta value significant difference from zero indicates that the curve is asymmetric, which is not suitable for calculating the pooled value of accuracy [11]. A beta value equal to or close to zero indicates that the curve is symmetrical, and lambda can be used to evaluate the overall prediction accuracy [11]. The post-test probability was assessed by Fagan’s nomogram.

The heterogeneity was evaluated by the I2 test. If I2 ≤ 50%, P ≥ 0.1, the heterogeneity among the studies was acceptable; If I2 > 50%, P < 0.1, the heterogeneity among the studies is significant, and the source of heterogeneity should be analyzed. The threshold effect was not judged because the Stata Midas module was used for statistical analysis. Univariate meta-regression and subgroup analysis were used to explore the source of heterogeneity. Deek’ s funnel plot was made to assess the publication bias.

Results

Search results and description of studies

5809 articles were found in the initial search, repetitive articles were deleted and 135 articles were reviewed after careful evaluation of abstracts. 109 studies were excluded, of which 12 were letters, 3 were reviews, 73 were conference abstracts, 17 were unable to obtain relevant data, and 4 were unable to accessible. We attempted to contact the manuscript authors to retrieve the 17 articles for which relevant data was not available and 4 articles for which unable to accessible, but there was still no response. 26 studies [8, 12–36] were included according to the inclusion and exclusion criteria (Fig 1). The basic characteristics of the 26 included studies are shown in Table 1. Eight studies [8, 12, 15, 17, 20, 24, 30, 35] were prospective studies and the others [13, 14, 16, 18, 19, 21–23, 25–29, 31–34, 36] were retrospective studies. Five studies [12, 15, 23, 24, 33] were multi-center studies, and the others [8, 13, 14, 16–22, 25–32, 34–36] were described as single-center. The average age varied between 57 and 81, and the proportion of men varied between 46% and 65% in these studies. There were 24 studies [12–17, 18–22, 24–36] including qSOFA, 12 studies [12–15, 17, 19, 20, 25, 26, 30, 32, 36] including SIRS and 11 studies [8, 12–14, 16, 19, 23, 25, 26, 30, 32] including NEWS. The cut-off value of NEWS was 2 in one study [19], 5 in three studies [13, 23, 30], 6 in two studies [12, 25], 7 in two studies [14, 26], and 8 in three studies [8, 16, 32].

[Figure omitted. See PDF.]

Fig 1. Flow diagram of the study selection process.

https://doi.org/10.1371/journal.pone.0266755.g001

[Figure omitted. See PDF.]

Table 1. Characteristics of included studies.

https://doi.org/10.1371/journal.pone.0266755.t001

Quality assessment of studies

The individual and overall quality assessment results of the 26 included studies were shown in S1 Fig. Overall, the included studies showed that there were risks in two of the four areas. Except for the study of Brink, et al [14], all other studies included consecutive or random cases, avoiding inappropriate exclusion. Because the cut-off values of the 11 studies [8, 16, 17, 25, 26, 28, 32–36] were not determined in advance, there were high risks in the index test. Dick’s funnel plot suggested that there was no publication bias (Fig 2).

[Figure omitted. See PDF.]

Fig 2. Deek’ s funnel plot for publication bias.

A: qSOFA; B: SIRS; C: NEWS.

https://doi.org/10.1371/journal.pone.0266755.g002

Data synthesis and meta-analysis

Predictive validity of qSOFA.

Data of 60661 patients in 24 studies were associated with the accuracy of qSOFA in predicting mortality. The pooled sensitivity and specificity were 0.46 (95% CI: 0.39–0.53) and 0.82 (95% CI: 0.76–0.86), respectively (S2 Fig). The I2 of sensitivity and specificity were 95.10% and 99.25% indicating significant heterogeneity among the studies. The heterogeneity sources were analyzed by meta-regression and subgroup analyses, and the results were shown in S5A Fig. The types of studies (P<0.05), the types of patients (P<0.05), and the types of mortality measured (P<0.01) may be the source of heterogeneity. The HSROC curve and the estimated parameters of qSOFA were shown in Fig 3A and Table 2. Beta was 0.12 (Z = -0.74, P = 0.459), which indicates that the curve is symmetric. Lambda was 1.23 (95% CI: 0.91–1.55), which represents corresponding the DOR of 3.79. Fagan’ s nomogram (Fig 4A) showed that qSOFA increased the possibility of mortality in patients with suspected sepsis to 39% when the pretest probability of mortality was 20%.

[Figure omitted. See PDF.]

Fig 3. Hierarchical Summary Receiver Operating Characteristic (HSROC) curve for predicting mortality in patients with suspected sepsis.

A: qSOFA; B: SIRS; C: NEWS.

https://doi.org/10.1371/journal.pone.0266755.g003

[Figure omitted. See PDF.]

Fig 4. Fagan’ s nomogram for pre-test and post-test probability of mortality in patients with suspected sepsis.

A: qSOFA; B: SIRS; C: NEWS.

https://doi.org/10.1371/journal.pone.0266755.g004

[Figure omitted. See PDF.]

Table 2. The estimated parameters of HSROC.

https://doi.org/10.1371/journal.pone.0266755.t002

Predictive validity of SIRS.

The pooled sensitivity and specificity of SIRS for predicting mortality in suspected sepsis patients were 0.82 (95% CI: 0.78–0.85) and 0.24 (95% CI: 0.19–0.29), respectively (S3 Fig). The HSROC curve and the estimated parameters of SIRS were shown in Fig 3B and Table 2. Significant heterogeneity between studies can be observed in the combined results of sensitivity and specificity (I2 = 77.08% and I2 = 98.32%). Meta-regression analysis and subgroup analysis showed that the heterogeneity of sensitivity was caused by the types of studies, patients’ settings, the types of patients, and the types of mortality measured, and the heterogeneity of specificity was caused by the types of study and types of patients (S5B Fig). Beta was 0.32 (Z = 1.28, P = 0.200) and Lambda was 0.79 (95% CI: 0.09–1.50). Fagan’s nomogram (Fig 4B) showed that SIRS increased the possibility of mortality in patients with suspected sepsis to 21% when the pretest probability of mortality was 20%.

Predictive validity of NEWS.

The pooled sensitivity and specificity of NEWS for predicting mortality in suspected sepsis patients were 0.73 (95%CI, 0.63–0.81) and 0.52 (95%CI, 0.39–0.65), respectively (S4 Fig). Heterogeneity among studies was considered substantial in the analyses of sensitivity and specificity, with I2 values of 92.50% for sensitivity and 99.47% for specificity. Meta-regression analysis and subgroup analysis showed that the heterogeneity of sensitivity among the studies came from the types of studies (S5C Fig). Beta was 0.20 (Z = 1.11, P = 0.267), which indicates that the curve is symmetric. Lambda was 1.18 (95% CI: 0.87–1.50), which represents corresponding the DOR of 2.96 (Fig 3C and Table 2). Fagan’s nomogram (Fig 4C) showed that NEWS increased the possibility of mortality in patients with suspected sepsis to 28% when the pretest probability of mortality was 20%.

Performance comparison of qSOFA, SIRS and NEWS.

The performance of the qSOFA, SIRS and NEWS in predicting mortality in patients with suspected sepsis were presented in Fig 5 and Table 3. In direct comparisons, qSOFA showed higher overall prediction accuracy than SIRS and NEWS with the diagnostic odds ratio (DOR). In addition, the AUC in qSOFA and NEWS were higher than in SIRS. qSOFA provided the highest specificity for predicting mortality followed by NEWS and SIRS. On the contrary, the SIRS provided the highest sensitivity for predicting mortality followed by NEWS and qSOFA. qSOFA showed better post-test probability than SIRS and NEWS, representing patients with suspected sepsis met qSOFA the greater chance of die.

[Figure omitted. See PDF.]

Fig 5. Direct comparison among qSOFA, SIRS, and NEWS in predicting the accuracy of mortality in patients with suspected sepsis.

https://doi.org/10.1371/journal.pone.0266755.g005

[Figure omitted. See PDF.]

Table 3. Pooled performance characteristics of qSOFA, SIRS, and NEWS for predicting mortality in patients with suspected sepsis.

https://doi.org/10.1371/journal.pone.0266755.t003

Discussion

This is the first meta-analysis comparing the prognostic accuracy of qSOFA, SIRS, and NEWS to predict mortality in patients with suspected sepsis. Our meta-analysis identified 26 clinical studies, including 62338 patients with suspected sepsis. However, no scoring system has both high sensitivity and specificity for predicting the accuracy of mortality in patients with suspected sepsis. These scoring systems have their advantages and disadvantages.

qSOFA is a bedside assessment tool recommended by the Third International Sepsis Consensus Definitions Task Force used to assess patients with suspected sepsis outside the ICU [5]. Our findings suggest that qSOFA shows the highest overall prediction accuracy of mortality and has high specificity. Therefore, qSOFA is of great value in predicting the mortality of patients with suspected sepsis. It can more accurately identify patients with a high risk of death than the other two scoring systems. However, our results show that qSOFA has low sensitivity, which means that false negative is high and it is easy to miss or delay treatment. The reasons for the low sensitivity of qSOFA may be as follows. On the one hand, the supporting paper [6] derived and tested among critically ill patients and there will be deviation and sensitivity will be decreased for the general patient population. On the other hand, qSOFA ≥ 2 is too strictly and patients will be in a late disease state with a worse prognosis using it [37]. Due to the lethality of sepsis, a screening mechanism showing high sensitivity is needed [38]. Some studies have proposed that reducing the qSOFA cut-off to 1 or combined with lactate levels can improve the sensitivity [39].

It is universally acknowledged that any scoring system used to determine sepsis should tend to be higher sensitivity rather than specificity because the cost of delaying or missing treatment caused by false negatives is much greater than the cost of unnecessary antibiotics caused by false positives [13]. We found that SIRS had the highest sensitivity among the three scoring systems. However, SIRS is too poor to predict mortality in patients with suspected sepsis. Whether direct comparison or indirect comparison, SIRS has the lowest prediction ability. SIRS is more suitable as a screening tool for early care and the prevention of missed diagnoses.

NEWS is widely recommended to identify patients at risk of deterioration, which was launched by the Royal College of Physicians (RCP) in 2012 [40]. NEWS has the strongest ability to identify patients at risk of deterioration compared with other Early Warning Score (EWS) [41]. Meanwhile, more and more studies have proved that NEWS is a promising scoring system and can be used as an alternative screening tool for patients with suspected sepsis. In this meta-analysis of patients with suspected sepsis, we found that NEWS is slightly worse than qSOFA in terms of overall prediction ability, but avoids the extremely low sensitivity of qSOFA and has both an intermediate sensitivity and specificity. The strength of the NEWS is that it can be calculated based on physiological parameters alone, which is easier to implement than the other two scoring systems [42].

Our research also has some limitations. On the one hand, there is significant heterogeneity in our meta-analysis. The included studies were different types of studies (prospective or retrospective), different outcome indicators (in-hospital mortality or 30 / 28 / 60-day mortality), and different types of patients (sepsis or suspected sepsis). The above points are the sources of heterogeneity. On the other hand, the cut-off values of the included studies are different, and the cut-off values of some studies are not determined in advance. A predefined cut-off value help to reduce the sensitivity and specificity bias that may result from this data-driven method [43].

Conclusion

In conclusion, our results indicate that qSOFA showed higher overall prediction accuracy of mortality than SIRS and NEWS. The three scoring systems have limitations as a tool for predicting mortality in patients with suspected sepsis. A scoring system with both high sensitivity and specificity needs to be studied in the future.

Supporting information

S1 Fig. Summary of methodological quality in the included studies.

https://doi.org/10.1371/journal.pone.0266755.s001

(JPG)

S2 Fig. Forest plot for sensitivity and specificity of qSOFA for predicting mortality in suspected sepsis patients.

https://doi.org/10.1371/journal.pone.0266755.s002

(JPG)

S3 Fig. Forest plot for sensitivity and specificity of SIRS for predicting mortality in suspected sepsis patients.

https://doi.org/10.1371/journal.pone.0266755.s003

(JPG)

S4 Fig. Forest plot for sensitivity and specificity of NEWS for predicting mortality in suspected sepsis patients.

https://doi.org/10.1371/journal.pone.0266755.s004

(JPG)

S5 Fig. Univariate meta-regression and subgroup analysis for sensitivity and specificity.

Factors with asterisk are potential sources of heterogeneity. A: qSOFA; B: SIRS; C: NEWS.

https://doi.org/10.1371/journal.pone.0266755.s005

(JPG)

S1 File. PRISMA 2009 checklist.

https://doi.org/10.1371/journal.pone.0266755.s006

(DOC)

Citation: Wang C, Xu R, Zeng Y, Zhao Y, Hu X (2022) A comparison of qSOFA, SIRS and NEWS in predicting the accuracy of mortality in patients with suspected sepsis: A meta-analysis. PLoS ONE 17(4): e0266755. https://doi.org/10.1371/journal.pone.0266755

References

1. Rhee C, Dantes R, Epstein L, Murphy DJ, Seymour CW, Iwashyna TJ, et al. Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009–2014. JAMA. 2017;318(13):1241–1249. pmid:28903154.

2. Salomão R, Ferreira BL, Salomão MC, Santos SS, Azevedo LCP, Brunialti MKC. Sepsis: evolving concepts and challenges. Braz J Med Biol Res. 2019;52(4): e8595. pmid:30994733.

3. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–1655. pmid:1303622.

4. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, Cook D, et al. 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med. 2003;31(4):1250–1256. pmid:12682500.

5. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):801–810. pmid:26903338.

6. Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, et al. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):762–774. pmid:26903335.

7. Williams JM, Greenslade JH, McKenzie JV, Chu K, Brown AFT, Lipman J. Systemic Inflammatory Response Syndrome, Quick Sequential Organ Function Assessment, and Organ Dysfunction: Insights from a Prospective Database of ED Patients with Infection. Chest. 2017;151(3):586–596. pmid:27876592.

8. Almutary A, Althunayyan S, Alenazi K, Alqahtani A, Alotaibi B, Ahmed M, et al. National Early Warning Score (NEWS) as Prognostic Triage Tool for Septic Patients. Infect Drug Resist. 2020; 13:3843–3851. pmid:33149629.

9. Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–536. pmid:22007046.

10. Negeri ZF, Beyene J. Robust bivariate random-effects model for accommodating outlying and influential studies in meta-analysis of diagnostic test accuracy studies. Stat Methods Med Res. 2020;29(11):3308–3325. pmid:32469266.

11. Rivero-Santana A, Ferreira D, Perestelo-Pérez L, Westman E, Wahlund L, Sarría A, et al. Cerebrospinal Fluid Biomarkers for the Differential Diagnosis between Alzheimer’s Disease and Frontotemporal Lobar Degeneration: Systematic Review, HSROC Analysis, and Confounding Factors. J Alzheimers Dis. 2019;55(2):625–644. pmid:27716663.

12. Szakmany T, Pugh R, Kopczynska M, Lundin RM, Sharif B, Morgan P, et al. Defining sepsis on the wards: results of a multi-centre point-prevalence study comparing two sepsis definitions. Anaesthesia. 2018;73(2):195–204. pmid:29150856.

13. Goulden R, Hoyle MC, Monis J, Railton D, Riley V, Martin P, et al. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J. 2018;35(6):345–349. pmid:29467173.

14. Brink A, Alsma J, Verdonschot RJCG, Rood PPM, Zietse R, Lingsma HF, et al. Predicting mortality in patients with suspected sepsis at the Emergency Department; A retrospective cohort study comparing qSOFA, SIRS and National Early Warning Score. PLoS One. 2019;14(1): e0211133. pmid:30682104.

15. Kim KS, Suh GJ, Kim K, Kwon WY, Shin J, Jo YH, et al. Quick Sepsis-related Organ Failure Assessment score is not sensitive enough to predict 28-day mortality in emergency department patients with sepsis: a retrospective review. Clin Exp Emerg Med. 2019;6(1):77–83. pmid:30944293.

16. Pong JZ, Fook-Chong S, Koh ZX, Samsudin MI, Tagami T, Chiew CJ, et al. Combining Heart Rate Variability with Disease Severity Score Variables for Mortality Risk Stratification in Septic Patients Presenting at the Emergency Department. Int J Environ Res Public Health. 2019;16(10):1725. pmid:31100830.

17. Sivayoham N, Blake LA, Tharimoopantavida SE, Chughtai S, Hussain AN, Cecconi M, et al. The REDS score: a new scoring system to risk-stratify emergency department suspected sepsis: a derivation and validation study. BMJ Open. 2019;9(8): e030922. pmid:31455715.

18. Abdullah SMOB, Grand J, Sijapati A, Puri PR, Nielsen FE. qSOFA is a useful prognostic factor for 30-day mortality in infected patients fulfilling the SIRS criteria for sepsis. Am J Emerg Med. 2020;38(3):512–516. pmid:31171438.

19. Boonmee P, Ruangsomboon O, Limsuwat C, Chakorn T. Predictors of Mortality in Elderly and Very Elderly Emergency Patients with Sepsis: A Retrospective Study. West J Emerg Med. 2020;21(6):210–218. pmid:33207168.

20. Caramello V, Beux V, De Salve AV, Macciotta A, Ricceri F, Boccuzzi A. Comparison of different prognostic scores for risk stratification in septic patients arriving to the emergency department. Italian Journal of Medicine. 2020;14:2(79–87).

21. Guarino M, Gambuti E, Alfano F, de Giorgi A, Maietti E, Strada A, et al. Predicting in-hospital mortality for sepsis: a comparison between qSOFA and modified qSOFA in a 2-year single-centre retrospective analysis. Eur J Clin Microbiol Infect Dis. 2021;40(4):825–831. pmid:33118057.

22. Guirgis FW, Puskarich MA, Smotherman C, Sterling SA, Gautam S, Moore FA, et al. Development of a Simple Sequential Organ Failure Assessment Score for Risk Assessment of Emergency Department Patients With Sepsis. J Intensive Care Med. 2020;35(3):270–278. pmid:29141524.

23. Hargreaves DS, de Carvalho JLJ, Smith L, Picton G, Venn R, Hodgson LE. Persistently elevated early warning scores and lactate identifies patients at high risk of mortality in suspected sepsis. Eur J Emerg Med. 2020;27(2):125–131. pmid:31464702.

24. Mearelli F, Barbati G, Casarsa C, Giansante C, Breglia A, Spica A, et al. The Integration of qSOFA with Clinical Variables and Serum Biomarkers Improves the Prognostic Value of qSOFA Alone in Patients with Suspected or Confirmed Sepsis at ED Admission. J Clin Med. 2020;9(4):1205. pmid:32331426.

25. Phungoen P, Khemtong S, Apiratwarakul K, Ienghong K, Kotruchin P. Emergency Severity Index as a predictor of in-hospital mortality in suspected sepsis patients in the emergency department. Am J Emerg Med. 2020;38(9):1854–1859. pmid:32739856.

26. Wattanasit P, Khwannimit B. Comparison the accuracy of early warning scores with qSOFA and SIRS for predicting sepsis in the emergency department. Am J Emerg Med. 2021; 46:284–288. pmid:33046318.

27. Xia Y, Zou L, Li D, Qin Q, Hu H, Zhou Y, et al. The ability of an improved qSOFA score to predict acute sepsis severity and prognosis among adult patients. Medicine (Baltimore). 2020;99(5): e18942. pmid:32000414.

28. Zhou H, Lan T, Guo S. Prognostic Prediction Value of qSOFA, SOFA, and Admission Lactate in Septic Patients with Community-Acquired Pneumonia in Emergency Department. Emerg Med Int. 2020;2020:7979353. pmid:32322422.

29. Devia Jaramillo G, Ibáñez Pinilla M. Quick Sequential Organ Failure Assessment, Sequential Organ Failure Assessment, and Procalcitonin for Early Diagnosis and Prediction of Death in Elderly Patients with Suspicion of Sepsis in the Emergency Department, Based on Sepsis-3 Definition. Gerontology. 2021:1–10. pmid:34315158.

30. Oduncu AF, Kıyan GS, Yalçınlı S. Comparison of qSOFA, SIRS, and NEWS scoring systems for diagnosis, mortality, and morbidity of sepsis in emergency department. Am J Emerg Med. 2021;48:54–59. pmid:33839632.

31. Prasad PA, Fang MC, Martinez SP, Liu KD, Kangelaris KN. Identifying the Sickest During Triage: Using Point-of-Care Severity Scores to Predict Prognosis in Emergency Department Patients with Suspected Sepsis. J Hosp Med. 2021;16(8):453–461. pmid:34328843.

32. Ruangsomboon O, Boonmee P, Limsuwat C, Chakorn T, Monsomboon A. The utility of the rapid emergency medicine score (REMS) compared with SIRS, qSOFA and NEWS for Predicting in-hospital Mortality among Patients with suspicion of Sepsis in an emergency department. BMC Emerg Med. 2021;21(1):2. pmid:33413139.

33. Shi QF, Xu Y, Zhang BY, Qu W, Wang SY, Zheng WL, et al. External validation and comparison of two versions of simplified sequential organ failure assessment scores to predict prognosis of septic patients. Int J Clin Pract. 2021;75(12): e14865. pmid:34523203.

34. Sivayoham N, Hussain AN, Shabbo L, Christie D. An observational cohort study of the performance of the REDS score compared to the SIRS criteria, NEWS2, CURB65, SOFA, MEDS and PIRO scores to risk-stratify emergency department suspected sepsis. Ann Med. 2021;53(1):1863–1874. pmid:34686088.

35. Suttapanit K, Wisan M, Sanguanwit P, Prachanukool T. Prognostic Accuracy of VqSOFA for Predicting 28-day Mortality in Patients with Suspected Sepsis in the Emergency Department. Shock. 2021;56(3):368–373. pmid:33577246.

36. Kilinc Toker A, Kose S, Turken M. Comparison of SOFA Score, SIRS, qSOFA, and qSOFA + L Criteria in the Diagnosis and Prognosis of Sepsis. Eurasian J Med. 2021;53(1):40–47. pmid:33716529.

37. Sprung CL, Schein RMH, Balk RA. The new sepsis consensus definitions: the good, the bad and the ugly. Intensive Care Med. 2016;42(12):2024–2026. pmid:27815588.

38. Simpson SQ. New Sepsis Criteria: A Change We Should Not Make. Chest. 2016;149(5):1117–1118. pmid:26927525.

39. Baumann BM, Greenwood JC, Lewis K, Nuckton TJ, Darger B, Shofer FS, et al. Combining qSOFA criteria with initial lactate levels: Improved screening of septic patients for critical illness. Am J Emerg Med. 2020;38(5):883–889. pmid:31320214.

40. Scott LJ, Redmond NM, Garrett J, Whiting P, Northstone K, Pullyblank A. Distributions of the National Early Warning Score (NEWS) across a healthcare system following a large-scale roll-out. Emerg Med J. 2019;36(5):287–292. pmid:30842204.

41. Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84(4):465–470. pmid:23295778.

42. Graham CA, Leung LY, Lo RSL, Yeung CY, Chan SY, Hung KKC. NEWS and qSIRS superior to qSOFA in the prediction of 30-day mortality in emergency department patients in Hong Kong. Ann Med. 2020;52(7):403–412. pmid:32530356.

43. Leeflang MMG, Moons KGM, Reitsma JB, Zwinderman AH. Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clin Chem. 2008;54(4):729–737. pmid:18258670.