A comparison of host response strategies to distinguish bacterial and viral infection

Abstract

Objectives

Compare three host response strategies to distinguish bacterial and viral etiologies of acute respiratory illness (ARI).

Methods

In this observational cohort study, procalcitonin, a 3-protein panel (CRP, IP-10, TRAIL), and a host gene expression mRNA panel were measured in 286 subjects with ARI from four emergency departments. Multinomial logistic regression and leave-one-out cross validation were used to evaluate the protein and mRNA tests.

Results

The mRNA panel performed better than alternative strategies to identify bacterial infection: AUC 0.93 vs. 0.83 for the protein panel and 0.84 for procalcitonin (P<0.02 for each comparison). This corresponded to a sensitivity and specificity of 92% and 83% for the mRNA panel, 81% and 73% for the protein panel, and 68% and 87% for procalcitonin, respectively. A model utilizing all three strategies was the same as mRNA alone. For the diagnosis of viral infection, the AUC was 0.93 for mRNA and 0.84 for the protein panel (p<0.05). This corresponded to a sensitivity and specificity of 89% and 82% for the mRNA panel, and 85% and 62% for the protein panel, respectively.

Conclusions

A gene expression signature was the most accurate host response strategy for classifying subjects with bacterial, viral, or non-infectious ARI.

About the Authors:

Melissa Ross

Roles Data curation, Formal analysis, Investigation, Writing – original draft

Affiliation: Duke University School of Medicine, Durham, NC, United States of America

Ricardo Henao

Roles Formal analysis, Supervision, Writing – review & editing

Affiliations Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America, Department of Electrical and Computer Engineering, Duke University, Durham, NC, United States of America

ORCID logo https://orcid.org/0000-0003-4980-845X

Thomas W. Burke

Roles Data curation, Investigation, Methodology, Supervision, Writing – review & editing

Affiliation: Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America

ORCID logo https://orcid.org/0000-0003-0592-5822

Emily R. Ko

Roles Conceptualization, Resources, Supervision, Writing – review & editing

Affiliations Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America, Duke Regional Hospital, Durham, NC, United States of America

Micah T. McClain

Roles Conceptualization, Funding acquisition, Investigation, Resources, Writing – review & editing

Affiliations Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America, Medical Service, Durham Veterans Affairs Health Care System, Durham, NC, United States of America

Geoffrey S. Ginsburg

Roles Conceptualization, Funding acquisition, Supervision, Writing – review & editing

Affiliation: Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America

Christopher W. Woods

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Resources, Writing – review & editing

Affiliations Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America, Medical Service, Durham Veterans Affairs Health Care System, Durham, NC, United States of America

Ephraim L. Tsalik

Roles Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – original draft

* E-mail: [email protected]

Affiliations Duke Center for Applied Genomics & Precision Medicine, Duke University School of Medicine, Durham, NC, United States of America, Emergency Medicine Service, Durham Veterans Affairs Health Care System, Durham, NC, United States of America

ORCID logo https://orcid.org/0000-0002-6417-2042

Introduction

Emerging antibiotic resistance is one of the most pressing medical challenges of our time; in 2019 the Centers for Disease Control and Prevention estimated 2.8 million antibiotic resistant infections causing more than 35,000 deaths annually in the United States [1]. One contributing factor is a high rate of inappropriate antimicrobial use. Even among patients with upper respiratory tract infections, which are typically viral, antibacterials are prescribed in up to 42% of cases [2]. This is itself driven by difficulties identifying whether an illness is infectious and if so, whether it is bacterial or viral. To augment pathogen detection approaches, which have limitations, there has been growing interest in host response strategies. The best-studied clinically available biomarker to discriminate bacterial and non-bacterial etiologies of acute respiratory illness (ARI) is procalcitonin [3–6]. Previous studies have shown that despite being a relatively poor biomarker for distinguishing bacterial versus viral infection [7,8], procalcitonin-guided algorithms can reduce antibacterial consumption, decrease antibacterial-related side effects, and lower mortality [9,10].

Combining multiple host biomarkers in a multivariate index assay may offer better opportunities to discriminate bacterial and viral infections. Multi-protein biomarker panels have been developed [11,12]. Among them is a 3-peptide panel of IP-10 (interferon-inducible protein 10), TRAIL (TNF-related apoptosis-inducing ligand), and CRP (C-reactive protein), which has been shown to accurately distinguish bacterial ARI from viral ARI [12–14]. However, these studies did not include sufficient numbers of patients with acute respiratory illness of non-infectious etiology to gauge performance characteristics in a clinically relevant population. Moreover, subjects with equivocal test results were excluded from analysis, which inflates performance characteristics.

Host gene expression offers another diagnostic biomarker strategy for bacterial vs. viral infection [15–19]. Furthermore, patients with non-infectious illness can also be distinguished using host gene expression [18]. We have previously adapted this signature onto the BioFire FilmArray System, which is a rapid, sample-to-answer platform measuring host gene expression [20,21]. Validated in a large cohort of patients with ARI, this gene expression test accurately discriminated patients with bacterial infection, viral infection, coinfection, or no infection [22,23].

These biomarker strategies have been variably evaluated, precluding head-to-head comparisons. In this secondary analysis of the RADICAL study (Rapid Diagnostics in Categorizing Acute Lung Infection), we applied a uniform analytical strategy to directly compare the performance of multiple host response biomarkers including procalcitonin, a three-protein panel (IP-10, TRAIL, and CRP), and a 45-transcript mRNA panel. These biomarkers were evaluated in a cohort of 286 subjects with ARI of bacterial, viral, or non-infectious etiologies.

Materials and methods

Study cohort

Studies were approved by the Duke University Institutional Review Board (IRB), Durham VA Health Care System IRB, Henry Ford IRB, and University of North Carolina Medical Center IRB in accordance with institutional and federal regulations regarding human subjects’ protection. Written informed consent was obtained from all subjects or legally authorized representatives.

This study is reported as per the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline (S1 Checklist). This is a secondary analysis of the RADICAL study, focusing on subjects with microbiologically confirmed bacterial ARI, viral ARI, or non-infectious illness and who had samples available to support all biomarker testing [23]. The study protocol and analysis plan for this study were not pre-specified at the time the RADICAL study was designed and implemented. Subjects with a low-confidence reference standard were excluded, consisting of those without confirmatory microbiology or bacterial/viral co-infection. Enrollment occurred in the emergency departments of Duke University Medical Center, Durham VA Health Care System, Henry Ford Hospital, and University of North Carolina Medical Center from 2005 through 2016. They were enrolled as part of CAPSOD (Community-Acquired Pneumonia and Sepsis Outcome Diagnostics; ClinicalTrials.gov NCT00258869) or CAPSS (Community-Acquired Pneumonia and Sepsis Study). Patients ≥6-years were eligible if they had a known or suspected infection and at least two Systemic Inflammatory Response Syndrome (SIRS) criteria [24]. Additional subjects were enrolled at Duke and the Durham VA as part of RADICAL (Rapid Diagnostics in Categorizing Acute Lung Infections). RADICAL enrolled subjects ≥2-years with <28-days duration of suspected bacterial, viral, or non-infectious ARI.

Reference standard and case definitions

Since there is no gold standard to define whether ARI is bacterial, viral, or non-infectious, panel adjudication served as the reference standard [18,23,25,26]. Adjudications were performed by specialists in emergency medicine, infectious diseases, pulmonary medicine, or hospital medicine based on chart reviews performed >28 days after enrollment, and before measuring procalcitonin, the protein panel, or the mRNA panel. Due to the retrospective nature of the adjudications, the diagnostic accuracy of this reference standard is expectedly higher than what might be achieved in real-time by clinical providers. Adjudicators had access to medical records including but not limited to admission notes, consultant notes, discharge summaries, patient-reported symptoms, and laboratory and radiographic results (when clinically indicated). In addition, we performed supplemental microbiological testing including a multiplex viral respiratory pathogen panel (ResPlex V2.0, Qiagen; Respiratory Viral Panel, Luminex; or Respiratory Pathogen Panel, Luminex). Each case was independently adjudicated by two reviewers. If there was discordance with respect to the primary etiology, the case was reviewed by a panel of at least three reviewers. To be categorized as having a viral or bacterial ARI, a subject must have had a compatible clinical syndrome and an identified pathogen. The determination of “noninfectious illness” required an alternative non-infectious diagnosis and negative microbiological testing. Sepsis was defined based on the Sepsis-2 criteria, which requires the presence of systemic inflammatory response syndrome (SIRS) due to infection [24].

Procalcitonin measurement

Procalcitonin was measured for study purposes and was not available to adjudicators, limiting any potential incorporation bias. Serum measurements were made on a Roche Elecsys 2010 analyzer (Roche Diagnostics) by electrochemiluminescence immunoassay or the miniVIDAS immunoassay (bioMérieux). Measurements in plasma-EDTA were made by the Phadia Immunology Reference Laboratory by immunofluorescence using the B·R·A·H·M·S PCT sensitive KRYPTOR (Thermo Fisher Scientific). Since all three platforms are approved for this indication, results were treated equivalently regardless of test platform. Subjects were classified as bacterial (≥0.25 ng/mL) or non-bacterial (<0.25 ng/mL) (Fig 1).

[Figure omitted. See PDF.]

Fig 1. Study schematic.

The analysis cohort included 286 subjects, each of which had three bacterial vs. viral biomarkers measured. The mRNA panel results were based on a previously published study using the BioFire system to measure 45 host response transcripts. This test reports independent results for bacterial vs. non-bacterial (viral or non-infectious illness) and viral vs. non-viral (bacterial or non-infectious illness), each with its own threshold. The protein panel test measures IP-10, TRAIL, and CRP. Two sets of thresholds were applied. The first, based on published values, assigns values below 0.35 as Viral, values above 0.65 as Bacterial, and values from 0.35–0.65 as equivocal. A second scheme was utilized whereby a single optimized threshold of 0.30 was used to distinguish bacterial from viral infections. The third biomarker, procalcitonin, used the established threshold of 0.25 ng/ml to distinguish bacterial from non-bacterial etiologies.

https://doi.org/10.1371/journal.pone.0261385.g001

Protein panel measurement

Concentrations of plasma IP-10, TRAIL, and CRP were measured by sandwich immunoassay with electrochemiluminescent detection using the UPLEX Human IP-10 Assay, the UPLEX Human TRAIL Assay, and the VPLEX Human CRP Assay with imaging using the QuickPlex SQ 120 Imager (Meso Scale Discovery, Rockville, MD). Plasma samples stored at -80°C were thawed on ice and processed according to the manufacturer’s recommended protocol, similar to published methods [12]. Calibration curves were generated using 4-fold serial dilutions of the calibrator. Generating probabilities of bacterial vs. viral infection was performed as previously described: protein concentrations were used to build a multinomial logistic regression model, establishing a bacterial likelihood score from zero to one [12].

Subjects were classified as bacterial or viral using two different thresholds (Fig 1). The first used published thresholds: subjects with a score ≤0.35 were classified as viral or non-infectious, subjects with a score ≥0.65 were classified as bacterial, and scores from 0.35–0.65 were considered equivocal [12]. The second method identified a single ideal threshold, above which subjects were classified as bacterial and below which subjects were classified as viral. We used the Youden Index to identify a threshold that maximized sensitivity and specificity. Values ≥0.30 were considered bacterial and values <0.30 were considered non-bacterial.

mRNA panel measurement

The composition and development of the mRNA panel measured in this study was previously described [21]. In brief, the gene signature test is composed of two unique host response signatures: a bacterial host response signature that assigns a probability of having a bacterial vs. non-bacterial etiology and a viral host response signature that assigns a probability of having a viral vs. non-viral etiology (Fig 1). The two probabilities are independent and do not sum to 1, which allows for the identification of co-infection (i.e., both positive) or no infection (i.e., both negative). In order to generate these probabilities, the BioFire System measured transcript abundance, which was then used to build a logistic regression model trained on subjects with known phenotype.

Gene expression was measured via research-use-only FilmArray pouches as previously described [21]. At initial clinical presentation, subjects were enrolled and whole blood was collected directly into PAXgene Blood RNA tubes (PreAnalytix) and stored at -80°C until use. For testing, 100μL was used to measure the relative abundance of 45 target host mRNAs as defined by the PCR cycle at which the amplified target was detected using FilmArray chemistry and processing. Normalized target expression values were used to build two independent sparse logistic regression models: one to discriminate viral from non-viral etiologies and the other to discriminate bacterial from non-bacterial etiologies. The regularization parameter of the model and performance metrics were estimated using nested Leave-One-Out Cross-Validation (LOOCV), where the internal LOOCV was used for the regularization parameter and the outer LOOCV for performance estimates. This approach was selected to match the model training and validation process used for the protein panel. Thresholds for the bacterial (0.263) and viral (0.361) tests were chosen to optimize the average weighted accuracy [27].

Statistical analysis

Each of the three tests was evaluated for its ability to distinguish bacterial from non-bacterial subjects. The protein panel and gene signature were also evaluated for their ability to distinguish viral from non-viral subjects. Procalcitonin was not evaluated for viral vs. non-viral discrimination since it does not discriminate these two possibilities. Further, a combined model was created using a multinomial logistic regression model, treating each of the aforementioned tests as an individual input. Test performance was evaluated by comparing the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Biomarker strategies were compared using the Chi-squared test except for AUC, which was compared using the Delong test. Data is available in S1 Table.

Results

Demographics

Samples from 286 subjects were analyzed. Subjects had ARI due to clinically adjudicated bacterial (n = 47), viral (n = 162), or non-infectious (n = 77) etiologies. Demographics were similar across different disease states. Subjects with bacterial infection had a mean age of 52 years, were 58% male, 43% African American, and 2% Hispanic. Subjects with viral infection had a mean age of 43 years, were 40% male, 58% African American, and 4% Hispanic. Subjects with non-infectious illness had a mean age of 54 years, were 52% male, and 41% African American (Table 1). The most common bacterial pathogens were Streptococcus pyogenes, Streptococcus pneumoniae, and Staphylococcus aureus (Table 2). The most common viral pathogens were influenza A, rhinovirus/enterovirus, respiratory syncytial virus (RSV), and human metapneumovirus (Table 2). The most common non-infectious etiologies included asthma and congestive heart failure (Table 2).

[Figure omitted. See PDF.]

Table 1. Subject demographics.

https://doi.org/10.1371/journal.pone.0261385.t001

[Figure omitted. See PDF.]

Table 2. Adjudicated phenotypes and etiologies.

https://doi.org/10.1371/journal.pone.0261385.t002

Procalcitonin

Procalcitonin was used as a binary marker of bacterial infection, with a concentration ≥0.25 ng/ml denoting a bacterial infection and values <0.25 ng/ml indicating a non-bacterial process. In this cohort, procalcitonin differentiated bacterial from non-bacterial etiologies with 68.1% sensitivity (95% confidence interval [CI], 52.9%-80.9%), 86.6% specificity (95% CI, 81.6%-90.7%), and AUC of 0.837 (95% CI, 0.773 to 0.901) (Fig 2). These and additional test characteristics are shown in Table 3. Procalcitonin does not differentiate viral from non-infectious etiologies. As such, we did not apply it to the question of viral vs. non-viral (which includes bacterial and non-infectious etiologies).

[Figure omitted. See PDF.]

Fig 2. Bacterial and viral diagnosis receiver operating characteristic curves using different host biomarker strategies.

Bacterial vs. non-bacterial classification is shown on the left. The combined strategy includes procalcitonin, protein panel, and mRNA panel. Viral vs. non-viral classification is shown on the right. For viral classification, the combined model includes the mRNA and protein panels.

https://doi.org/10.1371/journal.pone.0261385.g002

[Figure omitted. See PDF.]

Table 3. Test performance for bacterial vs. non-bacterial etiologies and viral vs. non-viral etiologies.

https://doi.org/10.1371/journal.pone.0261385.t003

Protein panel test

Bacterial vs non-bacterial classification.

Consistent with their expected distributions, subjects with bacterial infections had higher concentrations of CRP and lower concentrations of TRAIL and IP-10 (Fig 3). Using previously published reporting recommendations, we classified results as bacterial (score ≥0.65), equivocal (score 0.35–0.65), or viral (score ≤0.35). Among 47 subjects with bacterial infection, 20 (43%) were classified as bacterial, 15 (32%) were equivocal, and 12 (25%) were viral. Of the 162 viral cases, 8 (5%) were classified as bacterial, 14 (9%) were equivocal, and 140 (86%) were correctly identified as viral. Finally, of the 77 subjects with non-infectious illness, 12 (16%) were classified as bacterial, 24 (31%) were equivocal, and 41 (53%) fell into the viral/non-infectious category. Overall, 19% of cases (n = 53) fell in the equivocal range and thus could not be classified. The overall accuracy accounting for all tested subjects using these previously published thresholds was 70.3% (95% CI, 64.6%-75.5%).

[Figure omitted. See PDF.]

Fig 3. Log concentrations of CRP, TRAIL, and IP-10 stratified by clinical category.

Data are presented as a box and whisker plot showing medians with interquartile ranges.

https://doi.org/10.1371/journal.pone.0261385.g003

When equivocal results apply to such a large number of patients, generalizability becomes limited. We therefore used the previously established threshold of ≥0.65 to indicate the presence of a bacterial infection and any values <0.65 to indicate a non-bacterial etiology (viral infection or non-infectious illness). This resulted in a test sensitivity of 42.6% (95% CI 28.3%-57.8%) and specificity of 91.6% (95% CI 87.4%-94.8%). Since these protein panel thresholds were established in other clinical cohorts, they may not be well-calibrated to this study population. We therefore selected a new threshold that optimized test performance in this cohort. Using a threshold of 0.30, we observed a more balanced sensitivity [80.9% (95% CI, 66.7%-90.9%)] and specificity [73.2% (95% CI, 67.1%-78.7%)], which corresponded to an AUC of 0.832 (95% CI, 0.771–0.893) (Table 3, Fig 2).

Viral vs non-viral.

Using the previously published threshold of ≤0.35, the protein panel had 86.4% sensitivity (95% CI, 80.2%-91.3%) but only 57.3% specificity (95% CI, 48.1%-66.1%) for the classification of viral infection. Most misclassifications were non-infectious subjects being classified as viral, which may have smaller clinical consequences as both groups would be spared antibacterial use.

Using the threshold optimized for this cohort described above (<0.30 for viral infection and ≥0.30 as non-viral), the protein panel had 84.6% sensitivity (95% CI, 78.1%-93.3%), 62.1% specificity (95% CI, 53.0%-70.7%), and AUC of 0.843 (95% CI, 0.799–0.888) for detecting viral infections (Table 3, Fig 2).

mRNA panel test

Bacterial vs non-bacterial.

Any subject with a bacterial probability ≥0.26 using the mRNA panel test was classified as having a bacterial infection. Using this threshold, the mRNA panel test had 91.5% sensitivity (95% CI, 79.6%-97.6%), 82.9% specificity (95% CI, 77.5%-87.4%), and AUC of 0.932 (95% CI, 0.902–0.962) (Table 3, Fig 2). Specificity was significantly better than the protein panel (p <0.01), and sensitivity was significantly better than both procalcitonin and the protein panel (p<0.001) (Table 3). A direct comparison of the predicted probabilities of bacterial infection using the mRNA and protein panel tests is shown in Fig 4. A model combining the protein panel, mRNA panel, and procalcitonin did not improve performance beyond that of the mRNA panel alone (Table 3, Fig 2).

[Figure omitted. See PDF.]

Fig 4.

Distribution of predicted bacterial (left) and viral (right) infection probabilities using the protein and mRNA panels. The probabilities predicted by the protein panel test (y-axis) are plotted relative to the probabilities predicted by the mRNA panel test (x-axis). The dashed horizontal line corresponds to the optimal protein panel threshold. The vertical line corresponds to the optimal mRNA panel threshold. Bacterial infection cases are represented by red dots. Viral infection cases are represented by blue circles. Non-infectious illness cases are represented by green circles.

https://doi.org/10.1371/journal.pone.0261385.g004

Viral vs non-viral.

Any subject with a viral probability of ≥0.361 was classified as having a viral infection. Using this threshold, the gene signature distinguished viral vs. non-viral subjects with a sensitivity of 88.9% (95% CI, 83.0%-93.3%), specificity of 82.3% (95% CI, 74.4%-88.5%), and AUC of 0.928 (95% CI, 0.899–0.956) (Table 3, Fig 2). Specificity was significantly better than the protein panel (p <0.0001) (Table 3). A direct comparison of the predicted probabilities of viral infection using the mRNA and protein panel tests is shown in (Fig 4). Comparison to procalcitonin was not performed since it does not discriminate viral from non-infectious etiologies. A model combining the protein panel and mRNA panel did not improve performance beyond that of the mRNA panel alone (Table 3, Fig 2).

Discordant classifications

We hypothesized there would be scenarios in which all three host response tests agreed with each other but disagreed with clinical adjudication. This occurred in nine cases. The only pattern we could discern among them were three cases adjudicated as having Group A Streptococcal pharyngitis, but all three host response tests identified a viral infection. In the entire cohort, there were 11 cases of Group A Streptococcal pharyngitis corresponding to a 27% (3/11) error rate.

We also stratified the cohort by the presence or absence of SIRS criteria, which in the case of bacterial and viral infections, would denote sepsis. SIRS was present in 155 subjects and absent in 131. We observed no differences in the sensitivity for any biomarker strategy, though this was limited by having only six bacterial infections without sepsis. In contrast, the specificity to diagnose a bacterial infection was lower for all three biomarker strategies among subjects with SIRS compared to those without: 78.1% vs. 94.4% for procalcitonin (p<0.001), 62.3% vs. 83.2% for the protein panel (p<0.001), and 71.9% vs. 92.8% for the mRNA panel (p<0.001).

There were no other identifiable patterns with respect to illness or pathogens among the other discordant predictions. Similarly, there was no clear pattern among cases where the different biomarker panels were discordant with each other.

Discussion

ARI is among the most common acute care presentations worldwide. This syndrome is commonly due to viral, bacterial, or non-infectious etiologies. The greatest challenge in clinical management is identifying when a bacterial infection is present and subsequently, which patients need antibacterial therapy. Pathogen-detection strategies are an important component of this evaluation but have limited utility. A complementary strategy measures the host’s molecular response to the infection, which can discriminate between etiologies. This study evaluated three different host-based diagnostic approaches in a common cohort of patients with bacterial, viral, and non-infectious ARI. We showed that all three strategies discriminate between bacterial and viral etiologies, although an mRNA panel test performed best, in part because of its ability to discriminate patients with non-infectious etiologies.

Currently, procalcitonin is the most widely used biomarker to aid in the differentiation of bacterial and non-bacterial etiologies of ARI. Procalcitonin has been cleared by the FDA to guide the management of antibiotic therapy in patients with lower respiratory tract infection based on previously published trial data [28]. However, a recent study failed to show an impact by procalcitonin on antibacterial use, the reasons for which are likely multifactorial [29]. Other recent publications have demonstrated a poor ability to discriminate bacterial and viral etiologies [29,30]. Further, high false positive rates for procalcitonin have been observed for patients with a variety of inflammatory and chronic disease conditions [31–38]. The limitations of procalcitonin may be inherent to the use of a single biomarker approach. Consequently, integrating multiple biomarkers from different biological pathways into a single test could improve ARI diagnostics.

One example is the use of CRP, IP-10, and TRAIL, which were identified in a proteomic study based on their ability to differentiate bacterial and viral infections [12]. This panel has been validated in several studies, demonstrating an ability to distinguish bacterial from viral infection with a sensitivity of 86%-94% and specificity of 90%-94% [13,14,39,40]. Equivocal test results were obtained in 11–17% of these cohorts compared to 19% in this study. Such a high rate of non-informative results limits the utility of such a test. Furthermore, these prior publications excluded subjects with equivocal results when calculating sensitivity and specificity. Removing subjects from the denominator (all subjects tested who had a valid test result) will artificially inflate performance estimates. When performance characteristics are calculated with all tested subjects accounted for, results are expectedly lower. This was observed in our study as well as another independent validation performed by van der Does et al [41]. In that study, sensitivity was 78% and specificity was 73% for the diagnosis of bacterial infection, which is similar to our observations (81% and 73%, respectively).

Transcriptomic approaches present another opportunity for disease classification. Many transcriptomic-derived classifiers for distinguishing between patients with bacterial or viral infections have been described [15–19,42–50]. Despite these discoveries, the means to measure these signatures in a clinically meaningful time was not previously available. To address that, we translated the gene expression signature using BioFire technology, taking advantage of the sample-to-answer format [23]. Using this research-use-only test, the mRNA panel test evaluated here was better than the protein panel and procalcitonin for classifying subjects with ARI. One potential reason is that a 45-gene signature captures more biology due to the increased number of analytes measured. Perhaps similar improvements could be achieved with a larger panel of proteins although adding procalcitonin to the protein panel did not improve results. Furthermore, protein biomarkers for measurement in serum or plasma will be restricted to those that are secreted, representing a biologically limited view of the underlying host response. Furthermore, the mRNA panel was discovered in a cohort of patients that had a higher proportion of non-infectious etiologies as compared to the alternative biomarker strategies. As such, it is expected to perform better in an undifferentiated and more heterogeneous patient population.

This study identified Group A streptococcal pharyngitis as one diagnostically challenging area. Specifically, all three biomarker strategies were discordant with clinical adjudication in 27% of subjects with streptococcal pharyngitis. Rates of pharyngeal S. pyogenes colonization are as high as 40% depending on how the carrier state is defined [51]. This suggests that the presence of Group A Streptococcus in and of itself is a poor indication of the presence of bacterial infection. Host response biomarkers may therefore offer additional and perhaps superior diagnostic information for such infections.

More broadly, all three biomarker strategies offered high NPVs for bacterial infection, ranging from 93% for procalcitonin to 98% for the mRNA test. These values highlight the potential utility for such biomarkers in mitigating inappropriate antibacterial use. The PPV for viral infection was lower: 75% for the protein panel and 87% for the mRNA panel. We note, however, that tests to positively identify viral infection are not currently available. As such, these tests, particularly the mRNA strategy, address an important diagnostic gap.

One study limitation includes the use of LOOCV rather than using independent training and validation cohorts. This was necessary since the protein panel demonstrated poor calibration using previously published model parameters. Although this may overestimate performance as compared to a truly independent validation, it enabled a head-to-head comparison between biomarker strategies. Another limitation is that we measured the protein panel using standardized research assays and not the proprietary ImmunoXpert™ platform. That could affect results, although we optimized the panel’s performance in this study cohort, which should mitigate technical differences. Most subjects in this study were adults. If distinguishing bacterial and viral etiologies in children is easier, then the differences observed between the three host response approaches may diminish. Children and adults may have distinct biological responses to infection. However, studies have shown that procalcitonin, the protein panel, and the mRNA panel evaluated here perform similarly in children and adults [12,18,21,40,52].Three of the four enrollment sites were in the same geographic area. To ensure generalizability of these results, further investigation must be done in a more heterogeneous population. We note that the indication for use for procalcitonin focuses on lower respiratory tract infection and therefore, its application to cases of upper respiratory tract ARI including pharyngitis may be more limited. Nevertheless, inclusion of a heterogeneous cohort strengthens the study’s generalizability. Lastly, all tests were performed on banked samples although the biomarkers evaluated here are not known to be affected by a single freeze-thaw cycle.

This study demonstrated a clear improvement in discriminating ARI etiologies using a gene expression approach, although it is unknown whether this will translate into improved patient management. Clinical trials are currently underway to investigate all three biomarker strategies and how best to apply the test results. Host response-based diagnostics offer new opportunities to characterize patients in ways not previously possible. Gene expression signatures, coupled with technical advances that bring these tests to clinical practice, potentially offer the best approach for host response-based infectious disease diagnosis.

Supporting information

S1 Checklist. STROBE statement checklist for cohort studies.

https://doi.org/10.1371/journal.pone.0261385.s001

(DOC)

S1 Table. Adjudicated phenotype and biomarker measurements for each subject.

https://doi.org/10.1371/journal.pone.0261385.s002

(XLSX)

Citation: Ross M, Henao R, Burke TW, Ko ER, McClain MT, Ginsburg GS, et al. (2021) A comparison of host response strategies to distinguish bacterial and viral infection. PLoS ONE 16(12): e0261385. https://doi.org/10.1371/journal.pone.0261385

References

1. CDC. Antibiotic Resistent Threats in the United States 2019 2019. Available from: https://www.cdc.gov/drugresistance/pdf/threats-report/2019-ar-threats-report-508.pdf.

2. Palin V, Molter A, Belmonte M, Ashcroft DM, White A, Welfare W, et al. Antibiotic prescribing for common infections in UK general practice: variability and drivers. The Journal of antimicrobial chemotherapy. 2019;74(8):2440–50. Epub 2019/05/01. pmid:31038162; PubMed Central PMCID: PMC6640319.

3. Assicot M, Bohuon C, Gendrel D, Raymond J, Carsin H, Guilbaud J. High serum procalcitonin concentrations in patients with sepsis and infection. The Lancet. 1993;341(8844):515–8. pmid:8094770

4. Muller B, White JC, Nylen ES, Snider RH, Becker KL, Habener JF. Ubiquitous expression of the calcitonin-i gene in multiple tissues in response to sepsis. J Clin Endocrinol Metab. 2001;86(1):396–404. pmid:11232031.

5. Moyer MW. New biomarkers sought for improving sepsis management and care. Nature Publishing Group; 2012.

6. Wacker C, Prkno A, Brunkhorst FM, Schlattmann P. Procalcitonin as a diagnostic marker for sepsis: a systematic review and meta-analysis. The Lancet Infectious diseases. 2013;13(5):426–35. Epub 2013/02/05. pmid:23375419.

7. Bergin SP, Tsalik EL. Procalcitonin: The right answer but to which question? Clinical Infectious Diseases. 2017;65(2):191–3. pmid:28407096

8. Self WH, Wunderink RG, Jain S, Edwards KM, Grijalva CG. Procalcitonin as a Marker of Etiology in Adults Hospitalized With Community-Acquired Pneumonia. Clinical infectious diseases: an official publication of the Infectious Diseases Society of America. 2018;66(10):1640–1. Epub 2018/01/24. pmid:29360954.

9. Schuetz P, Wirz Y, Sager R, Christ‐Crain M, Stolz D, Tamm M, et al. Procalcitonin to initiate or discontinue antibiotics in acute respiratory tract infections. Cochrane Database of Systematic Reviews. 2017;(10). pmid:29025194

10. Huang H-B, Peng J-M, Weng L, Wang C-Y, Jiang W, Du B. Procalcitonin-guided antibiotic therapy in intensive care unit patients: a systematic review and meta-analysis. Annals of intensive care. 2017;7(1):114. pmid:29168046

11. Self W, Rosen J, Sharp S, Filbin M, Hou P, Parekh A, et al. Diagnostic accuracy of FebriDx: a rapid test to detect immune responses to viral and bacterial upper respiratory infections. Journal of clinical medicine. 2017;6(10):94. pmid:28991170

12. Oved K, Cohen A, Boico O, Navon R, Friedman T, Etshtein L, et al. A novel host-proteome signature for distinguishing between acute bacterial and viral infections. PloS one. 2015;10(3):e0120012. Epub 2015/03/19. pmid:25785720; PubMed Central PMCID: PMC4364938.

13. van Houten CB, de Groot JAH, Klein A, Srugo I, Chistyakov I, de Waal W, et al. A host-protein based assay to differentiate between bacterial and viral infections in preschool children (OPPORTUNITY): a double-blind, multicentre, validation study. The Lancet Infectious diseases. 2017;17(4):431–40. Epub 2016/12/26. pmid:28012942.

14. Eden E, Srugo I, Gottlieb T, Navon R, Boico O, Cohen A, et al. Diagnostic accuracy of a TRAIL, IP-10 and CRP combination for discriminating bacterial and viral etiologies at the Emergency Department. The Journal of infection. 2016;73(2):177–80. Epub 2016/06/04. pmid:27255416.

15. Bhattacharya S, Rosenberg AF, Peterson DR, Grzesik K, Baran AM, Ashton JM, et al. Transcriptomic Biomarkers to Discriminate Bacterial from Nonbacterial Infection in Adults Hospitalized with Respiratory Illness. Sci Rep. 2017;7(1):6548. Epub 2017/07/28. pmid:28747714; PubMed Central PMCID: PMC5529430.

16. Suarez NM, Bunsow E, Falsey AR, Walsh EE, Mejias A, Ramilo O. Superiority of transcriptional profiling over procalcitonin for distinguishing bacterial from viral lower respiratory tract infections in hospitalized adults. J Infect Dis. 2015;212(2):213–22. Epub 2015/02/01. pmid:25637350; PubMed Central PMCID: PMC4565998.

17. Ramilo O, Allman W, Chung W, Mejias A, Ardura M, Glaser C, et al. Gene expression patterns in blood leukocytes discriminate patients with acute infections. Blood. 2007;109(5):2066–77. Epub 2006/11/16. pmid:17105821; PubMed Central PMCID: PMC1801073.

18. Tsalik EL, Henao R, Nichols M, Burke T, Ko ER, McClain MT, et al. Host gene expression classifiers diagnose acute respiratory illness etiology. Science translational medicine. 2016;8(322):322ra11. Epub 2016/01/23. pmid:26791949; PubMed Central PMCID: PMC4905578.

19. Hu X, Yu J, Crosby SD, Storch GA. Gene expression profiles in febrile children with defined viral and bacterial infection. Proceedings of the National Academy of Sciences of the United States of America. 2013;110(31):12792–7. Epub 2013/07/17. pmid:23858444; PubMed Central PMCID: PMC3732941.

20. Tsalik EL HR, Aydin M, Bullard C, Montgomery JL, Nawrocki J, Deneris M, et al. FilmArray® Measurement of Host Response Signatures Rapidly Discriminates Viral, Bacterial, and Non-Infectious Etiologies of Illness. Abstract 2012, 2018 IDWeek, San Francisco, CA.

21. Tsalik EL, Henao R, Montgomery JL, Nawrocki JW, Aydin M, Lydon EC, et al. Discriminating Bacterial and Viral Infection Using a Rapid Host Gene Expression Test. Crit Care Med. 2021;49(10):1651–63. Epub 2021/05/04. pmid:33938716; PubMed Central PMCID: PMC8448917.

22. Lydon EC, Henao R, Burke TW, Aydin M, Nicholson BP, Glickman SW, et al. Validation of a host response test to distinguish bacterial and viral respiratory infection. EBioMedicine. 2019;48:453–61. Epub 2019/10/22. pmid:31631046; PubMed Central PMCID: PMC6838360.

23. Tsalik EL, Henao R, Montgomery JL, Nawrocki JW, Aydin M, Lydon EC, et al. Discriminating Bacterial and Viral Infection Using a Rapid Host Gene Expression Test. Critical Care Medicine. 2021;In Press. pmid:33938716

24. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–55. Epub 1992/06/01. pmid:1303622

25. Lydon EC, Bullard C, Aydin M, Better OM, Mazur A, Nicholson BP, et al. A host gene expression approach for identifying triggers of asthma exacerbations. PloS one. 2019;14(4):e0214871. Epub 2019/04/09. pmid:30958855; PubMed Central PMCID: PMC6453459.

26. Lydon E, Bullard C, Aydin M, Better O, Mazur A, McClain MT, et al. 2014. TLDA Validation of a Host Response Signature to Discriminate Bacterial, Viral, and Non-infectious Causes of Illness. Open Forum Infectious Diseases. 2018;5(Suppl 1):S587. PubMed Central PMCID: PMC6253485.

27. Liu Y TE, Jiang Y, Ko ER, Woods CW, Henao R, Evans SR. Average Weighted Accuracy (AWA): Pragmatic Analysis for a RADICAL Study. Clinical Infectious Diseases. 2019:In Press. Epub In Press.

28. Hey J T-LP, Kirson NY, Zimmer L, Wilkins D, Rice B, Iankova I, et al. Procalcitonin guidance in patients with lower respiratory tract infections: a systematic review and meta-analysis. Clin Chem Lab Med. 2018;56(8):1200–9. pmid:29715176

29. Huang DT, Yealy DM, Filbin MR, Brown AM, Chang CH, Doi Y, et al. Procalcitonin-Guided Use of Antibiotics for Lower Respiratory Tract Infection. The New England journal of medicine. 2018;379(3):236–49. Epub 2018/05/22. pmid:29781385; PubMed Central PMCID: PMC6197800.

30. Bergin SP, Tsalik EL. Procalcitonin: The Right Answer but to Which Question? Clinical infectious diseases: an official publication of the Infectious Diseases Society of America. 2017;65(2):191–3. Epub 2017/04/14. pmid:28407096.

31. Mimoz O, Benoist JF, Edouard AR, Assicot M, Bohuon C, Samii K. Procalcitonin and C-reactive protein during the early posttraumatic systemic inflammatory response syndrome. Intensive care medicine. 1998;24(2):185–8. Epub 1998/04/16. pmid:9539079

32. Loebe M, Locziewski S, Brunkhorst FM, Harke C, Hetzer R. Procalcitonin in patients undergoing cardiopulmonary bypass in open heart surgery-first results of the Procalcitonin in Heart Surgery study (ProHearts). Intensive care medicine. 2000;26 Suppl 2:S193–8. Epub 2008/05/13. pmid:18470719.

33. Attar BM, Moore CM, George M, Ion-Nedelcu N, Turbay R, Zachariah A, et al. Procalcitonin, and cytokines document a dynamic inflammatory state in non-infected cirrhotic patients with ascites. World journal of gastroenterology. 2014;20(9):2374–82. Epub 2014/03/08. pmid:24605035; PubMed Central PMCID: PMC3942841.

34. Grace E, Turner RM. Use of procalcitonin in patients with various degrees of chronic kidney disease including renal replacement therapy. Clinical infectious diseases: an official publication of the Infectious Diseases Society of America. 2014;59(12):1761–7. Epub 2014/09/18. pmid:25228701.

35. Nylen ES, Al Arifi A, Becker KL, Snider RH Jr., Alzeer A. Effect of classic heatstroke on serum procalcitonin. Critical care medicine. 1997;25(8):1362–5. Epub 1997/08/01. pmid:9267950.

36. Carsin H, Assicot M, Feger F, Roy O, Pennacino I, Le Bever H, et al. Evolution and significance of circulating procalcitonin levels compared with IL-6, TNF alpha and endotoxin levels early after thermal injury. Burns: journal of the International Society for Burn Injuries. 1997;23(3):218–24. Epub 1997/05/01. pmid:9232281.

37. Nagata J, Kobayashi M, Nishikimi N, Komori K. Serum procalcitonin (PCT) as a negative screening test for colonic ischemia after open abdominal aortic surgery. European journal of vascular and endovascular surgery: the official journal of the European Society for Vascular Surgery. 2008;35(6):694–7. Epub 2008/02/26. pmid:18295515.

38. Scire CA, Cavagna L, Perotti C, Bruschi E, Caporali R, Montecucco C. Diagnostic value of procalcitonin measurement in febrile patients with systemic autoimmune diseases. Clin Exp Rheumatol. 2006;24(2):123–8. Epub 2006/06/10. pmid:16762145.

39. Stein M, Lipman-Arens S, Oved K, Cohen A, Bamberger E, Navon R, et al. A novel host-protein assay outperforms routine parameters for distinguishing between bacterial and viral lower respiratory tract infections. Diagnostic microbiology and infectious disease. 2018;90(3):206–13. Epub 2017/12/24. pmid:29273482.

40. Srugo I, Klein A, Stein M, Golan-Shany O, Kerem N, Chistyakov I, et al. Validation of a Novel Assay to Distinguish Bacterial and Viral Infections. Pediatrics. 2017;140(4). Epub 2017/09/15. pmid:28904072.

41. van der Does Y, Rood PPM, Ramakers C, Schuit SCE, Patka P, van Gorp ECM, et al. Identifying patients with bacterial infections using a combination of C-reactive protein, procalcitonin, TRAIL, and IP-10 in the emergency department: a prospective observational cohort study. Clinical microbiology and infection: the official publication of the European Society of Clinical Microbiology and Infectious Diseases. 2018;24(12):1297–304. Epub 2018/10/01. pmid:30268671.

42. Parnell GP, McLean AS, Booth DR, Armstrong NJ, Nalos M, Huang SJ, et al. A distinct influenza infection signature in the blood transcriptome of patients with severe community-acquired pneumonia. Critical care (London, England). 2012;16(4):R157. Epub 2012/08/18. pmid:22898401; PubMed Central PMCID: PMC3580747.

43. Herberg JA, Kaforou M, Wright VJ, Shailes H, Eleftherohorinou H, Hoggart CJ, et al. Diagnostic Test Accuracy of a 2-Transcript Host RNA Signature for Discriminating Bacterial vs Viral Infection in Febrile Children. Jama. 2016;316(8):835–45. Epub 2016/08/24. pmid:27552617; PubMed Central PMCID: PMC5997174.

44. Sweeney TE, Wong HR, Khatri P. Robust classification of bacterial and viral infections via integrated host gene expression diagnostics. Science translational medicine. 2016;8(346):346ra91. Epub 2016/07/08. pmid:27384347; PubMed Central PMCID: PMC5348917.

45. Mahajan P, Kuppermann N, Mejias A, Suarez N, Chaussabel D, Casper TC, et al. Association of RNA Biosignatures With Bacterial Infections in Febrile Infants Aged 60 Days or Younger. Jama. 2016;316(8):846–57. Epub 2016/08/24. pmid:27552618; PubMed Central PMCID: PMC5122927.

46. Zaas AK, Burke T, Chen M, McClain M, Nicholson B, Veldman T, et al. A host-based RT-PCR gene expression signature to identify acute respiratory viral infection. Science translational medicine. 2013;5(203):203ra126. Epub 2013/09/21. pmid:24048524; PubMed Central PMCID: PMC4286889.

47. Sampson DL, Fox BA, Yager TD, Bhide S, Cermelli S, McHugh LC, et al. A Four-Biomarker Blood Signature Discriminates Systemic Inflammation Due to Viral Infection Versus Other Etiologies. Scientific reports. 2017;7(1):2914. Epub 2017/06/08. pmid:28588308; PubMed Central PMCID: PMC5460227.

48. Henrickson SE, Manne S, Dolfi DV, Mansfield KD, Parkhouse K, Mistry RD, et al. Genomic Circuitry Underlying Immunological Response to Pediatric Acute Respiratory Infection. Cell reports. 2018;22(2):411–26. Epub 2018/01/11. pmid:29320737; PubMed Central PMCID: PMC5796675.

49. Zaas AK, Chen M, Varkey J, Veldman T, Hero AO 3rd, Lucas J, et al. Gene expression signatures diagnose influenza and other symptomatic respiratory viral infections in humans. Cell host & microbe. 2009;6(3):207–17. Epub 2009/08/12. pmid:19664979; PubMed Central PMCID: PMC2852511.

50. Heinonen S, Jartti T, Garcia C, Oliva S, Smitherman C, Anguiano E, et al. Rhinovirus Detection in Symptomatic and Asymptomatic Children: Value of Host Transcriptome Analysis. American journal of respiratory and critical care medicine. 2016;193(7):772–82. Epub 2015/11/17. pmid:26571305; PubMed Central PMCID: PMC4824929.

51. Shaikh N, Leonard E, Martin JM. Prevalence of streptococcal pharyngitis and streptococcal carriage in children: a meta-analysis. Pediatrics. 2010;126(3):e557–64. Epub 2010/08/11. pmid:20696723.

52. Baer G, Baumann P, Buettcher M, Heininger U, Berthet G, Schafer J, et al. Procalcitonin guidance to reduce antibiotic treatment of lower respiratory tract infection in children and adolescents (ProPAED): a randomized controlled trial. PLoS One. 2013;8(8):e68419. Epub 2013/08/13. pmid:23936304; PubMed Central PMCID: PMC3735552.