Development of Ebola virus disease prediction scores: Screening tools for Ebola suspects at the triage-point during an outbreak

Abstract

Background

The control of Ebola virus disease (EVD) outbreaks relies on rapid diagnosis and prompt action, a daunting task in limited-resource contexts.

This study develops prediction scores that can help healthcare workers improve their decision-making at the triage-point of EVD suspect-cases during EVD outbreaks.

Methods

We computed accuracy measurements of EVD predictors to assess their diagnosing ability compared with the reference standard GeneXpert® results, during the eastern DRC EVD outbreak. We developed predictive scores using the Spiegelhalter-Knill-Jones approach and constructed a clinical prediction score (CPS) and an extended clinical prediction score (ECPS). We plotted the receiver operating characteristic curves (ROCs), estimated the area under the ROC (AUROC) to assess the performance of scores, and computed net benefits (NB) to assess the clinical utility (decision-making ability) of the scores at a given cut-off. We performed decision curve analysis (DCA) to compare, at a range of threshold probabilities, prediction scores’ decision-making ability and to quantify the number of unnecessary isolation.

Results

The analysis was done on data from 10432 subjects, including 651 EVD cases. The EVD prevalence was 6.2% in the whole dataset, 14.8% in the subgroup of suspects who fitted the WHO Ebola case definition, and 3.2% for the set of suspects who did not fit this case definition. The WHO clinical definition yielded 61.6% sensitivity and 76.4% specificity. Fatigue, difficulty in swallowing, red eyes, gingival bleeding, hematemesis, confusion, hemoptysis, and a history of contact with an EVD case were predictors of EVD. The AUROC for ECPS was 0.88 (95%CI: 0.86–0.89), significantly greater than this for CPS, 0.71 (95%CI: 0.69–0.73) (p < 0.0001). At -1 point of score, the CPS yielded a sensitivity of 85.4% and specificity of 42.3%, and the ECPS yielded sensitivity of 78.8% and specificity of 81.4%. The diagnostic performance of the scores varied in the three disease contexts (the whole, fitting or not fitting the WHO case definition data sets). At 10% of threshold probability, e.g. in disease-adverse context, ECPS gave an NB of 0.033 and a net reduction of unnecessary isolation of 67.1%. Using ECPS as a joint approach to isolate EVD suspects reduces the number of unnecessary isolations by 65.7%.

Conclusion

The scores developed in our study showed a good performance as EVD case predictors since their use improved the net benefit, i.e., their clinical utility. These rapid and low-cost tools can help in decision-making to isolate EVD-suspicious cases at the triage point during an outbreak. However, these tools still require external validation and cost-effectiveness evaluation before being used on a large scale.

Introduction

Ebola virus disease (EVD) outbreaks have emerged in the tropical part of Africa with an increased incidence since the mid-nineties. The infection claimed tens of thousands of lives as its case fatality can reach ninety percent in the absence of any treatment [1–6]. Most EVD outbreaks have occurred in remote areas in which we note a lack of infrastructure and equipment, a few available diagnostic tools, and trained personnel. These conditions contribute to the further spread of the disease, compromising the outbreak control [6], and placing a heavy burden on the health systems and communities as well [7,8]. The control of EVD outbreaks relies on accurate and early detection of the pathogen to allow the rapid implementation of countermeasures [9]. Since the 2018 EVD outbreak [Equateur Province, Democratic Republic of the Congo (DRC)], the Cepheid® GeneXpert Ebola platform was used following the WHO’s instructions as the reference standard for EVD diagnosis at the point of care. The GeneXpert® is a highly sensitive and specific, rapid, and fully automated dual-purpose quantitative RT-PCR that diagnoses EVD with a few skills and a short turn-around time [10]. Although on-site GeneXpert technology has shown beneficial results in controlling the EVD outbreaks [11], timely scaling-up of this technology face to an expanding or quickly changing outbreak context remains challenging because of the requirements such as electricity supply, laboratory infrastructure, personnel training, and the cost. In these contexts, and an absence of validated rapid diagnostic tests (RDTs), clinical signs and symptoms associated with epidemiological information can serve as an immediate tool to identify suspect-cases and to support patients’ categorization and management. Early and accurate recognition of EVD can help improve the healthcare workers’ (HCW) decision-making and epidemic control.

In the early stage of the infection, sudden fever onset and other non-specific symptoms, such as severe headache, muscle and joint pain, and fatigue characterize the EVD [12]. In tropical settings, many infectious diseases have a similar initial clinical presentation. EBOV-specific symptoms appear at the last stage, further delaying the diagnostics in the absence of detection tools.

Often, HCWs must decide to isolate suspected cases without having any timely access to diagnostic tests. HCWs must weigh up the risk of sending true-positive EVD cases back to the community against the risk of admitting false-positive cases into the isolation wards. These errors in the triage result in the spread of the infection into the community or sustain nosocomial transmission in health settings, both delaying timely access to appropriate EVD care.

As for other epidemic pathologies, the clinical case definitions were developed, combining the signs and symptoms with risk factors (as having had contact with a confirmed case). The WHO clinical case definition (or local adaptation) is most commonly used through a recent meta-analysis showed its sub-optimal performance (pooled sensitivity of 81.5% (95% CI 74.1–87.2), and specificity 35.7% (28.5–43.6)) [13,14].

Thus, further research on simple clinical scoring systems is needed to add precision to the risk assessment in a single patient. Such simple tools, as for other conditions, are used, to better classify patients either for prognostic or management purposes [15–18].

Several prediction scores were developed to assess the individual risk of EVD using patient characteristics, clinical and epidemiological risk factors; and they were evaluated using accuracy metrics, e.g., sensitivity, specificity, and area under the curve [19–22]. However, the accuracy metrics only tell on how well a score discriminates EVD from non-EVD cases; but they do not assess the clinical usefulness of the score (i.e., their utility in clinical practice) by accounting for the clinical consequences of using them in healthcare practice. Consequently, none of the Ebola outbreak response teams has used these EVD risk scores during subsequent outbreaks. The score’s clinical utility assessment provides the potential benefits and harms of using a score relative to the predicted absolute risk of disease to inform clinical decision-making in clinical practice.

This study aims 1) to develop an EVD clinical prediction rule using surveillance data from the 2018–2020 DRC EVD outbreak; 2) to evaluate the clinical usefulness of the prediction rule to improve healthcare workers’ decision-making at the triage-point of EVD suspect-cased during outbreaks.

Materials and methods

Ethics statements

This study received the approval of the Ethics Committee of the University of Kisangani (Approbation No Ref: CER/005/GEAK/2017), and the authorization from the Secretariat Technique du Comité Multisectioriel de lutte contre la Maladie à Virus Ebola in DRC. This study used anonymized data from the surveillance and care unit collected as part of the EVD outbreak response (2018–2020). Since this study used de-identified data, the Ethics Committee did not consider individual patient consent as necessary.

Study subjects, data collection, and reference standard for EVD confirmation

In this retrospective cohort study, we analyzed the epidemiological and clinical data collected routinely upon identification or admission of an EVD suspect case. We used the datasets of the surveillance and care units teams deployed in the areas covered by the EVD coordination team of Butembo, in the North-Kivu province. We included data with alive EVD suspects and the community deaths identified in the Butembo area between September 3, 2018 and February 17, 2020.

In this outbreak, the surveillance and clinical teams used a standard clinical case definition adapted from the WHO Integrated Disease Surveillance and Response (IDSR) guidelines to identify EVD suspect-cases [23].

The case definition contained notions of residence, signs, or symptoms without referring to contact information. The case definition was stated as follows: “Any person alive or dead, living in North-Kivu, South-Kivu, or Ituri provinces or any person who traveled to these provinces during this period and who reported the following signs or symptoms; sudden fever onset and at least three of the following symptoms: vomiting, diarrhea, abdominal pain, conjunctivitis, rash, unexplained bleeding from any part of the body, muscle pain, intense fatigue, difficulty of swallowing, the difficulty of breathing, hiccups, or headache.”

For each EVD suspect case, the surveillance and/or clinical team collected, guided by a standard notification form, data on socio-demographics (age, gender, residence, profession), putative epidemiologic routes of contagion (contacts, type of contact), date of symptom onset, and signs and symptoms at the time of diagnostic.

After the clinical and epidemiological assessment, each EVD suspect-case was subjected to the sample collection process (blood or oral swab) for confirmatory testing. While waiting for the result, EVD suspect patients were kept in the isolation wards within the triage center (TC), Ebola treatment center (ETC), or in the community for cases associated with refusal to be transferred to the care units.

For the Butembo areas, all samples from suspect-cases were tested in the EVD field laboratory set by the Institut National de Recherche Biomédicale (INRB) in Butembo. The confirmation of EBOV infection relied on the detection of EBOV nucleoprotein (NP) and/or glycoprotein (GP) in the clinical specimens through GeneXpert® (Cepheid, Sunnyvale, CA, USA) [10,11].

In our analysis, we used the GeneXpert® results as the reference for the EVD status. EVD confirmed case was defined as a suspect case with a GeneXpert® positive, i.e., EBOV Cycle threshold-values (Ct-values) NP < 40. A probable EVD case was defined as a suspect case with high-risk contact for whom a test was not available because no sample was collected. A non-case was a suspect case with a GeneXpert® negative result.

The surveillance team was led by the experts of the Ministry of Health (MoH), supported by the World Health Organization (WHO) and other partner teams. Ebola care units (Transit, treatment, and triage) were managed or supported by different teams such as MoH, WHO, Médecins Sans Frontières (MSF), the Alliance International for Medical Action (ALIMA), Samaritan Purse, etc. DRC Red Cross National Society) and research partners (INRB, Institute of Tropical Medicine of Antwerp, and US National Institute of Health) over time.

Statistical analysis

Data description.

Study data were available on Microsoft Excel 2016® sheets whereas they were analyzed with SPSS® IBM statistic version 20 software and R® package version 3.6.1(R Foundation for Statistical Computing, Vienna, Austria). Most clinical and epidemiological information consisted of binary variables. We dichotomized continuous measurements at the cut-off where the sum of specificity and sensitivity was maximized using the receiver operating characteristic curves (ROC). We tested associations between each possible predictor and EVD status using the chi-square test or Fisher exact test, as appropriate, at 5% of significance.

Development of the scores.

Diagnostic accuracy, including the sensitivity, specificity, positive/negative likelihood ratios (LRs), and their 95% binomial confidence intervals for each demographic, clinical, and exposure information was calculated using 2x2 tables. For binary variables, we considered the presence of a sign/symptom to be a positive test for the disease compared to the gold standard test. We selected associated predictors with a crude LR+ ≥2.0 or LR—≤0.5 and adjusted in multivariate logistic regression analysis according to an adaptation of the Spiegelhalter-Knill-Jones approach [24] as described in [25]. This approach combines the independent Bayes method and the logistic regression to construct a scoring system [26]. We selected the adjusted diagnostic predictors with aLR ≥1.5 or ≤ 0.67 to construct the scoring system. We obtained the diagnostic score for each retained predictor from the natural log-transformation of the aLRs, rounded to their nearest integer. Positive scores favored the presence of EVD, whereas negative scores argued against it. We obtained the total predictive score (X) for each suspect by summing the diagnostic score for each predictor. We assigned a value of 0 to the missing data. Finally, we computed from the total score for each suspect, the posterior probability of having EVD as follows:(1)where X is the vector of all predictors X_i, D is the EVD status, β₀ is the constant of the regression equation and β_i the coefficient of the X_i predictor.

We developed two prediction models for the diagnosis of EVD: a clinical prediction model, the clinical prediction score (CPS), which included only clinical predictors, and an extended clinical prediction model, the extended clinical prediction score (ECPS), which included also demographic and epidemiologic predictors.

Evaluation of the developed scores.

The performance of the developed prediction models (i.e. the extent to which each model distinguishes between EVD and non-EVD) was analyzed with a focus on their discrimination power, as described in [15,27]. Thus, we evaluated the discrimination performance of the CPS and ECPS by constructing their receiver operating characteristic curves (ROC) and calculating the areas under the curve (AUROC) with 95% confidence intervals (95%CIs) in the whole dataset and in subgroups of the EVD suspect’s dataset (e.g., fitting or not the WHO case definition). An area of 1.0 indicates perfect discrimination power, whereas an area of 0.5 indicates no discrimination of the binary disease status (EVD versus non-EVD). Furthermore, we computed the diagnostic accuracy measures, e.g., sensitivity, specificity, PPV, and NPV, and their binomial 95%CIs at each cut-off point value of the score for each prediction model. Assuming that EVD prevalences in the two subgroups of suspects (e.g., those fitting or not fitting the WHO case definition) are different, we computed and described for each subgroup the diagnostic accuracy metrics to investigate the impact of the disease prevalence variation on the scores’ accuracy performance.

Finally, we compared the AUROCs for prediction models developed by computing the critical ratio z as described in Hanley and McNeil [28]. We performed a 10-fold cross-validation on the logistic regression step and plotted calibration plots to evaluate the out-of-sample performance of the model with the selected predictors. Calibration refers to the agreement between predicted probabilities and observed proportions, or the actual risk of a clinical outcome. We used the Hosmer-Lemeshow goodness-of-fit test, and a model was considered statistically fit data if the p-value was greater than 0.1.

Clinical usefulness of the prediction models.

To evaluate the clinical usefulness or utility of the developed prediction scores, we performed decision curve analysis, which depicts the net benefit (NB) of using a model at various ranges of threshold probabilities of interest, as described by Vickers and Elkin [29]. The net benefit defines the difference between the benefit for true cases (TP) who would receive correctly the intervention (isolation and care) based on the prediction of the model and the expected harm for non-cases (FP) who will erroneously receive the intervention weighted by the odds of the patient’s threshold probability. The model with the higher net benefit at a given threshold probability is the preferred model.

The net benefit (NB) for a given threshold probability was calculated as the following:(2)

In this equation, N denotes the total sample size and pt denotes the probability threshold above which a suspect is considered high-risk.

The weighting factor, the odds pt/(1-pt), captures the patient or health professional’s values regarding the risks of false-negative and false-positive results.

Additionally, we plotted the decision curves for models and computed the reduction in the number of unnecessary isolation of suspects by using these models at a range of threshold probabilities of interest to compare our developed models. Additionally, the models were compared to two extreme strategies, isolating all suspects (where TP/N is the prevalence of EVD among the suspects and FP/N is one minus the prevalence of EVD in the NB formula) and isolating none of the suspects (where TP = zero and FP = zero, thus NB = zero at any threshold probability). For more detail on the technical approach of our derivation, please consider the paper by Vickers (http://www.decisioncurveanalysis.org/). In practice, implementing the decision curve analysis requires the following height steps:

1. Choose a value for p_t, e.g., the threshold probability, a composite of patient and health workers’ preferences (a theoretical risk level where the expected benefit of isolating a suspect is equal to the expected benefit of avoiding it).

2. With this threshold probability as a cut-point, calculate the number of true and false positives using 2x2 tables that summarize the positive or negative results.

3. Compute the net benefit of the prediction model.

4. Repeat steps 2–3 by varying p_t over a range of interest.

5. Plot the net benefit on the y-axis against p_t on the x-axis to draw the decision curve for the model.

6. Repeat steps 1–5 for every model to compare.

7. Repeat steps 1–5 for the strategy that assumes all EVD suspects are positive.

8. Finally, a straight line drawn parallel to the x-axis at y = 0 represents the net benefit associated with the strategy of assuming that all suspects are negative.

We computed the reduction in the number of unnecessary isolations per 100 EVD suspects as described in [30].

(3)

As a comparison, the EVD predictive model with the highest NB or the highest percentage of reduction of unnecessary isolation at a particular point of threshold probability will yield the best clinical usefulness at this point, e.g., more true-positive cases and, or fewer false-positive cases.

Finally, we evaluated our prediction models according to two additional clinical practice approaches in healthcare settings: joint and conditional tests or approaches. In both approaches, the suspects with no reported risk of exposure would be considered to not have the disease and the clinical team would act accordingly. No additional action, e.g., isolation, would be required. In the joint approach, the clinical team should clinically examine all suspects at low-, intermediate-, and high-risk reported exposure and recommend for isolation only those with a predicted probability of EVD greater than 5% (the cut-off chosen to maximize sensitivity, about 90 percent, in disease adverse context). In the conditional approach, the clinical team should isolate all suspects with high-risk reported exposure irrespective of their predicted probability of the disease and then suspects at low and intermediate reported exposure having an EVD-predicted probability greater than 5%.

Thus, we first grouped EVD suspects into high-, intermediate-, and low-risk exposure according to their reported risk exposure to EVD. We define the categories of EVD risk exposure for suspects as follows: (1) no risk exposure if there has been no reported EVD risk exposure. (2) Low-risk exposure, any suspect who had direct contact with a living EVD patient who did not present diarrhea, vomiting, or bleeding at the moment of contact; or any suspect who had touched patient clothes or sheets which were not visibly soiled; or any funeral attendance or hospital visit but without reported direct contact. (3) Intermediate-risk exposure includes having had any contact with a living EVD patient who had diarrhea, vomiting, or bleeding, as well as any contact with clothes or sheets soiled with stool, vomit, or blood. (4) High-risk exposure: having cared for an EVD patient with diarrhea, vomiting, or bleeding; or having cleaned a patient room or clothes; or having handled patient waste; or any direct exposure to blood, body fluids (vomitus, urine, feces), or tissues from EVD patients; or direct skin contact with skin, blood, or body fluids from EVD patients; or having processed blood or body fluids from an EVD patient without protective equipment or direct contact with an EVD patient’s dead body.

Results

The surveillance system registered 10496 EVD suspect-cases during the outbreak in Butembo (DRC). Among them, 704 were confirmed-cases (including 673 confirmed EVD cases and 31 probable cases linked epidemiologically to a confirmed EVD case but no GeneXpert test done) and 9792 were classified as non-case cases during the epidemic. We excluded for analysis 31 suspects without laboratory results and those without clinical and epidemiological data (including 22 EVD cases and 11 non-EVD cases). Thus, we included for analyses 10432 subjects, among which 651 were EVD cases and 9781 were non-EVD cases as classified by the gold standard test used for screening during this outbreak. The EVD prevalence among Ebola-suspected people was 6.2% of the total sample. The disease prevalence was 14.8% among suspects who fitted the WHO case definition for the suspect (401 EVD cases in 2707 suspects) and 3.2% for those who did not fit (250 EVD cases in 7725 suspects). Fig 1 presents a flow diagram showing the number of EVD suspects included in the study and their outcomes (Fig 1).

[Figure omitted. See PDF.]

Classification of 10496 notified cases according to the WHO guideline case Definition for Ebola virus disease (EVD) during the Ebola virus Outbreak in BUTEMBO, RDC.

The Table 1 summarizes the associations between the EVD status and the epidemiologic and clinical characteristics of the patients. This analysis revealed a significant association with the EVD status for all variables except only four predictors.

[Figure omitted. See PDF.]

Table 2 details the accuracy metric results of epidemiologic and clinical predictors to predict the diagnosis of the EVD. Among the associated epidemiologically predictors, having reported any contact with EVD case had the highest sensitivity (65.4%, 95%CI: 61.5%–69.1%), followed by being more than 25 years old (61.7%, 95%CI: 57.8%–65.4%). For the clinically associated predictors, the highest sensitivity was 76.8% (95%CI: 73.3%–79.9%) for malaise and fatigue and the highest specificity was 99.9% (95%CI: 99.8%–99.9%) and 99.8% (95%CI: 99.7%–99.9%) for gingival bleeding and black-vomiting (vomito-negro), respectively. Additionally, the sensitivity for hemorrhagic signs was low, between 0.2% for hematuria and 11.7% for any bleeding, and their specificity was relatively high, up to 100% for petechiae (Table 2). Further, most of these variables met the criteria for being the predictors of EVD diagnosis based on their positive and negative crude LRs as they are different from one for most of them. The highest positive LRs were 5.7(95%CI: 5.1–6.4) for any history of contact with an EVD case, 4.4(95%CI: 3.2–5.9) for eye redness, 45(95%CI: 15.5–64.0) for hematemesis, and 7.8 (95%CI: 4.5–12.0) for confusion-disorientation. The negative crude LRs were 0.4(95%CI: 0.3–0.4) for any history of contact with an EVD case among the epidemiologic predictors, and 0.5(95%CI: 0.4–0.6) for malaise/fatigue among the general clinical predictors. None of the hemorrhagic or neurological symptoms demonstrated the ability to rule out EVD infection in the analyses, as their negative LRs were equal to one or included one in their related confidence interval.

[Figure omitted. See PDF.]

Based on their crude likelihood ratios, the epidemiologic predictors ‘any history of contact with EVD case’, and ‘any contact with the healthcare setting’ and the clinical predictors fatigue, sore throat, difficulty of swallowing, eye redness, confusion-disorientation, hematemesis, vomito-negro or black-vomiting, melena, gingival bleeding, and hemoptysis met the criteria to be included in multivariable analysis. After adjustment and shrinkage, for both the CPS and ECPS, predictors including fatigue, difficulty in swallowing, confusion-disorientation, red eyes, gingival bleeding, hematemesis, hemoptysis, and any history of contact with an EVD patient, had an adjusted likelihood ratio (aLR)≥1.5 or ≤0.67. Thus, they fulfilled the criteria to be included in the scores. Table 3 presents the details of the results of the (E) CPS development analyses and the weighted scores produced after natural log-transformation.

[Figure omitted. See PDF.]

The total score for each individual patient in the EVD surveillance dataset can range from -2 to +7 points for the CPS and from -4 to +9 points for the ECPS. Fig 2 and the S1 Table describe the distribution of the total of predictive scores CPS and ECPS in the EVD and non-EVD groups.

[Figure omitted. See PDF.]

Fig 3 shows areas under the ROC curve (AUROC) plotting true-positive against the false-positive rate for each model of prediction, in the overall sample and in subgroups of suspects, depending or not if they fit with the WHO case definition. Applied to the whole dataset, the AUROC is 0.69 (95%CI: 0.67–0.71), 0.71(95%CI: 0.69–0.73) and 0.88 (95%CI: 0.86–0.89) for the WHO criteria, clinical and extended clinical model, respectively. Fig 3B and 3C show the AUROCs for the subgroups of EVD suspects (e.g. fitting or not the WHO criteria) (Fig 3). The AUROC for the extended score (ECPS) was significantly higher than the AUROC of the clinical prediction score (z = 12.1, p < 0.0001).

[Figure omitted. See PDF.]

(A) depicts the ROCs for the overall set of EVD suspects, (B) the ROCs for the set of EVD suspects who fit, and (C) presents the ROCs for the set of EVD suspects who do not fit with the WHO case definition used.

Fig 4 presents the calibration plots, which depict the observed proportion versus predicted probability of EVD plots for our scores. Generally, CPS well fitted the data and ECPS did not (Fig 4). Fig 5 and S2 Table present the results of the 10-fold cross-validation analysis for the two clinical prediction scores (CPS and ECPS). The mean cross-validation ROC (AUCCV) over the cross-validated samples with the ECPS was 0.87 (0.88 for the full dataset), and the mean AUCCV over the cross-validated samples with CPS was 0.71, the same as for the full dataset.

[Figure omitted. See PDF.]

Calibration plots depict the observed proportion versus predicted probability of EVD. (A) and (B) present the calibration plot for the overall set of EVD suspects, (C) and (D), the set of suspects who fit with the WHO case definition, and (E) and (F) for those not fitting with the WHO case definition used.

[Figure omitted. See PDF.]

(A) is the ROC for the extended clinical prediction score, (B) the ROC for the extended clinical prediction score.

At cut-off point value lower or equal to -1 for the clinical prediction score, chosen arbitrarily, sensitivity of 87.3% (95%CI: 84.4%–89.7%), and specificity of 42% (95%CI: 41%–43%) were estimated for the diagnosis of EVD. For the extended prediction score, the sensitivity of 81.4% (95%CI: 78.2% –84.3%), and specificity of 80.5% (95%CI: 79.7%–81.3%) were obtained at the cut-off point value of the score equal or lower than -1 for instance (Table 4).

[Figure omitted. See PDF.]

Table 5 describes the performance of extended and clinical prediction scores on two subpopulations, one that fitted the WHO case definition for the suspect (with a disease prevalence of 14.8%) and the other that did not (with an EVD prevalence of 3.2%). At an arbitrarily chosen cut-off point value of -1 for the clinical prediction score, sensitivity of 91.0% (95% CI: 87.7%–93.6%) and specificity of 27.6% (95% CI: 25.8%–29.5%) were estimated in the 14.8% disease prevalence subgroup versus 76.4% (95% CI: 70.6%–81.5%) and 46.9% (95% CI: 45.7%–48.0%) in the 3.2% prevalence subgroup. For the extended prediction score, the sensitivity of 76.6% (95% CI: 72.0%–80.6%) and specificity of 78.0% (95% CI: 76.2%–79.6%) in 14.8% subgroup suspects versus 82.4% (95% CI: 77.1%–86.9%) and 85.4% (95% CI: 81.6%–83.4%) in 3.2% prevalence were obtained at the cut-off point value of the score equal to or lower than -1, for instance (Table 5).

[Figure omitted. See PDF.]

Fig 6 depicts the decision curves for isolating the suspects according to the prediction of the clinical and extended clinical scores compared to isolating all and isolating none of the suspects on a range of reasonable threshold probabilities. Fig 6A shows that the use of the clinical or extended clinical scores to isolate EVD-suspected patents offers higher net benefits than alternative reasonable strategies (e.g., isolating all or none of the suspects across all ranges of threshold probabilities of interest. When used to isolate EVD suspects, ECPS has more net benefit than any reasonable strategies on all ranges of probabilities below 70% and CPS has more net benefits than the "isolate all" or none suspects on the range of threshold probabilities between 2% and 67%.

[Figure omitted. See PDF.]

(A) represents the DCAs for the CPS and ECPS, and (B) the DCAs for the joint and conditional approaches with CPS and ECPS.

In other words, irrespective of any differences in patient or health professional preferences to isolating the suspect, using (E) CPS improves the clinical decision-making (value) at the point of triage (e.g., increase isolation of EVD case and reduce isolation of false-positive EVD cases) on the ranges of threshold probabilities mentioned above. Additionally, the figure also shows that using the extended clinical prediction score has more net benefit than the clinical one at all ranges of probability thresholds. For instance, at a threshold probability of 5%, the net benefit of isolating the suspects using the prediction of the extended clinical score is higher than that of isolating the suspects according to the clinical prediction score, 0.041 and 0.026, respectively.

That means that, compared to the isolating "none of the suspects" strategy (by assuming that all suspects are negative), isolating the suspects on the basis of the CPS and ECPS is equivalent to strategies that correctly identify and isolate approximately 4.1 and 2.6 EVD cases per 100 suspects, respectively, without an increase in the number of non-EVD cases isolated as EVD cases. Furthermore, at this threshold, the net benefit of the CPS and ECPS is 0.013 and 0.028 greater than assuming all suspects are EVD. When these scores are used, there are 24.7% [= 0.013*100/5/(100–5)] fewer non-EVD cases classified and isolated as EVD cases with CPS and 53.2% [= 0.028*100/5/(100–5)] fewer non-EVD cases classified and isolated as EVD cases with EPCS per 100 suspects. These percentages represent the proportion of the reduction in isolation rate without missing any EVD cases (See Table 6 and Fig 6A).

[Figure omitted. See PDF.]

Both prediction score models add clinical value in the entire range of threshold probability up to 65%; over this threshold, they add no clinical benefit than isolating none of the suspects, as their net benefits are negative.

Finally, Fig 6B shows the DCAs for using the joint and conditional approaches with the two clinical prediction models. The figure shows that, relative to isolating all and isolating none, joint approaches for both ECPS and CPS had better patient clinical outcomes (net benefit) than their conditional approaches in a large range of threshold probabilities. It is between 2% and 9% range of threshold probabilities for the joint approach with CPS, 4% and 16% for the conditional approach with CPS, 2%, and 22% for the joint approach with ECPS, and 4% and 26% for the conditional approach with ECPS. This means that, for the preferences (or threshold probabilities) below this range of threshold probabilities, the isolate all suspect strategy is the best, and above this threshold, the best option is to isolate none of the suspects. Additionally, choosing a home for isolating suspects would be an option, for example.

Discussion

Main findings

We need sensitive and affordable diagnostic tools and efficient strategies to detect and control EVD outbreaks, especially in the settings with limited-resources and increased risk occurrence for EVD outbreaks in Africa. This study developed clinical prediction scores to improve the triage of suspect-cases at the point of-entry of a health facility during EVD outbreaks.

Importantly, the study has identified seven symptoms, including fatigue, difficulty in swallowing, red eyes, gingival bleeding, hematemesis, mental confusion, and hemoptysis, and one epidemiologic, having a history of any contact with an EVD case, as diagnostic predictors to build the two prediction scores (CPS and ECPS). These latter showed a better and excellent power in distinguishing the EVD from other conditions, respectively. The two models of scores are robust for the selected predictors. The addition of epidemiological factors clearly increased the diagnostic accuracy, and ECPS is preferred.

Clinical symptoms and signs as well as risk exposure predictors

Our previous research on this data has compared the clinical and epidemiologic features of EVD and non-EVD to inform HCW on the case management decisions and infection control measures (Tshomba et al., in review). The study, along with others [31–33], identified and extensively discussed several epidemiological and clinical predictors associated with the diagnosis of EVD, most of which were also found in non-EVD patients during this outbreak. Importantly, the commonly reported risk factors and symptoms in EVD had previously been identified [34,35], and none of these clinical and epidemiologic predictors were found to be sensitive enough to better differentiate EVD cases. This study aimed to develop clinical prediction scores and evaluate their performance and clinical utility. Thus, our discussion will mainly focus on this aim.

Clinical accuracy of symptoms, signs and risk exposure predictors

This study developed two prediction scores, which included fatigue, difficulty in swallowing, confusion-disorientation, red eyes, gingival bleeding, hematemesis, hemoptysis, and any history of contact with an EVD patient as a diagnostic predictor, some of which and their accuracy performance were consistent with those previously developed [21,22,36]. Although the methods used to develop prediction scores were different from ours, both models demonstrated better performance in discriminating EVD cases from others. Additionally, this discriminative performance remained the same, thus robust, in the two stratified sets of sample, e.g., for the set that fitted or did not fit the WHO case definition for the EVD suspects. Our study found that specificity (the ability of screening tests to recognize non-EVD suspects) for almost all cut-off points of the developed scores to be variable. Low specificity for a screening test implies that a high number of individuals will be isolated even though they are not EVD cases (false positives). Although the mathematical expressions of the basic accuracy metrics of the screening test do not depend on the prevalence, variations in specificity and sensitivity (the ability of screening tests to recognize EVD suspects) rely on the clinical variability in clinical presentations, which is the result of the variability in disease prevalence. Increasing the disease prevalence increases sensitivity but at the same time lowers specificity or vice versa. This was the case for the suspect sample we studied, in which the EVD prevalence was only 6.2%. For high disease prevalence suspects with many severe EVD patients, e.g., in the subgroup who fits the WHO case definition for the suspect (disease prevalence of 14.8%), the test sensitivity may appear better in this group of suspected populations than when applied to less severe EVD suspects. The increase in the prevalence of EVD decreases ruling out EVD cases (fewer false negatives). As a result, in clinical practice, health workers involved in the screening of EVD suspects should be aware of the accuracy variability related to EVD prevalence to make informed decisions. For this reason, using predictive values that translate the pre-test probability of disease to the post-test probability based on screening test results can be a better choice to guide clinicians in making rational screening and isolation decisions to control EVD. This post-probability depends additionally on the prevalence, or pre-test probability, of EVD—how common Ebola infections are in the EVD suspect group under consideration. In the same sense, the better the disease prevalence, the better the sensitivity of the screening test applied to discriminate high-risk suspects from low-risk ones.

The highest sum sensitivity-specificity is at -1 point and -1 point, respectively, for the clinical prediction score and the extended one; the models are associated with sensitivity of 85% and 42.30% and specificity of 78.8% and 81.40%. This means, for our prediction scores, the presence of a positive numeric score will be a useful argument for more investigations and/or the isolation of a suspect case. However, instead of choosing a sole cut-point, it is worth dividing it into three or four categories based on the numerical score, describing different probability levels of being useful in clinical practice. For instance, for these two models, one can consider a score of prediction equal or less than -2 as having a very low probability of the disease (i.e., less than 20%) and a score of prediction greater than +2 as having very high probability of the disease and act accordingly. With intermediate probabilities, e.g., between these two levels of predictive disease probabilities, one should consider the prediction score as not conclusive and call for additional clinical evaluation. In this sense, using these models can allow the decision to be made on at least 40.7 percent (4141 non-EVD and 114 EVD cases) and 79.8% (7965 non-EVD and 360 EVD cases), respectively, for the clinical prediction score and the extended score.

Therefore, using these prediction scores can reduce the number of unnecessary isolations and/or additional procedures to be carried out for suspect-cases. The two prediction scores are associated with very low probability for EVD, and they avoid any delay in the isolation and case management for subjects with very high probabilities of disease in the isolation wards. Another way of using this score is to set up a threshold based on the greater sum sensitivity-specificity and then apply it to all suspect-cases. Those with scores above the threshold can be isolated, whereas subjects with a score under the threshold will not. However, the choice of the operational cut-off score should not only depend only on sensitivity and specificity but also on how professionals or patients weigh the errors of classification (false-positive and negative). Notably, the availability of human resources and health infrastructures and the context of use must be considered while managing the two prediction scores. For instance, in a context with adequate infrastructure, available effective drugs, a "disease-averse" context ("worried about the disease"), it is worth choosing a sensitive operational cut-off to catch a maximum number of EVD cases. A sensitive threshold can reduce the transmission in the community, and can be held to separate suspect-cases within the isolation wards to reduce cross-contamination between true-positive and false-positive cases. Conversely, in a context with poor health infrastructure and low density of the population, an "isolation-averse" context (e.g., "worried about the isolation" context), choosing a high specific cut-point (avoiding the FP) can reduce the risk of nosocomial infection among the patients with low-risk of contamination at the community level. With caution, prediction scores could be useful tools in the settings with limited-resources. The prediction scores do not intend to substitute the rational clinical or epidemiological judgments but to provide additional information, which can help healthcare workers to decide whether to isolate an EVD suspect case.

Clinical usefulness of the developed models

Furthermore, we assessed the clinical usefulness of the two predictions scoring models developed by AUROC and the net benefit of using them to isolate EVD suspect-cases. We noted that both diagnostic prediction scores provided significantly higher AUROC and NB values relative to using clinically reasonable strategies (e.g., isolating all or none of the suspects). The AUROC is a summary measure of discrimination between individuals who experienced the disease and those who did not. Though it is subjective to determine the good value of the AUROC for assessing the disease risk, the AUROC values for the two scores developed are considered excellent, according to D’Agostino’s AUROC classification [37]. Additionally, the two final scoring algorithms are quite robust for the selected variables. However, accuracy measures only show us about the models or tests’ discriminative performance; they do not consider the clinical consequences of using these clinical prediction models to inform on their clinical usefulness in making decisions [29]. While addressing this issue, the study has also shown that incorporating prediction models into the decision-making to isolate suspect-cases would improve clinical results or outcomes linked to the use of WHO case definition for suspect-cases while awaiting the confirmation results. Their use avoids many unnecessary isolation and only a few true EVD cases could miss the isolation. Our findings demonstrate the utility of these models while identifying suspect-cases regardless of patient or professional preferences. The application of the DCA corresponds to the situation in which patients, EVD suspect-cases do not have a definite diagnosis yet. The healthcare workers or clinical team must decide on whether the suspect-cases should be isolated or not, while the isolation exposes them to the cross-contamination among isolated people. This decision depends on the posterior probability of the disease for a suspect. Practically, the decision is straightforward if the suspected-case has very low or very high probability, e.g., less than 20% or greater than 80%, to have the disease. For intermediate probabilities, the decision is more complex and will require a consensus. The decision should consider the EVD’s risk of spread-out (i.e., in communities or healthcare stings), disease severity and the context in which the decision is taken, including the availability of the resources, and should also be discussed on the basis of the perspective of the decision, which can be patient-centered, healthcare-centered, or societal-centered. Again, the DCAs represent rapid tools in the decision-making of healthcare workers during the surveillance and at the screening at the point of-entry since they do not require much additional data in the analysis. Nonetheless, they do not substitute traditional decision analysis, e.g., cost-effectiveness analysis, when their use is to inform and direct a policy choice.

Strengths of developed predictive score systems

This study has explored the potential usefulness of clinical and epidemiological predictors in making clinical decisions in screening EVD suspect-cases with a large sample size. The usefulness of these predictive scores as rapid tests lies in their lack of cost and demand in terms of logistics Thus, standardized and applied screening tools for the health system frontline during outbreaks would be simple and beneficial in shortening the delay in decision-making by healthcare workers at the triage and/or case management.

Limitations of the study

The main limitations of the study are First, the database used for this analysis contained many community cases from surveillance, which had missed clinical and epidemiologic data.

Second, the study included a population with a high prevalence of severe patients from the Ebola treatment unit during the epidemic period. Thus, the study missed most clinically asymptomatic or mild cases. Although the performance of our scores was excellent, their use during the inter-epidemic period or in other settings will need further evaluation.

Finally, the DCAs are simple and quick tools to improve the making of decisions, as their construction requires only one parameter, the range of threshold probabilities. However, for more complex decision choices, i.e., to inform policy, further research must capture the uncertainty involved before their use in a large context.

In conclusion

An improved triage process and quick therapeutic decision could reduce the intra-hospital spread of infection and the disease-related mortality rate. This study developed diagnostic prediction scores that can aid in calculating the risk of being Ebola positive as a suspect case at the triage-point. These rapid and low-cost tools can help in decision-making to isolate EVD suspect cases at the triage point during an outbreak, as they constitute simple tools to assess the risk of having Ebola disease in suspected people. However, these tools still require external validation and a cost-effectiveness evaluation before being used on a large scale.

Supporting information

S1 Table. Distribution of clinical prediction score and extended clinical prediction score among suspect cases.

https://doi.org/10.1371/journal.pone.0278678.s001

(DOCX)

S2 Table. 10-fold cross-validation accuracy for the clinical prediction score and extended clinical prediction score.

https://doi.org/10.1371/journal.pone.0278678.s002

(DOCX)

S1 File. Notification chart used in the 2018–2020 DRC EVD outbreak.

https://doi.org/10.1371/journal.pone.0278678.s003

S2 File. TRIPOD checklist for prediction model development and valuation.

https://doi.org/10.1371/journal.pone.0278678.s004

(DOCX)

S3 File. North-Kivu/Butembo surveillance row data.

https://doi.org/10.1371/journal.pone.0278678.s005

(XLSX)

Acknowledgments

The authors thank the local and international staff whose efforts contained the epidemic and collected data used in this study.

Citation: Tshomba AO, Mukadi-Bamuleka D-R, De Weggheleire A, Tshiani OM, Kitenge RO, Kayembe CT, et al. (2022) Development of Ebola virus disease prediction scores: Screening tools for Ebola suspects at the triage-point during an outbreak. PLoS ONE 17(12): e0278678. https://doi.org/10.1371/journal.pone.0278678

About the Authors:

Antoine Oloma Tshomba

Roles: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Visualization, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliations: Department of Public Health, University of Kisangani, Kisangani, Democratic Republic of Congo, National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo

ORICD: https://orcid.org/0000-0003-2440-8468

Daniel-Ricky Mukadi-Bamuleka

Roles: Data curation, Investigation, Validation, Writing – original draft, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Medical Biology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

Anja De Weggheleire

Roles: Methodology, Validation, Writing – original draft, Writing – review & editing

Affiliation: Department of Clinical Sciences, Institute of Tropical Medicine, Antwerp, Belgium

ORICD: https://orcid.org/0000-0001-9377-4766

Olivier M. Tshiani

Roles: Data curation, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Medical Biology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

Richard O. Kitenge

Roles: Data curation, Validation, Writing – review & editing

Affiliation: National Emergency Program, Ministry of Health, Kinshasa, Democratic Republic of the Congo

Charles T. Kayembe

Roles: Conceptualization, Methodology, Supervision, Writing – review & editing

Affiliation: Department of Internal Medicine, University of Kisangani, Kisangani, Democratic Republic of the Congo

Bart K. M. Jacobs

Roles: Formal analysis, Methodology, Validation, Writing – review & editing

Affiliation: Department of Clinical Sciences, Institute of Tropical Medicine, Antwerp, Belgium

Lutgarde Lynen

Roles: Methodology, Validation, Writing – review & editing

Affiliation: Department of Clinical Sciences, Institute of Tropical Medicine, Antwerp, Belgium

ORICD: https://orcid.org/0000-0001-7183-4895

Placide Mbala-Kingebeni

Roles: Data curation, Formal analysis, Validation, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Medical Biology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

Jean-Jacques Muyembe-Tamfum

Roles: Project administration, Supervision, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Medical Biology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

Steve Ahuka-Mundeke

Roles: Funding acquisition, Project administration, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Medical Biology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

Dieudonné N. Mumba

Roles: Conceptualization, Methodology, Project administration, Supervision, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Tropical Medicine, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

ORICD: https://orcid.org/0000-0002-5886-2004

Désiré D. Tshala-Katumbay

Roles: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Neurology and School of Public Health, Oregon Health & Science University, Portland, Oregon, United States of America, Department of Neurology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

Sabue Mulangu

Roles: Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Validation, Writing – review & editing

Affiliations: National Institute for Biomedical Research, Kinshasa, Democratic Republic of the Congo, Department of Medical Biology, University of Kinshasa, Kinshasa, Democratic Republic of the Congo

References

1. Albariño CG, Shoemaker T, Khristova ML, Wamala JF, Muyembe JJ, Balinandi S et al.: Genomic analysis of filoviruses associated with four viral hemorrhagic fever outbreaks in Uganda and the Democratic Republic of the Congo in 2012. Virology 2013, 442: 97–100 Epub 2013 May 25. pmid:23711383

2. Coltart CE, Lindsey B, Ghinai I, Johnson AM, Heymann DL: The Ebola outbreak, 2013–2016: old lessons for new epidemics. Philos Trans R Soc Lond B Biol Sci 2017, 372: pii: 20160297. pmid:28396469

3. Georges AJ, Leroy EM, Renaut AA, Benissan CT, Nabias RJ, Ngoc MT et al.: Ebola hemorrhagic fever outbreaks in Gabon, 1994–1997: epidemiologic and health control issues. J Infect Dis 1999, 179 Suppl 1: S65–S75. pmid:9988167

4. Rosello A, Mossoko M, Flasche S, Van Hoek AJ, Mbala P, Camacho A et al.: Ebola virus disease in the Democratic Republic of the Congo, 1976–2014. Elife 2015, 4: pii: e09015. pmid:26525597

5. Aruna Aaron, Mbala Placide, Minikulu Luigi, Mukadi Daniel, Bulemfu Dorothée, Edidi Franck et al.: Ebola Virus Disease Outbreak—Democratic Republic of the Congo, August 2018-November 2019. MMWR Morb Mortal Wkly Rep 2019, 68: 1162–1165 Published online 2019 Dec 20. pmid:31856146

6. WHO Ebola Response Team.: After Ebola in West Africa—Unpredictable Risks, Preventable Epidemics. N Engl J Med 2016, 375: 587–596. pmid:27509108

7. The World Bank. The economic impact of the 2014 Ebola epidemic: short and medium term estimates for Guinea, Liberia and Sierra Leone. Access on. http://documents.worldbank.org/curated/en/2014/09/20214465/economic-impact-2014-ebola-epidemic-short-medium-term-estimates-guinea-liberia-sierra-leone. 17-9-2014.

8. Fortin A, Vroh Benie Bi J, Soulimane A.: [The lessons of the Ebola epidemic for a more effective preparedness face to health crises]. Sante Publique 2017, 29: 465–475. pmid:29034662

9. Broadhurst MJ, Brooks TJ, Pollock NR: Diagnosis of Ebola Virus Disease: Past, Present, and Future. Clin Microbiol Rev 2016, 29: 773–793. pmid:27413095

10. Towner JS, Rollin PE, Bausch DG, Sanchez A, Crary SM, Vincent M et al.: Rapid diagnosis of Ebola hemorrhagic fever by reverse transcription-PCR in an outbreak setting and assessment of patient viral load as a predictor of outcome. J Virol 2004, 78: 4330–4341. pmid:15047846

11. Semper AE, Broadhurst MJ, Richards J, Foster GM, Simpson AJ, Logue CH et al.: Performance of the GeneXpert Ebola Assay for Diagnosis of Ebola Virus Disease in Sierra Leone: A Field Evaluation Study. PLoS Med 2016, 13: e1001980. eCollection 2016 Mar. pmid:27023868

12. Geisbert T.W., Hensley L.E.: Ebola virus: New insights into disease aetiopathology and possible therapeutic interventions. Expert Rev Mol Med 6 2004, 6: 1–24. pmid:15383160

13. World Health Organization. Case definition recommendations for Ebola or Marburg Virus Diseases. https://www.who.int/csr/resources/publications/ebola/ebola-case-definition-contact-en.pdf. 9-8-0014.

14. Caleo G, Theocharaki F, Lokuge K, Weiss HA, Inamdar L, Grandesso F et al.: Clinical and epidemiological performance of WHO Ebola case definitions: a systematic review and meta-analysis. Lancet Infect Dis 2020, 20: 1324–1338. Epub 2020 Jun 25. pmid:32593318.

15. Barili F, Pacini D, D’Ovidio M, Ventura M, Alamanni F, Di Bartolomeo R et al.: Reliability of Modern Scores to Predict Long-Term Mortality After Isolated Aortic Valve Operations. Ann Thorac Surg 2016, 101: 599–605. Epub 2015 Oct 21. pmid:26499815

16. Kuo G, Yang SY, Chuang SS, Fan PC, Chang CH, Hsiao YC et al.: Using acute kidney injury severity and scoring systems to predict outcome in patients with burn injury. J Formos Med Assoc 2016, 115: 1046–1052. Epub 2016 Dec 3. pmid:27923537

17. Lynen L, An S, Koole O, Thai S, Ros S, De Munter P et al.: An algorithm to optimize viral load testing in HIV-positive patients with suspected first-line antiretroviral therapy failure in Cambodia. J Acquir Immune Defic Syndr 2009, 52: 40–48. pmid:19550349

18. van Griensven J, Phan V, Koole O, Lynen L: Simplified clinical prediction scores to target viral load testing in adults with suspected first line treatment failure in Phnom Penh, Cambodia. PLoS ONE 2014, 9: e87879. eCollection 2014. pmid:24504463

19. Hartley MA, Young Ai, Tran AM, Okoni-Williams HH, Suma M, Mancuso Bet al.: Predicting Ebola infection: A malaria-sensitive triage score for Ebola virus disease. PLOS Neglected Tropical Diseases 2017, pmid:28231242

20. Ingelbeen B, De Weggheleire A, Van Herp M, van Griensven J: Symptom-Based Ebola Risk Score for Ebola Virus Disease, Conakry, Guinea. Emerg Infect Dis 2018, 24: 1162. pmid:29774864

21. Levine AC, Shetty PP, Burbach R, Cheemalapati S, Glavis-Bloom J, Wiskel T et al.: Derivation and Internal Validation of the Ebola Prediction Score for Risk Stratification of Patients With Suspected Ebola Virus Disease. Ann Emerg Med 2015, 66: 285–293.e1. Epub 2015 Apr 3. pmid:25845607

22. Oza S, Sesay AA, Russell NJ, Wing K, Boufkhed S, Vandi L et al.: Symptom- and Laboratory-Based Ebola Risk Scores to Differentiate Likely Ebola Infections. Emerg Infect Dis 2017, 23: 1792–1799. pmid:29047428

23. WHO Africa: Technical Guidelines for Integrated Disease Surveillance and Response in the African Region: Third edn. Geneva https://www.afro.who.int/publications/technical-guidelines-integrated-disease-surveillance-and-response-african-region-third: 2019.

24. Spiegelhalter DJ.: Statistical methodology for evaluating gastrointestinal symptoms. Clin Gastroenterol 1985, 14: 489–515. pmid:4064353

25. Berkley JA, Ross A, Mwangi I, Osier FHA, Mohamed M, Shebbe M et al.: Prognostic indicators of early and late death in children admitted to district hospital in Kenya: Cohort study. BMJ 2003, 326: 361–366. pmid:12586667

26. Chan SF, Deeks JJ, Macaskill P, Irwig L: Three methods to construct predictive models using logistic regression and likelihood ratios to facilitate adjustment for pretest probability give similar results. J Clin Epidemiol 2008, 61: 52–63. Epub 2007 Jul 6. pmid:18083462

27. Lasko TA, Bhagwat JG, Zou KH, Ohno-Machado L.: The use of receiver operating characteristic curves in biomedical informatics. J Biomed Inform 2005, 38: 404–415. Epub 2005 Apr 2. pmid:16198999

28. Hanley JA, McNeil BJ: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983, 148: 839–843. pmid:6878708

29. Vickers AJ, Elkin EB: Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 2006, 26: 565–574. pmid:17099194

30. Zhang Zhongheng, Rousson Valentin, Lee Wen-Chung, Ferdynus Cyril, Chen Mingyu, Qian Xin et al.: Decision curve analysis: a technical note. Ann Transl Med 2018, 15: 308. pmid:30211196

31. Bah EI, Lamah MC, Fletcher T, Jacob ST, Brett-Major DM, Sall AA et al.: Clinical Presentation of Patients with Ebola Virus Disease in Conakry, Guinea. N Engl J Med 2014. pmid:25372658

32. Bwaka MA, Bonnet MJ, Calain P, Colebunders R, De Roo A, Guimard Y et al.: Ebola hemorrhagic fever in Kikwit, Democratic Republic of the Congo: clinical observations in 103 patients. J Infect Dis 1999, 179 Suppl 1: S1–S7. pmid:9988155

33. Gao X, Rong G, Lu Y, Duan X, Duan H: Symptoms closely associated with Ebola virus disease. Lancet Infect Dis 2016, 16: 18–19. pmid:26738827

34. Lado M, Walker NF, Baker P, Haroon S, Brown CS, Youkee D et al.: Clinical features of patients isolated for suspected Ebola virus disease at Connaught Hospital, Freetown, Sierra Leone: a retrospective cohort study. Lancet Infect Dis 2015, 15: 1024–1033. Epub 2015 Jul 23. pmid:26213248

35. Brainard J, Hooper L, Pond K, Edmunds K, Hunter PR: Risk factors for transmission of Ebola or Marburg virus disease: a systematic review and meta-analysis. Int J Epidemiol 2016, 45: 102–116. Epub 2015 Nov 20. pmid:26589246

36. Fitzgerald Felicity, Wing Kevin, Naveed Asad, Gbessay Musa, Ross J.C.G., Francesco Checchiet al.: Development of a Pediatric Ebola Predictive Score, Sierra Leone. Emerg Infect Dis 2018, 24: 311–319. pmid:29350145

37. D’Agostino RB Sr, Pencina MJ, Massaro JM, Coady S: Cardiovascular Disease Risk Assessment: Insights from Framingham. Glob Heart 2013, 8: 11–23. pmid:23750335