Area under the ROC Curve has the most consistent evaluation for binary classification

Abstract

The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables and the sample size are kept constant. Analyzing 156 data scenarios, 18 model evaluation metrics and five commonly used machine learning models as well as a naive random guess model, I find that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, Area Under the ROC Curve (AUC) which takes all decision thresholds into account when evaluating models has the smallest variance in evaluating individual models and smallest variance in ranking of a set of models. A close threshold analysis using all possible thresholds for all metrics further supports the hypothesis that considering all decision thresholds helps reduce the variance in model evaluation with respect to prevalence change in data. The results have significant implications for model evaluation and model selection in binary classification tasks.

1 Introduction

The proper use of evaluation metrics for binary classification tasks have received considerable scientific attention in recent years especially for imbalanced data [1–12]. In particular, Area under the Receiver Operating Characteristic (ROC) Curve (AUC) as the most widely used metric in model evaluation has received the most scholarly attention.

For binary classification, the Receiver Operating Characteristic (ROC) Curve considers all decision thresholds and plots the corresponding false positive rate (FPR) and true positive rate (TPR) on the x-axis and y-axis respectively. The Area under the ROC Curve (AUC) essentially depicts the probability that a randomly selected positive case ranks higher than a randomly selected negative case, and has one-to-one equivalence to the Wilcoxon statistic [13–16]. In comparison with accuracy and other metrics, AUC has shown a number of advantages: increased sensitivity in Analysis of Variance (ANOVA) tests; decreasing standard error with increasing sample size; threshold-independent and invariant to priori class probabilities [17]. As a result, AUC as a evaluation metric has a dedicated history not only for model evaluation but also for medical screening test such as early cancer detection [18], bankruptcy prediction [19], software defect prediction [20], and many other purposes [21–23], albeit the use of AUC has faced more scrutiny recently [24–27].

Critics of AUC have pointed out at least two issues associated with its use: it does not account for prevalence and does not work well with imbalanced data where either the positive or the negative class overwhelmingly dominates the AUC score [28–30]; not all regions of AUC are meaningful or represent realistic evaluation outcomes [31–33]. As a result, a number of alternatives have been proposed. For example, Matthew’s correlation coefficient (MCC) has been proposed as a better alternative because it has high values only when a model predicts well on both positive and negative classes [7, 8, 12, 34].

Additional proposed alternatives to the standard AUC include: average positive predictive value (AP) that varies with respect to prevalence in the data and places more emphasis on early positives than does AUC [35]; uniform AUC to make AUC not dependent on the range of predicted values [36]; not proper ROC Curve to identify hidden subclass in microarray data [37]; restricted ROC Curves to separate informative from noninformative values [38]; concentrated AUC to address the early retrieval problem [39]; partial AUC metrics [40, 41]; delimited region of interest in the ROC space [32, 33]; total operating characteristic [42]; expected cost representation [43]; precision-recall curve to better suit low-prevalence data [30, 44]; general performance score to combine several user-selected metrics [9, 45], etc.

In particular, one research [12] advises “MCC as the standard binary classification metric for any scientific study in any scientific field” because it is the only metric that has high values when true positive rate, true negative rate, positive predictive value and negative predictive value are all high. However, MCC, as the Pearson correlation coefficient of two binary variables, is a quite conservative evaluation metric and requires the two sequences of predicted and actual class labels to highly align with each other in order to have high values. For example, suppose the true class labels of 10 observations are (1, 0, 0, 0, 0, 1, 1, 0, 0, 1), and the predicted class labels are (1, 1, 0, 0, 0, 1, 1, 0, 0, 1). While the classification misses only one observation, the corresponding MCC is 0.82. If the predicted class labels are (0, 0, 0, 0, 0, 1, 1, 0, 0, 0) and the classification misses only two observations, MCC is only 0.61. If the predicted class labels are (0, 0, 1, 0, 0, 0, 1, 0, 0, 1) and the classification misses three observations, MCC is only 0.36 even when there is 70 percent accuracy. For MCC, not only does the number of correct classifications matter, but the order in which the correct and incorrect predictions happen matters significantly.

An existing study [6] also investigates the behavior of MCC by letting either true positive rate or true negative rate change while holding the other one constant. The results show that MCC is disproportionately concentrated in the low value range of -0.1 and 0.1 when data is highly class imbalanced even when the overall accuracy is high, an unsurprising result given the conservative nature of MCC. In addition, the study results show that classification accuracy involving positive predictive value and negative predictive value behaves undesirably for imbalanced data because of their formulation. As has been emphasized already [15], positive predictive value and negative predictive value are not properties of model performance or diagnostic test itself. They are more of an additional interpretation of a given test or classification result rather than a measure of intrinsic model performance. Therefore, a well-performing classification model does not need to always have high positive predictive and negative predictive values. Existing research also shows that AUC is more discriminating than MCC for both balanced and imbalanced data sets [46].

In general, there are at least three limitations associated with existing research on model evaluation for binary classification and the use of AUC in particular. One is that existing research mostly focuses on model evaluation on one data set and fails to consider the consistency of model evaluation across data sets of varying prevalence levels (the proportion of positive cases). A related issue is that existing research does not sufficiently separate out influence of the model being trained and sample size when assessing the proper use of evaluation metrics. Moreover, for metrics other than AUC, a close examination of the effect of decision threshold used and the number of decision thresholds used on evaluation metrics still does not exist.

This paper revisits model evaluation metrics for binary classification, and studies to what extent different metrics give consistent evaluation of model performance over data of varying prevalence while the relationships between different variables and sample size are kept constant. We measure consistency of model evaluation using variance of metric values for individual models and variance of rankings for a set of models across data of different prevalence. Using statistical simulation, we conduct analysis for 18 evaluation metrics and five commonly used machine learning models as well as a naive random guess model. The results show that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, AUC has the smallest variance with respect to prevalence and the smallest variance in its ranking of different models. We further hypothesize that the reason AUC is able to achieve more consistent evaluation is because it considers all decision thresholds, and a threshold analysis considering all possible thresholds supports this claim.

Consistency of model evaluation across data of different prevalence is important because repeated evaluations of same model across data of different prevalene are common. And we don’t want model evaluation results change simply because prevalence in the data changes and the metric used has a close to monotonic relationship with prevalence. If that is the case, evaluation metric values become artifact of the metrics being used rather than measures of intrinsic model performance. As a result, practitioners of model evaluation are left unsure about the true predictive power of the model being used. This can have undesirable consequences for model evaluation and model selection in various domains [47]. Therefore, it is essential that model evaluation results are consistent with prevalence change in the data.

The novelty of this paper resides in its focus on an underappreciated question in model evaluation and its use of statistical simulation with a real-world data set, which generates a much richer set of data scenarios compared with existing research (having only a small number of cases). Thus, we show a more complete picture of the relationships between data, evaluation metric and model while keeping sample size and the relationships between variables constant. In addition, a much larger set of evaluation metrics are considered here than existing research and an explicit link is made between prevalence, evaluation of individual models, and ranking of a set of models by different metrics. Moreover, we perform an extensive analysis of the effect of decision threshold on model evaluation outcomes and show that using a larger number of thresholds can improve model evaluation consistency.

This paper is organized as follows, in section 2, we first provide background information and notations on model evaluation metrics for binary classification. In section 3, we look at a prominent case of predicting crime recidivism and present a simulation study that examines how different confusion matrix metrics evaluate models differently under data with different prevalence. In section 4, we show the analysis results. And in section 5, we offer discussions and conclusions.

2 Methods

2.1 Background and basic notations

Evaluation metrics are statistics used to evaluate model performance especially for out of sample prediction tasks as is commonly the case for machine learning model applications. Below we compare several evaluation metrics and look at how accurate and reliable they are at capturing the model prediction performance for data of different prevalence level. We focus on binary classification tasks in this study. First, let us introduce a few concepts. For every binary prediction/classification task, the result could fall into four different categories:

* true positives (TP), positive cases correctly predicted as positive;

* false negatives (FN), positive cases incorrectly predicted as negative;

* true negatives (TN), negative cases correctly predicted as negative;

* false positives (FP), negative cases incorrectly predicted as positive.

The partition of these four categories can be represented in a 2 × 2 tabular format called confusion matrix: , TP + FN + FP + TN = n.

The first set of prediction performance metrics we look at are:

And a number of metrics that combine TPR and TNR in certain ways:

F-scores (we use β = 2, 0.5, two commonly used values) and Matthew’s correlation coefficient (MCC):

As will become clearer in subsequent analysis, each of these metrics emphasizes different aspects of the performance of a binary classifier. For example, metrics such as the F-scores, weigh heavily on TPR and will be most informative if the particular classification task cares most of the ability to detect true positive cases while the opposite is true for metrics that weigh heavily on TNR. A number of metrics including balanced accuracy, bookmaker informedness, geometric mean and MCC, in their formulations, try to balance model performance on both classes of data and are most useful when the purpose of model evaluation is to assess accuracy on both classes of data equally.

In addition, some metrics originate from specific application domains and can have their particular use cases. In particular, bookmaker informedness in the binary case measures how much a test or model is better than random guessing and is primarily used in medical diagnostic test. Markedness is a measure that comes from linguistics, and measures the extent to which a unit is different from another unit as abnormal or divergent. Jaccard index, also called intersection over union, is a measure of similarity or diversity between two sets of values, and a common metric for image object recognition in computer vision. Diagnostic odds ratio measures the effectiveness of medical diagnostic test, and is the ratio between the odds for a test result being positive if the unit has disease and the odds for a test result being positive if the unit has no disease. Fowlkes-Malloows Index is the square root of the product of TPR and PPV, and measures the similarity between two clustering/classification outcomes. Cohen’s Kappa, a measure of inter-rater reliability from psychology, measures the agreement between two raters each classifying N items into C categories. F1 score is the harmonic mean of TPR and PPV while the F_β scores treat TPR as β times more important than PPV. The F-scores are commonly used in information retrieval and named entity recognition and word segmentation tasks in natural language processing.

2.2 Alternative derivation of confusion matrix metrics

As mentioned in the literature [4, 48], all entries of a confusion matrix and confusion matrix metrics can be derived from the four quantities of sample size n, prevalence, TPR and TNR. To show that this is indeed the case, below we redefine the confusion matrix metrics introduced above in terms of these four quantities excluding metrics that are already expressed in these four quantities. We use ϕ to represent prevalence, and .(1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

3 Case study: Crime recidivism

First, we look at a real-world case where the outcome is more or less class balanced. We leverage the Broward County data set [49, 50] that consists of individuals arrested in Broward County, Florida between 2013 and 2014. The sample size is 6214, with 2775 positive cases, individuals who reoffended and 3439 negatives, individuals who did not reoffend. The predictors for the classifier include seven features of the defendants: gender, age, number of juvenile misdemeanors, number of juvenile felonies, number of prior (nonjuvenile) crimes, crime degree, and crime charge [49]. We train five machine learning models and see how the different evaluation metrics listed above capture the prediction performance of these models: logistic regression (GLM), random forest (RF), k-nearest neighbors (KNN), linear discriminant analysis (LDA), gradient boosting machine (GBM). In addition, we make a random guess model that randomly assign class labels to observations based on prevalence of the test set. For example, for a test set with 60 percent positive cases, random guess will randomly assign the positive label to 60 percent cases and negative label to the other 40 percent (the predicted probability of positive class is a random number between 0.51 and 0.99 while the predicted probability of negative class is a random number between 0.01 and 0.49). The sample data are randomly split into 80 percent training data and 20 percent test data. 10-fold cross-validation is used with the model training process. In short, 10-fold cross-validation splits the training data into 10 equally-sized random samples, iteratively uses 9 samples to estimate a model and the remaining one sample to validate the model, and choose the model that has the best performance across the 10 random samples. This procedure helps to address over-fitting and find the model that has best predictive power. Table 1 lists the rankings of the different models by the different metrics.

[Figure omitted. See PDF.]

Table 3 in Appendix lists the corresponding confusion matrix entries for the model predictions (first four rows) and values of the 18 evaluation metrics. The results indicate that all evaluation metrics rank naive random guess as the worst performing model. GBM is most frequently ranked as the best model, random forest is most frequently ranked as the second best. GLM is most frequently ranked as either third or fourth, and KNN is most frequently ranked as fifth but occasionally as third or fourth. LDA is ranked between third, fourth, and fifth or 1.5. Ties in ranking are broken by averages. As TP, FN, TN, FP, TPR, TNR concern positive or negative class only for either actual or predicted labels, it is no surprise that rankings are reversed between metrics focusing on TPR only and those focusing on TNR only. AUC calculations are based on all possible thresholds while all other metric calculations are based on the 0.5 cutoff. These patterns can also be observed from the corresponding metric values in Table 3.

While this example shows that different metrics can converge in their evaluation of classification outcomes, this is just one single case. And more importantly, this is a case where the positive and negative classes in the data are more or less balanced. To be precise, the corresponding prevalence is 0.452. When data becomes more imbalanced and either the positive or negative class dominates the data, different metrics’ evaluation behavior can be very different. To obtain a more complete picture of the dynamics between prevalence level and model evaluation by different metrics, we use statistical simulation to create a series of data sets with varying prevalence. The results below show how prevalence change affects TPR and TNR, which in turn affect different metrics. The sample size n is kept constant such that we are able to separate out the influence of sample size.

3.1 Simulation study

The following simulation is done with the Broward County data set [49, 50] used above. The original data set has 2775 positive cases and 3439 negative cases, which is equal to a prevalence of 0.452. Here, prevalence refers to prevalence of the sample data as a whole. However, all the above definitions of evaluation metrics in 2.1 refer to prevalence of the test set. In this study, as the test sets are randomly sampled from the sample data, prevalence of the test sets are the same as the sample data (differences are negligible). Prevalence of the simulation results presented below is for the test sets. To simulate the dynamics associated with changing prevalence, we apply up-sampling and down-sampling at the same time but for different classes of the data. First, we iteratively drop 30 randomly sampled positive cases while simultaneously adding 30 randomly sampled negative cases, thus incrementally reducing prevalence while keeping the sample size n constant. In total, we run 76 such iterations, which along with the original data sample give us 77 data sets. The 77th data sample has 495 positive cases and 5719 negative cases, equaling prevalence of about 0.08. We stop at this iteration because TPR decreases to almost 0 while TNR increases to almost 1. Next, we do the opposite. From the original data sample, we iteratively drop 30 randomly sampled negative cases while simultaneously adding 30 randomly sampled positive cases, thus incrementally increasing prevalence while keeping the sample size n constant. In this second phase, we run 79 iterations and stop when TPR increases to almost 1 while TNR decreases to almost 0. The 79th data sample has 5145 positive cases and 1069 negative cases, equaling prevalence of about 0.83.

As units are randomly dropped from or added to the positive and negative classes, the simulation steps above are able to keep the relationships among the predictors and between the predictors and the outcome largely unchanged. The bivariate correlations among all 8 variables have only minimal changes. The correlation heat maps shown in the S3 Fig showcase the bivariate correlations among all the variables for the original data as well as 79 simulated data samples (every other ones selected from all data samples). It can be observed that the correlations among all variables largely stay the same across all the simulated data samples (The other 76 simulated data samples have been verified to show the same pattern).

4 Results

4.1 Simulation results: Evaluation metrics

Figs 1–3 show the relationships between changing prevalence and various model evaluation metrics. The curves are LOESS-smoothed ones.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

First, we look at TPR, TNR, PPV and NPV. Across the six models, decreasing prevalence corresponds with decreasing TPR and increasing TNR. And the two rates intersect at around prevalence 0.5. Similarly, in general, PPV decreases and NPV increases as prevalence decreases. However, for all models except random guess, there are significant fluctuations in PPV when prevalence is low, while NPV shows most fluctuation when prevalence is high. The two curves for PPV and NPV again intersect at around prevalence 0.5, indicating that 0.5 is a critical point for PPV and NPV just as it is for TPR and TNR.

Mathematically, from expressions shown in section 2.2, we know that . As rate of change in TPR is similar to the rate of change in TNR and the rate of change in prevalence across the whole range of prevalence, and as prevalence decreases, (1 − TNR) increases, increases, increases, therefore, we know that increases, and PPV decreases.

Similarly, , and as prevalence decreases, , and decreases, and NPV increases. The observed patterns of PPV and NPV in Fig 1 largely correspond with their mathematically expected behavior. Noticeably, for changing prevalence, TPR and TNR are more stable than PPV and NPV especially at extreme prevalence. From corresponding expressions in section 2.1, we know that denominators of both TPR and TNR are known before predictions are made, therefore, only numerators of TPR and TNR are changing as prevalence changes. For PPV and NPV, both denominator and numerator are only known after predictions are made. Therefore, for PPV and NPV, both denominator and numerator are changing as prevalence changes. In other words, PPV and NPV vary more because of a particular prediction task while TPR and TNR vary less and are more affected by the independent influence of prevalence, a key distinction not often emphasized in existing literature.

The three F-scores show a similar downward trend as prevalence decreases. F1 score shows the highest steepness, F_β score with β = 2 next, and F_β score with β = 0.5 shows the least steepness. Apparently, F-scores have a monotonically increasing relationship with prevalence, which is verified by a simple mathematical derivation for F1 score shown in Appendix B.

Similarly, both Falkes-Mallows index and Jaccard index show a monotonically increasing relationship with prevalence. From Eqs 6 and 8 in section 2.2, we know that both metrics are close to PPV in form, therefore, it is unsurprising that both metrics show a similarly downtown trend as PPV though with more steepness.

It can be observed that accuracy has a convex curve with respect to prevalence, that is accuracy has lower values when prevalence is around 0.5 while it has higher values when prevalence goes to either the upper or lower extremes. Above prevalence 0.5, accuracy is dominated by the positive class and decreasing TPR while below 0.5, accuracy is dominated by the negative class and increasing TNR. Therefore, accuracy is dominated by the majority class. Even random guess shows the same behavior.

On the contrary, both balanced accuracy and bookmaker informedness show a concave curve with respect to prevalence. That is both metrics have higher values when class is more balanced and lower values when class is very imbalanced. This can be understood from their expression as certain form of summation of TPR and TNR. Naive random guess is constant around 0.5 and 0, median of the range for balanced accuracy and bookmaker informedness respectively, indicating the amount of prediction accuracy by random chance.

Cohen’s Kappa shows an concave curve with respect to prevalence. That is it is higher when class is more balanced and lower when class is more imbalanced. From Eq 9, we know that , and expAccuracy is dominated by decreasing TPR when prevalence is above 0.5 and by increasing TNR when prevalence is below 0.5, thus leading to the observed pattern for Kappa. Random guess is constant around 0, median of the range for Kappa indicating the amount of prediction accuracy by random chance.

Similarly, the geometric mean shows a concave curve with respect to prevalence, which can be understood from its expression . That is it values high TPR and high TNR at the same time. The same applies to MCC, which is essentially the Pearson’s correlation coefficient for two binary variables. And to have high values, MCC requires the two sequence of actual and predicted class labels to closely align with each other. From , we know that MCC also tries to achieve high TPR and high TNR at the same time. Random guess again is constant around 0, median of the range for MCC.

As Area under the ROC Curve (AUC) assesses performance of a classifier at all possible thresholds, it cannot be constructed from a single confusion matrix based on one classification threshold and cannot have an exact comparison with other confusion matrix evaluation metrics calculated with the 0.5 probability cutoff. However, AUC shows a most obvious pattern in Fig 3. AUC is mostly flat regardless of prevalence change except for minor fluctuations at extremely low prevalence.

Results for Markedness and DOR are shown in the S1 Fig. Markedness does not show an obvious pattern with prevalence across the six models. As it is a form of sum of PPV and NPV, and both PPV and NPV fluctuate a lot at extreme prevalence, it is unsurprising to see the large fluctuation in its trends including for random guess. DOR shows a somewhat upward trend at low prevalence. However, for most prevalence levels, it has a flat trend, and there is a lot of fluctuations at low prevalence. This can be understood from its alternative expression , where the diverging trends of TPR and TNR compete with each other.

Here is a summary of the above results. As TPR, TNR, PPV and NPV mostly concern prediction accuracy for one class only, they have a close to monotonic relationship with prevalence in either the positive or negative direction though PPV and NPV fluctuate much more. And a number of metrics appear to have a monotonically increasing relationship with prevalence including the three F-scores, Fowlkes-Mallows index and Jaccardindex. Meanwhile, balanced accuracy, bookmaker informedness, Cohen’s Kappa, Matthew’s correlation coefficient and geometric mean all have a concave curve with respect to decreasing prevalence though geometric mean has a much larger range of values. Accuracy has a convex curve with respect to decreasing prevalence. AUC is prevalence-independent and shows flat trends across the different models. Markedness and DOR do not have strong patterns in regard to prevalence.

4.2 Simulation results: Comparisons of models

Figs 4 and 5 show how the 6 models are evaluated by such metrics. For presentation convenience, we organize the 18 metrics into the upper, middle and lower panes. The results indicate that evaluation metrics that are more influenced by prevalence as shown in Figs 1–3 also do a poorer job differentiating between different models. As an example, we cannot visually determine from the three F scores which one or two models perform better among the 5 trained machine learning models, the only thing we can say is that these 5 models are better than random guess. This is also true for Fowlkes Mallows index and Jaccard index. On the contrary, for both MCC and AUC, it is clear to observe that random forest and Gradient Boosted Machine perform better than the other three models. This is also true for balanced accuracy, bookmaker informedness and Cohen’s Kappa. The general pattern is that metrics that are having a close to monotonic relationship with prevalence do a poorer job distinguishing among different models while metrics that are more prevalence-independent do a better job.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

We also use statistical tests to detect whether the variance in model evaluation between metrics that are more influenced by prevalence (the three F scores, Jaccard Index, TNR, TPR, and Fowlkes Mallows index) and those that are less are statistically significant. F-test, Bartlett test and Levene test results all show highly statistically significant p-values (we test the difference in variance for the interaction between metric type and model in Bartlett and Levene tests). That is the difference in variance of model evaluation with respect to changing prevalence is statistically significant between these two types of metrics.

4.3 Simulation results: Ranking of different models

Here we provide direct results for how different metrics rank the 6 different models across data of varying prevalence, and more specifically we look at the variance of rankings for each model. Table 2 presents the variance values while Figs 6 and 7 show the corresponding actual rankings for each model. The variance of rankings for each metric and model combination is obtained by ranking the 6 models for each of the 156 data sets and calculating the corresponding variance, so the variance is with respect to prevalence.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

Corresponding to results shown in section 4.1, it is clear to observe that metrics that are more affected by prevalence also show larger variance in ranking of different models. This type of metrics includes TPR, TNR, PPV, NPV, the three F scores, Fowlkes-Mallows index, and Jaccard index. Their variances as shown in Table 2 are larger compared with metrics that are less influenced by prevalence. The latter group of metrics includes BA, BI, Kappa, MCC, AUC, Markedness and DOR. Variances for the first group of metrics are almost always above 1, sometimes above 2 or 3 while variances for the second group are mostly below 1. Most notably, AUC has the smallest variance in ranking of all models as it is mostly prevalence-independent. MCC for the majority of models has the second smallest variance. The geometric mean also tries to balance prediction accuracy for both positive and negative classes, yet it does have a relatively large range of values with respect to prevalence as shown in Fig 3. Correspondingly, it has relatively large variance in ranking of different models. Figs 6 and 7 show the same patterns for actual model rankings: accuracy, balanced accuracy, bookmaker informedness, geometric mean, Kappa, MCC and AUC in Fig 7 are shown to fluctuate less than TPR, TNR, PPV, NPV, the F-scores, Fowlkes-Mallows index, and Jaccard index in Fig 6.

We also perform statistical tests to detect whether the variance in model ranking between metrics that are more influenced by prevalence (the three F-scores, Jaccard Index, TNR, TPR, and Fowlkes Mallows index) and those that are less are statistically significant. F-test, Bartlett test and Levene test results all show highly statistically significant p-values (we test the difference in ranking variance for the interaction between metric type and model in Bartlett and Levene tests). That is the difference in variance of model rankings with respect to prevalence change is statistically significant between the two types of metrics.

4.4 Threshold analysis

Figs 8 and 9 show the model evaluation results using different decision thresholds for data sets of different prevalence. The legend shows the upper bounds of five equally spaced intervals of the corresponding metric values with the last value being the maximum. Decision threshold refers to predicted probability for the positive class. This figure shows the results for the random forest model, results for other models show similar patterns and are included in the S2 Fig. As can be clearly observed, metrics that are more influenced by prevalence including TPR, TNR, the three F-scores, Fowlkes Mallows Index and Jaccard Index show strong correlation with prevalence. However, this relationship is significantly moderated by the decision threshold. For example, TPR has the highest value between 0.9 and 1 when prevalence is high and threshold is low while as expected the opposite is true for TNR. The three F-scores, Fowlkes Mallows Index and Jaccard Index show similar behavior as TPR. PPV shows high values when both prevalence and decision threshold are high and low values when both are low. The opposite is true for NPV.

[Figure omitted. See PDF.]

[Figure omitted. See PDF.]

For metrics that try to balance accuracy for both positive and negative classes including balanced accuracy, bookmaker informedness, geometric mean, MCC, the high values reside more or less along the diagonal line of the corresponding graph while low values are away from these diagonal lines. Accuracy show high values at either high or low prevalence and low values when prevalence is close to 0.5. However, it does have low values when the decision threshold is either close to 0 or 1 and the corresponding prevalence is low or high. As expected, AUC is invariant to decision threshold change as it always considers all possible thresholds already. And it is largely invariant to prevalence change as the range of its values is between 0.7 and 0.79, a narrower interval compared with all other metrics. Cohen’s kappa has high values at low prevalence and low values at high prevalence corresponding to what is shown in Fig 3 while there is no systematic pattern for DOR and markedness.

These patterns correspond to what is shown in Figs 1–3, and indicate that changing from one threshold to another does have an effect on actual metric values but does not fundamentally alter the relationship between a particular metric and prevalence. In the following, we show that increasing the number of thresholds used in model evaluation does reduce metrics’ variance with respect to prevalence. This provides support for the claim that the reason why AUC is able to offer more consistent model evaluation (with respect to prevalence) is because it considers all possible decision thresholds in its evaluation.

Fig 10 shows the variance of metrics that vary most with prevalence as indicated by results shown in Figs 1–3. Here, we start with a single threshold, that is 0.5, gradually add more thresholds until all possible thresholds are added. We take the average of model evaluation metric values each time additional thresholds are added into the calculation. The thresholds are added incrementally based on their distance from 0.5. Those closest to 0.5 are added first, thus we gradually increase the span of thresholds used. While there are fluctuations in metric values for the different models, the overall pattern is clearly that the more thresholds we consider, the smaller variance we have in regard to prevalence change.

[Figure omitted. See PDF.]

Fig 11 shows the OLS (Ordinary Least Squares) regression results of number of thresholds used on variance of evaluation metrics. Regressing the variance of evaluation metric values on number of thresholds used in evaluation, the results show that out of 108 possible combinations of metrics and models (18 × 6), 98 has a negative parameter estimate that is statistically significant (p-value less than 0.05). Compared with other metrics, DOR has a much larger negative parameter estimate that is always highly statistically significant, and is not shown here for better visualization of the other metrics. The 10 cases (in red) that either have very small non-negative parameter estimate or p-value larger than 0.05 include 8 for PPV and NPV (likely because these two metrics fluctuate so much at extreme prevalence), 1 for Cohen’s Kappa (only mildly influenced by prevalence), and 1 for AUC (least influenced by prevalence and already considering all possible thresholds). As expected the parameter estimates for AUC is always very close to zero and is not exactly zero only because the regression includes data for all models. The size of the parameter estimates is small because the maximum number of thresholds considered is above 1200 as shown in Fig 10, thus effect size of adding just one more threshold is small yet it is highly statistically significant.

[Figure omitted. See PDF.]

5 Discussions and conclusion

5.1 Discussions

While model evaluation for binary classification is a longstanding question with a large literature, existing research suffers from at least three shortcomings. Firstly, there is no clear attempt to establish a clean relationship between metric and prevalence level while separating out the influence of model and sample size. This paper does what is missing in the literature by simultaneously down-sampling and up-sampling different classes of data to keep both relationships among all relevant variables and sample size constant. Secondly, existing research mostly focuses on evaluating model performance on a single data set and thus ignores the question that different metrics can have systematically different behavior with respect to prevalence change. This paper addresses this question by looking at the variance of evaluation metrics across 156 data sets for 5 machine learning models as well as a naive random guess model. Lastly, existing literature still lacks a threshold analysis that can show how different metric values change as both decision threshold and prevalence change. This paper does exactly that by using all possible thresholds both individually and collectively for each metric.

The results indicate that evaluation metrics can largely be categorized into three types: monotonically influenced by prevalence such as TPR, TNR, the F-scores, Fowlkes Mallows Index and Jaccard Index; having a close to parabolic-shaped relationship with prevalence such as balanced accuracy, bookmaker informedness, geometric mean and MCC; not having a systematic relationship with prevalence such as markedness and DOR. AUC is the only metric that is mostly prevalence-independent. Moreover, metrics that are more influenced by prevalence also have larger variance both in evaluation of individual models and in ranking of a set of models. In other words, metrics that are less influenced by prevalence offer more consistent model evaluation (the specific model being trained does have an influence, overall GBM and Random forest have smaller variance in rankings compared with GLM, KNN and LDA). As expected, random guess is most frequently being ranked as the least performing model and evaluations of almost all metrics overlap on the rank 6th line in Figs 6 and 7. In particular, AUC offers the most consistent model evaluation, and this is not a result by definition. The predicted probabilities for the positive and negative classes are influenced by prevalence. However, the fact that AUC evaluates model using all possible thresholds effectively offsets the influence of changing prevalence. We show evidence for this claim through a threshold analysis that includes all possible thresholds for all the data samples, models and metrics. The results show that the more decision thresholds we consider, the most consistent model evaluations we obtain across data of different prevalence.

An implication for researchers and practitioners of model evaluation is that awareness ought to be raised for the underappreciated issue of consistency of model evaluation with respect to prevalence change. Ensuring consistency of model evaluation is important because repeated evaluation of the same model across different data samples is common. And prevalence in data collected across different time periods can change. As evaluators of model performance, we don’t want metric values change just because prevalence in the data change. A much more desirable evaluation outcome is that we obtain consistent evaluation results of a model across data of different prevalence. This ensures that we are measuring the intrinsic performance of the model itself. As shown in Fig 5, more consistent evaluation metrics also do a better job differentiating between different models.

As a practical guide for action, if practitioners are interested in the complete picture of model performance for a binary classification task, then they need to look at all four basic quantities (TP, TN, FN, FP). If practitioners are interested in one single metric to summarize the model performance, then the particular goal of the specific evaluation task must be taken into consideration as no single metric can summarize the four basic quantities satisfactorily in general. If the goal of evaluation is to obtain consistent rating of a single model or consistent ranking of a set of models across different data scenarios, then AUC should be the metric to be used.

One limitation of this study is that the simulation exercises performed above focuses on one data set which has a more or less balanced outcome variable, similar analysis with other data sets with a more imbalanced outcome can be performed to further confirm the current findings and we expect similar patterns to hold. Moreover, in training the five models above, seven features are used as predictor variables. In other settings of applied machine learning, the number of predictor variables can be much larger and that can be an additional source of variation to a prediction task. Further analysis of data with a much larger set of feature variables can potentially bring additional insights to the complex interplay of data, model and model evaluation metrics.

Regarding the implications for different domains or types of data, no definite answer can be given at this point. However, we expect similar results to hold in settings with a much large sample size and a lot more feature variables. The number of feature variables do not have an effect on evaluation metric values as long as the relationships between the feature variables and the outcome variable are kept constant and the same model is used. Likewise, sample size alone should not have an effect on evaluation metrics as sample size is not in the formulation of almost all metrics shown in section 2.1. When the number of variables is higher than sample size, we are in a high-dimensional setting and things can become different as a different modeling strategy is needed.

Another limitation of this study is that we only consider binary classification tasks, similar analysis for data sets with multi-class outcome can be conducted. For a multi-class classification question, often we need to binarize the outcome in certain ways: either adopting a one-versus-rest approach where one class is compared against all the other classes or taking a one-versus-one approach where comparison is made for every pairwise class combination. And the formulation for most metrics can be easily extended to a multi-class scenario. This will enable generalizing the current study to the multi-class setting. For a one-versus-rest comparison, we expect similar results to hold, while for one-versus-one pairwise comparison, we expect similar results to hold conditional on the relative size of other classes not in the comparison.

5.2 Conclusion

In conclusion, this study provides new analytical results on machine learning model evaluation for binary classification and more specifically for the consistency of model evaluation metrics with respect to prevalence change while holding the influence of model and data constant. The analysis performed covers a distribution of data scenarios, from an overwhelming positive sample to an overwhelming negative sample and everything in between. Across this full span of data scenarios and five commonly used machine learning models as well as naive random guess, clear patterns are obtained for 18 evaluation metrics.

It is hoped that the results presented above can provide a different angle on thinking about the complex relationships between data, model and evaluation metrics, and can spur further research into developing best practices for machine learning model evaluation in various scientific domains. As has been shown above, statistical simulation can be an effective tool for scientific discovery even in an age of big data. Only analyzing observed empirical data is not enough for full-scale scientific advances. Theory-driven simulation analysis can provide new findings that are not available otherwise.

6 Appendices

6.1 Prediction results for crime recidivism

Table 3 shows the crime recidivism prediction results for 5 machine learning models (logistic regression (GLM), random forest (RF), k-nearest neighbors (KNN), linear discriminant analysis (LDA), gradient boosting machine (GBM)) as well as naive random guess evaluated with 18 metrics. The data set [50] consists of individuals arrested in Broward County, Florida between 2013 and 2014. The sample size is 6214, with 2775 positive cases, individuals who reoffended and 3439 negatives, individuals who did not reoffend. The predictors for the classifiers include seven features of the defendants: gender, age, number of juvenile misdemeanors, number of juvenile felonies, number of prior (nonjuvenile) crimes, crime degree, and crime charge.

6.2 F1 score as a result of prevalence change when only sample size n is constant

From the alternative definition of F1 score in Eq 2, we know that , in addition, , increases, and (1 − TPR) increases. And across the whole spectrum of prevalence, the rate of change in TPR is similar to the rate of change in both TNR and prevalence itself, and as only (1 − TNR) goes down while all other terms in the denominator of F1 score goes up, we have that increases and F1 score decreases as prevalence level decreases.

[Figure omitted. See PDF.]

Supporting information

S1 Fig. Model evaluation for data of different prevalence using markedness and diagnostic odds ratio.

https://doi.org/10.1371/journal.pone.0316019.s001

S2 Fig. Model evaluation using all thresholds for data of different prevalence for models other than random forest.

https://doi.org/10.1371/journal.pone.0316019.s002

S3 Fig. Correlation heat maps showing bivariate correlations among variables for original data sample as well as 79 simulated data samples (every other ones selected from all the data samples).

https://doi.org/10.1371/journal.pone.0316019.s003

Acknowledgments

I thank Wendy K. Tam Cho and three anonymous reviewers for very helpful comments on an earlier version of this paper.

References

1. 1. Powers DMW. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. ArXiv. 2011;abs/2010.16061.

2. 2. Garcia V SSJ, R AM. On the suitability of numerical performance measures for class imbalance problems. Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods, Algarve, Portugal. 2012; p. 310–313.

3. 3. Lever J, Krzywinski M, Altman N. Classification evaluation. Nature Methods. 2016;13(8):603–604.

* View Article

* Google Scholar

4. 4. Luque A, Carrasco A, Martín A, De Las Heras A. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition. 2019;91:216–231.

* View Article

* Google Scholar

5. 5. Jadhav AS. A novel weighted TPR-TNR measure to assess performance of the classifiers. Expert Systems with Applications. 2020;152:113391.

* View Article

* Google Scholar

6. 6. Zhu Q. On the performance of Matthews correlation coefficient (MCC) for imbalanced dataset. Pattern Recognition Letters. 2020;136:71–80.

* View Article

* Google Scholar

7. 7. Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics. 2020;21(1):6. pmid:31898477

* View Article

* PubMed/NCBI

* Google Scholar

8. 8. Chicco D, Tötsch N, Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining. 2021;14(1):13. pmid:33541410

* View Article

* PubMed/NCBI

* Google Scholar

9. 9. De Diego IM, Redondo AR, Fernández RR, Navarro J, Moguerza JM. General Performance Score for classification problems. Applied Intelligence. 2022;52(10):12049–12063.

* View Article

* Google Scholar

10. 10. Hicks SA, Strümke I, Thambawita V, Hammou M, Riegler MA, Halvorsen P, et al. On evaluation metrics for medical applications of artificial intelligence. Scientific Reports. 2022;12(1):5979. pmid:35395867

* View Article

* PubMed/NCBI

* Google Scholar

11. 11. Lavazza L, Morasca S. Common Problems With the Usage of F-Measure and Accuracy Metrics in Medical Research. IEEE Access. 2023;11:51515–51526.

* View Article

* Google Scholar

12. 12. Chicco D, Jurman G. The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining. 2023;16(1).

* View Article

* Google Scholar

13. 13. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. pmid:7063747

* View Article

* PubMed/NCBI

* Google Scholar

14. 14. Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine. 1978;8(4):283–298. pmid:112681

* View Article

* PubMed/NCBI

* Google Scholar

15. 15. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical chemistry. 1993;39 4:561–77. pmid:8472349

* View Article

* PubMed/NCBI

* Google Scholar

16. 16. Hoo ZH, Candlish J, Teare D. What is an ROC curve? Emergency Medicine Journal. 2017;34(6):357–359. pmid:28302644

* View Article

* PubMed/NCBI

* Google Scholar

17. 17. Bradley AP. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition. 1997;30(7):1145–1159.

* View Article

* Google Scholar

18. 18. Baker SG. The Central Role of Receiver Operating Characteristic (ROC) Curves in Evaluating Tests for the Early Detection of Cancer. JNCI Journal of the National Cancer Institute. 2003;95(7):511–515. pmid:12671018

* View Article

* PubMed/NCBI

* Google Scholar

19. 19. Pérez-Pons ME, Parra-Dominguez J, Hernández G, Herrera-Viedma E, Corchado JM. Evaluation metrics and dimensional reduction for binary classification algorithms: a case study on bankruptcy prediction. The Knowledge Engineering Review. 2022;37:e1.

* View Article

* Google Scholar

20. 20. Morasca S, Lavazza L. On the assessment of software defect prediction models via ROC curves. Empirical Software Engineering. 2020;25(5):3977–4019.

* View Article

* Google Scholar

21. 21. Parodi S, Muselli M, Fontana V, Bonassi S. ROC curves are a suitable and flexible tool for the analysis of gene expression profiles. Cytogenetic and Genome Research. 2003;101(1):90–91. pmid:14571143

* View Article

* PubMed/NCBI

* Google Scholar

22. 22. Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging. 2015;15(1):29. pmid:26263899

* View Article

* PubMed/NCBI

* Google Scholar

23. 23. Yang S, Berdine G. The receiver operating characteristic (ROC) curve. The Southwest Respiratory and Critical Care Chronicles. 2017;5(19):34.

* View Article

* Google Scholar

24. 24. Berrar D, Flach P. Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them). Briefings in Bioinformatics. 2012;13(1):83–97. pmid:21422066

* View Article

* PubMed/NCBI

* Google Scholar

25. 25. Cook NR. Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction. Circulation. 2007;115(7):928–935. pmid:17309939

* View Article

* PubMed/NCBI

* Google Scholar

26. 26. Powers DMW. The problem of Area Under the Curve. In: 2012 IEEE International Conference on Information Science and Technology. IEEE; 2012. p. 567–573. Available from: http://ieeexplore.ieee.org/document/6221710/.

27. 27. Muschelli J. ROC and AUC with a Binary Predictor: a Potentially Misleading Metric. Journal of Classification. 2020;37(3):696–708. pmid:33250548

* View Article

* PubMed/NCBI

* Google Scholar

28. 28. Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach. European Radiology. 2015;25(4):932–939. pmid:25599932

* View Article

* PubMed/NCBI

* Google Scholar

29. 29. Wald N, Bestwick J. Is the area under an ROC curve a valid measure of the performance of a screening or diagnostic test? Journal of Medical Screening. 2014;21(1):51–56. pmid:24407586

* View Article

* PubMed/NCBI

* Google Scholar

30. 30. Movahedi F, Padman R, Antaki JF. Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores. The Journal of Thoracic and Cardiovascular Surgery. 2023;165(4):1433–1442.e2. pmid:34446286

* View Article

* PubMed/NCBI

* Google Scholar

31. 31. Lobo JM, Jiménez‐Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography. 2008;17(2):145–151.

* View Article

* Google Scholar

32. 32. McClish DK. Analyzing a Portion of the ROC Curve. Medical Decision Making. 1989;9(3):190–195. pmid:2668680

* View Article

* PubMed/NCBI

* Google Scholar

33. 33. Lavazza L, Morasca S. Considerations on the region of interest in the ROC space. Statistical Methods in Medical Research. 2022;31(3):419–437. pmid:34928729

* View Article

* PubMed/NCBI

* Google Scholar

34. 34. Chicco D, Starovoitov V, Jurman G. The Benefits of the Matthews Correlation Coefficient (MCC) Over the Diagnostic Odds Ratio (DOR) in Binary Classification Assessment. IEEE Access. 2021;9:47112–47124.

* View Article

* Google Scholar

35. 35. Yuan Y, Su W, Zhu M. Threshold-Free Measures for Assessing the Performance of Medical Screening Tests. Frontiers in Public Health. 2015;3. pmid:25941668

* View Article

* PubMed/NCBI

* Google Scholar

36. 36. Jiménez-Valverde A. The uniform AUC: Dealing with the representativeness effect in presence–absence models. Methods in Ecology and Evolution. 2022;13(6):1224–1236.

* View Article

* Google Scholar

37. 37. Parodi S, Pistoia V, Muselli M. Not proper ROC curves as new tool for the analysis of differentially expressed genes in microarray experiments. BMC Bioinformatics. 2008;9(1):410. pmid:18834513

* View Article

* PubMed/NCBI

* Google Scholar

38. 38. Parodi S, Muselli M, Carlini B, Fontana V, Haupt R, Pistoia V, et al. Restricted ROC curves are useful tools to evaluate the performance of tumour markers. Statistical Methods in Medical Research. 2016;25(1):294–314. pmid:22735161

* View Article

* PubMed/NCBI

* Google Scholar

39. 39. Swamidass SJ, Azencott CA, Daily K, Baldi P. A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval. Bioinformatics. 2010;26(10):1348–1356. pmid:20378557

* View Article

* PubMed/NCBI

* Google Scholar

40. 40. Vivo JM, Franco M, Vicari D. Rethinking an ROC partial area index for evaluating the classification performance at a high specificity range. Advances in Data Analysis and Classification. 2018;12(3):683–704.

* View Article

* Google Scholar

41. 41. Carrington AM, Fieguth PW, Qazi H, Holzinger A, Chen HH, Mayr F, et al. A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Medical Informatics and Decision Making. 2020;20(1):4. pmid:31906931

* View Article

* PubMed/NCBI

* Google Scholar

42. 42. Pontius RG Jr, Si K. The total operating characteristic to measure diagnostic ability for multiple thresholds. International Journal of Geographical Information Science. 2014;28(3):570–583.

* View Article

* Google Scholar

43. 43. Drummond C, Holte RC. Explicitly representing expected cost: an alternative to ROC representation. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD’00. New York, NY, USA: Association for Computing Machinery; 2000. p. 198–207. Available from: https://doi.org/10.1145/347090.347126.

44. 44. Ozenne B, Subtil F, Maucort-Boulch D. The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. Journal of Clinical Epidemiology. 2015;68(8):855–859. pmid:25881487

* View Article

* PubMed/NCBI

* Google Scholar

45. 45. Redondo AR, Navarro J, Fernández RR, de Diego IM, Moguerza JM, Fernández-Muñoz JJ. Unified Performance Measure for Binary Classification Problems. In: Analide C, Novais P, Camacho D, Yin H, editors. Intelligent Data Engineering and Automated Learning—IDEAL 2020. Cham: Springer International Publishing; 2020. p. 104–112.

46. 46. Halimu C, Kasem A, Newaz SHS. Empirical Comparison of Area under ROC curve (AUC) and Mathew Correlation Coefficient (MCC) for Evaluating Machine Learning Algorithms on Imbalanced Datasets for Binary Classification. In: Proceedings of the 3rd International Conference on Machine Learning and Soft Computing. ACM; 2019. p. 1–6. Available from: https://dl.acm.org/doi/10.1145/3310986.3311023.

47. 47. Yao J, Shepperd M. Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters. In: Proceedings of the Evaluation and Assessment in Software Engineering; 2020. p. 120–129. Available from: http://arxiv.org/abs/2003.01182.

48. 48. Kruschke J. Bayes’ rule: Doing Bayesian Data Analysis. Amsterdam: Elsevier; 2015.

49. 49. Dressel J, Farid H. The accuracy, fairness, and limits of predicting recidivism. Sci Adv. 2018;4(1):eaao5580. pmid:29376122

* View Article

* PubMed/NCBI

* Google Scholar

50. 50. Bansak K. Can nonexperts really emulate statistical learning methods? A comment on “The accuracy, fairness, and limits of predicting recidivism”. Polit Anal. 2019;27(3):370–380.

* View Article

* Google Scholar

Citation: Li J (2024) Area under the ROC Curve has the most consistent evaluation for binary classification. PLoS ONE 19(12): e0316019. https://doi.org/10.1371/journal.pone.0316019

About the Authors:

Jing Li

Roles: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

E-mail: [email protected]

Affiliation: Department of Political Science, University of Illinois at Urbana-Champaign, Urbana, Illinois, United States of America

ORICD: https://orcid.org/0000-0001-9332-399X

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

[/RAW_REF_TEXT]

References

1. Powers DMW. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. ArXiv. 2011;abs/2010.16061.

2. Garcia V SSJ, R AM. On the suitability of numerical performance measures for class imbalance problems. Proceedings of the 1st International Conference on Pattern Recognition Applications and Methods, Algarve, Portugal. 2012; p. 310–313.

3. Lever J, Krzywinski M, Altman N. Classification evaluation. Nature Methods. 2016;13(8):603–604.

4. Luque A, Carrasco A, Martín A, De Las Heras A. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition. 2019;91:216–231.

5. Jadhav AS. A novel weighted TPR-TNR measure to assess performance of the classifiers. Expert Systems with Applications. 2020;152:113391.

6. Zhu Q. On the performance of Matthews correlation coefficient (MCC) for imbalanced dataset. Pattern Recognition Letters. 2020;136:71–80.

7. Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics. 2020;21(1):6. pmid:31898477

8. Chicco D, Tötsch N, Jurman G. The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining. 2021;14(1):13. pmid:33541410

9. De Diego IM, Redondo AR, Fernández RR, Navarro J, Moguerza JM. General Performance Score for classification problems. Applied Intelligence. 2022;52(10):12049–12063.

10. Hicks SA, Strümke I, Thambawita V, Hammou M, Riegler MA, Halvorsen P, et al. On evaluation metrics for medical applications of artificial intelligence. Scientific Reports. 2022;12(1):5979. pmid:35395867

11. Lavazza L, Morasca S. Common Problems With the Usage of F-Measure and Accuracy Metrics in Medical Research. IEEE Access. 2023;11:51515–51526.

12. Chicco D, Jurman G. The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining. 2023;16(1).

13. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. pmid:7063747

14. Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine. 1978;8(4):283–298. pmid:112681

15. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical chemistry. 1993;39 4:561–77. pmid:8472349

16. Hoo ZH, Candlish J, Teare D. What is an ROC curve? Emergency Medicine Journal. 2017;34(6):357–359. pmid:28302644

17. Bradley AP. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition. 1997;30(7):1145–1159.

18. Baker SG. The Central Role of Receiver Operating Characteristic (ROC) Curves in Evaluating Tests for the Early Detection of Cancer. JNCI Journal of the National Cancer Institute. 2003;95(7):511–515. pmid:12671018

19. Pérez-Pons ME, Parra-Dominguez J, Hernández G, Herrera-Viedma E, Corchado JM. Evaluation metrics and dimensional reduction for binary classification algorithms: a case study on bankruptcy prediction. The Knowledge Engineering Review. 2022;37:e1.

20. Morasca S, Lavazza L. On the assessment of software defect prediction models via ROC curves. Empirical Software Engineering. 2020;25(5):3977–4019.

21. Parodi S, Muselli M, Fontana V, Bonassi S. ROC curves are a suitable and flexible tool for the analysis of gene expression profiles. Cytogenetic and Genome Research. 2003;101(1):90–91. pmid:14571143

22. Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging. 2015;15(1):29. pmid:26263899

23. Yang S, Berdine G. The receiver operating characteristic (ROC) curve. The Southwest Respiratory and Critical Care Chronicles. 2017;5(19):34.

24. Berrar D, Flach P. Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them). Briefings in Bioinformatics. 2012;13(1):83–97. pmid:21422066

25. Cook NR. Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction. Circulation. 2007;115(7):928–935. pmid:17309939

26. Powers DMW. The problem of Area Under the Curve. In: 2012 IEEE International Conference on Information Science and Technology. IEEE; 2012. p. 567–573. Available from: http://ieeexplore.ieee.org/document/6221710/.

27. Muschelli J. ROC and AUC with a Binary Predictor: a Potentially Misleading Metric. Journal of Classification. 2020;37(3):696–708. pmid:33250548

28. Halligan S, Altman DG, Mallett S. Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: A discussion and proposal for an alternative approach. European Radiology. 2015;25(4):932–939. pmid:25599932

29. Wald N, Bestwick J. Is the area under an ROC curve a valid measure of the performance of a screening or diagnostic test? Journal of Medical Screening. 2014;21(1):51–56. pmid:24407586

30. Movahedi F, Padman R, Antaki JF. Limitations of receiver operating characteristic curve on imbalanced data: Assist device mortality risk scores. The Journal of Thoracic and Cardiovascular Surgery. 2023;165(4):1433–1442.e2. pmid:34446286

31. Lobo JM, Jiménez‐Valverde A, Real R. AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography. 2008;17(2):145–151.

32. McClish DK. Analyzing a Portion of the ROC Curve. Medical Decision Making. 1989;9(3):190–195. pmid:2668680

33. Lavazza L, Morasca S. Considerations on the region of interest in the ROC space. Statistical Methods in Medical Research. 2022;31(3):419–437. pmid:34928729

34. Chicco D, Starovoitov V, Jurman G. The Benefits of the Matthews Correlation Coefficient (MCC) Over the Diagnostic Odds Ratio (DOR) in Binary Classification Assessment. IEEE Access. 2021;9:47112–47124.

35. Yuan Y, Su W, Zhu M. Threshold-Free Measures for Assessing the Performance of Medical Screening Tests. Frontiers in Public Health. 2015;3. pmid:25941668

36. Jiménez-Valverde A. The uniform AUC: Dealing with the representativeness effect in presence–absence models. Methods in Ecology and Evolution. 2022;13(6):1224–1236.

37. Parodi S, Pistoia V, Muselli M. Not proper ROC curves as new tool for the analysis of differentially expressed genes in microarray experiments. BMC Bioinformatics. 2008;9(1):410. pmid:18834513

38. Parodi S, Muselli M, Carlini B, Fontana V, Haupt R, Pistoia V, et al. Restricted ROC curves are useful tools to evaluate the performance of tumour markers. Statistical Methods in Medical Research. 2016;25(1):294–314. pmid:22735161

39. Swamidass SJ, Azencott CA, Daily K, Baldi P. A CROC stronger than ROC: measuring, visualizing and optimizing early retrieval. Bioinformatics. 2010;26(10):1348–1356. pmid:20378557

40. Vivo JM, Franco M, Vicari D. Rethinking an ROC partial area index for evaluating the classification performance at a high specificity range. Advances in Data Analysis and Classification. 2018;12(3):683–704.

41. Carrington AM, Fieguth PW, Qazi H, Holzinger A, Chen HH, Mayr F, et al. A new concordant partial AUC and partial c statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Medical Informatics and Decision Making. 2020;20(1):4. pmid:31906931

42. Pontius RG Jr, Si K. The total operating characteristic to measure diagnostic ability for multiple thresholds. International Journal of Geographical Information Science. 2014;28(3):570–583.

43. Drummond C, Holte RC. Explicitly representing expected cost: an alternative to ROC representation. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD’00. New York, NY, USA: Association for Computing Machinery; 2000. p. 198–207. Available from: https://doi.org/10.1145/347090.347126.

44. Ozenne B, Subtil F, Maucort-Boulch D. The precision–recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. Journal of Clinical Epidemiology. 2015;68(8):855–859. pmid:25881487

45. Redondo AR, Navarro J, Fernández RR, de Diego IM, Moguerza JM, Fernández-Muñoz JJ. Unified Performance Measure for Binary Classification Problems. In: Analide C, Novais P, Camacho D, Yin H, editors. Intelligent Data Engineering and Automated Learning—IDEAL 2020. Cham: Springer International Publishing; 2020. p. 104–112.

46. Halimu C, Kasem A, Newaz SHS. Empirical Comparison of Area under ROC curve (AUC) and Mathew Correlation Coefficient (MCC) for Evaluating Machine Learning Algorithms on Imbalanced Datasets for Binary Classification. In: Proceedings of the 3rd International Conference on Machine Learning and Soft Computing. ACM; 2019. p. 1–6. Available from: https://dl.acm.org/doi/10.1145/3310986.3311023.

47. Yao J, Shepperd M. Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters. In: Proceedings of the Evaluation and Assessment in Software Engineering; 2020. p. 120–129. Available from: http://arxiv.org/abs/2003.01182.

48. Kruschke J. Bayes’ rule: Doing Bayesian Data Analysis. Amsterdam: Elsevier; 2015.

49. Dressel J, Farid H. The accuracy, fairness, and limits of predicting recidivism. Sci Adv. 2018;4(1):eaao5580. pmid:29376122

50. Bansak K. Can nonexperts really emulate statistical learning methods? A comment on “The accuracy, fairness, and limits of predicting recidivism”. Polit Anal. 2019;27(3):370–380.