Binary Classification with Imbalanced Data

Abstract

When the binary response variable contains an excess of zero counts, the data are imbalanced. Imbalanced data cause trouble for binary classification. To simplify the numerical computation to obtain the maximum likelihood estimators of the zero-inflated Bernoulli (ZIBer) model parameters with imbalanced data, an expectation-maximization (EM) algorithm is proposed to derive the maximum likelihood estimates of the model parameters. The logistic regression model links the Bernoulli probabilities with the covariates in the ZIBer model, and the prediction performance among the ZIBer model, LightGBM, and artificial neural network (ANN) procedures is compared by Monte Carlo simulation. The results show that no method can dominate the other methods regarding predictive performance under the imbalanced data. The LightGBM and ZIBer models are more competitive than the ANN model for zero-inflated-imbalanced data sets.

1. Introduction

When a data set contains an excess of zero counts over the expectation of a standard statistical distribution, a zero-inflated model helps to analyze such data sets. In many scenarios, count data have two sources of zeros. They are the structural zeros and sampling zeros. Structural zeros mean that the response variable cannot take positive values because of inherent constraints or conditions, and sampling zeros mean the zeros result from random chance. The idea behind the zero-inflated model is to account for two sources of zeros using a mixture of two separated distributions. One model characterizes the probability of structural zeros, and the other describes the non-zero counts. This mixture mechanism of the zero-inflated model allows for better representing the complexities when the data set has excess zeros. The zero-inflated Poisson (ZIP) model and zero-inflated negative Binomial (ZINB) model are two major zero-inflated models.

Imbalanced data refers to the distribution of categories in the data set that is highly disproportionate. For example, in a data set with two types, one type has significantly more instances than another. The situation results in an unequal representation of the two categories. Data sets could be inherently imbalanced in real-world scenarios for various reasons. One example of imbalanced data in medical applications can be diagnosing a rare disease with only a tiny percentage of patients with the condition. In this case, the class size of the rare event will be severely smaller than the common event.

Traditional machine learning algorithms and statistical models could be challenged to handle imbalanced data. The prediction results could be biased toward the majority class and lead to poor performance of the minority class. However, the minority class is often more significantly interesting in many applications. The well-known drawbacks of machine learning algorithms caused by imbalanced data sets are model bias, poor generalization, and misleading evaluation metrics. Several techniques can be used to address the issue of imbalanced data, including resampling, synthetic data generation, using ensemble methods, and more.

The artificial neural network (ANN) has been widely used to implement categorical classification. Rosenblatt [1] proposed a theory of perception about a hypothetical nervous system that can sense, remember, and recognize information from the physical world. Rumelhart et al. [2] discussed a learning algorithm for backpropagation ANNs. The algorithm adjusts the weights of the connections in the network to minimize the error between the realistic output and the desired output. Hirose et al. [3] modified the backpropagation algorithm by varying the number of hidden units in the ANN. Their algorithm can reduce the probability of becoming trapped in local minima and speed up the convergence compared to conventional backpropagation ANN. Sietsma and Dow [4] studied the relationship between the size and structure of an ANN and its ability to generalize from a training set. Using a complex classification task, they investigated the effects of pruning and noise training on network performance.

In the previous decade, Devi et al. [5] emphasized the classification of cervical cancer using ANN methods to develop a strategy for accurately categorizing cervical cancer cases. Sze et al. [6] provided a comprehensive tutorial and survey on the efficient processing of deep neural networks. Abiodun et al. [7] surveyed the ANN applications to provide a taxonomy of ANN models and provided readers with knowledge of current and emerging trends in ANN research. Nasser and Abu-Naser [8] addressed the detection of lung cancer using an ANN method to explore the feasibility of employing ANN technology for identifying lung cancer, which is a significant advancement in medical diagnostics. Muhammad et al. [9] delved into predicting pancreatic cancer using an ANN method. Umar [10] predicted student academic performance using ANN methods to forecast students’ academic achievements. Shen et al. [11] introduced a novel ensemble classification model for imbalanced credit risk evaluation. The model integrates neural networks with a classifier optimization technique. The study addressed the challenge of imbalanced data sets in credit risk assessment by leveraging the strengths of neural networks and advanced optimization techniques.

Lambert [12] proposed a ZIP regression model to process count data with excess zeros. The author applied the ZIP regression model to a soldering defects data set on the printed wiring board and compared the performance of the proposed model with other models. Hall [13] presented a case study of ZIP and zero-inflated binomial (ZIB) regression with random effects. The author showed that zero-inflated models are useful when the data contains an excess of zeros, and random effects can be used to account for the correlation between observations within clusters. Cheung [14] discussed the theoretical underpinnings of zero-inflated models, provided examples of their application, and discussed the implications of using these models for accurate inference in the context of growth and development studies. Gelfand and Citron-Pousty [15] presented how to use zero-inflated models for spatial count data within the environmental and ecological statistics field. They explored how zero-inflated models can account for both the spatial correlation and the excess zero counts often found in environmental and ecological applications. Rodrigues [16] discussed how Bayesian techniques can be applied to estimate the parameters of zero-inflated models and provided practical examples to demonstrate the approach.

Ghosh et al. [17] provided a Bayesian analysis of zero-inflated regression models. Their Bayesian approach suggested a coherent way to handle the challenges of excess zeros in count data, offering insights into parameter estimation and uncertainty quantification. Harri and Zhao [18] introduced the zero-inflated-ordered probit model to handle ordered response data with excess zeros. The proposed method is applied to analyze tobacco consumption data. Loeys et al. [19] explored the analysis of zero-inflated count data, going beyond the scope of ZIP regression. Diop et al. [20] studied the maximum likelihood estimation method in the logistic regression model with a cure fraction. The authors introduced the concept of cure fraction models, which analyze data where many individuals do not experience the event of interest. Staub and Winkelmann [21] studied the consistent estimation of zero-inflated count models. He et al. [22] discussed the concept of structural zeros and their relevance to zero-inflated models. Diop et al. [23] studied simulation-based inference in a zero-inflated Bernoulli regression model. This research is valuable for practitioners working with binary data exhibiting excess zeros.

Alsabti et al. [24] present a novel decision tree classifier called CLOUDS for classification using big data. The authors showed that techniques such as discretization and data set sampling can be used to scale up decision tree classifiers to large data sets. However, both methods can cause a significant accuracy loss. Friedman [25] introduced Gradient Boosting Machines (GBMs) with a powerful technique for building predictive models by iteratively combining weak learners. GBMs are known for their effectiveness in handling complex, high-dimensional data with robustness against overfitting. Jin and Agrawal [26] addressed the challenge of constructing decision trees efficiently in a parallel computing environment. They improved communication and memory efficiency during the parallel construction of decision trees.

Chen and Guestrin [27] developed the “XGBoost”, a scalable tree-boosting system. XGBoost is a powerful and efficient algorithm for boosting decision trees. Ke et al. [28] introduced “LightGBM”, a highly efficient gradient boosting decision tree algorithm. This tool dramatically contributes to machine learning, particularly boosting techniques. LightGBM has gained attention for its speed and scalability, making it suitable for handling large data sets and complex problems. Wang et al. [29] applied “LightGBM” for the miRNA classification in breast cancer patients. Ma et al. [30] used multi-observation and multi-dimensional data cleaning methods and the algorithms of LightGBM and XGboost for prediction. The authors concluded that the LightGBM algorithm based on multiple observational data set classification prediction results is the best. Machado et al. [31] introduced LightGBM, showing it is an excellent decision tree gradient boosting method. The authors applied LightGBM to predict customer loyalty within the finance industry. Minstireanu and Mesnita [32] employed the LightGBM algorithm for online click fraud detection. Daoud [33] compared three popular gradient boosting algorithms, XGBoost, LightGBM, and CatBoost, to evaluate the strengths and weaknesses of each algorithm in the context of credit risk assessment or related tasks.

Some open challenges for implementing imbalanced data analysis have been comprehensively discussed by Krawczyk [34] and Nalepa and Kawulok [35]. The two works provided insights into imbalanced data modeling. Krawczyk [34] pointed out the open challenges including the binary and multi-class classification problems, how to do multi-label and multi-instance learning, and unsupervised and semi-supervised handling for imbalanced data sets. It is also challenging to perform regression on skewed cases and a well-designed procedure for learning from imbalanced data streams under stationary and drifting environments, then extending the model for large-scale and big-data cases. Nalepa and Kawulok [35] reviewed the issue of selecting training sets for support vector machines. They did an extensive survey on existing methods for using the support vector machine method to train data in big-data applications. Nalepa and Kawulok [35] separated these helpful techniques into several categories. This work helps users understand the underlying ideas behind the algorithms used.

Even a ZIBer regression model has been proposed in the literature. How to obtain the maximum likelihood estimates (MLEs) of the ZIBer regression model parameters can be a gap in this topic. We are also interested in studying the performance of the ZIBer regression model compared with machine learning classifiers, for example, the LighGBM and ANN methods. In this study, we propose an EM algorithm to obtain reliable MLEs of the ZIBer regression model parameters based on zero-inflated-imbalanced binary data. The logistic model links the event or non-event probability with explanatory variables. The innovation is to propose the theoretical process of the EM algorithm to obtain reliable MLEs for the ZIBer regression model using imbalanced data sets. Monte Carlo simulations were conducted to show the predictive performance of the ZIBer, LightGBM, and ANN methods for binary classification under imbalanced data.

The rest of this article is organized as follows. Section 2 presents the ZIBer regression model and the proposed EM procedure that produces reliable MLEs of the model parameters. In Section 3, two examples are used to illustrate the applications of using the proposed EM algorithm to the ZIBer regression model. In Section 4, Monte Carlo simulations were conducted to compare the classification rate of the ZIBer model, LighGBM, and ANN methods in three measures. The design of Monte Carlo simulations and their implementation are studied in this section. Some concluding remarks are given in Section 5.

2. Zero-Inflated Bernoulli Regression Model and EM Algorithm

Consider a special case about the infection status of a specific disease. $Y_i$ denotes the infection status for the specific disease of the i-th individual in a sample of size n; $Y_i=1$ if the individual is infected and $Y_i=0$ otherwise. The conditional distribution of $Y_i$ can be a Bernoulli distribution, denoted by $Y_i|{\mathit{x}}_i\sim B(1,p_i)$, where $p_i=P(Y_i=1|{\mathit{x}}_i)$ is the conditional probability of $Y_i=1$, given a $p\times 1$ vector of explanatory variables, ${\mathit{x}}_i^T=(x_{i1}=1,x_{i2},\cdots ,x_{ip})$. The logistic function to link $Y_i$ and ${\mathit{x}}_i$ can be expressed by

(1)$log\left(\frac{p_i}{1-p_i}\right)={\mathit{x}}_i^T\mathsf{\beta }={\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+\cdots +{\beta }_p{x}_{ip},i=1,2,\cdots ,n,$

where ${\mathsf{\beta }}^T=({\beta }_1,{\beta }_2,\cdots ,{\beta }_p)$ is the coefficient vector. Failure to account for the extra zeros in a zero-inflated model can result in biased estimates and inferences.

Let ${\delta }_i$ control the probability of $Y_i$ being a structural zero. ${\delta }_i$ can be an unconditional probability of $Y_i$ being a structural zero controlled by an unobserved factor, say $w_i$. When $w_i=1$, $Y_i=0$, otherwise, $w_i=0$, $Y_i$ can be 0 or 1, determined by the model $B(1,p_i)$, where $p_i=P(Y_i=1|{\mathit{x}}_i))$. It can be shown that the unobserved factor $w_i\sim B(1,{\delta }_i)$ for $i=1,2,\cdots ,n$. Two situations result in $Y_i=0$; the first situation is that $Y_i$ is a structural zero with probability ${\delta }_i$, and the other is the situation that $Y_i$ is not a structural zero but has the probability $1-p_i$ to be zero. The unconditional probability of $Y_i=0$ can be presented by

(2)$P(Y_i=0)={\delta }_i+(1-{\delta }_i)(1-p_i)=1-(1-{\delta }_i)p_i=1-{\pi }_i,i=1,2,\cdots ,n,$

where ${\pi }_i=(1-{\delta }_i)p_i$. The unconditional probability of $Y_i=1$ can be presented by

(3)$\begin{array}{c}\hfill P(Y_i=1)=1-P(Y_i=0)={\pi }_i,i=1,2,\cdots ,n.\end{array}$

We can obtain the unconditional distribution of $Y_i$ by $Y_i\sim B(1,{\pi }_i)$.

Using the second logistic function to link ${\delta }_i$ and the $k\times 1$ covariate vector of ${\mathit{z}}_i^T=(z_{i1}=1,z_{i2},\cdots ,z_{ik})$, we obtain

(4)$log\left(\frac{{\delta }_i}{1-{\delta }_i}\right)={\mathit{z}}_i^T\mathsf{\theta }={\theta }_1{z}_{i1}+{\theta }_2{z}_{i2}+\cdots +{\theta }_p{z}_{ik},i=1,2,\cdots ,n,$

where ${\mathsf{\theta }}^T=({\theta }_1,{\theta }_2,\cdots ,{\theta }_k)$ is the vector of model parameters. After algebraic formulation, we can show that

(5)$\begin{array}{c}\hfill {\pi }_i=\frac{1}{1+e^{{\mathit{z}}_i^T\mathsf{\theta }}}\times \frac{e^{{\mathit{x}}_i^T\mathsf{\beta }}}{1+e^{{\mathit{x}}_i^T\mathsf{\beta }}},i=1,2,\cdots ,n.\end{array}$

Assume that sample $\mathit{D}=(y_1,\cdots ,y_n,{\mathit{x}}_1,\cdots ,{\mathit{x}}_n,{\mathit{z}}_1,\cdots ,{\mathit{z}}_n)$ is available. The likelihood function can be constructed as follows:

(6)$\begin{array}{cc}\hfill L(\mathsf{\beta },\mathsf{\theta }|\mathit{D})& =\prod _{i=1}^n{\pi }_i^{y_i}{(1-{\pi }_i)}^{1-y_i}\hfill \\ \hfill & =\prod _{i=1}^n{\left(\frac{e^{{\mathit{x}}_i^T\mathsf{\beta }}}{\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}\right)\left(1+e^{{\mathit{x}}_i^T\mathsf{\beta }}\right)}\right)}^{y_i}\times {\left(\frac{1+e^{{\mathit{z}}_i^T\mathsf{\theta }}+e^{{\mathit{z}}_i^T\mathsf{\theta }+{\mathit{x}}_i^T\mathsf{\beta }}}{\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}\right)\left(1+e^{{\mathit{x}}_i^T\mathsf{\beta }}\right)}\right)}^{1-y_i}.\hfill \end{array}$

After algebraic formulation, we can obtain the log-likelihood equation,

(7)$\begin{array}{cc}\hfill \mathit{\ell }(\mathsf{\beta },\mathsf{\theta }|\mathit{D})& =\sum _{i=1}^n{y}_i{\mathit{x}}_{\mathit{i}}^T\mathsf{\beta }+(1-y_i)log\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}+e^{{\mathit{x}}_i^T\mathsf{\beta }+{\mathit{z}}_i^T\mathsf{\theta }}\right)\hfill \\ \hfill & -\sum _{i=1}^{n}log\left(1+e^{{\mathit{x}}_i^T\mathsf{\beta }}\right)-\sum _{i=1}^{n}log\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}\right).\hfill \end{array}$

To obtain reliable maximum likelihood estimators, it could be challenging to directly maximize the log-likelihood function in Equation (7). So, we suggest an EM algorithm to maximize the log-likelihood function.

Assume that an unobserved probability $w_i$ can determine whether $Y_i$ is a structural zero or not; when $w_i=1$ implies that $Y_i=0$ is a structural zero, and $w_i=0$ implies that the Bernoulli distribution determines the response $Y_i$ is 1 or 0. The likelihood function based on complete data $\mathit{D}$ can be represented by

(8)$\begin{array}{cc}\hfill L(\mathsf{\beta },\mathsf{\theta }|\mathit{D})& =\prod _{i=1}^{n}f\left(w_i\right|{\mathit{z}}_i)f\left(y_i\right|w_i,{\mathit{x}}_i)\hfill \\ \hfill & =\prod _{i=1}^{n}f(w_i=1|{\mathit{z}}_i)\prod _{i=1}^{n}f(w_i=0|{\mathit{z}}_i)f\left(y_i\right|w_i=0,{\mathit{x}}_{\mathit{i}}).\hfill \end{array}$

The log-likelihood function can be represented by

(9)$\begin{array}{c}\hfill L(\mathsf{\beta },\mathsf{\theta }|\mathit{D},\mathit{w})=\prod _{i=1}^n{e}^{w_i{\mathit{z}}_i^T\mathsf{\theta }}{\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}\right)}^{-1}{e}^{y_i(1-w_i){\mathit{x}}_i^T\mathsf{\beta }}{\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}\right)}^{-(1-w_i)},\end{array}$

and the log-likelihood function can be obtained by

(10)$\begin{array}{c}\hfill \ell (\mathsf{\beta },\mathsf{\theta }|\mathit{D},\mathit{w})={\ell }_c\left(\mathsf{\theta }\right|\mathit{w},\mathit{z})+{\ell }_c\left(\mathsf{\beta }\right|\mathit{D},\mathit{w}),\end{array}$

where

(11)$\begin{array}{cc}\hfill {\ell }_c\left(\mathsf{\theta }\right|\mathit{w},\mathit{z})& =\sum _{i=1}^n{w}_i{\mathit{z}}_i^T\mathsf{\theta }-log\left(1+e^{{\mathit{z}}_i^T\mathsf{\theta }}\right)\hfill \end{array}$

and

(12)$\begin{array}{cc}\hfill {\ell }_c\left(\mathsf{\beta }\right|\mathit{D},\mathit{w})& =\sum _{i=1}^n(1-w_i)\left\{y_i{\mathit{x}}_i^T\mathsf{\beta }-log\left(1+e^{{\mathit{x}}_i^T\mathsf{\beta }}\right)\right\}.\hfill \end{array}$

Using Equations (11) and (12) to implement the following E-step and M-step until convergences of the estimates of parameters, $\mathsf{\theta }$ and $\mathsf{\beta }$, are obtained.

• E-step:. Iteration $(h+1)$: Estimate $w_i$ by its posterior mean $w_i^{(h+1)}$ given the estimates ${\mathsf{\beta }}^{\left(h\right)}$ and ${\mathsf{\theta }}^{\left(h\right)}$. If $y_i=1$, then $w_i^{(h+1)}=0$; otherwise,

  (13)$\begin{array}{cc}\hfill w_i^{(h+1)}& =\frac{P(Y_i=0|A)P\left(A\right)}{P(Y_i=0|A)P\left(A\right)+P(Y_i=0|A^c)P\left(A^c\right)}\hfill \end{array}$

  (14)$\begin{array}{cc}\hspace{1em}\hspace{1em}\hspace{1em}& =\frac{1\times {\delta }_i}{1\times {\delta }_i+(1-p_i)(1-{\delta }_i)}=\frac{{\delta }_i}{1-p_i(1-{\delta }_i)},\hfill \end{array}$

  where A and $A^c$ denote that $Y_i$ has a structure zero and is from a Bernoulli distribution.

• M-step:. Maximizing

  (15)$\begin{array}{c}\hfill {\ell }_c\left(\mathsf{\theta }\right|\mathit{z},{\mathit{w}}^{(h+1)})=\sum _{i=1}^n{w}_i^{(h+1)}{\mathit{z}}_i^T{\mathsf{\theta }}^{\left(h\right)}-log\left(1+e^{{\mathit{z}}_i^T{\mathsf{\theta }}^{\left(h\right)}}\right)\end{array}$

  to obtain ${\mathsf{\theta }}^{(h+1)}$, and maximizing

  (16)$\begin{array}{c}\hfill {\ell }_c\left(\mathsf{\beta }\right|\mathit{D},{\mathit{w}}^{(h+1)})=\sum _{i=1}^n(1-w_i)\left\{y_i{\mathit{x}}_i^T{\mathsf{\beta }}^{\left(h\right)}-log\left(1+e^{{\mathit{x}}_i^T{\mathsf{\beta }}^{\left(h\right)}}\right)\right\}\end{array}$

  to obtain ${\mathsf{\beta }}^{(h+1)}$.

3. Examples

3.1. Example 1

A diabetes data set is used for illustration. The data set can be obtained from the R package version 2.1–3.1 “mlbench”. There are 768 records with eight independent variables including

• The number of pregnancies.

• The glucose concentration in the 2-h oral glucose tolerance test.

• The diastolic blood pressure in mm Hg.

• The triceps skin-fold thickness in millimeters.

• Two-hour serum insulin in mu U/mL.

• Body mass.

• Diabetes pedigree function.

• Age in years.

The response variable is Diabetes, which can be negative or positive. After removing incomplete records, we have 392 records for the final sample for illustration. We labeled all the independent variables by $x_i$, $i=1,2,\cdots ,8$. Moreover, the variables of “The diastolic blood pressure in mm Hg”, “Body mass”, and “Age” are selected as the independent variables, labeled by $z_h$, $h=1,2,3$ to develop the second logistic regression model. The diabetes rate is 0.332 in this example.

All independent variables are scaling with $x_{ij}^{*}=\frac{x_{ij}-min\left(x_j\right)}{max\left(x_j\right)-min\left(x_j\right)}$ and $z_{ih}^{*}=\frac{z_{ih}-min\left(z_h\right)}{max\left(z_h\right)-min\left(z_h\right)}$ for $i=1,2,\cdots ,n$. Using the proposed EM algorithm, we can obtain the following ZIBer models,

(17)$\begin{array}{cc}\hfill log\left(\frac{p_i}{1-p_i}\right)& =-7.562+2.665x_{i1}^{*}+6.931x_{i2}^{*}+1.99x_{i3}^{*}-0.19x_{i4}^{*}+3.34x_{i5}^{*}\hfill \\ \hfill & +5.246x_{i6}^{*}+5.313x_{i7}^{*}-2.863x_{i8}^{*},\hfill \end{array}$

and

(18)$\begin{array}{cc}\hfill log\left(\frac{{\delta }_i}{1-{\delta }_i}\right)& =0.005+2.232z_{i1}^{*}-1.94z_{i2}^{*}-10.494z_{i3}^{*},i=1,2,\cdots ,392.\hfill \end{array}$

When the estimated probability $p_i\ge 0.53$, $i=1,2,\cdots ,392$, the corresponding person is identified for diabetes. Based on the obtained model, the accuracy is 0.806. We want to study how much efficiency is lost using the typical logistic regression model. Based on the typical logistic regression model, we obtain the following estimation function form,

(19)$\begin{array}{cc}\hfill log\left(\frac{p_i}{1-p_i}\right)& =-5.771+1.397x_{i1}^{*}+5.434x_{i2}^{*}-0.122x_{i3}^{*}+0.628x_{i4}^{*}-0.687x_{i5}^{*}\hfill \\ \hfill & +3.449x_{i6}^{*}+2.664x_{i7}^{*}+2.037x_{i8}^{*},i=1,2,\cdots ,392.\hfill \end{array}$

When the estimated probability $p_i\ge 0.39$, $i=1,2,\cdots ,392$, the corresponding person is identified for diabetes. Based on the obtained model, the accuracy is 0.798. The efficiency of the typical logistic regression model can be improved, but the improvement is not significant based on only 0.008 increment on the Accuracy. The diabetes rate is 0.332. If the success rate is high, the improvement of replacing the typical logistic regression model with the ZIBer regression model could not be significant. The finding makes sense because the imbalance in this example is insignificant. The R codes using the proposed EM algorithm to obtain the MLEs of the ZIBer regression model parameters are given in Appendix A. We use the best cut-rate of 0.53 to predict the value of Diabetes and identify it as positive if $p_i\ge 0.53$ and 0 otherwise. We can obtain the Accuracy, Sensitivity, and Specificity as 0.806, 0.608, and 0.905, respectively.

3.2. Example 2

A Taiwan credit data set is used for illustration. This data set can be obtained from the UC Irvine Machine Learning Repository by the hyperlink https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients; this data set was donated to UC Irvine Machine Learning Reponsitory on 25 January 2016. The response variable is the default payment, Yes = 1 for default and Yes = 0 for non-default. There are 30,000 records with the following 23 explanatory variables:

• The Given Credit Amount of NT dollars, including the individual consumer and his (or her) family (supplementary) credits.

• Gender, 1 for male and 2 for female.

• Education, 1 for graduate school, 2 for university, 3 for high school, and 4 for other.

• Marital Status, 1 for married, 2 for single, and 3 for other.

• Age in years.

• The history of past payments in 2005 from September to April in decent order of the columns. The measurement scale is −1 for pay duly, 1 for a one-month payment delay, 2 for a two-month payment delay, …, 8 for an eight-month payment delay, and 9 for a nine-month payment delay and above.

• The number of bill statements in 2005 with NT dollars from September to April in decent order of the columns.

• The amount of previous payments in 2005 with NT dollars from September to April in decent order of the columns.

The response variable of Default Payment is labeled by Y. We use the columns of The Given Credit Amount of NT dollars, Gender, Education, Marital Status, Age, and the payment statuses of April, May, June, July, and August as the independent variables, labeled by $X_1$, $X_2$, …, and $X_{10}$, respectively, to develop the ZIBer regression model. In many instances, we need to subjectively determine the independent variables of $Z^{\prime }s$ for establishing the second logistic regression model. In this example, we select the columns of Gender, Education, Marital Status, and Age as the independent variables of $Z^{\prime }s$ for the second logistic regression model.

No missing records were found in this data set. First, we searched the rows with labels that were not well-defined in the independent variables and removed all records that were not well-defined from the data set. Finally, 4030 records are used in the illustrative data set for constructing the ZIBer regression model.

The summary of the credit card users in the categories of Default Payment, Gender, Education, and Marriage is given below.

• 1436 default and 2594 non-default credit card users are included in the illustrative data set. The proportion is 35.63%.

• 2385 female and 1645 male credit card users are in the illustrative data set. The proportions are 59.18% and 40.82%, respectively.

• The number of credit card users in the education categories of Senior High School, College, Graduate, and Others are 624, 1713, 1683, and 10, respectively. The proportions are 15.48%, 42.51%, 41.76%, and 0.25%.

• 2076 credit card users are married, 1920 credit card users are single, and 34 credit card users are other. The proportions are 51.51%, 47.64%, and 0.85%.

Before model construction, we transform the categorical variable with three or larger categories into dummy variables. Hence, the Marital Status needs to be transformed into two dummy variables. If $X_4=1,2$, and 3, the new dummy variables of M1 and M2 can be obtained by $(M_1,M_2)=(1,0),(0,1)$ and (1,1), respectively. Let $Z_1=X_2$, $Z_2=X_3$, $Z_3=M_1$, $Z_4=M_2$, and $Z_5=X_5$. All independent variables are scaling with $x_{ij}^{*}=\frac{x_{ij}-min\left(x_j\right)}{max\left(x_j\right)-min\left(x_j\right)}$ and $z_{ih}^{*}=\frac{z_{ih}-min\left(z_h\right)}{max\left(z_h\right)-min\left(z_h\right)}$ for $i=1,2,\cdots ,n$. Using the proposed EM algorithm, we can obtain the following ZIBer models,

(20)$\begin{array}{cc}\hfill log\left(\frac{p_i}{1-p_i}\right)& =0.639-1.266x_{i1}^{*}-0.458x_{i2}^{*}+0.554x_{i3}^{*}-3.713M_{i1}^{*}-3.039M_{i2}^{*}\hfill \\ \hfill & +1.789x_{i5}^{*}+0.179x_{i6}^{*}+4.520x_{i7}^{*}+6.366x_{i8}^{*}-0.908x_{i9}^{*}+3.876x_{i10}^{*},\hfill \end{array}$

and

(21)$\begin{array}{cc}\hfill log\left(\frac{{\delta }_i}{1-{\delta }_i}\right)& =4.860-0.441z_{i1}^{*}+0.024z_{i2}^{*}-3.970z_{i3}^{*}-3.754z_{i4}^{*}-0.495z_{i5}^{*},\hfill \\ \hfill & i=1,2,\cdots ,4030.\hfill \end{array}$

When the estimated probability $p_i\ge 0.41$, $i=1,2,\cdots ,4030$, the corresponding customer is identified as a default. Based on the obtained model, the Accuracy is 0.911.

4. Monte Carlo Simulations

In this section, we would like to study the performance of the proposed ZIBer regression method and compare its performance with that of the ANN and LightGBM methods. To help readers have a comprehensive understanding of the Monte Carlo simulation design, the design of the Monte Carlo simulation section is explained in Section 4.1, and the implementation of performance comparison among models can be found in Section 4.2.

4.1. The Simulation Design

Before implementing Monte Carlo simulations, we need to generate imbalanced data. Hence, we need a baseline model to do a fair performance comparison. First, we generate zero-inflated samples. Assume that $w_1,w_2,\cdots ,w_n$ is a random sample from the uniform distribution over 0 and 1. if $w_i\le {\delta }_i$, let $Y_i=0$ and $Y_i$ follows a Bernoulli distribution with probability ${\pi }_i=(1-{\delta }_i)p_i$ for $i=1,2,\cdots ,n$. In the simulation studies, we use three covariates: first logistic model, $x_{i1}=1$, generate $x_{i2}$ from the standard normal distribution, and generate $x_{i3}$ from the Bernoulli distribution with probability 0.75. Let ${\beta }_1=-1$, ${\beta }_2=1$ and ${\beta }_3=-2$. Present the corresponding logistic model by

$\begin{array}{c}\hfill log\left(\frac{p_i}{1-p_i}\right)={\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+{\beta }_3{x}_{i3}={\beta }_1+{\beta }_2{x}_{i2}+{\beta }_3{x}_{i3},i=1,2,\cdots ,n.\end{array}$

We can obtain the value of $p_i$ from the equation,

$\begin{array}{c}\hfill p_i=\frac{e^{{\beta }_1+{\beta }_2{x}_{i2}+{\beta }_3{x}_{i3}}}{1+e^{{\beta }_1+{\beta }_2{x}_{i2}+{\beta }_3{x}_{i3}}},i=1,2,\cdots ,n.\end{array}$

Using two $z^{\prime }s$ for the second logistic model, let $z_{i1}=1$ and generate $z_{i2}$ from the Weibull distribution with a shape parameter 3.6 and scale parameter 1 for $i=1,2,\cdots ,n$. Let ${\theta }_1=-1$, ${\theta }_2=2$. Present the corresponding logistic model by

$\begin{array}{c}\hfill log\left(\frac{{\delta }_i}{1-{\delta }_i}\right)={\theta }_1{z}_{i1}+{\theta }_2{z}_{i2}={\theta }_1+{\theta }_2{z}_{i2},i=1,2,\cdots ,n.\end{array}$

we can obtain the value of ${\delta }_i$ from the equation,

$\begin{array}{c}\hfill {\delta }_i=\frac{e^{{\theta }_1+{\theta }_2{z}_{i2}}}{1+e^{{\theta }_1+{\theta }_2{z}_{i2}}},i=1,2,\cdots ,n.\end{array}$

Then, we can generate $Y_i$ from a Bernoulli distribution with probability ${\pi }_i=(1-{\delta }_i)p_i$, $i=1,2,\cdots ,n$. Repeat the above process n times to obtain a zero-inflated random sample $\mathit{D}$.

4.2. The Implementation of Performance Comparison

To guarantee that the sample size is large enough to make the machine learning method perform stable, we consider $n=5000$ in this simulation study. The ZIBer algorithm can reach the stop rule if 50 iterative runs are up or all the parameter differences between two adjacent iterations are less than ${10}^{-5}$. For the comparison purpose, we use $80\%$ or 4000 generated data to train the LightGBM and ANN models and obtain the MLEs of ZIBer model parameters. R codes were prepared to implement the prediction performance comparison. When the MLEs of the model parameters are obtained, the model global searches for the best cut point with the lowest error classification rate as the training ZIBer model. Because the separation of training and testing groups depends on random chance, we repeated the whole process 1000 times and calculated the average performance of accuracy, sensitivity, and specificity,

(22)$\begin{array}{cc}\hfill \mathrm{Accuracy}& =\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}},\hfill \end{array}$

(23)$\begin{array}{cc}\hfill \mathrm{Sensitivity}& =\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},\hfill \end{array}$

(24)$\begin{array}{cc}\hfill \mathrm{Specificity}=& \frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}},\hfill \end{array}$

which TP denotes true positive, FN denotes false negative, FP denotes false positive, and TN denotes true negative. The simulation results are given in Table 1. The dispersion of the metrics Accuracy, Sensitivity, and Specificity of 1000 repetitions is given in Figures 3–5.

Table 2 shows that no method dominates the others. These three methods have similar performance for Accuracy and Specificity. The differences between these three methods are almost ignored. However, we found that the ANN method has extreme results for Sensitivity and Specificity. The average sensitivity based on 1000 repetitions is nearly 0, and the average Specificity based on 1000 repetitions is 1. The finding means the ANN methods could not be suitable for the case, emphasizing the Sensitivity measure. The Sensitivity in zero-inflated-imbalanced data is not easy to measure correctly. The LightGBM and ZIBer methods outperform the ANN methods in Sensitivity and Specificity measures. We also find that the ZIBer and LighGBM methods are competitive.

From Figure 1, Figure 2 and Figure 3, we can see the spread of 1000 measures of Accuracy, Sensitivity, and Specificity, respectively. Figure 1 indicates the Accuracy of LightGBM has a higher median than the other two competitors with a shorter dispersion than the ZIBer method. Figure 2 shows the strength of the ZIBer method with a higher median than the other two competitors and a similar dispersion with the LightGBM method. Figure 3 shows the ZIBer method has the lowest median and is widespread compared with the other two competitors.

Let us revisit Examples 1 and 2, these two data sets can be zero-inflated because the proportions of non-diabetes and non-default payments are 66.8% and 64.4%, respectively. Because the simulation design used is the same and comparison results are similar, Example 2 will be kept for explanation. In practical applications, some non-default credit card users have extremely low default probability. They could have a different model structure from users that could be default but not default in the current state. The performance of the ANN, LightGBM, and ZIBer models is compared using model validation. Randomly splitting the illustrative data set into the training data set with 80% observations, and the remaining 20% observations comprise the testing data set. Meanwhile, the random sampling maintains the same proportion of default payments in the training data set as the illustrative data set. The training data set is used to train the ANN and LightGBM models and construct the ZIBer model. Then, the testing data set is used to evaluate the Accuracy, Sensitivity, and Specificity. The split operation is repeated 300 times. Accuracy, Sensitivity, and Specificity are evaluated using their mean values in 300 repetitions. Moreover, the standard deviations of these three models in 300 repetitions are computed, too.

All computation results are given in Table 2, which shows that these three models are competitive. No one model dominates the others. The LightGBM model has the highest Accuracy and Specificity. The ZIBer model has the highest Sensitivity. We also note that the ZIBer model and LightGBM are competitive for the Accuracy measurement. The mean Sensitivity of ANN is significantly lower than the other two competitors and has the largest standard deviation. We also find that the standard deviation of the Sensitivity of LightGBM is over two times higher than that of the ZIBer model.

Figure 4 and Figure 5 show the scatter plots of 300 Sensitivity measurements versus the Accuracy measurements and 300 Sensitivity measurements versus the Specificity measurements for the ANN, LightGBM, and ZIBer models. These two scatter plots show that the Accuracy, Sensitivity, and Specificity measures of the ZIBer model spread at the top position among all three methods. In this example, the ZIBer model looks to have a more stable performance than its competitors.

The ZIBer method is a statistical model-based method. Hence, the ZIBer method does not require big data to train models. When the sample size is small, the ZIBer method can be expected to perform best. The ZIBer and LightGBM methods are more competitive than the ANN, and we recommend using them to process zero-inflated-imbalanced data.

5. Conclusions

This paper proposes the EM maximum likelihood estimation procedure to obtain reliable MLEs of the ZIBer model parameters. The EM algorithm is simple for practical use. The machine learning models of ANN and LightGBM are used for comparing the performance of the proposed ZIBer model in terms of Accuracy, Sensitivity, and Specificity. Monte Carlo Simulations were conducted, and the simulation results show that no method can dominate the other two methods under the zero-inflated-imbalanced data.

Two examples of diabetes and credit card defaults are used for illustration. We demonstrate how to use the proposed EM algorithm to obtain the reliable parameters of the ZIBer regression model. Moreover, we use a random sampling technique for Example 2 to study the performance of the ANN, LightGBM, and ZIBer models under zero-inflated-imbalanced binary data sets in Section 4. We find that the LightGBM and ZIBer models are competitive. The LightGBM has the highest Accuracy and Specificity, and the ZIBer model has the highest Sensitivity. The ANN model performs worse in the measure of Sensitivity than the two competitors. Overall, the ZIBer model has a more stable performance than its competitors.

The strength of a parametric statistical model is the functional form is clear, and the explanation between the response and explanatory variables is feasible. However, screening out critical explanatory variables from a large set in the model is challenging. Enhancing the abilities of the model with higher sensitivity of the ZIBer model is also challenged. These two issues could be studied in the future.

Author Contributions

Conceptualization, investigation, writing and editing, project administration, funding acquisition: T.-R.T.; validation, investigation, writing and editing: Y.L.; methodology: C.-Y.H.; investigation: C.-L.H. and J.-Y.C.; project administration, funding acquisition: T.-R.T. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The research data can be free access by checking https://archive.ics.uci.edu/dataset/350/default+of+credit+card+clients, this data set was donated to UC Irvine Machine Learning Repository on 25 January 2016.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. The box plots of accuracy for the ANN, LightGBM, and ZIBer models.

Figure 2. The box plots of sensitivity for the ANN, LightGBM, and ZIBer models.

Figure 3. The box plots of Specificity for the ANN, LightGBM, and ZIBer models.

Figure 4. The scatter plot of the Accuracy versus Sensitivity for the ANN, LightGBM, and ZIBer models.

Figure 5. The scatter plot of the Specificity versus Sensitivity for the ANN, LightGBM, and ZIBer models.

Table 1

Performance comparison among ZIBer, LightGBM, and ANN based on 1000 repetitions.

	Average Values
Methods	Accuracy	Sensitivity	Specificity
ANN	0.9554	0.0010	1
LightGBM	0.9596	0.1082	0.9994
ZIBer	0.9563	0.1256	0.9952

Table 2

Performance comparison among the ANN, LightGBM, and ZIBer models based on 100 split operations for the illustrative data set.

	Mean Values (Standard Deviation*100)
Methods	Accuracy	Sensitivity	Specificity
ANN	0.7402 (3.9677)	0.5400 (21.2622)	0.8513 (6.5206)
LightGBM	0.7718 (1.4739)	0.6042 (7.4150)	0.8647 (3.6120)
ZIBer	0.7692 (1.4731)	0.7020 (3.0221)	0.8064 (2.4367)

Appendix A. R Codes for the EM Algorithm

Figure 5. The scatter plot of the Specificity versus Sensitivity for the ANN, LightGBM, and ZIBer models.

Figure 5. The scatter plot of the Specificity versus Sensitivity for the ANN, LightGBM, and ZIBer models.

Figure 5. The scatter plot of the Specificity versus Sensitivity for the ANN, LightGBM, and ZIBer models.

References

1. Rosenblatt, F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychol. Rev.; 1958; 65, pp. 386-408. [DOI: https://dx.doi.org/10.1037/h0042519] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/13602029]

2. Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning representations by back-propagating errors. Nature; 1986; 323, pp. 533-536. [DOI: https://dx.doi.org/10.1038/323533a0]

3. Hirose, Y.; Yamashita, K.; Hijiya, S. Back-propagation algorithm which varies the number of hidden units. Neural Netw.; 1991; 4, pp. 61-66. [DOI: https://dx.doi.org/10.1016/0893-6080(91)90032-Z]

4. Sietsma, J.; Dow, R.J. Creating artificial neural networks that generalize. Neural Netw.; 1991; 4, pp. 67-79. [DOI: https://dx.doi.org/10.1016/0893-6080(91)90033-2]

5. Devi, M.A.; Ravi, S.; Vaishnavi, J.; Punitha, S. Classification of cervical cancer using artificial neural networks. Procedia Comput. Sci.; 2016; 89, pp. 465-472. [DOI: https://dx.doi.org/10.1016/j.procs.2016.06.105]

6. Sze, V.; Chen, Y.H.; Yang, T.J.; Emer, J.S. Efficient processing of deep neural networks: A tutorial and survey. Proc. IEEE; 2017; 105, pp. 2295-2329. [DOI: https://dx.doi.org/10.1109/JPROC.2017.2761740]

7. Abiodun, O.I.; Jantan, A.; Omolara, A.E.; Dada, K.V.; Mohamed, N.A.; Arshad, H. State-of-the-art in artificial neural network applications: A survey. Heliyon; 2018; 4, e00938. [DOI: https://dx.doi.org/10.1016/j.heliyon.2018.e00938] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30519653]

8. Nasser, I.M.; Abu-Naser, S.S. Lung cancer detection using artificial neural network. Int. J. Eng.; 2019; 3, pp. 17-23.

9. Muhammad, W.; Hart, G.; Nartowt, B.; Farrell, J.J.; Johung, K.; Liang, Y.; Deng, J. Pancreatic cancer prediction through an artificial neural network. Front. Artif. Intell.; 2019; 2, 2. [DOI: https://dx.doi.org/10.3389/frai.2019.00002]

10. Umar, M.A. Student academic performance prediction using artificial neural networks: A case study. Int. J. Comput. Appl.; 2019; 178, pp. 24-29.

11. Shen, F.; Zhao, X.; Li, Z.; Li, K.; Meng, Z. A novel ensemble classification model based on neural networks and a classifier optimisation technique for imbalanced credit risk evaluation. Phys. A Stat. Mech. Its Appl.; 2019; 526, 121073. [DOI: https://dx.doi.org/10.1016/j.physa.2019.121073]

12. Lambert, D. Zero-inflated Poisson regression, with an application to defects in manufacturing. Technometrics; 1992; 34, pp. 1-14. [DOI: https://dx.doi.org/10.2307/1269547]

13. Hall, D.B. Zero-inflated Poisson and binomial regression with random effects: A case study. Biometrics; 2000; 56, pp. 1030-1039. [DOI: https://dx.doi.org/10.1111/j.0006-341X.2000.01030.x] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/11129458]

14. Cheung, Y.B. Zero-inflated models for regression analysis of count data: A study of growth and development. Stat. Med.; 2002; 21, pp. 1461-1469. [DOI: https://dx.doi.org/10.1002/sim.1088] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/12185896]

15. Gelfand, A.E.; Citron-Pousty, S. Zero-inflated models with application to spatial count data. Environ. Ecol. Stat.; 2002; 9, pp. 341-355.

16. Rodrigues, J. Bayesian analysis of zero-inflated distributions. Commun. Stat.-Theory Methods; 2003; 32, pp. 281-289. [DOI: https://dx.doi.org/10.1081/STA-120018186]

17. Ghosh, S.K.; Mukhopadhyay, P.; Lu, J.C. Bayesian analysis of zero-inflated regression models. J. Stat. Plan. Inference; 2006; 136, pp. 1360-1375. [DOI: https://dx.doi.org/10.1016/j.jspi.2004.10.008]

18. Harris, M.N.; Zhao, X. A zero-inflated ordered probit model, with an application to modelling tobacco consumption. J. Econom.; 2007; 141, pp. 1073-1099. [DOI: https://dx.doi.org/10.1016/j.jeconom.2007.01.002]

19. Loeys, T.; Moerkerke, B.; De Smet, O.; Buysse, A. The analysis of zero-inflated count data: Beyond zero-inflated Poisson regression. Br. J. Math. Stat. Psychol.; 2011; 65, pp. 163-180. [DOI: https://dx.doi.org/10.1111/j.2044-8317.2011.02031.x]

20. Diop, A.; Diop, A.; Dupuy, J.-F. Maximum likelihood estimation in the logistic regression model with a cure fraction. Electron. J. Stat.; 2011; 5, pp. 460-483.

21. Staub, K.E.; Winkelmann, R. Consistent estimation of zero-inflated count models. Health Econ.; 2012; 22, pp. 673-686. [DOI: https://dx.doi.org/10.1002/hec.2844] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22623339]

22. He, H.; Tang, W.; Wang, W.; Crits-Christoph, P. Structural zeroes and zero-inflated models. Shanghai Arch. Psychiatry; 2014; 26, pp. 236-242. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25317011]

23. Diop, A.; Diop, A.; Dupuy, J.-F. Simulation-based inference in a zero-inflated Bernoulli regression model. Commun. Stat.-Simul. Comput.; 2016; 45, pp. 3597-3614. [DOI: https://dx.doi.org/10.1080/03610918.2014.950743]

24. Alsabti, K.; Ranka, S.; Singh, V. CLOUDS: A decision tree classifier for large datasets. Electr. Eng. Comput. Sci.-All Scholarsh.; 1998; 41, Available online: https://surface.syr.edu/eecs/41 (accessed on 20 September 2010).

25. Friedman, J.H. Greedy function approximation: A gradient boosting machine. Ann. Stat.; 2001; 29, pp. 1189-1232. [DOI: https://dx.doi.org/10.1214/aos/1013203451]

26. Jin, R.; Agrawal, G. Communication and memory efficient parallel decision tree construction. Proceedings of the 2003 SIAM International Conference on Data Mining; San Francisco, CA, USA, 1–3 May 2003; [DOI: https://dx.doi.org/10.1137/1.9781611972733.11]

27. Chen, T.; Guestrin, C. XGBoost: A scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; San Francisco, CA, USA, 13–17 August 2016; pp. 785-794. [DOI: https://dx.doi.org/10.1145/2939672.2939785]

28. Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; Liu, T. LightGBM: A highly efficient gradient boosting decision tree. Neural Inf. Process. Syst.; 2017; 30, pp. 3149-3157.

29. Wang, D.; Zhang, Y.; Zhao, Y. LightGBM: An effective miRNA classification method in breast cancer patients. Proceedings of the 2017 International Conference on Computational Biology and Bioinformatics; New York, NY, USA, 18 October 2017; pp. 7-11.

30. Ma, X.; Sha, J.; Wang, D.; Yu, Y.; Yang, Q.; Niu, X. Study on a prediction of P2P network loan default based on the machine learning LightGBM and XGboost algorithms according to different high dimensional data cleaning. Electron. Commer. Res. Appl.; 2018; 31, pp. 24-39. [DOI: https://dx.doi.org/10.1016/j.elerap.2018.08.002]

31. Machado, M.R.; Karray, S.; De Sousa, I.T. LightGBM: An effective decision tree gradient boosting method to predict customer loyalty in the finance industry. Proceedings of the 14th International Conference on Computer Science & Education (ICCSE); Toronto, ON, Canada, 19–21 August 2019.

32. Minstireanu, E.A.; Mesnita, G. Light GBM machine learning algorithm to online click fraud detection. J. Inf. Assur. Cyber Secur.; 2019; 2019, 263928. [DOI: https://dx.doi.org/10.5171/2019.263928]

33. Daoud, E.A. Comparison between XGBoost, LightGBM and CatBoost using a home credit dataset. Int. J. Comput. Inf. Eng.; 2019; 13, pp. 6-10.

34. Krawczyk, B. Learning from imbalanced data: Open challenges and future directions. Prog. Artif. Intell.; 2016; 5, pp. 221-232. [DOI: https://dx.doi.org/10.1007/s13748-016-0094-0]

35. Nalepa, J.; Kawulok, M. Selecting training sets for support vector machines: A review. Artif. Intell. Rev.; 2019; 52, pp. 857-900. [DOI: https://dx.doi.org/10.1007/s10462-017-9611-1]