Entropic Statistics: Concept, Estimation, and Application in Machine Learning and Knowledge Extraction

Abstract

The demands for machine learning and knowledge extraction methods have been booming due to the unprecedented surge in data volume and data quality. Nevertheless, challenges arise amid the emerging data complexity as significant chunks of information and knowledge lie within the non-ordinal realm of data. To address the challenges, researchers developed considerable machine learning and knowledge extraction methods regarding various domain-specific challenges. To characterize and extract information from non-ordinal data, all the developed methods pointed to the subject of Information Theory, established following Shannon’s landmark paper in 1948. This article reviews recent developments in entropic statistics, including estimation of Shannon’s entropy and its functionals (such as mutual information and Kullback–Leibler divergence), concepts of entropic basis, generalized Shannon’s entropy (and its functionals), and their estimations and potential applications in machine learning and knowledge extraction. With the knowledge of recent development in entropic statistics, researchers can customize existing machine learning and knowledge extraction methods for better performance or develop new approaches to address emerging domain-specific challenges.

1. Introduction of Entropic Statistics

Entropic statistics is a collection of statistical procedures that characterize information from non-ordinal spaces with Shannon’s entropy and its generalized functionals. Such procedures includes but not limited to statistical methods involving Shannon’s entropy (entropy) and Mutual Information (MI) [1], Kullback–Leibler divergence (KL) [2], entropic basis and diversity index [3,4], and Generalized Shannon’s Entropy (GSE) and Generalized Mutual Information (GMI) [5]. The field of entropic statistics is at the intersection of information theory and statistics. Entropic statistics quantities are also referred as information-theoretic quantities [6,7].

There are two general data types—ordinal and non-ordinal (nominal). Ordinal data are data with an inherent numerical scale. For example, {52 F, 50 F, 49 F, 53 F}—a set of daily high temperatures at Nuuk, Greenland—is ordinal. Ordinal data are generated from random variables (which map outcomes from sample space to the real numbers). For ordinal data, classical concepts, such as moments (mean, variance, covariance, etc.) and characteristics functions, are powerful tools to induce various statistical methods, including but not limited to regression analysis [8] and analysis of variance (ANOVA) [9].

Non-ordinal data are data without an inherent numerical scale. For example, {androgen receptor, clock circadian regulator, epidermal growth factor, Werner syndrome RecQ helicase-like}—a subset of human genes names—is a set of data without inherent numerical scale. Non-ordinal data are generated from random elements (which map outcomes from sample space to alphabet). Due to the absence of inherent numerical scale, the concept of random variable is undefined according to its definition. Therefore, statistical concepts involving ordinal scale (e.g., mean, variance, covariance, and characteristic functions) no longer exist. For example, consider the mentioned data of human genes names; what is the mean or variance of the data? Such questions cannot be answered because the concepts of mean and variance do not exist, while in practice, researchers need to measure the level of dependence in non-ordinal joint space between gene types and genetic phenotype to study the gene’s functionalities. One would use covariance and its generated methods in ordinal data. However, the concept of covariance no longer exists in such non-ordinal space. Furthermore, all well-established statistical methods that require ordinal scale (e.g., regression and ANOVA) cannot be directly applied anymore.

Non-ordinal data have several variant names, such as categorical data, qualitative data, and nominal data. A common situation is a dataset is mixed with ordinal and non-ordinal data. On such a dataset, a common practice is to introduce coded (dummy) variables [10]. However, introducing dummy variables is equivalent to separating the mixed dataset according to the classes in non-ordinal variables to induce multiple purely ordinal subsets and then utilizing ordinal methods (such as regression analysis) case-by-case on the induced subsets. Unfortunately, this approach sometimes could be impractical because of the curse of dimensionality, particularly when there are too many categorical variables or when some categorical variable has too many categories (classes).

With the challenges from non-ordinal data, entropic statistics methods focus on underlying probability distribution instead of associated labels. As a result, all the entropic statistical quantities are location (permutation) invariant. The main strengths of entropic statistics lie within non-ordinal alphabets, or a mixture data space that significant bulk of information lies within the non-ordinal sub-space. For ordinal spaces, although ordinal variables can be binned as categorical variables, the strength of entropic statistics are generally incapable of overcoming the loss of ordinal information during discretization. Therefore, ordinal statistical methods are preferred when they are capable of the needs. In summary, potential scenarios for entropic statistics are:

• The data lie within non-ordinal space.

• The data are a mixture of ordinal and non-ordinal spaces, and the non-ordinal space is expected to carry unneglectable bulk of information.

• The data lie within ordinal space, yet the performance of ordinal statistics methods fails to meet the expectation.

The following notations are used throughout the article. They are listed here for convenience.

• Let $\mathcal{X}=\{x_i;i=1,2,\cdots \}$ and $\mathcal{Y}=\{y_j;j=1,2,\cdots \}$ be two countable alphabets with cardinalities $K_1\le \infty$ and $K_2\le \infty$, respectively.

• Let the Cartesian product $\mathcal{X}\times \mathcal{Y}$ be with a joint probability distribution ${\mathbf{p}}_{XY}=\left\{p_{i,j}\right\}$.

• Let the two marginal distributions be respectively denoted by ${\mathbf{p}}_X=\left\{p_{i,·}\right\}$ and ${\mathbf{p}}_Y=\left\{p_{·,j}\right\}$ where $p_{i,·}={\sum }_j{p}_{i,j}$ and $p_{·,j}={\sum }_i{p}_{i,j}$; hence X is a variable on $\mathcal{X}$ with distribution ${\mathbf{p}}_X$ and Y is a variable on $\mathcal{Y}$ with distribution ${\mathbf{p}}_Y$.

• For uni-variate situations, K stands for $K_1$, and $\mathbf{p}$ stands for ${\mathbf{p}}_X$.

• Let $\{X_1,X_2,\cdots ,X_n\}$ be an independent and identically distributed ($i.i.d.$) random sample of size n from $\mathcal{X}$. Let $C_r={\sum }_{i=1}^{n}1[X_i=l_r]$; hence $C_r$ is the count of occurrence of letter $l_r$ in a sample. Let $\hat{\mathbf{p}}=\{{\hat{p}}_1,{\hat{p}}_2,\cdots \}=\{C_1/n,C_2/n,\cdots \}$. $\hat{\mathbf{p}}$ is called the plug-in estimator of $\mathbf{p}$. Similarly, one can construct the plug-in estimators for ${\mathbf{p}}_Y$ and ${\mathbf{p}}_{XY}$ and name them as ${\hat{\mathbf{p}}}_Y$ and ${\hat{\mathbf{p}}}_{XY}$, respectively.

• For any two functions f and g taking values in $(0,\infty )$ with ${lim}_{n\to \infty }f\left(n\right)={lim}_{n\to \infty }$$g\left(n\right)=0$, the notation $f\left(n\right)=\mathcal{O}\left(g\right(n\left)\right)$ means

  $0<\underset{n\to \infty }{lim\; inf}\frac{g\left(n\right)}{f\left(n\right)}\le \underset{n\to \infty }{lim\; sup}\frac{g\left(n\right)}{f\left(n\right)}<\infty .$

• For any two functions f and g taking values in $(0,\infty )$, the notation $f\left(n\right)=\mathcal{O}\left(g\right(n\left)\right)$ means

  $\underset{n\to \infty }{lim}\frac{f\left(n\right)}{g\left(n\right)}=0.$

Many concepts discussed in the following sections have continuous counterparts under the same concept name. The results reviewed in this article focus on non-ordinal data space. Therefore, some notable results on ordinal space are not reviewed (for example, [11,12,13]). In Section 2, estimation on some classic entropic statistics quantities are discussed. Section 3 reviews estimation results and properties for some recently developed information-theoretic quantities. Entropic statistics’ application potentials in machine learning (ML) and knowledge extraction are discussed in Section 4. Finally, some remarks are given in Section 5.

2. Classic Entropic Statistics Quantities and Estimation

This section reviews three classic entropic concepts and their estimations, including Shannon’s entropy (Section 2.1.1) and mutual information (Section 2.1.2), and Kullback–Leibler divergence (Section 2.2). These three concepts are among the earliest entropic concepts and have been intensively studied over the past decades. Enormous amounts of statistical methods and computational algorithms are designed based on these three concepts [14,15,16]. Nevertheless, most of those methods and algorithms use naive plug-in estimation, which could be improved for a smaller estimation bias and better performance. For this reason, this section reviews several notable estimation methods as a reference. Some asymptotic properties are also presented as a reference. The asymptotic properties provide a theoretical guarantee for the corresponding estimators with statistical procedures such as hypothesis testing and confidence intervals.

2.1. Shannon’s Entropy and Mutual Information

2.1.1. Shannon’s Entropy

Established by Shannon in his landmark paper [1], the concept of entropy is the first and still the most important building brick in characterizing information from non-ordinal spaces. Many of the established information-theoretic quantities are linear functions of entropy. Shannon’s entropy, H, is defined as

$H=-\sum _i{p}_{i}ln{p}_i.$

Some remarkable properties of entropy are:

Property 1

(Entropy).

• 1.

  H is a measurement of dispersion. It is always non-negative by definition.

• 2.

  H = 0 if and only if the probability of a letter l in $\mathcal{X}$ is 1; hence no dispersion.

• 3.

  For a finite alphabet with cardinality K, H is bounded from the above by $lnK$, and the maximum is achieved when its distribution is uniform ($p_i=1/K,i=1,2,\cdots ,K$); hence maximum dispersion.

• 4.

  For a countably infinite alphabet, H may not exist (See Example 4 in Section 3).

Entropy Estimation-The Plug-in Estimator

Estimation of entropy has been a core research topic for decades. Due to the curse of “High Dimensionality” and “Discrete and Non-ordinal Nature”, entropy estimation is a technically difficult problem. Advances in this area have been slow to come. The plug-in estimator of entropy (also known as empirical entropy estimator), $\hat{H}$, defined as

$\hat{H}=-\sum _i{\hat{p}}_{i}ln{\hat{p}}_i,$

is inarguably the most naive entropy estimator. $\hat{H}$ has been studied thoroughly in recent decades. Ref. [17] provided the asymptotic properties for $\hat{H}$ when K is finite, namely,

Theorem 1

(Asymptotic property of $\hat{H}$ when K is finite).

$\sqrt{n}\left(\hat{H}-H\right)/\hat{\sigma }\stackrel{D}{⟶}N(0,1),$

where $\hat{\sigma }=\sqrt{{\sum }_i{\hat{p}}_i{ln}^2{\hat{p}}_i-{\hat{H}}^2}$.

Ref. [18] derived the bias of $\hat{H}$ for finite K

(1)$\mathrm{E}\left(\hat{H}\right)-H=-\frac{K-1}{2n}+\frac{1}{12n^2}\left(1-\sum _{i=1}^K\frac{1}{p_i}\right)+\mathcal{O}\left(n^{-3}\right).$

Ref. [19] derived the asymptotic properties for $\hat{H}$ when K is countable infinite. Namely,

Theorem 2

(Asymptotic property of $\hat{H}$ when K is countable infinite). For any nonuniform distribution $\left\{p_i;i\ge 1\right\}$ satisfying ${\sum }_i\left(p_i{ln}^2{p}_i\right)<\infty$, if there exists an integer-valued function $K\left(n\right)$ such that, as $n\to \infty$,

• 1.

  $K\left(n\right)\to \infty$,

• 2.

  $K\left(n\right)=\mathcal{O}\left(\sqrt{n}\right)$, and

• 3.

  $\sqrt{n}{\sum }_{i\ge K\left(n\right)}{p}_{i}ln{p}_i\to 0$;

then

$\sqrt{n}\left(\hat{H}-H\right)/\hat{\sigma }\stackrel{D}{⟶}N(0,1),$

where $\hat{\sigma }=\sqrt{{\sum }_k{\hat{p}}_k{ln}^2{\hat{p}}_k-{\hat{H}}^2}$.

As discussed in [19], the conditions with $K\left(n\right)$ hold if $p_i\sim 1/\left(i^2{ln}^{2}i\right)$; the conditions do not hold if $p_i\sim 1/\left(i^{2}lni\right)$.

Entropy Estimation-The Miller–Madow and Jackknife Estimators

${\hat{H}}_{MM}$[20] and ${\hat{H}}_{JK}$ [21] are two notable entropy estimators with bias adjustments. Namely,

(2)${\hat{H}}_{MM}=\hat{H}+\frac{\hat{K}-1}{2n},$

where $\hat{K}$ is the observed sample cardinality. For finite K, the bias of ${\hat{H}}_{MM}$ is

$\mathrm{E}\left({\hat{H}}_{MM}\right)-H=\mathcal{O}\left(n^{-2}\right).$

${\hat{H}}_{JK}$ is calculated in three steps:

• for each $i\in \{1,2,\cdots ,n\}$, construct ${\hat{H}}^{\left(i\right)}$, which is a plug-in estimator based on a sub-sample of size $n-1$ obtained by leaving the ith observation out;

• obtain ${\hat{H}}_{\left(i\right)}=n\hat{H}-(n-1){\hat{H}}^{\left(i\right)}$ for $i=1,\cdots ,n$; and then

• compute the jackknife estimator

  (3)${\hat{H}}_{JK}=\frac{{\sum }_{i=1}^n{\hat{H}}_{\left(i\right)}}{n}.$

Equivalently, (3) can be written as

${\hat{H}}_{JK}=n\hat{H}-(n-1)\frac{{\sum }_{i=1}^n{\hat{H}}^{\left(i\right)}}{n}.$

When $K<\infty$, it can be shown that the bias of ${\hat{H}}_{JK}$ is

$E\left({\hat{H}}_{JK}\right)-H=\mathcal{O}\left(n^{-2}\right).$

Asymptotic properties for ${\hat{H}}_{MM}$ and ${\hat{H}}_{JK}$ were derived in [22]. ${\hat{H}}_{MM}$ and ${\hat{H}}_{JK}$ reduce the rate of bias to a higher order power-decaying. Ref. [23] proved the convergence of $\hat{H}$ could be arbitrarily slow. Ref. [24] proved that for finite K, an unbiased estimator for entropy does not exist. As a result, it is only possible to reduce the bias to a smaller extent.

Entropy Estimation-The Z-Estimator

Recent studies on entropy estimation have reduced the bias to exponentially decaying. For example,

${\hat{H}}_z=\sum _{v=1}^{n-1}\left\{\frac{1}{v}\frac{n^{1+v}[n-(1+v)]!}{n!}\sum _i\left[{\hat{p}}_i\prod _{j=0}^{v-1}\left(1-{\hat{p}}_i-\frac{j}{n}\right)\right]\right\}$

is the entropy estimator provided in [25] with an exponentially decaying bias (Interested readers may refer to [26] for discussion on an entropy estimator that is algebraically equivalent to ${\hat{H}}_z$). Ref. [27] derived the asymptotic properties for ${\hat{H}}_z$. Namely,

Theorem 3

(Asymptotic property of ${\hat{H}}_z$ when K is finite).

$\sqrt{n}\left({\hat{H}}_z-H\right)/\hat{\sigma }\stackrel{D}{⟶}N(0,1),$

where $\hat{\sigma }=\sqrt{{\sum }_i{\hat{p}}_i{ln}^2{\hat{p}}_i-{\hat{H}}^2}$.

The following asymptotic properties for ${\hat{H}}_z$ when K is countable infinite were provided in [28].

Theorem 4

(Asymptotic property of ${\hat{H}}_z$ when K is countable infinite). For a nonuniform distribution $\left\{p_i;i\ge 1\right\}\in \mathcal{P}$ satisfying ${\sum }_i\left(p_i{ln}^2{p}_i\right)<\infty$, if there exists an integer-valued function $K\left(n\right)$ such that, as $n\to \infty$,

• 1.

  $K\left(n\right)\to \infty$,

• 2.

  $K\left(n\right)=o(\sqrt{n}/lnn)$, and

• 3.

  $\sqrt{n}{\sum }_{i\ge K\left(n\right)}{p}_{i}ln{p}_i\to 0$;

then

$\sqrt{n}\left({\hat{H}}_z-H\right)/\hat{\sigma }\stackrel{D}{⟶}N(0,1),$

where $\hat{\sigma }=\sqrt{{\hat{H}}_{2z}-{\hat{H}}_z^2}$ and ${\hat{H}}_{2z}={\sum }_i\left\{{\hat{p}}_i{\sum }_{v=1}^{n-n{\hat{p}}_i}\left[\left({\sum }_{s=1}^{v-1}\frac{1}{s(v-s)}\right){\prod }_{j=1}^v\left(1-\frac{n{\hat{p}}_i-1}{n-j}\right)\right]\right\}$.

The sufficient condition given in Theorem 4 for the normality of ${\hat{H}}_z$ is slightly more restrictive than that of the plug-in estimator $\hat{H}$ as stated in Theorem 2, and consequently supports a smaller class of distributions. The sufficient conditions of Theorem 4 still holds for $p_i=C_{\lambda }{i}^{-\lambda }$ where $\lambda >2$, but not for $p_i=$$C/\left(i^2{ln}^{2}i\right)$, which satisfies the sufficient conditions of Theorem 2. However, it is discussed in [28] that simulation results indicate that the asymptotic normality of ${\hat{H}}_z$ in Theorem 4 may still hold for $p_i=C/\left(i^2{ln}^{2}i\right)$ for $i\ge 1$ though not covered by the sufficient condition.

Remarks

Another perspective of entropy estimation is to combine ${\hat{H}}_z$ and ${\hat{H}}_{JK}$. Namely, one could use ${\hat{H}}_z$ in place of each ${\hat{H}}_{\left(i\right)}$ in (3). Interested readers may refer to [25] where a single layer combination of ${\hat{H}}_z$ and ${\hat{H}}_{JK}$ was discussed. In addition, ref. [29] presented a non-parametric entropy estimator (${\hat{H}}_{chao}$) when there are unseen species in the sample. ${\hat{H}}_{chao}$ has a smaller sample root mean squared error than ${\hat{H}}_{MM}$ and a smaller bias than ${\hat{H}}_{JK}$, according to the simulation study. Unfortunately, the bias decaying rate for ${\hat{H}}_{chao}$ was not theoretically offered. Based on their simulation study of ${\hat{H}}_{chao}$, it seems that the bias decaying rate is $\mathcal{O}(1/n^2)$, which is slower than ${\hat{H}}_z$. Asymptotic properties of ${\hat{H}}_{chao}$ are not developed in the literature.

There are several parametric entropy estimators for specific interests. For example, Dirichlet prior Bayesian estimator of entropy [30,31] and shrinkage estimator of entropy [32]. This review article focuses on results from non-parametric estimation methods. To conclude this section, a small scale comparison between $\hat{H}$ and ${\hat{H}}_z$ from [33] is provided in Table 1.

2.1.2. Mutual Information

In the same paper defining Shannon’s entropy, the concept of Mutual Information (MI) was also described [1]. Shannon’s entropies for $\mathcal{X}$, $\mathcal{Y}$, and $\mathcal{X}\times \mathcal{Y}$ are defined as

$\begin{array}{cc}\hfill H\left(X\right)& =-\sum _i{p}_{i,·}ln{p}_{i,·},\hfill \\ \hfill H\left(Y\right)& =-\sum _j{p}_{·,j}ln{p}_{·,j},\hfill \\ \hfill H(X,Y)& =-\sum _i\sum _j{p}_{i,j}ln{p}_{i,j},\hfill \end{array}$

and MI between $\mathcal{X}$ and $\mathcal{Y}$ is defined as

$MI(X,Y)=H\left(X\right)+H\left(Y\right)-H(X,Y).$

Some notable properties of MI are:

Property 2

(Mutual Information).

• 1.

  MI is a measurement of dependence. It is always non-negative by definition.

• 2.

  $MI=0$ if and only if the two marginals are independent.

• 3.

  $MI>0$ if and only if the two marginals are dependent.

• 4.

  A non-zero MI does not always indicate the degree (level) of dependence.

• 5.

  MI may not exist when the cardinality of joint space is countably infinite.

MI Estimation-The Plug-in Estimator and Z-Estimator

Since MI is a function of entropy, estimation of MI is essentially entropy estimation. Let $\{(X_1,Y_1),(X_2,Y_2),\cdots ,(X_n,Y_n)\}$ be an $i.i.d.$ random sample of size n from the joint alphabet $(\mathcal{X},\mathcal{Y})$. Based on the sample, plug-in estimators of the component entropy of MI can be obtained. Namely,

$\begin{array}{cc}\hfill \hat{H}\left(X\right)& =-\sum _i{\hat{p}}_{i,·}ln{\hat{p}}_{i,·},\hfill \\ \hfill \hat{H}\left(Y\right)& =-\sum _j{\hat{p}}_{·,j}ln{\hat{p}}_{·,j},\hfill \\ \hfill \hat{H}(X,Y)& =-\sum _i\sum _j{\hat{p}}_{i,j}ln{\hat{p}}_{i,j},\hfill \end{array}$

where ${\hat{p}}_{i,·}$ is the plug-in estimator for $p_{i,·}$, ${\hat{p}}_{·,j}$ is the plug-in estimator for $p_{·,j}$, and ${\hat{p}}_{i,j}$ is the plug-in estimator for $p_{i,j}$. Then the plug-in estimator of mutual information between $\mathcal{X}$ and $\mathcal{Y}$ is defined as

$\widehat{MI}(X,Y)=\hat{H}\left(X\right)+\hat{H}\left(Y\right)-\hat{H}(X,Y).$

With various entropy estimation methods, one could estimate MI by replacing $\hat{H}$ with a different entropy estimator. For example, using the entropy estimator with the fastest bias decaying rate, ${\hat{H}}_z$, the resulting estimator (${\widehat{MI}}_z$) also has a bias with an exponentially decaying rate [34], namely,

${\widehat{MI}}_z={\hat{H}}_z\left(X\right)+{\hat{H}}_z\left(Y\right)-{\hat{H}}_z(X,Y).$

The asymptotic properties for $\widehat{MI}$-s ($\widehat{MI}$ and ${\widehat{MI}}_z$) shall be discussed under two situations: (1) $MI=0$, and (2) $MI>0$.

The first situation of $MI=0$ is used for testing independence. For example, in feature selection, irrelevant (to the outcome) non-ordinal features shall be dropped, and a feature is irrelevant if it is independent of the outcome. Let A be the potential irrelevant feature and B be the outcome; hence one must test $H_0:MI(A,B)=0$ against $H_a:MI(A,B)>0$. To test such a hypothesis, one needs the asymptotic properties of $\widehat{MI}$-s under the null hypothesis: $MI=0$, derived in [35]. Namely,

Theorem 5

(Asymptotic properties of $\widehat{MI}$ and ${\widehat{MI}}_z$ when $MI=0$). Provided that $MI=0$,

$2n\widehat{MI}\stackrel{D}{⟶}{\chi }_{(K_1-1)(K_2-1)}^2$

and

$2n{\widehat{MI}}_z+(K_1-1)(K_2-1)\stackrel{D}{⟶}{\chi }_{(K_1-1)(K_2-1)}^2\prime$

where n is the sample size and ${\chi }_{(K_1-1)(K_2-1)}^2$ stands for chi-squared distribution with degrees of freedom $(K_1-1)(K_2-1)$.

For the second situation of $MI>0$ (recall that $MI>0$ if and only if the two marginals are dependent), the following asymptotic properties were due to [34].

Let

$\begin{array}{cc}\hfill \hat{v}& ={\left({\hat{p}}_1,\cdots ,{\hat{p}}_{K_1{K}_2-1}\right)}^{\tau }\hfill \\ & ={\left({\hat{p}}_{1,1},{\hat{p}}_{1,2},\cdots ,{\hat{p}}_{1,K_2},{\hat{p}}_{2,1},{\hat{p}}_{2,2},\cdots ,{\hat{p}}_{2,K_2},\cdots ,{\hat{p}}_{K_1,1},{\hat{p}}_{K_1,2},\cdots ,{\hat{p}}_{K_1,K_2-1}\right)}^{\tau }\hfill \end{array}$

be the enumeration of joint probabilities plug-in estimators. Let

$\Sigma \left(\hat{v}\right)=\left(\begin{array}{cccc}{\hat{p}}_1\left(1-{\hat{p}}_1\right)& -{\hat{p}}_1{\hat{p}}_2& \cdots & -{\hat{p}}_1{\hat{p}}_{K_1{K}_2-1}\\ -{\hat{p}}_2{\hat{p}}_1& {\hat{p}}_2\left(1-{\hat{p}}_2\right)& \cdots & -{\hat{p}}_2{\hat{p}}_{K_1{K}_2-1}\\ \cdots & \cdots & \cdots & \cdots \\ -{\hat{p}}_{K_1{K}_2-1}{\hat{p}}_1& -{\hat{p}}_{K_1{K}_2-1}{\hat{p}}_2& \cdots & {\hat{p}}_{K_1{K}_2-1}\left(1-{\hat{p}}_{K_1{K}_2-1}\right)\end{array}\right),$

and

$g\left(\hat{v}\right)=\left\{\begin{array}{cc}ln\left({\hat{p}}_{K_1,·}{\hat{p}}_{·,K_2}{\hat{p}}_k\right)-ln\left({\hat{p}}_{i,·}{\hat{p}}_{·,j}{\hat{p}}_{K_1,K_2}\right),\hfill & \mathrm{if}k’\mathrm{s}\mathrm{corresponding}\{i,j\}\mathrm{satisfying}\hfill \\ & i\ne K_1\mathrm{and}j\ne K_2\hfill \\ ln\left({\hat{p}}_{·,K_2}{\hat{p}}_k\right)-ln\left({\hat{p}}_{·,j}{\hat{p}}_{K_1,K_2}\right),\hfill & \mathrm{if}k’\mathrm{s}\mathrm{corresponding}\{i,j\}\mathrm{satisfying}\hfill \\ & i=K_1\mathrm{and}j\ne K_2\hfill \\ ln\left({\hat{p}}_{K_1,·}{\hat{p}}_k\right)-ln\left({\hat{p}}_{i,·}{\hat{p}}_{K_1,K_2}\right),\hfill & \mathrm{if}k’\mathrm{s}\mathrm{corresponding}\{i,j\}\mathrm{satisfying}\hfill \\ & i\ne K_1\mathrm{and}j=K_2\hfill \end{array}\right.,$

where $k\in \{1,2,\cdots ,K_1{K}_2-1\}$. Then,

Theorem 6

(Asymptotic properties of $\widehat{MI}$ and ${\widehat{MI}}_z$ when $MI>0$). Provided that $MI>0$,

$\sqrt{n}(\widehat{MI}-MI){\left[g^{\tau }\left(\hat{v}\right)\Sigma \left(\hat{v}\right)g\left(\hat{v}\right)\right]}^{-\frac{1}{2}}\stackrel{L}{\to }N(0,1)$

and

$\sqrt{n}\left({\widehat{MI}}_z-MI\right){\left[g^{\tau }\left(\hat{v}\right)\Sigma \left(\hat{v}\right)g\left(\hat{v}\right)\right]}^{-\frac{1}{2}}\to N(0,1).$

The following examples describe a proper use of MI and properties in Theorems 5 and 6.

Example 1

(Genes TMEM30A and MTCH2—data and descriptions are in Example 1 of [34]). In the example, data were from two different genes in 191 patients. It has been calculated in [34] that ${\widehat{MI}}_z=0.0552$. The hypothesis test in Example 1 of [35] gave a p-value of 0.0567, which suggests $MI=0$ at $\alpha =0.05$. However, one shall use the property in Theorem 6 to obtain a confidence interval of $MI$. One must not use the property in Theorem 5 for the purpose of the confidence interval in this situation (because the asymptotic distribution in Theorem 5 assumes a specific location for $MI$ under the null hypothesis).

Example 2

(Genes ENAH and ENAH—data and descriptions are in Example 2 of [34]). In the example, data were from different probes of the same genes on 191 patients. It has been calculated in [34] that ${\widehat{MI}}_z=0.1157$. The hypothesis test in Example 2 of [35] gave a p-value of 0.0012, which suggests $MI>0$ at $\alpha =0.05$. Furthermore, one shall use the property in Theorem 6 to obtain a confidence interval of $MI$. One must not use the property in Theorem 5 for the purpose of the confidence interval in this situation.

Example 3

(Compare the MI between Examples 1 and 2). From Examples 1 and 2, ${\widehat{MI}}_z(TME$$M30A,MTCH2)=0.0552$ and ${\widehat{MI}}_z(ENA{H}_1,ENA{H}_2)=0.1157$. Although the second estimation value is higher, one cannot conclude that the level of dependence between $ENA{H}_1$ and $ENA{H}_2$ is higher than that between TMEM30A and MTCH2 due to the limitation described in the 4-th property in Property 2. To compare the level of dependence, one shall refer to the standardized mutual information in Section 3.1.

Recall that MI is always non-negative. For the same reason, $\widehat{MI}$ is always non-negative (note that $\widehat{MI}$ can be viewed as the $MI$ for the distribution $\hat{\mathbf{p}}$). Nevertheless, ${\widehat{MI}}_z$ can be negative under some scenarios. A negative ${\widehat{MI}}_z$ suggests the level of dependence between the two random elements is extremely weak. If one uses the results from Theorem 5 to test if $H_0:MI=0$, a negative ${\widehat{MI}}_z$ would lead to a fail-to-reject for most settings of $\alpha$ (level of significance).

Remarks

There is another line of research on multivariate information-theoretic methods, the Partial Information Decomposition (PID) framework [36,37,38]. The PID may be viewed as a direct extension of MI to a measures of information provided by two or more variables about a third. Interesting applications of the PID are, for example, in explaining representation learning in neural networks [39] or in feature selection from dependent features [40]. PID aims to characterize redundancy with information decomposition. Another approach to characterize redundancy is to utilize MI on a joint feature space [33]. Additional research to compare the two approaches is needed.

2.2. Kullback–Leibler Divergence

Kullback–Leibler divergence (KL) [2], also known as relative entropy, is the distance between two probability distributions, introduced by [2], and is an important measure of information in information theory. The notations to define KL and describe its properties differ slightly from other sections. Let $P=\left\{p_k:k=1,\cdots ,K\right\}$ and $Q=\left\{q_k:k=1,\cdots ,K\right\}$ be two discrete probability distributions on the same finite alphabet, $\mathcal{X}=\left\{{\ell }_k:k=1,\cdots ,K\right\}$, where $K\ge 2$ is a finite integer. KL is defined to be

$KL=KL(P\parallel Q)=\sum _{k=1}^K{p}_{k}ln\left(p_k/q_k\right)=\sum _{k=1}^K{p}_{k}ln\left(p_k\right)-\sum _{k=1}^K{p}_{k}ln\left(q_k\right).$

Note that many also use D as the notation of KL, namely, $D(P\parallel Q)$. KL is not a metric since it does not satisfy the triangle inequality and is not symmetric. Some notable properties of KL are:

Property 3

(Kullback–Leibler divergence).

• 1.

  KL is a measurement of non-metric distance between two distributions on the same alphabet (with the same discrete support). It is always non-negative because of Gibbs’ inequality.

• 2.

  $KL=0$ if and only if the two underlying distributions are the same. Namely, $P=Q$ for each $k=1,\cdots ,K$.

• 3.

  $KL>0$ if and only if the two underlying distributions are different. Namely, $p_k\ne q_k$ for some k.

The use of KL has several variants, including but not limited to, (1) P and Q are unknown; (2) Q is known; (3) P and Q are continuous distributions. The second variant is an alternative method of the Pearson goodness-of-fit test. Interested readers may refer to [41] for more discussion on the second variant. Although utilizing entropic statistics on continuous spaces is generally not recommended, interested readers may refer to [42,43] for discussions on the third variant.

2.2.1. KL Point Estimation-The Plug-in Estimator, Augmented Estimator, and Z-Estimator

Although KL is not exactly a function of entropy, it still carries many similarities with entropy. For that reason, KL estimation is very similar to entropy estimation. For example, KL can be estimated from a plug-in perspective. Let ${\hat{p}}_k$ be the plug-in estimator of $p_k$ and ${\hat{q}}_k$ be the plug-in estimator of $q_k$, then the KL plug-in estimator is

$\widehat{KL}=\widehat{KL}(P\parallel Q)=\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{p}}_k\right)-\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{q}}_k\right).$

Because $\widehat{KL}$ could have an infinite bias [44], an augmented plug-in estimator of KL was presented in [44]:

${\widehat{KL}}^{*}=\sum _{i=1}^K{\hat{p}}_{k}ln\left({\hat{p}}_k\right)-\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{q}}_k^{*}\right),$

where

${\hat{q}}_k^{*}={\hat{q}}_k+\frac{1\left[{\hat{q}}_k=0\right]}{m},$

and m is the sample size of the sample from Q. The bias of $\widehat{KL}$ is no faster than $\mathcal{O}(1/n)$, where n is the sample size of the sample from P [44].

Since the $\widehat{KL}$ could have an infinite bias, its estimation in perspectives of ${\hat{H}}_{MM}$ or ${\hat{H}}_{JK}$ will not help in reducing the bias to a finite extent. In the perspective of ${\hat{H}}_z$, a KL estimator with exponentially decaying bias was offered in [44]:

$\begin{array}{c}\hfill {\widehat{KL}}_z={\widehat{KL}}_z(P\parallel Q)=\sum _{k=1}^K{\hat{p}}_k\left[\sum _{v=1}^{m-m{\hat{q}}_k}\frac{1}{v}\prod _{j=1}^v\left(1-\frac{m{\hat{q}}_k}{m-j+1}\right)-\sum _{v=1}^{n-n{\hat{p}}_k}\frac{1}{v}\prod _{j=1}^v\left(1-\frac{n{\hat{p}}_k-1}{n-j}\right)\right].\end{array}$

2.2.2. Symmetrized KL and Its Point Estimation

As mentioned in the first property of Property 3, KL is generally an asymmetric measurement. For certain interests that require a symmetric measurement, a symmetrized KL is defined to be

$\begin{array}{cc}\hfill S& =S(P,Q)=\frac{1}{2}[KL(P\parallel Q)+KL(Q\parallel P)]\hfill \\ \hfill & =\frac{1}{2}\left(\sum _{k=1}^K{p}_{k}ln\left(p_k\right)-\sum _{k=1}^K{p}_{k}ln\left(q_k\right)\right)+\frac{1}{2}\left(\sum _{k=1}^K{q}_{k}ln\left(q_k\right)-\sum _{k=1}^K{q}_{k}ln\left(p_k\right)\right).\hfill \end{array}$

The symmetrized KL S, as a function of $KL$, can be similarly estimated in the perspective of $\widehat{KL}$, ${\widehat{KL}}^{*}$, and ${\widehat{KL}}_z$. The respective estimators are

$\hat{S}=\frac{1}{2}\left(\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{p}}_k\right)-\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{q}}_k\right)\right)+\frac{1}{2}\left(\sum _{k=1}^K{\hat{q}}_{k}ln\left({\hat{q}}_k\right)-\sum _{k=1}^K{\hat{q}}_{k}ln\left({\hat{p}}_k\right)\right),$

${\hat{S}}^{*}=\frac{1}{2}\left(\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{p}}_k\right)-\sum _{k=1}^K{\hat{p}}_{k}ln\left({\hat{q}}_k^{*}\right)\right)+\frac{1}{2}\left(\sum _{k=1}^K{\hat{q}}_{k}ln\left({\hat{q}}_k\right)-\sum _{k=1}^K{\hat{q}}_{k}ln\left({\hat{p}}_k^{*}\right)\right),$

where ${\hat{p}}_k^{*}={\hat{p}}_k+1\left[{\hat{p}}_k=0\right]/n$ (n is the sample size of the sample from P), and

$\begin{array}{cc}\hfill {\hat{S}}_z=& \frac{1}{2}\sum _{k=1}^K{\hat{p}}_k\left[\sum _{v=1}^{m-m{\hat{q}}_k}\frac{1}{v}\prod _{j=1}^v\left(1-\frac{m{\hat{q}}_k}{m-j+1}\right)-\sum _{v=1}^{n-n{\hat{p}}_k}\frac{1}{v}\prod _{j=1}^v\left(1-\frac{n{\hat{p}}_k-1}{n-j}\right)\right]\hfill \\ \hfill & +\frac{1}{2}\sum _{k=1}^K{\hat{q}}_k\left[\sum _{v=1}^{n-n{\hat{p}}_k}\frac{1}{v}\prod _{j=1}^v\left(1-\frac{n{\hat{p}}_k}{n-j+1}\right)-\sum _{v=1}^{m-m{\hat{q}}_k}\frac{1}{v}\prod _{j=1}^v\left(1-\frac{m{\hat{q}}_k-1}{m-j}\right)\right].\hfill \end{array}$

2.2.3. Asymptotic Properties for KL and Symmetrized KL Estimators

The asymptotic properties for $\widehat{KL}$, ${\widehat{KL}}^{*}$, ${\widehat{KL}}_z$, $\hat{S}$, ${\hat{S}}^{*}$, and ${\hat{S}}_z$ are all presented in [44]. All the asymptotic properties therein require $P\ne Q$ (namely, $KL>0$). When $P=Q$, the asymptotic property of KL (or S) estimators are currently missing from the literature. The derivation of such asymptotic properties is not complicated yet unnecessary. The only purpose of such asymptotic property under $P=Q$ is to test if $H_0:P=Q$ against $H_a:P\ne Q$. For such a purpose, the two-sample goodness-of-fit chi-squared test can be used (see p. 616 in [45]).

3. Recently Developed Entropic Statistics Quantities and Estimation

In this section, various recently developed entropic statistics quantities are introduced and discussed. Some quantities are quite new with limited estimation properties developed other than the plug-in estimation. Therefore, some of the following discussions focus on conceptual spirits and application potentials.

3.1. Standardized Mutual Information

Mutual information between two random elements (on non-ordinal alphabets) is similar to the covariance between two random variables (on ordinal spaces) regarding properties and drawbacks. For example, the covariance does not provide general information on the degree of correlation, and the concept correlation of coefficient was defined to fill the gap. Similarly, recall the fourth property of MI that MI generally does not provide information about the degree of dependence, standardized mutual information (SMI), $\kappa$, has been studied and defined in various ways. To name a few, provided $H(X,Y)<\infty$,

(4)$\begin{array}{cc}\hfill {\kappa }_1& =\frac{MI(X,Y)}{H(X,Y)},\hfill \\ \hfill {\kappa }_2& =\frac{MI(X,Y)}{min\left\{H\right(X),H(Y\left)\right\}},\hfill \\ \hfill {\kappa }_3& =\frac{MI(X,Y)}{\sqrt{H\left(X\right)H\left(Y\right)}},\hfill \\ \hfill {\kappa }_4& =\frac{MI(X,Y)}{\left(H\right(X)+H(Y\left)\right)/2},\hfill \\ \hfill {\kappa }_5& =\frac{MI(X,Y)}{max\left\{H\right(X),H(Y\left)\right\}},\hfill \\ \hfill {\kappa }_6& =\frac{MI(X,Y)}{H\left(X\right)}.\hfill \end{array}$

The quantity ${\kappa }_6$ is also called information gain ratio [46]. The benefits of SMI are supported by Theorem 7.

Theorem 7

(Theorem 5.4 in [28]). Suppose $H(X,Y)<\infty$. Then

$0\le \kappa \le 1$

Moreover, (1) $\kappa =0$ if and only if X and Y are independent, and (2) $\kappa =1$ if and only if X and Y have a one-to-one correspondence.

Interested readers may refer to [47,48,49,50] for discussions on SMI. A detailed discussion of the estimation of various SMI may be found in [51].

3.2. Entropic Basis: A Generalization from Shannon’s Entropy

Shannon’s entropy and MI are powerful tools to quantify dispersion and dependence on non-ordinal space. More concepts and statistical tools are needed to characterize non-ordinal space information from different perspectives.

Generalized Simpson’s diversity indices were established in [52] and coined in [3].

Definition 1

(Generalized Simpson’s Diversity Indices). For a given $\mathbf{p}=\left\{p_k;k\ge 1\right\}$ and an integer pair $(u\ge 1,v\ge 0)$, let ${\zeta }_{u,v}={\sum }_{k\ge 1}{p}_k^u(1-$${\left.p_k\right)}^v$. Let

$\zeta =\left\{{\zeta }_{u,v};u\ge 1,v\ge 0\right\}$

be defined as the family of generalized Simpson’s diversity indices.

Generalized Simpson’s Diversity Indices are the foundation of entropic basis and entropic moment. Interested readers may refer to [53] for discussions on entropic moments and a goodness-of-fit test under permutation based on entropic moments. In estimating ${\zeta }_{u,v}$,

$z_{u,v}=\frac{[n-(u+v)]!n^{u+v}}{n!}\times \sum _{k\ge 1}\left\{1\left[{\hat{p}}_k\ge \frac{u}{n}\right]\left[\prod _{i=0}^{u-1}\left({\hat{p}}_k-\frac{i}{n}\right)\right]\left[\prod _{j=0}^{v-1}\left(1-{\hat{p}}_k-\frac{j}{n}\right)\right]\right\}$

was derived in [52], where n is the sample size; u and v are given constants; ${\hat{p}}_k$ is the sample proportion of the k-th letter(category); $1[·]$ stands for indicator function. $z_{u,v}$ is an uniformly minimum-variance unbiased estimator (UMVUE) of ${\zeta }_{u,v}$ for any combination of $(u,v)$ non-negative integers pair as long as $u+v\le n$, where n is the corresponding sample size.

Based on ${\zeta }_{u,v}$, ref. [3] defined the entropic basis.

Definition 2

(Entropic Basis). Given Definition 1, the entropic basis is the sub-family

${\zeta }_1=\{{\zeta }_{1,v};v\ge 0\}$

of ζ.

All diversity indices can be represented as a function of ${\zeta }_1$ [3] (most representations are due to Taylor’s expansion). For example,

• Simpson’s index [54]:

  $\lambda =\sum _{k\ge 1}{p}_k^2={\zeta }_{1,0}-{\zeta }_{1,1}$

• Gini–Simpson index [54,55]:

  $1-\lambda =\sum _{k\ge 1}{p}_k\left(1-p_k\right)={\zeta }_{1,1}$

• Shannon’s entropy:

  $H=-\sum _{k\ge 1}{p}_{k}ln\left(p_k\right)=\sum _{v=1}^{\infty }\frac{1}{v}{\zeta }_{1,v}$

• Rényi equiv. entropy [56]:

  $h_r=\sum _{k\ge 1}{p}_k^r={\zeta }_{1,0}+\sum _{v=1}^{\infty }\prod _{i=1}^v\left(\frac{i-r}{i}\right){\zeta }_{1,v}$

• Emlen’s index [57]:

  $D=\sum _{k\ge 1}{p}_k{e}^{-p_k}=\sum _{v=0}^{\infty }\frac{e^{-1}}{v!}{\zeta }_{1,v}$

• Richness index (population size):

  $K=\sum _{k\ge 1}1\left[p_k>0\right]=\sum _{v=0}^{\infty }{\zeta }_{1,v}$

• Generalized Simpson’s index:

  ${\zeta }_{u,m}=\sum _{k\ge 1}{p}_k^u{\left(1-p_k\right)}^m=\sum _{v=0}^{u-1}{(-1)}^v\left(\begin{array}{c}u-1\\ v\end{array}\right){\zeta }_{1,m+v}.$

In practice, plug-in estimation is used in estimating diversity indices. The representations of the diversity index on an entropic basis allow a new estimation method with a smaller bias. Namely, $z_{1,v}$, the UMVUE for ${\zeta }_{1,v}$, exist for all v up to $n-1$. If one replaces $\{{\zeta }_{1,0},{\zeta }_{1,1},\cdots ,{\zeta }_{1,n-1}\}$ with $\{z_{1,0},z_{1,1},\cdots ,z_{1,n-1}\}$, and let all the other $\zeta$s (namely, $\{{\zeta }_{1,n},{\zeta }_{1,n+1},\cdots \}$) to be zero, then the resulting estimator is exactly the same as plug-in estimator. However, the estimation can be further improved if one estimate $\{{\zeta }_{1,n},{\zeta }_{1,n+1},\cdots \}$ based on $\{z_{1,0},z_{1,1},\cdots ,z_{1,n-1}\}$.

For example, let $\hat{K}$ (the observed number of categories) be the plug-in estimator of K. Meanwhile, ${\sum }_{v=0}^{n-1}{z}_{1,v}$ (the estimator in perspective of entropic basis representation) is algebraically equivalent to $\hat{K}$ [58]. Namely,

$K=\sum _{k\ge 1}1\left[p_k>0\right]=\sum _{v=0}^{\infty }{\zeta }_{1,v}=\sum _{v=0}^{n-1}{\zeta }_{1,v}+\sum _{v=n}^{\infty }{\zeta }_{1,v}$

is a decomposition of K, then

$\hat{K}=\sum _{k\ge 1}1\left[{\hat{p}}_k>0\right]=\sum _{v=0}^{n-1}{\zeta }_{1,v};$

whereas

${\hat{K}}_{entropic}=\sum _{v=0}^{n-1}{\zeta }_{1,v}+\sum _{v=n}^{\infty }{\zeta }_{1,v}=\hat{K}+\mathrm{estimator}\mathrm{of}{\sum }_{v=n}^{\infty }{\zeta }_{1,v}.$

${\hat{K}}_{entropic}$ has a smaller bias than $\hat{K}$. Interested readers may refer to [58] for details on the estimation of ${\sum }_{v=n}^{\infty }{\zeta }_{1,v}$. Similar estimation could benefit the estimation of Rényi equiv. entropy, Emlen’s index, and any other diversity indices or theoretical quantities which contain the terms ${\sum }_{v=n}^{\infty }{\zeta }_{1,v}$ after Taylor’s expansion.

3.3. Generalized Shannon’s Entropy and Generalized Mutual Information

Because of the advantages in characterizing information in non-ordinal space, Shannon’s entropy and MI have become the building blocks of information theory and essential aspects of ML methods. Yet, they are only finitely defined for distributions with fast decaying tails on a countable alphabet.

Example 4

(Unbounded entropy). Let $X=\{2,3,4,\cdots \}$ be a random variable following the distribution $P(X=k)=p_k=c/\left(k{ln}^{2}k\right)$, where $k=2,3,4,\cdots$, and c is the constant to make the distribution valid (total probability add up to 1). Such a constant uniquely exists because the summation ${\sum }_{i=2}^{\infty }1/\left(i{ln}^{2}i\right)$ converges. Then

$\begin{array}{cc}\hfill H\left(X\right)& =\sum _{k=2}^{\infty }{p}_{k}ln\frac{1}{p_k}=\sum _{k=2}^{\infty }\left(\frac{c}{k{ln}^{2}k}ln\frac{k{ln}^{2}k}{c}\right)\hfill \\ \hfill & =\sum _{k=2}^{\infty }\left(\frac{c}{k{ln}^{2}k}(lnk+2lnlnk-lnc)\right)\hfill \\ \hfill & =\sum _{k=2}^{\infty }\frac{c}{klnk}+\sum _{k=2}^{\infty }\left(\frac{c(2lnlnk-lnc)}{k{ln}^{2}k}\right)=\infty ,\hfill \end{array}$

because

$\sum _{k=2}^{\infty }\frac{c}{klnk}=\infty$

and

$\sum _{k=2}^{\infty }\left(\frac{c(2lnlnk-lnc)}{k{ln}^{2}k}\right)<\infty .$

Therefore the entropy $H\left(X\right)$ is unbounded.

The unboundedness of Shannon’s entropy and MI over the general class of all distributions on an alphabet prevents their potential utility from being fully realized. Ref. [5] proposed GSE and GMI, which are finitely defined everywhere. To state the definition of GSE and GMI, Definition 3 is stated first.

Definition 3

(Conditional Distribution of Total Collision (CDOTC)). Given $\mathcal{X}=\left\{x_i;i\ge 1\right\}$ and $\mathbf{p}=\left\{p_i\right\}$, consider the experiment of drawing an identically and independently distributed (iid) sample of size m ($m\ge 2)$. Let $C_m$ denote the event that all observations of the sample take on the same letter in $\mathcal{X}$, and let $C_m$ be referred to as the event of a total collision. The conditional probability, given $C_m$, that the total collision occurs at the letter $x_i$ is

$p_{m,i}=\frac{p_i^m}{{\sum }_{s\ge 1}{p}_s^m},$

where $m\ge 2$. ${\mathbf{p}}_m=\left\{p_{m,i}\right\}$ is defined as the m-th order CDOTC.

The idea of CDOTC is to adopt a special member of the family of the escort distributions introduced in [59]. The utility of CDOTC is endorsed by Lemmas 1 and 2, which are proved in [5].

Lemma 1.

For any order m, $\mathbf{p}$ and ${\mathbf{p}}_m$ uniquely determine each other.

Lemma 2.

For any order m, ${\mathbf{p}}_{XY}=\left\{p_{i,j}\right\}=\{p_{i,.}\times p_{.,j}\}$ if and only if ${\mathbf{p}}_{XY,m}=\left\{p_{m,i,j}\right\}=\{p_{m,i,.}\times p_{m,.,j}\}$.

It is clear that ${\mathbf{p}}_m$ is a probability distribution induced from $\mathbf{p}=\left\{p_k\right\}$. An example is provided to help understand Definition 3.

Example 5

(The second-order CDOTC). Let $\mathbf{p}=\left\{p_i\right\}=\{6i^{-2}/{\pi }^2;i=1,2,3,\cdots \}$, the second-order CDOTC is then defined as

${\mathbf{p}}_2=\left\{p_{2,i}\right\},$

where

$p_{2,i}=\frac{p_i^2}{{\sum }_{s\ge 1}{p}_s^2}=\frac{36i^{-4}/{\pi }^4}{{\sum }_{s\ge 1}\left[36s^{-4}/{\pi }^4\right]}=\frac{i^{-4}}{{\sum }_{s\ge 1}{s}^{-4}}$

for $i=1,2,3,\cdots$.

Based on Definition 3, GSE and GMI are defined as follows.

Definition 4

(Generalized Shannon’s Entropy (GSE)). Given $\mathcal{X}=\left\{x_i;i\ge 1\right\}$, ${\mathbf{p}}_X=\left\{p_i\right\}$, and ${\mathbf{p}}_{X,m}=\left\{p_{m,i}\right\}$, generalized Shannon’s entropy (GSE) is defined as

$H_m\left(X\right)=-\sum _{i\ge 1}{p}_{m,i}ln{p}_{m,i},$

where $p_{m,i}$ is defined in Definition 3, and $m=2,3,\cdots$ is the order of GSE. GSE with order m is called the m-th order GSE.

Definition 5

5(Generalized Mutual Information (GMI)). Let $\mathcal{X}=\left\{x_i;i\ge 1\right\}$ and $\mathcal{Y}=\left\{y_j;j\ge 1\right\}$. Let ${\mathbf{p}}_{XY}=\left\{p_{i,j}\right\}$ and ${\mathbf{p}}_{XY,m}=\left\{p_{m,i,j}\right\}$. Let $X^{*}$ and $Y^{*}$ be the marginal distributions of ${\mathbf{p}}_{XY,m}$. The m-th order generalized mutual information (GMI) between X and Y is defined as

$M{I}_m(X,Y)=H\left(X^{*}\right)+H\left(Y^{*}\right)-H_m(X,Y);$

or equivalently

$M{I}_m(X,Y)=H\left(X^{*}\right)+H\left(Y^{*}\right)-H(X^{*},Y^{*});$

To help understand Definitions 4 and 5, Examples 6 and 7 are provided as follows.

Example 6

(The second-order GSE). Let $\mathcal{X}=\left\{x_i;i\ge 1\right\}$ and $\mathbf{p}=\left\{p_i\right\}=\{6i^{-2}/{\pi }^2;i=1,2,3,\cdots \}$. The second-order GSE, $H_2\left(X\right)$, is then defined as

$H_2\left(X\right)=-\sum _{i\ge 1}{p}_{2,i}ln{p}_{2,i},$

where

$p_{2,i}=\left\{\frac{i^{-4}}{{\sum }_{s\ge 1}{s}^{-4}};i=1,2,\cdots \right\}$

is given in Example 5.

Example 7

(The second-order GMI). Let $\mathcal{X}=\left\{x_{i,.};i\ge 1\right\}$, $\mathcal{Y}=\left\{x_{.,j};j\ge 1\right\}$, and $\mathcal{X}\times \mathcal{Y}=\left\{x_{i,j};i\ge 1,j\ge 1\right\}$. Let

${\mathbf{p}}_{XY}=\left\{p_{i,j}\right\}=\left\{\frac{24{\left[{(i+j-1)}^2+i-j+1\right]}^{-2}}{{\pi }^2};i\ge 1,j\ge 1\right\},$

and

${\mathbf{p}}_{XY,2}=\left\{p_{2,i,j}\right\}=\left\{\frac{p_{i,j}^2}{{\sum }_{s\ge 1,t\ge 1}{p}_{s,t}^2};i\ge 1,j\ge 1\right\}.$

Further, let

${\mathbf{p}}_{X^{*}}=\left\{\sum _j{p}_{2,i,j};i\ge 1\right\}$

and

${\mathbf{p}}_{Y^{*}}=\left\{\sum _i{p}_{2,i,j};j\ge 1\right\}.$

The second-order GMI, $M{I}_2(X,Y)$, is then defined as

$M{I}_2(X,Y)=H\left(X^{*}\right)+H\left(Y^{*}\right)-H(X^{*},Y^{*}),$

where $H\left(X^{*}\right)$, $H\left(Y^{*}\right)$, and $H(X^{*},Y^{*})$ are Shannon’s entropy based on ${\mathbf{p}}_{X^{*}}$, ${\mathbf{p}}_{Y^{*}}$, and ${\mathbf{p}}_{XY,2}$, respectively.

GSE’s and GMI’s plug-in estimators are stated in Definitions 6 and 7.

Definition 6

(GSE’s plug-in estimators). Let $X_1,X_2,\cdots ,X_n$ be i.i.d. random variables taking values in $\mathcal{X}=\left\{x_i;i\ge 1\right\}$ with distribution ${\mathbf{p}}_X$. The plug-in estimator for ${\mathbf{p}}_X$ is ${\hat{\mathbf{p}}}_X=\left\{{\hat{p}}_i\right\}$. The plug-in estimator for the m-th order GSE, ${\hat{H}}_m\left(X\right)$, is

$\begin{array}{cc}\hfill {\hat{H}}_m\left(X\right)=& -\sum _{i\ge 1}\left[{\hat{p}}_{m,i}ln{\hat{p}}_{m,i}\right]\hfill \\ \hfill =& -\sum _{i\ge 1}\left[\frac{{\hat{p}}_i^m}{{\sum }_{s\ge 1}{\hat{p}}_s^m}ln\frac{{\hat{p}}_i^m}{{\sum }_{s\ge 1}{\hat{p}}_s^m}\right].\hfill \end{array}$

Definition 7

(GMI’s plug-in estimators). Let $(X_1,Y_1),(X_2,Y_2),\cdots ,(X_n,Y_n)$ be i.i.d. random variables taking values in $\mathcal{X}\times \mathcal{Y}=\left\{(x_i,y_j);i\ge 1,j\ge 1\right\}$ with distribution ${\mathbf{p}}_{XY}=\left\{p_{i,j}\right\}$. Let ${\hat{\mathbf{p}}}_{XY}=\left\{{\hat{p}}_{i,j}\right\}$ be the plug-in estimator of ${\mathbf{p}}_{XY}$. The plug-in estimator for the m-th order GMI, ${\widehat{MI}}_m(X,Y)$, is

${\widehat{MI}}_m(X,Y)=\hat{H}\left(X^{*}\right)+\hat{H}\left(Y^{*}\right)-\hat{H}(X^{*},Y^{*}),$

where

$\begin{array}{cc}\hfill \hat{H}\left(X^{*}\right)=& -\sum _{i\ge 1}\left[\left(\sum _{j\ge 1}\frac{{\hat{p}}_{i,j}^m}{{\sum }_{s\ge 1,t\ge 1}{\hat{p}}_{s,t}^m}\right)ln\left(\sum _{j\ge 1}\frac{{\hat{p}}_{i,j}^m}{{\sum }_{s\ge 1,t\ge 1}{\hat{p}}_{s,t}^m}\right)\right],\hfill \\ \hfill \hat{H}\left(Y^{*}\right)=& -\sum _{j\ge 1}\left[\left(\sum _{i\ge 1}\frac{{\hat{p}}_{i,j}^m}{{\sum }_{s\ge 1,t\ge 1}{\hat{p}}_{s,t}^m}\right)ln\left(\sum _{i\ge 1}\frac{{\hat{p}}_{i,j}^m}{{\sum }_{s\ge 1,t\ge 1}{\hat{p}}_{s,t}^m}\right)\right],\hfill \\ \hfill \hat{H}(X^{*},Y^{*})=& -\sum _{i\ge 1,j\ge 1}\left[\frac{{\hat{p}}_{i,j}^m}{{\sum }_{s\ge 1,t\ge 1}{\hat{p}}_{s,t}^m}ln\frac{{\hat{p}}_{i,j}^m}{{\sum }_{s\ge 1,t\ge 1}{\hat{p}}_{s,t}^m}\right].\hfill \end{array}$

The following asymptotic properties for GSE’s plug-in estimators are given in [60].

Theorem 8.

Let ${\mathbf{p}}_X=\{p_k;k\ge 1\}$ be a probability distribution on a countably infinite alphabet $\mathcal{X}$, without any further conditions,

$\sqrt{n}\left(\frac{{\hat{H}}_m\left(X\right)-H_m\left(X\right)}{{\hat{\sigma }}_m}\right)\stackrel{d}{\to }N(0,1),$

where

(5) ${\hat{\sigma }}_m^2=\sum _{k=1}^{\infty }{\left[\frac{m^2}{{\hat{p}}_k}\left({\hat{p}}_{m,k}ln{\hat{p}}_{m,k}+{\hat{p}}_{m,k}{\hat{H}}_m\left(Z\right)\right)\right]}^2.$

Theorem 9.

Let ${\mathbf{p}}_X=\{p_k;k=1,2,\cdots ,K\}$ be a non-uniform probability distribution on a countably finite alphabet $\mathcal{X}$, without any further conditions,

$\sqrt{n}\left(\frac{{\hat{H}}_m\left(X\right)-H_m\left(X\right)}{{\hat{\sigma }}_m}\right)\stackrel{d}{\to }N(0,1),$

where

${\hat{\sigma }}_m^2=\sum _{k=1}^K{\left[\frac{m^2}{{\hat{p}}_k}\left({\hat{p}}_{m,k}ln{\hat{p}}_{m,k}+{\hat{p}}_{m,k}{\hat{H}}_m\left(Z\right)\right)\right]}^2.$

The properties in Theorems 8 and 9 allow interval estimation and hypothesis testing with ${\hat{H}}_m$. The advantage of shifting the original distribution to an escort distribution is reflected in Theorem 8-the asymptotic normality requires no assumption on a countable infinite alphabet. Theorem 9 can be viewed as a special case of Theorem 8 under a finite situation, where the uniform distribution shall be omitted because a uniform distribution has no variation between different category probabilities and hence results in a zero GSE and degenerate asymptotic distribution.

Nevertheless, suppose one is certain that the cardinality of distribution is finite. In that case, one shall use Shannon’s entropy instead of GSE because Shannon’s entropy always exists under a finite distribution. There are various well-studied estimation methods with Shannon’s entropy (whereas only plug-in estimation on GSE has been studied by far).

Asymptotic properties for GMI plug-in estimator have not been studied yet. Nonetheless, a test of independence with modified GMI [61] has been studied. The test does not require the knowledge of the number of columns or rows of a contingency table; hence it yielded an alternative other than Pearson’s chi-squared test of independence, particularly when a contingency table is large or sparse.

4. Application of Entropic Statistics in Machine Learning and Knowledge Extraction

Applications of entropic statistics in ML and knowledge extraction can be clustered in two directions. The first direction is to solve an existing question from a new perspective by creating a new information-theoretic quantity [61] or revisiting an existing information-theoretic quantity for additional insights [62]. The second direction is to use different estimation methods in existing methods to improve the performance by reducing bias and/or variation [32]. Application potentials in the second direction are very promising because theoretical results from recent-developed estimation methods suggest the performance of many existing ML methods could be improved, yet not much research has been conducted in the direction. In this section, many established ML and knowledge extraction methods are discussed with their potential to improve in the second direction.

4.1. An Entropy-Based Random Forest Model

Ref. [63] proposed an entropy-importance-based random forest model for power quality feature selection and disturbance classification. The method used a greedy search based on entropy and information gain for node segmentation. Nevertheless, only the plug-in estimation of entropy and information gain was considered. The method could be improved by replacing the plug-in estimation with smaller bias estimation methods, such as ${\hat{H}}_z$ in [25]. Further, one can also combine ${\hat{H}}_z$ with the jackknife procedure in (3) to obtain

${\hat{H}}_{zJK}=\frac{{\sum }_{i=1}^n{\hat{H}}_{z\left(i\right)}}{n},$

where ${\hat{H}}_{z\left(i\right)}=n{\hat{H}}_z-(n-1){\hat{H}}_z^{\left(i\right)}$, and use ${\hat{H}}_{zJK}$ in place of the adopted plug-in estimation. The benefit of using ${\hat{H}}_{zJK}$ is the potential smaller bias, and variance [25]. However, asymptotic properties for ${\hat{H}}_{zJK}$ are yet developed. When asymptotic properties are desired (e.g., for confidence interval or hypothesis testing purposes), one shall consider estimators with established asymptotic properties (also called theoretical guarantee), such as $\hat{H}$, ${\hat{H}}_{MM}$, ${\hat{H}}_{JK}$, and ${\hat{H}}_z$.

4.2. Feature Selection Methods

In [16,64], various information-theoretic feature selection methods were reviewed and discussed. The two review articles did not mention that all the discussed methods adopted plug-in estimators for the corresponding information-theoretic quantities. Improving the performance with different estimation methods is possible, and investigation is needed. For example, some of the discussed methods are summarized in Table 2 with suggestions to utilize smaller bias and/or variance estimation methods.

4.3. A Keyword Extraction Method

Ref. [79] proposed a keyword extraction method with Rényi’s entropy

$S_R(w,q)=\frac{1}{1-q}{log}_2\sum _{i=1}^{F_w}{p}_i^q,$

and used the plug-in estimator therein. Namely, ${\hat{S}}_R(w,q)=\frac{1}{1-q}{log}_2{\sum }_{i=1}^{F_w}{\hat{p}}_i^q$. Nevertheless, $S_R(w,q)$ can be represented as

$\begin{array}{cc}\hfill S_R(w,q)=& \frac{1}{1-q}{log}_2\left[{\zeta }_{1,0}+\sum _{v=1}^{\infty }\left(\prod _{j=1}^v\left(\frac{j-q}{j}\right){\zeta }_{1,v}\right)\right]\hfill \\ \hfill =& \frac{1}{1-q}{log}_2\left[{\zeta }_{1,0}+\sum _{v=1}^{n-1}\left(\prod _{j=1}^v\left(\frac{j-q}{j}\right){\zeta }_{1,v}\right)+\sum _{v=n}^{\infty }\left(\prod _{j=1}^v\left(\frac{j-q}{j}\right){\zeta }_{1,v}\right)\right],\hfill \end{array}$

where ${\zeta }_{1,v}$ has the UMVUE $z_{1,v}$ for $v=0,1,2,\cdots ,n-1$. For $v\ge n$, ${\zeta }_{1,v}$ could be estimated based on regression analysis [58]. Hence, $S_R(w,q)$ can be estimated as

${\hat{S}}_{Rz}(w,q)=\frac{1}{1-q}{log}_2\left[z_{1,0}+\sum _{v=1}^{n-1}\left(\prod _{j=1}^v\left(\frac{j-q}{j}\right)z_{1,v}\right)+\sum _{v=n}^{\infty }\left(\prod _{j=1}^v\left(\frac{j-q}{j}\right)z_{1,v}^{*}\right)\right],$

where the construction of $z_{1,v}^{*}$ needs investigation using regression analysis [58]. The resulting ${\hat{S}}_{Rz}(w,q)$ would have a smaller bias than that of ${\hat{S}}_R(w,q)$ to help improve the established keyword extraction method. Note that if one wishes to use $z_{1,v}$ up to $n-1$ only, the resulting estimator would become

(6)${\hat{S}}_{Rz}^{*}(w,q)=\frac{1}{1-q}{log}_2\left[z_{1,0}+\sum _{v=1}^{n-1}\left(\prod _{j=1}^v\left(\frac{j-q}{j}\right)z_{1,v}\right)\right].$

(6) is the same as ${\hat{H}}_r^{♯}\left(n\right)$ in [3]. Asymptotic properties for ${\hat{H}}_r^{♯}\left(n\right)$ were provided therein (Corollary 3 in [3]) for interested readers.

5. Conclusions

Entropic statistics is effective in characterizing information from non-ordinal space. Meanwhile, it is essential to realize that non-ordinal information is inherently difficult to identify due to its non-ordinal and permutation invariant nature. This survey article aims to provide a comprehensive review of recent advances in entropic statistics, including classic entropic concepts estimation, recent-developed entropic statistics quantities, and their applications potentials in ML and knowledge extraction. This article first introduces the concept of entropic statistics and emphasizes challenges from non-ordinal data. Then this article reviews the estimation for classic entropic quantities. These classic entropic concepts, including Shannon’s entropy, MI, and KL, are widely used in established machine learning and knowledge extraction methods. Most, if not all, of the established methods use plug-in estimation, which is computation efficient yet with a large bias. The surveyed different estimation methods would help researchers to potentially improve existing methods’ performance by adopting a different estimation method or adding theoretical guarantee to the existing methods. Recent-developed entropic statistics concepts are also reviewed with their estimation and applications. These new concepts not only allow researchers to estimate existing quantities in a new perspective, but also support additional aspects in characterizing non-ordinal information. In particular, the generalized Simpson’s diversity indices (with the induced entropic basis and entropic moments) have significant application and theoretical potential to either customize existing ML and knowledge extraction methods or to establish new methods considering domain-specific challenges. Further, this article provides some examples of how to apply the surveyed results to some of the existing methods, including a random forest model, fourteen feature selection methods, and a keyword extraction model. It should be mentioned that the aim of the survey is not to claim the superiority of some estimation methods over others but to provide a comprehensive list of recent advances in entropic statistics research. Specifically, although an estimator with a faster-decaying bias seems theoretically preferred, it has a longer calculation time even with the convenient R functions, particularly when multiple layers of jackknife (bootstrap) are involved. The preference of estimation varies case by case—some may prefer an estimator with a smaller bias, some may prefer one with a smaller variance, while some may need a trade-off between them. Furthermore, the article focuses on non-parametric estimation, while parametric estimation would perform better if the specified model fits the domain-specific reality. In summary, one should always investigate if a new estimation method fits the needs.

Enormous additional works are still needed in entropic statistics. For example, (1) the asymptotic properties for many established estimators (such as ${\hat{H}}_{chao}$ and ${\hat{H}}_{zJK}$) are not clear when cardinality is infinite. (2) With the transition from original distribution to escort distribution, GSE and GMI fill the void left by Shannon’s entropy and MI. However, only plug-in estimations of GSE and GMI have been studied. The biases of these plug-in estimators have not been studied, and additional estimation methods are undoubtedly needed. (3) Calculations for many entropic statistics are not yet supported in R, such as entropic basis, GSE, and GMI. Furthermore, more work is needed to implement the new entropic statistics concepts in programming software other than R (some of the reviewed estimators are implemented in R and are listed in Appendix A as a reference), particularly in Python. With additional theoretical development and application support, entropic statistics methods would be a more efficient tool to characterize more non-ordinal information and better serve the demands arose from the emerging domain-specific challenges.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

ANOVA	Analysis of Variance
GSE	Generalized Shannon’s Entropy
GMI	Generalized Mutual Information
i.i.d.	independent and identically distributed
KL	Kullback–Leibler Divergence
MI	Mutual Information
ML	Machine Learning
PID	Partial Information Decomposition
SMI	Standardized Mutual Information
UMVUE	Uniformly Minimum-Variance Unbiased Estimator

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Tables

Table 1

Estimation comparison between $\hat{H}$ and ${\hat{H}}_z$ via simulation. In the simulation, the real underlying distribution is $p_i=i/2001000$, where $i=1,2,\cdots ,2000$ (i.e., a triangle distribution). Under this setting, the true entropy $H=7.408005$. To compare the two estimators, 10,000 samples were independently generated following the triangle distribution for each of the six sample size settings in the table (i.e., we generate 60,000 random samples in total). The average values of $\hat{H}$ and ${\hat{H}}_z$ under different sample sizes are summarized and reported in the table. The simulation shows that $\hat{H}$ would consistently underestimate H more than ${\hat{H}}_z$. The underestimation is more severe when the sample size is smaller.

n	100	300	500	1000	1500	2000
avg. of $\hat{H}$	4.56	5.57	6.00	6.51	6.75	6.89
avg. of ${\hat{H}}_z$	5.11	6.09	6.49	6.92	7.11	7.21

Table 2

Selected information-theoretic feature selection methods reviewed in [16,64] with potential perspectives to improve the performance using smaller bias and/or variance estimation methods. The proposed criterion uses the same notations as their original forms for readers’ ease in tracing them back to the original articles, except some terms are equivalently re-denoted as $\widehat{MI}$ or $\hat{\kappa }$ to help readers follow the same notations in the previous sections. To further clarify the notations, $\widehat{MI}(X_1,X_2)$ and $\widehat{MI}(X_1;X_2)$ are the same. $\widehat{MI}(X_1,X_2|Y)$ is the plug-in estimator of conditional mutual information. By its definition, $\widehat{MI}(X_1,X_2|Y)=\hat{H}(X_1,Y)+\hat{H}(X_2,Y)-\hat{H}(X_1,X_2,Y)-\hat{H}\left(Y\right)$. Similarly, ${\widehat{MI}}_z(X_1,X_2|Y)={\hat{H}}_z(X_1,Y)+{\hat{H}}_z(X_2,Y)-{\hat{H}}_z(X_1,X_2,Y)-{\hat{H}}_z\left(Y\right)$, where each ${\hat{H}}_z$ could be further replaced using ${\hat{H}}_z$ with jackknife procedure.

MIM [65] MIFS [66] JMI [67]	Proposed Criterion (Score)	$\widehat{\mathit{M}\mathit{I}}$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
IF [68]	Proposed Criterion (Score)	$\widehat{MI}(C,F)$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
FCBF [69]	Proposed Criterion (Score)	${\hat{\kappa }}_4$
	Different Estimation Method	Use ${\hat{\kappa }}_{4z}$ [51] with jackknife procedure
AMIFS [70]	Proposed Criterion (Score)	$\widehat{MI}\left(C;f_i\right)-\beta {\sum }_{s\in S}{\hat{\kappa }}_6\left(C;f_s\right)\widehat{MI}\left(f_s;f_i\right)$
	Different Estimation Method	Use ${\widehat{MI}}_z$ and ${\hat{\kappa }}_{6z}$ [51] with jackknife procedure
CMIM [71]	Proposed Criterion (Score)	$\widehat{MI}(Y;X_n)$ and $\widehat{MI}(Y;X_n|X_m)$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
MRMR [72]	Proposed Criterion (Score)	$max\left[\widehat{MI}\left(x_i;c\right)-\frac{1}{\left|s\right|}{\sum }_{x_i,x_j\in S}\widehat{MI}\left(x_i,x_j\right)\right]$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
ICAP [73]	Proposed Criterion (Score)	$arg{max}_{X\in \overline{\mathcal{A}}}\left(\widehat{MI}(X;Y)+{\sum }_{A\in \mathcal{A}}min\{0,\widehat{MI}(X;Y;A)\}\right)$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
CIFE [74]	Proposed Criterion (Score)	$\underset{{\theta }_t}{argmax}\left\{\widehat{MI}\left(y^{\left(t\right)};c\right)-{\sum }_{u=1}^{t-1}\left[\widehat{MI}\left(y^{\left(u\right)};y^{\left(t\right)}\right)-\widehat{MI}\left(y^{\left(u\right)};y^{\left(t\right)}\mid c\right)\right]\right\}$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
DISR [75]	Proposed Criterion (Score)	$arg{max}_{X_i\in X_{-S}}\left\{{\sum }_{X_j\in X_S}{\hat{\kappa }}_1\left(X_{i,j};Y\right)\right\}$
	Different Estimation Method	Use ${\hat{\kappa }}_{1z}$ [51] with jackknife procedure
IGFS [76]	Proposed Criterion (Score)	$arg{max}_{X\in X_{-s}}\left(\widehat{MI}(X_i;Y)+\frac{1}{d}{\sum }_{X_j\in X_S}\widehat{MI}\left(X_i;X_j;Y\right)\right)$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
SOA [77]	Proposed Criterion (Score)	${\sum }_i\widehat{MI}\left(X_i,Y\right)-{\sum }_i{\sum }_{j>i}\widehat{MI}\left(X_i,X_j\right)+{\sum }_i{\sum }_{j>i}\widehat{MI}\left(X_i,X_j\mid Y\right)$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure
CMIFS [78]	Proposed Criterion (Score)	$\widehat{MI}\left(C;f_i\mid f_1\right)+\left[\widehat{MI}\left(f_n;f_i\mid C\right)-\widehat{MI}\left(f_n;f_i\mid f_1\right)\right]$
	Different Estimation Method	Use ${\widehat{MI}}_z$ with jackknife procedure

Appendix A. R Functions

Statistic	R Package Name	Function Name
$\hat{H}$	entropy [80]	entropy.plugin
${\hat{H}}_{MM}$	entropy	entropy.MillerMadow
${\hat{H}}_{JK}$	bootstrap [81]	jackknife
${\hat{H}}_{chao}$	entropy	entropy.ChaoShen
${\hat{H}}_z$	EntropyEstimation [82]	Entropy.z
$\hat{\sigma }$ in Theorems 1 and 3	EntropyEstimation	Entropy.sd
$\widehat{MI}$	entropy	mi.plugin
${\widehat{MI}}_z$	EntropyEstimation	MI.z
${\left[g^{\tau }\left(\hat{v}\right)\Sigma \left(\hat{v}\right)g\left(\hat{v}\right)\right]}^{\frac{1}{2}}$ in Theorem 6	EntropyEstimation	MI.sd
$\widehat{KL}$	entropy	KL.plugin
${\widehat{KL}}_z$	EntropyEstimation	KL.z
$\hat{S}$	EntropyEstimation	SymKL.plugin
${\hat{S}}_z$	EntropyEstimation	SymKL.z
${\hat{H}}_r^{♯}\left(n\right)$	EntropyEstimation	Renyi.z

References

1. Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J.; 1948; 27, pp. 379-423. [DOI: https://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x]

2. Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat.; 1951; 22, pp. 79-86. [DOI: https://dx.doi.org/10.1214/aoms/1177729694]

3. Zhang, Z.; Grabchak, M. Entropic representation and estimation of diversity indices. J. Nonparametr. Stat.; 2016; 28, pp. 563-575. [DOI: https://dx.doi.org/10.1080/10485252.2016.1190357]

4. Grabchak, M.; Zhang, Z. Asymptotic normality for plug-in estimators of diversity indices on countable alphabets. J. Nonparametr. Stat.; 2018; 30, pp. 774-795. [DOI: https://dx.doi.org/10.1080/10485252.2018.1482294]

5. Zhang, Z. Generalized Mutual Information. Stats; 2020; 3, pp. 158-165. [DOI: https://dx.doi.org/10.3390/stats3020013]

6. Burnham, K.P.; Anderson, D.R. Practical use of the information-theoretic approach. Model Selection and Inference; Springer: Berlin/Heidelberg, Germany, 1998; pp. 75-117.

7. Dembo, A.; Cover, T.M.; Thomas, J.A. Information theoretic inequalities. IEEE Trans. Inf. Theory; 1991; 37, pp. 1501-1518. [DOI: https://dx.doi.org/10.1109/18.104312]

8. Chatterjee, S.; Hadi, A.S. Regression Analysis by Example; John Wiley & Sons: Hoboken, NJ, USA, 2006.

9. Speed, T. What is an analysis of variance?. Ann. Stat.; 1987; 15, pp. 885-910. [DOI: https://dx.doi.org/10.1214/aos/1176350472]

10. Hardy, M.A. Regression with Dummy Variables; Sage: Newcastle upon Tyne, UK, 1993; Volume 93.

11. Kent, J.T. Information gain and a general measure of correlation. Biometrika; 1983; 70, pp. 163-173. [DOI: https://dx.doi.org/10.1093/biomet/70.1.163]

12. Lin, J. Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory; 1991; 37, pp. 145-151. [DOI: https://dx.doi.org/10.1109/18.61115]

13. Van Erven, T.; Harremos, P. Rényi divergence and Kullback-Leibler divergence. IEEE Trans. Inf. Theory; 2014; 60, pp. 3797-3820. [DOI: https://dx.doi.org/10.1109/TIT.2014.2320500]

14. Sethi, I.K.; Sarvarayudu, G. Hierarchical classifier design using mutual information. IEEE Trans. Pattern Anal. Mach. Intell.; 1982; 4, pp. 441-445. [DOI: https://dx.doi.org/10.1109/TPAMI.1982.4767278] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/21869061]

15. Safavian, S.R.; Landgrebe, D. A survey of decision tree classifier methodology. IEEE Trans. Syst. Man Cybern.; 1991; 21, pp. 660-674. [DOI: https://dx.doi.org/10.1109/21.97458]

16. Li, J.; Cheng, K.; Wang, S.; Morstatter, F.; Trevino, R.P.; Tang, J.; Liu, H. Feature selection: A data perspective. ACM Comput. Surv. (CSUR); 2017; 50, pp. 1-45. [DOI: https://dx.doi.org/10.1145/2996357]

17. Basharin, G.P. On a statistical estimate for the entropy of a sequence of independent random variables. Theory Probab. Appl.; 1959; 4, pp. 333-336. [DOI: https://dx.doi.org/10.1137/1104033]

18. Harris, B. The Statistical Estimation of Entropy in the Non-Parametric Case; Technical Report Wisconsin Univ-Madison Mathematics Research Center: Madison, WI, USA, 1975.

19. Zhang, Z.; Zhang, X. A normal law for the plug-in estimator of entropy. IEEE Trans. Inf. Theory; 2012; 58, pp. 2745-2747. [DOI: https://dx.doi.org/10.1109/TIT.2011.2179702]

20. Miller, G.A.; Madow, W.G. On the Maximum Likelihood Estimate of the Shannon-Weiner Measure of Information; Operational Applications Laboratory, Air Force Cambridge Research Center, Air Research and Development Command, Bolling Air Force Base: Washington, DC, USA, 1954.

21. Zahl, S. Jackknifing an index of diversity. Ecology; 1977; 58, pp. 907-913. [DOI: https://dx.doi.org/10.2307/1936227]

22. Chen, C.; Grabchak, M.; Stewart, A.; Zhang, J.; Zhang, Z. Normal Laws for Two Entropy Estimators on Infinite Alphabets. Entropy; 2018; 20, 371. [DOI: https://dx.doi.org/10.3390/e20050371]

23. Antos, A.; Kontoyiannis, I. Convergence properties of functional estimates for discrete distributions. Random Struct. Algorithms; 2001; 19, pp. 163-193. [DOI: https://dx.doi.org/10.1002/rsa.10019]

24. Paninski, L. Estimation of entropy and mutual information. Neural Comput.; 2003; 15, pp. 1191-1253. [DOI: https://dx.doi.org/10.1162/089976603321780272]

25. Zhang, Z. Entropy estimation in Turing’s perspective. Neural Comput.; 2012; 24, pp. 1368-1389. [DOI: https://dx.doi.org/10.1162/NECO_a_00266]

26. Schürmann, T. A note on entropy estimation. Neural Comput.; 2015; 27, pp. 2097-2106. [DOI: https://dx.doi.org/10.1162/NECO_a_00775] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26313604]

27. Zhang, Z. Asymptotic normality of an entropy estimator with exponentially decaying bias. IEEE Trans. Inf. Theory; 2013; 59, pp. 504-508. [DOI: https://dx.doi.org/10.1109/TIT.2012.2217393]

28. Zhang, Z. Statistical Implications of Turing’s Formula; John Wiley & Sons: Hoboken, NJ, USA, 2016.

29. Chao, A.; Shen, T.J. Nonparametric estimation of Shannon’s index of diversity when there are unseen species in sample. Environ. Ecol. Stat.; 2003; 10, pp. 429-443. [DOI: https://dx.doi.org/10.1023/A:1026096204727]

30. Nemenman, I.; Shafee, F.; Bialek, W. Entropy and inference, revisited. arXiv; 2001; arXiv: physics/0108025

31. Agresti, A.; Hitchcock, D.B. Bayesian inference for categorical data analysis. Stat. Methods Appl.; 2005; 14, pp. 297-330. [DOI: https://dx.doi.org/10.1007/s10260-005-0121-y]

32. Hausser, J.; Strimmer, K. Entropy inference and the James-Stein estimator, with application to nonlinear gene association networks. J. Mach. Learn. Res.; 2009; 10, pp. 1469-1484.

33. Shi, J.; Zhang, J.; Ge, Y. CASMI—An Entropic Feature Selection Method in Turing’s Perspective. Entropy; 2019; 21, 1179. [DOI: https://dx.doi.org/10.3390/e21121179]

34. Zhang, Z.; Zheng, L. A mutual information estimator with exponentially decaying bias. Stat. Appl. Genet. Mol. Biol.; 2015; 14, pp. 243-252. [DOI: https://dx.doi.org/10.1515/sagmb-2014-0047]

35. Zhang, J.; Chen, C. On “A mutual information estimator with exponentially decaying bias” by Zhang and Zheng. Stat. Appl. Genet. Mol. Biol.; 2018; 17, 20180005. [DOI: https://dx.doi.org/10.1515/sagmb-2018-0005]

36. Williams, P.L.; Beer, R.D. Nonnegative decomposition of multivariate information. arXiv; 2010; arXiv: 1004.2515

37. Bertschinger, N.; Rauh, J.; Olbrich, E.; Jost, J.; Ay, N. Quantifying unique information. Entropy; 2014; 16, pp. 2161-2183. [DOI: https://dx.doi.org/10.3390/e16042161]

38. Griffith, V.; Koch, C. Quantifying synergistic mutual information. Guided Self-Organization: Inception; Springer: Berlin/Heidelberg, Germany, 2014; pp. 159-190.

39. Tax, T.M.; Mediano, P.A.; Shanahan, M. The partial information decomposition of generative neural network models. Entropy; 2017; 19, 474. [DOI: https://dx.doi.org/10.3390/e19090474]

40. Wollstadt, P.; Schmitt, S.; Wibral, M. A rigorous information-theoretic definition of redundancy and relevancy in feature selection based on (partial) information decomposition. arXiv; 2021; arXiv: 2105.04187

41. Mori, T.; Nishikimi, K.; Smith, T.E. A divergence statistic for industrial localization. Rev. Econ. Stat.; 2005; 87, pp. 635-651. [DOI: https://dx.doi.org/10.1162/003465305775098170]

42. Wang, Q.; Kulkarni, S.R.; Verdú, S. Divergence estimation for multidimensional densities via k-Nearest-Neighbor distances. IEEE Trans. Inf. Theory; 2009; 55, pp. 2392-2405. [DOI: https://dx.doi.org/10.1109/TIT.2009.2016060]

43. Nguyen, X.; Wainwright, M.J.; Jordan, M.I. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Trans. Inf. Theory; 2010; 56, pp. 5847-5861. [DOI: https://dx.doi.org/10.1109/TIT.2010.2068870]

44. Zhang, Z.; Grabchak, M. Nonparametric estimation of Küllback-Leibler divergence. Neural Comput.; 2014; 26, pp. 2570-2593. [DOI: https://dx.doi.org/10.1162/NECO_a_00646]

45. Press, W.H.; Teukolsky Saul, A. Numerical Recipes in Fortran: The Art of Scientific Computing; Cambridge University Press: Cambridge, UK, 1993.

46. De Mántaras, R.L. A distance-based attribute selection measure for decision tree induction. Mach. Learn.; 1991; 6, pp. 81-92. [DOI: https://dx.doi.org/10.1023/A:1022694001379]

47. Kvalseth, T.O. Entropy and correlation: Some comments. IEEE Trans. Syst. Man Cybern.; 1987; 17, pp. 517-519. [DOI: https://dx.doi.org/10.1109/TSMC.1987.4309069]

48. Strehl, A.; Ghosh, J. Cluster ensembles—A knowledge reuse framework for combining multiple partitions. J. Mach. Learn. Res.; 2002; 3, pp. 583-617.

49. Yao, Y. Information-theoretic measures for knowledge discovery and data mining. Entropy Measures, Maximum Entropy Principle and Emerging Applications; Springer: Berlin/Heidelberg, Germany, 2003; pp. 115-136.

50. Vinh, N.X.; Epps, J.; Bailey, J. Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance. J. Mach. Learn. Res.; 2010; 11, pp. 2837-2854.

51. Zhang, Z.; Stewart, A.M. Estimation of Standardized Mutual Information; Technical Report UNC Charlotte Technical Report: Charlotte, NC, USA, 2016.

52. Zhang, Z.; Zhou, J. Re-parameterization of multinomial distributions and diversity indices. J. Stat. Plan. Inference; 2010; 140, pp. 1731-1738. [DOI: https://dx.doi.org/10.1016/j.jspi.2009.12.023]

53. Chen, C. Goodness-of-Fit Tests under Permutations. Ph.D. Thesis; The University of North Carolina at Charlotte: Charlotte, NC, USA, 2019.

54. Simpson, E.H. Measurement of diversity. Nature; 1949; 163, 688. [DOI: https://dx.doi.org/10.1038/163688a0]

55. Gini, C. Measurement of inequality of incomes. Econ. J.; 1921; 31, pp. 124-126. [DOI: https://dx.doi.org/10.2307/2223319]

56. Rényi, A. On measures of entropy and information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability; Berkeley, CA, USA, 1 January 1961; Volume 1.

57. Emlen, J.M. Ecology: An Evolutionary Approach; Addison-Wesley: Boston, MA, USA, 1977.

58. Zhang, Z.; Chen, C.; Zhang, J. Estimation of population size in entropic perspective. Commun.-Stat.-Theory Methods; 2020; 49, pp. 307-324. [DOI: https://dx.doi.org/10.1080/03610926.2018.1536786]

59. Beck, C.; Schögl, F. Thermodynamics of Chaotic Systems; Cambridge University Press: Cambridge, UK, 1995.

60. Zhang, J.; Shi, J. Asymptotic Normality for Plug-In Estimators of Generalized Shannon’s Entropy. Entropy; 2022; 24, 683. [DOI: https://dx.doi.org/10.3390/e24050683]

61. Zhang, J.; Zhang, Z. A Normal Test for Independence via Generalized Mutual Information. arXiv; 2022; arXiv: 2207.09541

62. Kontoyiannis, I.; Skoularidou, M. Estimating the directed information and testing for causality. IEEE Trans. Inf. Theory; 2016; 62, pp. 6053-6067. [DOI: https://dx.doi.org/10.1109/TIT.2016.2604842]

63. Huang, N.; Lu, G.; Cai, G.; Xu, D.; Xu, J.; Li, F.; Zhang, L. Feature selection of power quality disturbance signals with an entropy-importance-based random forest. Entropy; 2016; 18, 44. [DOI: https://dx.doi.org/10.3390/e18020044]

64. Brown, G.; Pocock, A.; Zhao, M.J.; Luján, M. Conditional likelihood maximisation: A unifying framework for information theoretic feature selection. J. Mach. Learn. Res.; 2012; 13, pp. 27-66.

65. Lewis, D.D. Feature selection and feature extraction for text categorization. Proceedings of the Speech and Natural Language: Proceedings of a Workshop Held at Harriman; New York, NY, USA, 23–26 February 1992.

66. Battiti, R. Using mutual information for selecting features in supervised neural net learning. IEEE Trans. Neural Netw.; 1994; 5, pp. 537-550. [DOI: https://dx.doi.org/10.1109/72.298224] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18267827]

67. Yang, H.; Moody, J. Feature selection based on joint mutual information. Proceedings of the International ICSC Symposium on Advances in Intelligent Data Analysis; Rochester, NY, USA, 22–25 June 1999; Volume 1999, pp. 22-25.

68. Ullman, S.; Vidal-Naquet, M.; Sali, E. Visual features of intermediate complexity and their use in classification. Nat. Neurosci.; 2002; 5, pp. 682-687. [DOI: https://dx.doi.org/10.1038/nn870] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/12055634]

69. Yu, L.; Liu, H. Efficient feature selection via analysis of relevance and redundancy. J. Mach. Learn. Res.; 2004; 5, pp. 1205-1224.

70. Tesmer, M.; Estévez, P.A. AMIFS: Adaptive feature selection by using mutual information. Proceedings of the 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No. 04CH37541); Budapest, Hungary, 25–29 July 2004; Volume 1, pp. 303-308.

71. Fleuret, F. Fast binary feature selection with conditional mutual information. J. Mach. Learn. Res.; 2004; 5, pp. 1531-1555.

72. Peng, H.; Long, F.; Ding, C. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell.; 2005; 27, pp. 1226-1238. [DOI: https://dx.doi.org/10.1109/TPAMI.2005.159]

73. Jakulin, A. Machine Learning Based on Attribute Interactions. Ph.D. Thesis; Univerza v Ljubljani: Ljubljana, Slovenia, 2005.

74. Lin, D.; Tang, X. Conditional infomax learning: An integrated framework for feature extraction and fusion. Proceedings of the European Conference on Computer Vision; Graz, Austria, 7–13 May 2006; pp. 68-82.

75. Meyer, P.E.; Bontempi, G. On the use of variable complementarity for feature selection in cancer classification. Proceedings of the Workshops on Applications of Evolutionary Computation; Budapest, Hungary, 10–12 April 2006; pp. 91-102.

76. El Akadi, A.; El Ouardighi, A.; Aboutajdine, D. A powerful feature selection approach based on mutual information. Int. J. Comput. Sci. Netw. Secur.; 2008; 8, 116.

77. Guo, B.; Nixon, M.S. Gait feature subset selection by mutual information. IEEE Trans. Syst. Man-Cybern.-Part Syst. Hum.; 2008; 39, pp. 36-46.

78. Cheng, G.; Qin, Z.; Feng, C.; Wang, Y.; Li, F. Conditional Mutual Information-Based Feature Selection Analyzing for Synergy and Redundancy. Etri J.; 2011; 33, pp. 210-218. [DOI: https://dx.doi.org/10.4218/etrij.11.0110.0237]

79. Singhal, A.; Sharma, D. Keyword extraction using Renyi entropy: A statistical and domain independent method. Proceedings of the 2021 7th International Conference on Advanced Computing and Communication Systems (ICACCS); Coimbatore, India, 19–20 March 2021; Volume 1, pp. 1970-1975.

80. R Package Entropy. Available online: https://cran.r-project.org/web/packages/entropy/index.html (accessed on 27 September 2022).

81. R Package Bootstrap. Available online: https://cran.r-project.org/web/packages/bootstrap/index.html (accessed on 27 September 2022).

82. R Package EntropyEstimation. Available online: https://cran.r-project.org/web/packages/EntropyEstimation/index.html (accessed on 27 September 2022).