Understanding Collections of Related Datasets Using Dependent MMD Coresets

Abstract

Understanding how two datasets differ can help us determine whether one dataset under-represents certain sub-populations, and provides insights into how well models will generalize across datasets. Representative points selected by a maximum mean discrepancy (MMD) coreset can provide interpretable summaries of a single dataset, but are not easily compared across datasets. In this paper, we introduce dependent MMD coresets, a data summarization method for collections of datasets that facilitates comparison of distributions. We show that dependent MMD coresets are useful for understanding multiple related datasets and understanding model generalization between such datasets.

1. Introduction

When working with large datasets, it is important to understand your data. If a dataset is not representative of your population of interest, and no appropriate correction is made, then models trained on this data may perform poorly in the wild. Sub-populations that are under-represented in the training data are likely to be poorly served by the resulting algorithm, leading to unanticipated or unfair outcomes—something that has been observed in numerous scenarios including medical diagnoses [1,2] and image classification [3,4].

In low-dimensional settings, it is common to summarize data using summary statistics such as marginal moments or label frequencies, or to visualize univariate or bivariate marginal distributions using histograms or scatter plots. As the dimensionality of our data increases, such summaries and visualizations become unwieldy, and ignore higher-order correlation structure. In structured data such as images, such summary statistics can be hard to interpret, and can exclude important information about the distribution [5,6]—the per-pixel mean and standard deviation of a collection of images tells us little about the overall distribution. Further, if our data are not labeled, or are only partially labeled, we cannot make use of label frequencies to assess class balance.

In such settings, we can instead choose to present a set of exemplars that capture the diversity of the data. This is particularly helpful for structured, high-dimensional data such as images or text, that can easily be qualitatively assessed by a person. A number of algorithms have been proposed to find such a set of exemplars [7,8,9,10,11,12,13,14,15,16,17]. Many of these algorithms can be seen as constructing a coreset for the dataset—a (potentially weighted) set of exemplars that behave similarly to the full dataset under a certain class of functions. In particular, coresets that minimize the maximum mean discrepancy [18] (MMD) between coreset and data have recently been used for understanding data distributions [11,13]. Further, evaluating models on such MMD-coresets have been shown to aid in understanding model performance [11].

In addition to summarizing a single dataset, we may also wish to compare and contrast multiple related datasets. For example, a company may be interested in characterizing differences and similarities between different markets. A machine learning practitioner may wish to know whether their dataset is similar to that used to train a given model. A researcher may be interested in understanding trends in posts or images on social media. Here, summary statistics offer interpretable comparisons: we can plot the mean and standard deviation of a given marginal quantity over time, and easily see how it changes [19,20]. By contrast, coresets are harder to compare, since the exemplars selected for two datasets $X_1$ and $X_2$ will not in general overlap.

In this paper, we introduce dependent MMD coresets, a new tool for characterizing related datasets and understanding model behavior across such datasets. These dependent MMD coresets provide a low-dimensional summary of a collection of datasets, that allows easy comparison across datasets. A dependent MMD coreset for a collection of datasets constructs a collection of exemplars, that is shared across all datasets. Each dataset assigns a different weight vector to these exemplars, so that the weighted exemplars approximate the dataset. These weights allow us to easily see which exemplars are relevant to which datasets, and comparing two sets of weights provides a simple way of showing how the corresponding datasets differ.

The use of shared exemplars makes it easy to compare two or more datasets, by providing a common language. Consider comparing two datasets of faces. If we independently constructed representations of each dataset—for example, using two independent MMD coresets—we would obtain two disjoint sets of weighted exemplars. Visually assessing the similarity between two sets would involve considering both the similarities of the images and the similarities in the weights. Conversely, with a dependent MMD coreset, the exemplars would be shared between the two datasets. Similarity can be assessed by considering the relative weights assigned in the two marginal coresets. This in turn leads to easy summarization of the difference between the two datasets, by identifying exemplars that are highly representative of one dataset, but less representative of the other.

In addition to understanding the difference between multiple datasets, dependent MMD coresets allow us to qualitatively explore the behavior of algorithms on these datasets. The shared set of exemplars provides representative points at which to evaluate the algorithm. Looking at the relative weights of these exemplars in the different datasets paints a picture of the relative performances we would expect between those datasets. This is particularly useful when a model has been trained on one dataset, but we wish to apply it to a second dataset: looking at exemplars that are highly representative of the second dataset, but not the first, allows us to identify potential failure modes.

We begin by considering existing coreset methods for data and model understanding in Section 2, before discussing their limitations and proposing our dependent MMD coreset in Section 3. A greedy algorithm to select dependent MMD coresets is provided in Section 3.4. In Section 4, we evaluate the efficacy of this algorithm, and show how the resulting dependent coresets can be used for understanding collections of image datasets and probing the generalization behavior of machine learning algorithms. We summarize notation used in this paper in Table 1.

2. Background and Related Work

2.1. Coresets and Measure-Coresets

A coreset is a “small” summary of a dataset X, which can act as a proxy for the dataset under a certain class of functions $\mathcal{F}$. Concretely, a weighted set of points ${\left\{(w_i,u_i)\right\}}_{i\in S}$ are an $ϵ$ strong coreset for a size-n dataset X with respect to $\mathcal{F}$ if

$\left|\frac{1}{n}\sum _{i=1}^{n}f\left(x_i\right)-\frac{1}{\left|S\right|}\sum _{j\in S}{w}_{j}f\left(u_j\right)\right| \le ϵ$

for all $f\in \mathcal{F}$ [21].

A measure coreset [22] generalizes this idea to assume that X are independently and identically distributed samples from some distribution $\mathbb{P}$. A measure $\mathbb{Q}$ is an $ϵ$-measure coreset for $\mathbb{P}$ with respect to some class $\mathcal{F}$ of functions if

(1)$\underset{f\in \mathcal{F}}{sup}\left|{\mathbb{E}}_{X\sim \mathbb{P}}\right[f\left(X\right)\left] - {\mathbb{E}}_{Y\sim \mathbb{Q}}\right[f\left(Y\right)]|.$

The left hand side of Equation (1) describes an integral probability metric [23], a class of distances between probability measures parametrized by some class $\mathcal{F}$ of functions. Different choices of $\mathcal{F}$ yield different distributions (Table 2).

2.2. MMD-Coresets

In this paper, we consider the case where $\mathcal{F}$ is the class of all functions that can be represented in the unit ball of some reproducing kernel Hilbert space (RKHS) $\mathcal{H}$—a very rich class of continuous functions on $\mathcal{X}$. This corresponds to a metric known as the maximum mean discrepancy [18] (MMD),

(2)$\mathrm{MMD}(\mathbb{P},\mathbb{Q})=\underset{f\in \mathcal{H}}{sup}|{\mathbb{E}}_{X\sim \mathbb{P}}[f\left(X\right)] - {\mathbb{E}}_{Y\sim \mathbb{Q}}[f\left(Y\right)]|.$

An RKHS can be defined in terms of a mapping $\mathsf{\Phi }:\mathcal{X}\to \mathcal{H}$, which in turn specifies a kernel function $k(x,x^{\prime })={⟨\mathsf{\Phi }\left(x\right),\mathsf{\Phi }\left(x^{\prime }\right)⟩}_{\mathcal{H}}$. A distribution $\mathbb{P}$ can be represented in this space in terms of its mean embedding, ${\mu }_P={\mathbb{E}}_{\mathbb{P}}\left[\mathsf{\Phi }\left(x\right)\right]$. The MMD between two distributions equivalently can be expressed in terms of their mean embeddings, $\mathrm{MMD}{(\mathbb{P},\mathbb{Q})}^2=\left|\right|{\mu }_P-{\mu }_Q{\left|\right|}_{\mathcal{H}}^2$,

An $ϵ$-MMD coreset for a distribution $\mathbb{P}$ is a finite, atomic distribution $\mathbb{Q}={\sum }_{i\in S}{w}_i{\delta }_{u_i}$ such that $\mathrm{MMD}{(\mathbb{P},\mathbb{Q})}^2\le {ϵ}^2$. We will refer to the set ${\left\{u_i\right\}}_{i\in S}$ as the support of $\mathbb{Q}$, and refer to individual locations in the support of $\mathbb{Q}$ as exemplars.

In practice, we are unlikely to have access to $\mathbb{P}$ directly, but instead have samples $X:=(x_1,\dots ,x_n)\sim \mathbb{P}$. If $\mathbb{Q}={\sum }_{i\in S}{w}_i{\delta }_{u_i}$, we can estimate $\mathrm{MMD}{(\mathbb{P},\mathbb{Q})}^2$ as

$\widehat{{\mathrm{MMD}}^2}\left(X,\mathbb{Q}\right) =\frac{1}{n^2}\sum _{i=1}^n\sum _{j=1}^{n}k(x_i,x_j)+\sum _{i\in S}\sum _{j\in S}{w}_i{w}_{j}k(u_i,u_j)-\frac{2}{n}\sum _{i=1}^n\sum _{j\in S}{w}_{j}k(x_i,u_j).$

We, therefore, define an $ϵ$-MMD coreset for a dataset X as a finite, atomic distribution $\mathbb{Q}$ such that $\widehat{{\mathrm{MMD}}^2}(X,\mathbb{Q})\le {ϵ}^2$—or equivalently, whose mean embedding ${\mu }_Q$ in $\mathcal{H}$ is close to the empirical mean embedding ${\widehat{\mu }}_X$ so that $\left|\right|{\mu }_Q-{\widehat{\mu }}_X{\left|\right|}_{\mathcal{H}}^2\le {ϵ}^2$.

A number of algorithms have been proposed that correspond to finding $ϵ$-MMD coresets, under certain restrictions on $\mathbb{Q}$ (While most of these algorithms do not explicitly use coreset terminology, the resulting set of samples, exemplars or prototypes meet the definition of an MMD coreset for some value of $ϵ$). Many of these algorithms greedily construct an MMD coreset, adding exemplars one-by-one based on some criteria. For example, kernel herding [14,24,25] can be seen as finding an MMD coreset $\mathbb{Q}$ for a known distribution $\mathbb{P}$, with no restriction on the support of $\mathbb{Q}$. The greedy prototype-finding algorithm used by [11] can be seen as a version of kernel herding, where $\mathbb{P}$ is only observed via a set of samples X, and where the support of $\mathbb{Q}$ is restricted to be some subset of a collection of candidates U (often chosen to be the data set X). Versions of this algorithm that assign weights to the atoms in $\mathbb{Q}$ are proposed in [13].

Other methods start from the full dataset, and repeatedly discard points to construct a coreset [15,16,17,26]. Loosely, these methods repeatedly partition the dataset based on a discrepancy criterion, and then discard one half of the partition. Compared with the greedy methods, these approaches typically obtain smaller coresets for a given $ϵ$ [15,17]. As shown by [27], random sampling also provides a way to construct an MMD-coreset.

As in [11,13], in this paper we require the support of our coreset to be a subset of some finite set of candidates U, indexed by $1,\dots ,n_U$. In other words, our measure coresets will take the form $\mathbb{Q}={\sum }_{i\in S}{w}_i{\delta }_{u_i}$, where $S\subset \left[n_U\right]$.

2.3. Coresets for Understanding Datasets and Models

The primary application of coresets is to create a compact representation of a large dataset, to allow for fast inference on downstream tasks (see [28] for a recent survey). However, such compact representations have also proved beneficial in interpretation of both models and datasets.

While humans are good at interpreting visual data [29], visualizing large quantities of data can become overwhelming due to the sheer quantity of information. Coresets can be used to filter such large datasets, while still retaining much of the relevant information.

The MMD-critic algorithm [11] uses a fixed-dimension, uniformly weighted MMD coreset, which they refer to as “prototypes”, to summarize collections of images. Gurumoorthy et al. [13] extends this to use a weighted MMD coreset, showing that weighted prototypes allow us to better model the data distribution, leading to more interpretable summaries. Zheng et al. [30] show how unweighted MMD coresets can be used to represent spatial point processes such as spatial location of crimes.

Techniques such as coresets that produce representative points for a dataset can also be used to provide interpretations and explanations of the behavior of models on that dataset. Case-based reasoning approaches use representative points to describe “typical” behavior of a model [11,31,32,33]. Considering the model’s output on such representative points can allow us to understand the model’s behavior.

Viewing the model’s behavior on a collection of “typical” points in our dataset also allows us to get an idea of the overall model performance on our data. Evaluating a model on a coreset can give an idea of how we expect it to perform on the entire dataset, and can help identify failure modes or subsets of the data where the model performs poorly.

2.4. Criticising MMD Coresets

While MMD coresets are good at summarizing a distribution, since the coreset is much smaller than the original dataset, there are likely to be outliers in the data distribution that are not well explained by the coreset. The MMD-critic algorithm supplements the “prototypes” associated with the MMD coreset with a set of “criticisms”—points that are poorly modeled by the coreset [11].

Recall from Equation (5) that the MMD between two distributions $\mathbb{P}$ and $\mathbb{Q}$ corresponds to the maximum difference in the expected value on the two spaces, of a function that can be represented in the unit ball of a Hilbert space $\mathcal{H}$. The function f that achieves this maximum is known as the witness function, and is given by

$f\left(x^{\ast }\right)={\mathbb{E}}_{X\sim \mathbb{P}}\left[k(x^{\ast },X)\right]-{\mathbb{E}}_{Y\sim \mathbb{Q}}[k(x^{\ast },Y)].$

When we only have access to $\mathbb{P}$ via a size-n sample X, and where $\mathbb{Q}={\sum }_{i\in S}{w}_i{\delta }_{u_i}$, we can approximate this as

$\widehat{f}\left(x^{\ast }\right)=\frac{1}{n}\sum _{i=1}^{n}k(x^{\ast },X_i)-\sum _{j\in S}{w}_{j}k(x^{\ast },u_j).$

Criticisms of an MMD coreset for a data set X are selected as the points in X with the largest values of the witness function. Kim et al. [11] show that the combination of prototypes and criticisms allow us to visually understand large collections of images: the prototypes summarize the main structure of the dataset, while the criticisms allow us to represent the extrema of a distribution. Criticisms can also augment an MMD coreset in a case-based reasoning approach to model understanding, by allowing us to consider model behavior on both “typical” and “atypical” exemplars.

2.5. Dependent and Correlated Random Measures

Dependent random measures [34,35] are distributions over collections of countable measures ${\mathbb{P}}_t={\sum }_{i=1}^{\infty }{w}_{t,i}{\delta }_{u_{t,i}}$, indexed by some set $\mathcal{T}$, such that the marginal distribution at each $t\in \mathcal{T}$ is described by a specific distribution. In most cases, this marginal distribution is a Dirichlet process, meaning that the ${\mathbb{P}}_t$ are probability distributions. Most dependent random measures either keep the weights $w_{t,i}$ or the atom locations $u_{t,i}$ constant accross t, to assist identifiability and interpretability.

In a Bayesian framework, dependent Dirichlet processes are often used as a prior for time-dependent mixture models. In settings where the atom locations (i.e., mixture components) are fixed but the weights vary, the posterior mixture components can be used to visualize and understand data drift [36,37]. The dependent coresets presented in this paper can be seen as deterministic, finite-dimensional analogues of these posterior dependent random measures.

3. Understanding Multiple Datasets Using Coresets

As we have seen, coresets provide a way of summarizing a single distribution. In this section, we discuss interpretational limitations that arise when we attempt to use coresets to summarize multiple related datasets (Section 3.1), before proposing dependent MMD coresets in Section 3.2 and discussing their uses in Section 3.3.

3.1. Understanding Multiple Datasets Using MMD-Coresets

If we have a collection ${\left\{X_t\right\}}_{t\in \mathcal{T}}$ of datasets, we might wish to find $ϵ$-MMD coresets for each of the $X_t$, in the hope of not just summarizing the individual datasets, but also of easily comparing them. However, if we want to understand the relationships between the datasets, in addition to their marginal distributions, comparing such coresets in an interpretable manner is challenging.

An MMD coreset selects a set ${\left\{u_i\right\}}_{i\in S}$ of points from some set of candidates U. Even if two datasets X and Y are sampled from the same underlying distribution (i.e., $X,Y\stackrel{\mathrm{iid}}{\sim }\mathbb{P}$), and the set U of available candidates is shared, the optimal MMD-coreset for the two datasets will differ in general. Sampling error between the two distributions means that $\widehat{{\mathrm{MMD}}^2}(X,\mathbb{Q})\ne \widehat{{\mathrm{MMD}}^2}(Y,\mathbb{Q})$ for any candidate coreset Q unless $X\equiv Y$, and so the optimal coreset will typically differ between the two datasets.

Figure 1 shows that, even if two distributions X and Y are sampled from the same underlying distribution, and their coreset locations are selected from the same collection U, the two coresets will not be identical. Here, we see two datasets (Figure 1b,c) generated from the same mixture of three equally weighted Gaussians (Figure 1a). Below (Figure 1d,e), we have selected a coreset for each dataset (using the algorithm that will be introduced in Section 3.4), with locations selected from a shared set U. While the associated coresets are visually similar, they are not the same.

This is magnified if we look at a high-dimensional dataset. Here, the relative sparsity of data points (and candidate points) in the space means that individual locations in ${\mathbb{Q}}_X$ might not have close neighbors in ${\mathbb{Q}}_Y$, even if X and Y are sampled from the same distribution. Further, in high dimensional spaces, it is harder to visually assess the distance between two exemplars. These observations make it hard to compare two coresets, and gain insights about similarities and differences between the associated datasets.

To demonstrate this, we constructed two datasets, each containing 250 randomly selected, female-identified US highschool yearbook photos from the 1990s. Figure 2 shows MMD-coresets obtained for the two datasets (See Section 4.2.1 for full details of dataset and coreset generation.) While both datasets were selected from the same distribution, there is no overlap in the support of the two coresets. Visually, it is hard to tell that these two coresets are representing samples from the same distribution.

3.2. Dependent MMD Coresets

The coresets in Figure 2 are hard to compare due to their disjoint supports (i.e., the fact that there are no shared exemplars). Comparing the two coresets involves comparing multiple individual photos and assessing their similarities, in addition to incorporating the information encoded in the associated weights. To avoid the lack of interpretability resulting from dissimilar supports, we introduce the notion of a dependent MMD coreset.

Given a collection of datasets ${\left\{X_t\right\}}_{t\in \mathcal{T}}$, the collection of finite, atomic measures ${\left\{{\mathbb{Q}}_t\right\}}_{t\in \mathcal{T}}$, is an $ϵ$-dependent MMD coreset if

(3)$\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t)\le {ϵ}^2$

for all $t\in \mathcal{T}$, and if the ${\mathbb{Q}}_t$ have common support, i.e.,

(4)${\mathbb{Q}}_t=\sum _{i\in S}{w}_{t,i}{\delta }_{u_i},$

where ${\left\{u_i\right\}}_{i\in S}$ is a subset of some candidate set U.

In Equation (4), the exemplars $u_i$ are shared between all $t\in \mathcal{T}$, but the weights $w_{t,i}$ associated with these exemplars can vary with t. Taking the view from Hilbert space, we are restricting the mean embeddings ${\mu }_{Q_t}$ of the marginal coresets to all lie within a convex hull defined by the exemplars ${\left\{u_i\right\}}_{i\in S}$.

By restricting the support of our coresets in this manner, we obtain data summaries that are easily comparable. Within a single dataset, we can look at the weighted exemplars that make up the coreset and use these to understand the spread of the data, as is the case with an independent MMD coreset. Indeed, since ${\mathbb{Q}}_t$ still meets the definition of an MMD coreset for $X_t$ (see Equation (3)), we can use it analogously to an independently generated coreset. However, since the exemplars are shared across datasets, we can directly compare the exemplars for two datasets. We no longer need to intuit similarities between disjoint sets of exemplars and their corresponding weights; instead we can directly compare the weights for each exemplar to determine their relative relevance to each dataset. We will show in Section 4.2.1 that this facilitates qualitative comparison between the marginal coresets, when compared to independently generated coresets.

We note that the dependent MMD coresets introduced in this paper are directly extensible to other integral probability measures; we could, for example, construct a dependent version of the Wasserstein coresets introduced by [22].

3.3. Model Understanding and Extrapolation

As we discussed in Section 2.3, MMD coresets can be used as tools to understand the performance of an algorithm on “typical” data points. Considering how an algorithm performs on such exemplars allows the practitioner to understand failure modes of the algorithm, when applied to the data. In classification tasks where labeling is expensive, or on qualitative tasks such as image modification, looking at an appropriate coreset can provide an estimate of how the algorithm will perform across the dataset.

In a similar manner, dependent coresets can be used to understand generalization behavior of an algorithm. Assume a machine learning algorithm has been trained on a given dataset $X_a$, but we wish to apply it (without modification) to a dataset $X_b$. This is frequently done in practice, since many machine learning algorithms require large training sets and high computational cost; however if the training distribution differs from the deployment distribution, the algorithm may not perform as intended. In general, we would expect the algorithm to perform well on data points in $X_b$ that have many close neighbors in $X_a$, but perform poorly on data points in $X_b$ that are not well represented in $X_a$.

Creating a dependent MMD coreset $\left({\mathbb{Q}}_a={\sum }_i{w}_{a,i}{\delta }_{u_i},{\mathbb{Q}}_b={\sum }_i{w}_{b,i}{\delta }_{u_i}\right)$ for the pair $(X_a,X_b)$ allows us to identify exemplars that are highly representative of $X_a$ or $X_b$ (i.e., have high weight in the corresponding weighted coreset). Further, by comparing the weights in the two coreset measures—e.g., by calculating $f_i=w_{b,i}/w_{a,i}$—we can identify exemplars that are much more representative of one dataset than another. Rather than look at all points in the coreset, if we are satisfied with the performance of our model on $X_a$, we can choose to only look at points with high values of $f_i$—points that are representative of the new dataset $X_b$, but not the original dataset $X_a$. Further, if we wish to consider generalization to multiple new datasets, a shared set of exemplars reduces the amount of labeling or evaluation required.

An MMD coreset, dependent or otherwise, will only contain exemplars that are representative of the dataset(s). There are likely to be outliers that are less well represented by the coreset. Such outliers are likely to be underserved by a given algorithm—for example, yielding low accuracy or poor reconstructions.

As we saw in Section 2.4, MMD coresets can be augmented by criticisms—points in X that are poorly approximated by $\mathbb{Q}$. We can equivalently construct criticisms for each dataset represented by a dependent MMD coreset. In the example above, we would select criticisms for the dataset $X_b$ by selecting points in $X_b$ that maximize

${\widehat{f}}_t\left(x^{\ast }\right)=\frac{1}{n_t}\sum _{i=1}^{n_t}k(x^{\ast },X_{t,i})-\sum _{j\in S}{w}_{t,j}k(x^{\ast },u_j).$

In addition to evaluating our algorithm on the marginal dependent coreset for dataset $X_b$, or the subset of the coreset with high values of $f_i$, we can evaluate on the criticisms $C_b$. In conjunction, the dependent MMD coreset and its criticisms allow us to better understand how the algorithm is likely to perform on both typical, and atypical, exemplars of $X_b$.

3.4. A Greedy Algorithm for Finding Dependent Coresets

Given a collection ${\left\{X_t\right\}}_{t\in \mathcal{T}}$ of datasets, where we assume $X_t:=\{x_{t,1},\dots ,x_{t,n_t}\}\sim {\mathbb{P}}_t$, and a set of $n_U$ candidates U, our goal is to find a collection ${\left\{{\mathbb{Q}}_t\right\}}_{t\in \mathcal{T}}$ with shared support $\{u_i:i\in S\subset \left[n_U\right]\}$ such that $\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t)\le {ϵ}^2$ for all $t\in \mathcal{T}$. We begin by constructing an algorithm for a related task: to minimize ${\sum }_{t\in \mathcal{T}}MMD({\mathbb{Q}}_t,{\mathbb{P}}_t)$, where

(5)$\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t)=\frac{1}{n_t^2}\sum _{i=1}^{n_t}\sum _{j=1}^{n_t}k(x_{t,i},x_{t,j})+\sum _{i\in S}\sum _{j\in S}{w}_{t,i}{w}_{t,j}k(u_i,u_j)-\frac{2}{n_t}\sum _{i=1}^{n_t}\sum _{j\in S}{w}_{t,j}k(x_i,u_j)$

where ${\mathbb{Q}}_t={\sum }_{i\in S}{w}_{t,i}{\delta }_{u_i}$. If we ignore terms in Equation (5) that do not depend on the ${\mathbb{Q}}_t$, we obtain the following loss:

(6)$\begin{array}{cc}\hfill \mathcal{L}\left({\left\{{\mathbb{Q}}_t\right\}}_{t\in \mathcal{T}}\right)=& \sum _{t\in \mathcal{T}}{\ell }_t\left({\mathbb{Q}}_t\right)\hfill \\ \hfill {\ell }_t\left({\mathbb{Q}}_t\right)=& \frac{1}{2}\sum _{i\in S}\sum _{j\in S}{w}_{t,i}{w}_{t,j}k(u_i,u_j)-\frac{1}{n_t}\sum _{i=1}^{n_t}\sum _{j\in S}k(x_i,u_j).\hfill \end{array}$

We can use a greedy algorithm to minimize this loss. Let ${\mathbb{Q}}_t^{\left(m\right)}={\sum }_{i\in S^{\left(m\right)}}{w}_{t,i}^{\left(m\right)}{\delta }_{u_i^{\left(m\right)}}$, where $S^{\left(m\right)}$ indexes the first m exemplars to be added. We wish to select the exemplar $u_{\ast }$, and set of weights $\left\{w_{t,\ast }^{(m+1)},{\left\{w_{t,i}^{(m+1)}\right\}}_{i\in S^{\left(m\right)}}\right\}$ for each dataset $X_t$, that minimize the loss. However, searching over all possible combinations of exemplars and weights is prohibitively expensive, as it involves a non-linear optimization to learn the weights associated with each candidate. Instead, we assume that, for each $t\in \mathcal{T}$, there is some ${\alpha }_t^{\ast }>0$ such that $w_{t,\ast }^{(m+1)}=\frac{{\alpha }_t^{\ast }}{{\alpha }_t^{\ast }+1}$ and $w_{t,i}^{(m+1)}=\frac{w_{t,i}^{\left(m\right)}}{{\alpha }_t^{\ast }+1}$ for all $i\in S^{\left(m\right)}$. In other words, we assume that the relative weights in each ${\mathbb{Q}}_t$ of the previously added exemplars do not change as we add more exemplars.

Fortunately, the value of ${\alpha }_t^{\ast }$ that minimizes ${\ell }_t\left(\frac{1}{1+{\alpha }_t^{\ast }}{\mathbb{Q}}_t^{\left(m\right)}+\frac{{\alpha }_t^{\ast }}{{\alpha }_t^{\ast }+1}{\delta }_{u_{\ast }}\right)$ can be found analytically for each candidate $u_{\ast }$ by differentiating the loss in step 1, yielding

(7)${\alpha }_t^{\ast }=\frac{{\displaystyle \sum _{i,j\in S^{\left(m\right)}}{w}_{t,i}{w}_{t,j}k(u_i,u_j)-\sum _{i\in S^{\left(m\right)}}{w}_{t,i}k(u_i,u_{\ast })-\frac{1}{n_t}\sum _{i=1}^{n_t}\left(k(x_i,u_{\ast })-\sum _{j\in S^{\left(m\right)}}{w}_{j}k(x_{t,i},u_j)\right)}}{{\displaystyle k(u_{\ast },u_{\ast })-\sum _{i\in S^{\left(m\right)}}{w}_{t,i}k(u_i,u_{\ast })-\frac{1}{n_t}\sum _{i=1}^{n_t}\left(k(x_i,u_{\ast })-\sum _{j\in S^{\left(m\right)}}{w}_{j}k(x_{t,i},u_j)\right)}}.$

We can, therefore, set

(8)$i^{\ast },\left\{{\alpha }_t^{\ast }\right\}\leftarrow \underset{\stackrel{i\in \left[n_U\right]\backslash S^{\left(m\right)},}{{\alpha }_t\in {\mathbb{R}}_{+}}}{arg\; min}\sum _{t\in \mathcal{T}}{\ell }_t\left(\frac{1}{1+{\alpha }_t}{\mathbb{Q}}_t^{\left(m\right)}+\frac{{\alpha }_t}{{\alpha }_t+1}{\delta }_{u_i}\right)$

and let $S^{(m+1)}=S^{\left(m\right)}\cup i^{\ast }$, $w_{t,i^{\ast }}^{(m+1)}=\frac{{\alpha }_t^{\ast }}{{\alpha }_t^{\ast }+1}$ for all $t\in \mathcal{T}$, and $w_{t,i}^{(m+1)}=\frac{w_{t,i}^{\left(m\right)}}{{\alpha }_t^{\ast }+1}$ for all $t\in \mathcal{T}$ and $i\in S^{\left(m\right)}$.

As written, the procedure will greedily minimize the sum of the per-dataset losses. However, the definition of an MMD coreset involves satisfying, not minimizing: we want $\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t)\le {ϵ}^2$ for all $t\in \mathcal{T}$. To achieve this, we modify the sum in Equation (8) so that it only includes terms for which $\widehat{{\mathrm{MMD}}^2}\left(X_t,{\mathbb{Q}}_t^{\left(m\right)}\right)>{ϵ}^2$. The resulting procedure is summarized in Algorithm 1.

Algorithm 1.dmmd: Selecting dependent MMD coresets

Require: Datasets ${\left\{X_t\right\}}_{t\in \mathcal{T}}$; candidate set U; kernel $k(·,·)$; threshold ${ϵ}^2>0$  $S^{\left(0\right)}\leftarrow \oslash$; $w_t^{\left(0\right)}\leftarrow \left[\right]$ for all $t\in \mathcal{T}$; $D\leftarrow \mathcal{T}$; $m\leftarrow 0$  while $D\ne \oslash$ do   for all $i\in \left[n_U\right]\backslash S^{\left(m\right)}$ do    for all $t\in \mathcal{T}$ do     Calculate ${\alpha }_{t,i}^{\ast }$ using Equation (7)    end for    $L_i\leftarrow 0$    for all $t\in D$ do     $L_i\leftarrow L_i+{\ell }_t\left(\frac{1}{1+{\alpha }_t^{\ast }}{\mathbb{Q}}_t^{\left(m\right)}+\frac{{\alpha }_t^{\ast }}{{\alpha }_t^{\ast }+1}{\delta }_{u_{\ast }}\right)$    end for   end for   $i^{\ast }= {arg\; min}_{i\in \left[n_U\right]\backslash S^{\left(m\right)}}{L}_i$   $S^{(m+1)}\leftarrow S^{\left(m\right)} \cup \{i^{\ast }\}$   $w_{i^{\ast }}^{(m+1)}\leftarrow \frac{{\alpha }_{i^{\ast }}}{{\alpha }_{i^{\ast }}+1}$   for all $t\in \mathcal{T}$ do    for all $i\in S^{\left(m\right)}$ do     $w_i^{(m+1)}\leftarrow \frac{w_i^{\left(m\right)}}{{\alpha }_{i^{\ast }}+1}$    end for    ${\mathbb{Q}}_t^{(m+1)}\leftarrow {\sum }_{i\in S^{(m+1)}}{w}_{t,i}^{(m+1)}{\delta }_{u_i}$   end for   $D=\{X_t:\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t^{(m+1)})>{ϵ}^2\}$   $m\leftarrow m+1$  end while

3.5. Limitations

As discussed in Section 2, if we can bound the MMD between two distributions by $ϵ$, then for any function f in the unit ball $\mathcal{H}$ of the Hilbert space associated with our kernel, the expectations of f with respect to the two distributions will differ by at most $ϵ$. However, we have no guarantee for functions that cannot be represented in that Hilbert space. If we use an MMD coreset (dependent or otherwise) to understand the output of a model, and that output cannot be well approximated by the expectation with respect to a function in $\mathcal{H}$, we cannot use performance on the coreset to bound performance on the full dataset. For this reason, we focus on the use of coresets as a qualitative, diagnostic tool for exploring model performance.

Beyond the question of whether functions of interest lie in a Hilbert space, we must also question which Hilbert space. Our choice of kernel will impact the nature of the resulting coresets. If we assume the popular squared exponential kernel, then different lengthscales will cause the algorithm to prioritize capturing variation at different scales. In this work, we have used median heuristics to set the lengthscale [38]; however if we were interested in capturing differences on a specific task, a better approach might be to learn the kernel. An alternative approach would be to use a different integral probability metric in place of the MMD, such as the Wasserstein distance, which has been used to construct (non-dependent) measure coresets [22].

Conversely, a limitation of MMD is that calculating $\widehat{{\mathrm{MMD}}^2}(X,\mathbb{Q})$ scales cubically with the size of the data. Similarly, calculating the Wasserstein distance is typically computationally expensive, as in general it requires solving a linear programming problem. This limits the scalability of our algorithm; however, since subsampling a dataset yields a valid MMD coreset with high probability [27]), our algorithm could be used on samples from larger datasets. Similarly, we could replace our initial datasets with (non-dependent) MMD coresets obtained using an existing algorithm [14,15,16,17], although this would be more expensive than random sampling. In either setting, we would need to incorporate the approximation error of the random sample or coreset, into our overall approximation error $ϵ$.

When working with complex datasets such as images, we often work with lower-dimensional representations or embeddings [39,40,41]—for example, in Section 4, we will use ResNet [42] to generate embeddings for yearbook photos. However, this can make notions of “similarity” opaque, since the representations can capture properties of the image that are not immediately obvious to the viewer, or do not register as important [43]. Concerningly, recent research has suggested that image representations can encode harmful human-like biases [44].

Our algorithm greedily constructs dependent coresets. Recent work on MMD coresets has found that discrepancy-based algorithms, where the full dataset is successively divided based on some discrepancy measure, can obtain smaller $ϵ$-coresets than greedy methods or random sampling [15,16,17]. Unfortunately, it is not clear how to extend such a partitioning algorithm to the dependent setting; however, these results suggest that it is worth exploring alternative constructions for dependent coresets.

4. Experimental Evaluation

In Section 3.2, we introduced dependent MMD coresets, a summarization technique designed to allow easy comparison between related datasets, and proposed a greedy algorithm to construct dependent MMD coresets in Section 3.4. We also described, in Section 3.3, how dependent MMD coresets can be used to understand performance of models and algorithms, particularly in the context of generalization to new datasets.

In this section, we will empirically evaluate the performance of our algorithm in Section 4.1. Previous greedy algorithms for weighted MMD coresets (without dependence) proceed by first selecting a new exemplar, and then updating weights once the exemplar has been added to the coreset. While such an approach could be adapted to the dependent setting, we show that our algorithm (Algorithm 1), which pre-selects weights based on a single calculation, achieves comparable coresets with lower computational cost.

After evaluating the algorithm used to select the coresets, we will go on to explore the coresets themselves, in Section 4.2. We begin by showing how, when comparing two datasets, the shared support offered by dependent MMD coresets allows for easier comparison than two standard MMD coresets. We then go on to show, in an example comparing 12 related datasets, that dependent MMD coresets can allow us to capture trends and similarities in an interpretable manner.

In Section 4.3, we turn our attention to coresets for model understanding. Here, we simulate a scenario where we wish to deploy algorithms trained on one dataset, to a slightly different datasets. By looking at performance on exemplars that are highly weighted in the second dataset, but not the first, we can obtain qualitative insights on the generalization properties of the algorithms. Adding evaluation on criticisms of the dependent MMD coreset leads to a deeper understanding of the model behavior.

4.1. Evaluation of Dependent MMD Coreset Algorithm

In Section 3.4, we proposed a greedy algorithm for selecting dependent MMD coresets (Algorithm 1, which we will denote dmmd). This algorithm selected weights (one for each dataset) for each candidate data point, and then greedily selected a data point and its associated weights. Since dependent coresets are introduced in this work, there is no direct comparison algorithm; however, a natural alternative would have been to adapt protodash, an existing greedy algorithm for weighted MMD coresets, to the dependent setting. Such an approach differs from Algorithm 1 in that weights are optimized after a candidate has been selected.

Below, we review the protodash algorithm, and introduce two alternative greedy algorithms for dependent MMD coresets: a dependent version of protodash, that selects unweighted candidates then optimizes weights; and a hybrid algorithm that pre-selects weights for candidate points, but further optimizes them after an exemplar has been added to the coreset. We quantitatively compare these variants with Algorithm 1, showing that pre-selecting weights provides comparable coresets to methods that optimize weights, at a much lower computational cost.

The protodash algorithm [13] for weighted MMD coresets greedily selects exemplars that minimize the gradient of the loss in Equation (6) (for a single dataset). Having selected an exemplar to add to the coreset, protodash then uses an optimization procedure to find the weights that minimize $\widehat{{\mathrm{MMD}}^2}(X,{\mathbb{Q}}^{(m+1)})$. We modify this algorithm for the dependent MMD setting by summing the gradients across all datasets for which the ${ϵ}^2$ threshold is not yet satisficed, leading to the dependent protodash algorithm shown in Algorithm 2.

Unlike the dependent version of protodash in Algorithm 2, our algorithm assigns weights before selection, which should encourage adding points that would help some of the marginal coresets, but not others. However, there is no post-exemplar-addition optimization of the weights. Inspired by the post-addition optimization in protodash, we also compare our algorithm with a variant of Algorithm 1 that optimizes the weights after each step—allowing the relative weights of the exemplars to change between each iteration. We will refer to this variant of dmmd with post-exemplar-addition optimization as dmmd-opt.

Algorithm 2. A dependent protodash algorithm

Require: Datasets ${\left\{X_t\right\}}_{t\in \mathcal{T}}$, candidate set U, kernel $k(·,·)$, threshold ${ϵ}^2>0$  $S^{\left(0\right)}\leftarrow \oslash$, $w_t^{\left(0\right)}\leftarrow \left[\right]$ for all $t\in \mathcal{T}$, $D\leftarrow \mathcal{T}$, $m\leftarrow 0$  for all $i\in \left[n_U\right]$ do   $g_i={\sum }_{t\in \mathcal{T}}\frac{1}{n_t}{\sum }_{j=1}^{n_t}k(x_j,u_i)$  end for  while $D\ne \oslash$ do   $i^{\ast }= {arg\; min}_{i\in \left[n_U\right]\backslash S^{\left(m\right)}}{g}_i$   $S^{(m+1)}\leftarrow S^{\left(m\right)}\cup \left\{i^{\ast }\right\}$   for all $t\in \mathcal{T}$ do $\hspace{1em}\hspace{1em}\hspace{1em}{\{w_{t,i}^{(m+1)}\}}_{i\in S^{(m+1)}}\leftarrow {arg\; max}_{{\left\{w_{t,i}\right\}}_{i\in S^{(m+1)}}}{\ell }_t\left({\sum }_{i\in S^{(m+1)}}{w}_{t,i}{\delta }_{u_i}\right)$   end for   ${\mathbb{Q}}_t^{(m+1)}\leftarrow {\sum }_{i\in S^{(m+1)}}{w}_{t,i}^{(m+1)}{\delta }_{u_i}$   for all $i\in \left[n_U\right]\backslash S^{(m+1)}$ do    $g_i={\sum }_{t\in D}\nabla {\ell }_t({\mathbb{Q}}^{(m+1)}$   end for   $D=\{X_t:\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t^{(m+1)})>{ϵ}^2\}$   $m\leftarrow m+1$  end while

We evaluate all three methods using a dataset of photographs of 15,367 female-identified students, taken from yearbooks between 1905 and 2013 [45]. We show a random subset of these images in Figure 3. We generated 512-dimensional embeddings of the photos using the torchvision pre-trained implementation of ResNet [42,46]. We then partitioned the collection into 12 datasets, each containing photos from a single decade.

In order to capture lengthscales appropriate for the variation in each decade, we use an additive kernel, setting

$K=\frac{K_{all}+{\sum }_{t\in \mathcal{T}}{K}_t}{\left|\mathcal{T}\right|+1},$

where $K_{all}$ is a squared exponential kernel with bandwidth given by the overall median pairwise distances; $\mathcal{T}$ is the set of decades that index the datasets; $K_t$ is a squared exponential kernel with bandwidth given by the median pairwise distance between images in dataset t.

We begin by considering how good a dependent MMD coreset each algorithm is able to construct, for a given number of exemplars $m=\left|S\right|$. To do so, we ran all algorithms without specifying a threshold ${ϵ}^2$, recording $\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t^{\left(m\right)})$ for each value of m. All algorithms were run for one hour on a 2019 Macbook Pro (2.6 GHz 6-Core Intel Core i7, 32 GB 2667 MHz DDR4), excluding time taken to generate and store the kernel entries, which only occurs one time. As much code as possible was re-used between the three algorithms. Where required, optimization of weights was carried out using a BFGS optimizer. Code is available at https://github.com/sinead/dmmd (accesson 22 September 2021).

Figure 4a shows the per-dataset estimates $\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t^{\left(m\right)})$, and Figure 4b shows the average performance across all 12 datasets. We see that the three algorithms perform comparably in terms of coreset quality. dmmd-opt seems to perform slightly better than dmmd, as might be expected due to the additional optimization step. protodash, by comparison, seems to perform slightly worse, which we hypothesise is because it has no mechanism by which weights can be incorporated at selection time. However, in both cases, the difference is slight.

dmmd is however much faster at generating coresets, since it does not optimize the full set of weights at each iteration. This can be seen in Figure 5, which shows the time taken to generate coresets of a given size. The cost of the optimization-based algorithms grows rapidly with coreset size $\left(m\right)$; the rate of growth of the dmmd coresets is much smaller.

In practice, rather than endlessly minimizing ${\sum }_{t\in \mathcal{T}}\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t)$, we will aim to find ${\mathbb{Q}}_t$ such that $\widehat{{\mathrm{MMD}}^2}(X_t,{\mathbb{Q}}_t)<{ϵ}^2$ for all $t\in \mathcal{T}$. In Figure 6, we show the coreset sizes required to obtain an $ϵ$-MMD dependent coreset on the twelve decade-specific yearbook datasets, for each algorithm. Again, a maximum runtime of one hour was specified. When all three algorithms were able to finish, the coreset sizes are comparable (with dmmd-opt finding slightly smaller coresets than dmmd, and protodash finding slightly larger coresets). However the optimization-based methods are hampered by their slow runtime.

Based on these analyses, it appears there is some advantage to additional optimization of the weights. However, in most cases, we do not feel the additional computational cost merits the improved performance.

4.2. Interpretable Data Summarizations

Summarizations of datasets can allow us to quickly understand properties of their distributions, and allow us to convey such properties to others, for example in a document explaining the data and its providence [47,48]. In high-dimensional, highly structured datasets such as collections of images, traditional summary statistics such as the mean of a dataset are particularly uninterpretable, as they convey little of the shape of the underlying distribution. A better approach is to show the viewer a collection of images that are representative of the dataset. MMD coresets allow us to obtain such a representative set, making them a better choice than displaying a random subset.

As we discussed in Section 3.1, if we wish to summarize a collection of related datasets, independently generated MMD coresets can help us understand each dataset individually, but it may prove challenging to compare datasets. This challenge becomes greater in high dimensional settings such as image data, where we cannot easily intuit a distance between exemplars. To showcase this phenomenon, and demonstrate how dependent MMD coresets can help, we return to the yearbook photos introduced in Section 4.1. For all experiments in this section, we use the additive kernel described in Section 4.1.

4.2.1. A Shared Support Allows for Easier Comparison of Datasets

In Section 3.2, we argued that the shared support provided by dependent MMD coresets facilitates comparison of datasets, since we only need to consider differences in weights. To demonstrate this, we constructed four datasets, each a subset of the entire yearbook dataset containing 250 photos. The first two datasets contained only faces from the 1990s; the second two, only faces from the 2000s. The datasets were generated by sampling without replacement from the associated decades, to ensure no photo appeared more than once across the four datasets. Our goal is to provide a visual way to compare these four datasets.

We begin by independently generating (non-dependent) MMD coresets for the four datasets, using Algorithm 1 independently on each dataset, with a threshold of ${ϵ}^2=0.01$. The set of candidate images, U, was the entire dataset of 15,367 images. The resulting coresets are shown in Figure 7; the areas of the bubbles correspond to the weights associated with each exemplar (The top row of Figure 2 duplicates Figure 7).

We can see that, considered individually, each coreset appears to be doing a good job of capturing the variation in students for each dataset. However, if we compare the four coresets, it is not easy to tell that Figure 7a,b represent the same underlying distribution, and Figure 7c,d represent a second underlying distribution—or to interpret the difference between the two distributions. We see that the highest weighted exemplar for the two 2000s datasets is the same(top left of Figure 7c,d), but only one other image is shared between the two coresets. Meanwhile, the first coreset for the 1990s shares the same highest-weighted image with the two 2000s datasets—but this coreset does not appear in the first 1990s coreset, and the two 1990s coresets have no overlap. Overall, it is hard to compare between the marginal coresets.

By contrast, the shared support offered by dependent coresets means we can directly compare the distributions using their coresets. In Figure 8, we show a dependent MMD coreset (${ϵ}^2=0.01$) for the same collection of datasets. The shared support allows us to see that, while the two decades are fairly similar, there is clearly a stronger similarity between the pairs of datasets from the same year (i.e., similarly sized photos), than between pairs from different years. We can also identify images that exemplify the difference between the two decades, by looking at the difference in weights. We see that many of the faces towards the top of the bubble plot have high weights in the 2000s, but low weights in the 1990s. Examining these exemplars suggests that straight hair became more prevalent in the 2000s. Conversely, many of the faces towards the bottom of the bubble plot have high weights in the 1990s, but low weights in the 2000s. These photos tend to have wavy/fluffy hair and bangs. In conjunction, these plots suggest a tendency in the 2000s away from bangs and towards straight hair, something the authors remember from their formative years. However, there is still a significant overlap between the two decades: many of the exemplars have similar weights in the 1990s and the 2000s.

We can also see this in Figure 9, a bar chart shows the average weights associated with each exemplar in each decades (i.e., the blue bar above a given image is the average weight for that exemplar across the two 1990s datasets, and the red bar is the average weight across the two 2000s datasets). We see that most of the exemplars have similar weights in both scenarios, but that we have a number of straight-haired exemplars disproportionally representing the 2000s, and a number of exemplars with bangs and/or wavy hair disproportionately representing the 1990s. These insights would have been hard to intuit from the standard MMD coresets, where it is hard to identify what variation is due to true underlying differences in the dataset, and what is due to sampling error.

4.2.2. Dependent Coresets Allow Us to Visualize Data Drift in Collections of Images

Next, we show how dependent MMD coresets can be used to understand and visualize variation between collections of multiple datasets. As in Section 4.1, we partition the 15,367 yearbook images into twelve datasets based on their decade, with the goal of understanding how the distribution over yearbook photos changes over time. Table 3 shows the number of photos in each resulting dataset.

Figure 10 shows the exemplars in the resulting dependent MMD coreset, with a threshold of ${ϵ}^2=0.01$. The corresponding plots show how the weights vary with time. The exemplars are ordered based on their average weight across the 12 datasets. In each case, a red, vertical line indicates the year of the yearbook from which the exemplar was taken. We are able to see how styles change over time, moving away from the formal styles of the early 20th century, through waved hairstyles popular in the midcentury, towards longer, straighter hairstyles in later decades. In general, the relevance of an exemplar peaks around the time it was taken (although, this information is not used to select exemplars). However, some styles remain relevant over longer time periods (see many exemplars in the first column). Most of the early exemplars are highly peaked on the 00 s or 10 s; this is not surprising since these pre-WW1 photos tend to have very distinctive photography characteristics and hair styles. Note that we do not include a comparison to standard, independent MMD coresets, as it would not be possible to produce an analogous set of plots—the exemplars in each decade’s coreset would, in general, not overlap.

Figure 10 appears to show that the marginal coresets have high weights on exemplars from the corresponding decade. To look at this in more detail, we consider the distributions over the dates of the exemplars associated with each decade. Figure 11 shows the weighted mean and standard deviation of the years associated with the exemplars, with weights given by the coreset weights. We see that the mean weighted year of the exemplars increases with the decade. However, we notice that it is pulled towards the 1940s and 1950s in each case: this is because we must represent all datasets using a weighted combination of points taken from the convex hull of all datapoints.

4.3. Dependent Coresets Allow Us to Understand Model Generalization

To see how dependent coresets can be used to understand how a model trained on one dataset will generalize to others, we simulate a scenario where we wish to deploy a machine learning model on a given dataset, but where the model was trained on a different dataset. In this scenario, we are interested in learning whether the model generalizes well to the new dataset.

We generate two datasets—one to represent the training data, and one to represent the data used in deployment—by partitioning a collection of image digits. We started with the USPS handwritten digits dataset [49], which comprises a train set of 2791 handwritten digits, and a test set of 2001 handwritten digits. We split the train set into two datasets, $X_a$ and $X_b$, where $X_a$ is skewed towards the earlier digits and $X_b$ towards the later digits. Figure 12 shows the resulting label counts for each dataset: we see there is a clear distributional imbalance. Note that, in general, we will not have such a concise summary of the difference between two datasets; however using image digits as our example allows us to get an idea of the “ground truth” difference between the two datasets.

We selected three classification algorithms to assess generalization performance. We chose classification algorithms because it is easy for us to obtain “ground truth” generalization performance by applying these algorithms to our second dataset $X_b$, allowing us to compare our insights with the true generalization performance. In general, we may not be able to easily estimate generalization performance in this manner: we may have unlabeled data, or our task may not be easily qualitatively evaluated (e.g., evaluating quality of auto-generated captions); we expect our approach to have greatest utility in such scenarios.

We trained three classifiers—a decision tree with maximum depth of 8, a random forest with 100 trees, and a multilayer perceptron (MLP) with a single hidden layer with 100 units—on $X_a$ and the corresponding labels. In each case, we used the implementation in scikit-learn [50], with parameters chosen to have similar train set accuracy on $X_a$. These three models were chosen to have varying generalization accuracy. Table 4 shows the associated classification accuracies on the datasets $X_a$ and $X_b$, which we will use as a quantitative representation of the algorithms’ generalization performance on $X_b$. We also show confusion matrices in Figure 13 and Figure 14. We see that all three algorithms perform comparably on $X_a$, the dataset on which they were trained. However, when applied to $X_b$, we see in Figure 14 that the decision tree struggles in classifying 8 s and 9 s, and that the random forest struggles with 9 s.

We begin our analysis by generating a dependent MMD coreset for the two datasets, with ${ϵ}^2=0.005$. To ensure the exemplars in our coreset have not been seen in training, we let our set of candidate points U be the union of $X_b$ and the USPS test set. As with the yearbook data, we use an additive squared exponential kernel, with bandwidths of the composite kernels being the median within-class pairwise distances, and the overall median pairwise distance. Distances were calculated using the raw pixel values. Figure 15 shows the resulting dependent MMD coreset, with the bars showing the weights $w_{a,i},w_{b,i}$ associated with the two datasets, and the images below the x axis showing the corresponding images $u_i$. In Figure 16, the $u_i$ and $w_i$ have been grouped by number, so that if $y\left(u\right)$ is the label of image u, the jth bar for ${\mathbb{Q}}_a$ has weight ${\sum }_{i\in S:y\left(u_i\right) = j}{w}_{a,j}$.

We can see that the coreset has selected points that cover the spread of the overall dataset. However, looking at Figure 16, we see that the weights assigned to these exemplars in ${\mathbb{Q}}_a$ and ${\mathbb{Q}}_b$ mirror the relative frequencies of each digit in the corresponding datasets $X_a$ and $X_b$ (Figure 12).

We then considered all points $u_i$ in our dependent coreset $({\mathbb{Q}}_a,{\mathbb{Q}}_b)$ where $f_i=w_{b,i}/w_{a,i}>2$—i.e., points that are much more representative of $X_b$ than $X_a$. We then looked at the class probabilities of the three algorithms, on each of these points, as shown in Figure 17. We see that the decision tree mis-classifies nine of the 21 exemplars, and is frequently highly confident in its misclassification. The random forest misclassifies three examples, and the MLP misclassifies two. We see this agrees with the ordering provided by empirically evaluating generalization in Table 4—the MLP generalizes best, and the decision tree worst. As suggested by our confusion matrices in Figure 14, we see that all algorithms generalize worst to the numbers 8 and 9—this is to be expected, since these digits are most under-represented in $X_a$. The decision tree in particular appears to fail on these digits, mirroring the quanitative results in the confusion matrix.

For comparison, in Figure 18 we show the points where $f_i<0.5$—i.e., points that are much more representative of $X_a$ than $X_b$. Note that, since our candidate set did not include any members of $X_a$, none of these points were in our training set. Despite this, the accuracy is high, and fairly consistent between the three classifiers (the decision tree misclassifies two exemplars; the other two algorithms make no errors).

The dependent coreset only provides information about performance on “representative” members of $X_b$. Since classifiers will tend to underperform on outliers, looking only at the dependent MMD coreset does not give us a full picture of the expected performance. We can augment our dependent MMD coreset with criticisms—points that are poorly described by the dependent coreset. Figure 19 shows the performance of the three algorithms on a size-20 set of criticisms for $X_b$. Note that, overall, accuracy is lower than for the coreset—unsurprising, since these are outliers. However, as before, we see that the decision tree performs worst on these criticisms (nine mis-classifications), with the other two algorithms performing slightly better (six mis-classifications for the random forest, and seven for the MLP).

Note that, since accuracy does not correspond to a function in a RKHS, we cannot expect to use the coreset to bound the expected accuracy of an algorithm on the full dataset. Indeed, while the coresets and critics correctly suggest that the decision tree generalizes poorly, they do not give conclusive evidence on the relative generalization abilities of the other two algorithms. However, they do highlight what sort of data points are likely to be poorly modeled under each algorithm. By providing a qualitative assessment of performance modalities and failure modes on either typical points for a dataset, or points that are disproportionately representative of a dataset (vs the original training set) dependent MMD coresets allows users to identify potential generalization concerns for further exploration.

4.4. Discussion

MMD coresets have already proven to be a useful tool for summarizing datasets and understanding models. However, as we have shown in Section 4.2, their interpretability wanes when used to compare related datasets. Dependent MMD coresets provide a tool to jointly model multiple datasets using a shared set of exemplars. This shared set of exemplars makes it easy to compare two datasets, providing an interpretable summary not just of each dataset in isolation, but also of the difference between datasets. As such, we believe they will prove useful in understanding related datasets, and summarizing such collections of datasets.

In addition to facilitating understanding of data, we have also shown that dependent MMD coresets can be used to better understand model performance. By considering the weights associated with two different datasets, we can identify areas of domain mis-match. By exploring performance of algorithms on such points, we can glean insights about the ability of a model to generalize to new datasets.

In principle, dependent MMD coresets can be applied to any number of datasets. However, as we discuss in Section 3.5, the computational cost of our algorithm will scale cubically in the number of datapoints in the union of the datasets. This cost can be reduced by representing each dataset with an independent coreset, either obtained by subsampling the data or by applying a coreset selection algorithm such as [14,15,16,17]; however, the approximation error of this coreset would need to be incorporated into the overall approximation error $ϵ$.

An alternative approach might be to develop streaming algorithms for constructing dependent MMD coresets. In the non-dependent setting, streaming algorithms such as [16] allow us to construct a coreset in an online manner, at a lower computational cost than batch algorithms. Such an approach would be particularly appealing in the case of time-stamped data, since it would allow us to update our dependent MMD coreset to include a new dataset.

Dependent MMD coresets are just one example of a dependent coreset that could be constructed using this framework. Future directions include exploring dependent analogues of other measure coresets [22].

Author Contributions

Conceptualization, S.A.W. and J.H.; methodology, software, and experiments, S.A.W.; data curation: S.A.W. and J.H.; writing and visualization: S.A.W. and J.H. Both authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding. Part of the work was completed while S.A.W. was employed by CognitiveScale.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Datasets and code available at https://github.com/sinead/dmmd (accessed on 22 September 2021).

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. (a) Three equally weighted Gaussians (lines show 1, 2, 3 standard deviations of each component). (b,c) Independently sampled datasets from the mixture of three Gaussians. (d,e) MMD coresets for the three-Gaussian datasets.

Figure 2. Independently learned MMD coresets for two randomly selected dataset of 250 female-identified photographs from US yearbooks in the 1990s. Area of each bubble is proportional to weight of the corresponding exemplar.

Figure 3. A random subset of 100 images taken from the yearbook dataset.

Figure 4. Evaluating how coreset quality varies with number of exemplars, for dependent MMD coresets generated using three algorithms, on 12 yearbook datasets.

Figure 5. Time (in seconds) taken to construct MMD dependent coresets of a given size, for three algorithms, on the 12 yearbook datasets. Algorithms ran for a maximum of one hour.

Figure 6. Coreset size required to obtain an [Forumla omitted. See PDF.]-MMD dependent coreset on the 12 yearbook datasets, for three algorithms. Algorithms ran for a maximum of one hour; protodash failed to complete coresets for [Forumla omitted. See PDF.] and [Forumla omitted. See PDF.]. dmmd-opt failed to complete a coreset for [Forumla omitted. See PDF.].

Figure 7. Independently learned, weighted 0.1-MMD coresets based on 250 random samples from a given decade. Area of each bubble is proportional to the weight of the corresponding exemplar in the coreset.

Figure 8. Dependent 0.1-MMD coreset for a collection of 4 datasets, each including 250 random samples from a given decade. Area of each bubble is proportional to the weight of the corresponding exemplar in the marginal coreset. Positioning is constant across all four examples.

Figure 9. Summary of a dependent 0.1-MMD coreset for four datasets of yearbook faces from the 1990s and 2000s. Exemplars are shown along the x axis. The average weight for each exemplar in the coresets associated with the 1990s is shown in blue [Image omitted. Please see PDF.]; the average weight for the 2000s is shown in red [Image omitted. Please see PDF.].

Figure 10. Visualization of an 0.1-MMD dependent coreset for 12 datasets, each containing yearbook photos from a given decades. Photos show the exemplars [Forumla omitted. See PDF.], ordered by their average weight across the 12 marginal coresets. To the left of each photo is a plot of the corresponding weight over time; a red vertical line marks the year the photo was taken.

Figure 11. Distribution over the year associated with the marginal coresets for each decade. Plot shows weighted mean ± weighted standard deviation.

Figure 12. Frequency with which each digit occurs in two datasets of handwritten digits.

Figure 13. Confusion matrices on [Forumla omitted. See PDF.], for three classification algorithms trained on [Forumla omitted. See PDF.].

Figure 14. Confusion matrices on [Forumla omitted. See PDF.], for three classification algorithms trained on [Forumla omitted. See PDF.].

Figure 15. Dependent MMD coreset for two datasets of handwritten digits. Bars show the weight in each marginal coreset, with [Forumla omitted. See PDF.] shown in blue [Image omitted. Please see PDF.] and [Forumla omitted. See PDF.] shown in red [Image omitted. Please see PDF.]; images along axis show corresponding exemplars.

Figure 16. Summary of a dependent MMD coreset for two datasets of handwritten digits. The weights and exemplars from Figure 15 have been combined based on their label. Weights for [Forumla omitted. See PDF.] are shown in blue [Image omitted. Please see PDF.] and weights for [Forumla omitted. See PDF.] are shown in red [Image omitted. Please see PDF.].

Figure 17. Exemplars over-represented in [Forumla omitted. See PDF.], with class probabilities under three algorithms trained on [Forumla omitted. See PDF.]. The true class is shown in blue [Image omitted. Please see PDF.]; where the highest probability class differs from the true class, the highest probability class is shown in red [Image omitted. Please see PDF.].

Figure 18. Exemplars over-represented in [Forumla omitted. See PDF.], with class probabilities under three algorithms trained on [Forumla omitted. See PDF.]. The true class is shown in blue [Image omitted. Please see PDF.]; where the highest probability class differs from the true class, the highest probability class is shown in red [Image omitted. Please see PDF.].

Figure 19. Criticisms of [Forumla omitted. See PDF.] from the dataset [Forumla omitted. See PDF.], with class probabilities under three algorithms trained on [Forumla omitted. See PDF.]. The true class is shown in blue [Image omitted. Please see PDF.]; where the highest probability class differs from the true class, the highest probability class is shown in red [Image omitted. Please see PDF.].

Table 1

Notation used in this paper.

$$\mathcal{T}$$	set that indexes datasets and associated measures
$$X_t=(x_{t,1},\dots ,x_{t,n_t})\in {\mathcal{X}}^{n_t}$$	a dataset indexed by $$t\in \mathcal{T}$$
$${\mathbb{P}}_t$$	true distribution at $$t\in \mathcal{T}$$, $$X_t\sim {\mathbb{P}}_t$$
$$U=(u_1,\dots ,u_{n_u})$$	set of candidate locations
$${\delta }_u$$	Dirac measure (i.e., point mass) at u.
$${\mathbb{Q}}_t$$	a probability measure used to approximate $${\mathbf{P}}_t$$, that takes the form $${\sum }_{i\in S}{w}_{t,i}{\delta }_{u_i}$$, where $$S\subset \left[n_u\right]$$

Table 2

Some examples of integral probability metrics.

Distance	$$\mathcal{F}$$
1-Wasserstein distance	$$\{f:||\nabla f|{|}_1\le 1\}$$
Maximum mean discrepancy	$$\{f:|\left|f\right|{|}_{\mathcal{H}}\le 1\}$$ for some RKHS $$\mathcal{H}$$
Total variation	$$\{f:|\left|f\right|{|}_{\infty }\le 1\}$$

Table 3

Number of yearbook photos for each decade.

1900s	1910s	1920s	1930s	1940s	1950s	1960s	1970s	1980s	1990s	2000s	2010s
35	98	308	1682	2650	2093	2319	2806	2826	2621	2208	602

Table 4

Accuracies of three classification algorithms, on datasets $$X_a$$ and $$X_b$$. All algorithms were trained on $$X_a$$.

Model	Accuracy on $${\mathit{X}}_{\mathit{a}}$$	Accuracy on $${\mathit{X}}_{\mathit{b}}$$
MLP	0.9998	0.8531
Random Forest	1.0	0.7129
Decision Tree	0.9585	0.5880

References

1. Larrazabal, A.J.; Nieto, N.; Peterson, V.; Milone, D.H.; Ferrante, E. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl. Acad. Sci. USA; 2020; 117, pp. 12592-12594. [DOI: https://dx.doi.org/10.1073/pnas.1919012117] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32457147]

2. Chen, I.Y.; Johansson, F.D.; Sontag, D. Why is my classifier discriminatory?. Proceedings of the 32nd International Conference on Neural Information Processing Systems; Montreal, QC, Canada, 3–8 December 2018; pp. 3543-3554.

3. Buolamwini, J.; Gebru, T. Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency; New York, NY, USA, 23–24 February 2018; pp. 77-91.

4. Shankar, S.; Halpern, Y.; Breck, E.; Atwood, J.; Wilson, J.; Sculley, D. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. arXiv; 2017; arXiv: 1711.08536

5. Alexander, R.G.; Schmidt, J.; Zelinsky, G.J. Are summary statistics enough? Evidence for the importance of shape in guiding visual search. Vis. Cogn.; 2014; 22, pp. 595-609. [DOI: https://dx.doi.org/10.1080/13506285.2014.890989]

6. Lauer, T.; Cornelissen, T.H.; Draschkow, D.; Willenbockel, V.; Võ, M.L.H. The role of scene summary statistics in object recognition. Sci. Rep.; 2018; 8, 14666. [DOI: https://dx.doi.org/10.1038/s41598-018-32991-1]

7. Kaufmann, L.; Rousseeuw, P. Clustering by means of medoids. Statistical Data Analysis Based on the L1-Norm and Related Methods; Springer: Berlin/Heidelberg, Germany, 1987; pp. 405-416.

8. Bien, J.; Tibshirani, R. Prototype selection for interpretable classification. Ann. Appl. Stat.; 2011; 5, pp. 2403-2424. [DOI: https://dx.doi.org/10.1214/11-AOAS495]

9. Mak, S.; Joseph, V.R. Projected support points: A new method for high-dimensional data reduction. arXiv; 2017; arXiv: 1708.06897

10. Mak, S.; Joseph, V.R. Support points. Ann. Stat.; 2018; 46, pp. 2562-2592. [DOI: https://dx.doi.org/10.1214/17-AOS1629]

11. Kim, B.; Khanna, R.; Koyejo, O.O. Examples are not enough, learn to criticize! Criticism for interpretability. Proceedings of the 30th International Conference on Neural Information Processing Systems; Barcelona, Spain, 5–10 December 2016; pp. 2280-2288.

12. Wilson, D.R.; Martinez, T.R. Reduction techniques for instance-based learning algorithms. Mach. Learn.; 2000; 38, pp. 257-286. [DOI: https://dx.doi.org/10.1023/A:1007626913721]

13. Gurumoorthy, K.S.; Dhurandhar, A.; Cecchi, G.; Aggarwal, C. Efficient data representation by selecting prototypes with importance weights. Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM); Beijing, China, 8–11 November 2019; pp. 260-269.

14. Chen, Y.; Welling, M.; Smola, A. Super-samples from kernel herding. Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence Uncertainty in Artificial Intelligence; Catalina Island, CA, USA, 8–11 July 2010; pp. 109-116.

15. Phillips, J.M.; Tai, W.M. Near-optimal coresets of kernel density estimates. Discret. Comput. Geom.; 2020; 63, pp. 867-887. [DOI: https://dx.doi.org/10.1007/s00454-019-00134-6]

16. Karnin, Z.; Liberty, E. Discrepancy, coresets, and sketches in machine learning. Proceedings of the 32nd Conference on Learning Theory Conference on Learning Theory; Phoenix, AZ, USA, 25–28 June 2019; pp. 1975-1993.

17. Tai, W.M. Optimal Coreset for Gaussian Kernel Density Estimation. arXiv; 2021; arXiv: 2007.08031

18. Gretton, A.; Borgwardt, K.M.; Rasch, M.J.; Schölkopf, B.; Smola, A. A kernel two-sample test. J. Mach. Learn. Res.; 2012; 13, pp. 723-773.

19. Pratt, K.B.; Tschapek, G. Visualizing concept drift. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Washington, DC, USA, 24–27 August 2003; pp. 735-740.

20. Hohman, F.; Wongsuphasawat, K.; Kery, M.B.; Patel, K. Understanding and visualizing data iteration in machine learning. Proceedings of the CHI Conference on Human Factors in Computing Systems; Honolulu, HI, USA, 25–30 April 2020; pp. 1-13.

21. Agarwal, P.K.; Har-Peled, S.; Varadarajan, K.R. Approximating extent measures of points. J. ACM; 2004; 51, pp. 606-635. [DOI: https://dx.doi.org/10.1145/1008731.1008736]

22. Claici, S.; Solomon, J. Wasserstein coresets for Lipschitz costs. Stat; 2018; 1050, 18.

23. Müller, A. Integral probability metrics and their generating classes of functions. Adv. Appl. Probab.; 1997; 29, pp. 429-443. [DOI: https://dx.doi.org/10.2307/1428011]

24. Bach, F.; Lacoste-Julien, S.; Obozinski, G. On the equivalence between herding and conditional gradient algorithms. Proceedings of the 29th International Conference on Machine Learning; Edinburgh, UK, 26 June–1 July 2012.

25. Lacoste-Julien, S.; Lindsten, F.; Bach, F. Sequential kernel herding: Frank-Wolfe optimization for particle filtering. Proceedings of the 18th International Conference on Artificial Intelligence and Statistics; San Diego, CA, USA, 9–12 May 2015; pp. 544-552.

26. Phillips, J.M. ε-samples for kernels. Proceedings of the 24th Annual ACM-SIAM Symposium on Discrete Algorithms; New Orleans, LA, USA, 6–8 January 2013; pp. 1622-1632.

27. Lopez-Paz, D.; Muandet, K.; Schölkopf, B.; Tolstikhin, I. Towards a learning theory of cause-effect inference. Proceedings of the 32nd International Conference on Machine Learning; Lille, France, 7–9 July 2015; pp. 1452-1461.

28. Feldman, D. Introduction to core-sets: An updated survey. arXiv; 2020; arXiv: 2011.09384

29. Potter, M.C.; Wyble, B.; Hagmann, C.E.; McCourt, E.S. Detecting meaning in RSVP at 13 ms per picture. Atten. Percept. Psychophys.; 2014; 76, pp. 270-279. [DOI: https://dx.doi.org/10.3758/s13414-013-0605-z] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/24374558]

30. Zheng, Y.; Ou, Y.; Lex, A.; Phillips, J.M. Visualization of big spatial data using coresets for kernel density estimates. Proceedings of the IEEE Visualization in Data Science (VDS); Phoenix, AZ, USA, 1 October 2017; pp. 23-30.

31. Kim, B.; Rudin, C.; Shah, J.A. The Bayesian case model: A generative approach for case-based reasoning and prototype classification. Proceedings of the 28th International Conference on Neural Information Processing Systems; Montreal, QC, Canada, 8–13 December 2014; pp. 1952-1960.

32. Aamodt, A.; Plaza, E. Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Commun.; 1994; 7, pp. 39-59. [DOI: https://dx.doi.org/10.3233/AIC-1994-7104]

33. Murdock, J.W.; Aha, D.W.; Breslow, L.A. Assessing elaborated hypotheses: An interpretive case-based reasoning approach. Case-Based Reasoning Research and Development, Proceedings of the 5th International Conference on Case-Based Reasoning, Trondheim, Norway, 23–26 June 2003; Springer: Berlin/Heidelberg, Germany, 2003; pp. 332-346.

34. MacEachern, S.N. Dependent nonparametric processes. ASA Proceedings of the Section on Bayesian Statistical Science; American Statistical Association: Alexandria, VA, USA, 1999; Volume 1, pp. 50-55.

35. Quintana, F.A.; Mueller, P.; Jara, A.; MacEachern, S.N. The dependent Dirichlet process and related models. arXiv; 2020; arXiv: 2007.06129

36. De Iorio, M.; Müller, P.; Rosner, G.L.; MacEachern, S.N. An ANOVA model for dependent random measures. J. Am. Stat. Assoc.; 2004; 99, pp. 205-215. [DOI: https://dx.doi.org/10.1198/016214504000000205]

37. Dubey, A.; Hefny, A.; Williamson, S.; Xing, E.P. A nonparametric mixture model for topic modeling over time. Proceedings of the 13th SIAM International Conference on Data Mining; Austin, TX, USA, 2–4 May 2013; pp. 530-538.

38. Garreau, D.; Jitkrittum, W.; Kanagawa, M. Large sample analysis of the median heuristic. arXiv; 2017; arXiv: 1707.07269

39. Kiela, D.; Bottou, L. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP); Doha, Qatar, 25–29 October 2014; pp. 36-45.

40. Chen, M.; Radford, A.; Child, R.; Wu, J.; Jun, H.; Luan, D.; Sutskever, I. Generative pretraining from pixels. Proceedings of the International Conference on Machine Learning; Online, 13–18 July 2020; pp. 1691-1703.

41. Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning; Online, 13–18 July 2020; pp. 1597-1607.

42. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA, 27–30 June 2016; pp. 770-778.

43. Athalye, A.; Engstrom, L.; Ilyas, A.; Kwok, K. Synthesizing robust adversarial examples. Proceedings of the 35th International Conference on Machine Learning; Stockholm, Sweden, 10–15 July 2018; pp. 284-293.

44. Steed, R.; Caliskan, A. Image representations learned with unsupervised pre-training contain human-like biases. Proceedings of the 4th Conference on Fairness, Accountability, and Transparency; Online, 3–10 March 2021; pp. 701-713.

45. Ginosar, S.; Rakelly, K.; Sachs, S.; Yin, B.; Efros, A.A. A century of portraits: A visual historical record of American high school yearbooks. Proceedings of the IEEE International Conference on Computer Vision Workshops; Santiago, Chile, 7–13 December 2015; pp. 1-7.

46. Marcel, S.; Rodriguez, Y. Torchvision the machine-vision package of torch. Proceedings of the 18th ACM International Conference on Multimedia; Firenze, Italy, 25–29 October 2010; pp. 1485-1488.

47. Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J.W.; Wallach, H.; Daumé, H., III; Crawford, K. Datasheets for datasets. Proceedings of the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning; Stockholm, Sweden, 13–15 July 2018.

48. Chmielinski, K.S.; Newman, S.; Taylor, M.; Joseph, J.; Thomas, K.; Yurkofsky, J.; Qiu, Y.C. The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence. Proceedings of the NeurIPS 2020 Workshop on Dataset Curation and Security; Online, 11 December 2020.

49. Hull, J.J. A database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell.; 1994; 16, pp. 550-554. [DOI: https://dx.doi.org/10.1109/34.291440]

50. Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V. et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res.; 2011; 12, pp. 2825-2830.