1. Introduction
Radar sensing has been classically used in an extensive range of applications, due to its ability to operate under all-weather and scene illumination geometry independence acquisition conditions. This would be a critical advantage, under specific circumstances, when compared, for instance, to optical sensing. Advances in radar hardware and software technology have made it possible to reliably detect and track objects, under competitive classification accuracy conditions, in underwater, air, and ground environments [1,2]. However, this framework seems to have been applied only to relatively big objects in specific scenarios, i.e., airplanes, ships, or submarines [3,4].
Radar technology has also found an important field of application in remote sensing data classification and health monitoring for environmental preservation purposes, usually combined in a common optical remote sensing framework [5,6].
Apart from these research fields, radar sensors are also being applied nowadays in other areas, as in action and gesture recognition or in autonomous driving, to cite a few cases. In particular, there is an increasing interest in the use of radar technology in human gesture and action recognition because it is aimed at solving the problem of low recognition accuracy that vision based systems may have. Not only gestures [7] but also more complex actions are aimed at by the new radar acquisition technology and classification strategies [8]. Even daily or ordinary activities (e.g., cooking, eating, and resting) may be classified using this type of acquisition technology [9]. Usually, human action recognition is made using sensors whose location is fixed, but new approaches using unmanned aerial vehicles (UAVs) are starting to appear [10].
Another area where radar technology is being applied is related to object and people classification for systems aimed at helping create a safer driving environment, or even in autonomous driving [11,12].
Some of the newest radar sensing applications do not try to obtain information from objects over long, mid, or short distances (in the range of meters), but just within a few centimeters. A sensor developed by Google, called Soli, was announced in 2015 and obtained great media interest. It is a millimeter-wave radar, and it has found several research applications. In [13], this sensor is used to identify up to 26 types of materials and 10 body parts from several participants. In [14], it is used to classify and distinguish five common types of materials, namely aluminum, ceramic, plastic, wood, and water, regardless of their different sizes and thicknesses. A hand gesture recognition system, which successfully distinguishes 10 gestures, is proposed in [15]. This type of radar sensor has even been used for face verification [16] or to differentiate between blood samples of disparate glucose concentrations in the range of 0.5 to 3.5 mg/mL [17]. However, the number of research studies focused on the application of this technology for daily object material classification and other interactions seems to still be scarce [18]. A Tangible User Interface (TUI) allows the user to interact with digital information through real actions. This type of user interfaces is known to be more usable and easier to understand, especially for elderly people [19], since, although not everyone knows how to operate a keyboard or mouse, everyone is familiar with grasping or moving common objects. The idea behind TUIs is that there is a direct link between the digital system and the way the physical objects are manipulated.
Miniature milimeter-wave radar sensing has been proposed to enhance the interactions by identifying materials or estimating the orientation or distance at which the physical elements are located [18]. Millimeter-wave radar technology could be considered a cost effective option for material identification, counting objects/items, or estimating their position, even when they might be partially covered or occluded. This property is a great advantage when compared to other sensors such as cameras.
The aim of this paper is to apply a diverse group of classification strategies and feature extraction techniques on a series of datasets of different nature, acquired by a portable radar sensor, aiming at validating this technology as a good candidate to be used in TUI sensing problems. The classification strategies include Random Forests [20] and Support Vector Machines [21]. Moreover, different classifiers are combined in ensembles, using Stacked Generalization [22].
The feature extraction techniques applied in this paper are the basic aggregation features (e.g., averages and root mean squares) previously used by Yeo et al. [18] and two techniques originally introduced for time series classification: ROCKET [23] and TSFRESH [24]. The datasets used in this study, obtained using a radar system, are not time series. Nevertheless, as in time series, the features are arranged and therefore their order is important. For instance, functions on the values in an interval (e.g., the first half of the series or channel) can be used. Hence, methods proposed for times series classification can also be used for this kind of data. It is common practice to use time series classification methods (including multivariate time series) for datasets where the feature order is important even though the features do not represent different times [25,26], for instance, from spectroscopy [27,28]. Image contours or outlines, such as arrowheads [29], leaves [30], or fish species [31], can be represented as time series.
The rest of the paper is organized as follows. Section 2 describes the group of datasets, as well as the corresponding feature extraction and classification algorithms used. Section 3 discusses the results. These results are validated and analyzed using average rankings and post hoc tests such as the Nemenyi test, as well as by using Bayesian Signed-Rank Test, all of them in order to determine which combination of feature vector and classification method is best, and whether this improvement (i.e., difference in accuracy performance) is statistically significant or not. Section 4 presents the conclusions and discusses potential future lines of research.
2. Datasets, Feature Extraction, and Classification Methods
Radar sensing uses an electromagnetic signal that hits an object. This signal might be directly reflected or scattered/absorbed by it, therefore giving an overview of the properties of the material the object is made of or of the distance or position the object is at. The type of radar whose data form the repository we used is a Frequency-Modulated Continuous-Wave (FMCW) radar, and this type of radar system has shown its potential to be used for detection/recognition purposes.
Soli [18] is a mono-static, multi-channel (8 channels = 2 transmitters × 4 receivers) radar device, operating in the 57–64 GHz range (center frequency of 60 GHz), using an FMCW principle, where the radar transmits and receives information on a continuous basis. When an object is placed on the top or nearby the Soli sensor, the energy transmitted from the sensor is absorbed and scattered by the object, which varies depending on its distance, thickness, shape, density, internal composition, and surface properties, commonly described as the Radar Cross Section (RCS). As a result, the signals that are reflected back represent rich information about the contributions from a range of surface and internal properties.
Nevertheless, the complex mixing nature of the signals obtained by the radar sensor (due to these above-mentioned multiple factors that are contributing to the final, detected, and signal) make them have non-smooth and complex shapes, which contribute with an additional complexity degree to a potential signal classification strategy.
The aim of Section 2 is to present a coherent explanation of the different feature extraction techniques and classification strategies applied on a series of datasets obtained by Yeo et al. [18], using the Soli radar sensor. It is composed of the following subsections: Section 2.1 gives a detailed overview about the different types of datasets used in our classification performance analysis framework. Section 2.2 explains the feature extraction methods that are applied. Feature selection methods are considered taking into account their number in relation to the number of instances of each dataset (the so-called curse of dimensionality). Section 2.3 describes the different classification strategies applied.
2.1. Radar Datasets
A previous study [18] showed that Soli could be used to identify materials, estimate number of objects, their orientation, or the the distance to the sensor. Yeo et al. [18] created a series of supervised classification datasets, formed by on the one hand the signal acquired by this sensor, and the class of the material on the other hand. The type of material, distance, and other features were also included. The following is a brief schematic description about the different types of categories these datasets are formed by:
⋄. Material identification (of the material an object is made of), from a limited list of materials.
⋄. Object identification, from a series of objects in the same category (i.e., a credit card of a specific bank from the rest of the credit cards).
⋄. Counting the number of elements that might be piled up on the surface of the sensor.
⋄. Distance estimation, from an object to the sensor.
⋄. Order identification, of the items in a battery of objects.
⋄. Flipping identification, where the orientation of an object may be inferred.
⋄. Movement, where the angular position of the object or changes in the movement of one of them in relation to other(s) are obtained.
The complete group of datasets can be found at:
Figure 1a,b shows the eight 64D signals obtained by the radar sensor for two instances, each from a different dataset. We can see the complex and irregular shape of the signals detected by the sensor.
Table 1 shows the total number of samples and classes for each one of the files. These datasets show different acquisition conditions over a series of distinct objects, including playing cards, poker chips, and Lego blocks. For particular details and a deeper description about the acquisition conditions for the creation of these sets, the reader is referred to Section 6 in [18].
2.2. Feature Extraction
The most straightforward way to apply a classification strategy on this dataset would be to use the complete 512-dimensional vector (raw data) as the feature vector. However, raw data often contain noise, redundancies, or irrelevant information; thus, there are different feature selection/extraction techniques that could be applied on such a feature space and subsequently be used instead or added to this feature vector. The following additional feature extraction techniques are considered in our case:
Basic aggregation features: A series of basic features are extracted. They are defined as aggregations at different levels:
∘. Along the signals of the eight channels: Average (AVG) and the average of the absolute values (ABS). There are 64 values in each case and 128 (=) values in total.
∘. For each channel (×8): Absolute mean square (AMS) and root mean square (RMS). There are 8 for each channel and 16 values in total.
∘. At a global level: Maximum, minimum, mean, AVG, average of absolute values (ABS), and root means square (RMS). There are five values in total.
ROCKET (RandOm Convolutional KErnel Transform [23]): It is a method that achieves state-of-the-art classification accuracy in several benchmarks, but it only requires a fraction of the training time used by other existing standard methods. Other methods for time series classification focus only on a single type of representation such as shape, frequency, or signal variance. Convolutional kernels are able to represent multiple characteristics of different types simultaneously.
TSFRESH (Time Series FeatuRe Extraction based on Scalable Hypothesis tests): This methodology aims to avoid the time-consuming process of meaningful feature identification and extraction, from time series data. It consists of an algorithm and a Python package. The Python package implements the extraction of 794 features using multiple characterization methods, each of them executed with various sets of parameters. This framework also includes feature selection to identify those features that are statistically significant. The algorithm is described in [32]. The software package is presented in [24]. The total number of features is therefore 6352 (=).
In this framework, each feature vector is individually analyzed in relation to its significance for predicting the target class label. As a result, a vector of p-values is obtained. This vector is assessed against the Benjamini–Yekutieli procedure [33], which allows the method to decide which features to keep. This selection strategy considers the application of a threshold, whose default value is 0.05. However, this selection is so restrictive in some cases that most of the features are discarded. An iterative method is applied that considers the threshold values and selects the first one that keeps at least a 10% of the total number of attributes.
2.3. Classification Methods
Two main blocks of classification strategies were considered to be applied on the datasets, summarized as the use of: (a) a Support Vector Machine and a Random Forest classifier, being these two classifiers those that were used in [18] (presented in Section 2.3.1); (b) a Stacked Generalization approach (a particular type of ensemble machine learning algorithm) (Section 2.3.2), where we try to take advantage of the diversity that different classification methods may give, in order to help improve classification accuracy, when using them in a coordinated/simultaneous way.
2.3.1. Single Type Classifiers
Two classification methodologies were applied: (1) Support Vector Machines (SVM); (2) Random Forests (RFs). SVM is a widely used classification method, in different and varied areas of research, partly because of its capability and good behavior when dealing with problems with a small number of samples in (relation to) high-dimensional feature spaces. Originally developed and applied in linearly separable problems, it is aimed at obtaining the hyperplane whose distance to the two groups of data points (called margin), representing the two classes, was maximal. SVM was generalized later to deal with nonlinearly separable problems using the so-called (transformational) Kernel trick [21]. A mathematical transformation function is applied to map the nonlinear separable dataset into a higher dimensional space where the samples can be linearly separated using an hyperplane.
Under this mathematical framework, two parameters emerge . The optimal value of these parameters is problem dependent. A Grid Search strategy was applied to assess their optimal values. The parameter C search interval and step size was , ,…, , and the corresponding interval and step size was , ,…, . Whenever the best best pair of parameters was obtained in the interval limits, the Grid Search was automatically extended by a factor of two. This procedure follows the guidelines given in [34].
Ensemble learning methods are based on the idea of using multiple learning algorithms to obtain better predictive performance than one could obtain from any of them, separately. The idea behind ensemble learning is the way an expert committee works in real life, i.e., it is usually easier to properly predict something when the prediction is made by more than one expert, and a consensus is obtained from them. Ensembles are combinations of several classifiers, which are often called base classifiers. There are several types of ensembles, which may be divided into two large groups: (a) homogeneous ensembles, where all the base classifiers are built using the same algorithm (but with different versions of the dataset or different training parameters); (b) heterogeneous ensembles, where the base classifiers are built using different algorithms.
Diversity is a key property in the search for an optimal ensemble strategy performance, since there is no benefit when combining base classifiers that always obtain the same predictions. There are several techniques to induce diversity in homogeneous ensembles. In Bagging [35], for instance, each classifier is trained with a variant of the training dataset, which uses different random samples of the training set. Random Forests [20] are ensembles of Decision Trees [36]. In this method, the diversity during the training process is enforced by combining the sampling of the training set, as Bagging does, with the random selection of subsets of attributes in each node of the tree. This way, in each node, the splits only consider the selected subset of attributes. Later, on the prediction stage, each base classifier predicts a class, and the class selected the most (the mode) is the final prediction of the ensemble. RFs are used to correct the tendency of the decision trees to overfit. The main parameter of an RF is its size (i.e., the number of trees that are generated into the ensemble). In our study, 100 decision trees were used because it is a usual and sufficient number [37].
2.3.2. Stacked Generalization
Stacked Generalization is an ensemble method where a new model learns how to best combine the predictions from multiple existing models. In this approach, any learning algorithm could be used to combine them.
In particular, the generation of classifiers that are accurate and diverse is only the first part in an ensemble classifier generation process. The second part (as important as the former one) is the method used to obtain the ensemble outputs by combining the outputs of the base classifiers. Two of the approaches used to combine the outputs of the base classifiers are: (a) majority voting; (b) average of probabilities. Alternatively, there are methods that may be able to learn the so-called combination rules. These methods (called meta-classifiers) are particularly useful when the base classifiers do not have the same success rate (among them) when classifying instances. This may happen when the base classifiers are generated using different training sets or different training algorithms.
Stacked Generalization (also called Stacking) [22] builds a classifier that takes as inputs values, the output values of the base classifiers, and learns to map these values of the base classifiers into the correct final output value. In other words, no voting strategy is applied in order to combine the predictions of the base classifiers. In this case, a meta-classifier is used. The base classifiers are trained with the training set and the meta-classifier is trained with the predictions of the base classifiers. Figure 2 shows a scheme of one of the stacking approaches used.
The predictions of the classifiers are obtained from a different partition set than the one used for training. This is achieved by dividing the training set into several partitions. Therefore, Stacking can also be seen as a sophisticated form of attribute extraction. Stacking base classifiers are usually trained using different algorithms. Another strategy would be to use different views, i.e., subsets of attributes obtained by each feature extraction method. This type of Stacking strategy is often called multi-view stacking and has been successfully used when applied to other (but somehow similar) problems [38,39,40].
Figure 3 shows a flow diagram of the general data processing strategy followed in the paper. The figure shows the processing chain divided into two parts. The first of them shows that different types of features and feature combination strategies are obtained from the raw data values, and different classifiers are trained and used to obtain the classification accuracy results for the different datasets. In the second part, these classification accuracy results, for each dataset, are ranked, and the statistical significance of their differences are obtained in order to assess which method is better, and whether the differences among classification strategies are statistically significant or not.
3. Results and Discussion
Accuracy (for the Random Forest, SVM Linear with default parameters (rhe same configuration as the one used in [18]), and the optimized Gaussian SVM classifiers) was assessed using different combinations of the attributes explained in Section 2.2. Table 2 shows these combination strategies: The symbol `&’ means that the referred attributes are concatenated. Stacking was applied using two different configurations, called as follows:
⋄. Stacking:All: Eight base classifiers were assembled (four RF classifiers and four SVM linear classifiers), one for each one of the four extracted feature sets (raw features (O), basic aggregation features (B), ROCKET with 1000 kernels (R1000), and TSFRESH with feature selection (t)).
⋄. Stacking+:All: Ten base classifiers were assembled: the eight base classifiers described in the previous case, a linear SVM, and an RF, trained in both cases with the concatenation of all the attributes.
In all the results that follow, we use SVM-L (SVM with Linear Kernel) to refer to SVM with default parameters and SVM-G to refer to Grid search-optimized SVM with RBF Kernel. We simply use RF for the Random Forest classifier. Features used for training the classifiers use the same abbreviation as that shown in Table 2.
In order to compare the performance of the different classification methods, we might use the average accuracy of each one of the pairs (Classifier:Feature Set) evaluated throughout the 34 datasets shown in Table 1. Nevertheless, when comparing multiple methods on multiple datasets, an alternative to (and sometimes more appropriate way than) comparing average accuracies is to use average ranks [41]. Average ranks are computed in the following way: For a given dataset, the methods (in this case, a method is a pair formed by Classifier and Feature set) are sorted from best to worst. The best method receives rank = 1, the second best receives rank = 2, etc. In the case of a tie, average ranks are assigned. For instance, if two methods tie for the top rank, they both receive rank = 1.5. The average ranks across all the datasets are then computed for each method.
Post hoc tests were applied in order to identify statistically significant differences among the performance results. Some of these tests are strict in the conclusions that might be obtained from them. We found that the classification accuracy results from some of the 34 datasets are substantially high, for a considerable number of methods. Therefore, aiming at inferring the classifiers and attributes that work best with the hardest datasets (i.e., those that are more interesting), the statistical comparison of the methods was carried out twice: first using all the datasets and then using the subset formed by the difficult ones (the division between easy and difficult datasets was determined based on the performance of a baseline classifier).
Post hoc tests based on mean-ranks are commonly used, but their application has been questioned recently [42]. Hence, the results are also compared with the Bayesian Signed-Rank Test [43].
3.1. Results Corresponding to All the Datasets
Table 3 shows a selection of the results, for a subset of the pairs (Classifier:Feature Set). Given the large number of pairs, it is not possible to include all the methods in a single table (Table A1, Table A2, Table A3 and Table A4 (in the Appendix A) show the complete set of results for the RF, SVM-L, and SVM-G classifiers and the Stacking strategy, respectively, considering all the datasets). The pairs in the subset were selected so that for all data sets there was some method with the highest accuracy. In some datasets, many methods share the best accuracy, so it is not possible to include all of them in the subset. Therefore, the subset of pairs was further reduced according to the average accuracy across all the datasets. Moreover, the pairs with the feature set OB were also included because it was used by Yeo et al. [18].
Table 4 presents the average accuracy of each one of the pairs (Classifier:Feature Set) assessed throughout the 34 datasets. It also shows the average ranks computed using all the methods in the experimental setup. In terms of average accuracies, the best results obtained by Shyong Yeo et al. [18] appear in the lower third of the table; SVM-L:OB (Linear SVM trained using the concatenation of Raw and Basic features) achieves an average accuracy of 91.36%. The same classifier, when trained using all features or TSFRESH with feature selection, obtains an accuracy higher than 94.5%. In terms of average ranks, the two pairs with top ranks are SVM-L:All and Stacking+:All, with average ranks below 12.7. The average rank for SVM-L:OB is 18.13.
The best five pairs according to the average accuracy and rank (in Table 4) use the feature sets (t), (All), and (Ot). Table 5 summarizes the results in Table 4 averaging for each feature set the corresponding values of RF, SVM-L and SVM-G. According to both the average accuracies and ranks, the three best feature sets are (t), (All), and (Ot).
Figure 4 shows, for each one of the three classification methods, the differences in terms of accuracy between each feature set and the feature set (OB) used by Yeo et al. [18]. Each boxplot is from the corresponding differences from the 34 datasets. The average differences are clearly favorable for several of the alternative feature sets. Medians of the differences are close to 0 or negative. As shown in Table A1, Table A2, Table A3 and Table A4, there are several datasets with 100% accuracy for all or many of the feature sets. For several feature sets, the boxplot are mostly in the positive region, positive differences are greater than negative differences. For SVM-G the boxplots are less favorable to the alternative feature sets, but, as shown in Table 4, SVM-L has better results than SVM-G.
Given the large number of (Classifier:Feature Set) tested methods, it is preferable to obtain the average ranks by considering smaller pair groups. Therefore, methods were divided depending on the type of classifier used: RF, SVM-L, and SVM-G. These average ranks are shown in Table 6.
The use of the Nemenyi test [44] was also proposed by Demšar [41] to compare methods in a pairwise way. For a certain level of confidence (), the test determines a critical difference (CD) value. If the difference between the average rankings of two methods is greater than CD, the null hypothesis, , that both methods have equal performance, is rejected. Figure 5 shows the Nemenyi’s CD diagrams. In these diagrams, thick horizontal lines are used to connect methods whose difference in average ranks is smaller than CD.
The methods were also compared using the Bayesian Signed-Rank Test [43], the Bayesian framework equivalent version of the Wilcoxon Signed-Rank Test. In this test, the value of the Region of Practical Equivalence (ROPE) was set to 1% for accuracy. Two methods were considered equivalent when the difference in their performance was smaller than this ROPE value. The test determines three probability values, corresponding to the following cases: (1) one method is better than the other; (2) vice versa; (3) they are in the ROPE.
Figure 6, Figure 7 and Figure 8 show the Bayesian Signed-Rank Tests posteriors, for the RF, SVM-L, and SVM-G classifiers, respectively. In these figures, the (OB) feature set is compared against each one of the other feature sets, for the corresponding classifier. For each feature set, there is a triangle. In these triangles [43], the bottom-left and bottom-right regions correspond to the case where one method is better than the other or vice versa. The top region represents the case where the ROPE is more probable. The corner triangles show the probability of each region. The left region in the triangle is for OB and the right region for the other feature set.
Figure 6 shows that, for RF, the feature set with more favorable results when compared to (OB) is (All), with a probability of 0.767, while it is 0.000 for (OB). Figure 7 shows that, for SVM-L, the best feature set is (Ot): its probability is 0.821, while it is 0.000 for (OB). In Figure 8, for SVM-G, the results are less favorable for the alternative cases to (OB). The best feature set is (Ot), with a probability of 0.244, being 0.001 the probability for (OB). The classification results for the three different classifiers therefore show an improvement that can be considered as significant, when using the different types of proposed feature sets, versus the features proposed in [18].
3.2. Results for the So-Called Difficult Datasets
An important part of the datasets reached a classification accuracy near or equal (Table A1, Table A2, Table A3 and Table A4, in Appendix A) , while others did not even reach . This was the reason we considered splitting up the dataset into two groups, one being formed only by what we may call the difficult datasets, for which the classification accuracy using SVM-L (best classifier in the previous work) was ≤90% using the original set of raw features. The list of difficult datasets (in Table A2) is the following: (1)
For the other datasets, simple methods may have good accuracy results, with little room for improvement. The entire experimental framework was repeated considering only this (difficult) subgroup of datasets, and the results are as follows.
Table 7 shows the average accuracies and average ranks, obtained using only the difficult datasets. The average accuracy for SVM-L:OB is 78.199 and for SVM-L:t is 87.173. The average rank of SVM-L:OB is 23.115 and 6.654 for SVM-L:All. Table 8 summarizes the results in Table 7 for each feature set, averaging the results of RF, SVM-L and SVM-G. The best feature set is (t), with an average accuracy of 85.58% and an average rank of 10.987. The average accuracy is 77.922 and the rank is 23.090 for (OB).
Given again the large number of methods (Classifier:Feature Set) tested, they were divided depending on the type of classifier used: RF, SVM-L, and SVM-G. These average ranks (for the difficult datasets) are shown in Table 9.
Figure 9 shows the critical difference diagrams for the Nemenyi test, for the three classifiers. The difference of the average ranks between RF:OB and the best feature sets with RF is greater than the critical difference. The distance of SVM-L:OB to the best feature sets with SVM-L is also greater. Nevertheless, the differences for SVM-G:OB and other feature sets with SVM-G are smaller than the critical difference.
Figure 10, Figure 11 and Figure 12 show the Bayesian Signed-Rank Tests posteriors, for the RF, SVM-L, and SVM-G classifiers, respectively, for the difficult datasets. In Figure 10, for RF, the feature sets with more favorable results when compared to (OB) are (t), (OT), (Ot), and (All), with a probability of 1.000 for the corresponding feature set, and 0.000 for (OB). For SVM-L (Figure 11), the best feature sets are, again, (t), (OT), (Ot), and (All), with a probability of 1.000 for the corresponding feature set, and 0.000 for (OB). In Figure 12, for SVM-G, the results are best for (Ot), with a probability of 0.997, and a probability of 0.001 for (OB), followed by (All), with a probability of 0.983, and a probability of 0.000 for (OB), and by (t), with a probability of 0.989, and a probability of 0.002 for (OB).
4. Conclusions
This paper presents a comparative analysis of different types of classification methodologies, applied on a series of datasets of raw signals acquired by a portable radar sensor, for different types of materials. In particular, twelve different types of feature vectors obtained from the original raw dataset were obtained, applying different types of feature extraction strategies. These feature vectors were subsequently combined with two classification methods (Random forests and SVM with linear and radial kernel types). A stacked generalization (Stacking) approach was also considered which involved base classifiers created using Random Forest and SVM trained using a subset of the different sets of features. The classification results shown outperformed the corresponding ones obtained by Shyong Yeo et al. [18], when considering the complete collection of datasets, as well as in a wider margin when using the partial so-called difficult datasets. In particular, the difference between the use of the TSFRESH with feature selection (t) features and the original and basic (OB) features (used in [18]), for the complete group of datasets, is almost in accuracy. Moreover, this difference increases to almost , for (t) vs. (OB) as well, for the so-called difficult subgroup.
From a classifier performance point-of-view, SVM with linear kernel (with default options) has the best global results (the methods with best average accuracy and rank in Table 4 and Table 7 use SVM-L), being much less costly than SVM with Gaussian kernel (with parameter adjustment) and Stacking. This suggest that it is not necessary to use expensive methods when using adequate feature extraction methods.
Potential future lines of research include the creation of our own datasets to explore the use of the radar sensor in problems that may have an industrial interest, for instance, in non-destructive testing or in the signal analysis of trash composites, to discern or classify them.
Author Contributions
J.F.D.-P., P.L.-C. and J.J.R. made the design of experiments, and carried them out. All the authors analysed the results and wrote the paper up. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by the Spanish Ministry of Science and Innovation under project PID2020-119894GB-I00, Junta de Castilla y León under project BU055P20 (JCyL/FEDER, UE) co-financed through European Union FEDER funds. José Luis Garrido-Labrador was supported by the predoctoral grant (BDNS 510149) awarded by the Universidad de Burgos, Spain.
Institutional Review Board Statement
Not applicable.
Informed Consent Statement
Not applicable.
Data Availability Statement
Data and codes will be available upon demand.
Acknowledgments
We would like to thank the authors of [18] for making the datasets available (
Conflicts of Interest
Authors declare that they do not have any conflict of interests.
Appendix A. Tables with Classification Results for All Datasets
This section includes four tables corresponding to the complete classification results, for the 34 datasets, when applying RF, SVM-L, SVM-G, and Stacking classification strategies.
Table A1
Results for classifier RF with different feature sets.
| O | B | OB | T | OT | R100 | R1000 | t | Ot | OR100 | OR1000 | All | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Count + Order Lego | 97.09 | 90.45 | 96.09 | 94.27 | 97.09 |
|
|
|
97.09 | 96.18 |
|
96.09 |
| Count 20 chips NO case ×30 sorted | 65.68 | 57.57 | 63.60 | 65.51 | 66.95 | 49.76 | 57.24 |
|
66.95 | 67.74 | 65.18 | 68.68 |
| Count 20 chips WITH case ×30 sorted | 70.79 | 61.11 | 70.32 | 73.33 | 75.56 | 66.98 | 69.84 | 75.24 |
|
72.06 | 72.86 | 75.71 |
| Count 20 papers ×10 |
|
75.71 | 75.71 | 72.86 | 73.33 | 74.76 | 76.67 | 75.24 | 75.24 | 76.19 | 77.14 | 78.57 |
| Distance 3 mugs 10 distances | 48.56 | 52.78 | 47.22 | 67.89 | 64.78 | 75.11 | 68.89 |
|
64.44 | 49.67 | 65.67 | 73.22 |
| Distance 3 mugs grouped by material | 71.33 | 92.67 | 84.33 | 97.89 | 99.00 | 80.78 | 80.00 |
|
|
85.33 | 84.11 |
|
| Distance 7 slotting |
|
98.33 |
|
|
|
|
|
|
|
|
|
|
| Flip 10 creditcards NO case | 82.27 | 80.00 | 83.64 | 86.82 | 85.00 | 79.55 | 79.55 |
|
86.36 | 80.91 | 83.64 | 83.64 |
| Flip 10 creditcards WITH case |
|
|
|
|
|
|
|
|
|
|
|
|
| Flip 52 cards | 94.36 | 96.27 | 96.18 | 97.18 | 97.27 | 95.36 | 97.27 |
|
|
96.27 | 97.18 | 97.27 |
| Identify 10 creditcards NO case | 85.45 | 83.64 | 87.27 |
|
90.91 | 80.91 | 85.45 | 90.91 |
|
85.45 | 83.64 | 90.00 |
| Identify 10 creditcards WITH case |
|
|
|
|
|
|
|
|
|
|
|
|
| Identify 12 printed designs |
|
|
|
|
|
|
|
|
|
|
|
|
| Identify 12 touch on numpad | 97.56 | 91.67 | 98.33 | 97.82 | 98.08 | 96.15 | 97.05 | 97.95 | 97.82 | 98.21 |
|
98.46 |
| Identify 5 colors × 20 chips | 75.00 | 73.33 | 75.42 | 82.08 | 81.25 | 78.75 | 80.83 |
|
83.33 | 76.25 | 82.08 | 82.50 |
| Identify 6 tagged plastic cards |
|
|
|
|
|
|
|
|
|
|
|
|
| Identify 6 users by palm | 82.86 | 79.52 | 82.86 | 89.52 | 89.05 | 82.86 | 85.24 |
|
90.95 | 85.71 | 85.24 | 90.95 |
| Identify 6 users by touch behavior sorted | 80.38 | 71.41 | 82.56 | 95.26 | 94.74 | 73.21 | 80.38 |
|
96.03 | 82.95 | 80.90 | 94.49 |
| Identify 7 dominoes | 96.25 |
|
98.75 |
|
98.75 | 97.50 |
|
98.75 | 98.75 | 98.75 |
|
|
| Identify 9 touch on half-sphere | 99.00 | 94.67 |
|
98.00 | 99.00 | 96.00 | 97.33 | 98.17 | 98.67 | 98.33 | 99.17 | 99.00 |
| Order 3 coasters NO case | 80.42 | 71.25 | 79.58 |
|
82.71 | 76.25 | 75.21 | 82.92 | 83.13 | 80.21 | 79.17 | 83.33 |
| Order 3 creditcards NO case sorted | 80.40 | 69.08 | 82.39 | 80.40 |
|
73.20 | 78.12 | 84.26 |
|
82.32 | 82.28 | 85.40 |
| Order 4 creditcards NO case sorted | 56.85 | 43.59 | 55.51 | 56.40 | 56.69 | 47.31 | 49.25 | 56.10 | 56.10 | 55.94 | 54.47 |
|
| Order 4 creditcards WITH case sorted |
|
98.46 |
|
|
|
|
|
|
|
|
|
|
| Rotation interval half numeric |
|
|
|
|
|
|
|
|
|
|
|
|
| Rotation interval one numeric | 99.17 |
|
99.17 |
|
99.17 |
|
|
|
|
99.17 | 99.17 |
|
| Slide inside-out on desk surface |
|
|
|
|
|
|
|
|
|
|
|
|
| Slide outside-in ruler |
|
|
|
|
|
|
|
|
|
|
|
|
| session1 × 10 | 99.81 | 99.25 | 99.81 | 99.81 | 99.81 | 99.81 |
|
99.81 | 99.81 | 99.81 | 99.81 | 99.62 |
| session2 × 10 | 99.25 | 98.30 |
|
99.43 | 99.43 | 99.25 | 99.43 | 99.43 | 99.43 |
|
99.43 |
|
| session3 × 10 |
|
99.62 | 99.62 |
|
99.62 |
|
|
|
|
|
|
99.62 |
| session4 × 10 | 99.81 | 99.81 |
|
|
99.81 |
|
|
99.81 |
|
|
99.81 |
|
| session5 × 10 | 97.92 | 97.55 | 97.92 |
|
98.68 | 97.55 | 97.74 | 98.68 | 98.49 | 98.11 | 97.92 | 98.49 |
| session6 × 10 |
|
|
|
|
|
99.06 | 99.06 |
|
|
|
|
|
| Mean | 89.33 | 87.51 | 89.82 | 92.03 | 92.10 | 88.75 | 89.76 |
|
92.48 | 90.11 | 90.71 | 92.72 |
| Average rank | 7.51 | 9.38 | 7.01 | 5.50 | 5.75 | 8.41 | 7.16 |
|
4.94 | 6.50 | 6.46 | 4.69 |
Table A2
Results for classifier SVM-L with different feature sets.
| O | B | OB | T | OT | R100 | R1000 | t | Ot | OR100 | OR1000 | All | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Count + Order Lego |
|
96.09 |
|
96.00 | 96.00 | 98.00 | 98.00 | 96.00 | 96.00 |
|
98.00 | 97.00 |
| Count 20 chips NO case ×30 sorted | 60.11 | 60.42 | 62.49 | 71.71 | 72.35 | 60.90 | 71.88 | 72.35 | 71.87 | 69.17 | 73.78 |
|
| Count 20 chips WITH case ×30 sorted | 79.52 | 67.78 | 79.52 | 78.73 | 79.37 | 73.33 | 78.89 | 79.68 | 80.00 | 81.27 |
|
80.63 |
| Count 20 papers ×10 | 88.57 | 79.52 | 87.14 | 82.38 | 82.38 | 88.10 |
|
84.29 | 84.76 | 88.10 | 89.52 | 86.19 |
| Distance 3 mugs 10 distances | 39.11 | 27.00 | 47.56 | 64.67 | 61.33 | 77.22 | 80.56 |
|
72.22 | 63.44 | 75.11 | 77.33 |
| Distance 3 mugs grouped by material | 63.56 | 77.00 | 80.89 |
|
|
90.44 | 93.44 |
|
|
85.22 | 90.33 |
|
| Distance 7 slotting |
|
|
|
98.33 |
|
|
|
98.33 |
|
|
|
|
| Flip 10 creditcards NO case | 79.55 | 75.45 | 82.73 | 86.82 | 87.27 | 79.55 | 84.09 | 88.18 |
|
82.73 | 85.91 | 88.64 |
| Flip 10 creditcards WITH case |
|
96.82 |
|
|
|
|
|
|
|
|
|
|
| Flip 52 cards | 96.36 | 92.45 | 96.36 | 98.18 | 98.18 | 97.09 | 98.18 |
|
|
94.36 | 98.18 | 98.18 |
| Identify 10 creditcards NO case | 80.00 | 81.82 | 83.64 | 88.18 | 87.27 | 79.09 | 81.82 |
|
|
83.64 | 84.55 | 86.36 |
| Identify 10 creditcards WITH case |
|
|
|
98.18 | 98.18 |
|
|
|
|
|
|
|
| Identify 12 printed designs |
|
|
|
|
|
|
|
|
|
|
|
|
| Identify 12 touch on numpad |
|
94.62 | 98.97 | 98.59 | 98.85 | 97.82 | 97.95 | 98.59 | 98.59 | 98.97 | 98.72 | 98.59 |
| Identify 5 colors × 20 chips | 85.00 | 76.67 | 87.50 | 94.58 | 94.58 | 81.25 | 85.83 |
|
|
87.08 | 88.33 |
|
| Identify 6 tagged plastic cards |
|
|
|
97.14 | 97.14 |
|
|
|
|
|
|
|
| Identify 6 users by palm | 87.62 | 85.71 | 90.95 | 92.86 | 92.86 | 86.19 | 90.00 | 95.24 | 94.29 | 90.00 | 91.43 |
|
| Identify 6 users by touch behavior sorted | 82.18 | 61.03 | 85.38 | 98.08 | 97.95 | 71.03 | 85.64 |
|
98.72 | 85.64 | 88.72 | 98.59 |
| Identify 7 dominoes |
|
|
|
96.25 | 97.50 |
|
|
|
|
|
|
|
| Identify 9 touch on half-sphere | 97.50 | 95.17 | 98.67 | 98.83 | 98.83 | 97.67 | 99.50 | 99.00 | 99.00 | 98.50 |
|
99.33 |
| Order 3 coasters NO case | 82.92 | 77.50 | 81.04 | 84.58 | 85.42 | 76.67 | 80.00 | 85.21 | 85.21 | 82.71 | 83.12 |
|
| Order 3 creditcards NO case sorted | 82.32 | 75.59 | 86.58 | 89.01 | 89.01 | 81.07 | 84.12 | 89.01 |
|
84.74 | 87.87 | 89.63 |
| Order 4 creditcards NO case sorted | 60.13 | 45.54 | 61.16 | 66.36 | 66.37 | 51.93 | 58.64 | 65.91 |
|
61.01 | 60.72 | 65.47 |
| Order 4 creditcards WITH case sorted |
|
99.23 |
|
|
|
|
99.23 |
|
|
|
99.23 |
|
| Rotation interval half numeric |
|
|
|
99.17 | 99.17 |
|
|
|
|
|
|
|
| Rotation interval one numeric |
|
|
|
99.17 | 99.17 |
|
|
|
|
|
|
|
| Slide inside-out on desk surface |
|
|
|
97.27 | 97.27 |
|
|
|
|
|
|
|
| Slide outside-in ruler |
|
|
|
98.18 | 99.09 |
|
|
99.09 | 99.09 |
|
|
99.09 |
| session1 × 10 | 99.81 | 99.62 | 99.81 | 99.62 | 99.62 |
|
|
99.81 | 99.81 | 99.81 | 99.81 | 99.81 |
| session2 × 10 |
|
99.06 |
|
98.87 | 99.25 |
|
99.25 | 99.06 | 99.43 |
|
99.25 | 99.43 |
| session3 × 10 |
|
99.43 |
|
99.62 | 99.62 |
|
|
99.43 | 99.62 |
|
|
|
| session4 × 10 |
|
|
|
99.62 | 99.62 |
|
|
99.43 | 99.43 |
|
|
99.43 |
| session5 × 10 | 98.87 | 97.55 | 98.49 |
|
|
98.87 |
|
|
|
|
|
|
| session6 × 10 |
|
|
|
|
|
99.06 | 99.06 |
|
|
|
99.06 |
|
| Mean | 89.99 | 87.07 | 91.35 | 93.20 | 93.28 | 90.71 | 92.84 |
|
94.38 | 92.14 | 93.28 | 94.52 |
| Average rank | 6.96 | 9.43 | 6.41 | 7.66 | 6.96 | 7.43 | 6.29 | 5.53 | 5.06 | 6.00 | 5.51 |
|
Table A3
Results for classifier SVM-G with different feature sets.
| O | B | OB | T | OT | R100 | R1000 | t | Ot | OR100 | OR1000 | All | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Count + Order Lego |
|
97.00 |
|
95.09 | 95.09 | 97.00 |
|
95.09 | 95.09 |
|
|
96.00 |
| Count 20 chips NO case ×30 sorted | 71.39 | 60.74 | 71.23 | 72.50 | 73.93 | 63.60 | 73.62 | 72.35 | 74.25 | 73.94 | 73.46 |
|
| Count 20 chips WITH case ×30 sorted | 79.68 | 69.68 | 79.05 | 79.52 | 81.43 | 75.40 | 80.63 | 80.95 | 81.90 |
|
82.06 | 81.75 |
| Count 20 papers ×10 | 88.57 | 79.05 | 87.62 | 82.86 | 83.33 | 87.62 |
|
84.29 | 83.81 | 88.10 | 89.05 | 85.71 |
| Distance 3 mugs 10 distances | 46.56 | 58.00 | 50.78 | 63.67 | 60.33 | 77.22 | 80.56 |
|
71.11 | 62.56 | 75.11 | 77.33 |
| Distance 3 mugs grouped by material | 71.33 | 88.22 | 80.89 |
|
|
86.33 | 92.44 |
|
|
94.67 | 88.33 |
|
| Distance 7 slotting |
|
|
|
98.33 | 96.67 |
|
|
98.33 | 98.33 |
|
|
98.33 |
| Flip 10 creditcards NO case | 87.27 | 85.00 |
|
85.45 | 86.36 | 81.82 | 85.45 | 88.18 | 87.73 | 85.00 | 85.45 | 87.73 |
| Flip 10 creditcards WITH case |
|
|
|
|
|
|
|
|
|
|
|
|
| Flip 52 cards |
|
96.27 |
|
|
|
96.18 |
|
|
|
97.18 | 96.18 |
|
| Identify 10 creditcards NO case |
|
85.45 | 84.55 | 85.45 | 86.36 | 79.09 | 83.64 | 89.09 | 88.18 | 88.18 | 83.64 | 84.55 |
| Identify 10 creditcards WITH case |
|
|
|
99.09 | 99.09 |
|
|
99.09 | 99.09 |
|
|
99.09 |
| Identify 12 printed designs |
|
99.23 | 99.23 |
|
|
|
|
|
|
|
|
99.23 |
| Identify 12 touch on numpad | 99.10 | 96.15 |
|
98.33 | 98.33 | 97.82 | 98.21 | 98.46 | 98.46 | 98.85 | 98.46 | 98.46 |
| Identify 5 colors × 20 chips | 86.67 | 85.00 | 88.33 | 94.58 | 94.58 | 75.83 | 85.00 |
|
|
87.08 | 87.50 | 95.83 |
| Identify 6 tagged plastic cards |
|
|
|
98.57 | 98.57 |
|
|
98.57 | 98.57 |
|
|
98.57 |
| Identify 6 users by palm | 89.52 | 86.67 | 91.90 | 92.86 | 91.90 | 85.71 | 89.52 |
|
93.33 | 92.38 | 90.48 | 93.81 |
| Identify 6 users by touch behavior sorted | 91.41 | 82.05 | 91.41 | 97.82 | 97.82 | 82.95 | 89.10 |
|
|
91.92 | 90.26 | 98.08 |
| Identify 7 dominoes |
|
98.75 |
|
98.75 | 98.75 |
|
|
98.75 | 98.75 |
|
|
98.75 |
| Identify 9 touch on half-sphere | 98.33 | 97.00 | 98.50 | 98.83 | 98.83 | 98.33 |
|
99.00 | 99.00 | 99.00 |
|
99.33 |
| Order 3 coasters NO case | 83.75 | 76.04 | 85.00 | 85.00 |
|
76.25 | 80.63 | 85.42 | 85.21 | 83.54 | 83.75 | 86.04 |
| Order 3 creditcards NO case sorted | 88.38 | 78.13 | 89.63 | 89.63 | 88.38 | 79.89 | 84.15 | 87.79 |
|
87.79 | 87.17 | 89.04 |
| Order 4 creditcards NO case sorted | 63.84 | 51.48 | 63.40 | 66.50 |
|
53.28 | 58.04 | 66.65 | 66.51 | 63.09 | 61.32 | 66.06 |
| Order 4 creditcards WITH case sorted | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 | 99.23 |
|
| Rotation interval half numeric |
|
|
|
|
99.17 |
|
|
|
|
99.17 | 99.17 |
|
| Rotation interval one numeric | 99.17 | 99.17 |
|
99.17 | 99.17 |
|
99.17 |
|
|
|
|
|
| Slide inside-out on desk surface |
|
|
|
|
|
99.09 | 99.09 |
|
|
|
99.09 |
|
| Slide outside-in ruler |
|
|
|
99.09 | 99.09 |
|
|
99.09 | 99.09 |
|
|
99.09 |
| session1 × 10 | 99.81 | 99.62 | 99.81 | 99.62 | 99.62 |
|
|
99.81 | 99.81 | 99.81 | 99.81 | 99.81 |
| session2 × 10 |
|
98.87 | 99.25 | 98.87 | 99.06 |
|
99.06 | 99.06 | 99.25 |
|
99.43 | 99.25 |
| session3 × 10 |
|
99.43 |
|
99.43 | 99.43 |
|
|
99.43 | 99.43 |
|
|
99.62 |
| session4 × 10 |
|
|
|
99.43 | 99.62 |
|
|
99.25 | 99.25 |
|
99.81 | 99.43 |
| session5 × 10 | 98.68 | 97.92 | 98.30 | 99.06 | 99.06 | 98.87 | 98.87 | 99.06 | 99.06 | 99.06 |
|
98.87 |
| session6 × 10 |
|
|
|
|
|
98.87 | 99.06 |
|
|
|
99.06 |
|
| Mean | 91.99 | 90.10 | 92.38 | 93.36 | 93.34 | 90.88 | 92.97 |
|
94.02 | 93.25 | 93.07 | 94.19 |
| Average rank | 5.94 | 8.97 | 5.94 | 7.26 | 6.99 | 7.74 | 6.47 | 5.87 | 5.65 |
|
6.13 | 5.65 |
Table A4
Results for Stacking.
| Stacking | Stacking+ | |
|---|---|---|
| Count + Order Lego |
|
|
| Count 20 chips NO case ×30 sorted |
|
76.17 |
| Count 20 chips WITH case ×30 sorted |
|
80.79 |
| Count 20 papers ×10 | 83.33 |
|
| Distance 3 mugs 10 distances |
|
63.67 |
| Distance 3 mugs grouped by material | 99.00 |
|
| Distance 7 slotting |
|
|
| Flip 10 creditcards NO case |
|
86.82 |
| Flip 10 creditcards WITH case |
|
|
| Flip 52 cards |
|
|
| Identify 10 creditcards NO case |
|
|
| Identify 10 creditcards WITH case |
|
|
| Identify 12 printed designs |
|
|
| Identify 12 touch on numpad | 98.97 |
|
| Identify 5 colors × 20 chips | 94.58 |
|
| Identify 6 tagged plastic cards |
|
|
| Identify 6 users by palm | 94.29 |
|
| Identify 6 users by touch behavior sorted | 98.46 |
|
| Identify 7 dominoes |
|
|
| Identify 9 touch on half-sphere | 98.83 |
|
| Order 3 coasters NO case | 82.08 |
|
| Order 3 creditcards NO case sorted | 88.38 |
|
| Order 4 creditcards NO case sorted | 64.43 |
|
| Order 4 creditcards WITH case sorted |
|
|
| Rotation interval half numeric |
|
|
| Rotation interval one numeric |
|
|
| Slide inside-out on desk surface |
|
|
| Slide outside-in ruler |
|
|
| session1 × 10 |
|
|
| session2 × 10 |
|
|
| session3 × 10 |
|
|
| session4 × 10 |
|
|
| session5 × 10 | 98.49 |
|
| session6 × 10 |
|
|
| Mean | 93.94 |
|
| Average rank | 1.60 |
|
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Figures and Tables
Enlarge this image.Figure 1. Plots of the eight 64D signals (one 64D signal per channel) for two instances, each of a different dataset.
Enlarge this image.Figure 3. Flow chart describing the general data processing pipeline, divided into two steps.
Enlarge this image.Figure 4. Boxplots of the differences in accuracy between the different feature sets and the feature set (OB), for the three considered classifiers. The boxplots on the right do not include the outliers. The average differences are marked with a red dot (•).
Enlarge this image.Figure 6. Posteriors for the Bayesian sign-rank tests for RF, from all the datasets.
Enlarge this image.Figure 7. Posteriors for the Bayesian sign-rank tests for SVM-L, from all the datasets.
Enlarge this image.Figure 8. Posteriors for the Bayesian sign-rank tests for SVM-G, from all the datasets.
Enlarge this image.Figure 9. Critical difference diagrams for the Nemenyi test, for the difficult datasets (α=0.05).
Enlarge this image.Figure 10. Posteriors for the Bayesian sign-rank tests for RF, from the difficult datasets.
Enlarge this image.Figure 11. Posteriors for the Bayesian sign-rank tests for SVM-L, from the difficult datasets.
Enlarge this image.Figure 12. Posteriors for the Bayesian sign-rank tests for SVM-G, from the difficult datasets.
Columns show radar datasets used, including the problem type (C, Counting; M, Material Identification; D, Distance estimation; O, Order identification; F, Flipping identification (Up/Down)); I, Object identification; R, Movement, rotation; P, Movement, position), the total number of samples, and the number of classes for each dataset.
| Dataset | Problem | # of Samples | # of Classes | |
|---|---|---|---|---|
| 1 | Count + Order Lego | C | 106 | 11 |
| 2 | Count 20 chips NO case x30 sorted | C | 629 | 21 |
| 3 | Count 20 chips WITH case x30 sorted | C | 630 | 21 |
| 4 | Count 20 papers x10 | C | 210 | 21 |
| 5 | Distance 3 mugs 10 distances | M + D | 93 | 31 |
| 6 | Distance 3 mugs grouped by material | M | 93 | 4 |
| 7 | Distance 7 slotting | D | 60 | 8 |
| 8 | Flip 10 creditcards NO case | F + I | 220 | 21 |
| 9 | Flip 10 creditcards WITH case | F + I | 220 | 21 |
| 10 | Flip 52 cards | F | 105 | 3 |
| 11 | Identify 10 creditcards NO case | I | 110 | 11 |
| 12 | Identify 10 creditcards WITH case | I | 110 | 11 |
| 13 | Identify 12 printed designs | I | 130 | 13 |
| 14 | Identify 12 touch on numpad | I | 780 | 13 |
| 15 | Identify 5 colors × 20 chips | I | 240 | 6 |
| 16 | Identify 6 tagged plastic cards | I | 70 | 7 |
| 17 | Identify 6 users by palm | I | 210 | 7 |
| 18 | Identify 6 users by touch behavior sorted | I | 780 | 7 |
| 19 | Identify 7 dominoes | I | 80 | 8 |
| 20 | Identify 9 touch on half-sphere | I | 600 | 10 |
| 21 | Order 3 coasters NO case | O | 480 | 16 |
| 22 | Order 3 creditcards NO case sorted | O | 164 | 16 |
| 23 | Order 4 creditcards NO case sorted | O | 672 | 65 |
| 24 | Order 4 creditcards WITH case sorted | O | 390 | 65 |
| 25 | Rotation interval half numeric | R | 120 | 12 |
| 26 | Rotation interval one numeric | R | 120 | 12 |
| 27 | Slide inside-out on desk surface | P | 110 | 11 |
| 28 | Slide outside-in ruler | P | 110 | 11 |
| 29 | session1 × 10 | C | 530 | 53 |
| 30 | session2 × 10 | C | 530 | 53 |
| 31 | session3 × 10 | C | 530 | 53 |
| 32 | session4 × 10 | C | 530 | 53 |
| 33 | session5 × 10 | C | 530 | 53 |
| 34 | session6 × 10 | C | 530 | 53 |
Combination of features used, including their total number. In the case of variable-size feature extraction methods, the minimum, maximum, and average values, respectively, are also given (in parenthesis).
| Features | Abbreviation | No of Attributes |
|---|---|---|
| Raw Original Data | (O) | 512 |
| Basic aggregation features | (B) | 149 |
| ROCKET with 100 kernels | (R100) | 200 |
| ROCKET with 1000 kernels | (R1000) | 2000 |
| TSFRESH without feature selection | (T) | 6352 |
| TSFRESH with feature selection | (t) | Variable (885/3567/3235) |
| Original & Basic | (OB) | 661 |
| Original & ROCKET (100) | (OR100) | 712 |
| Original & ROCKET (1000) | (OR1000) | 2512 |
| Original & TSFRESH (without feat. sel.) | (OT) | 6864 |
| Original & TSFRESH (with feat. sel.) | (Ot) | Variable (1397/4079/3747) |
| Original, basic, ROCKET (1000) and TSFRESH (with feat. sel.) | (All) | Variable (3546/6228/5896) |
Results for a subset of the classifiers with different feature sets.
| RF:OB | RF:Ot | SVM-L:R1000 | SVM-L:t | SVM-L:OB | SVM-L:OR1000 | SVM-L:Ot | SVM-L:All | SVM-G:R1000 | SVM-G:OB | SVM-G:OR100 | Stack:All | Stack+:All | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Count + Order Lego | 96.09 | 97.09 | 98.00 | 96.00 |
|
98.00 | 96.00 | 97.00 | 98.00 | 98.00 | 98.00 | 98.00 | 98.00 |
| Count 20 chips NO case ×30 sorted | 63.60 | 66.95 | 71.88 | 72.35 | 62.49 | 73.78 | 71.87 | 73.94 | 73.62 | 71.23 | 73.94 |
|
76.17 |
| Count 20 chips WITH case ×30 sorted | 70.32 | 77.14 | 78.89 | 79.68 | 79.52 | 81.27 | 80.00 | 80.63 | 80.63 | 79.05 |
|
80.95 | 80.79 |
| Count 20 papers ×10 | 75.71 | 75.24 |
|
84.29 | 87.14 | 89.52 | 84.76 | 86.19 | 90.00 | 87.62 | 88.10 | 83.33 | 83.81 |
| Distance 3 mugs 10 distances | 47.22 | 64.44 | 80.56 |
|
47.56 | 75.11 | 72.22 | 77.33 | 80.56 | 50.78 | 62.56 | 64.78 | 63.67 |
| Distance 3 mugs grouped by material | 84.33 |
|
93.44 |
|
80.89 | 90.33 |
|
|
92.44 | 80.89 | 94.67 | 99.00 |
|
| Distance 7 slotting |
|
|
|
98.33 |
|
|
|
|
|
|
|
|
|
| Flip 10 creditcards NO case | 83.64 | 86.36 | 84.09 | 88.18 | 82.73 | 85.91 |
|
88.64 | 85.45 | 88.18 | 85.00 | 87.73 | 86.82 |
| Flip 10 creditcards WITH case |
|
|
|
|
|
|
|
|
|
|
|
|
|
| Flip 52 cards | 96.18 | 98.18 | 98.18 |
|
96.36 | 98.18 |
|
98.18 | 98.18 | 98.18 | 97.18 | 98.18 | 98.18 |
| Identify 10 creditcards NO case | 87.27 |
|
81.82 | 90.00 | 83.64 | 84.55 | 90.00 | 86.36 | 83.64 | 84.55 | 88.18 | 88.18 | 88.18 |
| Identify 10 creditcards WITH case |
|
|
|
|
|
|
|
|
|
|
|
|
|
| Identify 12 printed designs |
|
|
|
|
|
|
|
|
|
99.23 |
|
|
|
| Identify 12 touch on numpad | 98.33 | 97.82 | 97.95 | 98.59 | 98.97 | 98.72 | 98.59 | 98.59 | 98.21 |
|
98.85 | 98.97 | 99.10 |
| Identify 5 colors × 20 chips | 75.42 | 83.33 | 85.83 |
|
87.50 | 88.33 |
|
|
85.00 | 88.33 | 87.08 | 94.58 | 96.25 |
| Identify 6 tagged plastic cards |
|
|
|
|
|
|
|
|
|
|
|
|
|
| Identify 6 users by palm | 82.86 | 90.95 | 90.00 | 95.24 | 90.95 | 91.43 | 94.29 |
|
89.52 | 91.90 | 92.38 | 94.29 |
|
| Identify 6 users by touch behavior sorted | 82.56 | 96.03 | 85.64 | 98.72 | 85.38 | 88.72 | 98.72 | 98.59 | 89.10 | 91.41 | 91.92 | 98.46 |
|
| Identify 7 dominoes | 98.75 | 98.75 |
|
|
|
|
|
|
|
|
|
|
|
| Identify 9 touch on half-sphere | 99.33 | 98.67 | 99.50 | 99.00 | 98.67 |
|
99.00 | 99.33 | 99.50 | 98.50 | 99.00 | 98.83 | 99.00 |
| Order 3 coasters NO case | 79.58 | 83.13 | 80.00 | 85.21 | 81.04 | 83.12 | 85.21 |
|
80.63 | 85.00 | 83.54 | 82.08 | 83.54 |
| Order 3 creditcards NO case sorted | 82.39 | 85.99 | 84.12 | 89.01 | 86.58 | 87.87 |
|
89.63 | 84.15 | 89.63 | 87.79 | 88.38 | 89.01 |
| Order 4 creditcards NO case sorted | 55.51 | 56.10 | 58.64 | 65.91 | 61.16 | 60.72 |
|
65.47 | 58.04 | 63.40 | 63.09 | 64.43 | 65.47 |
| Order 4 creditcards WITH case sorted |
|
|
99.23 |
|
|
99.23 |
|
|
99.23 | 99.23 | 99.23 |
|
|
| Rotation interval half numeric |
|
|
|
|
|
|
|
|
|
|
99.17 |
|
|
| Rotation interval one numeric | 99.17 |
|
|
|
|
|
|
|
99.17 |
|
|
|
|
| Slide inside-out on desk surface |
|
|
|
|
|
|
|
|
99.09 |
|
|
|
|
| Slide outside-in ruler |
|
|
|
99.09 |
|
|
99.09 | 99.09 |
|
|
|
|
|
| session1 × 10 | 99.81 | 99.81 |
|
99.81 | 99.81 | 99.81 | 99.81 | 99.81 |
|
99.81 | 99.81 | 99.81 | 99.81 |
| session2 × 10 |
|
99.43 | 99.25 | 99.06 |
|
99.25 | 99.43 | 99.43 | 99.06 | 99.25 |
|
99.43 | 99.43 |
| session3 × 10 | 99.62 |
|
|
99.43 |
|
|
99.62 |
|
|
|
|
|
|
| session4 × 10 |
|
|
|
99.43 |
|
|
99.43 | 99.43 |
|
|
|
99.81 | 99.81 |
| session5 × 10 | 97.92 | 98.49 |
|
|
98.49 |
|
|
|
98.87 | 98.30 | 99.06 | 98.49 | 98.87 |
| session6 × 10 |
|
|
99.06 |
|
|
99.06 |
|
|
99.06 |
|
|
|
|
| Mean | 89.82 | 92.48 | 92.84 |
|
91.35 | 93.28 | 94.38 | 94.52 | 92.97 | 92.38 | 93.25 | 93.94 | 94.07 |
Average accuracies and ranks from all the datasets.
| Method | Accuracy | Method | Rank |
|---|---|---|---|
| SVM-L:t | 94.678 | SVM-L:All | 12.632 |
| SVM-L:All | 94.524 | Stacking+:All | 12.676 |
| SVM-G:t | 94.466 | SVM-L:Ot | 13.456 |
| SVM-L:Ot | 94.382 | Stacking:All | 14.265 |
| SVM-G:All | 94.188 | SVM-L:t | 14.853 |
| Stacking+:All | 94.075 | SVM-L:OR1000 | 15.368 |
| SVM-G:Ot | 94.025 | SVM-G:OR100 | 15.441 |
| Stacking:All | 93.938 | SVM-G:O | 16.824 |
| SVM-G:T | 93.359 | SVM-L:OR100 | 16.971 |
| SVM-G:OT | 93.338 | SVM-G:OB | 17.176 |
| SVM-L:OR1000 | 93.283 | SVM-G:All | 17.309 |
| SVM-L:OT | 93.283 | SVM-L:R1000 | 17.338 |
| SVM-G:OR100 | 93.252 | SVM-G:OR1000 | 17.588 |
| SVM-L:T | 93.205 | SVM-G:Ot | 17.662 |
| SVM-G:OR1000 | 93.070 | RF:t | 17.794 |
| SVM-G:R1000 | 92.969 | SVM-G:t | 17.853 |
| RF:t | 92.926 | SVM-L:OB | 18.132 |
| SVM-L:R1000 | 92.840 | RF:All | 18.176 |
| RF:All | 92.720 | SVM-G:R1000 | 18.265 |
| RF:Ot | 92.479 | RF:Ot | 18.353 |
| SVM-G:OB | 92.375 | RF:T | 19.279 |
| SVM-L:OR100 | 92.141 | SVM-L:O | 19.603 |
| RF:OT | 92.100 | SVM-L:OT | 20.176 |
| RF:T | 92.028 | RF:OT | 20.426 |
| SVM-G:O | 91.987 | SVM-L:R100 | 21.044 |
| SVM-L:OB | 91.355 | SVM-G:OT | 21.103 |
| SVM-G:R100 | 90.877 | SVM-G:T | 21.382 |
| SVM-L:R100 | 90.712 | SVM-L:T | 21.544 |
| RF:OR1000 | 90.706 | RF:OR1000 | 21.779 |
| RF:OR100 | 90.110 | SVM-G:R100 | 22.765 |
| SVM-G:B | 90.100 | RF:OR100 | 22.897 |
| SVM-L:O | 89.989 | RF:OB | 23.176 |
| RF:OB | 89.823 | RF:R1000 | 23.338 |
| RF:R1000 | 89.761 | RF:O | 24.706 |
| RF:O | 89.333 | RF:R100 | 25.971 |
| RF:R100 | 88.747 | SVM-G:B | 26.603 |
| RF:B | 87.508 | SVM-L:B | 28.088 |
| SVM-L:B | 87.068 | RF:B | 28.985 |
Average accuracies and ranks for each feature set, from all the datasets. For each feature set, the values in these tables are the averages for RF, SVM-L, and SVM-G in Table 4.
| Feature Set | Accuracy | Feature Set | Rank |
|---|---|---|---|
| t | 94.024 | All | 16.039 |
| All | 93.811 | Ot | 16.490 |
| Ot | 93.628 | t | 16.833 |
| OT | 92.907 | OR1000 | 18.245 |
| T | 92.864 | OR100 | 18.436 |
| OR1000 | 92.353 | OB | 19.495 |
| R1000 | 91.857 | R1000 | 19.647 |
| OR100 | 91.834 | O | 20.377 |
| OB | 91.184 | OT | 20.569 |
| O | 90.437 | T | 20.735 |
| R100 | 90.112 | R100 | 23.260 |
| B | 88.226 | B | 27.892 |
Average ranks for each classification method.
| RF | SVM-L | SVM-G | |||
|---|---|---|---|---|---|
| Method | Rank | Method | Rank | Method | Rank |
| RF:t | 4.676 | SVM-L:All | 4.765 | SVM-G:OR100 | 5.397 |
| RF:All | 4.691 | SVM-L:Ot | 5.059 | SVM-G:Ot | 5.647 |
| RF:Ot | 4.941 | SVM-L:OR1000 | 5.515 | SVM-G:All | 5.647 |
| RF:T | 5.500 | SVM-L:t | 5.529 | SVM-G:t | 5.868 |
| RF:OT | 5.750 | SVM-L:OR100 | 6.000 | SVM-G:O | 5.941 |
| RF:OR1000 | 6.456 | SVM-L:R1000 | 6.294 | SVM-G:OB | 5.941 |
| RF:OR100 | 6.500 | SVM-L:OB | 6.412 | SVM-G:OR1000 | 6.132 |
| RF:OB | 7.015 | SVM-L:O | 6.956 | SVM-G:R1000 | 6.471 |
| RF:R1000 | 7.162 | SVM-L:OT | 6.956 | SVM-G:OT | 6.985 |
| RF:O | 7.515 | SVM-L:R100 | 7.426 | SVM-G:T | 7.265 |
| RF:R100 | 8.412 | SVM-L:T | 7.662 | SVM-G:R100 | 7.735 |
| RF:B | 9.382 | SVM-L:B | 9.426 | SVM-G:B | 8.971 |
Average accuracies and ranks from the difficult datasets .
| Method | Accuracy | Method | Rank |
|---|---|---|---|
| SVM-L:t | 87.173 | SVM-L:All | 6.654 |
| SVM-G:t | 87.092 | SVM-L:Ot | 7.154 |
| SVM-L:All | 86.550 | SVM-L:t | 7.538 |
| SVM-G:All | 86.278 | SVM-G:t | 7.731 |
| SVM-L:Ot | 86.226 | SVM-G:Ot | 7.846 |
| SVM-G:Ot | 85.924 | SVM-G:All | 8.308 |
| Stacking+:All | 85.215 | Stacking+:All | 9.577 |
| Stacking:All | 84.908 | Stacking:All | 11.654 |
| SVM-L:T | 84.458 | SVM-G:OT | 11.808 |
| SVM-G:OT | 84.407 | SVM-L:OT | 12.654 |
| SVM-L:OT | 84.319 | SVM-L:T | 13.077 |
| SVM-G:T | 84.297 | SVM-G:T | 13.500 |
| SVM-G:OR100 | 83.199 | SVM-G:OR100 | 14.462 |
| SVM-L:OR1000 | 83.128 | SVM-L:OR1000 | 15.115 |
| SVM-G:OR1000 | 82.891 | SVM-G:OR1000 | 15.654 |
| SVM-G:R1000 | 82.522 | SVM-G:OB | 16.423 |
| RF:t | 82.475 | SVM-G:O | 17.462 |
| SVM-L:R1000 | 82.026 | RF:t | 17.692 |
| RF:All | 81.967 | SVM-G:R1000 | 18.077 |
| RF:Ot | 81.345 | RF:All | 19.000 |
| SVM-G:OB | 80.921 | RF:Ot | 19.346 |
| RF:OT | 80.458 | SVM-L:R1000 | 19.885 |
| RF:T | 80.400 | SVM-L:OR100 | 20.808 |
| SVM-L:OR100 | 80.365 | RF:T | 20.808 |
| SVM-G:O | 79.875 | RF:OT | 22.115 |
| SVM-L:OB | 78.199 | SVM-L:OB | 23.115 |
| SVM-G:R100 | 77.307 | SVM-L:O | 27.115 |
| SVM-L:R100 | 76.674 | SVM-L:R100 | 27.885 |
| RF:OR1000 | 76.643 | SVM-G:R100 | 28.077 |
| SVM-G:B | 75.809 | RF:OR100 | 28.846 |
| RF:OR100 | 75.442 | SVM-G:B | 28.923 |
| SVM-L:O | 74.660 | RF:OR1000 | 29.000 |
| RF:OB | 74.647 | RF:OB | 29.731 |
| RF:R1000 | 74.360 | RF:O | 30.154 |
| RF:O | 73.736 | RF:R1000 | 31.731 |
| RF:R100 | 72.264 | RF:R100 | 33.538 |
| RF:B | 70.128 | RF:B | 34.231 |
| SVM-L:B | 68.541 | SVM-L:B | 34.308 |
Average accuracies and ranks for each feature set, from the difficult datasets. For each feature set, the values in these tables are the averages for RF, SVM-L, and SVM-G in Table 7.
| Feature Set | Accuracy | Feature Set | Rank |
|---|---|---|---|
| t | 85.580 | t | 10.987 |
| All | 84.932 | All | 11.321 |
| Ot | 84.498 | Ot | 11.449 |
| OT | 83.062 | OT | 15.526 |
| T | 83.051 | T | 15.795 |
| OR1000 | 80.887 | OR1000 | 19.923 |
| OR100 | 79.669 | OR100 | 21.372 |
| R1000 | 79.636 | OB | 23.090 |
| OB | 77.922 | R1000 | 23.231 |
| O | 76.090 | O | 24.910 |
| R100 | 75.415 | R100 | 29.833 |
| B | 71.492 | B | 32.487 |
Average ranks for each classification method, for difficult datasets.
| RF | SVM-L | SVM-G | |||
|---|---|---|---|---|---|
| Method | Rank | Method | Rank | Method | Rank |
| RF:t | 2.808 | SVM-L:All | 2.923 | SVM-G:Ot | 3.308 |
| RF:All | 2.923 | SVM-L:Ot | 3.192 | SVM-G:t | 3.577 |
| RF:Ot | 3.538 | SVM-L:t | 3.346 | SVM-G:All | 3.808 |
| RF:OT | 4.692 | SVM-L:OT | 4.885 | SVM-G:OT | 5.000 |
| RF:T | 4.731 | SVM-L:OR1000 | 5.231 | SVM-G:T | 6.000 |
| RF:OR100 | 6.962 | SVM-L:T | 5.423 | SVM-G:OR100 | 6.192 |
| RF:OR1000 | 7.346 | SVM-L:R1000 | 7.154 | SVM-G:OB | 6.962 |
| RF:O | 7.692 | SVM-L:OR100 | 7.231 | SVM-G:OR1000 | 7.000 |
| RF:OB | 8.000 | SVM-L:OB | 7.923 | SVM-G:O | 7.269 |
| RF:R1000 | 8.846 | SVM-L:O | 9.385 | SVM-G:R1000 | 7.692 |
| RF:R100 | 10.038 | SVM-L:R100 | 9.692 | SVM-G:R100 | 10.308 |
| RF:B | 10.423 | SVM-L:B | 11.615 | SVM-G:B | 10.885 |
© 2021 by the authors.
Details
; José Miguel Ramírez-Sanz
; Rodríguez, Juan J