Large‐Area Piezoresistive Tactile Sensor Developed by Training a Super‐Simple Single‐Layer Carbon Nanotube‐Dispersed Polydimethylsiloxane Pad

Abstract

The revolutionary concept of creating a large‐area tactile sensor by training a simple, bulky material through deep learning (DL) is proposed. This enables the replacement of the conventional tactile sensor comprising a patterned array structure with a super‐simple, single‐layer, large‐area tactile sensor pad. A crude carbon nanotube‐dispersed polydimethylsiloxane pad—with a bias applied to the center and the resultant piezoresistive current detected at several electrodes located on the pad edge—plays a smart sensory role without the need for complicated fabrication of microengineered structures. The piezoresistive current while recording the indented location and the pressure thereon is measured, and then various DL models (a multimodel arrangement is necessary due to the viscoelasticity of the pad) using the collected data are trained. The proposed concept is realized using a tandem model comprising a combination of algorithms selected from deep neural networks, convolutional neural networks, long short‐term memory networks, and 16 state‐of‐the‐art machine learning algorithms. The hold‐out dataset test accuracy for the indented location identification reaches 98.89%, and the goodness of fit for pressure prediction is evaluated with mean squared error of 2.5 × 10⁻³ and coefficient of determination of 98.05%.

Introduction

There have been significant advancements in the development of tactile sensors in recent decades,^[1–24] and the state of the art has achieved not only the recognition of mechanical force or strain but also other functions such as the detection of temperature,^[1–9] moisture,^[5–7] and chemical substances.^[6–11] The present research seeks to develop a large-area tactile sensing pad that enables the precise detection of indented location and simultaneous evaluation of pressure at a touched location. Many types of tactile sensors have been investigated based on several transduction principles, such as capacitive, piezoresistive, piezoelectric, and electromagnetic principles.^[25–44] Piezoresistive composite materials play a pivotal role in constituting a tactile sensory device.^[45–49] A representative piezoresistive material is a composite consisting of a soft insulating polymer matrix (e.g., polydimethylsiloxane [PDMS], polymethylmethacrylate [PMMA], and epoxy resin) dispersed with conducting materials (e.g., multiwalled carbon nanotubes [CNTs]^[46,47] and metal powders^[48,49]). To constitute a large-area tactile sensor pad enabling the instantaneous detection of both the indented location and the pressure thereon, the conventional approach incorporates an orderly patterned array of macro- or microscale piezoresistive sensor device units on a large area for appropriate addressing and read-out based on a logic circuit.^[1–20] In addition, pattern-free (nonlayer) tactile sensors have been extensively investigated.^[50–57]

In contrast to these typical concepts for array- or nonarray-type, multilayer, large-area tactile sensors based on microengineered active layer structures, we recently reported a completely different type of super-simple piezoresistive composite-based tactile sensor for use in e-skin^[58] and portable keypad^[59] applications. The reported sensor uses a crude, nonarray, single-layer piezoresistive pad (a CNT-dispersed PDMS pad) without the need for high-cost fabrication of microengineered structures. Moreover, we achieved state-of-the-art tactile sensing accuracy and spatial resolution^[58] and created a smart keypad that is portable, disposable, and extremely flexible, to the extent that it can be carried and crushed inside the pocket of a pair of trousers.^[59]

The key to converting such a super-simple, single-layer, nonarray, piezoresistive pad into a smart tactile sensor is the incorporation of machine learning (ML) and deep learning (DL) techniques. Under a certain bias applied to the electrode located at the center of the pad, the piezoresistive current is detected at several electrodes connected to the edges of the pad, while recording the indented location and pressure, after which we collect these data and use them as a training dataset to train the DL and ML models. The fully trained DL and ML models can predict the exact location and pressure from the piezoresistive current signal detected at several edge electrodes. No complicated device fabrication is required; a single-layer, as-cast, large-area piezoresistive pad (CNT-dispersed PDMS pad) is sufficient to constitute a smart tactile sensory system, demonstrating that a bulky material in a crude state can replace complicated microengineered structures. This has led to the unfamiliar but revolutionary concept of creating a smart large-area tactile sensory system simply by training a crude bulky material instead of using a high-cost fabrication process based on microengineering.

Despite previous success,^[58,59] this type of tactile sensing approach needs to be further improved. In particular, the viscoelastic (or anelastic) behavior of the piezoresistive pad has hindered the device's capacity for accurate real-time tactile sensing, a problem which affects not only our proposed technology but piezoresistive tactile sensors in general.^[26,60–62] We solved this problem to a certain extent not by improving the material or sensor structure (which would be time-consuming and expensive) but by reinforcing the data and algorithm aspects—a more economical approach. We used only a basic deep neural network (DNN) model together with noncurated raw data signal in the previous investigations,^[58,59] based on a presumption that the piezoresistive pad would act as a perfect linear elastic body. In the present investigation, however, more advanced DL and ML algorithms—specifically, a convolutional neural network (CNN),^[63] a long short-term memory (LSTM) network,^[64] and 16 ML algorithms—were introduced, along with the transformation of data into images to resolve the complications originating from the viscoelastic (or anelastic) piezoresistive pad. An appropriate tandem model system consisting of a certain combination of the DNN, CNN, LSTM, and ML algorithms was developed to achieve real-time tactile sensing despite using a viscoelastic (or anelastic) piezoresistive pad.

The 16 ML algorithms used for the tandem model on top of DNN, CNN, and LSTM fall into three categories. First, we used the regularized linear regression algorithms Ridge,^[65] Lasso,^[66] Elastic Net,^[67] kernel ridge regression (KRR),^[68] least-angle regression (LARS) Lasso,^[69] Bayesian ridge regression (BRR),^[70] automatic relevance determination (ARD)^[71] regression, and partial least squares (PLS).^[72] We also adopted the decision tree-based ensemble algorithms random forest (RF),^[73] AdaBoost,^[74] gradient boost,^[75] and XGBoost.^[76] Furthermore, we used the well-known nonlinear ML algorithms k-nearest neighbors (KNN),^[77] support vector machines (SVM),^[78] and Gaussian process regression (GPR).^[79] In contrast to other conventional ML-related tactile sensing approaches wherein a single or at most a few DL (or ML) algorithms are used, almost every possible DL (and ML) algorithm was introduced in the present investigation, resulting in a more robust tactile sensing system.

Our approach differs from other nonarray approaches^[50–57] in three key aspects. We use a super-simple, single-layer pad, whereas the other nonarray-type approaches generally adopt multilayer structures, and some layers have a type of micro-engineered structure that is not a sensory pattern array. In addition, we incorporate 19 state-of-the-art DL and ML algorithms for the processing of the detected signal, whereas either complex mathematical algorithms or only a few ML algorithms are used in the other approaches. Another crucial distinction of the present study is the enhanced pressure level. We increase the detection of the indented pressure level to 2.3 MPa, which is far beyond the typical tactile range (0–50 kPa),^[2–8] without losing precision in the low-pressure range. High-pressure detection is necessary when the sensory pad is applied in extreme circumstances, such as disasters, warfare, and other similarly harsh situations. Furthermore, when applied to e-skin, it can serve as a pain detector.

In the ensuing section, we discuss how several DL (and ML) algorithms were selected out of the 19 algorithms to constitute the tandem model, how the tandem model was customized for accurate detection of the indented location and pressure, and the performance of the tandem model. In addition, we discuss how the problem caused by the viscoelastic (or anelastic) nature of the sensor pad could be solved for real-world applications. Finally, some disadvantages of the proposed tactile sensing system at the current state of development are also discussed in detail, along with suggestions on how they might potentially be ameliorated.

Results and Discussion Classification Models for Indented Location Recognition Classification_10

CNN models were primarily adopted for the 10-class classification (Classification_10) to recognize either the current indented location or the untouched state. Namely, nine touched positions and an untouched state constitute the ten classes. It is noted that the tactile sensor is in the untouched state most of time, so that the recognition of untouched state should be prioritized in the classification process. The best CNN architecture (three convolutional layers with 64, 128, and 256 kernels and a kernel size of 16 × 16, with max pooling layers) was obtained by varying the number of convolution layers, kernel size, pooling, and number of fully connected layers. The size of the image and the number of images extracted from the original time series data were also varied along with the CNN architecture to achieve the optimal test accuracy. The number of available images is much larger than the number used for training, so the number of images in Table 1 is shown along with the total number of images. The two most important variables were the image size and number of images, which more heavily influenced the test accuracy than the details of the CNN architecture. Images with sizes of 16 × 50 and 16 × 100 were randomly extracted from the typical time series data. Table 1 shows the test accuracy as a function of these two variables based on the final determined CNN architecture, as shown in Figure 1a. All test accuracy values were obtained from hold-out test datasets that were not used for the training and validation processes. Two hold-out test datasets were prepared using individual image-based and cycle-based splitting schemes. Even the minimum number of images (50) with the minimum image size (16 × 50) yielded a test accuracy of 96.49% (94.55%), and the highest accuracy obtained was 99.61% (99.15%) for the number of images (100) with an image size of 16 × 100. Percentages within parentheses denote the test accuracy from the cycle-based data-splitting scheme. Other conditions also gave even higher test accuracies, which were close to 100%, although, for brevity, the results are not presented here.

Table 1 Test accuracy for hold-out datasets. When preparing the hold-out test datasets, both the individual data-based and cycle-based splitting schemes were used. The accuracy values inside the parentheses denote the test accuracy for the hold-out test dataset prepared by the cycle-based splitting scheme. The individual data-based splitting differs between CNN and DNN approaches, that is, individual image-based splitting was used for CNN and individual snapshot data-based splitting was used for DNN. The LSTM models adopt the cycle-based splitting scheme only, whereas the CNN and DNN models adopt both data-splitting schemes. The CNN test accuracy is given as a function of the image size and the number of images used for training, and the LSTM test accuracy is given as a function of the sequence length. A more detailed hyperparameter effect on the CNN test accuracy is described in a systematic manner in Table S3, supporting information

CNN
Image size		16 × 50			16 × 100
No. of images/total number of available images		50/950	100/950	200/950	50/900	100/900	200/900
Accuracy	Classification_10	96.49% (94.55%)	98.78% (95.25%)	98.10% (95.60%)	98.33% (98.69%)	99.61% (99.15%)	99.07% (99.07%)
	Classification_3	95.74% (94.30%)	97.47% (94.37%)	99.55% (94.67%)	99.08% (97.56%)	98.87% (97.67%)	99.69% (98.89%)
No. of images/total number of available images		50/450	100/450	200/450	50/400	100/400	200/400
Accuracy	Classification_9_P	97.78% (95.11%)	99.94% (94.33%)	100% (94.86%)	100% (98.00%)	99.65% (98.11%)	100% (98.72%)
	Classification_9_R	100% (100%)	100% (100%)	100% (100%)	100% (100%)	100% (100%)	100% (100%)
LSTM
Sequence length		50			100
Accuracy	Classification_10	(83.11%)			(85.78%)
	Classification_3	(87.57%)			(93.82%)
	Classification_9_P	(100%)			(100%)
	Classification_9_R	(100%)			(100%)
DNN
Accuracy	Classification_10	82.80% (80.06%)
	Classification_9_P	89.81% (84.04%)
	Classification_9_R	100% (100%)

Figure 1. Architecture of a) CNN, b) LSTM, and c) DNN. All architectures were determined through numerous trials with different architectures selected from scratch.

Using an LSTM model, we repeated the aforementioned Classification_10 task that the CNN models had already successfully accomplished. The original data acquired from the piezoresistive pad during the experimental training process were in time series format; hence, the LSTM models could be used effectively. The input data unit was a 16 × 1 vector-type snapshot with a sequence length of 50 (equivalent to 100 ms in the time series) or 100 (equivalent to 200 ms). The sequence length was set to be compatible with the image size used in the CNN approach. The hidden layer size was set to 64 after many trials. In addition, several LSTM layers were stacked and tested, but a single LSTM layer was finally adopted because no significant improvement was observed for multilayer LSTM. Many different structures for the fully connected layer at the end of the sequence were tested, and the final selected architecture included three hidden layers, which were directly connected to the output layer with ten nodes, denoting nine sectors and an untouched state. The final LSTM architecture is shown schematically in Figure 1b. The LSTM classification model for Classification_10 yielded a test accuracy of 85.78%, and more detailed results are shown in Table 1 as a function of the sequence length. The LSTM accuracy was obtained from only the cycle-based split hold-out test dataset, and thereby within the parenthesis. All LSTM test accuracies for Classification_10 were inferior to those of the CNN.

DNN classification models for the same Classification_10 task were also implemented using 16 × 1 vector-type snapshot input data. Finally, we pinpointed a three-hidden-layer architecture with (16):512:1024:512:(10) nodes per layer, as shown in Figure 1c. Note that many other architectures were also tested, but for brevity the results are not presented here. According to the present classification task (Classification_10), the number of nodes in the output layer was 10. Dropout and/or batch normalization did not affect the best architecture. The training was executed using a fivefold cross-validation strategy, after which a test was implemented using two hold-out datasets prepared by individual snapshot-based splitting and cycle-based data splitting. The test accuracy of DNN classification reached a maximum of 82.80% (80.06%), as shown in Table 1. The DNN-based classification models based on the snapshot data were proven to be unacceptable in comparison with the CNN results. The viscoelastic (or anelastic) trait leading to the asymmetric peak-shaped piezoresistive response might be the key reason for the deteriorated test accuracy of the DNN approaches.

Classification_3

In parallel with the CNN models for Classification_10 that achieved high test accuracy for indented location recognition, we also pursued a perfect level of accuracy using a tandem classification model. The tandem classification model adopted CNN models for the 3-class classification to recognize the current tactile status, indicating a pressing, releasing, or untouched state. The CNN model for the 3-class classification provided a nearly perfect accuracy of 99.69%, which is similar to the maximum 10-class classification result of 99.61%. The CNN architecture and its training routine were identical to those for the 10-class classification CNN model described in Section 2.1.1. Table 1 also shows the test accuracy for Classification_3 as a function of the size of the image and the number of images, based on the final pinpointed CNN architecture. Even the minimum number of images (50) with the minimum image size (16 × 50) yielded a test accuracy of 95.74% (94.30%), and the highest accuracy obtained was 99.69% (98.89%) for the maximum number of images (200) with an image size of 16 × 100.

Using LSTM models to complete the tactile status identification (Classification_3), an accuracy of 93.82% was attained. The LSTM architecture was identical to that of the LSTM model described in Section 2.1.1. The test accuracy data for LSTM-based Classification_3 are shown in Table 1. The hold-out test dataset was prepared using the cycle-based splitting scheme only, because the individual image-based splitting was not used for LSTM. The DNN is not appropriate for Classification_3 because the snapshot input data cannot be used to determine the tactile status. Thus, the DNN Classification_3 results are not shown in Table 1.

Classification_9 Following Classification_3

The tandem tactile sensing process consists of Classification_3 and Classification_9_P (or Classification_9_R), which enabled us to distinguish 19 states almost concurrently with an indentation (touch) on the piezoresistive pad, that is, the 19 states indicate 9 sectors under pressing, releasing, or untouched states. Classification_9_P (or Classification_9_R) includes CNN, LSTM, and DNN models for the identification of the indented sector location after the completion of Classification_3. Once this 19-state recognition was completed by the tandem tactile sensing model, the result immediately allowed for pressure regressions for each state afterward (except, of course, for the untouched state). In contrast, the 10-class classification (Classification_10) never allowed for an acceptable posterior pressure regression, although the Classification_10 task itself was successful. This is what the tandem tactile sensing system is for.

Considering the fact that the test accuracy for Classification_3 was guaranteed only to be ≈100%, rather than exactly 100%, the total test accuracy of the tandem tactile sensing model would deteriorate unless the indented sector location recognition, referred to as Classification_9_P (or Classification_9_R), exhibited a perfect test accuracy of 100%. If Classification_3 was completed with nearly perfect test accuracy, the ensuing 9-class classification CNN model could be trained independently either for the pressing (PP) or releasing part (RP). Two CNN models for Classification_9_P and Classification_9_R were trained separately using Dataset_P and Dataset_R, respectively. As a result, the fully trained CNNs were able to recognize the indented sector location with relatively high test accuracy. Table 1 shows the test accuracy of the CNNs for both Classification_9_P and Classification_9_R as a function of the training dataset size (number of images) and image size. Even the minimum number of images (50) with the minimum image size (16 × 50) yielded a test accuracy of 97.78% (95.11%) for Classification_9_P and 100% (100%) for Classification_9_R. The best performance obtained was 100% (98.72%) for Classification_9_P and 100% (100%) for Classification_9_R for 200 images with an image size of 16 × 100.

In addition to the CNN models, LSTM models were used for both 9-class classifications: Classification_9_P and Classification_9_R. The LSTM architecture was also identical to that of the Classification_10 LSTM model. Table 1 shows the test accuracy of the two LSTM models for Classification_9_P and Classification_9_R as a function of the sequence length. The LSTM results are even better than those of the CNNs, as every condition gave 100% accuracy. DNN models were also used for both Classification_9_P and Classification_9_R, wherein the instantaneous data from the time series (snapshot data) were used as input data units. The test accuracy of the DNN model is shown in Table 1. The DNN-based test accuracy was 89.81% (84.04%) for Classification_9_P and 100% (100%) for Classification_9_R. This discrepancy between the results of Classification_9_P and Classification_9_R implies that the viscoelasticity problem is severe with a DNN-based approach.

Pressure Regression DNN for Pressure Regression

In addition, pressure regression was implemented for each sector location under the premise that the indented sector location was already identified perfectly by the aforementioned tandem model system, and each sector required two DNN regression models working separately for both the pressing and releasing regimes. Consequently, two DNN models for pressure prediction were assigned to every indented sector location, one for pressing and the other for releasing; thus, we obtained 18 trained DNN models. Furthermore, it should be noted that every DNN model was trained twice based on two different training/test data-splitting schemes: individual snapshot-based splitting and cycle-based splitting. In total, 36 independent pressure regressions were implemented using DNN models with the same architecture as that used for Classification_3 and Classification_10, as shown in Figure 1c. In addition, the DNN model was trained using a five-fold cross-validation scheme along with the hold-out test dataset acquired from both the training/test data-splitting schemes. It should be noted that all the other pressure regression algorithms to be introduced henceforth also followed the same training scheme; specifically, fivefold cross validation, the two different hold-out test datasets, and separate training for pressing and releasing regimes were used for all other ML algorithms.

The 36 separate pressure regressions were implemented based on the premise that the preceding identification of tactile status and indented sector location could be achieved by the tandem tactile sensing model, and the total test accuracy of the tandem tactile sensing model could reach nearly 100% under all circumstances. Figure 2a shows 36 plots of DNN-predicted versus real pressures, that is, for each of the nine sectors, the pressing state and releasing state and the individual snapshot-based and cycle-based training/test data-splitting schemes are shown. All plots show acceptable coincidence between the predicted and real data. A slight deviation can be found in some sectors for the releasing state in the case of cycle-based data splitting. In particular, the deviation is more conspicuous in the center sector, wherein the deviated curves designate two indentation cycles belonging to the hold-out test dataset—one below and the other above the standard line. However, this deviation is acceptable. The DNN-based pressure regression following the exact identification of both the tactile status and indented sector location finally realized complete tactile sensing involving both indented location recognition and pressure prediction at the indented location.

Figure 2. Plot of real versus ML-predicted pressures for the hold-out test dataset for a) DNN, b) KRR, c) XGBoost, and d) GPR. 36 plots are shown for each ML algorithm, that is, one for each of the nine sectors, for either pressing (left) or releasing (right) state and for the individual data-based (top) and cycle-based (bottom) training/test data-splitting schemes. The MSE and R2 values averaged over nine sectors for the hold-out test dataset are summarized in Table S1, Supporting Information.

Regularized Linear Regression Algorithms for Pressure Regression

We introduced the regularized linear regression algorithms Ridge, Lasso, Elastic Net, KRR, LARS, BRR, ARD, and PLS. These regression algorithms work well for relatively small problems with a dearth of training data. Regularization indicates a special penalty included in the loss function to address the high-variance problem, which becomes serious when the training data are deficient. The L2 regularization approaches (Ridge), which is also known as the maximum a posteriori (MAP), L1 regularization (Lasso), and L1 + L2 regularization (Elastic Net), are representative regularization algorithms. The others are variants that originate from these basic regularization techniques. It is also interesting to use Bayesian regression, using fitted parameters that are not definite values but given as a distribution within a confidence interval; correspondingly, the ML model-predicted output is also given as a distribution within a confidence interval. A basic linear regression was also adopted as a baseline reference.

As a representative example, only the results of KRR are shown in Figure 2b; the results of the other algorithms are shown in Figure S1, Supporting Information. The other algorithms’ fitting quality was less than that of the KRR algorithm. The PLS algorithm, which is regarded as a type of linear regression algorithm with no regularization, exhibited the worst results among all the algorithms adopted here for the pressure regression. KRR has a nonlinear trait, ascribed to the use of kernels (the Matérn kernel was used herein), although it was categorized into a linear regression group. As shown in Figure 2a,b, the regression fitting quality is very similar to that of DNN, and a slight deviation in some sectors for the releasing state in the case of cycle-based data splitting was also similarly detected. The goodness of fit for the DNN and KRR algorithms was similar in terms of the mean-squared-error (MSE) and the coefficient of determination (R²) for the hold-out test dataset. Table S1, Supporting Information, shows the MSE and R² values for all pressure regression ML algorithms used in the present investigation.

Tree-Based Ensemble Algorithms for Pressure Regression

We introduced four ensemble algorithms: RF, AdaBoost, gradient boost, and XGBoost. The RF algorithm uses so-called bagging and the others use boosting, while both approaches adopt a decision tree as a base estimator. AdaBoost uses only stumps and not trees. Bagging decreases the variance in the prediction by generating additional data for training from the dataset using combinations with repetitions to produce multiple sets of the original data. In bagging, each model receives an equal weight. In boosting, however, models are weighted based on their performance. Boosting decreases bias, not variance. In bagging, the result is obtained by averaging the responses of the N learners (base estimators) or by a majority vote. In contrast, boosting deals with a weighted average of the estimates.

Figure 2c shows the results of XGBoost as a representative ensemble algorithm. As shown in Figure S1, Supporting Information, other ensemble algorithms such as RF and gradient boost also gave results as good as those of the XGBoost algorithm. A slight deviation in some sectors for the releasing state in the case of cycle-based data splitting was also inevitable in this case. As in the previous case, however, the deviation would not deteriorate the pressure predictability, even in real-world applications. Unlike the RF, gradient boost, and XGBoost algorithms, however, AdaBoost yielded slightly deteriorated fitting quality, particularly in the pressing state, which might be attributed to the use of stumps instead of trees. According to Table S1, Supporting Information, the training MSE values for RF, gradient boost, and XGBoost were relatively small compared with those for the other algorithms; this does not indicate overfitting because both the validation and hold-out test MSE were low as well.

Nonlinear ML Algorithms for Pressure Regression

We introduced some other traditional algorithms, specifically, KNN, SVR, and GPR. These three are well-known representative nonlinear regression algorithms. KNN and GPR are also known as model-free (parameter-free) regression algorithms, although some crucial hyperparameters are used. Figure 2d shows the results of GPR as a representative algorithm for this category. The others also provide good fitting quality, but the results of SVR are slightly inferior to those of GPR. Considering the finding that nonlinear algorithms yield promising fitting quality but linear regression algorithms such as basic linear, Ridge, and Lasso yield unacceptable fitting quality, the present pressure regression problem can be characterized as a typical nonlinear problem. It should be noted that KRR can be also characterized as a nonlinear algorithm due to the kernel.

Figure 2d shows the results of the GPR algorithm as a representative nonlinear algorithm. This algorithm is based on the Bayesian approach. Bayesian approach-based algorithms provide a range of confidence around the predicted mean rather than a deterministic prediction. The amber and green dots for the results of GPR indicate the upper- and lower-confidence boundaries (i.e., the standard deviation range), although the range is so narrow that these three colors (blue: predicted mean, amber: upper bound, and green: lower bound) cannot be discerned, as shown in Figure 2d. This type of Bayesian approach would be more desirable than other customary regularization strategies because uncertainty in the prediction can also be formulated. The KNN and SVR results are shown in Figure S1 and Table S1, Supporting Information. The GPR algorithm yielded relatively promising test accuracy in comparison with all other algorithms; therefore, we nominate GPR as the most appropriate constituent for the tandem tactile sensing model.

Discussion

Although the 10-class classification was successful in the identification of indented locations, it never led to successful pressure regression. The recommended sequence for complete tactile sensing based on the tandem model to achieve both indented location identification and indented pressure detection was Classification_3 → Classification_9_P (or Classification_9_R) → pressure regression. Several different classification/regression algorithms (CNN, LSTM, DNN, KRR, XGBoost, GPR, etc.) could be adopted for this sequence; in other words, some combinations of ML (and DL) models can constitute the sensing sequence, whereas other combinations cannot. Both the CNN and LSTM models effectively conducted the 3-class classification (Classification_3), but the CNN slightly outperformed the LSTM model. Classification_9_P and Classification_9_R were also successfully completed by the CNN, LSTM, and DNN models, but the LSTM slightly outperformed the others. There are many combinations of DL algorithms (CNN, LSTM, and DNN) that constitute the suggested tandem model system to achieve acceptable recognition of indented locations. Figure 3 schematically shows the possible indented−location−recognition routes (sequences), and their corresponding total test accuracy values are shown. The test accuracy from the cycle-based split hold-out test dataset (accuracies in parentheses) was regarded as a baseline because these accuracy values were always slightly lower than those from the individual data-based splitting. CNN models were found to be the best option for Classification_3 (1st step), whereas LSTM models were found to be ideal for Classification_9_P and Classification_9_R (2nd step). Despite the fact that the CNN (1st step) → CNN (2nd step) route obtained the highest total accuracy of 99.69% from the individual image-based split hold-out test dataset, the CNN (1st step) → LSTM (2nd step) route is highlighted as a recommendation in Figure 3, and the total test accuracy for this route is 98.89%, which is still acceptable; therefore, it is possible to proceed to the pressure prediction at each of the identified sector locations.

Figure 3. Possible routes (sequences) for indented location recognition along with the corresponding total test accuracy values, obtained by multiplying the test accuracies at each step. The classification tasks belonging to each step are summarized on the left side. DL algorithms are shown interconnected in red and blue lines, such that CNN-initiated routes are marked with red lines and the LSTM-initiated routes are marked with green lines, and the thick red line designates a recommended route. The accuracy values inside the parentheses designate the indentation cycle-based split hold-out test dataset result. Note that routes involving LSTM exhibit only the accuracy values written inside parentheses and include no individual data-based splitting results, as only the cycle-based training/test data-splitting scheme was applied to LSTM.

Furthermore, all the pressure regression algorithms involving 17 ML algorithms were applied to the ensuing pressure regression, and DNN, KRR, XGBoost, and GPR algorithms were revealed to outperform the others because these nonlinear ML algorithms were able to account for the nonlinear data characteristics. Correspondingly, some other nonlinear ML algorithms also performed well, but the conventional linear regression ML algorithms performed worse than these nonlinear ML algorithms. Accordingly, it is evident that many possible combinations of ML (and DL) models could be nominated as reliable constituents for the tandem tactile sensing model, which consists of three parts: tactile status identification (first step), indented location identification (second step), and pressure prediction (third step). The tentative best option was determined to be CNN (first step) → LSTM (second step) → GPR (third step), leading to an acceptable total test accuracy of 98.89% for the final recognition of the indented location (first and second steps), an MSE of 4.2 × 10⁻⁵ (2.5 × 10⁻³), and an R² of 99.97% (98.05%) for the pressure prediction (third step).

In contrast to the indentation experiment described earlier, which used a fixed depth of 5 mm, random indentations with depths of 1–5 mm were also implemented to achieve more robust tactile sensing. Consequently, a much larger training/test dataset was prepared to train the same tandem tactile sensing model. Figure S2, Supporting Information, shows the random indentation data collected for the nine sectors. These random indentation data were used to train the CNN models for Classification_10, Classification_3, and Classification_9_P (or Classification_9_R). For convenience, LSTM training was skipped. Table S2, Supporting Information, shows the test accuracies for random indentation obtained from the hold-out test datasets prepared using both the individual image-based and cycle-based splitting schemes. All the test accuracies for CNN models dealing with Classification_10, Classification_3, and Classification_9_P (or Classification_9_R) were slightly inferior to those for single-depth indentation but still sufficiently high to be conventionally accepted. However, although the indented location identification was viable for random indentation, the ensuing pressure regression was in question in this case, irrespective of whether the pressure regression result was successful. The pressure regression result was acceptable; nonetheless, the significance of the pressure regression result was reduced because the total accuracy of the indented location identification was not nearly 100%, only reaching ≈98.08% (84.58%) and 98.28% (90.94%) at best for the Classification_3 + Classification_9_P and Classification_3 +Classification_9_R tandem models, respectively. Further effort should be made to improve the total accuracy of indented location detection for random indentation. In this regard, the suggested tandem tactile sensing model should be improved, and thereby the result of the pressure regression, which has already proven to be promising, could be considered as reliable. The tentative pressure regression results, which would be ineffectual unless the indented location detection was improved significantly, are not presented here for brevity. Further efforts are being made to improve both indented location identification and pressure regression using the random indentation dataset.

Unlike the acceptable test accuracy of CNN and LSTM, the deteriorated test accuracy of the DNN models for the indented location recognition seemed to result from the viscoelastic (or anelastic) nature of the CNT-dispersed PDMS pad. Therefore, we introduced the CNN algorithms along with the image-type data and LSTM algorithms with a certain sequence length to tackle the viscoelasticity (anelasticity) problem, which caused a slight asymmetric indentation peak shape and, in turn, the more conspicuous asymmetric shape of piezoresistive response. However, the introduction of other ML (or DL) algorithms that constitute the tandem tactile sensing model would generate another complication. Regardless of which types of ML (or DL) algorithms were introduced to lead to improved accuracy, such an improvement would be ad hoc. Ubiquitous tactile sensing would be available only by either improving the pad materials or by introducing microengineered structures. Improvement of the ML (or DL) algorithms, in parallel with a change in data curation, is not an ultimate solution for the problem. An improvement in either the pad material or pad structure should be the final solution. Because we pursue a super-simple, single-layer structure in the present investigation, the introduction of a microengineered structure would be counterproductive.

Furthermore, a search for novel piezoresistive composite materials that behave like a perfect linear elastic body would likely be futile. The discovery of a perfect linear elastic pad material would resolve all the current complications, and there would be no need to use such a complicated tandem model-based system or transform the training data into image data; instead, a simple DNN model with instantaneous snapshot-type data would suffice. Thus, we have experimented with many other materials (particularly host polymer materials) such as PMMA, Ecoflex, and epoxy resins and also tried many other conducting materials such as CNTs, graphene, and aluminum powder. In addition to these various combinations of insulating matrix materials and conducting reinforcement materials, we also tried many different processing conditions as well as different mixing ratios. Unfortunately, we have failed to discover an appropriate linear elastic (more correctly “less viscoelastic”) piezoresistive composite material thus far. Rather than expending further effort to discover a less viscoelastic piezoresistive composite material, improving the algorithm is definitely a more plausible and reasonable approach. Accordingly, we should improve the algorithms and data curation to achieve more robust tandem tactile sensing.

Despite the improvement of the algorithm component, the present approach is far from complete and has several restrictions in comparison with state-of-the-art nonarray tactile sensing technologies.^[50–57] The most conspicuous restriction is that the present approach is only applicable to the detection of vertical force (indentation). However, other types of tactile sensing would be possible if we altered the training process to fit the desired force application mode. For instance, if the detection of slip force was desired, we could train the pad in a slip mode. Although the training details would need to be meticulously set up, it would be possible to realize slip-mode tactile detection. In addition, the present technique is confined to single-point detection and is incapable of detecting multiple touches. Multitouch detection could be possible if multitouch training was implemented. Note, however, that the multitouch training would be difficult, as a larger amount of more complicated training data would be required. The increased amount of training data would solve any tactile sensing problem regardless of complexity, but it would also require a more expensive training process. All others who have used ML (or DL) techniques in their tactile sensing approaches still maintained the conventional sensory device concept involving a microengineered, multilayer structure. We hope that our approach involving a super-simple, single-layer pad with no microengineered structure could become comparable with conventional state-of-the-art tactile sensor technologies, although the current status of development is far behind the state of the art. In addition to the mechanical force detection that we are concerned with, other sensory functions such as temperature detection, moisture detection, and chemical (hazardous) substance detection have been recently addressed within the framework of tactile sensing.^[1–20] This type of advanced multipurpose tactile sensing is also a great challenge at the current state of development of the proposed tactile sensor pad.

Conclusion

Rather than either the fabrication of a patterned array of small sensory devices or the microengineering of multilayer, nonarray type sensors, we developed a super-simple single-layer, nonarray, tactile sensor pad through the use of a crude bulky material: in this case, a piezoresistive CNT-dispersed PDMS pad. The ML (or DL) algorithms converted this simple piezoresistive pad into a smart tactile sensor merely by training it. This unprecedented idea was experimentally proven to be plausible and applicable to many sensor domains.

The viscoelastic (or anelastic) nature of the piezoresistive CNT-dispersed PDMS pad is an obstacle for the widespread adoption of the proposed idea. The tandem model concept was introduced along with a data grouping defined as “data conversion into image-type data” to tackle this—viscoelastic (or anelastic)—problem. However, an ultimate solution for the problem should be established in future work through improvement of either the pad material or structure.

For the tandem model system, many different combinations (sequential routes) of ML (DL) algorithms with various training schemes were extensively tested. From the results, we determined that a tandem tactile sensing model sequentially consisting of CNN for Classification_3, LSTM for Classification_9_P (or Classification_9_R), and DNN, KRR, XGBoost, or GPR for pressure prediction could be an interim solution. The CNN → LSTM → GPR route yielded an acceptable total test accuracy of 98.89% for indented location recognition, an MSE of 4.2 × 10⁻⁵ (2.5 × 10⁻³), and an R² of 99.97% (98.05%) for the pressure prediction.

Experimental Section Material Preparation

A homogeneous piezoresistive composite pad was synthesized from commercially available MWCNTs with a diameter of 20 nm and a length of 5 μm (Carbon Nano-material Technology Co. Ltd) and polydimethylsiloxane (PDMS) (Sylgard 184 Silicone Elastomer). To construct a piezoresistive composite material, 1 wt% of the CNTs was mixed with liquid PDMS in a cylindrical plastic container. To achieve homogeneous dispersion of the CNTs in PDMS and avoid agglomeration, a few 10 mm alumina grinding balls were added, and the container was transferred to a planetary shear mixer at a mixing speed of 400 rpm for 2 h. A PDMS curing agent was then added at a weight ratio of 1:10, and the container was transferred to a planetary shear mixer for 20 min. Finally, the composites were degassed under vacuum for 20 min to remove the entrapped bubbles. Using the doctor blade method, pristine PDMS was cast in a 73 mm × 73 mm × 10 mm acrylic mold as a base layer, after which a piezoresistive CNT/PDMS composite was cast on top of the base layer. The CNT/PDMS composite was then solidified at 60 °C for 45 min. The surface was also coated with an insulating PDMS layer. Sixteen copper wires were connected to the electrode terminal of an MWCNT−PDMS extension at a fixed interelectrode distance from each MWCNT−PDMS pad edge, and a graphite fiber bundle was embedded at the center of the pad and connected to the center electrode. The detailed schematic drawing and the photograph of the tactile sensor is shown in Figure S3, Supporting Information. In addition, the entire sample preparation procedures are schematically described in Figure S4, Supporting Information. The basic property of proposed tactile sensor is described in the previous DNN approach.^[58,59]

Sample Training and Testing

The conceptual training process is schematically shown in Figure 4. The piezoresistive pad was trained using an Instron E3000. A smaller area (36 mm × 36 mm) inside the piezoresistive pad was virtually divided into 3 × 3 sectors. Because we compartmentalized the pad area into nine sectors, the nine-class classification constituted the basic location identification task. The area of a single sector and the indenting rod diameter simulated the fingertip of a human. The Instron E3000 was equipped with a wooden rod with a cross-sectional diameter of 10 mm, which could be used to systematically press a selected sector of the piezoresistive pad. The piezoresistive pad was placed on a movable x−y stage so that the pressing location could be changed. By repetitively pressing each sector in sequence, we collected electric current data from 16 probe terminal electrodes on the pad edge. A video clip elucidating the testing process is available in Supporting Information.

Figure 4. Overall schematics for the experimental training process, including the piezoresistive sensor pad, data acquisition apparatus (center), computing facility (middle right), adopted algorithm schematics (lower right), and final detection results (lower left). An inset inside the yellow circle shows the microstructure schematics of the CNT-dispersed PDMS composite, the inset inside the green circle describes the indentation on the pad using the actuator machine (Instron E3000), and the inset inside the red circle indicates a circuit for data acquisition at each electrode. The voltage drop (Vout) at the reference resistor (RO = 100 kΩ) can be converted to the piezoresistive current (and piezoresistance). RP denotes piezoresistance and Vin is the bias applied on the center of the backside of the pad.

Dataset Preparation

The data were collected during 19 loading/deloading (or pressing/releasing) cycles on each sector location; a single pressing/releasing cycle lasted for 2 s with an indentation depth of 5 mm. The pressure values reached far beyond the conventional tactile range, as shown in Figure 2. The piezoresistive current and resistance were calculated from the voltage drop at the reference resistor (100 kΩ) connected to every terminal electrode, while a 7 V bias was applied to the center of the piezoresistive pad. The voltage signal sampling speed was 500 samples/s. The data acquisition process entailed a single indentation (a single pressing/releasing cycle) with 1000 samples, which means that 1000 data points were collected from each of the 16 electrodes during a single indentation over 2 s. The acquired data involved 19 indentation cycles for each sector. In addition to the pressing/releasing data, the data under no (or negligible) pressure—a condition that is referred to as the untouched state—should be taken into account. Therefore, we collected the data for the untouched state as well, the amount of which should be set to the same as the amount of data assigned to each sector to avoid an unbalanced amount of data for the untouched state. Consequently, the input data with a size of (3 × 3 + 1) × (1000 × 16) × 19 were used in the classification task to identify 3 × 3 sectors and an untouched state (referred to as Classification_10) and for pressing/releasing/untouched status classification (referred to as Classification_3). The term “+1” designates the dataset for the untouched state, and “x 16” denotes the piezoresistive current signal (voltage signal at the reference resistor) collected from the 16 electrodes. The term “19” refers to the 19 pressing/releasing cycles at each sector. The sector label and pressure data were simultaneously collected as an output (label) dataset. The entire dataset is referred to as Dataset_Cls.

Dataset_Cls was also subdivided into three subdatasets: pressing, releasing, and untouched state subdatasets. Consequently, we prepared three subdatasets: a PP (Dataset_P), RP (Dataset_R), and untouched part (Dataset_U). Each of the subdatasets was also split into training and test datasets. Dataset_Cls, Dataset_P, and Dataset_R were split into training and test datasets using two different schemes. In the first scheme, we randomly split the total image dataset on the basis of individual images, and ≈10% of the total image data was set as hold-out test data. The other splitting scheme involved grouping based on the indentation cycle. All images belonging to each indentation peak (each pressing/releasing cycle) were grouped when the training/test data were split, that is, two indentation cycles were set aside as hold-out test data, whereas the remaining 17 cycles were used as a training dataset. The test accuracy, MSE, and R² results from the indentation cycle-based splitting are always presented inside parentheses.

The validation appeared to have no effect on the training process because the training dataset size was sufficiently large. In fact, fivefold cross-validation was also implemented, but it was proven to make no improvement. Only a hold-out test dataset, the size of which was around 10% of the total dataset size or two indentation cycles out of 19 cycles, was set aside to evaluate the final test accuracy. When the CNN model was used, the test accuracy values from both data-splitting schemes were similar, but only the indentation cycle-based data-splitting scheme was used for LSTM because the LSTM algorithm would never allow for reasonable training if the individual snapshot-based random data-splitting scheme was adopted. The DNN models adopted both data-splitting schemes (individual snapshot-based and indentation cycle-based splitting schemes); these models are not concerned with the image-type data, instead they use the snapshot-type data.

Problem Setting

The key issue of the present investigation is restricted to the accurate detection of the indented location and pressure, which is a central function of tactile sensing despite additional, more complicated issues such as the detection of temperature, chemicals, and moisture. The indented location detection was accomplished via classification in the present investigation. For the classification-based identification of the indented location, we segmented the pad area into 3 × 3 sectors, experimentally trained the pad by indenting every sector 19 times, and eventually produced a very large dataset, Dataset_Cls. Although this miniature system could be criticized for being too small, we already created a much larger pad with a sufficiently large number of sectors to constitute a keypad.^[59] As the main purpose of the present investigation was to determine a solution to the viscoelastic problem, this miniature system was sufficient. The entire classification problem was subdivided into several settings. A single DL model based on the pristine (noncurated) Dataset_Cls could not achieve acceptable DL-driven tactile sensing. Manifold settings for the classification problem and a crucial data transformation are required, rather than a single DL model with the pristine dataset. The data curation here briefly designated a sort of data split and transformation—the experimental data in the form of time series were split into three parts with respect to time, and the sections were labeled as PP, RP, and untouched parts; thereafter, the PP and RPs were processed to train two independent DL models. This type of data-splitting process is necessary to realize real-time tactile detection. In addition to the data splitting, data transformation was also necessary for some DL algorithms, which is discussed in more detail later in this subsection.

Figure 5a shows the entire set of collected data for the classification (the time series data of both the electric current detected at 16 terminal electrodes and the instantaneous pressures) on nine indented sectors for 74 s with a sampling speed of 500 samples s⁻¹. Each of the nine graphs representing the corresponding sectors had 16 lines with different colors that were signals from the 16 terminal electrodes. We implemented 19 indentations (19 pressing/releasing cycles) over 74 s on each indented sector. Figure 5b shows a single pressing/releasing cycle (a single indentation peak) for each sector. The viscoelastic (or anelastic) nature of the CNT-dispersed PDMS pad resulted in an asymmetric piezoresistive current signal, although the asymmetry of the indentation pressure was less prominent. We split a single indentation peak into three segments: pressing, releasing, and untouched. This time series data compartmentation is necessary for correct real-time tactile sensing when the tactile sensing pad is viscoelastic (or anelastic). The instantaneous tactile status indicating whether the location of interest on the pad is being pressed, being released, or untouched should be identified during a period much shorter than the human tactile response time; thereafter, complete tactile sensing, that is, the identification of the indented location and pressure prediction, should be achieved almost concurrently by referring to the preceding tactile status identification result, indicating pressing, releasing, or untouched states. Consequently, the identification of the instantaneous tactile status fell into a three-class classification task, which should precede the identification of indented location and pressure prediction.

Figure 5. Schematic representation of all indentation data consisting of piezoresistive current signals in time series form (voltage drop at each reference resistor) from the 16 edge electrodes (marked in different colors) and pressure signal data. Each of nine graphs representing the corresponding sector has 16 signals (the lines with different colors). The data are separately arranged for each sector. a) All indentation cycles with an indentation depth of 5 mm and b) a single pressing/releasing cycle (called an indentation peak). The color of each piezoresistive signal curve is directly matched to the corresponding color of the electrode in the schematics on the bottom. The pressure curve was drawn downward from the top of the plot frame for convenience.

In the previous DNN approach,^[58,59] the electric current data (so-called snapshot data) in the shape of a 16 × 1 vector were instantaneously extracted from the 16 electric current time series and used as input data for real-time tactile sensing. That is, the instantaneous identification of the indented location and the pressure evaluation were implemented using the snapshot data based on the assumption that the piezoresistive CNT-dispersed PDMS pad was completely linear elastic. In sharp contrast to the previous simple DNN approach based on the snapshot data, we rearranged the time series data in such a manner that data collected within every period of 100 ms can be grouped and treated as an input data unit with dimensions of 16 × 50. This means that the 16 piezoelectric current signals detected from the 16 edge terminal electrodes over a period of 100 ms, corresponding to 50 samples due to the sampling speed of 500 samples s⁻¹, were grouped as an input data unit, that is, for every moment we predicted an output, the output was predicted from the input data history for the last 100 ms. In addition, the data were grouped not only in units of 100 ms, but also in longer periods of 200 ms, which is equivalent to a size of 16 × 100. The human tactile response time is generally known to be ≈200 ms, and the response is retarded as the pressure increases up to the pain level.^[80–83] Therefore, the data grouped in units of 100 and 200 ms would pose no problem even when the DL-based tactile sensing models were applied for artificial electronic skin (e-skin).

Now that the input data had a 2D shape of 16 × 50 (or 16 × 100), these data could be treated as a sort of image, and thus 2D convolution was available for use in CNNs. For convenience, we refer to the 16 × 50 (or 16 × 100) input data unit as “image-type input” and the 16 × 1 input data unit as “vector-type snapshot input.” The image-type input data were used for CNNs conducting the three-class classification described earlier (Classification_3) and the other ensuing CNN-based classification tasks in the present investigation. Although the original time series data were used for the LSTM models, the input data unit for LSTM can be considered identical to the 16 × 50 (or 16 × 100) image type used for CNNs by setting the sequence length in the LSTM to 50 (or 100). This philosophy behind data transformation (grouping), which indicates that the instantaneous output is predicted from the past history of the input for a certain period, is effectively analogous to the basic concept of LSTM.^[64] We introduced the image-type input data to consider the viscoelastic (or anelastic) behavior of the CNT-dispersed PDMS pad, the details of which are discussed later.

The conventional DNN model along with the vector-type snapshot input data was applied to accomplish the instantaneous recognition of indented location and pressure based on the premise that the piezoresistive CNT-dispersed PDMS pad behaves like a perfect linear elastic body.^[58,59] In this regard, a relatively slow loading/deloading rate was applied during the training and testing process to make it possible to use the simple ANN (or DNN) model along with the snapshot input data.^[58,59] However, the piezoresistive pad that we used for the present investigation exhibited a viscoelastic (or anelastic) nature. Note that the anelasticity differed from the viscoelasticity because a complete recovery was ensured under any circumstances.^[84] The piezoresistive pad that we used seemed more like an anelastic body in this regard, but we did not precisely discern viscoelasticity and anelasticity for convenience in the present investigation. Due to the viscoelastic (or anelastic) trait, the data (16 piezoresistive current signals represented in different colors) in both the RPs and RPs in the time series were not symmetric, although the pressure data appear roughly symmetric, as shown in Figure 5b. Accordingly, ANN (or DNN) models based on snapshot input data were not accurate. Therefore, as described earlier, we separated the total data into three parts: pressing, releasing, and untouched (or negligibly weakly touched). We use the term “tactile state” (or “tactile status”) to refer to the pressing, releasing, or untouched state. If a certain DL classification model enabled us to recognize immediately which tactile state the pad was currently in (pressing, releasing, or untouched), then we achieved more accurate location/pressure detection immediately after such a tactile state identification. This location/pressure sensing was achieved separately for both the pressing and releasing tactile states by introducing two separate DL models that were trained independently in two different data regimes, that is, pressing and releasing regimes. Further, these separately trained models were utilized for accurate indented location recognition and reliable pressure prediction following the tactile state identification.

This sort of tandem model-based tactile sensing system would be practical for conventional e-skin applications, considering the fact that the typical human tactile response time (≈200 ms) is much longer than the total processing time (latency) for the tandem DL model. The latency was at the most ≈100 μs for the tandem DL model to complete all tasks involving tactile state identification (Classification_3), indented location recognition (Classification_9_P or Classification_9_R), and pressure regression in the present case, wherein a typical GPU-based calculation was used. Even when using a practical device (such as a Jetson Nano) in real-world contexts, the latency should still be reasonable in comparison with the human tactile response time.

The indented location detection was also implemented using a 10-class classification model that was trained with the entire dataset (Dataset_Cls), without having to identify the tactile state in advance, and the resultant test accuracy for the location recognition reached 99.07% (99.07%), as shown on the green top-right side in Figure 6. This 10-class classification, referred to as Classification_10, designated the identification of nine sectors and an untouched state. Although the total dataset-based 10-class classification provides acceptable test accuracy for indented location recognition, pressure prediction was not possible when using the total dataset without data splitting. Considering the fact that the indented location was nearly perfectly identified with a test accuracy of 99.07% (99.07%), the pressure regression could be conducted independently for each sector location. However, the pressure regression for every sector was unviable because of the viscoelastic (or anelastic) nature of the pad. A prerequisite for acceptable pressure prediction was to separate the dataset into pressing and RPs. If the preliminary detection of tactile status, indicating whether a particular indented location on the pad is under pressure or release, is correctly completed, we can establish a pressure prediction model for each sector location. The pressure prediction model for every sector should also be divided into two parts corresponding to two tactile statuses, as shown by the PP and RP on the blue bottom area in Figure 6. As a result, we developed a tandem tactile sensing system enabling the sequential (almost concurrent in view of human tactile response time) recognition of tactile status and the indented location and in turn, the prediction of pressure on the identified sector location. This tandem tactile sensing system is schematically shown in the green, yellow, and blue areas on the left side of Figure 6.

Figure 6. Total problem setting for both the indented location recognition and pressure prediction. The top-left green area designates the current tactile status recognition (Classification_3), that is, identifying whether the relevant indentation is being pressed, being released, or untouched. The yellow area in the middle designates the classification-based indented location recognition for nine (3 × 3) compartmentalized sectors for both the pressing and releasing regimes; the ensuing DNN regression-based pressure evaluation at the indented sector location is shown in the blue area, wherein the plot of predicted versus real pressure is presented, and the other ML-based pressure regressions (so-called integrated ML platform) are described in the larger blue region on the bottom right. The top-right green area designates the identification of indented sector locations and untouched state (Classification_10) using all data (Dataset_Cls) without subsplitting. The large green arrow indicates that reliable pressure regression is unviable following Classification_10.

The tactile status identification (Classification_3) was primarily executed using CNN and LSTM models, as shown in green in the top of Figure 6. Classification_3 led to a nearly perfect test accuracy, 99.69% (99.89%), for the identification of current tactile status, indicating whether the location was in the pressing, releasing, or untouched state. According to the tactile status identification result, indented location identification was promptly completed using either of the two 9-class classification DL models that were separately trained with the pressing and releasing data subsets (Dataset_P and Dataset_R), respectively. Classification_9_P and Classification_9_R, used for indented sector location identification, are marked in yellow in Figure 6. This indented location recognition process immediately led to the pressure regression on the identified sector location, as shown in the blue area of Figure 6. The pressure regression was implemented separately at all nine sector locations. The regression result, that is, the plot of real versus predicted pressure, is shown for every sector location in Figure 6. In addition to the conventional DNN regression model, 16 additional ML models were used for the pressure regression in this case. The regression results for all 16 ML models are shown in Figure S1 and Table S1, Supporting Information. In addition to the CNN-based classification, we also used LSTM and DNN to solve the Classification_3, Classification_9_P, Classification_9_R, and Classification_10 problems.

The indented location recognition procedure was subdivided into two classification steps. The first step was Classification_3, which simply distinguished whether the current state was pressing, releasing, or untouched. The ensuing classification step was location identification, which was referred to as either Classification_9_P (for the PP) or Classification_9_R (for the RP). If the current tactile state was correctly identified as pressing, releasing, or untouched, then the indented location can be immediately identified based on the predetected tactile status; thereafter, the pressure on the indented location can be evaluated. Specifically, the 9-class classification (Classification_9_P or Classification_9_R) was executed following the prior 3-class classification (Classification_3). A prerequisite condition for this type of tandem tactile sensing to be successful is that both Classification_3, used for tactile state recognition, and Classification_9_P (Classification_9_R), used for indented location recognition, should be perfect or nearly perfect, as the total test accuracy should be the product of the test accuracies from both classification tasks. Accordingly, we achieved nearly 100% test accuracy for both the prior Classification_3 and posterior Classification_9_P (or Classification_9_R).

Acknowledgements

M.-Y.C. and J.-W.L. contributed equally to this work. This research was supported by Hyundai Mobis Co., Ltd., and partly by the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT, and Future Planning (2015M3D1A1069705, 2021R1I1A1A01059583, and 2021R1A2C1009144) and by Digital manufacturing platform (N0002598) funded by MOTIE, Korea.

Conflict of Interest

The authors declare no conflict of interest.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request. The codes used for this study are available from the GitHub link at https://github.com/socoolblue/tactile-sensor.