Evaluation of Machine Learning Algorithms for Classification of EEG Signals

Abstract

In brain–computer interfaces (BCIs), it is crucial to process brain signals to improve the accuracy of the classification of motor movements. Machine learning (ML) algorithms such as artificial neural networks (ANNs), linear discriminant analysis (LDA), decision tree (D.T.), K-nearest neighbor (KNN), naive Bayes (N.B.), and support vector machine (SVM) have made significant progress in classification issues. This paper aims to present a signal processing analysis of electroencephalographic (EEG) signals among different feature extraction techniques to train selected classification algorithms to classify signals related to motor movements. The motor movements considered are related to the left hand, right hand, both fists, feet, and relaxation, making this a multiclass problem. In this study, nine ML algorithms were trained with a dataset created by the feature extraction of EEG signals.The EEG signals of 30 Physionet subjects were used to create a dataset related to movement. We used electrodes C3, C1, CZ, C2, and C4 according to the standard 10-10 placement. Then, we extracted the epochs of the EEG signals and applied tone, amplitude levels, and statistical techniques to obtain the set of features. LabVIEW™2015 version custom applications were used for reading the EEG signals; for channel selection, noise filtering, band selection, and feature extraction operations; and for creating the dataset. MATLAB 2021a was used for training, testing, and evaluating the performance metrics of the ML algorithms. In this study, the model of Medium-ANN achieved the best performance, with an AUC average of 0.9998, Cohen’s Kappa coefficient of 0.9552, a Matthews correlation coefficient of 0.9819, and a loss of 0.0147. These findings suggest the applicability of our approach to different scenarios, such as implementing robotic prostheses, where the use of superficial features is an acceptable option when resources are limited, as in embedded systems or edge computing devices.

1. Introduction

The central nervous system is composed of the spinal cord and the brain; the human brain resides in the skull and is considered an essential part of the central nervous system [1]. The human brain is composed of 100 billion neurons on average [2], the joint action of which is responsible for thoughts, actions, and emotional states. The brain is divided into the right and left hemispheres, where the right hemisphere is in charge of regulating the muscular activity of the left side of the human body, while the left hemisphere regulates the activities of the right side [3]. In order to measure and record the brain’s activities, different neuroimaging techniques are used, including magnetoencephalography (MEG) [4], electrocorticography (ECoG) [5], intracortical neuronal recording, functional magnetic resonance (fMRI) [6], near-infrared spectroscopy (NIRS) [7], and electroencephalography (EEG), the latter being the neurophysiological technique [8] most widely accepted by the scientific community and the private sector in the development of research in fields such as neuroscience, robotics, home automation, the Internet of Things, education, etc. [9].

Electroencephalography (EEG) is a non-invasive procedure for measuring the electrical activity generated within the brain as a result of different mental processes [10]. The electrical signals are acquired through electrodes placed on the scalp’s surface; thus, waves with different amplitudes and frequencies that refer to a person’s mental state are obtained [11]. The frequency ranges span from 0 Hz to 100 Hz. Based on these ranges, the signals are classified as follows: delta, which ranges from 0 Hz to 4 Hz; theta, which contains signals from 4 Hz to 7 Hz; alpha, where the information range is between 8 Hz and 12 Hz; beta, where the range is between 12 Hz and 30 Hz; and gamma, with a range that covers from 30 Hz to 100 Hz [12,13]. Different ranges of signals are essential for identifying different clinical problems, such as schizophrenia [14], Alzheimer’s, insomnia, epileptic disorders, brain tumors, and different injuries and infections related to the central nervous system. Furthermore, classification of motor impairment in neural disorders by means of EEG signals processing has been a successful method for identifying central nervous system roots of motor disabilities [15]. Compared with other methods, this neuroimaging technique offers advantages such as portability, temporal resolution, safety, cost, small time constants, simple equipment, and effectiveness [16].

The EEG neuroimaging method is the preferred method for developing brain–computer interfaces (BCIs), both in the academic community and the private sector. Historically, BCI has been clinically applied for understanding motor impairment, both in verbal communication [17] and limb movement [18], as well as cognitive impairment [19], and offers a great advantage over electromyography pattern recognition [20] due to the lack of neuromuscular signals under amputation conditions. BCIs are direct communication and control channels between users’ brains and computers where muscle activity is not involved [21,22]. They are currently considered a powerful communication technology as they do not involve muscular routes to complete tasks such as communication, commands, and actions. The basis of these systems is the computer, whose central role is the analysis of EEG signals [23,24]. BCIs are classified as exogenous and endogenous. Exogenous BCIs require external conditions or stimuli so that the brain can generate a particular response based on the stimulus. Endogenous BCIs do not require external stimulation; however, they require some training on the user’s part so that they can regulate brain rhythms [24]. Despite the differences mentioned, most BCI models contain the following elements: signal acquisition, information preprocessing, feature extraction, and classification [16,25]. The acquisition of signals is carried out by employing electrodes placed on the scalp’s surface [26], through which analog signals are obtained and then digitized by means of analog–digital converters. The next step is the preprocessing of the signals, whereby the following are removed: noise induced by the electrical line; the background noise of the brain; various artifacts that the EEG signals present as a result of some muscular activity such as eye movement, facial muscle activity, etc. [27]. Feature extraction is one of the crucial steps due to its impact on the performance of classification algorithms [28]. Some of the obtained features are in the domains of time and frequency [28], i.e., mean, median, variance, maximum, and minimum, among others [29,30]. The feature extraction process produces a vector containing the most relevant features of the EEG signals, used as input for classification algorithms. The next step is classification, which is carried out by different algorithms, including LDA, SVM [31], KNN [32], D.T. [33], N.B., and ANN [34].

Currently, there are different fields of science, engineering, and research that evaluate and make use of BCIs to develop applications that present solutions to complex problems [35,36]. These have been possible due to advances in high-density electronics, data acquisition systems that allow high-quality EEG signals to be acquired, intelligent systems that use machine and deep learning algorithms, and neural networks that allow pattern recognition and signal classification to be performed with high precision. In [25], the authors explain that BCIs can be used in the following six application scenarios: replace, restore, augment, enhance, supplement, and research tools. The authors of [37] commented that current and future BCI application areas are device control, user status monitoring, assessment, training and education, gaming and entertainment, cognitive enhancement, safety, and security. Intelligent systems commonly incorporate machine learning (ML) approaches [38,39,40]. ML refers to a system able to learn from training data from certain activities so that the analytical model generation process is automated, and associated tasks can be completed or supplemented [41,42]. Deep learning (DL) is a paradigm within ML based on the use of artificial neural networks (ANNs) [41]. Commonly, ML algorithms focus on classifying EEG signals related to the motor and imaginary movements of hands and feet to carry out control actions, as presented in [43,44,45,46]. DL is useful in areas with vast and high-dimensional data; therefore, deep neural networks outperform ML algorithms for most text, images, video, voice, and audio processing techniques [47]. Nevertheless, for low-dimensional data input, especially with insufficient training data, ML algorithms may still achieve superior results [48], which are even more interpretable than deep neural network results [49]. The authors of [50] used power, mean, and energy as features to classify EEG signals related to the right and left hands through artificial neural networks (ANNs) and support vector machine (SVM). In [51], the authors used SVM to control the direction of a wheelchair by extracting the mean, energy, maximum value, minimum value, and dominant frequency characteristics of the EEG signals. In [52], the authors used the fast Fourier transform and principal component analysis as characteristics of the EEG signals to feed the SVM classifier to control a robotic arm. The authors of [53] reported the use of EEG signals to control an exoskeleton and the use of SVM, LDA, and NN for their respective classification. Studies such as the one presented in [54] have used pretrained neural network models to classify EEG signals through time–frequency characteristics. Recent studies have focused on the proper selection of EEG signal characteristics and its effect on the accuracy of ML and DL algorithms, as presented in [30]. ML and DL techniques are widely accepted and help to develop specific tasks within different applications [55,56,57,58,59,60,61]. Moreover, they are increasingly used to obtain EEG data for pattern analysis, classification of group membership, and BCIs [29,62,63,64,65,66,67]. However, there are still open research problems, such as the real-time processing of EEG signal classification and the optimization of ML algorithms for implementation on embedded systems or edge computing devices. Hence, research on and development of reliable, efficient, and robust systems for EEG signal classification, among others, should be pursued [16,68]. The complexity of human movements for the manipulation of tools is very high and diverse; for an adult human brain that has automated different movements, it does not represent a major effort, however, for ML it requires the management of precise information inputs that allow programming and execution of free movement. Previous studies offer multiple classes of motor imagery limb movements based on EEG spectral and time domain descriptors [69]; in this sense, there continues to be a need in machine learning to increase the reliability and accuracy of EEG signals used for programming human-like movements.

For the reasons stated above, the aim of this paper is to evaluate nine ML algorithms for the classification of EEG signals. The purpose is to find which ML model presents the best performance metrics for the identification of movement patterns in EEG signals for the control of a mechatronic system, in this case, a robotic hand prosthesis. The selected dataset consists of more than 1500 EEG recordings of 1–2 min in length from 109 subjects and is publicly available in [70]. In this study, we randomly selected 30 subjects to train, validate, and test the proposed method. The ultimate aim is to facilitate the development of robotic limb prosthetics, which is possible because ML algorithms can recognize patterns in EEG signals with complex dynamics. The hypothesis is that ML algorithms perform better in tasks of signal classification than standard methods. The novelty of this study is to provide a methodology for the classification of EEG signals by training several ML algorithms and employing processing, analysis, and feature extraction techniques in the time domain of various lapses of EEG signals related to motor tasks, which can be translated into commands for the control of mechanisms or mechatronic systems such as wheelchairs, robotic prostheses, and mobile robots.

The rest of this paper is organized as follows: Section 2 presents the materials used for the development of the proposed method; additionally, the description of the dataset used for this paper is presented. Section 3 presents the performance metrics obtained from the proposed ML models and the discussion of the main findings obtained in this study. Section 5 presents the proposed usage scenario in a real-world application. Conclusions and future work are described in Section 6.

2. Materials and Methods

2.1. Hardware and Software

The hardware used for the implementation of the proposed method had the following specifications: Microsoft Windows 10 Pro operating system, system model OptiPlex 3070, system type ×64-based PC, Processor Intel Core i5-9500 at 3.00 GHz, six Cores, six logical processors, memory (RAM) of 16.0 GB DDR4 2666 MHz (2 × 8 GB), and NVIDIA GeForce GT 1030 GDDR5 2 GB PCI-Express ×16. The software used for reading the EEG signals, electrode selection, signal segmentation, preprocessing, analysis, feature extraction, and preparation of the dataset was LabVIEW 2015. Furthermore, the following libraries, which are part of the development environment of LabVIEW, were used: Biomedical Toolkit and Signal Express. The MATLAB 2021a version was used for training and testing the different ML algorithms, which are part of the Statistics, Machine Learning and Deep Learning Toolbox.

2.2. Machine Learning Algorithm Training

In this paper, we selected nine ML algorithms to evaluate their performance in the classification of EEG signals related to the motor movements of right hand, left hand, both fists, feet, and relaxation. The nine selected algorithms are naive Bayes (N.B.), k-nearest neighbors (KNN), decision tree (D.T.), support vector machine (SVM), linear discriminant analysis (LDA), Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN. These ML algorithms are part of the statistical and machine learning toolbox of MATLAB, which has various tools that can be used for both the pre- and post-processing of data.

Figure 1 shows the block diagram to train, test, and evaluate the selected ML algorithms. First, the dataset is loaded; the chosen dataset is constituted of more than 1500 EEG recordings from 109 subjects that become between 1 and 2 minutes long and can be found in [70]. In this study, 30 people were randomly chosen to train, test, and validate the proposed method. Subsequently, the data are normalized between 0 and 1 to obtain better results. Next, we randomly split the dataset into 80% for training and 20% for testing. Then, the ML model is trained. The next step is to obtain the performance metrics of the ML models (for example, using the confusion matrix), i.e., the performance metrics to evaluate the ML algorithms, such as the area under the curve (AUC) and accuracy, among others.

A typical system for EEG signal classification is conceptually divided into signal acquisition, preprocessing, feature extraction, and classification [23,71]. The EEG signals are acquired by electrodes located on the scalp’s surface that transfer information on the electrical neuronal activity to the data acquisition system. In preprocessing, line noise and muscle artifacts are removed from EEG signals. Feature extraction uses several digital signal processing techniques to obtain feature vectors. These vectors are used to train the ML or DL algorithms to classify the EEG signals. The result of the algorithms is a specific class, as illustrated in Figure 2. The following subsections describe the procedure in detail.

2.3. Input Data

The dataset used for EEG signal classification was developed by Schalk and colleagues at Nervous System Disorders Laboratory and is publicly available on Physionet [70]. The data consist of more than 1500 EEG recordings of 1–2 min in length from 109 subjects. Patients performed 14 tasks (experiments) while 64 electrodes acquired and recorded the EEG signals through the BCI2000 system [72]. The data are in EDF+ format [73], and they contain 64 EEG signals, each displayed at a rate of 160 samples per second, and an annotation channel, which refers to the actions performed during the task. Table 1 shows the protocol of the Schalk agreement experiment. The diagram of the position of the electrodes used to record the data is the standard 10-10 placement. The dataset consists of 109 folders, and each folder contains 28 files, where 14 of these have the *.edf extension, and the other 14 have the *.edf.event extension. The files that contain the EEG signals are those that contain the *.edf extension. The *.edf.event files refer to the events during the development of the different tasks. Although the original set of recorded data consists of continuous multichannel data, and the number of users that comprise it is extensive, we only used the EEG signals of 30 randomly selected subjects, and the tasks that are related to the real movements that take place in tasks 3, 5, 7, 9, 11, and 13. In tasks 3, 7, and 11, real movements related to the right and left fists and relaxation are carried out, while in tasks 5, 9, and 13, real movements of both fists and both feet are carried out. Table 1 summarizes the dataset used in the proposed approach.

2.4. Proposed Method for EEG Signal Processing

Figure 3 depicts the proposed method for EEG signal processing, described in detail in the following subsections.

2.5. EEG Signal Acquisition and Channel Selection

The LabVIEW software 2015 version was employed as the development platform, while the Biomedical Toolkit was used to import the EEG signals, due to the signals being in EDF format. The selected electrodes are shown in Figure 4b. These electrodes present neuronal activity correlated to the execution of the left- and right-hand movements (contained in electrodes C3, C4, and CZ [74,75]) and the neuronal activity related to the movement of both feet (contained in electrodes C1 and C2 [76]); because the different EEG channels tend to represent redundant information, as mentioned in [77], electrodes C3, C1, CZ, C2, and C4 were selected in our study. The selected electrodes were located around the center of the skull, within the motor cortex area; their characteristic is that these electrodes are the least affected by different artifacts [78], which allows the reliable extraction of features to be obtained.

2.6. Preprocessing

The EEG signals used, with a sampled frequency of 160 Hz, are available online [70]. Bandpass filters were required to select only the frequencies of interest and eliminate line noise and some other interferences. For this study, we processed the EEG signals through an IIR bandpass filter, with third-order Butterworth topology from 0.1 to 50 Hz. After this, a 50 Hz notch filter was applied to the signals to eliminate noise from the signal power line. Figure 5 shows the original readings of the electrodes used before and after applying the different filters related to the signal preprocessing operations.

2.7. EEG Band Separation

Within EEG signal analysis, it is common to separate a signal into different frequency bands, including Delta (1–4 Hz), Theta (4–8 Hz), Alpha (8–12 Hz), Beta (12–30 Hz), and Gamma (30–50 Hz). As shown in Table 2, third-order bandpass Butterworth IIR filters with different cut-off frequencies were used to carry out this separation.

2.8. Feature Extraction

The features of the EEG rhythm can be obtained by using several digital signal processing techniques. These features were used for training the nine ML algorithms. These analysis techniques included measurements of tone, amplitude, and level, as well as statistical analyses. Table 3 shows the type of measurements and features obtained when these techniques were applied to the EGG signal epochs.

Signal Analysis

• Tone measurements. The tone measurements carried out in the EEG signal epochs were the following: amplitude, frequency, and phase.

• Level measurements. The level measurements implemented in the EEG signal epochs were the following: peak-to-peak, negative peak, and positive peak.

• Statistical features. The statistical measurements applied to the different signal epochs were the following:

  • -. Median [30,79]:

    (1)$\begin{array}{c}\hfill Median=\left\{\begin{array}{c}\frac{(N+1)}{2},\mathrm{when}N\mathrm{is}\mathrm{odd}\hfill \\ \frac{N}{2}+\frac{(N+1)}{2},\mathrm{when}N\mathrm{is}\mathrm{ever}\hfill \end{array}\right.\end{array}$

  • -. Mode is the number that occurs most frequently in the set;

  • -. Mean [80]:

    (2)$\tilde{x}=\frac{1}{N}\sum _{i=1}^N{x}_i$

  • -. Root mean square (RMS) [80]:

    (3)$RMS=\sqrt{\frac{1}{N}\sum _{i=1}^N{x}_i^2}$

  • -. Standard deviation [80]:

    (4)$S=\sqrt{\frac{1}{N}\sum _{i=1}^N{(x_i-\tilde{x})}^2}$

  • -. Summation:

    (5)$\sum _{i=1}^N{x}_i$

  • -. Variance [80]:

    (6)$S^2=\frac{1}{N}\sum _{i=1}^N{(x_i-\tilde{x})}^2$

    where $\tilde{x}$ is the mean;

  • -. Kurtosis [80]:

    (7)$Kurtosis=\sum _{i=1}^N\frac{{(x_i-\tilde{x})}^4}{(N-1)s^4}$

  • -. Skewness [80]:

    (8)$Skewness=\sum _{i=1}^N\frac{{(x_i-\tilde{x})}^3}{(N-1)s^3}$

2.9. Dataset Preparation

The data vectors consist of 15 features, 3 features for each electrode; the electrodes correspond to positions C3, C1, CZ, C2, and C4, which are related to motor movements, and these belong to one of the five classes of “relaxation”, “Right hand”, “Left hand”, and “Fist and Feet”. The dataset has 2792 samples, where 558 samples correspond to the “Relaxation” class, 567 to the right hand, 555 to the left hand, 561 to both fists, and 547 to the feet. On average, there are 557 samples per class, which preserves the balance among the classes. Figure A1 in Appendix A shows a fragment of the dataset created by processing EEG signals when different users performed different motor tasks. Figure A2 in Appendix B depicts the graphic user interface (GUI) of the software (App) developed for the feature extraction process. The proposed App allows features to be extracted in different frequency bands, where each frequency band corresponds to a different class. Table 3 shows the features obtained for training the different ML algorithms. Each line represents a vector of features consisting of five electrodes. Three different measurements were made for each electrode, which resulted in a vector with 15 different characteristics used for the training and testing of the ML and DL models. It can be observed that the feature vector is labeled with its respective class. For each of the five classes, 15 different features were obtained in five different frequency bands to improve the classification accuracy of the ML algorithms [81,82]. The proposed dataset can be downloaded at the link from Supplementary Materials.

3. Results

To evaluate the performance of the ML algorithms, we used the following scoring metrics: $Accuracy$, $Error$, $Recall$, $Specificity$, $Precision$, and $F1$-$Score$.The performance evaluation of the proposed ML models was initiated by calculating $Sensitivity$, $Specificity$, $Precision$, and $Accuracy$ [83,84]. $Sensitivity$, also known as $Recall$ [84], measures the proportion of positives that are correctly identified as such; it can be calculated by (9). Similarly, $Specificity$ measures the proportion of negatives that are correctly identified as such [84]; it can be calculated by (10). $Precision$ is the proportion of true positives among the positive predictions [84]; it can be calculated by (11). $Accuracy$ can be calculated using (12):

(9)$Recall=\frac{TruePositives}{FalseNegative+TruePositives},$

(10)$Specificity=\frac{TrueNegatives}{FalsePositives+TrueNegatives},$

(11)$Precision=\frac{TruePositives}{TruePositives+FalsePositives},$

(12)$Accuracy=\frac{TruePositives+TrueNegatives}{TruePositives+FalsePositives+TrueNegatives+FalseNegatives}.$

$F1$-$Score$ is a method for combining $Precision$ and $Recall$ into a single measure that includes both [85]. Neither $Accuracy$ nor $Recall$ can analyze the complete situation on their own. We might have outstanding $Precision$ but poor $Recall$, or vice versa, poor $Precision$ but good $Recall$. With $F1$-$Score$, one can represent both concerns with a single score [86]. Once $Accuracy$ and $Recall$ for a binary or multiclass classification task have been computed, the two scores may be combined to calculate the $F1$-$Score$ metric; it can be calculated by (13):

(13)$F\mathrm{1-}Score=\frac{2\ast Precision\ast Recall}{Precision+Recall}.$

Equations (9)–(13) are valid for binary classification and multiclass issues; however, when used for multiclass problems, they must be calculated for each class and then averaged to obtain each metric per model.

Table 4 shows the average scores obtained in each performance metrics by the nine ML algorithms selected in this study. The first parameter analyzed was accuracy, where the LDA model presented an accuracy score of 0.9229; D.T. obtained 0.9803; KNN obtained 0.8996; N.B. obtained 0.9373; SVM obtained 0.9803; Narrow-ANN, Medium-ANN, and Bilayered-ANN obtained 0.9857; finally, Wide-ANN obtained 0.9821. The Narrow-ANN, Medium-ANN, and Bilayered-ANN models obtained the best accuracy score (0.9857). Regarding the error metric, we can see that the LDA, D.T., N.B., SVM, Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN algorithms achieved a score less than 0.1, while the KNN model obtained an Error greater than 0.1; therefore, the models with the lowest error were Narrow-ANN, Medium-ANN, and Bilayered-ANN (0.0143). Considering the recall parameter, we observed that the Narrow-ANN algorithm presented the highest score of 0.9863, while the KNN algorithm obtained the lowest score of 0.9037. Regarding the specificity metric, all the algorithms achieved a score greater than 0.9; the ML models with the best results were the Narrow-ANN, Medium-ANN, and Bilayered-ANN models, all scoring 0.9964. Regarding the precision metric, the Bilayered-ANN algorithm is the one that presented the best result, with 0.9859, while the KNN algorithm presented the lowest score, with 0.9099. Regarding the F1-score parameter, the LDA, D.T., N.B., SVM, Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN algorithms achieved scores greater than 0.91, while the KNN model obtained a score below 0.91. The algorithm that presented the best F1-score result was Narrow-ANN, with 0.9859.

Table 5 presents the performance metrics achieved by each ML algorithm. The metrics used to evaluate the performance of the ML algorithms were the area under the average curve (AUC average), Cohen’s Kappa coefficient [87], Matthews correlation coefficient [88], and model loss. Concerning the AUC average metric, all algorithms achieved a score greater than 0.90, where the top three ML models were the SVM, Medium-ANN, and Bilayered-ANN models, which obtained the highest scores (AUC scores). Regarding Cohen’s Kappa coefficient, a score above 0.8 indicates exemplary commitment, while zero or less indicates poor commitment. The LDA and KNN algorithms obtained Cohen’s Kappa coefficients less than 0.80 but greater than zero, while the D.T., N.B., SVM, Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN algorithms achieved Cohen’s Kappa coefficients of 0.9384, 0.8040, 0.9384, 0.9552, 0.9552, 0.9440, and 0.9552, respectively, where the Narrow-ANN, Medium-ANN, and Bilayered-ANN algorithms achieved the highest scores. In addition, we used the Matthews correlation coefficient, which has been widely used as a performance metric for ML algorithms since 2000. The best scores obtained were presented by the D.T, N.B., SVM, Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN models (0.9736, 0.9225, 0.9757, 0.9824, 0.9819, 0.9783, and 0.9820, respectively), with Narrow-ANN obtaining the best score, while the KNN algorithm achieved the lowest score of 0.8810. The ML model with the lowest loss was Narrow-ANN, with 0.0136, followed by the Medium-ANN and Bilayered-ANN models, both with 0.0147, while the ML algorithm with the highest loss was KNN.

Figure 6 shows the ROC curves of the top four ML algorithms trained for the classification of EEG signals related to the state of relaxation, right hand, left hand, both hands, and both feet. These algorithms are LDA, SVM, D.T., and N.B. The algorithm that presented the best performance metrics was SVM, with an AUC average of 0.9988. The ROC curves showed a compromise between sensitivity and specificity. The SVM algorithm was the closest to the upper-left corner of the ROC space, while the D.T. model was closer to the 45-degree diagonal. Classifiers that obtain curves closer to the upper-left corner indicate better performance, while classifiers with ROC curves closer to the 45-degree diagonal of the ROC space are less accurate.

Figure 7 shows the ROC curves of the top four DL algorithms (neural networks) trained for the classification of EEG signals. These algorithms are Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN. The algorithm that presented the best performance metrics was Medium-ANN, with an AUC average of 0.9998; it was the closest to the upper-left corner of the ROC space.

In machine learning, the presumably best model is chosen from a collection of model candidates obtained by evaluating various model types, hyperparameters, or feature subsets, among others. In this paper, it is proposed to use ConfusionVis, a model-agnostic technique for evaluating and comparing multiclass classifiers based on their confusion matrices [56]. Figure 8 depicts the ConfusionVis achieved for the nine ML models chosen for EEG signal classification. Figure 8a shows the average accuracy score per ML model, where it can be observed that Narrow-ANN had the best accuracy score. Figure 8b illustrates the confusion matrix similarity results, where it can be seen that the D.T., SVM, Narrow-ANN, and Medium-ANN models obtained the best similarity. Figure 8c depicts the error by class scores, where it can be observed that Medium-NN and Narrow-ANN achieved the lowest error score in most classes of movements classified from the EEG signals. Figure 8d shows the error by model scores, where it can also be seen that Medium-ANN obtained the lowest error score, followed by Bilayered-NN, Narrow-ANN, and decision tree (D.T.).

Figure 9 shows the training time of the nine ML algorithms tested, with N.B., LDA, and KNN having the shortest training time. However, the results shown in Table 4 and Table 5 show that these algorithms had the lowest performance metrics, with the exception of D.T. In contrast, the SVM, Narrow-ANN, Medium-ANN, Wide-ANN, and Bilayered-ANN algorithms had the most considerable training times of 0.13546, 0.37135, 0.16956, 0.36255, and 0.45722 s, respectively, with the Bilayered-ANN algorithm having the longest training time. However, these algorithms had the best performance metrics, as shown in Table 4 and Table 5 and Figure 7 and Figure 8. Therefore, the data science engineer or researcher must perform a cost–benefit analysis regarding accuracy and processing time. In most circumstances, engineers favor accuracy over training time, because training is only performed a few times and only the trained ML model is employed. For this reason, in this study, it is more convenient to select the Narrow-ANN model.

4. Discussion

In this study, we observed that the different features used were helpful for the classification of EEG signals, as proposed in our hypothesis. The presented features are based on the time domain: amplitude, frequency, phase, peak–peak value, negative peak, positive peak, median, mode, average, mean square error value, standard deviation, summation, variance, kurtosis, and skewness. We consider that they are good features for classifying EEG signals related to movements. Using these features, the ML model that achieved the best performance was Medium-ANN, with average area under the curve of 0.9998, Cohen’s Kappa coefficient of 0.9552, Matthew correlation coefficient of 0.9819, and loss of 0.0147.

We observed that the performance metrics obtained from the nine machine learning algorithms were good. Using standard features in different frequency bands and related to a particular class allowed machine learning and deep learning algorithms to obtain excellent performance metrics, as shown in Table 4 and Table 5 and Figure 6, Figure 7 and Figure 8; this is because the proposed frequency bands and features improved the separability of the data, making the classification algorithms substantially better.

Regardless, the data science engineer/scientist is in charge of carrying out the corresponding analysis in terms of costs–benefits and precision concerning the information processing time. In most cases, ML models with better precision are chosen, and training time is usually sacrificed. Since the training of the ML algorithms is performed once, only the trained model is used for the assigned task. The Medium-ANN algorithm was selected for this reason and because its performance metrics were the best. Therefore, feature extraction is worth mentioning among the processes that improve relevant information acquisition and ensure better performance metrics when training EEG signal classification algorithms, as shown in different studies. Our results are consistent with other spectrogram methods implemented for identifying EEG patterns in persons with motor impairment using similar brain sources that were analyzed in this study [15]. Many human behavior fields are still a challenge for BCIs; findings from this study may provide complementary data for other studies reporting findings from central nervous system damage with residuals of motor impairment of upper limb movement [18]. In addition to limb paralysis, limb loss represents an obstacle to quality of life for which the results of this study offer a comprehensive and reliable technique for extracting electrical brain sources for human movement programming. As in other research [20], results of the present study provide consistent and accurate information for future controlling inputs for the adaptation of prosthesis. As reported elsewhere [69], we conclude that it is necessary to increase movement classes in EEG features extraction for providing mechatronic systems controlled by means of BCI, suitable and reliable patterns corresponding for target movements.

5. Proposed Usage Scenario

The ML algorithms proposed in this research study could be implemented in high-performance embedded systems or edge computing devices as verified in previous studies [59,89]. These act as the central control system, which is in charge of communicating with the BCI to acquire EEG signals. Likewise, the control system is in charge of carrying out the digital processing of the EEG signals, the extraction of features, the classification, and the translation (decoding and execution) of the control commands. The mechatronic control system would have a trained ML model which would allow a user with some motor disability to perform some motor activities, such as opening and closing the right fist, left fist, or both fists through the classification of EEG signals.

Figure 10 depicts a conceptual diagram of the prospective mechatronic control system. We could consider this model the first step in developing intelligent prostheses that integrate the system’s several components. The future characteristics to be developed are lower cost, size, portability, low power consumption, and reliable communication with the BCI.

Limitations of the Study

One of the drawbacks of this research study is the need for a BCI; users should have short hair, as the BCI must be comfortable and enjoyable to them. Furthermore, the electrodes must be maintained in saline solution. Successful implementation also relies on the BCI battery life. Finally, if the emotional state of the participants is altered, accurate measurements cannot be acquired.

6. Conclusions

In this study, a methodology for classifying motor movements by processing the EEG signals of 30 users is presented. The classification of the EEG signals was related to left hand, right hand, both fists, feet, and relaxation movements. As a result of EEG signal processing, a customized dataset was created and used to train the ML algorithms. The dataset was obtained by reading the EEG signal files in EDF+ format, extracting the different segments of the EEG signals, filtering the signals, extracting the features, and labeling their corresponding classes. The customized dataset was created to train and evaluate the performance metrics of different ML algorithms in the classification of EEG signals related to motor movements. The model of Medium-ANN achieved the best performance metrics, with an AUC average of 0.9998, Cohen’s Kappa coefficient of 0.9552, Matthews correlation coefficient of 0.9819, and loss of 0.0147. These findings enable the approach to be applied to different scenarios, such as robotic prosthesis implementation, where the utilization of physical qualities is an acceptable alternative when hardware resources are restricted, or in embedded systems or edge computing devices, which have the advantages of low cost, small size, portability, low power consumption, and reliable communication with the BCI.

Furthermore, with the proposed method, we estimate that quantifiable information about motor movement can be obtained through the feature extraction and performance metrics of ML algorithms. We also believe that the proposed method could allow us to generate different datasets that could be used for future studies, as the proposed software was developed and customized to analyze EEG signals.

Author Contributions

Conceptualization, E.E.G.-G. and E.I.-G.; Data curation, J.M.C.-V., E.G.-C. and O.R.L.-B.; Formal analysis, E.E.G.-G., E.T.-C. and E.G.-C.; Investigation, F.J.R.-A., E.T.-C., G.M.G.-A. and E.I.-G.; Methodology, F.J.R.-A., G.M.G.-A. and E.I.-G.; Project administration, O.R.L.-B.; Resources, F.J.R.-A., O.R.L.-B. and E.I.-G.; Software, F.J.R.-A.; Supervision, E.E.G.-G. and E.I.-G.; Validation, J.M.C.-V., E.G.-C. and G.M.G.-A.; Visualization, J.M.C.-V. and O.R.L.-B.; Writing—original draft, F.J.R.-A.; Writing—review and editing, E.E.G.-G., E.T.-C., E.G.-C. and E.I.-G. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

We share the dataset as supplementary material.

Acknowledgments

The authors are very grateful to PRODEP for supporting the academic groups to increase their degree of consolidation. We also want to thank the Faculty of Engineering and Technology Sciences (FCITEC) for all the facilities provided for the development of this project. F.J.R.-A. would like to thank SPSU-UABC (Union of Teachers University Improvement of the Autonomous University of Baja California) for the scholarship supporting their postgraduate studies.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. Block diagram for training, testing, and evaluating the ML algorithms.

Figure 2. Proposed method for classifying EEG signals.

Figure 3. Proposed method for EEG signal classification.

Figure 4. Selected electrodes for EEG signal classification. (a) Biomedical toolkit and (b) electrodes selected.

Figure 5. EEG signals acquired from electrodes C3, C1, CZ, C2, and C4. (a) Original EEG signal and (b) filtered EEG signal.

Figure 6. The receiver operating characteristic (ROC) curves of the top four ML algorithms trained for EEG signals classification, related to the movements of hands and feet: (a) ROC curves of the LDA algorithm, (b) ROC curves of the SVM algorithm, (c) ROC curves of the D.T. algorithm, and (d) ROC curves of the N.B. algorithm.

Figure 7. The receiver operating characteristic (ROC) curves of the four ANN algorithms trained for EEG signals classification, related to the movements of hands and feet: (a) ROC curves of the Narrow-ANN algorithm, (b) ROC curves of the Medium-ANN algorithm, (c) ROC curves of the Wide-ANN algorithm, and (d) ROC curves of the Bilayered-ANN algorithm.

Figure 8. ConfusionVis [56]: Comparative evaluation of the multiclass classifiers based on confusion matrices. (a) Averaged accuracy per model, (b) confusion matrix similarity, (c) error by class, and (d) error by model.

Figure 9. Training time of the nine ML models.

Figure 10. Suggested usage scenario for an application of mechatronic control system.

Table 1

Tasks presented in the dataset to train the ML algorithms for EEG signal classification.

Task	Real Movement	Imaginary Movement	To	T1	T2	Duration
1	Open Eyes	-	Relaxing	-	-	1 min
2	Close Eyes	-	Relaxing	-	-	1 min
3	Fist	-	Relaxing	Left	Right	2 min
4	-	Fist	Relaxing	Left	Right	2 min
5	Fist/Feet	-	Relaxing	Fist	Feet	2 min
6	-	Fist/Feet	Relaxing	Fist	Feet	2 min
7	Fist	-	Relaxing	Left	Right	2 min
8	-	Fist	Relaxing	Left	Right	2 min
9	Fist/Feet	-	Relaxing	Fist	Feet	2 min
10	-	Fist/Feet	Relaxing	Fist	Feet	2 min
11	Fist	-	Relaxing	Left	Right	2 min
12	-	Fist	Relaxing	Left	Right	2 min
13	Fist/Feet	-	Relaxing	Fist	Feet	2 min
14	-	Fist/Feet	Relaxing	Fist	Feet	2 min

Table 2

Cut-off frequencies of bandpass filters for band extraction of EEG signals.

Band of EEG Signal	Low Cut-Off Frequency	High Cut-Off Frequency
Delta	0.1 Hz	3.99 Hz
Theta	4.0 Hz	7.99 Hz
Alpha	8.0 Hz	11.99 Hz
Beta	12.0 Hz	29.99 Hz
Gamma	30.0 Hz	49.99 Hz

Table 3

Features of the EEG signal used to train the ML algorithms.

	Features of the Channels for the Different Electrode Positions
Band	C3	C3	C3	C1	C1	C1	Cz	Cz	Cz	C2	C2	C2	C4	C4	C4	Class
Delta	Amplitude	Frequency	Phase	Peak to Peak	Neg.Peak	Pos.Peak	Median	Mode	Mean	RMS	S.D.	Summation	Variance	Kurtosis	Skewness	Relaxing
Theta	Amplitude	Frequency	Phase	Peak to Peak	Neg.Peak	Pos.Peak	Median	Mode	Mean	RMS	S.D.	Summation	Variance	Kurtosis	Skewness	Left Hand
Alpha	Amplitude	Frequency	Phase	Peak to Peak	Neg.Peak	Pos.Peak	Median	Mode	Mean	RMS	S.D.	Summation	Variance	Kurtosis	Skewness	Right Hand
Beta	Amplitude	Frequency	Phase	Peak to Peak	Neg.Peak	Pos.Peak	Median	Mode	Mean	RMS	S.D.	Summation	Variance	Kurtosis	Skewness	Fist
Gamma	Amplitude	Frequency	Phase	Peak to Peak	Neg.Peak	Pos.Peak	Median	Mode	Mean	RMS	S.D.	Summation	Variance	Kurtosis	Skewness	Feet

Table 4

The average score parameters of the EEG classification algorithms.

	Average Scoring Parameters
ML Algorithm	Accuracy	Error	Recall	Specificity	Precision	F1-Score
LDA	0.9229	0.0771	0.9219	0.9807	0.9332	0.9228
D.T.	0.9803	0.0197	0.9777	0.9951	0.9792	0.9783
KNN	0.8996	0.1004	0.9037	0.9747	0.9099	0.9047
N.B.	0.9373	0.0627	0.9384	0.9844	0.9382	0.9378
SVM	0.9803	0.0197	0.9789	0.9950	0.9827	0.9803
Narrow-ANN	0.9857	0.0143	0.9863	0.9964	0.9857	0.9859
Medium-ANN	0.9857	0.0143	0.9854	0.9964	0.9856	0.9855
Wide-ANN	0.9821	0.0179	0.9834	0.9955	0.9824	0.9828
Bilayered-ANN	0.9857	0.0143	0.9854	0.9964	0.9859	0.9856

Table 5

Performance metrics of the nine ML algorithms trained for EEG signal classification.

	Performance Metrics
ML Algorithm	AUC Average	Cohen’s Kappa Coefficient	Matthews Correlation Coefficient	Loss
LDA	0.9889	0.7592	0.9072	0.0787
D.T.	0.9873	0.9384	0.9736	0.0229
KNN	0.9392	0.6864	0.8810	0.0961
N.B.	0.9935	0.8040	0.9225	0.0616
SVM	0.9988	0.9384	0.9757	0.0217
Narrow-ANN	0.9982	0.9552	0.9824	0.0136
Medium-ANN	0.9998	0.9552	0.9819	0.0147
Wide-ANN	0.9984	0.9440	0.9783	0.0165
Bilayered-ANN	0.9988	0.9552	0.9820	0.0147

Supplementary Materials

The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/technologies10040079/s1.

References

1. Savadkoohi, M.; Oladunni, T.; Thompson, L. A machine learning approach to epileptic seizure prediction using Electroencephalogram (EEG) Signal. Biocybern. Biomed. Eng.; 2020; 40, pp. 1328-1341. [DOI: https://dx.doi.org/10.1016/j.bbe.2020.07.004]

2. Lent, R.; Azevedo, F.C.; Andrade-Moraes, C.; Pinto, A. How many neurons do you have? Some dogmas of quantitative neuroscience under revision. Eur. J. Neurosci.; 2012; 35, pp. 1-9. [DOI: https://dx.doi.org/10.1111/j.1460-9568.2011.07923.x] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/22151227]

3. Carter, R. The Human Brain Book: An Illustrated Guide to Its Structure, Function, and Disorders; Penguin: London, UK, 2019; Google-Books-ID: S8bhDwAAQBAJ

4. Gross, J. Magnetoencephalography in Cognitive Neuroscience: A Primer. Neuron; 2019; 104, pp. 189-204. [DOI: https://dx.doi.org/10.1016/j.neuron.2019.07.001] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31647893]

5. Ince, N.F.; Goksu, F.; Tewfik, A.H. ECoG Based Brain Computer Interface with Subset Selection. Biomedical Engineering Systems and Technologies; Fred, A.; Filipe, J.; Gamboa, H. Springer: Berlin/Heidelberg, Germany, 2009; pp. 357-374. [DOI: https://dx.doi.org/10.1007/978-3-540-92219-3_27]

6. Son, J.; Ai, L.; Lim, R.; Xu, T.; Colcombe, S.; Franco, A.R.; Cloud, J.; LaConte, S.; Lisinski, J.; Klein, A. et al. Evaluating fMRI-Based Estimation of Eye Gaze During Naturalistic Viewing. Cereb. Cortex; 2020; 30, pp. 1171-1184. [DOI: https://dx.doi.org/10.1093/cercor/bhz157] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31595961]

7. Coyle, S.; Ward, T.; Markham, C.; Mcdarby, G. On the suitability of near-infrared (NIR) systems for next-generation brain-computer interfaces. Physiol. Meas.; 2004; 25, pp. 815-822. [DOI: https://dx.doi.org/10.1088/0967-3334/25/4/003]

8. Ramadan, R.A.; Vasilakos, A.V. Brain computer interface: Control signals review. Neurocomputing; 2017; 223, pp. 26-44. [DOI: https://dx.doi.org/10.1016/j.neucom.2016.10.024]

9. Fontanillo Lopez, C.A.; Li, G.; Zhang, D. Beyond Technologies of Electroencephalography-Based Brain-Computer Interfaces: A Systematic Review From Commercial and Ethical Aspects. Front. Neurosci.; 2020; 14, 611130. [DOI: https://dx.doi.org/10.3389/fnins.2020.611130]

10. Esqueda-Elizondo, J.J.; Juárez-Ramírez, R.; López-Bonilla, O.R.; García-Guerrero, E.E.; Galindo-Aldana, G.M.; Jiménez-Beristáin, L.; Serrano-Trujillo, A.; Tlelo-Cuautle, E.; Inzunza-González, E. Attention Measurement of an Autism Spectrum Disorder User Using EEG Signals: A Case Study. Math. Comput. Appl.; 2022; 27, 21. [DOI: https://dx.doi.org/10.3390/mca27020021]

11. Teplan, M. Fundamentals of Eeg Measurement. Meas. Sci. Rev.; 2002; 2, 11.

12. Tiwari, N.; Edla, D.R.; Dodia, S.; Bablani, A. Brain computer interface: A comprehensive survey. Biol. Inspired Cogn. Archit.; 2018; 26, pp. 118-129. [DOI: https://dx.doi.org/10.1016/j.bica.2018.10.005]

13. Luján, M.Á.; Jimeno, M.V.; Sotos, J.M.; Ricarte, J.J.; Borja, A.L. A Survey on EEG Signal Processing Techniques and Machine Learning: Applications to the Neurofeedback of Autobiographical Memory Deficits in Schizophrenia. Electronics; 2021; 10, 3037. [DOI: https://dx.doi.org/10.3390/electronics10233037]

14. Shoeibi, A.; Sadeghi, D.; Moridian, P.; Ghassemi, N.; Heras, J.; Alizadehsani, R.; Khadem, A.; Kong, Y.; Nahavandi, S.; Zhang, Y.D. et al. Automatic Diagnosis of Schizophrenia in EEG Signals Using CNN-LSTM Models. Front. Neuroinform.; 2021; 15, 777977. [DOI: https://dx.doi.org/10.3389/fninf.2021.777977] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34899226]

15. Vrbancic, G.; Podgorelec, V. Automatic Classification of Motor Impairment Neural Disorders from EEG Signals Using Deep Convolutional Neural Networks. Electron. Electr. Eng.; 2018; 24, pp. 3-7. [DOI: https://dx.doi.org/10.5755/j01.eie.24.4.21469]

16. Abdulkader, S.N.; Atia, A.; Mostafa, M.S.M. Brain computer interfacing: Applications and challenges. Egypt. Inform. J.; 2015; 16, pp. 213-230. [DOI: https://dx.doi.org/10.1016/j.eij.2015.06.002]

17. Neuper, C.; Müller, G.; Kübler, A.; Birbaumer, N.; Pfurtscheller, G. Clinical application of an EEG-based brain–computer interface: A case study in a patient with severe motor impairment. Clin. Neurophysiol.; 2003; 114, 399. [DOI: https://dx.doi.org/10.1016/S1388-2457(02)00387-5]

18. Bartur, G.; Pratt, H.; Soroker, N. Changes in mu and beta amplitude of the EEG during upper limb movement correlate with motor impairment and structural damage in subacute stroke. Clin. Neurophysiol.; 2019; 130, pp. 1644-16451. [DOI: https://dx.doi.org/10.1016/j.clinph.2019.06.008]

19. Sergeev, K.; Runnova, A.; Zhuravlev, M.; Kolokolov, O.; Akimova, N.; Kiselev, A.; Titova, A.; Slepnev, A.; Semenova, N.; Penzel, T. Wavelet skeletons in sleep EEG-monitoring as biomarkers of early diagnostics of mild cognitive impairment. Chaos; 2021; 31, 073110. [DOI: https://dx.doi.org/10.1063/5.0055441]

20. Samuel, O.W.; Xiangxin, L.; Yanjuan, G.; Pang, F.; Shixiong, C.; Guanglin, L. Motor imagery classification of upper limb movements based on spectral domain features of EEG patterns. Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC); Jeju, Korea, 11–15 July 2017.

21. Wang, H.; Song, Q.; Ma, T.; Cao, H.; Sun, Y. Study on Brain-Computer Interface Based on Mental Tasks. Proceedings of the The 5th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent Systems; Shenyang, China, 8–12 June 2015; pp. 841-845. [DOI: https://dx.doi.org/10.1109/CYBER.2015.7288053]

22. Zander, T.O.; Kothe, C. Towards passive brain–computer interfaces: Applying brain–computer interface technology to human–machine systems in general. J. Neural Eng.; 2011; 8, 025005. [DOI: https://dx.doi.org/10.1088/1741-2560/8/2/025005]

23. Aggarwal, S.; Chugh, N. Signal processing techniques for motor imagery brain computer interface: A review. Array; 2019; 1–2, 100003. [DOI: https://dx.doi.org/10.1016/j.array.2019.100003]

24. Mudgal, S.K.; Sharma, S.K.; Chaturvedi, J.; Sharma, A. Brain computer interface advancement in neurosciences: Applications and issues. Interdiscip. Neurosurg.; 2020; 20, 100694. [DOI: https://dx.doi.org/10.1016/j.inat.2020.100694]

25. Brunner, C.; Birbaumer, N.; Blankertz, B.; Guger, C.; Kübler, A.; Mattia, D.; Millán, J.d.R.; Miralles, F.; Nijholt, A.; Opisso, E. et al. BNCI Horizon 2020: Towards a roadmap for the BCI community. Brain-Comput. Interfaces; 2015; 2, pp. 1-10. [DOI: https://dx.doi.org/10.1080/2326263X.2015.1008956]

26. Jurcak, V.; Tsuzuki, D.; Dan, I. 10/20, 10/10, and 10/5 systems revisited: Their validity as relative head-surface-based positioning systems. NeuroImage; 2007; 34, pp. 1600-1611. [DOI: https://dx.doi.org/10.1016/j.neuroimage.2006.09.024]

27. Peng, H.; Hu, B.; Qi, Y.; Zhao, Q.; Ratcliffe, M. An improved EEG de-noising approach in electroencephalogram (EEG) for home care. Proceedings of the 2011 5th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops; Dublin, Ireland, 23–26 May 2011; pp. 469-474. [DOI: https://dx.doi.org/10.4108/icst.pervasivehealth.2011.246021]

28. Khalid, S.; Khalil, T.; Nasreen, S. A survey of feature selection and feature extraction techniques in machine learning. Proceedings of the 2014 Science and Information Conference; Karlovy Vary, Czech Republic, 27–28 December 2014; pp. 372-378. [DOI: https://dx.doi.org/10.1109/SAI.2014.6918213]

29. Saeidi, M.; Karwowski, W.; Farahani, F.V.; Fiok, K.; Taiar, R.; Hancock, P.A.; Al-Juaid, A. Neural Decoding of EEG Signals with Machine Learning: A Systematic Review. Brain Sci.; 2021; 11, 1525. [DOI: https://dx.doi.org/10.3390/brainsci11111525] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34827524]

30. Stancin, I.; Cifrek, M.; Jovic, A. A Review of EEG Signal Features and Their Application in Driver Drowsiness Detection Systems. Sensors; 2021; 21, 3786. [DOI: https://dx.doi.org/10.3390/s21113786] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34070732]

31. Subasi, A.; Ismail Gursoy, M. EEG signal classification using PCA, ICA, LDA and support vector machines. Expert Syst. Appl.; 2010; 37, pp. 8659-8666. [DOI: https://dx.doi.org/10.1016/j.eswa.2010.06.065]

32. Yazdani, A.; Ebrahimi, T.; Hoffmann, U. Classification of EEG signals using Dempster Shafer theory and a k-nearest neighbor classifier. Proceedings of the 2009 4th International IEEE/EMBS Conference on Neural Engineering; Antalya, Turkey, 29 April–2 May 2009; pp. 327-330. [DOI: https://dx.doi.org/10.1109/NER.2009.5109299]

33. Edla, D.R.; Mangalorekar, K.; Dhavalikar, G.; Dodia, S. Classification of EEG data for human mental state analysis using Random Forest Classifier. Procedia Comput. Sci.; 2018; 132, pp. 1523-1532. [DOI: https://dx.doi.org/10.1016/j.procs.2018.05.116]

34. Saragih, A.S.; Pamungkas, A.; Zain, B.Y.; Ahmed, W. Electroencephalogram (EEG) Signal Classification Using Artificial Neural Network to Control Electric Artificial Hand Movement. IOP Conf. Ser. Mater. Sci. Eng.; 2020; 938, 012005. [DOI: https://dx.doi.org/10.1088/1757-899X/938/1/012005]

35. Han, Y.; Ma, Y.; Zhu, L.; Zhang, Y.; Li, L.; Zheng, W.; Guo, J.; Che, Y. Study on Mind Controlled Robotic Arms by Collecting and Analyzing Brain Alpha Waves; Atlantis Press: Amsterdam, The Netherlands, 2018; pp. 145-148. [DOI: https://dx.doi.org/10.2991/ammsa-18.2018.30]

36. Casey, A.; Azhar, H.; Grzes, M.; Sakel, M. BCI controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients. Disabil. Rehabil. Assist. Technol.; 2021; 16, pp. 525-537. [DOI: https://dx.doi.org/10.1080/17483107.2019.1683239]

37. Van Erp, J.; Lotte, F.; Tangermann, M. Brain-Computer Interfaces: Beyond Medical Applications. Computer; 2012; 45, pp. 26-34. [DOI: https://dx.doi.org/10.1109/MC.2012.107]

38. Navarro-Espinoza, A.; López-Bonilla, O.R.; García-Guerrero, E.E.; Tlelo-Cuautle, E.; López-Mancilla, D.; Hernández-Mejía, C.; Inzunza-González, E. Traffic Flow Prediction for Smart Traffic Lights Using Machine Learning Algorithms. Technologies; 2022; 10, 5. [DOI: https://dx.doi.org/10.3390/technologies10010005]

39. Cerrada, M.; Trujillo, L.; Hernández, D.E.; Correa Zevallos, H.A.; Macancela, J.C.; Cabrera, D.; Vinicio Sánchez, R. AutoML for Feature Selection and Model Tuning Applied to Fault Severity Diagnosis in Spur Gearboxes. Math. Comput. Appl.; 2022; 27, 6. [DOI: https://dx.doi.org/10.3390/mca27010006]

40. Enríquez Zárate, J.; Gómez López, M.d.l.A.; Carmona Troyo, J.A.; Trujillo, L. Analysis and Detection of Erosion in Wind Turbine Blades. Math. Comput. Appl.; 2022; 27, 5. [DOI: https://dx.doi.org/10.3390/mca27010005]

41. Janiesch, C.; Zschech, P.; Heinrich, K. Machine learning and deep learning. Electron. Mark.; 2021; 31, pp. 685-695. [DOI: https://dx.doi.org/10.1007/s12525-021-00475-2]

42. Fong-Mata, M.B.; García-Guerrero, E.E.; Mejía-Medina, D.A.; López-Bonilla, O.R.; Villarreal-Gómez, L.J.; Zamora-Arellano, F.; López-Mancilla, D.; Inzunza-González, E. An Artificial Neural Network Approach and a Data Augmentation Algorithm to Systematize the Diagnosis of Deep-Vein Thrombosis by Using Wells’ Criteria. Electronics; 2020; 9, 1810. [DOI: https://dx.doi.org/10.3390/electronics9111810]

43. Cho, J.H.; Jeong, J.H.; Shim, K.H.; Kim, D.J.; Lee, S.W. Classification of Hand Motions within EEG Signals for Non-Invasive BCI-Based Robot Hand Control. Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC); Miyazaki, Japan, 7–10 October 2018; pp. 515-518. [DOI: https://dx.doi.org/10.1109/SMC.2018.00097]

44. Roy, G.; Bhoi, A.K.; Bhaumik, S. A Comparative Approach for MI-Based EEG Signals Classification Using Energy, Power and Entropy. IRBM; 2021; [DOI: https://dx.doi.org/10.1016/j.irbm.2021.02.008]

45. You, Y.; Chen, W.; Zhang, T. Motor imagery EEG classification based on flexible analytic wavelet transform. Biomed. Signal Process. Control; 2020; 62, 102069. [DOI: https://dx.doi.org/10.1016/j.bspc.2020.102069]

46. Faiz, M.Z.A.; Al-Hamadani, A.A. Online Brain Computer Interface Based Five Classes EEG To Control Humanoid Robotic Hand. Proceedings of the 2019 42nd International Conference on Telecommunications and Signal Processing (TSP); Budapest, Hungary, 1–3 July 2019; pp. 406-410. [DOI: https://dx.doi.org/10.1109/TSP.2019.8769072]

47. LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature; 2015; 521, pp. 436-444. [DOI: https://dx.doi.org/10.1038/nature14539] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26017442]

48. Zhang, Y.; Ling, C. A strategy to apply machine learning to small datasets in materials science. NPJ Comput. Mater.; 2018; 4, 25. [DOI: https://dx.doi.org/10.1038/s41524-018-0081-z]

49. Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell.; 2019; 1, pp. 206-215. [DOI: https://dx.doi.org/10.1038/s42256-019-0048-x]

50. Alomari, M.H.; Samaha, A.; AlKamha, K. Automated Classification of L/R Hand Movement EEG Signals using Advanced Feature Extraction and Machine Learning. arXiv; 2013; [DOI: https://dx.doi.org/10.14569/IJACSA.2013.040628] arXiv: 1312.2877

51. Pinheiro, O.; Alves, L.; Souza, J. EEG Signals Classification: Motor Imagery for Driving an Intelligent Wheelchair. IEEE Lat. Am. Trans.; 2018; 16, pp. 254-259. [DOI: https://dx.doi.org/10.1109/TLA.2018.8291481]

52. Bousseta, R.; El Ouakouak, I.; Gharbi, M.; Regragui, F. EEG Based Brain Computer Interface for Controlling a Robot Arm Movement Through Thought. IRBM; 2018; 39, pp. 129-135. [DOI: https://dx.doi.org/10.1016/j.irbm.2018.02.001]

53. Tang, Z.; Sun, S.; Zhang, S.; Chen, Y.; Li, C.; Chen, S. A Brain-Machine Interface Based on ERD/ERS for an Upper-Limb Exoskeleton Control. Sensors; 2016; 16, 2050. [DOI: https://dx.doi.org/10.3390/s16122050] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27918413]

54. Kant, P.; Laskar, S.H.; Hazarika, J.; Mahamune, R. CWT Based Transfer Learning for Motor Imagery Classification for Brain computer Interfaces. J. Neurosci. Methods; 2020; 345, 108886. [DOI: https://dx.doi.org/10.1016/j.jneumeth.2020.108886] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32730917]

55. Kaur, D.; Uslu, S.; Rittichier, K.J.; Durresi, A. Trustworthy Artificial Intelligence: A Review. ACM Comput. Surv.; 2022; 55, pp. 1-38. [DOI: https://dx.doi.org/10.1145/3491209]

56. Theissler, A.; Thomas, M.; Burch, M.; Gerschner, F. ConfusionVis: Comparative evaluation and selection of multi-class classifiers based on confusion matrices. Knowl.-Based Syst.; 2022; 247, 108651. [DOI: https://dx.doi.org/10.1016/j.knosys.2022.108651]

57. Sabharwal, R.; Miah, S.J. An intelligent literature review: Adopting inductive approach to define machine learning applications in the clinical domain. J. Big Data; 2022; 9, 53. [DOI: https://dx.doi.org/10.1186/s40537-022-00605-3]

58. Haque, R.; Islam, N.; Islam, M.; Ahsan, M.M. A Comparative Analysis on Suicidal Ideation Detection Using NLP, Machine, and Deep Learning. Technologies; 2022; 10, 57. [DOI: https://dx.doi.org/10.3390/technologies10030057]

59. Contreras-Luján, E.E.; García-Guerrero, E.E.; López-Bonilla, O.R.; Tlelo-Cuautle, E.; López-Mancilla, D.; Inzunza-González, E. Evaluation of Machine Learning Algorithms for Early Diagnosis of Deep Venous Thrombosis. Math. Comput. Appl.; 2022; 27, 24. [DOI: https://dx.doi.org/10.3390/mca27020024]

60. Aboneh, T.; Rorissa, A.; Srinivasagan, R. Stacking-Based Ensemble Learning Method for Multi-Spectral Image Classification. Technologies; 2022; 10, 17. [DOI: https://dx.doi.org/10.3390/technologies10010017]

61. Bi, L.; Fan, X.; Liu, Y. EEG-Based Brain-Controlled Mobile Robots: A Survey. IEEE Trans. Hum.-Mach. Syst.; 2013; 43, pp. 161-176. [DOI: https://dx.doi.org/10.1109/TSMCC.2012.2219046]

62. Abbasi, B.; Goldenholz, D.M. Machine learning applications in epilepsy. Epilepsia; 2019; 60, pp. 2037-2047. [DOI: https://dx.doi.org/10.1111/epi.16333] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31478577]

63. Majidov, I.; Whangbo, T. Efficient Classification of Motor Imagery Electroencephalography Signals Using Deep Learning Methods. Sensors; 2019; 19, 1736. [DOI: https://dx.doi.org/10.3390/s19071736] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30978978]

64. Craik, A.; He, Y.; Contreras-Vidal, J.L. Deep learning for electroencephalogram (EEG) classification tasks: A review. J. Neural Eng.; 2019; 16, 031001. [DOI: https://dx.doi.org/10.1088/1741-2552/ab0ab5] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30808014]

65. Padfield, N.; Zabalza, J.; Zhao, H.; Masero, V.; Ren, J. EEG-based brain-computer interfaces using motor-imagery: Techniques and challenges. Sensors; 2019; 19, 1423. [DOI: https://dx.doi.org/10.3390/s19061423] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30909489]

66. Lawhern, V.J.; Solon, A.J.; Waytowich, N.R.; Gordon, S.M.; Hung, C.P.; Lance, B.J. EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces. J. Neural Eng.; 2018; 15, 056013. [DOI: https://dx.doi.org/10.1088/1741-2552/aace8c]

67. Lotte, F.; Bougrain, L.; Cichocki, A.; Clerc, M.; Congedo, M.; Rakotomamonjy, A.; Yger, F. A review of classification algorithms for EEG-based brain-computer interfaces: A 10 year update. J. Neural Eng.; 2018; 15, 031005. [DOI: https://dx.doi.org/10.1088/1741-2552/aab2f2]

68. Mridha, M.F.; Das, S.C.; Kabir, M.M.; Lima, A.A.; Islam, M.R.; Watanobe, Y. Brain-Computer Interface: Advancement and Challenges. Sensors; 2021; 21, 5746. [DOI: https://dx.doi.org/10.3390/s21175746]

69. Samuel, O.W.; Geng, Y.; Li, X.; Li, G. Towards Efficient Decoding of Multiple Classes of Motor Imagery Limb Movements Based on EEG Spectral and Time Domain Descriptors. J. Med. Syst.; 2017; 41, 194. [DOI: https://dx.doi.org/10.1007/s10916-017-0843-z]

70. Goldberger, A.L.; Amaral, L.A.N.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.G.; Mietus, J.E.; Moody, G.B.; Peng, C.K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet. Circulation; 2000; 101, pp. e215-e220. [DOI: https://dx.doi.org/10.1161/01.CIR.101.23.e215]

71. Hosseini, M.P.; Hosseini, A.; Ahi, K. A Review on Machine Learning for EEG Signal Processing in Bioengineering. IEEE Rev. Biomed. Eng.; 2021; 14, pp. 204-218. [DOI: https://dx.doi.org/10.1109/RBME.2020.2969915] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32011262]

72. Schalk, G.; McFarland, D.J.; Hinterberger, T.; Birbaumer, N.; Wolpaw, J.R. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng.; 2004; 51, pp. 1034-1043. [DOI: https://dx.doi.org/10.1109/TBME.2004.827072] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/15188875]

73. Kemp, B.; Olivan, J. European data format ‘plus’ (EDF+), an EDF alike standard format for the exchange of physiological data. Clin. Neurophysiol.; 2003; 114, pp. 1755-1761. [DOI: https://dx.doi.org/10.1016/S1388-2457(03)00123-8]

74. Neuper, C.; Pfurtscheller, G. Evidence for distinct beta resonance frequencies in human EEG related to specific sensorimotor cortical areas. Clin. Neurophysiol.; 2001; 112, pp. 2084-2097. [DOI: https://dx.doi.org/10.1016/S1388-2457(01)00661-7]

75. Deecke, L.; Weinberg, H.; Brickett, P. Magnetic fields of the human brain accompanying voluntary movement: Bereitschaftsmagnetfeld. Exp. Brain Res.; 1982; 48, pp. 144-148. [DOI: https://dx.doi.org/10.1007/BF00239582]

76. Hashimoto, Y.; Ushiba, J. EEG-based classification of imaginary left and right foot movements using beta rebound. Clin. Neurophysiol.; 2013; 124, pp. 2153-2160. [DOI: https://dx.doi.org/10.1016/j.clinph.2013.05.006]

77. Sleight, J.; Pillai, P.J.; Mohan, S. Classification of Executed and Imagined Motor Movement EEG Signals. Comput. Sci.; 2009.

78. Lee, K.; Liu, D.; Perroud, L.; Chavarriaga, R.; Millán, J.d.R. A brain-controlled exoskeleton with cascaded event-related desynchronization classifiers. Spec. Issue New Res. Front. Intell. Auton. Syst.; 2017; 90, pp. 15-23. [DOI: https://dx.doi.org/10.1016/j.robot.2016.10.005]

79. Phinyomark, A.; Thongpanja, S.; Hu, H.; Phukpattaranont, P.; Limsakul, C. The Usefulness of Mean and Median Frequencies in Electromyography Analysis. Computational Intelligence in Electromyography Analysis: A Perspective on Current Applications and Future Challenges; InTech: London, UK, 2012; [DOI: https://dx.doi.org/10.5772/50639]

80. Zwillinger, D.; Kokoska, S. CRC Standard Probability and Statistics Tables and Formulae; CRC Press: Boca Raton, FL, USA, 1999; Google-Books-ID: TB3RVEZ0UIMC

81. Jatupaiboon, N.; Pan-ngum, S.; Israsena, P. Emotion classification using minimal EEG channels and frequency bands. Proceedings of the 2013 10th International Joint Conference on Computer Science and Software Engineering (JCSSE); Lampang, Thailand, 2–30 June 2013; pp. 21-24. [DOI: https://dx.doi.org/10.1109/JCSSE.2013.6567313]

82. Al-Ani, A.; Al-Sukker, A. Effect of Feature and Channel Selection on EEG Classification. Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society; New York, NY, USA, 30 August–3 September 2006; pp. 2171-2174. [DOI: https://dx.doi.org/10.1109/IEMBS.2006.259833]

83. Sadrawi, M.; Sun, W.Z.; Ma, M.M.; Yeh, Y.T.; Abbod, M.; Shieh, J.S. Ensemble Genetic Fuzzy Neuro Model Applied for the Emergency Medical Service via Unbalanced Data Evaluation. Symmetry; 2018; 10, 71. [DOI: https://dx.doi.org/10.3390/sym10030071]

84. Guang-Hui, F.; Feng, X.; Bing-Yang, Z.; Lun-Zhao, Y. Stable variable selection of class-imbalanced data with precision-recall criterion. Chemom. Intell. Lab. Syst.; 2017; 171, pp. 241-250. [DOI: https://dx.doi.org/10.1016/j.chemolab.2017.10.015]

85. Castro, C.; Vargas-Viveros, E.; Sánchez, A.; Gutiérrez-López, E.; Flores, D. Parkinson’s Disease Classification Using Artificial Neural Networks. Proceedings of the VIII Latin American Conference on Biomedical Engineering and XLII National Conference on Biomedical Engineering, CLAIB 2019; Cancun, Mexico, 2–5 October 2019; Volume 75, [DOI: https://dx.doi.org/10.1007/978-3-030-30648-9_137]

86. Metrics for Multi-Class Classification: An Overview. arXiv; 2020; arXiv: 2008.05756

87. Vieira, S.M.; Kaymak, U.; Sousa, J.M.C. Cohen’s kappa coefficient as a performance measure for feature selection. Proceedings of the International Conference on Fuzzy Systems; Yantai, China, 10–12 August 2010; pp. 1-8. [DOI: https://dx.doi.org/10.1109/FUZZY.2010.5584447]

88. Matthews, B.W. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim. Biophys. Acta (BBA)-Protein Struct.; 1975; 405, pp. 442-451. [DOI: https://dx.doi.org/10.1016/0005-2795(75)90109-9]

89. Aguirre-Castro, O.; García-Guerrero, E.; López-Bonilla, O.; Tlelo-Cuautle, E.; López-Mancilla, D.; Cárdenas-Valdez, J.; Olguín-Tiznado, J.; Inzunza-González, E. Evaluation of underwater image enhancement algorithms based on Retinex and its implementation on embedded systems. Neurocomputing; 2022; 494, pp. 148-159. [DOI: https://dx.doi.org/10.1016/j.neucom.2022.04.074]