Motion Cognitive Decoding of Cross-Subject Motor Imagery Guided on Different Visual Stimulus Materials

Abstract

Background: Motor imagery (MI) plays an important role in brain-computer interfaces, especially in evoking event-related desynchronization and synchronization (ERD/S) rhythms in electroencephalogram (EEG) signals. However, the procedure for performing a MI task for a single subject is subjective, making it difficult to determine the actual situation of an individual's MI task and resulting in significant individual EEG response variations during motion cognitive decoding. Methods: To explore this issue, we designed three visual stimuli (arrow, human, and robot), each of which was used to present three MI tasks (left arm, right arm, and feet), and evaluated differences in brain response in terms of ERD/S rhythms. To compare subject-specific variations of different visual stimuli, a novel cross-subject MI-EEG classification method was proposed for the three visual stimuli. The proposed method employed a covariance matrix centroid alignment for preprocessing of EEG samples, followed by a model agnostic meta-learning method for cross-subject MI-EEG classification. Results and Conclusion: The experimental results showed that robot stimulus materials were better than arrow or human stimulus materials, with an optimal cross-subject motion cognitive decoding accuracy of 79.04%. Moreover, the proposed method produced robust classification of cross-subject MI-EEG signal decoding, showing superior results to conventional methods on collected EEG signals.

Headnote

Abstract

Background: Motor imagery (MI) plays an important role in brain-computer interfaces, especially in evoking event-related desynchronization and synchronization (ERD/S) rhythms in electroencephalogram (EEG) signals. However, the procedure for performing a MI task for a single subject is subjective, making it difficult to determine the actual situation of an individual's MI task and resulting in significant individual EEG response variations during motion cognitive decoding. Methods: To explore this issue, we designed three visual stimuli (arrow, human, and robot), each of which was used to present three MI tasks (left arm, right arm, and feet), and evaluated differences in brain response in terms of ERD/S rhythms. To compare subject-specific variations of different visual stimuli, a novel cross-subject MI-EEG classification method was proposed for the three visual stimuli. The proposed method employed a covariance matrix centroid alignment for preprocessing of EEG samples, followed by a model agnostic meta-learning method for cross-subject MI-EEG classification. Results and Conclusion: The experimental results showed that robot stimulus materials were better than arrow or human stimulus materials, with an optimal cross-subject motion cognitive decoding accuracy of 79.04%. Moreover, the proposed method produced robust classification of cross-subject MI-EEG signal decoding, showing superior results to conventional methods on collected EEG signals.

Keywords: motion cognitive decoding; guided visual stimulus; cross-subject MI-EEG decoding; event-related desynchronization/synchronization; brain-computer interface

(ProQuest: ... denotes formulae omited.)

1. Introduction

A brain-computer interface (BCI) system can sense brain signals and use them to control external devices directly by using neural activity in the brain instead of the muscular system [1-3]. An efficient and accessible BCI system would have wide-ranging applications in clinicallyassisted rehabilitation [4]. Recently, BCI systems that combine virtual reality and motor imagery (MI) have become the mainstay of auxiliary rehabilitation, with many researchers investigating Mi-based BCI (MI-BCI) for clinical exoskeleton-assisted rehabilitation [5-7]. Based on the theory of neural plasticity, the MI-BCI exoskeleton-assisted rehabilitation system is an effective approach for motorrelated complications resulting from stroke [8]. The MI-BCI-based exoskeleton was found to increase upper limb strength after regular training time, resulting in an increased Fugl-Meyer motor assessment score [9,10]. The MI-BCI system can be constructed by invasive and non-invasive means. Electroencephalogram (EEG) signals are the main input for non-invasive MI-BCI systems, and have the advantages of low-cost and high temporal resolution, with no requirement for craniotomy. MI-EEG signals encode human motor cognitive intention or process. They use signal processing and pattern recognition methods to decode the specific category of MI task from EEG signals, thereby ensuring the establishment of an exoskeleton rehabilitation system [11].

To decode the motion cognitive categories from MIEEG signals, an important research field is the design of the motion cognitive stimulus paradigm [12]. The neural mechanism of the human motor is complex, and it plays an important role in rehabilitation with BCI systems. The extraction of brain activity components from the neural mechanism can be translated to the decoding of motion cognitive intention and process from the MI-EEG signals [13,14]. An excellent stimulus paradigm encodes the detailed neural activities of motor intention or process into MI-EEG signals. During the decoding of MI-EEG signals, discriminative features can be obtained for different MI categories and are recognized with high average accuracy to ensure external device control [15]. At present, studies for the design of stimulus paradigms consist of three parts: visual stimulusguided motion imagery [16], MI [17], and motor execution (ME) [18]. Of these, MI and ME are the most commonly used to design BCI stimulus paradigms, and have achieved excellent decoding results in different rehabilitation scenarios.

The visual stimulus-guided motion imagery paradigms are different from the MI and ME paradigms in that they provide rich information about motion cognitive intentions and emotions, which influence the understanding of motion by the human brain. Neuro-imagc research of visual motion cognitive paradigms has shown that observing an individual's action activates a cortical network known as the mirror neuron system (MNS) [19]. Activation of the MNS was first discovered during the observation by a monkey of an individual's motion. A meta-analysis found that human MNS (hMNS) activity also included the monkeys' mirror cortical network [20]. A broad cortical network in the frontal and parietal lobes of the human brain contains the hMNS regions [21,22] and is functionally connected to key regions of the hMNS, referred to as the "extended hMNS" [23]. Generally, neural activities of MI paradigms increase the alpha and beta rhythms of hMNS in the frontal and parietal lobes of the brain from MI-EEG signals. This is called the event-related desynchronization/synchronization (ERD/S) phenomenon [24]. The ERD/S-based BCI system has shown potential for the assisted rehabilitation of motor disorders [25,26], and the patterns encoded by ERD/S phenomenon arc key elements of motion cognitive decoding [27]. To enhance such phenomenon, some researchers have used training of motion planning [28], while others provided visual guidance of motion cognitive intention [29].

Another research field in motion cognitive decoding is the recognition method for ERD/S phenomenon from MI-EEG signals. Due to the non-linear and non-stationary characteristics of MI-EEG signals, task-related ERD/S features are hidden in the noise when recoding MI-EEG signals. To analyze hidden features within MI-EEG signals, the analysis method consists of two parts. First, the discriminative features are extracted from EEG signals, such as filter bank common spatial patterns (FBCSP) [30]. Next, the extracted features are classified using machine learning methods, such as robust support matrix machine (RSMM) [31]. With the development of deep neural networks [3236], the convolutional neural networks (CNNs) represented by ShallowNet and EEGNet [37,38] have gradually become the end-to-end MI-EEG classification methods.

Recent studies on visual stimulus-guided motion cognitive intention have focused on hand movements, such as catching a ball with different hands [39], and grasping a cup with both hands [40]. Other researchers have concentrated on the visual guidance of various body movements, such as balanced standing with feet [41], walking posture [42], the exchange process of sitting down and standing up [43], and self-initiated and cued movements [44]. Two different conditions have been used for motor task initiation: self-paced and cued. Studies have also explored how the MI cue influences the ERD/S and movement-related cortical potentials. The results showed that more influence was linked to motor related cortical potential (MRCP), but no significant differences were recorded for ERD/S. The goal of the present study was to explore the influence of ERD/S caused by different stimulus materials. As far as we are aware, there has been less research on how visually-guided stimuli influence the motion cognitive process. Therefore, we explored the issue of how and when to perform MI tasks, since this is very important for the motion cognitive process [45,46]. We designed three objective visual stimuli to guide the process of MI tasks, and then explored differences in ERD/S phenomena on the same subjects. Specifically, we designed arrow, human, and robot visual stimuli to study the guidance of left arm, right arm, and feet MI tasks. To overcome differences in ERD/S responses between different subjects for the same stimuli, and to allow a fair comparison of different visual stimuli, we adopted a cross-subject MI-EEG signal decoding method. Specifically, the covariance matrix centroid alignment (CMCA) method [47] was applied to align filtered cross-subject MIEEG signals. The commonly used ShallowNet [37] and EEGNet [38] models were then used to identify discriminative features. To achieve high accuracy classification, the model-agnostic meta-learning (MAME) method [48] was used to perform cross-subject, motion-cognitive decoding of different objective visual stimulus materials.

With the loss of generality, the average cross-subject MI-EEG signal classification will objectively reflect the ERD/S responses of different visually-guided stimuli. The stimulus with the best accuracy will be selected as the MI-BCI guidance. The main objectives of this paper are therefore:

(1) To explore the neural mechanism of MI through the different guidance provided by visual stimuli. Three visual stimuli (arrow, human, and robot) were designed and presented to subjects, who were then asked to perform three MI tasks (left arm, right arm, and feet). At the same time, MI-EEG signals were recorded for the cross-subject classification analysis.

(2) To analyze the guidance effects of different visual stimuli during MI, the leave-one-rest cross-subject MI-EEG signal classification method was constructed with the three stimulus materials. The CMCA method was first used to align EEG samples, and the CNNs model was then used to extract features. Next, the MAME method was used for cross-subject classification. The proposed CMCA-MAML method showed better classification performance compared to conventional methods.

Section 2 of this paper describes the three kinds of visual stimuli and the EEG recording experiment. Section 3 describes the data pre-processing and the cross-subject classification method of MI-EEG signals for the three kinds of visual stimuli. Section 4 presents the experimental results and ablation studies, section 5 discusses the experimental findings and results, and finally the conclusions and future research directions are presented in section 6.

2. Materials and Experiments

2.1 Participants

A total of 14 subjects (8 females and 6 males) with an age range of 19-21 years (mean age = 20.3) were recruited for the EEG recording experiments. None had any history of mental illness, and all had normal (or correctedto-normal) vision. Prior to EEG signal recording, all subjects were requested to practice the MI process for several days. Before the beginning of each experiment, detailed information concerning the experimental study (including purpose, content, estimated time, etc.) was given to each subject. All subjects signed an informed consent form as per the Declaration of Helsinki. The experiment was approved by the institutional review board of Shaoxing University for the research purpose of studying motion cognitive intention and process.

2.2 Stimulus Materials

The neural mechanism of MI through different guidance by visual stimuli was studied using three different carriers to construct visual stimulus materials: plane graphics, the human body, and a biped humanoid robot. To evoke the ERD/S phenomena when subjects perform MI tasks under the guidance of carriers, three MI task schemes were designed for the three carriers: left arm, right arm, and feet. Fig. 1 illustrates the three MI tasks for the three different stimulus materials. The left and right arrows are used to instruct the left arm and right arm imagery of the subject him/herself, and the down arrow is used to instruct the feet imagery of the subject him/herself (Fig. la). The left arm, right arm, and feet MI arc instructed by the actual movements of the human body (Fig. lb) and humanoid robot (Fig. 1c). The above settings confirm the variations of actual actions, and can thus be used to evaluate the different neural mechanisms of MI tasks guided by different stimuli.

2.3 Experimental Paradigm

During MI tasks, the above three schemes of visual stimulus materials use the same experimental paradigm (Fig. 2).

There are a total of 210 trials for each scheme in this experimental paradigm, with each trial consisting of three steps: concentration, MI, and rest. In the concentration step, a specific symbol (e.g., "+") is displayed on the computer monitor for 1 second, with the aim of focusing the attention of subjects. In the MI step, an image of one of the directional elements (e.g., left arm, right arm, or feet) in a specific scheme (e.g., plane graphics, biped humanoid robot, or human body) is randomly displayed for 4 seconds. In the rest step, a blank screen is displayed on the computer monitor for 1.5 seconds. The psychological software tool "E-Prime 1.1" (Psychology Software Tools, Inc. Pittsburgh, PA, USA) was used to instantiate the experimental paradigm.

For the experimental paradigm based on a specific scheme of visual stimulus material, 70 trials were performed for each directional element (left arm, right arm, and feet), giving a total of 210 trials. After observing a specific directional element during an experiment (regardless of the scheme), the subject carried out MI guided by the directional element. Specifically, for the three directional elements of the visual stimulus materials, corresponding contents for the subjects' MI were: lift and straighten the left or right arm as 'left arm' and 'right arm' tasks, respectively, and squat down as a 'feet' task.

2.4 EEG Signal Recording

EEG signals from subjects were recorded using the mobile EEG device EMOTIV EPOC+ (Emotiv Systems, Inc., San Francisco, CA, USA), as shown in Fig. 3. The parameters for this device were set to a sampling resolution of 16 bits and a sampling frequency of 256 Hz.

During the experiments, subjects were told to move as little as possible, and to imagine body movements as many times as possible during the emergence of a visual stimulus. A dimly lit room with attenuated sound served as the experimental site, with subjects arranged to sit in front of a computer.

3. Cross-Subject MI-EEG Classification Methods

3.1 Pre-Processing of EEG Signals

After the recording of EEGs, MI-EEG signals from 14 subjects guided by three schemes of visual stimuli were collected. Results for two subjects were removed from the MI-EEG datasets due to excessive noise. EEG time-series collected from each subject were examined first, and artifacts exceeding 100 pV and below -100 mV were manually eliminated. These artifacts were caused by the subject's muscle movements, and were removed before preprocessing. Prior to analysis, pre-processing of MI-EEG signals from the remaining 12 subjects [49] was performed as follows:

(1) Signal filtering. A notch filter (48-52 Hz) was used to filter power frequency interference, and a band-pass filter (0.1-30 Hz) was used to retain the Mi-related alpha (4-12 Hz) and beta (12-25 Hz) rhythms.

(2) Signal Segmentation. Each trial in the MI experimental paradigm lasted 4000 ms. The sampling rate for the 'EMOTIV' device was set to 256 Hz, resulting in 1024 sampling points for each MI-EEG sample. After MI processing, 10% (102) of the sampling points of a trial were selected for calibration, and each MI-EEG sample was segmented to a length of [-102, 1024], in which the "zero point" was the starting time for the MI task.

(3) Baseline calibration. As per the segmentation principle of EEG experiments [50], baseline calibration was conducted on all trials to ensure the baseline of each MIEEG sample was zero at the start of each MI task. For baseline calibration, signals from the time range of [-102, 0] were used to calibrate the signals from the time range of [0, 1024]. The final number of sampling points for each MI-EEG sample was 1126.

After pre-processing, three EEG sample sets for the arrow, human, and robot visual stimuli were established for 12 subjects. Each MI-EEG sample can be defined as XeRc·t, where c=i4 is the number of channels, and r=ii26 is the number of sampling points. Each MI task (left arm/right arm/feet) has 70 samples for three schemes of visual stimuli (arrow, human, and robot). Three-class, crosssubject MI-EEG classification of the 210 samples was then constructed for arrow, human, and robot.

3.2 Covariance Matrix Centroid Alignment

To avoid different subjects having different ERD/S responses to the same stimulus, the method of covariance matrix centroid alignment (CMCA) [47] was used to align EEG samples from each subject. It was assumed the ith (z = 1,2, ...,N) MI-EEG sample from the j-th (j = 1,2,..., L) subject could be denoted as Xji ∈ Rc· T, noting that MI-EEG sample sets from the three visual stimuli are processed in the same way. The first step was to compute the mean of covariance matrix for each sample set:

*·· (1)

where Nt represents the number of trials for the /-th subject. Řj represents the mean of covariance matrix for the j-lh subject. The matrix of R^2 thus obtained was termed the covariance matrix centroid. The centroid matrix was used to align each EEG sample for the j-th subject:

*·· (2)

where Xji represents the z-th aligned MI-EEG trial for the /-lh subject. After processing by the CMC A method, the mean of the covariance matrix for the aligned MI-EEG trials of the j-th subject can be computed:

*·· (3)

From the above computation, the MI-EEG samples aligned by the Ä-1/2 centroid alignment have the same mean of covariance matrix as the identity matrix. Therefore, the same alignment process for the MI-EEG trials set was performed for each subject. The maximum mean discrepancy between MI-EEG trials set for different subjects was close to zero. The EEG trials set after covariance matrix centroid alignment can be used to construct a crosssubject classifier for the recognition of MI categories.

3.3 Feature Extractor by CNNs

MI-EEG samples from 12 subjects were aligned to a similar distribution and the process of cross-subject MI-EEG classification was then performed. Before classification, it is important to label MI-EEG samples and to extract discriminative features [51]. For the three visual stimuli (arrow, human, and robot), the pre-processed and aligned MI-EEG samples were divided into three categories (left arm, right arm, and feet), and the labels of left arm, right arm, and feet were set to 0, 1, and 2, respectively. In this study, the fine-tuned strategy was used to construct a subject-invariant classification model. Since the weights of the convolutional neural network (CNN) models can be fine-tuned during error-propagation, this model has excellent ability to extract discriminative and robust features [52]. For simplicity, the widely used ShallowNet [37] and EEGNet [38] models were employed for feature extraction. The ShallowNet and EEGNet architectures are illustrated in Fig. 4. The convolutional layers, pooling layer, and fully-connected layer are used to extract features. EEGNet extracts 200 dimension features and ShallowNet extracts 840 dimension features.

Large differences between channels occur for each time series in individual EEG trials. To reduce feature differences caused by the dimensions, a common strategy is to use a sliding window to divide the original EEG trial into multiple time segments [53]. In this way, the original temporal higher dimensions are divided into multiple and overlapping time segments of lower dimensions. A series of EEG segments are then combined and incorporated into the CNN model, thereby improving the recognition accuracy and robustness of MI. During pre-processing, a dimension of 750 sampling points was selected for classification, and an overlapped cropping strategy was used to obtain 7 segments with 400 sampling points [54]. After the feature extraction process, the SoftMax function was used for classi- fication, and true labels were used to compute the loss for back-propagation:

*·· (4)

where exp ^fe [X? j j was the conditional probability of the Z-th MI-EEG samples for they-th subject. This step was referred to as pre-training, and used only the EEG sampies and labels 1^X1 to extract the preliminary weights 0 of the feature extractor, such as EEGNet and ShallowNet models.

3.4 Meta-learning-Based Domain Adaptation

Although end-to-end convolutional neural networks (EEGNet and ShallowNet) can extract discriminative features from MI-EEG signals, such signals recorded from different subjects show considerable variation in MI patterns, duration, and power spectra. Therefore, the models perform with different accuracy levels for different subjects. A fine-tuning strategy is thus urgently needed to confirm the reliability of deep learning models. A study has shown how to adjust deep learning models to serve different subjects [55]. The present study employed the advanced meta-learning method called Model Agnostic Metalearning (MAME) [48] for cross-subject MI-EEG classification to train the models on existing multi-subjects and apply them to new subjects. Recently, the MAME method was applied to cross-subject MI-EEG classification scenarios to improve the generalization of deep learning models such as multi-domain model-agnostic meta-learning (MDMAML) [56] and meta UPdate strategy (MUPS) [57]. The current work employed EEGNet and ShallowNet for base learning, and adopted the MAME method to perform crosssubject meta-learning.

Similar to the original settings of the cross-subject MIEEG classification, we used the leave-one-rest strategy with the MAME method. EEG signals were recorded from 12 subjects for three visual stimuli (arrow, human, and robot), with one subject selected as the target subject for validation and the remaining 11 subjects used as the source domain for training. For three different MI stimulus materials, there were a total of 12 meta-training and validation tasks in the cross-subject MI-EEG classification. The average accuracy of the 12 tasks was recorded as the final classification result. To describe the process for the MAME method with "subject 1" as an example of the target domain, subjects l/=[s2,S3,.-%!2] are useð f°r pre-training, and "subject 1" for validation. MI-EEG signals from the source domain were randomly sampled to construct г meta-tasks·. Tk = {(xf, У1 ) ,, ,k = 1,2,... ,T, where each task contains s MI-EEG samples.

Formally, the MAME method endeavors to find the optimal parameter 0· from the deep learning model that performs well on all meta-tasks {п, ..., тт^ . For the pre-trained stage of EEGNet or ShallowNet, the initial parameter 0 from the source domain was obtained by model fe in Eqn. 4, which can be called meta-weights. According to the training process of the MAME method, in each meta-training iteration, the meta-weights are dispensed to each meta-task: 0^ g- 0. Based on the setting for the MAME method, two MI-EEG samples sets are sampled from each meta-task Tk for training (т^тагп) and validation Çr^alY Thus, for a binary MI classification, a total of ^=2way and p-shot epochs are selected for each т1гагп (referred to as the support set), and w=2-way and Ç-shot epochs are selected to T^al (referred to as the query set).

To adapt the meta-w eights 6 k to all subjects from the source domain, a forward pass of model training on the support set Tf~am is first executed to update 6 k·.

*·· (5)

where α is the base learning rate and refers to the temporarily updated model parameters for the Tk meta-task. The common stochastic gradient descent (SGD) is adopted for base learner. The update is then evaluated to achieve CTk on the query set r£aZ. For the MI-EEG classification task, the cross-entropy loss is computed in CTk :

*·· (6)

After all of the т sampled meta-tasks are trained and evaluated, the summation of all cross-entropy losses are computed as the meta-loss:

*·· (7)

According to the MAME method, after meta-training iteration is completed on all meta-tasks, the parameter 6 of the deep learning model can be fine-tuned by the summation loss Cmeta- The same SGD method can be adopted for the fine-tuned stage:

*·· (8)

where ß is the base learning rate to update the meta-weights. Note that the actual fine-tune is updated on the deep learning model parameter instead of the temporary random sampled model parameter 6k- In other words, after the finetune process, the distribution discrepancy and domain shift among all source domains can be minimized, and subjectinvariant knowledge can be learned by the MAME method to adapt to the new-coming subject. This framework has two steps: meta-train and meta-tcst. For each domain-generalized MI-EEG decoding task based on the meta-learning theory, the training set is sampled to shot-set and query-set. The "shot" set is used to pre-train (meta-train) the network, while the "query" set is used to fine-tune (meta-tcst) the network. Hence, the framework contains the generalization ability, which can be used to generalize to the real-world test set. In the present work, left arm, right arm, and foot MI-EEG samples were collected from 12 subjects, and each Mi-class contained almost 210 samples. Therefore, for each domain-generalized MI-EEG decoding task, 11 x 630 = 6930 samples were used as source domain to train the networks, while the remaining 630 samples were used as target domain to test the networks. The current work focused on offline applications, and the 630 unlabeled MI-EEG trials from the target domain were classified by the learned meta-learning model. The proposed CMCA+MAML method is described with the pseudo-code in Algorithm 1.

4. Results

4.1 Visualization of EEG Signals during MI

Analysis of the motion cognitive process for three visual stimuli (arrow, human, and robot) was separated into two parts. First, the MI-EEG signals recorded from different stimuli were visualized. The MI-EEG signals of "Subject 7" showed excellent classification performance for all three stimuli and were thus selected for visualization. The ERD/S curves were visualized to compare the three visual stimuli. Afterwards, the spatial, temporal, and spectral features of EEG signals were commonly used for analysis and visualization, since MI-EEG signals represent the multivariate time series.

Subjects exposed to Arrow, Human, and Robot stimuli had their preprocessed EEG data utilized for computing ERD/S curves. For the EEG recording experiment, data were collected using an emotive device with 14 channels. Channels FC5 and FC6 in this device represent the sensorymotor areas of the left and right hemispheres of the brain, respectively. Therefore, we selected to display the power (voltage square) ratio of MI tasks under different stimuli using the FC5 and FC6 electrodes. For the same MI task, the FC5 and FC6 electrodes were separately averaged across all trials for each subject, and the ERD/S curves were computed using a sliding time window. The specific calculation process is as follows: for the z-th sample's time series xc¿ on electrode c, the voltage square is first computed as:

*·· (9)

where n represents the number of sampling points. Then, a smooth method is used for the power value of each channel Yc,n *

*·· (10)

where l = 200 is the smooth time window. Last, to give the ERD/S curve, we compute the baseline of the first 125 time samples of Yc, termed as Ymean, which is the zero point in Fig. 5. The ERD/s curves is given as:

*·· (11)

Finally, the proportion (%) of ERD/S at each sampling point during the MI process was computed to validate the impact of different stimuli on ERD/S during MI tasks. In Fig. 5, the average ERD/S curves of all 12 subjects under different stimulus materials were drawn, and the standard deviation across all subjects was also drawn to show the confidence intervals. The x-axis represents the number of sampling points during the MI process, while they-axis indicates the percentage of ERD/S (%).

For MI tasks guided by arrow, human, and robot stimuli, it is clear that ERD/S ratios on channel FC5 for left arm MI tasks are higher than those for right arm MI tasks (Fig. 5). Conversely, on channel FC6, the ERD/S ratios for right arm MI tasks are higher than those for left arm MI tasks. These outcomes align with the fundamental conclusion that ERD/S phenomena typically exhibit decreased energy in contralateral brain areas and increased energy in ipsilateral brain areas. Furthermore, regardless of the channel (FC5 or FC6), differences in ERD/S for left and right-hand MI tasks are most pronounced for robot stimuli. This was followed by arrow stimuli, with human stimuli yielding the least effective results. Based on variations in the ERD/S curves, we infer that robot stimuli guidance for MI tasks have greater discriminative power and better performance.

In order to show the spatial features of MI-EEG signals recorded from three different stimuli, common spatial pattern (CSP) features [30] are used to visualize binary MI tasks. The CSP feature distributions are compared with each binary category in Fig. 6. Arrow and robot stimuli are clearly better than human stimuli in terms of their ability to discriminate pairwise comparisons for the three MI tasks (left arm, right arm, and feet). However, CSP features are fixed when given a certain number of MI-EEG samples, and this cannot be fine-tuned during cross-subject classification. To solve this problem, the CNN model (Shallo wNet and EEGNet) was used to fully extract discriminative features and to dynamically fine-tune during crosssubject classification.

To visualize EEG signal variations during MI tasks, brain electrical activity mapping (BEAM) [58] was employed to show the motion cognitive process of each MI task for the three stimuli. BEAMs of 500 ms, 1000 ms, 1500 ms, 2000 ms, 2500 ms, and 2950 ms after the onset of arrow, human, and robot stimuli are shown in Figs. 7,8,9, respectively.

The motion cognitive process of the left arm guided by arrow materials suggests an obvious ERD phenomenon in the right parietal brain region (Fig. 7a). The BEAMs for the right arm also indicate an obvious ERD phenomenon in the right frontal and occipital brain regions (Fig. 7b). For the feet, the ERS phenomenon appears in the frontal brain region, while the ERD phenomenon appears in the central brain region, with no obvious difference in lateralization.

For the motion cognitive process of the left arm guided by human materials, the BEAMs suggest an obvious ERD phenomenon in the right frontal brain region (Fig. 8a). For the right arm, they show that an obvious ERS phenomenon arises in the left parietal and occipital brain regions, and another ERD phenomenon arises in the left frontal brain region (Fig. 8b). For the feet, they show an obvious ERS phenomenon in the left central brain region, and an obvious ERD phenomenon in the right frontal brain region, with a certain lateralization effect (Fig. 8c).

For the motion cognitive process of the left arm guided by robot materials, an ERD phenomenon was observed in the right frontal brain region before 1500 ms, while an ERS phenomenon was observed in the right frontal brain region between 2000 ms and 2500 ms (Fig. 9a). For the right arm, an ERD/S phenomenon was observed in the left parietal and occipital brain regions, and an ERD phenomenon in the left frontal brain region (Fig. 9b). Finally, the motion cognitive process of feet showed that an ERS phenomenon was evoked in the left central brain region, and an obvious ERD phenomenon was evoked in the right frontal brain region, with a certain lateralization effect (Fig. 9c).

In addition to the spatial visualization of MI-EEG signals, the statistical time domain features and frequency domain features were also compared. This allowed validation of the effects of MI tasks influenced by different visual stimuli from additional perspectives. Two motor cortexrelated electrodes (FCS and FC6) were selected to illustrate the statistical features. The area under the EEG curve and the EEG amplitude power were computed as the time domain features (Fig. 10).

For the FC5 electrode, the average area under the EEG curve was greatest for MI tasks guided by human visual stimuli (Fig. 10a), i.e., human materials produced the greatest changes in EEG curves from the left motor cortex. For the FC6 electrode, the highest average area under the EEG curve was observed with robot visual stimuli, which produced the most variation in the right hemisphere (Fig. 10b). For the FCS electrode, MI tasks guided by human visual stimuli had the highest EEG amplitude power (Fig. 10c), whereas MI tasks guided by robot visual stimuli had the highest EEG amplitude power with the FC6 electrode (Fig. 10d). Therefore, human materials produce the most amplitude increase in the left hemisphere, whereas robot materials act on the right hemisphere. Human and robot materials showed greater variation in these two domain features than arrow materials for both motion cognitive processes, thus improving the accuracy of MI task decoding.

Research has shown that ERD/S phenomena arise from power variations in alpha and beta rhythms. The average power variation of MI tasks guided by different visual stimuli on FCS and FC6 is shown in Fig. 11 for the two rhythms.

For both alpha and beta rhythms, human materials have the advantage of the left hemisphere and achieved higher average power for left arm and feet MI tasks with the FCS electrode (Fig. 1 la,c). For both rhythms, robot materials have the advantage of the right hemisphere and achieved higher average power for left arm, right arm, and feet MI tasks with the FC6 electrode (Fig. llb,d). Irrespective of the left or right motor cortex, the results for the two frequency domain features show that human and robot materials caused greater variations in the frequency band related to the motor cortex than arrow materials, thus improving the classification accuracy of MI-EEG signals.

According to the above temporal-spatial-spectral analysis of MI-EEG signals guided by the three stimuli, human and robot materials were found to provide stronger discriminative features compared with autonomous imagination of arrow visual materials. Therefore, we hypothesize that refined human and robot materials will improve the motion cognitive process of MI tasks. Next, the CMCA+MAML method was employed to conduct cross-subject MI-EEG decoding experiments. Through the average accuracy, this further verified the hypothesis by classifying ERD/S phenomena caused by different visual stimuli.

4.2 Classification Results from the Meta-Learning Framework

The second part in comparing the motion cognitive process guided by three visual stimuli was the use of cross-subject MI-EEG signal classification. During preprocessing, the three motion cognitive processes (left arm, right arm, and feet) were guided by three visual stimuli (arrow, human, and robot), and 70 EEG samples were collected for each subject. To perform cross-subject MI-EEG classification, the three MI tasks were divided into three binary classification tasks: left arm vs. right arm, left arm vs. feet, and right arm vs. feet. Similar to state-of-the-art methods [47,59-61], the commonly used leave-one-rest strategy was employed to perform cross-subject EEG sample classification. For each binary classification task of 12 subjects, one subject was selected as the target domain, while the remaining 11 subjects served as the source domain. Crosssubject MI-EEG classification was then carried out for the target subject. Each binary classification task contained 11 x 70 x 2 EEG samples for training, and 1 x 70 x 2 EEG samples for testing. This meant a total of 12 testing classification accuracies for each stimulus material, with the average accuracy used as the measurement index.

During training of the MAME method, the step learning decay strategy (stepLR) [62] was adopted to adjust the learning rate:

*··

With the increase in training epochs, stepLR uses 7 and step_size to decay the learning rate, enabling the model to emerge from the locally optimal solution and keep the best generalization ability.

The parameters for feature extraction using the CNN models (ShallowNet and EEGNet) were set to 7 = 0.05, step_size = 20,7 = 0.5. The number of iterations was set to 10 for feature extraction, and the feature with the least loss was saved and used as the basis for the MAME method. The parameters for meta-learning were set to w ay = 2, shot = 20, query = 15, and the two learning rates were set to a = le-4, ß = 5e-4. The parameters for stepLR were set to step_size = 20,7 = 0.8. The number of epochs for the training and testing of all MI-EEG samples was set to 40. For the binary classification tasks (left arm vs. right arm, left arm vs. feet, and right arm vs. feet), the best testing accuracy obtained with each stimulus material is shown in Tables 1,2,3, respectively. The tables also show the average accuracy and standard deviation (SD) for each classification condition. The classification results for two CNN models (ShallowNet and EEGNet) are shown, and the /-test statistical method was used to compare pairs with the same model. The Benjamini-Hochberg procedure was used to multi /-test the results for ShallowNet and EEGNet, giving ing higher confidence for the p-value.

For the three compared materials (human, robot, arrow), the ShallowNet feature extractor achieved an average classification accuracy of77.60%, 78.98%, and 77.73%, spectively, for left arm vs. right arm MI tasks, with SDs of 1.31%, 3.51%, and 1.77%, respectively (Table 1). The EEGNet feature extractor achieved an average classification accuracy of 73.19%, 77.73%, and 75.67%, with SDs of 2.77%, 3.13%, and 3.7%, respectively. Robot materials achieved the most discriminative ERD/S responses for the left arm and right arm motion cognitive processes. No significant differences (¿-test) between the three stimulus materials were observed for the binary cross-subject classification of left arm MI and right arm MI tasks.

For the human, robot and arrow materials, the ShallowNet feature extractor achieved an average classification accuracy of 74.17%, 74.92%, and 73.35%, respectively, for left arm vs. feet MI tasks, with SDs of 2.30%, 4.19%, and 3.06%, respectively (Table 2). The EEGNet feature extractor achieved an average classification accuracy of 69.56%, 74.60%, and 73.54%, with SDs of 2.58%, 2.09%, and 2.46%, respectively. The robot materials again achieved the most discriminative ERD/S responses for the left arm and feet motion cognitive processes. No signifi- cant differences between the three stimulus materials were observed for the binary cross-subject classification of left arm MI and feet MI tasks.

Finally, the ShallowNet feature extractor achieved an average classification accuracy for the human, robot and arrow materials of 77.81%, 79.88%, and 76.67%, respectively, for right arm vs. feet MI tasks, with SDs of 1.17%, 2.72%, and 2.13%, respectively (Table 3). The EEGNet feature extractor achieved an average classification accuracy of 72.29%, 79.04%, and 75.31%, with SDs of 2.02%, 3.78%, and 3.15%, respectively. The robot materials once again achieved the most discriminative ERD/S responses for the right arm and feet motion cognitive processes. For the binary cross-subject classification of right arm MI and feet MI tasks, significant improvement was observed when comparing robot stimulus materials to human and arrow stimuli (p < 0.05). However, no significant difference was observed between human and arrow stimuli.

To better illustrate the binary classification results shown in Tables 1,2,3, box-plots of the classification accuracies were constructed for the three binary classification tasks (Fig. 12). The maximum and average classification accuracies achieved with the ShallowNet and EEGNet feature extractors were highest for the robot materials. The three binary classification tasks also achieved the highest median accuracy with the robot materials. Moreover, robot materials showed larger inter-quartile ranges with the ShallowNet feature extractor than with the EEGNet feature extractor. Therefore, statistical analysis of cross-subject classification accuracies showed that robot materials achieved better guidance of the motion cognitive process compared to arrow and human materials.

The iteration process of the MAML method was demonstrated by first selecting the accuracy and Fl-score of the EEGNet feature extractor. Iterations of all three binary classification tasks for the three materials are shown in Fig. 13. At the start, the iterative accuracy of the MAME method begins to increase. However, for all three materials, the accuracy and F1 -score gradually decrease once the number of iterations exceeded 20 rounds. Overfitting occurred when the number of iterations exceeded 20, resulting in a lower average accuracy and Fl-score. It can also be seen that the MAME method is able to generalize the extracted features among subjects, which is more effective for real-world BCIs.

4.3 Ablation Study

With cross-subject classification, two factors may influence the proposed CMCA-MAML method. The first is the comparison with state-of-the-art methods, and the second is the parameter settings during training of the MAML method. The ablation study is focused on these factors. Since "Subject 7" reported a classification result that approached the average, samples from this subject were selected for the parameters setting experiments.

4.3.1 Comparison with State-of-the-art Methods

To validate the feasibility and effectiveness of the proposed CMCA-MAML method, four state-of-the-art domain adaptation methods were selected to compare cross-subject MI-EEG classification with the aligned samples:

(1) CMCA-CSP [47]: MI-EEG signals from different subjects are first aligned by CMC A, the spatial features extracted by CSP, and then classified by the linear discriminant analysis (EDA) classifier.

(2) CMCA-CSP-transfer component analysis (TCA) [60]: This method is the same as CMCA-CSP, except that CSP features are adapted by the TCA domain adaptation method.

(3) CMCA-CSP-joint distribution adaptation (IDA) [61]: This method is the same as CMCA-CSP, except that CSP features are adapted by the IDA domain adaptation method.

(4) Manifold embedded knowledge transfer (MEKT) [59]: The tangent space vector on covariance features is extracted from the aligned MI-EEG signals, and the MEKT method is then used to adapt the features and classify them according to the EDA classifier.

The comparative experiments employed the same settings as the CMCA-MAML method, including the three binary classification tasks. The leave-one-rest strategy was also adopted in 12 subjects for comparison with the conventional methods. The average accuracy of 12 cross-subject MI-EEG classifications for three binary tasks with the arrow, human and robot stimulus materials are shown in Table 4.

The CMCA method is the common pre-processing method used for cross-subject EEG samples. Based on the aligned MI-EEG samples, the conventional domain adaptation methods first extract the spatial features, such as supervised common spatial patterns, and unsupervised tangent space vectors on covariance. These two spatial feature methods can extract discriminative features from different MI tasks. However, for the given EEG samples, the extracted features are fixed and cannot be fine-tuned according to the adaptation and classification procedure. Fixed spatial features are unsuitable for EEG samples that are difficult to distinguish. As shown in Table 4, the present study used the "EMOTIV" device. This provides a low signal-tonoise ratio and strong non-linear and non-stationary characteristics for EEG samples, resulting in greater differences in motion cognitive processes among subjects. It is therefore difficult for conventional domain adaptation methods to decode different MI tasks.

The current study adopted the CMCA-MAML method to construct a cross-subject, meta-learning procedure, with feature extraction using the CNN models of ShallowNet and EEGNet. By employing a fine-tuned strategy to adjust the parameters for the feature extractor during meta-training, domain generalization can be achieved by adjusting specific features for different subjects. This allows the highest MI-EEG signal classification accuracy to be achieved. The meta-learning procedure based on the MAME method requires a longer training time compared with conventional methods. However, after training, the model is generalized for different subjects, and the time complexity for verification is similar to conventional domain adaptation methods.

4.3.2 Parameter Settings for the MAME Method

To examine the influence of various parameters on the MAME method, the best performing MI-EEG samples on robot materials from "Subject 7" (the testing target domain) were selected for the ablation study, with the binary classification task of right arm vs. feet used for testing. Different parameter settings were evaluated by setting to: shot = [10, 20, 30, 40, 50], query = [5, 10, 15, 20, 25], a = [1 x 10"1, 1 x IO"2, 1 x IO"3, 1 x 10"4, 1 x IO"5], ß = [5 x 10"1, 5 x 10 2, 5 x 10 3, 5 x 101, 5 x 10 5], step_size = [10, 20, 30, 40, 50], and 7 = [0.5, 0.6, 0.7, 0.8, 0.9]. The EEGNet feature extractor was used for the ablation study, and the results are shown in Fig. 14, including comparison of the ShallowNet and EEGNet feature extractors.

As the shot and query parameters increase, the average accuracy continues to rise, thereby improving the performance of the decoding motion cognitive process (Fig. 14a). With the MAME method, a larger shot means that more MIEEG samples are sampled in each sampler process. This results in more MI-EEG samples being used to train the inner base learner, making the EEGNet more capable of extracting discriminative features. Furthermore, a larger query means that more testing MI-EEG samples are used to train the outer meta learner, thereby increasing the generalization ability of the meta-learning for cross-subject. In summary, the average classification results shown in Fig. 14a concur with the theory of meta-learning.

As shown in Fig. 14b, the best average classification accuracy obtained by the MAME method is at q = 1 x 10-4 and Ő = 5 x 101. Whena and ß are both larger, the average classification accuracy is lower, consistent with known characteristics of the EEGNet feature extractor and MAME method. There is some reduction in classification performance when a and ß are both too small. Therefore, for MI-BCI applications a slightly smaller initial learning rate should be set for the EEGNet feature extractor and the MAME method. Eqn. 9 should be included to continuously decay the learning rates as the iterations progress.

The results shown in Fig. 14c suggest that the two parameters used to adjust the decay of learning rates during the iterations, 7 and step_size, have less impact on the MAME method. The best average classification accuracy was obtained when 7 = 0.9 and step_size = 20. However, these two parameters had little effect on the average classification accuracy. Therefore, the learning process for the MAME method is related only to the determination of initial learning rates, as well as the selection of suitable learning rate decay parameters.

The average classification accuracy for the two CNN feature extractors is compared in Fig. 14d. Results for the three binary classification tasks with human materials reveal the ShallowNet model achieves better accuracy than the EEGNet model. The ShallowNet model also showed better classification accuracy than the EEGNet model for the left arm vs. right arm, and right arm vs. feet classification tasks with robot and arrow materials, although these differences were not statistically significant. The average accuracy was the same for the left arm vs. feet classification task with robot and arrow materials. Generally ing, the model is more suitable for the MAME method than the EEGNet model, since it adopts fewer convolutional operations while providing a greater number of features. 5.

Discussion

5.1 ERD/S Characteristics of Motion Cognitive Processes Guided by Different Stimuli

This study used cross-subject MI-EEG classification results to explore ERD/S responses during different motion cognitive processes. It also comprehensively explored the guidance roles provided by three kinds of visual stimulus material: arrow, human, and robot. Specifically, two schemes of humanoid-based materials (healthy human body and biped humanoid robot) were used to guide the specific motion cognitive process. Classical MI is followed by the instruction of arrow materials. Subjects were asked to imagine the same left arm, right arm, and feet motions during experiments. In our previous study [63], we found that a visual stimulus of moderate-speed arm movements by a biped humanoid robot evoked an effective ERD/S phenomenon. The findings from this earlier research supported the humanoid robot as visual stimulus material for guiding the motion cognitive process.

The present study compared the evoked ERD/S characteristics between a healthy human body and a biped humanoid robot. As shown in Tables 1,2,3, and Fig. 14, the results provide theoretical support for the design of further visual stimuli for guiding the motion cognitive process. On one hand, the use of a biped humanoid robot to present stimulus material will optimize the actual stimulation process according to the quality of the rehabilitation process, thereby ensuring the exoskeleton-assisted rehabilitation system is always in the optimal state. On the other hand, according to the neuroplasticity theory, it is helpful for the rehabilitation of motor disorders by providing stimulus materials for different patients to evoke optimal ERD/S ERD/S responses. Stimulus materials must be continuously adjusted according to the rehabilitation cycle [64], and can be easily adjusted on the robot platform.

The results of our cross-subject MI-EEG classification showed that robot materials achieve the best ERD/S characteristics during the motion cognitive process, with arrow materials being the second-best stimulus and human materials the worst. BEAM variations of arrow materials are consistent with the classical MI paradigm (see Figs. 7,10,11) [65,66]. Moreover, robot materials have corresponding ERD/S characteristics in the frontal, parietal, and occipital brain regions, and have a certain lateralization effect (see Figs. 9,10,11). The human materials are related to a static material form or motor design, but require further experimentation and evaluation (see Figs. 8,10,11). Iterations of the MAML method for the robot material were consistently higher (see Fig. 13). In conclusion, robot materials provided the best results. In addition to the importance of designing the motors, the choice of stimulus materials that guide the MI process is also critical.

Other important issues concerning guidance of the motion cognitive process is whether the materials are in static form, and the personal perspective with regard to the presentation of stimuli. In our previous study [63], video frames of dynamic robot materials were presented to the subjects. We showed that dynamic visual stimuli will stably evoke the ERD/S phenomenon, with excellent classification performance. Furthermore, whereas arrow materials are simulated in the first-person perspective, the robot and human materials are simulated in the third-person perspective. The first-person perspective was not considered when designing the stimulus materials. Indeed, some studies have shown that subjects who observe their own hand movements (first-person perspective) evoke a stronger ERD/S phenomenon compared to observing the hand movements of other people (third-person perspective) [67,68]. In future research, we will compare more stimulus materials of complex movements, static and dynamic, as well as firstperson and third-person perspectives. This should allow us to identify more suitable stimulus materials for guiding the motion cognitive process.

5.2 Meta-learning Based Cross-subject MI-EEG Classification Method

A stratified cross-subject MI-EEG classification method was employed in this study. The effectiveness and performance of the proposed method was compared to that of four conventional domain adaptation methods. Visualization of ERD/S curves revealed differences between the three stimulus materials (see Fig. 5). As shown in Table 4, the average cross-subject classification with the proposed method showed >70% accuracy for three binary classification tasks, which was much higher than the conventional methods. The performance of conventional methods relics on the discrimination of CSP features, but this was not effective with the extracted MI-EEG samples from our experiments (see Fig. 6). The ShallowNet and EEGNet CNN models employed in this study extracted discriminative features by utilizing a fine-tuned strategy. The results showed a range of 69% to 79% for the average cross-subject MIEEG classification. The proposed CMCA+MAML method produces iterative table classification results and is highly robust.

In recent years, the CMCA [47] has proven to be an excellent cross-domain, multivariate time series alignment method. The CMCA method aligns the mean covariance of different domains, thereby whitening the samples of all domains. For cross-session and cross-subject situations, EEG signals show large changes and do not conform to the assumption of independent and identical distribution. This leads to major difficulties when constructing the MI-EEG signal decoding method [69]. After performing the CMCA method, the distribution of MI-EEG samples is more in line with a normal distribution, thereby improving the classification performance of the subsequent MAME method.

The MAML method adopts a dual fine-tuning process of base-learner and meta-learner for the generalization of different subjects [48]. In the present study, two simple CNN models (ShallowNet and EEGNet) were embedded in the MAML method. MAML is a general end-to-end framework. Recently, more complex CNN models have also been embedded to further improve the cross-subject MI-EEG classification performance. We performed ablation studies for the three groups of influential parameters (shot and query, a and ß, step_size and 7) to instruct parameter selections that consider both performance and efficiency for a given number of samples (see Fig. 14).

5.3 Visual Stimulus-guided MI-B CI

Due to subjective individual differences in the motion cognitive process, considerable time is currently required to calibrate and train an individual-specific MI-EEG classifier. This causes fatigue when individual patients use the MI-BCI system [70,71]. The present study guided the subjects' motion cognitive process through three different stimulus schemes, and found that robot materials provided optimal guidance. Recently, many researchers have guided the MI process through action observations [43,72,73], thus avoiding the lower performance of MI-BCI systems caused by individual differences in subjective imagination. In contrast to most action observation research that is focused on the human body, the current study compared the motion cognitive process between a biped humanoid robot and healthy human body. The humanoid robot showed better consistency in the cross-subject motion cognitive paradigm. When using the MI-BCI system to construct an exoskeleton-assisted rehabilitation system, designing the stimulus materials as a humanoid robot consistent with the exoskeleton could greatly improve the rehabilitation process by using a familiar scene.

The present work employed a biped humanoid robot platform to design static motions as stimulus material. This is more invariant across different subjects than motions by healthy humans. Using the motion cognitive process based on the robot platform, the desired motion paradigms can be designed to consistently maintain a high level of classification accuracy and Fl-score, especially for the binary classification task of right arm vs. feet. Conventional MI-BCI systems rely only on subjective MI and establish subject-specific classification methods from a limited number of MI-EEG samples. In comparison, the robot stimulus paradigm with the cross-subject classification method proposed here has more advantages and better outcomes in exoskeleton-assisted rehabilitation BCI systems.

5.4 Limitations and Future Directions

Guided by different visual stimulus materials, the method proposed here explores the classification of crosssubject MI-EEG signal decoding. Moreover, it is superior to conventional methods on collections of EEG signals. However, our method has the following limitations: (1) The EEG device (EMOTIV EPOC+) used in this study has only 14 channels. EEG devices with more channels can comprehensively collect more signal changes in brain regions. (2) Individual differences in MI ability presented a challenge in this study, and may have decreased the overall accuracy of cross-subject motor cognitive decoding. (3) This study used three kinds of visual stimulus material (arrow, human, and robot) presented in a static manner. However, the type of visual stimulus material and the manner in which they are presented may impact the research results. (4) In this study, centroid alignment and the MAML framework were employed to achieve cross-subject MI-EEG classification. However, the ShallowNet and EEGNet backbones utilized here are relatively simple. Hence, the characterization of non-linear and non-stationary MI-EEG characteristics was not profound, leaving room for improvement in the performance of MI-EEG decoding.

In future investigations of motion cognitive decoding of cross-subject MI, the following aspects could be improved: (1) An advanced EEG device named 'Brain Products' with 32 electrodes will be considered for EEG signal acquisition. More channels mean more comprehensive collection of signal changes in brain regions. (2) Several more universal models of cross-subject motor cognitive decoding will be designed to help reduce individual differences in MI ability. (3) New visual stimulus materials (e.g., animals, cartoon characters, etc.) and a dynamic manner of presentation (e.g., videos) will be used for MI experiments. (4) In the cross-subject MI-EEG decoding framework, it would be beneficial to incorporate additional preprocessing steps, filter banks and time segmentations so as to reduce variations in MI-EEG. Simultaneously, the construction of more complex backbones to obtain comprehensive feature representations, such as transformers and attention-based networks, would enhance the performance of MI-EEG decoding.

6. Conclusion

This study investigated the motion cognitive process guided by different visual stimuli. Three kinds of materials (arrow, human, and robot) were used to design left arm motion, right arm motion, and feet motion as the visual stimuli. To validate the influence of the motion cognitive process, three binary classification tasks were constructed to measure ERD/S responses on different visual stimulus materials. To eliminate differences between subjects during classification, a CMCA-MAML method was proposed to classify the cross-subject MI-EEG signals guided by three visual stimuli. Spatial-temporal-spectral analysis and classification results showed that robot materials evoke more discriminative ERD/S phenomena, thus facilitating the construction of subject-free, high-performance MI-BCI systems.

Future work by our group will be focused on the following areas: (1) the use of dynamic visual stimulus materials to comprehensively measure differences in motion cognitive processes; (2) the collection of additional MI-EEG samples to explore more useful and robust cross-subject EEG classification methods; (3) the design of new deep neural network models to decode the motion cognitive process of the human brain; (4) combining MI-EEG with intelligent robots to develop hybrid intelligence.

Availability of Data and Materials

Data will be made available on reasonable request, please contact with: [email protected].

Author Contributions

Conceptualization, TL, JL and HP; Methodology, TL and IE; Software, TL; validation, XZ, SW and HP; formal analysis, RE; investigation, SW; resources, TL and HP; data curation, TL and HP; writing-original draft preparation, TL and JL; writing-review and editing, TL and HP; visualization, SW; supervision, HP and TL; project administration, TL; funding acquisition, TL, HP and RL. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript. All authors have participated sufficiently in the work and agreed to be accountable for all aspects of the work.

Ethics Approval and Consent to Participate

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Ethics Committee of Shaoxing University (No. 2021012). Informed consent was obtained from all subjects involved in the study.

Acknowledgment

Not applicable.

Funding

This work was supported by Planning Project of Philosophy and Social Science of Zhejiang Province (Grant No. 24JCXK01YB, 24JCXK02YB), National Natural Science Foundation of China (Grant No. 62106049, 61662025, 61871289, 62007016), Natural Science Foundation of Fujian Province of China (Grant No. 2022101655).

Conflict of Interest

The authors declare no conflict of interest.

References

References

[1] Lebedev MA, Nicolelis MAL. Brain-machine interfaces: past, present and future. Trends in Neurosciences. 2006; 29: 536-546.

[2] Lebedev MA. Brain-machine interfaces: an overview. Translational Neuroscience. 2014; 5: 99-110.

[3] Abiri R, Borhani S, Sellers EW, Jiang Y, Zhao X. A comprehensive review of EEG-based brain-computer interface paradigms. Journal of Neural Engineering. 2019; 16: 011001.

[4] Baniqued PDE, Stanyer EC, Awais M, Alazmani A, Jackson AE, Mon-Williams MA, et al. Brain-computer interface ro-botics for hand rehabilitation after stroke: a systematic review. Journal of Neuroengineering and Rehabilitation. 2021; 18: 15.

[5] Ang KK, Chua KSG, Phua KS, Wang C, Chin ZY, Kuah CWK, et al. A Randomized Controlled Trial of EEG-Based Mo-tor Imagery Brain-Computer Interface Robotic Rehabilitation for Stroke. Clinical EEG and Neuroscience. 2015; 46: 310-320.

[6] De Venuto D, Mezzina G. Multisensing architecture for the balance losses during gait via physiologic signals recognition. IEEE Sensors Journal. 2020; 20: 13959-13968.

[7] Yang N, An Q, Kogami H, Yamakawa H, Tamura Y, Takahashi K, et al. Temporal Features of Muscle Synergies in Sit-to-Stand Motion Reflect the Motor Impairment of Post-Stroke Patients. IEEE Transactions on Neural Systems and Rehabilitation gineering: a of the IEEE Engineering in Medicine and Biology Society. 2019; 27: 2118-2127.

[8] Frisoli A, Loconsole C, Leonardis D, Banno F, Barsotti M, Chisari C, et al. A New Gaze-BCI-Driven Control of an Upper Limb Exoskeleton for Rehabilitation in Real-World Tasks. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews). 2012; 42: 1169-1179.

[9] Ang KK, Guan C. Brain-computer interface for neurorehabilitation of upper limb after stroke. Proceedings of the IEEE. IEEE. 2015; 103: 944-953.

[10] Foong R, Ang KK, Quek C, Guan C, Phua KS, Kuah CWK, et al. Assessment of the Efficacy of EEG-Based MI-BCI With Visual Feedback and EEG Correlates of Mental Fatigue for UpperLimb Stroke Rehabilitation. IEEE Transactions on Bio-medical Engineering. 2020; 67: 786-795.

[11] Cheng N, Phua KS, Lai HS, Tam PK, Tang KY, Cheng KK, et al. Brain-Computer Interface-Based Soft Robotic Glove Rehabilitation for Stroke. IEEE Transactions on Bio-medical Engineering. 2020; 67: 3339-3351.

[12] Xu L, Xu M, Jung TP, Ming D. Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface. Cognitive Neurodynamics. 2021; 15: 569-584.

[13] Dong R, Zhang X, Li H, Masengo G, Zhu A, Shi X, et al. EEG generation mechanism of lower limb active movement intention and its virtual reality induction enhancement: a preliminary study. Frontiers in Neuroscience. 2024; 17: 1305850.

[14] Fox NA, Bakermans-Kranenburg MJ, Yoo KH, Bowman LC, Cannon EN, Vanderwert RE, et al. Assessing human mir-ror activity with EEG mu rhythm: A meta-analysis. Psychological Bulletin. 2016; 142: 291-313.

[15] Bablani A, Edla DR, Tripathi D, Cheruku R. Survey on BrainComputer Interface: An emerging computational intelligence para-digm. ACM Computing Surveys (CSUR). 2019; 52: 1-32

[16] Bianco V, Berchicci M, Perri RE, Spinelli D, Di Russo E The proactive self-control of actions: Time-course of underlying brain activities. NeuroImage. 2017; 156: 388-393.

[17] Hardwick RM, Caspers S, Eickhoff SB, Swinnen SP. Neural correlates of action: Comparing meta-analyses of imagery, observa-tion, and execution. Neuroscience and Biobehavioral Reviews. 2018; 94: 31-44.

[18] Alíalo T, Kellis S, Klaes C, Lee B, Shi Y, Pejsa K, et al. Neurophysiology. Decoding motor imagery from the posterior parietal cortex of a tétraplégie human. Science (New York, N.Y). 2015; 348: 906-910.

[19] Rizzolatti G, Craighero L. The mirror-neuron system. Annual Review of Neuroscience. 2004; 27: 169-192.

[20] Caspers S, Zilles К, Laird AR, Eickhoff SB. ALE meta-analysis of action observation and imitation in the human brain. NeuroImage. 2010; 50: 1148-1167.

[21] Gazzola V, Aziz-Zadeh L, Keysers C. Empathy and the somatotopic auditory mirror system in humans. Current Biology: CB. 2006; 16: 1824-1829.

[22] Jiang J, Liu F, Zhou L, Jiang C. The neural basis for understanding imitation-induced musical meaning: The role of the human mirror system. Behavioural Brain Research. 2019; 359: 362369.

[23] Holmes PS, Wright DJ. Motor cognition and neuroscience in sport psychology. Current Opinion in Psychology. 2017; 16: 43-47.

[24] Copelli F, Rovetti J, Ammirante P, Russo FA. Human mirror neuron

system responsivity to unimodal and multimodal presenta-tions of action. Experimental Brain Research. 2022; 240: 537-548.

[25] Pfurtscheller G, Lopes da Silva FH. Event-related EEG/MEG synchronization and desynchronization: basic principles. Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology. 1999;110: 1842-1857.

[26] Kitahara K, Hayashi Y, Yano S, Kondo T. Target-directed motor imagery of the lower limb enhances event-related desynchroniza-tion. PloS One. 2017; 12: eO 184245.

[27] Cho H, Ahn M, Ahn S, Kwon M, Jun SC. EEG datasets for motor imagery brain-computer interface. GigaScience. 2017; 6: 1-8.

[28] McFarland DJ, Sarnacki WA, Wolpaw JR. Effects of training pre-movement sensorimotor rhythms on behavioral performance. Journal of Neural Engineering. 2015; 12: 066021.

[29] Abdelhaleem N, Taher S, Mahmoud M, Hendawy A, Hamed M, Mortada H, et al. Effect of action observation therapy on motor function in children with cerebral palsy: a systematic review of randomized controlled trials with meta-analysis. Clinical Rehabilitation. 2021; 35: 51-63.

[30] Ang KK, Chin ZY, Wang C, Guan C, Zhang H. Filter Bank Common Spatial Pattern Algorithm on В CI Competition IV Datasets 2a and 2b. Frontiers in Neuroscience. 2012; 6: 39.

[31] Zheng Q, Zhu F, Heng PA. Robust Support Matrix Machine for Single Trial EEG Classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering: a Publication of the IEEE Engineering in Medicine and Biology Society. 2018; 26: 551-562.

[32] Rithani M, Kumar RP, Doss S. A review on big data based on deep neural network approaches. Artificial Intelligence Review. 2023;56: 14765-14801.

[33] Li Z, Li H, Meng L. Model compression for deep neural networks: A survey. Computers. 2023; 12: 60.

[34] Marino GC, Petrini A, Malchiodi D, Frasca M. Deep neural networks compression: a comparative survey and choice recommen-dations. Neurocomputing. 2023; 520: 152-170.

[35] Kaur M, Upadhyay R, Kumar V. A Hybrid Deep Learning Framework Using Scaling-Basis Chirplet Transform for Motor Im-agery EEG Recognition in Brain-Computer Interface Applications. International Journal of Imaging Systems and Technology. 2024; 34: e23127.

[36] Bassi PRAS, Dertkigil SSJ, Cavalli A. Improving deep neural network generalization and robustness to background bias via lay-er-wise relevance propagation optimization. Nature Communications. 2024; 15: 291.

[37] Lawhern VJ, Solon AJ, Waytowich NR, Gordon SM, Hung CP, Lance B J. EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. Journal of Neural Engineering. 2018; 15: 056013.

[38] Schirrmeister RT, Springenberg JT, Fiederer LD J, Glasstetter M, Eggensperger К, Tangermann M, et al. Deep learning with convolutional neural networks for EEG decoding and visualization. Human Brain Mapping. 2017; 38: 5391-5420.

[39] Angelini M, Fabbri-Destro M, Lopomo NF, Gobbo M, Rizzolatti G, Avanzini P. Perspective-dependent reactivity of sensorimotor mu rhythm in alpha and beta ranges during action observation: an EEG study. Scientific Reports. 2018; 8: 12429.

[40] Neuper C, Scherer R, Wriessnegger S, Pfurtscheller G. Motor imagery and action observation: modulation of sensorimotor brain rhythms during mental control of a brain-computer interface. Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology. 2009; 120: 239247.

[41] Gonzalez-Rosa JJ, Natali F, Tettamanti A, Cursi M, Velikova S, Comi G, et al. Action observation and motor imagery in mance of movements: evidence from EEG and kine- and kinematics analysis. Behavioural Brain Research. 2015; 281: 290300.

[42] Kaneko N, Yokoyama H, Masugi Y, Watanabe K, Nakazawa K. Phase dependent modulation of cortical activity during action observation and motor imagery of walking: An EEG study. NeuroImage. 2021; 225: 117486.

[43] Chaisaen R, Autthasan P, Mingchinda N, Leelaarporn P, Kunaseth N, Tammajarung S, et al. Decoding EEG Rhythms during Action Observation, Motor Imagery, and Execution for Standing and Sitting. IEEE Sensors Journal. 2020; 20: 13776-13786.

[44] Savié A, Lontis R, Jiang N, Popovié M, Farina D, Dremstrup K, et al. Movement related cortical potentials and sensory motor rhythms during self initiated and cued movements. Replace, Repair, Restore, Relieve-Bridging Clinical and Engineering Solutions in Neurorehabilitation: Proceedings of the 2nd International Conference on NeuroRehabilitation (ICNR2014), Aalborg, 24-26 June, 2014 (pp. 701-707). Cham: Springer International Publishing: Switzerland. 2014.

[45] Annett J. Motor imagery: Perception or action? Neuropsychologia. 1995; 33: 1395-1417.

[46] de Lange FP, Roelofs K, Toni I. Motor imagery: a window into the mechanisms and alterations of the motor system. Cortex; a Journal Devoted to the Study of the Nervous System and Behavior. 2008; 44: 494-506.

[47] He H, Wu D. Transfer Learning for Brain-Computer Interfaces: A Euclidean Space Data Alignment Approach. IEEE Transactions on Bio-medical Engineering. 2020; 67: 399-410.

[48] Finn C, Abbééi P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on ma-chine learning. Proceedings of Machine Learning Research. 2017;70: 1126-1135.

[49] George O, Smith R, Madiraju P, Yahyasoltani N, Ahamed SI. Motor Imagery: A review of existing techniques, challenges and potentials. 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC). IEEE. 2021; 1893- 1899.

[50] Mishuhina V, Jiang X. Complex common spatial patterns on time-frequency decomposed EEG for brain-computer interface. Pat-tern Recognition. 2021 ; 115: 107918.

[51] Gaur P, Chowdhury A, McCreadie K, Pachori RB, Wang H. Logistic Regression with Tangent Space-Based Cross-Subject Learning for Enhancing Motor Imagery Classification. IEEE Transactions on Cognitive and Developmental Systems. 2022; 14: 1188-1197.

[52] Cecotti H, Gräser A. Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2011; 33: 433-445.

[53] Gaur P, Gupta H, Chowdhury A, McCreadie K, Pachori RB, Wang H. A Sliding Window Common Spatial Pattern for Enhanc-ing Motor Imagery Classification in EEG-BCI. IEEE Transactions on Instrumentation and Measurement. 2021; 70: 1-9.

[54] Luo TJ, Zhou CL, Chao F. Exploring spatial-frequencysequential relationships for motor imagery classification with recurrent neural network. BMC Bioinformatics. 2018; 19: 344.

[55] Banluesombatkul N, Ouppaphan P, Leelaarporn P, Lakban P, Chaitusaney B, Jaimchariyatam N, et al. MetaSleepLearner: A Pilot Study on Fast Adaptation of Bio-Signals-Based Sleep Stage Classifier to New Individual Subject Using MetaLearning. IEEE Journal of Biomedical and Health Informatics. 2021; 25: 1949-1963.

[56] Li S, Wu H, Ding L, Wu D. Meta-Learning for Fast and PrivacyPreserving Source Knowledge Transfer of EEG-Based BCIs. IEEE Computational Intelligence Magazine. 2022; 17: 16-26.

[57] Duan T, Chauhan M, Shaikh MA, Chu J, Srihari S. Ultra efficient transfer learning with meta update for cross subject EEG classification. arXiv. 2020. (preprint)

[58] Michel CM, Murray MM. Towards the utilization of EEG as a brain imaging tool. NeuroImage. 2012; 61: 371-385.

[59] Zhang W, Wu D. Manifold Embedded Knowledge Transfer for Brain-Computer Interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering: a Publication of the IEEE Engineering in Medicine and Biology Society. 2020; 28: 1117- 1127.

[60] Pan SJ, Tsang I W, Kwok JT, Yang Q. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks. 2011; 22: 199-210.

[61 ] Long M, Wang J, Ding G, Sun J, Yu PS. Transfer feature learning with joint distribution adaptation. Proceedings of the IEEE International Conference on Computer Vision. IEEE. 2013; 22002207.

[62] Cutler J, Drusvyatskiy D, Harchaoui Z. Stochastic optimization under time drift: Iterate averaging, step decay, and high probability guarantees. arXiv. 2021. (preprint)

[63] Luo TJ, Lv J, Chao F, Zhou C. Effect of Different Movement Speed Modes on Human Action Observation: An EEG Study. Frontiers in Neuroscience. 2018; 12: 219.

[64] Ertelt D, Small S, Solodkin A, Dettmers C, McNamara A, Binkofski F, et al. Action observation has a positive impact on rehabilitation of motor deficits after stroke. NeuroImage. 2007; 36: T164-T173.

[65] Eaves DL, Riach M, Holmes PS, Wright DJ. Motor Imagery during Action Observation: A Brief Review of Evidence, Theory and Future Research Opportunities. Frontiers in Neuroscience. 2016; 10: 514.

[66] Quandt LC, Marshall PJ. The effect of action experience on sensorimotor EEG rhythms during action observation. Neuropsychologia. 2014; 56: 401-408.

[67] Nagai H, Tanaka T. Action Observation of Own Hand Movement Enhances Event-Related Desynchronization. IEEE Transactions on Neural Systems and Rehabilitation Engineering: a Publication of the IEEE Engineering in Medicine and Biology Society. 2019; 27: 1407-1415.

[68] Song M, Kim J. A Paradigm to Enhance Motor Imagery Using Rubber Hand Illusion Induced by Visuo-Tactile Stimulus. IEEE Transactions on Neural Systems and Rehabilitation Engineering: a Publication of the IEEE Engineering in Medicine and Biology Society. 2019; 27: 477-486.

[69] Altaheri H, Muhammad G, Alsulaiman M, Amin SU, Altuwaijri GA, Abdul W, et al. Deep learning techniques for classi-fication of electroencephalogram (EEG) motor imagery (MI) signals: a review. Neural Computing and Applications. 2023; 35: 14681- 14722.

[70] Zhang K, Robinson N, Lee SW, Guan C. Adaptive transfer learning for EEG motor imagery classification with deep Convolutional Neural Network. Neural Networks: the Official Journal of the International Neural Network Society. 2021; 136: 1-10.

[71] Jacquet T, Lepers R, Poulin-Charronnat B, Bard P, Pfister P, Pageaux В. Mental fatigue induced by prolonged motor imagery increases perception of effort and the activity of motor areas. Neuropsychologia. 2021; 150: 107701.

[72] Rannaud Monany D, Papaxanthis C, Guillot A, Lebon F. Motor imagery and action observation following immobiliza-tioninduced hypoactivity: A narrative review. Annals of Physical and Rehabilitation Medicine. 2022; 65: 101541.

[73] Sarasso E, Agosta F, Piramide N, Gardoni A, Ganu E, Leocadi M, et al. Action Observation and Motor Imagery Improve Dual Task in Parkinson's Disease: A Clinical/fMRI Study. Movement Disorders: Official Journal of the Movement Disorder Society. 2021; 36: 2569-2582.