Advancements in Image Feature-Based Classification of Motor Imagery EEG Data: A Comprehensive Review

Electroencephalography (EEG) is a neuroimaging method for measuring, recording, and examining the brain activity. It has been used in brain-computer interfaces (BCIs), consumer neuroscience, psychiatry, diagnosis, rehabilitation, and treatment. The most common use case is BCIs, which establish a communication and control environment without muscles and peripheral devices. Some examples include limb motor imagery, controlling unmanned aerial vehicles, virtual drones, robotic arms, and virtual reality environments. Motor imagery (MI) is a universal way to develop such BCIs, where actions are performed by mentally simulating the motor action in the brain without using the motor system. Researchers have proposed various machine/deep learning techniques, but the flourishing applications of MI-BCIs still need to be improved. Some difficulties encountered include noise and artefacts, non-stationarity, nonlinearity, inconsistency, being distorted, session-to-session and subject-to-subject variability, trade-offs between training time and performance, and the time-consuming calibration. The problems become even tougher when considering non-invasive real-time MI-EEG systems.

Nowadays, deep learning (DL) has become very popular due to its success in automatic feature extraction and classification capabilities, which differs from classical machine learning. It has been used to classify different data types and has demonstrated great success. Most deep learning research on MI-EEG has used one-dimensional time series data. However, further studies are needed to investigate the processing of MI-EEG signals in deep learning aspects. For example, some deep learning studies used 2D inputs and processed them in 2D instead of 1D time series MI-EEG signals or features. These approaches have increasing applications and successful results have been achieved. In this respect, examining studies on this subject is highly important. So far, most reviews have focussed only on machine learning or deep learning. In this paper, we mainly focus on a review of the techniques for converting MI-EEG signals to images. This is important because the processing of MI-EEG signals, especially in deep learning, is a significant issue. In the literature, only some researchers have investigated this topic. This present study will contribute to this gap in the literature. The basic research questions are: (1) What are the signal-to-image conversion methods for MI-EEG signals? (2) What are the classification techniques using these image features? and (3) What are the challenges and future directions in the classification of MI-EEG signals?

The MI-EEG datasets used for model development and testing are detailed in Sec. 2. The signal-to-image conversion and classification methods are analysed in Sec. 3. The last sections of this article present challenges, future directions, and conclusions.

The conversion of MI-EEG signals to 2D images has usually performed by time-frequency representation approaches. They transform MI-EEG signals into 2D time-frequency images, i.e., spectrograms or scalograms. Many other characteristic methods have also been researched extensively. Earlier studies on the classifications of MI-EEG signal images were mostly conducted with deep learning. It is mainly due to their recent success, where they work well for automatic feature extraction and classification. Apart from deep learning, several traditional machine learning approaches have also been engaged. Deep learning models need large sample data sets to achieve good performance. Nevertheless, collecting large-scale and high-quality MI-EEG data is difficult due to the strict requirements for subjects and experimental environments. Deep transfer learning and data augmentation are promising candidates to solve these problems. They have much potential; nonetheless, the number of studies in this field is limited.

A comprehensive review of the classification and signal-to-image conversion methods is shown in Table 2. Studies are first grouped according to the data sets. In some studies, specific MI categories were classified, while in others, additional categories were constructed. Consequently, the classified MI tasks are given in the third column. The next column gives the signal-to-image conversion and classification methods, respectively. Important features specific to the application of these methods are also emphasised. In the last column, performance results are given regarding accuracy or Kappa. It also explains how the results are obtained. For example, some studies use the original training and test sets, some use only the training or test set, and some use k-fold cross-validation.

Table 2. An extensive review of the classification and signal-to-image conversion methods

Some acronyms are as follow: STFT (Short-time Fourier transform), CWT (continuous wavelet transform), FFT (fast Fourier transform), ETR (EEG topographical representations), TPCT (Clough-Tocher interpolation-based imaging), ST (Stockwell Transform), AAG (angle-amplitude graph), CNN (convolutional neural network), SAE (Stacked Autoencoders), VAE (Automatic Variation Encoder), VMD (variational mode decomposition), PCNN (parallel CNN), 3DCNNs (three-dimensional CNNs), modified visual geometry group network (mVGG), BGRU/Bi-GRUs (bidirectional gated recurrent units), (PMMCL) parallel multimodule CNN and long short-term memory network, DCGAN (deep convolutional generative adversarial network), SIFT (scale-invariant feature transform), BOW (bag-of-words), kNN (k-nearest neighbour algorithm), DA (data augmentation), KBIM (key band imaging method), CV (cross-validation), (K) Kappa, subj (subject).

3.1 Time-frequency image representation methods

The human brain is a complex nonlinear system that produces non-stationary EEG oscillations [28]. Conventional Fourier-based methods are successful in investigating the spectrum of stationary signals. However, they often fail in MI-EEG, where the spectrum changes with time [29]. Time-frequency image representation (TFIR) methods have been used in the literature to comprehend which frequency components exist in different time intervals and how they vary over time. They transform 1D signals into 2D time-frequency representations and provide information about local frequency variations [30]. Short-time Fourier transform (STFT), and Continuous wavelet transform (CWT) are widespread examples applied to numerous problems [14]. TFIR especially has constraints in dealing with fast varying instantaneous frequencies [30], and one of the problems is finding suitable time-frequency resolution [29]. For instance, traditional STFT with invariable window width provides a constant resolution over the entire spectrum [14], and therefore, it suffers poor resolution [29]. Compared to STFT’s constant resolution over the entire spectrum, CWT can provide a higher resolution [14]. On the other hand, CWT can also fail because its window dilation rule is not signal-dependent [29]. In the literature, STFT, CWT and Stockwell transform has been employed as TFIR for MI-EEG signals.

3.1.1 Short-time Fourier transform-based methods

STFT have been one of the most used methods to convert MI-EEG signals into spectrums. In some studies, these spectrums were given to the classifiers as image input directly and without processing, while in others, sub-images of the spectrum were extracted and combined. Thus, useful MI-EEG information was used together. In many studies, to capture event-related desynchronization and synchronization, μ and β bands were extracted and combined. In such combinations, sub-images related to frequency bands were usually resized to the same size to ensure similar effects for all bands. If STFTs from different electrodes were to be combined, the electrodes neighbouring information were also considered. Thus, STFT-based time-frequency and electrode location information was fused. STFT-based approaches are summarized in Table 3. The table shows the data sets, EEG channels, STFT parameter values (signal length, window size, time-lapse, overlap), and frequency bands from which MI information was obtained.

Tabar and Halici [10] combined time, frequency, and electrode location information to create a new input form for MI-EEG signals. They used it in CNNs, with one 1D convolutional and one max-pooling layer. First, they applied STFT to MI-EEG signals and calculated 257×32-sized spectrum images. After that, to capture and use event-related desynchronizations and synchronizations, they extracted 16×32 and 23×32-sized sub-images related to μ and β frequency bands, respectively. To ensure similar effects of both bands, they resized the 23×32-sized sub-images to 15×32. After that, they vertically combined sub-images of β and μ bands, respectively, and constructed 31×32-sized images for a one-channel MI-EEG signal. Lastly, they vertically combined images of C3, Cz, and C4 channels while preserving neighbouring information and formed a 93×32-sized final input image, as shown in Figure 1. Thus, they used MI signals' time and frequency properties to design network input. The experiments were conducted on DS1 and DS6 to classify left/right-hand tasks. This study combined the CNN and stacked autoencoders (SAE), and the features extracted in CNN were classified through the deep network SAE. The STFT + CNN-SAE approach showed 90.0% accuracy with a kappa of 0.8 on DS1 and 77.6±2.1% with a kappa of 0.547±0.083 on DS6. The results for DS6 are from nine subjects and are slightly lower as they include data from different sessions. However, the proposed method achieved a 9% improvement over the competition winner.

1.png

Figure 1. A sample input MI-EEG image where STFT spectrums are extracted, resized, and fused [10]

The same paradigm was used differently, where spectrum images were extracted, resized, and combined. For example, Dai et al. [6] used the electrode location, time, and frequency information and defined new multidimensional input features. They extracted sub-images for β and μ frequency bands from 257×32-sized spectrums. The sub-images were resized to 15×32 to establish the similar effects of both bands. After that, these sub-images were combined vertically, and 31×32-sized images were constructed per channel. In the last stage, images from different channels were vertically combined while preserving the electrodes' neighbouring information. They presented a new classification framework that uses an Automatic Variation Encoder (VAE) after CNN to deal with artefacts, noise, channel correlation, and high-dimensionality problems. The STFT + CNN-VAE method showed an average kappa of 0.564 on DS6 with a 3% improvement. Furthermore, their dataset DS8 achieved the best average kappa of 0.568 with three electrodes and 0.603 with five electrodes.

Xu et al. [23] investigated an STFT-based preprocessing procedure based on time-frequency images. After converting MI-EEG signals to spectrums, they extracted 20×32 and 33×32-sized sub-images for 4–14 Hz μ and 16–32 Hz β bands, respectively, and resized each of these sub-images to 112×224. Then, they combined them and generated 224×224-sized spectrums. Finally, a 224×224×3-sized input image was created by combining spectrum images converted from three EEG channels. After that, to solve the insufficient labelled data problem and enhance the training efficiency of the CNN, they adopted a VGG-16-based deep transfer CNN framework. The model was pre-trained on ImageNet, and the parameters of the VGG-16 CNN were directly transferred to the target CNN. It shares the same structure as VGG-16 except for the softmax output layer. In the target model, the front-layer parameters were frozen, while later-layer parameters were fine-tuned by the target dataset. For the classification of left/right-hand tasks on DS6, the proposed framework achieved 74.2% accuracy, better than most known approaches like standard CNN and support vector machines.

Chaudhary et al. [14] introduced a new deep convolutional neural network for classifying right-hand and right-foot tasks. They applied time-frequency techniques to convert MI-EEG signals into images. In this paper, they obtained spectrograms using the Hamming window, resized them to 227×227×3 and used them as input. After that, they employed a transfer learning strategy to fine-tune the pre-trained Alexnet DCNN. In this network, the last two layers were replaced with new ones to discover features unique to the dataset. The proposed STFT + CNN approach acquired 98.7% accuracy with a kappa of 0.98. In the same study, CWT showed a slightly better result than STFT. Ha and Jeong [24] transformed MI-EEG signals into 2D images and gave them to the capsule network (CapsNet). They formed 3×65×14-sized images by separately applying STFT to three channels. After selecting β and μ bands, they constructed three channel 14×14-sized 2D spectrum images as the initial input. They mainly focused on solving the problems of distorted and inconsistent MI-EEG signals by automatic feature learning and determined the optimal network architecture by analysing various CapsNet configurations. Experiments were conducted on DS6 to classify two-class left/right-hand MI tasks. The STFT + CapsNet approach showed an average accuracy of 78.44%, higher than CNN-based methods (ShallowNet et al.) and various conventional machine learning techniques (SVM et al.). On the other hand, for some subjects, the network architecture failed to capture better features and patterns, and due to this, some of the CNN-based approaches outperformed the proposed method.

Zhang et al. [15] decoded MI-EEG signals based on deep neural networks. They created STFT images of C3, Cz, and C4 channels and mosaicked them into an image while preserving the channel's neighbouring information. They investigated data augmentation methods for classifying spectrogram images. For this purpose, they generated artificial MI-EEG data using five data augmentation methods, i.e., geometric transformation, noise addition, generative model, autoencoder, variational autoencoder, and deep convolutional generative adversarial network (DCGAN). Among them, DCGAN performed better than traditional methods. It showed 17% and 21% accuracy improvements for DS4 and DS6, respectively, for the classification of left/right-hand tasks. The proposed CNN-DCGAN showed high consistency and outperformed the other methods, with average kappa values of 0.564 and 0.677 for these datasets. The results indicated that the data augmentation using operations like rotation may have adversely affected the MI-EEG information.

Al-Saegh et al. [18] employed the raw time-series MI-EEG signals to train a deep end-to-end CNN. They used STFT and converted MI-EEG signals to 2D images in this approach. These images preserve event-related desynchronisation and synchronisation-related information. They applied STFT to capture power spectral density and constructed 250×67-sized images per channel. After that, as with previous studies, μ and β band-related sub-images were extracted, resized, and concatenated. Finally, 96×67-sized final images were formed by combining the electrodes' location information. This paper also presented an augmentation method and enlarged the MI-EEG dataset to solve the same data scarcity problem. It generated trials from inter and intra- subjects and trials. The generated images were then input to the shallow and deep CNNs. The STFT + Deep CNN and CutCat data augmentation showed 75.81% accuracy with a kappa of 0.68 for a four-class (left/right hand, both feet and tongue) problem on DS5. Besides, the STFT + Shallow CNN + CutCat data augmentation acquired 78.44% accuracy with a kappa of 0.57 for a two-class (left/right hand) problem on DS6. Thanks to the favourable augmentation method for small-scale datasets, the improved MI-EEG decoding and implemented networks achieved good results compared to the state-of-the-art.

Table 3. Literature review of STFT-based TFIR approaches

Some other studies, e.g., [9, 12, 16, 25], have sought to increase the performance of MI-EEG systems using different STFT-based approaches. In these studies, STFT was generally used with different techniques, e.g., common spatial patterns or spectrum fusion methods. For instance, Han et al. [16] combined spatial filtering and frequency band extracting to create a new image form for the raw MI-EEG signal representation. They used STFT both to obtain MI-related frequency bands and describe MI-EEG signals. They applied STFT after a regularised common spatial pattern (RCSP) and proposed a new method called RCSP-STFT. It has the advantage of combining spatial projection and time–frequency analysis. As in previous studies, sub-images of μ and β bands were extracted, resized (using neighbour interpolation method), and combined vertically in this study. For a single trial, 30 representation images were calculated with the help of 30 pairs of two regularisation parameters. These images were fed to a parallel convolutional neural network (PCNN) architecture. In addition to the usual 2D kernel, 1D kernels for both frequency channel and time were added to optimise the PCNN and sufficiently capture features from the 2D images. Batch normalisation and dropout strategies were also engaged to optimise the network's performance. The proposed STFT + PCNN method yielded an average accuracy of 83.0% with a kappa of 0.66 on DS6 to classify left/right-hand tasks. It outperformed the state-of-the-art by at least 5.2% accuracy and 20.5% kappa improvements.

MI-EEG has high time resolution and frequency-spatial characteristics. Because of the nature of the MI-EEG, the features of α and β frequency band features might not fully be extracted by neural networks. Considering these and for automatic classification, Li et al. [25] constructed two key band image sequences containing temporal-spectral-spatial features. They constructed time-frequency images with STFT and then extracted sub-images of α and β bands. Afterwards, they fused the sub-images from all electrodes for α and β bands. Lastly, each band was arranged to the electrode coordinate using the nearest neighbour interpolation. In this paper, they further investigated the performance of different interpolation methods. They applied α and β band image sequences to a hybrid deep neural network, i.e., parallel multimodule CNN and long short-term memory network (PMMCL) network, to extract and fuse spatial-spectral and temporal features of key band image sequences. In the hybrid DNN, parallel multimodule CNNs were used to simultaneously acquire the global features of frequency band image sequences. LSTM was used to extract the temporal features among key band image sequences. A sliding window technique was also employed to expand the dataset. The proposed STFT + KBIM + PMMCL method showed 77.33% accuracy on DS5 for a four-class MI problem and 97.42% on DS7 for a five-class MI problem, better than the state-of-the-art. However, computational costs due to data augmentation and networking should be reduced. Keerthi Krishnan and Soman [12] aimed to generate spectrum images for preprocessing and inputting to CNNs. For this purpose, they decomposed MI-EEG signals into four variational mode decomposition (VMD) modes and applied STFT to each. Then, they combined the STFT of each VMD mode by stacking and formed the final spectrum image. This study used ConvNet, EEGNet, AlexNet, and LeNet CNN architectures. ConvNet and EEGNet were modified with three channel layers as the RGB spectrum images. AlexNet and LeNet adopted a network without layer modifications and fine-tuned these CNN architectures for spectral image input. The STFT + EEGNet + VMD framework showed average accuracies of 91.37% and 94.41% for the four-class problems on DS2, DS5, respectively, and 90.20% for the five-class problem on DS10. Furthermore, the STFT + DeepConvNet + VMD acquired an accuracy of 88.51% for a two-class problem on DS9. These results reveal the potential of VMD-STFT in the recognition of MI-EEG signals.

Shi et al. [9] investigated a BCI that uses monocular vision and MI-EEG for unmanned aerial vehicle (UAV) indoor space target searching. In this BCI, the navigation subsystem offers the precise, 3D-space-feasible flying direction to the decision subsystem. The decision subsystem first filtered raw MI-EEG signals with two fifth-order Butterworth band-pass filters. After that, common spatial patterns were applied to realise the spatial transformation. Then, spatially filtered signals were converted to images using stacked spectrograms and data augmentation. These spectrograms were inputted into a single convolutional layer CNN for feature extraction and classification. The proposed method yielded a calibration accuracy of 88.06% and an indoor 3D space target searching accuracy of 88.95% for a four-class (left/right hand, feet, and tongue) problem on DS11. The BCI system showed good adaptability and control stability for indoor 3D space target searching.

3.1.2 Wavelet-based methods

Wavelet-based methods are another means of extracting time-frequency patterns and converting MI-EEG signals to images. As in STFT-based methods, Wavelet-based TFIR has been constructed in different ways. In some studies, these images were given as input directly and without processing to the classifiers, while in others, some MI-related parts were extracted and fused. In most of them, electrode neighbouring information was considered, and topographic interpolation approaches were used. Chaudhary et al. [14] analysed non-stationary MI-EEG signals in time-frequency domains using CWT because of higher time-frequency resolution compared to the constant resolution of STFT. They applied CWT to MI-EEG signals, constructed scalograms, resized them to the size of 227×227×3, and used them as DCNN inputs. This paper also investigated a deep convolutional neural network (DCNN) based technique for classifying two-class tasks (right-hand and right-foot). A transfer learning strategy was employed to fine-tune the pre-trained Alexnet DCNN. The proposed STFT+CWT approach achieved 99.35% accuracy with a kappa of 0.99. In the same study, STFT showed slightly worse results. Collazos-Huertas et al. [19] created an input image set for CNNs. They calculated CWT coefficients and TFIR with the help of a complex Morlet wavelet. The representations were extracted within each time window, generating an image containing temporal, spectral, and spatial information. Besides, a topographic interpolation technique was used to project multi-channel data and preserve the spatial knowledge of electrodes. The performance was tested using MI-related different spectral bandwidths. Four combining scenarios of µ and β rhythms were further evaluated, showing that CWT is more suitable for the non-stationary data decomposition compared to power spectral density. In this paper, a CNN approach based on non-sequential Wide&Deep neural networks was proposed. A spatial dropping technique was also employed to eliminate the learning weights reflecting the not engaged localities. This paper focused on the improved interpolation of spatial neural patterns and acquired good performance values on DS5. The CNN architecture was tuned for discriminative MI, and the spatial dropping algorithm was evaluated. CWT without spatial dropping + CNN approach showed 87.6% accuracy with a kappa of 0.75 for a two-class problem, and CWT with spatial dropping + CNN showed 71.2% accuracy with a kappa of 0.56 for a three-class problem. The results showed that the CWT-based vectors were preferable for interpretation, learned weights were less sensitive to overtraining, and the CWT-based weights smoothly changed over time. Wei et al. [11] used a wavelet transform threshold denoising and a finite impulse response filter to eliminate extraneous signals and artefacts from MI-EEG recordings. They used CWT to create 2D images containing time, scale, and amplitude. They adopted the Morlet wavelet as the base function, which has the feature of equal variance in time and frequency, and employed the wavelet transform threshold denoising. Ultimately, they horizontally combined CWT images of the C3 and C4 electrodes. In experiments, a deep transfer learning strategy was developed for a small MI-EEG data set. The final layers of Alexnet were fine-tuned, and transfer learning was used to train the pre-trained AlexNet CNN. The experiments were conducted on DS1, and the proposed CWT + AlexNet approach achieved 96.43% accuracy with a kappa of 0.93 for the classification of the left/right hand MI tasks. Qiao et al. [20] offered a novel feature learning and preprocessing approach. They studied multi-channel MI-EEG signals and represented them while preserving spatial and temporal information. They used Morlet wavelet and cubic spline interpolation in preprocessing and constructed spectral frames. Lastly, they used an interpolation technique to transform each spectral irregularity map into a rectangle, which was given as input to the CNNs. They proposed a spatial-temporal hybrid deep learning model combining IncepCNN and bidirectional gated recurrent unit (BGRU) in classification. The proposed CNN structure was abbreviated as IncepCNN and used to learn and extract spatial features. After that, the output of IncepCNN was used as the input of BGRU to learn the temporal information existing in MI-EEG. The proposed method achieved 76.62% accuracy with a kappa of 0.69 for the classification four-class MI task on DS5, encouraging the problem of inter- and intra-class variations. Wei and Lin [17] sought to solve the problems of poor performance, low efficiency and weak robustness using a multi-dimensional fusion features-based classification. They used an improved Morlet wavelet algorithm to extract stable features from the frequency spectrum, especially for nonlinear and non-stationary signals. After applying Morlet, spectral power maps from 1 Hz to 40 Hz were obtained with frequency bands of 1 Hz. After that, a cubic spline interpolation method was used to convert each spectral map to the corresponding rectangular spectrum energy map. Finally, the entire EEG trial was sampled into time slices, and the temporal-spatial-frequency event-related spectral power features were extracted due to the changes in event-related desynchronisation/synchronisation over time. This paper used a three-dimensional convolutional neural network (3DCNNs) model to extract features. They were subsequently put into the bidirectional gated recurrent units (Bi-GRUs) models to extract the spatial-frequency-sequential multi-dimensional fusion features. The proposed Morlet Wavelet + 3DCNNs + Bi-GRUs method achieved 64.93% accuracy on DS4 for some two-class MI problems. The experiment was also conducted for action observation and action execution tasks. The proposed method also achieved stable classification results in dealing with these tasks.

3.1.3 Stockwell transform

Stockwell Transform (ST) is another method to transform MI-EEG signals from time to the frequency domain. Due to this, it has been used to convert 1D EEG signals to 2D images. However, there are very few studies with ST compared to STFT and CWT. For example, Alwasiti et al. [26] introduced a triplet network to categorise MI-EEG signals for the first time. Unlike the state-of-the-art approaches, in this paper, log-scaling provided better spectrograms and emphasised the μ and β rhythms more. Due to this, they used the log-scaled frequency range of 2-78 Hz in the TFIR. Besides, according to experiments, incorporating a larger frequency range improved the accuracy of deep metric learning. In this paper, ST was applied to each channel of single-epoch signals, and 64 spectrograms were generated, as shown in Figure 2.

2.png

Figure 2. (a) ST spectrograms of 64 electrodes placed according to the 10-10 system, (b) 512×512-sized fused image generated from all spectrograms [26]

After that, the 64 spectrograms were fused on a 512×512-sized rectangular topogram area. Finally, the image dataset was normalised using the mean and standard deviation of the pixel values. Using only a small training set, the authors developed a triplet deep metric network (DML) to solve a three-class problem (left/right hand or rest classes). Thanks to this network, the distance between the embeddings of different labelled images was increased while the distance between the embeddings of spectrograms of the same class was minimised. The experiments were conducted on DS7, which has 109 untrained subjects, and an average accuracy of 0.647 was achieved. It meant the DML could converge with and obtain good results even using a small number of training data. Furthermore, the Stockwell transform consistently outperformed STFT in almost all subjects. Due to these successful results, the techniques used with STFT and CWT for classifying MI-EEG signals should also be evaluated with Stockwell Transform. At the same time, more successful MI-EEG systems can be developed by combining the outputs of all these TFIR methods.

3.2 Other characteristic methods

Converting MI-EEG signals to images has not been carried out with only TFIR approaches. Unlike the TFIR methods discussed above, several studies have used different techniques. For example, the true electrode location information can be lost when converting MI-EEG signals to images, decreasing the classification performance. For this purpose, Li et al. [21] investigated the time domain power and Clough-Tocher interpolation-based imaging (TPCT) to construct 64x64-sized time-frequency-space features-based TPCT images, where each electrode's real positions were used to locate time-frequency characteristics. In this method, a fast Fourier transform (FFT) was first applied to each time window for each channel. The 8-30 Hz frequency range was divided into three sub-bands, i.e., 8-13 Hz, 13-21 Hz and 21-30 Hz, respectively. Then, an inverse FFT was applied, and the average power of the time domain was calculated independently for each of the three sub-bands. After that, a 64×64-sized grid system was established, time-frequency features were placed in a 2D space using the Clough-Tocher algorithm, and final MI-EEG input images were constructed, respectively. This study also modified the visual geometry group network (VGG) to MI-EEG BCIs and proposed a new deep CNN abbreviated mVGG. In this network, the convolution layer was used instead of max pooling, and average pooling was used instead of the fully connected layer for efficient recognition. The convolution kernel size was also modified, and the layers were deepened. The proposed TPCT + mVGG approach achieved 92.13% accuracy on DS5 for a four-class problem and 96.82% on DS6 for a two-class problem when evaluating with a 10-fold CV. The effectiveness of the imaging method, which had lower class skew and error costs, was also demonstrated by kappa values and ROC curves. Wang and Li [27] aimed to decode MI-EEG tasks using a deep parallel CNN (pDCNN) in an efficient way. They used 8-13Hz, 13-21 Hz and 21-30 Hz sub-bands by redividing μ and β bands. They divided the raw data into segments, transformed each time segment into a frequency domain using FFT, and calculated the average power of three sub-bands. They calculated the 2D positions of 64 3D electrode locations using an azimuthal equidistant projection and constructed three MI-EEG spatial-frequency images using the Clough-Tocher interpolation. In this paper, the authors aimed to overcome the insufficient spatial-frequency feature extraction for MI-EEG signals. Due to this, they extracted three groups of features corresponding to the three sub-bands and fused them using pDCNN. The methods were evaluated on DS7, including 109 subjects, and 90.25% average accuracy with a kappa value of 0.81 was achieved for classifying two-class MI tasks (open & close left or right fist).

Finding a reliable approach to support high-dimensional MI-EEG data with poor signal-to-noise ratios is a difficult problem. Xu et al. [22] investigated a new EEG topographical representations (ETR) approach to address this problem and efficiently learn brain activities. They produced ETR topologies that could be functional and correct for spatial location, temporal onset, and stability. The proposed ETR data structure was also designed to be useful for dimension reduction and reflecting intrinsic brain activity connections. In the last stage, ETR data-based spectral-spatial inputs were given to an ETR-based CNN (ETRCNN) learning framework. The performance of ETR + CNN was evaluated on DS5, and the framework achieved 87.66% accuracy (with a kappa of 0.82) for a four-class and 85.57% (with a kappa of 0.801) for a five-class problem. These results showed that the framework could accomplish multi-period and multi-object recognition. Yilmaz et al. [13] used an angle-amplitude transformation technique, a simple signal-to-image transformation approach for the MI-EEG and MEG signals and formed angle-amplitude graph (AAG) images. Then, they employed scale-invariant feature transform (SIFT)-based bag-of-words (BOW) features to extract image features. These features were then inputted into the k-nearest neighbour algorithm for classification. The experiments were conducted on DS2, and the proposed AAG + SIFT+ BoW + kNN approach showed an average accuracy of 97.99% for four-class and 96.50% for two-class MI problems. This study used a straightforward method for classification compared to other methods using deep learning.