Emotion recognition from EEG signals allows the direct assessment of the “inner” state of a user, which is considered an important factor in human-machine-interaction. Many methods for feature extraction have been studied and the selection of both appropriate features and electrode locations is usually based on neuro-scientific findings. Their suitability for emotion recognition, however, has been tested using a small amount of distinct feature sets and on different, usually small data sets. A major limitation is that no systematic comparison of features exists. Therefore, we review feature extraction methods for emotion recognition from EEG based on 33 studies. An experiment is conducted comparing these features using machine learning techniques for feature selection on a self recorded data set. Results are presented with respect to performance of different feature selection methods, usage of selected feature types, and selection of electrode locations. Features selected by multivariate methods slightly outperform univariate methods. Advanced feature extraction techniques are found to have advantages over commonly used spectral power bands. Results also suggest preference to locations over parietal and centro-parietal lobes.

Feature Extraction and Selection for Emotion Recognition from EEG

Emotions have an important role in daily life, not only in human interaction, but also in decision-making processes, and in the perception of the world around us. Due to the recent interest shown by the research community in establishing emotional interactions between humans and computers, the identification of the emotional state of the former became a need. This can be achieved through multiple measures, such as subjective self-reports, autonomic and neurophysiological measurements. In the last years, Electroencephalography (EEG) received considerable attention from researchers, since it can provide a simple, cheap, portable, and ease-to-use solution for identifying emotions. In this paper, we present a survey of the neurophysiological research performed from 2009 to 2016, providing a comprehensive overview of the existing works in emotion recognition using EEG signals. We focus our analysis in the main aspects involved in the recognition process (e.g., subjects, features extracted, classifiers), and compare the works per them. From this analysis, we propose a set of good practice recommendations that researchers must follow to achieve reproducible, replicable, well-validated and high-quality results. We intend this survey to be useful for the research community working on emotion recognition through EEG signals, and in particular for those entering this field of research, since it offers a structured starting point.

Emotions Recognition Using EEG Signals: A Survey

Thanks to the decreasing cost of whole-body sensing technology and its increasing reliability, there is an increasing interest in, and understanding of, the role played by body expressions as a powerful affective communication channel. The aim of this survey is to review the literature on affective body expression perception and recognition. One issue is whether there are universal aspects to affect expression perception and recognition models or if they are affected by human factors such as culture. Next, we discuss the difference between form and movement information as studies have shown that they are governed by separate pathways in the brain. We also review psychological studies that have investigated bodily configurations to evaluate if specific features can be identified that contribute to the recognition of specific affective states. The survey then turns to automatic affect recognition systems using body expressions as at least one input modality. The survey ends by raising open questions on data collecting, labeling, modeling, and setting benchmarks for comparing automatic recognition systems.

https://web4.cs.ucl.ac.uk/uclic/people/n.berthouze/KleinsmithBerthouze12.pdf

Affective Body Expression Perception and Recognition: A Survey

In this paper, we investigate stable patterns of electroencephalogram (EEG) over time for emotion recognition using a machine learning approach. Up to now, various findings of activated patterns associated with different emotions have been reported. However, their stability over time has not been fully investigated yet. In this paper, we focus on identifying EEG stability in emotion recognition. We systematically evaluate the performance of various popular feature extraction, feature selection, feature smoothing and pattern classification methods with the DEAP dataset and a newly developed dataset called SEED for this study. Discriminative Graph regularized Extreme Learning Machine with differential entropy features achieves the best average accuracies of 69.67 and 91.07 percent on the DEAP and SEED datasets, respectively. The experimental results indicate that stable patterns exhibit consistency across sessions; the lateral temporal areas activate more for positive emotions than negative emotions in beta and gamma bands; the neural patterns of neutral emotions have higher alpha responses at parietal and occipital sites; and for negative emotions, the neural patterns have significant higher delta responses at parietal and occipital sites and higher gamma responses at prefrontal sites. The performance of our emotion recognition models shows that the neural patterns are relatively stable within and between sessions.

Identifying Stable Patterns over Time for Emotion Recognition from EEG

In this paper, we present a multimodal emotion recognition framework called EmotionMeter that combines brain waves and eye movements To increase the feasibility and wearability of EmotionMeter in real-world applications, we design a six-electrode placement above the ears to collect electroencephalography (EEG) signals We combine EEG and eye movements for integrating the internal cognitive states and external subconscious behaviors of users to improve the recognition accuracy of EmotionMeter  The experimental results demonstrate that modality fusion with multimodal deep neural networks can significantly enhance the performance compared with a single modality, and the best mean accuracy of 8511% is achieved for four emotions (happy, sad, fear, and neutral) We explore the complementary characteristics of EEG and eye movements for their representational capacities and identify that EEG has the advantage of classifying happy emotion, whereas eye movements outperform EEG in recognizing fear emotion To investigate the stability of EmotionMeter over time, each subject performs the experiments three times on different days EmotionMeter obtains a mean recognition accuracy of 7239% across sessions with the six-electrode EEG and eye movement features These experimental results demonstrate the effectiveness of EmotionMeter within and between sessions

EmotionMeter: A Multimodal Framework for Recognizing Human Emotions

This study investigates recognition of affect in human walking as daily motion, in order to provide a means for affect recognition at distance. For this purpose, a data base of affective gait patterns from non-professional actors has been recorded with optical motion tracking. Principal Component Analysis (PCA), Kernel PCA (KPCA) and Linear Discriminant Analysis (LDA) are applied to kinematic parameters and compared for feature extraction. LDA in combination with Naive Bayes leads to an accuracy of 91% for person-dependent recognition of four discrete affective states based on observation of barely a single stride. Extra-success comparing to inter-individual recognition is twice as much. Furthermore, affective states which differ in arousal or dominance are better recognizable in walking. Though primary task of gait is locomotion, cues about a walker's affective state are recognizable with techniques from machine learning.

A comparison of PCA, KPCA and LDA for feature extraction to recognize affect in gait kinematics

This paper describes recognition of emotions of an unkown person during natural walking. As gait data is redundant, high dimensional and variable, effective feature extraction is essential. A combination of two consecutive Principal Component Analyses (PCA) and Fourier Transformation is used for data reduction. Naive Bayes, 1-Nearest Neighbor and Support Vector Machine (SVM) are compared for classification. Best accuracy is achieved for Naive Bayes with 72% for the four emotions sad, neutral, happy and angry within 1.65 seconds recording time. Slight increase in recognition accuracy is achieved, if the first PCA is replaced by Kernel PCA for SVM classification. The emotions angry and sad, both with high contrast in arousal, are better recognizable than neutral and happy. It is concluded that gait is capable to reveal the emotional state of a walker above chance level.

A Two-fold PCA-Approach for Inter-Individual Recognition of Emotions in Natural Walking

Emotion recognition from EEG signals allows the direct assessment of the “inner” state of the user which is considered an important factor in Human-Machine-Interaction. Given the vast amount of possible features from scalp recordings and the high variance between subjects, a major challenge is to select electrodes and features that separate classes well. In most cases, this decision is made based on neuro-scientific knowledge. We propose a statistically-motivated electrode/feature selection procedure, based on Cohen's effect size f2. We compare inter- and intra-individual selection on a self-recorded database. Classification is evaluated using quadratic discriminant analysis (QDA). We found both feature selection versions based on f2 yield comparable results. While highest accuracies up to 57,5% (5 classes) are reached by applying intra-individual selection, inter-individual analysis successfully finds features that perform with lower variance in recognition rates across subjects than combinations of electrodes/features suggested in literature.

Effect-size-based electrode and feature selection for emotion recognition from EEG

Early stages in the development of a Brain-and-Body-Computer Interface controlled robot avatar are presented. The robot is aimed at performing well-defined daily tasks upon the choice and on behalf of a user. We built on recent advances in neuroscience, robotics and machine learning to demonstrate that it is possible to control a robot, accurately and reliably, by decoding scalp-recorded non-invasive Electroencephalographic (EEG) potentials into user intentions.

Robert Jenke

Papers

Feature Extraction and Selection for Emotion Recognition from EEG

A comparison of PCA, KPCA and LDA for feature extraction to recognize affect in gait kinematics

A Two-fold PCA-Approach for Inter-Individual Recognition of Emotions in Natural Walking

Effect-size-based electrode and feature selection for emotion recognition from EEG

Towards robotic re-embodiment using a Brain-and-Body-Computer Interface