Multimodal Emotion Recognition Based on EEG and EOG Signals Evoked by the Video-Odor Stimuli

• Main Authors: Minchao Wu (Author), Wei Teng (Author), Cunhang Fan (Author), Shengbing Pei (Author), Ping Li (Author), Guanxiong Pei (Author), Taihao Li (Author), Wen Liang (Author), Zhao Lv (Author)
• Format: Book
• Published: IEEE, 2024-01-01T00:00:00Z.
• Subjects: article
• Online Access: Connect to this object online.
• Notes: 1534-4320; 1558-0210; 10.1109/TNSRE.2024.3457580
• Keywords: Electroencephalogram (EEG); electrooculogram (EOG); emotion recognition; video-odor stimuli; multi-modal fusion; Medical technology; R855-855.5; Therapeutics. Pharmacology; RM1-950
• Source: IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol 32, Pp 3496-3505 (2024)

Affective data is the basis of emotion recognition, which is mainly acquired through extrinsic elicitation. To investigate the enhancing effects of multi-sensory stimuli on emotion elicitation and emotion recognition, we designed an experimental paradigm involving visual, auditory, and olfactory senses. A multimodal emotional dataset (OVPD-II) that employed the video-only or video-odor patterns as the stimuli materials, and recorded the electroencephalogram (EEG) and electrooculogram (EOG) signals, was created. The feedback results reported by subjects after each trial demonstrated that the video-odor pattern outperformed the video-only pattern in evoking individuals’ emotions. To further validate the efficiency of the video-odor pattern, the transformer was employed to perform the emotion recognition task, where the highest accuracy reached 86.65% (66.12%) for EEG (EOG) modality with the video-odor pattern, which improved by 1.42% (3.43%) compared with the video-only pattern. What’s more, the hybrid fusion (HF) method combined with the transformer and joint training was developed to improve the performance of the emotion recognition task, which achieved classify accuracies of 89.50% and 88.47% for the video-odor and video-only patterns, respectively.