Emotion Recognition Based on EEG Signals Using Deep Learning Based on Bi-Directional Long Short-Term Memory and Attention Mechanism
Subject Areas : electrical and computer engineeringSeyyed Abed Hosseini 1 * , M. Houshmand 2
1 - 1Department of Electrical Engineering, Mashhad Branch, Islamic Azad University, Mashhad, Iran.
2 -
Keywords: Emotion recognition, bi-directional long short-term memory, electroencephalogram, attention mechanism, deep learning,
Abstract :
This research deals with the recognition of emotions from EEG signals using deep learning based on bi-directional long short-term memory (LSTM) and attention mechanism. In this study, two SEED and DEAP databases are utilized for the emotion recognition. The SEED database includes EEG signals in 62 channels from 15 participants in three categories of positive, neutral, and negative emotions. The DEAP dataset includes EEG signals in 32 channels from 32 participants in two categories of valence and arousal. LSTM has shown its efficiency in extracting temporal information from long physiological signals. The innovations of this research include the use of a new loss function and Bayesian optimizer to find the initial learning rate. The accuracy of the proposed method for the classification of emotions in the SEED database is 96.72%. The accuracy of the proposed method for classifying emotions into two categories of valence and arousal is 94.9% and 97.1%, respectively. Finally, comparing the obtained results with recent research studies.
[1] A. S. A. Hans and S. Rao, "A CNN-LSTM based deep neural networks for facial emotion detection in videos," International J. of Advances in Signal and Image Sciences, vol. 7, no. 1, pp. 11-20, Jun. 2021.
[2] L. Mou, et al., "Driver stress detection via multimodal fusion using attention-based CNN-LSTM," Expert Systems with Applications, vol. 173, Article ID: 114693, Jul. 2021.
[3] N. S. Suhaimi, J. Mountstephens, and J. Teo, "EEG-based emotion recognition: a state-of-the-art review of current trends and opportunities," Computational Intelligence and Neuroscience, vol. 2020, Article ID: 8875426, 16 Sept. 2020.
[4] Y. Kim, H. Lee, and E. M. Provost, "Deep learning for robust feature generation in audiovisual emotion recognition," in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 3687-3691, Vancouver, Canada, 26-31 May 2013.
[5] C. Herrando, J. Jiménez-Martínez, M. J. Martín-De Hoyos, and E. Constantinides, "Emotional contagion triggered by online consumer reviews: evidence from a neuroscience study," J. of Retailing and Consumer Services, vol. 67, Article ID: 102973, Jul. 2022.
[6] M. Ali, A. H. Mosa, F. Al Machot, and K. Kyamakya, "EEG-based emotion recognition approach for e-healthcare applications," in Proc. 8th Int. Conf. on Ubiquitous and Future Networks, pp. 946-950, Vienna, Austria, 5-8 Jul. 2016.
[7] S. A. Hosseini, M. A. Khalilzadeh, and S. Changiz, "Emotional stress recognition system for affective computing based on bio-signals," J. of Biological Systems, vol. 18, no. spec01, pp. 101-114, 2010.
[8] A. Sakalle, P. Tomar, H. Bhardwaj, D. Acharya, and A. Bhardwaj, "A LSTM based deep learning network for recognizing emotions using wireless brainwave driven system," Expert Systems with Applications, vol. 173, Article ID: 114516, Jul. 2021.
[9] C. Li, Z. Bao, L. Li, and Z. Zhao, "Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition," Information Processing & Management, vol. 57, no. 3, Article ID: 102185, May 2020.
[10] A. Bhattacharyya, R. K. Tripathy, L. Garg, and R. B. Pachori, "A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition," IEEE Sensors J., vol. 21, no. 3, pp. 3579-3591, Feb. 2021.
[11] Y. Luo, et al., "EEG-based emotion classification using spiking neural networks," IEEE Access, vol. 8, pp. 46007-46016, 2020.
[12] Y. Wang, et al., "EEG-based emotion recognition with prototype-based data representation," in Proc. 41st Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, pp. 684-689, Berlin, Germany, 23-27 Jul. 2019.
[13] M. M. Rahman, et al., "Recognition of human emotions using EEG signals: a review," Computers in Biology and Medicine, vol. 136 Article ID: 104696, Sept. 2021.
[14] J. Yang, X. Huang, H. Wu, and X. Yang, "EEG-based emotion classification based on bidirectional long short-term memory network," Procedia Computer Science, vol. 174, pp. 491-504, 2020.
[15] R. Andreasson, B. Alenljung, E. Billing, and R. Lowe, "Affective touch in human-robot interaction: conveying emotion to the nao robot," International J. of Social Robotics, vol. 10, no. 4, pp. 473-491, Dec. 2018.
[16] X. Wang, Y. Ren, Z. Luo, W. He, J. Hong, and Y. Huang, "Deep learning-based EEG emotion recognition: current trends and future perspectives," Frontiers in Psychology, vol. 14, Article ID: 1126994, Feb. 2023.
[17] M. K. Chowdary, J. Anitha, and D. J. Hemanth, "Emotion recognition from EEG signals using recurrent neural networks," Electronics, vol. 11, no. 15, Article ID: 2387, Jul. 2022.
[18] R. C. Dhingra and S. Ram Avtar Jaswal, "Emotion recognition based on EEG using DEAP dataset," European J. of Molecular & Clinical Medicine, vol. 8, no. 3, pp. 3509-3517, 2021.
[19] N. Zhuang, et al., "Emotion recognition from EEG signals using multidimensional information in EMD domain," BioMed Research International, vol. 2017, Article ID: 8317357, 2017.
[20] V. M. Joshi and R. B. Ghongade, "IDEA: intellect database for emotion analysis using EEG signal," J. of King Saud University-Computer and Information Sciences, vol. 34, no. 7, pp. 4433-4447, Jul. 2022.
[21] O. Atila and A. Şengür, "Attention guided 3D CNN-LSTM model for accurate speech based emotion recognition," Applied Acoustics, vol. 182, Article ID: 108260, Nov. 2021.
[22] X. Zheng and W. Chen, "An attention-based bi-LSTM method for visual object classification via EEG," Biomedical Signal Processing and Control, vol. 63, Article ID: 102174, Jan. 2021.
[23] D. Huang, et al., "Differences first in asymmetric brain: a bi-hemisphere discrepancy convolutional neural network for EEG emotion recognition," Neurocomputing, vol. 448, pp. 140-151, 11 Aug. 2021.
[24] S. Koelstra, et al., "Deap: a database for emotion analysis; using physiological signals," IEEE Trans. on Affective Computing, vol. 3, no. 1, pp. 18-31, Jun. 2011.
[25] M. Algarni, F. Saeed, T. Al-Hadhrami, F. Ghabban, and M. Al-Sarem, "Deep learning-based approach for emotion recognition using electroencephalography (EEG) signals using bi-directional long short-term memory (Bi-LSTM)," Sensors, vol. 22, no. 8, Article ID: 2976, Apr. 2022.
[26] W. L. Zheng and B. L. Lu, "Investigating critical frequency bands and channels for EEG-based emotion recognition with deep neural networks," IEEE Trans. on Autonomous Mental Development, vol. 7, no. 3, pp. 162-175, Sep. 2015.
[27] Q. Ma, M. Wang, L. Hu, L. Zhang, and Z. Hua, "A novel recurrent neural network to classify EEG signals for customers’ decision-making behavior prediction in brand extension scenario," Frontiers in Human Neuroscience, vol. 15, Article ID: 610890, Mar. 2021.
[28] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Computation, vol. 9, no. 8, pp. 1735-1780, Nov. 1997.
[29] S. Kumar, A. Sharma, and T. Tsunoda, "Brain wave classification using long short-term memory network based OPTICAL predictor," Scientific Reports, vol. 9, Article ID: 9153, Jun. 2019.
[30] M. Z. I. Ahmed and N. Sinha, "EEG-based emotion classification using LSTM under new paradigm," Biomedical Physics & Engineering Express, vol. 7, no. 6, Article ID: 065018, Sept. 2021.
[31] G. Liu and J. Guo, "Bidirectional LSTM with attention mechanism and convolutional layer for text classification," Neurocomputing, vol. 337, pp. 325-338, 14 Apr. 2019.
[32] J. C. Nunez, R. Cabido, J. J. Pantrigo, A. S. Montemayor, and J. F. Velez, "Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition," Pattern Recognition, vol. 76, pp. 80-94, Apr. 2018.
[33] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, "LSTM: a search space odyssey," IEEE Trans. on Neural Networks and Learning Systems, vol. 28, no. 10, pp. 2222-2232, Mar. 2015.
[34] T. Shen, et al., "Disan: directional self-attention network for rnn/cnn-free language understanding," in Proc. of the AAAI Conf. on Artificial Intelligence, pp. 5446-5455, New Orleans, LA, USA, 2-7 Feb. 2018.
[35] S. Mirsamadi, E. Barsoum, and C. Zhang, "Automatic speech emotion recognition using recurrent neural networks with local attention," in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 2227-2231, New Orleans, LA, USA, 5-9 Mar. 2017.
[36] R. Dutta and M. Majumder, "Attention-based bidirectional LSTM with embedding technique for classification of COVID-19 articles," Intelligent Decision Technologies, vol. 16, no. 1, pp. 205-215, Apr. 2022.