EEG Emotion Recognition using Deep Neural Network (DNN) in Virtual Reality Environments
DOI:
https://doi.org/10.24076/intechnojournal.2024v6i2.1903Keywords:
EEG, DNN, VR, Brain Signal, Emotion ClassificationAbstract
Purpose: The purpose of this study is to explore the integration of EEG technology with virtual reality (VR) systems to enhance therapeutic interventions, improve cognitive state recognition, and develop personalized immersive experiences. Specifically, it investigates the classification of EEG signals in a VR environment using machine learning models and identifies the most effective methods for individual-level analysis.
Methods: The study utilized EEG data collected from 31 participants using the Muse 2016 headset, with electrodes positioned according to the 10-20 international system. EEG signals were analyzed for features such as statistical metrics (mean, median, standard deviation, skewness, and kurtosis) and Hjorth parameters (activity, mobility, complexity). Machine learning models, including K-Nearest Neighbors (KNN), Random Forest (RF), and Support Vector Machine (SVM), were evaluated for their performance in classifying emotional and cognitive states in a VR environment.
Result: The results indicate that the Deep Neural Network (DNN) outperformed SVM and KNN models, achieving the highest average classification accuracy. SVM demonstrated consistent performance, with accuracy values consistently above 0.8 across subjects, while KNN showed greater variability and lower overall performance. DNN's architecture, incorporating two hidden layers with ReLU activation and a softmax output layer, demonstrated superior capability in modeling complex EEG patterns. The findings emphasize the effectiveness of DNN in handling high-dimensional and non-linear data, particularly for multi-class classification tasks.
Novelty: This study is novel in its focus on personalized machine learning model performance in a VR-EEG setup. Instead of a one-size-fits-all approach, it emphasizes individualized analysis, identifying the most effective model for each participant.
Downloads
References
J. GomezRomero-Borquez, C. Del-Valle-Soto, J. A. Del-Puerto-Flores, R. A. Briseño, and J. Varela-Aldás, “Neurogaming in Virtual Reality: A Review of Video Game Genres and Cognitive Impact,” Electronics, vol. 13, no. 9, p. 1683, Apr. 2024, doi: 10.3390/electronics13091683.
C. Tacca, B. A. Kerr, C. McLamb, K. L. Ridgway, and E. A. Friis, “Efficacy of a remote virtual reality and EEG enabled psychotherapy system for the treatment of depressive symptoms,” Front. Virtual Real., vol. 5, May 2024, doi: 10.3389/frvir.2024.1281017.
S. Cheng, D. Sheng, Y. Gao, Z. Dong, and T. Han, “Enhancing Positive Emotions through Interactive Virtual Reality Experiences: An EEG-Based Investigation,” in 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), Mar. 2024, pp. 189–199, doi: 10.1109/VR58804.2024.00042.
J. GomezRomero-Borquez, C. Del-Valle-Soto, J. A. Del-Puerto-Flores, F. R. Castillo-Soria, and F. M. Maciel-Barboza, “Implications for Serious Game Design: Quantification of Cognitive Stimulation in Virtual Reality Puzzle Games through MSC and SpEn EEG Analysis,” Electronics, vol. 13, no. 11, p. 2017, May 2024, doi: 10.3390/electronics13112017.
W. Ma, Z. Li, T. Li, Y. Zheng, and L. Wang, “P?4.35: Application of Virtual Reality Technology in the Diagnosis and Treatment of Psychological Disorders: An Electroencephalography (EEG)?Based Approach,” SID Symp. Dig. Tech. Pap., vol. 55, no. S1, pp. 882–886, Apr. 2024, doi: 10.1002/sdtp.17228.
S. Sargolzaei, S. Hatchett, M. Okashita, and A. Sargolzaei, “Decoding Driving Neural Dynamics: An Exploratory Survey of EEG Applications in Unraveling Cognitive Processes in Virtual Reality Environments,” in SoutheastCon 2024, Mar. 2024, pp. 830–836, doi: 10.1109/SoutheastCon52093.2024.10500163.
Y. Bai, M. Yu, and Y. Li, “Dynamic Neural Patterns of Human Emotions in Virtual Reality: Insights from EEG Microstate Analysis,” Brain Sci., vol. 14, no. 2, p. 113, Jan. 2024, doi: 10.3390/brainsci14020113.
P. Wang, X. Zhang, X. Ai, and S. Wang, “Modulation of EEG Signals by Visual and Auditory Distractors in Virtual Reality-Based Continuous Performance Tests,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 32, pp. 2049–2059, 2024, doi: 10.1109/TNSRE.2024.3405549.
Y.-T. Cho, N. Yoshimura, and L. A. Martinez-Tejada, “Decoding Locomotion intention in Virtual Reality Using EEG,” in 2023 15th Biomedical Engineering International Conference (BMEiCON), Oct. 2023, pp. 1–5, doi: 10.1109/BMEiCON60347.2023.10321978.
H. Tadayyoni, M. S. Ramirez Campos, A. J. U. Quevedo, and B. A. Murphy, “Biomarkers of Immersion in Virtual Reality Based on Features Extracted from the EEG Signals: A Machine Learning Approach,” Brain Sci., vol. 14, no. 5, p. 470, May 2024, doi: 10.3390/brainsci14050470.
H. Ren, Z. Zheng, J. Zhang, Q. Wang, and Y. Wang, “Electroencephalography (EEG)-Based Comfort Evaluation of Free-Form and Regular-Form Landscapes in Virtual Reality,” Appl. Sci., vol. 14, no. 2, p. 933, Jan. 2024, doi: 10.3390/app14020933.
H. Liang, S. Liu, Y. Wang, J. Pan, J. Fu, and Y. Yuan, “EEG-Based VR Scene Adaptive Generation System for Regulating Emotion,” in 2023 9th International Conference on Virtual Reality (ICVR), May 2023, pp. 361–368, doi: 10.1109/ICVR57957.2023.10169325.
G. R. F. Suwandi, S. N. Khotimah, F. Haryanto, and - Suprijadi, “Electroencephalography Signal Analysis for Virtual Reality Sickness: Head-mounted Display and Screen-based,” Int. J. Adv. Sci. Eng. Inf. Technol., vol. 13, no. 4, pp. 1449–1455, Aug. 2023, doi: 10.18517/ijaseit.13.4.18165.
M. Korkiakoski et al., “Exploring the Potential of EEG for Real-Time Interactions in Immersive Virtual Reality,” WSEAS Trans. Inf. Sci. Appl., vol. 20, pp. 98–108, Mar. 2023, doi: 10.37394/23209.2023.20.12.
A. Apicella et al., “Electroencephalography correlates of fear of heights in a virtual reality environment,” Acta IMEKO, vol. 12, no. 2, pp. 1–7, May 2023, doi: 10.21014/actaimeko.v12i2.1457.
G. Pei, C. Fan, T. Li, J. Jin, and R. Wang, “EEG-based Emotion Recognition in Immersive Virtual Reality: Meeting the Requirement of Accuracy and Computational Efficiency,” in Proceedings of the 2023 7th International Conference on Digital Signal Processing, Feb. 2023, pp. 8–15, doi: 10.1145/3585542.3585544.
N. S. Suhaimi, J. Mountstephens, and J. Teo, “A Dataset for Emotion Recognition Using Virtual Reality and EEG (DER-VREEG): Emotional State Classification Using Low-Cost Wearable VR-EEG Headsets,” Big Data Cogn. Comput., vol. 6, no. 1, 2022, doi: 10.3390/bdcc6010016.
R. Jenke, S. Member, A. Peer, and M. Buss, “Feature Extraction and Selection for Emotion Recognition from EEG,” IEEE Trans. Affect. Comput., vol. 5, no. 3, pp. 327–339, Jul. 2014, doi: 10.1109/TAFFC.2014.2339834.
B. Hjorth, “EEG analysis based on time domain properties,” Electroencephalogr. Clin. Neurophysiol., vol. 29, no. 3, pp. 306–310, Sep. 1970, doi: 10.1016/0013-4694(70)90143-4.
J. Atkinson and D. Campos, “Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers,” Expert Syst. Appl., vol. 47, pp. 35–41, 2016, doi: 10.1016/j.eswa.2015.10.049.
H. He, Y. Tan, J. Ying, and W. Zhang, “Strengthen EEG-based emotion recognition using firefly integrated optimization algorithm,” Appl. Soft Comput. J., vol. 94, p. 106426, 2020, doi: 10.1016/j.asoc.2020.106426.
