Identification of Imagery Based Affective States using Decision Level Fusion of Multimodal Physiological Signals

Khezri, Mahdi; Firoozabadi, Seyed Mohammad; Sharafat, Seyed Ahmad Reza

doi:10.22041/ijbme.2014.14703

Document Type : Full Research Paper

Authors

¹ Ph.D Student, Biomedical Engineering Department, Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran

² Professor, Medical Physics Department, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran

³ Professor, Electrical Engineering Department, Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran

10.22041/ijbme.2014.14703

Abstract

In this study, we propose decision level fusion of multimodal physiological signals to design an affect identification system using the MIT database. Four types of physiological signals, including blood volume pressure (BVP), respiration rate (RSP), skin conductance and facial muscles activities (fEMG) were utilized as affective modalities. To collect the above-mentioned database, researchers used personalized imagery to elicit the desired affective states from a single subject and recorded the corresponding physiological signals simultaneously. In this study, the best subset of features for each signal was determined using previously calculated time and frequency domain features. To this end, sequential floating forward selection (SFFS) and RELIEF feature selection algorithms were evaluated. A new feature set, formed by concatenating the selected features, was partitioned into three subsets. Each subset was then fed into a classifier to identify the desired affective states. The majority voting method was applied to fuse the results obtained by the subsystems. Three types of classification methods, namely SVM, LDA and KNN were evaluated to design an affect identification system. The results showed remarkable performance from the system in identifying the desired scenarios with an acceptable accuracy and speed of response. Using the RELIEF feature selection method, along with SVM as a classifier, an overall recognition accuracy of 93.8% was obtained, which is better than the results reported with the use of the above-mentioned database so far.

Keywords

Main Subjects

Biomedical Signal Processing / Medical Signal Processing / Biosignal Processing

References

[1] W. R. Picard, “Affective computing: challenges” Int J hum Comput Stud-Application of affective computing in human-Computer interaction 59, 55-64, 2003.

[2] P. R. Kleinginna, A. M. Kleinginna, “A categorized list of emotion definitions with suggestions for a consensual definition” Motiv Emot, 5 (4), 345-379, 1981.

[3] C. Muhl. Koelstra, M. Soleymani, J. S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras, “DEAP: A Database for Emotion Analysis using Physiological Signals” IEEE Trans Affect Comput 3, 18-31, 2001.

[4] J. A. Russell, L. F. Barrett, “Core affect, prototypical emotional episodes, and other things called emotion: Dissecting the elephant”, J Pers Soc Psychol, 76 (5): 805-819, 1999.

[5] R. W. Picard, “Affective Computing” the MIT Press, 1997.

[6] S. D’Mello, S. Craig, B. Gholson, S. Franklin, R. Picard, A. Graesser, “Integrating Affect Sensors in an Intelligent Tutoring System” Proc Computer in the Affective Loop Workshop Int Conf Intell User Interfac, 7-13, 2005.

[7] I. M. Rezazadeh, S. M. P. Firoozabadi, H. Hu, SMP. H. Golpayegani, “Co-Adaptive and Affective Human-Machine Interface for Improving Training Performances of Virtual Myoelectric Forearm Prosthesis” IEEE Trans Affect Comput3, 285-297, 2012.

[8] P. Ekman, W. V. Friesen, “Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion” J Pers Soc Psychol 53, 712-714, 1987.

[9] P. J. Lang, “the Emotion Probe: Studies of Motivation and Attention” Am Psychol 50, 372-385, 1995.

[10] M. M. Bradley, P. J. Lang, “Measuring emotion: the self-assessment mankin and the semantic differential” J Behav Ther Exp Psychiatr 25, 49-59, 1994.

[11] C. E. Izard, “Basic emotions, relations among emotions and emotion-cognition relations”Psychol Rev 99 (3), 561-565, 1992.

[12] K. Kim, S. Bang, S. Kim, “Emotion recognition system using short term monitoring of physiological signals” Med Biol Eng Comput 42, 419-427, 2004.

[13] J. Kim, E. Andre, “Emotion Recognition Based on Physiological Changes in Music Listening” IEEE Trans Pattern Anal Mach Intell 30, 2067-2083, 2008.

[14] E. Vyzas, R. W. Picard, “Affective pattern classification” Emot Intell: The Tangled Knot of Cognition 176-182, 1998.

[15] R. W. Picard, E. Vyzas, J. Healey, “Toward Machine Emotional Intelligence: Analysis of Affective Physiological State” IEEE Trans Pattern Anal Mach Intell 23 (10), 1175-1191, 2001.

[16] R. Cowie, E. Douglas Cowie, “Emotion recognition in human-computer interaction” IEEE Signal Process Mag, 18 (1), 32-80, 2001.

[17] Z. Khalili, M. H. Moradi, “Emotion recognition system using brain and peripheral signals: Using Correlation Dimension to Improve the Results of EEG” Proc Int Joint Conf Neural Network, Georgia, USA, 1571-1575, 2009.

[18] F. Nasoz, K. Alvarez, “Emotion recognition from physiological signals using wireless sensors for presence technologies” Cognit Tech Work, 6 (1), 4-14, 2004.

[19] M. Naji, S. M. P. Firoozabadi, P. Azadfallah, “Classification of Music-Induced Emotions Based on Information Fusion of Forehead Biosignals and Electrocardiogram” Cogn Comput, Springer, 6, 241-252, 2014.

[20] Z. Long, G. Liu, X. Dai, “Extracting Emotional Features from ECG by Using Wavelet Transform” Int Conf Biomed Eng Comput Sci (ICBECS), 1-4, 2010.

[21] G. Rigas, C. D. Katsis, G. Ganiatsas, D. I. Fotiadis, “A User Independent, Biosignal Based, Emotion Recognition Method” Proc int conf User Model Corfu, Greece, 2007.

[22] E. Andre, R. L. Dybkjae, W. Minker, P. Heisterkamp, A. Haag, S. Goronzy, P. Schaich, J. Williams, “Emotion Recognition Using Biosensors: First Steps towards an Automatic System” Affect Dialog Syst, 3068, 36-48, 2004.

[23] Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, J. H. Chen, “EEG based emotion recognition in music listening” IEEE Trans Biomed Eng, 57: 1798-1806, 2010.

[24] G. Chanel, J. J. M. Kierkels, M.Soleymani, T. Pun, “Short-term emotion assessment in a recall paradigm” Int J Hum Comput Stud, 67, 607-627, 2009.

[25] Z. Zeng, M. Pantic, G. I. Roisman, T. S. Huang, “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions” IEEE Trans Pattern Anal Mach Intell, 31 (1), 39-58, 2009.

[26] G. Castellano, L. Kessous, G. Caridakis, “Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech” Affect Emot Hum Comput Interact, Springer, 92-103, 2008.

[27] S. D’Mello, A. Graesser, “Multimodal Semi-Automated Affect Detection from Conversational Cues, Gross Body Language, and Facial Features” User Modeling and User-Adapted Interaction, 10, 147-187, 2010.

[28] F. Alkoot, J. Kittler, “Experimental Evaluation of Expert Fusion Strategies” Pattern Recogn Lett, 20, 1361–1369, 1999.

[29] J. Kittler, M. Hatef, R. Duin, J. Matas, “On combining classifiers” IEEE Trans Pattern Anal Mach Intell, 20 (3), 226-239, 1998.

[30] L. I. Kuncheva, “A theoretical study on six classifier fusion strategies” IEEE Trans Pattern Anal Mach Intell, 24 (2), 281-286, 2002.

[31] http://affect.media.mit.edu/share-data.php.

[32] http://www.ai.mit.edu/projects/humanoid-robotics-group/kismet/kismet.html.

[33] C. D. Katsis, N. Katertsidis, G. Ganiatsas, D. I. Fotiadis, “Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach” IEEE Trans Syst Man Cybern Syst Hum, 38, 502-512, 2008.

[34] F. Honig, J. Wagner, A. Batliner, E. Noth, “Classification of user states with physiological signals: On-line Generic features vs. Specialized feature sets” Euro Signal Process Conf, 2357-2361, 2009.

[35] B. Cheng, G. Liu, “Emotion Recognition from Surface EMG Signal Using Wavelet Transform and Neural Network” Int Conf Bioinformatics Biomed Eng, Shanghai, China, 1363-1366, 2008.

[36] Z. Cong, M. Chetouani, “Hilbert-Huang transform based physiological signals analysis for emotion recognition” Int Symp Signal Process Inf Technol (ISSPIT), 334-339, 2009.

[37] P. C. Petrantonakis, L. J. Hadjileontiadis, “Emotion Recognition from EEG Using Higher Order Crossings” IEEE Trans Inform Tech Biomed, 14 (2), 186-197, 2010.

[38] C. A. Frantzidis, C.Bratsas, C. L Papadelis, E. Konstantinidis, C. Pappas, P. D. Bamidis, “Toward Emotion Aware Computing: An Integrated Approach Using Multichannel Neurophysiological Recordings and Affective Visual Stimuli”, IEEE Trans Inform Tech Biomed, 14 (3), 589-597, 2010.

[39] P. Ekman, W. Friesen, “Facial Action Coding System: A Technique for the Measurement of Facial Movement” Investigator’s Guide 2 Parts Consult Psychol Press, 1978.

[40] E. Mower, M. J. Mataric, S. Narayanan, “A Framework for Automatic Human Emotion Classification Using Emotion Profiles” IEEE Trans Audio Speech Lang Process, 9 (5), 1057-1070, 2011.

[41] I. Arroyo, D. G. Cooper, W. Burleson, B. P. Woolf, K. Muldner, R. Christopherson, “Emotion Sensors Go to School” Proc Conf Artif Intell Educ, 17-24, 2009.

[42] P. Y. Oudeyer, “The production and recognition of emotions in speech: features and algorithm” Int J Hum Comput Stud, 59 (1-2), 157-183, 2003.

[43] C. H. Wu, W. B. Liang, “Emotion Recognition of Affective Speech Based on Multiple Classifiers Using Acoustic-Prosodic Information and Semantic Labels” IEEE Trans Affect Comput, 2 (1), 10-21, 2011.

[44] M. Coulson, “Attributing Emotion to Static Body Postures: Recognition Accuracy, Confusions, and Viewpoint Dependence” Nonverbal Behav Jour, 28, 117-139, 2004.

[45] S. Mota, R. Picard, “Automated Posture Analysis for Detecting Learner’s Interest Level” Proc Computer Vision and Pattern Recognition Workshop, 5, 49, 2003.

[46] S. D’Mello, A. Graesser, “Automatic Detection of Learner’s Affect from Gross Body Language” Appl Artif Intell, 23: 123-150, 2009.

[47] D. Glowinski, N. Dael, A. Camurri, G. Volpe, M. Mortillaro, K. Scherer, “Towards a Minimal Representation of Affective Gestures” IEEE Trans Affect Comput, 2 (2), 106-118, 2011.

[48] W. B. Cannon, “The James-Lang Theory of Emotions: A Critical Examination and an Alternative Theory” Am Psychologist, 39, 106-124, 1927.

[49] P. Ekman, R. W. Levenson, “Autonomic nervous system activity distinguishes among emotions” Science, 221 (4616), 1208-1210, 1983.

[50] H. Liu, H. Motoda, “Feature Selection for Knowledge Discovery and Data Mining” Boston: Kluwer Academic Publishers, 1998.

[51] P. Pudil, J. Novovicova, J. Kittler, “Floating Search Methods in Feature Selection” Pattern Recogn Lett, 15, 1119-1125, 1994.

[52] K. Kira, L. A. Rendell, “A practical approach to feature selection” Proc Int Conf Mach Learn, 249 –256, 1992.

[53] M. Dash, H. Liu, “Feature selection for classifications” Intell Data Anal Int J, 1, 131-156, 1997.

[54] I. Kononenko, “Estimating attributes: Analysis and extension of RELIEF” Proc Euro Conf Mach Learn, 171–182, 1994.

[55] C. W. Hsu, C. C. Chang, C. J. Lin, “A practical guide to support vector classification” http://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf, 2009.

[56] J. Li, “Linear Discriminant Analysis; Department of Statistics” The Pennsylvania State University, http://www.stat.psu.edu/_jiali.

[57] J. Kim, E. Andre, “Emotion-specific Dichotomous Classification and Feature-level Fusion of Multichannel Biosignals for Automatic Emotion Recognition” Proc IEEE Int Conf Multisensor Fusion and Integration for Intelligent Systems, Korea, 20–22, 2008.

[58] M. Paleari, C. L. Lisetti, “Toward Multimodal Fusion of Affective Cues” Proc Workshop on Human Centered Multimedia at ACM Multimedia, California, 99-108, 2006.

[59] A. K. Akobeng, “Understanding diagnostic tests 1: sensitivity, specificity and predictive values” Acta Pædiatrica, 96, 338–341, 2006.

Iranian Journal of Biomedical Engineering

Identification of Imagery Based Affective States using Decision Level Fusion of Multimodal Physiological Signals

References

References

Volume 8, Issue 4 - Serial Number 4
February 2015
Pages 339-358

Identification of Imagery Based Affective States using Decision Level Fusion of Multimodal Physiological Signals

References

References

Volume 8, Issue 4 - Serial Number 4February 2015Pages 339-358

Volume 8, Issue 4 - Serial Number 4
February 2015
Pages 339-358