A Comprehensive study of Geometric and Appearance based Facial Expression Recognition Methods

Authors(2) :-Nazeerah A. Sheth, Mahesh M. Goyani

It is a well-known fact that facial expressions are one of the key reflectors of the emotional state of a person and the research on the same has been spanning for a long time. Being an essential requirement in Human Computer Interaction as well as other applications such as automobile safety, mental health detection, animations, etc. recognizing facial expressions with precision has become vital. This paper presents a survey on various important and effective techniques present in literature along with their variations used recently. Prominent techniques of each step and a detailed discussion on feature extraction methods have been provided along with a detailed comparison of few recent approaches.

Authors and Affiliations

Nazeerah A. Sheth

Mahesh M. Goyani

Facial Expressions, Survey, Emotion Recognition

  1. A. Mehrabian, "Communication without words," Psychology Today, vol. 2. pp. 53-55, 1968.
  2. F. Cid, J. A. Prado, P. Bustos, and P. Nunez, "A Real Time and Robust Facial Expression Recognition and Imitation approach for Affective Human-Robot Interaction Using Gabor filtering," IEEE International Conference on Intelligent Robots and Systems, pp. 2188-2193, 2013.
  3. W. Gaebel and W. Wolwer, "Facial expression and emotional face recognition in schizophrenia and depression," European Archives of Psychiatry and Clinical Neuroscience, vol. 242, no. 1, pp. 46-52, 1992.
  4. F. Nasoz, K. Alvarez, C. L. Lisetti, and N. Finkelstein, "Emotion recognition from physiological signals using wireless sensors for presence technologies," Cognition, Technology & Work, vol. 6, no. 1, pp. 4-14, 2004.
  5. A. Butalia, M. Ingle, and P. Kulkarni, "Facial Expression Recognition for Security," International Journal of Modern Engineering Research, vol. 2, no. 4, pp. 1449-1453, 2012.
  6. M. S. Bartlett, G. Littlewort, I. Fasel, and J. R. Movellan, "Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction," IEEE Conference on Computer Vision and Pattern Recognition Workshop, vol. 5, pp. 53-53, 2003.
  7. R. Cowie et al., "Emotion recognition in human-computer interaction," Signal Processing Magazine, IEEE, vol. 18, no. 1, pp. 32-80, 2001.
  8. B. Fasel and J. Luettin, "Automatic facial expression analysis: A survey," Pattern Recognition, vol. 36, no. 1, pp. 259-275, 2003.
  9. V. Bettadapura, "Face Expression Recognition and Analysis : The State of the Art," arXiv preprint arXiv:1203.6722, pp. 1-27, 2012.
  10. Charles Darwin, The expression of the emotions in man and animals. Oxford University Press, USA, 1998.
  11. P. Ekman, "Universal and Cultural differences in facial expressions of emotions," Nebraska Symposium on Motivation, vol. 19. pp. 207-282, 1971.
  12. M. J. Lyons, "Automatic classification of single facial images," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 12, pp. 1357-1362, 1999.
  13. P. Ekman, "Strong Evidence for universals in facial expressions," Psychol. Bull, vol. 115, no. 2. pp. 268-287, 1994.
  14. S. Deshmukh, M. Patwardhan, and A. Mahajan, "Survey on real-time facial expression recognition techniques," IET Biometrics, vol. 5, no. 3, pp. 243-251, 2016.
  15. C. A. Corneanu, M. O. Simón, J. F. Cohn, and S. E. Guerrero, "Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 8, pp. 1548-1568, 2016.
  16. E. Sariyanidi, H. Gunes, and A. Cavallaro, "Automatic analysis of facial affect: A survey of registration, representation, and recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 6, pp. 1-22, 2015.
  17. M. Pantic and L. J. M. Rothkrantz, "Automatic analysis of facial expressions : the state of the art," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424-1445, 2000.
  18. M. Goyani, "A Comprehensive Study on Texture Feature Based Facial Expression Recognition," 2017.
  19. D. Huang, C. Shan, M. Ardebilian, and L. Chen, "Facial Image Analysis Based on Local Binary Patterns : A Survey," IEEE Transactions on Image Processing, 2011.
  20. R. P. Holder and J. R. Tapamo, "Improved gradient local ternary patterns for facial expression recognition," EURASIP Journal on Image and Video Processing, vol. 2017, no. 1, pp. 42-57, 2017.
  21. B. Martinez, M. F. Valstar, B. Jiang, and M. Pantic, "Automatic Analysis of Facial Actions : A Survey," IEEE Transactions on Affective Computing, 2017.
  22. S. K. A. Kamarol, M. H. Jaward, J. Parkkinen, and R. Parthiban, "Spatiotemporal feature extraction for facial expression recognition," IET Image Processing, vol. 10, no. 7, pp. 534-541, 2016.
  23. Y. Sun and J. Yu, "Facial Expression Recognition by Fusing Gabor and Local Binary Pattern Features," International Conference on Multimedia modeling. Springer, pp. 209-220, 2017.
  24. T. Yu and X. Gu, "Facial Expression Recognition Using Double-Stage Sample-Selected SVM," International Conference on Intelligent Computing, Springer, pp. 304-315, 2017.
  25. M. H. Siddiqi, R. Ali, A. M. Khan, Y. T. Park, and S. Lee, "Human Facial Expression Recognition Using Stepwise Linear Discriminant Analysis and Hidden Conditional Random Fields," IEEE Transactions on Image Processing, vol. 24, no. 4, pp. 1386-1398, 2015.
  26. B. Ryu, A. Ramirez Rivera, J. Kim, and O. Chae, "Local Directional Ternary Pattern for Facial Expression Recognition," IEEE Transactions on Image Processing, vol. 26, no. 12, pp. 6006-6018, 2017.
  27. M. Liu, S. Shan, R. Wang, and X. Chen, "Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition," in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2014, pp. 1749-1756.
  28. D. Ghimire, S. Jeong, J. Lee, and S. H. Park, "Facial expression recognition based on local region specific features and support vector machines," Multimedia Tools and Applications, vol. 76, no. 6, pp. 1-19, 2016.
  29. A. Uçar, Y. Demir, and C. Güzeliş, "A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering," Neural Computing and Applications, vol. 27, no. 1, pp. 131-142, 2016.
  30. A. Mollahosseini, D. Chan, and M. H. Mahoor, "Going deeper in facial expression recognition using deep neural networks," IEEE Winter Conference on Applications of Computer Vision, pp. 1-10, 2016.
  31. S. Arshid, A. Hussain, A. Munir, A. Nawaz, and S. Aziz, "Multi-stage binary patterns for facial expression recognition in real world," Cluster Computing, pp. 1-9, 2017.
  32. Y. Luo, T. Zhang, and Y. Zhang, "A novel fusion method of PCA and LDP for facial expression feature extraction," International Journal for Light and Electron Optics, vol. 127, no. 2, pp. 718-721, 2016.
  33. P. Ekman and W. Friesen, Manual for the Facial Action Coding System. Consulting Psychologist Press, Palo Alto, 1977.
  34. D. Acevedo, P. Negri, M. E. Buemi, F. G. Fernandez, and M. Mejail, "A Simple Geometric-Based Descriptor for Facial Expression Recognition," 12th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 802-808, 2017.
  35. D. Ghimire, J. Lee, Z. N. Li, and S. Jeong, "Recognition of facial expressions based on salient geometric features and support vector machines," Multimedia Tools and Applications, vol. 76, no. 6, pp. 7921-7946, 2016.
  36. S. Berretti, B. Ben Amor, M. Daoudi, and A. Del Bimbo, "3D facial expression recognition using SIFT descriptors of automatically detected keypoints," Visual Computer, vol. 27, no. 11, pp. 1021-1036, 2011.
  37. W. Zeng, H. Li, L. Chen, J.-M. Morvan, and X. D. Gu, "An automatic 3D expression recognition framework based on sparse representation of conformal images," 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1-8, 2013.
  38. P. Lemaire, M. Ardabilian, L. Chen, and M. Daoudi, "Fully automatic 3D facial expression recognition using differential mean curvature maps and histograms of oriented gradients," 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1-7, 2013.
  39. Dahmane, Mohamed, and J. Meunier, "Emotion recognition using dynamic gridbased HoG features," in IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, 2011, pp. 884-888.
  40. Lowe and G. David, "Distinctive image features from scale-invariant key points," International Journal of Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
  41. T. Ojala, M. Pietikäinen, and D. Harwood, "A comparative study of texture measures with classification based on featured distributions," Pattern Recognition, vol. 29, no. 1, pp. 51-59, 1996.
  42. T. Ahonen, A. Hadid, and M. Pietikäinen, "Face description with local binary patterns: application to face recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 12, pp. 2037-2041, 2006.
  43. T. Ojala, M. Pietikäinen, and T. Mäenpää, "A generalized local binary pattern operator for multiresolution gray scale and rotation invariant texture classification," Advances in Pattern Recognition, pp. 399-408, 2001.
  44. C. Shan, S. Gong, and P. W. Mcowan, "Facial expression recognition based on Local Binary Patterns : A comprehensive study," Image and Vision Computing, vol. 27, no. 6, pp. 803-816, 2009.
  45. C. Shan, S. Gong, and P. W. McOwan, "Robust facial expression recognition using local binary patterns," IEEE International Conference on Image Processing, vol. 2, pp. 170-173, 2005.
  46. S. Moore and R. Bowden, "Local binary patterns for multi-view facial expression recognition," Computer Vision and Image Understanding, vol. 115, no. 4, pp. 541-558, 2011.
  47. R. Hablani, N. Chaudhari, and S. Tanwani, "Recognition of facial expressions using Local Binary Patterns of important facial parts," International Journal of Image Processing, vol. 7, no. 2, pp. 163-170, 2013.
  48. X. Zhao and S. Zhang, "Facial expression recognition using local binary patterns and discriminant kernel locally linear embedding," EURASIP Journal on Advances in Signal Processing, vol. 2012, no. 1, p. 20, 2012.
  49. D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen, "Local Binary Patterns and Its Application to Facial Image Analysis: A Survey," IEEE Transactions on Systems Man and Cybernetics Part C-Applications and Reviews, vol. 41, no. 6, pp. 765-781, 2011.
  50. H. Jin, Q. Liu, H. Lu, and X. Tong, "Face detection using improved LBP under Bayesian framework," in International Conference on Image and Graphics, 2004, pp. 306-309.
  51. T. Jabid, M. H. Kabir, and O. Chae, "Facial expression recognition using Local Directional Pattern (LDP)," 17th IEEE International Conference on Image Processing, pp. 1605-1608, 2010.
  52. F. Ahmed and M. H. Kabir, "Directional ternary pattern (dtp) for facial expression recognition," in IEEE International Conference on Consumer Electronics, 2012, pp. 265-266.
  53. B. Jun and D. Kim, "Robust face detection using local gradient patterns and evidence accumulation," Pattern Recognition, vol. 45, no. 9, pp. 3304-3316, 2012.
  54. H. Kabir, T. Jabid, and O. Chae, "Local Directional Pattern Variance (LDPv):Robust Feature Descriptor for Facial Expression Recognition," The International Arab Journal of Information Technology, vol. 9, no. 4, pp. 382-392, 2012.
  55. X. Tan and B. Triggs, "Enhanced local texture feature sets for face recognition under difficult lighting conditions," in Analysis and Modeling of Faces and Gestures, 2007, pp. 168-182.
  56. F. Ahmed and E. Hossain, "Automated facial expression recognition using gradient-based ternary texture patterns," Chinese Journal of Engineering, pp. 1-8, 2013.
  57. R. Ahmed, A. Meyer, H. Konik, and S. Bouakaz, "Framework for reliable , real-time facial expression recognition for low resolution images," Pattern Recognition Letters, vol. 34, no. 10, pp. 1159-1168, 2013.
  58. M. Goyani and N. Patel, "Robust Facial Expression Recognition using Local Mean Binary Pattern," Electronic Letters on Computer Vision and Image Analysis, vol. 16, no. 1, pp. 54-67, 2017.
  59. M. Goyani and N. Patel, "Multi-Level Haar Wavelet based Facial Expression Recognition using Logistic Regression," Indian Journal of Science and Technology, vol. 10, no. 9, pp. 1-9, 2017.
  60. G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski, "Classifying facial actions," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, 1999.
  61. M. Lyons and S. Akamatsu, "Coding Facial Expressions with Gabor Wavelets," 3rd IEEE Conference on Automatic Face and Gesture Recognition, pp. 200-205, 1998.
  62. M. Lades et al., "Distortion Invariant Object Recognition in the Dynamic Link Architecture," IEEE Transactions on Computers, vol. 42, pp. 300-311, 1993.
  63. W. Gu, C. Xiang, Y. V. Venkatesh, D. Huang, and H. Lin, "Facial expression recognition using radial encoding of local Gabor features and classifier synthesis," Pattern Recognition, vol. 45, no. 1, pp. 80-91, 2012.
  64. F. Chen and K. Kotani, "Facial Expression Recognition by SVM-based Two-stage Classifier on Gabor Features," IAPR Conference on Machine Vision Applications, pp. 453-456, 2007.
  65. M. J. Lyons, J. Budynek, A. Plante, and S. Akamatsu, "Classifying Facial Attributes using a 2-D GaborWavelet Representation and Discriminant Analysis," in 4th International Conference on Automatic Face and Gesture Recognition, 2000, pp. 202-207.
  66. L. Zhang, D. Tjondronegoro, and V. Chandran, "Random Gabor based templates for facial expression recognition in images with facial occlusion," Neurocomputing, vol. 145, pp. 451-464, 2014.
  67. Y. Xing and W. Luo, "Facial Expression Recognition Using Local Gabor Features and Adaboost Classifiers," IEEE Internation al Conference on Progress in Informatics and Computing, pp. 1-5, 2016.
  68. V. Ojansivu and J. Heikkil, "Blur insensitive texture classification using local phase quantization," Image and Signal Processing, pp. 236-243, 2008.
  69. A. Dhall, A. Asthana, R. Goecke, and T. Gedeon, "Emotion recognition using PHOG and LPQ features," IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, pp. 878-883, 2011.
  70. N. Dalal and B. Triggs, "Histograms of Oriented Gradients for Human Detection," International Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886-893, 2005.
  71. Z. Li, J. Imai, and M. Kaneko, "Facial-component-based Bag of Words and PHOG Descriptor for Facial Expression Recognition," in International Conference on Systems, Man, and Cybernetics San, 2009, pp. 1353-1358.
  72. Y. Bai, L. Guo, L. Jin, and Q. Huang, "A novel feature extraction method using pyramid histogram of orientation gradients for smile recognition," in 16th IEEE International Conference on Image Processing, 2009, pp. 3305-3308.
  73. P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman, "Eigenfaces vs . Fisherfaces : Recognition Using Class Specific Linear Projection," vol. 19, no. 7, pp. 711-720, 1997.
  74. M. Sugiyama, "Dimensionality Reduction of Multimodal Labeled Data by Local Fisher Discriminant Analysis," Journal of Machine Learning Research, vol. 8, pp. 1027-1067, 2007.
  75. Y. Rahulamathavan, R. C. Phan, J. A. Chambers, and D. J. Parish, "Facial Expression Recognition in the Encrypted Domain Based on Local Fisher Discriminant Analysis," vol. 4, no. 1, pp. 83-92, 2013.
  76. S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K. Muller, "Fisher Discriminant Analysis with Kernels," in IEEE Signal Processing Soceity Workshop Neural Network Signal Process IX, 1999, pp. 41-48.
  77. G. Baudat and F. Anouar, "Generalized Discriminant Analysis using a Kernel Approach," Neural Computing, vol. 12, no. 10, pp. 2385-2404, 2000.
  78. W. Zhang, S. Shan, W. Gao, and X. Chen, "Local gabor binary pattern histogram sequence (lgbphs): a novel non-statistical model for face representation and recognition," IEEE 10th International Conference on Computer Vision, vol. 1, pp. 786-791, 2005.
  79. C. Loob et al., "Dominant and Complementary Multi-Emotional Facial Expression Recognition Using C-Support Vector Classification," IEEE 12th International Conference on Automatic Face & Gesture Recogition, pp. 833-838, 2017.
  80. Y. Luo, C. Wu, and Y. Zhang, "Facial expression recognition based on fusion feature of PCA and LBP with SVM," International Journal for Light and Electron Optics, vol. 124, no. 17, pp. 2767-2770, 2013.
  81. J. Kumari, R. Rajesh, and A. Kumar, "Fusion of features for the effective facial expression recognition," International Conference on Communication and Signal Processing, pp. 457-461, 2016.
  82. G. Y. Zhao and M. Pietikainen, "Dynamic Texture Recognition using Local Binary Pattern with an Application to Facial Expressions," Transactions on Pattern Analysis and Machine Intelligence, vol. 2, no. 6, pp. 915-928, 2007.
  83. B. Jiang, M. Valstar, B. Martinez, and M. Pantic, "A dynamic appearance descriptor approach to facial actions temporal modeling," IEEE Transactions on Cybernetics, vol. 44, no. 2, pp. 161-174, 2014.
  84. T. R. Almaev and M. F. Valstar, "Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition," in Humaine Association Conference on Affective Computing and Intelligent Interaction, 2013, pp. 356-361.
  85. S. K. A. Kamarol, M. H. Jaward, H. Kalviaineninen, J. Parkkinen, and R. Parthiban, "Joint facial expression recognition and intensity estimation based on weighted votes of image sequences," Pattern Recognition Letters, vol. 92, pp. 25-32, 2017.
  86. D. Ghimire, S. Jeong, J. Lee, and S. H. Park, "Facial expression recognition based on local region specific features and support vector machines," Multimedia Tools and Applications, vol. 76, no. 6, pp. 7803-7821, 2017.
  87. C. Loob et al., "Dominant and Complementary Multi-Emotional Facial Expression Recognition Using C-Support Vector Classification," IEEE 12th International Conference on Automatic Face & Gesture Recogition, pp. 833-838, 2017.
  88. H. Naveen Kumar, S. Jagadeesha, and A. K. Jain, "Human Facial Expression Recognition from static images using shape and appearance feature," 2nd International Conference on Applied and Theoretical Computing and Communication Technology, pp. 598-603, 2016.
  89. F. Bashar, A. Khan, F. Ahmed, and M. H. Kabir, "Robust facial expression recognition based on median ternary pattern (MTP)," International Conference on Electrical Information and Communication Technology, pp. 1-5, 2013.

Publication Details

Published in : Volume 4 | Issue 2 | January-February 2018
Date of Publication : 2018-01-20
License:  This work is licensed under a Creative Commons Attribution 4.0 International License.
Page(s) : 163-175
Manuscript Number : IJSRSET184229
Publisher : Technoscience Academy

Print ISSN : 2395-1990, Online ISSN : 2394-4099

Cite This Article :

Nazeerah A. Sheth, Mahesh M. Goyani, " A Comprehensive study of Geometric and Appearance based Facial Expression Recognition Methods, International Journal of Scientific Research in Science, Engineering and Technology(IJSRSET), Print ISSN : 2395-1990, Online ISSN : 2394-4099, Volume 4, Issue 2, pp.163-175, January-February-2018. Citation Detection and Elimination     |     
Journal URL : https://ijsrset.com/IJSRSET184229

Article Preview