Skip to main content

Advertisement

Robust Feature Detection for Facial Expression Recognition

Article metrics

Abstract

This paper presents a robust and adaptable facial feature extraction system used for facial expression recognition in human-computer interaction (HCI) environments. Such environments are usually uncontrolled in terms of lighting and color quality, as well as human expressivity and movement; as a result, using a single feature extraction technique may fail in some parts of a video sequence, while performing well in others. The proposed system is based on a multicue feature extraction and fusion technique, which provides MPEG-4-compatible features assorted with a confidence measure. This confidence measure is used to pinpoint cases where detection of individual features may be wrong and reduce their contribution to the training phase or their importance in deducing the observed facial expression, while the fusion process ensures that the final result regarding the features will be based on the extraction technique that performed better given the particular lighting or color conditions. Real data and results are presented, involving both extreme and intermediate expression/emotional states, obtained within the sensitive artificial listener HCI environment that was generated in the framework of related European projects.

[1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162]

References

  1. 1.

    Mehrabian A: Communication without words. Psychology Today 1968,2(9):52-55.

  2. 2.

    Fasel B, Luettin J: Automatic facial expression analysis: a survey. Pattern Recognition 2003,36(1):259-275. 10.1016/S0031-3203(02)00052-3

  3. 3.

    Ekman P, Friesen WV: Facial Action Coding Systems: A Technique for the Measurement of Facial Movement. Consulting Psychologist Press, Palo Alto, Calif, USA; 1978.

  4. 4.

    Tomasi C, Kanade T: Detection and tracking of point features. In Tech. Rep. CMU-CS-91-132. Carnegie Mellon University, Pittsburgh, Pa, USA; 1991.

  5. 5.

    Tian Y-L, Kanade T, Cohn JF: Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(2):97-115. 10.1109/34.908962

  6. 6.

    Tekalp AM, Ostermann J: Face and 2-D mesh animation in MPEG-4. Signal Processing: Image Communication 2000,15(4):387-421. 10.1016/S0923-5965(99)00055-7

  7. 7.

    Raouzaiou A, Tsapatsoulis N, Karpouzis K, Kollias S: Parameterized facial expression synthesis based on MPEG-4. EURASIP Journal on Applied Signal Processing 2002,2002(10):1021-1038. 10.1155/S1110865702206149

  8. 8.

    Essa IA, Pentland AP: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997,19(7):757-763. 10.1109/34.598232

  9. 9.

    Lanitis A, Taylor CJ, Cootes TF, Ahmed T: Automatic interpretation of human faces and hand gestures using flexible models. Proceedings of the 1st International Workshop on Automatic Face and Gesture Recognition (FG '95), September 1995, Zurich, Switzerland 98-103.

  10. 10.

    Yacoob Y, Devis LS: Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 1996,18(6):636-642. 10.1109/34.506414

  11. 11.

    Lisetti CL, Rumelhart DE: Facial expression recognition using a neural network. In Proceedings of the 11th International Florida Artificial Intelligence Research Society Conference, May 1998, Sanibel Island, Fla, USA. AAAI Press; 328-332.

  12. 12.

    Kaiser S, Wehrle T: Automated coding of facial behavior in human-computer interactions with facs. Journal of Nonverbal Behavior 1992,16(2):67-84. 10.1007/BF00990323

  13. 13.

    Edwards GJ, Cootes TF, Taylor CJ: Face recognition using active appearance models. Proceedings of the 5th European Conference on Computer Vision (ECCV '98), June 1998, Freiburg, Germany 2: 581-595.

  14. 14.

    Cohn JF, Zlochower AJ, Lien JJ, Kanade T: Feature-point tracking by optical flow discriminates subtle differences in facial expression. Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98), April 1998, Nara, Japan 396-401.

  15. 15.

    Black MJ, Yacoob Y: Recognizing facial expressions in image sequences using local parameterized models of image motion. International Journal of Computer Vision 1997,25(1):23-48. 10.1023/A:1007977618277

  16. 16.

    Lam K-M, Yan H: An analytic-to-holistic approach for face recognition based on a single frontal view. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998,20(7):673-686. 10.1109/34.689299

  17. 17.

    Gu H, Su G-D, Du C: Feature points extraction from face images. Proceedings of the Image and Vision Computing Conference (IVCNZ '03), November 2003, Palmerston North, New Zealand 154-158.

  18. 18.

    Leung S-H, Wang S-L, Lau W-H: Lip image segmentation using fuzzy clustering incorporating an elliptic shape function. IEEE Transactions on Image Processing 2004,13(1):51-62. 10.1109/TIP.2003.818116

  19. 19.

    Sarris N, Grammalidis N, Strintzis MG: FAP extraction using three-dimensional motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 2002,12(10):865-876. 10.1109/TCSVT.2002.804888

  20. 20.

    Tian Y, Kanade T, Cohn JF: Robust lip tracking by combining shape, color and motion. Proceedings of the 4th Asian Conference on Computer Vision (ACCV '00), January 2000, Taipei, Taiwan 1040-1045.

  21. 21.

    Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS: Authentic facial expression analysis. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG '04), May 2004, Seoul, South Korea 517-522.

  22. 22.

    DeCarlo D, Metaxas D: The integration of optical flow and deformable models with applications to human face shape and motion estimation. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '96), June 1996, San Francisco, Calif, USA 231-238.

  23. 23.

    Pantic M, Rothkrantz LJM: Expert system for automatic analysis of facial expressions. Image and Vision Computing 2000,18(11):881-905. 10.1016/S0262-8856(00)00034-2

  24. 24.

    Cootes TF, Edwards GJ, Taylor CJ: Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(6):681-685. 10.1109/34.927467

  25. 25.

    Huang C-L, Huang Y-M: Facial expression recognition using model-based feature extraction and action parameters classification. Journal of Visual Communication and Image Representation 1997,8(3):278-290. 10.1006/jvci.1997.0359

  26. 26.

    Lyons MJ, Budynek J, Akamatsu S: Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999,21(12):1357-1362. 10.1109/34.817413

  27. 27.

    Hong H, Neven H, von der Malsburg C: Online facial expression recognition based on personalized galleries. Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98), April 1998, Nara, Japan 354-359.

  28. 28.

    Hsu R-L, Abdel-Mottaleb M, Jain AK: Face detection in color images. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(5):696-706. 10.1109/34.1000242

  29. 29.

    McKenna SJ, Raja Y, Gong S: Tracking colour objects using adaptive mixture models. Image and Vision Computing 1999,17(3-4):225-231. 10.1016/S0262-8856(98)00104-8

  30. 30.

    ERMIS : Emotionally Rich Man-machine Intelligent System IST-2000-29319. http://www.image.ntua.gr/ermis/

  31. 31.

    HUMAINE IST : Human-Machine Interaction Network on Emotion. 2007.http://www.emotion-research.net/

  32. 32.

    ISTFACE : MPEG-4 Facial Animation System—Version 3.3.1 Gabriel Abrantes. (Developed in the context of the European Project ACTS MoMuSys © 97-98 Instituto Superior Tecnico)

  33. 33.

    Young JW: Head and Face Anthropometry of Adult U.S. Civilians. FAA Civil Aeromedical Institute, 1963–1993, (final report 1993)

  34. 34.

    Yang M-H, Kriegman DJ, Ahuja N: Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(1):34-58. 10.1109/34.982883

  35. 35.

    Papageorgiou CP, Oren M, Poggio T: A general framework for object detection. Proceedings of the 6th IEEE International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 555-562.

  36. 36.

    Viola P, Jones M: Rapid object detection using a boosted cascade of simple features. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '01), December 2001, Kauai, Hawaii, USA 1: 511-518.

  37. 37.

    Fasel I, Fortenberry B, Movellan J: A generative framework for real time object detection and classification. Computer Vision and Image Understanding 2005,98(1):182-210. 10.1016/j.cviu.2004.07.014

  38. 38.

    Fransens R, De Prins J, van Gool L: SVM-based nonparametric discriminant analysis, an application to face detection. Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV '03), October 2003, Nice, France 2: 1289-1296.

  39. 39.

    Kollias S, Anastassiou D: An adaptive least squares algorithm for the efficient training of artificial neural networks. IEEE Transactions on Circuits and Systems 1989,36(8):1092-1101. 10.1109/31.192419

  40. 40.

    Hagan MH, Menhaj MB: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 1994,5(6):989-993. 10.1109/72.329697

  41. 41.

    Yin L, Basu A: Generating realistic facial expressions with wrinkles for model-based coding. Computer Vision and Image Understanding 2001,84(2):201-240. 10.1006/cviu.2001.0949

  42. 42.

    Lyons MJ, Haehnel M, Tetsutani N: The mouthesizer: a facial gesture musical interface. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01), August 2001, Los Angeles, Calif, USA 230.

  43. 43.

    Arca S, Campadelli P, Lanzarotti R: An automatic feature-based face recognition system. Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS '04), April 2004, Lisboa, Portugal

  44. 44.

    Lam K-M, Yan H: Locating and extracting the eye in human face images. Pattern Recognition 1996,29(5):771-779. 10.1016/0031-3203(95)00119-0

  45. 45.

    Canny J: A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 1986,8(6):679-698. 10.1109/TPAMI.1986.4767851

  46. 46.

    Gorodnichy DO: On importance of nose for face tracking. Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition (FG '02), May 2002, Washington, DC, USA 181-186.

  47. 47.

    Aung SC, Ngim RCK, Lee ST: Evaluation of the laser scanner as a surface measuring tool and its accuracy compared with direct facial anthropometric measurements. British Journal of Plastic Surgery 1995,48(8):551-558. 10.1016/0007-1226(95)90043-8

  48. 48.

    Vincent L: Morphological grayscale reconstruction in image analysis: applications and efficient algorithms. IEEE Transactions on Image Processing 1993,2(2):176-201. 10.1109/83.217222

  49. 49.

    Vincent L: Morphological grayscale reconstruction in image analysis: applications and efficient algorithms. IEEE Transactions on Image Processing 1993,2(2):176-201. 10.1109/83.217222

  50. 50.

    Krogh A, Vedelsby J: Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems. Volume 7. Edited by: Tesauro G, Touretzky D, Leen T. The MIT Press, Cambridge, Mass, USA; 1995:231-238.

  51. 51.

    Tresp V: Committee machines. In Handbook for Neural Network Signal Processing. Edited by: Hu YH, Hwang J-N. CRC Press, Boca Raton, Fla, USA; 2001.

  52. 52.

    Whissel CM: The dictionary of affect in language. In Emotion: Theory, Research and Experience. The Measurement of Emotions. Volume 4. Edited by: Plutchnik R, Kellerman H. Academic Press, New York, NY, USA; 1989:113-131.

  53. 53.

    Ioannou S, Raouzaiou AT, Tzouvaras VA, Mailis TP, Karpouzis K, Kollias S: Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Networks 2005,18(4):423-435. 10.1016/j.neunet.2005.03.004

  54. 54.

    Klir GJ, Yuan B: Fuzzy Sets and Fuzzy Logic: Theory and Applications. Prentice-Hall, Upper Saddle River, NJ, USA; 1995.

  55. 55.

    Lee MA, Takagi H: Integrating design stages of fuzzy systems using genetic algorithms. Proceedings of the 2nd IEEE International Conference on Fuzzy Systems (FUZZY'93), March-April 1993, San Francisco, Calif, USA 612-617.

  56. 56.

    Wallace M, Kollias S: Possibilistic evaluation of extended fuzzy rules in the presence of uncertainty. Proceedings of the 14th IEEE International Conference on Fuzzy Systems (FUZZ '05), May 2005, Reno, Nev, USA 815-820.

  57. 57.

    Weizenbaum J: ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM 1966,9(1):36-45. 10.1145/365153.365168

  58. 58.

    Zhang YJ: A survey on evaluation methods for image segmentation. Pattern Recognition 1996,29(8):1335-1346. 10.1016/0031-3203(95)00169-7

  59. 59.

    Williams GW: Comparing the joint agreement of several raters with another rater. Biometrics 1976,32(3):619-627. 10.2307/2529750

  60. 60.

    Chalana V, Kim Y: A methodology for evaluation of boundary detection algorithms on medical images. IEEE Transactions on Medical Imaging 1997,16(5):642-652. 10.1109/42.640755

  61. 61.

    Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schröder M: 'Feeltrace': an instrument for recording perceived emotion in real time. Proceedings of the ISCA Workshop on Speech and Emotion, September 2000, Belfast, Northern Ireland 19-24.

  62. 62.

    Cristinacce D, Cootes TF: A comparison of shape constrained facial feature detectors. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG '04), May 2004, Seoul, South Korea 375-380.

Download references

Author information

Correspondence to Spiros Ioannou.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Ioannou, S., Caridakis, G., Karpouzis, K. et al. Robust Feature Detection for Facial Expression Recognition. J Image Video Proc 2007, 029081 (2007) doi:10.1155/2007/29081

Download citation

Keywords

  • Feature Extraction
  • Facial Expression
  • Video Sequence
  • Facial Feature
  • Extraction Technique