Skip to main content
EURASIP Journal on Image and Video Processing
Download PDF

Robust Feature Detection for Facial Expression Recognition

EURASIP Journal on Image and Video Processing volume 2007, Article number: 029081 (2007) Cite this article

Abstract

This paper presents a robust and adaptable facial feature extraction system used for facial expression recognition in human-computer interaction (HCI) environments. Such environments are usually uncontrolled in terms of lighting and color quality, as well as human expressivity and movement; as a result, using a single feature extraction technique may fail in some parts of a video sequence, while performing well in others. The proposed system is based on a multicue feature extraction and fusion technique, which provides MPEG-4-compatible features assorted with a confidence measure. This confidence measure is used to pinpoint cases where detection of individual features may be wrong and reduce their contribution to the training phase or their importance in deducing the observed facial expression, while the fusion process ensures that the final result regarding the features will be based on the extraction technique that performed better given the particular lighting or color conditions. Real data and results are presented, involving both extreme and intermediate expression/emotional states, obtained within the sensitive artificial listener HCI environment that was generated in the framework of related European projects.

[1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162]

References

  1. Mehrabian A: Communication without words. Psychology Today 1968,2(9):52-55.

    Google Scholar 

  2. Fasel B, Luettin J: Automatic facial expression analysis: a survey. Pattern Recognition 2003,36(1):259-275. 10.1016/S0031-3203(02)00052-3

    Article  MATH  Google Scholar 

  3. Ekman P, Friesen WV: Facial Action Coding Systems: A Technique for the Measurement of Facial Movement. Consulting Psychologist Press, Palo Alto, Calif, USA; 1978.

    Google Scholar 

  4. Tomasi C, Kanade T: Detection and tracking of point features. In Tech. Rep. CMU-CS-91-132. Carnegie Mellon University, Pittsburgh, Pa, USA; 1991.

    Google Scholar 

  5. Tian Y-L, Kanade T, Cohn JF: Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(2):97-115. 10.1109/34.908962

    Article  Google Scholar 

  6. Tekalp AM, Ostermann J: Face and 2-D mesh animation in MPEG-4. Signal Processing: Image Communication 2000,15(4):387-421. 10.1016/S0923-5965(99)00055-7

    Google Scholar 

  7. Raouzaiou A, Tsapatsoulis N, Karpouzis K, Kollias S: Parameterized facial expression synthesis based on MPEG-4. EURASIP Journal on Applied Signal Processing 2002,2002(10):1021-1038. 10.1155/S1110865702206149

    Article  MATH  Google Scholar 

  8. Essa IA, Pentland AP: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997,19(7):757-763. 10.1109/34.598232

    Article  Google Scholar 

  9. Lanitis A, Taylor CJ, Cootes TF, Ahmed T: Automatic interpretation of human faces and hand gestures using flexible models. Proceedings of the 1st International Workshop on Automatic Face and Gesture Recognition (FG '95), September 1995, Zurich, Switzerland 98-103.

    Google Scholar 

  10. Yacoob Y, Devis LS: Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 1996,18(6):636-642. 10.1109/34.506414

    Article  Google Scholar 

  11. Lisetti CL, Rumelhart DE: Facial expression recognition using a neural network. In Proceedings of the 11th International Florida Artificial Intelligence Research Society Conference, May 1998, Sanibel Island, Fla, USA. AAAI Press; 328-332.

    Google Scholar 

  12. Kaiser S, Wehrle T: Automated coding of facial behavior in human-computer interactions with facs. Journal of Nonverbal Behavior 1992,16(2):67-84. 10.1007/BF00990323

    Article  Google Scholar 

  13. Edwards GJ, Cootes TF, Taylor CJ: Face recognition using active appearance models. Proceedings of the 5th European Conference on Computer Vision (ECCV '98), June 1998, Freiburg, Germany 2: 581-595.

    Google Scholar 

  14. Cohn JF, Zlochower AJ, Lien JJ, Kanade T: Feature-point tracking by optical flow discriminates subtle differences in facial expression. Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98), April 1998, Nara, Japan 396-401.

    Chapter  Google Scholar 

  15. Black MJ, Yacoob Y: Recognizing facial expressions in image sequences using local parameterized models of image motion. International Journal of Computer Vision 1997,25(1):23-48. 10.1023/A:1007977618277

    Article  Google Scholar 

  16. Lam K-M, Yan H: An analytic-to-holistic approach for face recognition based on a single frontal view. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998,20(7):673-686. 10.1109/34.689299

    Article  Google Scholar 

  17. Gu H, Su G-D, Du C: Feature points extraction from face images. Proceedings of the Image and Vision Computing Conference (IVCNZ '03), November 2003, Palmerston North, New Zealand 154-158.

    Google Scholar 

  18. Leung S-H, Wang S-L, Lau W-H: Lip image segmentation using fuzzy clustering incorporating an elliptic shape function. IEEE Transactions on Image Processing 2004,13(1):51-62. 10.1109/TIP.2003.818116

    Article  Google Scholar 

  19. Sarris N, Grammalidis N, Strintzis MG: FAP extraction using three-dimensional motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 2002,12(10):865-876. 10.1109/TCSVT.2002.804888

    Article  Google Scholar 

  20. Tian Y, Kanade T, Cohn JF: Robust lip tracking by combining shape, color and motion. Proceedings of the 4th Asian Conference on Computer Vision (ACCV '00), January 2000, Taipei, Taiwan 1040-1045.

    Google Scholar 

  21. Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS: Authentic facial expression analysis. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG '04), May 2004, Seoul, South Korea 517-522.

    Google Scholar 

  22. DeCarlo D, Metaxas D: The integration of optical flow and deformable models with applications to human face shape and motion estimation. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '96), June 1996, San Francisco, Calif, USA 231-238.

    Chapter  Google Scholar 

  23. Pantic M, Rothkrantz LJM: Expert system for automatic analysis of facial expressions. Image and Vision Computing 2000,18(11):881-905. 10.1016/S0262-8856(00)00034-2

    Article  Google Scholar 

  24. Cootes TF, Edwards GJ, Taylor CJ: Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(6):681-685. 10.1109/34.927467

    Article  Google Scholar 

  25. Huang C-L, Huang Y-M: Facial expression recognition using model-based feature extraction and action parameters classification. Journal of Visual Communication and Image Representation 1997,8(3):278-290. 10.1006/jvci.1997.0359

    Article  Google Scholar 

  26. Lyons MJ, Budynek J, Akamatsu S: Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999,21(12):1357-1362. 10.1109/34.817413

    Article  Google Scholar 

  27. Hong H, Neven H, von der Malsburg C: Online facial expression recognition based on personalized galleries. Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98), April 1998, Nara, Japan 354-359.

    Chapter  Google Scholar 

  28. Hsu R-L, Abdel-Mottaleb M, Jain AK: Face detection in color images. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(5):696-706. 10.1109/34.1000242

    Article  Google Scholar 

  29. McKenna SJ, Raja Y, Gong S: Tracking colour objects using adaptive mixture models. Image and Vision Computing 1999,17(3-4):225-231. 10.1016/S0262-8856(98)00104-8

    Article  Google Scholar 

  30. ERMIS : Emotionally Rich Man-machine Intelligent System IST-2000-29319. http://www.image.ntua.gr/ermis/

  31. HUMAINE IST : Human-Machine Interaction Network on Emotion. 2007.http://www.emotion-research.net/

    Google Scholar 

  32. ISTFACE : MPEG-4 Facial Animation System—Version 3.3.1 Gabriel Abrantes. (Developed in the context of the European Project ACTS MoMuSys © 97-98 Instituto Superior Tecnico)

  33. Young JW: Head and Face Anthropometry of Adult U.S. Civilians. FAA Civil Aeromedical Institute, 1963–1993, (final report 1993)

  34. Yang M-H, Kriegman DJ, Ahuja N: Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(1):34-58. 10.1109/34.982883

    Article  Google Scholar 

  35. Papageorgiou CP, Oren M, Poggio T: A general framework for object detection. Proceedings of the 6th IEEE International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 555-562.

    Google Scholar 

  36. Viola P, Jones M: Rapid object detection using a boosted cascade of simple features. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '01), December 2001, Kauai, Hawaii, USA 1: 511-518.

    Google Scholar 

  37. Fasel I, Fortenberry B, Movellan J: A generative framework for real time object detection and classification. Computer Vision and Image Understanding 2005,98(1):182-210. 10.1016/j.cviu.2004.07.014

    Article  Google Scholar 

  38. Fransens R, De Prins J, van Gool L: SVM-based nonparametric discriminant analysis, an application to face detection. Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV '03), October 2003, Nice, France 2: 1289-1296.

    Article  Google Scholar 

  39. Kollias S, Anastassiou D: An adaptive least squares algorithm for the efficient training of artificial neural networks. IEEE Transactions on Circuits and Systems 1989,36(8):1092-1101. 10.1109/31.192419

    Article  Google Scholar 

  40. Hagan MH, Menhaj MB: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 1994,5(6):989-993. 10.1109/72.329697

    Article  Google Scholar 

  41. Yin L, Basu A: Generating realistic facial expressions with wrinkles for model-based coding. Computer Vision and Image Understanding 2001,84(2):201-240. 10.1006/cviu.2001.0949

    Article  MATH  Google Scholar 

  42. Lyons MJ, Haehnel M, Tetsutani N: The mouthesizer: a facial gesture musical interface. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01), August 2001, Los Angeles, Calif, USA 230.

    Google Scholar 

  43. Arca S, Campadelli P, Lanzarotti R: An automatic feature-based face recognition system. Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS '04), April 2004, Lisboa, Portugal

    Google Scholar 

  44. Lam K-M, Yan H: Locating and extracting the eye in human face images. Pattern Recognition 1996,29(5):771-779. 10.1016/0031-3203(95)00119-0

    Article  MathSciNet  Google Scholar 

  45. Canny J: A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 1986,8(6):679-698. 10.1109/TPAMI.1986.4767851

    Article  Google Scholar 

  46. Gorodnichy DO: On importance of nose for face tracking. Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition (FG '02), May 2002, Washington, DC, USA 181-186.

    Google Scholar 

  47. Aung SC, Ngim RCK, Lee ST: Evaluation of the laser scanner as a surface measuring tool and its accuracy compared with direct facial anthropometric measurements. British Journal of Plastic Surgery 1995,48(8):551-558. 10.1016/0007-1226(95)90043-8

    Article  Google Scholar 

  48. Vincent L: Morphological grayscale reconstruction in image analysis: applications and efficient algorithms. IEEE Transactions on Image Processing 1993,2(2):176-201. 10.1109/83.217222

    Article  Google Scholar 

  49. Vincent L: Morphological grayscale reconstruction in image analysis: applications and efficient algorithms. IEEE Transactions on Image Processing 1993,2(2):176-201. 10.1109/83.217222

    Article  Google Scholar 

  50. Krogh A, Vedelsby J: Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems. Volume 7. Edited by: Tesauro G, Touretzky D, Leen T. The MIT Press, Cambridge, Mass, USA; 1995:231-238.

    Google Scholar 

  51. Tresp V: Committee machines. In Handbook for Neural Network Signal Processing. Edited by: Hu YH, Hwang J-N. CRC Press, Boca Raton, Fla, USA; 2001.

    Google Scholar 

  52. Whissel CM: The dictionary of affect in language. In Emotion: Theory, Research and Experience. The Measurement of Emotions. Volume 4. Edited by: Plutchnik R, Kellerman H. Academic Press, New York, NY, USA; 1989:113-131.

    Google Scholar 

  53. Ioannou S, Raouzaiou AT, Tzouvaras VA, Mailis TP, Karpouzis K, Kollias S: Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Networks 2005,18(4):423-435. 10.1016/j.neunet.2005.03.004

    Article  Google Scholar 

  54. Klir GJ, Yuan B: Fuzzy Sets and Fuzzy Logic: Theory and Applications. Prentice-Hall, Upper Saddle River, NJ, USA; 1995.

    MATH  Google Scholar 

  55. Lee MA, Takagi H: Integrating design stages of fuzzy systems using genetic algorithms. Proceedings of the 2nd IEEE International Conference on Fuzzy Systems (FUZZY'93), March-April 1993, San Francisco, Calif, USA 612-617.

    Google Scholar 

  56. Wallace M, Kollias S: Possibilistic evaluation of extended fuzzy rules in the presence of uncertainty. Proceedings of the 14th IEEE International Conference on Fuzzy Systems (FUZZ '05), May 2005, Reno, Nev, USA 815-820.

    Google Scholar 

  57. Weizenbaum J: ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM 1966,9(1):36-45. 10.1145/365153.365168

    Article  Google Scholar 

  58. Zhang YJ: A survey on evaluation methods for image segmentation. Pattern Recognition 1996,29(8):1335-1346. 10.1016/0031-3203(95)00169-7

    Article  Google Scholar 

  59. Williams GW: Comparing the joint agreement of several raters with another rater. Biometrics 1976,32(3):619-627. 10.2307/2529750

    Article  MATH  Google Scholar 

  60. Chalana V, Kim Y: A methodology for evaluation of boundary detection algorithms on medical images. IEEE Transactions on Medical Imaging 1997,16(5):642-652. 10.1109/42.640755

    Article  Google Scholar 

  61. Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schröder M: 'Feeltrace': an instrument for recording perceived emotion in real time. Proceedings of the ISCA Workshop on Speech and Emotion, September 2000, Belfast, Northern Ireland 19-24.

    Google Scholar 

  62. Cristinacce D, Cootes TF: A comparison of shape constrained facial feature detectors. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG '04), May 2004, Seoul, South Korea 375-380.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Image, Video and Multimedia Systems Laboratory, National Technical University of Athens, 9 Iroon Polytechniou Street, Zographou, Athens, 157 80, Greece

    Spiros Ioannou, George Caridakis, Kostas Karpouzis & Stefanos Kollias

Authors
  1. Spiros Ioannou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George Caridakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kostas Karpouzis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefanos Kollias
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Spiros Ioannou.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Ioannou, S., Caridakis, G., Karpouzis, K. et al. Robust Feature Detection for Facial Expression Recognition. J Image Video Proc 2007, 029081 (2007). https://doi.org/10.1155/2007/29081

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1155/2007/29081

Keywords

  • Feature Extraction
  • Facial Expression
  • Video Sequence
  • Facial Feature
  • Extraction Technique