Skip to content


  • Research Article
  • Open Access

Robust Feature Detection for Facial Expression Recognition

  • 1Email author,
  • 1,
  • 1 and
  • 1
EURASIP Journal on Image and Video Processing20072007:029081

  • Received: 1 May 2006
  • Accepted: 18 May 2007
  • Published:


This paper presents a robust and adaptable facial feature extraction system used for facial expression recognition in human-computer interaction (HCI) environments. Such environments are usually uncontrolled in terms of lighting and color quality, as well as human expressivity and movement; as a result, using a single feature extraction technique may fail in some parts of a video sequence, while performing well in others. The proposed system is based on a multicue feature extraction and fusion technique, which provides MPEG-4-compatible features assorted with a confidence measure. This confidence measure is used to pinpoint cases where detection of individual features may be wrong and reduce their contribution to the training phase or their importance in deducing the observed facial expression, while the fusion process ensures that the final result regarding the features will be based on the extraction technique that performed better given the particular lighting or color conditions. Real data and results are presented, involving both extreme and intermediate expression/emotional states, obtained within the sensitive artificial listener HCI environment that was generated in the framework of related European projects.


  • Feature Extraction
  • Facial Expression
  • Video Sequence
  • Facial Feature
  • Extraction Technique


Authors’ Affiliations

Image, Video and Multimedia Systems Laboratory, National Technical University of Athens, 9 Iroon Polytechniou Street, Zographou, Athens, 157 80, Greece


  1. Mehrabian A: Communication without words. Psychology Today 1968,2(9):52-55.Google Scholar
  2. Fasel B, Luettin J: Automatic facial expression analysis: a survey. Pattern Recognition 2003,36(1):259-275. 10.1016/S0031-3203(02)00052-3View ArticleMATHGoogle Scholar
  3. Ekman P, Friesen WV: Facial Action Coding Systems: A Technique for the Measurement of Facial Movement. Consulting Psychologist Press, Palo Alto, Calif, USA; 1978.Google Scholar
  4. Tomasi C, Kanade T: Detection and tracking of point features. In Tech. Rep. CMU-CS-91-132. Carnegie Mellon University, Pittsburgh, Pa, USA; 1991.Google Scholar
  5. Tian Y-L, Kanade T, Cohn JF: Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(2):97-115. 10.1109/34.908962View ArticleGoogle Scholar
  6. Tekalp AM, Ostermann J: Face and 2-D mesh animation in MPEG-4. Signal Processing: Image Communication 2000,15(4):387-421. 10.1016/S0923-5965(99)00055-7Google Scholar
  7. Raouzaiou A, Tsapatsoulis N, Karpouzis K, Kollias S: Parameterized facial expression synthesis based on MPEG-4. EURASIP Journal on Applied Signal Processing 2002,2002(10):1021-1038. 10.1155/S1110865702206149View ArticleMATHGoogle Scholar
  8. Essa IA, Pentland AP: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997,19(7):757-763. 10.1109/34.598232View ArticleGoogle Scholar
  9. Lanitis A, Taylor CJ, Cootes TF, Ahmed T: Automatic interpretation of human faces and hand gestures using flexible models. Proceedings of the 1st International Workshop on Automatic Face and Gesture Recognition (FG '95), September 1995, Zurich, Switzerland 98-103.Google Scholar
  10. Yacoob Y, Devis LS: Recognizing human facial expressions from long image sequences using optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence 1996,18(6):636-642. 10.1109/34.506414View ArticleGoogle Scholar
  11. Lisetti CL, Rumelhart DE: Facial expression recognition using a neural network. In Proceedings of the 11th International Florida Artificial Intelligence Research Society Conference, May 1998, Sanibel Island, Fla, USA. AAAI Press; 328-332.Google Scholar
  12. Kaiser S, Wehrle T: Automated coding of facial behavior in human-computer interactions with facs. Journal of Nonverbal Behavior 1992,16(2):67-84. 10.1007/BF00990323View ArticleGoogle Scholar
  13. Edwards GJ, Cootes TF, Taylor CJ: Face recognition using active appearance models. Proceedings of the 5th European Conference on Computer Vision (ECCV '98), June 1998, Freiburg, Germany 2: 581-595.Google Scholar
  14. Cohn JF, Zlochower AJ, Lien JJ, Kanade T: Feature-point tracking by optical flow discriminates subtle differences in facial expression. Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98), April 1998, Nara, Japan 396-401.View ArticleGoogle Scholar
  15. Black MJ, Yacoob Y: Recognizing facial expressions in image sequences using local parameterized models of image motion. International Journal of Computer Vision 1997,25(1):23-48. 10.1023/A:1007977618277View ArticleGoogle Scholar
  16. Lam K-M, Yan H: An analytic-to-holistic approach for face recognition based on a single frontal view. IEEE Transactions on Pattern Analysis and Machine Intelligence 1998,20(7):673-686. 10.1109/34.689299View ArticleGoogle Scholar
  17. Gu H, Su G-D, Du C: Feature points extraction from face images. Proceedings of the Image and Vision Computing Conference (IVCNZ '03), November 2003, Palmerston North, New Zealand 154-158.Google Scholar
  18. Leung S-H, Wang S-L, Lau W-H: Lip image segmentation using fuzzy clustering incorporating an elliptic shape function. IEEE Transactions on Image Processing 2004,13(1):51-62. 10.1109/TIP.2003.818116View ArticleGoogle Scholar
  19. Sarris N, Grammalidis N, Strintzis MG: FAP extraction using three-dimensional motion estimation. IEEE Transactions on Circuits and Systems for Video Technology 2002,12(10):865-876. 10.1109/TCSVT.2002.804888View ArticleGoogle Scholar
  20. Tian Y, Kanade T, Cohn JF: Robust lip tracking by combining shape, color and motion. Proceedings of the 4th Asian Conference on Computer Vision (ACCV '00), January 2000, Taipei, Taiwan 1040-1045.Google Scholar
  21. Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS: Authentic facial expression analysis. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG '04), May 2004, Seoul, South Korea 517-522.Google Scholar
  22. DeCarlo D, Metaxas D: The integration of optical flow and deformable models with applications to human face shape and motion estimation. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '96), June 1996, San Francisco, Calif, USA 231-238.View ArticleGoogle Scholar
  23. Pantic M, Rothkrantz LJM: Expert system for automatic analysis of facial expressions. Image and Vision Computing 2000,18(11):881-905. 10.1016/S0262-8856(00)00034-2View ArticleGoogle Scholar
  24. Cootes TF, Edwards GJ, Taylor CJ: Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(6):681-685. 10.1109/34.927467View ArticleGoogle Scholar
  25. Huang C-L, Huang Y-M: Facial expression recognition using model-based feature extraction and action parameters classification. Journal of Visual Communication and Image Representation 1997,8(3):278-290. 10.1006/jvci.1997.0359View ArticleGoogle Scholar
  26. Lyons MJ, Budynek J, Akamatsu S: Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999,21(12):1357-1362. 10.1109/34.817413View ArticleGoogle Scholar
  27. Hong H, Neven H, von der Malsburg C: Online facial expression recognition based on personalized galleries. Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG '98), April 1998, Nara, Japan 354-359.View ArticleGoogle Scholar
  28. Hsu R-L, Abdel-Mottaleb M, Jain AK: Face detection in color images. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(5):696-706. 10.1109/34.1000242View ArticleGoogle Scholar
  29. McKenna SJ, Raja Y, Gong S: Tracking colour objects using adaptive mixture models. Image and Vision Computing 1999,17(3-4):225-231. 10.1016/S0262-8856(98)00104-8View ArticleGoogle Scholar
  30. ERMIS : Emotionally Rich Man-machine Intelligent System IST-2000-29319.
  31. HUMAINE IST : Human-Machine Interaction Network on Emotion. 2007. Scholar
  32. ISTFACE : MPEG-4 Facial Animation System—Version 3.3.1 Gabriel Abrantes. (Developed in the context of the European Project ACTS MoMuSys © 97-98 Instituto Superior Tecnico)Google Scholar
  33. Young JW: Head and Face Anthropometry of Adult U.S. Civilians. FAA Civil Aeromedical Institute, 1963–1993, (final report 1993)Google Scholar
  34. Yang M-H, Kriegman DJ, Ahuja N: Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 2002,24(1):34-58. 10.1109/34.982883View ArticleGoogle Scholar
  35. Papageorgiou CP, Oren M, Poggio T: A general framework for object detection. Proceedings of the 6th IEEE International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 555-562.Google Scholar
  36. Viola P, Jones M: Rapid object detection using a boosted cascade of simple features. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '01), December 2001, Kauai, Hawaii, USA 1: 511-518.Google Scholar
  37. Fasel I, Fortenberry B, Movellan J: A generative framework for real time object detection and classification. Computer Vision and Image Understanding 2005,98(1):182-210. 10.1016/j.cviu.2004.07.014View ArticleGoogle Scholar
  38. Fransens R, De Prins J, van Gool L: SVM-based nonparametric discriminant analysis, an application to face detection. Proceedings of the 9th IEEE International Conference on Computer Vision (ICCV '03), October 2003, Nice, France 2: 1289-1296.View ArticleGoogle Scholar
  39. Kollias S, Anastassiou D: An adaptive least squares algorithm for the efficient training of artificial neural networks. IEEE Transactions on Circuits and Systems 1989,36(8):1092-1101. 10.1109/31.192419View ArticleGoogle Scholar
  40. Hagan MH, Menhaj MB: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 1994,5(6):989-993. 10.1109/72.329697View ArticleGoogle Scholar
  41. Yin L, Basu A: Generating realistic facial expressions with wrinkles for model-based coding. Computer Vision and Image Understanding 2001,84(2):201-240. 10.1006/cviu.2001.0949View ArticleMATHGoogle Scholar
  42. Lyons MJ, Haehnel M, Tetsutani N: The mouthesizer: a facial gesture musical interface. Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01), August 2001, Los Angeles, Calif, USA 230.Google Scholar
  43. Arca S, Campadelli P, Lanzarotti R: An automatic feature-based face recognition system. Proceedings of the 5th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS '04), April 2004, Lisboa, PortugalGoogle Scholar
  44. Lam K-M, Yan H: Locating and extracting the eye in human face images. Pattern Recognition 1996,29(5):771-779. 10.1016/0031-3203(95)00119-0MathSciNetView ArticleGoogle Scholar
  45. Canny J: A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 1986,8(6):679-698. 10.1109/TPAMI.1986.4767851View ArticleGoogle Scholar
  46. Gorodnichy DO: On importance of nose for face tracking. Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition (FG '02), May 2002, Washington, DC, USA 181-186.Google Scholar
  47. Aung SC, Ngim RCK, Lee ST: Evaluation of the laser scanner as a surface measuring tool and its accuracy compared with direct facial anthropometric measurements. British Journal of Plastic Surgery 1995,48(8):551-558. 10.1016/0007-1226(95)90043-8View ArticleGoogle Scholar
  48. Vincent L: Morphological grayscale reconstruction in image analysis: applications and efficient algorithms. IEEE Transactions on Image Processing 1993,2(2):176-201. 10.1109/83.217222View ArticleGoogle Scholar
  49. Vincent L: Morphological grayscale reconstruction in image analysis: applications and efficient algorithms. IEEE Transactions on Image Processing 1993,2(2):176-201. 10.1109/83.217222View ArticleGoogle Scholar
  50. Krogh A, Vedelsby J: Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems. Volume 7. Edited by: Tesauro G, Touretzky D, Leen T. The MIT Press, Cambridge, Mass, USA; 1995:231-238.Google Scholar
  51. Tresp V: Committee machines. In Handbook for Neural Network Signal Processing. Edited by: Hu YH, Hwang J-N. CRC Press, Boca Raton, Fla, USA; 2001.Google Scholar
  52. Whissel CM: The dictionary of affect in language. In Emotion: Theory, Research and Experience. The Measurement of Emotions. Volume 4. Edited by: Plutchnik R, Kellerman H. Academic Press, New York, NY, USA; 1989:113-131.Google Scholar
  53. Ioannou S, Raouzaiou AT, Tzouvaras VA, Mailis TP, Karpouzis K, Kollias S: Emotion recognition through facial expression analysis based on a neurofuzzy network. Neural Networks 2005,18(4):423-435. 10.1016/j.neunet.2005.03.004View ArticleGoogle Scholar
  54. Klir GJ, Yuan B: Fuzzy Sets and Fuzzy Logic: Theory and Applications. Prentice-Hall, Upper Saddle River, NJ, USA; 1995.MATHGoogle Scholar
  55. Lee MA, Takagi H: Integrating design stages of fuzzy systems using genetic algorithms. Proceedings of the 2nd IEEE International Conference on Fuzzy Systems (FUZZY'93), March-April 1993, San Francisco, Calif, USA 612-617.Google Scholar
  56. Wallace M, Kollias S: Possibilistic evaluation of extended fuzzy rules in the presence of uncertainty. Proceedings of the 14th IEEE International Conference on Fuzzy Systems (FUZZ '05), May 2005, Reno, Nev, USA 815-820.Google Scholar
  57. Weizenbaum J: ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM 1966,9(1):36-45. 10.1145/365153.365168View ArticleGoogle Scholar
  58. Zhang YJ: A survey on evaluation methods for image segmentation. Pattern Recognition 1996,29(8):1335-1346. 10.1016/0031-3203(95)00169-7View ArticleGoogle Scholar
  59. Williams GW: Comparing the joint agreement of several raters with another rater. Biometrics 1976,32(3):619-627. 10.2307/2529750View ArticleMATHGoogle Scholar
  60. Chalana V, Kim Y: A methodology for evaluation of boundary detection algorithms on medical images. IEEE Transactions on Medical Imaging 1997,16(5):642-652. 10.1109/42.640755View ArticleGoogle Scholar
  61. Cowie R, Douglas-Cowie E, Savvidou S, McMahon E, Sawey M, Schröder M: 'Feeltrace': an instrument for recording perceived emotion in real time. Proceedings of the ISCA Workshop on Speech and Emotion, September 2000, Belfast, Northern Ireland 19-24.Google Scholar
  62. Cristinacce D, Cootes TF: A comparison of shape constrained facial feature detectors. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG '04), May 2004, Seoul, South Korea 375-380.Google Scholar


© Spiros Ioannou et al. 2007

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.