Face recognition using color local binary pattern from mutually independent color channels
© Anbarjafari; licensee Springer. 2013
Received: 19 September 2012
Accepted: 1 November 2012
Published: 22 January 2013
In this article, a high performance face recognition system based on local binary pattern (LBP) using the probability distribution functions (PDFs) of pixels in different mutually independent color channels which are robust to frontal homogenous illumination and planer rotation is proposed. The illumination of faces is enhanced by using the state-of-the-art technique which is using discrete wavelet transform and singular value decomposition. After equalization, face images are segmented by using local successive mean quantization transform followed by skin color-based face detection system. Kullback–Leibler distance between the concatenated PDFs of a given face obtained by LBP and the concatenated PDFs of each face in the database is used as a metric in the recognition process. Various decision fusion techniques have been used in order to improve the recognition rate. The proposed system has been tested on the FERET, HP, and Bosphorus face databases. The proposed system is compared with conventional and the state-of-the-art techniques. The recognition rates obtained using FVF approach for FERET database is 99.78% compared with 79.60 and 68.80% for conventional gray-scale LBP and principle component analysis-based face recognition techniques, respectively.
KeywordsIllumination robust Local binary pattern Face recognition Probability distribution function Discrete wavelet transform Kullback–Leibler distance
Face recognition has been one of the most interesting research topics for over the past half century. During this period, many methods such as principle component analysis (PCA), linear discriminant analysis (LDA), independent component analysis (ICA), etc., have been introduced [1–5]. Many of these methods are based on gray-scale images; however, color images are increasingly being used since they add additional biometric information for face recognition [6–8]. As reported by Demirel and Anbarjafari [6, 8], color probability distribution functions (PDFs) of a face image can be considered as the signature of the face, which can be used to represent the face image in a low-dimensional space. It is known that PDF of an image is a normalized version of an image histogram . PDF recently has been used in many applications of image processing such as object detection, face localization, and face recognition [6, 8–12].
One of the most important steps in a face recognition system is face segmentation. There are various methods for segmentation of the faces such as skin color-based face segmentation [13, 14], Viola–Jones  face detection system, local successive mean quantization transform (SMQT)-based face detection [16, 17]. In this study, we are using local SMQT-based face segmentation followed by skin color-based face segmentation. This procedure will reduce the effect of background on the rectangle-shape segmented face image.
In this article, the PDF-based face recognition will be studied analytically and then LBP will be used in order to boost the recognition performance. Also in this article, instead of experimentally choosing PDFs of HSI and YCbCr color channels [6, 8], analytically specific color channels have been selected. Furthermore, analytical studies of false acceptance rate (FAR) and false rejection rate (FRR) analysis are included in the third section. The head pose (HP) face database  with 15 subjects, a subset of 50 subjects from the FERET  database with faces containing varying poses changing from –90° to +90° of rotation around the vertical axis passing through the neck (the same subset as Demirel and Anbarjafari used in [6, 8]), and Bosphorus face database  with 105 subjects with varying frontal illuminations, poses, expressions, and occlusions were used to test the proposed system.
Facial images pre-processing
Image illumination enhancement
In many image processing applications, the general histogram equalization (GHE) is one of the simplest and most effective primitives for contrast enhancement , which attempts to produce an output histogram that is uniform . One of the disadvantages of the GHE is that the information laid on the histogram or PDF of the image will be lost. Demirel and Anbarjafari  have showed that the PDF of face images can be used for face recognition, hence preserving the shape of PDF of the image is of vital importance. Therefore, GHE is not a suitable technique for illumination enhancement of face images. Also, it is known that GHE often produces unrealistic effects in images. After the introduction of GHE, researchers came out with better technique which deals with equalization of portion of the image at a time, called local histogram equalization (LHE). LHE can be expressed as follows: GHE can be applied independently to small regions of the image. Most small regions will be very self-similar. If the image is made up of discrete regions, most small regions will lie entirely within one or the other region. If the image has more gradual large-scale variation, most small regions will contain only a small portion of the large-scale variation.
However, the contrast issue is yet to be improved and even these days many researchers are proposing new techniques for image equalization. DHE is obtained from dynamic histogram specification  which generates the specified histogram dynamically from the input image.
where U A and V A are orthogonal square matrices known as hanger and aligner, respectively, and Σ A matrix contains the sorted SVs on its main diagonal. The idea of using SVD for image equalization comes from this fact that Σ A contains the intensity information of the given image . The objective of SVE  is to equalize a low-contrast image in such a way that the mean moves towards the neighborhood of 8-bit mean gray value 128 in the way that the general pattern of the PDF of the image is preserved. Demirel and Anbarjafari  used SVD to deal with the illumination problem in their proposed face recognition system. SVE can be described in the following way: the ratio of the largest SV of the generated normalized matrix over a normalized image. This coefficient can be used to regenerate an equalized image. This task is eliminating the illumination problem. It is important to mention that techniques such as DHE or SVE are preserving the general pattern of the PDF of an image.
Face localization and segmentation
A face is naturally recognizable by a human regardless of its many point of variation such as skin tone, facial hair, etc. Face detection is a required first step in face recognition systems [16, 29]. The most straight forward variety of face localization is the detection of a single face at a known scale and orientation, which is yet a non-trivial problem. Efficient fast face detection is an impressive goal, which is subject to face tracking that required no knowledge of previous frames . Another reason that face detection is an important research issue is its role as a challenging case of a more general problem, object detection.
Skin is a widely used feature in human image processing with applications ranging from face detection  and person tracking  to content filtering . Human skin can be detected by identifying the presence of skin color pixels. Many methods have been proposed for achieving this. Chai and Ngan  modeled the skin color in YCbCr color space. In their technique, pixels are classified into “skin” and “non-skin” by using four threshold values, which form a rectangular region in CbCr space.
All color spaces such as RGB, HIS, and YCbCr can be used for face recognition . The advantage of using HSI color space is its independence of knowledge of the exact percentage of red, green, or blue. Many applications such as machine vision use HSI color space in identifying the color of different objects.
Kjeldson and Kender  stated a color preference in HSI color space to distinguish skin regions from other segments. Skin color classification in HSI color space is based on hue and saturation values.
In order to eliminate the illumination effect from the input images, the intensity component of an image in HSI color space has been equalized.
A different approach to separating faces and non-faces in image space is proposed by Osuna et al.  and later developed and modified by Romdhani et al. . Both are based on support vector machines (SVM) . The key to the SVM model is the choice of a manifold that separates the face set from the non-face set. Romdhani et al. had chosen a hyperplane which maximizes minimum distance on either side. Romdhani et al. worked further on reducing the vector set in order to improve performance.
where M(x) is a new set of values which are insensitive to gain and bias . These two properties are desired for the formation of the intensity image which is a product of reflection and illumination. A common approach to separate the reflection and illumination is based on this assumption that illumination is spatially smooth so that it can be taken as a constant in a local area. Therefore, each local pattern with similar structure will yield the similar SMQT features for a specified level, L. The spare network of winnows (SNoW) learning architecture is also employed in order to create a look-up table for classification .
As Nilsson et al.  proposed, in order to scan an image for faces, a patch of 32 × 32 pixels is used and also the image is downscaled and resized with a scale factor to enable the detection of faces with different sizes. The choice of the local area and the level of the SMQT are vital for successful practical operation. The level of the transform is also important in order to control the information gained from each feature. As reported in , the 3 × 3 local area and level L = 1 are used to be a proper balance for the classifier. The face and non-face tables are trained in order to create the split up SNoW classifier. Overlapped detections are disregarded using geometrical locations and classification score. Hence, given two detections overlapping each other, the detection with the highest classification score is kept and the other one is removed. This operation is repeated until no overlapping detection is found.
it is fast and very accurate;
it is a state-of-the-art technique;
if the input image does not have a face image, there will be no output, therefore in the proposed face recognition system there will be no issue of having a noise image, whose PDF is the same as a face image, as an input.
PDF based face recognition by using LBP
Analytical point of view
Recognition rate performance (%) of the PDF-based face recognition system for gray-scale PDFs of HP face database with 15 subjects and 10 samples per each subject obtained by using four different metrics
# of Training Images
Recognition of face images of the HP face database by using different bin numbers
Number of bins
Recognition rates (%)
The discrimination of the PDF by using KLD in different color channels and different databases
AVERAGE WITHIN CLASS DISTANCE
AVERAGE BETWEEN CLASS DISTANCE
CLASS DISCRIMINATION øc
In Table 3 class discrimination, ø c, is defined to be the ratio of the average between-class distance and the average within class distance which is indicating the discrimination power of different color channels. Class discrimination values show that KLD provides enough separation between classes in different color channels in PDF-based face recognition.
The correlation between HSI and YCbCr color channels in percentage
Table 4 shows the average mutual information in percentage between the various color channels in HSI and YCbCr color spaces for Bosphorus face databases where there exist over 4,500 face images. The high correlation between I-Y, I-Cb, and I-Cr color channels shows that instead of using both color spaces, using only HSI will have enough information in order to get conclusive recognition rate after the fusion. Also Table 4 indicates that the color channels in YCbCr are highly correlated with each other.
Local binary pattern
By definition, the LBP operator is unaffected by any monotonic gray-scale transformation which preserves the pixel intensity order in a local neighborhood. Note that each bit of the LBP code has the same significance level and that two successive bit values may have a totally different meaning. Sometimes, the LBP code is referred as a kernel structure index.
Ojala et al.  extended their previous study to a circular neighborhood of different radius size. They used LBP P,R notation which refers to P equally spaced pixels on a circle of radius R. Two of the main motivations of using LBP are its low computational complexity and its texture discriminative property. LBP has been used in many image processing applications such as motion detection , visual inspection , image retrieval , face detection , and face recognition [47, 48].
In most aforementioned applications, a face image was usually divided into small regions. For each region, a cumulative histogram of LBP code computed at each pixel location within the region was used as a feature vector.
In this histogram, a description of the face on three different levels of locality exists: the labels for the histogram contain information about the patterns on a pixel level, the labels are summed over a small region to produce information on a regional level, and the regional histograms are concatenated to build a global description of the face.
Although Ahnon et al.  have mentioned several dissimilarity measures such as histogram intersections, log-likelihood statistics, and Chi square statistics, they used nearest neighbor classifier in their study.
When the image has been divided into several regions, it can be expected that some of the regions contain more useful information than others in terms of distinguishing between people, such as eyes [49, 50]. In order to contribute such information, a weight can be set for each region based on the level of information it contains.
The proposed LBP-based face recognition
Performance of different decision-making techniques for the proposed face recognition system
# of training image
Performance of the proposed LBP based face recognition system using FVF, PCA, LDA, conventional gray scale LBP, PDF based face recognition, NMF, and INMF based face recognition system for the FERET face databases with 50 subjects and 10 samples per each subject
# of training images
The median rule and FVF-based results are 97.17 and 99.78% for the FERET face database, when five samples per subject are available in the training set, respectively. These results are significant, when compared with the recognition rates achieved by conventional PCA and LDA and the state-of-the-art techniques such as LBP, NMF, and INMF-based face recognition system.
In this article, we have studied a high performance frontal homogenous illumination robust face recognition system using LBP and PDFs in different mutually independent color channels. A face localization which is a combination of local SMQT technique followed by skin tone-based face detection method was employed in this study. DWT + SVD-based image illumination enhancement technique was also applied in order to reduce the effect of illumination. The article analytically analyzed and justified the use of KLD and PDF with a bin size of 256 which was introduced and used in [6, 8]. Several well-known decision fusion techniques have been used in order to combine the decisions obtained from mutually independent color channels. Also an FAR and FRR analysis has been done in this study. Finally, comparison between the proposed method and the conventional and the state-of-the-art techniques has been done which showed the superiority of the proposed method.
The author would like to thank Asst. Prof. Dr. Mikael Nilsson from Blekinge Institute of Technology, for providing the algorithm for local SMQT-based face recognition. Also the author would like to thank Prof. Dr. Ivan Selesnick from Polytechnic University for providing the DWT codes in MATLAB.
- Bledsoe WW: The Model Method in Facial Recognition, Technical report pri 15. Panoramic Research, Inc, Palo Alto, California; 1964.Google Scholar
- Turk M, Pentland A: Face recognition using eigenfaces. In Proceeding of International Conference On Computer Vision and Pattern Recognition. Maui, Hawaii: IEEE; 1991:586-591.Google Scholar
- Belhumeur P, Hespanha J, Kreigman D: Eigenfaces vs. Fisherfaces: Recognition using Class Specific Linear Projection. IEEE Trans Pattern Anal Mach Intell 1997, 19: 711-720. 10.1109/34.598228View ArticleGoogle Scholar
- Liu C, Wechsler H: Comparative assessment of independent component analysis (ICA) for face recognition. In Proceedings of the Second International Conference on Audio- and Video-based Biometric Person Authentication. Washington D.C., USA; 1999:211-216. 22-24 MarchGoogle Scholar
- Bartlett MS, Movellan JR, Sejnowski TJ: Face Recognition by Independent Component Analysis. IEEE Trans Neural Netw 2002, 13(6):1450-1464. 10.1109/TNN.2002.804287View ArticleGoogle Scholar
- Demirel H, Anbarjafari G: Pose Invariant Face Recognition using Probability Distribution Functions in Different Color Channels. IEEE Signal Process Lett 2008, 15: 537-540.View ArticleGoogle Scholar
- Marcel S, Bengio S: Improving face verification using skin color information. In Proceeding 16th International Conference on Pattern Recognition. Society Press, Quebec, Canada; 2002:378-381.Google Scholar
- Demirel H, Anbarjafari G: Data fusion boosted face recognition based on probability distribution functions in different colour channels. EURASIP J Adv Signal Process 2009, 2009: 1-10.View ArticleGoogle Scholar
- Anbarjafari G, Demirel H: Face Recognition Using Colour Statistics. VDM Verlag: Biometric Recognition; 2010.Google Scholar
- Laptev I British Machine Vision Conference. In Improvements of object detection using boosted histograms. Edinburgh, UK; 2006:949-958.Google Scholar
- Tae-WoongYoo I-SO: A fast algorithm for tracking human faces based on chromatic PDFs. Pattern Recognit Lett 1999, 20: 967-978. 10.1016/S0167-8655(99)00053-7View ArticleGoogle Scholar
- Rodriguez Y, Marcel S Proceedings of the 9th European Conference on Computer Vision. In Face authentication using adapted local binary pattern histograms. Graz, Austria; 2006:321-332.Google Scholar
- Yang MH, Kriegman D, Ahuja N: Detecting faces in images: A survey. IEEE Trans Pattern Anal Mach Intell 2002, 24: 34-58. 10.1109/34.982883View ArticleGoogle Scholar
- Chai D, Ngan KN: Face segmentation using skin color map in videophone application. IEEE Trans Circ Syst Video Tech 9th edition. 1999, 551-564.Google Scholar
- Viola P, Jones M: Robust real-time face detection. Int J Comput Vis 2004, 52(2):137-154.View ArticleGoogle Scholar
- Nilsson M, Nordberg J, Claesson I IEEE International Conference on Acoustics, Speech, and Signal Processing. In Face detection using local SMQT features and split up snow classifier. 2nd edition. Hawaii, USA; 2007:589-592.Google Scholar
- A face detector software written by Mike Nilsson, provided in MathWorks exchange file. 2012. http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13701
- Gourier N, Hall D, Crowley JL: Estimating Face Orientation from Robust Detection of Salient Facial Features. In Proceedings of Pointing 2004, ICPR, International Workshop on Visual Observation of Deictic Gestures. Cambridge, UK;
- Philips PJ, Moon H, Rizvi SA, Rauss PJ: The FERET Evaluation Methodology for Face Recognition Algorithm. IEEE Trans Pattern Anal Mach Intell 2000, 22(10):1090-1104. 10.1109/34.879790View ArticleGoogle Scholar
- Savran A, Alyüz N, Dibeklioğlu H, Celiktutan O, Gökberk B, Sankur B, Akarun L: Bosphorus database for 3D face analysis. In The First COST 2101 Workshop on Biometrics and Identity Management (BIOID 2008). Denmark: Roskilde University; 2008:47-56.Google Scholar
- Kim TK, Paik JK, Kang BS: Contrast enhancement system using spatially adaptive histogram equalization with temporal filtering. IEEE Trans Consum Electron 1998, 44(1):82-86. 10.1109/30.663733View ArticleGoogle Scholar
- Weeks AR, Sartor LJ, Myler HR: Histogram specification of 24-bit colour images in the colour difference (C-Y) colour space. Proc SPIE 1999, 3646: 319-329. 10.1117/12.341098View ArticleGoogle Scholar
- Sun CC, Ruan SJ, Shie MC, Pai TW: Dynamic contrast enhancement based on histogram specification. IEEE Trans Consum Electron 2005, 51(4):1300-1305. 10.1109/TCE.2005.1561859View ArticleGoogle Scholar
- Demirel H, Anbarjafari G, Jahromi MNS: Image equalization based on singular value decomposition. In 23rd IEEE International Symposium on Computer and Information Sciences. Istanbul, Turkey: Istanbul, Turkey; 2008:1-5. 27-29 OctoberGoogle Scholar
- Tian Y, Tan T, Wang Y, Fang Y: Do singular values contain adequate information for face recognition? Pattern Recogn 2003, 36: 649-655. 10.1016/S0031-3203(02)00105-XView ArticleGoogle Scholar
- Ozcinar C, Demirel H, Anbarjafari G: Satellite Image Contrast Enhancement Using Discrete Wavelet Transform and Singular Value Decomposition. IEEE Geosci Rem Sens Lett 2010, 7(2):334-338.Google Scholar
- Kahraman F, Kurt B, Gokmen M: Robust face alignment for illumination and pose invariant face recognition. In IEEE Conference on Computer Vision and, Pattern Recognition. Minnesota, USA; 2007:1-7. 17-22 JuneGoogle Scholar
- Abdullah-Al-Wadud M, Kabir MH, Dewan A, Chae O: A Dynamic Histogram Equalization for Image Contrast Enhancement. IEEE Trans Consum Electron 2007, 53(2):593-600.View ArticleGoogle Scholar
- Sung KK, Poggio T: Example based learning for view based human face detection. IEEE Trans Pattern Recogn Mach Intell 1998, 20: 39-51. 10.1109/34.655648View ArticleGoogle Scholar
- Matsumoto Y, Zelinsky A: Real-time stereo face tracking system for visual interface. In International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time. Corfu, Greece; 1999:77-82.Google Scholar
- Schwerdt K, Crowley J: Robust face tracking using colour. In 4th IEEE International Conference on Automatic Face and Gesture Recognition. Grenoble, France; 2000:90-95.Google Scholar
- Huicheng Z, Hongmei L, Daoudi M IEEE Conference on Multimedia and Expo. In Blocking objectionable images: adult images and harmful symbols. Lausanne, Switzerland; 2004:1223-1226.Google Scholar
- Singh SK, Chauhan DD, Vatsa M, Singh R: “A robust skin colour based face detection algorithm”. Tamkang J Sci Eng 2003, 6(4):227-234.Google Scholar
- Kjeldsen R, Kender J: Finding skin in colour images. In Proceedings of the Second International Conference on Automatic Face and Gesture Recognition. Vermont, USA; 1996:312-317. 14-16 OctoberView ArticleGoogle Scholar
- Anbarjafari G: A new face recognition system based on colour statistics. MS Thesis: Eastern Mediterranean University (EMU), Gazimağusa, TRNC; 2008.Google Scholar
- Osuna E, Freund R, Girosi F: Training support vector machines: an application to face detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR ’97). San Juan, Puerto Rico; 1997:130-136.View ArticleGoogle Scholar
- Romdhani S, Torr P, Scholkopf B, Blake A 8th IEEE International Conferences on Computer Vision. In Computationally efficient face detection. British Columbia, Canada; 2001:695-700.Google Scholar
- Cortes C, Vapnik V: Support vector networks. Mach Learn 1995, 20(3):273-297.Google Scholar
- Nilsson M, Dahl M, Claesson I IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). In The successive mean quantization transform. Philadelphia, USA; 2005:429-432.Google Scholar
- Roth D, Yang M, Ahuja N Advances in Neural Information Processing Systems. In A snow based face detector. Edited by: Solla SA, Leen TK, Muller K-R. MIT Press; 2000:855-861.Google Scholar
- Ojala T, Pietikainen M, Harwood D: A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 1996, 29: 51-59. 10.1016/0031-3203(95)00067-4View ArticleGoogle Scholar
- Ojala T, Pietikainen M, Maenpaa T: Multiresolutiongray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 2002, 24: 971-987. 10.1109/TPAMI.2002.1017623View ArticleGoogle Scholar
- Heikkila M, Pietikainen M, Heikkila J British Machine Vision Conference. In A texture based method for detecting moving object. London, England; 2004:187-196.Google Scholar
- Turtinen M, Pietikainen M, Silven O: Visual characterization of paper using isomap and local binary patterns. Conference Mach Vision Applic 2005, E-89-D: 1-8.Google Scholar
- Takala V, Ahonen T, Pietikainen M: Block-based methods for image retrieval using local binary patterns. In 14th Scandinavian Conference on Image Analysis. Joensuu, Finland; 2005:882-891.View ArticleGoogle Scholar
- Jin H, Liu Q, Lu H, Tong X: Face detection using improved LBP under Bayesian framework. In 3rd International Conference on Image and Graphics. Porto, Portugal; 2004:306-309. 18-20 DecemberGoogle Scholar
- Zhang W, Shan S, Gao W, Chen X, Zhang H 10th IEEE International Conference on Computer Vision. In Local Gabor binary pattern histogram sequence: a novel non-statistical model for face representation and recognition. Beijing, China; 2005:786-791. 17-21 OctoberGoogle Scholar
- Ahnon T, Hadid A: Pietikainen, Face recognition with local binary patterns. In European Conference on Computer Vision. Prague, Czech Republic; 2004:469-481.Google Scholar
- Gong S, McKenna SJ, Psarrou A: Dynamic Vision, From Images to Face Recognition. London: Imperial College Press; 2000.View ArticleGoogle Scholar
- Zhao W, Chellapa R, Rosenfeld A, Philips PJ: Face Recognition: A Literature Survey, Technical report CAR-TR-948. University of Maryland: Centre for Automation Research; 2002.Google Scholar
- Polikar R: Ensemble Based Systems in Decision Making. IEEE Circ Syst Mag 2006, 6(3):21-45.View ArticleGoogle Scholar
- Lee DD, Seung HS: Learning the parts of objects by nonnegative matrix factorization. Nature 1999, 401: 788-791. 10.1038/44565View ArticleGoogle Scholar
- Lee DD, Seung HS: Algorithms for nonnegative matrix factorization. Adv Neural Inform Process Syst 2001, 13: 556-562.Google Scholar
- Chen WS, Pan BB, Fang B, Li M, Tang JL: Incremental nonnegative matrix factorization for face recognition. Mathematical Problems in Engineering 2008, 1-17. VolGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.