Efficient face detection method with eye region judgment

Abstract

Real-time face detection technology can be applied in many industrial or commercial products. Many face detection applications use the traditional Adaboost face detection system which is proposed by Viola and Jones in 2004. Viola and Jones used the Adaboost training algorithm and the Haar-like features in their proposed traditional Adaboost face detection system, which has a high detection rate but long training time. Many studies have attempted to reduce the training time and retain the high detection rate of the traditional Adaboost face detection system. However, the detection rate of the Adaboost-based face detection system cannot compete with the traditional Adaboost face detection system when the training time is reduced significantly. This study proposes the judging existence of eye region (JEER) method to enhance the detection rate of the previous Adaboost-based face detection systems. The eyes are more salient and representative features than the other facial parts such as the mouth or ears, especially when mask is worn on the human face. Therefore, the proposed face detection system with the JEER method achieves higher detection rate than the Adaboost-based face detection systems. Although the JEER computation results in a slightly longer training time, the training time of the proposed face detection system is still much shorter than the traditional Adaboost face detection system owing to the efficient JEER computation. The experimental results obtained using the gray FERET and CMU databases show that the proposed face detection system is effective in detection and efficient in training.

1. Introduction

Face detection technology is important owing to its applicability in many fields such as in statistics and security services [1, 2]. Many useful face detection systems use various basic theories [3–6]. Among these basic theories, Viola and Jones chose the Adaboost algorithm to train every Haar-like feature for a weak classifier [7, 8]. Cascading all weak classifiers yields a strong face detection classifier in the face detection system. Although the detection performance of this strong classifier is high, its training time is exceedingly long because of the large number of Haar-like features that needs to be trained for the weak classifiers. The recent Adaboost-based face detection systems have attempted to improve the traditional Adaboost face detection system in two aspects: one is the enhancement of the detection performance, and the other is the decrement of the training time [9–13]. Lienhart et al. presented the rotated Haar-like features to enhance the detection performance of rotated faces [14]. Guo et al. presented a two-stage hybrid face detection system composed of the probability-based face mask pre-filtering (PFMPF) and the pixel-based hierarchical feature Adaboosting (PBHFA) to reduce the training time [15]. First, PFMPF does not use the training features as classifiers but employ a probability-based face mask to exclude the nonface. Therefore, no training time is required by PFMPF. Subsequently, PBHFA trains the hierarchical features, which are much different from the Haar-like features, as the Adaboost strong classifier. Because the number of hierarchical features is significantly fewer than that of the Haar-like features, the training time of the two-stage hybrid face detection system is much shorter than the traditional Adaboost face detection system.

However, the detection rate of the Adaboost-based face detection systems, which reduces much the training time similar to the two-stage hybrid face detection system proposed by Guo et al., cannot compete with the traditional Adaboost face detection system. Therefore, some critical facial properties must be considered to enhance the detection rate of the Adaboost-based face detection systems, where the training time is much shorter than the traditional Adaboost face detection system. The eyes are the most representative features of the human face, especially when masks are put on the face. Although many studies have accurately detected the location of the eyes on human faces, the computation of these eye detection methods is very complex and is only suitable for high-resolution face images [16, 17]. Since the faces detected in natural images have a low resolution, these eye detection methods cannot be applied in face detection systems.

Therefore, the present study proposes the judging existence of eye region (JEER) algorithm, which is suitable for low-resolution facial image in natural images, to enhance the detection rate of the Adaboost-based face detection systems. In this study, JEER is applied to the Adaboost-based face detection system, which is a hybrid of PFMPF and PBHFA, proposed by Guo et al.

The proposed face detection system achieves a higher detection rate and a lower false positive rate than the Adaboost-based face detection system which is hybrid from PFMPF and PBHFA, and the training time of the proposed system is still much shorter than the traditional Adaboost face detection system.

The remaining parts of this paper are structured as follows: Section 2 briefly reviews the traditional Adaboost face detection system that uses the Haar-like features. Section 3 introduces the JEER algorithm. Section 4 presents the application of JEER to the hybrid face detection system of PFMPF and PBHFA. Section 5 presents the experimental results of the proposed face detection system. Section 6 presents the conclusion.

2. Background

The traditional Adaboost face detection system has been used in many industrial and commercial applications. Viola and Jones used the Adaboost training algorithm and the Haar-like features in proposing the traditional Adaboost face detection system in 2004, which has a high detection rate but a long training time. The Haar-like feature f _j is a rectangular region at a specific location in the detection window of an image. Figure 1 shows four Haar-like features overlaid on a typical training face image, which is defined as a detection window. A Haar-like feature can be at any location and scale within the detection window. Therefore, the number of Haar-like features in the detection window is very large. The feature value f _j (x _i ) of a Haar-like feature f _j is defined as the difference in the sum of the pixel values of the white and block areas inside the rectangle in the detection window x _i .

Figure 1

Haar-like features in a detection window.

The traditional Adaboost face detection system employs the Adaboost algorithm to train each Haar-like feature f _j for the weak classifier h _j (x, f _j , p, θ) in the detection window [8]. The weak classifier h _j (x, f _j , p, θ) is a function of the detection window x _i , Haar-like feature f _j , polarity p, and threshold θ. Subsequently, the Adaboost algorithm selects the best weak classifier h _t (x, f _t , p, θ) in each round. Finally, it cascades every best weak classifier of the rounds to a strong classifier. Algorithm 1 describes the Adaboost algorithm [8].

Algorithm 1 Adaboost algorithm

Since the number of Haar-like features in a detection window is very large, training all Haar-like features for the weak classifiers and selecting the best weak classifier in each round to a strong classifier require an exceedingly long time.

3. Judging existence of eye region

The eyes are more salient and representative than other facial features such as the mouth or ears when there is mask on a human face. A human face must be present in a detection window when human eyes are present in the window. Conversely, a human face could not be present in a detection window when human eyes are not present in the window. Therefore, this section presents the JEER algorithm, which assists the Adaboost-based face detection system in detecting human faces.

JEER circles the possible areas of the eye region in a detection window and judges whether a human eye might be present in the detection window, that is, human eyes might be present in the eye region area, which is defined by JEER from the possible areas of the eye region. Although the presence of a human face in the detection window is not consequential when the eye region area is present, a human face in the detection window cannot be present when no eye region area is present. Therefore, no human face can exist in a detection window when JEER judges that no eye region area is present. Because the JEER computation is extremely simple, it can exclude quickly the nonface. Furthermore, JEER indirectly reduces the face detection time because most parts of the natural images are nonface.

Although JEER can only filter out nonface and cannot detect a human face by itself, the eye region area, which can be defined by JEER, can be applied to PFMPF and PBHFA to assist effectively in detecting the human face to obtain a higher detection rate and a lower false positive rate, as presented in Sections 4.1 and 4.2, respectively.

The JEER procedure is divided into two steps, namely, Step 1 and Step 2. Step 1 (‘possible areas of the eye region’), which is described in Section 3.1, circles the possible areas of the eye region in the detection window. Step 2 (‘necessary condition for existence of the eye region area’), which is described in Section 3.2, indicates the necessary condition for the existence of the eye region area in the detection window. Finally, the nonface image is filtered, or the eye region area is defined. The details of the JEER algorithm are presented in Algorithm 2.

Algorithm 2 Judging existence of eye region (JEER) algorithm

3.1 Step 1 (possible areas of the eye region)

First, human eyes in a human face image are analyzed. Figure 2a shows a human face in the detection window. The nose tip is often near the center. Drawing four straight lines from the center to each boundary of the detection window would hit four pixels, as shown in Figure 2a. Because the frontal bones reflect more light when a light source is near the front, the gray pixel value of the frontal bone is higher than those of the other three pixels. As soon as the frontal bone is identified, the bridge of the nose can be defined as the straight line from the nose tip to the frontal bone. Because human eyes are located symmetrically relative to the bridge of the nose, they are symmetrically relative to the straight line from the center to the brightest pixel among the four pixels (Figure 2a).

Figure 2

Human face in a detection window. (a) Light source is near the front. (b) Light source is approximately lateral.

Therefore, the eye region area of the human face image is circled, as shown in Figure 3. In other words, the possible areas of the eye region in the detection window that could possibly contain a face can be circled, as shown in Figure 3. Figure 3a shows the possible areas of the eye region when the size N of the detection window is odd, whereas Figure 3b shows those when N is even.

Figure 3

The possible areas of the eye region in a detection window. (a) The size N of the detection window is odd. (b) The size N of the detection window is even.

However, the frontal bone is not the brightest pixel among the four pixels when the light source is far from the front. The side that faces the light is the brightest pixel when the light source is completely from the side. Figure 2b shows that the brightest pixel is not the frontal bone but the right side that faces the light. Therefore, the straight line from the center to the brightest pixel is not the bridge of the nose. Under this condition, human eyes are nonsymmetrical relative to the straight line from the center to the brightest pixels. Thus, JEER uses Step 2 (necessary condition for existence of the eye region area) to determine whether human eyes might be present in the possible areas of the eye region chosen by Step 1.

3.2 Step 2 (necessary condition for existence of the eye region area)

First, the characteristic of the eye region area in a human face image is analyzed. The eye region area is composed of two parts: the left and right eyes. Because the human face is symmetrical, the summation of the gray pixel values in each part in the eye region area is similar regardless whether glasses are worn or not. The summation of the gray pixel values in each part in the eye region area is very close when the light source is near the front, as shown in Figure 2a. Even when the light source is far from the front but completely lateral, as shown in Figure 2b, the summation of the gray pixel values in one part is not more than triple that of the other part. This characteristic, which is found in this study, is named as the symmetry of the summation of the gray values in the eye region area.

The possible areas of the eye region chosen by Step 1 are defined as the eye region area only when the symmetry of the summation of the gray values in the eye region area is satisfied. For example, the first possible eye region area chosen by Step 1 can be the right eye and the right cheek, as shown in Figure 2b. Because the first possible eye region area does not satisfy the symmetry of the summation of the gray values in the eye region area characteristic, it cannot be an eye region area. Thus, JEER subsequently chooses the second brightest pixel among the four pixels as the front bone and decides the second possible eye region area. Because the second possible eye region area satisfies the symmetry of the summation of the gray values in the eye region area characteristic, it is chosen as an eye region area. The details are described in Algorithm 2. Therefore, no human face exists in the detection window when all four possible areas of the eye region do not satisfy the symmetry of the summation of the gray values in the eye region area characteristic, that is, this characteristic is a necessary condition for the existence of an eye region area in the detection window.

4. Proposed face detection system

This section presents the proposed face detection system that applies JEER to the hybrid Adaboost-based face detection system of PFMPF and PBHFA, which was proposed by Guo et al., to reduce the training time of the traditional system [15]. PFMPF does not use the training features as classifiers but employ a probability-based face mask to exclude the nonface. PBHFA trains the hierarchical features as the Adaboost strong classifier. PFMPF does not need a training time, and the number of PBHFA features is significantly fewer than that of the Haar-like features. Therefore, the training time of the two-stage hybrid face detection system of PFMPF and PBHFA is much shorter than that of the traditional Adaboost system. However, the detection rate of the two-stage hybrid face detection system cannot compete with the traditional system. To enhance the detection rate, JEER is applied to PFMPF and PBHFA, as presented in Sections 4.1 and 4.2, respectively.

In Section 4.1, the eye region area defined by JEER is applied to PFMPF as a crucial area whose initial weight must be emphasized. In Section 4.2, the four possible areas of the eye region identified by Step 1 of the JEER procedure are applied to PBHFA as crucial hierarchical features.

4.1 Applying JEER to PFMPF

The PFMPF algorithm is described in Algorithm 3. When a human face is present in the detection window, PFMPF observes that the pixel values of some specific positions are within some specific ranges, such as the position of the eyes, eyebrows, and cheek. Therefore, PFMPF gathers the statistics of the gray pixel value probability of a positive training image (face image) set. PFMPF builds the probability table P(x, y, GreyValue) for each position in the detection window from a positive training image set. The coordinate of the detection window is (x, y), and the gray pixel value is GreyValue. The probability table P(x, y, GreyValue) records the appearance probability of the gray pixel value GreyValue for each position (x, y) in the detection window from the positive training image set. Thereafter, PFMPF calculates the standard deviation D(x, y) of the gray pixel value of each position (x, y) in the detection window from the positive training image set. The initial weight W(x, y) is the reciprocal of D(x, y). The initial weight W(x, y) of the pixels with less variation in the gray value is higher, whereas that of the pixels with more variation in the gray value is lower.

Algorithm 3 PFMPF algorithm

When a test image X _i is present in the detection window with the size of 19 × 19, the Weight_Sum of the test image is computed as follows:

$$\mathrm{Weight}\_\mathrm{Sum}={\displaystyle \sum _{x=0}^{x=18}{\displaystyle \sum _{y=0}^{y=18}P\left(x,y,X_i\left(x,y\right)\right)}\times W\left(x,y\right)}.$$

(1)

From P(x, y, GreyValue), the value of P(x, y, X _i (x, y)) can be obtained, where X _i (x, y) is the gray pixel value in position (x, y) of the test image X _i . Therefore, the Weight_Sum of the positive test images is higher than that of the negative test images. If a threshold is set to classify the face and nonface test images, the following classification result can be obtained:

$$\mathrm{classify}=\left\{{}_{\mathrm{nonface},\begin{array}{}\end{array}\mathrm{otherwise}}^{\mathrm{face},\begin{array}{}\end{array}\mathrm{if}\begin{array}{}\end{array}\mathrm{Weight}\_\mathrm{Sum}\ge \mathrm{threshold}}\right..$$

(2)

PFMPF is used to exclude quickly the nonface using the hybrid Adaboost-based face detection system. The detection and the false positive rates of PFMPF are approximately 95.35% and 5.68%, respectively.

However, the detection and the false positive rates of PFMPF are insufficient because the probability table P(x, y, GreyValue) is inaccurate when a mask covers the face. Therefore, JEER is applied to PFMPF to obtain a higher detection rate and lower false positive rate than that of PFMPF alone.

The flowchart of the JEER application to PFMPF is shown in Figure 4, which shows that most nonface is excluded after the JEER is applied. Meantime, the eye region area defined by JEER is circled. Because the eye region area is the most representative evidence of a human face, the initial weight W(x, y) of the eye region area is emphasized before PFMPF is executed. The initial weight of the eye region area in the detection window is multiplied by 0.7, and those of the other regions are multiplied by 0.3. Thereafter, PFMPF computes the Weight_Sum and detects the human face in the detection window.

Figure 4

Flow chart of applying JEER to PFMPF.

Applying JEER to PFMPF yields a higher detection rate and a lower false positive rate than PFMPF can do alone. Further, PFMPF with JEER excludes nonface faster than PFMPF alone because the JEER computation is more efficient than the PFMPF. The MIT-CBCL face database is employed to calculate the probability table P(x, y, GreyValue) and initial weights W(x, y) [18]. In addition, the MIT-CBCL face database is employed to test the performance of the detection system. Figure 5a shows some positive training images, and Figure 5b shows some face and nonface test images of the MIT-CBCL face database. The detection system is tested on 600 images. A slightly improved detection performance by PFMPF with JEER is achieved compared with that of PFMPF alone. For the test samples, the detection and false positive rates are approximately 97.62% and 3.58%, respectively.

Figure 5

Examples of (a) positive training images and (b) face and nonface test images.

4.2 Applying JEER to PBHFA

PBHFA trains the hierarchical PBH features, which are much different from the Haar-like features, as Adaboost strong classifiers. The generation of the PBH features is described:

• 1. First, input training positive images X _i of size M × N.

• 2. The average value of each training image X _i is T _i , which is used as the threshold for the corresponding training image X _i to produce the thresholded binary image B _i . An example is shown in Figure 6a.

• 3. Given all the B _i , calculate the probability of black pixel occurrence at each position (x, y) and obtain P(x, y). Subsequently, the order table O(x, y) is created according to the probability P(x, y) distribution. Enter the value (1 ~ M × N) into O(x, y) with the probability value in position (x, y). This means, when the probability of position (x, y) is the maximum, set O(x, y) = 1, and O(x, y) = 2 when the probability of position (x, y) is the second highest, and so forth. An example is shown in Figure 6b.

• 4. All possible features F _{j _PBH} and the corresponding feature values in a M × N sub-window can be produced according to the order table O(x, y). First, the summation of the pixel values in the training image is calculated and denoted as sum. The PBH feature value F _{j _PBH} is computed as

  $$F_{j\_\mathrm{PBH}}=\mathrm{sum}-\left({\displaystyle \sum _{k=1}^{k=j}{X}_i\left(x,y\right)},\mathrm{where}O\left(x,y\right)=k\right),$$

  (3)

where X _i (x, y) denotes the pixel value of the training image X _i at position (x, y). Since PBH feature is obtained by subtracting the overall pixel values in the black region from the whole pixel values in a detection window, the sum of all the possible PBH features in M × N images is not over M × N.

• 5. Finally, these features and training image X _i are fed into the Adaboost algorithm. Similar to the Haar-like features, each PBH feature can be considered as a weak classifier. Through the Adaboost algorithm, the threshold θ _j and inequality p _j can be derived to classify face and nonface candidates.

Figure 6

Example of PBH feature selection. (a) A training image and corresponding binary image. (b) Probability of black pixel occurrence at each position (x, y) and corresponding order table.

PBHFA is used as the main classifier of the hybrid Adaboost-based face detection system. Because the number of hierarchical PBH features is significantly fewer than that of the Haar-like features, the training time of the two-stage hybrid face detection system is much shorter than the traditional Adaboost system.

However, the detection and false positives rate of the hybrid Adaboost-based face detection system cannot compete with the traditional Adaboost face detection system. Therefore, JEER is applied to PBHFA to obtain a higher detection rate and a lower false positive rate than that of the PBHFA alone.

The concept of applying JEER to PBHFA is shown in Figure 7. Four crucial JEER features are added to the PBH feature. These four JEER crucial features represent the four angles of the possible areas of the eye region identified by Step 1. The JEER feature value F _{j _JEER} is computed as

$$F_{j\_\mathrm{JEER}}=\mathrm{sum}-\left({\displaystyle \sum _x{\displaystyle \sum _y{X}_i\left(x,y\right)}},\mathrm{\text{where}}x,y\mathrm{\text{∈ the possible areas of the eye region}}\right),$$

(4)

where X _i (x, y) denotes the pixel value of the training image X _i at position (x, y).

Figure 7

Four crucial JEER features are added to the PBH feature.

5. Experimental results

In this section, the experimental results are presented and obtained using the CMU and FERET databases to compare the detection performance of the proposed face detection system, the hybrid Adaboost-based face detection system, and the traditional Adaboost face detection system that uses the Haar-like features [19, 20]. The MIT-CBCL face database is employed for the training of classifier, as presented in Section 4. The images in the CMU, FERET and MIT-CBCL face databases include variations to imitate real-world conditions, such as slight rotation, different facial expressions, different race, and different illumination directions. The experimental results were obtained from the tests on 450 images. Figure 8 shows some detection results of the proposed face detection system for the CMU and FERET test images. The CMU database has copious illumination variations. The FERET database has copious facial expressions.

Figure 8

Some detection results of the proposed face detection system.

The performance of a face detection system is normally evaluated based on the detection rate, false positive rate, training time, and detection time. Table 1 shows the detection and false positive rate comparisons among the proposed face detection system, the hybrid Adaboost-based face detection system, and the traditional Adaboost face detection system that uses the Haar-like features. All face detection systems use 200 features. The detection and false positive rates of the proposed system are better than that of the hybrid Adaboost-based face detection system and are close to that of the traditional Adaboost face detection system that uses the Haar-like features.

Table 1 Detection performance comparisons

Because training time mostly depends on the number of features, an increased number of features will result in additional training time in the detection window. The number of features in the traditional Adaboost face detection system that uses the Haar-like features is 300 times that of the PBH features. However, the practical training time is approximately 1,500 times that of the PBH features. The reason is that the large number of Haar-like features causes data swapping between the memory and the hard drive, whereas that of the proposed PBH features can be completely stored in the memory, which results in the large difference in the training time [15]. Because the number of features in the proposed system only added four crucial JEER features to the PBH features, its training time is slightly longer than that of the hybrid Adaboost-based face detection system but is significantly shorter than that of the traditional Adaboost face detection system.

Detection time depends on the computation complexity. Table 2 shows the computation complexity comparison among the proposed face detection system, the hybrid Adaboost-based face detection system, and the traditional Adaboost face detection system. The upper part of Table 2 shows the complexity proportion when the detection window size is N × N, and the lower part shows the number of operations when the detection window size is 24 × 24. The computation complexity of the traditional Adaboost face detection system is divided into integral image and K weak classifier calculations. The traditional Adaboost face detection system employs an integral image that uses 3 × N × N operations in the detection window of size N × N to calculate the difference in the sum of the pixel values between the white and block areas inside the rectangular feature, as described in Section 2. Thereafter, the integral image uses seven operations to add the feature values for the weak classifier when the weak classifier is a two-rectangle feature, which entails the least number of operations. Therefore, the best scenario for the integral image to sum up the feature values of the K weak classifiers is 7 × K [15]. The lower part of Table 2 shows that the best scenario is 3 × N × N + 7 × K = 3 × 24 × 24 + 7 × 17 = 1,847 when all 17 weak classifiers are two-rectangle features. Furthermore, if three-rectangle or four-rectangle features are involved, the number of operations will be higher than 1,847.

Table 2 Computation complexity comparisons

The computational complexity of the hybrid Adaboost-based face detection system is much lesser than that of the traditional Adaboost face detection system. PFMPF, which can filter 85% of the nonface images, uses N × N operations in the detection window of size N × N, and the PBH features use less than 2 × N × N operations for the weak classifiers because the sum of the PBH features in a detection window of size N × N is not greater than N × N [15]. Therefore, 85% of the images use N × N = 24 × 24 = 576 operations, and 15% of the images use less than (N × N) + 2 × (N × N) = (24 × 24) + 2 × (24 × 24) = 1,728 operations.

The computational complexity of the proposed face detection system is slightly higher than that of the hybrid Adaboost-based face detection system but is significantly shorter than that of the traditional Adaboost face detection system. Because the pixel number of the eye region area in the detection window is approximately 1/20, each detection window needs 2 × 0.05 × (N × N) operations to calculate the JEER when the four angles of the possible areas of the eye region are calculated. In other words, the proposed face detection system needs more than 0.1 × (N × N) operations than PFMPF and PBHFA because the four angles of the possible areas of the eye region is applied to PFMPF and PBHFA. The number of nonface images filtered by applying JEER to PFMPF is more than that of PFMPF alone. Therefore, 91% of the images use less than (N × N) + 0.1 × (N × N) = (24 × 24) + 0.1 × (24 × 24) ≒ 633 operations, and 9% of the images use less than (N × N) + 0.1(N × N) + (2 × N × N) + 0.1(N × N) = (24 × 24) + 0.1(24 × 24) + (2 × 24 × 24) + 0.1(24 × 24) ≒1,843 operations.

Therefore, the proposed face detection system not only has a significantly shorter training time than the traditional Adaboost face detection system when the detection and false positive rates are comparable with that of the traditional Adaboost face detection system but also a shorter detection time than the traditional Adaboost face detection system.

6. Conclusion

This study has proposed the JEER algorithm to improve the previous Adaboost-based face detection systems generally used in many industrial or commercial products. In this study, JEER was applied to the Adaboost-based face detection system, which is a hybrid of PFMPF and PBHFA, proposed by Guo et al.

Although the training and detection times of the proposed face detection system were slightly longer than the hybrid Adaboost-based face detector, they were still much shorter than those of the traditional Adaboost face detection system that uses the Haar-like features. Furthermore, the proposed face detection system achieved higher detection and lower false positive rates than the hybrid Adaboost-based face detector, as documented in the experimental results. Therefore, the detection rate of the proposed face detection system is comparable with that of the traditional Adaboost face detection system that uses the Haar-like features, whereas the training time of the proposed face detection system is much shorter. Finally, the proposed approach significantly improved the face detection systems.

Consent

The images which are used in this study are extracted from CMU.FERET and MIT-CBCL databases which are public database.