Automatic landmark point detection and tracking for human facial expressions

2.1 Preprocessing

Since the faces are non-rigid and have a high degree of variability in location, color and pose, it is difficult to detect face automatically in a complex environment. Occlusion and illumination artifacts can also change the overall appearance of a face. We, therefore, propose detecting facial regions in the input video sequence using a face detector with local illumination compensation for normalization and optimal adaptive correlation [18]. Specifically, each frame of the input video sequence is extracted and regularized using an illumination compensation process, including gamma intensity correction (GIC), difference of Gaussian (DoG), local histogram matching (LHM) and local normal distribution (LND). Face candidate regions are then located by the OAC technique with kernel canonical correlation analysis (KCCA). Compare to Viola and Johns’ algorithm [19], the local normalization based method is adaptive to the normalized input image and designed to complete the segmentation in a single iteration. With the local normalization based method, the proposed method tends be more robust under different illumination conditions.

Before the raw data sequences can be used for automatic landmark point detection and tracking, it is necessary to normalize the size of the sequence such that they were in the format required by the system. Since the displacement of landmark point in each frame depends on each individual, we use the Inter-ocular Distance (IOD) for size normalization. The distance between left and right eye pupils is determined in the first input frame. We also manually marked the landmarks for the selected sequences to create the ground truth data.

After the facial region being detected, we propose to use the scale space extrema method to find the locations of candidate points in Section 2.2. The scale invariant feature for each candidate point is extracted and the 26 landmark detectors are constructed as described in Section 2.3.

2.2 Candidates selection

where D(x,y,σ) is the DoG function of the input image. In this work, we set the interval number n to 3 to form n + 2 DoG images, and k to 2^1/3. Each pixel in a DoG image is compared to its eight neighbors on the same scale and each of its nine neighbors one scale up and down. If this value is the minimum or maximum among the pixels compared, it is an extremum. These pixels are chosen as interest candidate points, including the adjacent scale, the position and scale of the local extreme point. Since the success of landmark detection depends on the quantity of the selected candidates, we used a larger number of scale samples. (Those points are generally the feature points of the image, located on contours, corners and edges.) DoG extrema are repeatedly assigned in the scale space. They are stable features across all possible scales and are invariant to scale and rotation. These points are highly distinctive and are located on contours, corners and edges in a facial region. Since there are 5 DoG images in our work, all the interest candidates are examined to determine location and scale. The landmarks are detected based on the measurements from these local decisions.

2.3 Feature vectors extraction

where L is the image at scale σ. We choose a neighborhood F centered at the interest point. By calculating the directions of points in F, we obtain the histogram of gradient directions. The orientation has a range of 360 degrees calculated by Eqs. (4) and (5). However, it is complex and computationally expensive to use the original orientation histogram with 360 bins. To reduce the computing cost, we equally divide the histogram into 36 phases each covering a range of 10 degrees of the orientations. As a result, the orientation histogram has 36 bins. The direction of the interest candidate point is the maximal component of the 36 phases in the histogram.

To detect the landmarks from the interest candidate points, a set of landmark detectors with the feature description from the gradient orientation histogram of the input images are constructed. The descriptor is constructed from a vector containing the values of all the orientation histogram entries. At the center of each landmark, a neighborhood window is selected and divided into 16 subregions of 4 × 4. Using (4) and (5), the directions and amplitudes of all pixels in the subregions are obtained, and then accumulated into orientation histograms summarizing the contents over the 4 × 4 subregions. Using the orientation histogram, we can calculate the eight direction distributions in the ranges of (0,π/4,π/2,3π/4,π,5π/4,3π/2,7π/4) with the length corresponding to the sum of the gradient magnitudes near that direction within the region. The amplitude and Gaussian function are also applied on the eight direction distributions to create the direction histogram of subregions. The feature description of each landmark point is obtained by connecting the direction descriptions of all subregions. The total number of the direction descriptions is 16 since we have 4 × 4 subregions of the landmark descriptor. So the length of a landmark point detector is 128 = 16 × 8, and should be normalized in order to ensure illumination invariance.

3.1 DE-MC particle filter

In (6), the state X _k is a 2 M-component vector that represents the location of landmarks, the observation Y _1:k is the set of image frames up to the current time instant. The normalization constant λ _k is independent of X _k . The motion model p(X _k |X _k - 1) is conditioned directly on the immediate preceding state and independent of the earlier history if the motion dynamics are assumed to form a temporal Markov chain. The distribution is represented by discrete samples N through particle filtering. The N samples (particles) are drawn from a proposed distribution $p\left(\left.X_k^{\left(i\right)}\right|X_k^{\left(i-1\right)},Y_k\right)$, i = 1,2,…,N and assigned with weights $w\left(X_k^{\left(i\right)}\right)$.

where ${\widehat{X}}_k^{\left(i\right)}$ is a new set of samples selected at time k, and V _k-1 is the velocity vector computed in time step k-1.

The particle set ${\left\{X_k^{\left(i\right)-}\right\}}_{i=1}^N$ acts as the initial N population for a T-iteration DE-MC processing. For any one landmark in the T-iteration processing, two different integers, r ₁ r ₂ that r ₁ ≠ r ₂ ≠ k, are randomly chosen from the population of previous iteration. A new member $\left\{X_k^{*\left(i\right)}\right\}$ that $\left\{X_k^{*\left(i\right)}\right\}=\left\{X_{k-1}^{\left(i\right)}\right\}+\lambda \left(\left\{X_{k-1}^{\left(r_1\right)}\right\}-\left\{X_{k-1}^{\left(r_2\right)}\right\}\right)+g$ is created, where λ is a scalar whose value is found to be optimal when $\lambda =2.38/\sqrt{2N}$, g is drawn from a symmetric distribution with small variance compared to that of ${\left\{X_k^{\left(i\right)}\right\}}_{i=1}^N$. A target function is given based on the ratio between the populations of current and previous step until a convergence or a preset end point is reached. Then the weights of particles are subject to update by the DE-MC. At the end of this step, we take the output population as the particle set of current time step ${\left\{X_k^{\left(i\right)},w\left(X_k^{\left(i\right)}\right)\right\}}_{i=1}^N$.

and update the velocity vector of current time step V _k  = X _k  - X _k - 1. The step size of random jumping for current DE-MC iteration is reduced if the survival rate of the last DE-MC iteration is high or inflated otherwise [37]. The update scheme for the maximum likelihood decision on the weights w can be summarized as follows:

While the tracker updates and tracks the X _k vector that represents the coordinates of the 26 landmark points, the samples are already drawn. The DE-MC particle filter is able to make a more reasonable sampling and keeps them from running off into implausible shapes even if they are placed in the positions far away from the solution point or are trapped in the local cost basin of the state space. The observation model can help the sample points for positions close to the solution in regard to their starting points. The measurement module provides necessary feedback to the sampling module, according to which, the hypothesis moves to the regions where it is more likely for the global maximum of the measurement function to be found.

3.2 Kernel correlation-based observation likelihood

where τ _i is a scaling parameter, which helps the result evaluated by (12) be more reasonably distributed in the range of (0,1).

3.3 Landmark point tracking

In this section, we present using multiple DE-MC filters for facial landmarks tracking over time. Once the observation model is defined we need to model the transition density and to specify the scheme for reweighting the particles. The single particle filters weight particles based on a likelihood score and then propagate these weighted particles according to a motion model. Simply running particle filters for multiple landmarks tracking needs a complex motion model for the identity between targets. Such an approach suffers from exponential complexity in the number of tracked targets [40]. In contrast to traditional methods, our approach addresses the multi-target tracking problem using the M-component non-parametric mixture model, where each component (every landmark point) is modeled with an individual particle filter that forms part of the mixture. The landmark states have multi-modal distribution functions and the filters in the mixture interact only through the computation of the importance weights. In particular, we combined color based kernel correlation technique for the observation likelihood with DE-MC particle filtering distribution. A set of weighted particles are used to approximate a density function corresponding to the probability of the location of the target given observations.

The particles are sampled from the training data to obtain the appropriate distribution in the M-mixture model. The prediction step and the measurement step are integrated together instead of functioning separately. The use of the priors provides sufficient constrains for reliable tracking at the presence of appearance changes due to facial expressions. The measurement function evaluates the resemblance between image features generated by hypothesis and those generated by ground truth positions, as the criterion for judging the correctness of hypothesis.

In our work, scaling is normalized by person-related scaling factors that are estimated from the positions of the facial features at the first frame, such as the dimensions of the mouth. This scheme simply processes with the prior knowledge by sampling from the transition priors and updating the particles using importance weights derived from (17).

4.1 Facial landmark detection

In this section, we present the experimental results using the proposed facial landmarks detection method. Adaboost algorithm is applied for training the 26 facial landmark detectors. We use ten frames from each training sequence with the manually labeled ground truth points. The surrounding eight positions of the true point are also selected as the positive examples in a training image. Another five arbitrary points in the same frame are chosen as the set of negative examples. The prototypical 128-dimensional feature vector is used for each sample point. In the testing images, candidate points are first extracted from facial region using scale invariant feature. For a certain facial landmark, Adaboost classifier outputs a response depicting the similarity between the representations of the candidate points compared to the learned training model. After checking the entire facial region, the position with the highest response reveals the landmark point.

Boost algorithm has been proposed to reduce the redundancies of the high dimensional feature space and computational cost. The Adaboost algorithm by Viola and Jones [19] for face detection is a typically successful example as it has a very low false positive rate and can detect faces in real time. It can be trained for different levels of computational complexity, speed and detection rate which are suitable for specific applications. The performances of RealAdaboost [44], GentleAdaboost [45] and ModestAdaboost [46] for fiducail point detectors are compared in our work using GML AdaBoost Matlab Toolbox [47] and shown in Figure 3. GentleAdaboost returns the best detection rates from the results. In contrast to other Adaboost algorithms, GentleAdaboost uses real valued features and converges faster. It gives less emphasis to misclassified examples since the increase in the weight of the example is quadratic in the negative margin, rather than exponential. Thus, GentleAdaboost is selected as the classification algorithm in our system.

The overall detection rates for each point are shown in Figure 4, and the proposed method achieves 91% average detection rate of the facial landmarks. We illustrate some representative cases in Figure 5. The proposed method is applied on each frame of the input video sequences, and the 26 facial landmarks are automatically detected.

4.2 Tracking results

In this section, we present the experimental results using the proposed multiple DE-MC filters. The positions of the facial landmarks in the first frame of an input sequence are automatically found using the detection method. The positions in all subsequent frames are then determined by the multiple particle filters with the color based observation likelihood. The observation model is built from the training data of manually labeled sequences using a finite set of particles within the feature point centered window. We approximate the posterior p(X _k |Y _1 : k ) from an appropriate proposal distribution to maintain a particle based representation for the a posteriori probability of the state. Since the calculation of the weights of the particles is a critical step of multiple points tracking, in the proposed M-mixture model, we sample the particles from the training data to obtain the appropriate proposal distribution. The proposed method simply proceeds by sampling from the transition priors and updating the particles using importance weights derived from Eq. (17). In the DE-MC iterations, the measurement module provides necessary feedbacks to the sampling module. According to them the sampling moves to regions in the state space where it is more possible to find the global maximum of the measurement function. Since we are interested in the global optimal state, we place denser sampling grids in the region of interest. This approach yields a result reasonably close to that obtained by sampling strictly according to the ground truth posterior distribution.

We present some representative cases using the proposed method, exploring various practical aspects for the facial landmark detection and tracking. Figures 6 and 7 summarize the experimental results for two different emotional expressions. The facial landmarks are first detected by the point detectors in the first frame and then tracked by the kernel correlation based multiple DE-MC filters. For all figures, the white dots represent the positions of facial landmarks to be detected and tracked, which are all labeled with the associate numbers. In Figure 6, the subject exhibits a set of sadness expressions from a neutral face at the beginning and ends at the apex of that expression. Figure 7 shows the anger expression with talking at the same time. As expected, all the points are tracked reliably for the whole sequence. Since the motions of the faces are not intensive and the facial appearances are not heavily changed, the features extracted from consecutive frames are highly correlated and the results achieve a very impressive tracking rate.

We apply the proposed method to the zoomed case, as shown in Figure 8. When the camera zooms, the factors assigned with the color based kernel correlation keep changing and are going to descend, as a result of (9) and (10) which can be seen from frame 320. However, the facial landmark can still be tracked with the updating weights using (16), as we keep track of the points from the previous frame. It shows that the use of the priors for the multiple filters provides constraints that are sufficient for the reliable tracking of the points at the presence of the facial appearances.

While performing the experiments, we also consider the cases with head in rotation or point occlusion, as shown in Figure 9. In this case we can see the points 3, 7, 15, 20 and 23 are lost after frame 78 when a frontal face is rotating to a profile view. So far, the multiple detectors and trackers are based on different configurations of color intensity regions. If both detectors and particle trackers fail for several consecutive frames, the proposed approach will eventually lose the landmark points.

To solve this problem, we execute a conservative way to update the trackers temporally with the response distribution [48] for the next n frames when the missing points first occurred. This step length n can be changed by the user and should not be crucial to the system. If the trackers respond correctly after a few frames, the trackers are able to recover due to the accumulation of probabilities. However, when the step length n continues to grow, due to incorrect responses of the detector, the color correlation of the observation likelihood drops and the trackers will begin to lose points. After that, “point lost” will be declared. We then stop estimating its motion V _k and discard the motion likelihood term. The trackers will be reinitialized by the point detectors in the following frames. All the 26 points can be detected with a new set of parameters if the facial region appears again in the scene. The improved result is shown in Figure 10 that reinitialization executes and all facial landmarks are found again after frame 183.

4.3 Performance evaluate

where N _SUCCESS stands for the number of SUCCESS point from the detection and tracking, N _miss stands for the number of missed points, and N _false stands for the number of false alarms. The sum N _SUCCESS + N _miss is the total number of manually labeled facial landmarks in the entire video sequence.

The overall performance of the system in term of false alarm rate using the aforementioned datasets is illustrated in Figure 11. From this figure, we can see that the precision is decreasing and recall is increasing with the increment of false alarms. Note, in the graph, a system performance of recall 94.15% and precision 92.86% is achieved simultaneously.

We also checked the displacement accuracy for the proposed methods. The Euclidean distances between each individual landmark point are used for the measurement. Since we use the scale normalization for the variation in size of each individual face, therefore the distance measurement is invariant for the experimental datasets. We calculate the average accuracy for the displacement of the proposed automatic method compared to the manually labeled ground truth. The distributions of the displacement accuracy are shown in Figure 12. From the figure we can see that given a 10% normalized distance, the proposed method achieves a 93% average accuracy from the ground truth.

4.4 Comparison with state-of-the-art

To distinguish person-independent affective states, subtle changes of facial expressions should be extracted for feature construction. Automatic facial landmark detection and tracking are crucial for analyzing the current facial appearance since it will facilitate the examination of the fine structural changes inherent in the spontaneous expressions. A key motivation for developing landmark point techniques is that they lay the foundation for developing 3D models and associated dynamic feature extraction and recognition techniques which are highly likely superior to 2D-based and static 3D-based techniques. We therefore first compare with the result reported in [9] which also used the RML Emotion database, but with static visual features extracted by 2D Gobar filters. The comparison shows that working on the same database, facial landmark based 3D dynamic features [49] (90% recognition rate) substantially outperforms the recognition rate by the 2D Gabor features (approximately 50%), and also the bimodal features (approximately 82%).

As is evident from these results, our method achieves the best overall performance of 90.8% average rate. In contrast with other approaches, the most evident improvement of the proposed method is that the prediction step and the measurement step are integrated together instead of functioning separately. The use of the priors provides sufficient constrains for reliable tracking at the presence of appearance changes due to facial expressions. The measurement function evaluates the resemblance between image features generated by hypothesis and those generated by ground truth positions, as the criterion for judging the correctness of hypothesis.

The proposed method has demonstrated its ability to handle pose variations problems and can be used for both image and video based facial expression recognition. Computationally, the proposed method has the advantages of automatic initialization by using the scale invariant features extraction over the other methods that examine pixels one by one. Note that the method proposed in [27] achieved a better overall detection rate. However, this method is only tested on perfect manually aligned image sequences and no experiments in fully automatic conditions were reported. In addition, only 13 sequences were experimented on in [27]. Therefore, the result is far from conclusive.