Boosting Discriminant Learners for Gait Recognition Using MPCA Features
© Haiping Lu et al. 2009
Received: 24 January 2009
Accepted: 9 July 2009
Published: 13 October 2009
This paper proposes a boosted linear discriminant analysis (LDA) solution on features extracted by the multilinear principal component analysis (MPCA) to enhance gait recognition performance. Three-dimensional gait objects are projected in the MPCA space first to obtain low-dimensional tensorial features. Then, lower-dimensional vectorial features are obtained through discriminative feature selection. These feature vectors are then fed into an LDA-style booster, where several regularized and weakened LDA learners work together to produce a strong learner through a novel feature weighting and sampling process. The LDA learner employs a simple nearest-neighbor classifier with a weighted angle distance measure for classification. The experimental results on the NIST/USF "Gait Challenge" data-sets show that the proposed solution has successfully improved the gait recognition performance and outperformed several state-of-the-art gait recognition algorithms.
Automated human identification at a distance is important in visual surveillance and monitoring applications in security-sensitive environments such as airports, banks, shopping malls, parking lots, and large civic structures [1, 2]. However, many conventional biometrics, such as iris, face, and fingerprint, require the person to be recognized to be in close distance or even in contact with the capturing device. At a distance, these biometrics are usually not available in high enough resolution for recognition purposes.
Gait, the style of walking of an individual, is an emerging behavioral biometric that offers the potential for vision-based recognition at a distance [3–6]. In 1975 , Johansson used point light displays to show humans' ability to distinguish human locomotion from other motion patterns. Later, experiments demonstrate the capability of identifying familiar individuals or the gender of a person [8, 9]. Nonetheless, researches on gait recognition from video sequences are only receiving significant attentions recently. Vision-based gait recognition is particularly attractive in human identification at a distance because gait capture is unobtrusive, requiring no cooperation or attention of the observed subject, and gait is difficult to hide [5, 10].
There are two approaches to gait recognition: the model-based approach [11–13], where human body structure is explicitly modeled, and the appearance-based approach [5, 6, 10, 14–18], where gait is treated as a sequence of holistic binary patterns (silhouettes). It should be noted that although the EigenGait approach  makes use of silhouettes as well, a motion-based recognition approach is taken where features are extracted from the image self-similarity plots rather than from the silhouettes directly. The appearance-based approach has been more successful working on practical data . Appearance-based approaches take binary gait silhouette sequences extracted from raw gait sequences [19–21] as the input. These gait silhouette sequences are naturally three-dimensional objects, also called third-order tensors, and the three dimensions are the spatial row, column, and the temporal modes . These tensor objects are in a very high-dimensional tensor space. To apply traditional linear feature extraction algorithms such as the Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA) on these tensorial data, they need to be first reshaped (vectorized) into vectors in a very high dimensional space. This reshaping does not only result in high computation and memory demand but also breaks the structure and correlation in the original data. This problem has motivated the development of multilinear subspace learning algorithms operating directly on the gait sequences in their tensorial representation rather than their vectorized forms. In particular, the multilinear PCA (MPCA) algorithm  aims to determine a multilinear projection that projects the original tensor objects into a lower-dimensional tensor subspace while preserving the variation in the original data as much as possible. For gait recognition, a number of discriminative features in the projected tensor space can be selected. The MPCA-based gait recognition algorithm has achieved better overall performance when compared with the state-of-the-art gait recognition algorithms.
Although progresses have been made in gait recognition, it remains a very challenging problem. A person's gait can be affected by many factors, such as viewing angles, walking surfaces and shoes. Similar to face patterns, the distribution of gait patterns is expected to be nonlinear and complex. Furthermore, the gait data in training and those in testing may be captured under different conditions and this makes generalization very difficult, as studied in the Gait Challenge problem . There are many methods proposed in literature to handle complex and nonlinear patterns. The ensemble-based machine-learning method named boosting is a very promising one that offers good generalization capability. Traditional boosting design works through the combination of a set of weak classifiers repeatedly trained on weighted training samples [23, 24], which tends to be an adaptive feature-selection process . Feature extraction is not a concern in these boosters and the requirement of an appropriate weak learner in boosting has restricted its applicability [24, 26]. A recent work in  has broken this limitation by proposing a boosting algorithm that puts the learning focus on the feature extractor rather than the classifier so that the new boosting scheme works with LDA-style learners. The effectiveness of the boosting scheme proposed in  has been demonstrated on the problem of face recognition. A cross-validation mechanism is employed to weaken the LDA learner, and the pairwise class discriminant distribution (PCDD) is introduced for interaction between the booster and the learner.
In this paper, the boosting work in  is enhanced and extended so it can be successfully applied to the problem of gait recognition through combination with the recent development of MPCA . It should be noted that, to the best of the authors' knowledge, this is the first work that has applied boosting to gait recognition although boosting has been well studied for face recognition [27–29]. In the proposed processing scheme, MPCA  first produces EigenTensorGaits (ETGs) in a lower-dimensional tensor space and then only a number of discriminative ETGs are selected as the input to the LDA-based booster. There are two main advantages in this scheme. On one hand, the MPCA feature extractor applied before the booster reduces the processing cost greatly (in both training and testing) such that the very-high-dimensional tensorial gait data can be handled efficiently. On the other hand, the number of selected ETGs provides another way (in addition to the cross-validation mechanism in ) to control the weakness of the LDA learner. In addition, in order to improve the generalization performance further, a regularization mechanism is incorporated since the within-class scatter of gait patterns under the capturing conditions in testing is expected to be larger than that of gait patterns in training. Furthermore, the training sample selection scheme in the original LDA-style boosting scheme proposed in  tends to prevent the inclusion of "difficult" (hard to classify correctly) samples in subsequent boosting steps. Therefore, a new training sample selection method is introduced in this paper to include more "difficult" samples in subsequent boosting steps to get better boosting results.
The rest of the paper is organized as follows. Section 2 briefly reviews the MPCA-based gait feature extraction method introduced in . Section 3 proposes the LDA-based boosting algorithm operating on MPCA features for enhancing gait recognition performance. In Section 4, experimental results on the NIST/USF "Gait Challenge" datasets are presented and the proposed algorithm is compared with the state-of-the-art gait recognition algorithms to illustrate the effectiveness of the proposed solution. Finally, conclusions are drawn in Section 5.
2. Review of MPCA-Based Gait Feature Extraction
MPCA  is a multilinear subspace learning method that extracts features directly from tensorial representation of multidimensional objects. In this section, the notations are introduced and the MPCA-based gait feature extraction algorithm is briefly reviewed.
2.2. Gait Feature Extraction through MPCA
In the MPCA-based gait feature extraction algorithm proposed in , a gait sample is a half cycle of gait silhouette sequences, represented naturally as a third-order tensor. The procedures described in  are followed to obtain these gait samples, where the foreground pixels in the lower-half of the silhouettes are counted and the minimums of the foreground pixel number sequence partition a gait sequence into half cycles. Another choice is to use full cycles as gait samples, which results in larger sample size in the time mode but fewer samples available for both training and test, while asymmetry between two adjacent half cycles could be potentially useful for discrimination in this case. In addition, half cycles may not always be an appropriate choice for gait samples. For example, when a luggage is carried on one side, full cycles are more appropriate to be used as gait samples. Thus, it will be worthwhile to study the effects of this choice on the gait recognition performance. However, this issue is out of the scope of this paper and it is left for future works since this paper focuses on the incorporation of the boosting scheme in gait recognition. There are two types of gait datasets in a typical gait recognition problem: the gallery and the probe . Gait samples in the gallery set are labeled with their identities and they are used as training data, while the probe set contains the test data, which are gait samples of unknown identities that need to be matched against those included in the gallery set.
such that the total tensor scatter
where denotes the number of classes (subjects), denotes the number of gait samples for class (subject) , and denotes the class label for the th gallery gait sample . Also, is the feature tensor of in the projected MPCA subspace, the mean feature tensor and the class mean feature tensor . For the ETG selection, the entries in are arranged into a feature vector according to in descending order. Only the first entries of are kept for subsequent recognition task . It should be noted that discriminability is only considered in the ETG selection process, while the selected ETG features are extracted in an unsupervised way by MPCA.
3. Boosting LDA-Style Learners on MPCA Features for Gait Recognition
3.1. The Boosting Scheme
The introduction of the mislabel distribution enhances the communication between the learner and the booster, so that the AdaBoost.M2 can focus the weak learner not only on hard-to-classify samples but also on the incorrect labels that are the hardest to discriminate .
Algorithm 1: The pseudocode implementation of the LDA-based booster.
the floor and ceil functions, respectively.
- (ii)Do for
fier with the MAD measure  on .
and normalize it:
Output: The final hypothesis:
3.2. The LDA Learner
In building the LDA learner, the approach in  is adopted with several enhancement.
( ) In , samples per class are used as the input to the LDA learner in order to get weaker but more diverse LDA learners; random samples per class are taken for the first boosting step; the hardest (with the largest ) samples per class are selected for subsequent steps. Let denote the selected samples, where for the sample selection scheme in .
The average weights of the samples selected according to the sample selection scheme proposed above are shown in Figure 8(a) as well, denoted as "New selection." As seen from the figure, the new sample selection scheme results in samples with much larger weights selected compared to the scheme in .
where is a regularization parameter to increase the estimated within-class scatter and is an identity matrix of size . The regularization term is added because in the gait recognition problem, the actual within-class scatter of gait sequences captured under various conditions is expected to be greater than the within-class scatter that can be estimated from the gallery set, which is captured under a single condition.
It should be noted that beside the algorithmic difference, the proposed solution has an important difference in design with that in . Direct application of the algorithm in  on the gait recognition problem requires the vectorization of the tensorial input to . For a gait sample of typical size , the resulted vectors are of size . In contrast, the LDA-based learners in the proposed booster take the gait feature vectors extracted by MPCA , rather than the original data . The proposed scheme has two benefits.
( ) The number of selected discriminative ETGs, which is the gait feature vector dimension , gives us one more degree (besides the number of samples used for LDA learners) to control the weakness of the LDA learners. Similar to the case of PCA+LDA, where the recognition performance is often affected by the number of principal components for input to LDA, affects the recognition performance of LDA on the MPCA features as well, as observed in . Therefore, by choosing a value of that is not optimal for a single LDA learner, the obtained LDA learner is weakened. On the other hand, the LDA learner cannot be made "too weak" either. Otherwise, the boosting scheme will not work.
( ) Using feature vectors of dimensionality instead of the original high-dimensional data as the booster input is computationally advantageous. Since boosting is an iterative algorithm with rounds, the computational cost is about times of that of a single learner with the same input, both in training and testing. When the booster works on lower-dimensional features extracted by MPCA, it becomes much more efficient since it needs to deal with low-dimensional vectors only in each round. For instance, the dimension of the input vectors to the booster is around in this paper, which is much smaller than the dimension for face data in  and the original gait data dimension . Therefore, the computational cost is reduced significantly this way.
4. Experimental Results
the comparison of gait recognition performance against the state-of-the-art gait recognition algorithms,
the effects of the gait feature vector dimension for input to LDA learners, the LDA feature vector dimension , the number of LDA training samples for LDA learner input , and the regularization parameter on boosting recognition performance,
the effectiveness of the new sample selection scheme proposed in this paper in improving the booster performance.
4.1. The Datasets
The NIST/USF "Gait Challenge" datasets version 1.7 [10, 34] is chosen to carry out the gait recognition experiments. All the recognition results reported and compared in this paper are obtained from this database. It consists of sequences from subjects walking in elliptical paths in front of the camera, with two viewpoints (left or right), two shoe types (A or B) and two surface types (grass or concrete). There is a newer version 2.1 available, which is of much larger size with two additional differences in briefcase carrying condition and time (including clothing). Version 1.7 is chosen in this work because this version is widely used in the research community as well and the performance on it is far from saturated [4, 5, 14, 16, 17, 22]. In addition, version 1.7 is much smaller than version 2.1 so the computational demand is much lower in experimental evaluation.
The characteristics of the gait data from the NIST/USF "Gait Challenge" dataset version 1.7.
Number of sequences
Difference from the gallery
Surface, shoe, view
4.2. Comparison of Gait Recognition Results with the State-of-the-Art Algorithms
Comparison of the gait recognition results on the NIST/USF "Gait Challenge" datasets version 1.7: the rank 1 identification rate (%).
Comparison of the gait recognition results on the NIST/USF "Gait Challenge" datasets version 1.7: the rank 5 identification rate (%).
From the results, the B-LDA-MPCA algorithm has achieved the best rank 1 and rank 5 recognition results on all probes except the rank 1 identification rate on probe B and the rank 5 identification rate on probe D, demonstrating its superior recognition performance. Compared to the MPCA+LDA algorithm, the B-LDA-MPCA algorithm has improved the rank 1 identification rate by an average of and the rank 5 identification rate by an average of . The greatest improvement in rank 1 identification rate is on probe F, and the greatest improvement in rank 5 identification rate is on probe E. In particular, in rank 1 identification rates, the performance improvement on the more difficult probes, D, E, F, and G, are more significant than the improvement on the easier probes, A, B, and C, showing that the B-LDA-MPCA algorithm indeed generalizes better than the MPCA+LDA algorithm.
The proposed method introduces an additional learner weakness control mechanism by . From , gives the best gait recognition performance with the MAD measure and the NNC classifier. From Figure 4, the weaker learners with give much better boosting results than the stronger learners with . This confirms that can improve the boosting performance through controlling the weakness of the learners.
The dimensionality of the LDA features affects the recognition performance of the proposed solution as well. Since , the maximum dimensionality of the features extracted by LDA learners is . Nonetheless, as pointed out in , if , the resulted LDA learner will be very strong, deteriorating the performance of the booster. From Figure 5, it can be seen that the value of giving the best performance is a medium value. It is also evident from the figure that the strong learner with collapsed around the th boosting step, as expected in boosting . This set of experiments demonstrate that appropriate weakness is again required and the best boosting performance cannot be reached with too strong or too weak learners.
The value of determines the number of training samples for the LDA learners. As discussed in , the diversity of the LDA learners is necessary to ensure good boosting performance. Therefore, by choosing only a subset of the available training samples, the diversity among learners at different boosting steps is enhanced. On the other hand, needs to be sufficiently large to enable learners to achieve a certain classification accuracy. Figure 6 illustrates the effects of discussed here, showing that an appropriate choice of is neither too small nor too large.
The effects of regularization are depicted in Figure 7, where it is shown that an appropriate regularization parameter does result in better generalization. This study confirms that gait recognizer can benefit from making use of the fact that the within-class scatter of gait patterns under various capturing conditions is greater than that under the same capturing condition.
4.4. The Effectiveness of the Proposed Sample Selection Scheme for LDA Learners in Boosting the Recognition Performance
Figure 8 demonstrates the effectiveness of the new sample selection scheme proposed in Section 3.2. As discussed in Section 3.2 and illustrated in Figure 8(a), the proposed scheme selects samples with much larger weights for subsequent boosting steps, compared with the scheme in . Thus, the new scheme focuses more on the difficult samples, which agrees with the working principle behind boosting. The effects of the new sample selection scheme on the recognition performance are shown in Figures 8(b) and 8(c), where the corresponding average rank 1 and rank 5 identification rates are compared, respectively. From the figure, it can be seen that the proposed new sample selection scheme results in approximately improvement in both rank 1 and rank 5 identification rates.
This paper proposes a gait recognition solution through combining the MPCA algorithm  and the ensemble-based discriminant learning method in . The MPCA algorithm in  is used to extract features from tensorial gait data and a subset of the extracted features are fed into an enhanced LDA-style booster. This scheme gives another way of learner weakness control in addition to computational efficiency. The LDA learner in  is modified by adopting a simpler weighted pairwise between-class scatter matrix and introducing a regularization term in the within-class scatter matrix so that the gait challenge due to various capturing conditions is taken into account. Furthermore, a new sample selection scheme of the LDA-based booster is proposed to concentrate more on the "difficult" samples in the boosting process. Experiments carried out on the gait challenge datasets show that the proposed scheme is effective in boosting the gait recognition performance and outperforms several state-of-the-art gait recognition algorithms.
The authors would like to thank the anonymous reviewers for their insightful comments. The authors would also like to thank Professor Sudeep Sarkar from the University of South Florida for kindly providing us with the Gait Challenge datasets. This work is partially supported by the Ontario Centres of Excellence through the Communications and Information Technology Ontario Partnership Program and the Bell University Laboratories at the University of Toronto.
- Jain AK, Ross A, Prabhakar S: An introduction to biometric recognition. IEEE Transactions on Circuits and Systems for Video Technology 2004,14(1):4-20. 10.1109/TCSVT.2003.818349View ArticleGoogle Scholar
- Lu H, Wang J, Plataniotis KN: A review on face and gait recognition: system, data and algorithms. In Advanced Signal Processing: Theory and Implementation for Sonar, Radar, and Non-Invasive Medical Diagnostic Systems. 2nd edition. Edited by: Stergiopoulos S. CRC Press, Boca Raton, Fla, USA; 2009:303-330.Google Scholar
- Nixon MS, Carter JN: Automatic recognition by gait. Proceedings of the IEEE 2006,94(11):2013-2024.View ArticleGoogle Scholar
- Kale A, Sundaresan A, Rajagopalan AN, et al.: Identification of humans using gait. IEEE Transactions on Image Processing 2004,13(9):1163-1173. 10.1109/TIP.2004.832865View ArticleGoogle Scholar
- Boulgouris NV, Hatzinakos D, Plataniotis KN: Gait recognition: a challenging signal processing technology for biometrics. IEEE Signal Processing Magazine 2005,22(6):78-90.View ArticleGoogle Scholar
- Wang L, Tan T, Ning H, Hu W: Silhouette analysis-based gait recognition for human identification. IEEE Transactions on Pattern Analysis and Machine Intelligence 2003,25(12):1505-1518. 10.1109/TPAMI.2003.1251144View ArticleGoogle Scholar
- Johansson G: Visual motion perception. Scientific American 1975,232(6):76-88. 10.1038/scientificamerican0675-76View ArticleGoogle Scholar
- Cutting J, Kozlowski L: Recognizing friends by their walk: gait perception without familiarity cues. Bulletin of the Psychonomic Society 1977,9(5):353-356.View ArticleGoogle Scholar
- Barclay CD, Cutting JE, Kozlowski LT: Temporal and spatial factors in gait perception that influence gender recognition. Perception and Psychophysics 1978,23(2):145-152. 10.3758/BF03208295View ArticleGoogle Scholar
- Sarkar S, Phillips PJ, Liu Z, Robledo I, Grother P, Bowyer KW: The human ID gait challenge problem: data sets, performance, and analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 2005,27(2):162-177.View ArticleGoogle Scholar
- Lu H, Plataniotis KN, Venetsanopoulos AN: A layered deformable model for gait analysis. Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition (FGR '06), April 2006 249-256.Google Scholar
- Yam CY, Nixon MS, Carter JN: Automated person recognition by walking and running via model-based approaches. Pattern Recognition 2004,37(5):1057-1072. 10.1016/j.patcog.2003.09.012View ArticleGoogle Scholar
- Lu H, Plataniotis KN, Venetsanopoulos AN: A full-body layered deformable model for automatic model-based gait recognition. EURASIP Journal on Advances in Signal Processing 2008, 2008:-13.Google Scholar
- Lu H, Plataniotis KN, Venetsanopoulos AN: Uncorrelated multilinear discriminant analysis with regularization and aggregation for tensor object recognition. IEEE Transactions on Neural Networks 2009,20(1):103-123.View ArticleGoogle Scholar
- Lu H, Plataniotis KN, Venetsanopoulos AN: Multilinear principal component analysis of tensor objects for recognition. Proceedings of the International Conference on Pattern Recognition, August 2006 2: 776-779.Google Scholar
- Boulgouris NV, Plataniotis KN, Hatzinakos D: Gait recognition using linear time normalization. Pattern Recognition 2006,39(5):969-979. 10.1016/j.patcog.2005.10.013View ArticleMATHGoogle Scholar
- Han J, Bhanu B: Individual recognition using gait energy image. IEEE Transactions on Pattern Analysis and Machine Intelligence 2006,28(2):316-322.View ArticleGoogle Scholar
- Cutler R, Benabdelkader C, Davis L: Motion-based recognition of people in eigengait space. Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, May 2002 254-259.Google Scholar
- Lu H, Plataniotis KN, Venetsanopoulos AN: Coarse-to-fine pedestrian localization and silhouette extraction for the gait challenge data sets. Proceedings of the IEEE International Conference on Multimedia and Expo (ICME '06), July 2006, Toronto, Canada 1009-1012.Google Scholar
- Migdal J, Grimson WEL: Background subtraction using Markov thresholds. Proceedings of the IEEE Workshop on Motion and Video Computing (MOTION '05), January 2007 58-65.Google Scholar
- Lee L, Dalley G, Tieu K: Learning pedestrian models for silhouette refinement. Proceedings of the IEEE International Conference on Computer Vision, October 2003 1: 663-670.View ArticleGoogle Scholar
- Lu H, Plataniotis KN, Venetsanopoulos AN: MPCA: multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks 2008,19(1):18-39.View ArticleGoogle Scholar
- Freund Y, Schapire RE: Experiments with a new boosting algorithm. Proceedings of the 13th International Conference on Machine Learning, July 1996, Bari, Italy 148-156.Google Scholar
- Schapire RE: The boosting approach to machine learning: an overview. In Proceedings of the MSRI Workshop on Nonlinear Estimation and Classification, March 2003, Berkeley, Calif, USA. Edited by: Denison DD, Hansen MH, Holmes C, Mallick B, Yu B. Springer;Google Scholar
- Viola P, Jones MJ: Robust real-time face detection. International Journal of Computer Vision 2004,57(2):137-154.View ArticleGoogle Scholar
- Skurichina M, Duin RPW: Bagging, boosting and the random subspace method for linear classifiers. Pattern Analysis and Applications 2002,5(2):121-135. 10.1007/s100440200011View ArticleMathSciNetMATHGoogle Scholar
- Lu J, Plataniotis KN, Venetsanopoulos AN, Li SZ: Ensemble-based discriminant learning with boosting for face recognition. IEEE Transactions on Neural Networks 2006,17(1):166-178.View ArticleGoogle Scholar
- Yang P, Shan S, Gao W, Li SZ, Zhang D: Face recognition using Ada-Boosted Gabor features. Proceedings of the 6th IEEE International Conference on Automatic Face and Gesture Recognition, May 2004 356-361.Google Scholar
- Li SZ, Chu RF, Liao SC, Zhang L: Illumination invariant face recognition using near-infrared images. IEEE Transactions on Pattern Analysis and Machine Intelligence 2007,29(4):627-639.View ArticleGoogle Scholar
- Lathauwer LD, Moor BD, Vandewalle J: A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications 2000,21(4):1253-1278. 10.1137/S0895479896305696View ArticleMathSciNetMATHGoogle Scholar
- Bader BW, Kolda TG: Algorithm 862: MATLAB tensor classes for fast algorithm prototyping. ACM Transactions on Mathematical Software 2006,32(4):635-653. 10.1145/1186785.1186794View ArticleMathSciNetMATHGoogle Scholar
- Lathauwer LD, Moor BD, Vandewalle J:On the best rank-1 and rank- approximation of higher-order tensors. SIAM Journal on Matrix Analysis and Applications 2000,21(4):1324-1342. 10.1137/S0895479898346995View ArticleMathSciNetMATHGoogle Scholar
- Loog M, Duin RPW, Haeb-Umbach R: Multiclass linear dimension reduction by weighted pairwise Fisher criteria. IEEE Transactions on Pattern Analysis and Machine Intelligence 2001,23(7):762-766. 10.1109/34.935849View ArticleGoogle Scholar
- Phillips PJ, Sarkar S, Robledo I, Grother P, Bowyer KW: The gait identification challenge problem: data sets and baseline algorithm. Proceedings of the 16th International Conference on Pattern Recognition (ICPR '02), August 2002, Quebec, Canada 1: 385-388.Google Scholar
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.