- Open Access
Background initialization and foreground segmentation for bootstrapping video sequences
© Hsiao and Leou; licensee Springer. 2013
- Received: 4 February 2012
- Accepted: 8 January 2013
- Published: 28 February 2013
In this study, an effective background initialization and foreground segmentation approach for bootstrapping video sequences is proposed. First, a modified block representation approach is used to classify each block of the current video frame into one of four categories, namely, “background,” “still object,” “illumination change,” and “moving object.” Then, a new background updating scheme is developed, in which a side-match measure is used to determine whether the background is exposed. Finally, using the edge information, an improved noise removal and shadow suppression procedure with two morphological operations is adopted to enhance the final segmented foreground. Based on the experimental results obtained in this study, as compared with three comparison approaches, the proposed approach produces better background initialization and foreground segmentation results.
- Background initialization
- Foreground segmentation
- Side-match measure
- Block representation
- Shadow suppression
The main purpose of foreground/background segmentation, a basic process of a computer vision application system, is to extract some interesting objects (the foreground) from the rest (the background) of each video frame in a video sequence . Background subtraction is a popular foreground/background segmentation approach, which detects the foreground by thresholding the difference between the current video frame and the modeled background in a pixel-by-pixel manner . The correctness of the modeled background is usually affected by three factors : (1) illumination changes; (2) dynamic backgrounds: some “moving” objects, such as waving trees, fountains, and flickering monitors, are not interested for a vision-based surveillance system; and (3) shadows: foreground objects often cast shadows, which are different from the modeled background.
A background subtraction approach usually considers three main issues: background representation, background updating, and background initialization . For the popular background subtraction approach called the Gaussian background model, Stauffer and Grimson  presented a pixel-wise background representation scheme using the mixture of Gaussians (MoG) and pixel-wise background updating to update the intensity mean and variance of each pixel in real-time. The MoG-based methods are effective for dynamic background scenes with multiple background variations, but they are sensitive to noise and illumination changes. Several existing MoG-based approaches are proposed to improve their performances by adaptation of some MoG parameters , such as the number of components [6, 7], weights, mean, and variance [8–11], learning rate [8, 9, 12, 13], and feature type [9, 14–17], and by smoothing among spatially and temporally neighboring pixels using spatial and temporal dependencies . In general, a training duration without foreground objects (non-bootstrapping) is required and some ghost (false positive) objects may be detected when some foreground objects change their motion status (static or moving) suddenly.
Recently, the background subtraction methods focused on background initialization for bootstrapping video sequences [19–24], in which a training duration without foreground objects is not available in some cluttered environments [3, 19]. That is, background initialization for bootstrapping video sequences can be defined as follows: given a video sequence captured by a stationary camera, in which the background is occluded by some foreground objects in each frame of the video sequence, the aim is to estimate a background frame without foreground objects [22, 24]. Background initialization for bootstrapping video sequences (or simply background initialization) is widely used in the intelligent video surveillance systems for monitoring crowded infrastructures, such as banks, subway, airports, and lobby.
Two simple background initialization techniques are the pixel-wise temporal mean and median filters over a large number of video frames [20, 21]. For the pixel-wise temporal median filter, it is assumed that for each pixel within the estimation duration, the exposure of the background must be more than that of the foreground. Based on the block-wise strategy, Farin et al.  used a block similarity matrix to segment the input video frames into foreground and background regions, which contain the block-wise temporal differences between any video frame pair. Reddy et al.  proposed a block selection approach using the discrete cosine transform (DCT) among some neighboring blocks to estimate the unconstructed parts of the background. This approach is usually degraded by similar frequency content within a block candidate set and error propagation if some blocks in a video frame are erroneously estimated. Note that, to obtain the processing results, the whole video sequence should be available to Reddy et al.’s approach. Then, the DCT is replaced by the Hadamard transform to reduce the computation time for block selection . In addition, a block selection refinement step using spatial continuity along block borders is added to prevent erroneous block selection. Most block-wise background initialization approaches need large memories and are computationally expensive. Furthermore, one free-background video frame is usually obtained as its output during the “learning” duration.
For the frame-wise strategy with temporal smoothing, the first video frame of a video sequence is usually treated as the initial background for background initialization. Most background initialization approaches maintain a modeled background by iterative updating with temporal smoothing between each input video frame and the modeled background. Liu and Chen  proposed a background modeling method, in which the background similarity using the mean and variance information is adopted to identify the background image. Moreover, Scott et al.  updated the mean and variance information by Kalman filter updating equations for maintaining the modeled background. Maddalena and Petrosino  automatically generated the background model without prior knowledge by using self-organizing artificial neural networks. Each color pixel is represented by n × n weight vectors to form a neural map. It is claimed that they can handle bootstrapping scenes containing dynamic backgrounds, gradual illumination changes, and shadows. Using the growing self-organizing map, Ghasemi and Safabakhsh  generated a codebook for detecting moving objects in the dynamic background scenes. The major advantage of the methods using variant self-organizing maps [27, 28] is low computational complexity. Chiu et al.  proposed a pixel-wise color background modeling approach using probability theory and clustering. To estimate the modeled background completely, a suitable time duration is required, because each of the R, G, and B color components is iteratively updated by increasing/decreasing 1 in the range of 0–255. The main weakness for the background initialization and foreground segmentation approaches using the frame-wise strategy with temporal smoothing is that the “erroneous” parts in the modeled background are slowly updated. Furthermore, this type of approaches can work properly only when the video sequence contains fast “moving” foreground objects so that the background is exposed most of the time.
On the other hand, within some existing approaches [30–34], temporal smoothing is not adopted in background updating. Chein et al.  proposed a pixel-wise video segmentation approach with adaptive thresholding to determine each pixel as a moving or stationary one. Each pixel in the modeled background is then replaced by the corresponding pixel in the current video frame if the pixel is detected as a stationary one for some time duration. That is, this type of approaches might not work well in illumination-changing environments. Verdant et al.  proposed three analog-domain motion detection algorithms in video surveillance, namely, the scene-based adaptive algorithm, the recursive average with estimator algorithm, and the adaptive wrapping thresholding algorithm, in which background estimation and variance of each pixel are computed with nonlinear operations to perform adaptive local thresholding. Lin et al.  used a classifier to determine whether an image block belongs to the background for block-wise background updating. The classifier using two learning methods, namely, the support vector machine and column generation boost, is trained by some training data, which are manually labeled as foreground/background blocks before background initialization. In addition, some foreground prediction approaches may segment accuracy foreground without background modeling. For example, Tang et al.  proposed a foreground prediction algorithm, which estimates each pixel in the current video frame belonging to the foreground one. Given a segmentation result (an alpha matte) of the previous video frame as an opacity map, the opacity values [0–1] in an opacity map are propagated from the previous video frame to the current video frame using the foreground prediction algorithm. It was claimed that the foreground can be predicted accurately in sudden illumination changes. Zhao et al.  proposed a learning-based background subtraction approach based on sparse representation and dictionary learning. They made two important assumptions, which enabled their approach to handle both sudden and gradual background changes.
In this study, an effective background initialization and foreground segmentation approach for bootstrapping video sequences is proposed, which contains a block-wise background initialization procedure and a pixel-wise foreground segmentation procedure. First, a modified block representation approach is used to classify each block of the current video frame into one of four categories. Then, a new background updating scheme is developed, in which a side-match measure is used to determine whether the background is exposed so that the modeled background can be well determined. Finally, using the edge information, an improved noise removal, and shadow suppression procedure with two morphological operations is adopted to enhance the final foreground segmentation results. The main contributions of the proposed approach include: (1) using motion estimation and correlation coefficient computation to perform block representation (classification); (2) developing four types of background updating for four types of block representation; (3) using side-match measure to perform background updating of “moving object” blocks; and (4) using a modified noise removal and shadow suppression procedure to improve final foreground segmentation results.
This article is organized as follows. In Section 2, the proposed background initialization and foreground segmentation approach is addressed. Experimental results are described in Section 3, followed by concluding remarks given in Section 4.
2.1. Initial modeled background processing
Finally, as shown in Figure 2q, the initial modeled background frame contains no “undefined” block. Here, for the illustrated example shown in Figure 2, the performance index T 1(=19) is defined as the frame index for initial modeled background processing. Afterwards, the initial modeled background frame (t = 20,21,…) is duplicated from the “updated” modeled background frame B t–1, i.e., (t = 20,21,…) .
2.2. Block representation
Motion estimation is performed between the two consecutive video frames, I t and I t– 1 using a block matching algorithm so that each block in I t is determined as either “static” or “moving.” In this study, the sum of absolute differences (SAD) is used as the cost function for block matching between block b (i,j) t in I t and the corresponding block in I t–1 and the search range for motion estimation is set to ±N/2 [35, 36]. For a block in I t , if the minimum SAD, D mv(u,v), for motion vector (u,v), is smaller than 90% of the SAD for the null-vector (0,0), D mv(0,0), the block is determined as a “moving” block; otherwise, it is determined as a “static” block [19, 35].
where μ b is the mean of the pixel values in block b. As shown in Figure 4, based on C B (i,j) and the threshold THCB a “static” block can be further classified into either a “background” block (if C B (i,j) ≥ THCB) or a “still object” block (otherwise), whereas a “moving” block can be further classified into either an “illumination change” block (if C B (i,j) ≥ THCB) or a “moving object” block (otherwise). Afterwards, four different block representations are obtained.
2.3. Background updating
By background updating, each block in the initial modeled background frame can be updated to obtain the corresponding block in the modeled background frame B t as follows. Both the “background” and “illumination change” blocks are updated by temporal smoothing, i.e., block in B t is updated as the linearly weighted sum of block in and block b (i,j) t in I t . On the other hand, both the “still object” and “moving object” blocks are updated by block replacement.
where α, the updating weight, is empirically set to 0.9 in this study.
(c) Illumination change: the modeled background block in B t is similarly updated by Equation (2).
2.4. Initial segmented foreground
2.5. Noise removal and shadow suppression with two morphological operations
As shown in Figure 8, usually contains some fragmented (noisy) parts and shadows. To obtain the precise segmented foreground frame F t , a noise removal and shadow suppression procedure is adopted, which combines the shadow suppression approach in  and the edge information extracted from I t with being the (binary) operation mask.
The six bootstrapping video sequences and their categories
To evaluate the performance of the proposed approach, three comparison approaches, namely, MoG , Reddy background estimation (Reddy) , and self-organizing background subtraction (SOBS) , are implemented in this study. In MoG and Reddy, only the gray-level component of each video frame is employed, in SOBS, the H, S, and V components of each video frame are employed, and in the proposed approach, the gray-level video frames are used and additionally the S component is only used for shadow suppression. Note that, for the SOBS approach, each SOBS high-resolution video frame (3 W × 3H pixels) is downsampled to a video frame of the original resolution (W × H pixels) by local averaging.
3.1. Parameter setting
Actually, THstill depends on the sizes of moving objects, the velocities of moving objects, and the frame rate (frames per second, fps) of each bootstrapping video sequence. Let A t be the minimum bounding rectangle of a moving object in frame I t and A t-FR be the minimum bounding rectangle of the moving object in frame I t-FR where FR (fps) is the frame rate of a bootstrapping video sequence. Note that the time difference between the two frames, I t-FR and I t , is 1 s. Here, the moving object is roughly determined as “high-motion” if A t-FR and A t do not contain any overlapping part. Otherwise, the moving object is roughly determined as “low-motion.” In this study, if a bootstrapping video sequence contains “high-motion” moving object(s), then (FR/2) ≤ THstill ≤ FR. Otherwise, FR ≤ THstill ≤ (FR + FR/2). The threshold values THstill for the six video sequences, namely, “Highway-1,” “Highway-2,” “S1-T1-C-3,” “S1-T1-C-4,” “Vignal,” and “Granguardia,” by the proposed approach are empirically set to 15, 15, 35, 35, 20, and 20, respectively.
3.2. Subjective comparisons
The average frame processing times (s) for the six bootstrapping video sequences by MoG, Reddy, SOBS, and the proposed approach with block size 16 × 16
0.068 ± 0.004
0.403 ± 0.252
0.389 ± 0.025
0.615 ± 0.024
The average frame processing times (s) of the three processing steps, namely, block representation, background updating, and foreground segmentation, for the six bootstrapping video sequences by the proposed approach with block size 16 × 16
0.488 ± 0.0243
0.002 ± 0.0005
0.124 ± 0.0026
3.3. Objective comparisons
false positive rate (FPR): FP/(FP + TN),
false negative rate (FNR): FN/(TN + FP),
percentage of wrong classifications (PWC):
f-measure (FM): 2 × (PR × RE)/(PR + RE).
Objective performance comparisons by four evaluation metrics FPR, FNR, PWC, and FM for the four video sequences in the “baseline” category of “changedetection.net” video dataset by MoG, SOBS, and the proposed approach
Objective performance comparisons by four evaluation metrics FPR, FNR, PWC, and FM for the four video sequences in the “baseline” category of “changedetection.net” video dataset by the proposed approach with different block sizes (8 × 8, 16 × 16, and 32 × 32)
8 × 8
16 × 16
32 × 32
For background initialization, including the evaluation of foreground masks, we can also evaluate the performance of the estimated background. In this study, the PSNR value of the estimated background, with respective to one “free” background (the groundtruth), is employed. The “free” background (the groundtruth) is synthesized by the “static” parts in different frames of the whole bootstrapping video sequence. The average PSNR values of SOBS and the proposed approach for the “baseline” category of “changedetection.net” video dataset  are 26.46 and 28.96 dB, respectively.
In this study, an effective background initialization and foreground segmentation approach for bootstrapping video sequences is proposed, in which a modified block representation approach, a new background updating scheme, and an improved noise removal and shadow suppression procedure with two morphological operations are employed. Based on the experimental results obtained in this study, as compared with MoG , Reddy  and SOBS , the proposed approach has better background initialization and foreground segmentation results. In addition, bootstrapping video sequences with jiggled capture, shadow effect, and heavy clutter can be well handled by the proposed approach.
This study was supported in part by the National Science Council, Taiwan, Republic of China under Grants NSC 99-2221-E-194-032-MY3 and NSC 101-2221-E-194-031.
- Moeslund TB, Hilton A, Kruer V: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Understand. 2006, 104(2):90-126. 10.1016/j.cviu.2006.08.002View ArticleGoogle Scholar
- Piccardi M: Background subtraction techniques: a review. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. The Hague, The Netherlands; 2004:3099-3104. vol. 4Google Scholar
- Toyama K, Krumm J, Brumitt B, Meyers B: Wallflower: principles and practice of background maintenance. In Proceedings of the 7th IEEE International Conference on Computer Vision. Kerkyra, Greece; 1999:255-261. vol. 1View ArticleGoogle Scholar
- Stauffer C, Grimson WEL: Adaptive background mixture models for real-time tracking. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. Ft. Collins, CO, USA; 1999:246-252. vol. 2Google Scholar
- Bouwmans T, Baf FE, Vachon B: Background modeling using mixture of Gaussians for foreground detection—a survey. Recent Patents Comput. Sci. 2008, 1(3):219-237. 10.2174/2213275910801030219View ArticleGoogle Scholar
- Shimada A, Arita D, Taniguchi R: Dynamic control of adaptive mixture-of-Gaussians background model. In Proceedings of the IEEE International Conference on Video and Signal Based Surveillance. Sydney, Australia; 2006:1-5.Google Scholar
- Tan R, Huo H, Qian J, Fang T: Traffic video segmentation using adaptive-k Gaussian mixture model. In Proceedings of the International Workshop on Intelligent Computing in Pattern Analysis/Synthesis. Xi'An, China; 2006:125-134.Google Scholar
- Wang H, Suter D: A re-evaluation of mixture-of-Gaussian background modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing. Philadelphia, PA, USA; 2005:1017-1020. vol. 2Google Scholar
- Lindstrom J, Lindgren F, Ltrstrom K, Holst J, Holst U: Background and foreground modeling using an online EM algorithm. In Proceedings of the IEEE International Workshop on Visual Surveillance. Graz, Austria; 2006:9-16.Google Scholar
- Han B, Lin X: Update the GMMs via adaptive Kalman filtering. In Proc SPIE. Beijing, China; 2005:1506-1515. vol. 5960Google Scholar
- Zhang Y, Liang Z, Hou Z, Wang H, Tan M: An adaptive mixture Gaussian background model with online background reconstruction and adjustable foreground mergence time for motion segmentation. In Proceedings of the IEEE International Conference on Industrial Technology. Istanbul, Turkey; 2005:23-27.Google Scholar
- White B, Shah M: Automatically tuning background subtraction parameters using particle swarm optimization. In Proceedings of the IEEE International Conference on Multimedia and Expo. Beijing, China; 2007:1826-1829.Google Scholar
- Lin HH, Chuang JH, Liu TL: Regularized background adaptation: a novel learning rate control scheme for Gaussian mixture modeling. IEEE Trans Image Process. 2011, 20(3):822-836.MathSciNetView ArticleGoogle Scholar
- Jain V, Kimia B, Mundy J: Background modeling based on subpixel edges. In Proceedings of the IEEE International Conference on Image Processing. San Antonio, Texas, USA; 2007:321-324. vol. 6Google Scholar
- Tian YL, Lu M, Hampapur A: Robust and efficient foreground analysis for real-time video surveillance. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition 2005, 1182-1187. vol. 1Google Scholar
- Tang P, Gao L, Liu Z: Salient moving object detection using stochastic approach filtering. In Proceedings of IEEE International Conference on Image and Graphics. Chengdu, China; 2007:530-535.Google Scholar
- Caseiro R, Henriques JF, Batista J: Foreground segmentation via background modeling on Riemannian manifolds. In Proceedings of International Conference on Pattern Recognition. Istanbul, Turkey; 2010:3570-3574.Google Scholar
- Quast K, Kaup A: Auto GMM-SAMT: an automatic object tracking system for video surveillance in traffic scenarios. EURASIP J. Image Video Process. 2011, 2011(814285):1-14.View ArticleGoogle Scholar
- Farin D, de With PHN, Effelsberg W: Robust background estimation for complex video sequences. In Proceedings of the International Conference on Image Processing. Barcelona, Spain; 2003:145-148.Google Scholar
- Massey M, Bender W: Salient stills: process and practice. IBM Syst J. 1996, 35(3–4):557-573.View ArticleGoogle Scholar
- Wang H, Suter D: A novel robust statistical method for background initialization and visual surveillance. In Proceedings of the 7th Asian Conf. on Computer Vision, Part I. Hyderabad, India; 2006:328-337.Google Scholar
- Reddy V, Sanderson C: BC Lovell, A Bigdeli, An efficient background estimation algorithm for embedded smart cameras. In Proceedings of the IEEE International Conference on Distributed Smart Cameras. Como, Italy; 2009:1-7.Google Scholar
- Baltieri D, Vezzani R, Cucchiara R: Fast background initialization with recursive Hadamard transform. In Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance. Boston, MA, USA; 2010:165-171.Google Scholar
- Colombari A, Fusiello A: Patch-based background initialization in heavily cluttered video. IEEE Trans. Image Process. 2010, 19(4):926-933.MathSciNetView ArticleGoogle Scholar
- Liu H, Chen W: An effective background reconstruction method for complicated traffic crossroads. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. San Antonio, TX, USA; 2009:1376-1381.Google Scholar
- Scott J, Pusateri MA, Cornish D: Kalman filter based video background estimation. In Proceedings of the IEEE Applied Imagery Pattern Recognition Workshop. Washington DC, USA; 2009:1-7.Google Scholar
- Maddalena L, Petrosino A: A self-organizing approach to background subtraction for visual surveillance applications. IEEE Trans. Image Process. 2008, 17(7):1168-1177.MathSciNetView ArticleGoogle Scholar
- Ghasemi A, Safabakhsh R: Unsupervised foreground-background segmentation using growing self organizing map in noisy backgrounds. In Proceedings of the IEEE International Conference on Computer Research and Development. Shanghai, China; 2011:334-338. vol. 1Google Scholar
- Chiu CC, Ku MY, Liang LW: A robust object segmentation system using a probability-based background extraction algorithm. IEEE Trans. Circuits Syst. Video Technol. 2010, 20(4):518-528.View ArticleGoogle Scholar
- Chien SY, Huang YW, Hsieh BY, Ma SY, Chen LG: Fast video segmentation algorithm with shadow cancellation, global motion compensation, and adaptive threshold techniques. IEEE Trans. Multimed. 2004, 6(5):732-748. 10.1109/TMM.2004.834868View ArticleGoogle Scholar
- Verdant A, Villard P, Dupret A, Mathias H: Three novell analog-domain algorithms for motion detection in video surveillance. EURASIP J. Image Video Process. 2011, 2011(698914):1-13.View ArticleGoogle Scholar
- Lin HH, Liu TL, Chuang JH: Learning a scene background model via classification. IEEE Trans. Signal Process. 2009, 57(5):1641-1654.MathSciNetView ArticleGoogle Scholar
- Tang Z, Miao Z, Wan Y, Jesse FF: Foreground prediction for bilayer segmentation of videos. Pattern Recognit. Lett. 2011, 32(14):1720-1734. 10.1016/j.patrec.2011.07.025View ArticleGoogle Scholar
- Zhao C, Wang X, Cham WK: Background subtraction via robust dictionary learning. EURASIP J. Image Video Process. 2011, 2011(972961):1-12.View ArticleGoogle Scholar
- Hsiao HH, Leou JJ: An effective foreground/background segmentation approach for bootstrapping video sequences. In Proceedings of the 2011 IEEE International Conference on Acoustics, Speech, and Signal Processing. Prague, Czech Republic; 2011:1177-1180.Google Scholar
- Ghanbari M: Standard Codecs: Image Compression to Advanced Video Coding. The Institution of Engineering and Technology, London, UK; 2003.View ArticleGoogle Scholar
- Atzori L, de Natale FGB, Perra C: A spatio-temporal concealment technique using boundary matching algorithm and mesh-based warping (BMA-MBW). IEEE Trans. Multimed. 2001, 3(3):326-338. 10.1109/6046.944476View ArticleGoogle Scholar
- Thaipanich T, Wu PH, Kuo CC: Video error concealment with outer and inner boundary matching algorithms. In Proc. SPIE. San Diego, CA, USA; 2007:1-11. vol. 6696Google Scholar
- Guan YP: Spatio-temporal motion-based foreground segmentation and shadow suppression. IET Comput. Vis. 2010, 4(1):50-60. 10.1049/iet-cvi.2008.0016View ArticleGoogle Scholar
- Gonzalez RC, Woods RE: Digital Image Processing. 3rd edition. Pearson Prentice Hall, Upper Saddle River, NJ, USA; 2008.Google Scholar
- Goyette N, Jodoin P, Porikli F, Konrad J, Ishwar P: Changedetection.net: a new change detection benchmark dataset. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition. Sherbrooke, Canada; 2012:1-8.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.