Low-complexity background subtraction based on spatial similarity

Abstract

Robust detection of moving objects from video sequences is an important task in machine vision systems and applications. To detect moving objects, accurate background subtraction is essential. In real environments, due to complex and various background types, background subtraction is a challenging task. In this paper, we propose a pixel-based background subtraction method based on spatial similarity. The main difficulties of background subtraction include various background changes, shadows, and objects similar in color to background areas. In order to address these problems, we first computed the spatial similarity using the structural similarity method (SSIM). Spatial similarity is an effective way of eliminating shadows and detecting objects similar to the background areas. With spatial similarity, we roughly eliminated most background pixels such as shadows and moving background areas, while preserving objects that are similar to the background regions. Finally, the remaining pixels were classified as background pixels and foreground pixels using density estimation. Previous methods based on density estimation required high computational complexity. However, by selecting the minimum number of features and deleting most background pixels, we were able to significantly reduce the level of computational complexity. We compared our method with some existing background modeling methods. The experimental results show that the proposed method produced more accurate and stable results.

Introduction

As security monitoring emerges as an important issue, there has been an increasing demand for intelligent surveillance systems. Key operations in intelligent surveillance include object tracking, abnormal behavior detection, and behavior understanding. Accurate background subtraction plays an important role. The goal of background subtraction is to eliminate background components and detect meaningful moving objects. In real environments, due to various and complex background types such as moving escalators, waving tree branches, water fountains, and flickering monitors, background subtraction is a difficult task. Researchers have overcome these problems by using background modeling. Simple background models assume static background images. Background components can generally be eliminated by computing the difference between an input image and the background image that was modeled using average, low-pass filtering, and median filtering[1–4]. For instance, in[1], the median background image was used to subtract the background components. Since temporal median filtering is time-consuming, a fast algorithm utilizing the characteristics of adjacent frames was proposed[2]. Cheng et al. applied a recursive mean procedure to compute background images[3]. In[4], low-pass filtering was utilized to estimate a static background image. However, these approaches cannot handle dynamic backgrounds and are sensitive to threshold values.

In order to handle various background types, statistical approaches were introduced. Among these approaches, Gaussian modeling methods have been widely used. Initially, uni-modal distribution was used to model pixel values[5]. In[6], a background subtraction method using the HSV color space was presented based on single Gaussian modeling. A fast and stable linear discriminant approach based on uni-modal distribution and Markov random field was proposed[7]. Rambabu and Woo proposed a background subtraction method which is robust against noisy and changing illumination based on single Gaussian modeling[8]. Although these models have low complexity levels and produce satisfactory performances in controlled backgrounds, it is difficult to use them for dynamic scenes. The Gaussian mixture model (GMM) is usually used to model various background types. Stuffer and Grimson used the GMM for background subtraction in[9], and it is still a popular method for background subtraction[10–20]. A spatio-temporal GMM (STGMM) was proposed to handle complex background[10]. Using a GMM, a statistical framework was investigated to localize a foreground object[11] and a dynamic background was modeled for highly dynamic conditions such as active cameras and high motion activities in background regions[12]. Also, the subtraction of two Gaussian kernels (difference of Gaussians) was used to eliminate background regions in embedded platforms[13]. A general framework of regularized online classification EM for GMM was proposed[14]. Wang et al. proposed an adaptive local-patch GMM to detect moving objects in dynamic background regions[15]. In[16], a new update algorithm was proposed for learning adaptive mixture models, and Bin et al. proposed a self-adaptive moving object detection algorithm. The method improved the original GMM in order to adapt to sudden or gradual illumination changes[17]. In[18], in order to improve GMM performance, a new rate control method based on high-level feedback was developed. An improved adaptive-K GMM method was presented for updating background regions[19], and GMM was used for modeling background regions in a Bayer-pattern domain[20]. A disadvantage of these multimodal Gaussian modeling methods is that they require pre-defined parameters such as the number of the Gaussian distributions and the standard deviations of those distributions. Also, dynamic backgrounds cannot be accurately modeled by a few Gaussian distributions. In order to overcome parameter background modeling methods, nonparametric background modeling techniques have been developed for estimating background probabilities. Nonparametric background modeling methods have been used to estimate background distribution based on pixel values observed in the past. In[21], the Gaussian kernel was used for pixel-based background modeling. This nonparametric method is usually used to handle multiple modes of dynamic backgrounds without pre-defined parameters. However, these nonparametric methods use kernel density estimation (KDE), which requires heavy computational complexity and a large amount of memory. Various efforts have been made to address these problems. Using Parzen density estimation and foreground object detection, a fast estimation method was presented[22] and an automatic background modeling based on multivariate non-parametric KDE was proposed[23]. In[24], a non-parametric method was proposed for foreground and background modeling, which did not require any initialization. Han et al. proposed an efficient algorithm for recursive density approximation based on density mode propagation[25]. Also, depth information, on-line auto-regressive modeling, and Gaussian family distribution were used to eliminate background regions[26–28]. In[29], new object segmentation was proposed based on a recursive KDE. It used the mean-shift method to approximate the local maximum value of the density function. The background was modeled using real-time KDE based on online histogram learning[30].

Also, alternative approaches were proposed based on neural network techniques or the support vector machine (SVM) method[31–35]. A method was proposed based on self-organization through artificial neural networks[31]. Furthermore, a self-organization method was combined with fuzzy approach to update background[32]. In[33–35], an automatic algorithm was proposed to perform background modeling using SVM.

To develop a robust model with low complexity, we used a pixel-based background subtraction method based on spatial similarity computed using the structural similarity method (SSIM)[36]. Using spatial similarity, we measured the pixel similarity and eliminated background pixels. The remaining pixels were classified as either background or foreground pixels using KDE. Since we eliminated most background pixels and used only two features for KDE, the complexity of the proposed method was significantly reduced. The proposed method was evaluated using two datasets (Wallflower's and Li's datasets) and showed favorable performance over some existing methods.

The overall algorithm for efficient background subtraction

Preparation

The structure similarity for eliminating background components

To eliminate background components while preserving potential foreground components, we first computed the spatial similarity using the SSIM method that was developed for image quality assessment[36]. The SSIM was computed as follows:

$$\begin{array}{l}\text{Lum}\text{inance}:\mathit{l}\left(\mathit{x},\mathit{y}\right)=\frac{2{\mathit{\mu }}_{\mathit{x}}{\mathit{\mu }}_{\mathit{y}}+{\mathit{C}}_1}{{\mathit{\mu }}_{\mathit{x}}^2+{\mathit{\mu }}_{\mathit{y}}^2+{\mathit{C}}_1}\\ \text{Contrast}:\mathit{c}\left(\mathit{x},\mathit{y}\right)=\frac{2{\mathit{\sigma }}_{\mathit{x}}{\mathit{\sigma }}_{\mathit{y}}+{\mathit{C}}_2}{{\mathit{\sigma }}_{\mathit{x}}^2+{\mathit{\sigma }}_{\mathit{y}}^2+{\mathit{C}}_2}\\ \text{Structure}:\mathit{s}\left(\mathit{x},\mathit{y}\right)=\frac{{\mathit{\sigma }}_{\mathit{xy}}+{\mathit{C}}_3}{{\mathit{\sigma }}_{\mathit{x}}{\mathit{\sigma }}_{\mathit{y}}+{\mathit{C}}_3}\\ \begin{array}{l}\text{SSIM}\left(\mathit{x},\mathit{y}\right)={\left[\mathit{l}\left(\mathit{x},\mathit{y}\right)\right]}^{\mathit{\alpha }}{\left[\mathit{c}\left(\mathit{x},\mathit{y}\right)\right]}^{\mathit{\beta }}{\left[\mathit{s}\left(\mathit{x},\mathit{y}\right)\right]}^{\mathit{\gamma }}\\ =\frac{\left(2{\mathit{\mu }}_{\mathit{x}}{\mathit{\mu }}_{\mathit{y}}+{\mathit{C}}_1\right)\left(2{\mathit{\sigma }}_{\mathit{\text{xy}}}+{\mathit{C}}_2\right)}{\left({\mathit{\mu }}_{\mathit{x}}^2+{\mathit{\mu }}_{\mathit{y}}^2+{\mathit{C}}_1\right)\left({\mathit{\sigma }}_{\mathit{x}}^2+{\mathit{\sigma }}_{\mathit{y}}^2+{\mathit{C}}_2\right)}\end{array}\end{array}$$

(1)

where α, β, and γ are parameters which determine the relative importance of l(x, y), c(x, y), and s(x, y) and we set α, β, and γ to 1. μ _x and μ _y are the local means, σ _x and σ _y are the local standard deviations, σ _xy is the local covariance coefficient between regions x and y, and we set C₃ to C₂/2 and C₁ and C₂ are constants that were set to 6.5025 and 58.5225 as proposed in[36]. In Equation 1, l, c, and s represent the luminance, contrast, and structure of two images. In this paper, we computed the SSIM for local regions (e.g., a 3 × 3 block) to eliminate background components. Figure 1a and b show the input and reference background images, respectively. Figure 1c and g show the intensity and hue difference images between Figure 1a and b, respectively. The SSIM difference image between Figure 1a and b is shown in Figure 1k. Thresholding (if a pixel value of the difference image was larger than the given threshold value, the pixel was eliminated) was applied to the difference images with various threshold values (low, medium, large), and the resulting images are shown in Figure 1d,e,f,h,i,j,l,m,n. For the intensity component (Figure 1c,d,e,f), the differences between the shadow regions and the corresponding background regions were high. The thresholding operation still left shadows when using a low threshold value (e.g., 80). When we used a larger threshold value (e.g., 120) to eliminate the shadows, potential foreground objects were also eliminated (Figure 1f).

Figure 1

The characteristic of features. (a) An input image, (b) the reference background image, (c) intensity difference image between (a) and (b), (d) thresholding of (c) with a low value (80), (e) thresholding of (c) with a middle value (100), (f) thresholding of (c) with a large value (120), (g) hue difference image between (a) and (b), (h) thresholding of (g) with a low value (80), (i) thresholding of (g) with a middle value (100), (j) thresholding of (g) with a large value (120), (k) SSIM difference image between (a) and (b), (l) thresholding of (k) with a low value (0.4), (m) thresholding of (k) with a middle value (0.5), and (n) thresholding of (k) with a large value (0.6).

For the hue component (Figure 1g,h,i,j), shadows were not retained, but many of the background regions contained high difference values. To eliminate these background regions, we tried using a larger threshold value (e.g., 120). However, the top portion of the person with the blue jacket and the red portion of the person on the right were also eliminated. Furthermore, the small object in the lower-left corner was almost deleted when the intensity component or the hue component was used. However, the method based on the SSIM correctly retained the object (Figure 1k,l,m,n). In the SSIM, global intensity and contrast changes were not determined as forms of distortion[36]. Therefore, the proposed method proved to be robust against shadows with lower intensity values while retaining internal structures. Furthermore, since the proposed method used the variances and covariance of two local regions, it could detect objects with similar colors. In Figure 2a, a person's head color was similar to the background regions. The proposed method showed improved performance compared to the other method[31] (http://www.na.icar.cnr.it/~maddalena.l/MODLab/SoftwareSOBS.html). Similarly, in Figure 2b, the woman's jacket color was similar to the background regions. The proposed method correctly classified the woman as a foreground object while the other method missed the jacket.

Figure 2

The effect for detecting the objects which are similar with backgrounds. (a) An input image 1, (b) the results of other method[31], (c) the results of the proposed method, (d) an input image 2, (e) the results of other method[31], and (f) the results of the proposed method.

To apply the SSIM to local regions, we used a sliding window approach. For each pixel, we computed the SSIM of a 3 × 3 window centered at the pixel. Let A(i, j) = ⌊A^R(i, j), A^G(i, j), A^B(i, j)⌋ be a pixel in the RGB color space. Then, the similarity image (SI) between intensity images A^I(i, j) and B^I(i, j) was calculated as follows:

$${\text{SI}}_{{\mathit{A}}^{\mathit{I}},{\mathit{B}}^{\mathit{I}}}\left(\mathit{i},\mathit{j}\right)=\text{SSIM}\left({\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right),{\mathit{B}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)\right)$$

(2)

where

$$\begin{array}{ll}{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)& =\frac{1}{3}\left({\mathit{A}}^{\mathit{R}}\left(\mathit{i},\mathit{j}\right)+{\mathit{A}}^{\mathit{G}}\left(\mathit{i},\mathit{j}\right)+{\mathit{A}}^{\mathit{B}}\left(\mathit{i},\mathit{j}\right)\right),{\mathit{\mu }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}\\ =\frac{1}{9}\left({\displaystyle \sum _{\mathit{v}=-1}^1{\displaystyle \sum _{\mathit{u}=-1}^1{\mathit{A}}^{\mathit{I}}\left(\mathit{i}+\mathit{u},\mathit{j}+\mathit{v}\right)}}\right)\end{array}$$

(3)

$$\begin{array}{l}{\mathit{\sigma }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}^2=\frac{1}{9}\left({\displaystyle \sum _{\mathit{v}=-1}^1{\displaystyle \sum _{\mathit{u}=-1}^1\left({\mathit{A}}^{\mathit{I}}{\left(\mathit{i}+\mathit{u},\mathit{j}+\mathit{v}\right)}^2-{\mathit{\mu }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}\right)}}\right),\\ {\mathit{\sigma }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right){\mathit{B}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}=\frac{1}{9}\left(\sum _{\mathit{v}=-1}^1\sum _{\mathit{u}=-1}^1\left({\mathit{A}}^{\mathit{I}}\left(\mathit{i}+\mathit{u},\mathit{j}+\mathit{v}\right)-{\mathit{\mu }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}\right)\right.\\ \times \left.\left({\mathit{B}}^{\mathit{I}}\left(\mathit{i}+\mathit{u},\mathit{j}+\mathit{v}\right)-{\mathit{\mu }}_{{\mathit{B}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}\right)\right)\end{array}$$

A^I(i, j) represents an intensity value,${\mathit{\mu }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}$ and${\mathit{\mu }}_{{\mathit{B}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}$ are intensity means,${\mathit{\sigma }}_{{\mathit{A}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}$ and${\mathit{\sigma }}_{{\mathit{B}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}$ are intensity standard deviations, and${\mathit{\sigma }}_{{\mathit{A}}^{\mathit{I}}{\mathit{B}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}$ is the intensity covariance.$\mathit{S}{\mathit{I}}_{{\mathit{A}}^{\mathit{I}},{\mathit{B}}^{\mathit{I}}}\left(\mathit{i},\mathit{j}\right)$ is close to 1 when two window regions were similar. C₁ and C₁ were set to 6.5025 and 58.5225, respectively[36]. By assuming that one image was a reference background image, we obtained a binary background image (BBI) by applying a thresholding operation:

$${\text{BBI}}_{{\mathit{A}}^{\mathit{I}},{\mathit{B}}^{\mathit{I}}}\left(\mathit{i},\mathit{j}\right)=\left\{\begin{array}{l}0\left(\text{background}\right)\text{if}\left({\text{SI}}_{{\mathit{A}}^{\mathit{I}},{\mathit{B}}^{\mathit{I}}}\left(\mathit{i},\mathit{j}\right)>{\mathit{T}}_1\right)\\ 1\left(\text{foreground}\text{candidate}\right)\text{otherwise}\end{array}\right.$$

(4)

T₁ is a threshold value which was empirically determined and set to 0.55. Figure 3 shows the effect of the threshold value. When we used a small value for T₁, most pixels were classified as background regions (Figure 3c). When we used a large value for T₁, most pixels were classified as foreground regions (Figure 3o). Based on this observation, we set T₁ to 0.55, though any value between 0.1 and 0.9 provided good performance.

Figure 3

The effect of threshold T ₁ . (a) An input image, (b) the reference background image, (c) T₁ = 0.1, (d) T₁ = 0.2, (e) T₁ = 0.3, (f) T₁ = 0.35, (g) T₁ = 0.4, (h) T₁ = 0.45, (i) T₁ = 0.50, (j) T₁ = 0.55, (k) T₁ = 0.60, (l) T₁ = 0.65, (m) T₁ = 0.70, (n) T₁ = 0.80, and (o) T₁ = 0.90.

Since we calculated the means and the variances, the computational complexity was low. However, some background pixels were still retained. In order to eliminate the background pixels, we used nonparametric kernel density estimation.

Determining foreground and background areas using KDE

Generally, KDE can model multi-modal probability distributions without requiring any prior information. It is effective for modeling the arbitrary densities of real environments. KDE was applied to each pixel of the training images. In other words, we extracted training samples at each pixel location of the training images. Let s₁, s₂, …, s _N be training samples and we used the Gaussian kernel function. Then, the probability of x _t was calculated as follows[21]:

$$\mathit{p}\left({\mathit{x}}_{\mathit{t}}\right)=\frac{1}{\mathit{N}}{\displaystyle \sum _{\mathit{i}=1}^{\mathit{N}}\frac{1}{\sqrt{2\mathit{\pi \sigma }}}}{\mathit{e}}^{-\frac{1}{2\mathit{\sigma }}{\left({\mathit{s}}_{\mathit{i}}-{\mathit{x}}_{\mathit{t}}\right)}^2}$$

(5)

where σ represents the kernel function bandwidth and N is the number of training samples. A pixel was classified as a background pixel if the estimated probability was larger than the given threshold. It was observed that a large value of N produced more robust results. Consequently, a typical KDE method requires a large number of operations. On the other hand, we first eliminated most background pixels using the spatial similarity (SS) method and used only two features (one of the RGB components and one of the normalized RGB components). Also, we used a small number of samples (one hundred samples). Therefore, we were able to significantly reduce the computational complexity of the KDE without sacrificing performance. Figure 4 shows an example of the proposed method. We eliminated most background pixels using the SS method (Figure 4c). However, some background pixels were still retained and we eliminated these pixels using KDE. In this case, the candidate pixels made up 5% to 6% of the entire image. The processing time was also reduced accordingly.Based on this observation, we propose a computationally efficient background subtraction method by eliminating background regions using spatial similarity in the spatial domain and the KDE method in the temporal domain. By combining spatial and temporal features, the proposed method produced better performance than the conventional KDE method. Figure 5 shows the comparison results. These sequences contain dynamic background regions. Tree branches were swaying and the curtain was moving in the wind. In dynamic background regions, it is difficult to accurately model the background in the conventional KDE method. Therefore, many background components are often classified as foreground components. However, since most of the background components in the proposed method were eliminated with spatial similarity, most of the background components misclassified as foreground components were correctly classified as background components.

Figure 4

An example of the proposed method. (a) An input image, (b) the reference background image, (c) the similarity image, and (d) the final result.

Figure 5

The results of the proposed method and the single KDE method. First column: input image; second column: results of single KDE method; and third column: the proposed method.

The proposed method

Determine the background type

The reference background image (RBI) was computed as the average of the training intensity images:

$${\mathrm{RBI}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)=\frac{1}{\mathit{N}}{\displaystyle \sum _{\mathit{t}=0}^{\mathit{N}-1}{\mathit{A}}_{\mathit{t}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)}$$

(6)

where${\mathit{A}}_{\mathit{t}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)$ represents a pixel of the t-th intensity image of a video sequence and N is the number of training images, which was set to 100. In other words, the first 100 images of a given video sequence generally were used as training images. We also computed the averages of the RGB channels of the training images:

$${\text{RBI}}^{\mathrm{\Omega }}\left(\mathit{i},\mathit{j}\right)=\frac{1}{\mathit{N}}{\displaystyle \sum _{\mathit{t}=0}^{\mathit{N}-1}{\mathit{A}}_{\mathit{t}}^{\mathrm{\Omega }}\left(\mathit{i},\mathit{j}\right)}\text{where}\mathrm{\Omega }\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}$$

(7)

Then, a similarity image between the reference background and training intensity images was computed using Equation 2 and the reference binary background image (RBBI) was computed:

$$\begin{array}{l}\text{For}\text{each}\text{pixel}\left(\mathit{i},\mathit{j}\right)\\ \mathit{r}\left(\mathit{i},\mathit{j}\right)=\frac{1}{\mathit{N}}{\displaystyle \sum _{\mathit{t}=0}^{\mathit{N}-1}{\text{SI}}_{{\text{RBI}}^{\mathit{I}},{\mathit{A}}_{\mathit{t}}^{\mathit{I}}}\left(\mathit{i},\mathit{j}\right)}\\ \text{RBBI}\left(\mathit{i},\mathit{j}\right)=\left\{\begin{array}{l}0\left(\text{static}\text{background}\right)\text{if}\left(\mathit{r}\right(\mathit{i},\mathit{j})>0.8)\\ 1\left(\text{moving}\text{background}\right)\text{otherwise}\end{array}\right.\end{array}$$

(8)

The RBBI successfully detected moving background components such as moving escalators, waving tree branches, and water fountains.

Determine the foreground candidate pixels

When a new image was entered, a BBI was computed between the RBI and the input intensity image using Equations 3 to 4. If BBI(i, j) = 1, the pixel could have been either a foreground pixel or a moving background pixel. If RBBI(i, j) = 1 (moving background), we computed the difference between the intensity input image and the RBI^I. If the difference between the input intensity image and the RBI^I was small, the pixel could have been a background pixel. Also, the pixel was classified as a foreground candidate when the difference was larger than the given threshold, and the pixel was classified as a foreground candidate if BBI(i, j) = 1 and RBBI(i, j) = 0. The following procedure was used to classify a pixel:

$$\begin{array}{l}\text{For}\text{each}\text{pixel}\left(\mathrm{i},\mathrm{j}\right)\\ \text{If}({\text{BBI}}_{{\text{RBI}}^{\mathit{I}},{\mathit{I}}_{\mathit{k}}^{\mathit{I}}}\left(\mathit{i},\mathit{j}\right)=1),\text{then}\\ \text{If}(\text{RBBI}\left(\mathit{i},\mathit{j}\right)=1),\text{then}\\ {\text{FCI}}_{\mathit{k}}\left(\mathit{i},\mathit{j}\right)=\left\{\begin{array}{l}\mathit{if}\left(|{\text{RBI}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)-{{\mathit{I}}_{\mathit{k}}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)|>{\mathit{T}}_2\right)\\ 1\left(\text{foreground}\text{candidate}\right)\\ \text{otherwise}\\ 0\left(\text{background}\right)\end{array}\right.\\ \text{Otherwise}\\ \text{FC}{\mathit{I}}_{\mathit{k}}\left(\mathit{i},\mathit{j}\right)=1\left(\text{foreground}\text{candidate}\right)\end{array}$$

(9)

where FCI _k (i, j) represents a candidate image,${\mathit{I}}_{\mathit{k}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)$ represents the k-th input intensity image (see Equation 3), and T₂ was empirically set to 30. If T₂ was too large, most pixels were classified as background pixels. In other words, many foreground pixels were misclassified as background pixels when T₂ was too large. Figure 6 shows the results for various values of T₂. Figure 6a,b shows an input image and the BBI. Figure 6c,d,e,f shows the FCI for various values of T₂. Most foreground pixels were eliminated when T₂ was set to 80 (Figure 6f), while most moving background pixels were retained when T₂ was set to 10 (Figure 6c). In order to choose an optimal threshold value, we tested the proposed method with various values of T₂ using some video sequences with dynamic background regions and chose the threshold value (T₂ = 30). At this point, most background regions were removed.

Figure 6

The results with various threshold ( T ₂ ) values. (a) An input image, (b) BBI, (c) T₂ = 10, (d) T₂ = 30, (e) T₂ = 60, and (f) T₂ = 80.

Subtract the background pixels using KDE

We classified only the foreground candidate pixels (i.e., FCI _k (i, j) = 1) using KDE. Since there were high correlations among the R, G, and B components, and using all three channels produced only slight improvements, we used only the color with the largest difference. To improve performance, we also used one of the normalized RGB components that were robust against illumination changes and that represented the chrominance information well. We selected one of the RGB component channels as follows:

$$\begin{array}{l}\text{For}\text{each}\text{pixel}\left(\mathit{i},\mathit{j}\right)\text{of}\text{the}\mathit{k}‒\text{th}\text{frame}\\ \text{If}({\text{FCI}}_{\mathit{k}}\left(\mathit{i},\mathit{j}\right)=1)\\ {\mathit{d}}_{\text{max}}=max\left({\mathrm{Diff}}_{\mathrm{R}},{\mathrm{Diff}}_{\mathrm{G}},{\mathrm{Diff}}_{\mathrm{B}}\right)\end{array}$$

(10)

where

$$\begin{array}{l}{\text{Diff}}_{\mathrm{R}}=|{\text{RBI}}^{\mathrm{R}}\left(\mathit{i},\mathit{j}\right)-{\mathit{I}}_{\mathit{k}}^{\mathrm{R}}(\mathit{i},\mathit{j}\left)\right|\\ {\text{Diff}}_{\mathrm{G}}=|{\text{RBI}}^{\mathrm{G}}\left(\mathit{i},\mathit{j}\right)-{\mathit{I}}_{\mathit{k}}^{\mathrm{G}}(\mathit{i},\mathit{j}\left)\right|\\ {\text{Diff}}_{\mathrm{B}}=|{\text{RBI}}^{\mathrm{B}}\left(\mathit{i},\mathit{j}\right)-{\mathit{I}}_{\mathit{k}}^{\mathrm{B}}(\mathit{i},\mathit{j}\left)\right|\end{array}$$

where d_max represents the maximum difference. Let Ω_max be the channel with the maximum difference.

A foreground candidate pixel was classified as a background pixel when the estimated probability density function of the pixel value was larger than the given threshold as follows:

$$\begin{array}{l}\text{if}\frac{1}{\mathit{N}}{\displaystyle \sum _{\mathit{m}=0}^{\mathit{N}-1}\frac{1}{\sqrt{2\mathit{\pi \sigma }}}}{\mathit{e}}^{-\frac{1}{2\mathit{\sigma }}{\left({\mathit{I}}_{\mathit{k}}^{{\mathrm{\Omega }}_{\text{max}}}\left(\mathit{i},\mathit{j}\right)-{\mathit{A}}_{\mathit{m}}^{{\mathrm{\Omega }}_{\text{max}}}\left(\mathit{i},\mathit{j}\right)\right)}^2}>{\mathit{T}}_3,\\ \text{decide the pixel as background}\\ \text{otherwise},\\ \text{decide the pixel as foreground}\end{array}$$

(11)

where σ represents the kernel width. Since the probability density function of the background pixel was unknown, we assumed that the probability densities for all intensity values were identical. Therefore, we set T₃ to 1/256. We used the standard deviation of the training images as the kernel width. This procedure was repeated using the normalized RGB color components, which were computed as follows:

$${\mathit{I}}_{\text{normalized}}^{\mathrm{\Omega }}\left(\mathit{i},\mathit{j}\right)=\frac{255\cdot {\mathit{I}}^{\mathrm{\Omega }}\left(\mathit{i},\mathit{j}\right)}{{\mathit{I}}^{\mathrm{R}}\left(\mathit{i},\mathit{j}\right)+{\mathit{I}}^{\mathrm{G}}\left(\mathit{i},\mathit{j}\right)+{\mathit{I}}^{\mathrm{B}}\left(\mathit{i},\mathit{j}\right)}\text{with}\mathrm{\Omega }\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}$$

(12)

where I(i, j) represents the input image. If either the estimated probability density function of the pixel using the original RGB channels or the estimated probability density function of the pixel value of the normalized RGB channels was classified as a foreground component, the pixel was determined to be a foreground component. After this procedure, there were several small holes inside the foreground regions and some noise elements in the background regions. Most pixel-based methods suffer from this kind of problem. In order to address this, we applied a morphological operation to remove the small holes and noise elements. In particular, we used erosion followed by dilation and then a region filling technique was applied to the results[37].

Updating

After the decision procedure, the RBI and the pixels of the training images had to be updated to adapt to the changing background areas. We used a simple IIR filter to update the RBI as follows[38]:

$$\begin{array}{l}\text{If}\text{pixel}\left(\mathit{i},\mathit{j}\right)\text{is classified as background},\\ {\text{RBI}}^{\mathrm{\Omega }}\left(\mathit{i},\mathit{j}\right)=(1-\mathit{\alpha }){\text{RBI}}^{\mathrm{\Omega }}(\mathit{i},\mathit{j})+\mathit{\alpha }{\mathit{I}}_{\mathit{k}}^{\mathrm{\Omega }}(\mathit{i},\mathit{j})\\ \text{where}\mathrm{\Omega }\in \left\{\mathrm{R},\mathrm{G},\mathrm{B}\right\}\end{array}$$

(13)

where α represents the learning rate and was set to 0.01. The training images were updated by replacing the oldest pixel with the new background pixel. There is a trade-off in the choice of α. If a value for α was large, the RBI quickly reflected background changes. Figures 7 and8 show the RBI changes for various learning rate values. As can be seen in Figure 7, the RBI was affected by shadows when we used a large value for α. Figure 7a,b shows the 372nd input and the initial RBI images. Figure 7c shows the RBI image when α was 0.6. Because of a large value for α, the RBI was quickly affected by the shadows. If we used a small value for α, the RBI did not quickly reflect background changes.

Figure 7

The RBI images at different values of the learning rate 1. (a) Initial RBI^I, (b) 372th input image, (c) RBI^I with the α = 0.6, (d) RBI^I with the α = 0.3, (e) RBI^I with the α = 0.05, and (f) RBI^I with the α = 0.01.

Figure 8

The RBI images at different values of the learning rate 2. (a) The first input image, (b) the 1,386th input image, (c) the 1,386th RBI^I with the α = 0.001, and (d) the 1,386th RBI^I with the α = 0.01.

In some test sequences, the background gradually became brighter over a period (Figure 8). The RBI did not reflect this gradual background change with a small value of α (Figure 8c). Thus, we set α = 0.01, and the learning rate was able to handle background changes adequately (Figure 8d).

If sudden background changes occurred, the results may have been erroneous. In order to handle such sudden background changes, we calculated the image intensity difference between the input image and the RBI and determined that sudden background changes occurred if the difference was larger than the given threshold:

$$\begin{array}{l}\text{if}\left(\frac{1}{{\mathit{N}}_{\mathit{x}}\cdot {\mathit{N}}_{\mathit{y}}}{\displaystyle \sum _{\mathit{i}=0,\mathit{j}=0}^{\mathit{i}={\mathit{N}}_{\mathit{x}}-1,\mathit{j}={\mathit{N}}_{\mathit{y}}-1}\left|{\mathit{I}}_{\mathit{k}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)-{\text{RBI}}^{\mathit{I}}\left(\mathit{i},\mathit{j}\right)\right|}>30\right),\\ \text{a sudden background change occurs}\\ \text{at the k}‒\text{th sequence}.\end{array}$$

(14)

When a sudden background change was detected at the k-th image, we calculated the image differences between the previous 100 images (from the (k-99)-th image to the k-th image) and the RBI. We selected the previous images that had larger frame differences than the threshold. The selected images were temporarily used as the training images. If the number of selected images was smaller than 15, all the pixels of the k-th image were classified as background components. However, the RBI was not updated when sudden changes were detected.Figure 9 shows an example of the proposed background subtraction procedure. Figure 9a is an input image, and Figure 9b shows the reference background image. Figure 9c is the reference binary background image where the white areas represent moving backgrounds (the waving trees). Figure 9d shows the binary background image between Figure 9a and b. Figure 9e shows the foreground candidate image. Figure 9f shows the result obtained using the original RGB components, and Figure 9g shows the final result using the normalized RGB components and the morphological operation.

Figure 9

An example of the overall procedure of the proposed method. (a) An input image, (b) RBI, (c) RBBI, (d) the BBI between (a) and (b), (e) the foreground candidate image, (f) the result obtained using the original RGB components, and (g) the final result.

Experimental results

Experiments were performed using two datasets (Li's dataset and the Wallflower's dataset). Li's dataset contained several dynamic background video sequences (water surface (WS), campus (CAM), fountain (FT), and meeting room (MR)) and static background video sequences (shopping center (SC), subway station (SS), airport (AP), lobby (LB), bootstrap (B)). The Wallflower's dataset contained various background types (bootstrap (B), camouflage (C), foreground aperture (FA), light switch (LS), moved object (MO), time of day (TD), and waving tree (WT)).

First, we measured the processing time of the proposed method. The proposed method took about 0.015 s per 10,000 pixels, while the processing time of a conventional method[38] was about 1.475 s per 10,000 pixels (using a 2.8-GHz Pentium IV with 1 GB of RAM) when the number of sample images was 100. For instance, the proposed method processed 66.7 frames of video per second when working with 160 × 128 video sequences. The complexity of KDE is O_KDE(MN) evaluations (the kernel function, multiplications and additions), assuming N image pixels and M sample points (N pixels per image and M training images). In the proposed method, we applied ‘spatial similarity’ to eliminate potential background pixels using a window processing operation (size of window = w). The computational complexity for calculating spatial similarity is O_similarity(w²N) operations (multiplications and additions). Then, the remaining pixels (the number of remaining pixels: K = τN) are further processed using KDE (O_KDE(KM)). Therefore, the computational complexity of the proposed method is calculated as follows:

$$\begin{array}{l}\text{Number of operation}={\mathit{O}}_{\text{similarity}}\left({\mathit{w}}^2\mathit{N}\right)+{\mathit{O}}_{\text{KDE}}\left(\mathit{\text{KM}}\right)\\ ={\mathit{O}}_{\text{similarity}}\left({\mathit{w}}^2\mathit{N}\right)+{\mathit{O}}_{\text{KDE}}\left(\mathit{\tau }\mathit{\text{NM}}\right)\end{array}$$

(15)

In the proposed method, the window size is 3 (w = 3), and the average remaining pixels were about 5% ~ 6% of the entire image pixels (τ ≅ 0.05). In other words, the KDE operation was reduced by approximately 95%. Although we needed to compute additional spatial similarity, it had a minor effect on the overall complexity. With 100 training images, the computational complexity for KDE and the proposed method was O(100 N) and O((9 + 0.05 × 100)N) = O(14 N), respectively. In this case, the complexity of the proposed method was about 14% of KDE.

Next, the proposed method was compared with some existing algorithms[31, 38–40]. The Jaccard similarity was used as a performance measure[41]:

$$\text{JS}=\frac{\text{TP}}{\text{TP}+\text{FP}+\text{FN}}$$

(16)

where TP represents the number of true positive pixels, FP represents the number of false positive pixels, and FN represents the number of false negative pixels. Generally, a higher Jaccard similarity index indicates better performance.

Results using Li's dataset

Table 1 shows a performance comparison with Li's dataset based on Jaccard similarity.Figure 10 shows the background subtraction results of the proposed method and Li's method using Li's dataset. The first column shows a test image, the second column shows the ground truth data of the test image, the third column shows the results of Li's method, and the fourth column shows the results of the proposed method. Using spatial similarity, the proposed method was robust against shadows. Noticeable improvements were observed in the SC, LB, B, and AP sequences which contained significant shadows. For these sequences, the proposed method showed about 8.4% ~ 14.9%, 7.8% ~ 20.6%, and 2.91% ~ 12.7% improvement compared to SOBS, Li's method, and Park's method, respectively, in terms of the Jaccard similarity. Since the proposed method used covariance, the variances of two local regions, and the normalized RGB color components, it was able to detect some objects that were similar to the background intensity. Therefore, in the WS and the FT sequences that contained objects whose intensity values were similar to the background regions, the proposed method showed improved performance compared to the other methods. For instance, a main difficulty of the WS sequence was detecting a person's leg when the intensity value of the leg was similar to the background intensity value. The other methods missed parts of the leg while the proposed method accurately detected the leg. For this WS sequence, the proposed method showed about 10.4%, 7.8%, and 2.91% improvements compared to SOBS, Li's method, and Park's method. A main difficulty of the FT sequence was that a person's pants color was similar to the background region when the person stood against the fountain. For the FT sequence, the Jaccard similarity of the proposed method was 0.820, and the proposed method showed about 16.5%, 14.6%, and 10.3% improvements compared to SOBS, Li's method, and Park's method, respectively. However, some sequences (e.g., CAM, SS, and MR) contained complex dynamic background sequences. For instance, in the CAM sequence, the background included tree branches that were constantly swayed by a strong wind. The SS sequence contained moving escalators and the MR sequence contained moving curtains). In these kinds of dynamic background sequences, Park’s method (in CAM, SS, and MR) and Li's method (in MR) performed slightly better than the proposed method.

Table 1 Performance comparison with Jaccard similarity (Li's dataset)

Figure 10

Background subtraction results of the proposed method and Li's method using Li's dataset.

Results using Wallflower's dataset

Table 2 shows a performance comparison with Wallflower's dataset based on FP + FN. Figure 11 shows the results of the proposed method and Wallflower method using the Wallflower's dataset. The first column shows a test image, the second column shows the ground truth data of the test image, the third column shows the results of Wallflower method, and the fourth column shows the results of the proposed method. The proposed method showed noticeable improvements for the C and B sequences. In the B sequence, the proposed method successfully detected objects that were similar to the background areas. On the other hand, since some moving trees of the WT sequence were classified as foreground components, the proposed method was not as good as Park's method. The LS sequence contained a sudden background change and the proposed method showed better performance. In the MO sequence, the proposed method classified the relocated objects (the chair and the phone) as foreground components. To handle this kind of problem, higher level processing such as that used in the Wallflower method might be required. The proposed method missed an object whose color was similar to that of the background area in the TD sequence.

Table 2 Performance comparison with the number of false positive and false negative pixels (Wallflower's dataset)

Figure 11

Background subtraction results of the proposed method and the Wallflower method using the Wallflower's dataset.

The effects of thresholds

Next, we investigated the effects of thresholds (T₁ and T₂ in Equations 8 to 9). Figure 12 shows the Jaccard similarity of the proposed method as the T₁ and T₂ values increased with Li's dataset and wallflower's dataset. In order to analyze the effect of T₁, we computed the false positive ratio (FPR) and false negative ratio (FNR) metrics as follows:

$$\begin{array}{l}\text{FPR}=\frac{\text{FP}}{\text{TP}+\text{FP}+\text{FN}}\\ \text{FNR}=\frac{\text{FN}}{\text{TP}+\text{FP}+\text{FN}}\end{array}$$

(17)

Figure 12

The effects of thresholds ( T ₁ and T ₂ ).

When we used a large value for T₁, most foreground pixels were correctly classified as foreground pixels. However, many background pixels also were classified as foreground pixels. Therefore, FPR increased and FNR decreased. When we used a small value for T₁, most background pixels were classified as background pixels. However, many foreground pixels were classified as background pixels. Therefore, FPR decreased and FNR increased when we used a small value for T₁. Figure 13 shows the Jaccard similarity, and the FPR and FNR metrics with various values for T₁ (T₂ was fixed and set at 30).

Figure 13

The evaluation metrics with various T ₁ values.

We selected the optimal value for T₁ and T₂. When we set T₁ and T₂ to 0.55 and 30 respectively, the foreground candidate pixels were about 5% of the entire number of pixels, and the Jaccard similarity of the proposed method was about 0.78 with Li's dataset and the FP and FN numbers were about 6,888 with Wallflower's dataset. Experiments with various values of T₁ and T₂ show that the proposed method produced stable performance when the value of T₁ was from 0.5 to 0.65 and the value of T₂ was from 25 to 35.

Conclusions

In this paper, we proposed a background subtraction method that utilized structural similarity, which was robust against various background areas. The proposed method also significantly reduced the level of computational complexity since most pixels were eliminated using the similarity image. We tested the proposed method with two datasets and then compared the proposed method with some existing methods. The experimental results demonstrated that the proposed method was effective for various background scenes and compared favorably with some existing algorithms.

Authors’ information

Sangwook Lee received the BS and MS degrees in electrical and electronic engineering from Yonsei University, Seoul, Repiblic of Korea in 2004 and 2006, respectively. He is currently working toward the PhD degree from Yonsei University and a senior engineer at Samsung Electronics Co. Ltd., Republic of Korea. His research interests include machine vision, image/signal processing, and video quality measurement.

Chulhee Lee received the BS and MS degrees in electronic engineering from Seoul National University in 1984 and 1986, respectively, and a PhD degree in electrical engineering from Purdue University, West Lafayette, Indiana, in 1992. In 1996, he joined the faculty of the Department of Electrical and Computer Engineering, Yonsei University, Seoul, Republic of Korea. His research interests include image/signal processing, pattern cognition, and neural networks.