Moving shadow detection based on stationary wavelet transform

Abstract

Many surveillance and forensic applications face problems in identifying shadows and their removal. The moving shadow points overlap with the moving objects in a video sequence leading to misclassification of the exact object. This article presents a novel method for identifying and removing moving shadows using stationary wavelet transform (SWT) based on a threshold determined by wavelet coefficients. The multi-resolution property of the stationary wavelet transform leads to the decomposition of the frames into four different bands without the loss of spatial information. The conventional discrete wavelet transform (DWT), which has the same property, suffers from the problem of shift invariance due to the decimation operation leading to a shift in the original signal during reconstruction. Since SWT does not have the decimation operation, the problem of shift invariance is solved which makes it feasible for change detection, pattern recognition and feature extraction and retrieves the original signal without the loss of phase information also. For detection and removal of shadow, a new threshold in the form of a variant statistical parameter—“skewness”—is proposed. The value of threshold is determined through the wavelet coefficients without the requirement of any supervised learning or manual calibration. Normally, the statistical parameters like mean, variance and standard deviation does not show much variation in complex environments. Skewness shows a unique variation between the shadow and non-shadow pixels in various environments than the previously used thresholds—standard deviation and relative standard deviation. The experimental results prove that the proposed method works better than other state-of-art-methods.

Introduction

The automated visualization applications are shaped mainly to acquire the appearance of moving points from a video sequence using various pattern recognition and machine learning techniques. In particular, people and vehicles are the most important subjects for monitoring in any typical surveillance and video forensics applications. They usually describe humans with their appearance, size and behaviour and vehicles by their make, model and type. Similarly annotations of humans and vehicles help in video forensic applications to search for people or vehicle with a recognized portrayal by a witness. The major issue in such surveillance and forensic videos is moving shadows. They prevent the correct identification of the suspects in case of any unlawful activities. Serious flaws like merging of more than one object, incorrect object shape and deformation of its contour and cues may happen if moving shadows are not removed from the target object. Proper induction of the moving object segmentation algorithms may help the video surveillance and forensic applications, object tracking applications, etc., to perform well up to the customer satisfaction. If not, categorization or evaluation of moving object position will produce erroneous results.

Sanin et al. [1] classified the moving shadow detection methods under various categories. Readily available cues which make a distinction between the target object, its shadow and the place where it is cast (background) are discussed to get a clear understanding of the shadow detection concept.

Sanin et al. [1] discussed few shadow detection algorithms which consider the physical and geometrical properties of the shadows to identify them. They are categorized under model-based techniques. Many of the shadow detection algorithms consider the colour-based methods [2,3,4,5,6,7,8] and texture-based [9,10,11,12] methods to differentiate shadows from the foreground objects. They are categorized under property-based techniques.

The colour-based methods use the following facts about the shadow and background. The red-green-blue (RGB) values, hue and saturation in hue-saturation-value (HSV) colour space and grey level of a shadow are inferior to the background in the corresponding pixel [12, 13]. The variation between those values of shadow and background show a gradual growth between adjacent pixels [14].

The shadow and the background have similar texture. Texture-rich object has a texture-less shadow [12, 14]. The entropy value or the Gaussian filter derivatives are helpful in extracting the texture of the image segments [15]. There is a difference in the illumination of shadow and background which also helps to differentiate shadow pixels. Lesser boundary when compared to background and not many interior edges when compared to objects are used to identify shadow pixels. But still, the shadows remain attached to their objects [16]. Even though object and shadow have the same movement, their position separates them [17]. Skewness is a variant statistical feature with respect to shadow and non-shadow areas for locating shadows provided a proper edge detection algorithm is used [15]. An invariant feature across shadow boundary is the image gradient.

For majority of the moving shadow detection methods, chromatic analysis is the first step. The hue to intensity ratio [18] is used to categorize shadow and non-shadow pixels. Also, c1c2c3 colour space is used to distinguish shadow areas by introducing a variation in the colour space transformation. But still, if the RGB values remain the same, the variation in the transformation formula will not show any difference. Colour space-based methods [19] such as HSI, HSV, YIQ and YCbCr analyse the intensity and colour properties of shadows in aerial images and calculate a hue to intensity ratio for each pixel. The shadow regions are extracted based on a threshold, which is not clearly classifying the dark blue and dark green surfaces. To improve the shadow detection results, a consecutive thresholding [20] can be done. Based on the lighting conditions, an invariant colour model [21] which identifies the shadow can be used, but this may not work for all types of images. HSI colour space along with the colour attenuation relationship [22] is analysed to detect the shadow in colour aerial images.

The shadow properties discussed so far can be utilized in image and video forensics to expose the falsification of photos and videos by detecting the shadows using their geometric and shading information [23]. Those images and videos can then be properly annotated for future forensic analysis after the removal of fake content. In image and video forensics field, the detection of shadow regions decides whether they are really formed or manipulated. The inconsistency of the texture between the shadow and background regions reveal the tampered content from the images [24] since it is known that the shadow does not change the texture of the background [12, 14]. Farid [25] gave three geometric techniques for detecting the traces of digital manipulation: vanishing points, reflections and shadows. The author reported with both real-time image and video forgery by analysing the shadows in them. This supports the need of detection of shadow in image and video forensics by which we were motivated to propose a moving shadow detection method. All the methods discussed so far dealt the shadow detection or removal process in spatial domain techniques that suffer from inaccurate segmentation due to the non-removal of noise or failure to detect new appearance automatically. Only few methods are available in wavelet domain that rectifies the problems in spatial domain techniques for a better identification of the moving cast shadows.

Guan [26] explored the properties of the HSV colour model for shadow detection and removal using dyadic multi-scale wavelet transform. The standard deviation of the wavelet coefficients from the value component helps to identify the shadow pixels from the foreground pixels. In a similar fashion, Khare et al. [27] used relative standard deviation of the wavelet coefficients to separate the shadows from the foreground pixels. Since the wavelet transforms decompose the input into high- and low-frequency values, the threshold values get updated automatically. Combining the saturation component with the value component, the threshold identifies the region of the shadows to be removed. The algorithms discussed so far still have the deficiency of the accurate shadow detection and removal.

In the proposed approach, we have also used the HSV colour space as it corresponds closely to the human perception of colour and separates chromaticity and luminosity easily. We also used variant statistical parameter “skewness” of the wavelet coefficients to detect and remove the shadows.

In the rest of the paper, Section 2 describes stationary wavelet transform (SWT); Sections 3 and 4 explains the threshold selection and the proposed method respectively. Section 5 explains the experimental results, and Section 6 gives the comparisons of the proposed method and other existing and state-of-art-methods [1, 7, 12, 26, 27]. Finally, discussions and conclusions of the work are given in Sections 7 and 8.

Stationary wavelet transform (SWT)

In this section, we have briefly discussed SWT and the reasons for using SWT for detection and removal of shadow from moving objects. Guan [26], Khare et al. [27] used discrete wavelet transform (DWT) that lacks in phase information and translation invariance and that creates problems in reconstruction of the image. So, we propose a method which uses SWT.

Fourier transform (FT) analyses a signal by decomposing it into constituent sinusoids of different frequencies. It has a major drawback of losing the temporal information. But cooperation of short-time Fourier transform (STFT) with a predefined window helps to retrieve the temporal information along with their frequencies.

DWT has an advantage over the Fourier transform in terms of localisation in frequency domain as well as in spatial domain. The DWT can be applied on a discrete signal containing N samples. The signal is decomposed into low-frequency band (L) using low-pass filter and high-frequency band (H) using high-pass filter. Each band is sub-sampled by a factor of two. In the case of 2D signal (image), each row of an image is filtered by a low-pass filter l[m] and high-pass filter h[m]. But the decimation operation of DWT makes it a shifted version of signal but not equivalent to the shift in DWT of signal.

SWT solves this problem of shift invariance [28]. SWT differs from conventional DWT in terms of decimation and shift invariance, which makes it feasible for change detection, pattern recognition and feature extraction. In SWT, the input signal is convolved with low l[m]- and high h[m]- pass filter in a similar manner as in DWT, but no decimation is performed to obtain wavelet coefficients of different sub-bands. As there is no decimation involved in SWT, the number of coefficients is twice that of the samples in the input signal that helps for a better reconstruction of the given image.

Threshold selection

An optimal threshold has to be determined for the accurate shadow detection process. Guan [26] used standard deviation (σ) as the threshold, Khare et al. [27] used relative standard deviation (σ/μ) as the threshold and we have chosen skewness as the threshold in SWT domain. For a sample of n values, skewness (Skew) parameter is defined as:

$$ \mathrm{Skewness}=E\left[{\left(\frac{x-\mu }{\sigma}\right)}^3\right] $$

(1)

where x is the one of the sample value, μ is the sample mean and σ is the standard deviation of the n values. Skewness is a measure of the degree of asymmetry of a distribution. The skewness value can be positive, negative or even undefined [29]. A negative skewness indicates that the tail on the left side of the probability density function is longer than the right side and the bulk of the values lie to the right of the mean. A positive skewness indicates that the tail on the right side is longer than the left side and the bulk of the values lie to the left of the mean as shown in Fig. 1.

Fig. 1

a Negative skewness—the left tail is longer and the mass of the distribution is concentrated on the right of the figure. b Positive skewness—the right tail is longer and the mass of the distribution is concentrated on the left of the figure

The motivation for selecting skewness as one of the distinguishing features to represent shadow is it always shows a variation from the skewness of the creating object and the casted background [30]. For an image having homogeneous reflective surface, the skewness of the luminance histogram and sub-band histograms are correlated with the shadows regions present in it [29]. In Eq. 1, μ is the average luminance of the object and σ is the standard deviation of the luminance of the object along with its neighbours in wavelet domain. The acceleration of (x − μ) with the cubic value enhances the edges around the object present in a scene; the summation and normalization make the edges of shadows blurry since they are soft and fasten the process of shadow detection.

So, the skewness of the wavelet coefficients has the boundary information of the objects which helps us to clearly segregate them from the shadows. Also, the existing thresholds may not show any differentiation of either as object or shadow for those pixels whose values of saturation and hue are undefined. The mean μ and standard deviation σ of such patterns as shown in Fig. 2 are identical leading to misclassification. They differ only in the sign of skewness because of the cubic acceleration and normalization of the luminance value of the object.

Fig. 2

a Example Image1 having similar patterns. b Histogram of images in a. c Example Image2 having similar patterns. d Histogram of images in c

Therefore, skewness is selected as a stable threshold in our work for classification of moving shadow pixels from the objects. Though the thresholds σ and (σ/μ) are able to detect and remove shadow, their performance degrades when the foreground object and the background share the same or dark colour.

In the proposed method, we have applied the threshold on stationary wavelet coefficients of value component of HSV to detect moving object with shadow. For removal of shadow, we have applied logical AND on thresholded stationary wavelet coefficients of value component and those of saturation component. Instead of manual calibration of threshold which needs some predefined parameters, we propose a variant statistical parameter, i.e. skewness as a new threshold for detection and removal of shadow in SWT domain. Guan [26] and Khare et al. [27] applied the thresholds to the discrete wavelet coefficients (DWT) that losses the phase information. The loss of phase information in DWT and the difficulty faced when there are similar patterns of input frames are rectified by applying SWT and skewness as a threshold respectively.

Proposed algorithm

The proposed approach considers a reference frame as the background model from the video sequence and the consecutive frames are processed one by one to detect the foreground and shadow pixels. The reference frame and the current frame are converted from RGB colour space to HSV colour space. The absolute difference of the two frames is taken with respect to hue, saturation and value component. They are represented as ΔH, ΔV and ΔS respectively. The stationary wavelet transform is applied to the absolute difference components of value (ΔV) and saturation (ΔS) to get the wavelet coefficients denoted as W_ΔV and W_ΔS. Also, the variant statistical parameter skewness is calculated for the wavelet coefficients for the value and saturation component denoted as (Skew)_WΔV and (Skew)_WΔS respectively. They act as an automatic threshold to classify the moving pixels as foreground or shadow pixels.

The pixels having a greater value than the automatic threshold “variant statistical value − skewness” of the value component are categorized as the moving pixels which include the foreground and shadow pixels. Equation 2 represents this situation.

$$ M=\left\{\begin{array}{l}1,\mathrm{if}\ {W}_{\varDelta V}\ge {\left(\mathrm{Skew}\right)}_{W_{\varDelta V}}\\ {}0,\mathrm{otherwise}\end{array}\right. $$

(2)

Also, to remove the shadow pixels, we consider the automatic threshold “variant statistical value − skewness” of the saturation component. Equation 3 represents this situation clearly.

$$ M=\left\{\begin{array}{l}1,\mathrm{if}\ {W}_{\varDelta V}\ge {\left(\mathrm{Skew}\right)}_{W_{\varDelta V}}\varLambda {W}_{\varDelta S}\ge {\left(\mathrm{Skew}\right)}_{W_{\varDelta S}}\\ {}0,\mathrm{otherwise}\end{array}\right. $$

(3)

Finally, the reconstruction of the shadow-detected image and shadow-removed image is done by the inverse wavelet transform. Binary closing morphological operations are applied to smoothen the reconstructed images.

Experimental results

To make our performance effective and systematic, an extensive result on several well-known benchmarks for which ground truth data was available is presented. The chosen benchmarks consist of indoor and outdoor scenes from UCSD CVRR Laboratory, CAVIAR Test Scenarios and Institute for Infocomm Research (I2R). A regional language movie’s song of our country named “Sarvam” is also used to test the proposed method with the other existing methods. Specifically, Intelligent Room, Hallway and CAVIAR are typical indoor environments while Highway I is an outdoor scenario of roadways. In addition, comparison with several existing methods to prove the superiority from the aspects of quality and quantity of our method, including Sanin et al. [12], Cucchiara et al. [7], Guan et al. [26] methods is done. The results are shown in Fig. 3. The first image (a) displays the original frames and (b) gives the ground truths of the corresponding frames in (a). Then the figure (c), (d) and (e) presents the results of the various methods [7, 12, 26] and finally (f) gives a visualization of the results of the proposed method.

Fig. 3

Visualized comparison results of the proposed method with others. a Input frame from Highway I, Intelligent Room, and Hallway. b Ground truths of the respective frames from video sequence. c Results of Sanin et al. [12]. d Results of Cucchiara et al. [7]. e Results of Guan et al. [26]. f Results of the proposed method

Performance measures

The performance of the proposed method against previously used thresholds is discussed. Figures 4 and 5 shows the vertical, horizontal and diagonal wavelet coefficients of the absolute difference of the reference frame and current frame of Highway I video sequence (frame no: 100, outdoor) and Intelligent Room video sequence (frame no: 130, indoor) plotted against their frequency of occurrence respectively. The shadow regions have lower values than the object region regions [12, 13, 29]. In both the figures, the original shadow regions are highlighted with the grey line and arrow mark indicates the end of the shadow points. The blue square indicates the detected shadow pixels using our proposed method—skewness as threshold. The black and red squares indicate the detected shadow pixels using standard deviation [26] (σ) and relative standard deviation [27] (σ/μ). The plots clearly indicate that our proposed method detects perfectly the shadow pixels than the previously used thresholds.

Fig. 4

Plot of a vertical, b horizontal and c diagonal wavelet coefficients of frame no. 100 from Highway I video sequence: The grey colour line shows the original shadow pixels, and the arrow mark in grey colour indicates the end of the shadow points. The blue colour square indicates the detected shadow pixels using skewness, and the arrow mark in blue colour indicates the end of the shadow points. The black and red squares indicate the detected shadow pixels using standard deviation [26] (σ) and relative standard deviation [27] (σ/μ)

Fig. 5

Plot of a vertical, b horizontal and c diagonal wavelet coefficients of frame no. 130 from Intelligent Room video sequence: The grey colour line shows the original shadow pixels, and the arrow mark in grey colour indicates the end of the shadow points. The blue colour square indicates the detected shadow pixels using skewness, and the arrow mark in blue colour indicates the end of the shadow points. The black and red squares indicate the detected shadow pixels using standard deviation [26] (σ) and relative standard deviation [27] (σ/μ)

In order to analyse the proposed method objectively and quantitatively, the shadow detection rate η and shadow discrimination rate ε [20] are considered. The effectiveness of the proposed method is shown along with the other methods in Table 1 giving the shadow detection rate (η) and Table 2 giving the shadow discrimination rate (ε). Also, the computation time needed to process for a single frame in seconds of the proposed method along with the other methods is given in Table 3.

Table 1 Shadow detection rate (η) for representative video sequences

Table 2 Shadow discrimination rate (ε) for representative video sequences

Table 3 Total computation time of various methods

The shadow discrimination rate of our method for various videos compared with the thresholds used by Guan [26] and Khare et al. [27] is shown in Fig. 6. The visualized comparison of the proposed method with the state-of-art-methods is shown in Fig. 7. In all the sub-figures, the first image shows the input frame, the second shows the corresponding ground truth, the third shows the proposed method results and the fourth and fifth show the results of the state-of-art-methods [26, 27] respectively. The video sequences used are Highway sequence, regional language movie song (Sarvam) and Lobby Hall. All the performance measures show that the proposed method outperforms than the existing and state-of-art-methods in the literature.

Fig. 6

Shadow discrimination rate of various videos using skewness (proposed method), standard deviation [26] and relative standard deviation [27]. a Shadow discrimination rate in Highway I sequence. b Shadow discrimination rate in Shopping Mall. c Shadow discrimination rate in “Sarvam” regional movie song. d Shadow discrimination rate in Lobby Hall

Fig. 7

Visualized comparison results of the proposed method with the state-of-art-methods. The first frame is the input, second is the ground truth, third is the proposed method result, fourth is the Guan [26] method result and fifth is the Khare et al. [27] method result. a Highway I video sequence and Lobby Hall video sequence. b “Sarvam” regional movie song—long and short shadows

Discussions

The existing moving shadow detection methods in wavelet domain using DWT lack in translation invariance and loss phase information during reconstruction. To address these issues, we have used stationary wavelet transform. The novelty of our method lies in the usage of a variant statistical parameter skewness that shows a difference for any kind of pattern. Previously used thresholds like standard deviation (σ) and relative standard deviation (σ/μ) will remain the same for undefined values of saturation and hue leading to misclassification. Whenever there is equality in RGB values, the conversion of such pixels to HSV colour space produces an undefined saturation and hue value [31]. The proposed threshold has been tested with various benchmark videos and compared with the existing thresholds to prove its usability. The limitation of the proposed method is the redundancy of the wavelet coefficients which can be reduced by incorporating global feature preservation techniques like principal component analysis (PCA) or linear discriminant analysis (LDA) [32] with the wavelet transform as in many object and face detection methods.

Conclusions

In this article, a stationary wavelet transform (SWT)-based shadow detection method is proposed. A new threshold based on the variant statistical feature, calculated with the help of the wavelet coefficients, is used to classify the moving objects and shadows. When compared with the discrete wavelet transform (DWT), the shift invariance and multi-resolution property of SWT supports the reconstruction of the image from the sub-bands without the loss of phase information. Also, the determined threshold from the wavelet coefficients helps in a unique discrimination of the shadows and objects. The promising results of the proposed method highlight the advantages of both the SWT and the variant statistical threshold. The performance of the method degrades with high speed of the object and non-stationary background; therefore, this work can be extended for the non-stationary background for the future work. Also, there is a redundancy of the wavelet coefficients which can be reduced by incorporating PCA or LDA with the wavelet transform in the future.