Robust multiple-vehicle tracking via adaptive integration of multiple visual features

Abstract

This article presents a robust approach to tracking multiple vehicles with integration of multiple visual features. The observation is modeled by democratic integration strategies according to the reliability of the information in the current multi-visual features to adjust their weights. The appearance model is also embedded in a particle filter (PF) tracking framework. Furthermore, we propose a new model updating algorithm based on the PF. In order to avoid incorrect results caused by "model drift" introduced into the observation model, model updating should only be controlled in a reliable manner, and the rate of updating is based on reliability. This article also presents the experiments using a real video sequence to verify the proposed method.

1. Introduction

With the rapid process of urbanization, the concept of developing a "smart city" has gained prominence. As an important part of this trend, Intelligent Transportation Systems (ITS) will be critical for effective management of urban traffic. Vehicle tracking under different traffic scenarios is one of the key issues in ITS. Vehicle motion parameters, such as location, velocity, orientation, and acceleration, can be obtained to further recognize and understand vehicular behavior. However, the challenges of robust tracking come from uncertain and dynamic conditions of speed, occlusion, deformation, illumination variation, background clutter, real-time restriction, etc. In order to handle these problems, great effort has been made to devise robust tracking algorithms. In general, the following three key problems should be solved in tracking: (1) an effective framework to locate vehicles in motion; (2) modeling observation of vehicles; and (3) reliable updating of vehicle models.

An ideal locating framework should be able to predict and update the motion state and observation model of an object, and even track multiple objects under various conditions. Probabilistic tracking, which is a process utilizing posterior probability density of target states in a Bayesian framework, is a highly effective approach. Kalman Filter (KF) [1], Hidden Markov Model [2], and Particle Filter (PF) [3–7] techniques have been used in different tracking applications. PF recursively constructs the posterior probability distribution function of the state space using a Monte Carlo integral. A PF-based tracking algorithm has the added advantage that any visual feature can be used for the observation model. Meanwhile, it has the ability to integrate multiple visual features.

The observation model depicts similarity measurements between a template region and the candidate region of a vehicle, and plays an even critical role in visual tracking associated with PF. Many visual features can be selected for vehicle observation modeling, including color [8], edge [3, 9], feature descriptors [10], color-spatial features [11], wavelets [12], etc. For instance, it is sensitive to the variation of a given illumination environment using a color-based method when the illumination varies. An edge-based method can avoid disturbances caused by illumination variances, but it is either time-consuming or limited to a single shape model, and presents difficulties in achieving accurate real-time tracking. The algorithms based on these methods have achieved good tracking performance, but relying on a single visual feature is often inadequate and unstable in some complex tracking scenarios. Various complementary features can be combined to derive more robust tracking results. It is our interest to employ multiple visual features under a robust tracking framework. The advantage of this kind of method is that vehicle information can complement other visual features. When one visual feature fails, others can be used to maintain tracking. However, the difficulty is how to design a good strategy to integrate visual features reliably. Methods proposed in [13–15] are based on the fixed weight integration. If one visual feature with a fixed weight changes markedly, the observation model after integration will be unreliable. This leads to tracking drift away from the true location, and even tracking failure. Spengler and Schiele [16] proposed an algorithm with an adaptive integration strategy using an EM algorithm to adjust the weight of each visual feature online. However, this algorithm is based on a matching algorithm under a local search. When partial or complete occlusion occurs, tracking performance will seriously decline.

How to update a vehicle model to deal with appearance changes during tracking is very important for the robustness of an algorithm. Many algorithms assume the appearance of an object as being invariable during tracking. The appearance model of an object is usually extracted in the first frame image, and then the most probable location of the object is found in the following frame. This assumption is reasonable for short-term tracking. However, for long-term tracking, appearance changes in an object are inevitable. Jepson et al. [17] proposed an adaptive texture-based model named WSL. This model consists of three components to describe object appearance changes, where W describes the rapid changes in object appearance, S is used to characterize the stability in an object whose change in appearance is slow, and L is defined to depict abnormal variations in object appearance. A Gaussian Mixture Model (GMM) is constructed by these components, and the parameters of GMM are updated through the EM algorithm online. The proposed model has strong robustness to changes in both illumination and shape. However, it fails to track when an object is occluded by one with the same visual features for even a moment. The reason is that the same information presented by the occluding object is added into the model of the occluded object during updating. After occlusion, the appearance model cannot correctly reflect the object. This phenomenon is called "model drift". A fixed number of pre-learned exemplars are used as templates by Toyama and Blake [18]. The problem with this method is that only a fixed number of examples can be used as templates to model the appearance of an object. Yang and Wu [19] introduced a closed-form solution by "discriminative training" of a generative model to alleviate model drift. They optimize a convex combination of the generative and discriminative log likelihood functions to obtain the model. Avidan [20] treated tracking as a classification problem. The ensemble of weak classifiers is combined into a strong classifier using AdaBoost. The strong classifier is then used to label pixels in the next frame as either belonging to the object or the background, creating a confidence map. The new position of the object is found in the peak of the map by using a mean shift. However, only the color of each pixel is used to classify, and the classifier needs to update background information around the object. When two objects of a similar color are very near each other, tracking will fail.

This article proposes a robust tracking approach with an adaptive integration of multiple visual features for vehicles. A color histogram and an edge orientation histogram (EOH) are selected as visual features to model the observation of the vehicle and integrated by a democratic integration strategy proposed by Triesch and Malsburg [21]. It is suitable for dynamic scenes due to the adaptive adjustment of the weight of each visual feature with its reliability in the current frame. However, deterministic integration is vulnerable to occlusion for a few frames because the present iteration is initialized according to the previous one. Thus, the observation model is embedded in the PF tracking framework. In order to improve the robustness of object representation, spatial information is incorporated into the observation model by dividing the object to be tracked into a number of fragments. We then analyze the reason for model drift during the model update process, and propose a new model updating method under a PF. In order to avoid errors caused by model drift, the updating process should only be implemented in a reliable manner, and the rate of updating can be controlled according to this reliability. The posterior probability density function of distribution of state vector and similarity between the candidate and reference observation of an object are used to define the valid measurement of the reliability to model updating during tracking. Experimental results in real traffic surveillance video sequences show that our approach outperforms others in vehicle tracking under complex conditions.

The remainder of this article is organized as follows. The preprocessing before tracking is described in Section 2. The state model for multiple vehicles is built in Section 3. The adaptive and robust observation model is presented in Section 4. The reliable model updating strategy is introduced in Section 5. The PF-based tracking algorithm is completely summarized in Section 6. The experimental results are given in Section 7, and finally, the conclusion is given in Section 8.

2. Preprocessing before tracking

2.1. Background modeling

In surveillance video, it can be seen that the background changes along with illumination, weather, and other conditions. So, we must process the surveillance video scene first. Our previous research has given a self-adaptive modeling for real-time background modeling with lower computational-complexity and higher accuracy [22].

In the (n + 1)th frame, the gray value of point p can be described as follows:

$$G\left(n+1,p\right)=G\left(n,p\right)+L\left(n,p\right)+{\text{noise}}^1\left(n,p\right)$$

(1)

where G(n, p) is the pixel p's gray value in the n th frame, L(n, p) is the model to describe the change of illumination with the change of time, and noise¹(n, p) is the Gaussian noise taking zero as the center. The gray value of pixel p in the input image can be described as:

$$I\left(n,p\right)=G\left(n,p\right)+{\text{noise}}^2\left(n,p\right)$$

(2)

where noise²(n, p) is Gaussian noise taking zero as the center. A comparison between (1) and (2) can easily indicate that

$$I\left(n+1,p\right)=G\left(n,p\right)+\omega \left(n+1,p\right)$$

(3)

where ω(n+1, p) = L(n, p)+noise¹(n, p)+noise²(n+1, p). ω(n, p) is a Gaussian distribution. We use a mean value to represent m(n, p) and s(n, p), respectively, and use a variable to represent ω(n, p). In traffic surveillance video, illumination and noise distribution change little in a triangular region. Therefore, m(n, p) and s(n, p) are independent of the position of pixel p. Then, a histogram can be derived from the difference between {I(n+1, p)} and {G(n, p)} in a triangular region. From this histogram, the mean value of m(n) and s(n) can be estimated by a self-adaptive filter based on a recursive least square method.

Figure 1 gives the background in four video surveillance scenes, where the regions masked in red are not our monitoring driveways.

Figure 1

The results of background estimation. (a) Straight driveway. (b) Turning driveway. (c) Straight driveway in the evening. (d) Turning driveway late at night.

2.2. Detecting the ROI of a vehicle

The aim is to track vehicle targets in real time and as robustly as possible. The first step is to detect the vehicle targets automatically. The initial detection and the tracking regions are set in the field of vision, respectively, as shown in Figure 2.

Figure 2

Initial detection region and tracking region.

Then, due to its success in vehicle detection of a real surveillance scene, a fast-constrained Delaunay triangulation (FCDT) algorithm [23] is used as follows:

• (1) Extract contour information with a Canny filter

• (2) Extract lines from image contours with a Hough transformer

• (3) Achieve a set of corners at both ends of the lines

• (4) Initialize the CDT based on all constrained edges

• (5) Insert all corner points in turn to reconstruct the CDT

• (6) Extract corner density, horizontal straight line density, vertical straight line density, triangle density, and average intensity of a vehicle region to construct the feature vector

• (7) Put the feature vector into SVM to determine the ROI of the vehicle target

The detection results are shown in Figure 3. The blue lines construct the Delaunay triangulation net. The red rectangle bounding box is the ROI of the vehicle.

Figure 3

Results of detecting the ROI of the vehicle targets using FCDT.

3. State modeling for multiple vehicles

According to the characteristics of vehicle motion, we build the prediction equation of the motion state using the second-order linear regression. The state model is built by a centroid and the area of a rectangular bounding box:

$$\mathbf{S}={\left(x,y,s\right)}^T$$

(4)

where C = (x, y) and s are the centroid and area bounding box, respectively.

The current state S _t is predicted by three parts: the previous state S_t-1, the last state displacement S_t-1- S_t-2, and a zero-mean Gaussian stochastic component ω _t with covariance matrix ∑:

$$\left\{\begin{array}{c}{\mathbf{S}}_t-{\mathbf{S}}_{t-1}={\mathbf{S}}_{t-1}-{\mathbf{S}}_{t-2}+{\omega }_t\hfill \\ {\omega }_k~N\left(0,\Sigma \right)\hfill \end{array}\right.$$

(5)

Hence, the model can be denoted as a Gaussian distribution as follows:

$$p\left({\mathbf{S}}_t|{\mathbf{S}}_{t-1},{\mathbf{S}}_{t-2},...,{\mathbf{S}}_1\right)~N\left({\mathbf{S}}_t;2{\mathbf{S}}_{t-1}-{\mathbf{S}}_{t-2},\Sigma \right)$$

(6)

For multiple vehicles, we suppose that vehicles are independent from each other and there are M vehicles in a video scene. So, the model is regarded as

$$p\left({\mathbf{S}}_t\left(m\right)|{\mathbf{S}}_{t-1}\left(m\right),...,{\mathbf{S}}_1\left(m\right)\right)~N\left({\mathbf{S}}_t\left(m\right);2{\mathbf{S}}_{t-1}\left(m\right)-{\mathbf{S}}_{t-2}\left(m\right),\Sigma \left(m\right)\right)$$

(7)

where S _t (m) is the state vector of the m th vehicle in the k th frame.

4. Adaptive integration-based observation model

The observation models encode the visual information of a vehicle's appearance. Since a single visual feature does not work in all cases, we utilize the Hue-Saturation-Value (HSV) color histogram to capture the color information of a vehicle, and an EOH to encode shape information, indicating that O = {O _t ; t ∈ N} is denoted as the vehicle's observation model.

4.1. Color features

We obtain the color information of a vehicle by a two-part color histogram based on the HSV color space. We use the HSV color histogram because it decouples the intensity from Hue and Saturation, and thus it is less sensitive to illumination effects than a histogram from the RGB color space. The exploitation of the spatial layout of the color is also crucial due to the fact that different vehicles usually have different colors.

In the non-Gaussian state space, state model S is assumed to be a hidden Markov process, with an initial distribution p(S₀) and a transfer distribution p(S _t |S_t-1). A color histogram-based observation model ${\mathbf{O}}_t^c$ is obtained through the marginal distribution $p\left({\mathbf{O}}_t^c|{\mathbf{S}}_t\right)$. Our color observation model is composed of a 2D histogram based on Hue and Saturation and a 1D histogram based on value. Both histograms are normalized such that all bins sum to one. We assign the same number of bins for each color component, i.e., N_h = N_s = N_v = 10, resulting in an N = N_h × N_s+N_v = 110-dimensional HSV histogram.

Assume that R(S _t ) is the candidate region of vehicle at time t, the kernel density estimation of color distribution is

$$k\left(n;{\mathbf{S}}_t\right)=\kappa \sum _{\mathbf{d}\in R\left({\mathbf{S}}_t\right)}\delta \left[b_t\left(\mathbf{d}\right)-n\right]$$

(8)

where b _t (d) ∈ {1, ..., N} is the index of color bins of a pixel at position d; δ[·] is the delta function; κ is a normalized factor to subject to ${\sum }_{n=1}^{N}k\left(n;{\mathbf{S}}_t\right)=1$; position d is a pixel in the candidate region R(S _t ). Suppose that ${\mathbf{K}}^{*}\triangleq {\left\{k^{*}\left(n;{\mathbf{S}}_0\right)\right\}}_{n=1,...,N}$ is the reference template and $\mathbf{K}\left({\mathbf{S}}_t\right)\triangleq {\left\{k\left(n;{\mathbf{S}}_t\right)\right\}}_{n=1,...,N}$ is the candidate model, the similarity measurement is defined based on Bhattacharyya coefficient:

$${\rho }_c\left({\mathbf{K}}^{*},\mathbf{K}\left({\mathbf{S}}_t\right)\right)={\left(1-\sum _{n=1}^N\sqrt{k^{*}\left(n;{\mathbf{S}}_0\right)k\left(n;{\mathbf{S}}_t\right)}\right)}^{\frac{1}{2}}$$

(9)

Therefore, the color-based observation model is denoted as follows:

$$p\left({\mathbf{O}}_t^c|{\mathbf{S}}_t\right)\propto e^{-{\lambda }_c{\rho }_c^2\left({\mathbf{K}}^{*},\mathbf{K}\left({\mathbf{S}}_t\right)\right)}$$

(10)

where λ _c is a factor determined by the variation of color Gaussian distribution. Figure 4 shows the HSV color histograms of two vehicles.

Figure 4

HSV color histograms of vehicles.

4.2. Shape features

We apply an EOH to describe shape information of a vehicle. In order to detect the edge, the color image must be converted to grayscale at first. The gradient at pixel (x, y) in the image I can be computed by the Sobel operator mask:

$$G_h\left(x,y\right)={\text{Sobel}}_h\times I\left(x,y\right)$$

(11)

$$G_v\left(x,y\right)={\text{Sobel}}_v\times I\left(x,y\right)$$

(12)

where Sobel_h and Sobel_v are horizontal and vertical masks of the Sobel operator. The strength of an edge is computed as follows:

$$G\left(x,y\right)=\sqrt{G_h^2\left(x,y\right)+G_v^2\left(x,y\right)}$$

(13)

In order to suppress noise we threshold G(x, y) such that

$$G^{\prime }\left(x,y\right)=\left\{\begin{array}{cc}\hfill G\left(x,y\right)\hfill & \hfill \text{if}G\left(x,y\right)\ge T\hfill \\ \hfill 0\hfill & \text{otherwise}\hfill \end{array}\right.$$

(14)

where the value of T was suggested to be set between 80 and 110 in [24]. The orientation of the edge is

$$\theta \left(x,y\right)=\left\{\begin{array}{cc}\hfill \text{arctan}\left(\frac{G_v\left(x,y\right)}{G_h\left(x,y\right)}\right)\hfill & \hfill \text{if}{G}_h\left(x,y\right)\ne 0\hfill \\ \hfill \frac{\pi }{2}\hfill & \hfill \text{if}{G}_h\left(x,y\right)=0\hfill \end{array}\right.$$

(15)

Then, the edges are divided into K bins. The value of the k th bin is denoted as

$${\psi }_k\left(x,y\right)=\left\{\begin{array}{cc}\hfill G^{\prime }\left(x,y\right)\hfill & \hfill \text{if}\theta \left(x,y\right)\in {\mathrm{bin}}_k\hfill \\ \hfill 0\hfill & \text{otherwise}\hfill \end{array}\right.$$

(16)

Figure 5 shows the EOHs of the vehicles in Figure 4.

Figure 5

EOHs of vehicles.

Levi and Weiss [25] introduced three extended features based on EOH. However, direct use of these features for vehicles has some limitations. First, the values of both the ratio of edge strength of any two orientations and the dominant orientation feature have a large range, but the values of the discriminative features distribute into a relatively small scope. It cannot reflect characteristics of the majority of edges. Second, the orientations of symmetrical edges should be complementary instead of equal, because of the symmetry of two regions. Hence, we provide an enhanced feature set. These features are used to improve robustness in Section 4.3.

• (1) Edge Strength Features in any two orientations ϕ:

  $${\varphi }_{i,j}\left(q^l\right)=arctan\left(\frac{E_i\left(q^l\right)+\epsilon }{E_j\left(q^l\right)+\epsilon }\right)$$

  (17)

• (2) Dominant orientation features φ:

  $${\phi }_i\left(q^l\right)=arctan\left(\frac{E_i\left(q^l\right)+\epsilon }{\sum _{j\in K}{E}_j\left(q^l\right)+\epsilon }\right)$$

  (18)

• (3) Symmetry features ζ:

  $${\zeta }_1\left(R_1,R_2\right)=arctan\left(\frac{E_i\left(R_1\right)-E_{\pi \left(i\right)}\left(R_2\right)+\epsilon }{\sum _{j\in K}\left(E_j\left(R_1\right)+E_j\left(R_2\right)\right)+\epsilon }\right)$$

  (19)
  $${\zeta }_2\left(R_1,R_2\right)=arctan\left(\frac{E_i\left(R_1\right)+E_{\pi \left(i\right)}\left(R_2\right)+\epsilon }{\sum _{j\in K}\left(E_j\left(R_1\right)+E_j\left(R_2\right)\right)+\epsilon }\right)$$

  (20)

where R₁ and R₂ are regions of the same size and are positioned at opposite sides of the symmetry axes. π(i) = (M _ζ - i)%M _ζ ; M _ζ is interval numbers of [0, π] and M _ζ = 6 in the experiment.

Suppose that ${\mathbf{E}}^{*}\triangleq {\left\{e^{*}\left(n;{\mathbf{S}}_0\right)\right\}}_{n=1,...,K}$ is the reference template and $\mathbf{E}\left({\mathbf{S}}_t\right)\triangleq {\left\{e\left(n;{\mathbf{S}}_t\right)\right\}}_{n=1,...,K}$ is the candidate model of EOH, the similarity measurement is defined as follows:

$${\rho }_e\left({\mathbf{E}}^{*},\mathbf{E}\left({\mathbf{S}}_t\right)\right)={\left(\sum _{n=1}^K\left(e^{*}\left(n;{\mathbf{S}}_0\right)e\left(n;{\mathbf{S}}_t\right)\right)\right)}^{\frac{1}{2}}$$

(21)

Therefore, the color-based observation model is denoted as follows:

$$p\left({\mathbf{O}}_t^e|{\mathbf{S}}_t\right)\propto e^{-{\lambda }_e{\rho }_e^2\left({\mathbf{E}}^{*},\mathbf{E}\left({\mathbf{S}}_t\right)\right)}$$

(22)

where O^eis denoted as the observation model based on an EOH, and λ _e is a factor determined by the variation of the EOH distribution.

4.3. Improving robustness

Both visual features introduced above are based on histograms, while all spatial information is discarded. This may lead to false objects and local minima, and even tracking failure under occlusion. On the other hand, methods incorporating the spatial information are computationally intensive. Motivated by the approaches proposed in [26, 27], spatial information is incorporated into the observation model by dividing the vehicle to be tracked into a number of fragments.

The reference observation of a vehicle is represented by multiple fragments using multiple feature histograms {q^l}_{l = 1, ..., L}instead of one global histogram, where L is the number of fragments. Let the target candidate centered at position C be represented by {p^l(C)}_{l = 1, ..., L}, where p^l(C) is built in the same manner as the observation model. With this definition, we propose the similarity function as follows:

$$\rho \left(\mathbf{C}\right)={\lambda }^{\left(1\right)}{\rho }^{\left(1\right)}+\cdot \cdot \cdot +{\lambda }^{\left(L\right)}{\rho }^{\left(L\right)}=\sum _{l=1}^L{\lambda }^{\left(l\right)}{\rho }^{\left(l\right)}$$

(23)

where λ^(l)describes the important weight of each fragment and subjects to ${\sum }_{l=1}^L{\lambda }^{\left(l\right)}=1$. The similarity function of each fragment is calculated by similarity measurements of different features between p^l(C) and q^l. During tracking, each fragment should play a role at different levels due to occlusions or other kinds of appearance changes. A higher value λ^(l)means that the tracking algorithm will refer more to the l th fragment. Conversely, a fragment with little weight will count less for the final tracking result. Here, we regard a fragment as being more important if it is more similar with the reference fragment, and at the same time less similar with the background:

$${\lambda }^{\left(l\right)}=\gamma {\lambda }_{\mathrm{fg}}^{\left(l\right)}+\left(1-\gamma \right){\lambda }_{\mathrm{bg}}^{\left(l\right)}$$

(24)

where γ tunes the proportion of ${\lambda }_{\mathrm{fg}}^{\left(l\right)}$ and ${\lambda }_{\mathrm{bg}}^{\left(l\right)}$, that we set it 0.8 in the following experiments. The background region for each fragment is selected as the neighborhood surrounding region with a double size excluding the fragment. Accordingly, the feature histogram of the background region is extracted. To measure the similarity more properly, we use the metric proposed by Nummiaro et al. [28]:

$${\lambda }_{\mathrm{fg}}^{\left(l\right)}=\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{{\left(d_{\mathrm{fg}}^{\left(l\right)}\right)}^2}{2{\sigma }^2}\right)$$

(25)

$${\lambda }_{\mathrm{bg}}^{\left(l\right)}=1-\frac{1}{\sqrt{2\pi }\sigma }exp\left(-\frac{{\left(d_{\mathrm{bg}}^{\left(l\right)}\right)}^2}{2{\sigma }^2}\right)$$

(26)

where d^(l)is the distance of two feature histograms.

Many suggested methods [29–31] divide an object into multiple non-overlapping fragments. Note that both the number of fragments and their delineation have an impact on tracking efficiency and accuracy. Although the robustness increases with the number of fragments, too many fragments mean an increased processing time for each frame. Since the computation required for each frame greatly depends on the size of each fragment, which also needs to be restricted. Further, selecting very small fragments will result in tracking drift, or some information about the vehicle being discarded. So, a trade-off is required. We prompt the use of some overlapping fragments. A set of non-overlapping horizontal fragments and a set of non-overlapping vertical fragments are overlapped. Horizontal and vertical fragments are obtained by the dominant orientation features and the symmetry feature introduced in Section 4.2, respectively. When the size of the fragment is less than 8 × 8, it will be discarded. The satisfactory results of fragmentation are shown in Figure 6.

Figure 6

The satisfactory results of fragmentation.

Hence, the color and EOH-based observation with fragmentation are denoted as follows:

$$p\left({\mathbf{O}}_t^c|{\mathbf{S}}_t\right)\propto e^{-{\lambda }_c\sum _{l=1}^L{\left({\rho }_c^{\left(l\right)}\left({\mathbf{K}}^{*},\mathbf{K}\left({\mathbf{S}}_t\right)\right)\right)}^2}$$

(27)

$$p\left({\mathbf{O}}_t^e|{\mathbf{S}}_t\right)\propto e^{-{\lambda }_e\sum _{l=1}^L{\left({\rho }_e^{\left(l\right)}\left({\mathbf{E}}^{*},\mathbf{E}\left({\mathbf{S}}_t\right)\right)\right)}^2}$$

(28)

4.4. Adaptive integration

We employ an adaptive integration of the multiple visual features mentioned above, i.e., democratic integration. This integration strategy changes each feature's weight adaptively, according to its reliability in the previous frame, and improves the performance robustness of the visual features.

The complete observation model is defined as

$$p\left(\mathbf{O}|\mathbf{S}\right)={\alpha }^{c}p\left({\mathbf{O}}^c|\mathbf{S}\right)+{\alpha }^{e}p\left({\mathbf{O}}^e|\mathbf{S}\right)$$

(29)

where α^c and α^e are the weights of color histogram and EOH features, respectively, and α^c +α^e = 1. The final state vector can be obtained by the maximum likelihood estimation:

$$\widehat{\mathbf{S}}=\underset{\mathbf{s}}{arg\; max}\left\{p\left(\mathbf{O}|\mathbf{S}\right)\right\}$$

(30)

In order to verify the consistency between results by integration of multiple visual features and by a single feature, a quality function ${\gamma }_t^f$ is introduced and normalized as follows:

$${\bar{\gamma }}_t^f=\frac{{\gamma }_t^f}{{\sum }_f{\gamma }_t^f}$$

(31)

where f is a sign to indicate the type of feature, i.e., color or EOH. In general, the change between two adjacent frames is small, so the weight of a feature can be predicted by

$$\tau \frac{{\alpha }_t^f-{\alpha }_{t-1}^f}{\Delta t}={\bar{\gamma }}_{t-1}^f-{\alpha }_{t-1}^f$$

(32)

where τ is a constant to determine the adaptive rate of change of weight; Δt is a continuous time interval between two frames. From Equation (32), the weight of feature whose current weight is less than the value of ${\bar{\gamma }}_{t-1}^f$ may be increased. That is to say that this strategy always increases the weight of a feature with a high reliability and reduces the weight with a low reliability.

In fact, ${\gamma }_t^f$ can be treated as the feedback of the tracking result ${\widehat{\mathbf{S}}}_t$. The weight of each visual feature is adaptively calculated by the normalized quality function in the previous frame. In order to define ${\gamma }_t^f$, we employ the probabilistic distribution map in [31]: p _f (x _i , t) ∝p _f (Z _i |M _{f, F} ). Z _i is the observation at pixel i; M _{f, F} is the foreground model of feature f, p _f (Z _i |M _{f, F} ) represents the observation likelihood of the pixel i given the foreground model M _{f, F} of feature f. The higher the pixel's value in p _f (x _i , t) is, the higher the likelihood of pixel i belongs to the foreground. Hence, ${\gamma }_t^f$ is defined as the ratio between the numbers of probabilistic pixels of foreground and background in the probabilistic distribution map:

$${\gamma }_t^f=\frac{\mathrm{Sum}\left(p_f\left({\mathbf{x}}_i,t\right),{\widehat{\mathbf{S}}}_t\right)}{\mathrm{Sum}\left(p_f\left({\mathbf{x}}_i,t\right),{\widehat{\mathbf{S}}\prime }_t-{\widehat{\mathbf{S}}}_t\right)}$$

(33)

where the background is defined as the area between the tracking box and a larger window ${\widehat{\mathbf{S}}\prime }_t$, which shares the same centroid of a bounding box. Sum(·,·) is the sum of probabilistic pixels in the window W:

$$\mathrm{Sum}\left(p\left({\mathbf{x}}_i\right),W\right)=\sum _{i}p\left({\mathbf{x}}_i\right),{\mathbf{x}}_i\in W$$

(34)

5. Model updating

Tracking is usually performed by searching for a location in the image that is similar to a given reference model. The updating of the observation model is implemented by the new appearance and a previous observation model O₁, ..., O _t to estimate to the observation model O_t+1in the next frame. Assume that the appearance of a vehicle remains the same during tracking, the observation model in the coming frame is

$${\mathbf{O}}_{t+1}={\mathbf{O}}_t$$

(35)

It is reasonable for short-term tracking under some conditions. However in reality, vehicles will change appearance due to a variety of factors, such as turning, scale, camera angle, etc. Therefore, this assumption will eventually lead to some errors where the observation model cannot correctly represent the actual appearance of the vehicle. In order to obtain the latest and real observation model of a vehicle, a simple model updating strategy is proposed where the observation model in the next frame is estimated by the state vector of tracking results from the previous frame:

$${\mathbf{O}}_{t+1}=p\left({\bar{\mathbf{S}}}_t\right)$$

(36)

where ${\bar{\mathbf{S}}}_t$ is the state of the vehicle at time t, and $p\left({\bar{\mathbf{S}}}_t\right)$ is the observation estimation covered by ${\bar{\mathbf{S}}}_t$.

This updating strategy can make the observation model of a vehicle respond to appearance changes, but it easily leads to model drift when the vehicle is occluded by other vehicles or tracking errors, or the rapid deviations from the ground-truth of vehicle observation present during the updating process. Thus, we have created an adaptive update method to maintain stability over observation changes:

$${\mathbf{O}}_{t+1}=\left(1-{\beta }_t\right){\mathbf{O}}_t+{\beta }_{t}p\left({\bar{\mathbf{S}}}_t\right)$$

(37)

where β _t is named as a forgetting factor, and it is used to minimize the impact on the observation model by specific frames and to control the speed of model updating. It is inevitable that some kinds of errors will be made during tracking. There exist two kinds of errors: errors caused by accumulation, and errors caused by object distortion. The former is caused by the accumulation of small errors from frequent updating; the latter is usually a fatal error which is induced by maintaining the same observation model during tracking. Therefore, the key problems are when to update the model and the rate of updating.

In a traffic scene, the changes of tracked vehicles usually fall into two categories: change of a vehicle's scale and changes in appearance. Therefore, we define two factors, η₁(t) and η₂(t), to determine the forgetting factor β _t at time t:

$${\beta }_t=k{\eta }_1\left(t\right){\eta }_2\left(t\right)$$

(38)

where k is a constant.

First, when the appearance of a vehicle is obviously changed by occlusion, illumination, etc., a significant difference appears between the reference observation and the candidate one. At this time, updating should be avoided. Thus, η₁(t) is defined using the similarity measurement between the candidate and the reference observation:

$${\eta }_1\left(t\right)=\left\{\begin{array}{cc}\hfill \rho \left({\mathbf{O}}_t,{\bar{\mathbf{S}}}_t\right),\hfill & \hfill \rho \left({\mathbf{O}}_t,{\bar{\mathbf{S}}}_t\right)>T{h}_1\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\right.$$

(39)

where ρ(·,·) is the similarity measurement between O _t and ${\bar{\mathbf{S}}}_t$. Th₁ is empirical and is set to 0.8 in the experiments. The bounding box scale changes due to the vehicle's motion trajectory. We employ the bounding box scale recursion introduced by McKenna et al. [32]:

$${\mu }_{t+1}=c{\mu }_t+\left(1-c\right)s_{t+1}^{\prime }$$

(40)

$${\sigma }_{t+1}^2=c\left({\sigma }_t^2+{\left({\mu }_{t+1}-{\mu }_t\right)}^2\right)+\left(1-c\right){\left(s_{t+1}^{\prime }-{\mu }_{t+1}\right)}^2$$

(41)

where μ_t+1and ${\sigma }_{t+1}^2$ represent the new mean and the new variance of the recursive bounding box scale, respectively, and $s_{t+1}^{\prime }$ represents the newly detected bounding box scale. C is used to control the forgotten rate of the recursive bounding box scale of the vehicle. If c is large, the history of the bounding box scale will fade out slowly. This is good for a vehicle as a rigid object with a fixed shape, and the history of the bounding box scale will be kept through the large c. In the experiments, c is set to 0.9. Here, η₂(t) is defined according to the new mean and variance:

$${\eta }_2\left(t\right)=\left\{\begin{array}{cc}\hfill CV\left(t\right),\hfill & \hfill 0<CV\left(t\right)<T{h}_2\hfill \\ \hfill 0,\hfill & \hfill \text{otherwise}\hfill \end{array}\right.$$

(42)

where CV(t) = σ _t /μ _t is the dispersion coefficient. Th₁ is empirical and is set to 0.2 in the experiments.

6. Robust tracking under PF

According to the state and observation model, multi-vehicle tracking is performed by running multiple-independent PFs for every vehicle in the scene. Algorithm 1 summarizes the fully automatic multi-vehicle tracking algorithm.

Algorithm 1. Robust Tracking under PF

Input: {I _t }_{t = 1, ..., T};

Output: ${\left\{{\widehat{\mathbf{S}}}_t\left(m\right)\right\}}_{t=1,\dots ,T;m=1,\dots ,M}$;

• 1. Detect the ROI of vehicle;

• 2. Divide (0, 1] into N independent intervals, and N is the number of initial particles, i.e. $\left(0,1\right]=\left(0,\frac{1}{N}\right]\cup \cdot \cdot \cdot \cup \left(\frac{N-1}{N},1\right]$, where N is the number of initial particles;

• 3. For each initial particle set {Sⁱ}_{i = 1,2,...,N}, which is independent identical distribution, Sⁱ is denoted as ${\mathbf{S}}^i=U\left(\left(\frac{i-1}{N},\frac{i}{N}\right]\right)$, where U((u, v]) is uniform distribution in (u, v];

• 4. The vehicle is fragmented according to the set of features generated by the EOH;

• 5. Compute the initial HSV color histogram of each fragments of vehicle;

• 6. Compute the initial EOH histogram of each fragments of vehicle;

• 7. Initialize the weights of integration of the color and EOH features: α^c = α^e = 0.5;

• 8. For t = 1,2,...

  For i = 1,...,N

  Predict the state of the vehicle by Equation (5): ${\bar{\mathbf{S}}}_t^i=E\left({\mathbf{S}}_t^i\right)=2{\mathbf{S}}_{t-1}^i-{\mathbf{S}}_{t-2}^i$;

  Compute the observation likelihood of color $p\left({\mathbf{O}}_t^c|{\mathbf{S}}_t^i\right)$ by Equation (27)

  Compute the observation likelihood of EOH $p\left({\mathbf{O}}_t^e|{\mathbf{S}}_t^i\right)$ by Equation (28)

  Generate the observation likelihood integrating both color and EOH $p\left({\mathbf{O}}_t|{\mathbf{S}}_t^i\right)$ by Equation (29);

  Update the importance weights: ${\omega }_t^i={\omega }_{t-1}^{i}p\left({\mathbf{O}}_t|{\mathbf{S}}_t^i\right)$;

  End For

• 9. If it is necessary to do re-sampling

  Obtain a new set of particles: $\left\{{\mathbf{S}}_t^i,1/N\right\}~\left\{{\mathbf{S}}_t^i,{\omega }_t^i\right\}$;

  End if

• 10. Generate the final state vector by Eqn. (30);

• 11. Compute the quality function ${\gamma }_t^f$ of color and EOH by Equation (33), respectively;

• 12. Compute the integrated weight ${\alpha }_t^f$ of color and EOH by Equation (32), respectively;

• 13. According to probability density distribution of the posterior of a vehicle's state, compute the two factors η₁(t) and η₂(t) by Equations (39) and (42), respectively;

• 14. Obtain the forgetting factor β _t by Equation (38) to update the vehicle's observation model by Equation (37);

  End For

7. Experimental results

In this section, the proposed approach is used to track vehicles on the road. In our experiments, the dataset is composed of video sequences which were obtained from a real surveillance camera. The camera is fixed on a pole in highway and has a high-angle shot to one side of a driveway. All the experiments were carried out on 640 × 480 pixel sequences with an Intel^® Core™ Duo CPU T7500 2.93 GHz PC. A real-life scenario, including partial occlusion, large-area occlusion in a short time and scale variation, is considered. We verify the performance of our approach via single and multiple vehicle target trackings. In the experiments, the length of each video sequence is 100 frames, and the number of particles is set to 50.

7.1. Quantitative evaluation

We evaluate our algorithm quantitatively in order to show the robustness for tracking. The evaluation compares the position and scale estimation of our approach with the ground-truth. Root mean squared error (RMSE) is used as the performance metric. The RMSE of a vehicle's centroid and bounding box scale are defined as follows:

$${\text{RMSE}}_{\text{centroid}}\left(t\right)=\sqrt{\frac{1}{M}\sum _{m=1}^M\left[{\left(x_{t,m}-{x^{\prime }}_t\right)}^2+{\left(y_{t,m}-{y^{\prime }}_t\right)}^2\right]}$$

(43)

$${\text{RMSE}}_{\text{scale}}\left(t\right)=\sqrt{\frac{1}{M}\sum _{m=1}^M{\left(s_{t,m}-{s^{\prime }}_t\right)}^2}$$

(44)

where $\left(x_t^{\prime },y_t^{\prime }\right)$ and $s_t^{\prime }$ are the ground-truth centroid and scale of a vehicle at time t, respectively. M is the measurement time.

7.2. Tracking results and discussion

For comparison, we conducted our experiments with four different types of trackers: a color-based PF tracker (Tracker 1), an EOH-based PF tracker (Tracker 2), a PF tracker based on fixed-weight multiple visual features (Tracker 3), and our approach. The former three trackers had no adaptive updating during tracking, and the weights of color histogram and EOH features are 0.5 in Tracker 3, respectively.

In Figure 7, the vehicle traveled on a straight driveway. The tracking results of the four trackers are shown in Figure 7a-d. From Figure 7e, f, we can see that the RMSEs of a vehicle's centroid and bounding box scale of the four trackers are all maintained at a lower level, and the four trackers provide nearly similar tracking results. Figure 7g gives the curves of weights of different visual features. It can be seen that the features are in a relatively stable state with no dramatic change throughout the video sequence because of no obvious change of illumination or translation, rotation, etc. So, the tracking results of Track 3 and our approach are more similar to each other.

Figure 7

Vehicle tracked when moving straight forward in the evening.

Figure 8 shows the vehicle turning. Figure 8a-d presents the tracking results of the four trackers and we can see from Figure 8e, f that the RMSE of our approach is lower than that of the other three trackers, especially obvious when the vehicle turns between frames 30 and 40. In Figure 8g, due to translation and rotation, the EOH of the vehicle has a greater change. The decline of the EOH weight makes the color histogram more reliable. From the comparison of the RMSEs of Tracker 3 and our approach in Figure 8e, f, we can see that the RMSEs of a vehicle's centroid are provided more similar tracking results than the bounding box scale because of the advantage of multiple visual features integration for vehicles.

Figure 8

Vehicle tracked during turning in daytime.

The third sequence is captured at night, as shown in Figure 9. The color of the vehicle and the background are very similar, but the edge feature is obvious due to the streetlight and the headlight of the rear vehicle. Figure 9a-d is still the tracking results of four trackers. From Figure 9e, f, we can see that the RMSEs of all trackers increase significantly, but our approach is still more accurate than the other methods. Furthermore, the accuracy of Tracker 3 decreases much faster than our approach, caused by the fixed weights of multiple visual features. In Figure 9g, the changes of weight show that the color histogram is unreliable with low weight because the color of the vehicle shows weak discrimination from the background. Instead, the EOH feature plays a dominant role at this moment.

Figure 9

Vehicle tracked during turning at night.

When vehicles with very similar colors appear close to each other, tracking algorithms using color, EOH, or fixed-weight multiple visual features fail. As shown in Figure 10, the precision of tracking with fixed-weight multiple visual features is slightly better than color-based tracking, but the deviation caused by color similarity is just prolonged and cannot be prevented completely. And the adaptive integration of multiple features makes a contribution to distinguish the vehicles to get a robust tracking result. Partial occlusion between vehicles, even large area occlusion, is the key issue to influence robustness and tracking accuracy. Figures 11, 12, 13, and 14 show the cases of occurrence of occlusion. As illustrated in Figures 11 and 12, partial and large area occlusion appear and last for about 50 frames, respectively. While the proposed approach incorporates spatial information, i.e., fragmentation, the proposed approach can track vehicles with non-occluded fragments. Since some fragments were occluded, the observation model of these regions is unreliable and the forgetting factor is determined to be equal to 0. Therefore, the observation model is stopped from updating. When occlusion is finished, the proposed tracker can still give continuous tracking. Figure 14 shows vehicles traveling under occlusion at night. As Figure 14a-c demonstrates, the former three trackers may lead to inaccurate results due to the ambiguities inherent in the processing of the video sequence when considering single modalities. There are objects in the background which have a similar appearance to the vehicle. Therefore, soon after the initialization, the color-based tracking framework starts on the vehicle and gradually deviates from the ground-truth.

Figure 10

Two similarly colored vehicles when slight occlusion occurs in daytime.

Figure 11

Vehicles tracked when partial occlusion occurs in daytime.

Figure 12

Vehicles tracked when large-area occlusion occurs in daytime.

Figure 13

More vehicles tracked under occlusion in the evening.

Figure 14

Vehicles tracked by our approach at night.

8. Conclusions

This article presents a robust tracking approach for multiple vehicles using adaptive integration of multiple visual features. Color histograms and EOHs are selected as visual features to model the observation of vehicles and integrated by a democratic integration strategy, and the observation model is embedded in a PF tracking framework. The spatial information is incorporated into the observation model to improve the robustness of object representation by dividing the object to be tracked into a number of fragments. Further, in order to avoid errors caused by model drift, the updating process should only be implemented in a reliable manner, and the rate of updating can be controlled according to this reliability. The posterior probability density function of distribution of state vector and similarity between the candidate and reference observation of an object are used to define the valid measurement of reliability to model updating during tracking. Experimental results in real traffic surveillance video sequences show that our approach outperforms others in vehicle tracking under complex conditions.