Digital video stabilizer by adaptive fuzzy filtering

Abstract

Digital video stabilization (DVS) allows acquiring video sequences without disturbing jerkiness, removing unwanted camera movements. A good DVS should remove the unwanted camera movements while maintains the intentional camera movements. In this article, we propose a novel DVS algorithm that compensates the camera jitters applying an adaptive fuzzy filter on the global motion of video frames. The adaptive fuzzy filter is a simple infinite impulse response filter which is tuned by a fuzzy system adaptively to the camera motion characteristics. The fuzzy system is also tuned during operation according to the amount of camera jitters. The fuzzy system uses two inputs which are quantitative representations of the unwanted and the intentional camera movements. The global motion of video frames is estimated based on the block motion vectors which resulted by video encoder during motion estimation operation. Experimental results indicate a good performance for the proposed algorithm.

1. Introduction

Digital video stabilization (DVS) techniques have been studied for decades to improve visual quality of image sequences captured by compact and lightweight digital video cameras. When such cameras are hand held or mounted on unstable platforms, the captured video generally looks shaky because of undesired camera motions. Unwanted video vibrations would lead to degraded view experience and also greatly affect the performances of applications such as video encoding [1–4] and video surveillance [5, 6]. With recent advances in wireless technology, video stabilization systems are also considered for integration into wireless video communication equipments for the stabilization of acquired sequences before transmission, not only to improve visual quality but also to increase the compression performance [1]. Solutions to the stabilization problem involve either hardware or software to compensate the unwanted camera motion. The hardware-based stabilizers are generally expensive and lack the kind of compactness that is crucial for today’s consumer electronic devices [7, 8]. On the contrary, a DVS system that is implemented by software can easily be miniaturized and updated. Consequently, DVS system is suitable for portable digital devices, such as digital camera and mobile phone.

In general, a DVS system consists of two principal units including motion estimation (ME) and motion correction (MC) units. The ME unit estimates a global motion vector (GMV) between every two consecutive frames of the video sequence. Using the GMVs, the MC unit then generates smoothing motion vectors (SMVs) needed to compensate the frame jitters and warp the frames to create a more visual stable image sequence.

According to the motion models being considered, the already proposed global ME techniques for DVS system can roughly be divided into two categories: (1) two-dimensional stabilization techniques which deal with translational jitter only [9–20] and (2) multi-dimensional stabilization techniques which aim at stabilizing more complicated fluctuations in addition to translation [21–25]. Most of the existing algorithms fall into the first category because the translation is the most commonly encountered motion and the complexity of estimating translation parameters is relatively low for real-time stabilization.

Regarding to the ME task of DVS systems, most previous approaches attempt to reduce the computational cost by using fast ME algorithms, e.g., gray-coded bit-plane matching [9], two-bit transform [10], multiplication-free one-bit transform [11], Laplacian two-bit transform [12], and binary image matching of color weight [13]. In another approach, the global ME is limited to small, pre-defined regions [16, 17]. Such approaches consider DVS and video encoding separately and attempt to trade the accuracy of motion vectors (MVs) for the computational efficiency; nevertheless they improve the computational efficiency at the expense of degradation in the accuracy in ME and thereafter in MC tasks.

Since both the video encoder and the digital stabilizer of a digital video camera use a ME unit, we can integrate digital stabilizer with video encoder [2, 4, 26] by making the two modules of a digital video camera share a common local motion vectors (LMVs) estimation process, as shown in Figure 1. The ME task in video encoders usually is implemented on frame blocks by a block matching process to estimate anMV for each block (BMV).

Figure 1

Integration scheme of the video stabilizer and the video encoder.

In video frames with smooth or complex texture regions, the estimated BMVs may not be in coincidence with the real motion of the blocks. Although such LMVs are applicable to the local motion compensation task which is executed in the encoder, they cannot be used for the global motion compensation which is executed by the DVS. These LMVs include some noises that degrade the global ME task. In order to remove the noisy LMVs in these regions some algorithms are proposed in [27–30]. The valid BMVs as LMVs are used for the global ME and MC compensation in next steps.

After global ME, the next essential task of a DVS system is MC in which the unwanted camera jitters are separated and removed from the intentional camera movement. Among the various MC algorithms proposed in the literature, smoothing of the GMV by low-pass filtering is the most popular. For instance, anMV integration method is used in [9, 31] which utilizes a first-order infinite impulse response (IIR) low-pass filter to integrate differential motion and to smoothen the global movement trajectory. A frame position smoothing (FPS) algorithm, based on smoothing absolute frame positions that achieve successful stabilization performance with retained smooth camera movements, is utilized for MC in [17, 32–39]. Off-line discrete Fourier transform (DFT) domain filtering is proposed for FPS-based stabilization in [32]. Kalman filter and fuzzy systems have widely been used in DVS applications [33–39]. Real-time FPS-based stabilizer using Kalman filtering of absolute frame positions has been proposed in [17, 33]. It is shown that the stabilization performance can be improved by a fuzzy adaptive Kalman filter; introducing a stabilization system that is adjusted according to the camera motion characteristics in [34]. Fuzzy stabilization systems improve the stabilization performance when their membership functions (MFs) are optimized to motion dynamics [35]. A membership selective fuzzy stabilization, in which the stabilization system selects between a pre-determined set of MFs according to instantaneous motion characteristics is proposed in [36]. A MF adaptive fuzzy filter for video stabilization is presented in [37] and a fuzzy Kalman system consists of a fuzzy system with a Kalman filter is presented in [38].

Regarding to the MC task of DVS system, almost all published algorithms try to smoothen the global movement trajectory by a kind of low-pass filtering. An important drawback of the low-pass filtering is that smoothened movement trajectory is delayed with respect to the desired camera displacements. A stricter filtering provides more stabilization at the expense of more trajectory delay and vice versa. More trajectory delay means losing more image content after stabilization.

A good MC unit should remove the unwanted camera motion while tracks the intentional motion without any delay. For this purpose, it should discriminate the unwanted and intentional camera motions while adjust the smoothing filter adaptively according to the amount of unwanted and intentional camera motions. The studied published MC algorithms lack some of these features. For example,algorithms presented in [27, 37] suffer from the lack of discrimination of unwanted and intentional camera motions. Moreover, the proposed adaptive algorithm in [27] suffers from a continuous and well adaptation. They use an adaptive filter with a smoothing factor that is switched between only two values and therefore it leads to undesirable jumps in frame position. The proposed algorithm in [38] shows a high performance but still suffers from well adaptation.

In this article, we propose a DVS algorithm with new features in ME and MC units. The ME unit estimates a GMV based on the BMVs which are estimated by the video encoder. Therefore, accurate motion information is used without extra computation cost. Moreover, an adaptive thresholding algorithm is used to remove the noisy invalid LMVs. The MC unit of the proposed DVS system applied a fuzzy adaptive IIR filter to smooth the camera movement trajectory adaptively to the characteristics of unwanted and intentional camera motions. The fuzzy system adjusts the IIR filter by using two novel inputs which are quantitative representations of the unwanted and the intentional camera motions. Experimental results show a good performance for the proposed DVS algorithm.

The remainder of this article is organized as follows. The details of the proposed video stabilization algorithm are described in Section 2. Some experimental results are presented in Section 3, and the article is concluded in Section 4.

2. The proposed method

A flowchart of the proposed DVS system is depicted in Figure 2. The details of the proposed system are described in the sequel.

Figure 2

Flowchart of the proposed DVS system.

2.1. Block-based ME

The block-based ME is used to generate the LMVs. Since the ME is done by the video encoder, the computational complexity of the DVS is very low. In this article, to test the proposed DVS system independent of the encoder, a full search ME algorithm with full-pixel resolution is taken for 8 × 8 blocks over a search range of 33 × 33 pixel to achieve the BMVs.

The ME algorithm works as follows. First, the current frame is divided into a number of N × N blocks and an MV for each block is computed. The resulting MV points to the most correlated reference block in the previous frame within the search area. To measure the goodness of each candidate MV (x, y), the mean absolute difference (MAD) measure is used as

$$\begin{array}{l}\text{MAD}\left(i,j\right)=\frac{1}{N^2}\sum _{k=0}^{\mathrm{N}-1}\sum _{L=0}^{\mathrm{N}-1}\left|C\left(x+k,y+L\right)\right.\\ \left.-\mathrm{R}\left(x+i+k,y+j+L\right)\right|\end{array}$$

(1)

where C(x + k, y + L) and R(x + i + k, y + j + L) denote the block pixels in the target frame and the displaced block pixels in the reference frame, respectively. The candidate MV(i, j) with the smallest MAD is chosen as the MV of the current block according to

$$V=argmin\mathrm{MAD}\left(i,j\right),-p\le i,j\le p-1$$

(2)

where p defines the motion search range.

2.2. LMV validation

The ME unit plays an important role in DVS system and its estimation accuracy is a decisive factor for the overall performance of stabilization system. Block ME process typically computes some wrong MVs which are not in coincidence to the real motion direction of the blocks. Although, such MVs can be useful for the motion compensation in encoder, they include noise and should not be used for the global motion compensation and video stabilization operations. The noisy MVs are mostly obtained from two types of regions including: very smooth regions with lack of features and very complex uneven regions [27–30]. Inspiring from the algorithm presented in [27], two qualifying tests, namely “Smoothness Test” and “Complexity Test”, are used to detect and remove the noisy MVs by an adaptive thresholding method as follows.

2.2.1. Smoothness test

The noisy MVs corresponding to the smooth regions such as sky image are detected by thresholding of the average of MAD as

$${\text{MAD}}_{\mathit{avg}}^n<{\text{th}}_1,$$

(3)

where MAD _avg ⁿ denotes the average of calculated MADs within the search area, during ME of n th block. th₁ is also defined as

$${\text{th}}_1={\text{MAD}}_{\text{min}}^{\mathrm{n}}+T_1\mathrm{x}\text{Mean}\left({\text{MAD}}_{\text{avg}}^{\mathrm{n}}\right),$$

(4)

where MAD _min ⁿ and MAD _Avg ⁿ denote the minimum and the average values of computed MADs, respectively, during ME of n th block within the search area. T₁ is an experimentally defined constant coefficient about 0.45 and Mean(MAD _avg ⁿ) denotes the average of MAD _avg ⁿ, over all blocks of the frame. In fact the threshold th₁ includes a global average value over the frame plus a margin.

2.2.2. Complexity test

The noisy MVs corresponding to the complex texture regions are identified by another thresholding as

$${\text{MAD}}_{\text{min}}^{\mathrm{n}}>{\text{th}}_2,$$

(5)

where threshold th₂ is defined adaptively as:

$${\text{th}}_2={\mathrm{T}}_2\mathrm{x}\text{Max}\left({\text{MAD}}_{min}^{\mathrm{n}}\right),$$

(6)

where T₂ is an experimentally defined constant coefficient about 0.45, and Max Max(MAD_minⁿ) denotes the maximum value of MAD_minⁿ, over all blocks of the frame. According to the equations above, the MAD_minⁿ is compared against a portion of its global maximum over a frame.

It is notable that MAD is computed during ME by encoder. Therefore, the smoothness test and complexity test have no additional computational complexity cost for the proposed DVS system.

A similar thresholding approach is presented in [27–30], in which fixed values for thresholds th₁ and th₂ are used. Our simulation results on different video contents show that using fixed thresholds for different video contents may cause a remarkable amount of invalid noisy LMVs remain or a notable amount of valid LMVs be removed. To solve this problem, the values of thresholds th₁ and th₂ are adjusted adaptively based on the video content for each frame. Note, if ME is executed by a fast search algorithm rather than full-search algorithm at the encoder, the MADs calculated during ME are used for adaptation of thresholds th₁ and th₂.

Original LMVs and validated LMVs for a sample frame are presented in Figure 3. This figure shows that many noisy LMVs have been removed by the LMV validation process.

Figure 3

Example of noisy LMV removal on a frame of the avenue sequence. (a) Original MVs. (b) Valid MVs.

2.3. Global ME

The global ME unit produces a unique GMV for each video frame, which represents the camera movement during the time interval of two frames. Since the LMVs obtained from the image background tend to be very similar in both magnitude and direction, we used a clustering process to classify the motion field into clusters corresponding to the background and foreground objects. The global motion induced by camera movement is determined by a clustering process that consists of the following steps.

Step 1. Construct the histogram H of the valid LMVs. The value of H(x, y) is incremented by one each time the LMV(x, y) is encountered.

Step 2. As long as the scene is not dominated by moving objects, the cluster corresponding to background blocks has the maximum votes in the clustering process. The position (x, y) of the largest cluster or histogram bin is considered as the GMV.

As an example, Figure 4 shows the largest histogram bin at coordinates (5, 12), yields the GMV.

Figure 4

Clusters of motion field.

2.4. Unwanted ME and correction

An estimated GMV may consist of two major components: an intentional motion component (e.g., corresponding to camera panning) and unintentional motion component (e.g., corresponding to handshake). A good MC algorithm should only remove the unwanted motion while maintain and track the intentional motion. Assuming that the unwanted motion is corresponding to the high-frequency components, the proposed algorithm uses a low-pass filter to remove the unwanted motion component. AnSMV is resulted by a low-pass filtering on the GMVs that resembles the intentional camera movement. An adaptive first-order IIR filter is applied as

$$\text{SMV}\left(\mathrm{n}\right)=\alpha \left(\mathrm{n}\right)\text{SMV}\left(\mathrm{n}-1\right)+\left[1-\alpha \left(\mathrm{n}\right)\right]\text{GMV}\left(\mathrm{n}\right).$$

(7)

where the index n indicates the frame number. The parameter α, (0 ≤ α ≤ 1), can be regarded as the smoothing factor of the filter which is adjusted by the fuzzy system for each frame. A larger smoothing factor leads to a smoother, but a larger lag during intentional camera motion that makes artificially stabilized, image sequence. Therefore, a fixed value of α hardly leads to good stabilized image sequences. To avoid the lag of intentional movement and to smooth the unwanted camera motion efficiently the following fuzzy adaptation mechanism of α is proposed.

2.4.1. Fuzzy adaptation of smoothing filter

The smooth filtering is implemented on the vertical and horizontal components of the GMVs separately. The smoothing factor of filter, i.e., α(n) is adjusted by a fuzzy system continuously for MC of each frame. In facts, two fuzzy systems with a similar structure are used corresponding to the vertical and horizontal motion components. The fuzzy system has two inputs (Input1, Input2) and one output. The fuzzy inputs are defined as

$${\mathrm{x}}_1=\frac{1}{M}{\displaystyle \sum _{i=n-M+1}^n\left|\mathit{GM}{V}_{\mathrm{x}}\left(i\right)-\mathit{GM}{V}_{\mathrm{x}}\left(i-1\right)\right|,}$$

(8)

$${\mathrm{x}}_2=\left|{\text{GMV}}_{\mathrm{x}}\left(\mathrm{n}\right)-{\text{GMV}}_{\mathrm{x}}\left(\mathrm{n}-\mathrm{M}\right)\right|.$$

(9)

$$y_1=\frac{1}{M}{\displaystyle \sum _{i=n-M+1}^n\left|{\text{GMV}}_{\mathrm{y}}\left(i\right)-{\text{GMV}}_{\mathrm{y}}\left(i-1\right)\right|,}$$

(10)

$$y_2=\left|{\text{GMV}}_{\mathrm{y}}\left(\mathrm{n}\right)-{\text{GMV}}_{\mathrm{y}}\left(\mathrm{n}-\mathrm{M}\right)\right|.$$

(11)

where x₁ and x₂ denote the inputs of fuzzy system used for the adaptive filtering of the horizontal motion component and also y₁ and y₂ are the inputs of fuzzy system used for the adaptive filtering of the vertical motion component. GMV _x (n) and GMV _y (n) indicate the horizontal and vertical components of the GMV of last frame and M + 1 is the number of last GMVs used for decision. The fuzzy system inputs, Input1 (x₁,y₁) and Input2 (x₂,y₂), are used as quantitative representations of unwanted and intentional camera movements, respectively. The value of Input1 is proportional to the noise amplitude and the value of Input2 is proportional to the intentional camera motion when it has an accelerating movement.

Defining suitable inputs for an adaptive DVS system has a great impact on the performance of system. Only relevant inputs can provide precise discriminating between unwanted and intentional camera motions to be used for the adaptation of smoothing filter. Different scenarios for the combination of unwanted and intentional camera motion can be considered. As examples, some scenarios are presented graphically in Figure 5. In graphs (a) and (b), camera has an intentional accelerating movement plus noise or unwanted motion. The noise amplitude is high in (a) while it can be ignored in (b). Graph (e) is corresponding to a camera movement path while panning in which the camera is moving with a constant velocity without any acceleration and noise. The explanations of all graphs are summarized in Table 1.

Figure 5

Sample scenarios for combination of unwanted and intentional camera motions: (a) high acceleration with high noise, (b) high acceleration with no noise, (c) high noise with no acceleration, (d) low acceleration with low noise, (e) constant velocity without noise and acceleration, (f) constant velocity with noise.

Table 1 Sample scenarios for combination of unwanted and intentional camera motion

From the adaptive filtering point of view it is important to measure the amount of noise and the intentional camera movement velocity and acceleration. A stricter smoothing filter is needed when the noise amplitude is high to remove the noise. On the other hand, the strict smoothing filter prevents following of camera path when it has an intentional high acceleration. Therefore, the smoothing factor of filter should be tuned carefully proportional to the amount of noise and camera movement acceleration. According to this, we defined the fuzzy inputs so that Input1 gives information about the amount of noise and Input2 gives information about the amount of camera movement acceleration. It is notable that amount of camera movement velocity itself does not have any constrain on the filtering so it is not measured and used here.

The proposed fuzzy system tunes the smoothing factor of the IIR filter adaptively according to the amount of noise and the camera intentional accelerating movement. In the proposed fuzzy system, trapezoidal and triangular MFs are used for the inputs and the outputs, respectively. The number of MFs has been selected so as to obtain decent performance with as few MFs as possible to maintain low system complexity. The experimentally designed input and output MFs and also the surface of desired outputs are shown in Figure 6. According to experimental results, the performance of used IIR filter is more sensitive to α’s changes where α has a large value. Therefore, more MFs of the fuzzy output are concentrated in this operating area. The constructed rule base is containing 30 rules as presented in Table 2. The proposed fuzzy system was implemented while the min function was used for the fuzzy implication and the max function used for the fuzzy aggregation. Furthermore, the centroid defuzzification method was applied. The output of fuzzy system defines the smoothing factor of the IIR filter, i.e., α(n), for MC of each video frame.

Figure 6

The experimentally designed inputs and output MFs and also surface of desired input and output of the proposed fuzzy system. (a) MFs of fuzzy Input1 x₁&y₁, (b) MFs of fuzzy Input2 x₂&y₂, (c) MFs of fuzzy output, (d) surface of desired outputs.

Table 2 Central values of fuzzy system output

2.4.2. Adaptive fuzzy MFs

Study on a number of video sequences has shown that the range of fuzzy inputs (Input1, Input2) is very variable on different video contents. Therefore, fixed MFs for the inputs of fuzzy system cannot provide a good stabilization performance over all video contents. In order to have a good performance for the proposed DVS system over different video contents, it is proposed to adjust the MFs of fuzzy inputs adaptively to recently received video frames. The range of MFs for the fuzzy inputs, i.e., (0, Input1(max)) and (0, Input2(max)) are modified adaptively as

$$\text{Input}1\left(\text{max}\right)=\text{Max}\mathrm{of}\text{input}1\text{over}K\text{recent}\text{frames},$$

(12)

$$\text{Input}2\left(\text{max}\right)=\text{Max}\text{of}\text{input}2\text{over}K\mathrm{recent}\text{frames},$$

(13)

where Input1 and Input2 are clipped to a range from 1 to 10% of video frame height in term of pixel, and the K corresponds to the number of frames received in last few seconds, e.g., 2 s. This means that the system is adapted to the time-varying noise conditions while the frame size and frame rate are considered.

2.4.3. MC

After computing the smoothing factor α(n) by the fuzzy system, SMV is calculated by Equation (7). For the first three frames, a fixed large value for α(n) is used. After computing SMV, the unwanted motion vector (UMV) is obtained by

$$\text{UMV}\left(\mathrm{n}\right)=\text{GMV}\left(\mathrm{n}\right)-\text{SMV}\left(\mathrm{n}\right).$$

(14)

To restore the current frame to its stabilized position, we offset the current frame by the accumulated UMV, AMV, defined by

$$\text{AMV}\left(\mathrm{n}\right){\displaystyle \sum _{i=m}^n\text{UMV}}\left(i\right)$$

(15)

where m is the frame number of the last scene cut frame.

3. Experimental results

The performance of the proposed DVS method is evaluated against 15 video sequences covering different types of scenes.

Since there is no well-known video sequence in this research field, the algorithm is tested on a number of sequences which are easily available. For example, some used video sequences are available at [40, 41]. These sequences have a frame rate of 25 fps and a picture size of 352 × 288 pixels. Sample frames of used video sequences are shown in Figure 7. Moreover, the performance of proposed DVS system has been evaluated with several datasets extracted from movement curves published in the literature, not from ME on video sequences. Sample simulation results are presented in Figures 8 and 9. The results presented in Figure 8 correspond to a case in which the camera has no intentional movement,whereas the results presented in Figure 9 correspond to a case in which the camera has intentional camera movements. We worked with both gray-scale and color test sequences where in both cases ME is implemented on the luminance component. Good experimental results are obtained with M = 3. However, a larger M provides more smoothness at the expense of more tracking delay and vice versa. The stabilizer performance is assessed according to the smoothness of the resultant global motion compared to the original sequence and the gross movement preservation capability. The results of the proposed DVS algorithm are compared with results provided by the presented algorithms in [27, 38], as the most relevant anchor algorithms. An adaptive IIR filtering technique and a fuzzy Kalman system are proposed for MC in [27, 38], respectively. Some graphical comparison results are presented Figures 8 and 9. The results provided by the presented algorithm in [27] show that it leads to smooth camera movement trajectory but at expense of a relative large tracking delay when the camera has intentional accelerating movement. The results provided by the presented algorithm in [38] show that it closely tracks the intentional camera movements but at the cost of slightly reduced stabilization capabilities. Whereas results demonstrate that our proposed DVS system provides expanded stabilization, while enables the close tracking of the intentional camera movements. Small-scale subjective quality test also demonstrated that human eyes have better visual perception to the stabilized videos by the proposed DVS system than the original videos in all cases.

Figure 7

Sample frames of used video sequences. Images taken by a camera (a–c) held by a hand; (d) in a moving vehicle.

Figure 8

Comparison results of DVS algorithms: the absolute frame positions before and after stabilization correspond to a case in which the camera has no intentional movement.

Figure 9

Comparison results of DVS algorithms: the absolute frame positions before and after stabilization correspond to a case in which the camera has intentional movement.

One way of visualizing the effectiveness of image stabilization is to subtract consecutive frames of the original and stabilized image sequences. Figure 10 shows an example of this technique. The upper left image in (a) shows a frame of the original sequence while the upper right image in (b) is the stabilized frame at the same time instance. The bottom left image in (c) shows the difference between the frame in (a) and its previous frame from original sequence. Similarly, Figure 10 shows the difference between the stabilized frame shown in (b) and its previous frame in the stabilized sequence. The decrease in luminance of the difference signal means removing unwanted camera motion. Expressing the difference of consecutive frames in the original and the stabilized image sequences in terms of squared error, for a video sequence, namely road [40], is plotted in Figure 11. As expected, the frame difference measure has been reduced considerably in the stabilized image sequence.

Figure 10

Visualizing the effectiveness of image stabilization. (a) Original frame, (b) stabilized frame, (c, d) difference between two frames before and after MC, respectively.

Figure 11

Frame difference for the road video sequence before and after stabilization.

Numerical performance assessment of a DVS system is a difficult task since the ground truth of unwanted motions as reference is not available. To solve the lack of reference problem, we choose some video sequences in which the camera has been fixed without any intentional movement. Therefore, the reference GMVs of these sequences are zero. Samples of such video sequences are shown in Figure 8. Finding a reference path for intentional camera motion, we computed the meansquare error (MSE) between the smoothed GMVs and the GMVs of reference as a numerical measure to compare the MC performance of proposed adaptive fuzzy DVS algorithms with those of presented algorithms in [27, 38]. The MSE measure is defined as

$${\text{MSE}}_{\text{MV}}=\frac{1}{N}{\displaystyle \sum _{n=0}^N{\left[{\text{GMV}}_{\mathrm{R}}\left(\mathrm{n}\right)-\text{SMV}\left(\mathrm{n}\right)\right]}^2},$$

(16)

where GMV _R (n) and SMV(n) denote the n th GMV of the reference and the smoothed video sequence, respectively. Numerical performance assessment results for the three video sequences, as shown in Figure 8, are presented in Table 3. The numerical results show lower MSE values for our proposed method that means a higher performance in this case.

Table 3 MSE resulted by MC units of different DVSs

To evaluate the performance of the proposed algorithm numerically in case of intentional camera movement, we produced a synthetic signal as reference. The reference signal includes three parts corresponding to intentional camera movements with positive, negative, and zero value accelerations. Moreover, a sequence of normal random numbers was generated and the random numbers were rounded to integer values to simulate the unwanted camera motion in terms of pixel displacement. Provided sequence was added to the reference signal to obtain a synthetic signal including both the intentional and the unwanted camera motions. Two copies of the synthetic signal (y₁y₂) were generated to be processed by MC units of the proposed DVS algorithm and also by the anchor algorithms presented in [27, 38]. While the reference signal is known that the performance of compared algorithms can be evaluated numerically by computing the MSE measure between the reference signal and the processed signals. It is noted that the performance of the proposed algorithm in [38] depends on the values of Q parameter (process noise covariance) of Kalman filter, so it cannot provide a well adaptation. Therefore, the two copies of synthetic signal were processed by two values of Q (0.005 and 0.6) and graphical simulation results are shown in Figures 12 and 13, respectively. Moreover, numerical comparison results in terms of MSE for the synthetic signals are presented in Table 4. According to the graphical results, the proposed algorithm in [27] removes the unwanted camera motions in case of no intentional camera movement. It results a considerable tracking delay in case of intentional accelerating camera movement. The proposed algorithm in [38], depending on the Q value, performs only good filtering of the unwanted camera motion or only the tracking of intentional camera movement as shown in Figures 12 and 13, respectively. The proposed algorithm in this article shows a high performance in both the removing of unwanted motion and the tracking of intentional camera movement. Furthermore, the numerical results presented in Table 4 confirm the graphical results shown in Figures 12 and 13. The least MSE for our proposed algorithm means the best performance.

Figure 12

Synthetic camera movement path y ₁ processed by MC units of different DVSs.

Figure 13

Synthetic camera movement path y ₂ , processed by MC units of different DVSs.

Table 4 Resulted by MC units of different DVSs

4. Conclusion

In this article, we proposed a computationally efficient DVS algorithm using motion information obtained from a hybrid block-based video encoder. Since some of the obtained MVs are not valid, an adaptive thresholding was developed to filter out valid MVs and to compute an accurate GMV for each frame. The GMVs are smoothened with an IIR low-pass filter that is tuned adaptively to unwanted and intentional camera movements. The filter is adjusted by a fuzzy system with two inputs which quantify the unwanted and intentional camera movements. The proposed method fulfills two apparently conflicting requirements: close follow-up of the intentional camera movement and removal of the unwanted camera motion. In order to improve the stabilization performance, inputs MFs of the fuzzy system are continuously adapted according to the motion properties of a number of recently received video frames. Simulation results show a high performance for the proposed algorithm. With a low degree of computational complexity, the proposed scheme can effectively be used for the mobile video communications as well as for the conventional video coding applications to improve the visual quality of digital video and to provide a higher compression performance.