Modeling of SSIM-based end-to-end distortion for error-resilient video coding

Abstract

Conventional end-to-end distortion models for videos measure the overall distortion based on independent estimations of the source distortion and the channel distortion. However, they are not correlating well with the perceptual characteristics where there is a strong inter-relationship among the source distortion, the channel distortion, and the video content. As most compressed videos are represented to human users, perception-based end-to-end distortion model should be developed for error-resilient video coding. In this paper, we propose a structural similarity (SSIM)-based end-to-end distortion model to optimally estimate the content-dependent perceptual distortion due to quantization, error concealment, and error propagation. Experiments show that the proposed model brings a better visual quality for H.264/AVC video coding over packet-switched networks.

1 Introduction

Most video coding standards achieve high compression using transform coding and motion-compensated prediction, which creates a strong spatial-temporal dependency in compressed videos. Thus, transmitting highly compressed video streams over packet-switched networks may suffer from spatial-temporal error propagation and may lead to severe quality degradation at the decoder side [1]. To protect compressed videos from packet loss, error-resilient video coding becomes a crucial requirement. Given transmission conditions, such as bit rate and packet loss ratio, the target of error resilient video coding is to minimize the distortion at the receiver [2]:

$$min\left\{D\right\}\mathrm{s}.\mathrm{t}.R\le R_T\mathrm{and}\rho$$

(1)

where D and R denote the distortion at the receiver and the bit rate, respectively. R _T is the target bit rate and ρ is the packet loss ratio. Note that we assume the transmission conditions are available at the encoder throughout this paper. This can be either specified as part of the initial negotiations or adaptively calculated from information provided by the transmission protocol [3].

Assume packet containing video data is lost in the channel and the decoder performs error concealment. Clearly, the resulting reconstruction at the decoder is different from the reconstruction at the encoder and the difference will propagate to the following frames due to the prediction chain. Therefore, the key challenge of the error-resilient video coding is to estimate at the encoder the reconstruction error and error propagation of the decoder, which is useful to optimize the coding options to solve the above minimization problem.

A number of end-to-end distortion models (also known as joint source-channel distortion models) for video transmission over lossy channels have been proposed in the literature. In [4, 5], several low-complexity estimation models were presented for low error rate applications. For a more accurate distortion estimation model, the work in [2] developed a frame-level recursion distortion model, which relates to the channel-induced distortion due to bit errors. Another efficient approach is the well-known recursive optimal pixel estimation (ROPE) model [3] and its extensions [6–10], which estimate the overall distortion due to quantization, error concealment, and error propagation. Recently, several novel source-channel distortion models were developed for distributed video coding [11], generic multi-view video transmission [12], and error-resilient schemes based on forward error correction [13].

However, these models are derived in terms of mean squared error (MSE), which has been criticized for weak correlation with perceptual characteristics. As most compressed videos are presented to humans, it is meaningful to incorporate visual features into the error-resilient video coding to protect important visual information of compressed videos from packet loss. Thus, several region-of-interest (ROI)-based approaches were presented to better evaluate the visual quality [14, 15]. However, ROI-based approaches do not provide accurate distortion estimation, and ROI determination may be difficult for most videos, especially for videos with natural scenes. Therefore, it is expected that a perception-based end-to-end distortion model could provide a more general and accurate perceptual distortion estimation.

In [16], the structural similarity (SSIM)-based end-to-end distortion was predicted by several factors extracted from the encoder. Although the variation trend is very similar at the block level, the estimated SSIM cannot reach the peak points of the actual SSIM. In [17], a parametric model was proposed to accurately estimate the degradation of SSIM over error-prone networks, in which the content, encoding, and network parameters are considered. However, the encoding parameters only included the number of slices per frame and the GOP length. The proposed model cannot estimate the relative quality of a block given different coding modes. In our earlier work [18], we introduced a block-level SSIM-based distortion model into the error-resilient video coding to minimize the perceptual distortion. In [19], improved SSIM-based distortion model and Lagrange multiplier decision method are proposed for better coding performance. In [18] and [19], the expected SSIM scores were estimated by the expected decoded frames. Due to the nonlinear variation of SSIM, the estimated SSIM scores may be less accurate, especially at high bit rate.

In this paper, we develop an SSIM-based end-to-end distortion model to estimate the overall perceptual distortion for H.264/AVC coded video transmission over packet-switched networks. Unlike the traditional end-to-end distortion model, the perceptual quantization distortion and the perceptual error propagation distortion are dependent on the video content, which makes the end-to-end distortion become complex or difficult to estimate at the encoder. Therefore, this paper provides two major contributions: 1) a SSIM-based reconstruction quality model; 2) a SSIM-based error propagation model. Both models are useful to estimate the content-dependent perceptual distortion at the encoder. Our extensive experimental results demonstrate that the proposed end-to-end distortion model can bring visual quality improvement for H.264/AVC video coding over packet-switched networks. We would like to mention that the scheme presented in this paper is an enhanced approach based on our preliminary work in [20]. Different settings are considered in this paper, including additional descriptions of related works, technical and implementation details, and comparison experiment results to better evaluate the efficiency of the proposed scheme.

The rest of the paper is organized as follows. Section 2 states the problem and motivation. Section 3 describes the proposed SSIM-based end-to-end distortion model. Section 4 introduces the distortion model into the error-resilient video coding. Section 5 provides the simulation results and Section 6 concludes the paper.

2 Problem and motivation

For H.264/AVC coded video transmission over packet-switched networks, the general formulation of the widely used MSE-based end-to-end distortion can be defined as

$$D=\left(1-\rho \right)\cdot D_Q+\rho \cdot D_C+\left(1-\rho \right)\cdot D_{P\_f}+\rho \cdot D_{P\_c}$$

(2)

where ρ is the packet loss ratio. D is the estimated overall distortion. D _Q denotes the source distortion due to the quantization. D _c , D _{P_f} , and D _{P_c} represent the channel distortion due to the error concealment, error propagation from the reference frames, and error propagation from the concealment frames, respectively.

With such a model in Equation 2, the end-to-end distortion can be individually and independently estimated by the quantization distortion, error concealment distortion, and error propagation distortion. This model is appealing because it is easy to calculate and has clear physical meanings. However, since the perceptual distortion is dependent on the video content, the individual and independent objective distortion estimation does not correspond well with human perceptual characteristics. For instance, as shown in Figure 1, since ten compressed or lossy transmitted videos (Live video quality database [21, 22]) have different perceptual characteristics, a similar objective distortion may result in different levels of perceptual quantization distortion or transmission distortion. Therefore, we aim to propose a perception-based end-to-end distortion model for more accurate estimation of the overall perceptual distortion in the following section.

Figure 1

Quality comparison. (a) PSNR results for compressed videos; (b) DMOS results for compressed videos; (c) PSNR results for lossy transmitted videos; (d) DMOS results for lossy transmitted videos.

3 SSIM-based end-to-end distortion model

To estimate the overall perceptual distortion of decoded videos, we adopt the SSIM index [23] as the perceptual distortion metric due to its best trade-off among simplicity and efficiency [24]. Three important perceptual components, luminance, contrast, and structure, are combined as an overall similarity measure. For two images x and y, the SSIM index is defined as follows:

$$\begin{array}{ll}\mathrm{SSIM}\left(x,y\right)& =l\left(x,y\right)\cdot c\left(x,y\right)\cdot s\left(x,y\right)\\ =\frac{2{\mu }_x{\mu }_y+c_1}{{\mu }_x^2+{\mu }_y^2+c_1}\cdot \frac{2{\sigma }_{\mathit{xy}}+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}\end{array}$$

(3)

where l(x,y), c(x,y), and s(x,y) represent the luminance, contrast and structure perceptual components, respectively. μ, σ², and σ _xy are the mean, variance, and cross covariance, respectively. c₁ and c₂ are used to avoid the instability when means or variances are close to zero.

Based on the perceptual distortion metric, we develop a novel end-to-end distortion model as follows. In Figure 2, b denotes the original block and $\tilde{b}$ is the corresponding reconstruction block at the decoder. $\widehat{r}$ and $\tilde{r}$ represent the prediction block of b at the encoder and at the decoder, respectively. e denotes the prediction residual and its reconstruction value is $\widehat{e}$. If the block is received correctly, $\tilde{b}=\tilde{r}+\widehat{e}$. When the block is lost, an error concealment technique is used to estimate the missing content. Let $\widehat{c}$ and $\tilde{c}$ represent the concealment block of b at the encoder and at the decoder, respectively. In this case, $\tilde{b}=\tilde{c}$. For a given packet loss ratio ρ, the general SSIM-based end-to-end distortion can be expressed as

Figure 2

End-to-end distortion model in lossy transmission channel.

$$\begin{array}{ll}{D}_{\mathrm{SSIM}}\left(b,\tilde{b}\right)& =\left(1-\rho \right)\cdot E\left\{1-\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)\right\}\\ +\rho \cdot E\left\{1-\mathrm{SSIM}\left(b,\tilde{c}\right)\right\}\end{array}$$

(4)

with

$$\begin{array}{ll}E\left\{1-\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)\right\}& =1-E\left\{\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)\right\}\\ =1-{\mathrm{\phi }}_r\cdot E\left(b,\widehat{r}+\widehat{e}\right)\end{array}$$

(5)

$$\begin{array}{ll}E\left\{1-\mathrm{SSIM}\left(b,\tilde{c}\right)\right\}& =1-E\left\{\mathrm{SSIM}\left(b,\tilde{c}\right)\right\}\\ =1-{\mathrm{\phi }}_c\cdot \mathrm{SSIM}\left(b,\widehat{c}\right)\end{array}$$

(6)

where E { } is the expectation operator. φ is the error propagation factor. It indicates how the transmission errors from prediction block or concealment block influence the quality of current block. $\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)$ and $\mathrm{SSIM}\left(b,\tilde{c}\right)$ denote the quality of prediction coding and error concealment at the decoder, respectively. $\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)$ and $\mathrm{SSIM}\left(b,\widehat{c}\right)$ are the quality of prediction coding and error concealment at the encoder, respectively.

With this formula, the reconstruction quality $\mathrm{SSIM}\left(b,\widehat{r}+\widehat{e}\right)$ and error propagation factor φ are the key terms of the SSIM-based end-to-end distortion model. In the following section, we will make a development of the two terms based on content dependency.

3.1 Development of reconstruction quality model

In this section, we aim to estimate the content-dependent reconstruction quality $\mathrm{SSIM}\left(b,\widehat{r}+\widehat{e}\right)$ at the block level (the 4 × 4 transform and quantization unit is used throughout this paper). Since the accurate reconstruction quality can only be obtained after de-quantization, the proposed quality estimation reduces the computational complexity of de-quantization process for each candidate modes.

According to the SSIM index, the reconstruction quality is derived as

$$\mathrm{SSIM}\left(b,\widehat{r}+\widehat{e}\right)=l\left(b,\widehat{r}+\widehat{e}\right)\cdot c\left(b,\widehat{r}+\widehat{e}\right)\cdot s\left(b,\widehat{r}+\widehat{e}\right)$$

(7)

with

$$l\left(b,\widehat{r}+\widehat{e}\right)=\frac{2{\mu }_b{\mu }_{\widehat{r}}+2{\mu }_b{\mu }_{\widehat{e}}+c_1}{{\mu }_b^2+{\mu }_{\widehat{r}}^2+{\mu }_{\widehat{e}}^2+2{\mu }_{\widehat{r}}{\mu }_{\widehat{e}}+c_1}$$

(8)

$$c\left(b,\widehat{r}+\widehat{e}\right)\cdot s\left(b,\widehat{r}+\widehat{e}\right)=\frac{2{\sigma }_{b\widehat{r}}+2{\sigma }_{b\widehat{e}}+c_2}{{\sigma }_b^2+{\sigma }_{\widehat{r}}^2+{\sigma }_{\widehat{e}}^2+2{\sigma }_{\widehat{r}\widehat{e}}+c_2}$$

(9)

From Equations 8 and 9, we can see that the estimation of content-dependent reconstruction quality is converted to the estimation problem of three content-independent parameters: 1) the variance of reconstructed prediction residual; 2) the cross-covariance between the reconstructed prediction residual and current block; 3) the cross-covariance between the reconstructed prediction residual and prediction block.

It is reported that the DCT coefficients of prediction residual closely follows a zero-mean Laplacian distribution [25]. Based on this phenomenon, the work in [26] proved that the reconstruction distortion from the prediction residual can be estimated by the Laplacian parameter and the quantization step. Extending the derivation in [26] into pixel-domain, we establish the following two estimation models for above parameters:

$${\sigma }_{\widehat{e}}^2=M_{\text{var}}\left(\alpha ,\mathit{Q}\mathit{P}\right)\cdot {\sigma }_e^2,\alpha =\sqrt{2/{\sigma }_e^2}$$

(10)

$${\sigma }_{b\widehat{e}}=M_{\text{cov}}\left(\beta ,\mathit{Q}\mathit{P}\right)\cdot {\sigma }_{\mathit{be}},\beta =\sqrt{2/{\sigma }_{\mathit{be}}}$$

(11)

where α and β denote the Laplacian parameters. QP is the quantization parameter in H.264/AVC. M_var and M_cov indicate the scaling maps, which vary from 0 to 1.

The scaling maps M_var and M_cov are modeled based on four video sequences [27]: ‘Crow_run’, ‘In_to_tree’, ‘Ducks_take_off’, and ‘Old_town_cross’, which have abundant and various structural information. Each sequence is coded as intra-frame (I frame) and inter-frame (P frame), respectively. To cover various reconstruction variances, 11 different QP values are tested, ranging from 15 to 45 uniformly with the step size of 3.

Firstly, we calculate the variance of initial and reconstructed prediction residual with different QP values. Secondly, we obtain the scaling curve by doing statistics analysis for each test QP. Finally, we interpolate the eleven scaling curves to establish the scaling map. Based on the simulations, the fitted scaling map M_var and M_cov are shown in Figure 3, which can be constructed as look-up tables.

Figure 3

The scaling maps. (a) For residual variance by intra-coding; (b) for residual variance by inter-coding; (c) for residual cross covariance by intra-coding; (d) for residual cross covariance by inter-coding.

To demonstrate the accuracy of the proposed reconstruction quality models, 250 frames of each sequence are coded with constant quantization parameters: 20, 25, 30, and 35, respectively. Table 1 shows the average mean absolute deviation (MAD) between the actual and estimated variance, cross covariance, and reconstruction quality. The first two terms denote the accuracy of the fitted models (10) and (11), respectively. The following three terms show the accuracy of the final estimated SSIM scores. It can be seen that the proposed models are valid to predict the reconstruction quality.

Table 1 MAD between actual and estimated scores

3.2 Development of error propagation model

The error propagation is the key component of the end-to-end distortion model. Different from the independent estimation in conventional MSE-based end-to-end distortion model, the perceptual error propagation depends on the source distortion or the concealment distortion. In this section, our primary goal is to develop the error propagation models to estimate the overall perceptual quality for given transmission errors of prediction block or concealment block.

The error propagation models are motivated by three observations. The first observation is related to the impact of error propagation on the three components of SSIM. Let Q_att denote the quality attenuation of a given block b due to the error propagation.

$$Q_{\mathrm{att}}\left(b\right)=\mathrm{SSIM}\left(b,\tilde{b}\right)/\mathrm{SSIM}\left(b,\widehat{b}\right)$$

(12)

To illustrate the fact, Q_att is measured by three different similarity metrics: 1) luminance component of SSIM, 2) the contrast and structure components of SSIM, 3) SSIM index. Figure 4 illustrates an example where the quality attenuation is calculated by Equation 12 for each frame suffering from random transmission errors. As shown, the contrast and structure components have the similar changes with SSIM. On the other hand, the impact of error propagation on the luminance component is limited.

Figure 4

The quality attenuation due to transmission errors.

The second observation is made on the relationship f _p between the quality attenuation of block b and the quality attenuation of its compensation block p. p indicates the compensation block of b, which may contain the transmission errors. Thus, p can be used to represent the prediction block r or concealment block c of b.

$$f_p=Q_{\mathrm{att}}\left(b_p\right)/Q{}_{\mathrm{att}}\left(p\right)=\frac{\mathrm{SSIM}\left(b,\tilde{p}\right)}{\mathrm{SSIM}\left(b,\widehat{p}\right)}/\frac{\mathrm{SSIM}\left(p,\tilde{p}\right)}{\mathrm{SSIM}\left(p,\widehat{p}\right)}$$

(13)

Usually, the quality attenuation of block b correlates with the quality attenuation of its compensation block p. In addition, the structural similarity between current block and its compensation block may be another factor in estimation of f _p . x _p is defined as follows to explore the effect of quality attenuation and structural similarity on f _p .

$$x_p=Q_{\mathrm{att}}\left(p\right)\cdot \mathrm{SSIM}\left(b,\widehat{p}\right)$$

(14)

The simulation results are carried out on the same four sequences as Section 3.1. Each sequence is coded with four different QPs: 15, 25, 35, and 45. One I frame followed by all inter frames (IPPP). To cover various error propagation, each block is tested with random transmission errors propagated from prediction block and concealment block, respectively. Note that the prediction residuals of block b are not included in this observation.

Figure 5a displays the simulation results. The mean of each test frame is recorded as one blue sample, and the fitted curve of f _p (x _p ) is shown as the red line. It shows that the quality attenuation of block, in terms of SSIM, is related with the quality attenuation of its compensation block and the structural similarity. Moreover, it demonstrates that less quality attenuation of compensation block or less structural similarity between current block and compensation block leads to less quality attenuation.

Figure 5

Quality attenuation due to error propagation. (a) The relationship between f _p and x _p ; (b) the relationship between f _e and x _e .

The third observation is related to the impact of prediction residual on the decoded quality. Let Q_enh denote the quality enhancement of a given block b due to its prediction residuals e. f _e represents the relationship between the quality attenuation of block b with and without the prediction residual.

$$Q_{\mathrm{enh}}\left(b_{p,e}\right)=\mathrm{SSIM}\left(b,p+e\right)/\mathrm{SSIM}\left(b,p\right)$$

(15)

$$f_e=Q_{\mathrm{att}}\left(b_{p,e}\right)/Q{}_{\mathrm{att}}\left(b_p\right)=\frac{\mathrm{SSIM}\left(b,\tilde{p}+\widehat{e}\right)}{\mathrm{SSIM}\left(b,\widehat{p}+\widehat{e}\right)}/\frac{\mathrm{SSIM}\left(b,\tilde{p}\right)}{\mathrm{SSIM}\left(b,\widehat{p}\right)}$$

(16)

In this observation, the quality attenuation of block b may link with the quality attenuation of its compensation block p and the quality enhancement of its prediction residuals e. x _e is defined as follows to explore the effect of quality attenuation and quality enhancement on f _p :

$$x_e=Q_{\mathrm{att}}\left(b_p\right)/Q_{\mathrm{enh}}\left(b_{\widehat{p},\widehat{e}}\right)$$

(17)

The simulation set-up is the same as that in the second observation. In this observation, each block including the prediction residuals is tested with random transmission errors propagated from its prediction blocks. Figure 5b shows the simulation results. The mean of each test frame is recorded as one blue sample, and the fitted curve of f _e (x _e ) is shown as the red line. The results show that a larger prediction residual leads to a better decoded quality of current block, and the influence of error propagation from prediction blocks will be smaller.

According to Equations 13 and 16, the effective approximation of $\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)$ and $\mathrm{SSIM}\left(b,\tilde{c}\right)$ can be developed as

$$\begin{array}{ll}\mathrm{SSIM}\left(b,\tilde{r}+\widehat{e}\right)& \approx f_e\left(x_e\right)\cdot \mathrm{SSIM}\left(b,\widehat{r}+\widehat{e}\right)\cdot \frac{\mathrm{SSIM}\left(b,\tilde{r}\right)}{\mathrm{SSIM}\left(b,\widehat{r}\right)}\\ \approx f_e\left(x_e\right)\cdot \mathrm{SSIM}\left(b,\widehat{r}+\widehat{e}\right)\cdot f_p\left(x_r\right)\cdot \frac{\mathrm{SSIM}\left(r,\tilde{r}\right)}{\mathrm{SSIM}\left(r,\widehat{r}\right)}\\ \approx Q_{\mathrm{att}}\left(r\right)\cdot f_e\left(x_e\right)\cdot f_p\left(x_r\right)\cdot \mathrm{SSIM}\left(b,\widehat{r}+\widehat{e}\right)\end{array}$$

(18)

$$\begin{array}{ll}\mathrm{SSIM}\left(b,\tilde{c}\right)& \approx f_p\left(x_c\right)\cdot \mathrm{SSIM}\left(b,\widehat{c}\right)\cdot \frac{\mathrm{SSIM}\left(c,\tilde{c}\right)}{\mathrm{SSIM}\left(c,\widehat{c}\right)}\\ \approx Q_{\mathrm{att}}\left(c\right)\cdot f_p\left(x_c\right)\cdot \mathrm{SSIM}\left(b,\widehat{c}\right)\end{array}$$

(19)

Where $\mathrm{SSIM}\left(r,\tilde{r}\right)$ and $\mathrm{SSIM}\left(c,\tilde{c}\right)$ represent the end-to-end distortion of prediction block and concealment block, respectively. The approximations used in the equations represent the estimation of f _p and f _e .

Based on Equations 18 and 19, the error propagation factors in Equations 5 and 6 can be obtained by

$${\mathrm{\phi }}_r=Q_{\mathrm{att}}\left(r\right)\cdot f_p\left(x_r\right)\cdot f_e\left(x_e\right)$$

(20)

$${\mathrm{\phi }}_c=Q_{\mathrm{att}}\left(c\right)\cdot f_p\left(x_c\right)$$

(21)

To better demonstrate the accuracy of the proposed error propagation models, Table 2 shows the average MAD between the actual and estimated SSIM of the four test sequences. It indicates that video quality at the decoder, in terms of SSIM, can be approximately calculated by the fitted models (20) and (21) at the encoder.

Table 2 MAD between actual and estimated SSIM

4 Error-resilient video coding

It is widely recognized that intra-update is an effective approach for error-resilient video coding because decoding of an intra-coding block does not require information from its previous frames. To better evaluate the performance of our proposed model, we incorporate the proposed SSIM-based end-to-end distortion model into the mode selection to improve the RD performance over packet-switched networks. Thus, the optimization problem in Equation 1 can be converted to the problem of mode selection between intra-coding and inter-coding as follows:

$$\begin{array}{ll}min\left\{J\left(\mathrm{mode}\right)\right\}& =min\left\{D_{\mathrm{SSIM}}\left(\mathrm{mode}|\rho ,\mathrm{QP}\right)\right.\\ \left.+{\lambda }_{\mathrm{SSIM}}\cdot R\left(\mathrm{mode}|\mathrm{QP}\right)\right\}\end{array}$$

(22)

where D_SSIM and R denote the end-to-end distortion and bit-rate of current coding block. ρ is the packet loss ratio. mode denotes the coding mode. QP is the quantization parameter, which is determined by the target bit rate. According to [8, 28, 29], the Lagrange multiplier λ_SSIM is determined as follows:

$${\lambda }_{\mathrm{SSIM}}=\left(1-\rho \right)\cdot \overline{D_{\mathrm{SSIM}}}\cdot f\left(R_T\right)$$

(23)

where $\overline{D_{\mathrm{SSIM}}}$ denotes the average distortion of previous coding units. f (R _T ) is an look-up experimental function [20], which is inversely proportional to the target bit rate R _T .

5 Experimental results

5.1 Evaluation of end-to-end distortion model

To validate the effectiveness of our proposed models, the end-to-end distortion models proposed in [16] and [19] are used as comparison. In [17], the SSIM-based estimation model is not suitable for a block given different coding modes. Thus, we did not compare the performance with the proposed model in [17].

We performed the simulation with ten LIVE sequences [21, 22]. First 100 frames of each sequence are encoded by four different QPs: 24, 28, 32, and 36, respectively. Random packet losses (10% and 20%) are used. Each experiment is repeated 200 times and the results are averaged. Table 3 shows the average MAD between the actual and estimated end-to-end distortion, in terms of SSIM. It is obvious that our proposed model achieves better performance for most sequences.

Table 3 MAD between actual and estimated end-to-end distortion

5.2 Evaluation of RD performance

To validate the RD performance of error-resilient video coding, the MSE-based error resilient video coding scheme (MSE-ER) and the SSIM-based error-resilient video coding scheme (SSIM-ER) in [19] are used as the comparison schemes. For MSE-ER, the end-to-end distortion is estimated by the ROPE model [3], which is well studied and regarded as an advanced MSE-based distortion model, and the Lagrange multiplier is calculated by the model presented in [8].

We evaluate the performance on the platform of JM 15.1 [30], in which the SSIM index is adopted as an optimal quality metric. Five CIF sequences [27], five 640 × 360 sequences [27] and ten LIVE sequences [21, 22] are tested in the experiments. The first frame is coded as I frame and the rest are coded as P frames. The rate control is turned on. Table 4 shows the detail information of target bit rates for test sequences. Corresponding to four different target bit rates, the initial QP is equal to 24, 28, 32, and 36 for the first I frame and P frame. Frames are partitioned into one or more slices (each slice contains no more than 1,200 bytes), and the slices are organized in packets for transmission where each slice is packed into one packet. The test sequences are encoded with 5%, 10%, and 20% random packet loss ratio, respectively. For each packet loss ratio, four different target bit rates are tested in the experiments. Each experiment is repeated by 200 times, and the results are averaged, which are used as the final result.

Table 4 Detail information of the test sequences

Figure 6 illustrates the results of Rate-SSIM performance comparison for four test sequences. Moreover, we choose the MSE-ER scheme as the baseline and calculate all the simulation results of average SSIM gain with different bit rates and packet loss ratios, which are tabulated in Table 5.

Figure 6

Comparison of Rate-SSIM performance. (a) ‘Ducks_take_off’ by with 5% packet loss. (b) ‘Football’ by with 10% packet loss. (c) ‘Park run’ by with 10% packet loss. (d) ‘Shields’ by with 20% packet loss.

Table 5 Simulation results of SSIM again with different packet loss ratios and bit rates

It can be seen that the proposed model yields consistent gains over the MSE-ER for all sequences except ‘Pedestrian area’. Our proposed scheme achieves an average SSIM gain of 0.0218 or equivalently a bit rate saving of 15.7%. Comparing to the SSIM-ER [19], our proposed scheme has better performance of most sequences and obtains an average gain of 0.0068. For some sequences, such as ‘Mobile calendar’ and ‘Pedestrian area’, although our proposed scheme cannot achieve the best performance, the quality of the two SSIM-ER schemes is similar.

5.3 Evaluation of subjective quality

Finally, we show the visual quality comparison of reconstructed images by different error-resilient video coding schemes. Figure 7 compares the subjective quality of the 25th frame of ‘Stefan’ encoded at 1.7 Mbps with 10% packet loss. Figure 8 shows the comparison on visual quality of the 38th frame of ‘In_to_tree’ encoded at 1 Mbps with 20% packet loss. Figure 9 represents the visual quality of the 29th frame of ‘Station’ encoded at 0.85 Mbps with 5% packet loss.

Figure 7

Subjective quality comparison of one CIF sequence. (a) Original frame of ‘Stefan’; (b) ‘Stefan’ by ‘MSE-ER’ (SSIM 0.913); (c) ‘Stefan’ by SSIM-ER [19] (SSIM 0.917); (d) ‘Stefan’ by our proposed (SSIM 0.929).

Figure 8

Subjective quality comparison of one 640 × 360 sequence. (a) Original frame of ‘In_to_tree’; (b) ‘In_to_tree’ by MSE-ER (SSIM 0.833); (c) ‘In_to_tree’ by SSIM-ER [19] (SSIM 0.860); (d) ‘In_to_tree’ by our proposed (SSIM 0.873).

Figure 9

Subjective quality comparison of one LIVE sequence. (a) Original frame of ‘Station’; (b) ‘Station’ by MSE-ER (SSIM 0.826); (c) ‘Station’ by SSIM-ER [19] (SSIM 0.851); (d) ‘Station’ by our proposed (SSIM 0.869).

For the similar bit rate, the reconstructed images based on the SSIM-based error-resilient video coding can provide a better visual quality due to more image details being protected from transmission errors. On the other side, the reconstructed images based on the conventional MSE-based error-resilient video coding suffer from larger perceptual distortion. Compared to SSIM-ER [19], our proposed scheme obtains similar or better visual quality.

5.4 Evaluation of coding complexity

Our proposed SSIM-based error resilient video coding scheme improves the RD performance for lossy transmission over packet-switched network. However, the computational complexity of codec is increased due to SSIM-based distortion calculation and mode selection.

We compare the coding efficiency with different bit rates and packet loss ratios. Table 6 shows the average encoding time ratio of SSIM-ER [19] and proposed scheme to MSE-ER, respectively. The experiments are performed on a laptop with 3.4 GHz Intel Core i7-3770 CPU and 4G memory running on Microsoft Windows 7 professional platform. Each experiment is repeated 100 times and the results are averaged.

Table 6 Average encoding time ratio of SSIM-ER[19] and proposed scheme to MSE-ER, respectively

Comparing to MSE-ER, the average computation of SSIM-ER [19] and proposed scheme increase by 3.65% and 5.58%, respectively. In addition, different sequences have inconsistent degree of encoding time, as can be seen in Table 6. That is because the computation complexity is also affected by the characteristics of video content and the results of mode selection. The SSIM-based schemes may take more time to code the image details, such as ‘Sunflower’ and ‘Mobile’.

6 Conclusions

In this paper, we propose an SSIM-based end-to-end distortion model for H.264/AVC video coding over packet-switched networks. This model is useful to estimate the content-dependent perceptual distortion of quantization, error concealment, and error propagation. We integrate the proposed end-to-end distortion model into the error-resilient video coding framework to optimally select the coding mode. Simulation results show that the proposed scheme outperforms the state-of-the-art schemes in terms of SSIM.