No reference quality assessment for MPEG video delivery over IP

Abstract

Abstract

Video delivering over Internet protocol (IP)-based communication networks is widely used in the actual information sharing scenario. As well known, the best-effort Internet architecture cannot guarantee an errorless data delivering. In this paper, an objective no-reference video quality metric for assessing the impact of the degradations introduced by video transmission over heterogeneous IP networks is presented. The proposed approach is based on the analysis of the inter-frame correlation measured at the output of the rendering application. It does not require any information on errors, delays, and latencies affecting the links and on the countermeasures introduced by decoders in order to face the potential quality loss. Experimental results show the effectiveness of the proposed algorithm in approximating the assessments obtained by using human visual system (HVS)-inspired full reference metrics.

1 Introduction

In the last decade, a fast market penetration of new multimedia services has been experienced. UMTS/CDMA2000-based videotelephony, multimedia messaging, video on demand over Internet, and digital video broadcasting are a growing share of nowadays’ economy. The large-scale spreading of portable media players indicates the increasing end user demand for portability and mobility. At the same time, media sharing portals and social networks are mostly based on user-contributed video content, showing a different perspective of user relation with digital media: from being media consumers to being part of content creation, distribution, and sharing. This information sharing evolution presents many challenges; in particular, Internet protocol (IP)-based multimedia services require a transparent delivery of media resources to end users that has to be independent on the network access, type of connectivity, or current network conditions [1]. Furthermore, the quality of the rendered media should be adapted to the end users’ system capabilities and preferences. An overview of the future challenges of video communication are addressed in [2] and in [3]. As can be noted, there are many factors that can prejudice the quality of the delivered video.

For this reason, the effectiveness of each video service must be monitored and measured for verifying its compliance with the system performance requirements, for benchmarking competing service providers, for service monitoring and automatic parameters setting, network optimization, evaluation of customer satisfaction, and adequate price policies setting.

To this aim, ad hoc designed tools have to be employed. In fact, the systems adopted for assessing the performances of traditional voice-based transmission systems are usually inadequate for evaluating the quality of multimedia data. In fact, a single training sequence comparison or the bit-wise error collection, which is used for measuring the quality of the received signal in the voice-based communication model, is not able to catch the dynamic feature of a video stream and its correlation with the overall experience of the final user.

To cope with this task, the research on video quality assessment has mainly been focused on the development of objective video quality metrics able to mimic the average subjective judgment. This task is of difficult solution being dependent on many factors as:

• Video characteristics and content: size, smoothness, amount of motion, of sharp details, and the spatial and temporal resolution

• Actual network conditions: congestion, packet loss, bit error rate, time delay, time delay variation

• Viewer’s condition: display size and contrast, processing power, available memory, viewing distance

• Viewer’s status: feeling, expectations, experience, involvement

These factors are often difficult or impossible to be measured, especially in real-time communication services. Furthermore, each factor has a different impact on the overall perceived distortion whose visibility strongly depends on the content (e.g., salt and pepper noise can be not noticeable in highly textured areas of a frame while it can become highly visible in the next uniform frame).

In this contribution, a no-reference metric for assessing the degradations introduced by transmission of an original video, encapsulated in a MPEG2 TS, over a heterogeneous IP network is presented. Variable channel conditions may cause both isolated and clustered packet losses resulting in data and temporal integrity loss at the decoder side. This could lead to the impossibility of decoding isolated or clustered blocks, tiles, and even entire frames. Considering the continuous increase of computing power of both mobile and wired terminals, a wide spread of error concealment techniques aimed at increasing the perceived quality can be expected. The proposed system can be considered blind with respect to errors, such as delays and latencies that affected the link, and to the concealment strategies implemented at the decoders to face the potential quality loss. The proposed method is based on the evaluation of perceived quality of video delivered by a packet switched network. These networks are characterized by loss, delayed, or out-of-sequence delivery of packets. The proposed metric is therefore valid for mobile networks exploiting IP, i.e., UMTS, long-term evolution (LTE), and LTE Advanced.

The rest of the paper is organized as follows. In Section 2, the state-of-the-art research addressing video quality metrics is addressed, while in Section 3, the model of the proposed metric is presented. In Section 4, the evaluation of the video distortion is described. In Section 5, the results of the performed experiments are presented based on two types of evaluation: comparison with the National Telecommunications and Information Administration (NTIA) scores and comparison with subjective tests. Finally, in Section 6, the conclusions are drawn.

2 State of the art

A set of parameters describing the possible distortion in a video has been defined and classified in ‘ITU-T SG9 for RRNR project’ [4]. Here, temporal distortion, temporal complexity, blockiness, blurring, image properties, activity, and structure distortion are independently evaluated and then linearly combined with the aim of reliably fitting the measured mean opinion score (MOS) with the calculated MOS. This general model has been recently improved in [5] taking into account the dynamic range distortion. Objective quality metrics can be classified according to different criteria [6–8]. One of this is the amount of side information required to compute a given quality measurement; depending on this, three classes of objective metrics can be described:

• Full reference metrics (FR): the evaluation system has access to the original media. Therefore, a reliable measure for video fidelity is usually provided. A drawback presented by these metrics is that they require the knowledge of the original signal at the receiver side.

• Reduced reference metrics (RR): the evaluation system has access to a small amount of side information regarding the original media. In general, certain features or physical measures are extracted from the reference and transmitted to the receiver as side information to help the evaluation of the video quality. The metrics belonging to this class may be less accurate than the FR metrics, but they are also less complex, and make real-time implementations more affordable.

• No-reference metrics (NR): the evaluation system has no knowledge of the original media. This class of metrics is the most promising in the context of video broadcast scenario since the original images or videos are in practice not accessible to end users.

The FR class includes all methods based on pixel-wise comparison between the original and the received frame. Among them, the most relevant example of objective metric is the peak signal-to-noise ratio (PSNR), which is widely used to perform a fast and simple quality evaluation. It is based on the computation of the ratio between the mean square error (MSE) between the image to be evaluated and the reference image, and the maximum range of pixel values. Even if the metrics belonging to this class are easy to compute, they do not always well correlate with quality as perceived by human observers. In fact, these metrics do not consider the masking effects of the human visual system (HVS) and each pixel degradation contributes to the overall error score even if the error is not perceived by a human subject.

A novel and effective approach has been proposed in [9] with the NTIA-video quality metric (VQM) which combines in a single score the perceptual impact of different video artifacts (block distortion, noise, jerkiness, blurring, and color modifications). The NTIA-VQM is a general purpose video quality model designed and tested in a wide range of quality and bit rates. It is based on a preprocessing consisting in spatial and temporal alignment of reference and impaired sequences, region extractions, gain, and offset correction. Following, feature extraction, spatio-temporal parameter estimation, and local indexes polling are performed for computing the overall quality score. The features considered in VQM are extracted from the spatial gradients of the luminance and chrominance components and from measuring contrast and temporal information extracted from the luminance component only. It has been validated by exploiting extensive subjective and objective tests. The high correlation with MOS shown in the performed subjective tests is the reason for the wide use of this metric as a standard tool (ANSI) for FR video assessment[10].

Similarly, moving picture quality metric (MPQM) [11] and its colored version color moving picture quality metric [12] are based on the assumption that the degradation of the quality of a video is strictly related to the visibility of the artifacts. In more detail, it is based on the decomposition of the original and of the impaired videos in visual channel, and on the distortion computation performed by considering the masking effect and the sensitivity contrast function. The main limitation of this, and similar methods based on error sensitivity model, is in the simple (often linear) model adopted for the HVS, which badly approximates the complex, non-linear, and still partially disclosed vision system.

A different approach is based on the hypothesis that the human brain is very efficient in extracting the structural information of the scene rather than the error information. Therefore, as proposed by Wang et al. in [13], a perceptual-based metric should be able to extract information about structural distortions. The SSIM metric, proposed in [13], shows a good correlation with the subjective scores obtained by campaigns of subjective tests. Other classical FR metrics inspired by the HVS are in the works by Wolf et al. [14] and Watson et al. [15], while a survey of available FR video quality metrics can be found in [16]. It is worth noticing that for an effective frame-to-frame comparison, both the original video and the one under test must be synchronized.

Recently, Shnayderman et al. [17] compared the singular value decomposition coefficients of the original and the coded signal, while in [18], the authors computed the correlation between the original and the impaired images after a 2D Gabor-based filter bank, based on the consideration that the cell of the visual cortex can be modeled as 2D Gabor functions. As can be noticed, these metrics can also be applied to videos by computing a frame by frame quality evaluation.

As previously stated, the usability of FR metrics is often limited in real scenarios due to the need for availability of the original video. Nevertheless, being the perceived quality dependent on the content, it cannot be directly inferred from the knowledge of parameters such as channel reliability and temporal integrity. To partially overcome this problem, RR and NR quality metrics have been devised.

With respect to FR metrics, only few attempts of RR and NR metrics have been presented in literature. Among RR ones, Carnec et al. in [19] present a still image metric based on color perception and masking effects, resulting in a small overhead. Blocking, blurring, ringing, masking, and lost blocks are linearly combined in [20] for a frame by frame comparison. Wang and Simoncelli in [21], based on the frequency distribution characteristics of natural images, proposed to use the statistics of the coded image to predict the visual quality.

Different approaches are proposed by Kanumuri et al. in [22]: the RR metric is based on a two-step approach. The information gathered from the original, received, and decoded video are used in a classifier whose output will be used in the evaluation of artifact visibility performed on a decision tree trained by subjective tests. Similarly, in [23], a general linear model (GLM) is adopted for estimating the visibility threshold of packet loss in H.264 video streaming. In [24], GLM is modified by computing a saliency map, for weighting the pixel-wise errors, and by taking into account the influence of the temporal variation of saliency map and packet loss. The results show that if the HVS features are considered, the prediction of subjective scores is improved.

Finally, a novel approach is represented using different communication systems for delivering information on the original media. In this classification are, for example, the data hiding-based RR metrics. In these approaches, a thumbnail of the original frame [25–28], a perceptually weight watermark [29], or a particular image projection [30] are used in the quality evaluation as fingerprint of the original frame quality. The main vulnerability of these methods is in the robustness of the watermarking method. In fact, any alteration, wanted or not, of the inserted data may strongly affect the objective assessment.

The need for the reference video or for partial information about it is a considerable drawback in many real-time applications. For this reason, the design of effective NR metrics is a big challenge. In fact, although human observers are able to assess the quality of a video without using the reference, the creation of a metric that could mimic such a task is difficult and, most frequently, it results in a loss of performances in comparison to the FR/RR approaches. To achieve effective evaluations, many existing NR metrics estimate the annoyance by detecting and estimating the strength of common artifacts (e.g., blocking and ringing).

NR techniques are the most promising because their final score can be considered, for an ideally perfect metric, as an absolute quality value, independent from the knowledge of the original content. Few metrics have been designed for the evaluation of impairments due to single artifacts as blockiness [31], blurriness [32], and jerkiness [33].

Different strategies have been proposed to evaluate the impact of impairments caused by compression algorithms and transmission over noisy channels. These can be classified according to the parts of the communication channel that are involved:

• Source coder errors: MSE estimation due to compression [34], example-based objective reference in [35], motion-compensation edge artifacts in [36];

• Variable delay [37, 38];

• Packet loss effects [39–41]. In [41], the NR metric is based on the estimation of mean square error propagation among the group of picture (GOP) in motion compensation-based encoded videos. The idea is to consider the motion activity in the block as initial guess of the distortion caused by the initial packet loss;

• Bitstream-based video quality metric: in these systems, several bitstream parameters, such as motion vector length or number of slice losses, are used for predicting the impairments visibility in MPEG-2, SVC [42], or H.264 [43, 44] and HD H.264 [45] videos. Recently, the bitstream metric proposed in [46] has been modified with pixel-based features to cope with HDTV stream [47];

• Rendering system errors: [48–50].

Other examples include the works presented by Webster et al. [51] and Brétillon et al. [52]. The estimation of the pattern of lost macroblocks based on the knowledge of the decoded pixels is used as input to a no-reference quality metrics for noisy channel transmission. The metrics by Wu and Yuen and Wang et al. estimate quality based on blockiness measurements [31, 53], while the metric by Caviedes and Jung takes into account measurements of five types of artifacts [54]. Recently, in [55], a methodology for fusing metrics feature for assessing video quality has been presented. This work has also been adopted in the ITU-T Recommendation P.1202.2.

3 The NR procedure

In the following, the motivations behind each step of the NR procedure are briefly described and then detailed in Subsections 3.1 to 3.4. As previously stated, channel errors and end-to-end jitter delays can produce different artifacts on the received video. The effect of these errors can have a dramatic impact on the quality perceived by users since the loss of a single packet can result in a corrupted macro-block. Corrupted information can affect both spatial (to neighboring blocks) and temporal (over adjacent frames) quality due to the predictive, motion-compensated coding scheme adopted by most of existing video coders. The visual impact of these errors strictly depends on the effectiveness of the decoder scheme and on the concealment strategy that is adopted.

In order to recover transmission errors, decoders can exploit several strategies depending on the error resilience or concealment techniques adopted in the communication scheme. Error resilience is based on the addition of redundant information at the encoder side for allowing the decoder to recover some transmission errors: the drawback is the increase in the amount of transmitted data. On the other hand, error concealment is a post-processing technique in which the decoder tries to mask the impairments caused by packet losses and bit stream errors that have been detected but not corrected. In this case, even if the quality of the recovered data is usually lower than the original one, the system does not require encoder/decoder modification or extra information delivering. Several concealment techniques have been proposed in literature whose effectiveness increases with complexity. The simplest proposed strategy consists in filling the missing areas with a constant value or with information extrapolated by considering the last correctly decoded block. More sophisticated techniques apply prediction/interpolation of the lost block(s) by exploiting spatial and temporal redundancy [28]. Concealment effectiveness is largely affected by the spatial and temporal extension of the missing information, with best performances obtained in the case of small clusters and isolated blocks. An example of visual artifacts on a test sequence ‘Brick’, when transmitted on a noisy channel affected by a 15% packet loss, is shown in Figure 1. When an error affects the entire frame or a large part of it, the decoder may decide to completely drop the corrupted frame and to freeze the last error-free video frame until a new valid frame is correctly decoded. In this case, the perceived quality of the played video will depend on the dynamics of the scene. In fact, although only error-free frames are played, the motion of objects composing the scene may appear unnatural, due to its stepwise behavior (jerkiness effect). The dropping mechanism can also be caused by a playback system that is not fast enough to decode and display each video frame at full nominal speed. It is worth noticing that the same experience is perceived in the presence of frame freezing artifacts or by repetition of the same frame.

Figure 1

Impact of a noisy channel (15% PLR) on the transmission of a test video sequence. In particular, by considering the rendered frame (A) versus the original one (B), several artifacts can be noted: isolated blocks, repeated lines, blurring and wrong color reconstruction.

The NR metric proposed in this paper is independent from the error concealment techniques implemented in the video player; however, since frame repetition is a very common concealment method, here, the assessment of the quality loss produced by freezing the video in correspondence of frame losses is specifically addressed. More in details, before applying the NR jerkiness metric proposed by the authors in [56], the played sequence is analyzed in order to detect the presence of repeated frames.

To this aim, the rendered sequence is first partitioned into static and dynamical shots, on the basis of the amount of changes between consecutive frames. Next, the shots classified as static are evaluated in order to detect if the identified small amount of changes corresponds to a real static scene or to the freeze of entire frames or part of them. At the same time, the dynamical shots are tested to verify the presence of isolated and clustered corrupted blocks. These analyses result in temporal variability and spatial degradation maps that are used to assess the video quality by evaluating the overall distortion as shown in Figure 2. In the following, the details of the proposed system are presented.

Figure 2

Block diagram of the proposed metric.

3.1 Frame segmentation in dynamic and static shots based on a global temporal analysis

As previously described, the first step of the NR procedure is the grouping of frames in dynamic or static shots based on a temporal analysis.

Let F={F _k ,k=1,…,L} denote a video sequence composed by L frames of m×n pixels.

The generic k th frame can be partitioned in N _r ×N _c blocks ${\mathbf{\text{B}}}_k^{(i,j)}$ of r×c pixels with top left corner located in (i,j). Let ${\overline{\mathbf{\text{F}}}}_k$ be the mean luminance value for the k th frame and ${\overline{\mathbf{\text{B}}}}_k^{(i,j)}$ the mean luminance value of block ${\mathbf{\text{B}}}_k^{(i,j)}$. Let $\Delta {\mathbf{F}}_k={\mathbf{F}}_k-{\overline{\mathbf{\text{F}}}}_k$ and $\Delta {\mathbf{B}}_k^{(i,j)}={\mathbf{B}}_k^{(i,j)}-{\overline{\mathbf{\text{B}}}}_k^{(i,j)}$ denote the deviation of the luminance of the k th frame and of the block ${\mathbf{\text{B}}}_k^{(i,j)}$ from the corresponding mean values.

The normalized inter-frame correlation coefficient ρ _k between the k th and the (k-1)th frames is defined as:

$${\rho }_k=\frac{〈\Delta {\mathbf{F}}_k,\Delta {\mathbf{F}}_{k-1}〉}{{∥\Delta {\mathbf{F}}_k∥}_{L_2}{∥\Delta {\mathbf{F}}_{k-1}∥}_{L_2}},$$

(1)

where <∙,∙> denotes the inner product and ${∥\bullet ∥}_{L_2}$ the L₂ norm. Similarly, the inter-block correlation ${\rho }_k^{B(i,j)}$ can be computed as:

$${\rho }_k^{\mathbf{B}(i,j)}=\frac{〈\Delta {\mathbf{B}}_k^{(i,j)},\Delta {\mathbf{B}}_{k-1}^{(i,j)}〉}{{∥\Delta {\mathbf{B}}_k^{(i,j)}∥}_{L_2}{∥\Delta {\mathbf{B}}_{k-1}^{(i,j)}∥}_{L_2}}.$$

(2)

It is possible to group the frames into static and dynamical shots by comparing the inter-frame correlation ρ _k ,k=1,…,L, with a threshold λ _s :

$$\left\{\begin{array}{cc}{\rho }_{{}_{\mathrm{k}}}<{\lambda }_S:& \text{dynamical}\text{shot}\\ {\rho }_{{}_{\mathrm{k}}}>{\lambda }_S:& \text{static}\text{shot}\end{array}\right.$$

(3)

where the threshold λ _s is set to the equal error rate (EER) between the classification of a static block as dynamic and vice versa.

As illustrated in Figure 3, the inter-frame correlation presents a spiky behavior with values close to one in correspondence of frozen frames. It is important to underline that the detection of such a behavior is not sufficient to identify a partial or total frame loss. In fact, in the case of static scenes, consecutive frames present a high inter-frame correlation.

Figure 3

Test video sequence affected by 6% packet loss. (A) The normalized interframe correlation among the first 100 frames extracted from the original video sequence Taxi and the first 100 frames extracted from the same sequence affected by 6% packet loss. (B) One frame extracted by the video sequence.

Therefore, it is important to be able to distinguish between frames that are affected by errors and the ones belonging to a static scene. This can be achieved by using a system for assessing the presence of jerkiness. In fact, jerkiness is the phenomenon that leads to perceive a video as consisting of a sequence of individual still images. In this contribution, we adopt the approach that has been presented in [56].

After the segmentation into dynamic and static shots, the task of quality evaluation gets easier. In fact, for static shot sequences, it is possible to evaluate the quality of the first frame and to extend the obtained score to the frames belonging to the static cluster. In this way, a degradation map is computed for the first frame (that can still be affected by artifacts) and is inherited by the frames belonging to the same static shot. When dealing with the distortion associated to isolated and clustered impaired blocks, it is estimated by means of a two-step procedure based on temporal and spatial degradation analysis. In the following, this will be referred to as ‘degradation map’ computation.

3.2 Local temporal analysis

The local temporal analysis is performed in two stages. The aim of the first one is to identify and to extract from each frame the blocks that are potentially affected by artifacts. This analysis is performed by classifying the blocks as:

• With medium content variations

• Affected by large temporal variations

• With small content variations

depending on their temporal correlation ${\rho }_k^{B^{(i,j)}}$.

The corresponding temporal variability map ${\mathbf{\Gamma }}_k^V=\left\{{\Gamma }_k^{{\text{VB}}^{(i,j)}}\right\}$ is computed by comparing the inter-frame correlation of each block with two thresholds θ _l and θ _h :

$${\Gamma }_k^{{\text{VB}}^{(i,j)}}=\left\{\begin{array}{cc}1,& \mathit{\text{if}}{\rho }_k^{B^{(i,j)}}<{\theta }_l\\ 0,& \mathit{\text{if}}{\theta }_l\le {\rho }_k^{B^{(i,j)}}\le {\theta }_h\\ 2,& \mathit{\text{if}}{\rho }_k^{B^{(i,j)}}>{\theta }_h\end{array}\right.$$

(4)

The selection of the two thresholds, θ _l and θ _h , is performed based on the assumption that:

• The correlation, between corresponding blocks belonging to consecutive frames, is close to one in the presence of a repeated block or of a block belonging to a static region.

• The correlation value is close to zero in case of a sudden content change (usually occurring after shot boundaries) or in the presence of an error.

For this reason, as can be noted in Equation 4, the highest temporal variability index is assigned to blocks considered as unchanged from the previous frame, while zero distortion index is assigned to blocks with medium content variation. In more details, let us define with probability of false alarm (P_fa) the probability of detecting the repeated blocks as affected by errors in the absence of distortion and with probability of miss detection (P_md) the probability of considering as unaltered a frame in the presence of errors. The two thresholds, θ _l and θ _h have been selected in order to grant

$$|P_{\text{fa}}-P_{\text{md}}|<{\epsilon }_1$$

where ε₁ has been experimentally determined, during the training phase, by comparing the performances achieved by the temporal analysis algorithm with the scores provided by a group of video quality experts in an informal subjective test.

3.3 Spatial analysis

The blocks classified as potentially affected by packet loss during the temporal analysis phase undergo a spatial analysis. The spatial analysis is performed in several steps:

• Static regions detection: it aims at verifying whether a high correlation between the current block ${\mathbf{B}}_k^{(i,j)}$ and the previous one ${\mathbf{B}}_{k-1}^{(i,j)}$ is due to the loss of a single or multiple blocks or to a static region. To perform this task, for each block with ${\Gamma }_k^{{\text{VB}}^{(i,j)}}=2$, it is checked if at least v among the surrounding blocks present a strong temporal correlation. In case of positive result, the block is classified as belonging to a static region and its potential distortion index ${\Gamma }_k^{{\text{CB}}^{(i,j)}}$ is set to zero. The parameter v has been identified by experimental test. Practically, a set of expert viewers has been presented with a set of short videos presenting different content situations affected by increasing blocking artifacts. The parameter v has been selected as the one resulting in the highest correlation between the people score and the algorithmically performed spatial analysis block. That is:

  $$\begin{array}{c}{\Gamma }_k^{{\text{CB}}^{(i,j)}}=\left\{\begin{array}{cc}0& \text{if}\left|\right\{(p,q)\left|\right(p,q)\in N(i,j),\text{and}{\Gamma }_k^{{\text{VB}}^{(p,q)}}=2\}|>\nu \\ {\Gamma }_k^{{\text{VB}}^{(i,j)}}:& \text{otherwise}\end{array}\right.\end{array}$$

  (5)

• Edge consistency check: the presence of edge discontinuities in block boundaries can be used as an evidence of distortions. For the sake of simplicity, we detail the procedure for the case of gray scale images. It can be easily extended to the color case by evaluating separately the edge consistency for each color component. Let E_l and E_r be the L₁ norms of the vertical edges, respectively, on the left and on the right boundary of the block, and with A_c, A_l and A_r the average values of the L₁ norms of the vertical edges inside the current block and of the left and right adjacent blocks. A block with ${\Gamma }_k^{{\text{CB}}^{(i,j)}}\ne 0$ is classified as affected by visible distortion if:

  $$\left|E_{\mathrm{l}}-\frac{\left(A_{\mathrm{c}}+A_{\mathrm{l}}\right)}{2}\right|>\theta \text{or}\left|E_{\mathrm{r}}-\frac{\left(A_{\mathrm{c}}+A_{\mathrm{r}}\right)}{2}\right|>\theta$$

  (6)

• where the threshold θ has been defined on the basis of experimental trials. In particular, it corresponds to just noticeable distortion collected evaluated for the 90% of subjects. The same procedure is then applied to the horizontal direction. If the block edges are consistent (i.e., no visible distortion has been detected along horizontal and vertical directions), ${\Gamma }_k^{{\text{CB}}^{(i,j)}}$ is reset to 0.

• Repeated lines test: it is performed to detect frames that have been partially correctly decoded. A very common concealment strategy is based on the fact that when the packet loss affects an intra-frame encoded image, and a portion of the frame is properly decoded, the remaining part is replaced with the last row correctly decoded. As can be noted in Figure 4, the procedure results in a region containing vertical stripes.

• Let f _k [ i] be the i th row of the k th frame. Starting from the m th line of the frame, the L₁ norm of the horizontal gradient component is computed and compared to a threshold λ _H . If

  $${∥\Delta f_k\left[i\right]∥}_{{\mathrm{L}}_1}>{\lambda }_{\mathrm{H}},$$

  (7)

• the procedure is repeated on the previous line (i-1) to check if consecutive lines are identical by comparing the L₁ norm of their difference with a threshold λ _V

  $${∥f_k\left[i\right]-f_k[i-1]∥}_{{\mathrm{L}}_1}<{\lambda }_{\mathrm{V}}.$$

  (8)

• This procedure is iterated until the test fails, thus meaning that there is a different information carried out by consecutive lines. After the repeated lines test has been performed, a binary spatial degradation map, ${\Gamma }_k^{{\mathrm{RLB}}^{(i,j)}}$ of [0,1] entries, is created where ‘1’ corresponds to a block belonging to a vertical stripes region and ‘0’ otherwise. The two thresholds, λ_V and λ_H have been set after a training process with a pool of experts trying to match the subjective impression of repeated lines.

Figure 4

Frame affected by vertical stripes.

3.4 Reference frame detection

The previous procedure allows to assess the presence of blocks belonging to the current frame which are affected by distortions caused by packet loss. Nevertheless, due to error propagation, the impairment can propagate until an intra-frame encoded image (I-frame), is received. Figure 5 shows the normalized inter-frame correlation of a sequence extracted from an action movie. As can be noted, an I-frame is usually characterized by a low correlation with the previous frame and a high correlation with the next frame. This behavior is always verified unless the same scene is shown for a long period.

Figure 5

Normalized inter-frame correlation of the original sequence.

Let us denote with ${\nu }_k^{\text{CB}}$ the number of corrupted blocks, i.e.,

$${\nu }_k^{\text{CB}}=\left|\right\{{\Gamma }_k^{{\text{CB}}^{(i,j)}}\ne 0\left\}\right|.$$

(9)

Then, the k th frame is classified as an I-frame if

$${\rho }_{k-1}-{\rho }_k>2{\eta }_P\text{and}{\rho }_{k+1}-{\rho }_k>2{\eta }_S$$

(10)

and no more than P _p out of the Q _p previous frames and no more than P _n out of the Q _n following frames are characterized by a number ${\nu }_k^{\text{CB}}$ of blocks with inconsistent edges exceeding a threshold λ _I .

The decision thresholds are adapted to the current video content. In particular, η _P and η _S are proportional to the mean absolute differences of the correlation coefficients in the intervals [ k-M _l ,k] and [ k,k+M _h ], i.e.:

$${\eta }_P=\frac{1}{M_l}\sum _{h=k-M_l+1}^k\left|{\rho }_h-{\rho }_{h-1}\right|$$

(11)

and

$${\eta }_S=\frac{1}{M_h}\sum _{n=k+1}^{k+M_h}\left|{\rho }_n-{\rho }_{n-1}\right|.$$

(12)

The value M _l is selected to guarantee that the time interval needed for the adaptation of η _P starts at the frame following the last correctly detected I-frame. When processing the k th frame, no information about location of next I-frames is available and the length of the interval employed for the adaptation of η _S is considered to be constant. When the time interval between two I-frames is less than M _h , only the I-frame with the lowest correlation with the previous frame is retained.

4 Distortion map evaluation

The evaluation of the video quality metric VQM _{N R} is based on the degradation index maps ${\Gamma }_k^{{\text{RLB}}^{(i,j)}}$ and ${\Gamma }_k^{{\text{CB}}^{(i,j)}}$ whose computation has been illustrated in the previous section. To account for the error propagation induced by predictive coding, a low-pass temporal filtering is applied to the degradation index maps. To this aim, let D_k-1 denote the generic distortion map at time (k-1); at time k, the distortion map D _k of frames belonging to dynamical shots is evaluated as follows:

$${\mathbf{\text{D}}}_k^{\text{CB}}=\mu \left[{\mathbf{\Gamma }}_k^{\text{CB}}+\phi \left({\rho }_k\right){\mathbf{\text{D}}}_{k-1}^{\text{CB}}\right];$$

(13)

$${\mathbf{\text{D}}}_k^{\text{RLB}}=\mu \left[{\mathbf{\Gamma }}_k^{\text{RLB}}+\phi \left({\rho }_k\right){\mathbf{\text{D}}}_{k-1}^{\text{RLB}}\right]$$

(14)

where μ(x) is a non-linearity shown in Figure 6 and defined as follows:

$$\mu \left(x\right)=\left\{\begin{array}{cc}0& x<\gamma \\ x& \gamma \le x<2\\ 2& x\ge 2\end{array}\right.$$

(15)

Figure 6

Non linearity μ(x) used for distortion map evaluation.

This non-linearity shrinks small distortions and allows to account for saturation in case of consecutive degradations of the same block through an operation of hard limiting.

The number of frames to be low-pass filtered is determined by the inter-frame correlation and in the following it will be indicated as φ.

More in details:

• For a given block b(i,j), φ is set to zero if the corresponding block in the previous frame is affected by repeated line distortion and the inter-block correlation is below a predefined threshold (i.e., ${\rho }_k^{B^{(i,j)}}<{\lambda }_{\text{RLB}}$) indicating that the block has been updated by I-frame coding.

• φ is set to zero when processing I-frames.

• φ is set to one for frames belonging to static shots.

In order to evaluate the overall distortion index, the map ${\mathbf{\text{D}}}_k^{\text{CB}}$ of corrupted blocks is decomposed into two groups: the first one, denoted in the following with ${\mathbf{\text{D}}}_k^{\text{CCB}}$, contains the entries of ${\mathbf{\text{D}}}_k^{\text{CB}}$ associated to clustered corrupted blocks, while the second one, denoted in the following with ${\mathbf{\text{D}}}_k^{\text{ICB}}$, contains the contributions corresponding to the remaining, isolated, blocks. A block b(i,j) for which $D_k^{\text{CB}}\left[i,j\right]>0$ is considered member of a cluster if at least for one of its eight surrounding neighbors, b(p,q), the condition $D_k^{{\text{CB}}^{(p,q)}}>0$ holds. Let •

$${\mathbf{N}}_k^{\text{CCB}}={∥{\mathbf{\text{D}}}_k^{\text{CCB}}∥}_{L_1}$$

be the the L₁ norm of clustered corrupted block map•

$${\mathbf{N}}_k^{\text{ICB}}={\eta }^{\text{ICB}}\left({∥{\mathbf{\text{D}}}_k^{\text{ICB}}∥}_{L_1}\right)$$

be the L₁ norm of isolated corrupted blocks where

$${\eta }^{\text{ICB}}\left(x\right)=\left\{\begin{array}{cc}0& x\le {\lambda }_{\text{ICB}}\\ x& \text{otherwise}\end{array}\right.$$

(16)

•

$${\mathbf{N}}_k^{\text{RL}}={∥{\mathbf{\text{D}}}_k^{\text{RLB}}∥}_{L_1}$$

be the column vector of the number of repeated lines for each image color component

• ρ^LOSS the packet loss rate

Then, denoting with ξ is the column vector

$$\mathit{\xi }={\left[\begin{array}{ccccc}1& {\overline{N}}^{\text{CCB}}& {\overline{N}}^{\text{ICB}}& {\overline{N}}^{\text{RL}}& {\rho }^{\text{LOSS}}\end{array}\right]}^T$$

(17)

where

$${\overline{N}}^{\text{CCB}}=\frac{1}{L}\sum _{k=1}^L{N}_k^{\text{CCB}},$$

(18)

$${\overline{N}}^{\text{ICB}}=\frac{1}{L}\sum _{k=1}^L{N}_k^{\text{ICB}},$$

(19)

$${\overline{N}}^{\text{RL}}=\frac{1}{L}\sum _{k=1}^L{N}_k^{\text{RL}},$$

(20)

are the average values of the corresponding L₁ norms and L is the length of the video sequence, the NR metric ${\text{VQM}}_{\text{NR}}^{\left(\mathrm{Y}\right)}$ based on the luminance component can be computed as follows:

$${\text{VQM}}_{\text{NR}}^{\left(\mathrm{Y}\right)}=\alpha {\left[{\mathit{\xi }}^T{Q}^{\left(Y\right)}\mathit{\xi }\right]}^{1/2}+\mathrm{\beta .}$$

(21)

The weighting matrix Q^(Y) (Table 1) and the regression coefficients α and β can be estimated by fitting subjective experiments, as illustrated in the next section.

Table 1 Weighting matrix Q ^(Y)

We remark that, since ξ₁=1, the quadratic form includes both linear and quadratic terms.

The above relationships can be directly extended to color images by building a degradation map for each color component. Therefore, assuming that each frame is represented by the luminance Y and color difference components C _b , C _r , the proposed video quality metric specifies as follows

$${\text{VQM}}_{\text{NR}}^{(Y,C_b,C_r)}={\alpha }_c{\left[{\mathit{\zeta }}^T{Q}^{(Y,C_b,C_r)}\mathit{\zeta }\right]}^{1/2}+{\beta }_c.$$

(22)

where ζ is the column vector:

$$\begin{array}{c}\mathit{\zeta }=\left[1{\overline{N}}_Y^{\text{CCB}}{\overline{N}}_Y^{\text{ICB}}{\overline{N}}_Y^{\text{RL}}{\overline{N}}_{C_b}^{\text{CCB}}{\overline{N}}_{C_b}^{\text{ICB}}\right.\\ {\left.{\overline{N}}_{C_b}^{\text{RL}}{\overline{N}}_{C_r}^{\text{CCB}}{\overline{N}}_{C_r}^{\text{ICB}}{\overline{N}}_{C_r}^{\text{RL}}{\rho }^{\text{LOSS}}\right]}^T\end{array}$$

(23)

having demoted with •

$${\overline{N}}_Y^{\text{CCB}},{\overline{N}}_{C_b}^{\text{CCB}},{\overline{N}}_{C_r}^{\text{CCB}}$$

the average numbers of clustered corrupted blocks for the three color components,•

$${\overline{N}}_Y^{\text{ICB}},{\overline{N}}_{C_b}^{\text{ICB}},{\overline{N}}_{C_r}^{\text{ICB}}$$

the average numbers of isolated corrupted blocks for the three color components,•

$${\overline{N}}_Y^{\text{RL}},{\overline{N}}_{C_b}^{\text{RL}},{\overline{N}}_{C_r}^{\text{RL}}$$

the average numbers of repeated lines for the three color components.

It is important to notice that the reduction of the impact of isolated corrupted blocks allows to mitigate the effects produced by misclassifications and to account for the lower sensitivity to artifacts in small areas compared to those in wider areas.

5 Experimental results

To identify the parameters specifying a NR metric and to verify its effectiveness, experiments involving human subjects should be performed. As already stated, this procedure is expensive, time-consuming, and often impossible to be performed. The alternative is to compare, under the same testing conditions, the gathered results with those provided by reliable full reference metrics. In the performed test, the NTIA video quality metric (VQM_NTIA) whose software implementation is publicly available and freely downloadable at the URL http://www.its.bldrdoc.gov/vqm, has been adopted.

5.1 Experimental setup

The experimental setup is shown in Figure 7, and it is composed by a streaming source, a network segment, and a receiver. The video server consists of a personal computer equipped with the open source VideoLAN server [57] and the FFmpeg tool [58]. The original video is encapsulated in a MPEG2 TS, packetized in the RTP/UDP/IP protocol stack and transmitted on a 100/1000Base-T Ethernet interface. The network segment has been accounted for by means of an open source network emulator: NETwork EMulator (NETEM) [59]. The emulator has been used for introducing packet losses in the incoming stream, in accordance to the statistics of real networks based on the best-effort paradigm.

Figure 7

Block diagram of the experimental setup.

Each considered media stream has been processed in order to simulate a set of increasing packet loss rates (PLRs). The selected PLRs are: 0.1%, 0.5%, 0.7%, 0.9%, 1.1%, 1.3%, 1.5%, 2.0%, 3.0%, 5.0%, 10%, 15%, and 20%. At the receiver side, the VLC client receives the media packets and uses concealment techniques for reconstructing the original video.

To evaluate the increase in VQM_NR performance achievable when full color information is employed with respect to the use of the luminance alone, the parameter identification and the performance assessment have been performed for both gray scale and color videos. Two sets of sequences have been used in our tests. The first one (test set 1) is composed by eight video sequences and it has been used for calibrating the ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ metric parameters. The sequences are of different content and characterized by still scenes and slow or fast motion rates. The second one (test set 2) has been used for evaluating the effectiveness of the proposed metric. The sequences have been extracted from the online database ‘The Consumer Digital Video Library’ [60]. All analyzed videos have a VGA resolution (640 × 480 pixels, progressive) and a frame rate equal to 30 fps and they are composed by 360 frames. The test set 2 video dataset characteristics are reported in Table 2 while sample frames from the videos are shown in Figure 8.

Table 2 Video dataset characteristics

Figure 8

Sample frames extracted from the videos in the dataset used in the experimental tests. (A) Bells. (B) Cargas. (C) Cartalk. (D) Catjoke. (E) Diner. (F) Drmset. (G) Fish. (H) Guitar. (I) Magic. (J) Music. (K) Rfdev. (L) Schart. (M) Wboard.

5.2 Gray scale video tests

The luminance component of a the test set 1 training set has been used for calibrating the ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ metric parameters. In this phase, the goal has been to mimic the VQM_NTIA score computed on the training sequences as much as possible. Based on the achieved results, the thresholds θ _l and θ _h in Equation 4 have been set to 0.3 and 0.9, respectively. The parameters λ _H and λ _V , defined in Equation 7 and 8 have been set to 5 and 1, respectively. From the performed test, it can be noticed that λ _V , although small, is not null to account for small variations induced by partial decoding of a tile affected by errors. The length of the interval employed for the adaptation of η _S (as in Equation 12) is considered constant and M _h =7 has been employed in the reported results. The parameters λ _I , P, and Q, as defined in Equation 10 have been set to 0.25, 2, and 5, respectively.

The capability of the proposed metric to mimic the behavior of the VQM_NTIA for the training set is illustrated in Figures 9, 10, and 11. In more details, in Figure 9, the results concerning the Taxi sequence affected by three packet loss rates (1.9% top row, 3% middle row, and 6.7% bottom row) are reported. As can be noticed from the plots, the proposed metric scores are coherent with the VQM_NTIA ones, especially for the PLR = 3% and PLR = 6.7% cases.

Figure 9

${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ scores versus VQM ones for the Taxi sequence. (A) PLR = 1.9%. (B) PLR = 3%. (C) PLR = 7%.

Figure 10

Degraded version of the Taxi sequence. Original (A) and impaired (B) version of the frame number 85 of the Taxi sequence.

Figure 11

Field sequence: ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ versus VQM one at PLR = 6.7%.

It is worth noticing that in Figure 9A, around frame 85, the VQM_NR presents a peak not corresponding to a similar quality variation detected by the VQM_NTIA. This behavior highlights the differences between the two metrics. As can be easily noticed by a visual inspection of the considered frames in the Taxi sequence and in its degraded version in Figure 10, there are errors resulting in block artifacts affecting both the main object and the road curbs. In this case, the proposed metric is able to cope with the masking effect of textures and with the perceived impact of silhouette definition and text readability.

The same behavior can be noticed for the sequence Field as reported in Figure 11. For almost the whole sequence, the two indexes show the same behavior. There is a slight tendency in overestimating the video artifacts by the ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ index. Only for a few frames, the quality assessments provided by the two metrics are opposite: the value is over or below the quality threshold.

In the sequence Horse ride, the overlapping between the two curves is not homogeneous, as shown in Figure 12. Moreover, if the average behavior is compared, among the 25th and the 38th frame, the ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ indicator shows high degradation while VQM_NTIA only shows a slight degradation. The same different degradation rate can also be noticed in the last part of the sequence.

Figure 12

Horse ride sequence: ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ versus VQM at PLR = 6.7%.

In order to evaluate the performances of the gray scale ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ with respect to the quality estimation provided by the full color VQM_NTIA metric, the test set 2 has been employed.

For comparing the performance achieved with the proposed gray scale and full color no-reference metrics, a Monte Carlo simulation of the transmission of the set of full color videos over an IP channel affected by packet losses for several packet loss rates has been performed. Then, only the luminance component of the decoded videos has been employed for computing ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ while both luminance and color differences have been employed for computing both ${\text{VQM}}_{\text{NR}}^{(Y,c_b,C_r)}$ and VQM_NTIA.

In Figure 13, the results obtained for each sequence with the ${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ have been plotted versus VQM_NTIA ones. As can be noticed, there is good matching between the two metrics and the root mean square error (RMSE) value is 0.14. The regression value is 0.86.

Figure 13

${\text{VQM}}_{\text{NR}}^{\left(Y\right)}$ scores versus VQM _NTIA ones for the gray scale sequences. Regression value = 0.86 and RMSE = 0.14.

5.3 Color video tests

To verify the gain achieved when chrominance is employed, the multivariate regression procedure has then been applied to the full reference metric and to the packet loss rate, the average numbers of clustered corrupted blocks, isolated corrupted blocks, and repeated lines extracted from Y, C _b , and C _r decoded components, thus, obtaining the regression coefficients α _c = 1.0419 and β _c =-0.0465. The weighting matrix is reported in Table 3.

Table 3 Weighting matrix Q

In Figure 14, the plot of ${\text{VQM}}_{\text{NR}}^{(Y,C_b,C_r)}$ versus the VQM_NTIA is reported for the selected videos. The plot shows improved performances of the proposed metric in matching the full reference score. In fact, the use of color information increases the fitting performances resulting in a regression value of 0.91 and on a RMSE value equal to 0.11.

Figure 14

${\text{VQM}}_{\text{NR}}^{(Y,C_b,C_r)}$ scores versus VQM _NTIA ones for color videos. Regression value = 0.91 and RMSE = 0.11.

By analyzing the results, a few issues are open for future investigation. First of all, from the performed experiments for both metric parameters tuning and metric performance effectiveness assessment, a key factor for a successful comparison of NR and FR metrics is represented by the temporal realignment algorithm. In the presence of highly textured backgrounds, severe frame losses, and medium to high compression ratios, at least our implementation of the VQM_NTIA algorithm does not provide reliable estimates of the variable delay between the original and decoded videos. This implies a potential bias in the estimated NR metrics induced by the wrong selection of the reference frame to be used for the comparison. Furthermore, we noticed that the adopted key-frame detection algorithm has an impact on overall distortion evaluation, since many elements, considered in the proposed metric, depend on the shot boundaries detection.

5.4 Subjective experiment

Finally, in order to further verify the effectiveness of the proposed metric, a subjective experiment has been performed.

Sixteen test subjects drawn from a pool of students of the University of Roma TRE have participated to the test. The students are thought to be relatively naive concerning video artifacts and the associated terminology. They were asked to wear any vision correcting devices (glasses or contact lenses) they normally wear to watch television. The subjects were asked to rate the quality of the videos in the test database (listed in Table 2) through a single stimulus quality evaluation method [61].

A Panasonic Viera monitor (46") is used to display the test video sequences. The experiment is run with one subject at a time. Each subject was seated straight ahead in front of the monitor, located at or slightly below eye height for most subjects. The subjects are positioned at a distance of four screen heights (80 cm) from the video monitor in a controlled light environment. The experimental session consisted of four stages. In the first stage, the subject was verbally given instructions for performing the test. In the second stage, training sequences were shown to the subject. The training sequences represent the impairment extremes for the experiment and are used to establish the annoyance value range. In the third stage, the test subjects run through several practice trials. The practice trials are identical to the experimental trials and are used to familiarize the test subject with the experiment. Finally, the experiment is performed on the complete set of test sequences. After each video was displayed, the subject was asked to enter his/her judgment in a scale from 1 to 5, where 5 corresponds to best quality and 1 to worst quality.

In Figure 15, the comparison between the collected MOS and the two objective metrics are reported. The MOS has been normalized in the range 0 (best quality) to 1 (worst quality). The RMS is 0.10 for the ${\text{VQM}}_{\text{NR}}^{(Y,C_b,C_r)}$ metric and 0.16 for VQM_NTIA metric, respectively. As can be noticed, the proposed metric is able to predict the subjective judgment.

Figure 15

Normalized MOS versus ${\text{VQM}}_{\text{NR}}^{(Y,C_b,C_r)}$ and VQM _NTIA metrics.

6 Conclusions

In this paper, a no-reference metric for assessing the quality of MPEG-based video transmissions over IP-based networks is presented. The proposed approach is based on the analysis of the inter-frame correlation measured at the receiver side. Several tests have been performed for tuning and evaluating the performances of the proposed metric. The scores collected by this tool in evaluating impaired videos have been compared with the ones gathered with the full reference VQM_NTIA metrics and with the MOS collected by means of a subjective experiment. The overall analysis demonstrates the effectiveness of the VQM_NR. Current investigation is devoted to solve the problems arising when using evaluation methods that are not based on reference signals. In particular, for the temporal realignment algorithm that is needed for the FR metrics in order to correctly estimate the NR parameters, we plan to test a novel re-synchronization procedure. Recently, the NTIA group announced the release of a new version of VQM metrics especially tuned for variable packet loss rate. Even if the problem of realignment is still to be solved, the use of such a metric could probably be used for a more effective parameters tuning. As a general remark, the influence of the adopted key-frame detection algorithm should be investigated. In fact, if a fake key-frame is selected due to estimation errors, the quality metric immediately decreases. Another issue is related to the amount of motion characterizing the sequences. We noticed a difference in the scores when slow or almost null motion rate is present. The choice of the parameter λ _s should be based on the consideration, confirmed by many studies, that human attention is attracted by objects whose movement is relevant with respect to the other elements in the scene. Therefore, λ _s should probably be adapted to the relative motion of the surrounding areas. Finally, a key issue to be further investigated is the influence of the adopted error concealment technique implemented in the decoder. With the improving of error concealment masking techniques, the concealed video may present different error patterns from the ones we are experiencing at the moment. For example, we noticed that the latest version of VLC is able to mask, in a more effective way, some transmission errors like the presence of isolated blocks. This means that in the future, the weight of such parameters may be different depending on the improvements achieved in the field of error concealment techniques.