Distortion-specific feature selection algorithm for universal blind image quality assessment

Abstract

Blind image quality assessment (BIQA) aims to use objective measures for predicting the quality score of distorted images without any prior information regarding the reference image. Several BIQA techniques are proposed in literature that use a two-step approach, i.e., feature extraction for distortion classification and regression for predicting the quality score. In this paper, a three-step approach is proposed that aims to improve the performance of BIQA techniques. In the first step, feature extraction is performed using existing BIQA techniques to determine the distortion type. Secondly, features are selected for each distortion type based on the mean value of Spearman rank ordered correlation constant (SROCC) and linear correlation constant (LCC). Lastly, distortion-specific features are used by regression model to predict the quality score. Experimental results show that the predicted quality score using distortion-specific features strongly correlates with the subjective quality score, improves the overall performance of existing BIQA techniques, and reduces the processing time.

Introduction

In recent years, multimedia content has become a significant part of our lives. Delivery of images at the highest quality to the end user is an essential requirement for many modern imaging applications. Therefore, estimation of perceived image quality by humans, also known as subjective evaluation has gained importance. Subjective evaluation is used as a benchmark for image quality assessment (IQA), but the constraint of time and the tedious nature of the task make it unsuitable for many applications [1]. IQA techniques aim to replicate the behavior of human visual system to evaluate the quality score of images using objective parameters or measures. Objective IQA is divided into full reference (FR), reduced reference (RR), and blind IQA techniques. FR-IQA techniques require the pristine version of the image to predict the quality score of images [2–10]. RR-IQA techniques do not require the whole reference image but some information extracted from the reference image to perform IQA [11–17]. Techniques that evaluate the image quality score without the use of any prior information of reference image are called blind image quality assessment (BIQA) techniques [1]. BIQA techniques usually follow a two-step approach. Firstly, the distortion type affecting the image is determined using extracted features and then regression is applied to predict the quality score of the image.

Most BIQA techniques extract features, which are altered in the presence of distortion [18–37]. Features are extracted either in spatial domain, wavelet domain, and discrete cosine transform (DCT) domain or using edge information of image. Discrete wavelet transform and complex wavelet transform have been used in [38] and [24], respectively, to extract statistical features from distorted images to determine the distortion type and predicting the quality score. The ability of wavelet transforms to extract high frequencies can be exploited for edge analysis and IQA. In [31], curvelet transform is utilized to extract statistical features for the purpose of BIQA, which has rich information of scale and orientation in the image. Blind image integrity notator [39], which is an extended version of [40] has used DCT-based features along with the Bayesian inference model for predicting the quality score of the image. It requires no statistical model and utilizes sampled DCT coefficients for IQA. Natural scene statistics (NSS)-based features have been extracted in spatial domain for the evaluation of image quality score using luminance information [41]. It is simple and computationally less expensive since no transform has to be computed. In [42], the collection of quality aware features in a spatial domain are utilized for BIQA that measures the deviation in statistical properties between a distorted and natural images. In [43], two-dimensional features called atoms are introduced, which used sparse representation of coefficients in feature set to assess the quality of images. Shearlet transform has been used in [44] to model the NSS characteristics of images. Natural undistorted parts of the image are compared with the distorted parts to assess the quality score. Laplacian of Gaussian and Gaussian magnitude along with support vector regression (SVR) are used in [30] for the predicting the quality score of images using edge information. In [45], spatial and spectral entropy that captures information over different scales are used to compute features for the evaluation of quality score. Since, utilizing entropy as features to perform FR-IQA showed promising results; therefore, entropy was utilized as features for BIQA. Recently, BIQA is performed on multiple distorted images by augmenting the features extracted using blind image quality index (BIQI) [46], blind/referenceless image spatial quality evaluator (BRISQUE) [41], and sparse representation for natural scene statistics (SRNSS) [47] and selecting the top three features based on the average value spearman rank ordered correlation constant (SROCC) and linear correlation (LCC) [48]. But the performance of feature selection in [48] is only limited to the LIVE multiply-distorted image database and features extracted in [42, 46, 47]. The performance of [48] cannot be generalized to other BIQA techniques.

All of the aforementioned BIQA techniques employ the same set of features for each distortion type to evaluate the quality score of images. Each distortion type affects individual BIQA feature in a distinct manner because every type of distortion exhibits different characteristics, e.g., the Gaussian blur affects the edge information in an image, whereas JPEG distortion introduces blockiness. Therefore, the same set of features used for every distortion type will not yield the optimum results. This paper introduces a distortion-specific feature selection algorithm, which is based on Spearman rank ordered correlation constant (SROCC) and linear correlation constant (LCC) scores. All features having SROCC and LCC score computed over individual features, greater than the mean value of SROCC and LCC, are selected for the specific distortion type. The major contributions of this work are as follows:

• 1 A new SROCC- and LCC-based feature selection algorithm is proposed, which can be utilized with any two-step BIQA framework.

• 2 The proposed algorithm improves the performance of BIQA techniques in terms of better correlation of predicted quality score with the mean observer score (MOS) and reduces the processing time.

• 3 The proposed three-step approach is robust, database independent, and applicable in real-time scenarios.

The rest of the paper is organized as follows. Section 2 explains the proposed methodology along with distortion-specific feature selection algorithm. Section 3 presents the experimental results for six different BIQA techniques followed by the conclusion in Section 4.

Proposed methodology

The proposed methodology for BIQA is shown in Fig. 1, which follows a three-step approach, i.e., feature extraction, distortion-specific feature selection, and support vector regression, in contrast to the traditional two-step approach that is feature extraction and regression. The details of each step are as follows.

Fig. 1

Block diagram of the proposed three-step BIQA approach

Feature extraction for BIQA

In the first step, the N number of features F=f₁,···,f_N are extracted using existing BIQA techniques. Since noise in the image usually disrupts the high-frequency information in the image such as edges and corners, therefore, established BIQA techniques usually extract features in spatial and transform domains that model the deviation in characteristics of distorted images in comparison to natural images. To validate the performance of the proposed feature selection algorithm over features extracted in different domains, six BIQA techniques are selected that extract features in spatial, DCT transform, wavelet transform, curvelet transform, and spectral domains. All the selected BIQA techniques follow a two-step approach, i.e., feature extraction and distortion classification, followed by the computation of the quality score using SVR. The six BIQA techniques include BRISQUE [41], gradient magnitude and Laplacian of Gaussian-based IQA (GM-LOG) [30], blind image integrity notator based in DCT statistics II (BLIINDS II) [39], spatial-spectral entropy-based quality (SSEQ) [45], distortion identification-based image verity and integration evaluation (DIIVINE) [38], and curvelet quality assessment (CurveletQA) [31]. The details of BIQA techniques used for feature extraction are as follows:

Blind/referenceless image spatial quality evaluator (BRISQUE)

BRISQUE [41] extracts features in the spatial domain by utilizing locally normalized luminance coefficients and their products over two scales. Local mean displacements are removed to normalize the local variance of log contrast, which has de-correlating properties.Eighteen features are extracted over each scale using variance, shape, mean value, right variance parameters for horizontal, left variance,vertical, and diagonal pairwise products. BRISQUE uses a total of 36 features for the evaluation of quality score.

Gradient magnitude and Laplacian of Gaussian-based IQA (GM-LOG)

GM-LOG [30] uses the joint statistical relationship between the local contrast features of Laplacian of Gaussian (LoG) and gradient magnitude (GM) for BIQA. An adaptive procedure called joint adaptive normalization based on gain control and divisive normalization models on the local neighborhood is used to remove the spatial redundancies of GM and LOG coefficients. The technique follows a two-step approach, i.e., identification of distortion type and quality score prediction. A total of 40 features are extracted, which describe the structural information of the images for assessing the qualityusing SVR.

Blind image integrity notator based in DCT statistics II (BLIINDS II)

BLIINDS II [39] extracts NSS features in DCT domain on 17×17 patches of image. Each DCT block is divided into three directional regions and Gaussian fitting is performed on each region. BLIINDS II extracts four types of features, namely, coefficients of frequency variation, generalized energy subband ratio measure, Gaussian model shape parameters, and orientation model-based features to obtain 24 features. These features are utilized with a Bayesian inference model for predicting the image quality.

Spatial-spectral entropy-based quality (SSEQ)

SSEQ [45] extracts features at three scales of image resolution. To avoid aliasing, bi-cubic interpolation is used during down sampling. Each image is partitioned into subregions consisting of 8×8 pixels. Spatial and spectral entropy in DCT domain are computed for each patch. The spatial and spectral entropies are sorted in an ascending order, and 60% of the central elements are selected, which constitute a feature vector of length 12 to predict the quality score. These features are used as input to a pre-trained support vector classification (SVC) for identifying the distortion type affecting the image and then given as input to a SVR model for predicting the quality score.

Distortion Identification-based Image Verity and IntegratioN Evaluation (DIIVINE)

DIIVINE [38] uses a loose discrete wavelet transform to compute five groups of statistical features over two scales and six orientations by using steerable pyramids and statistical distribution curve fitting. Five groups of features, namely scale and orientation selective features, orientation selective statistics, correlation across scales, spatial correlation, and across orientation statistics constitute a feature vector of length 88. The feature vector is given as input to the SVC for the determination of distortion type, and SVR is utilized for the computation of image quality score.

Curvelet Quality Assessment (CurveletQA)

CurveletQA [31] extracts three types of features from curvelet subbands. Four NSS features are computed on the finest scale of the curvelet subbands using asymmetric Gaussian distribution, two features are extracted on the finest detail layer using the mean value of kurtosis and the ratio of sample mean and standard deviation of the non-cardinal orientation energies and six features are computed for scalar energy distribution by taking the difference of mean values of logarithmic magnitude of subbands at adjacent layers. These 12 features are used for the prediction of distortion type using SVC and prediction of quality score using SVR respectively.

The extracted features using BIQA techniques are given as input to the SVC to determine the distortion type D.

Distortion-specific feature selection

Natural images are highly structured, which possess properties that are affected when distortion is introduced in an image. These properties are known as natural scene statistics (NSS) [22]. BIQA techniques that utilize NSS try to assess the image quality based on the deviation between the NSS properties of distorted and natural images. The features that can effectively represent the deviation between the NSS properties of distorted and natural images show better performance, and they can predict the image quality more accurately. The main objective of this work is to introduce a generalized approach for distortion-specific feature selection that aim to improve the performance of existing BIQA techniques using features that can effectively represent the deviation in characteristics of an images from that of a natural image for a particular distortion type, which is also validated by Fig. 2. Therefore, second step involves selection of distortion-specific features based on SROCC and LCC score represented as F_D={f₁,···,f_M}, where M≤N. Generally, SROCC and LCC are utilized to assess the similarity between the MOS and the quality score predicted using BIQA techniques. A value close to 1 suggests a superior performance. Therefore, we select those features that have individual SROCC and LCC scores greater than the mean SROCC and mean LCC computed over individual features. The selected features have SROCC and LCC values closer to 1 resulting in enhancement of the prediction score. SROCC and LCC score is computed for each individual feature as,

$$ \begin{aligned} SROCC=1-\frac{6\sum\limits_{i=1}^{T} d_{i}^{2}}{T\left(T^{2}-1\right)}, \end{aligned} $$

(1)

Fig. 2

Normalized histogram of features values averaged over all the distortion type for selected BIQA techniques. a BRISQUE, b GM-LOG, c BLIINDS II, d SSEQ, e DIIVINE, and f CurveletQA, using original, all features, and after proposed feature selection algorithm

where d_i is the difference between paired ranks and T is the total number of samples, and

$$ \begin{aligned} LCC = \frac{\sum\limits_{i=1}^{T}(x_{i}-\bar{x})(y_{i}-\bar{y}))}{\sqrt{\sum\limits_{i=1}^{T}(x_{i}-\bar{x})^{2}}\sqrt{\sum\limits_{i=1}^{T}(y_{i}-\bar{y})}}, \end{aligned} $$

(2)

where x_i and y_i are the ith instance in first and second dataset, respectively, and $\bar {x}$ and $\bar {y}$ are mean values of datasets x and y, respectively. To perform feature selection, SROCC_i and LCC_i, for each individual feature F_i belonging to a BIQA technique, is computed. The predicted quality score using each individual feature is calculated using a pre-trained SVR model. Eighty percent of images in the dataset are used to train the SVR model, and testing is performed using the remaining 20% images. Mean score of SROCC (μ_S) and mean score of LCC (μ_L) computed over 1000 iterations are utilized for selecting distortion-specific features F_D. The proposed feature selection approach for BIQA is presented as Algorithm 1.

Support vector regression

In the third step, selected features F_D by Algorithm 1 are given as input to SVR for the prediction of quality score. Each distortion-specific regression model have different features as input. SVR is given as,

$$ \begin{aligned} \psi(F_{D})=\alpha\beta(F_{D})+b, \end{aligned} $$

(3)

where F_D is the input feature vector, α is the weight constant, β(·) is the feature space, and b is the bias value.

Experimental results and discussion

The proposed methodology is evaluated on four IQA databases, i.e., LIVE [49], CSIQ [50], TID2013 [51], and LIVE in the wild image quality challenge database [52]. LIVE database contains 779 images and five distortion types, namely fast fading (FF), Gaussian blur (GB), JPEG2000 compression (JP2KC), JPEG compression (JPEG), and white noise (WN). The CSIQ databases consists of 900 images and six distortion types, i.e., GB, JP2KC, JPEG, WN, global contrast (GC), and pink noise (PN). TID2013 database consists of 3000 images and 24 distortion types.

Figure 2 shows the normalized histograms of features averaged over all the distortion types and three databases, i.e., LIVE, TID2013, and CSIQ. Most BIQA techniques assess the quality of a distorted image by measuring the deviation of image characteristics from the characteristics of non-distorted images. Therefore, BIQA techniques should perform well if the deviation in the characteristic of images is represented by the extracted features. It can be observed that the deviation in characteristics of features of the distorted images are increased from the non-distorted image when the proposed feature selection is performed as compared to using all the features.

Performance comparison

For estimation of results, support vector machine requires pre-trained models for determining the distortion type and prediction of quality score. Therefore, we divide the dataset into two non-overlapping disjoint sets, i.e., training and testing. Eighty percent of the images are selected for training, whereas 20% of images are utilized for testing. The training and testing is repeated 1000 times with random disjoint set of images to predict the quality score. The SVR parameters c and γ used in this paper are the same as those mentioned by respective BIQA techniques.

Median scores of SROCC, LCC, Kendall correlation constant (KCC), and root mean squared error (RMSE) are reported for the performance evaluation of the proposed approach. The SROCC, LCC, and KCC scores measure the similarity between mean observer score and predicted quality score, whereas RMSE measure the error.

Figures 3, 4, and 5 show the performance of the proposed distortion-specific feature selection algorithm for selected BIQA techniques over each distortion type for TID2013, LIVE, and CSIQ IQA databases, respectively. The horizontal axis in Fig. 3 represents the distortion type label as given in the TID2013 database. It is evident from the results that the distortion-specific feature selection algorithm improves the SROCC score for majority of distortion types on selected BIQA techniques. It can be observed from Fig. 3 that the proposed technique consistently outranks the BIQA techniques and the proposed algorithm shows better or at par performance on 15, 18, 14, 15, 14, and 17 out of a total of 24 distortion types as compared to using all the features for BRISQUE [41], BLIINDS II [39], GM-LOG [30], SSEQ [45], DIIVINE [38], and CurveletQA [31], respectively, on the TID2013 database. Similarly, the proposed technique shows better or at par performance on 4, 4, 4, 4, 3, and 4 out of a total of five distortion types as compared to using all the features for BRISQUE [41], BLIINDS II [39], GM-LOG [30], SSEQ [45], DIIVINE [38], and CurveletQA [31], respectively, on the LIVE database. On the CSIQ database, the proposed technique shows better or at par performance on 6, 3, 4, 4, 6, and 3 out of a total of 6 distortion types as compared to using all the features for BRISQUE [41], BLIINDS II [39], GM-LOG [30], SSEQ [45], DIIVINE [38], and CurveletQA [31], respectively.

Fig. 3

Performance comparison of proposed algorithm for each distortion on TID2013 database (median SROCC), a BRISQUE, b GM-LOG, c BLIINDS II, d SSEQ, e DIIVINE, and f CurveletQA, using original, all features, and after proposed feature selection algorithm

Fig. 4

Performance comparison of proposed algorithm for each distortion on LIVE database (median SROCC), a BRISQUE, b GM-LOG, c BLIINDS II, d SSEQ, e DIIVINE, and f CurveletQA, using original, all features, and after proposed feature selection algorithm

Fig. 5

Performance comparison of proposed algorithm for each distortion on CSIQ database (median SROCC), a BRISQUE, b GM-LOG, c BLIINDS II, d SSEQ, e DIIVINE, and f CurveletQA, using original, all features and after proposed feature selection algorithm

Figures 6, 7, 8, and 9 shows the overall performance comparison of each BIQA technique along with the proposed distortion-specific feature selection algorithm and feature selection algorithm given in [48] on four IQA databases, i.e., LIVE [49], CSIQ [50], TID2013 [51], and LIVE, in the wild image quality challenge database [52], respectively. It can be observed that the proposed distortion-specific feature selection algorithm improves the overall performance of all the six state-of-the-art BIQA techniques as compared to using all the features, and by using feature selection algorithm of [48], the performance is worse than the original BIQA technique. The proposed algorithm also improves the performance of BIQA techniques on real images. The performance on LIVE in the wild image quality challenge database shows that the proposed algorithm can be used in real-time scenarios with real images taken in daylight and night time conditions.

Fig. 6

Overall performance comparison of proposed algorithm on different BIQA techniques for LIVE database, a SROCC, b LCC, c KCC, d RMSE

Fig. 7

Overall performance comparison of proposed algorithm on different BIQA techniques for CSIQ database, a SROCC, b LCC, c KCC, d RMSE

Fig. 8

Overall performance comparison of proposed algorithm on different BIQA techniques for TID2013 database, a SROCC, b LCC, c KCC, d RMSE

Fig. 9

Overall performance comparison of proposed algorithm on different BIQA techniques for wild in the LIVE challenge database, a SROCC, b LCC, c KCC, d RMSE

The performance of proposed algorithm can be validated by Fig. 10 that represents the comparison between the performance using all features against the proposed feature selection algorithm in terms of box plots for each BIQA technique. Box plots measure the dispersion or variance in data utilizing interquartile range and standard deviation represented by a five-number summary that includes the minimum value, first quartile (Q1), median value, third quartile (Q3), and maximum value of samples. The interquartile range is computed using the difference between Q1 and Q3. Q1 in box plots denotes the 25 percentile of the SROCC values, i.e., 25% of the SROCC values lie below Q1, and Q3 denotes the 75 percentile of SROCC values, i.e., 75% SROCC values lie below Q3. The box plots are computed for SROCC scores computed over 1000 runs averaged over all the IQA databases. It can be observed that the predicted quality score using BIQA techniques shows higher correlation with MOS when feature selection is performed. The interquartile range of the box plot for SROCC is reduced when feature selection is applied, which depicts the reduction in standard deviation of quality score prediction for BIQA techniques.

Fig. 10

Box plots of SROCC score for different BIQA techniques, a BRISQUE, b GM-LOG, c BLIINDS II, d SSEQ, e DIIVINE, and f CurveletQA, using all features and after features selection algorithm

Table 1 shows the overall performance of the proposed feature selection algorithm for cross-database validation when training is performed on one database and testing is performed on the other two databases. Four common type of distortions, i.e., GB, JP2KC, JPEG, WN, are considered for cross database evaluation. It can be observed that the proposed feature selection algorithm performs better than using all the features over all the BIQA techniques considered in this work. The cross database evaluation results show that the proposed feature selection algorithm is database independent and the proposed distortion-specific feature selection algorithm improves the overall performance of BIQA techniques irrespective of database.

Table 1 Overall performance comparison of proposed algorithm for cross-database validation

Table 2 shows the comparison of the overall proposed feature selection algorithm with BIQA techniques in terms of number of features and total processing time using a core i7 processor with 8 GB of RAM operating at 2.3 GHz. It can be observed that the proposed distortion-specific feature selection algorithm outperforms all BIQA techniques by reducing the number of features and improving the performance of existing BIQA techniques. The proposed feature selection also shows slight reduction in total processing time since reduction in number of features leads to reduced training and testing time for the SVR. The time taken for computation of SROCC and LCC score over individual features is not added as it is performed once to indicate, which features are selected, and therefore, it will be unfair to add it to the processing time for prediction of quality score, each time a test image is given as an input. The largest reduction in processing time of 2.94% is obtained for GM-LOG, and the lowest reduction of 0.013% is obtained for BLINDS-II IQA technique.

Table 2 Reduction in number of features by using proposed algorithm for different BIQA techniques on LIVE database

Conclusion

BIQA techniques proposed in literature use the same set of features for all the distortion types to evaluate the quality score of images. Each distortion type affects the individual BIQA feature in a distinct manner because each type of distortion exhibits different characteristics. Therefore, using the same set of features for all the distortion type will not yield optimum results. This paper presents a distortion-specific feature selection algorithm based on mean values of SROCC and LCC scores for blind image quality assessment. All features having individual SROCC and LCC scores greater than the mean values of SROCC and LCC computed over all the features are selected for the specific distortion type. The proposed algorithm is tested on six BIQA techniques and over most commonly used four IQA databases. The experimental results show that the proposed approach not only improves the performance of existing BIQA techniques but also reduces number of features that result in reduction of processing time. The proposed distortion-specific feature selection algorithm can be used with any BIQA technique that follows a two-step approach. Results on cross database evaluation show that the proposed algorithm is robust and database independent. Furthermore, experimental results on the LIVE in the wild image quality challenge database shows that the proposed algorithm is also valid for real images.