Phase congruency based on derivatives of circular symmetric Gaussian function: an efficient feature map for image quality assessment

Abstract

Image quality assessment (IQA) has become a hot issue in the area of image processing, which aims to evaluate image quality automatically by a metric being consistent with subjective evaluation. The first stage of conventional IQA model design is the quality-aware feature selection. Taking advantages of early visual feature, Phase congruency (PC) operates in frequency domain to measure local structures such as edges, corners, lines, etc., by computing the local amplitudes and local energies in multiple scales. Conventional local PC features are calculated with log-Gabor-based filtrations in several orientations, and usually combined with other features for IQA model design. Generally, a directional filter is sensitive to the changes on specific direction, and insensitive to other directions. This leads to multi-directional calculation and much time consumption in practical applications. Recently, researchers suggested that spatially circular symmetric filters, such as gradient magnitude (GM) and Laplacian of Gaussian (LoG), are highly efficient quality features and hence have been widely used in various IQA model designs. With the odd-symmetric and even-symmetric properties of GM and LoG operators, the two features are a suitable pair for PC compositions and can be computed uniformly by a Gaussian function with one scale factor. In this regard, we propose to combine GM and LoG signals to construct a new PC model with non-directional property. With ability to catch different types of distortions, the proposed PC feature can be promoted to a full-reference IQA model simply with average pooling or standard deviation pooling, and shows state-of-the-art performance compared with existing methods. Furthermore, our proposed PC algorithm can take the place of conventional PC component in well-known FSIM metric, which achieves improved performance and spends less in computation cost. This study suggests that the proposed circular symmetric PC feature is a highly efficient quality feature and can be exclusively used in IQA model designs.

1 Introduction

With the rapid growth of the technologies in digital communications and multimedia applications, more and more image data are produced for human observations. Human visual system (HVS) is the ultimate observer to judge the image quality. In order to improve efficiency, it is necessary to evaluate image quality automatically with a critical metric for different systems. Image quality assessment (IQA) model is developed to aim at estimating the objective quality of images as closely to subjective judgements as possible. Among different IQA algorithms, full-reference (FR) IQA works when the original reference image is completely provided, no-reference (NR) is employed when the pristine reference image is not available, and reduced-reference (RR) works at the situation where partial information of reference image is provided. Up to now, FR-IQA has been extensively applied for various cases, such as image reconstruction, network transmission, image coding and compression, etc. In the premise of simultaneous presence of reference and distorted images, FR-IQA metric can be applied in optimization of image processing systems [1,2,3,4], including the help of training deep neural networks for various vision tasks [2,3,4].

Conventional FR-IQA metrics such as mean squared error (MSE) and the peak signal-to-noise ratio (PSNR), which compute the image quality index on the intensity domain, evaluate the distortion degree by an arithmetic difference between reference and distorted images. The structural similarity (SSIM) index [5] can capture structural similarity information based on the assumption that the HVS is sensitive to local structure of visual signals. Based on that, the multi-scale SSIM (MS-SSIM) metric [6] compute the contrast and structural similarity at five scales altogether. Another variant of SSIM is the information-weighted SSIM (IW-SSIM) metric [7], in which different types of local regions are considered to make different contributions to the quality of an image. Riesz transforms based feature similarity (RFSIM) [8] and spectral residual-based similarity (SR-SIM) [9] are also improvements based on SSIM. The information fidelity criterion (IFC) [10] was proposed using the information theory, and was upgraded to more efficient metric named visual information fidelity (VIF) [11]. Based on property of HVS understanding the image in low-level vision [12, 13], the feature similarity (FSIM) index [14] measures the local structure by the value of phase congruency (PC) and image gradient. Image gradient has also been extensively applied to evaluate image distortion which yielded the gradient similarity (GSM) algorithm [15] and the gradient magnitude similarity deviation (GMSD) [16]. Another method to measure local structure of visual signal is to employ the Laplacian of Gaussian (LoG) filter, which proves to be approximate to the de-correlating mechanism of the retinal ganglion receptive field in HVS [17, 18]. Non-shift edge based ratio (NSER) [19] makes use of image edges produced by LoG filters, which are quality-aware in representing structural distortions. More related researches prove that LoG is highly efficient in FR [20, 21], RR [22,23,24], and blind IQA [25] model design. In these related studies, LoG shows ability to retain structural distortions in all directions because of its circularly symmetric property. Especially, the joint distribution of GM and LoG in [25] has proven efficient in IQA feature representation and IQA model design, and the relationship between GM and LoG was explored for the first time. Since the non-directional filters have proven quality-aware, GM and LoG generated from Gaussian function on the same scale are more universal in theoretical calculation and subsequent optimization in IQA related applications. Comprehensive surveys and detailed comparisons of modern IQA metrics are discussed in many literature [26,27,28,29].

Aside from conventional IQA methods, convolutional neural networks (CNN) have been applied to IQA issues in recent researches [30,31,32]. Although existing CNNs have reached good performance in predicting image quality, studies on IQA models without training is still meaningful in practical applications. On the other hand, quality-aware feature maps have also been employed as similarity maps [33] and quality-aware loss [34] which helps to predict the discrepancy map. Therefore, quality-aware feature design is still a valuable issue in related fields.

In general, an FR-IQA model is usually consisted of three methodical steps: feature extraction from the reference image and distortion image, point-wise quality measure between the features of the reference and distortion images, and pooling the local quality measures over the image [5]. The quality score is attained based on the pooling result [5]. Meanwhile, the image quality feature is acquired by handcrafted design or machine learning tech [35]. The quality measure is normally carried out by a distance metric [16] or a learnt network [25]. The pooling strategy is mostly either of mean or deviation computation over the local quality measures [5, 16]. Image feature reflects a specific aspect of image information by which meaningful image signal is represented and synthesized, and hence an image processing algorithm can be designed to realize a specified processing purpose. There are numerous image features proposed for various image processing tasks, such as image gradient, LoG signal, Gabor-like function, etc. However, conventional image features were proposed in accordance with natural images. They are efficient in representing natural image structures and have been used for usual image processing tasks, such as image denoising, super-resolution, image restoration, etc. In IQA model design, image feature extraction is not only for natural (reference) images, but also for distorted images. For example, in low-level vision, a natural image consists of a plenty of directional features. In this type of image structures, pixel values are consistent along with its direction, so that Gabor-like functions are highly efficient to represent image low-level structures as independent components [36]. However, in distortion images, image structures are distorted to varied ones and IQA model measures the variation to assess image quality. In this case, distorted image structures may not be well represented by the directional features such as Gabor-like functions since the image values may be changed a lot along with its direction. Alternatively, circular symmetric filters, e.g., gradient magnitude, LoG signal, are more efficient in IQA design since circular symmetric filters do not have a preferred direction and hence easily sense the distortion information of the image. Indeed, there are a lot of successful IQA models that have been proposed based on the circular symmetric filters [16,17,18,19,20,21,22,23,24,25], as mentioned in the previous paragraph.

As one of the most important components in visual signal processing, phase information carries more structural information than the spectral amplitude does in an image [37], where low-level features such as edges and corners show consistence in phase according to Fourier translation. Based on physiological and psychophysical evidences, the PC theory provides a simple but biologically plausible model of how mammalian visual systems detect and identify features in an image [38,39,40]. The experiment based on odd and even symmetry of visual receptive fields [38] explains that it is an efficient means for the visual system to locate the edges by the sum of the squared output of odd and even-symmetric filters that always peak at points of phase congruence. As the result, points of high PC value represent highly informative features. As a dimensionless index, conventional PC algorithm was defined by Morrone et al. in 1986 [38] and was developed by Kovesi [41, 42] based on a local energy model, which assumed that features are more evidently perceived at points where the Fourier components are maximally in phase. Many scholars have made use of PC features in relevant fields, where the computation is operated in frequency domain after filtration with multi-orientations [41, 42]. Multiscale PC has been applied in edge visual saliency detection [43], and the feature map can reflect fundamental structures and textures. Combined with Complex Wavelet Transform (CWT), the concept of PC is also efficient in image representation [44] and redundancy removal. The phase-based algorithms usually employ 2D Discrete Fourier Transform (DFT), Gabor filters [45], or log-Gabor filters [46] to calculate phase information. In studies on image quality evaluation, FSIM algorithm [14] combined PC with gradient magnitude, which is computed as the secondary feature to encode contrast information [47]. Combined GM and local binary pattern (LBP) in PC domain at multiple scales to design the NR-IQA method with training a support vector regression model. A recent proposed metric based on symmetry phase congruency (SPCM) [48] also combined PC with GM in similarity computation.

In calculation, conventional PC-based metrics for IQA models use the even-symmetric and odd-symmetric components of Gabor-like functions with multi-orientations, for example, the four-orientation Gabor-based PC on four scales employed in FSIM index [14]. In low-level vision, natural image consists of directional features, so that Gabor-based PC is highly efficient to represent low-level structures. However, structures are changed on arbitrary directions in distortion images. When pixel value varies along with the initial structural direction, distortion structure would be neglected by Gabor-based PC, since a single Gabor-based operator is only highly sensitive to changes orthogonal to the edge. Consequently, more orientations are needed in calculation, and thus computational complexity is increased. On the contrary, circular symmetric filters treat changes on all directions equally, which are more efficient in IQA design. This motivates us to design a new term of PC computing method where only circular symmetric filters are used. As is well known, gradient magnitude and LoG filter are the first-order and second-order derivatives of Gaussian function, respectively. GM and LoG are quality-aware features as mentioned above, and can be uniformly generated from circular symmetric Gaussian function. Obviously, gradient filter is odd-symmetric, and LoG filter is even-symmetric. Therefore, GM and LoG maps can represent the odd-symmetric and even-symmetric components of an image separately.

In this study, we utilize GM and LoG maps for PC computing to obtain a non-directional PC operator. The proposed PC feature can be promoted to an FR-IQA model by simply utilizing a similarity calculation and an average or standard deviation pooling strategy, and the model proves to be state-of-the-art compared with the competitors. Furthermore, we replaced the PC algorithm in well-known FSIM with our PC computation method to test the accuracy of the proposed method in representing phase information. The experimental results revealed that the proposed PC feature map can correctly take the place of conventional PC algorithm, and the calculation method is faster than the original one.

The rest of this paper is organized as follows. The proposed phase congruency method and a new FR-IQA metric are introduced in Sect. 2. Section 3 describes the experimental setups. In Sect. 4, results and comparisons on three benchmark databases are presented. Section 5 concludes the paper.

2 Methods

2.1 Phase congruency

The phase congruency (PC) that is a dimensionless quantity was first proposed as a frequency-based algorithm [38] instead of spatially processing on images. The basic concept of PC algorithm is that the Fourier components are maximal in phase where the local structure is perceived in an image. According to the extensively used PC algorithm developed by Kovesi in [41], one consider a one-dimensional signal $f(x)$, and denote the even-symmetric filter and the odd-symmetric filter by ${M}_{n}^{e}$ and ${M}_{n}^{o}$ separately on scale n and define a vector to represent the responses of the signal $f(x)$ after filtered by ${M}_{n}^{e}$ and ${M}_{n}^{o}$ on scale n as follows:

$$\left[{e}_{n}\left(x\right), {o}_{n}\left(x\right)\right]=\left[f\left(x\right)*{M}_{n}^{e}, f\left(x\right)*{M}_{n}^{o}\right],$$

(1)

where ${e}_{n}\left(x\right)$ and ${o}_{n}\left(x\right)$ are the output of ${M}_{n}^{e}$ and ${M}_{n}^{o}$ filtering at position x. The local amplitude on scale n is defined as:

$${A}_{n}\left(x\right)=\sqrt{{{e}_{n}\left(x\right)}^{2}+{{o}_{n}\left(x\right)}^{2}.}$$

(2)

The local energy function can be written as:

$$E\left(x\right)=\sqrt{{F}^{2}\left(x\right)+{H}^{2}\left(x\right),}$$

(3)

where

$$F\left(x\right)=\sum_{n}{e}_{n}\left(x\right),$$

(4)

$$H\left(x\right)=\sum_{n}{o}_{n}\left(x\right).$$

(5)

The PC of one-dimensional signal is defined as:

$$\mathrm{PC}\left(x\right)=\frac{E(x)}{\varepsilon +{\sum }_{n}{A}_{n}(x)},$$

(6)

where $\varepsilon$ is a small positive constant to prevent the denominator from being zero.

Different from conventional methods, in this study we apply the GM and LoG filters, which are the first-order and the second-order derivatives of Gaussian function, instead of the log-Gabor-based directional filters ${M}_{n}^{o}$ and ${M}_{n}^{e}$. As shown in Fig. 1, the normalized first-order and second-order derivatives of 1D Gaussian function are odd-symmetric and even-symmetric, respectively.

Fig. 1

The first-order and second-order derivatives of 1D Gaussian function after normalization. The first-order derivate of Gaussian is an odd-symmetric filter that can generate signal gradient, and the second-order derivate of Gaussian is an even-symmetric filter, which is an LoG filter

For 2D signals, the image gradient magnitude defined as the root mean square of image directional gradients along two orthogonal directions is still the first-order derivative of 2D Gaussian filter. We denote the Gaussian function by $G$, then the gradient filter on horizontal direction and vertical direction are defined as:

$${{\varvec{h}}}_{x}(x,y|\sigma )=(-\frac{1}{2\pi {\sigma }^{4}})x{e}^{-\frac{{x}^{2}+{y}^{2}}{2{\sigma }^{2}},}$$

(7)

$${{\varvec{h}}}_{y}(x,y|\sigma )=(-\frac{1}{2\pi {\sigma }^{4}})\mathrm{y}{e}^{-\frac{{x}^{2}+{y}^{2}}{2{\sigma }^{2}},}$$

(8)

where the variables $x$ and $y$ denote the coordinate of the input image, parameter $\sigma$ denotes the scale factor of the Gaussian function. We denote the image by ${\varvec{I}}$, and convolve the image with the two directional derivative filters to produce the horizontal and vertical gradient images ${{\varvec{d}}}_{n,x}$ and ${{\varvec{d}}}_{n,y}$ on scale $n$, thus the GM of an image is computed as:

$$D_{n} \left( {x,y} \right) = \sqrt {{\varvec{d}}_{n,x}^{2} + {\varvec{d}}_{n, y}^{2} } = \sqrt {\left( {{\varvec{I}} \otimes {\varvec{h}}_{x} } \right)^{2} + \left( {{\varvec{I}} \otimes {\varvec{h}}_{y} } \right)^{2} } .$$

(9)

As the second-order derivative of 2D Gaussian function, the LoG filter is defined as:

$${{\varvec{h}}}_{LOG}(x,y|\sigma )=-\frac{1}{\pi {\sigma }^{4}}(1-\frac{{x}^{2}+{y}^{2}}{2{\sigma }^{2}}){e}^{-\frac{{x}^{2}+{y}^{2}}{2{\sigma }^{2}},}$$

(10)

where the variables $x$ and $y$ denote the coordinate of the input image, parameter $\sigma$ denotes the scale factor of the Gaussian function. Thus, the LoG map on scale $n$ can be computed as:

$$L_{n} \left( {x,y} \right) = {\varvec{I}} \otimes {\varvec{h}}_{{{\text{LOG}}}} .$$

(11)

In order to remove the contrast variation in the image of a large scale, we use divisive normalization [20, 25] as:

$${V}_{n}\left(x,y\right)=\frac{{D}_{n}(x,y)}{\sqrt{{G}_{n}\left(x,y\right)*{D}_{n}^{2}(x,y)}+{c}_{0}},$$

(12)

$${U}_{n}\left(x,y\right)=\frac{{L}_{n}(x,y)}{\sqrt{{G}_{n}\left(x,y\right)*{L}_{n}^{2}(x,y)}+{c}_{0}},$$

(13)

where ${c}_{0}$ is a positive constant to ensure the stability of calculation, and ${G}_{n}\left(x,y\right)$ represents a large-scale Gaussian filter employed for each scale n.

Therefore, the 2D local amplitude and local energy on scale $n$ can be written as:

$${A}_{n}\left(x,y\right)=\sqrt{{{U}_{n}\left(x,y\right)}^{2}+{{V}_{n}\left(x,y\right)}^{2},}$$

(14)

$$E\left(x,y\right)=\sqrt{{F}^{2}\left(x,y\right)+{H}^{2}\left(x,y\right),}$$

(15)

where

$$F\left(x,y\right)=\sum_{n}{U}_{n}\left(x,y\right),$$

(16)

$$H\left(x,y\right)=\sum_{n}{V}_{n}\left(x,y\right).$$

(17)

Thus the PC can be computed by:

$$\mathrm{PC}\left(x,y\right)=\frac{E(x,y)}{\varepsilon +{\sum }_{n}{A}_{n}(x,y)}.$$

(18)

The value of PC ranges from 0 to 1. According to the definition, PC value equals 0 means no significance feature here, whereas the value 1 means the most important feature existed. Therefore, the PC map constructed from the odd-symmetric and the even-symmetric components can reflect the structural information of an image.

An analysis of the GM and LoG responses for different types of edge signals and distorted edges is shown in Fig. 2. A pristine 1-D edge signal, a Gaussian blurred version, a Gaussian noise corrupted signal, and a DCT compressed signal are demonstrated in the first column of Fig. 2a–d, respectively. The next two columns are the corresponding GM and LoG responses on two different scales. The last column shows the phase congruency curve computed by GM and LoG responses. The result validated that the proposed PC feature gives the highest value at the edge position for both original ideal edge and the corresponding distorted version, no matter which type of edge is to be processed. Based on this property, the proposed PC feature is suggested to have the ability to reflect the structural information of reference and distorted images.

Fig. 2

Analysis on the GM and LoG responses of step function. The four columns from left to right are: step edges, GM and LoG responses, and PC value. a Ideal edge. b Gaussian blur. c Gaussian noise. d DCT compression

In order to explain why non-directional PC feature is better than directional Gabor-based PC in representing changed structures, we demonstrate a comprehensible comparison of edge extraction from structural changes in Fig. 3. The first row shows the proposed PC map along with Gabor-based PC maps using one and four orientations in computation separately. The horizontal and vertical edges are not completely consistent with each other in a 1-orient horizontal Gabor-based PC feature map. Although multi-orientation neutralizes the difference a lot, some artifacts still exist. In the second row, the input image has small changes on both directions. Obviously, the feature map generated by 1-orient Gabor-based PC gives different responses on horizontal and vertical edges, although the two changes share the same shape and value. When the number of orientations increases in Gabor-based PC, the difference between directions is reduced. Nevertheless, the feature map generated by 4-orient Gabor PC is not as smooth as the proposed PC map. The third row is the same as the second row except the images are rotated with 45 degrees. 1-orient Gabor PC responses the same on the two directions, but cannot clearly represent the image structure. The feature map of 4-orient Gabor PC seems to be better in extracting edges, while artifacts are still difficult to eliminate. However, the proposed non-directional PC operator does not suffer from this problem. Moreover, the proposed PC keeps steady on arbitrary directions and would not be affected by image rotation, which is more in line with the function of the HVS.

Fig. 3

Variation extraction on different directions by proposed PC and Gabor-based PC

For further comparison and better illustration on how PC describe the image structure, Fig. 4 shows the proposed PC feature map on two reference images and corresponding distorted images, compared with GM, LoG, and conventional Gabor-based PC feature proposed by Kovesi [41]. Note that the proposed PC is computed on small scales. In the reference image, the proposed PC map reflects the significance of local structures even if the local contrast is low, thus PC is able to capture more details of structural information than GM and LoG maps. For distorted images, the new PC map still shows more distorted structures than GM and LoG maps, regardless of distortion types. Especially, the distorted edges in JPEG image can be clearly sensed by the proposed PC map, which are hard to directly emerge by GM and LoG maps according to Fig. 4. It is obvious that the Gabor-based PC operator cannot describe the distortion as clearly as the proposed PC map. For example, in the JPEG image, the proposed PC map displays the blocky contours clearly, but Gabor-based PC leads to ineluctable artifacts beyond edges. In the blur image, Gabor-based PC gives high response to the background, where human observers do not notice. Therefore, the proposed PC can figure out distortion structures more completely and clearly. This comparison proclaims that the PC map constructed by GM and LoG is an efficient feature map which contains enough structural distortion information to distinguish the faint features in distorted images, thus can be helpful to improve the prediction accuracy for IQA.

Fig. 4

The proposed PC feature of reference image and its corresponding distorted images compared with GM, LoG, and Kovesi’s PC map

2.2 FR-IQA model based on the proposed PC algorithm

Since the PC value represents the significance of edges, the quality map that measures local similarity can use PC between signal ${f}_{1}(x)$ and ${f}_{2}(x)$, as defined by Eq. (19):

$${Q}_{\mathrm{PC},g}\left(x,y\right)=\frac{2{PC}_{1}\left(x,y\right){\bullet \mathrm{PC}}_{2}\left(x,y\right)+{c}_{1}}{{PC}_{1}^{2}\left(x,y\right)+{\mathrm{PC}}_{2}^{2}\left(x,y\right)+{c}_{1}},$$

(19)

where ${c}_{1}$ is a positive constant to prevent division by zero and increase the stability, the subscript $g$ means the calculation is done on grayscale images or luma channel of color images. This is a commonly applied measure to define the similarity of two positive real numbers [5], and the result of each image pixel ranges within (0, 1]. Higher result means higher similarity between distorted and reference images.

The calculation of quality map can be directly applied on grayscale images. As for color images, we transform RGB signals to YIQ color space by a formula in [49]:

$$\left[\begin{array}{c}Y\\ I\\ Q\end{array}\right]=\left[\begin{array}{ccc}0.299& 0.587& 0.114\\ 0.596& -0.274& -0.322\\ 0.211& -0.523& 0.312\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right].$$

(20)

The similarity between chromatic channels is generated from:

$${S}_{I}\left(x,y\right)=\frac{2{I}_{1}\left(x,y\right){\bullet I}_{2}\left(x,y\right)+{c}_{2}}{{I}_{1}^{2}\left(x,y\right)+{I}_{2}^{2}\left(x,y\right)+{c}_{2}},$$

(21)

$${S}_{Q}\left(x,y\right)=\frac{2{Q}_{1}\left(x,y\right){\bullet Q}_{2}\left(x,y\right)+{c}_{3}}{{Q}_{1}^{2}\left(x,y\right)+{Q}_{2}^{2}\left(x,y\right)+{c}_{3}},$$

(22)

where ${I}_{1}$, ${I}_{2}$, ${Q}_{1}$, ${Q}_{2}$ are chromatic channels of the reference and distorted images, ${c}_{2}$ and ${c}_{3}$ are constants that balance the data. Then the quality map for color images is defined as follows:

$${Q}_{\mathrm{PC},c}\left(x,y\right)={Q}_{\mathrm{PC},g}\left(x,y\right)\bullet {\left[{S}_{I}\left(x,y\right)\bullet {S}_{Q}\left(x,y\right)\right]}^{\lambda },$$

(23)

where $\lambda$ is a constant to regulate the influence level of chromatic channels, the subscript $c$ means the calculation is for color images.

In order to yield the overall score of an image, the pixel-based similarity map should be converted to a scalar score with a proper pooling strategy. Weighted pooling methods are widely discussed and many researches on pooling strategy have been done for image and video quality assessment [7, 50,51,52,53]. Average pooling is employed based on the hypothesis that each part of the image contributes the same importance in overall quality, which is the most commonly used method for pooling process. We compute the quality map with average pooling method as:

$${\mathrm{mean}(Q}_{PC,i})=\frac{1}{N}{\sum }_{x,y}{Q}_{\mathrm{PC},i}\left(x,y\right),$$

(24)

where N represents the number of pixels in the image, $i\in \{g,c\}$ denotes whether the input images are grayscale or color images.

For further comparison, we utilize a standard deviation pooling strategy that considers different local structures with different degradations. It has been proven to be efficient for gradient similarity-based IQA method in [16], thus we compute the standard deviation of the similarity map of PC as follows:

$${\mathrm{std}(Q}_{PC,i})=\sqrt{\frac{1}{N}{\sum }_{x,y}{{(Q}_{PC,i}(x,y)-{\mathrm{mean}(Q}_{PC,i}))}^{2}.}$$

(25)

The average pooling result gives higher score to better quality image since it measures the average similarity between reference and distorted images, whereas the standard deviation pooling gives higher score to lower quality image with larger distortion, on account of the ability to measure difference between distorted and reference images.

We made statistics on LIVE database [54] and found a nonlinear relationship of 1/3rd power law between the predicted scores and subjective scores. This nonlinearity also exists in CSIQ [55] and TID2013 [56] databases. Hence, in order to obtain a balanced relation between predicted quality scores and subjective scores, we use a nonlinear transformation to calculate the score as follows:

$${q}_{m,i}={[{1-\mathrm{mean}(Q}_{PC,i}\left(x\right))]}^\frac{1}{3},$$

(26)

$${q}_{\mathrm{sd},i}={{[\mathrm{std}(Q}_{PC,i}\left(x\right))]}^\frac{1}{3},$$

(27)

where $i\in \{g,c\}$ denotes whether the input images are grayscale or color images. The modification makes the output of our model more reasonable and practical owing to the uniform distribution. Note that the transformation does not change the rank order of the estimated scores of distorted images, hence it has no influence in the evaluation of monotonicity. Nevertheless, we will explore the nonlinear relationship between objective score and subjective score in our further investigation.

2.3 Replace the PC computation in FSIM

FSIM metric [14] is a well-known FR-IQA model that has achieved outstanding quality evaluation performance and has been widely applied for various applications [57,58,59]. FSIM separates the feature similarity measurement between signal ${f}_{1}(x)$ and ${f}_{2}(x)$ into two components, each for PC or GM. The feature ${S}_{L}\left(x\right)$ combined PC with GM is defined as:

$${S}_{L}\left(x\right)={S}_{PC}\left(x\right){S}_{G}\left(x\right),$$

(28)

where ${S}_{G}(x)$ is the similarity measure of image gradient. We replace the PC computation with our proposed PC metric, and compute the objective score in the way as the FSIM algorithm does:

$${\mathrm{S}}_{\mathrm{FSIM}}=\frac{{\sum }_{x}{S}_{L}(x)\bullet {PC}_{m}(x)}{{\sum }_{x}{PC}_{m}(x)},$$

(29)

where

$${PC}_{m}\left(x\right)=\mathrm{max}\left({PC}_{1}\left(x\right),{PC}_{2}\left(x\right)\right).$$

(30)

We compared this result with the original FSIM performance to test the validity and accuracy of our computation to express the structural features as a phase congruency expression.

3 Experimental setup

We test the proposed FR-IQA model on three benchmark databases: LIVE [54], CSIQ [55], and TID2013 [56]. LIVE database contains 29 reference images and 779 distorted images generated with 5 distortions types: JPEG compression, JPEG2000 compression, white noise, Gaussian blur and simulated fast fading. CSIQ database consists of 30 reference images and 866 distorted images generated with 6 different distortions types: JPEG compression, JPEG2000 compression, additive white noise, additive pink Gaussian noise, Gaussian blur, and global contrast decrements. The Difference Mean Opinion Score (DMOS) values are provided in LIVE and CSIQ databases as the subjective score for distorted images, which is a positive score representing the degree of distortion from human evaluation. The TID2013 database is the largest of the three databases which contains 3000 distorted images created from 25 reference images with 24 types of distortions at 5 levels. The mean opinion score (MOS) are provided as subjective score of human evaluation, which gives higher value to higher subjective image quality. The most commonly applied methodology in evaluation of IQA models is the Spearman rank-order correlation coefficient (SROCC). It takes consideration of prediction monotonicity, which is a typical aspect of IQA performance [60].

The SROCC between predicted score and reference subjective score is defined as:

$$\mathrm{SROCC}\left(X,S\right)=1-\frac{6{\sum }_{i=1}^{n}{d}_{i}^{2}}{n\left({n}^{2}-1\right)},$$

(31)

where $X$ and $S$ are the vectors of the predicted results and subjective scores of the test images, and ${d}_{i}$ is the difference between the rank of an objective score in $X$ and the rank of its corresponding subjective score in $S$.

The Pearson Linear Correlation Coefficient (PLCC) metric, which measures the prediction accuracy, should be applied after a nonlinear regression. A logistic function with an added linear term [26] is employed as follows:

$${X}_{r}={\beta }_{1}\left(\frac{1}{2}-\frac{1}{1+\mathrm{exp}\left({\beta }_{2}\left(X-{\beta }_{3}\right)\right)}\right)+{\beta }_{4}X+{\beta }_{5},$$

(32)

where ${\beta }_{i}, i=1, 2, \dots , 5,$ are parameters to be fitted in the regression function. ${X}_{r}$ denotes the IQA scores after nonlinear regression. The PLCC is defined as:

$$\mathrm{PLCC}\left({X}_{r}, S\right)=\frac{{\overline{X} }_{r}^{T}\overline{S} }{\sqrt{{\overline{X} }_{r}^{T}{\overline{X} }_{r}{\overline{S} }^{T}\overline{S}} },$$

(33)

where ${\overline{X} }_{r}^{T}$ and $\overline{S }$ denote the vectors of scores with mean value removed.

The root mean square error (RMSE), which evaluates the prediction consistency of the IQA performance, is computed as:

$$\mathrm{RMSE}\left({X}_{r}, S\right)=\sqrt{{\left({X}_{r}-S\right)}^{T}\left({X}_{r}-S\right)/n}.$$

(34)

In the experimental setup, the constant ${c}_{0}$ in divisive normalization by Eqs. (12) and (13) is selected as 120. According to the relationship between adjacent scales, the PC calculation is operated on two scales that larger filtering window is twice the width of the smaller one. We set the standard deviation of the original multi-scale Gaussian functions as 0.3 and 0.6, respectively. The constant $\varepsilon$ in PC calculation by Eq. (18) is selected as 25, and the constant ${c}_{1}$ in similarity map calculation by Eq. (19) is selected as $3\times {10}^{-5}$, which show the best property in the experimental performance of the proposed model on grayscale images. ${c}_{2}$ and ${c}_{3}$ in Eq. (21) and Eq. (22) are set as 200, and $\lambda$ in Eq. (23) is selected as 0.03 according to [14]. When replacing the PC algorithm in FSIM, we select the scale factor of Gaussian functions as 2 and 4, since the image gradient in FSIM is calculated by Prewitt operator with a small-scale window, thus the phase computation needs to catch structural information with a larger range. According to the chosen scale of Gaussian function, the constant ${c}_{0}$ is adjusted to 60, $\varepsilon$ is selected as 5.5, and ${c}_{1}$ is selected as 0.03.

4 Results and discussion

4.1 Experimental results on different databases

In order to validate the performance of the proposed metric, we investigate the model scores for images from the three benchmark databases, and compute the SROCC, PLCC, and RMSE between the model scores and subjective opinion scores provided by the databases as the performance criteria. The performances of proposed metric and competitors are shown in Table 1, where ${q}_{m,g}$ and ${q}_{\mathrm{sd},g}$ are the results on grayscale images, ${q}_{m,c}$ and ${q}_{\mathrm{sd},c}$ are results on color images. The source codes of the competitors in “Grayscale” group are publicly available and we have verified that they worked correctly according to the results in the original papers. Since the databases and calculation accuracies provided in the original literatures are not completely consistent with one another, we ran all the conventional algorithms on the three major databases with the same software and hardware systems in the whole process of our experiment to ensure the authenticity and fairness. On each database, the top three metric performances are presented in boldface. The “Color” group shows experimental results on color images, and the best metric for each database is shown in bold face. In addition, for a more comprehensive comparison, “Deep-Learning” group shows results of two deep-learning metrics [61, 62], where the data are provided by their original papers.

Table 1 Performances of proposed metric and competitors on three benchmark databases in terms of SROCC, PLCC and RMSE

In Table 1, the result of standard deviation pooling method is much more efficient than the average pooling method in grayscale group. For grayscale images, the proposed model with standard deviation pooling ranks 4th on LIVE database, 2nd on CSIQ, 1st on TID2013 database, and 1st on average across the three databases. Particularly, the proposed method performs significantly better than other metrics on TID2013 database. According to the experimental results, the proposed model shows stability and efficiency on a large range of different distortion types, since TID2013 is one of the most extensively used database which contains the most types of distortion and varies of image content. Besides, ${q}_{\mathrm{sd},g}$ performs better than DISTS-Gray [1, 61] on grayscale images, and ${q}_{\mathrm{sd},c}$ performs better than DISTS-Color model in terms of SROCC. Although DeepSim [62] shows better performance on LIVE database, the proposed model still achieves stable performances on CSIQ and TID2013, and no training process is required. The result of ${q}_{\mathrm{sd},c}$ on color images is not higher than FSIMc, however, we checked the parameter settings and found that adjusting Gaussian scale factor can improve the accuracy of color image quality prediction, especially on TID2013 where chromatic distortions lead to a wide range of color change. According to the relationship between adjacent scales, the PC calculation is operated on two scales that larger filtering window is twice the width of the smaller one. The relationship between SROCC and the small-scale factor of Gaussian function is shown in Fig. 5. Although we selected small scales for PC in our proposed IQA model, the scale factor is an adjustable parameter for the proposed PC operator. That is, the scale factor can be selected according to the specific situation where PC feature is applied.

Fig. 5

Relationship between SROCC and the small-scale factor of Gaussian function in PC, computed on LIVE, CSIQ, and TID2013 databases

In Table 2, we tested the effectiveness of the proposed PC feature in representing phase congruency information in FSIM metric where PC is applied as a dimensionless measure for the significance of a local structure. The data of ${\mathrm{S}}_{\mathrm{FSIM}}$ and ${\mathrm{S}}_{\mathrm{FSIM}}$-c in Table 2 represent the results that we replace the PC algorithm in FSIM and FSIMc metrics with our proposed PC metric, compared with the original FSIM and FSIMc results in terms of SROCC.

Table 2 Performance of ${\mathrm{S}}_{\mathrm{FSIM}}$ and ${\mathrm{S}}_{\mathrm{FSIMc}}$ in terms of SROCC, where the PC algorithm in FSIM metric has been replaced by the proposed PC

It is shown that the performances of ${\mathrm{S}}_{\mathrm{FSIM}}$ and ${\mathrm{S}}_{\mathrm{FSIMc}}$ are very close to FSIM and FSIMc methods, and the result has been slightly improved on CSIQ and TID2013 databases. In fact, the proposed PC method reduces the computational complexity and shows better performance in average than the original PC algorithm. Such results validate that our PC calculation method is effective for phase information representation for IQA tasks compared with traditional PC method. The computational PC model constructed from GM and LoG maps works stably as a phase-based mechanism without calculating the phase information directly from the multi-scale frequency domain.

Figure 6 shows the scatter plots of predicted quality scores versus subjective scores of the proposed model, compared with other metrics on LIVE database. The horizontal axis denotes the objective scores computed by different IQA metrics, while the vertical axis denotes the DMOS values. In this figure, we can see the monotonicity and consistency of the proposed and comparison IQA metrics more intuitively, since the scatter plots reflects the relationship between objective and subjective evaluations by pairs of coordinates.

Fig. 6

Scatter plots of predicted quality scores versus the subjective scores in term of MOS or DMOS for images in LIVE database

4.2 Experimental results on individual distortion types

For further comparison of the performance between the proposed model and the competitors, we present the performance of proposed model and the competition metrics on each individual distortion type in terms of SROCC in Table 3. For each distortion type, the top three algorithms are presented in boldface. The last row counts the number of times that each algorithm reaches the top three across all distortion types.

Table 3 Performances of proposed metric and competitors on each individual distortion type in terms of SROCC. The top three algorithms are presented in boldface for each distortion type

According to the table, the proposed model with standard deviation pooling works stably and robustly on most distortion types across the three databases, and finally reaches the highest hit number compared with all the competitors. Particularly, it performs better on the distortion types where structural changes occurs rather than contrast and intensity changes, since the PC operator measures how salient the edge is.

4.3 Comparison of running time

Since the computational efficiency plays an important role in practical applications, it is necessary to improve the operation speed and efficiency of IQA metrics. In Table 4, we present the running time of the proposed method and 9 competing FR-IQA models on each 512 × 512 image in average. Particularly, the running time of ${\mathrm{S}}_{FSIM}$ represents the result that we replace the PC algorithm in FSIM metric. We ran our algorithm and competitive conventional metrics using MATLAB R2019a on a personal desktop computer with Intel Core i5-6400 CPU @2.7 GHz and 8G RAM. The source codes of the competitors were provided by their authors.

Table 4 Running time of the proposed PC-based model and the competitors

According to Table 4, PSNR, GMSD and SSIM are the three fastest metrics owing to the low computational complexities, while the proposed ${q}_{\mathrm{sd},g}$ model ranks the 4th, since phase computation needs multi-scale calculation. The proposed PC-based model runs 5.38 times faster than FSIM, 10.62 times faster than IW-SSIM, and 22.71 times faster than VIF. In particular, when replacing the PC computation in FSIM with the proposed PC algorithm, the code runs 3.73 times faster than the original FSIM model, which validated that the proposed PC based on derivatives of Gaussian function is more efficient and less complex in computation than traditional PC method.

As a supplement for running time comparison, Table 5 shows the comparison of the running time of our proposed PC metric and conventional PC on 512 × 512 image where only the PC feature maps are calculated. Notice that the PC calculation here is done without down-sampling, which is different from the IQA metric calculation in Table 4. Obviously, the proposed PC metric runs more than 2 times faster than Gabor-based PC. More importantly, we only calculate on two different scales when replacing the PC algorithm in FSIM, since the two-scale feature shows better performance. The experimental results show that the proposed PC algorithm saves a lot of time compared with Gabor-based PC metric. This is mostly because of the simplicity of the computation on symmetric Gaussian function and the non-directional features generated by Gaussian derivatives, which shows less complexity compared with the Gabor-based features. In the experiment, both algorithms are implemented by MATLAB code only. Gabor-based PC uses Fast Fourier Transform (FFT) to separate odd and even components, where at least six parameters are necessary to control the scale and orientation selections. Meanwhile, a threshold is needed to penalize low PC values in order to reduce artifacts. On the contrary, the proposed circular symmetric PC is calculated in spatial domain with one scale factor for each scale, without consideration of direction selection and artifacts issues.

Table 5 Running time solely on PC calculation

5 Conclusion

In this paper, we proposed a novel algorithm of phase congruency map computation to represent quality-aware structural information of an image, and then proposed an FR-IQA model based on the quality feature. Instead of traditional multi-scale log-Gabor filters with multi-orientations, we utilized image gradient magnitude and Laplacian of Gaussian filters, which are the first-order and the second-order derivatives of Gaussian function, to generate the odd-symmetric and even-symmetric components of an image when computing the dimensionless phase congruency index. This calculation with Gaussian-based filters is much simpler in computation and more concise than traditional PC algorithm with log-Gabor filters. We have also validated that this phase congruency map contains enough structural information and can extract faint features such as the edges, lines, corners, and other local structures from both reference and distorted images, which makes it available to measure the degree of distortions.

The experimental results have indicated that the proposed method performs consistently and stably on different distortion types across three benchmark databases, while it is less computationally complex (faster to compute) compared with other outstanding metrics. Especially, the experiment on FSIM metric where we replaced the original PC algorithm with the proposed PC feature map shows that the first-order and second-order derivatives of Gaussian function can be constructed as an efficient PC alternative. Meanwhile, with performing similarly in prediction results but much faster in running time compared with conventional PC algorithms, the proposed PC shows to be a state-of-the-art feature map for IQA model design.

Although the proposed model works slightly better than GMSD, the PC feature based on circular symmetric Gaussian derivatives actually reflect the characteristics of image information where image components in different frequencies show similar responses in phase. Therefore, PC feature detects structural information at all kinds of phase angles, whereas image gradient mostly focuses on step features with a phase angle of 0 or 180 degrees. Despite that the gradient map in the proposed feature resembles the gradient magnitude in GMSD and FSIM, Gaussian derivatives are strictly circular symmetric filters, which are different from Prewitt or Sobel operator. Because of the non-directional properties and the ability to reflect image information in both odd and even phases, the proposed PC feature is expected to play an important role in image enhancement applications based on IQA features.

In conclusion, this paper proposes an efficient PC feature map based on derivatives of non-directional Gaussian function. This symmetric operator proves to be quality-aware and works stably in the proposed FR model with reduction in running time compared with conventional PC metrics. Therefore, the proposed feature map would play an important role in the image quality-related applications in future researches.