Self-feedback image retrieval algorithm based on annular color moments

In the era of Web 2.0, especially with the popularity of social networking sites such as Flickr and Facebook, unstructured data such as images, videos, and audios are growing at an alarming rate every day. How to quickly and accurately find the images one needs from a large-scale database has become the focus of research. Techniques for content-based image retrieval (CBIR) use only visual features of an image as a query, storing them in an image feature library [1]. Since these features are usually high-dimensional, storing and retrieving massive visual features are the main challenges for developing the CBIR technology. Hashing is one of the emerging technologies for supporting fast and accurate image retrieval that can be applied as an effective technique for CBIR [2]. The core idea is to map high-dimensional visual features to compact binary codes in the low-dimensional Hamming space, so that visual similarities of images can be efficiently measured using simple yet efficient bit operations. Using hashing as underlying indexing, the storage can be significantly reduced and the retrieval process can be completed rapidly [3].

The traditional cryptographic hashing method can map any input data to a fixed-length digital sequence; even if only one bit of the input data is changed, the output value will change drastically. In the field of image processing, image enhancement and compression are very common operations. These operations can change the specific representation data of an image, but cannot change the visual essence of the image, that is, the image hashing is maintained constant. Therefore, the hashing function in traditional cryptography is not suitable for image hashing calculation. Hashing functions for images exhibit two basic properties: perceptual robustness and uniqueness. Perceptual robustness means that visually similar images generate the same or a similar hashing value. Uniqueness means that the hashing value is completely different for images with different content [4, 5].

Recently, several image hashing methods have been proposed. The previous image hashing methods extracted some simple and effective global statistical characteristics to represent image hashes, including various histograms of images (e.g., luminance histogram, height histogram, color histogram, cumulative histogram, and cross histogram), mean, variance, color coherence vector, and color moments [6]. The image hashing methods based on global statistical characteristics can obtain a certain robustness of hashing, that is, the hashing extraction method is robust against a series of conventional operations such as image JPEG compression and filtering. However, this method has a serious flaw; it is not particularly sensitive to some minor illegal disturbances, which can lead to serious safety hazards. For example, the image content may completely be altered by malicious attackers during network transmission. Simultaneously, the tampered illegal images may maintain the same histogram, mean, and variance characteristics as those before tampering. Therefore, approaches to extract hashes based on this method are less secure. Later studies have proposed image hashing algorithms based on the transformation and time domains. Among them, the algorithms based on transformation domain can be divided into the discrete Fourier transform (DFT) domain [5, 7], discrete wavelet transform (DWT) domain [2], discrete cosine transform (DCT) domain [8], the integral transform Radon domain [9], etc. The image transformation domain can represent both image detailed characteristics and global contours. Different transformation domains can characterize different characteristics of images, and image characteristics extracted based on different domains can be adapted to different scenarios. Therefore, an appropriate transformation domain for image feature extraction can be selected as needed.

While many effective hashing algorithms have been proposed, there are still many problems to be solved in practical applications. For example, a hashing algorithm based on DWT can resist JPEG compression better, while that based on DCT has a better uniqueness; at the same time, their rotation robustness still needs to be improved. In addition, the existing hashing techniques suffer from two major limitations. First, most existing hashing algorithms are based on gray images. If a color image is input, it is represented by the luminance component. Because the algorithms do not consider information such as hue and saturation, the uniqueness is limited. Second, most hashing schemes employ only low-level visual features, and the well-known semantic gap degrades the CBIR performance [3]. To solve these problems, this paper uses the annular geometric segmentation method to extract the color moments of color images and combines fuzzy clustering and virtual correlation feedback technology to realize the self-feedback of image retrieval [10, 11]. The experimental results show that the proposed method is robust to rotation, scaling, and noise and exhibits good uniqueness.

The remaining paper is organized as follows. Section 2 describes the proposed color image hashing algorithm based on annular color moments. Section 3 describes the self-feedback image retrieval algorithm based on fuzzy clustering. Section 4 presents the experimental results and their discussion. Section 5 draws a conclusion of this paper.

Currently, most image hashing algorithms mainly deal with gray images. If a color image is input, it is represented by the luminance component of the YCbCr color space, ignoring the hue and saturation information, thus limiting the uniqueness of the algorithm. To solve this problem, annular color moments are proposed to represent hashing values of color images. This method is divided into three steps: preprocessing, feature extraction, and hashing generation.

2.2 Feature extraction

The color moment proposed by Stricker and Orengo is a very simple and effective color feature. The mathematical basis of this method is that any color distribution in an image can be represented by its moments. Because the color distribution information is mainly concentrated in the low-order moments, only the mean, variation, and skewness of a color are sufficient to express the color distribution of the image. The mathematical expressions of the three low-order color moments are

$$ {\mu}_i=\frac{1}{N}\sum \limits_{j=1}^N{P}_{ij}\kern5em ,i=1,2,3 $$

(7)

$$ {\sigma}_i={\left(\frac{1}{N}\sum \limits_{j=1}^N{\left({p}_{ij}-{\mu}_i\right)}^2\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.},i=1,2,3 $$

(8)

$$ {s}_i={\left(\frac{1}{N}\sum \limits_{j=1}^N{\left({p}_{ij}-{\mu}_i\right)}^3\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.},i=1,2,3 $$

(9)

where N represents the total number of pixels in the image and p_ij is the ith color component of the j-th pixel in the image. Therefore, the total color moment of the image requires only nine components: three color components (L*, a*, and b* in this paper) and three low-order moments per component.

The spatial information of the image is also a very important feature in image retrieval. Because color moments only reflect the overall characteristics of an image and do not express the spatial position of image colors, we use the annular geometrical segmentation method to record the color spatial information [13]. The process of extracting color moments based on annular geometric segmentation involves dividing an image into a circle, an M-1 annular, and a residual part at equal intervals according to the coordinates of the center point and then calculating the L*a*b* color moment feature vector of the segmented regions separately. The annular geometric segmentation of the image is shown in Fig. 1.Assuming W and H to be the width and height of an image, the radius of the circle and the annulus can be expressed as

$$ r=\left\{\begin{array}{c}W/2M,W\le H\\ {}H/2M,W>H\end{array}\right. $$

(10)

The image is divided into M annular parts according to its size. The color moment eigenvectors of the central annular region, the middle M-1 annular region, and the segmented remainder are calculated. The relation between the coordinates of an image pixel and the region number can be expressed as

$$ ZN=\frac{\sqrt{{\left(x-{O}_x\right)}^2+{\left(y-{O}_y\right)}^2}}{r}+1 $$

(11)

In the above equation, ZN denotes the region number, O_x denotes the abscissa of the center point of the part of the image that is of interest, O_y denotes the vertical coordinate of the center point of the image part of interest, and r denotes the image segmentation radius [13].

The final image hashing value can be obtained by connecting the color moments of three different color components in turn. In image retrieval, the results can be obtained by calculating and comparing the similarity between the corresponding color moment feature vector sequences of two images.

At present, most image retrieval systems are based on low-level visual features (such as color and texture), and the retrieval process is computer-centric, which makes its retrieval performance unsatisfactory. This is mainly due to the gap between low-level visual features and high-level semantic concepts. In addition, the standards of judging the similarity between images by human and computer image retrieval systems are very different. To solve these problems, a CBIR method based on relevance feedback (RF) is proposed. This method embeds the user model in the retrieval system and establishes a correlation between the low-level features of the image and its high-level semantics via human-computer interaction, thereby reducing the semantic gap and achieving a high retrieval accuracy [14, 15]. However, RF’s unique complex human-computer interaction interface increases the burden on users. In each feedback, the user must indicate which images are related and score the degree of correlation between these images and the query image. In some cases, the user will be confused and face difficulty in giving an appropriate judgment [16]. Therefore, some scholars have proposed the pseudo-relevance feedback (PRF) technique, which is also called local feedback or blind feedback. The main idea is to improve the retrieval performance of a system by extending the query on the basis of a reasonable assumption that some documents ranked at the top are assumed to be related to the query documents and related documents are selected automatically. The strategies of selecting positive examples and the query expansion method are the key issues in applying the virtual relevance feedback technology [15–17]. Based on the above research, this paper proposes a method for image retrieval self-feedback based on fuzzy clustering and the virtual relevance feedback technology. This method is mainly based on the similarity measure between the image content features; it applies the fuzzy clustering algorithm to automatically expand the query image features without user intervention and to improve the search performance.

3.1 Related technologies

Content-based relevance feedback retrieval is a process of gradual refinement. The basic idea is that the user is allowed to evaluate and mark the retrieval results in the search process, indicating which of the results are related to the query image and which are irrelevant, i.e., “feedback positive examples” and “feedback negative examples.” The system receives the user feedback on the current retrieval results and then automatically adjusts the query based on the feedback information. Finally, the optimized query is used to recalculate the retrieval results. In this way, the system and user gradually make the retrieval result to move toward the direction of the user’s expectation through the interaction and to finally reach the user’s request.

At present, the vector model is often used in image retrieval. That is, the image is represented in the vector form in the feature space; hence, the essence of image retrieval is to find the images corresponding to those closest to the query vector in the feature space. From the viewpoint of the vector model, the correlation feedback techniques can be divided into two categories: the point moving algorithm based on the query vector and the feature weight adjusting algorithm. These correlation feedback methods are based on the premise that the image visual features can completely describe the image semantics. Under this assumption, the user’s retrieval target can be described by a global distribution function. That is, similar images in the feature space are clustered around the central point. Therefore, the retrieval effect can be improved by moving the query points and modifying the distance measurement functions.

In practical applications, however, there are significant differences between the low-level features of images in the same category due to the diversity of semantically similar images. For example, the elephant class in the Corel image library includes elephants with different postures (e.g., standing and lying) and in different environments (e.g., forest, grassland, and water edge), as shown in Fig. 2. In terms of low-level features, the assumption that semantically similar images cluster around the central point in the feature space is not always true. As shown in Fig. 3, where the dot Q represents the query image, the solid dot represents the retrieval target, the hollow dot denotes irrelevant images, and L(Q) indicates the distribution of the user’s retrieval targets; clearly, the distribution does not have an ideal global distribution center. It is difficult to describe it with some kind of canonical geometry. In this case, if the query image is taken as the center in the feature space and the image in the hypersphere with a constant radius is returned as the retrieval result, it is obvious that some of the related images will be missed, while some uncorrelated images will be misdirected. However, as shown in Fig. 4, when L(Q) is divided into three sub-clusters, each cluster is uniformly distributed. The model can reflect the distribution of the correlated images in the feature space well; it also overcomes the shortcomings of the previous correlation feedback methods, which assume that semantically similar images cluster around the central point in the feature space, so it is helpful in finding more correlated images [15].

Based on the above ideas, a self-feedback strategy based on fuzzy clustering is proposed in this paper. First, the initial retrieval results generate multiple substitution queries using fuzzy clustering to form a new query. To some extent, this compensates for the shortage of image information in a single query sample. Then, these substitution queries are retrieved and merged with the initial retrieval results to form the final retrieval results.

This paper proposed an image hashing method combining annular color moments and virtual correlation feedback. The color moments based on annular segmentation were used to describe the color images, and the red, green, and blue components of the pixels were fully utilized to improve the performance of the method. Experimental results showed that the proposed method is robust to JPEG compression, brightness and contrast adjustment, image scaling, and rotation. Furthermore, this paper proposed a self-feedback method for image retrieval based on fuzzy clustering and virtual correlation feedback. This method is mainly based on the similarity measurement between image content features and applies the fuzzy clustering algorithm to automatically extend query image features without user participation to improve retrieval performance. The experimental results showed that the method can improve the retrieval accuracy by using a limited number of feedbacks.

The current algorithm only considers the color feature, and the future research will include designing a hashing algorithm that can extract the image texture, its shape, and other related features to further optimize the algorithm and improve its retrieval performance. At the same time, we will combine hashing and other machine learning methods to solve the large-scale search problems effectively.