A hierarchical graph model for object cosegmentation

2.1 Clustering based methods

Under the assumption that similar objects often recur in multiple images, clustering based methods employ clustering models to discover such frequent regions. The well-known clustering models include topic discovery models like probabilistic latent semantic analysis (PLSA) [20], and geometry based models like normalized cuts (NCut) [21]. Motivated by the success of topic discovery in text analysis, Russell et al. [5] first adopt PLSA to address the cosegmentation problem. Later, Cao et al. [6] and Zhao et al. [7] both present spatially coherent topic models to encode the spatial relationship of image patches which is ignored by the traditional topic models. Combining NCut and supervised classification technique, Joulin et al. [19] utilize a discriminative clustering scheme to tackle the cosegmentation problem. For speeding up, all clustering based methods take superpixels as computation nodes. The major limitation of these methods is the lower segmentation accuracy caused by the over-segmentation methods.

2.2 Labeling based methods

Considering the Markov property in the images, labeling based methods formulate cosegmentation as a Markov random field (MRF) energy minimization problem. Over the past decade, methods that use graphcut [12] to minimize MRF energy have become the standard for figure-ground separation.

One technique is to minimize an energy function that is a combination of a pairwise MRF energy and a histogram matching term. The histogram matching terms such as L 1 norm model [1], L 2 norm model [22] and “reward” model [23] force foreground histograms between a pair of images to be similar. Vicente et al. [3] review these models and make a comparison. Yet these methods are limited to two images. Another technique, also called unsupervised object segmentation such as LOCUS [8], ClassCut [9], Arora et al. [10] and Chen et al. [11], performs object cosegmentation by iteratively learning the object geometric models and segmenting the foreground objects. The initialization stages of these methods play an important role for energy minimization. For example, LOCUS [8] takes the pre-trained mask and edge probability maps as the initial object models, ClassCut [9] uses a general object detector [24] to locate objects. However, these methods are limited to segmenting objects with similar geometric shape. In contrast, the recently proposed cosegmentation method—BiCos [18] is more general and can be applicable for any non-rigid objects. BiCos [18] operates at the two levels: the bottom level treats each image separately and uses graphcut [12] to refine foreground objects in pixel level, whereas the top level takes superpixels as computation units and employs a discriminative classification to propagate information among images.

Our method falls into the last category. The main idea is to combine multiple schemes along a three-level hierarchical graph to refine foreground objects successively. In contrast to other labeling based methods [3, 8–11, 18], this method has the following characteristics: (1) utilization of heat sources for message propagation among images, which can significantly reduce computation time; (2) a saliency detection based initialization, which can remove the impact of background stuff; (3) instead of using graphcut [12] to refine objects in the pixel level, we introduce guided filtering [16] for local refinement. In experiments, we compare our method quantitatively and qualitatively with other state-of-the-art methods over several public datasets. As a outcome, our method achieves better segmentation quality as well as lower computation time.

3.1 Problem formulation

where $E_{d_1}\left(I_k\right|L_k)$ is derived from saliency detection, and $E_{d_2}\left(I_k\right|L_k,{\theta }_k)$ is inferred under the guide of an inherent object model. θ _k is the latent parameter set for the object model of I _k .

The pairwise term can be considered as a smooth term, which penalizes the inconsistent labeling among images. Ideally, this term should be formulated in the pixel level. For computational efficiency, we define it in the heat source level using appearance information (see Equation (8)). Minimizing the above energy with respect to all discrete labels is intractable. Instead, we relax the labels firstly, i.e., let L _k (x) ∈[0,1] be the object likelihood, and iteratively update them along a hierarchical graph model, finally obtain the segmentation results by rounding.

3.2 The hierarchical graph and our method

As shown in Figure 1, the graph model is composed of three types of nodes: pixels, superpixels and heat sources. For each image, superpixels are the clustering units of coherent pixels, and heat sources are the representative superpixels located in the centers of the clustering regions formed by coherent superpixels. In our implementation, the superpixels are extracted by an over-segmentation method—Turbopixels [14]. The generation of heat sources will be described in detail in Section 4.2.

Based on the graph model, our method successively updates the object likelihoods by the following iteration: (1) estimating the latent parameters and refining object segmentation, (2) transferring message among images and diffusing heat energy within individual image. Specifically, we first obtain the object likelihoods in each image with saliency detection [17], and then estimate the latent parameters to update the object likelihoods. The likelihoods of the heat sources are further updated among images via message transferring which is fulfilled by belief propagation [13], and diffused to other superpixels using random walks [15] within individual image. Now the likelihoods can be considered as input for further iteration. In the following sections, we denote the updated object likelihoods at different stages by $L_k^{\ast ,t},t=0,\dots ,3,$ k=1,…,K. To summarize the cosegmentation method presented in this article, we provide a high level overview of the method pipeline asfollows.

4.1 Initialization and local refinement

One major visual characteristic of objects is that they often stand out as saliency [24]. Based on this characteristic, we apply saliency detection to initially detect foreground regions in each image. Over various of saliency detection methods, we choose a recently proposed histogram based method [17] for its efficiency and effectiveness. Figure 2b demonstrates the saliency detection result of Figure 2a. We define the initial object likelihoods $L_k^{\ast ,0}$ as the saliency likelihoods.

Segmenting objects by directly thresholding the updated object likelihoods will result in noises, as shown in Figure 2c. We use guided filtering [16] to remove noises. The main idea of guided filtering [16] is that, given the filter input p, the filter output q is locally linear to the guidance map I, q _i  = a _x I _i  + b _x , ∀i ∈ w _x , where w _x is a window with radius r centered at the pixel x. By minimizing the difference between the filter input p and the filter output q, i.e., $\mathit{\text{Err}}(a_x,b_x)={\Sigma }_{i\in w_x}({(p_i-q_i)}^2+ϵa_x^2)$, we can obtain a _x , b _x and the filter output q.

Based on guided filtering [16], we perform local refinement with three steps: (1) obtaining the foreground likelihood map L _k,F ={p(I _k (x)|θ k F)} and the background likelihood map L _k,B  = {p(I _k (x)|θ k B)}; (2) taking the grayscale image of I _k as the guidance map, the two likelihood maps are filtered, respectively (denoting the filter outputs as ${\widehat{L}}_{k,F}$ and ${\widehat{L}}_{k,B}$); (3) defining the refined object likelihoods as $L_k^{\ast ,1}={\widehat{L}}_{k,F}/({\widehat{L}}_{k,F}+{\widehat{L}}_{k,B})$. Figure 2d shows the refinement result of Figure 2c. As can be seen, the guided filtering based scheme can significantly improve segmentation quality.

4.2 Global message transferring

Due to the diversity of realistic scenes, saliency based object segmentation sometimes fails to extract objects of the same class (see Figure 3c). The segmentation quality can be further boosted by sharing appearance similarity among images. Unlike other cosegmentation methods [8, 9, 18] which propagate the distributions of visual appearance in the pixel or superpixel level, we perform message propagation in the heat source level to reduce computation time.

As stated in Section 3, heat sources are the representative superpixels located in the centers of the clustering regions formed by coherent superpixels. The regions are formed by a bottom-up agglomerative clustering scheme. Specifically, given an image I, we first partition it into a collection of superpixels via Turbopixels [14] (see Figure 4b, in which superpixels are encircled with red boundaries). Then we build an intra-image graph G _S  = <S,Y _S  >, where S = {s _i } is the superpixel set and Y _S  = {(s _i ,s _j )} is the edge set connecting all pairs of adjacent superpixels. The edge weight is defined by Gaussian similarity between the normalized mean RGB color of the nodes, i.e., w(s _i ,s _j ) = exp(-∥I(s _i )-I(s _j )∥²)/σ _s , where σ _s is a variance constant. Based on the graph G _S , we use a greedy scheme to merge nodes one by one. Each time, we select the edge with the maximum weight value and merge its two nodes. This step is repeated until all nodes are merged into N regions. The central superpixel of each region is chosen as a heat source. Figure 4c demonstrates the clustering regions overlaid by the heat sources, in which the regions are encircled with green boundaries and the heat sources are colored in blue.

where λ is the weighting value balancing the trade off between the unary terms and the pairwise terms.

where w(z _i ,z _j ) is the edge weight, defined as w(z _i ,z _j )= exp(-∥f(z _i )-f(z _j )∥²)/σ _z , σ _z is a variance constant. f(z) is a nine-dimensional descriptor for the heat source z, including three-dimensional mean Lab color feature, four-dimensional mean texture feature^a and two-dimensional mean position feature. This definition suggests that the larger the weight for the edge, the more similar the labels for its two nodes.

Finally, a belief vector is computed for each node, $b_{z_i}\left(L\right(z_i\left)\right)=E_1\left(z_i\right)+\sum _{z_j\in Z}{m}_{z_j\to z_i}^T\left(L\right(z_i\left)\right)$, and the updated object likelihoods are expressed as: $L^{\ast ,2}\left(z_i\right)=b_{z_i}\left(0\right)/\left(b_{z_i}\right(0)+b_{z_i}(1\left)\right)$.

4.3 Local heat energy diffusion

where w(x _i ,x _j ) is the edge weight for the adjacent node pair (x _i ,x _j ).

where Q = D-A is the Laplacian matrix, in which A = {w(s _i ,s _j )} is the edge weight matrix, and D is a diagonal matrix with the entities $D(s_i,s_i)=\sum _{j}w(s_i,s_j)$.

where $Q_{Z_k}$ and $Q_{U_k}$ correspond to the Laplacian matrix for the node set Z _k and U _k , respectively.

Minimizing E(L(S _k )) is equal to differentiating E(L(S _k )) with respect to L(U _k ) and yields: $L\left(U_k\right)=-B^{T}L\left(Z_k\right)/Q_{U_k}$. L(Z _k ) are the object likelihoods acquired in the previous stage, i.e., L(Z _k ) = L^∗,2(Z _k ). The diffused object likelihoods for U _k are obtained by: $L^{\ast ,2}\left(U_k\right)=-B^T{L}^{\ast ,2}\left(Z_k\right)/Q_{U_k}$. The nonsingularity of $Q_{U_k}$ guarantees that the solution exists and is unique.

For each pixel x, its object likelihood L^∗,2(x) is assigned as the object likelihood of the superpixel it belongs to. Taking $L_k^{\ast ,3}\left(x\right)=\left(L_k^{\ast ,0}\right(x)+L_k^{\ast ,1}(x)+L_k^{\ast ,2}(x\left)\right)/3$ as input, we further invoke local refinement (see Section 4.1) to optimize object segmentation. Figure 3 demonstrates the segmentation results obtained before and after heat energy diffusion. As can be seen, although the saliency based initialization stage sometimes fails to extract the foreground objects, the stages of message transferring and heat energy diffusion can boost segmentation quality via sharing visual similarity of objects among images.

5.1 Datasets and implementation details

5.2 Evaluation on Weizmann horses and Caltech-4

As can be seen, LOCUS [8], ClassCut [9] and Arora et al. [10] achieve better performance on the horse, motorbike and face categories, respectively. The reason is that the geometric models employed in those methods can strongly separate the foreground and background regions. Yet BiCos [18] and our method can still achieve competitive performance even without geometric models. Our method outperforms BiCos [18] on the airplane, face and motorbike categories, while BiCos [18] performs better on the horse category.

5.3 Evaluations on Oxford flowers, UCSD birds and CMU iCoseg

As baselines, three state-of-the-art methods (Joulin et al. [19], CoSand [4], and ClassCut [9]) are evaluated using their implementations with the default parameter settings. Joulin et al. [19] is a clustering based method, which takes superpixels as basic units and utilizes discriminative clustering to find common objects. CoSand [4] takes the large coherent, appearance similar regions among images as the foreground objects. ClassCut [9] is an energy iteration based method, which first obtains object bounding boxes by [24], and then builds a common class model with color, shape and position cues, finally extracts foreground objects via iteratively optimizing an MRF energy function and updating the class model.

The segmentation accuracy is defined as the proportion of pixels correctly classified as foreground or background by comparing the segmentation results with the ground truth. We take the form: F _Measure = 2 ∗ pre ∗ rec/(pre+rec), where pre is defined as the ratio of true positive pixels (i.e., the pixels labeled as foreground actually belong to foreground) to all labeled foreground pixels, and rec is defined as the ratio of true positive pixels to ground truth pixels. The average segmentation accuracies across all images are shown in Table 2. Several examples from the Oxford flowers, UCSD birds and CMU iCoseg datasets can be seen in Figure 5.

	Oxford flowers		UCSD birds		CMU iCoseg
Method	Accuracy	Time(s)	Accuracy	Time(s)	Accuracy	Time(s)
CoSand [4]	0.68	39.21	0.42	37.50	0.52	23.90
ClassCut [9]	0.72	95.96	0.32	93.71	0.51	78.43
Joulin et al. [19]	0.70	33.07	0.35	19.44	0.43	19.19
Our method (initial)	0.67	-	0.52	-	0.64	-
Our method (final)	0.84	24.14	0.68	13.11	0.74	11.19

The values in bold indicate the best results.

5.3.1 Overall performance

As illustrated in Table 2 and Figure 5, our method outperforms the three methods in terms of segmentation accuracy as well as computation time. The method of Joulin et al. [19] takes superpixels as basic units, thus the objects’ boundaries are not clearly delineated as some superpixels merge foreground and background regions together. CoSand [4] only focuses on extracting the large coherent regions, it performs poorly for the figure-ground separation task. For example, it only extracts the black regions in the panda image set, failing to detect the white regions as foreground objects. ClassCut [9] can extract most of foreground regions, while it tends to omit some fragile regions like the petals in the Oxford flowers dataset. This is because the over-segmentation method it adopted has merged the boundaries with backgrounds. In contrast, our method can extract the whole foreground object accurately, no matter it is composed of one or several appearance distributions. We attribute this to the initialization scheme and the appearance sharing among images.

The benefit of segmenting all images together has been qualitatively shown in Figure 3. In Table 2, we quantitatively compare the segmentation accuracies obtained before and after sharing appearance similarity among images, obtaining that the accuracies are improved from 0.67, 0.52, 0.64 to 0.84, 0.68, 0.74 for the Oxford flowers, UCSD birds and CMU iCoseg datasets, respectively. Figure 6 compares some segmentation results obtained in the initialization and last stages. We can observe that most errors induced in the initialization stage are rectified finally.

5.4 Failure cases

Our method works under an assumption that the interested objects should stand out as saliency. Yet such an assumption may not hold in some cases. Figure 7 illustrates some failure cases of our method for the images from the UCSD birds, Oxford flowers and CMU iCoseg datasets. As illustrated, although the bird, flower, and panda regions recur in the image sets, they are not too distinct with other regions to be detected as saliency. Our method fails to separate them from the backgrounds under such cases.