Exemplar-based image inpainting using angle-aware patch matching

With the advancement of society and the rapid development of the Internet, image completion, also called image inpainting, has been applied to many fields proverbially, such as the protection of ancient relics, image editing, medical field, and military field. The technology is originally developed to renovate damaged photos and films or to remove unwanted texts and the occlusion from images in a plausible way, etc. All image completion algorithms are based on a hypothesis that the completed region and the missing region have the same statistical property and geometric structure. In addition, the inpainted image should satisfy the human visual consistency requirement as much as possible. That is, the colors, textures, and geometric structures of the inpainted regions should be similar to those of the ambient regions. Based on this assumption, many researchers have put forward their researches and obtained significant success. In the next paragraphs, we introduce several main completion methods and the classification diagram is shown in Fig. 1.

Currently, image inpainting methods have been mainly divided into two categories. The first category is diffusion-based methods [1–4], where parametric models are established by partial differential equations in order to propagate the local structures from known regions to unknown regions. Bertalmio et al. [1] proposed the Bertalmio–Sa-piro–Caselles–Ballester-based inpainting method, in which the information around the defective area was propagated from the outside to the inside along the direction of the isophotes in incomplete regions, thereby obtaining the restored image from the damaged one. Motivated by the idea, Chan and Shen [2] proposed the total variation (TV) model which can restore and maintain edge information, while performing denoising by using anisotropic diffusion. But, the method merely depended on its gradient values rather than geometric information of the isophotes, which led to the applicability of small missing regions. Furthermore, Chan and Shen proposed curvature-driven diffusions (CCD) model [3] in which the diffusion process took structure information of contour (curvature term) into account. When there was a large curvature anywhere, the isophotes became strong and then gradually weakened as the isophotes extended. It suppressed the large curvature and protected the small in the restoring process so that the “connectivity criterion” was satisfied. Therefore, compared with TV model, the CCD model can repair not only images with large damaged areas but also fine edges, especially for gray-scale images. In recent years, other diffusion-based methods have been studied for restoring images [5–8]. Biradar and Kohir [6] made use of median filter to preserve important properties of edges through diffusing median information of pixels from the outside to the inside in inpainted region. Prasath et al. [8] combined total variation with regularization to solve ill-posed image processing problems. In short, the diffusion-based methods can only restore natural and small-scale images with lower structures of texture and geometric. When containing large holes (e.g., the missing regions) or complex textures, the resorted part will be over-smoothed and produce unpleasant artifacts which led to the inconsistency of structure and texture.

Therefore, the second category is exemplar-based methods which have been presented to complete an image with a large missing region [9–13]. Exemplar-based inpainting methods filled in the missing region at a pixel level or a patch level. Due to massive runtime and inconsistent texture synthesis at a pixel level, Efors and Leung [9] proposed a patch-based inpainting method to fill in the unknown region by using texture synthesis. Afterward, Criminisi et al. [10] presented a classical inpainting method to remove a large object according to the computation of priority and similarity of patches, while preserving important information of texture and structure. After that, many other exemplar-based methods have been proposed. For example, Sun et al. [11] proposed a global structure propagation method which connected the geometry structure of the whole image manually and then propagated the known texture to an unknown region. Wong et al. [12] proposed a non-local mean by utilizing multiple samples in an image to obtain more similar source patches. Based on the method achieved in [11], Li and Zhao [14] proposed an automatic structure completion method which avoided manual intervention and enhanced the efficiency of the algorithm. In addition, the dropping effect of a confidence term was also an obvious drawback, so Wang et al. [15] introduced a regularized factor to limit the adverse effect and improved matching accuracy through a two-round search. Hereafter, a multi-scaled space method using a multi-resolution hierarchy was proposed by Kim et al. [16] and Liu et al. [17] that it can reduce workload and restore more textural information and structural features. Wang et al. [18] improved the priority estimation and structure consistency considered in patch matching. These methods were proud of accomplishing the textural and structural coherence inside the cavity.

Unlike the aforementioned exemplar-based methods, other methods were also applied to image inpainting and got satisfactory inpainted results. For example, Komodakis et al. [19] proposed the discrete global optimization tactics optimized by priority-BP based on a Markov random field. But this method was time-consuming. Subsequently, Ruzic et al. [20] presented context-aware patch-based image inpainting which divided the image into patches with variable sizes. MRF encoded consistency of neighboring patches. To fill large holes surrounded by different types of structure and texture, Alotaibi and Labrosse [21] compared with the previous methods in terms of speed and performance and obtained significant improvement. Ge et al. [22] made use of an optimal seam method to synthesize texture seamlessly and proposed patch subspace learning to limit patch selection. The problem was solved by EM-like algorithm, but it was sensitive to the initial values. Zhao et al. [23] proposed a coherent direction-aware patch alignment scheme based on GPU to enhance the similarity between matching patches, so the runtime can be reduced compared with similar methods. Huang et al. [24] proposed an automatically guiding patch-based image completion which considered patch transformation and used the mid-level constraints to guide the filling process.

In this paper, our novel scheme is performed in a dynamic manner. We initialize the unknown regions of a damaged image with the help of the MLS method. According to a dynamic patch selection process, small target patches are applied in the high-frequency region to maximize the restoration of the structural information, while large patches can reduce the computational workload in the low-frequency region. By developing the angle-aware patch matching with a Jaccard similarity coefficient, the process of patch matching can be performed better to advance matching accuracy. And the proposed approach can select multiple matching patches automatically from the available region by using an angle-aware rotation strategy to increase the probability of obtaining the optimal matching patch. It is worth mentioning that we also improve the priority function based on [10] for the selection of the target patch. In summary, the main contributions of this paper are as follows:

The remainder of this paper is organized as follows. Section 2 briefly narrates the classical exemplar-based method. Section 3 introduces the proposed approach in detail. Section 4 compares the proposed method with other state-of-the-art methods in some real images. The proposed method is compared with deep learning for image inpainting in Section 5. Finally, we draw a conclusion for this paper in Section 6.

In exemplar-based image inpainting methods, the most famous method was proposed by Criminisi et al. [10], which can restore structural and textural information of the large damaged region simultaneously. Given an image I decomposed into two parts as shown in Fig. 2, the source region (pixels are known) and the target region (pixels are unknown), are represented by Φ and Ω respectively, where Φ = I − Ω. δΩ denotes the boundary line connecting Φ and Ω together. Since the point filled earlier can affect the points filled afterward [18], we need to determine the priority of every point p ∈ δΩ by a priority function. The process is described as follows briefly. Firstly, a target patch Ψ_p centered at a point p with the highest priority is selected preferentially and it contains both the known pixels and the unknown pixels. Secondly, a source patch Ψ_q centered at a point q ∈ Φ is selected to match the above target patch. Finally, the defective region is filled by copying the corresponding pixels in Ψ_q to the unknown pixels within Ψ_p.

where ⊥ and ∇I_p are an orthogonal operation and the gradient centered at pixel p respectively. Thus, \( \nabla {I}_p^{\perp } \) denotes the isophotes vector orthogonal to the gradient ∇I_p, and n_p is a unit vector orthogonal to the boundary δΩ. α is a normalization factor (α = 255 if I is a gray image). Finally, through calculating the priority function, we can find the point \( \hat{p} \) with the highest priority, and construct the target patch centered at it.

After filling\( {\Psi}_{\hat{p}} \), the boundary δΩ is updated iteratively until the unknown region is filled entirely.

In this section, we test the proposed approach on all kinds of natural and textural images which are selected from references as well as the Berkeley image dataset. We compare our approach with other inpainting methods including Criminisi’s algorithm [10], image melding using patch-based synthesis by Darabi et al. [28], image restoration via group-based sparse representation (GSR) by Zhang et al. [29], and annihilating filter-based low-rank Hankel matrix approach (ALOHA) proposed by Jin and Ye [30]. Our experiments are simulated using Matlab 2016b in WIN10 system with Intel(R) Core(TM) i5-4590 CPU (3.30 GHz). We apply our algorithm to inpainting of missing blocks and object removal task for color images. And our approach can also be applied to a grayscale image. In our experiments, the parameters are set as follows: λ₁ and λ₂ is set as 0.7 and 0.3 respectively. Furthermore, we search for n = 3 the most similar source patches for every target patch. And, we set η as 0.3 in Eq. (13) to balance the contribution of the gradient features. We set the search range as \( \left[-\frac{\pi }{2},\frac{\pi }{2}\right] \) for rotation. The approach is fairly robust in the variation of this range.

4.2 Recovery of multiple block losses and large holes

We first apply our approach to restore the damaged images with multiple missing blocks. We compare our approach to Crinimisi’s algorithm [10] and the experimental result is shown in Fig. 7, in which nine missing blocks are all set to 20 × 20. We can observe clearly that her face is mottled and the stripes on the headscarf are also blurred in Fig. 7c. That is, Crinimisi's algorithm fails to recover textures and structures of missing regions. However, Fig. 7d displays the inpainted result of our approach which can restore both complex textural and structural details. In contrast, our method can produce a satisfactory visual effect.

To further demonstrate the advancement of our approach, we give average image recovery accuracy according to PSNR and SSIM on a dataset of thirty images with nine missing blocks taken from the Berkeley image dataset. In Fig. 8, we compare the average PSNR and SSIM of every missing block with other methods respectively. From Fig. 8a, b, we notice that average PSNR and SSIM of the proposed algorithm are higher than those of other methods in a majority of blocks. When missing regions are smooth, Criminisi’s and GSR algorithms can also obtain satisfactory and natural effects. But if the missing regions have complex textures and structures, these regions will be filled using error contents. In addition, ALOHA’s result is generally satisfactory, but it is still slightly inferior to our method. Since the proposed method restores the damaged regions using dynamic patches whose sizes are determined by gradient values, our method makes the inpainted image more reasonable in visual, and decreases the run-time. When gradient value is smaller than γ, we set the size of the target patch as 5 × 5, otherwise set as 7 × 7. Therefore, according to the above experimental results, we can prove that the proposed method is more effective for images with rich textures and structures.

Then our method is also suitable for filling a large hole inside images. For example, in Fig.9 we mainly give the inpainting results about filling a large hole set as 70 × 60 in an image of size 256 × 256. Fig.9 (a) is an original image, and Fig. 9 (b) is the masked image with a target region of a rectangular block. Figs. 9 (c)-(g) are the inpainting results of the Criminisi algorithm, GSR, ALOHA, image melding, and the proposed algorithm. Clear errors can be observed in Figs. 9 (c)-(e) in which the textures of the butterfly wing are very different from the original one. From Fig. 9 (f) we observe that the inpainting result is better than the previous methods in terms of inpainting textures, but it is slightly inferior to ours as shown in Fig. 9(g). Therefore our approach can better restore not only a large hole but also textural and structural information.

4.3 Object removal

We also test our approach on the object removal task in Fig. 10. The performance of the object removal task relies on the two main aspects: image naturalness and the runtime. In this subsection, we show the results of the various algorithms on a variety of color images of size 383 × 256 by discussing these aspects. The removed objects in these images are shown in Fig. 10 row 2 using black block masks. From the first image in row 3, we notice that the cloud generated by Criminisi’s algorithm is not smooth and not natural enough, and the connection with the rainbow also produces the mottled content. In contrast, the proposed algorithm generates relatively natural contents compared with the original image. As we have seen in row 4, GSR-based algorithm gives rise to the most unsatisfactory result, and we can clearly observe that the inpainted regions have serious inconsistency with surrounding contents. From Fig. 10 row 5 of column 2, we observe that the reflection of a person has not been reconstructed by Melding algorithm and is discontinuous. However, our algorithm and Crinimisi’s algorithm can both restore the shadow better. In column 3, these methods yield the severely blurred contents in the auditorium. Overall, compared with these methods, our method generates more plausible details without sacrificing the naturalness. From these results, we discover that the restoration of texture and structure plays an important role in the whole inpainting process. By comparing with other methods, experimental results prove that our method can complete more realistic contents in visual.

In Section 3.2, we proposed that the size of the target patch is selected dynamically. The patch size is a pivotal parameter in patch-based algorithms. Larger patches capture more structures and edge details, if good matches are found. However, if such matches are not found, the result can easily converge to a blurry solution. Thus, we suggest that if the gradient of point \( \hat{p} \) with the highest priority is higher than γ, we set the size of the target patch centered at the point \( \hat{p} \) as 5 × 5. Otherwise, it is set as 10 × 10. If patches are too large, error contents will be yielded. However, smaller patches may consume much runtime. Therefore, we exploit dynamic patches to preserve the structures as well as strong edges in transitional areas, meanwhile, to advance the performance efficiency of our approach. Consequently, Fig. 11 gives the comparison between a fixed patch set as 7 × 7 and dynamic patches, where we mark the inpainted roof with a red bold square. Obviously, the chasm is generated on the inpainted roof and structures are discontinuous as shown in Fig. 11b. In contrast, the inpainted result using dynamic patches is more acceptable and reasonable in visual in Fig. 11c. To prove the advantage of dynamic patches, we compare the mean runtime of our approach for 30 images with Criminisi’s method, GSR, image melding, and ALOHA on the Berkeley image dataset shown in Table 1. We find that our approach is faster than other inpainting methods except for Criminisi’s method. This is mainly because we use the Gaussian convolution in the computation of the similarity metric and search for multiple source patches for each target patch.

Algorithms	Runtime (average)
Criminisi	00:02:10
GSR	00:10:16
Melding	00:08:57
Aloha	02:24:36
Ours	00:03:45

Recently, deep learning has been diffusely applied to image completion to extract high-level features of images. Pathak et al. [37] proposed context encoders motivated by feature learning which used a convolutional neural network (CNN) trained with a reconstruction plus an adversarial loss to generate the contents of an arbitrary image region. Iizuka et al. [38] proposed a novel architecture based on context encoders with global and local context discriminators, which resulted in both globally and locally consistent images. Yu et al. [39] presented a unified feed-forward generative network using a novel contextual attention layer for image inpainting, which is trained end to end with reconstruction losses and two Wasserstein GAN losses to generate higher-quality inpainting results.

Undoubtedly, deep learning-based inpainting methods may achieve more meticulous results by utilizing a convolutional neural network to extract high-level features. However, image inpainting algorithms using the high-level features require very complex models and a large amount of parameters to adjust, especially the higher the accuracy of the model is, the worse the general robustness will be. And they need also a number of external training data that needs to be collected in the same circumstances as the data set used. In a fast-developing society, the quality of the inpainting result is not the only criterion to evaluate the effectiveness of the algorithm, since the runtime is also a significant factor that cannot be ignored. Therefore, the proposed method is more robust for image inpainting based on low-level features. And our method has fewer parameters and can be adjusted easily. Furthermore, the computational time is much shorter than that of deep-learning methods. Our method will be further improved in the future to achieve better visual quality.

In this paper, we have proposed a novel inpainting method which estimates the missing pixels by MLS method in 3D subspace as the prior knowledge utilized to solve the confidence term. And we also add a curvature factor for the data term to avoid its value of zero. We propose an angle-ware rotation patch matching strategy which considers the different angles of the same source patch in order to find multiple candidate patches for every target patch, thereby increasing the matching accuracy. Meanwhile, a Jaccard similarity coefficient is used to enhance the textural and structural similarity between the target patch and each corresponding source patche. In order to improve the restored efficiency, we divide the whole image into high-frequency and low-frequency components and introduce the gradient to determine the size of target patch dynamically. Our method is applied to the filling with a large hole, the restoration of multiple missing blocks and object removal task. And a large number of experimental results demonstrate that the proposed method is superior to other advanced methods and the runtime is shorter. In the future, we will extend our approach to image compression field.