A de-noising method based on L0 gradient minimization and guided filter for ancient Chinese calligraphy works on steles

Abstract

A clear stele image of ancient Chinese calligraphy pieces is very useful for studying ancient Chinese calligraphy. However, due to hundreds of or even thousands of years of natural or artificial damage on stele, images of ancient Chinese stele calligraphy works usually suffer from a large amount of image noise, and which usually leads to a poor visibility. To address this problem, in this paper, we propose a de-noising method based on L0 gradient minimization and guided filter. It consists of two main operations in sequence: First, L0 gradient minimization is utilized to obtain a random-noise free map, and then the random-noise free map is used as a guided image, and convoluted with its corresponding original noised stele image by a guided filter to obtain an edge preserved random-noise free image. Finally, the eight-connection region-based de-noising technique is followed to remove ant-like isolated blocks. Experiments demonstrate that the proposed method is superior to several recent published stele image de-noising techniques in terms of preserving the character structures.

1 Introduction

Ancient Chinese calligraphy works are very useful for people to learn and study Chinese history and culture [1]. From these works, we can enjoy the beauty of ancient Chinese culture and learn about how the ancient Chinese lived; furthermore, we can learn about where Chinese history and culture come from and where it will go. Of these calligraphy works, most of them used steles as their carriers. However, because of hundreds of or even thousands of years of natural or artificial damage on steles, images of ancient Chinese stele calligraphy works usually suffer from poor visibility with a large amount of noise, and the structure of characters in regions of the stele images is too spoiled by noise to see, as shown in Fig. 1. Hence, it is highly desirable to remove noise from ancient Chinese stele calligraphy images.

Fig. 1

An image of ancient Chinese calligraphy works on stele with a large amount of noise

However, to the best of our knowledge, though there are many works on image de-noising [2,3,4], few works have focused on ancient Chinese calligraphy images. For this reason, a de-noising method for ancient Chinese calligraphy works on steles based on L0 gradient minimization and guided filter is proposed in this paper. We first utilize L0 gradient minimization to smooth the noised stele image. The goal of this step is to obtain random-noise free maps. Then, we take the random-noise free map as a guided image and convolute it with its corresponding original noised stele image via a guided filter. The goal of this step is to obtain an edge preserved random-noise free image. Finally, we used the eight-connection region-based de-noising technique as a tool to remove isolated blocks. Experiments show that our method is superior to several recent published stele image de-noising techniques in preserving characters’ structures.

Our contributions in this work are summarized as follows:

• 1) We put forward a L0 gradient minimization guided image filter for random-noise removal. It consists of two operation steps: First, to smooth a noised stele image by using L0 gradient minimization technique. Then, to compute an edge-preserved noise free image by convoluting the smoothed image with the input noised stele image.

• 2) We employ the eight-connected domain technique for ant-like noise removal. The effectiveness of the proposed method is demonstrated by experimental results.

The remainder of this paper is structured as follows: Section 2 simply introduces the related works. Section 3 describes the proposed method in detail. Section 4 presents all the experimental results. Section 5 concludes the paper.

2 Related work

Interests regarding image preprocessing, including image de-noising [5] and image enhancement [6], especially on ancient Chinese calligraphy image enhancement [1,2,3,4, 7,8,9,10] have seen increasing in recent years; for instance, Zheng et.al. [1] presented a de-noising method for stele images using guided filter on the L channel. Wang et al. [7] used a threshold method to remove noise from a Chinese calligraphy image. In [3], an adaptive shock filter based on anisotropic diffusion is proposed. In [8], the Laplacian pyramid is combined with the fractional Fourier domain to extract the structures of characters with rotation angles at different scales. In [9], an anisotropic diffusion filter with the width information of different strokes was proposed for noise removal. In [10], structure characteristics of Chinese character strokes and run-length statistics were used for stele de-noising. In [11], an integration of multiple de-noising filters was proposed for removing block and line noise from stele images. In [12], KSVD dictionary learning was used for de-noising as well as for preserving the character structures.

Although there are many works on stele image processing, research on stele image de-nosing are still very limited. Solutions for ancient Chinese calligraphy image de-noising are still opened.

Recently, a piece wise filtering model termed L0 gradient minimization has been employed for image de-noising. By optimizing an energy function based on prior gradient sparsity, the L0 gradient minimization model shows a decent ability to perform image de-noising and a strong ability to preserve sharp features [13] in respect to traditional image de-noising method, such as Gaussian filter and shock filter. Similar works are also found in [14, 15].

Motivated by the aforementioned research, in this paper, we employ L0 gradient minimization as well as a guided filter for ancient stele calligraphy image de-noising.

3 Proposed method

3.1 Overview

Figure 2 shows the framework of our presented method. Firstly, a stele image is smoothed using L0 gradient minimization. The goal of this step is to obtain a random-noise free map. This random-noise free map is then convoluted with its corresponding original noised stele image to obtain an edge preserved random-noise free image. Next, a connection region based de-noising technique is used to remove isolated blocks. Details of each step are described as follows.

Fig. 2

The framework of the presented method. It consists of two steps: firstly, to compute an edge preserved random-noise free image using L0 gradient minimization; then, to remove isolated blocks by a connection region based de-noising technique

3.2 Random-noise free map computation with L0 gradient minimization

The goal of the computation in this section is to obtain a random-noise free map. For this purpose, there are several filters that can be chosen, such as the Gaussian filter, shock filter, and L0 gradient minimization [13]. Results with these filters were nearly identical. However, with respect to Gaussian filter and shock filter, the L0 gradient minimization model shows a decent ability to perform image de-noising and a strong ability to preserve sharp features by optimizing an energy function based on prior gradient sparsity [13]. For what was mentioned above, in this section, we select the L0 gradient minimization as the tools for random-noise free map computation. Firstly, we decompose the input image into two parts using L0 gradient minimization: a base image and detailed image. The L0 norm of the gradient specifies the number of non-zero gradients [13] and drives the minimization to produce an output that is comparable to input around strong edges (as shown in Fig. 2 by the color blocks), which is based on the previous gradient.

Let I(x,y) be an input image and B(x,y) be its corresponding base image, and D(x,y) be its corresponding detail image. The gradients of I(x,y) are denoted as ∇I(x, y). Formally, we define the energy function of L0 gradient minimization as:

$$ {\displaystyle \begin{array}{c}\min \\ {}B\left(x,y\right)\end{array}}\left\{\sum {\left|I\left(x,y\right)-B\left(x,y\right)\right|}^2+\upgamma \mathrm{\nabla}I\left(x,y\right)\right\} $$

(1)

where γ controls the level of detail in the final image and ∇I(x, y) represents the number of non-zero gradient and is described as:

$$ \nabla I\left(x,y\right)=\#\left\{\left(x,y\right)\Big\Vert \nabla B\left(x,y\right)|\ne 0\right\} $$

(2)

where # denotes the pixel number and ∇ indexes the operator difference.

The gradient L0 norm is usually computed at positions containing sharp edges because it denotes the numbers of non-zero gradients. In term of this view, we multiply a binary map Edge(x,y) on the original regularization term ∇I(x, y) to speed up the process.

$$ \nabla I\left(x,y\right)=\#\left\{\left(x,y\right)|\mathrm{Edge}\left(x,y\right)\bullet |\nabla B\Big(x,y\Big)|\ne 0\right\} $$

(3)

where Edge(x,y) denotes a binary map. Any technique that can obtain sharp edges can be used to get the binary map, such as applying the differential of Gaussian (DOG) filter on I(x,y) with a threshold, or applying the Laplacian of Gaussian filter on I(x,y) with a threshold. In this paper, we apply the differential of Gaussian (DOG) filter on I(x,y) with a threshold to get the binary map. The threshold value is determined based on experiments.

To easily calculate the L0 norm of gradient via Eq. (1), two dependent variables α_x(x, y) and α_y(x, y) are introduced, corresponding to$\frac{\partial B\left(x,y\right)}{\partial x}$ and$\frac{\partial B\left(x,y\right)}{\partial y}$, respectively. Equation (1) is reconstructed as:

$$ {\displaystyle \begin{array}{c}\begin{array}{c}\min \\ {}\left\{B,h,v\right\}\end{array}\sum {\left|I\left(x,y\right)-B\left(x,y\right)\right|}^2+\gamma \nabla I\left(x,y\right)\\ {}+\beta \cdot \left({\left|\frac{\partial B\left(x,y\right)}{\partial x}-{\alpha}_x\left(x,y\right)\right|}^2\right)\\ {}+\left({\left|\frac{\partial B\left(x,y\right)}{\partial y}-{\alpha}_y\left(x,y\right)\right|}^2\right)\end{array}} $$

(4)

where β is used to regulate the difference between the dependent variables and its corresponding gradients. To address this minimization, an alternating minimization strategy is used, viz. We keep one of the variable sets unchanged to obtain another variable set. In our work, we keep α_x(x, y) and α_y(x, y) unchanged and use the below cost function to first obtain B(x,y):

$$ {\displaystyle \begin{array}{c}\begin{array}{c}\min \\ {}B\left(x,y\right)\end{array}\frac{1}{\beta}\sum {\left|I\left(x,y\right)-B\left(x,y\right)\right|}^2\\ {}+{\left|\frac{\mathit{\partial B}\left(x,y\right)}{\partial x}-{\alpha}_x\left(x,y\right)\right|}^2\\ {}+{\left|\frac{\partial B\left(x,y\right)}{\partial y}-{\alpha}_y\left(x,y\right)\right|}^2\end{array}} $$

(5)

Then, we keep the evaluated B(x,y) unchanged and through the following cost function obtain α_x(x, y) and α_y(x, y)

$$ {\displaystyle \begin{array}{c}\begin{array}{c}\min \\ {}h,v\end{array}\frac{\gamma }{\beta}\nabla I\left(x,y\right)+{\left|\frac{\partial B\left(x,y\right)}{\partial x}-{\alpha}_x\left(x,y\right)\right|}^2\\ {}+{\left|\frac{\partial B\left(x,y\right)}{\partial y}-{\alpha}_y\left(x,y\right)\right|}^2\end{array}} $$

(6)

where we model the L0 norm of gradient as:

$$ \nabla I\left(x,y\right)=\mathrm{Edge}\left(x,y\right)\bullet H\left(|{\alpha}_x\left(x,y\right)|+|{\alpha}_y\left(x,y\right)|\right) $$

(7)

where H(| α_x(x, y)|  + | α_y(x, y)| ) is a binary function returning 1 when ∣α_x(x, y) ∣  +  ∣ α_y(x, y) ∣  ≠ 0; otherwise, it returns 0. By alternatively computing (5, 6), we obtain the final base image B(x,y) and its corresponding detail image D(x,y). Because all of the random-noise remains in the detail image D(x,y), we take the final base image B(x,y) as the random-noise free map. Figure 3 shows an example of random-noise removal results with the L0 gradient minimization operation. It is observed that the L0 gradient minimization smoothed image (Fig. 3b) contains little noise compared with the original image (Fig. 3a).

Fig. 3

Random-noise removal with L0 gradient minimization. a Input image. b Result of L0 gradient minimization

3.3 Recover of the Chinese character structure using a guided filter

After the L0 gradient minimization operation, we obtain a random-noise-free image, as shown in Fig. 3b. However, we find that some stroke edge details are removed as random noise by the L0 minimization operation, as indicated by the yellow box in Fig. 3. The guided filter is an edge-preserving filter [16]. Its goal is to smooth input images by calculating the content of the guidance image. Many studies shows that edges in an image after using guided filter will change differently. For step edges, it is still step edges after using guided filter, but their ranges become smaller, which means that the step edges become smoother after guided filter; For ridge edges, if the ridge edges with small size are unaffected by the other edges, their variances are close to 0, then the ridge edges will disappear and tend to the background; Valley edges will become larger than the input. From what was mentioned above, we can see that the guided filter has well preserving ability on image edges. Therefore, it can be used for image texture recover.

For what was mentioned above, to recover the over-smoothed stroke edge details, guided filter is employed in this section.

Formally, given a guidance image I and an input image I_in, the corresponding guided filtering output image${I}_{\mathrm{out}}^{\mathrm{guided}}$ is defined as:

$$ {I}_{\mathrm{out}}^{\mathrm{guided}}(x)={\alpha}_k{I}_i(x)+{b}_k,\forall i\in {\omega}_k $$

(8)

where ω_k is a window centered at the pixel x, α_k and b_k are constants in ω_k, respectively. These two parameters are determined as:

$$ {a}_k=\frac{\frac{1}{n_{\omega }}{\sum}_{i\in {\omega}_k}{I}_{\mathrm{in}}^i{p}_i-{\mu}_k{\overline{p}}_k}{\upsigma_k^2+\upvarepsilon} $$

(9)

$$ {b}_k={\overline{p}}_k-{a}_k{\mu}_k $$

(10)

where μ_k indexes the average value of input image I_in in window ω_k , and ${\upsigma}_k^2$ is the variance of input image I_in in window ω_k .n_ω is the number of pixels in window ω_k. ${\overline{p}}_k$ represents the mean of the guided image in window ω_k. ε represents the regularization parameter which is used to determine the changing of pixels’ intensity. The initial value of parameter ε in Eq. (9) is determined by referring [16], and its finial optimized value is determined by experiment.

In this work, for the use of guided filter, we employ the random-noise-free map obtained in above section as a guided image and convolute it with the input noised stele image to recover the Chinese character structure in a stele image. Figure 4 shows some of the results of recovering the Chinese character structure using the guided filter. As observed from the figure, the incomplete character strokes (as show in Fig. 4b) caused by the over-smoothing of the L0 gradient minimization operation are recovered well; these strokes are indicated by the yellow boxes in Fig. 4c.

Fig. 4

Partial results of the recovery of the Chinese character structure using the guided filter. a Original image. b Result of L0 gradient minimization. c Result of the guided filter

3.4 Ant-like blocks removal using the eight-connected region technique

From Fig. 4, we also find that although most of the noise in the input image is removed after L0 gradient minimization filtering, there remains a few isolated ant-like blocks in the smoothed images, as denoted by the red circle in Fig. 4c.

One simple relationship between pixels is connectivity. Based on the connectivity of pixels, we can easily determine a pixel is an image component or image noise. To do this, there are two connected region techniques can be used: the four-connected region technique and the eight-connected region technique [10]. Due to the four-connected region technique considered as neighbors, only the four pixels that share an edge (not a corner) with the pixel in question: (x + 1,y), (x − 1,y), (x,y + 1), and (x,y − 1); pixels on the diagonal could not be taken as an edge though they are serve as an effective insulator between two pixel set, and as a result, this creates undesirable topological anomalies, whereas the eight-connected region technique consider a pixel as connected not only pixels on the same row or column, but also the diagonal pixels. For this reason, also inspired by the work described [10], we utilize the eight-connected region technique [10] in this section to address the ant-like blocks problem. The processing to remove ant-like blocks using eight-connected region technique consists of the following three basic steps:

• 1) Compute the connected components using an eight-connected domain-based search technique;

• 2) Calculate each component’s area first. Then, sort all of connected components in terms of their area from large to small. Many studies index that if the area value of a block is smaller than the area value of 2/3 of the array of permutations, then the block could be taken as an isolated noise component and could be removed from an image. For this reason, we take the area value of 2/3 of the array of permutations as the value of threshold T;

• 3) Remove all of isolated components when the area is smaller than the value of threshold T.

Figure 5 shows partial results of the ant-like block removal via the eight-connected region technique. It is observed that almost all of the ant-like blocks are removed from the input stele image, and the characters in the image are highlighted and the structures of the strokes have been preserved perfectly (as shown in Fig. 5d) compared with that of the input stele image.

Fig. 5

Partial results of art-like blocks using the eight-connected region technique. a Original image. b Result of L0 gradient minimization. c Result of the guided filter. d Final result

4 Experimental results and discussions

4.1 Experimental setting

To the best of our knowledge, there is no standard ancient Chinese stele calligraphy image set available for benchmarking currently. Therefore, we collected 100 stele calligraphy images from the released image set by [1]. We also generated 50 synthetically noised stele images using Adobe Photoshop CS6 to quantitatively evaluate the proposed method.

For comparison, several state of the art image de-noising methods, including Zheng’s method [1], Zhang’s method [10], L0 gradient minimization smoothing [13], and block-matching 3D filtering (BM3D) [17], as well as methods in our two prior works [11, 12] are employed in our experiments. The results of these methods are produced in MatLab with authors suggested parameter settings. All experiments are performed on a PC with 4G RAM and a 2.60 GHz Intel Dual Core processor using MatLab R2015.

4.2 Qualitative results

Figure 6 shows partial processing results with our method; the top row shows the original images, and the bottom row shows our processing results. From Fig. 6, it is observed that our method can remove most of the random-noise and ant-like noise without spoiling the characters.

Fig. 6

De-noising results with our method. Top row: original images. Bottom row: output of our method

In Fig. 7, we compared our proposed method with Zheng’s method [1]. It can be observed that Zheng’s method failed to remove larger block noise from the images (Fig. 7b). However, the proposed method worked well for removing all types of noise (as shown in Fig. 7c).

Fig. 7

Comparison of the proposed method with Zheng’s method [1]. a Original images. b results of Zheng’s method [1]. c Results of our method

In Fig. 8, we compared our method with Zhang’s method [10]. It can be seen that both the two methods worked well in removing image noise, whereas our method obtained more smooth edges (as shown in Fig. 8c), compared with the results processed by Zhang’s method [10] where a small amount of “flash” remained in the edges (as shown in Fig. 8b).

Fig. 8

Comparison of the proposed method with Zhang’s method [10]. a Original images. b Results of Zhang’s method [10]. c Results of the proposed method

Figure 9 compares the proposed method with the L0 gradient minimization algorithm [13] and the block-matching 3D filtering (BM3D) algorithm [17]. It is observed that our proposed method retains more details than the other two methods and does not spoil the character structures.

Fig. 9

Comparison of our method with the L0 gradient minimization algorithm [13] and the block-matching 3D filtering (BM3D) algorithm [17]. a Original images. b Results with the L0 gradient minimization algorithm [13]. c The block-matching 3D filtering (BM3D) algorithm [17]. d Results with our method

In Fig. 10, we compare the proposed method in this paper with our two prior works, including an integrated de-noising method [11] and the KSVD de-noising method [12]. It is observed that the three methods were able to take out of most image noise from the input noisy images, whereas it should be noticed that the proposed method in this paper obtains the best results with clearer images.

Fig. 10

Comparison of the proposed method with our two prior works: a Original image. b KSVD de-noising method [12]. c Integrated de-noising method [11]. d Proposed method

4.3 Quantitative evaluation

In this section, we quantitative evaluate the performance of our method on synthetic and real noised stele images, respectively.

For evaluation on the synthetic noised stele calligraphy images, peak signal-to-noise ratio (PSNR) and Structure Similarity Index (SSIM) [18] are employed as evaluation measures because of available ground truth, where the larger the values are, the better the results are. Referred to [18], the definition formulation of PSNR and SSIM are described as Eq. (11, 12), respectively.

$$ \mathrm{PSNR}\left(I,K\right)=10\cdot {\log}_{10}\frac{255^2}{\mathrm{MSE}\left(I,K\right)} $$

(11)

where I is a noise-free image, K is its noisy approximation, and MSE is the mean squared error (MSE) of I and K.

$$ \mathrm{SSIM}\left(I,K\right)=\frac{\left(2{\mu}_I{\mu}_K+{c}_1\right)\left(2{\sigma}_{I,K}+{c}_2\right)}{\left({\mu}_I^2+{\mu}_K^2+{c}_1\right)\left({\sigma}_I^2+{\sigma}_K^2+{c}_2\right)} $$

(12)

where μ_I is the average of I, μ_K is the average of K, ${\sigma}_I^2$ is the variance of I, ${\sigma}_K^2$is the variance of K, σ_{I, K} is the covariance of I and K, c₁ and c₂ are two variables to stabilize the division with weak denominator.

Table 1 shows the evaluation result. From Table 1, as we can observe that the values of SNR and SSIM for our method in this work are 32.834 and 0.9952, that for Zheng’s method [1] are 26.801 and 0.9841, that for Zhang’s method [10] are 29.010 and 0.9858, that for L0 [13] are 27.062 and 0.9742, that for BM3D [17] are 27.960 and 0.9801, that for our prior work [11] are 32.690 and 0.9882, and that for our prior work [12] are 32.783 and 0.9916. These results clearly indexes that our proposed method in this work achieves the best de-noising performance of all evaluated method in this experiment in terms of both of PSNR and SSIM.

Table 1 Qualitative evaluation of different methods on synthetic noisy stele images

For evaluation on real noisy stele images, we employ the overall quality index [19] as evaluation measures because of unavailable ground truth. A larger overall quality index value indexes a better enhanced visibility. Table 2 shows the evaluation results in this experiment. From Table 2, it is observed that the proposed method obtained an average overall quality index value of 0.7056, which is much higher than that of other methods, including 0.6795 for Zheng’s method [1], 0.6865 for Zhang’s method [10], 0.6538 for L0 gradient minimization [13], 0.6895 for our prior work [11], and 0.6905 for our prior work [12]. This result indicates that our proposed method in this work remarkably improves the visibility of images in respect to other methods.

Table 2 Average overall quality index (AOQI) of different methods

5 Conclusions

We put forward a de-noising method based on L0 gradient minimization and guided filtering method for ancient Chinese stele calligraphy image de-noising in this paper. Experiments by comparing our presented method with several state of the art de-noising methods show the superior of our method for stele image de-noising.

In our next work, we will focus on how to restore spoiled character structures for the reason of preserving the beauty of art work.