Anisotropic mesh adaptation for image representation

Abstract

Triangular meshes have gained much interest in image representation and have been widely used in image processing. This paper introduces a framework of anisotropic mesh adaptation (AMA) methods to image representation and proposes a GPRAMA method that is based on AMA and greedy-point removal (GPR) scheme. Different than many other methods that triangulate sample points to form the mesh, the AMA methods start directly with a triangular mesh and then adapt the mesh based on a user-defined metric tensor to represent the image. The AMA methods have clear mathematical framework and provide flexibility for both image representation and image reconstruction. A mesh patching technique is developed for the implementation of the GPRAMA method, which leads to an improved version of the popular GPRFS-ED method. The GPRAMA method can achieve better quality than the GPRFS-ED method but with lower computational cost.

1 Introduction

Triangular meshes have recently received considerable interest in adaptive sampling for image representation [1–17]. One common approach is to find proper sample points then connect the points to form a mesh. For example, Ramponi and Carrato [5] have defined a sample skewness parameter and used a multi-resolution approach to obtain a grid with an almost uniform sample density along the edges and no sample in areas with constant or linearly changing gray level. Yang et al. [8] argue that small (in area) elements are needed in image region where the second directional directive is large and have introduced the error diffusion (ED) scheme. They first construct a feature map based on the largest entry (of absolute value) in the Hessian matrix of the image function, then use Floyd-Steinberg dithering scheme to generate sample points, and finally use Delaunay triangulation to connect the nodes into a mesh. Demaret et al. [11, 12] have introduced the greedy-point removal (GPR) scheme that first constructs a triangular mesh using all the image points and then removes the sample points that yield smallest reconstruction error repeatedly. Adams [17] has proposed the GPRFS method based on the GPR scheme by replacing the initial triangular mesh of all image points with a subset of the points and developed the GPRFS-ED method that selects the initial points using the ED scheme.

Another approach is to use a mesh directly to represent the image. For example, Terzopoulos and Vasilescu [2] have introduced an adaptive mesh approach where the mesh is considered as a dynamic node/spring system. They sample an image at a reduced rate and then reconstruct it by concentrating the nodes of the mesh at regions where the image values change rapidly (high-gradient region). They develop adaptive meshes with a feedback procedure that automatically adjusts spring parameters according to the observations made at the nodes to which they are attached, and use a Gaussian convolution of the Hessian for the adaptive image reconstruction. Isotropic triangles are used in their adaptive meshes. Courchesne et al. [13] use the Hessian matrix based on the gray level of MRI images as a metric tensor to adapt the triangular mesh for 3D reconstruction of human trunk. The Hessian matrix is reconstructed by linear or quadratic fitting. They then adapt the mesh based on the provided metric tensor and four constraint factors—minimum and maximum Euclidean edge lengths, maximum stretching of the metric, and target length of an edge in the metric. Bougleux et al. [15] have developed a progressive geodesic meshing algorithm that defines a geodesic distance using regularized Hessian as the metric tensor and exploits the anisotropy of images through a farthest point sampling strategy that forces the anisotropic Delaunay triangles to follow the geometry of the image. They have demonstrated the advantages of anisotropic triangular approximation over isotropic triangular approximation. Sarkis and Diepold [16] have used binary space partitions in combination with clustering scheme to approximate an image with a mesh. They first cluster the image into a few initial triangles (or rectangles) and then subdivide each triangle (or rectangle) into two or more smaller triangles recursively according to a predefined threshold.

Most of the adaptive sampling methods are “content-based” that use some information from the image such as edges, textures, or Hessian. Different sampling or meshing strategies have been developed. The GPR method provides high-quality meshes but requires significant computational cost. On the other hand, the ED method reduces computational cost but provides lower quality meshes. The GPRFS-ED method tries to find a balance between mesh quality and computational cost by combining the advantages of GPR and ED methods. It is worth mentioning that most of the sampling methods take the approach of finding the desired sample points first and then connect the points into a mesh. Only a few methods such as [2, 13, 15, 16] follow the approach that starts from an initial mesh and then adapt the mesh to represent the image.

On the other hand, anisotropic mesh adaptation (AMA) has been successfully applied to improve computational efficiency and accuracy when solving partial differential equations [18–23]. In this paper, we introduce a framework of AMA methods for image representation. AMA methods take the $\mathbb {M}$-uniform mesh approach for mesh adaptation and use finite element interpolation for image reconstruction. The methods start with an initial triangular mesh, then adapt the mesh according to a user-defined metric tensor $\mathbb {M}$, and finally reconstruct the image from the mesh. The framework has the flexibility for both mesh adaptation and image reconstruction. Various metric tensors can be chosen for mesh adaptation, and different orders of finite element interpolation can be applied for reconstruction. In this paper, we only consider linear finite element interpolation for triangular elements in the reconstruction step.

For reader’s convenience, the representation methods under consideration are summarized in the following list.

• ED: error diffusion method developed by Yang et al. [8].

• AMA methods such as $\mathbb {M}_{\text {aniso}, k}$: anisotropic mesh adaptation method using metric tensor $\mathbb {M}_{\text {aniso}}$ with the initial mesh being adapted k times to generate the desired mesh, proposed in Section 3.

• GPR: greedy-point removal scheme proposed by Demaret et al. [11, 12].

• GPRFS-ED: modified GPR scheme starting from a subset of points chosen by ED method, proposed by Adams [17].

• GPRED-CDT(γ): GPR starting from γ times of the desired number of sample points chosen by ED and utilizing constrained Delaunay triangulation for mesh patching, essentially the same as GPRFS-ED, proposed in Section 4.

• GPRED-EC(γ): same as GPRED-CDT(γ) except using ear clipping for mesh patching, proposed in Section 4.

• GPRAMA(γ): GPR starting from an AMA representation of γ times of the desired sample density and utilizing ear clipping for mesh patching, proposed in Section 4.

The remainder of this paper is organized as follows. Firstly, a brief introduction of the AMA methods is given in Section 2 where the details of the methods can be found in [24–26]. Then in Section 3, the AMA representation framework is introduced and some results obtained from different methods are presented. In Section 4, a GPRAMA representation method based on AMA and GPR is proposed and some results and computational complexity are discussed. Finally, some conclusions and comments are given in Section 5. For reader’s convenience, a brief summary of finite element interpolation for triangular elements is provided in the Appendix.

2 Anisotropic mesh adaptation (AMA) methods

Different adaptive sampling methods and mesh strategies have been applied in image representation by other researchers as summarized in Section 1. In this section, we introduce the “anisotropic mesh adaptation” (AMA) methods. AMA methods take the $\mathbb {M}$-uniform mesh approach, with which an adaptive mesh is generated as a uniform mesh in the metric specified by a tensor $\mathbb {M}$. The metric tensor $\mathbb {M}$ is required to be strictly positive definite and it determines the size, shape, and orientation of the triangular elements [25]. Once a metric tensor is specified, the free C++ code BAMG (bidimensional anisotropic mesh generator) developed by Hecht [27] is used to generate the corresponding triangular mesh. BAMG first generates an initial mesh based on the geometry of the domain provided in a file that defines the nodes and edges and the desired mesh size using constrained Delaunay triangulation. Then users have the choice to either provide a metric tensor on the initial background mesh or use the internal metric tensor computed by BAMG. Once a metric tensor $\mathbb {M}$ is provided, BAMG employs five local minimization tools including edge suppression, vertex suppression, vertex addition, edge swapping, and vertex relocation to generate the desired anisotropic mesh according to $\mathbb {M}$. One of our objectives in this paper is to build the framework for AMA in image representation that can take different metric tensors for different needs in image processing.

Firstly, we introduce some notations and the conditions for $\mathbb {M}$-uniform meshes. Let Ω be the spatial domain, K be any triangular element in a simplicial mesh $\mathcal {T}_{h}$, and $\hat {K}$ to be the reference element that is equilateral and unitary in area. Let F _K be the affine mapping from $\hat {K}$ to K. An $\mathbb {M}$-uniform 2-D triangular mesh $\mathcal {T}_{h}$ for a given metric tensor $\mathbb {M}=\mathbb {M}(\textbf {x})$ satisfies the following condition

$$ (F_{K}')^{T} \mathbb{M}_{K} F_{K}' = \frac{\sigma_{h}}{N} I, \quad \forall K \in \mathcal{T}_{h} $$

(1)

that is equivalent to the following two conditions [25]

$$\begin{array}{@{}rcl@{}} |K| \sqrt{\text{det}(\mathbb{M}_{K})} & = & \frac{\sigma_{h}}{N}, \quad \forall K \in \mathcal{T}_{h}, \end{array} $$

(2)

$$\begin{array}{@{}rcl@{}} \frac{1}{2} \text{tr} \left (\!(F_{K}')^{T} \mathbb{M}_{K} F_{K}' \!\right) & = &\text{det}\! \left (\!(F_{K}')^{T} \mathbb{M}_{K} F_{K}' \!\right)^{\frac{1}{2}}\!, \quad \!\!\forall K \!\in\! \mathcal{T}_{h}, \end{array} $$

(3)

where I is the identity matrix of size 2×2, |K| is the area of the element K, N is the number of mesh elements, F K′ is the Jacobian matrix of F _K ,

$$ \mathbb{M}_{K} = \frac{1}{|K|} \int_{K} \mathbb{M}(\textbf{x}) d \textbf{x}, \text{and}~ \sigma_{h} = \sum_{K \in \mathcal{T}_{h}} |K| \sqrt{\text{det}(\mathbb{M}_{K})}. $$

(4)

Condition (2) is called the equidistribution condition and determines the size of element K, while condition (3) is called the alignment condition and characterizes the shape and orientation of K.

In the framework of AMA methods, the goal is to develop and use proper metric tensors based on the needs of the problems. Different metric tensors will have different properties and features. It is worth mentioning that Hessian matrix H is not an optimal metric tensor [26] and may not be positive definite. In our framework, we replace the Hessian with its absolute form defined as follows

$$ |H|= Q \left[\begin{array}{cc} |\lambda_{1}| & 0 \\ 0 & |\lambda_{2}| \end{array} \right] Q^{-1}, \text{with}~H= Q \left[\begin{array}{cc} \lambda_{1} & 0 \\ 0 & \lambda_{2} \end{array} \right] Q^{-1}, $$

(5)

where λ ₁ and λ ₂ are the eigenvalues of H and Q is the matrix of the corresponding eigenvectors. The metric tensor |H| is denoted as $\mathbb {M}_{H}$ in this paper, and [13] can be considered as a specific example in our AMA framework. Some other metric tensors are described below.

For isotropic mesh adaptation, a metric tensor $\mathbb {M}_{iso}$ is defined for any triangular element K as follows [24]

$$ \mathbb{M}_{\text{iso},K}=\left(1 + \frac{1}{\alpha_{h}} \| H_{K} \|_{F} \right) \, I, $$

(6)

where H _K denotes the value of H at the center of element K,∥·∥ _F is the Frobenius matrix norm, and α _h is a regularization factor that is defined by

$$ \alpha_{h} = \frac{1}{| \Omega |} \left(\sum_{K\in \mathcal{T}_{h}} |K| \cdot \| H_{K} \|_{F} \right). $$

(7)

$\mathbb {M}_{\text {iso}}$ provides isotropic mesh adaptation where all triangles are of the same shape but may have different sizes, and more triangles will be concentrated in the high-gradient region.

For anisotropic mesh adaptation, a metric tensor $\mathbb {M}_{\text {aniso}}$ is developed in [24] that is based on minimization of a bound on the H ¹ semi-norm of linear interpolation error and is defined for any triangular element K as follows

$$ \mathbb{M}_{\text{aniso},K}=\rho_{K} \det \left(I+\frac{1}{\alpha_{h}}|H_{K}|\right)^{-\frac{1}{2}} \left[ I+\frac{1}{\alpha_{h}}|H_{K}|\right], $$

(8)

and

$$ \rho_{K} = \left\| I + \frac{1}{\alpha_{h}} | H_{K} | \right\|_{F}^{\frac{1}{2} }\,\det \left(I+\frac{1}{\alpha_{h}}|H_{K}|\right)^{\frac{1}{4}}, $$

(9)

where α _h is the regularization parameter and is defined implicitly through

$$ \sum_{K\in \mathcal{T}_{h}} \rho_{K} |K| = 2 | \Omega |. $$

(10)

With this choice of α _h , roughly 50 % of the triangular elements will be concentrated in large gradient regions [24]. The adaptation is anisotropic because the triangles in the mesh may have different size, shape, and orientation.

For image processing with anisotropic diffusion filters [28], a metric tensor $\mathbb {M}_{\text {DMP}}$ is developed in [20] that takes the inverse of the diffusion tensor. The elements of the mesh based on $\mathbb {M}_{\text {DMP}}$ will be aligned along the principle diffusion direction, and the corresponding numerical solution will satisfy the maximum principle under some conditions of time step [22]. Another metric tensor $\mathbb {M}_{\text {DMP+adap}}$ is also developed in [20] that combines the properties of both $\mathbb {M}_{\text {aniso}}$ and $\mathbb {M}_{\text {DMP}}$, that is, the mesh not only provides numerical solution that satisfies maximum principle but also performs adaptation based on the interpolation error.

In this paper, we only focus on the metric tensors $\mathbb {M}_{H}, \mathbb {M}_{\text {iso}}$, and $\mathbb {M}_{\text {aniso}}$. In the computations for those metric tensors, the Hessian matrix H at a point is reconstructed by the least-squares fitting from function values at neighboring vertices. For convenience, we use the metric tensor to denote the mesh as well as the corresponding representation. For example, $\mathbb {M}_{\text {aniso}}$ denotes the mesh and representation according to the metric tensor $\mathbb {M}_{\text {aniso}}$.

As demonstrated in [15, 26], anisotropic meshes have advantages over isotropic meshes in terms of computational efficiency and accuracy. Our results in Section 3 also confirm that anisotropic meshes provide better representation quality than isotropic meshes. Therefore, we will only use anisotropic mesh adaptation methods in the AMA image representation framework.

3 AMA image representation framework

In this section, we introduce the AMA framework for image representation. We consider an image as a function f that is defined on a set Λ of points on domain Ω=[0,1]×[0,1]. Let S denote the set of desired sample points and SD denote the sample density that is defined as

$$ SD=|S|/|\Lambda|, $$

(11)

where |·| is the cardinality of the set. The quality of the mesh (or representation) is measured by the peak-signal-to-noise-ratio (PSNR) that is calculated in decibels (dB) as follows [17]

$${} \text{PSNR} = 20 \log_{10} \left(\! \frac{2^{p}-1}{d}\!\right), d = \left(\! \frac{1}{|\Lambda|} \sum_{i \in \Lambda} | \hat{f}(i)-f(i) |^{2} \right)^{\frac{1}{2}}\!, $$

(12)

where $\hat {f}$ is the reconstructed image from the triangular mesh, and p is the sample precision in bits/sample. Larger value of PSNR indicates better mesh quality (or representation).

3.1 Framework

In this paper, we apply the AMA methods for image representation, in which a triangular mesh with fewer points is used to represent the original image f. Given the original image f, we use the free C++ code BAMG [27] to generate an initial triangular mesh with number of vertices N _v that is much smaller than |Λ|. Then the values on the vertices are interpolated from f, and the metric tensor $\mathbb {M}$ is computed for each triangle in the initial mesh. With the computed metric tensor, BAMG generates the desired anisotropic mesh using the procedures described in Section 2. Finally, the image is reconstructed from the final mesh using finite element interpolation.

It is difficult to generate a mesh that satisfies conditions (2) and (3) exactly. Moreover, the initial mesh, in general, may not contain the important information from the original image. Therefore, the mesh can be adapted multiple times in order to obtain a final mesh that is close to be an $\mathbb {M}$-uniform mesh, or the so-called quasi-$\mathbb {M}$-uniform mesh. The iteration can be terminated if further adaptation does not significantly improve image quality (measured by PSNR) and can be image dependent in order to obtain the best representation of a particular image. However, numerical results show that 2 to 5 iterations are sufficient to provide a quasi-$\mathbb {M}$-uniform mesh with good quality, and further adaptation does not improve the quality significantly.

More specifically, the AMA representation framework consists of the following four steps.

Step 1: Generate an initial mesh based on the desired sample density.

Step 2: Assign function values to mesh vertices (and interpolation nodes) from original image using linear finite element interpolation and compute the user-defined metric tensor $\mathbb {M}$ on the mesh.

Step 3: Adapt the mesh to be a quasi-$\mathbb {M}$-uniform mesh that almost fits the provided metric tensor $\mathbb {M}$.

Step 4: Reconstruct the image using the final quasi-$\mathbb {M}$-uniform mesh with finite element interpolation for triangles.

During the reconstruction step (step 4), for a particular image pixel, we first locate the triangle that the pixel lies on or in. Then, we compute the coordinates of the pixel in the reference element (see Fig. 1) and the corresponding basis functions at the interpolation nodes. Finally, we interpolate the function value from the interpolation nodes using the basis functions as the weights. For linear interpolation, only the three vertices are needed, while for quadratic interpolation, the midpoints are also needed. For reader’s convenience, a brief summary of finite element interpolation for triangular elements is provided in the Appendix.

Fig. 1

Sketch of triangular element K and its reference element $\hat K$ for finite element interpolation, where $\hat K$ is an isosceles right triangle with vertices $\hat {\textbf {a}}_{1} (0,0)$, $\hat {\textbf {a}}_{2} (1,0)$, and $\hat {\textbf {a}}_{3} (0,1)$

The above procedures are shown in Fig. 2 where steps 2 and 3 can be repeated multiple times in order to obtain better results. For convenience, we denote the number of iterations for steps 2 and 3 by k and the corresponding mesh as $\mathbb {M}_{k}$. For example, for metric tensor $\mathbb {M}_{H}$, the representation is denoted as $\mathbb {M}_{H,k}$ if there are k iterations of steps 2 and 3. For metric tensor $\mathbb {M}_{\text {aniso}}$, the corresponding mesh is denoted as $\mathbb {M}_{\text {aniso},k}$. When k=1, the mesh is only adapted from initial mesh once and no further adaptation is performed. As mentioned before, we take k∈[2,5] in our computations.

Fig. 2

Procedures for AMA representation method based on metric tensor $\mathbb {M}$

Note that we can start with a random initial mesh that has more number of vertices than desired (N _v >|S|) in Step 1, then iterate steps 2 and 3 to obtain a mesh with desired sample density. By this way, more information from the original image could be reserved by the mesh. Another approach is to obtain an initial mesh with N _v >|S| for steps 1 to 3, then reduce the number of mesh vertices to the desired number |S| using the GPR algorithm before Step 4. The particular representation using GPR before step 4 is denoted as GPRAMA and will be discussed in Section 4.

Note that in step 4, we can choose different orders of finite element interpolation methods. However, in this paper, we only consider linear finite element interpolation for triangular elements and the effects of higher order interpolation on representation quality is currently under investigation. In fact, quadratical interpolation provides higher representation quality; however, the sample density is also higher since it uses the midpoints on the edges of the triangles. There is no need to sample the midpoints because their coordinates can be computed; however, the function values at the midpoints need to be assigned in Step 2. A fair comparison is needed between higher order interpolation and linear interpolation with the same sample density.

3.2 Results

For evaluation purpose, we take the two widely used images, “Lena” and “peppers,” available from USC-SIPI Image Database [29]. Figure 3 shows the initial images of Lena and peppers with pixel resolution 512×512, while the RGB components of each pixel are converted to greyscale luminance using the weighted sum 0.2989·R+0.5870·G+0.1140·B. Three more images with different resolutions and features are also tested for comparison purpose, including “roof,” “lighthouse,” and “saturn.” Image roof has resolution 1024×1024 and is obtained from USC-SIPI Image Database [29]. “lighthouse” has resolution 480×640 and “saturn” has resolution 1500×1200; both are taken from MATLAB R2016a image data folder. All images are converted to greyscale images as done for images Lena and peppers.

Fig. 3

Images from USC-SIPI Image Database [29] with pixel resolution 512×512: a Lena; b peppers. The RGB components of each pixel are converted to luminance using the weighted sum 0.2989·R+0.5870·G+0.1140·B

Figure 4 shows the representation of the image Lena at S D=3 % using isotropic mesh according to $\mathbb {M}_{\text {iso,3}}$, and the quality of representation is PSNR = 28.26. Figure 5 shows the $\mathbb {M}_{\text {aniso,1}}$ and $\mathbb {M}_{\text {aniso,2}}$ meshes and the corresponding sample points of the image Lena at S D=3 %. The representation quality for $\mathbb {M}_{\text {aniso,1}}$ is PSNR = 29.81, and is PSNR = 30.81 for $\mathbb {M}_{\text {aniso,2}}$. After three iterations, the quality increases to PSNR = 31.00 for $\mathbb {M}_{\text {aniso,3}}$ as shown in Fig. 6. Further adaptation does not improve the representation quality for this case. It is clear that image representation based on $\mathbb {M}_{\text {aniso}}$ is better than the one based on $\mathbb {M}_{\text {iso}}$.

Fig. 4

Representation of image Lena at sample density of 3 % using $\mathbb {M}_{\text {iso,3}}$: a triangular mesh $\mathbb {M}_{\text {iso,3}}$; b reconstructed image, PSNR=28.47

Fig. 5

Meshes and sample points of image Lena at sample density of 3 % using $\mathbb {M}_{\text {aniso}}$: a $\mathbb {M}_{aniso,1}$ mesh; b sample points from the mesh in a; c $\mathbb {M}_{\text {aniso,2}}$ mesh; d sample points from the mesh in c

Fig. 6

Representations of image Lena at sample density of 3 % using different methods: a triangular mesh $\mathbb {M}_{\text {aniso,3}}$; b reconstructed image from a, PSNR=31.00; c mesh obtained using ED scheme; d reconstructed image from c, PSNR=28.41

Figure 6 also shows the representation and reconstruction of the image Lena at S D=3 % using ED scheme denoted as ED. For the ED scheme, we have applied the strategies recommended in [17], including B(3) smoothing for image data, zero extension for boundary points, and serpentine scan order for the error diffusion. The representation quality is PSNR=28.41 for ED.

Figure 7 shows similar results for the image peppers at S D=3 %. The quality for $\mathbb {M}_{\text {aniso,3}}$ is PSNR=30.89 and is PSNR=28.05 for ED. As can be seen from Figs. 6 and 7, the $\mathbb {M}_{\text {aniso,3}}$ meshes preserve key features of the original image by concentrating more triangular elements around the edges and texture regions. The quality of $\mathbb {M}_{\text {aniso,3}}$ representation is much better than that of the ED representation.

Fig. 7

Representations of image peppers at sample density of 3 % using different methods: a triangular mesh $\mathbb {M}_{\text {aniso,3}}$; b reconstructed image from a, PSNR=30.89; c mesh obtained using ED scheme; d reconstructed image from c, PSNR=28.05

The representation qualities for both image Lena and peppers using different AMA methods are shown in Table 1. The results obtained using ED scheme are also presented for comparison purpose. As can be seen, the quality of $\mathbb {M}_{\text {iso,3}}$ representation is comparable to ED but not as good as the anisotropic ones.

Table 1 Comparison of mesh qualities obtained with various methods

The results confirm that more iterations of steps 2 and 3, that is, increasing the values of k, do improve the mesh quality. For example, for image Lena at S D=3 %, PSNR increases from 29.67 for $\mathbb {M}_{H,1}$ to 30.45 for $\mathbb {M}_{H,3}$ and from 29.81 for $\mathbb {M}_{\text {aniso,1}}$ to 31.00 for $\mathbb {M}_{\text {aniso,3}}$. Similar results are observed for image peppers. The reason is that after each mesh adaptation, better information are preserved from the original image and the mesh is closer to the $\mathbb {M}$-uniform mesh. However, for the two images we are investigating, k=3 already provides a mesh with good quality and further adaptation does not make significant improvement. The results of PSNR values using $\mathbb {M}_{H,k}$ and $\mathbb {M}_{\text {aniso,k}}$ at different k values are presented in Table 2 for different images at sample density of 3 %. The optimal value of k depends on the given image. For example, k=2 is the best for image saturn, k=3 works the best for images Lena, and peppers. For image lighthouse, $\mathbb {M}_{H,k}$ and $\mathbb {M}_{\text {aniso,k}}$ have different optimal k values and k=3 is a good balance.

Table 2 PSNR (dB) of $\mathbb {M}_{H,k}$ and $\mathbb {M}_{\text {aniso,k}}$ representations at sample density of 3 %

Comparing $\mathbb {M}_{H,1}$ and $\mathbb {M}_{\text {aniso,1}}$ in Table 1 and 2, we see that the absolute Hessian |H| is not an optimal metric tensor, and the performance of $\mathbb {M}_{\text {aniso,1}}$ is better than $\mathbb {M}_{H,1}$ for all cases, although the difference is not significant. By adapting both $\mathbb {M}_{H}$ and $\mathbb {M}_{\text {aniso}}$ meshes three times, $\mathbb {M}_{\text {aniso,3}}$ performs better than $\mathbb {M}_{H,3}$ for image Lena while the opposite occurs for image peppers. The results of $\mathbb {M}_{H,3}$ and $\mathbb {M}_{\text {aniso,3}}$ for the other three images are presented in Table 3. The qualities of the representation depend on the specific image but overall performance is comparable for $\mathbb {M}_{H}$ and $\mathbb {M}_{\text {aniso}}$. In this paper, we choose $\mathbb {M}_{\text {aniso,3}}$ as the representative from the AMA framework and propose a new image representation method based on AMA and GPR in the next section.

Table 3 Comparison of mesh qualities obtained with $\mathbb {M}_{H,3}$ and $\mathbb {M}_{\text {aniso,3}}$

4 GPRAMA representation method

In this section, we apply the greedy-point removal scheme to AMA representation and propose a new method denoted as GPRAMA. Adams has proposed the GPRFS method in [17] that is based on the GPR scheme while replacing the initial triangular mesh of all image points with a subset S ₀⊆Λ. The GPRFS method starts with |S ₀|=γ|S| for γ∈[4,5.5] and then uses GPR scheme to reduce the number of points from |S ₀| to the desired number |S|. Adams employs ED method to choose S ₀ and denotes the method as GPRFS-ED in his paper.

As discussed in the previous section, our AMA representation methods provide better quality than ED scheme. Therefore, it is reasonable to consider the vertices of an AMA mesh as the initial subset S ₀ for the GPR scheme. In this sense, the GPRAMA method is a specific example of the GPRFS method. However, GPRFS method relies on Delaunay triangulation of the sample points that does not have the anisotropic feature as in AMA meshes. In fact, the mesh quality obtained via Delaunay triangulation may not be optimal for a given set of sample points, measured by PSNR value of the reconstructed image. Figure 8 shows one example, where the mesh is obtained by Delaunay triangulation of the provided sample points and the mesh quality is PSNR=28.57; however, for the same set of sample points, the $\mathbb {M}_{aniso,3}$ mesh (see Fig. 6 a) has better quality with PSNR=31.00.

Fig. 8

Representation of image Lena at sample density of 3 %: a sample points; b Delaunay mesh, PSNR=28.57

4.1 Mesh patching technique

In order to preserve the anisotropic feature of the AMA meshes when applying GPR scheme, we have developed a mesh patching technique that attaches a local mesh to the AMA mesh. This patching technique also works for any other triangular meshes. Let i be the index of a general vertex in the mesh, we denote the region covered by triangles sharing vertex i including the boundary edges by $\overline {\omega _{i}}$ and call it the patch of i. The polygon surrounding i formed by the boundary edges, is denoted as ∂ ω _i , and the inner region of the patch, that is, excluding the boundary edges, is denoted as ω _i . Before removing the vertex i, the patch $\overline {\omega _{i}}$ is partitioned by the triangular elements from the initial mesh, and linear finite element interpolation are used on those triangles for image reconstruction. If vertex i is chosen to be removed, $\overline {\omega _{i}}$ is triangulated again without vertex i. The new triangulation of $\overline {\omega _{i}}$ is then added to the global mesh structure. Figure 9 provides an illustration of the mesh patching technique with two different triangulation methods of the patch—one is the constrained Delaunay triangulation (CDT) [30] and the other is the ear clipping (EC) method [31]. Both CDT and EC work well for general polygons including concave ones. The specific procedures for the mesh patching of $\overline {\omega _{i}}$ are as follows.

Fig. 9

Illustration of mesh patching technique: a original mesh with the highlighted patch of the point to be removed; b mesh with polygon of the empty patch; c mesh patching using constrained Delaunay triangulation (CDT) for b; d mesh patching using ear clipping (EC) method for b

Step 1: Save the global indices of the vertices of the polygon ∂ ω _i .

Step 2: Triangulate $\overline {\omega _{i}}$ without vertex i (using CDT or EC) and save the mesh connectivity that lists the local indices of the vertices of each triangle.

Step 3: Map the local indices in Step 2 to the global indices from Step 1, and add the updated connectivity information of $\overline {\omega _{i}}$ to the global mesh.

4.2 GPRAMA method

With the mesh patching technique described above and the AMA representation framework introduced in Section 3, the GPRAMA method consists of the following four steps.

Step 1: Generate an AMA representation based on a metric tensor $\mathbb {M}$ with number of vertices N _v =γ|S| for γ≥1. Let V be the set of all vertices in the mesh and let V _p =V.

Step 2: For any vertex v _i ∈V _p , compute the significance measure δ e _i defined as the difference between the local mean square error of $\overline {\omega _{i}}$ after and before removing v _i , as shown below.

$$ \delta e_{i} = \sum_{j \in \Lambda \cap \overline{\omega_{i}}} |\, \hat{f}_{a} (j)-f(j) |^{2} - \sum_{j \in \Lambda \cap \overline{\omega_{i}}} |\, \hat{f}_{b} (j)-f(j) |^{2}, $$

(13)

where $\hat {f}_{a}$ is the reconstructed value using the new triangulation of the patch without vertex i and $\hat {f}_{b}$ is the one with vertex i.

Step 3: For the vertex v _i ∈V having minimal δ e _i , reset V _p as the set of vertices of the polygon ∂ ω _i , remove v _i and apply the mesh patching technique for $\overline {\omega _{i}}$. Reset V=V∖{v _i } and N _v =N _v −1.

Step 4: If N _v ≤|S|, output the mesh and stop; otherwise, go to step 2 with the updated V _p and V.

In our computations, we choose metric tensor $\mathbb {M}_{\text {aniso,3}}$ in the above procedures for GPRAMA due to its good representation quality as described in Section 3. In each iteration of Step 2, except the first, we only need to compute the significance measures for the neighboring vertices of v _i after it is removed. Other vertices in the mesh outside of $\overline {\omega _{i}}$ are not affected. For efficient implementation, we do not need to delete the information at vertex v _i such as coordinates, function value and neighboring triangles from the mesh data then triangulate the patch using the updated mesh data. We just need to replace the old triangulation of the patch containing v _i with the new one without v _i by updating the corresponding entries in the mesh data directly. A sorted index array for the significance measures can be used in step 3 for efficient selection of the vertex to be removed.

4.3 Results

According to the different triangulation methods for the mesh patching technique in step 3, the final mesh and the corresponding representation are denoted as GPRAMA(γ)-CDT if constrained Delaunay triangulation is used and GPRAMA(γ)-EC if ear clipping method is used, where γ specifies the number of initial points |S ₀|=γ|S|. Figure 10 shows the meshes obtained using GPRED and GPRAMA methods for image Lena at sample density 3 % with γ=4, and Fig. 11 presents two of the reconstructed images.

Fig. 10

Meshes of different GPR-related representations for image Lena at sample density of 3 %: a GPRED(4)-CDT, PSNR=33.49; b GPRED(4)-EC, PSNR=33.85; c GPRAMA(4)-CDT, PSNR=33.18; d GPRAMA(4)-EC, PSNR=34.51

Fig. 11

Reconstructed images using different GPR-related representations for image Lena at sample density of 3 %: a GPRED(4)-EC, PSNR=33.85; b GPRAMA(4)-EC, PSNR=34.51

GPRED(4)-CDT has quality PSNR=33.49 while GPRED(4)-EC has PSNR=33.85. GPRAMA(4)-CDT has quality PSNR=33.18 while GPRAMA(4)-EC has PSNR=34.51. The GPRED-CDT method is essentially the GPRFS-ED method in Adam’s paper [17] while GPRED-EC is an improved version of GPRED-CDT due to the different triangulation of the local patch. For GPRAMA, using constrained Delaunay triangulation for mesh patching does not preserve the anisotropy of the initial mesh, especially when significant amount of points are removed. Therefore, ear clipping method works better for GPRAMA, and GPRAMA-EC gives the best quality among the four GPR-related representations. Similar results are observed for image peppers as shown in Figs. 12 and 13.

Fig. 12

Meshes of different GPR-related representations for image peppers at sample density of 3 %: a GPRED(4)-CDT, PSNR=33.65; b GPRED(4)-EC, PSNR=33.74; c GPRAMA(4)-CDT, PSNR=33.44; d GPRAMA(4)-EC, PSNR=34.23

Fig. 13

Reconstructed images using different GPR-related representations for image peppers at sample density of 3 %: a GPRED(4)-CDT, PSNR=33.65; b GPRAMA(4)-EC, PSNR=34.23

The mesh qualities at different sample densities for both Lena and peppers using GPR-related representations are shown in Table 4, where mesh patching with ear clipping is used for GPR and GPRAMA methods. GPRAMA(4) provides better quality than the traditional GPR method for both images except at sample density of 1 % for image peppers. Furthermore, GPRAMA(3) performs better than GPRED(5)-CDT and is comparable with GPRED(5)-EC for both images. Therefore, GPRAMA can achieve better quality than GPRED while starting with smaller |S ₀| which indicates less computational cost, especially for high-resolution images. The computational costs for different methods are provided in the next subsection.

Table 4 Comparison of mesh qualities obtained with various GPR-related methods

Figure 14 shows a picture of the Golden Gate Bridge of pixel resolution 4000×3000 and its GPRAMA representation at sample density 1 % with γ=4. The reconstructed image is shown in Fig. 15 and the representation quality is PSNR=34.30. Some results using different representation methods are provided in Table 5, where ear clipping is used in the mesh patching for all the GPR-related methods. As can be seen, GPRAMA(2) has quality as good as GPRED(4) but starts with only half of the initial points. The results are consistent with our previous observations for other images.

Fig. 14

GPRAMA representation for a image of Golden Gate Bridge at sample density of 1 %: a original image, 4000×3000; b GPRAMA(4), PSNR=34.30

Fig. 15

Reconstructed image of GPRAMA(4) representation for Golden Gate Bridge in Fig. 14 a at sample density of 1 %, PSNR=34.30

Table 5 Comparison of mesh qualities for an image of Golden Gate Bridge^a

4.4 Computational complexity

Here, we compare the computational complexity of the various image representation methods considered in this paper. The computational complexity is measured in terms of CPU execution time in seconds (converted from clock ticks) and varies for different hardware and software environment. Our computations in this paper are performed in a MacBook Pro laptop with 2.6 GHz Intel Core i7 CPU, 8GB 1600MHz DDR3 memory, and OSX 10.9.5 operating system.

Note that our program code was developed with basic level of efficiency and has not been optimized for execution speed. Thus, the absolute CPU time for each method may be reduced by using highly optimized code. However, our focus here is to compare the computational complexity among the different methods. The CPU times for representations of images Lena and peppers with and without GPR are provided in Table 6, where ear clipping mesh patching technique is used in all the GPR-related methods.

Table 6 Comparison of CPU time (in s) for different representations

It is clear that ED method is the fastest (with low quality) and GPR is the most computationally expensive method (with good quality), and the result is consistent with the existing literatures. For example, at sample density of 3 % for image Lena, ED representation takes only 0.026 s, while GPR method takes 248 s. For image peppers at sample density of 3 %, ED method takes 0.027 s while GPR takes 318 s. The smaller sample density, the longer time GPR needs because more points need to be removed before reaching the desired sample density. For all other methods, the computational cost is lower for smaller sample density.

For representation methods without greedy-point removal technique, $\mathbb {M}_{H,3}$ and $\mathbb {M}_{\text {aniso,3}}$ provide better quality (see Table 1) but have higher computational cost than ED method (see Table 6). For example, at sample density of 3 % for image Lena, the $\mathbb {M}_{H,3}$ representation takes 0.45 s and $\mathbb {M}_{\text {aniso,3}}$ takes 0.47 s, which are about 16 times more than ED method but only 0.2 % of the time needed by GPR method. Similar results are observed for image peppers. Therefore, $\mathbb {M}_{H,3}$ and $\mathbb {M}_{\text {aniso,3}}$ are good balances between ED and GPR. In the meantime, $\mathbb {M}_{\text {aniso,3}}$ takes about 0.02 s longer than $\mathbb {M}_{H,3}$ which is due to the extra time needed to compute $\mathbb {M}_{\text {aniso}}$ (8) in addition to $\mathbb {M}_{H}$ (5). However, the extra cost is negligible (less than 5 %).

For GPR-related representation methods, both GPRED(5) and GPRAMA(3) provide comparable quality with GPR (see Table 4) but take much less time (see Table 6). For example, at sample density of 3 % for image Lena, GPRED(5) takes 10.5 s and GPRAMA(3) takes 4.93 s. GPRAMA(3) only takes 2 % of the time needed by GPR and less than half of the time needed by GPRED(5). Therefore, GPRAMA method can provide comparable quality with GPRED but with lower computational cost, which makes it another good balance between ED and GPR. The results for image peppers are also similar.

5 Conclusions

Adaptive sampling has become popular in image representation, among which triangular meshes have gained much interest. One common approach is to develop schemes to choose proper sample points then connect the points to form a triangular mesh. Another approach is to generate and adapt the mesh directly to represent the image. In this paper, we have introduced a framework of anisotropic mesh adaptation methods to image representation. The AMA methods take the $\mathbb {M}$-uniform mesh approach and use a metric tensor $\mathbb {M}$ to control the triangular mesh. Firstly, an initial Delaunay triangular mesh is generated based on the desired sample density. Then, the mesh is adapted using the software BAMG according to the provided metric tensor. Lastly, finite element interpolation is used to reconstruct the image from the mesh. The anisotropic metric tensor $\mathbb {M}_{\text {aniso}}$ in (8) provides the best representation in this framework among the considered metric tensors. Note that the method proposed by Courchesne et al. [13] (with minor modification) is a special case within this framework.

Within the AMA representation framework, we have developed a GPRAMA method based on the greedy-point removal scheme and a mesh patching technique. The local polygon (may be concave) surrounding a mesh vertex can be triangulated using constrained Delaunay triangulation or ear clipping method. When choosing the initial points using the error-diffusion scheme and CDT is chosen for mesh patching, the corresponding representation method is denoted as GPRED-CDT that is essentially the same as the GPRFS-ED method proposed by Adams [17]; while choosing EC for mesh patching leads to an improved version denoted as GPRED-EC. When starting the initial points from an AMA mesh, in particular, an $\mathbb {M}_{\text {aniso}}$ mesh, and EC is chosen for mesh patching, we obtain the GPRAMA representation method that provides better quality than the GPRFS-ED method but with lower computational cost. Overall, mesh patching with EC provides better quality for GPR-related representation methods than mesh patching with CDT. Numerical results on two standard test images, Lena and peppers, are presented, as well as on an image of the Golden Gate Bridge that has higher resolution. The observations are confirmed by the results from three other images. All the results demonstrate that AMA representation is superior than ED representation, and GPRAMA performs better than the GPRFS-ED method.

AMA representation methods have clear mathematic framework and provides flexibility for both adaptation using different metric tensors and reconstruction using different interpolation methods, although we have only focused on linear finite element interpolation in this paper. The AMA representation of the image and the mesh adaptation strategy will be useful for image scaling and PDE-based image processing such as image smoothing and edge enhancement using anisotropic diffusion filters, which are topics under our current investigation.

6 Appendix

7 Finite element interpolation for triangles

This appendix provides a brief introduction to the linear and quadratic finite element interpolation for triangles. The interpolation for quadrilateral elements are similar. Figure 1 shows a triangular element K and an isosceles right triangle $\hat {K}$ as the reference element. The vertices of K are denoted as a ₁,a ₂, and a ₃, and the midpoints of the corresponding sides are denoted by a ₄,a ₅, and a ₆. The vertices of the reference element $\hat {K}$ are located at $\hat {\textbf {a}}_{1} (0,0), \hat {\textbf {a}}_{2} (1,0)$, and $\hat {\textbf {a}}_{3} (0,1)$. The midpoints in $\hat {K}$ are located at $\hat {\textbf {a}}_{4} (0.5,0), \hat {\textbf {a}}_{5} (0.5,0.5)$, and $\hat {\textbf {a}}_{6} (0,0.5)$.

Denote the coordinates of the vertices of K as a ₁(x ₁,y ₁),a ₂(x ₂,y ₂), and a ₃(x ₃,y ₃). The corresponding function values are denoted by f ₁=f(x ₁,y ₁), f ₂=f(x ₂,y ₂), and f ₃=f(x ₃,y ₃). For any point a(x,y) in the element K, the corresponding point $\hat {\textbf {a}}(\xi,\eta)$ in the reference element $\hat {K}$ is given by

$$ \left[ \begin{array}{c} \xi \\ \eta \end{array} \right] = \left[ \begin{array}{ll} x_{2}-x_{1} & x_{3}-x_{1} \\ y_{2}-y_{1} & y_{3}-y_{1} \end{array}\right]^{-1} \times \left[ \begin{array}{c} x-x_{1} \\ y-y_{1} \end{array} \right]. $$

(14)

For linear interpolation, only the values at the three vertices are needed, and the function value at any point (ξ,η) is interpolated as follows

$$ f(\xi,\eta) = \sum_{i=1}^{3} f_{i} \cdot N_{i}(\xi,\eta), $$

(15)

where N _i (ξ,η) is the basis functions at $\hat {\textbf {a}}_{i}$ and is defined as follows

$$ N_{1}(\xi,\eta) = 1- \xi -\eta; \quad N_{2}(\xi,\eta)=\xi; \quad N_{3}(\xi,\eta)=\eta. $$

(16)

For quadratic interpolation, the midpoints of the sides are needed. Denote the function values at midpoints a ₄,a ₅, and a ₆ as f ₄,f ₅, and f ₆, respectively. Then, the function value at any point (ξ,η) in $\hat {K}$ is interpolated as follows

$$ f(\xi,\eta) = \sum_{i=1}^{6} f_{i} \cdot N_{i}(\xi,\eta), $$

(17)

where the basis functions are defined as follows

$$\begin{array}{@{}rcl@{}} && N_{1} = (1-\xi-\eta)(1-2\xi-2\eta); \quad N_{2} = \xi (2\xi-1); \\ && N_{3} = \eta (2\eta-1); \quad N_{4} = 4 \xi (1-\xi-\eta); \\ && N_{5} = 4 \xi \eta; \quad N_{6} = 4 \eta (1-\xi-\eta). \end{array} $$

(18)