# Fast colorization based image coding algorithm using multiple resolution images

- Kazunori Uruma†
^{1}Email author, - Katsumi Konishi†
^{2}, - Tomohiro Takahashi
^{1}and - Toshihiro Furukawa
^{1}

**2016**:7

https://doi.org/10.1186/s13640-016-0107-8

© Uruma et al. 2016

**Received: **9 January 2015

**Accepted: **1 February 2016

**Published: **10 February 2016

## Abstract

This paper proposes a representative pixel (RP) extraction algorithm and chrominance image recovery algorithm for the colorization-based digital image coding. The colorization-based coding methods reduce the color information of an image and achieve higher compression ratio than JPEG coding; however, they take much more computing time. In order to achieve low computational cost, this paper proposes the algorithm using the set of multiple-resolution images obtained by colorization error minimizing method. This algorithm extracts RPs from each resolution image and colorizes each resolution image utilizing a lower resolution color image, which leads to the reduction of the number of RPs and computing time. Numerical examples show that the proposed algorithm extracts the RPs and recovers the color image fast and effectively.

### Keywords

Image colorization Image coding Representative pixel Multiple-resolution images## 1 Introduction

The image colorization technique can recover a full-color image from a luminance image and several representative pixels (RPs) which have the chrominance values and their positions [1–4]. Levin et al. proposes a well known colorization algorithm, which achieves a high colorization performance using appropriately given RPs. However, this algorithm requires high computing cost and fails to colorize a whole image if only a few RPs are given. The authors have proposed the colorization algorithm, which can restore a color image from only a few RPs and takes a low computational cost [4].

The development of the colorization technique has led to the generation of a new color image coding method called colorization-based color image coding [5–12]. Because the luminance image can be compressed by standard coding methods such as JPEG coding and because the chrominance information is represented by a small number of RPs, the colorization-based image coding can compress full-color images more than these methods. The colorization-based image coding algorithms proposed in [10, 12] have achieved higher coding performance than JPEG2000, which is a highly optimized standard using many entropy coding techniques together. Therefore, the colorization-based image coding has potential to become standard image coding for the next generation. The performance of the colorization based coding is evaluated based on the number of RPs and the quality of a restore image, and therefore, the objective of this paper is to propose a new coding method which requires less RPs and gives more accurate chrominance information. In order to reduce the amount of information to represent the RPs, the method [5] assumes that all pixels in a line segment have only one color and proposes the algorithm where the RPs are described as a set of color line segments. In [9], the algorithm finds a set of two RPs which give the similar effect for colorization result, and the redundant RPs are deleted. In [12], under the assumption that the chrominance image is given as a sparse linear combination of the basis constructed from the luminance image, the algorithm achieves a high compression performance using the theory of compressed sensing [13, 14]. However, these algorithms require a high computing time depending on the image size. To reduce computing time, the method [5] proposes an algorithm to extract RPs from a single low-resolution image. This paper takes the same approach to reduce the computational costs and provides an RPs extraction algorithm using multiple low-resolution images. Though a low resolution image used in [5] is obtained by a simple downsampling, this paper gives low resolution images which are optimized such that a good color image is recovered in the colorization phase.

This paper is organized as follows. In Section 2.2, we give a colorization algorithm based on the algorithm proposed in [4]. Section 2.3 proposes the RPs extraction algorithm which uses recovery error feedback, and it is applied to multiple resolution images. In Section 2.4, a multi-resolution color image recovery algorithm is proposed. We consider the implementation of saving images with RPs in Section 2.5. Numerical examples show that the proposed algorithm extracts the RPs and recovers the color image fast and effectively in Section 3.

## 2 Main works

### 2.1 Notation

We provide here a brief summary of the notations used throughout the paper. (*A*)_{
i,j
} and (a)_{
i
} denote the (*i*,*j*)-element of a matrix A and the *i*th element of a vector a. ∥·∥_{2} and ∥·∥_{
F
} denote the *l*
_{2} norm of a vector and the Frobenius norm of a matrix, respectively. 0
_{
k
}∈*R*
^{
k
} is denoted as a *k*-dimension zero vector. We use diag(*A*
_{1},…,*A*
_{
m
}) to denote a block diagonal matrix consisting of *A*
_{1}, …, *A*
_{
m
} and |*S*| to denote the number of elements in a set *S*. We denote the ceiling function and the floor function by ⌈*a*⌉ and ⌊*a*⌋, respectively.

### 2.2 Colorization algorithm

In order to provide a colorization-based image coding, this paper applies the colorization algorithm proposed in [4] because it can recover a full color image from a luminance image using only a few color pixels. The algorithm restores chrominance images *C*
_{
b
} and *C*
_{
r
} independently to obtain a color image, and this subsection gives a chrominance image restoration method.

^{ m n }and y∈R

^{ m n }denote a desired chrominance image and a rasterized-given luminance image, respectively, where

*m*and

*n*denote height and width of the image. Define

*U*∈R

^{(m−1)×m },

*V*∈R

^{ m(n−1)×m n }, \(\bar {U} \in \boldsymbol {R}^{(m-1)n\times mn}\) and

*D*∈R

^{(2m n−m−n)×m n }as

*V*denote vertical and horizontal difference operators. Then

*D*x denotes the differences between the neighbor pixels of a whole image. Let us consider the following

*ℓ*

_{2}norm minimizing colorization problem,

*c*

_{ i }is a constant corresponding to a given color value, and the index set denotes a given set of vector indices of given color pixels. This problem recovers a color image by minimizing the sum of differences between neighbor pixels. In order to colorize an image more appropriately, [4] assumes that the differences between neighbor pixels of a chrominance image have relationship with those of a color image, which is defined by the following equality,

*α*is an unknown constant, and

*f*:

*R*→

*R*is a given function. Then, we obtain the following problem from (2) under the assumption (3),

where *F* is a diagonal matrix whose *i*th diagonal element is 1/*f*((*D*
y)_{
i
}). The details are described in [4]. While the problem (4) is a convex quadratic programming and can be solved exactly, [4] applies Lagrangian relaxation to reduce the computational cost. This paper provides an exact analytical solution of (4) which can be implemented simply.

*H*∈

*R*

^{(2m n−m−n)×m n }by

*FD*. Then, we define

*H*

^{∗}∈

*R*

^{(2m n−m−n)×(m n−p)}and \(\acute {H^{*}} \in R^{(2mn-m-n)\times p}\) as submatrices of

*H*composed of columns corresponding to unknown color pixels and given color pixels, respectively. Let x

^{ ∗ }∈

*R*

^{ m n−p }and \(\acute {\boldsymbol {x^{*}}} \in R^{p}\) denote unknown color vectors and given color vectors, respectively, where

*p*denotes the number of given color pixels. Then the objective function of (4) is rewritten as follows,

*c*

_{ i }, that is, they satisfy the constraint of (4), the problem (4) is equal to the following nonconstrained convex quadratic programming,

*c*

_{ i }. If

*H*

^{∗}

^{ T }

*H*

^{∗}is nonsingular, we obtain the solution of (6) as follows,

Now we move onto discuss the singularity of *H*
^{∗}
^{
T
}
*H*
^{∗}. For an image \(\boldsymbol {a} =\, [\!a_{1},a_{2},\dots,a_{\textit {mn}}]^{T} \in \boldsymbol {R}^{mn}\), the following lemma is provided.

###
**Lemma**
**1**.

It holds that *H*
**a**=**0**
_{2m
n−m−n
} if and only if \(a_{1}=a_{2}=\dots =a_{\textit {mn}}\).

###
*Proof*.

Because *F* is an invertible matrix, *H*
a=0
_{2m
n−m−n
} if and only if *D*
a=0
_{2m
n−m−n
}. Since *D* is the difference operator defined by (1), *D*
a is a zero vector if and only if all pixels have the same values, that is, \(a_{1}=a_{2}=\dots =a_{\textit {mn}}\).

Let a
^{(i)}∈*R*
^{
m
n−1} denotes a subvector of a and be composed by deleting an *i*th element of a, and *H*
^{(i)}∈*R*
^{(2m
n−m−n)×(m
n−1)} is denoted as a submatrix of *H* and is composed by deleting the *i*th column of *H*. Then the Lemma 1 leads to the following lemma:

###
**Lemma**
**2**.

For all *i*∈{1,2,…,*m*
*n*}, it holds that *H*
^{(i)}
a
^{(i)}=**0**
_{2m
n−m−n
} if and only if a
^{(i)}=**0**
_{
m
n−1}.

###
*Proof*.

###
**Theorem**
**1**.

*H*
^{∗}
^{
T
}
*H*
^{∗} is a nonsingular matrix

###
*Proof*.

Lemma 2 guarantees that the subspace of the column space of *H* is linear independent, that is, the column space of *H*
^{∗} is linear independent.

From this theorem, there exists the inverse of *H*
^{∗}
^{
T
}
*H*
^{∗}, and therefore we can obtain the exact solution of (4) by (7).

### 2.3 RPs extraction algorithm

This subsection proposes an algorithm to extract the RPs from an original image utilizing the error feedback between the original image and the colorized image. Let *Y*∈R
^{
m×n
}, *C*
*b*∈R
^{
m×n
} and *C*
*r*∈R
^{
m×n
} denote a given luminance image and two given chrominance images, respectively. First we provide the following RPs extraction scheme for a given color image, its luminance image, a given initial set of RPs and a small constant *ε*>0,

**Step 1.** Recover two chrominance images \(\bar {Cb}\) and \(\bar {Cr}\) using (7) with the luminance image and current RPs.

**Step 2.**Calculate the error between the original image and the colorized image obtained in Step 1 at each pixel as follows,

**Step 3.** Terminate if (*E*)_{
i,j
}≤*ε* for all (*i*,*j*), else go to Step 4.

**Step 4.** Add the pixel with the largest error *E*
_{
i,j
} to the set of RPs.

**Step 5.** Terminate if the number of iterations is larger than a given threshold, else go to Step 1.

Since this scheme extracts one pixel in one iteration, some pixels are selected appropriately as RPs by repeating Step 1–5. Although this scheme can steadily reduce the recovery error, it requires a lot of iterations to obtain an adequate amount of RPs and takes a lot of computing time for a large size image. In order to reduce the computing cost, this paper proposes an RP extraction algorithm using multiple resolution images.

*I*

_{0}= [

*Y*

_{0}

*C*

*b*

_{0}

*C*

*r*

_{0}]∈

*R*

^{ m×3n }and \(I_{k}=\,[\!Y_{k} \ Cb_{k} \ Cr_{k}] \in {R}^{m_{k}\times 3n_{k}}\ (k=1,2,\cdots,N)\) denote the original image and

*k*th reduced image, respectively, and

*N*≥1 is a given constant.

*Y*

_{ k }is obtained by simply multiple downsampling as follows,

*m*

_{ k }and

*n*

_{ k }denote horizontal and vertical sizes of the

*k*th reduced image. Next, we consider a way to generate

*C*

*b*

_{ k }and

*C*

*r*

_{ k }. Since

*C*

*b*

_{ k }and

*C*

*r*

_{ k }are composed by the same method, this paper gives a way to make the reduced image

*C*

*b*

_{ k }. The chrominance images of the

*k*th reduced image are generated so that the (

*k*−1)th chrominance image

*C*

*b*

_{ k−1}can be restored preciously using

*Y*

_{ k−1}and

*C*

*b*

_{ k }. Let \(\acute {\mathbf {c}}_{k-1}^{*} \in R^{m_{k}n_{k}}\) and \(\mathbf {c}_{k-1}^{\ast } \in R^{{m_{k-1}}{n_{k-1}}{-m_{k}}{n_{k}}}\) denote vectors formed by pixels of even numbered columns and rows, that is, they do not include pixels of odd numbered columns nor rows. Then, this paper proposes the chrominance image recovery problem of the

*k*th reduced image as follows,

*λ*>0 is a given constant. Because the chrominance images

*C*

*b*

_{ k−1}and

*C*

*r*

_{ k−1}are recovered using (7) with

*Y*

_{ k−1},

*C*

*b*

_{ k }and

*C*

*r*

_{ k }in the colorization phase as written Section 2.4, this problem gives

*C*

*b*

_{ k }to achieve a good colorization of the (

*k*−1)th image. However, problem (12) requires high computational cost to obtain the optimal solution since it takes high cost to calculate (

*H*

^{∗}

^{ T }

*H*

^{∗})

^{−1}. Therefore this paper provides its relaxed problem to obtain an approximate solution with low computational cost. Let

*J*denote the first term of the objective function in (12). Then, we have that

*J*is bounded by \(\|H^{*}\boldsymbol {c}_{k-1}^{*} +\acute {H^{*}}\boldsymbol {x}_{k}\|_{2}^{2}\), we consider the following problem to reduce

*J*instead of (12),

Utilizing (11) and (14) to obtain a set of multiple resolution images, this paper provides the RPs extraction method as shown in Algorithm ??, where \(\bar {Cb}_{k}\) and \(\bar {Cr}_{k}\) denote the colorized chrominance images of *I*
_{
k
}. This algorithm extracts RPs from *I*
_{
N
} to *I*
_{
N−K
} in turn for given constant *K*≥1, and the even numbered pixels of *I*
_{
k
} are used as initial RPs at each iteration. The performance of Algorithm ?? depends heavily on the value of *K*. If the value of *K* is large, lots of RPs are extracted, that is, the *K* determines the number of RPs, the calculation time and the volume of information to store RPs. The numerical examples in Section 3 show the effects of *K*.

*C*

*b*

_{ k+1}and

*C*

*r*

_{ k+1}in the previous iteration and are reused as RPs for recovering

*I*

_{ k }. Therefore Algorithm ?? does not store them in \(\Omega _{k}^{RP}\) to reduce the amount of information about RPs. All pixels of the

*N*th reduced image are members of \(\Omega _{N}^{RP}\). Figure 3 shows outline of the algorithm.

### 2.4 Color image recovery

*I*

_{0}= [

*Y*

_{0}

*C*

*b*

_{0}

*C*

*r*

_{0}]∈

*R*

^{ m×3n }from the stored RPs \(\Omega _{k}^{RP} \ (k = N,\ldots N-K)\). First,

*I*

_{ N }is completely obtained by \(\Omega _{N}^{RP}\). Next, we focus onto recover the image

*I*

_{ k }(

*k*=

*N*−1,

*N*−2,…,0). Since \(\Omega _{k}^{RP}\) includes RPs without the even numbered pixels, image

*I*

_{ k+1}is used as \(\Omega _{k}^{RP}\) corresponding to the even numbered pixels. Then the color images

*I*

_{ k }is restored by (7) using \(\Omega _{k}^{RP}\) and

*I*

_{ k+1}. Finally we obtain the colorization algorithm with multiple resolution images as shown in Algorithm ??, and Fig. 4 shows its outline.

*H*

^{∗}

^{ T }

*H*

^{∗})

^{−1}, and the size of (

*H*

^{∗}

^{ T }

*H*

^{∗})

^{−1}depends on the number of unknown color pixels. The number of unknown color pixels in Algorithm ?? is equal to the following,

This implies that the calculation cost does not depend on *N*. Numerical examples in Section 3 show that the computing time is independent of *N*.

### 2.5 Volume of information of RPs

This subsection gives an encoding method of RPs.

Let us consider the case that an *m*×*n* 24-bit image is compressed with p RPs. If the information of RPs is composed of their coodinates and two chrominance values, *p*(2×8+⌈log_{2}
*m*
*n*⌉) [bits] are required to represent all RPs because 2×8 [bits] and ⌈log_{2}
*m*
*n*⌉ [bits] are required to represent their chrominance values and coordinates, respectively. The proposed algorithm extracts RPs based on multiple resolution images and requires less information to represent them.

*I*

_{ N }, the \(\Omega _{N}^{RP}\) are given as the reduced image

*I*

_{ N }, that is, all pixels of

*I*

_{ N }are RPs. Therefore, the coordinates of \(\Omega _{N}^{RP}\) are not required, and only the value of

*N*is coded to represent \(\Omega _{N}^{RP}\). Next, we consider coding of the coordinates of RPs in \(\Omega _{k}^{RP} \ (k=N-1, \ldots,N-K)\). For

*m*×

*n*image, we store the coordinates of RPs as a rasterized vector of a binary matrix of size

*m*×

*n*whose entries are 1 if corresponding pixels are RPs and 0 if otherwise. Hence, this vector is sparse, and it is coded by run length encoding (RLE), which stores the length of consecutive 0s. Empirical results show that the length of consecutive 0s on the image

*Ω*

*N*−

*k*

*R*

*P*is less than

*z*

_{ k }defined by

*b*

*i*

*t*

*u*

*n*

*i*

*t*

_{ N−k }=3. This decoding method requires the run length data of \(\Omega _{k}^{RP} \ (k = N-1, \ldots,N-K)\) and the values of

*N*and

*t*

_{ max }to decode the coordinates of RPs.

Correspondence table (bit unit is 3)

Length | Sign |
---|---|

9 | 111001 |

8 | 111000 |

7 | 110 |

⋯ | ⋯ |

2 | 010 |

1 | 001 |

0 | 000 |

## 3 Numerical examples

*ε*=2 and

*λ*=0.3, which are selected empirically from

*ε*∈ [ 0,10] and

*λ*∈ [ 0.1,1] to achieve the best performance. We use

*t*

_{ max }=70 except for the second experiment.

*N*and

*K*using uncompressed luminance images. Table 2 shows PSNR, the computing time of Algorithm ??, the computing time of Algorithm ?? and the volume of information to store RPs. PSNR is obtained by averaging those of Cb and Cr. As can be seen, the results of

*N*≥5 are almost the same. Figure 6 shows the PSNR and the computing time of Algorithm ?? with

*N*=5 for

*N*−

*K*∈{0,1,2,3,4}. We can see that the algorithm with (

*N*,

*K*)=(5,3) achieves the best tradeoff between calculation time and PSNR, and therefore we use (

*N*,

*K*)=(5,3) in the rest of this section.

Numerical results for several (*N*,*K*)s

Milkdrop | N | N-K | Airplane | N | N-K | ||||||
---|---|---|---|---|---|---|---|---|---|---|---|

0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | ||||

PSNR [dB] | 4 | 38.0759 | 37.5711 | 37.1943 | 34.3947 | PSNR [dB] | 4 | 38.2035 | 33.2393 | 32.3945 | 32.0054 |

5 | 37.6920 | 37.2186 | 36.8684 | 34.2402 | 5 | 38.1185 | 33.2027 | 32.3641 | 31.9780 | ||

6 | 37.6314 | 37.1643 | 36.8181 | 34.2058 | 6 | 37.7937 | 33.1094 | 32.2888 | 31.9157 | ||

7 | 37.6244 | 37.1576 | 36.8116 | 34.2006 | 7 | 37.7754 | 33.1061 | 32.2874 | 31.9159 | ||

Encoding Time [s] | 4 | 14.1671 | 2.3608 | 0.6417 | 0.2239 | Encoding Time [s] | 4 | 14.6127 | 2.3328 | 0.6553 | 0.2307 |

5 | 13.9903 | 2.4329 | 0.7173 | 0.2841 | 5 | 14.3815 | 2.3910 | 0.7077 | 0.2871 | ||

6 | 13.9673 | 2.4428 | 0.7311 | 0.3021 | 6 | 14.6817 | 2.4039 | 0.7209 | 0.3002 | ||

7 | 14.0246 | 2.4431 | 0.7268 | 0.3098 | 7 | 14.7841 | 2.4064 | 0.7275 | 0.3042 | ||

Decoding Time [s] | 4 | 0.2144 | 0.2189 | 0.2122 | 0.2170 | Decoding Time [s] | 4 | 0.2099 | 0.2111 | 0.2033 | 0.1969 |

5 | 0.2137 | 0.1979 | 0.2156 | 0.1996 | 5 | 0.2240 | 0.1994 | 0.2011 | 0.1988 | ||

6 | 0.2382 | 0.2101 | 0.2075 | 0.1974 | 6 | 0.2193 | 0.2040 | 0.2087 | 0.2146 | ||

7 | 0.2242 | 0.2021 | 0.2336 | 0.2136 | 7 | 0.2181 | 0.1986 | 0.2042 | 0.2231 | ||

Volume [byte] | 4 | 1407 | 1146 | 913 | 702 | Volume [byte] | 4 | 1411 | 1148 | 915 | 706 |

5 | 1193 | 933 | 699 | 488 | 5 | 1197 | 935 | 702 | 492 | ||

6 | 1195 | 933 | 700 | 489 | 6 | 1154 | 892 | 659 | 449 | ||

7 | 1193 | 932 | 699 | 488 | 7 | 1149 | 886 | 653 | 444 | ||

Boat | N | N-K | Couple | N | N-K | ||||||

0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | ||||

PSNR [dB] | 4 | 34.2850 | 33.9851 | 33.6053 | 32.5525 | PSNR [dB] | 4 | 38.5062 | 38.2618 | 37.9755 | 37.0003 |

5 | 34.0394 | 33.7643 | 33.4117 | 32.4187 | 5 | 38.3833 | 38.1462 | 37.8793 | 36.9063 | ||

6 | 34.0133 | 33.7336 | 33.3841 | 32.3973 | 6 | 38.3118 | 38.0789 | 37.8241 | 36.8307 | ||

7 | 34.0132 | 33.7335 | 33.3840 | 32.3969 | 7 | 38.3085 | 38.0757 | 37.8214 | 36.8280 | ||

Encoding Time [s] | 4 | 14.117 | 2.3290 | 0.6535 | 0.2281 | Encoding Time [s] | 4 | 14.7000 | 2.3234 | 0.6541 | 0.2365 |

5 | 14.1622 | 2.4299 | 0.7107 | 0.2882 | 5 | 14.3138 | 2.3749 | 0.7144 | 0.2868 | ||

6 | 14.2034 | 2.4091 | 0.7348 | 0.3129 | 6 | 14.3077 | 2.3927 | 0.7312 | 0.3089 | ||

7 | 14.0931 | 2.4581 | 0.7512 | 0.3148 | 7 | 14.4525 | 2.4048 | 0.7412 | 0.3151 | ||

Decoding Time [s] | 4 | 0.2119 | 0.2098 | 0.2359 | 0.1954 | Decoding Time [s] | 4 | 0.2105 | 0.1967 | 0.1976 | 0.2081 |

5 | 0.2111 | 0.2058 | 0.2159 | 0.2064 | 5 | 0.2072 | 0.1983 | 0.1980 | 0.2156 | ||

6 | 0.2162 | 0.2037 | 0.2016 | 0.1980 | 6 | 0.2075 | 0.2159 | 0.2026 | 0.2246 | ||

7 | 0.2116 | 0.2069 | 0.2099 | 0.2536 | 7 | 0.2068 | 0.2166 | 0.2120 | 0.2006 | ||

Volume [byte] | 4 | 1413 | 1150 | 912 | 702 | Volume [byte] | 4 | 1414 | 1154 | 917 | 705 |

5 | 1198 | 936 | 698 | 487 | 5 | 1200 | 940 | 703 | 491 | ||

6 | 1195 | 933 | 695 | 484 | 6 | 1194 | 934 | 697 | 485 | ||

7 | 1198 | 936 | 698 | 487 | 7 | 1191 | 931 | 694 | 482 | ||

Lenna | N | N-K | Earth | N | N-K | ||||||

0 | 1 | 2 | 3 | 0 | 1 | 2 | 3 | ||||

PSNR [dB] | 4 | 36.3927 | 36.2937 | 36.0081 | 35.3116 | PSNR [dB] | 4 | 38.5740 | 38.4291 | 38.1530 | 37.4579 |

5 | 36.1112 | 36.0174 | 35.7481 | 35.1009 | 5 | 38.1631 | 38.0269 | 37.7843 | 37.0972 | ||

6 | 36.0689 | 35.9747 | 35.7079 | 35.0432 | 6 | 38.0894 | 37.9473 | 37.7087 | 37.0341 | ||

7 | 36.0716 | 35.9774 | 35.7105 | 35.0497 | 7 | 38.0962 | 37.9539 | 37.7153 | 37.0421 | ||

Encoding Time [s] | 4 | 14.0836 | 2.2984 | 0.6657 | 0.2247 | Encoding Time [s] | 4 | 14.5969 | 2.2922 | 0.6481 | 0.2284 |

5 | 14.1050 | 2.3610 | 0.7059 | 0.2838 | 5 | 14.2419 | 2.3374 | 0.7004 | 0.2842 | ||

6 | 14.4354 | 2.3993 | 0.7309 | 0.3194 | 6 | 14.3422 | 2.3709 | 0.7293 | 0.3078 | ||

7 | 14.5874 | 2.3534 | 0.7317 | 0.3150 | 7 | 14.1882 | 2.3787 | 0.7252 | 0.3131 | ||

Decoding Time [s] | 4 | 0.2023 | 0.2110 | 0.2272 | 0.1950 | Decoding Time [s] | 4 | 0.2106 | 0.2018 | 0.2078 | 0.1997 |

5 | 0.2278 | 0.2029 | 0.2078 | 0.2096 | 5 | 0.2291 | 0.2076 | 0.1970 | 0.2152 | ||

6 | 0.2624 | 0.2173 | 0.2230 | 0.2273 | 6 | 0.2057 | 0.2124 | 0.1964 | 0.2003 | ||

7 | 0.2388 | 0.2115 | 0.2162 | 0.1970 | 7 | 0.2082 | 0.2053 | 0.1983 | 0.2041 | ||

Volume [byte] | 4 | 1420 | 1153 | 915 | 704 | Volume [byte] | 4 | 1402 | 1143 | 909 | 702 |

5 | 1206 | 939 | 701 | 490 | 5 | 1188 | 929 | 695 | 488 | ||

6 | 1200 | 934 | 696 | 485 | 6 | 1184 | 926 | 692 | 485 | ||

7 | 1204 | 937 | 699 | 488 | 7 | 1183 | 924 | 690 | 484 | ||

Pepper | N | N-K | |||||||||

0 | 1 | 2 | 3 | ||||||||

PSNR [dB] | 4 | 32.6169 | 30.3128 | 30.0602 | 29.1197 | ||||||

5 | 32.0963 | 29.9893 | 29.7612 | 28.8955 | |||||||

6 | 32.0913 | 29.9865 | 29.7585 | 28.8929 | |||||||

7 | 32.0914 | 29.9867 | 29.7588 | 28.8932 | |||||||

Encoding Time [s] | 4 | 14.7880 | 2.3244 | 0.6564 | 0.2304 | ||||||

5 | 14.3656 | 2.3847 | 0.7095 | 0.2876 | |||||||

6 | 14.4174 | 2.4086 | 0.7376 | 0.3121 | |||||||

7 | 14.5826 | 2.4150 | 0.7427 | 0.3207 | |||||||

Decoding Time [sec] | 4 | 0.2103 | 0.2026 | 0.1971 | 0.2081 | ||||||

5 | 0.2246 | 0.2147 | 0.1985 | 0.2220 | |||||||

6 | 0.2037 | 0.1981 | 0.1977 | 0.1986 | |||||||

7 | 0.2061 | 0.2201 | 0.2230 | 0.1985 | |||||||

Volume [byte] | 4 | 1412 | 1143 | 911 | 702 | ||||||

5 | 1198 | 929 | 697 | 488 | |||||||

6 | 1206 | 937 | 705 | 496 | |||||||

7 | 1209 | 940 | 708 | 499 |

*λ*=5 in (12), which is selected empirically from

*λ*∈[1,10] to achieve the best performance. To evaluate the performance of the relaxed problem (14) in RPs extraction scheme, we also examine Algorithm ?? with (12) instead of (14). Because we cannot fix the number of RPs exactly in the algorithm proposed in [9], we adjust the parameter

*t*

_{ max }of Algorithm ?? such that the number of RPs is nearly equal to that of [9]. The random algorithm selects RPs randomly and gives the same number of RPs as the proposed algorithm, and we use colorization algorithm (7). In order to see the performance of colorization based coding method, these algorithms use uncompressed luminance images. Because the algorithm proposed in [9] and the proposed algorithm with (12) use a large amount of memory and take a high computational cost, they cannot be applied to a whole image of the test images. Therefore, we use their partial images as shown in Fig. 7. Table 3 shows the number of RPs, the volume of information to restore the RPs, SSIM, PSNR, the computing time to extract the RPs (encoding time), and to colorize the images (decoding time). As can be seen, the proposed algorithm with (14) works faster and achieves better performance than the other algorithms.

Numerical results of comparison with other colorization based image coding algorithms

Image | Algorithm | Number | Volume of | Time [sec] | PSNR [dB] | SSIM | |||
---|---|---|---|---|---|---|---|---|---|

of RPs | information | ||||||||

[pixels] | [byte] | encoding | decoding | Cb | Cr | Cb | Cr | ||

Milkdrop | algorithm proposed in [9] | 109 | 436 | 1.31 ×10 | 3.51 | 35.0257 | 27.1737 | 0.9232 | 0.8424 |

random algorithm | 108 | 432 | — | 0.17 | 33.2654 | 27.5243 | 0.9298 | 0.8818 | |

proposed algorithm with (12) | 108 | 273 | 4.17 ×10 | 0.11 | 38.2353 | 33.7145 | 0.9574 | 0.9332 | |

proposed algorithm with (14) | 108 | 273 | 0.23 | 0.11 | 37.9307 | 34.5197 | 0.9588 | 0.9371 | |

Boat | algorithm proposed in [9] | 117 | 468 | 1.19 ×10 | 3.53 | 33.9007 | 30.2299 | 0.8798 | 0.8216 |

random algorithm | 117 | 468 | — | 0.17 | 32.5359 | 28.6484 | 0.8766 | 0.8173 | |

proposed algorithm with (12) | 117 | 308 | 4.02 ×10 | 0.12 | 34.4766 | 32.3504 | 0.8962 | 0.8774 | |

proposed algorithm with (14) | 117 | 308 | 0.25 | 0.12 | 33.9637 | 32.1755 | 0.8946 | 0.8802 | |

Lenna | algorithm proposed in [9] | 117 | 468 | 1.20 ×10 | 3.57 | 33.6590 | 32.3297 | 0.8665 | 0.8335 |

random algorithm | 117 | 468 | — | 0.18 | 31.1721 | 30.8207 | 0.8546 | 0.8344 | |

proposed algorithm with (12) | 117 | 310 | 4.02 ×10 | 0.12 | 34.3554 | 33.1450 | 0.8860 | 0.8660 | |

proposed algorithm with (14) | 117 | 310 | 0.24 | 0.12 | 33.9673 | 32.8775 | 0.8846 | 0.8662 | |

Pepper | algorithm proposed in [9] | 113 | 452 | 1.25 ×10 | 3.50 | 31.8969 | 26.8608 | 0.8555 | 0.8260 |

random algorithm | 111 | 444 | — | 0.1716 | 29.9528 | 24.5312 | 0.8536 | 0.8209 | |

proposed algorithm with (12) | 111 | 285 | 3.90 ×10 | 0.12 | 33.2907 | 29.4019 | 0.8950 | 0.8809 | |

proposed algorithm with (14) | 111 | 284 | 0.24 | 0.12 | 32.9208 | 30.2535 | 0.8930 | 0.8852 | |

Airplane | algorithm proposed in [9] | 119 | 476 | 1.21 ×10 | 3.55 | 31.0753 | 34.4833 | 0.8166 | 0.9014 |

random algorithm | 117 | 468 | — | 0.17 | 29.8990 | 32.9087 | 0.8169 | 0.9035 | |

proposed algorithm with (12) | 117 | 307 | 3.90 ×10 | 0.12 | 33.7608 | 36.1246 | 0.8879 | 0.9154 | |

proposed algorithm with (14) | 117 | 307 | 0.27 | 0.12 | 33.8210 | 35.7528 | 0.8883 | 0.9155 | |

Couple | algorithm proposed in [9] | 118 | 472 | 1.26 ×10 | 3.47 | 35.9439 | 33.0840 | 0.9001 | 0.8795 |

random algorithm | 117 | 468 | — | 0.17 | 34.2172 | 31.2331 | 0.8992 | 0.8804 | |

proposed algorithm with (12) | 117 | 311 | 3.90 ×10 | 0.14 | 37.4989 | 35.2208 | 0.9210 | 0.9120 | |

proposed algorithm with (14) | 117 | 309 | 0.25 | 0.13 | 37.1587 | 35.2879 | 0.9188 | 0.9115 | |

Earth | algorithm proposed in [9] | 115 | 460 | 1.19 ×10 | 3.56 | 36.1886 | 33.8077 | 0.9133 | 0.8290 |

random algorithm | 114 | 456 | — | 0.17 | 34.1523 | 32.4901 | 0.9068 | 0.8277 | |

proposed algorithm with (12) | 114 | 288 | 3.90 ×10 | 0.12 | 37.6137 | 34.4186 | 0.9386 | 0.8598 | |

proposed algorithm with (14) | 114 | 289 | 0.26 | 0.12 | 37.9318 | 34.1140 | 0.9390 | 0.8602 |

Thirdly, the proposed algorithm is compared with the algorithm proposed in [12] using the test images as shown in Fig. 5. In [12], Lee et al. have proposed the colorization-based image coding algorithm, which achieves a high coding performance. In order to see the performance of colorization-based coding method, these algorithms use uncompressed luminance images. Since an arbitrary number of RPs can be used in Lee’s algorithm, we examine the algorithm in two ways by using (1) the same number of RPs as those of the proposed algorithm and (2) RPs whose volume of information to be represented is equal to that of the proposed algorithm.

Numerical results of comparison with Lee’s algorithm

Image | Algorithm | Number | Volume of | Time [sec] | PSNR [dB] | SSIM | |||
---|---|---|---|---|---|---|---|---|---|

of RPs | information | ||||||||

[pixels] | [byte] | encoding | decoding | Cb | Cr | Cb | Cr | ||

Milkdrop | proposed algorithm | 274 | 699 | 0.72 | 0.22 | 37.8741 | 35.8627 | 0.9635 | 0.9377 |

algorithm proposed in [12] | 274 | 959 | 81.78 | 31.01 | 37.1474 | 32.4973 | 0.9432 | 0.9049 | |

algorithm proposed in [12] | 200 | 700 | 53.41 | 29.27 | 36.8288 | 32.4900 | 0.9397 | 0.8978 | |

Boat | proposed algorithm | 274 | 698 | 0.71 | 0.22 | 34.7794 | 32.0441 | 0.8928 | 0.8734 |

algorithm proposed in [12] | 274 | 959 | 79.87 | 28.77 | 34.5627 | 32.8473 | 0.8815 | 0.8744 | |

algorithm proposed in [12] | 200 | 700 | 52.57 | 28.66 | 34.1386 | 32.5993 | 0.8753 | 0.8707 | |

Lenna | proposed algorithm | 274 | 701 | 0.71 | 0.21 | 36.1906 | 35.3057 | 0.9231 | 0.9018 |

algorithm proposed in [12] | 274 | 959 | 80.88 | 29.55 | 36.2129 | 34.5499 | 0.9086 | 0.8925 | |

algorithm proposed in [12] | 200 | 700 | 53.39 | 29.51 | 35.9824 | 34.5354 | 0.9059 | 0.8903 | |

Pepper | proposed algorithm | 274 | 697 | 0.71 | 0.20 | 29.4435 | 30.0789 | 0.8697 | 0.8762 |

algorithm proposed in [12] | 274 | 959 | 88.56 | 38.37 | 32.6670 | 29.7992 | 0.8702 | 0.8510 | |

algorithm proposed in [12] | 199 | 697 | 59.50 | 37.39 | 30.2358 | 28.8093 | 0.8458 | 0.8256 | |

Airplane | proposed algorithm | 274 | 702 | 0.71 | 0.22 | 33.0319 | 31.6955 | 0.9117 | 0.9149 |

algorithm proposed in [12] | 274 | 959 | 79.14 | 29.15 | 34.9049 | 33.9563 | 0.9124 | 0.9185 | |

algorithm proposed in [12] | 200 | 700 | 52.83 | 29.07 | 34.7446 | 33.8131 | 0.9100 | 0.9165 | |

Couple | proposed algorithm | 274 | 703 | 0.70 | 0.20 | 38.7016 | 37.0570 | 0.9216 | 0.9172 |

algorithm proposed in [12] | 274 | 959 | 84.51 | 34.82 | 38.4629 | 35.9642 | 0.9143 | 0.9013 | |

algorithm proposed in [12] | 201 | 704 | 56.13 | 32.76 | 38.3394 | 35.8256 | 0.9134 | 0.9012 | |

Earth | proposed algorithm | 274 | 695 | 0.71 | 0.21 | 39.4627 | 36.1058 | 0.9500 | 0.9049 |

algorithm proposed in [12] | 274 | 959 | 85.51 | 35.81 | 40.2759 | 37.7125 | 0.9536 | 0.9166 | |

algorithm proposed in [12] | 199 | 697 | 58.08 | 35.84 | 40.0729 | 37.6271 | 0.9525 | 0.9156 |

Finally, we compare the proposed algorithm with JPEG and JPEG2000 coding method. Luminance images are compressed using JPEG/JPEG2000 coding where the quality parameter (QP) is selected such that the volume of information is equal to about 4000 bytes.

Numerical results of comparison with JPEG and JPEG2000

Image | Algorithm | Volume of information [byte] | PSNR[dB] | SSIM | QP for JPEG/JPEG2000 |
---|---|---|---|---|---|

Milkdrop | JPEG | 4206 | 28.6925 | 0.7988 | 20 |

JPEG2000 | 4094 | 32.4263 | 0.8719 | 46 | |

proposed algorithm with JPEG | 4138 | 30.7470 | 0.8540 | 22 | |

proposed algorithm with JPEG2000 | 4036 | 32.6914 | 0.8891 | 20 | |

Boat | JPEG | 4139 | 24.7191 | 0.7488 | 10 |

JPEG2000 | 4068 | 26.9982 | 0.8219 | 46 | |

proposed algorithm with JPEG | 4125 | 25.8622 | 0.7849 | 10 | |

proposed algorithm with JPEG2000 | 4039 | 27.2043 | 0.8374 | 20 | |

Lenna | JPEG | 4136 | 26.794 | 0.7864 | 13 |

JPEG2000 | 4070 | 29.2353 | 0.8398 | 46 | |

proposed algorithm with JPEG | 4159 | 27.6452 | 0.8116 | 13 | |

proposed algorithm with JPEG2000 | 4061 | 29.2051 | 0.8568 | 20 | |

Pepper | JPEG | 4137 | 24.0496 | 0.7078 | 11 |

JPEG2000 | 4071 | 27.1189 | 0.7969 | 46 | |

proposed algorithm with JPEG | 4147 | 24.8098 | 0.7632 | 12 | |

proposed algorithm with JPEG2000 | 4051 | 25.9898 | 0.8127 | 20 | |

Airplane | JPEG | 4176 | 25.1615 | 0.7894 | 12 |

JPEG2000 | 4062 | 28.0905 | 0.8443 | 46 | |

proposed algorithm with JPEG | 4192 | 25.6759 | 0.8108 | 12 | |

proposed algorithm with JPEG2000 | 4044 | 26.7533 | 0.8471 | 20 | |

Couple | JPEG | 4158 | 30.5207 | 0.8112 | 21 |

JPEG2000 | 4018 | 32.0973 | 0.8373 | 46 | |

proposed algorithm with JPEG | 4184 | 31.0459 | 0.8314 | 21 | |

proposed algorithm with JPEG2000 | 4064 | 32.1436 | 0.8527 | 20 | |

Earth | JPEG | 4277 | 27.5088 | 0.7952 | 13 |

JPEG2000 | 4164 | 30.0662 | 0.8527 | 45 | |

proposed algorithm with JPEG | 4146 | 28.4777 | 0.8088 | 12 | |

proposed algorithm with JPEG2000 | 4017 | 30.2628 | 0.8604 | 20 |

## 4 Conclusions

This paper proposes a representative pixel (RP) extraction and colorization algorithm for the colorization-based digital image coding. In order to achieve low computing cost and high image coding performance, multiple reduced images are generated by colorization error minimizing method, and the RPs are extracted from these reduced images. Numerical examples show that the proposed algorithm can extract RPs and recover a color image fast and effectively comparing with other colorization-based algorithms, and numerical results show that the proposed algorithm achieves higher coding performance than JPEG/JPEG2000.

## Declarations

### Acknowledgements

We are grateful to Sukho Lee and his co-authors for providing the source code of the colorization-based coding proposed in [12].

This work was supported by JSPS KAKENHI Grant Numbers 26 ·6546.

**Open Access** This article is distributed under the terms of the Creative Commons Attribution 4.0 International License(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## Authors’ Affiliations

## References

- A Levin, D Lischinski, Y Weiss, Colorization using optimization. ACM Trans. Graph.
**23**(3), 689–694 (2004).View ArticleGoogle Scholar - L Yatziv, G Sapiro, Fast image and video colorization using chrominance blending. IEEE Trans. Image Process.
**15**(5), 1120–1129 (2006).View ArticleGoogle Scholar - J Pang, OC Au, K Tang, Y Guo, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP). Image colorization using sparse representation, (2013), pp. 1578–1582.Google Scholar
- K Uruma, K Konishi, T Takahashi, T Furukawa, in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process. (ICASSP). Image colorization algorithm using series approximated sparse function, (2013), pp. 1215–1219.Google Scholar
- T Miyata, Y Komiyama, Y Inazumi, Y Sakai, in Proc. Picture Coding Symp. Novel inverse colorization for image compression, (2009), pp. 1–4.Google Scholar
- H Noda, N Takao, M Niimi, in Proc. IEEE Int. Conf. Image Process. (ICIP). Colorization in YCbCr space and its application to improve quality of JPEG Color Images, (2007), pp. 385–388.Google Scholar
- L Cheng, S Vishwanathan, in Proc. Int. Conf. Mach. Learn. (ICML). Learning to compress images and videos, (2007), pp. 161–168.Google Scholar
- X He, M Ji, H Bao, in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR). A unified active and semi-supervised learning framework for image Compression, (2009), pp. 65–72.Google Scholar
- S Ono, T Miyata, Y Sakai, in Proc. Picture Coding Symp. Colorization-based coding by focusing on characteristics of colorization bases, (2010), pp. 230–233.Google Scholar
- Y Inoue, T Miyata, Y Sakai, Colorization based image coding by using local correlation between luminance and chrominance. IEICE Trans. Inf. Syst.
**95**(1), 247–255 (2012).View ArticleGoogle Scholar - T Ueno, T Yoshida, M Ikehara, Color image coding based on the colorization,Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), (2012), pp. 1–4.Google Scholar
- S Lee, S Park, P Oh, M Kang, Colorization-based compression using optimization. IEEE Trans. Image Proc.
**22**(7), 2627–2636 (2013).View ArticleGoogle Scholar - EJ Candes, J Romberg, T Tao, Stable signal recovery for incomplete and inaccurate measurements. Commun. Pure Appl. Math.
**59**(3), 1207–1223 (2013).MathSciNetGoogle Scholar - DL Donoho, Compressed sensing. IEEE Trans. Inf. Theory.
**52:**, 1289–1306 (2006).View ArticleMathSciNetMATHGoogle Scholar - C Rusu, SA Tsaftaris, in Proc. Int. Symp. Image and Signal Processing and Analysis (ISPA). Estimation of scribble placement for painting colorization, (2013), pp. 564–569.Google Scholar
- Z Wang, AC Bovik, HR Sheikh, EP Simoncelli, Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Proc.
**13**(4), 600–612 (2004).View ArticleGoogle Scholar