Demosaicking with two-dimensional continuous 3 × 3 order hidden Markov model

Abstract

Since most digital cameras use color filter arrays to sample red, green, and blue colors by a specific pattern, only one color sample would be taken at every pixel location. The process named demosaicking is exploited to recover the full-color image from the incomplete color samples. The paper presents a novel demosaicking method based on two-dimensional continuous 3 × 3 order HMM (2D 3 × 3 CHMM), which incorporates the statistics of high resolution images into the CFA interpolation process. The proposed new method adopts an approach of MAP sequence estimation and exploits high-order statistical dependency between missing pixels. Experiment results demonstrate that the proposed method outperforms several existing or state-of-the-art demosaicking techniques in terms of both objective and subjective evaluations.

1 Introduction

Digital imaging devices find a broad range of applications, particularly in digital cameras, surveillance devices, and mobile phones [1,2,3,4,5]. Every pixel must consist of three independent primary color components: red, green, and blue, when a digital camera obtains a color image [1, 6]. In order to reduce costs, most digital cameras use only one monochromatic image sensor with a color filter array (CFA) which makes each pixel to obtain only one color component. The most common CFA is the Bayer pattern that includes twice as much green information compared to blue (or red) information. To reconstruct the full-color image from such CFA samples, the missing color components need to be estimated by the color demosaicking (CDM) process.

The quality of the reconstructed image relies on not only the image contents but also the demosaicking methods [7, 8]. The earliest proposed approaches are the interpolation-based methods such as nearest-neighbor replication, bilinear, and bicubic interpolation. Although the implementation of these methods is simple and fast, they yield severe artifacts like zippering or false color information, especially along highly textural regions or the edges. The recently developed methods include the directional linear minimum mean square-error estimation (DLMMSE)-based CDM method [9], the variance of color difference (VCD)-based CDM method [10], the adaptive homogeneity CDM (AHD) method [11], and the successive approximation (SA)-based CDM method [12]. These demosaicking methods make the assumption that the local correlations are high. Such an hypothesis may be well effective for Kodak images, while the assumption may fail for images such as those in the McMaster dataset [7, 8]. The nonlocal similarity method (NS) and the nonlocal similarity fusion method (VCD-NS) presented recently exploit the image nonlocal redundancy to improve the CFA interpolation result [13, 14]. The demosaicking methods which use nonlocal similarity are called the nonlocal CDM methods. Similarly, we call the demosaicking methods that only use local correlations the local CDM methods.

Most block-based image classification algorithms decide the class of a block by examining only the feature vector of this block [15]. Since context information between blocks is ignored, the performance of such algorithms is limited [16]. In order to improve image classification by context, J. Li et al. [15] proposed a two-dimensional hidden Markov model (2D-HMM), in which the state transition probability for each block relies on the states of nearest neighboring blocks from vertical and horizontal directions. The 2D-HMM has wide applications in the field of pattern recognition and image processing [17,18,19,20]. However, the context information which a block depends on may arise from other directions, such as diagonal direction [21, 22]. In order to overcome the defect of the assumption of the 2D-HMM, the paper [22] presents the 2D 3 × 3 CHMM, where the probability density of the observation depends on not only current state but also immediate vertical and horizontal states, and where the state transition probability depends on not only immediate vertical and horizontal states but also immediate diagonal state.

In this paper, we present a novel demosaicking method based on 2D 3 × 3 CHMM, which incorporates the statistics of HR images into the interpolation process. In our method, the problem of CFA interpolation is converted into the MAP sequence estimation, which exploits high-order statistical dependency between pixels. Extensive experiments show that our proposed method outperforms the demosaicking algorithm based on two-dimensional continuous HMM. This is because that 2D continuous high-order HMM can better model spatial correlation in image data compared to the existing 2D continuous HMM.

The remainder of the paper is organized as follows: Section 2 provides the definition of the 2D continuous 3 × 3 order HMM. Section 3 gives some algorithms to solve the basic problems for the 2D continuous 3 × 3 order HMM. Section 4 describes in detail the proposed demosaicking algorithm. Sections 5 and 6 present the experimental results, and Section 7 concludes the paper.

2 Definition of the 2D continuous 3 × 3 order HMM

Let I_{M, N} = {(i, j)| 1 ≤ i ≤ M, 1 ≤ j ≤ N} be a M × N integer grid. If i^' < i, or i^' = i and j^' < j for (i^', j^'), (i, j) ∈ I_{M, N}, we call that (i^', j^') is before (i, j) and write (i^', j^') < (i, j).

Put φ_{i, j} = {(i^', j^')| (i^', j^') < (i, j)}.

$$ {U}_{i,j}=\left\{\begin{array}{c}\left\{\left(i-1,j\right),\Big(i-1,j-1\Big),\Big(i,j-1\Big)\right\},\kern0.5em i>1,j>1,\\ {}\left\{\left(i-1,j\right)\right\},\kern0.5em \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}\kern1.00em i>1,j=1,\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}& & i=1,j>1.\end{array}\end{array}\right. $$

$$ {V}_{i,j}=\left\{\begin{array}{c}\left\{\left(i-1,j\right),\Big(i,j-1\Big)\right\},\kern0.5em i>1,j>1,\\ {}\begin{array}{cccc}\left\{\left(i-1,j\right)\right\},& & & i>1,j=1,\end{array}\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& & & i=1,j>1.\end{array}\end{array}\right. $$

We denote the state and the observation at (i, j) by q_{i, j} and o_{i, j}, respectively. Then, we give the following two assumptions (1) and (2).

$$ P\left({q}_{i,j}|{q}_{k,l},{\mathbf{o}}_{k,l},\left(k,l\right)\in {\varphi}_{i,j}\right)=P\left({q}_{i,j}|{q}_{k,l},\left(k,l\right)\in {U}_{i,j}\right) $$

(1)

$$ P\left({\mathbf{o}}_{i,j}|{q}_{i,j},{q}_{k,l},{\mathbf{o}}_{k,l},\left(k,l\right)\in {\varphi}_{i,j}\right)=P\left({\mathbf{o}}_{i,j}|{q}_{i,j},{q}_{k,l},\left(k,l\right)\in {V}_{i,j}\right) $$

(2)

Let O = {o_{i, j}| (i, j) ∈ I_{M, N}} be an observation matrix and Q = {q_{i, j}| (i, j) ∈ I_{M, N}} be a state matrix. Suppose the observation o_{i, j} at a specific state follows multivariate normal distribution. Then, (Q, O) is 2D 3 × 3 CHMM when and only when (1) and (2) hold.

For convenience, the parameters of the 2D 3 × 3 CHMM (Q, O) are given as follows:

• (1) L represents the total state number.

• (2) The probability density of o_{i, j}(2 ≤ i ≤ M, 2 ≤ j ≤ N)

$$ {\displaystyle \begin{array}{c}\mathbf{B}=\left\{{b}_{k,m,n}\left({\mathbf{o}}_{i,j}\right)|1\le k,m,n\le L\right\}\\ {}=\left\{P\left({\mathbf{o}}_{i,j}|{q}_{i,j}=n,{q}_{i-1,j}=k,{q}_{i,j-1}=m,1\le k,m,n\le L\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_{k,m,n}\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{i,j}-{\boldsymbol{\upmu}}_{k,m,n}\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_{k,m,n}\right)}^{-1}\left({\mathbf{o}}_{i,j}-{\boldsymbol{\upmu}}_{k,m,n}\right)}|1\le k,m,n\le L\right\}\end{array}} $$

where μ_{k, m, n} and Σ_{k, m, n} are the mean vectors and the covariance matrix, respectively, and where p is the dimension of o_{i, j}.

• (3) The probability density of the observation in the first row

$$ {\displaystyle \begin{array}{c}{\mathbf{B}}^{\mathrm{h}}=\left\{{b}_{m,n}^{\mathrm{h}}\left({\mathbf{o}}_{1,j}\right)|1\le m,n\le L,2\le j\le N\right\}\\ {}=\left\{P\left({\mathbf{o}}_{1,j}|{q}_{1,j}=n,{q}_{1,j-1}=m,1\le m,n\le L,2\le j\le N\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_{m,n}^h\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{1,j}-{\boldsymbol{\upmu}}_{m,n}^h\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_{m,n}^h\right)}^{-1}\left({\mathbf{o}}_{1,j}-{\boldsymbol{\upmu}}_{m,n}^h\right)}|1\le m,n\le L,2\le j\le N\Big)\right\}\end{array}} $$

where ${\boldsymbol{\upmu}}_{m,n}^h$ and ${\boldsymbol{\Sigma}}_{m,n}^h$ are the mean vectors and the covariance matrix, respectively, and where p is the dimension of o_{1, j}.

• (4) The probability density of the observation in the first column

$$ {\displaystyle \begin{array}{c}{\mathbf{B}}^{\mathrm{v}}=\left\{{b}_{m,n}^{\mathrm{v}}\left({\mathbf{o}}_{i,1}\right)|1\le m,n\le L,2\le i\le M\right\}\\ {}=\left\{P\left({\mathbf{o}}_{i,1}|{q}_{i,1}=n,{q}_{i-1,1}=m,1\le m,n\le L,2\le i\le M\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_{m,n}^v\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{i,1}-{\boldsymbol{\upmu}}_{m,n}^v\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_{m,n}^v\right)}^{-1}\left({\mathbf{o}}_{i,1}-{\boldsymbol{\upmu}}_{m,n}^v\right)}|1\le m,n\le L,2\le i\le M\Big)\right\}\end{array}} $$

where ${\boldsymbol{\upmu}}_{m,n}^v$ and ${\boldsymbol{\Sigma}}_{m,n}^v$ are the mean vectors and the covariance matrix, respectively, and where p is the dimension of o_{i, 1}.

• (5) The probability density of the observation in the initial position

$$ {\displaystyle \begin{array}{c}{\mathbf{B}}^0=\left\{{b}_n^0\left({\mathbf{o}}_{1,1}\right)|1\le n\le L\right\}\\ {}=\left\{P\left({\mathbf{o}}_{1,1}|{q}_{1,1}=n,1\le n\le L\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_n^0\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{1,1}-{\boldsymbol{\upmu}}_n^0\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_n^0\right)}^{-1}\left({\mathbf{o}}_{1,1}-{\boldsymbol{\upmu}}_n^0\right)}|1\le n\le L\Big)\right\}\end{array}} $$

where ${\boldsymbol{\upmu}}_n^0$ and ${\boldsymbol{\Sigma}}_n^0$ are the mean vectors and the covariance matrix, respectively, and where p is the dimension of o_{1, 1}.

• (6) The state transition probability distribution A = {a_{k, l, m, n}| 1 ≤ k, l, m, n ≤ L},

where a_{k, l, m, n} = P(q_{i, j} = n| q_{i − 1, j} = k, q_{i − 1, j − 1} = l, q_{i, j − 1} = m),1 ≤ k, l, m, n ≤ L.

• (7) The probability of the state transfer about the first row

${\mathbf{A}}^{\mathrm{h}}=\left\{{a}_{m,n}^{\mathrm{h}}|1\le m,n\le L\right\}$, where ${a}_{m,n}^{\mathrm{h}}=P\left({q}_{1,j}=n|{q}_{1,j-1}=m\right)$, 1 ≤ m, n ≤ L.

• (8) The probability of the state transfer about the first column.

${\mathbf{A}}^{\mathrm{v}}=\left\{{a}_{m,n}^{\mathrm{v}}|1\le m,n\le L\right\}$, where ${a}_{m,n}^{\mathrm{v}}=P\left({q}_{i,1}=n|{q}_{i-1,1}=m\right)$, 1 ≤ m, n ≤ L.

• (9) The initial state probability distribution π = {π_n| 1 ≤ n ≤ L} where π_n = P(q_{1, 1} = n),1 ≤ n ≤ L.

Put τ = (π, A^h, A^v, A, B⁰, B^h, B^v, B). Then, τ may be used to represent the set of all model parameters.

Set ${T}_n=\left\{{q}_{1,n},{q}_{2,n},\cdots, {q}_{M,n}\right\}$, O_n = {o_{1, n}, o_{2, n}, ⋯, o_{M, n}}. Assume that Q and O are state matrix and observation matrix of τ, respectively. If the nth column of Q and O can be indicated by T_n and O_n respectively, 1 ≤ n ≤ N, then we have

$$ P\left({T}_n|{T}_{n-1},\cdots, {T}_1\right)=P\left({T}_n|{T}_{n-1}\right) $$

(3)

$$ P\left({O}_n|{T}_n,{T}_{n-1},\cdots, {T}_1\right)=P\left({O}_n|{T}_n,{T}_{n-1}\right) $$

(4)

By (3) and (4), we can deduce that τ equals a one-dimensional 1 × 2 order HMM $\lambda =\left({A}_{\tau },{B}_{\tau}^{\hbox{'}},{B}_{\tau },{\pi}_{\tau}\right)$, that is described as follows.

• (1) L^M represents the total state number. I is used to represent the individual state, where

$$ \Omega =\left\{\mathbf{I}|\mathbf{I}={\left({i}_1,{i}_2,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},{i}_k\in \Big\{1,2,\cdots, L\Big\},1\le k\le M\right\},\mathbf{I}\in \Omega . $$

• (2) B_τ = {b_IJ(W)} represents the observation probability distribution, where

$$ {\displaystyle \begin{array}{l}{b}_{\mathbf{IJ}}\left(\mathbf{W}\right)=P\left(\mathbf{W}={\left({\mathbf{o}}_{1,n},{\mathbf{o}}_{2,n},\cdots, {\mathbf{o}}_{M,n}\right)}^{\hbox{'}}|{T}_n=\mathbf{J}\right)={b}_{i_1{j}_1}^h\left({\mathbf{o}}_{1,n}\right)\cdot {b}_{j_1{i}_2{j}_2}\left({\mathbf{o}}_{2,n}\right)\cdots {b}_{j_{M\hbox{-} 1}{i}_M{j}_M}\left({\mathbf{o}}_{M,n}\right)\\ {}n>1,\mathbf{J}\in \Omega, \mathbf{W}\in \mathrm{E}.\end{array}} $$

• (3) The initial observation probability density is ${B}_{\tau}^{\hbox{'}}=\left\{{b}_{\mathbf{J}}\left(\mathbf{W}\right)\right\}$, in which

$$ {b}_{\mathbf{J}}\left(\mathbf{W}\right)=P\left(\mathbf{W}={\left({\mathbf{o}}_{1,1},{\mathbf{o}}_{2,1},\cdots, {\mathbf{o}}_{M,1}\right)}^{\hbox{'}}|{T}_1=\mathbf{J}\right)={b}_{j_1}^0\left({\mathbf{o}}_{1,1}\right){b}_{j_1{j}_2}^v\left({\mathbf{o}}_{2,1}\right)\cdots {b}_{j_{M\hbox{-} 1}{j}_M}^v\left({\mathbf{o}}_{M,1}\right),\mathbf{J}\in \Omega, \mathbf{W}\in E. $$

• (4) The initial state probability is ∏_τ = {π_J}, where

$$ {\pi}_{\mathbf{J}}=P\left({T}_1=\mathbf{J}\right)={\pi}_{j_1}\cdot {a}_{j_1{j}_2}^v\cdot {a}_{j_2{j}_3}^v\cdots {a}_{j_{M\hbox{-} 1}{j}_M}^v,\mathbf{J}\in \Omega . $$

• (5) The probability of the state transfer is A_τ = {a_{I, J}}, in which

$$ {a}_{\mathbf{I},\mathbf{J}}=P\left({T}_{n+1}=\mathbf{J}|{T}_n=\mathbf{I}\right)={a}_{i_1,{j}_1}^h\cdot {a}_{j_1,{i}_1,{i}_2,{j}_2}\times {a}_{j_2,{i}_2,{i}_3,{j}_3}\cdots {a}_{j_{M\hbox{-} 1},{i}_{M\hbox{-} 1},{i}_M,{j}_M} $$

$$ \mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},\mathbf{J}={\left({j}_1,\cdots, {j}_k,\cdots, {j}_M\right)}^{\hbox{'}},\mathbf{I},\mathbf{J}\in \Omega . $$

3 Basic problem solving for the proposed model

In order to put the proposed model into some practical applications, we need to solve these basic problems that are shown below.

• Problem 1: Given τ and O, how do we compute P(O| τ)?

• Problem 2: Given τ and O, how do we get the optimal state Q = {q_{i, j}| (i, j) ∈ I_{M, N}}?

• Problem 3: Given the observation matrix O, how do we adjust τ to maximize P(O| τ)?

3.1 Solution for problem 1

α_n(I) is called the forward variable, where α_n(I) = P(W₁, W₂, ⋯, W_n, T_n = I| λ) andW_n = (o_{1, n}, ⋯, o_{M, n})^', n = 1, ⋯, N. Then, we can calculate the forward variable as follows:

• (1) Initialization

$$ {\displaystyle \begin{array}{l}{\alpha}_1\left(\mathbf{I}\right)={\pi}_{\mathbf{I}}{b}_{\mathbf{I}}\left({\mathbf{W}}_1\right)={\pi}_{i_1}\cdot {b}_{i_1}^0\left({o}_{11}\right)\cdot {a}_{i_1{i}_2}^v\cdot {b}_{i_1{i}_2}^v\left({o}_{21}\right)\cdot {a}_{i_2{i}_3}^v\cdot {b}_{i_2{i}_3}^v\left({o}_{31}\right)\cdots {a}_{i_{M-1}{i}_M}^v\cdot {b}_{i_{M-1}{i}_M}^v\left({o}_{M1}\right),\\ {}\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}}\in \Omega .\end{array}} $$

• (2) Recursion.

${\displaystyle \begin{array}{c}{\alpha}_{n+1}\left(\mathbf{J}\right)=\sum \limits_{\mathbf{I}\in \Omega}\left({\alpha}_n\left(\mathbf{I}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{n+1}\right)\right)\\ {}=\sum \limits_{\mathbf{I}\in \Omega}\left({\alpha}_n\left(\mathbf{I}\right)\cdot \left({a}_{i_1,{j}_1}^{\mathrm{h}}\cdot {a}_{j_1,{i}_1,{i}_2,{j}_2}\cdots {a}_{j_{M-1},{i}_{M-1},{i}_M,{j}_M}\right)\cdot \left({b}_{i_1{j}_1}^h\left({o}_{1,n+1}\right)\cdot {b}_{j_1{i}_2{j}_2}\left({o}_{2,n+1}\right)\cdots {b}_{j_{\mathrm{M}-1}{i}_{\mathrm{M}}{j}_{\mathrm{M}}}\left({o}_{M,n+1}\right)\right)\right)\end{array}}\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},\mathbf{J}={\left({j}_1,\cdots, {j}_k,\cdots, {j}_M\right)}^{\hbox{'}},\mathbf{I},\mathbf{J}\in \Omega$

• (3) Termination

$$ \mathrm{P}\left(\mathbf{O}|\tau \right)=\mathrm{P}\left({\mathbf{W}}_1,\cdots, {\mathbf{W}}_N|\lambda \right)=\sum \limits_{\mathbf{I}\in \Omega}{\alpha}_N\left(\mathbf{I}\right) $$

(5)

3.2 Solution for problem 2

There is one way to solve problem 2 by the standard as follows:

Standard: Given τ and O, find the optimum state ${q}_{i,j}^{\ast }$ such that

$$ {q}_{i,j}^{\ast }=\arg \underset{1\le m\le \mathrm{L}}{\max }P\left({q}_{i,j}=m|\mathbf{O},\tau \right)=\arg \underset{1\le m\le \mathrm{L}}{\max }P\left({q}_{i,j}=m|\mathbf{O},\lambda \right) $$

(6)

We call β_n(I) backward variable, where β_n(I) = P(W_n + 1, W_n + 2, ⋯, W_N, T_n = I| λ) and

W_i = (o_{1, i}, ⋯, o_{M, i})^', i = n + 1, ⋯, N. Then, we can calculate the backward variable as follows:

• (1) Initialization

$$ {\beta}_{\mathrm{N}}\left(\mathbf{I}\right)=1,\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}}\in \Omega . $$

• (2) Recursion.

$$ {\displaystyle \begin{array}{c}{\beta}_n\left(\mathbf{I}\right)=\sum \limits_{\mathbf{J}\in \Omega}\left({\beta}_{n+1}\left(\mathbf{J}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{n+1}\right)\right)\\ {}=\sum \limits_{\mathbf{J}\in \Omega}\left({\beta}_{n+1}\left(\mathbf{J}\right)\cdot \left({a}_{i_1,{j}_1}^h\cdot {a}_{j_1,{i}_1,{i}_2,{j}_2}\cdots {a}_{j_{M-1},{i}_{M-1},{i}_M,{j}_M}\right)\cdot \left({b}_{i_1{j}_1}^h\left({o}_{1,n+1}\right)\cdot {b}_{j_1{i}_2{j}_2}\left({o}_{2,n+1}\right)\cdots {b}_{j_{\mathrm{M}-1}{i}_{\mathrm{M}}{j}_{\mathrm{M}}}\left({o}_{M,n+1}\right)\right)\right)\end{array}}\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},\mathbf{J}={\left({j}_1,\cdots, {j}_k,\cdots, {j}_M\right)}^{\hbox{'}},\mathbf{I},\mathbf{J}\in \Omega $$

Define γ_n(I) as the posteriori probability distribution of the nth state column q_n = I given λ and O, i.e., γ_n(I) = P(q_n = I| O, λ), in which ${\gamma}_n\left(\mathbf{I}\right)=\frac{P\left(\mathbf{O},{\mathbf{q}}_n=\mathbf{I}|\lambda \right)}{P\left(\mathbf{O}|\lambda \right)}=\frac{\alpha_n\left(\mathbf{I}\right)\cdot {\beta}_n\left(\mathbf{I}\right)}{\sum \limits_{\mathbf{I}\in \Omega}{\alpha}_n\left(\mathbf{I}\right){\beta}_n\left(\mathbf{I}\right)},\mathbf{I}\in \Omega$.

Given τ and O, η_{i, j}(k) is defined as the posteriori probability distribution of q_{i, j} = k, i.e., η_{i, j}(k) = P(q_{i, j} = k| O, τ), where q_{i, j} is the state at (i, j).

Then, we have ${\eta}_{i,j}(k)=\sum \limits_{\mathbf{I}\in \Omega, \mathbf{I}(i)=k}P\left({T}_j=\mathbf{I}|O,\lambda \right)=\sum \limits_{\mathbf{I}\in \Omega, \mathbf{I}(i)=k}{\gamma}_j\left(\mathbf{I}\right)$, in which I(i) is the ith component of I.

Thus, the optimal state ${q}_{i,j}^{\ast }$ can be found, where ${q}_{i,j}^{\ast }=\underset{1\le k\le L}{\arg \max}\left[{\eta}_{i,j}(k)\right]$, 1 ≤ i ≤ M, 1 ≤ j ≤ N.

3.3 Solution for problem 3

Define ξ_t(I, J) as the probability distribution being in state I at column t, and in state J at column t + 1, givenλ and O, i.e.,

$$ {\xi}_t\left(\mathbf{I},\mathbf{J}\right)=P\left({T}_t=\mathbf{I},{T}_{t+1}=\mathbf{J}|\lambda, \mathbf{O}\right),\mathbf{I},\mathbf{J}\in \Omega . $$

(7)

By α_n(I) and β_n(I), we have

$$ {\xi}_t\left(\mathbf{I},\mathbf{J}\right)=\frac{\alpha_t\left(\mathbf{I}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{t+1}\right)\cdot {\beta}_{t+1}\left(\mathbf{J}\right)}{\sum \limits_{\mathbf{I}\in \Omega}\sum \limits_{\mathbf{J}\in \Omega}{\alpha}_t\left(\mathbf{I}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{t+1}\right)\cdot {\beta}_{t+1}\left(\mathbf{J}\right)} $$

(8)

where α_n(I) is the forward variable and β_n(I) is the backward variable.

We denote the reestimation model and the initial model by $\overline{\tau}=\left(\overline{\boldsymbol{\uppi}},{\overline{\mathbf{A}}}^h,{\overline{\mathbf{A}}}^v,\overline{\mathbf{A}},{\overline{\mathbf{B}}}^0,{\overline{\mathbf{B}}}^h,{\overline{\mathbf{B}}}^v,\overline{\mathbf{B}}\right)$ and τ = (π, A^h, A^v, A, B⁰, B^h, B^v, B), respectively.

To solve problem 3, the reestimation formulae are given as follows:

$$ \overline{{\boldsymbol{\upmu}}_{m,n}^v}=\frac{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)\cdot {\mathbf{o}}_{i+1,1}}{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)} $$

(9)

$$ \overline{{\boldsymbol{\Sigma}}_{m,n}^v}=\frac{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)\cdot \left({\mathbf{o}}_{i+1,1}-\overline{{\boldsymbol{\upmu}}_{m,n}^v}\right)\cdot {\left({\mathbf{o}}_{i+1,1}-\overline{{\boldsymbol{\upmu}}_{m,n}^v}\right)}^{\hbox{'}}}{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)} $$

(10)

$$ \overline{{\boldsymbol{\upmu}}_{m,n}^h}=\frac{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)\cdot {\mathbf{o}}_{1,j+1}}{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)} $$

(11)

$$ \overline{{\boldsymbol{\Sigma}}_{m,n}^h}=\frac{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)\cdot \left({\mathbf{o}}_{1,j+1}-\overline{{\boldsymbol{\upmu}}_{m,n}^h}\right)\cdot {\left({\mathbf{o}}_{1,j+1}-\overline{{\boldsymbol{\upmu}}_{m,n}^h}\right)}^{\hbox{'}}}{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)} $$

(12)

$$ \overline{{\boldsymbol{\upmu}}_{k,m,n}}=\frac{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)\cdot {\mathbf{o}}_{i+1,t+1}}{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)} $$

(13)

$$ \overline{{\boldsymbol{\Sigma}}_{k,m,n}}=\frac{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)\cdot \left({\mathbf{o}}_{i+1,t+1}-\overline{{\boldsymbol{\upmu}}_{k,m,n}}\right)\cdot {\left({\mathbf{o}}_{i+1,t+1}-\overline{{\boldsymbol{\upmu}}_{k,m,n}}\right)}^{\hbox{'}}}{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)} $$

(14)

$$ \overline{{\boldsymbol{\upmu}}_k^0}={\boldsymbol{\upmu}}_k^0,\overline{{\boldsymbol{\Sigma}}_k^0}={\boldsymbol{\Sigma}}_k^0 $$

(15)

$$ {\overline{\pi}}_n=\frac{P\left({q}_{1,1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{k=1}^LP\left({q}_{1,1}=k,\mathbf{O}|\tau \right)} $$

(16)

$$ {{\overline{a}}^h}_{m,n}=\frac{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{n=1}^L\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)} $$

(17)

$$ {{\overline{a}}^v}_{m,n}=\frac{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{n=1}^L\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)} $$

(18)

$$ {\overline{a}}_{k,l,m,n}=\frac{\sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}P\left({q}_{i,j+1}=k,{q}_{i,j}=l,{q}_{i+1,j}=m,{q}_{i+1,j+1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{n=1}^L\sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}P\left({q}_{i,j+1}=k,{q}_{i,j}=l,{q}_{i+1,j}=m,{q}_{i+1,j+1}=n,\mathbf{O}|\tau \right)} $$

(19)

in which 1 ≤ k, l, m, n ≤ L.

By the formulas (9)–(19), we can get $\overline{\boldsymbol{\uptau}}$ such that $P\left(\mathbf{O}|\overline{\boldsymbol{\uptau}}\right)\ge P\left(\mathbf{O}|\boldsymbol{\uptau} \right)$.

4 Methods of CFA interpolation algorithm based on the 2D 3 × 3 CHMM

In the research, we find that the 2D continuous high-order HMM is suitable for the task of CFA interpolation, thanks to its ability to model two-dimensional spatial correlations.

The interpolation of a missing pixel could benefit from the property of the waveform of the true underlying HR image, in particular, whether the pixel is in an area with nonlocal similarity or in a region with local correlation. Since the knowledge of the waveform of the original HR image is unknown at the missing pixel location (i, j), we need to estimate the state s_{i, j} which is hidden by the down sampling process [23,24,25]. The estimation is based on the observable low resolution features that are exhibited by s_{i, j}.

We can construct the 2D continuous high-order HMM used to demosaicking, if the original 2D intensity function at the missing pixel x_{i, j} is classified into a set of states S = {s₁, s₂}, where s₁ and s₂ stand for the state with stronger local correlation and the state with stronger nonlocal similarity, respectively. For simplicity, the state set is denoted by {1, 2}. It should be pointed out that s_{i, j} ∈ {1, 2}.

Generally speaking, the observation vector ξ_{i, j} of s_{i, j} involves attributes of the LR image in a window W_{i, j} which is centered at (i, j); thus, we can measure the feature vector ξ_{i, j} by W_{i, j}. The contents below have detailed descriptions on the measure for ξ_{i, j}.

For convenience, we denote VCD, DLMMSE, NS, and VCD-NS demosaicking algorithm by method 1, method 2, method 3, and method 4, respectively. These four methods are known and can be directly used.

Without loss of generality, the original Bayer pattern is shown in Fig. 1a. By method i, the reconstructed image RI_i0 can be gotten through interpolating the missing colors. After RI_i0 is resampled according to the new Bayer pattern, the sampled red and blue values in the original CFA image are replaced by the green values in RI_i0, and the sampled green values in the original Bayer pattern are replaced by the blue values and the red values in RI_i0, which are shown in Fig. 1. By using the method i again, the full-resolution color image RI_i is obtained from the CFA samples which are shown in Fig. 1b. By sampling RI_i by the original Bayer pattern, we can obtain the improved Bayer pattern, where R, G, and B are the samples in the original Bayer pattern, and where r_i, g_i, and b_i are the values of R, G, and B channels of RI_i(i = 1, 2, 3, 4), respectively, as illustrated by Fig. 2.

Fig. 1

Schematic diagram of Bayer resampling. From left to right, the two kinds of Bayer pattern are labeled as a to b

Fig. 2

The improved Bayer pattern

Method 1 and method 2 only use local correlation while method 3 and method 4 utilize not only the local correlation but also nonlocal similarity. Hence, we call method i(i = 1, 2) and method j(j = 3, 4) local demosaicking method and nonlocal demosaicking method, respectively. At position (i, j), we denote the weight of method 1 versus method 3, method 1 versus method 4, method 2 versus method 3, and method 2 versus method 4 by ${\xi}_{i,j}^1$, ${\xi}_{i,j}^2$, ${\xi}_{i,j}^3$, and ${\xi}_{i,j}^4$, respectively, where ${\xi}_{i,j}^k\in \left[0,1\right]$, k = 1, 2, 3, 4. To be specific, ${\xi}_{i,j}^k$ can be calculated by the formulas (20)–(23).

$$ {\xi}_{i,j}^1={\underset{x}{\min}}^{-1}{f}_1(x)=\frac{\xi_{i,j\_\mathrm{num}}^1}{{\xi_{i,j}^1}_{\_\mathrm{den}}}, $$

where

$$ {\displaystyle \begin{array}{l}{f}_1(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_1\Big(m,n\left)-\left(1-x\right)\cdot {r}_3\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_1\Big(m,n\left)-\left(1-x\right)\cdot {g}_3\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_1\Big(m,n\left)-\left(1-x\right)\cdot {b}_3\right(m,n\Big)\right]}^2\\ {}{\xi}_{i,j\_\mathrm{num}}^1=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_3\left(m,n\right)-{r}_1\Big(m,n\Big)\right]\cdot \left[{r}_3\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_3\left(m,n\right)-{g}_1\Big(m,n\Big)\right]\cdot \left[{g}_3\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_3\left(m,n\right)-{b}_1\Big(m,n\Big)\right]\cdot \left[{b}_3\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{\xi}_{i,j\_\mathrm{den}}^1=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left\{{\left[{r}_3\left(m,n\right)-{r}_1\Big(m,n\Big)\right]}^2+{\left[{g}_3\left(m,n\right)-{g}_1\Big(m,n\Big)\right]}^2+{\left[{b}_3\left(m,n\right)-{b}_1\Big(m,n\Big)\right]}^2\right\}\end{array}} $$

(20)

$$ {\xi}_{i,j}^2={\underset{x}{\min}}^{-1}{f}_2(x)=\frac{\xi_{i,j\_\mathrm{num}}^2}{{\xi_{i,j}^2}_{\_\mathrm{den}}}, $$

where

$$ {\displaystyle \begin{array}{l}{f}_2(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_1\Big(m,n\left)-\left(1-x\right)\cdot {r}_4\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_1\Big(m,n\left)-\left(1-x\right)\cdot {g}_4\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_1\Big(m,n\left)-\left(1-x\right)\cdot {b}_4\right(m,n\Big)\right]}^2\\ {}{\xi}_{i,j\_\mathrm{num}}^2=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_4\left(m,n\right)-{r}_1\Big(m,n\Big)\right]\cdot \left[{r}_4\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_4\left(m,n\right)-{g}_1\Big(m,n\Big)\right]\cdot \left[{g}_4\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_4\left(m,n\right)-{b}_1\Big(m,n\Big)\right]\cdot \left[{b}_4\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{\xi}_{i,j\_\mathrm{den}}^2=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left\{{\left[{r}_4\left(m,n\right)-{r}_1\Big(m,n\Big)\right]}^2+{\left[{g}_4\left(m,n\right)-{g}_1\Big(m,n\Big)\right]}^2+{\left[{b}_4\left(m,n\right)-{b}_1\Big(m,n\Big)\right]}^2\right\}\end{array}} $$

(21)

$$ {\xi}_{i,j}^3={\underset{x}{\min}}^{-1}{f}_3(x)=\frac{\xi_{i,j\_\mathrm{num}}^3}{{\xi_{i,j}^3}_{\_\mathrm{den}}}, $$

where

$$ {\displaystyle \begin{array}{l}{f}_3(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_2\Big(m,n\left)-\left(1-x\right)\cdot {r}_3\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_2\Big(m,n\left)-\left(1-x\right)\cdot {g}_3\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_2\Big(m,n\left)-\left(1-x\right)\cdot {b}_3\right(m,n\Big)\right]}^2\\ {}{\xi}_{i,j\_\mathrm{num}}^3=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_3\left(m,n\right)-{r}_2\Big(m,n\Big)\right]\cdot \left[{r}_3\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_3\left(m,n\right)-{g}_2\Big(m,n\Big)\right]\cdot \left[{g}_3\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_3\left(m,n\right)-{b}_2\Big(m,n\Big)\right]\cdot \left[{b}_3\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{\xi}_{i,j\_\mathrm{den}}^3=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left\{{\left[{r}_3\left(m,n\right)-{r}_2\Big(m,n\Big)\right]}^2+{\left[{g}_3\left(m,n\right)-{g}_2\Big(m,n\Big)\right]}^2+{\left[{b}_3\left(m,n\right)-{b}_2\Big(m,n\Big)\right]}^2\right\}\end{array}} $$

(22)

$$ {\xi}_{i,j}^4={\underset{x}{\min}}^{-1}{f}_4(x)=\frac{\xi_{i,j\_\mathrm{num}}^4}{{\xi_{i,j}^4}_{\_\mathrm{den}}}, $$

where

$$ {\displaystyle \begin{array}{l}{\xi}_{i,j\_\mathrm{den}}^4=\sum \limits_{m,n\in \mathbf{W}}\left\{{\left[{r}_4\left(m,n\right)-{r}_2\Big(m,n\Big)\right]}^2+{\left[{g}_4\left(m,n\right)-{g}_2\Big(m,n\Big)\right]}^2+{\left[{b}_4\left(m,n\right)-{b}_2\Big(m,n\Big)\right]}^2\right\}\\ {}{\xi}_{i,j\_\mathrm{num}}^4=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_4\left(m,n\right)-{r}_2\Big(m,n\Big)\right]\cdot \left[{r}_4\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_4\left(m,n\right)-{g}_2\Big(m,n\Big)\right]\cdot \left[{g}_4\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_4\left(m,n\right)-{b}_2\Big(m,n\Big)\right]\cdot \left[{b}_4\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{f}_4(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_2\Big(m,n\left)-\left(1-x\right)\cdot {r}_4\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_2\Big(m,n\left)-\left(1-x\right)\cdot {g}_4\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_2\Big(m,n\left)-\left(1-x\right)\cdot {b}_4\right(m,n\Big)\right]}^2\end{array}} $$

(23)

It should be noted that if ${\xi}_{i,j\_\mathrm{den}}^k=0$ then ${\xi}_{i,j}^k=0.5$, where k = 1, 2, 3, 4.

So far, we might measure the observation vector ${\boldsymbol{\upxi}}_{i,j}=\left({\xi}_{i,j}^1,{\xi}_{i,j}^2,{\xi}_{i,j}^3,{\xi}_{i,j}^4\right)$ of the state s_{i, j} at pixel location (i, j).

Let ${\Omega}_{i,j}^1=\left\{{s}_{m,n}|\left(m,n\right)<\Big(i,j\Big)\right\}$ and ${\boldsymbol{\upxi}}_{i,j}^1=\left\{{\boldsymbol{\upxi}}_{m,n}|\left(m,n\right)<\Big(i,j\Big)\right\}$, where (m, n) < (i, j) is defined as m < i, or m = i and n < j. Let ${\Omega}_{i,j}^2=\left\{{s}_{m,n}|\left(m,n\right)\ge \Big(i,j\Big)\right\}$ and ${\boldsymbol{\upxi}}_{i,j}^2=\left\{{\boldsymbol{\upxi}}_{m,n}|\left(m,n\right)\ge \Big(i,j\Big)\right\}$, where (m, n) ≥ (i, j) is defined as n < j, or n = j and m < i. Set ${\Omega}_{i,j}^3={\Omega}_{i,j}^1\cup {\Omega}_{i,j}^2$, ${\boldsymbol{\upxi}}_{i,j}^3={\boldsymbol{\upxi}}_{i,j}^1\cup {\boldsymbol{\upxi}}_{i,j}^2$.

Put

$$ {\displaystyle \begin{array}{c}{U}_{i,j}=\left\{\begin{array}{c}\left\{\left(i-1,j\right),\Big(i-1,j-1\Big),\Big(i,j-1\Big)\right\},\kern0.5em i>1,j>1\\ {}\begin{array}{cccc}\left\{\left(i-1,j\right)\right\},& \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}& & i>1,j=1\end{array}\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}& & i=1,j>1\end{array}\end{array}\right.,\\ {}{V}_{i,j}=\left\{\begin{array}{c}\begin{array}{cc}\left\{\left(i-1,j\right),\Big(i,j-1\Big)\right\},& i>1,j>1\end{array}\\ {}\begin{array}{cccc}\left\{\left(i-1,j\right)\right\},& & & i>1,j=1\end{array}\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& & & i=1,j>1\end{array}\end{array}.\right.\end{array}} $$

Based on the definition of the 2D 3 × 3 CHMM, we have

$$ P\left({s}_{i,j}|{\Omega}_{i,j}^t,{\boldsymbol{\upxi}}_{i,j}^t\right)=P\left({s}_{i,j}|{s}_{k,l},\left(k,l\right)\in {U}_{i,j}\right),t=1,2,3; $$

$$ P\left({\boldsymbol{\upxi}}_{i,j}|{s}_{i,j},{\Omega}_{i,j}^t,{\boldsymbol{\upxi}}_{i,j}^t\right)=P\left({\boldsymbol{\upxi}}_{i,j}|{s}_{i,j},{s}_{k,l},\left(k,l\right)\in {V}_{i,j}\right),t=1,2,3. $$

The demosaicking method based on the 2D 3 × 3 CHMM is given below.

• (1) After the initial 2D 3 × 3 CHMM has been selected, we reestimate the model by using the formulas (9)–(19) and the given observation matrix O.

• (2) We obtain the state s_{i, j} at pixel location (i, j) according to the solution to problem 2 in Section 3.2.

• (3) Adaptive processing is as follows:

  • If s_{i, j} = 1 and ${\xi}_{i,j}^1+{\xi}_{i,j}^2\ge {\xi}_{i,j}^3+{\xi}_{i,j}^4$, then the CFA interpolation result for method 1 is selected.

  • If s_{i, j} = 1 and ${\xi}_{i,j}^1+{\xi}_{i,j}^2<{\xi}_{i,j}^3+{\xi}_{i,j}^4$, then the CFA interpolation result for method 2 is selected.

  • If s_{i, j} = 2 and $\left(1-{\xi}_{i,j}^1\right)+\left(1-{\xi}_{i,j}^3\right)\ge \left(1-{\xi}_{i,j}^2\right)+\left(1-{\xi}_{i,j}^4\right)$, then the CFA interpolation result for method 3 is selected.

  • If s_{i, j} = 2 and $\left(1-{\xi}_{i,j}^1\right)+\left(1-{\xi}_{i,j}^3\right)<\left(1-{\xi}_{i,j}^2\right)+\left(1-{\xi}_{i,j}^4\right)$, then the CFA interpolation result for method 4 is selected.

5 Experiments

The proposed demosaicking method, which is denoted by 2D 3 × 3 CHMM, is implemented and compared with five other CDM algorithms: VCD [10], DLMMSE [9], NS [13], VCD-NS [14], and 2D CHMM.

In the section, we assess the performance of various demosaicking methods on the two benchmark data sets, which are the Kodak dataset and the McMaster dataset. A set of testing images are generated by the 5 McMaster color images in Fig. 3 and the 5 Kodak color images in Fig. 4. CPSNR is used to quantify and measure the performance of the CDM algorithms.

Fig. 3

200 × 200 McMaster images used in the experiments. From left to right, these images are labeled as 1 to 5

Fig. 4

200 × 200 Kodak images used in the experiments. From left to right, these images are labeled as 6 to 10

In the experiments, the original images are down-sampled into the Bayer CFA images, and then, we have reconstructed the full-color images using the six algorithms. Table 1 tabulates the CPSNR performance of those six methods. Among the tested demosaicking algorithms, our proposed 2D 3 × 3 CHMM method obtains the best average performance. Specifically, our proposed method can produce 0.39 dB improvement over 2D CHMM method, 0.71 dB improvement over NS method, 0.74 dB improvement over DLMMSE method, 0.81 dB improvement over VCD-NS method, and 2.04 dB improvement over VCD method.

Table 1 CPSNR(dB) results using different demosaicking methods

Figures 5 to 6 show the cropped and zoomed demosaicking results of the six schemes on images 2 and 4. It should be clear that the proposed method yields much better demosaicking outputs than the other five methods.

Fig. 5

a–g Part of the demosaicking results of image 2 by different methods

Fig. 6

a–g Part of the demosaicking results of image 4 by different methods

6 Results and discussion

2D 3 × 3 CHMM and 2D CHMM significantly reduce the artifacts and color demosaicking errors and recover more accurately the missing samples than VCD-NS, DLMMSE, VCD, and NS. Their higher CPSNR measures in Table 1 also verify the powerful features in color reproduction. These are expected since the classification and optimization based on the 2D 3 × 3 CHMM and 2D CHMM to the VCD-NS, DLMMSE, VCD, and NS can improve the CFA interpolation effects.

Compared with 2D CHMM, the classification based on the 2D 3 × 3 CHMM takes more context into consideration, and hence, the 2D 3 × 3 CHMM reduces the false colors and zipper effects and more faithfully preserves the edges.

7 Conclusions

This paper has presented the 2D 3 × 3 CHMM in which a structured way is provided to incorporate more context information into classification. After several algorithms are given to solve the basic questions for the model, a novel adaptive demosaicking method based on the 2D 3 × 3 CHMM is proposed. In the method, the problem of CFA interpolation is converted to the MAP sequence estimation. The 2D 3 × 3 CHMM incorporates the statistics of HR images into the interpolation process and exploits high-order statistical dependency between missing pixels. Experimental results show that our proposed method achieves satisfactory images and outperforms several state-of-the-art demosaicking approaches.

In the future, we are planning to further speed up the proposed demosaicking algorithm and then extend the method to super-resolution. We are also interested in applying the proposed method to video analysis.