Skip to content

Advertisement

  • Research
  • Open Access

Demosaicking with two-dimensional continuous 3 × 3 order hidden Markov model

EURASIP Journal on Image and Video Processing20182018:131

https://doi.org/10.1186/s13640-018-0369-4

  • Received: 15 August 2018
  • Accepted: 26 October 2018
  • Published:

Abstract

Since most digital cameras use color filter arrays to sample red, green, and blue colors by a specific pattern, only one color sample would be taken at every pixel location. The process named demosaicking is exploited to recover the full-color image from the incomplete color samples. The paper presents a novel demosaicking method based on two-dimensional continuous 3 × 3 order HMM (2D 3 × 3 CHMM), which incorporates the statistics of high resolution images into the CFA interpolation process. The proposed new method adopts an approach of MAP sequence estimation and exploits high-order statistical dependency between missing pixels. Experiment results demonstrate that the proposed method outperforms several existing or state-of-the-art demosaicking techniques in terms of both objective and subjective evaluations.

Keywords

  • Color filter array
  • Demosaicking
  • Hidden Markov model
  • MAP estimation

1 Introduction

Digital imaging devices find a broad range of applications, particularly in digital cameras, surveillance devices, and mobile phones [15]. Every pixel must consist of three independent primary color components: red, green, and blue, when a digital camera obtains a color image [1, 6]. In order to reduce costs, most digital cameras use only one monochromatic image sensor with a color filter array (CFA) which makes each pixel to obtain only one color component. The most common CFA is the Bayer pattern that includes twice as much green information compared to blue (or red) information. To reconstruct the full-color image from such CFA samples, the missing color components need to be estimated by the color demosaicking (CDM) process.

The quality of the reconstructed image relies on not only the image contents but also the demosaicking methods [7, 8]. The earliest proposed approaches are the interpolation-based methods such as nearest-neighbor replication, bilinear, and bicubic interpolation. Although the implementation of these methods is simple and fast, they yield severe artifacts like zippering or false color information, especially along highly textural regions or the edges. The recently developed methods include the directional linear minimum mean square-error estimation (DLMMSE)-based CDM method [9], the variance of color difference (VCD)-based CDM method [10], the adaptive homogeneity CDM (AHD) method [11], and the successive approximation (SA)-based CDM method [12]. These demosaicking methods make the assumption that the local correlations are high. Such an hypothesis may be well effective for Kodak images, while the assumption may fail for images such as those in the McMaster dataset [7, 8]. The nonlocal similarity method (NS) and the nonlocal similarity fusion method (VCD-NS) presented recently exploit the image nonlocal redundancy to improve the CFA interpolation result [13, 14]. The demosaicking methods which use nonlocal similarity are called the nonlocal CDM methods. Similarly, we call the demosaicking methods that only use local correlations the local CDM methods.

Most block-based image classification algorithms decide the class of a block by examining only the feature vector of this block [15]. Since context information between blocks is ignored, the performance of such algorithms is limited [16]. In order to improve image classification by context, J. Li et al. [15] proposed a two-dimensional hidden Markov model (2D-HMM), in which the state transition probability for each block relies on the states of nearest neighboring blocks from vertical and horizontal directions. The 2D-HMM has wide applications in the field of pattern recognition and image processing [1720]. However, the context information which a block depends on may arise from other directions, such as diagonal direction [21, 22]. In order to overcome the defect of the assumption of the 2D-HMM, the paper [22] presents the 2D 3 × 3 CHMM, where the probability density of the observation depends on not only current state but also immediate vertical and horizontal states, and where the state transition probability depends on not only immediate vertical and horizontal states but also immediate diagonal state.

In this paper, we present a novel demosaicking method based on 2D 3 × 3 CHMM, which incorporates the statistics of HR images into the interpolation process. In our method, the problem of CFA interpolation is converted into the MAP sequence estimation, which exploits high-order statistical dependency between pixels. Extensive experiments show that our proposed method outperforms the demosaicking algorithm based on two-dimensional continuous HMM. This is because that 2D continuous high-order HMM can better model spatial correlation in image data compared to the existing 2D continuous HMM.

The remainder of the paper is organized as follows: Section 2 provides the definition of the 2D continuous 3 × 3 order HMM. Section 3 gives some algorithms to solve the basic problems for the 2D continuous 3 × 3 order HMM. Section 4 describes in detail the proposed demosaicking algorithm. Sections 5 and 6 present the experimental results, and Section 7 concludes the paper.

2 Definition of the 2D continuous 3 × 3 order HMM

Let IM, N = {(i, j)| 1 ≤ i ≤ M, 1 ≤ j ≤ N} be a M × N integer grid. If i' < i, or i' = i and j' < j for (i', j'), (i, j)  IM, N, we call that (i', j') is before (i, j) and write (i', j') < (i, j).

Put φi, j = {(i', j')| (i', j') < (i, j)}.
$$ {U}_{i,j}=\left\{\begin{array}{c}\left\{\left(i-1,j\right),\Big(i-1,j-1\Big),\Big(i,j-1\Big)\right\},\kern0.5em i>1,j>1,\\ {}\left\{\left(i-1,j\right)\right\},\kern0.5em \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}\kern1.00em i>1,j=1,\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}& & i=1,j>1.\end{array}\end{array}\right. $$
$$ {V}_{i,j}=\left\{\begin{array}{c}\left\{\left(i-1,j\right),\Big(i,j-1\Big)\right\},\kern0.5em i>1,j>1,\\ {}\begin{array}{cccc}\left\{\left(i-1,j\right)\right\},& & & i>1,j=1,\end{array}\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& & & i=1,j>1.\end{array}\end{array}\right. $$
We denote the state and the observation at (i, j) by qi, j and oi, j, respectively. Then, we give the following two assumptions (1) and (2).
$$ P\left({q}_{i,j}|{q}_{k,l},{\mathbf{o}}_{k,l},\left(k,l\right)\in {\varphi}_{i,j}\right)=P\left({q}_{i,j}|{q}_{k,l},\left(k,l\right)\in {U}_{i,j}\right) $$
(1)
$$ P\left({\mathbf{o}}_{i,j}|{q}_{i,j},{q}_{k,l},{\mathbf{o}}_{k,l},\left(k,l\right)\in {\varphi}_{i,j}\right)=P\left({\mathbf{o}}_{i,j}|{q}_{i,j},{q}_{k,l},\left(k,l\right)\in {V}_{i,j}\right) $$
(2)

Let O = {oi, j| (i, j)  IM, N} be an observation matrix and Q = {qi, j| (i, j)  IM, N} be a state matrix. Suppose the observation oi, j at a specific state follows multivariate normal distribution. Then, (Q, O) is 2D 3 × 3 CHMM when and only when (1) and (2) hold.

For convenience, the parameters of the 2D 3 × 3 CHMM (Q, O) are given as follows:
  1. (1)

    L represents the total state number.

     
  2. (2)

    The probability density of oi, j(2 ≤ i ≤ M, 2 ≤ j ≤ N)

     
$$ {\displaystyle \begin{array}{c}\mathbf{B}=\left\{{b}_{k,m,n}\left({\mathbf{o}}_{i,j}\right)|1\le k,m,n\le L\right\}\\ {}=\left\{P\left({\mathbf{o}}_{i,j}|{q}_{i,j}=n,{q}_{i-1,j}=k,{q}_{i,j-1}=m,1\le k,m,n\le L\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_{k,m,n}\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{i,j}-{\boldsymbol{\upmu}}_{k,m,n}\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_{k,m,n}\right)}^{-1}\left({\mathbf{o}}_{i,j}-{\boldsymbol{\upmu}}_{k,m,n}\right)}|1\le k,m,n\le L\right\}\end{array}} $$
where μk, m, n and Σk, m, n are the mean vectors and the covariance matrix, respectively, and where p is the dimension of oi, j.
  1. (3)

    The probability density of the observation in the first row

     
$$ {\displaystyle \begin{array}{c}{\mathbf{B}}^{\mathrm{h}}=\left\{{b}_{m,n}^{\mathrm{h}}\left({\mathbf{o}}_{1,j}\right)|1\le m,n\le L,2\le j\le N\right\}\\ {}=\left\{P\left({\mathbf{o}}_{1,j}|{q}_{1,j}=n,{q}_{1,j-1}=m,1\le m,n\le L,2\le j\le N\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_{m,n}^h\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{1,j}-{\boldsymbol{\upmu}}_{m,n}^h\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_{m,n}^h\right)}^{-1}\left({\mathbf{o}}_{1,j}-{\boldsymbol{\upmu}}_{m,n}^h\right)}|1\le m,n\le L,2\le j\le N\Big)\right\}\end{array}} $$
where \( {\boldsymbol{\upmu}}_{m,n}^h \) and \( {\boldsymbol{\Sigma}}_{m,n}^h \) are the mean vectors and the covariance matrix, respectively, and where p is the dimension of o1, j.
  1. (4)

    The probability density of the observation in the first column

     
$$ {\displaystyle \begin{array}{c}{\mathbf{B}}^{\mathrm{v}}=\left\{{b}_{m,n}^{\mathrm{v}}\left({\mathbf{o}}_{i,1}\right)|1\le m,n\le L,2\le i\le M\right\}\\ {}=\left\{P\left({\mathbf{o}}_{i,1}|{q}_{i,1}=n,{q}_{i-1,1}=m,1\le m,n\le L,2\le i\le M\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_{m,n}^v\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{i,1}-{\boldsymbol{\upmu}}_{m,n}^v\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_{m,n}^v\right)}^{-1}\left({\mathbf{o}}_{i,1}-{\boldsymbol{\upmu}}_{m,n}^v\right)}|1\le m,n\le L,2\le i\le M\Big)\right\}\end{array}} $$
where \( {\boldsymbol{\upmu}}_{m,n}^v \) and \( {\boldsymbol{\Sigma}}_{m,n}^v \) are the mean vectors and the covariance matrix, respectively, and where p is the dimension of oi, 1.
  1. (5)

    The probability density of the observation in the initial position

     
$$ {\displaystyle \begin{array}{c}{\mathbf{B}}^0=\left\{{b}_n^0\left({\mathbf{o}}_{1,1}\right)|1\le n\le L\right\}\\ {}=\left\{P\left({\mathbf{o}}_{1,1}|{q}_{1,1}=n,1\le n\le L\right)\right\}\\ {}=\left\{{\left(2\pi \right)}^{-p/2}{\left[\det \left({\boldsymbol{\Sigma}}_n^0\right)\right]}^{-1/2}{e}^{-\frac{1}{2}\left({\mathbf{o}}_{1,1}-{\boldsymbol{\upmu}}_n^0\right)\hbox{'}{\left({\boldsymbol{\Sigma}}_n^0\right)}^{-1}\left({\mathbf{o}}_{1,1}-{\boldsymbol{\upmu}}_n^0\right)}|1\le n\le L\Big)\right\}\end{array}} $$
where \( {\boldsymbol{\upmu}}_n^0 \) and \( {\boldsymbol{\Sigma}}_n^0 \) are the mean vectors and the covariance matrix, respectively, and where p is the dimension of o1, 1.
  1. (6)

    The state transition probability distribution A = {ak, l, m, n| 1 ≤ k, l, m, n ≤ L},

     
where ak, l, m, n = P(qi, j = n| qi − 1, j = k, qi − 1, j − 1 = l, qi, j − 1 = m),1 ≤ k, l, m, n ≤ L.
  1. (7)

    The probability of the state transfer about the first row

     
\( {\mathbf{A}}^{\mathrm{h}}=\left\{{a}_{m,n}^{\mathrm{h}}|1\le m,n\le L\right\} \), where \( {a}_{m,n}^{\mathrm{h}}=P\left({q}_{1,j}=n|{q}_{1,j-1}=m\right) \), 1 ≤ m, n ≤ L.
  1. (8)

    The probability of the state transfer about the first column.

     
\( {\mathbf{A}}^{\mathrm{v}}=\left\{{a}_{m,n}^{\mathrm{v}}|1\le m,n\le L\right\} \), where \( {a}_{m,n}^{\mathrm{v}}=P\left({q}_{i,1}=n|{q}_{i-1,1}=m\right) \), 1 ≤ m, n ≤ L.
  1. (9)

    The initial state probability distribution π = {πn| 1 ≤ n ≤ L} where πn = P(q1, 1 = n),1 ≤ n ≤ L.

     

Put τ = (π, Ah, Av, A, B0, Bh, Bv, B). Then, τ may be used to represent the set of all model parameters.

Set \( {T}_n=\left\{{q}_{1,n},{q}_{2,n},\cdots, {q}_{M,n}\right\} \), On = {o1, n, o2, n, , oM, n}. Assume that Q and O are state matrix and observation matrix of τ, respectively. If the nth column of Q and O can be indicated by Tn and On respectively, 1 ≤ n ≤ N, then we have
$$ P\left({T}_n|{T}_{n-1},\cdots, {T}_1\right)=P\left({T}_n|{T}_{n-1}\right) $$
(3)
$$ P\left({O}_n|{T}_n,{T}_{n-1},\cdots, {T}_1\right)=P\left({O}_n|{T}_n,{T}_{n-1}\right) $$
(4)
By (3) and (4), we can deduce that τ equals a one-dimensional 1 × 2 order HMM \( \lambda =\left({A}_{\tau },{B}_{\tau}^{\hbox{'}},{B}_{\tau },{\pi}_{\tau}\right) \), that is described as follows.
  1. (1)

    LM represents the total state number. I is used to represent the individual state, where

     
$$ \Omega =\left\{\mathbf{I}|\mathbf{I}={\left({i}_1,{i}_2,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},{i}_k\in \Big\{1,2,\cdots, L\Big\},1\le k\le M\right\},\mathbf{I}\in \Omega . $$
  1. (2)

    Bτ = {bIJ(W)} represents the observation probability distribution, where

     
$$ {\displaystyle \begin{array}{l}{b}_{\mathbf{IJ}}\left(\mathbf{W}\right)=P\left(\mathbf{W}={\left({\mathbf{o}}_{1,n},{\mathbf{o}}_{2,n},\cdots, {\mathbf{o}}_{M,n}\right)}^{\hbox{'}}|{T}_n=\mathbf{J}\right)={b}_{i_1{j}_1}^h\left({\mathbf{o}}_{1,n}\right)\cdot {b}_{j_1{i}_2{j}_2}\left({\mathbf{o}}_{2,n}\right)\cdots {b}_{j_{M\hbox{-} 1}{i}_M{j}_M}\left({\mathbf{o}}_{M,n}\right)\\ {}n>1,\mathbf{J}\in \Omega, \mathbf{W}\in \mathrm{E}.\end{array}} $$
  1. (3)

    The initial observation probability density is \( {B}_{\tau}^{\hbox{'}}=\left\{{b}_{\mathbf{J}}\left(\mathbf{W}\right)\right\} \), in which

     
$$ {b}_{\mathbf{J}}\left(\mathbf{W}\right)=P\left(\mathbf{W}={\left({\mathbf{o}}_{1,1},{\mathbf{o}}_{2,1},\cdots, {\mathbf{o}}_{M,1}\right)}^{\hbox{'}}|{T}_1=\mathbf{J}\right)={b}_{j_1}^0\left({\mathbf{o}}_{1,1}\right){b}_{j_1{j}_2}^v\left({\mathbf{o}}_{2,1}\right)\cdots {b}_{j_{M\hbox{-} 1}{j}_M}^v\left({\mathbf{o}}_{M,1}\right),\mathbf{J}\in \Omega, \mathbf{W}\in E. $$
  1. (4)

    The initial state probability is ∏τ = {πJ}, where

     
$$ {\pi}_{\mathbf{J}}=P\left({T}_1=\mathbf{J}\right)={\pi}_{j_1}\cdot {a}_{j_1{j}_2}^v\cdot {a}_{j_2{j}_3}^v\cdots {a}_{j_{M\hbox{-} 1}{j}_M}^v,\mathbf{J}\in \Omega . $$
  1. (5)

    The probability of the state transfer is Aτ = {aI, J}, in which

     
$$ {a}_{\mathbf{I},\mathbf{J}}=P\left({T}_{n+1}=\mathbf{J}|{T}_n=\mathbf{I}\right)={a}_{i_1,{j}_1}^h\cdot {a}_{j_1,{i}_1,{i}_2,{j}_2}\times {a}_{j_2,{i}_2,{i}_3,{j}_3}\cdots {a}_{j_{M\hbox{-} 1},{i}_{M\hbox{-} 1},{i}_M,{j}_M} $$
$$ \mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},\mathbf{J}={\left({j}_1,\cdots, {j}_k,\cdots, {j}_M\right)}^{\hbox{'}},\mathbf{I},\mathbf{J}\in \Omega . $$

3 Basic problem solving for the proposed model

In order to put the proposed model into some practical applications, we need to solve these basic problems that are shown below.
  • Problem 1: Given τ and O, how do we compute P(O| τ)?

  • Problem 2: Given τ and O, how do we get the optimal state Q = {qi, j| (i, j)  IM, N}?

  • Problem 3: Given the observation matrix O, how do we adjust τ to maximize P(O| τ)?

3.1 Solution for problem 1

αn(I) is called the forward variable, where αn(I) = P(W1, W2, , Wn, Tn = I| λ) andWn = (o1, n, , oM, n)', n = 1, , N. Then, we can calculate the forward variable as follows:
  1. (1)

    Initialization

     
$$ {\displaystyle \begin{array}{l}{\alpha}_1\left(\mathbf{I}\right)={\pi}_{\mathbf{I}}{b}_{\mathbf{I}}\left({\mathbf{W}}_1\right)={\pi}_{i_1}\cdot {b}_{i_1}^0\left({o}_{11}\right)\cdot {a}_{i_1{i}_2}^v\cdot {b}_{i_1{i}_2}^v\left({o}_{21}\right)\cdot {a}_{i_2{i}_3}^v\cdot {b}_{i_2{i}_3}^v\left({o}_{31}\right)\cdots {a}_{i_{M-1}{i}_M}^v\cdot {b}_{i_{M-1}{i}_M}^v\left({o}_{M1}\right),\\ {}\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}}\in \Omega .\end{array}} $$
  1. (2)

    Recursion.

     
\( {\displaystyle \begin{array}{c}{\alpha}_{n+1}\left(\mathbf{J}\right)=\sum \limits_{\mathbf{I}\in \Omega}\left({\alpha}_n\left(\mathbf{I}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{n+1}\right)\right)\\ {}=\sum \limits_{\mathbf{I}\in \Omega}\left({\alpha}_n\left(\mathbf{I}\right)\cdot \left({a}_{i_1,{j}_1}^{\mathrm{h}}\cdot {a}_{j_1,{i}_1,{i}_2,{j}_2}\cdots {a}_{j_{M-1},{i}_{M-1},{i}_M,{j}_M}\right)\cdot \left({b}_{i_1{j}_1}^h\left({o}_{1,n+1}\right)\cdot {b}_{j_1{i}_2{j}_2}\left({o}_{2,n+1}\right)\cdots {b}_{j_{\mathrm{M}-1}{i}_{\mathrm{M}}{j}_{\mathrm{M}}}\left({o}_{M,n+1}\right)\right)\right)\end{array}}\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},\mathbf{J}={\left({j}_1,\cdots, {j}_k,\cdots, {j}_M\right)}^{\hbox{'}},\mathbf{I},\mathbf{J}\in \Omega \)
  1. (3)

    Termination

     
$$ \mathrm{P}\left(\mathbf{O}|\tau \right)=\mathrm{P}\left({\mathbf{W}}_1,\cdots, {\mathbf{W}}_N|\lambda \right)=\sum \limits_{\mathbf{I}\in \Omega}{\alpha}_N\left(\mathbf{I}\right) $$
(5)

3.2 Solution for problem 2

There is one way to solve problem 2 by the standard as follows:

Standard: Given τ and O, find the optimum state \( {q}_{i,j}^{\ast } \) such that
$$ {q}_{i,j}^{\ast }=\arg \underset{1\le m\le \mathrm{L}}{\max }P\left({q}_{i,j}=m|\mathbf{O},\tau \right)=\arg \underset{1\le m\le \mathrm{L}}{\max }P\left({q}_{i,j}=m|\mathbf{O},\lambda \right) $$
(6)

We call βn(I) backward variable, where βn(I) = P(Wn + 1, Wn + 2, , WN, Tn = I| λ) and

Wi = (o1, i, , oM, i)', i = n + 1, , N. Then, we can calculate the backward variable as follows:
  1. (1)

    Initialization

     
$$ {\beta}_{\mathrm{N}}\left(\mathbf{I}\right)=1,\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}}\in \Omega . $$
  1. (2)

    Recursion.

     
$$ {\displaystyle \begin{array}{c}{\beta}_n\left(\mathbf{I}\right)=\sum \limits_{\mathbf{J}\in \Omega}\left({\beta}_{n+1}\left(\mathbf{J}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{n+1}\right)\right)\\ {}=\sum \limits_{\mathbf{J}\in \Omega}\left({\beta}_{n+1}\left(\mathbf{J}\right)\cdot \left({a}_{i_1,{j}_1}^h\cdot {a}_{j_1,{i}_1,{i}_2,{j}_2}\cdots {a}_{j_{M-1},{i}_{M-1},{i}_M,{j}_M}\right)\cdot \left({b}_{i_1{j}_1}^h\left({o}_{1,n+1}\right)\cdot {b}_{j_1{i}_2{j}_2}\left({o}_{2,n+1}\right)\cdots {b}_{j_{\mathrm{M}-1}{i}_{\mathrm{M}}{j}_{\mathrm{M}}}\left({o}_{M,n+1}\right)\right)\right)\end{array}}\mathbf{I}={\left({i}_1,\cdots, {i}_k,\cdots, {i}_M\right)}^{\hbox{'}},\mathbf{J}={\left({j}_1,\cdots, {j}_k,\cdots, {j}_M\right)}^{\hbox{'}},\mathbf{I},\mathbf{J}\in \Omega $$

Define γn(I) as the posteriori probability distribution of the nth state column qn = I given λ and O, i.e., γn(I) = P(qn = I| O, λ), in which \( {\gamma}_n\left(\mathbf{I}\right)=\frac{P\left(\mathbf{O},{\mathbf{q}}_n=\mathbf{I}|\lambda \right)}{P\left(\mathbf{O}|\lambda \right)}=\frac{\alpha_n\left(\mathbf{I}\right)\cdot {\beta}_n\left(\mathbf{I}\right)}{\sum \limits_{\mathbf{I}\in \Omega}{\alpha}_n\left(\mathbf{I}\right){\beta}_n\left(\mathbf{I}\right)},\mathbf{I}\in \Omega \).

Given τ and O, ηi, j(k) is defined as the posteriori probability distribution of qi, j = k, i.e., ηi, j(k) = P(qi, j = k| O, τ), where qi, j is the state at (i, j).

Then, we have \( {\eta}_{i,j}(k)=\sum \limits_{\mathbf{I}\in \Omega, \mathbf{I}(i)=k}P\left({T}_j=\mathbf{I}|O,\lambda \right)=\sum \limits_{\mathbf{I}\in \Omega, \mathbf{I}(i)=k}{\gamma}_j\left(\mathbf{I}\right) \), in which I(i) is the ith component of I.

Thus, the optimal state \( {q}_{i,j}^{\ast } \) can be found, where \( {q}_{i,j}^{\ast }=\underset{1\le k\le L}{\arg \max}\left[{\eta}_{i,j}(k)\right] \), 1 ≤ i ≤ M, 1 ≤ j ≤ N.

3.3 Solution for problem 3

Define ξt(I, J) as the probability distribution being in state I at column t, and in state J at column t + 1, givenλ and O, i.e.,
$$ {\xi}_t\left(\mathbf{I},\mathbf{J}\right)=P\left({T}_t=\mathbf{I},{T}_{t+1}=\mathbf{J}|\lambda, \mathbf{O}\right),\mathbf{I},\mathbf{J}\in \Omega . $$
(7)
By αn(I) and βn(I), we have
$$ {\xi}_t\left(\mathbf{I},\mathbf{J}\right)=\frac{\alpha_t\left(\mathbf{I}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{t+1}\right)\cdot {\beta}_{t+1}\left(\mathbf{J}\right)}{\sum \limits_{\mathbf{I}\in \Omega}\sum \limits_{\mathbf{J}\in \Omega}{\alpha}_t\left(\mathbf{I}\right)\cdot {a}_{\mathbf{I},\mathbf{J}}\cdot {b}_{\mathbf{I},\mathbf{J}}\left({\mathbf{W}}_{t+1}\right)\cdot {\beta}_{t+1}\left(\mathbf{J}\right)} $$
(8)
where αn(I) is the forward variable and βn(I) is the backward variable.

We denote the reestimation model and the initial model by \( \overline{\tau}=\left(\overline{\boldsymbol{\uppi}},{\overline{\mathbf{A}}}^h,{\overline{\mathbf{A}}}^v,\overline{\mathbf{A}},{\overline{\mathbf{B}}}^0,{\overline{\mathbf{B}}}^h,{\overline{\mathbf{B}}}^v,\overline{\mathbf{B}}\right) \) and τ = (π, Ah, Av, A, B0, Bh, Bv, B), respectively.

To solve problem 3, the reestimation formulae are given as follows:
$$ \overline{{\boldsymbol{\upmu}}_{m,n}^v}=\frac{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)\cdot {\mathbf{o}}_{i+1,1}}{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)} $$
(9)
$$ \overline{{\boldsymbol{\Sigma}}_{m,n}^v}=\frac{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)\cdot \left({\mathbf{o}}_{i+1,1}-\overline{{\boldsymbol{\upmu}}_{m,n}^v}\right)\cdot {\left({\mathbf{o}}_{i+1,1}-\overline{{\boldsymbol{\upmu}}_{m,n}^v}\right)}^{\hbox{'}}}{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)} $$
(10)
$$ \overline{{\boldsymbol{\upmu}}_{m,n}^h}=\frac{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)\cdot {\mathbf{o}}_{1,j+1}}{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)} $$
(11)
$$ \overline{{\boldsymbol{\Sigma}}_{m,n}^h}=\frac{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)\cdot \left({\mathbf{o}}_{1,j+1}-\overline{{\boldsymbol{\upmu}}_{m,n}^h}\right)\cdot {\left({\mathbf{o}}_{1,j+1}-\overline{{\boldsymbol{\upmu}}_{m,n}^h}\right)}^{\hbox{'}}}{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)} $$
(12)
$$ \overline{{\boldsymbol{\upmu}}_{k,m,n}}=\frac{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)\cdot {\mathbf{o}}_{i+1,t+1}}{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)} $$
(13)
$$ \overline{{\boldsymbol{\Sigma}}_{k,m,n}}=\frac{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)\cdot \left({\mathbf{o}}_{i+1,t+1}-\overline{{\boldsymbol{\upmu}}_{k,m,n}}\right)\cdot {\left({\mathbf{o}}_{i+1,t+1}-\overline{{\boldsymbol{\upmu}}_{k,m,n}}\right)}^{\hbox{'}}}{\sum \limits_{i=1}^{M-1}\sum \limits_{t=1}^{N-1}\sum \limits_{\mathbf{I},\mathbf{J}\in \Omega, \mathbf{I}\left(i+1\right)=m,\mathbf{J}(i)=k,\mathbf{J}\left(i+1\right)=n}{\xi}_t\left(\mathbf{I},\mathbf{J}\right)} $$
(14)
$$ \overline{{\boldsymbol{\upmu}}_k^0}={\boldsymbol{\upmu}}_k^0,\overline{{\boldsymbol{\Sigma}}_k^0}={\boldsymbol{\Sigma}}_k^0 $$
(15)
$$ {\overline{\pi}}_n=\frac{P\left({q}_{1,1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{k=1}^LP\left({q}_{1,1}=k,\mathbf{O}|\tau \right)} $$
(16)
$$ {{\overline{a}}^h}_{m,n}=\frac{\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{n=1}^L\sum \limits_{j=1}^{N-1}P\left({q}_{1,j}=m,{q}_{1,j+1}=n,\mathbf{O}|\tau \right)} $$
(17)
$$ {{\overline{a}}^v}_{m,n}=\frac{\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{n=1}^L\sum \limits_{i=1}^{M-1}P\left({q}_{i,1}=m,{q}_{i+1,1}=n,\mathbf{O}|\tau \right)} $$
(18)
$$ {\overline{a}}_{k,l,m,n}=\frac{\sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}P\left({q}_{i,j+1}=k,{q}_{i,j}=l,{q}_{i+1,j}=m,{q}_{i+1,j+1}=n,\mathbf{O}|\tau \right)}{\sum \limits_{n=1}^L\sum \limits_{i=1}^{M-1}\sum \limits_{j=1}^{N-1}P\left({q}_{i,j+1}=k,{q}_{i,j}=l,{q}_{i+1,j}=m,{q}_{i+1,j+1}=n,\mathbf{O}|\tau \right)} $$
(19)
in which 1 ≤ k, l, m, n ≤ L.

By the formulas (9)–(19), we can get \( \overline{\boldsymbol{\uptau}} \) such that \( P\left(\mathbf{O}|\overline{\boldsymbol{\uptau}}\right)\ge P\left(\mathbf{O}|\boldsymbol{\uptau} \right) \).

4 Methods of CFA interpolation algorithm based on the 2D 3 × 3 CHMM

In the research, we find that the 2D continuous high-order HMM is suitable for the task of CFA interpolation, thanks to its ability to model two-dimensional spatial correlations.

The interpolation of a missing pixel could benefit from the property of the waveform of the true underlying HR image, in particular, whether the pixel is in an area with nonlocal similarity or in a region with local correlation. Since the knowledge of the waveform of the original HR image is unknown at the missing pixel location (i, j), we need to estimate the state si, j which is hidden by the down sampling process [2325]. The estimation is based on the observable low resolution features that are exhibited by si, j.

We can construct the 2D continuous high-order HMM used to demosaicking, if the original 2D intensity function at the missing pixel xi, j is classified into a set of states S = {s1, s2}, where s1 and s2 stand for the state with stronger local correlation and the state with stronger nonlocal similarity, respectively. For simplicity, the state set is denoted by {1, 2}. It should be pointed out that si, j {1, 2}.

Generally speaking, the observation vector ξi, j of si, j involves attributes of the LR image in a window Wi, j which is centered at (i, j); thus, we can measure the feature vector ξi, j by Wi, j. The contents below have detailed descriptions on the measure for ξi, j.

For convenience, we denote VCD, DLMMSE, NS, and VCD-NS demosaicking algorithm by method 1, method 2, method 3, and method 4, respectively. These four methods are known and can be directly used.

Without loss of generality, the original Bayer pattern is shown in Fig. 1a. By method i, the reconstructed image RIi0 can be gotten through interpolating the missing colors. After RIi0 is resampled according to the new Bayer pattern, the sampled red and blue values in the original CFA image are replaced by the green values in RIi0, and the sampled green values in the original Bayer pattern are replaced by the blue values and the red values in RIi0, which are shown in Fig. 1. By using the method i again, the full-resolution color image RIi is obtained from the CFA samples which are shown in Fig. 1b. By sampling RIi by the original Bayer pattern, we can obtain the improved Bayer pattern, where R, G, and B are the samples in the original Bayer pattern, and where ri, gi, and bi are the values of R, G, and B channels of RIi(i = 1, 2, 3, 4), respectively, as illustrated by Fig. 2.
Fig. 1
Fig. 1

Schematic diagram of Bayer resampling. From left to right, the two kinds of Bayer pattern are labeled as a to b

Fig. 2
Fig. 2

The improved Bayer pattern

Method 1 and method 2 only use local correlation while method 3 and method 4 utilize not only the local correlation but also nonlocal similarity. Hence, we call method i(i = 1, 2) and method j(j = 3, 4) local demosaicking method and nonlocal demosaicking method, respectively. At position (i, j), we denote the weight of method 1 versus method 3, method 1 versus method 4, method 2 versus method 3, and method 2 versus method 4 by \( {\xi}_{i,j}^1 \), \( {\xi}_{i,j}^2 \), \( {\xi}_{i,j}^3 \), and \( {\xi}_{i,j}^4 \), respectively, where \( {\xi}_{i,j}^k\in \left[0,1\right] \), k = 1, 2, 3, 4. To be specific, \( {\xi}_{i,j}^k \) can be calculated by the formulas (20)–(23).
$$ {\xi}_{i,j}^1={\underset{x}{\min}}^{-1}{f}_1(x)=\frac{\xi_{i,j\_\mathrm{num}}^1}{{\xi_{i,j}^1}_{\_\mathrm{den}}}, $$
where
$$ {\displaystyle \begin{array}{l}{f}_1(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_1\Big(m,n\left)-\left(1-x\right)\cdot {r}_3\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_1\Big(m,n\left)-\left(1-x\right)\cdot {g}_3\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_1\Big(m,n\left)-\left(1-x\right)\cdot {b}_3\right(m,n\Big)\right]}^2\\ {}{\xi}_{i,j\_\mathrm{num}}^1=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_3\left(m,n\right)-{r}_1\Big(m,n\Big)\right]\cdot \left[{r}_3\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_3\left(m,n\right)-{g}_1\Big(m,n\Big)\right]\cdot \left[{g}_3\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_3\left(m,n\right)-{b}_1\Big(m,n\Big)\right]\cdot \left[{b}_3\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{\xi}_{i,j\_\mathrm{den}}^1=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left\{{\left[{r}_3\left(m,n\right)-{r}_1\Big(m,n\Big)\right]}^2+{\left[{g}_3\left(m,n\right)-{g}_1\Big(m,n\Big)\right]}^2+{\left[{b}_3\left(m,n\right)-{b}_1\Big(m,n\Big)\right]}^2\right\}\end{array}} $$
(20)
$$ {\xi}_{i,j}^2={\underset{x}{\min}}^{-1}{f}_2(x)=\frac{\xi_{i,j\_\mathrm{num}}^2}{{\xi_{i,j}^2}_{\_\mathrm{den}}}, $$
where
$$ {\displaystyle \begin{array}{l}{f}_2(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_1\Big(m,n\left)-\left(1-x\right)\cdot {r}_4\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_1\Big(m,n\left)-\left(1-x\right)\cdot {g}_4\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_1\Big(m,n\left)-\left(1-x\right)\cdot {b}_4\right(m,n\Big)\right]}^2\\ {}{\xi}_{i,j\_\mathrm{num}}^2=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_4\left(m,n\right)-{r}_1\Big(m,n\Big)\right]\cdot \left[{r}_4\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_4\left(m,n\right)-{g}_1\Big(m,n\Big)\right]\cdot \left[{g}_4\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_4\left(m,n\right)-{b}_1\Big(m,n\Big)\right]\cdot \left[{b}_4\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{\xi}_{i,j\_\mathrm{den}}^2=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left\{{\left[{r}_4\left(m,n\right)-{r}_1\Big(m,n\Big)\right]}^2+{\left[{g}_4\left(m,n\right)-{g}_1\Big(m,n\Big)\right]}^2+{\left[{b}_4\left(m,n\right)-{b}_1\Big(m,n\Big)\right]}^2\right\}\end{array}} $$
(21)
$$ {\xi}_{i,j}^3={\underset{x}{\min}}^{-1}{f}_3(x)=\frac{\xi_{i,j\_\mathrm{num}}^3}{{\xi_{i,j}^3}_{\_\mathrm{den}}}, $$
where
$$ {\displaystyle \begin{array}{l}{f}_3(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_2\Big(m,n\left)-\left(1-x\right)\cdot {r}_3\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_2\Big(m,n\left)-\left(1-x\right)\cdot {g}_3\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_2\Big(m,n\left)-\left(1-x\right)\cdot {b}_3\right(m,n\Big)\right]}^2\\ {}{\xi}_{i,j\_\mathrm{num}}^3=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_3\left(m,n\right)-{r}_2\Big(m,n\Big)\right]\cdot \left[{r}_3\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_3\left(m,n\right)-{g}_2\Big(m,n\Big)\right]\cdot \left[{g}_3\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_3\left(m,n\right)-{b}_2\Big(m,n\Big)\right]\cdot \left[{b}_3\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{\xi}_{i,j\_\mathrm{den}}^3=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left\{{\left[{r}_3\left(m,n\right)-{r}_2\Big(m,n\Big)\right]}^2+{\left[{g}_3\left(m,n\right)-{g}_2\Big(m,n\Big)\right]}^2+{\left[{b}_3\left(m,n\right)-{b}_2\Big(m,n\Big)\right]}^2\right\}\end{array}} $$
(22)
$$ {\xi}_{i,j}^4={\underset{x}{\min}}^{-1}{f}_4(x)=\frac{\xi_{i,j\_\mathrm{num}}^4}{{\xi_{i,j}^4}_{\_\mathrm{den}}}, $$
where
$$ {\displaystyle \begin{array}{l}{\xi}_{i,j\_\mathrm{den}}^4=\sum \limits_{m,n\in \mathbf{W}}\left\{{\left[{r}_4\left(m,n\right)-{r}_2\Big(m,n\Big)\right]}^2+{\left[{g}_4\left(m,n\right)-{g}_2\Big(m,n\Big)\right]}^2+{\left[{b}_4\left(m,n\right)-{b}_2\Big(m,n\Big)\right]}^2\right\}\\ {}{\xi}_{i,j\_\mathrm{num}}^4=\sum \limits_{m,n\in {\mathbf{W}}_{i,j}}\left[{r}_4\left(m,n\right)-{r}_2\Big(m,n\Big)\right]\cdot \left[{r}_4\left(m,n\right)-R\Big(m,n\Big)\right]+\left[{g}_4\left(m,n\right)-{g}_2\Big(m,n\Big)\right]\cdot \left[{g}_4\left(m,n\right)-G\Big(m,n\Big)\right]+\left[{b}_4\left(m,n\right)-{b}_2\Big(m,n\Big)\right]\cdot \left[{b}_4\left(m,n\right)-B\Big(m,n\Big)\right]\\ {}{f}_4(x)=\sum \limits_{m,n\in {W}_{i,j}}{\left[R\left(m,n\right)-x\cdot {r}_2\Big(m,n\left)-\left(1-x\right)\cdot {r}_4\right(m,n\Big)\right]}^2+{\left[G\left(m,n\right)-x\cdot {g}_2\Big(m,n\left)-\left(1-x\right)\cdot {g}_4\right(m,n\Big)\right]}^2+{\left[B\left(m,n\right)-x\cdot {b}_2\Big(m,n\left)-\left(1-x\right)\cdot {b}_4\right(m,n\Big)\right]}^2\end{array}} $$
(23)

It should be noted that if \( {\xi}_{i,j\_\mathrm{den}}^k=0 \) then \( {\xi}_{i,j}^k=0.5 \), where k = 1, 2, 3, 4.

So far, we might measure the observation vector \( {\boldsymbol{\upxi}}_{i,j}=\left({\xi}_{i,j}^1,{\xi}_{i,j}^2,{\xi}_{i,j}^3,{\xi}_{i,j}^4\right) \) of the state si, j at pixel location (i, j).

Let \( {\Omega}_{i,j}^1=\left\{{s}_{m,n}|\left(m,n\right)<\Big(i,j\Big)\right\} \) and \( {\boldsymbol{\upxi}}_{i,j}^1=\left\{{\boldsymbol{\upxi}}_{m,n}|\left(m,n\right)<\Big(i,j\Big)\right\} \), where (m, n) < (i, j) is defined as m < i, or m = i and n < j. Let \( {\Omega}_{i,j}^2=\left\{{s}_{m,n}|\left(m,n\right)\ge \Big(i,j\Big)\right\} \) and \( {\boldsymbol{\upxi}}_{i,j}^2=\left\{{\boldsymbol{\upxi}}_{m,n}|\left(m,n\right)\ge \Big(i,j\Big)\right\} \), where (m, n) ≥ (i, j) is defined as n < j, or n = j and m < i. Set \( {\Omega}_{i,j}^3={\Omega}_{i,j}^1\cup {\Omega}_{i,j}^2 \), \( {\boldsymbol{\upxi}}_{i,j}^3={\boldsymbol{\upxi}}_{i,j}^1\cup {\boldsymbol{\upxi}}_{i,j}^2 \).

Put
$$ {\displaystyle \begin{array}{c}{U}_{i,j}=\left\{\begin{array}{c}\left\{\left(i-1,j\right),\Big(i-1,j-1\Big),\Big(i,j-1\Big)\right\},\kern0.5em i>1,j>1\\ {}\begin{array}{cccc}\left\{\left(i-1,j\right)\right\},& \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}& & i>1,j=1\end{array}\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& \begin{array}{cccc}\begin{array}{cc}& \end{array}& & & \end{array}& & i=1,j>1\end{array}\end{array}\right.,\\ {}{V}_{i,j}=\left\{\begin{array}{c}\begin{array}{cc}\left\{\left(i-1,j\right),\Big(i,j-1\Big)\right\},& i>1,j>1\end{array}\\ {}\begin{array}{cccc}\left\{\left(i-1,j\right)\right\},& & & i>1,j=1\end{array}\\ {}\begin{array}{cccc}\left\{\left(i,j-1\right)\right\},& & & i=1,j>1\end{array}\end{array}.\right.\end{array}} $$
Based on the definition of the 2D 3 × 3 CHMM, we have
$$ P\left({s}_{i,j}|{\Omega}_{i,j}^t,{\boldsymbol{\upxi}}_{i,j}^t\right)=P\left({s}_{i,j}|{s}_{k,l},\left(k,l\right)\in {U}_{i,j}\right),t=1,2,3; $$
$$ P\left({\boldsymbol{\upxi}}_{i,j}|{s}_{i,j},{\Omega}_{i,j}^t,{\boldsymbol{\upxi}}_{i,j}^t\right)=P\left({\boldsymbol{\upxi}}_{i,j}|{s}_{i,j},{s}_{k,l},\left(k,l\right)\in {V}_{i,j}\right),t=1,2,3. $$
The demosaicking method based on the 2D 3 × 3 CHMM is given below.
  1. (1)

    After the initial 2D 3 × 3 CHMM has been selected, we reestimate the model by using the formulas (9)–(19) and the given observation matrix O.

     
  2. (2)

    We obtain the state si, j at pixel location (i, j) according to the solution to problem 2 in Section 3.2.

     
  3. (3)
    Adaptive processing is as follows:
    • If si, j = 1 and \( {\xi}_{i,j}^1+{\xi}_{i,j}^2\ge {\xi}_{i,j}^3+{\xi}_{i,j}^4 \), then the CFA interpolation result for method 1 is selected.

    • If si, j = 1 and \( {\xi}_{i,j}^1+{\xi}_{i,j}^2<{\xi}_{i,j}^3+{\xi}_{i,j}^4 \), then the CFA interpolation result for method 2 is selected.

    • If si, j = 2 and \( \left(1-{\xi}_{i,j}^1\right)+\left(1-{\xi}_{i,j}^3\right)\ge \left(1-{\xi}_{i,j}^2\right)+\left(1-{\xi}_{i,j}^4\right) \), then the CFA interpolation result for method 3 is selected.

    • If si, j = 2 and \( \left(1-{\xi}_{i,j}^1\right)+\left(1-{\xi}_{i,j}^3\right)<\left(1-{\xi}_{i,j}^2\right)+\left(1-{\xi}_{i,j}^4\right) \), then the CFA interpolation result for method 4 is selected.

     

5 Experiments

The proposed demosaicking method, which is denoted by 2D 3 × 3 CHMM, is implemented and compared with five other CDM algorithms: VCD [10], DLMMSE [9], NS [13], VCD-NS [14], and 2D CHMM.

In the section, we assess the performance of various demosaicking methods on the two benchmark data sets, which are the Kodak dataset and the McMaster dataset. A set of testing images are generated by the 5 McMaster color images in Fig. 3 and the 5 Kodak color images in Fig. 4. CPSNR is used to quantify and measure the performance of the CDM algorithms.
Fig. 3
Fig. 3

200 × 200 McMaster images used in the experiments. From left to right, these images are labeled as 1 to 5

Fig. 4
Fig. 4

200 × 200 Kodak images used in the experiments. From left to right, these images are labeled as 6 to 10

In the experiments, the original images are down-sampled into the Bayer CFA images, and then, we have reconstructed the full-color images using the six algorithms. Table 1 tabulates the CPSNR performance of those six methods. Among the tested demosaicking algorithms, our proposed 2D 3 × 3 CHMM method obtains the best average performance. Specifically, our proposed method can produce 0.39 dB improvement over 2D CHMM method, 0.71 dB improvement over NS method, 0.74 dB improvement over DLMMSE method, 0.81 dB improvement over VCD-NS method, and 2.04 dB improvement over VCD method.
Table 1

CPSNR(dB) results using different demosaicking methods

Images

VCD-NS [14]

DLMMSE [9]

VCD [10]

NS [13]

2D CHMM

2D 3 × 3 CHMM

1

38.21

37.87

37.41

38.23

38.01

38.26

2

31.77

28.70

29.22

31.25

31.65

31.85

3

33.87

33.40

32.56

33.99

33.82

33.99

4

27.09

24.49

23.43

27.15

27.16

27.17

5

38.89

38.53

38.16

38.93

38.82

38.98

6

39.64

41.04

39.17

39.49

39.45

41.09

7

37.15

38.54

36.10

37.76

38.26

38.61

8

39.45

40.01

39.28

38.87

39.11

40.05

9

35.82

36.76

33.91

36.55

36.74

36.83

10

36.91

40.11

37.22

37.56

40.01

40.08

Average

35.88

35.95

34.65

35.98

36.30

36.69

Figures 5 to 6 show the cropped and zoomed demosaicking results of the six schemes on images 2 and 4. It should be clear that the proposed method yields much better demosaicking outputs than the other five methods.
Fig. 5
Fig. 5

ag Part of the demosaicking results of image 2 by different methods

Fig. 6
Fig. 6

ag Part of the demosaicking results of image 4 by different methods

6 Results and discussion

2D 3 × 3 CHMM and 2D CHMM significantly reduce the artifacts and color demosaicking errors and recover more accurately the missing samples than VCD-NS, DLMMSE, VCD, and NS. Their higher CPSNR measures in Table 1 also verify the powerful features in color reproduction. These are expected since the classification and optimization based on the 2D 3 × 3 CHMM and 2D CHMM to the VCD-NS, DLMMSE, VCD, and NS can improve the CFA interpolation effects.

Compared with 2D CHMM, the classification based on the 2D 3 × 3 CHMM takes more context into consideration, and hence, the 2D 3 × 3 CHMM reduces the false colors and zipper effects and more faithfully preserves the edges.

7 Conclusions

This paper has presented the 2D 3 × 3 CHMM in which a structured way is provided to incorporate more context information into classification. After several algorithms are given to solve the basic questions for the model, a novel adaptive demosaicking method based on the 2D 3 × 3 CHMM is proposed. In the method, the problem of CFA interpolation is converted to the MAP sequence estimation. The 2D 3 × 3 CHMM incorporates the statistics of HR images into the interpolation process and exploits high-order statistical dependency between missing pixels. Experimental results show that our proposed method achieves satisfactory images and outperforms several state-of-the-art demosaicking approaches.

In the future, we are planning to further speed up the proposed demosaicking algorithm and then extend the method to super-resolution. We are also interested in applying the proposed method to video analysis.

Abbreviations

2D-HMM: 

Two-dimensional hidden Markov model

AHD: 

The adaptive homogeneity

CDM: 

Color demosaicking

CFA: 

Color filter array

DLMMSE: 

The directional linear minimum mean square-error estimation

SA: 

The successive approximation

VCD: 

The variance of color difference

Declarations

Acknowledgements

This work is partially supported by Shanxi Province Universities Science and Technology Innovation Project (2017107) and Shanxi Province Science Foundation for Youths (201701D12111421).

Thanks for the editor and reviewers.

Funding

The paper is subsidized by the Shanxi Province Universities Science and Technology Innovation Project (2017107) and Shanxi Province Science Foundation for Youths (201701D12111421).

Availability of data and materials

Data will not be shared; the reason for not sharing the data and materials is that the work submitted for review is not completed. The research is still ongoing, and those data and materials are still required by my team for further investigations.

Author’s contributions

GW designed the research, analyzed the data, wrote, and edited the manuscript. The author read and approved the final manuscript.

Authors’ information

Guogang Wang, Department of Electronic and Information Engineering, Shanxi University. His main research interests are computer vision, image processing, and pattern recognition.

Ethics approval and consent to participate

I approved.

Consent for publication

I agree.

Competing interests

The author declares no competing interests. The author has seen the manuscript and approved for submission to your journal. The author confirmed that the content of the manuscript has not been published or submitted for publication elsewhere.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors’ Affiliations

(1)
Department of Electronic and Information Engineering, Shanxi University, Taiyuan, 030013, China
(2)
College of Physics and Electronics, Shanxi University, Taiyuan, 030006, China

References

  1. J. Wang, W. JJ, W. ZS, G. Jeon, Filter-based Bayer pattern CFA demosaicking. Circuits Syst. Signal Process. 36(7), 2917–2940 (2017).View ArticleGoogle Scholar
  2. X. Chen, G. Jeon, J. Jeong, Voting-based directional interpolation method and its application to still color image demosaicking. IEEE Trans. Circuits Syst. Video Technol. 24(2), 255–262 (2014).View ArticleGoogle Scholar
  3. X. Chen, G. Jeon, J. Jeong, L. He, Multidirectional weighted interpolation and refinement method for Bayer pattern CFA demosaicking. IEEE Trans. Circuits Syst. Video Technol. 25(8), 1271–1282 (2015).View ArticleGoogle Scholar
  4. Z. Dengwen, S. Xiaoliu, D. Weiming, Colour demosaicking with directional filtering and weighting. IET Image Process. 6(8), 1084–1092 (2012).MathSciNetView ArticleGoogle Scholar
  5. C. Yan, H. Xie, D. Yang, J. Yin, Y. Zhang, Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Trans. Intell. Transp. Syst. 19(1), 284–295 (2018).View ArticleGoogle Scholar
  6. K.-L. Hua, S.C. Hidayati, F.-L. He, C.-P. Wei, Y.-C.F. Wang, Context-aware joint dictionary learning for color image demosaicking. J. Vis. Commun. Image Represent. 38(2016), 230–245 (2016).View ArticleGoogle Scholar
  7. L. Zhang, X. Wu, A. Buades, X. Li, Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. J. Electron. Imaging 20(2), 023016 (2011).View ArticleGoogle Scholar
  8. A. Buades, B. Coll, J.-M. Morel, C. Sbert, Self-similarity driven color demosaicking. IEEE Trans. Image Process. 18(6), 1192–1202 (2009).MathSciNetView ArticleGoogle Scholar
  9. L. Zhang, X. Wu, Color demosaicking via directional linear minimum mean square-error estimation. IEEE Trans. Image Process. 14(12), 2167–2178 (2005).View ArticleGoogle Scholar
  10. K.-H. Chung, Y.-H. Chan, Color demosaicing using variance of color differences. IEEE Trans. Image Process. 15(10), 2944–2955 (2006).View ArticleGoogle Scholar
  11. K. Hirakawa, T.W. Parks, Adaptive homogeneity-directed demosaicing algorithm. IEEE Trans. Image Process. 14(3), 360–369 (2005).View ArticleGoogle Scholar
  12. X. Li, Demosaicing by successive approximation. IEEE Trans. Image Process. 14(3), 370–379 (2005).MathSciNetView ArticleGoogle Scholar
  13. G.G. Wang, X.C. Zhu, Z.L. Gan. Image demosaicing by non-local similarity and local correlation. IEEE 11th International Conference on Signal Processing, Beijing 2012, 806–810.Google Scholar
  14. G.G. Wang, Z.L. Gan, G.J. Tang, Z.G. Cui, J.S. Liang, X.C. Zhu, Novel demosaicking method using nonlocal similarity fusion. Metallurgical Min. Ind. 7(9), 689–696 (2015).Google Scholar
  15. J. Li, A. Najmi, R.M. Gray, Image classification by a two-dimensional hidden Markov model. IEEE Trans. Signal Process. 48(2), 517–533 (2000).View ArticleGoogle Scholar
  16. X. Ma, D. Schonfeld, A. Khokhar. A general two-dimensional hidden markov model and its application in image classification. IEEE International Conference on Image Processing, San Antonio, 2007, 2837–2840.Google Scholar
  17. P.M. Baggenstoss, Two-dimensional hidden Markov model for classification of continuous-valued noisy vector fields[J]. IEEE Trans. Aerosp. Electron. Syst. 47(2), 1073–1080 (2011).View ArticleGoogle Scholar
  18. G. Jing, Z. Mingquan, L. Chao, Automatic 3D model annotation by a two-dimensional hidden Markov model. TELKOMNIKA Indones. J. Electrical Eng. 12(5), 3272–3280 (2014).View ArticleGoogle Scholar
  19. A. Behnad, X.L. Wu. Image interpolation with hidden Markov model. 2010 IEEE International Conference on Acoustics, Speech, and Signal Processing, Dallas, 2010, 874–877.Google Scholar
  20. A. Behnad, K. N. Plataniotis, X.L. Wu. A hidden Markov model-based methodology for intra-field video deinterlacing. 2011 18th IEEE International Conference on Image Processing, Brussels, 2011, 1189–1192.Google Scholar
  21. G.G. Wang, Z.L. Gan, G.J. Tang, Z.G. Cui, X.C. Zhu, Basic problems solving for two-dimensional discrete 3×4 order hidden Markov model. Chaos Solitons Fractals 89, 73–82 (2016).View ArticleGoogle Scholar
  22. G.G. Wang, G.J. Tang, Z.L. Gan, Z.G. Cui, X.C. Zhu, Basic problems and solution methods for two-dimensional continuous 3×3 order hidden Markov model. Chaos Solitons Fractals 89, 435–446 (2016).MathSciNetView ArticleGoogle Scholar
  23. Q. Zhou, J. Luo, Artificial neural network based grid computing of E-government scheduling for emergency management. Comput. Syst. Sci. Eng. 30(5), 327–335 (2015).Google Scholar
  24. Q. Zhou, Multi-layer affective computing model based on emotional psychology. Electron. Commer. Res. 18(1), 109–124 (2018). https://doi.org/10.1007/s10660-017-9265-8.View ArticleGoogle Scholar
  25. Q. Zhou, Z. Xu, N.Y. Yen, User sentiment analysis based on social network information and its application in consumer reconstruction intention. Comput. Hum. Behav. (2018) https://doi.org/10.1016/j.chb.2018.07.006.

Copyright

© The Author(s). 2018

Advertisement