Scalable kernel-based minimum mean square error estimate for light-field image compression

Light-field imaging, also referred to as plenoptic imaging, holoscopic imaging, and integral imaging, can capture both spatial and angular information of a 3D scene and enable new possibilities for digital imaging [1]. Light fields (LFs) captured by the light-field imaging represent the intensity and direction of the light rays emanated from a 3D scene. The full LFs can be represented by a seven-dimensional plenoptic function L = L(x, y, z, θ, ϕ, λ, t) introduced by Adelson and Bergen [2], where (x, y, z) is the viewing position, (θ, ϕ) is the light ray directions, λ is the light ray wavelength, andt is the time. The seven-dimensional plenoptic function is further simplified into four dimensions by considering only information taken in a region free of occlusions at a single time instance [3, 4]. The simplified 4D function L = L(u, v, x, y) can represent a set of light rays by being parametrized as an intersection of rays with two planes, where uvdescribing the ray position in aperture (object) plane and xy describing the rays position in image plane. Under this point of view, the LFs can be explored from the digital perspective with the advances in computational photography [5].

There are several techniques that can be utilized to capture a light-field image, such as by using coded apertures, multi-camera arrays, and micro-lens arrays. In the micro-lens array-based technique, the most appropriate way to capture the LF image is by using plenoptic camera which is produced by Lytro. The commercially available plenoptic cameras can be divided into two categories: standard plenoptic camera and focused plenoptic camera. The focused plenoptic camera can provide a trade-off between spatial and angular information by putting the focal plane of microlenses away from the image sensor plane. Since the wide range of possible applications and the rapid development of light-field technology, many research groups have considered the standardization of light-field application. The JPEG working group starts a new study, known as JPEG Pleno [6], aiming at richer image capturing, visualization, and manipulation. The MPEG group has also started the third phase of free-viewpoint television (FTV) since 2013, targeting super multi-view, free navigation, and full parallax imaging applications [7].

Since the acquired LF image records both spatial and angular information of a 3D scene, it can naturally provide the benefits of rendering views at different viewpoints and views at different focused planes, which expand its applications. However, a vast amount of data is needed for such a LF image during the acquisition step for its enhanced features. Even though many image coding methods [8–11] have been proposed, they cannot be directly used for LF image. Therefore, efficient compression schemes for such particular type of image are needed for effective transmission and storage.

According to the available techniques to capture and visualize LF images, the compression schemes of LF images can be mainly categorized into two different categories, where the general diagram of workflow for LF image acquisition and visualization is depicted in Fig. 1. The first kind of compression methods, called spatial correlation-based compression method, is to compress the acquired lenslet image directly (see Fig. 1a) based on the fact that the elementary images (EIs) of lenslet image exhibit repetitive patterns and a large amount of redundancy exists between the neighboring EIs, which can be seen in Fig. 2a. By exploiting the inherent non-local spatial redundancy of LF image, a coding method combining the locally linear embedding (LLE) is proposed in [12]. This work is further improved by combining the LLE based-method and self-similarity (SS)-based compensated prediction method in [13]. Paper [14] puts forward a disparity compensation-based light-field image coding algorithm by exploring the high spatial correlation existing in the LF images, which is further improved by using kernel-based minimum mean-square-error estimation prediction [15] and Gaussian process regression-based prediction method [16]. In [17], Conti et al. introduced the SS mode into HEVC to improve the coding efficiency of light-field image, which is similar to the intra-block copy (IntraBC) incorporated into the HEVC range extension to code the screen contents. To further improve the coding performance, a bi-predicted SS estimation and SS compensation are proposed in [18], where the candidate predictor can be also devised as a linear combination of two blocks within the same search window. A displacement intra-prediction scheme for LF contents is proposed in [19], where more than one hypothesis is used to reduce prediction errors.

The other kind of compression method, called pseudo-sequence-based compression method, considers creation of a 4D LF representation of LF image prior to compression (see Fig. 1b). The pseudo-sequence-based coding methods try to decompose the LF image into multiple views, which can be seen in Fig. 2b. The derived multiple views are then organized into a sequence to make full use of the inter-correlations among various views. In [20], a sub-aperture images streaming scheme is proposed to compress the lenslet images, in which rotation scan mapping is adopted to further improve compression efficiency. A pseudo-sequence-based scheme for LF image compression is proposed in [21], in which the specific coding order of views, prediction structure, and rate allocation have been investigated for encoding the pseudo-sequence. A new LF multi-view video coding prediction structure by extending the inter-view prediction into a two-directional parallel structure is designed in [22] to analyze the relationship of the prediction structure with its coding performance. In [23], a lossless compression method of the rectified LF images is presented to exploit the high similarity existing among the sub-aperture images or view images. A novel pseudo-sequence-based 2D hierarchical reference structure for light-field image compression is proposed in [24], where a 2D hierarchical reference structure is used with the distance-based reference frame selection and spatial-coordinate-based motion vector scaling to better characterize the inter-correlations among the various views decomposed from the light-field image.

Part of this pork has been published in [25]. In this paper, we give more details of theoretical analysis and propose a scalable kernel-based MMSE estimation method to accelerate LF image compression, and we also provide an adaptive layer management mechanism. Experimental results demonstrate the advantage of the proposed scalable compression method in terms of different quality metrics as well as the visual quality of views rendered from decompressed LF content.

The rest of this paper is organized as follows. An overview of the kernel-based MMSE estimation method is introduced in Section 2. The proposed scalable kernel-based MMSE estimate and its different reconstruction layers are described in Section 3. Section 4 gives the details of the layer selection mechanism. Experimental results are presented and analyzed in Section 5, and the concluding remarks are given in Section 6.

The kernel-based MMSE estimation method aims to predict the current coding block given its neighbor known context under a kernel-based point of view by constructing a statistical model and calculating a kernel-based MMSE estimation. In order to construct the statistical model, the pixel values in the coding block and its neighbor known context are arranged into a multidimensional formalism. Kernel density estimation (KDE) is used to estimate the probability density function (PDF) of the statistical model with a set of observed vectors. The coding block is then predicted from an MMSE estimator given the PDF.

Let the pixel values in the current coding block be stacked in a column vector x₀, and the pixel values in its neighbor templates with template thickness being T are compacted in a column vector y₀, shown in Fig. 3a. The current coding block and its neighbor templates with template thickness being T are called the prototype region in this paper. Therefore, the main goal of the prediction method can be expressed to derive the MMSE estimate E[x|y₀] of vector x₀ given its context y₀. In order to do so, we arrange the vector x₀ and y₀ into a multidimensional vector z₀ = (x₀, y₀), and a random vector variable z = (x, y) that has the same configuration as z₀ is considered to obtain the signal statistical behavior. If the PDF of z is acquired, the MMSE estimator E[x|y₀] can be derived from it. In the proposed scheme, we propose to utilize KDE to estimate the PDF of random variable z given a set of observed vectors {z_k| k = 1, …, K}. Here, the observed vectors are composed of K-NN patches, the top K closest templates that have the same configuration as the prototype region in terms of Euclidean distance, derived within the specified horizontal and vertical search windows, shown in Fig. 3b. Since the coding block in prototype region is unknown in the K-NN patches searching procedure, its neighbor blocks are used for searching. We set the template thickness to T₁ which equals to the size of the current coding block to increase the searching accuracy, shown in Fig. 3b.

There are two issues that should be tackled in K-MMSE estimation method. One is to derive the weight vector ω_k(y₀). The other is to estimate the kernel bandwidth matrix H.

The kernel-based MMSE estimation method is a powerful prediction method and can achieve a better prediction accuracy for LF contents in the homogenous texture areas. However, some shortcomings still exist. Firstly, since such method predicts the coding blocks given their neighbor known contexts, it does not always necessarily lead to a good prediction for the unknown block in some non-homogenous texture areas. Secondly, the kernel-based MMSE estimation method is time-consuming, especially in the decoder side. Thirdly, for some visually flat regions in homogenous texture areas, applying kernel-based MMSE estimation method would be an overkill since similar reconstruction quality could be achieved by simpler (and, therefore, faster) estimators when dealing with relatively simple structures. To this end, this paper proposes a scalable kernel-based MMSE estimation method, also called SK-MMSE, that aims at further improving the coding efficiency and accelerating the prediction process by decomposing the prediction procedure into different layers. The proposed SK-MMSE algorithm is comprised by three prediction layers. The layer division is based on the contents of the previously encoded blocks adjacent to the coding block. The higher the layer within the scalable hierarchy, the higher the computational complexity. The scalable layers are introduced in the next subsections, and the layer management mechanism will be illustrated in the next section.

In order to consider the two assumptions, we propose to use the prediction mode correlation to decide which blocks belong to HPL and use the gradient information to measure the visual flatness. The following will introduce the two layer switching mechanism.

For homogenous texture areas, the coding blocks and their neighboring blocks are closed linked. In most cases, they have the similar texture information and structural characteristics. As a results, the prediction modes of the coding blocks should be similar to their neighboring known blocks in the homogenous texture areas. Consequently, we can apply the prediction mode information to decide whether the coding block belongs to the homogenous texture areas and further determine whether the coding block belongs to HPL or KPL. Since the prediction mode information of the current coding block is unknown, the prediction mode information of its neighboring known blocks are utilized in the decision criterion.

From Eq. (12), we see that if PM_L = PM_U = PM_UL, the flag_CB is set to 1 and the current coding block is determined to belong to KPL. Otherwise, flag_CB is set to 0 and the current coding block is determined to belong to HPL. The main reason is that if PM_L = PM_U = PM_UL, the current coding block is likely to have the same prediction mode as its neighboring blocks (left neighboring block, up neighboring block, and up-left neighboring block), which indicates that the current coding block and its neighboring blocks have a similar texture information and structural characteristics and they belong to the homogenous texture areas. Therefore, the current coding block is grouped into KPL. If the flag_CB is equal to 0, which means that the current coding block is quite different from its neighboring blocks, the current coding block is grouped into HPL.

The proposed algorithm is summarized by the flow graph in Fig. 4. It is shown that the SK-MMSE algorithm is a scalable LF image coding method, where the whole coding blocks are divided into three layers. The HPL is used to improve the prediction accuracy for coding blocks in the non-homogenous texture areas while the KPL is applied to ensure the coding efficiency for the coding blocks in the homogenous texture areas where the texture information is abundant. Regarding to the LPL, a simplified prediction method is adopted to further reduce the whole computational complexity with negligible effect to the prediction accuracy, especially for the decoder side. Note that, the proposed framework can also be used to predict chrominance blocks.

In order to validate the efficiency of the proposed method, 12 LF test images including eight circular lens LF test images provided by the ICME 2016 grand challenge in light-field image compression [28] and four rectangular lens LF test images provided by Dr. T. Georgiev [29] are used in the test set. The used LF test images are all captured by the focused plenoptic camera. The resolution of the circular lens LF test images is 7728 × 5368. The original resolution of the rectangular lens LF test images is 7240 × 5432, and we cut the four test images into size of 3840 × 2160 for simplicity. The size of each EI in rectangular lens LF image is 75 × 75. All the LF test images are transformed into YUV 4:2:0 format. The central rendered views from each LF test image are shown in Fig. 5.

The HEVC SCC reference software SCM-3.0 [30] is modified for the proposed hybrid codec architecture. The coding configurations were set as the “All Intra,” which is defined in [31]. Four tested quantization parameters 22, 27, 32, and 37 are used. The proposed hybrid prediction method (referred to as SK-MMSE) is compared with three prediction schemes: the original HEVC (referred to as HEVC), the screen content coding extension Ver. 3.0 to HEVC (referred to as HEVC-SCC), and the kernel-based minimum mean-square-error estimation method [15] (referred to as K-MMSE). The K-MMSE method is realized in the HEVC SCC reference software SCM-3.0. The Y-PSNR and YUV-PSNR between the original LF image and decoded LF image shown in [28] are used as the objective quality metric.

The template thickness T in Fig. 3a is set to 4. The dimensions of the searching windows used in the SK-MMSE and K-MMSE method is given by V = 128, H = 128, shown in Fig. 3b. We have mentioned that the K-MMSE method is integrated into the HEVC SCC standard by replacing one of the 35 intra-directional prediction modes in SK-MMSE method. In our experiment, the intra-prediction mode “4” is replaced and K is set to 6.

Table 2 gives the rate-distortion gains of the three prediction methods over HEVC intra-standard with Y-PSNR as the objective quality metric. From Table 2, we see that the proposed SK-MMSE is clearly superior to the other methods. An average gain of up to 1.61 dB has been achieved by SK-MMSE to HEVC intra-standard. Compared to HEVC-SCC, around 30.9% BD-rate can be saved in average by using SK-MMSE. This is because integrating the K-MMSE method to the HEVC-SCC standard can effectively improve the prediction accuracy. When compared to K-MMSE, the proposed SK-MMSE can achieve about 0.17 dB average gains. The main reason is that the K-MMSE do not work well for the blocks in non-homogenous texture area. By combining the IBC mode, the proposed SK-MMSE can achieve a better prediction of the coding block in such non-homogenous texture area. From Table 2, we can also observe that the K-MMSE method allows an average of 22.4% rate saving compared to the HEVC-SCC, which means that K-MMSE mode can obtain a better prediction and is selected as the best prediction mode in most cases. Figure 6 shows the rate-distortion curve of the test image set using different coding schemes with Y-PSNR as the objective quality metric, which further confirms that the proposed SK-MMSE performs better than other prediction methods.

Images	HEVC-SCC		K-MMSE		SK-MMSE
	BD-PSNR (dB)	BD rate (%)	BD-PSNR (dB)	BD rate (%)	BD-PSNR (dB)	BD rate (%)
Fredo	2.73	− 40.40	3.11	− 43.07	3.52	−  48.59
Jeff	1.58	− 24.22	2.31	− 32.92	2.47	− 35.76
Sergio	2.36	− 32.99	2.94	− 38.61	3.18	− 41.99
Zhengyun1	1.70	− 29.56	2.24	− 36.24	2.41	− 39.63
I01_LF	0.47	− 10.34	0.64	− 13.94	0.77	− 16.60
I02_LF	0.36	− 8.56	0.43	− 10.07	0.59	− 13.56
I03_LF	0.13	− 3.02	0.17	− 4.04	0.23	− 5.42
I05_LF	0.55	− 16.41	0.89	− 26.03	1.00	− 29.72
I07_LF	0.65	− 17.07	0.64	− 17.07	0.83	− 21.72
I09_LF	1.26	− 24.30	2.03	− 36.63	2.13	− 38.40
I10_LF	0.17	− 5.92	0.28	− 9.90	0.31	− 10.73
I12_LF	0.97	− 26.52	1.55	− 40.00	1.72	− 44.14
Average	1.08	− 19.94	1.44	− 25.71	1.61	− 28.85

The rate-distortion gains of the three prediction methods over HEVC intra-standard with YUV-PSNR as the objective quality metric are given in Table 3. From Table 3, we can achieve a consistent conclusion as Table 2. Compared to HEVC intra-standard, an average gain of up to 1.42 dB has been achieved by the proposed SK-MMSE method. Likewise, around 10.5 and 32.8% BD rate can be saved in average by using SK-MMSE when compared to K-MMSE and HEVC-SCC method, respectively. This also validates that the proposed SK-MMSE architecture can effectively compress the LF data. Figure 7 gives the rate-distortion curve of the test image set using different coding schemes with YUV-PSNR as the objective quality metric, which further proves the validity of the proposed SK-MMSE.

Images	HEVC-SCC		K-MMSE		SK-MMSE
	BD-PSNR (dB)	BD rate (%)	BD-PSNR (dB)	BD rate (%)	BD-PSNR (dB)	BD rate (%)
Fredo	2.54	− 40.52	2.90	− 43.22	3.29	− 48.78
Jeff	1.42	− 23.91	2.16	− 33.35	2.31	− 36.17
Sergio	2.13	− 32.93	2.73	− 39.29	2.95	− 42.73
Zhengyun1	1.55	− 29.16	2.14	− 36.86	2.29	− 40.18
I01_LF	0.36	− 9.24	0.52	− 13.13	0.63	− 15.64
I02_LF	0.27	− 7.51	0.33	− 9.19	0.46	− 12.38
I03_LF	0.10	− 2.54	0.12	− 3.25	0.17	− 4.52
I05_LF	0.43	− 14.93	0.74	− 25.11	0.82	− 27.67
I07_LF	0.50	− 15.09	0.51	− 15.69	0.67	− 20.15
I09_LF	1.01	− 23.10	1.69	− 36.17	1.78	− 37.97
I10_LF	0.12	− 4.50	0.23	− 9.07	0.25	− 9.56
I12_LF	0.76	− 24.62	1.28	− 39.18	1.43	− 43.42
Average	0.93	− 19.00	1.28	− 25.29	1.42	− 28.26

Since LF image captures both spatial and angular information of a scene, view images can be rendered from the LF image data. In order to further validate the validity of the proposed coding scheme, we give a visual quality investigation of rendered view image from the decoded LF image in Fig. 8. As shown in Fig. 8, the proposed SK-MMSE can obtain a better visual quality, especially in some texture regions. The main reason mainly lies in two aspects. One is that the proposed scheme can achieve a better coding efficiency compared to other coding methods. The other is that the proposed SK-MMSE prediction method can keep the detail information of EIs in the prediction process effectively.

In this paper, we propose a scalable kernel-based MMSE estimation method to effectively compress the LF image. The coding blocks are divided into three layers. In the HPL, a hybrid kernel-based MMSE estimation and IntraBC method are proposed to predict the coding blocks to improve the prediction accuracy of coding blocks in non-homogenous texture area, which explores the idea of using the IntraBC scheme or intra-directional prediction to find the best prediction of the coding block when K-MMSE estimation method fails based on the rate-distortion optimization (RDO) procedure. In the KPL, we propose to use the K-MMSE estimation method to predict the coding blocks to ensure the coding efficiency for homogenous texture area. In the LPL, we propose to predict the coding blocks by directly using a linear prediction method and do not compute the correction vector. The linear prediction method can be seen as a simplified form of the K-MMSE estimation method. In order to decide which layer is belonged to for the current coding block accurately, we propose to use the prediction mode correlation to decide which blocks belong to HPL and use the gradient information to measure the visual flatness.

The experimental results demonstrate that the proposed SK-MMSE method can compress the light-field image efficiency. It outperforms the HEVC intra-standard with 1.61 and 1.42 dB average quality improvements with Y-PSNR and YUV-PSNR as the objective quality metric, respectively. With regard to the computation complexity, the proposed SK-MMSE method can save around 16.4 and 70.5% average coding time than the K-MMSE estimation method in encoder side and decoder side, respectively.

Future work will include complexity reduction and how to further improve the prediction accuracy for texture and edge regions.