Depth-image-based rendering with spatial and temporal texture synthesis for 3DTV

Year 2010 is considered to be the year of breakthrough for 3D video and 3D industry [1]. Numerous 3D films are produced and released to the market. Stereo movies provide people stereo perceptions by showing two slightly different images of the same scene. Consumers can have immersive feelings by watching them in theaters with stereo eyeglasses. Disks and players of 3D Blu-ray standard have entered the home entertainment. The prosperity of 3D industry gives an important opportunity for three-dimensional television (3DTV) system, which is believed to be the next generation of television broadcasting after high-definition television. The concept of 3DTV system is defined by European project ATTEST [2] and developed by Morvan et al. [3] and Kubota et al. [4]. To improve the depth perception of users, autostereoscopic display technology without any need of additional glasses is preferred in the display part of 3DTV. Autostereoscopic displays can provide comfortable stereo parallax and smooth motion disparity by displaying multiview images of the same scene simultaneously. A simple approach is to capture, compress, and transmit multiple views directly. The current multiview video coding standard [5, 6] with high compression efficiency, which exploits the spatial correlations of the neighboring views, is used to encode and decode the multiple video streams, generally more than eight views. But the transmission bandwidth cost remains a challenging and unresolved problem. Meanwhile, it is commonly suggested that the future 3DTV systems should have completely decoupled capture and display operations [7]. A proper abstract intermediate representation of the captured data, video plus depth format, is proposed by Fehn [8] to achieve such a decoupled operation with an acceptable increment of bandwidth. The depth-image-based rendering (DIBR) [2] algorithm will be used to render multiple perspective views from the video plus depth data according to the requirement of autostereoscopic displays. Thus, the DIBR method has attracted much attention, and become a key technology of the 3DTV system [1].

The video plus depth data format consists of one texture color image and its corresponding perpixel dense depth map. Theoretically, being provided with the intrinsic and extrinsic parameters of the virtual views, the DIBR algorithm can be used to synthesize any virtual perspective views from the video plus depth data. But there exists three problems [2], which are visibility, resampling, and disocclusion. Multiple pixels of the reference view may fall into the same position in the virtual image plane, which will cause the visibility problem. A Z-buffer algorithm [9] can solve this problem by recording the Z values and choosing the nearest pixel to the virtual camera plane. The phenomenon of an integer pixel position in the reference view image being projected to a subpixel position in the virtual view is called resampling problem, which can be coped with upsampling procedure or backwards warping with interpolation. The remaining disocclusion problem is the fact that some parts of the captured scene, which are occluded in the original views, become visible in the virtual views. It is caused by the lack of scene information occluded by the foreground objects in the original view position. As the distance from virtual view to reference view increases, the disoccluded area becomes larger, as shown in Figure 1.

The disocclusion problem is considered to be the most significant and difficult one of the DIBR algorithm. It is well handled in the interpolation operation [10–13], but will become severe in the extrapolation situation, where the missing image information needs to be reconstructed by appropriate algorithms. Lots of algorithms have been developed to solve this problem, which can be divided into three categories.

The first is layered-depth-image (LDI) [14, 15], which can achieve excellent rendering results by providing sufficient information of the scene. LDI data are composed of a number of color layers and their corresponding depth layers, which contain not only the texture and depth information of visible scene from the front view, but also that of the occluded regions. It is very simple to obtain high-quality multiview images from LDI data. However, the procedure of creating LDI is computationally complex and quite time-consuming. The transmission bandwidth of LDI data also increases drastically with the number of layers. A simplified data format of LDI, which is called the “Declipse” format [16], is proposed by Philips Corporation. The “Declipse” format data consist of foreground layer and background layer. It presents the advantage to improve the rendering quality with a quite small overhead in terms of complexity and bitrate.

The second approach is called depth image preprocessing. To reduce the disoccluded areas in virtual views, low pass filter is applied to smooth the depth image. Fehn [2] uses a suitable Gaussian filter preprocessing the depth image to eliminate the disocclusions with the cost of slightly geometric distortions. An asymmetric smoothing method is proposed by Zhang and Tam [17]. By enlarging the standard deviation and window size of Gaussian filter in vertical direction, the vertical structure distortion is reduced. The filtering effect is to smooth the sharp discontinuities in the depth image, thus reducing the hole areas near object boundaries. A consequence of these algorithms is that the whole depth map has been modified, which will severely blur the distance between scene objects in different depth layers. To cope with depth loss, different kinds of oriented filters [18–21] are designed with the same principle, i.e., smoothing the sharp edge in the depth image locally and keeping the depth of the other regions unchanged. The oriented filters can improve the image quality of the virtual views, but still induce geometric distortion. Although the depth image preprocessing methods can be used to handle the disoccluded regions in the virtual views of small baseline, obvious geometric distortions will occur when the baseline is getting larger.

The third approach to filling the disoccluded areas is image completing techniques. This approach can be further classified into statistical-based methods, partial differential equations (PDE)-based methods, and exemplar-based methods. Statistical-based methods [22–25] have good performance in pure texture synthesis applications, but fail to complete natural images with complex structure. PDE-based methods [26–29], which are also called image inpainting methods, propagate linear structures into the disoccluded areas smoothly via diffusion. The diffusion process is simulated by the PDE of physical heat flow. Inpainting methods are suitable for removing small image artifacts, such as speckles, scratches, and overlaid texts. When the disocclusion is getting larger, the diffusion process will over-smooth the image and cause visible blurring artifacts. Exemplar-based methods [30, 31] fill the hole regions by copying patches with the similar texture from the known neighborhood of the image. Criminisi et al. [30] use the exemplar-based method to remove objects from images. Komodakis and Tziritas [31] propose an efficient belief propagation method to obtain global optimization. Exemplar-based methods have been used for the case of video completion in [32, 33]. Multiple frames are provided as the searching source of best match patch by Cheng et al. [34] to achieve temporal continuity. Exemplar-based methods have been the most powerful techniques for dealing with large disoccluded regions. Schmeing and Jiang [35] first obtain the background information with a computed background model. But their approach cannot handle the uncovered areas caused by static foreground objects. For each virtual view, Ndjiki-Nya et al. [36] use a background sprite to update the texture and depth information of disoccluded areas. There are two major drawbacks of this method. One is the valuable background information of disocclusions, which cannot be reused during the generation of other virtual views. The other is the memory cost increases with the number of virtual views.

In this article, a new virtual view generation method with spatial and temporal texture synthesis is proposed. The structure information of the captured scene in the temporal domain is taken into account by maintaining an accumulated sprite of stationary scene. An oriented exemplar-based inpainting algorithm is applied to restore the rest disoccluded areas with background texture.

The remainder of this article is organized as follows. In Section 2, a brief description of the algorithm framework is given. The details of each processing modules are demonstrated in Sections 3, 4, 5, and 6. Experimental results are compared with state-of-the-art methods in Section 7. The conclusions and future works can be found in Section 8.

The framework of proposed DIBR method with spatial and temporal texture synthesis is shown in Figure 2. The proposed method is divided into four main stages, i.e., stationary scene extraction, backward DIBR, merging operation, and oriented exemplar-based inpainting.

In the first stage, a sprite of stationary scene is maintained throughout the view synthesis process, which stores the temporally accumulated structure and depth information of stationary image part. The Structural SIMilarity index (SSIM) [37] is utilized to distinguish the stationary scene from the moving foreground objects by combining the input depth images. For stationary scene, the SSIM index between adjacent frames is large, so the image part, which is stationary in both adjacent frames, can be extracted by using the SSIM index values. But there still are some stationary scenes, which cannot be distinguished due to the occlusions of moving foreground objects. By considering the spatial relationship provided by the input depth maps, the texture information of these occluded stationary scenes can also be obtained. In the demonstration of our algorithm, the camera of input view is supposed to be still for simplicity. If the camera is moving, an additional camera tracking module needs to be inserted before stationary scene extraction stage to compensate the global motions, which is beyond the discussion in this article.

In the second stage, current frame and stationary scene sprite are warped to the same virtual perspective view by a backward DIBR method to tackle the visibility problem and resampling problem.

The proposed algorithm merges these two virtual images obtained from the second stage together with the third stage. The merging operation needs to be done very carefully, because the foreground objects in virtual views may still exist inner hole pixels. The merging operation can take use of most of the scene information provided by the sprite of stationary scene.

After the merging operation, there still exists a few blank regions without pixel values. In the final stage, oriented exemplar-based inpainting approach is applied to fill the remaining holes by searching best matching exemplar with background texture. Current virtual image is used as the searching source of best matching patch. The filling order of the inpainting method is steered from background structures to foreground objects.

Note that the proposed method only uses the sequence of color images and depth images from one captured view as the input data. If image data of another view are also provided, the switch in the framework can directly be shifted from the extrapolation mode to the interpolation mode without any changes of the framework.

The DIBR algorithm warps the original view to the virtual view position by projecting current pixels to points in real 3D space and re-projecting the 3D points to virtual image plane. Large disocclusions will appear in the discontinuous edges of depth map, which is the transition place between foreground and background in texture image. The background image part occluded by foreground objects should be visible in the virtual views. But the occluded background information is lost during the procedure of recording a 3D scene by a 2D image. To solve this problem, the proposed stationary scene extraction module tries to recover the lost background structure from video sequences. For a video captured by a fixed camera or a short cut of video, the image consists of moving foreground objects and stationary background. The occluded background information in current image frame may appear in frames at other moments. If the information can effectively be used, the filling effect of disoccluded areas will be more convincing.

where p:(i j)corresponds to the pixel of column coordinate i and row coordinate j. I _t and D _t represent the color intensity frame and depth map frame of input original view at time t, respectively. D _t is represented as an 8-bits gray-scale image. The continuous depth range is quantized to 255 discrete depth values. The nearest object to the camera image sensor is assigned with 255 and the farthest object is assigned with 1. Pixels with depth value 0 are denoted as holes. The transform formula between discrete depth level and actual distance in real scene can be found in [12].

where ${\mu }_{{\Phi }_t}$, ${\mu }_{{\Phi }_{t-1}}$ represent the luminance mean value of ${\Phi }_t^I$ and ${\Phi }_{t-1}^I$, respectively. ${\sigma }_{{\Phi }_t}$ and ${\sigma }_{{\Phi }_{t-1}}$ represent the luminance standard deviation of ${\Phi }_t^I$ and ${\Phi }_{t-1}^I$. ${\sigma }_{{\Phi }_{t(t-1)}}$ denotes the luminance correlation coefficient between ${\Phi }_t^I$ and ${\Phi }_{t-1}^I$. K ₁ and K ₂ are constants. The value of K ₁and K ₂ can be determined according to the research work in [37]. The expressions of mean, standard deviation, and correlation coefficient can also be found in [37].

where ${\mu }_t^D$ and ${\mu }_{t-1}^D$, respectively, represent the average depth value of square areas in D _t and D _t-1. The square neighborhoods have the same window size L  ×  L with SSIM computation in Equation (2) and take the coordinates of pixel p as center position. T is a constant threshold, which defines the acceptable range of depth fluctuation. |·| is the absolute function.

where ${\mu }_{\text{TM}}^p$ and ${\mu }_M^p$, respectively, represent the average depth value of square areas in T M _SS and M _SS. The square neighborhoods have the same window size L  ×  L with SSIM computation and take the coordinates of pixel p as center position. T is the same constant threshold defined in Equation (4). Figure 3d shows T C _SS of Figure 3b. Figure 3h,i are C _SS and M _SS of Figure 3b, respectively. Almost all the texture and depth information of stationary scene are restored in Figure 3h,i.

So far, the appeared background information in past frames is stored in C _SS and M _SS, which can be used to partly solve the disocclusion problem of virtual view synthesis algorithm.

The backward DIBR method, which shares the same idea with the inverse warping method in [13], can efficiently eliminate the small cracks in virtual view caused by resampling problem in traditional DIBR process [2]. In general, the backward DIBR method can be divided into two steps: warping the depth map of the reference view to the virtual view position and generating the texture image of the virtual view.

where μ _ND is the average depth value of all the stored warped pixels in pixel q.

The virtual depth map VM _t projected from M _SS and the virtual texture image VC _t projected from C _SS can be obtained by the same backward DIBR method. Two results of our backward DIBR algorithm are given in Figure 4e,f.

To efficiently use the structure information in C _SS, the two virtual texture images (VI _t and VC _t ) need to be merged together. The merged virtual image and its depth map are denoted as MI _t and MD _t , respectively. The virtual view image VI _t is dominated in the merging process. Available background information in VC _t is used to fill the blank areas in VI _t . There may be holes in both foreground and background due to the inaccuracy of depth map, as shown in Figure 4e. We do the merging operation carefully to avoid filling holes in foreground with background structures.

where q _L and q _R represent the first left and first right non-hole pixel in horizontal column, respectively. ${\mu }_D^{q_L}$ and ${\mu }_D^{q_R}$ represent the average depth of the K  ×  K windows which take q _L and q _R as the center pixels in VD _t . T is the same constant defined in Equation (4).

where non-hole flag means there exists a meaningful value in this pixel position. The second condition in Equation (14) defines the situation, i.e., the pixel q is hole in VI _t , but meaningful pixel with available background texture in VC _t . This condition ensures that the holes in foreground objects will not be filled with the accumulated background information in VC _t . F represents the acceptable range of depth fluctuation in merging operation. In Figure 4g, the available texture of stationary background scene in Figure 4f is merged with the virtual image (Figure 4e) rendered from original view and the hole areas in foreground objects are reserved. The corresponding depth value of each non-hole pixel in merged virtual view MI _t is stored in MD _t , and the depth value of each hole pixel is set to zero.

The merging operation can solve the disocclusion problem partly, because the useful background information in C _SS and M _SS is limited. There still exist hole areas in the merged virtual view MI _t , which are divided into two kinds: the foreground holes caused by inaccurate depth map and the blank areas caused by occlusion in original view. The image part with known pixels is defined by Λ, and the remaining hole area is denoted as Γ. The border of hole area Γ is defined as ∂Γ, as shown in Figure 5a.

To restore the missing information of the remaining hole areas, we propose an oriented exemplar-based inpainting algorithm based on the previous work of Criminisi et al. [30]. They determine the filling order of hole pixel h ∈ ∂Γ by assigning each hole pixel a priority P(h). The hole pixel with the highest priority is first filled with the best match patch in Λ. The priority is the product of the confidence term C(h)and the data term D(h). The confidence term enforces to fill hole with large support set of known pixels first, while the data term ensures the continuous propagation of linear structure into hole regions. Noticing the fact that most remaining holes are due to a lack of scene information of the stationary background, we improve their algorithm in two ways. One is filling the border pixel in ∂Γ which is adjacent to background area, first. The other is choosing the texture of known background area to restore the disoccluded regions. The improvements are implemented by considering depth cue in the calculation of the priority term and the energy function, both of which are used for the best exemplar searching procedure.

where BG and FG represent the background areas and foreground objects, respectively. Q is a constant, which should be no less than the maximum of the product of C(h) and D(h). We set Q = 256 in our framework. The new priority term will steer the filling order from background to foreground and keep the advantage of linear structure propagation.

Let r denote the pixel with maximum priority in ∂Γ. The J  ×  J samples patch, which takes r as center, is defined as Ψ. A square area around r with W  ×  W samples is defined to be the searching area Ω. Then the oriented exemplar-based inpainting algorithm needs to search for the best match patch S in Ω, which has the most similar texture with Ψ. The center of S is denoted as s. The corresponding depth areas of Ψ and S are represented by Θ and O, respectively.

where T is a constant as defined in Equation (4).

The best match block in the searching area Ω is obtained by minimizing the energy cost function (17). The first term in energy function (17) represents the texture difference between the known pixels in target patch Ψ and the corresponding pixels in match patch S. In our approach, only the luminance component is considered. The second term in (17) indicates the depth similarity, which has lower importance than the first texture term. The third term is a penalization term. If there exist pixels of foreground objects in the corresponding area of Ψ _u in S, the penalization term will become larger. The likelihood of selecting patches with foreground pixels is greatly reduced by adding the penalization term. According to the definition of the energy function, the patches of the background scene, which contain similar texture and depth structure with the target block, will be selected to restore the missing information of the disoccluded image areas. We applied our oriented exemplar-based inpainting method to synthesize the missing texture information of disoccluded area in Figure 5a. The blank region is filled from background scene to foreground objects, and the linear structure is propagated into the hole in an appropriate way (see Figure 5c–g).

To evaluate the performance of the proposed method, we compare our approach with other methods, including the MPEG view synthesis reference software (VSRS, version 3.5) [38], the depth-based inpainting method in [29], and the Asymmetric Gaussian filtering method of Zhang and Tam [17].

Our experiments are carried out on three test sequences: “Book arrival”, “Breakdancers”, and “Ballet”. These sequences have 100 frames and a resolution of 1024 × 768samples. Multiple video plus depth data from different camera views are available. “Book arrival” sequence is captured by a parallel camera array and the others are obtained by a toed-in camera array. The baseline between two adjacent cameras is approximately 6.5 cm for “Book arrival” sequence and 20 cm for the other two sequences.

Subjective evaluation

The view synthesis results of these three test video sequences are shown in Figures 6, 7, and 8. All of the four presented approaches can handle the visibility and resampling problems and fill the disoccluded areas in virtual view. Our proposed algorithm has the best subjective effects compared to the others three methods.

The Asymmetric Gaussian filtering method causes noticeable geometric distortions. The vertical structure is curved in Figures 6c and 7c. The foreground objects become fat, as shown in Figures 6k and 7g,k. This method will slightly shift the object away from its correct position (see Figure 6g), which will reduce the disparity between reference image and virtual image and decrease the 3D feelings. For the purpose of autostereoscopic display, although the visual quality of Figure 6g is still pleasant, the depth perception of the scene is distorted due to these shifts. The distorted stereo display will make people fill uncomfortable and arouse visual fatigues. The depth-based inpainting method can restore the blank areas with color of background pixels, but induce severe blurring artifacts (see Figures 6l, 7h,l, and 8h) and some color bleeding defects (see Figures 7h,l and 8h). The filling results are very uncomfortable for visual experience. The VSRS method will lead to significant horizontal structure artifacts (as shown in Figures 6i,m, 7i,m, and 8i,m) and decrease the visual quality greatly.

The proposed approach utilizes the accumulated information of stationary scene to fill the disoccluded areas and achieves convincing effect, as shown in Figures 6n, 7j, and 8j. The missing structure of blank regions is restored with the true background structure. Even for the disoccluded areas caused by stationary foreground objects, our proposed method can obtain plausible filling results. As shown in Figures 6j and 7n, the hole areas are filled with the texture of background scene without losing the sharpness compared to Figure 6h,l. Figure 8l gives better visual effect than Figure 8n. Because the man’s leg is very close to the wall in Figure 8b, it is difficult to distinct the leg from the wall. In Figure 8n, our approach wrongly fill the hole with texture of the wall. Another important advantage of our approach is the temporary texture consistency of the filled disoccluded regions. For disoccluded areas caused by moving foreground objects, the missing texture is recovered from other frames. The true texture information in other frames is extracted and used to restore the hole areas. To demonstrate the consistency in temporal direction, a series of magnified virtual image subsection for “Ballet” sequence is shown in Figure 9. The disoccluded regions around the woman of adjacent frames are restored by the same true background structure, then the texture of filled image areas maintains consistent in time direction.

Objective comparison

We adopt peak-signal-to-noise ratio (PSNR) and SSIM [37] to compare the performance of proposed approach with the other three methods.

For every test case of each sequence, the PSNR and SSIM values are calculated for the whole image region of every virtual image frame. The mean values of PSNR and SSIM for each test case are stored in Table 2 and the best results are highlighted with boldface type. The “Camera” column indicates camera configuration of virtual view generation, i.e., “8→9” means synthesizing virtual view of the 9th camera’s perspective position from the 8th camera.

		PSNR (dB)				SSIM
Seq.	Camera
			VSRS	Depth-based	Asym.		VSRS	Depth-based	Asym.
		Prop.	3.5	inpainting	filter	Prop.	3.5	inpainting	filter
			[ [38]]	[ [29]]	[ [17]]		[ [38]]	[ [29]]	[ [17]]
Book arrival	8→9	32.91	32.85	32.69	28.85	0.9814	0.9803	0.9798	0.9500
	10→9	32.12	31.55	32.03	28.24	0.9817	0.9766	0.9811	0.9465
	8→10	29.74	29.50	29.53	28.85	0.9672	0.9647	0.9645	0.8876
	10→8	28.92	28.57	28.79	25.00	0.9684	0.9628	0.9660	0.8909
Break dancers	4→3	31.91	30.13	31.45	26.56	0.9470	0.9323	0.9440	0.8832
	3→4	31.94	29.67	31.78	26.70	0.9518	0.9357	0.9500	0.8848
	5→4	32.58	28.74	32.06	27.16	0.9532	0.9340	0.9503	0.8867
	5→6	32.47	31.51	32.12	26.59	0.9503	0.9441	0.9482	0.8876
Ballet	3→4	30.10	28.20	29.63	22.58	0.9388	0.9111	0.9288	0.8280
	5→4	31.91	26.87	31.85	23.28	0.9436	0.9151	0.9403	0.8304
	5→3	27.74	24.41	25.92	20.17	0.8884	0.8575	0.8820	0.7517
	3→5	27.38	25.98	27.10	20.82	0.8799	0.8447	0.8683	0.7489

This table shows the PSNR and SSIM values of four view synthesis methods. The best results are highlighted with boldface type.

From Table 2, we can observe that among these four methods the proposed framework has the best PSNR and SSIM performance for both the parallel and toed-in camera configuration. The Asymmetric Gaussian filtering method gets the lowest PSNR and SSIM values due to the geometric distortion. For the four test cases of “Book arrival” sequence, the baselines between the virtual view and reference view are small (6.5–13 cm). Because the holes around image boundary occupy great percentage of the whole disocclusions (see Figure 6b), the PSNR and SSIM gains of our proposed framework are small, i.e., 0.09–0.22 dB for PSNR and 0.0006–0.0027 for SSIM compared to depth-based inpainting method. For the test cases of “Breakdancers” and “Ballet” with large baseline (20–40 cm), our proposed approach obtains larger PSNR and SSIM gains compared to depth-based inpainting method, i.e., 0.16–1.82 dB for PSNR and 0.0018–0.0116 for SSIM. There are two important reasons for the improvements of PSNR and SSIM in our proposed framework. One is the available structure information from the stationary scene sprite; the other is the oriented exemplar-based inpainting process with reasonable filling orders. Figure 10 shows the PSNR and SSIM curves for two test cases. One is the virtual view of “Ballet” sequence, which is generated from the 3nd camera to the 4th camera. The other is the virtual view of “Breakdancers” sequence, which is generated from the 5th camera to the 4th camera.

Figure 11 gives the PSNR curves for a local area of “Book arrival” sequence. The concerned local area is the same subsection shown in Figure 6n. From the 1st frame to the 31st frame, the local area only covers background objects, so the performance is very close for these three algorithms. From the 32nd frame to the 99th frame, the local area contains not only background objects but also foreground objects. Then the disoccluded regions appear in the concerned local area due to the discontinuity of the depth. With the proposed stationary scene extraction algorithm, the true texture information of the background objects is utilized to recover the disoccluded regions. The temporal consistency of texture and structure is maintained for these frames using our algorithm. Compared to the VSRS and the depth-based inpainting algorithm, the fluctuation of the PSNR values is much smaller for the proposed method (as shown in Figure 11), which means that the temporal consistency of the rendered sequence is improved. It is obvious that the PSNR value drops at the 32nd frame and the 51st frame due to the sudden depth change in the input sequence. To obtain a more consistent rendered sequence, a temporal filtering procedure for the input depth sequence is beneficial.

This article presents a novel DIBR method combined with spatial and temporal texture synthesis. By maintaining a sprite of stationary scene of the original sequence, the useful structure information can be adopted to restore the missing texture of disocclusions in virtual view images. The remaining disoccluded areas are restored by proposed oriented exemplar-based inpainting approach. The oriented exemplar-based inpainting method fills the rest hole areas from background to foreground and propagates the structure and texture into the blank regions in an appropriate way. Combining these two algorithms, the proposed DIBR method solved the disocclusion problem well and achieved the spatial and temporal consistency. These features make the proposed approach very suitable for extrapolation of virtual view synthesis. Meanwhile, the proposed framework has the flexibility of shifting to the interpolation operation. Theoretical analysis and experimental results show that the proposed method outperforms state-of-the-art view synthesis methods. The increase of runtime cost is moderate and acceptable. Our future work will focus on the research of camera tracking and motion compensation to extend our proposed method to the situation with moving cameras.