Unified piecewise epipolar resampling method for pushbroom satellite images

Abstract

Computational stereo is in the fields of computer vision and photogrammetry. In the computational stereo and surface reconstruction paradigms, it is very important to achieve appropriate epipolar constraints during the camera-modeling step of the stereo image processing. It has been shown that the epipolar geometry of linear pushbroom imagery has a hyperbola-like shape because of the non-coplanarity of the line of sight vectors. Several studies have been conducted to generate resampled epipolar image pairs from linear pushbroom satellites images; however, the currently prevailing methods are limited by their pixel scales, skewed axis angles, or disproportionality between x-parallax disparities and height. In this paper, a practical and unified piecewise epipolar resampling method is proposed to generate stereo image pairs with zero y-parallax, a square pixel scale, and proportionality between x-parallax disparity and height. Furthermore, four criteria are suggested for performance evaluations of the prevailing methods, and experimental results of the method are presented based on the suggested criteria. The proposed method is shown to be equal to or an improvement upon the prevailing methods.

1 Introduction

The paradigm of computational stereo [1] is that it proceeds following six major steps: image acquisition, camera modeling, feature acquisition, image matching, distance or depth determination, and interpolation. It is very important to determine appropriate epipolar constraints as a preprocessing step for image matching after the camera or sensor modeling because many stereo image processing algorithms use these constraints to extract three-dimensional information such as positioning and surface data. Among the conditions determined using epipolar constraints are restrictions on the search space for stereo matching, identification and elimination of blunders or outliers from the matching results, and others.

It is well known that stereo images acquired by frame camera(s) have epipolar geometry, that is, there exist epipolar lines and the epipolar pairs on each line throughout the space. However, it has been shown [2] that the epipolar geometry of linear pushbroom camera images are not lines but hyperbola-like non-linear curves because of the non-coplanarity of line of sight vectors. Therefore, an epipolar curve for the entire image can only be approximated incrementally in linear segments, and epipolar pairs exist only locally.

Several studies have been conducted to establish the epipolar geometry of the pushbroom camera to achieve effective epipolar image resampling. For example, Oh et al. [3] proposed a piecewise approach to this problem with rational polynomial coefficient (RPC); this is an image space-based approach and was shown to achieve almost zero y-parallax, as well as a linear relationship between the x-parallax and the ground height. Alternatively, Wang et al. [4] suggest a method that implements a new epipolarity model based on the projection reference plane (PRP); this is an object space-based approach, and experimental results showed that the vertical parallaxes all attained sub-pixel levels in along-track and cross-track stereo images from actual equipment, including Earth observation satellites SPOT-5, IKONOS, IRS-P5, and QuickBird.

Because zero y-parallax is one of the most important features of epipolar constraints in stereo image processing applications, zero y-parallax should be used as an evaluation criterion. Therefore, the experimental results were presented to show whether the proposed algorithms satisfied this criterion and could be considered valid for use in applications.

In this paper, we suggest a modified piecewise epipolar resampling method to obtain an epipolar geometry that can be effectively applied to construct stereo image pairs from linear pushbroom sensors. The modified algorithm is based on the piecewise approach [3, 4]; however, in the proposed method, the distance in the epipolar image space is determined by the distance in the object space, as described in detail in Section 3.

Furthermore, we suggest four performance evaluation criteria for epipolar resampling algorithms: (1) geo-positioning accuracy of the resampled epipolar images relative to the original sensor model, (2) square pixel scale and perpendicularity of axis in object space, (3) near zero y-parallax in epipolar image space, and (4) proportionality between the disparity in the x-parallax and the ground height. Using these criteria, performance evaluations of the previously suggested algorithms and the algorithm proposed in this paper are presented based on experimental data from satellite images, including optical images such as those from IKONOS, Pleiades, and WorldView, as well as synthetic aperture radar (SAR) images as those from KOMPSAT-5 and TerraSAR-X.

2 Previous studies

2.1 Image space-based method

Oh et al. [3] suggested a piecewise approach to epipolar resampling of pushbroom satellite images with RPC [5, 6]. It was based on the researches presented by Habib et al. [7] and Kim [2]. These previous works showed that epipolar curves of linear pushbroom images are not lines but hyperbola-like non-linear curves and that an epipolar curve for the entire image can only be approximated by piecewise linear segments, with epipolar curve pairs existing only locally. Based on the previous results, they showed locally existing epipolar pairs for the entire space represented by the epipolar curve.

Figure 1 shows the essential elements of this method. For a selected point in the left image, p, the corresponding control point in the right image for pat the maximum ground height, h_max, is q1, and q2 is the corresponding control point for p at the minimum ground height, h_min. The curve between q1 and q2 in the right image can be approximated as a line segment. Likewise, the piecewise epipolar curve in the left image for q1, now a reference point in the right image, can be established in the same way, from control points p′ and p′′ This process is continued until the entire spaces covered by the left and/or right images are delineated.

Fig. 1

Piecewise epipolar curve pair generation [3]

Figure 2 shows all of the extracted points on the piecewise epipolar curves for the left and right images. Within this epipolar image space, the control points on each approximately horizontal line should be relocated to a truly horizontal line, and the intervals between points should all have the same pixel distance, guaranteeing (1) almost zero y-parallax, (2) x-parallax proportional to height.

Fig. 2

Extracted points on the piecewise epipolar curves [3]

This method efficiently generates piecewise resampled epipolar image pairs because most of the steps are performed in original or epipolar image space, except when calculating ground coordinates. They also described that the starting point in each epipolar curve, which is marked with triangle in Fig. 2, can be established along the orthogonal direction to the trajectory, i.e., orthogonal to the fitted line using the first epipolar curve generated, and a predefined interval. This process may produce unexpected results, such as a non-square pixel scale or a non-orthogonal axis in object space, discussed more detail in subsequent sections.

2.2 Object space-based method

Wang et al. [4] suggested a new epipolarity model based on the PRP in object space, which was first introduced using a virtual horizontal plane (VHP) [8]. Like the image space-based method, the PRP method also extracts control points from the original image space but assigns the control points’ coordinates to new coordinates on a PRP in object space instead of image space. This method does not require any ground control points, which is essential to the method suggested by Morgan et al. [9]. And this method may not cause a non-orthogonal axis in the object space.

The generated epipolar curve image points are first mapped on the PRP, which is defined such that points on the PRP have the same elevation in the local vertical coordinate system. After determining the direction of the approximated epipolar curve, only an affine transformation is needed to generate resampled epipolar images. Figure 3 shows the coordinate transformation between the epipolar images and the original stereo imagery.

Fig. 3

Coordinate transformation relation of epipolar and the original images [4]

Wang et al. described experimental results demonstrating feasibility of this method, with residuals from the fitted straight lines of less than 1 mm, and a root mean square of the vertical parallax of less than half of a pixel. Unlike the image space-based method, the PRP method uses the distance on the PRP as the pixel distance between the neighboring control points in epipolar image space; as a result, the horizontal parallax may not be proportional to the ground height.

3 Proposed method

This section describes a modified algorithm to perform piecewise epipolar resampling that ensures (1) preservation of the geo-positioning accuracy of the original sensor model in the resampled epipolar images, (2) scales with equidistant pixels and perpendicularity of the axis, (3) near zero y-parallax, and (4) proportionality between the disparity in the x-parallax and the ground height. The proposed algorithm is a unified one because it can be applied to both along-track and cross-track stereo images and can also be applied to SAR stereo images for radargrammetry, as shown by experimental results reported in Section 5. The prevailing methods can also be applied to all of these types of stereo images, but the proposed one uses only the CSM [10] interface, stated in Section 3.2, so that it can be applied to all of stereo image pairs regardless of the sensor types, and furthermore, it shows better performance in terms of the four performance evaluation criteria.

The basic steps of the proposed algorithm are shown in Fig. 4. The most significant innovation of the proposed algorithm is implemented in the first and the third step. In the first step, a starting control point on each horizontal line is determined such that it is located on the orthogonal line in the object space. In the third step, the generated epipolar curve image points are relocated to epipolar image space with the distances in the object space such that the two axes are perpendicular to each other and the image points satisfy the epipolar constraint.

Fig. 4

Basic steps of the proposed algorithm for piecewise epipolar resampling

3.1 Determine a starting control point

The first step is to determine a starting control point in the reference image to generate the control points on each epipolar curve pair. For example, the starting control point is p in Fig.1. The first starting control point can be determined as the one in the center of the reference image. The image control point ${I}_{00}^L$in Fig. 5 is a starting control point, and ${I}_{01}^L$ and ${I}_{02}^L$ are the neighbor image control points of ${I}_{00}^L$ on the same epipolar curve. To fit $\overline{I_{01}^L{I}_{00}^L}$ and $\overline{I_{00}^L{I}_{02}^L}$to a horizontal linelin epipolar image space, the next step is to determine two vertical points, i.e., two starting control points, on a vertical line v passing through the control point ${I}_{00}^L$, where l ⋅ v = 0. This vertical line can be approximately computed as shown in Fig. 5. The line l passes ${I}_{01}^L$ and ${I}_{02}^L$, and the line v is perpendicular to l. The two vertical points, ${I}_{10}^L$ and ${I}_{20}^L$, are selected among the points on the line v, and the two intervals, $\overline{I_{00}^L{I}_{10}^L}$ and $\overline{I_{00}^L{I}_{20}^L}$, can be determined to be the same as the interval between ${I}_{01}^L$ and ${I}_{00}^L$. The interval determination is not important in this step because all the distances between points, horizontal and vertical, are here calculated in the local reference plane prior to their relocation to epipolar image space, as detailed in the next section.

Fig. 5

Determination of two starting control points

3.2 Extract image control points on piecewise epipolar curve

The second step is to generate piecewise epipolar curve image points from original stereo images. As with the previously reported methods, this step is based on the fact that the hyperbola-like epipolar curve for a reference image point exists between two control points that correspond to the minimum and maximum ground height of the reference image point.

As shown in Fig. 6, a point ${I}_{00}^L\left({r}_{00}^L,{c}_{00}^L\right)$ in the reference image (left or right) is selected as the reference image point for determination of a piecewise epipolar curve with${G}_{00}^{\min}\left({X}_{00}^{\min },{Y}_{00}^{\min },{Z}^{\min}\right)$and${G}_{00}^{\max}\left({X}_{00}^{\max },{Y}_{00}^{\max },{Z}^{\max}\right)$ describing the reference image point at minimum and maximum ground height Z ^min and Z ^max, respectively. This can be done using one of the community sensor model (CSM) [10] application program interface (API) commands, “image2Ground(),” or any other type of rigorous sensor model or RPC [6].

Fig. 6

Piecewise epipolar curve generation using the proposed algorithm

The corresponding control points, ${I}_{00}^R\left({r}_{00}^R,{c}_{00}^R\right)$ and ${I}_{01}^L\left({r}_{01}^L,{c}_{01}^L\right)$, in the stereo image pair (right or left) for ${G}_{00}^{\min }$and${G}_{00}^{\max }$ are computed by direct coordinate transformation, using the command “ground2Image()” in CSM APIs. Subsequently, ${I}_{01}^L\left({r}_{01}^L,{c}_{01}^L\right)$ and ${I}_{02}^L\left({r}_{02}^L,{c}_{02}^L\right)$are computed for${I}_{00}^R\left({r}_{00}^R,{c}_{00}^R\right)$and${I}_{00}^R\left({r}_{00}^R,{c}_{00}^R\right)$using the ground points${G}_{01}^{\min}\left({X}_{01}^{\min },{Y}_{01}^{\min },{Z}^{\min}\right)$and${G}_{01}^{\max}\left({X}_{01}^{\max },{Y}_{01}^{\max },{Z}^{\max}\right)$, respectively. Successive iterations of this process continue until one of the image coordinates extends beyond the boundary of the reference image space.

The coordinate transformation between the object space and the original image space is denoted by the following equations, in which i is the i th line, j is thejth sample in i th line, and ${I}_{ij}^L$ and ${I}_{ij}^R$are the image coordinates in the left and right image spaces, respectively.

$$ \begin{array}{ll}{I}_{ij}^L=\hfill & \left\{\begin{array}{l}\mathrm{ground}2\mathrm{Image}\left({G}_{ij}^{\min}\right),\kern0.75em \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{odd}\hfill \\ {}\mathrm{ground}2\mathrm{Image}\left({G}_{i\left(j-1\right)}^{\max}\right),\ \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{even}\hfill \end{array}\right.\kern1em i\ge 0,\ j>0\hfill \\ {}{I}_{ij}^R=\hfill & \left\{\begin{array}{l}\mathrm{ground}2\mathrm{Image}\left({G}_{ij}^{\min}\right),\kern0.75em \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{even}\hfill \\ {}\mathrm{ground}2\mathrm{Image}\left({G}_{i\left(j-1\right)}^{\max}\right),\ \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{odd}\hfill \end{array}\right.\kern1em i\ge 0,\ j>0\hfill \\ {}{G}_{ij}^{\min }=\hfill & \left\{\begin{array}{l}\mathrm{image}2\mathrm{Ground}\left({I}_{ij}^R,\ {Z}^{\min}\right),\ \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{odd}\hfill \\ {}\mathrm{image}2\mathrm{Ground}\left({I}_{ij}^L,{Z}^{\min}\right),\ \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{even}\hfill \end{array}\right.\kern1em i\ge 0,\ j\ge 0\hfill \\ {}{G}_{ij}^{\max }=\hfill & \left\{\begin{array}{l}\mathrm{image}2\mathrm{Ground}\left({I}_{i\left(j-1\right)}^R,\ {Z}^{\max}\right),\ \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{odd}\hfill \\ {}\mathrm{image}2\mathrm{Ground}\left({I}_{i\left(j-1\right)}^L,{Z}^{\max}\right),\ \mathrm{if}\ \mathrm{j}\ \mathrm{is}\ \mathrm{even}\hfill \end{array}\right.\kern1em i\ge 0,\ j\ge 0\hfill \end{array} $$

(1)

3.3 Relocate the extracted points to epipolar image space

The next step is to relocate the control points to epipolar image space. The point ${I}_{00}^L$and its conjugate point ${I}_{00}^R$are relocated to the origin in epipolar image space. Note that ${I}_{00}^R$ represents the conjugate point for ${I}_{00}^L$at the minimum ground height; however, it can be any point between ${I}_{00}^R$ and ${I}_{01}^R$.

To relocate all of the control points to epipolar image space requires the determination of all of the intervals between neighboring control points on the horizontal lines approximated by each piecewise epipolar curve. In addition, the intervals between all of the neighboring control points on the vertical lines have to be computed in the same way.

A local reference plane is used to compute the intervals between control points. As shown in Fig. 7, the distance between two neighboring control points is computed on a local reference plane with ground height Z ^ref. For example, ${G}_{L_{0j}}^{\mathrm{ref}}$ is the ground point where the control point ${I}_{0j}^L$ is projected to a local plane with ground height of Z ^ref. The distance between ${G}_{L_{00}}^{\mathrm{ref}}$ and ${G}_{L_{01}}^{\mathrm{ref}}$ can be calculated, and this distance is used to determine the pixel distance between ${I}_{00}^L$ and ${I}_{01}^L$in epipolar image space. All of the pixel distances are determined in this way.

Fig. 7

Determination of epipolar image coordinates using ground space organized as a local reference plane

3.4 Determine the image transformation function

In the previous step, all of the original control points were relocated to epipolar image space, where they become epipolar image points, expressed as ${I}_{ij}^{L\hbox{'}}\left({r}_{ij}^{L\hbox{'}},{c}_{ij}^{L\hbox{'}}\right)$ and ${I}_{ij}^{R\hbox{'}}\left({r}_{ij}^{R\hbox{'}},{c}_{ij}^{R\hbox{'}}\right)$. Subsequently, the polynomial image transformation function, as well as the reverse transformation function, can be established as follows.

$$ \begin{array}{l}{r}_{ij}^{\hbox{'}}=f(a)={a}_1+{a}_2{r}_{ij}+{a}_3{c}_{ij}+{a}_4{r}_{ij}{c}_{ij}+\cdots +{a}_{\frac{\left(n+1\right)\left(n+2\right)}{2}-1}{r}_{ij}^n+{a}_{\frac{\left(n+1\right)\left(n+2\right)}{2}}{c}_{ij}^n\hfill \\ {}{c}_{ij}^{\hbox{'}}=g(b)={b}_1+{b}_2{r}_{ij}+{b}_3{c}_{ij}+{b}_4{r}_{ij}{c}_{ij}+\cdots +{b}_{\frac{\left(n+1\right)\left(n+2\right)}{2}-1}{r}_{ij}^n+{b}_{\frac{\left(n+1\right)\left(n+2\right)}{2}}{c}_{ij}^n\hfill \end{array} $$

(2)

The coefficients of each polynomial function can be computed using the least squares method. The order of the polynomial function can be adapted to achieve the desired precision for the image transformation. We used the fifth order polynomial function for the performance evaluation of the algorithms.

3.5 Calculate new sensor models for resampled epipolar image pair

It is required to calculate or generate new sensor models for each resampled epipolar image; for example, new RPCs can be easily generated from the equation proposed by Tao and Hu [5]. First, the ground coordinates in a cubic grid are generated within the effective ground space of the original stereo image pair. Subsequently, epipolar image coordinates for the pair are computed using a “ground2Image()” for each. It is possible to compute image coordinates in epipolar image space using the image transformation function established in the previous section. It has been shown that both the replacement sensor model (RSM) [11] and the RPC [12] can express the relationship between ground and image space for most of sensor types.

A strict requirement of the process of determining the image transformation function is that the function must preserve geo-positioning accuracy with respect to the original sensor model. Therefore, the newly generated RPCs should maintain the relationships expressed by the forward and reverse coordinate transformation functions.

4 Performance evaluation criteria and analysis procedure

As stated in the introduction, we suggest four performance evaluation criteria for piecewise epipolar resampling algorithms: geo-positioning accuracy, pixel scale and perpendicularity of axis in object space, near zero y-parallax in epipolar image space, and proportionality between x-parallax disparity and ground height. The third and the fourth criteria have already been mentioned in several previous studies [3, 4, 7, 8].

4.1 Geo-positioning accuracy

The geo-positioning accuracy of the epipolar resampling algorithm can be measured by comparing the original and generated sensor models’ forward coordinate transformation results for epipolar image point pairs. Ground coordinates, G, extracted from the cubic grid in the effective ground space are used to compute image coordinates from each of the original reference image points, and the image coordinates are relocated into epipolar image space using the image transformation polynomial function. New ground coordinates, G′, can be calculated from these epipolar image coordinates using the newly generated RPCs. The differences between the coordinates, G-G′, represent the geo-positioning accuracy of the resampled epipolar image pair.

4.2 Pixel scale and perpendicularity of axis in object space

Certain satellite images, especially SAR images, have different pixel scales, and their pixels are not always square, which inhibits manual stereo plotting. To confirm the degree to which the generated pixel scale is square, the pixel scale in meters of the grid points extracted within the effective ground space is computed along the x- and y-axis of the epipolar image space using the generated RPCs. To determine the perpendicularity of the axis in object space, the angle between the x- and y-axis in epipolar image space can be converted into different ground spaces including local tangential coordinate space and others.

4.3 Near zero y-parallax in epipolar image space

y-parallax between the resampled epipolar image pair can be measured by a stereo matching algorithm. All of the reliable matched image points from epipolar image pair are used to compute y-parallax. The points, which are used in stereo matching, are extracted using the Harris corner detector and the bi-directional area-based stereo matching algorithm with sub-pixel accuracy is used to get more reliable result.

4.4 Proportionality between disparity and height

Combined with the criteria for a square pixel scale, proportionality between the x-parallax disparity and ground height provides more favorable condition for stereo image processing. For example, in the field of computer vision technology, many dense stereo matching algorithms [13] are implemented to compute depth values from x-parallax disparity.

The proportionality of the x-parallax disparity and ground height can also be validated by area-based stereo matching algorithm described in the previous section. The x-parallax disparity between the conjugate image points from resampled epipolar image pair and the ground heights of the corresponding ground points, which can be calculated by the newly generated RPCs, are investigated the proportionality between them. The difference between the fitted lines is used as a performance metric for these criteria.

5 Performance evaluation of previous and proposed methods

In this section, we describe the experimental results from three different algorithms: the image space-based method proposed by Oh et al., the object space-based method proposed by Wang et al., and the method proposed in this paper. As detailed in the preceding section, the image space-based method uses pixel distance to reallocate reference image coordinates to epipolar image coordinates, and the object space-based method uses distance on the PRP. The suggested algorithm uses ground distance on the local plane.

5.1 Data set

For experimental verification, we used stereo image pairs from five different satellites: IKONOS-2(I), WV-2(W), Pleiades(P), Kompsat-5(K), and TerraSAR-X(T). These satellites cover different areas in South Korea and have been acquired between the years 2002 and 2014. The acquisition dates, image sizes, and nominal ground sample distance (GSD) for these images from these satellites are shown in Table 1, and the areas they cover are shown in Fig. 8.

Table 1 Features of stereo image pairs from different satellites for experimental evaluation

Fig. 8

Areas covered stereo image pairs from different satellites for experimental evaluation

5.2 Results by the proposed method

Figures 9, 10, 11, 12, and 13 show a segment of an IKONOS-2 anaglyph image generated by the proposed method, in which the horizontal line indicates that near zero y-parallax is achieved using the proposed method.

Fig. 9

Epipolar resampled stereo image pair generated by the proposed method (IKONOS-2). a Left image. b Right image. c Segment of anaglyph stereo image pair

Fig. 10

Epipolar resampled stereo image pair generated by the proposed method (WorldView-2). a Left image. b Right image. c Segment of anaglyph stereo image pair

Fig. 11

Epipolar resampled stereo image pair generated by the proposed method (Pleiades). a Left image. b Right image. c Segment of anaglyph stereo image pair

Fig. 12

Epipolar resampled stereo image pair generated by the proposed method (KOMPSAT-5). a Left image. b Right image. c Segment of anaglyph stereo image pair

Fig. 13

Epipolar resampled stereo image pair generated by the proposed method (TerraSAR-X). a Left image. b Right image. c Segment of anaglyph stereo image pair

Figure 14 illustrates the line fitting result showing proportionality between the x-parallax disparity and ground height of the resampled epipolar IKONOS image pair. In Fig. 14, the x-axis represents the disparity or the pixel distance between the matched points, and the y-axis represents the ground height. As is apparent from the figure, the slope of the line indicated by the 152 matched points is 1.53544 and negligible differences between actual ground heights and those computed from the x-parallax disparity.

Fig. 14

Proportionality between x-parallax disparity and ground height

Figure 15 shows an image matching result between the left and right images of a resampled epipolar IKONOS-2; (a) and (b) are the left and the right image, respectively, and (c) is again the left image which shows the reversed matching result. The image coordinates and the normalized correlation coefficient are also shown in the figure.

Fig. 15

An image matching result between the left and right epipolar image pair from IKONOS-2. a, b the left and the right image, respectively. c The left image which shows the reversed matching result

5.3 Geo-positioning accuracy

Geo-positioning accuracy can be evaluated by comparing the ground coordinates from the source image pair with those from the epipolar image pair using the generated RPCs. The differences between these ground coordinates for all three methods are shown in which it can be concluded that all of the three methods preserve the original geometric quality.

The geo-positioning accuracy is affected by the accuracy of image transformation and RPC fitting results. As shown in Fig. 16, the third or higher order polynomial function gives more accurate results than the second or lower order polynomial function. The fifth order polynomial function is used to transform original images to epipolar resampled images for all experiments.

Fig. 16

Accuracy of image transformation function according to the order

As shown in Table 2, the geo-positioning accuracy can be preserved, that is, the difference of geo-positioning results from before and after the resampling process is negligibly small.

Table 2 Geo-positioning accuracy of each method (unit: meter)

5.4 Pixel scale and perpendicularity of axis in object space

The pixel scales and the perpendicularity of the axes in object space for all three methods are shown in Tables 3 and 4. Regularly spaced image points are selected to compute pixel scale from epipolar resampled stereo image pair. The distance between the selected image point and the upper/right image point can be computed in meters. And, at each image point, axis angle between the two axes can be computed in ground space.

Table 3 Pixel scale of each method (unit: meter)

Table 4 Axis angle (unit: degree) in object space in each method

Because the image space-based method uses image space to relocate image control points, the pixel scale and the axis angle of the epipolar image pair depend on the source images. In contrast, the results show that the object space-based and the proposed methods can generate epipolar image pairs with almost square pixel scale and perpendicular axes.

5.5 Epipolarity and y-parallax

The y-parallax for each method was computed for each test data set by sub-pixel stereo matching, and the results are shown in Table 5. In this regard, all of the three methods show favorable performance for the generation of stereo image pair with epipolar constraints.

Table 5 Epipolarity in terms of y-parallax for each method

As explained in Section 4.3, bi-directional area-based stereo matching is used to determine conjugate points. Initially, epipolar image space is divided into 30 × 30 blocks, and three Harris corner points are extracted from each block (Fig. 17). Nine hundred corner points are used to extract matching points. The matched result whose normalized cross correlation value is below threshold, 0.8~0.9, is pruned. For example, in Fig. 15, 0.92 in (b) and 0.93 in (c) represents normalized cross correlation value. The unexpected matching results which caused by some changes between stereo images were manually discarded. For example, the matched results from the feature point extracted on the corner of floatages on the surface are eliminated.

Fig. 17

Image matching result to evaluate epipolarity and y-parallax (IKONOS-2). a Harris corner points. b, c Matching results after outlier pruned, left and right, respectively

5.6 Proportionality between x-parallax disparity and ground height

Table 6 shows the proportionality between x-parallax disparity and ground height for each method. The image space-based and the proposed methods generate resampled epipolar image pairs in which x-parallax disparities are proportional to the ground height. In contrast, the object space-based method uses a local tangential plane with a single origin for the entire space; its x-parallax disparity is not always proportional to the ground height.

Table 6 Proportionality between x-parallax disparity and ground height for each method

6 Conclusions

In this paper, we have suggested a practical and unified piecewise epipolar resampling method and have described the experimental results based on the suggested performance criteria. The proposed method showed performance equivalent to other methods in geo-positioning accuracy and near zero y-parallax. In addition, it showed an improved result in its square pixel scale, axis angle, and proportionality between x-parallax disparity and height. Table 7 shows the summary of the performance evaluation results.

Table 7 Summary of the performance evaluation results

The proposed method will be practical and useful in generating stereo image pairs for the many applications that employ dense matching algorithms [13], because the proposed method efficiently yields constraints on the search space, with near zero y-parallax and the x-parallax bounded by the minimum and maximum heights using any type of sensor models.