 Research
 Open Access
 Published:
A closed form unwrapping method for a spherical omnidirectional view sensor
EURASIP Journal on Image and Video Processing volume 2013, Article number: 5 (2013)
Abstract
This article proposes a novel method of image unwrapping for spherical omnidirectional images acquired through a nonsingle viewpoint (NSVP) omnidirectional sensor. It has three key steps i.e. (1) calibrate the camera to obtain parameters describing the spherical omnidirectional sensor, (2) map world points onto mirror points and, subsequently, onto image points, and (3) set up the projection plane for the final image unwrapping. Based on the projection plane selected, the algorithm is able to produce three common forms of unwrapping, namely cylindrical panoramic, cuboid panoramic, and ground plane view using closed form mapping equations. The motivation for developing this technique is to address the complexity in using a NSVP omnidirectional sensor and ultimately encouraging its application in robotics field. One of the main advantages of a NSVP omnidirectional sensor is that the mirror can often be obtained at a lower price as compared to the single viewpoint counterpart.
1 Introduction
The omnidirectional view sensor has gradually emerged as a popular and effective vision sensor in the field of robotics. In most cases of application, the large field of view (FOV) provided by the sensor allows simultaneous monitoring of the surrounding in different view angles and therefore enables a more flexible and responsive algorithmic behaviour. Among the several configurations, the catadioptric version has received relatively more attention than its dioptric counterpart. Rees [1] first suggested that a hyperboloidal mirror mounted on a perspective camera would enable a 360° FOV on the camera. Subsequently, it was realised by Yamazawa et al. [2] and concurrently several other types of mirror profile were also introduced, such as conical [3], spherical [4], and paraboloidal [5]. ^{a} A summary of the omnidirectional view sensor classification is provided in Figure 1.
However, as a tradeoff for a large FOV, the mirror’s curvature causes an unfavourable distortion in omnidirectional images. Therefore, a preprocessing termed “unwrapping”, is often necessary for a perspectivelycorrect alteration on the image acquired. Mirrors with a single viewpoint (SVP) property [5, 6] such as hyperboloidal, paraboloidal, and a specifically designed mirror [7] for example, can be represented by a common sphere model [8] that allows effective and unified camera calibration techniques [9–18] or otherwise by approximation models [19, 20]. These calibration methods subsequently allow unwrapping on the images acquired. For instance, one of the approaches was presented by Lei et al. [21], where they had demonstrated two forms of panoramic unwrapping—cylindrical and cuboid.
Calibration on nonSVP (NSVP) mirror profiles, such as those of conical and spherical, usually require specific modelling based on the particular mirror shape used [22–24], by approximation model [22, 23, 25], or otherwise, for example, by the use of polarised imaging [26]. NSVP mirrors are popular choices of catadioptric omnidirectional sensors due to the immediate availability and lower cost, particularly for the spherical one. Further evaluation on the advantage of spherical mirror profile is provided in Section 2. Among the earlier related works on NSVP omnidirectional image unwrapping, Gaspar et al. [27, 28] had worked on unwrapping of spherical omnidirectional images into ground plane view (bird’s eye view). Later, Jeng and Tsai [29] proposed a mirrorinvariant technique for panoramic unwrapping by means of groundtruth information calibration.
In this article, a closedform solution for spherical omnidirectional image unwrapping incorporating advantages from the different techniques is presented. Different from previous works, the proposed unwrapping technique (1) does not require any prior knowledge on sensor parameters or groundtruth information, (2) produces output that scales accordingly with the resolution of the image, (3) utilises closed form forward and backward mapping equations, and (4) is designed for multiple output forms such as cylindrical panoramic view, cuboid panoramic view, and ground plane view unwrapping.
The proposed approach attempts ray tracing of the light source in the surrounding captured by the camera of the omnidirectional sensor. By acquiring the mathematical model that describes the direction of ray, the desired form of unwrapping can be achieved by choosing an appropriate 3D projection plane. There are essentially three stages of procedure required in order to complete the unwrapping. The first stage requires the perspective camera of the omnidirectional system to be calibrated and thus an equivalent radius and height of the spherical mirror can be geometrically deduced based on the resolution of the image. The second stage would make use of the parameters to provide a closedform analytical solution to the ray tracing of the imaging system that enables corresponding mapping of world points and their equivalent image points. The last stage of the procedure is to set up the projection plane of the unwrapped image in a 3 dimensional (3D) space.
The rest of the article is organised as follows: In Section 2, the use of a spherical mirror profile is justified despite the complexity introduced, and in Sections 3, 4, and 5, the three key stages are explained. In Section 6, issue on the output quality of the algorithm is discussed. Conclusion of the work is provided in Section 7. Finally, Appendix provides detailed derivations of several important equations used in this article.
2 Evaluation on a spherical omnidirectional sensor
Similar to other NSVP mirror profiles, the spherical mirror profile is often unfavourable in various applications that require mapping between an image point and its corresponding unwrapped counterpart due to the lacking of a practical solution for SVP formation. However, the benefit of utilising the spherical mirror should not be overlooked as it may provide an omnidirectional sensor solution that is justifiably more practical at present. Firstly, a spherical object with a polished surface is easily accessible and so the cost is reasonably low as compared to other SVP mirror profiles because they are mostly custom made using computer numerical control (CNC) machine. Strictly speaking, a SVP formation has rather demanding requirements, which if not met, would render the SVP property of the mirrors an approximation at best. Therefore for most of the time, they are practically of NSVP setting.
Secondly, as a paracatadioptric sensor is not affected by vertical translational error in fabrication [5, 6], the spherical mirror is invariant to rotational error up to a certain degree because it is rotationally symmetrical in all axes as illustrated in Figure 2. Apart from that, since it is not crucial to maintain a specific distance between a NSVP mirror and the camera’s effective focal point along the optical axis, the spherical mirror is also invariant to translational error along the optical axis. In short, the design constraint for a spherical omnidirectional sensor is more relaxed and can tolerate fabrication error.
Thirdly, parameters describing a spherical model are as minimal as two parameters—its radius and its position on the optical axis. As compared to other more complex shapes, its parameter calibration is theoretically simpler and straight forward as described in Section 3.
3 Stage 1camera calibration
Prior to ray tracing, calibration is needed to estimate the two parameters describing the spherical mirror—radius, R, and the distance between its centre and the projection centre, h. Perspective cameras are generally modelled as shown in Figure 3. The parameters describing the model are usually grouped and termed as the intrinsic parameters. These information is useful in mapping the relationship between an image pixel and its corresponding world point in 3D space.
At present, methods to calibrate the intrinsic parameters of SVP omnidirectional cameras with the mirror attached are well established as reported in [10, 19, 30, 31]. For NSVP mirror profiles such as those of conical and spherical, a unified calibration algorithm requires polarisation imaging [26] for instance. However, since the sphere has only two parameters, it can be easily deduced from a calibrated camera instead [24], such as by the use of Zhang’s [32] method. Subsequently, the camera’s intrinsic parameter, K, can be obtained in the following form:
where f are the focal distances with subscript x and y denoting the respective axes, s is the skew of pixel, and c=[u _{ o },v _{ o }]^{T} is the principal point. Without any loss of generality, a sensor with rectangular pixel is assumed where s=0. In practice, the actual sensor in perspective camera may not be necessarily square and will therefore introduce an aspect ratio as in f _{ y }/f _{ x }. However, the two focal distances are often very similar rendering f _{ y }/f _{ x }≈1. With negligible margin of error, we have treated f _{ x }=f _{ y } in our application, and avoided extra computation to correct the input image due to a non 1 : 1 aspect ratio. The calibrated result of the perspective camera is shown in Table 1. For brevity, f _{ x } and f _{ y } are subsequently denoted by f.
When a sphere is projected onto an image plane, a circular feature is obtained as shown in Figure 4. The calibrated f is useful in estimating the radius of the spherical mirror based on the resolution of input image. The groundtruth radius of the sphere is not necessary because with limited quality of the input image, the resolution of the unwrapped image should be logically sufficient based on the resolution of the input image. Therefore, we establish the existence of a virtual sphere, assuming to be located at a constrained distance intersecting the image plane based on the calibrated f. R and h are then remapped from Figure 4 onto the virtual sphere as illustrated in Figure 5.
Prior to parameter estimation, the radius of the circular feature, ρ, on the image plane is first estimated using Hough Circle Transform [33]. Geometrically, it is understood that ρ≠R as illustrated in Figure 5. Due to the viewing angle, ζ, of the perspective projection from C, certain portion of the actual spherical mirror/virtual sphere will not appear on the image plane. To estimate the remapped R, two assumptions are made where (1) the centre of the spherical mirror, therefore the centre of the virtual sphere, coincides the optical axis, and (2) the spherical mirror is at least a “hemisphere” visible to the perspective camera. Although the first assumption may not be necessarily true as fabrication error will always results in misalignment, it is a reasonable approximation [15, 19, 20, 29] and will later greatly simplify the coordinate mapping in Section 4. The remapped R and h can be derived by the method of similar triangles as follows:
From Equations (2) and (3), $\sqrt{1+{t}^{2}}$ is easily observed as a correction factor for the parameters. The estimated R and h will be useful in completing the mapping of world points to equivalent image points, and vice verse. The estimated parameter values for our spherical mirror are shown in Table 2.
In Table 2, the estimated principal point, which is the centre of the circular feature on the image plane obtained using Hough Circle Transform, is easily translated into the centre of the virtual sphere. The error introduced when comparing with that from the intrinsic parameter calibration in Table 1 is marginal. Since the assumption that the optical axis coincides with the centre of the virtual sphere was made, the estimated principal point from Table 2 is used instead of those from Table 1.
4 Stage 2mapping of points
For a typical catadioptric omnidirectional sensor, incident rays would make contact with the mirror, be reflected with respect to the normal axis at the contacted mirror surface, and subsequently enter the pupil of the perspective camera. Figure 5 illustrates the actual image acquisition of a perspective camera at the presence of an effective pupil and effective focal length introduced by a perspective lens with the perspective camera model overlapping over. The perspective camera model has an equivalent structure where the centre of projection is analogous to the effective pupil. With this constraint, ray tracing from the image plane to an equivalent world coordinate is made possible. The image plane in an actual perspective camera is logically located behind the effective pupil but in the perspective camera model, it has been placed in front of the centre of projection. This is useful so that the image pixels are not mathematically flipped in both axes and the common image coordinate system can be applied directly for pixel referencing.
In Section 3, an assumption has been made where the spherical mirror’s centre coincides with the optical axis and subsequently the geometry of light ray made is assumed to be rotationally symmetrical about the optical axis. Therefore within a 3D space where points are represented in Cartesian form of P(x,y,z) with the optical axis labelled as zaxis, $\rho =\sqrt{({x}^{2}+{y}^{2})}$ can be introduced to reduce the dimension of our problem into that of a 2D. Without any loss of generality, we position our origin at the centre of the virtual sphere. Figure 6 illustrates the important geometry used for subsequent derivations with a reduced dimensionality.
In general, the point mapping process is a twostep translation process. The first step translates points between image and mirror while the second step translates points between mirror and world. Mirror points are points on the spherical mirror where lights are reflected while world points are the points of light source in the 3D space. In subsequent derivations, variables with subscript i, m, C, and w represent their respective image, mirror, caustics and world component. The idea of caustic is discussed in Section 4.2. For brevity, functions that map in the order of P _{ i } to P _{ m } to P _{ w } are subsequently denoted as “forward mapping functions” whereas the reversed counterparts are denoted as “backward mapping functions”.
Similar derivations of such a point mapping process had been previously attempted. Micusik and Pajdla [22, 23] had not provided a closedform twoway translation but relied on numerical search for the backward mapping function. That had been addressed by Agrawal et al. [24] in an approach which is rather comparable to our method. They had shown that the backward mapping is achievable via a forth order equation while we have employed a sixth order one due to a tight integration with the caustic geometry. However, on the forward mapping, our approach involves less operations to complete with 11 additions/subtractions, 23 multiplications/divisions, and 3 square roots (2 square roots for cylindrical unwrapping) instead of 38 addition/subtraction, 55 multiplications/divisions, and 2 square roots as in [24]. While both of our mapping functions operate in the same P(ρ z) space, Agrawal et al. [24] derivations use P(ρ z) space in the forward mapping but P(x y z) space in the backward mapping. Thus, our approach provides an advantage in cylindrical unwrapping that has a constant ρ _{ w }.
4.1 Mapping between image point and mirror point
The reflected light from the mirror is assumed to pass through the centre of projection of the perspective camera model (which is analogous to the camera pupil). In Figure 6, an arbitrary mirror point, P _{ m }(ρ _{ m },z _{ m }), and its corresponding image point, P _{ i }(ρ _{ i },f), are illustrated as the intersection points of the reflected light made with the virtual spherical mirror and the image plane, respectively. The backward mapping function of ρ _{ i }, given that P _{ m } is known, is straight forward by using the method of similar triangles:
On the other hand, the forward function of P _{ m } given ρ _{ i } demands the intersection point of a straight line equation representing the reflected light, k(ρ)=m _{ k } ρ+c _{ k } and the equation of the virtual sphere, $s\left(\rho \right)=\sqrt{{R}^{2}{\rho}^{2}}$. Geometrically, m _{ k } and c _{ k } can be easily identified as ${f}_{{\rho}_{i}}$ and h, respectively. Equating both equations at ρ=ρ _{ i } yields a quadratic equation as shown in Equation (5).
Mathematically, the other of solution of Equation (5) is always larger than Equation (6) and therefore is rejected by observing the geometry of the intersections. Subsequently z _{ m } can be obtained either from s(ρ) or k(ρ) as follows:
4.2 Mapping between mirror point and world point
In [5, 6, 34], caustics of several NSVP omnidirectional sensors were extensively reviewed. A caustic is defined as the locus of viewpoints forming a surface that is tangential to the incident rays. The centre of projection of the perspective camera model is analogous to the effective pupil of the camera, therefore a virtual caustic is applied onto the virtual sphere defined in Section 3. Assuming that the spherical mirror’s centre coincides with the optical axis, the caustic surface is rotationally symmetrical about the optical axis and is therefore reduced to a curve in ρz plane. Before subsequent mapping functions can be derived, the caustic model has to be first investigated.
4.2.1 Caustic curve
A general model for caustic curve that is compatible for all conic section mirror profiles has been provided by Swaminathan et al. [34]. However, since the generalisation is not important for our application, we opt for a simpler model derived specifically for a spherical mirror provided by Baker and Nayar [5, 6].
The parametric equations for the spherical mirror caustic curve are derived in [5, 6] assuming R=1. In order to serve our purpose, the parametric equations have been reformulated with R as a parameter instead. Firstly, an incident ray reflected at a mirror point ${P}_{m}\left({\rho}_{m},{z}_{m}\right)=\left({\rho}_{m},\sqrt{{R}^{2}{\rho}_{m}^{2}}\right)$ is described by a straight line equation, j(ρ _{ m }), and this is repeated for point ${P}_{m}^{\prime}\left({\rho}_{m}+d{\rho}_{m},{z}_{m}+d{z}_{m}\right)$. Secondly, the intersection point of j(ρ _{ m }) and j(ρ _{ m }+d ρ _{ m }) is obtained by taking the limit d ρ _{ m }→0 while applying a constraint on dz with (ρ _{ m }+d ρ _{ m })^{2}+(z _{ m }+d z _{ m })^{2}=R^{2}. Denoting a point on the caustic curve as P _{ c }(ρ _{ c } z _{ c }), the result of the derivation is shown in Equations (8) and (9). Detailed derivations can be found in Appendix. Figure 7 shows sample plots of the caustic curve of a spherical mirror with = 1 described in parametric equations with varying h.
4.2.2 Point mapping
The forward mapping function of P _{ m } to its corresponding world point, P _{ w }(ρ _{ w },z _{ w }) is obtained by differentiating Equations (8) and (9) with respect to ρ _{ m } using the chain rule of calculus. This thus forms the gradient, m _{ j }, of the incident ray, j(ρ). However, due to the lost of depth information during the image acquisition process, constraint on either z _{ w } or ρ _{ w } is needed. Mathematically, the constraint is needed because j(ρ) is valid for ρ=(ρ _{ m },∞). In practical application however, a constraint on ρ _{ w } is better than z _{ w } because at certain ρ _{ m }, m _{ j } will change sign. More discussion is provided in Section 6. Assuming ρ _{ w } is to be constrained, z _{ w } is then given by:
where m _{ j } is derived from the caustic curve provided in Equation (13).
In order to derive the backward mapping functions of P _{ w } to its corresponding P _{ m }, more measures have to be taken. An incident ray would pass through a P _{ w } and is tangential to the caustic curve at P _{ c }. This relationship can be exploited as a sixth order polynomial shown in Equation (11). Detailed derivation can be found in Section 4.3.
By solving Equation (11), the corresponding ρ _{ m } can be obtained and ${z}_{m}=\sqrt{{R}^{2}{\rho}_{m}^{2}}$ can be determined accordingly. Since 0≥ρ _{ m }≥ρ _{ m,max}, there will only be one valid solution for Equation (11). ρ _{ m,max} is easily observed as the maximum ρ _{ m } due to the maximum viewing angle, ζ, as follows:
4.3 Derivation of worldtomirror mapping function
In order to derive Equation (11), it is first considered that an incident ray passes through a P _{ w }(ρ _{ w },z _{ w }) and is tangential to the caustic curve at P _{ c }(ρ _{ c },z _{ c }). Therefore, the mapping of a P _{ w } to its corresponding P _{ m }(ρ _{ m },z _{ m }) can be done by making use of the gradient of the incident ray, m _{ j }, which can be obtained by differentiating Equations (8) and (9) with respect to ρ _{ m } using the chain rule of calculus.
Substituting Equations (8) and (9) into Equation (13) and subsequently rearranging them would then yield Equation (15). By discarding the denominator of Equation (15), the square root functions are then eliminated forming Equation (16).
Finally, not shown in this article due to the length of the equation, Equation (16) is summed to a common denominator and subsequently the denominator is again discarded. Collecting the ρ _{ m } terms hence forms Equation (11).
5 Stage 3projection plane for unwrapping
A virtual projection plane is assumed to be an imaginary 2 dimensional (2D) plane of light source. The illuminated light from the plane would travel towards the virtual sphere and thus be reflected into the camera’s pupil. By using the forward mapping functions derived in Section 4, the corresponding incident ray of a specific image point can be traced, and subsequently a world point where the incident ray coincides with the virtual projection plane can be obtained. Then, the backward mapping functions will be used to populate the virtual plane using common interpolation technique (i.e. bilinear interpolation). Finally, the plane itself results into an unwrapped image. In addition, by selecting the shape and position of the virtual plane, different forms of unwrapping is possible. Lei et al. [21] had documented the idea of virtual plane in details and they had demonstrated two forms of panoramic unwrapping—cylindrical and cuboid. Cuboid panoramic unwrapping is done by replacing the cylindrical virtual plane with a cuboid one. For our case, we will demonstrate that a ground plane view is also possible with our method by choosing an appropriate projection plane as illustrated in Figure 8c.
In order to take advantage of a unified platform with multiple form unwrapping output capability, the mapping functions have been conveniently made to accept and produce points in the Cartesian form. Therefore, the location of each element in a virtual plane (ends up as an image pixel) described in Cartesian points would be easily translated into their respective image points. Generally, a lookup table of corresponding points will be generated so that subsequent unwrapping can be speeded up. Such practice is commonly applied in omnidirectional image unwrapping field and is documented in details in Jeng and Tsai [29] work.
5.1 Cylindrical panoramic unwrapping
Cylindrical panoramic unwrapping can be done using an openended cylinder plane wrapping around the spherical mirror as shown in Figure 8a. Since a cylinder is rotationally symmetrical about its central axis and by letting it coincides with the optical axis, points on the projection plane described in cylindrical coordinate system (e.g. (ρ,φ,z)) will have a similar mapping of ρ and z for all φ. Therefore, only one set of mapped coordinate is required.
The cylinder will have a userdefined radius in pixel, r _{ k }, and thus the width of the unwrapped image is roughly ⌊2Π r _{ k }⌋ pixel. The height of the cylinder is dependent on the region of image to be unwrapped and is also userdefined. Due to the nature of the spherical mirror, unwrapping is only suitable up to a certain region of the mirror. More will be discussed in Section 6. Figure 9a shows the setup of our experiment in a controlled environment. The captured image is shown in Figure 9b with cylindrical panoramic unwrapping shown in Figure 10.
5.2 Cuboid panoramic unwrapping
Cuboid panoramic unwrapping is an enhanced version of the cylindrical one. As documented in [21], a cuboid projection plane will artificially create a perspective view of the surrounding. The output of this method results in a more natural view of the surrounding for human eye perception and is particularly effective if the surrounding is a rectangularly confined space. Figure 11 is the result of unwrapping Figure 9b using cuboid projection plane as illustrated in Figure 8b. The result shown assumed a cuboid placed at the centre of the virtual plane with upright orientation to the xaxis but xaxis but in practice, a rectangular one would work equally well. The shape, position, and orientation of the cuboid projection plane is mainly dependant on the boundary of surrounding space (e.g. walls, partitions, building etc.).
5.3 Ground plane view unwrapping
Ground plane view unwrapping generates an output that appears perspectively correct as if the image were captured from some height above. A more commonly known term for ground plane view is the bird’s eye view. As the name suggests, ground plane view unwrapping is mainly used to detect features on the ground. While panoramic unwrapping can include ground features, they would introduce low quality unwrapping due to insufficient data point (image pixel) near the centre of the mirror and rendering less useful interpolated data. Previous work by Hicks and Bajcsy [35] performed analogue ground plane correction using a specialised mirror profile. Another work by Gaspar and SantosVictor [27] corrects distortion on ground feature by solving the geometry made by the captured light rays.
In Figure 12, a controlled environment to demonstrate ground plane view unwrapping is set up. To adapt to such form of unwrapping using the existing derived mapping functions, a projection plane that is normal to the optical axis is simply placed some distance away from the virtual sphere instead of upright project planes used in the previous two panoramic unwrapping schemes as shown in Figure 8c. Illustration of the result is shown in Figure 13.
5.4 Assessment on accuracy
In order to assess the accuracy of the mapping functions, two experiments were conducted. The first experiment measures the fit of horizontal and vertical lines on a checkerbox pattern. The second experiment reconstructs 3D points of a trihedron with checkerbox pattern from two views.
5.4.1 Line fitting
In this experiment, a checkerbox pattern was initially captured and unwrapped into cuboid panoramic view. Then, Harris and Stephens [36] corner detection algorithm was used to capture the corner points in the unwrapped checkerbox pattern. For any ambiguities due to detection of multiple corners at the same point, the centroid of the cluster was used instead. Subsequently, horizontal and vertical lines were fitted using linear regression model to the points as shown in Figure 14. Note that for vertical lines, the axes were flipped so that fitting is possible. Throughout the entire process, human interaction was made minimal where user only specifies the total points to detect and a coarse estimation of the location of the points.
In Table 3, an analysis of the linefitting in Figure 14b is presented. The mean gradient suggested the “straightness” of the fitted lines while mean R^{2} suggested the “goodness” of fit of the points involved. As can be seen, the mean gradient and mean R^{2} approach 0 and 1 respectively, which imply a proportionally high degree of correctness in the relative position of the points involved as they are mapped from omnidirectional view to cuboid panoramic view. A second analysis was done on the spacing, Δ, between the fitted lines. Let Δ _{ y } be the mean spacing of horizontal lines while Δ _{ x } be mean spacing of vertical lines, the ideal benchmark checkerbox pattern of Figure 14a should produce a ratio of ${\Delta}_{y}\left(\right)close="">/$ neglecting lens distortion. On the unwrapped checkerbox pattern, it is found that ${\Delta}_{y}\left(\right)close="">/$, indicating an error of 7% in the ratio after unwrapping.
This simple line fitting experiment was meant to show a preliminary assessment of the mapping functions without involving complex algorithm. Since it is not a common practice in previous works, a benchmark for the result in Table 3 is not possible. Thus, another experiment that is more commonly conducted, i.e. 3D reconstruction in Section 5.4.2, was carried out to provide further evaluation on the mapping functions.
5.4.2 3D reconstruction
In this experiment, a 3D reconstruction [37] of a trihedron with checkerbox pattern was conducted. Initially, two images of the trihedron in Figure 15 were captured from two different viewing locations and unwrapped into cuboid form. Then, a total 92 pairs of corresponding points from the two views were sampled manually. From the corresponding points, the fundamental matrix was first computed and subsequently the two camera matrices, P _{1} and P _{2}, associated with the two views were deduced. As P _{1} is set at I0, this resulted in a projective reconstruction of the corresponding points by linear triangulation method. Finally, the reconstructed 3D points are upgraded to a metric reconstruction [37].
From the reconstructed trihedron points, several measurements were obtained to assess the accuracy of the mapping equations. Figure 16 shows the distribution of the calculated rootmeansquare error (RMSE) of the 3D points with an average of 14.54 mm. Without further optimisation on the reconstruction (i.e. bundle adjustment), the error obtained is observed to be within a similar range as previous works [20, 22]. Finally, the 3D points were reprojected back to the input image as shown in Figure 17 and the reprojection error of the points were measured as shown in Figure 18. The average reprojection error of the 3D points was found to be 0.22 pixels.
5.4.3 Sources of error
The assessments on accuracy revealed a certain degree of error introduced by the mapping functions. The possible sources of error include the assumptions made in the derivations of the mapping functions. An ideal spherical mirror is assumed in the derivations while this may not be always true in practice. Also, the spherical mirror’s centre might not coincides with the optical axis perfectly. Factors such as these could also affect the accuracy of parameter calibration for f and R, which eventually lead to error compounding as mapping is processed.
Other than that, the lens of the camera could be ideally assumed to provide a perfect perspective view projection. Slight fisheye distortion might be introduced as the omnidirectional view image is captured. Lastly, sampling error, either manually or using Harris corner detection algorithm [36], is inevitable.
6 Image quality and algorithm limitation
Due to heavy dependency on ray tracing in the proposed algorithm, different unwrapping forms are optimum only in certain regions on the omnidirectional image that is radially confined from the centre of the mirror. For ground plane view unwrapping, incident rays with m _{ j }≤0 do not intersect an xy plane placed below the virtual sphere, which translates to m _{ k }≥0 for reflected rays. Theoretically, unwrapping cannot be done with the mentioned condition. In practical unwrapping however, m _{ j } will never reach 0 but converges to it. At the converging region, there is insufficient data point at the input image (image pixel) to perform useful interpolation that results in highly detailed output and therefore should be avoided. Figure 19 shows a plot of ρ _{ i } versus ρ _{ w } in pixels, where ρ _{ i } gradually converges to a limit as ρ _{ w } progresses, indicating that ρ _{ w } is roughly represented by similar ρ _{ i } as the projection plane expands. The converging region suggests a low quality output region.
For panoramic unwrapping, upright projection planes suffer less from the above mentioned limitation. As shown in the solid line plot in Figure 20, ρ _{ i } is rather evenly “distributed” across z _{ w } indicating that the output is interpolated from a rather evenly spaced data point. However, for practical usage, as points on the input image are represented using Polar coordinates, region closing to the centre of the mirror should be avoided as it is stretched along the angular axis after unwrapped, which produces an output with degraded quality.
Also note in Figure 20, an overlapping dashed line plot is provided for the purpose of showing that m _{ j } changes sign during the complete mapping. In Section 4.2.2, as Equation (10) is derived, there is a need to provide additional constraint on either ρ _{ w } or z _{ w }. Mathematically, if ρ _{ w } is constrained, there might be two possible corresponding z _{ w } depending on the sign of m _{ j } at that instance. One of the z _{ w } would be invalid by observing the geometry. In forward mapping, this means that an arbitrary z _{ w } may not have a corresponding valid solution of ρ _{ w }. For practical implementation, constraint on ρ _{ w } can be effectively provided by the projection plane and thus the consideration made when deriving Equation (10).
Apart from quality degradation due to the mapping functions, the physical bounding of the surrounding will also affect the “correctness” of the unwrapped image output. Depending on the position, the camera would capture unknown portion of ground plane and upright plane (e.g. walls). While this external factor is beyond the control of the algorithm, a software can be written to facilitate user in providing information on the effective region for unwrapping as shown in using the forward mapping functions.
7 Conclusion
A novel technique of unwrapping for spherical omnidirectional images has been proposed. The algorithm comprises three key stages in which (1) the camera is first calibrated to obtain essential parameters, (2) ray tracing is then utilised to solve the functions that map points back and forth between the omnidirectional image and its unwrapped counterpart, and finally (3) a projection plane is set up for the unwrapping.
The proposed unwrapping scheme enables three commonly performed unwrapping forms, namely, cylindrical panoramic, cuboid panoramic, and ground plane view to be done. The different forms of unwrapping can be achieved by selecting an appropriate projection plane to be populated as the unwrapped image.
Finally, the accuracy of the mapping functions was accessed by conducting a simple line fitting and a 3D reconstruction. The line fitting experiment showed a 7% error in the checkerbox pattern ratio. For the 3D reconstruction experiment, the average RMSE was 14.54 mm while the average reprojection error was 0.22 pixel.
Appendix
Derivation of a spherical mirror’s caustic curve
In order to derive the caustic curve for a spherical mirror, the gradient of an incident ray, j(ρ,ρ _{ m })=m _{ j }(ρ _{ m })·ρ+c _{ j }(ρ _{ m }) at an arbitrary mirror point, P _{ m }(ρ _{ m },z _{ m }), is first derived. Note that for clarity, m _{ j } and c _{ j } are functions of ρ _{ m } whereas j is a function of ρ and ρ _{ m }. Prior to this section, they are omitted for brevity. Figure 6 shows that m _{ j }(ρ _{ m }) is in fact tanθ where θ=α2β.
where tanα can be deduced geometrically while tanβ can be obtained from the gradient of the spherical mirror curve, s(ρ). Substituting Equations (18) and (19) into Equation (17) yields:
Secondly, the zintercept of j(ρ,ρ _{ m }), c _{ j }(ρ _{ m }), is obtained by examine the relationship of z _{ m }=j(ρ _{ m },ρ _{ m }).
which implies that:
Thirdly, let two incident rays contacting at P _{ m }(ρ _{ m },z _{ m }) and ${P}_{m}^{\prime}\left({\rho}_{m}+d{\rho}_{m},{z}_{m}+d{z}_{m}\right)$, their intersection points would form the caustic curve. Points on the caustic curve are denoted as P _{ c }(ρ _{ c },z _{ c }).
Substituting Equations (20) and (21) into Equation (22) and taking the limit of d ρ _{ m }→0 thus results in Equation (8). Accordingly, z _{ c } is therefore derived from j(ρ _{ c },ρ _{ m })=m _{ j }(ρ _{ m })·ρ _{ c }+c _{ j }(ρ _{ m }), yielding Equation (9).
Endnote
^{a}Paraboloidal catadioptric camera and hyperboloidal catadioptric camera are also known as paracatadioptric and hypercatadioptric respectively in short mainly due to extensive utilisation.
Abbreviations
 FOV:

field of view
 SVP:

single viewpoint
 NSVP:

nonsingle viewpoint
 CNC:

computer numerical control
 3D:

3 dimensional
 2D:

2 dimensional.
References
 1.
Rees DW: Patent 3505465. 1970.
 2.
Yamazawa K, Yagi Y, Yachida M: Omnidirectional imaging with hyperboloidal projection. In Proceedings of the 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems ’93, IROS ’93. Tokyo, Japan; 1993:10291034.
 3.
Yagi Y, Kawato S: Panorama scene analysis with conic projection. 1990, 181187.
 4.
Hong J, Tan X, Pinette B, Weiss R, Riseman EM: Imagebased homing. IEEE Control Syst 1992, 12: 3845.
 5.
Baker S, Nayar SK: A theory of singleviewpoint catadioptric image formation. Int. J. Comput. Vis 1999, 35(2):175196. http://www.springerlink.com/index/WU62M18P65412043.pdf 10.1023/A:1008128724364
 6.
Baker S, Nayar SK: A theory of catadioptric image formation. In Proceedings of the Sixth International Conference on Computer Vision, 1998. Bombay, India; 1998:3542.
 7.
Sturzl W, Srinivasan W: Omnidirectional imaging system with constant elevational gain and single viewpoint. In Proccedings of the OMNIVIS  10th Workshop on Omnidirectional Vision, Camera Networks and Nonclassical Cameras. Zaragoza, Spain; 2010:17.
 8.
Geyer C, Daniilidis K: A unifying theory for central panoramic systems and practical applications. In Proceedings of the 6th European Conference on Computer VisionPart II, ECCV ’00. London, UK; 2000:445461. http://dl.acm.org/citation.cfm?id=645314.649434
 9.
Barreto JP, Araujo H: Geometric properties of central catadioptric line images and their application in calibration. IEEE Trans. Pattern Anal. Mach. Intell 2005, 27: 13271333. http://dl.acm.org/citation.cfm?id=1070616.1070819
 10.
Ying X, Hu Z: Catadioptric camera calibration using geometric invariants. IEEE Trans. Pattern Anal. Mach. Intell 2004, 26(10):12601271. 10.1109/TPAMI.2004.79
 11.
Ying X, Zha H: Simultaneously calibrating catadioptric camera and detecting line features using Hough transform. In 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2005. (IROS 2005). Alberta, Canada; 2005:412417.
 12.
Vasseur P, Mouaddib EM: Central catadioptric line detection. In British Machine Vision Conference. Kingston University, London, UK; 2004.
 13.
Mei C, Rives P: Single view point omnidirectional camera calibration from planar grids. In IEEE International Conference on Robotics and Automation, 2007. Roma, Italy; 2007:39453950.
 14.
Puig L, Bastanlar Y, Sturm P, Guerrero JJ, Barreto JA: Calibration of central catadioptric cameras using a DLTLike approach. Int. J. Comput. Vis 2011, 93: 101114. 10.1007/s1126301004111
 15.
Deng XM, Wu FC, Wu YH: An easy calibration method for central catadioptric cameras. Acta Automatica Sinica 2007, 33(8):801808.
 16.
Gasparini S, Sturm P, Barreto J: Planebased calibration of central catadioptric cameras. 2009 IEEE 12th International Conference on Computer Vision 2009, 11951202.
 17.
Wu F, Duan F, Hu Z, Wu Y: A new linear algorithm for calibrating central catadioptric cameras. Pattern Recogn 2008, 41(10):31663172. 10.1016/j.patcog.2008.03.010
 18.
Wu Y, Hu Z: Geometric invariants and applications under catadioptric camera model. In Tenth IEEE International Conference on Computer Vision, 2005. ICCV 2005. Beijing, China; 2005:15471554.
 19.
Scaramuzza D, Martinelli A, Siegwart R: A toolbox for easily calibrating omnidirectional cameras. In Proceedings of the 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. Beijing, China; 2006:56955701.
 20.
Scaramuzza D, Martinelli A, Siegwart R: A flexible technique for accurate omnidirectional camera calibration and structure from motion. In IEEE International Conference on Computer Vision Systems, 2006 ICVS ’06. New York, USA; 2006:4545.
 21.
Lei J, Du X, Zhu YF, Liu JL: Unwrapping and stereo rectification for omnidirectional images. J. Zhejiang Univ. SCI. A 2009, 10(8):11251139. http://www.springerlink.com/index/10.1631/jzus.A0820357 10.1631/jzus.A0820357
 22.
Micusik B, Pajdla T: Autocalibration 3D reconstruction with noncentral catadioptric cameras. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004, CVPR 2004. Washington, DC, USA; 2004:5865.
 23.
Micusik B, Pajdla T: Structure from motion with wide circular field of view cameras. IEEE Trans. Pattern Anal. Mach. Intell 2006, 28(7):11351149.
 24.
Agrawal A, Taguchi Y, Ramalingam S: Analytical forward projection for axial noncentral dioptric and catadioptric cameras. In Proceedings of the 11th European conference on computer vision conference on Computer vision: Part III. ECCV’10, SpringerVerlag, Berlin, Heidelberg; 2010:129143.
 25.
Derrien S, Konolige K: Approximating a single viewpoint in panoramic imaging devices. In Proceedings of IEEE Workshop on Omnidirectional Vision, 2000. South Carolina, USA; 2000:8590.
 26.
Shabayek A, Morel O, Fofi D: Autocalibration and 3D reconstruction with noncentral catadioptric sensors using polarization imaging. In The Proceedings of the 10th Workshop on Omnidirectional Vision (OMNIVIS) in conjunction with Robotics Systems and Science RSS. Zaragoza, Spain; 20102010.
 27.
Gaspar J, SantosVictor J: Visual path following with a catadioptric panoramic camera. In Proceedings of the International Symposium on Intelligent Robotic Systems  SIRS’99. Coimbra, Portugal; 1999. http://citeseerx.ist.psu.edu/viewdoc/summary?http://dx.doi.org/10.1.1.33.3379
 28.
Winters N, Gaspar J, Lacey G, SantosVictor J: Omnidirectional vision for robot navigation. In Proceedings of the IEEE Workshop on Omnidirectional Vision. South Carolina, USA; 2000:2128.
 29.
Jeng SW, Tsai WH: Using panomapping tables for unwarping of omniimages into panoramic and perspectiveview images. IET Image Process 2007, 1(2):149155. 10.1049/ietipr:20060201
 30.
Geyer C, Daniilidis K: Catadioptric camera calibration. Kerkyra, Greece; 1999.
 31.
Geyer C, Daniilidis K: Paracatadioptric camera calibration. IEEE Trans. Pattern Anal. Mach. Intell 2002, 24(5):687695. 10.1109/34.1000241
 32.
Zhang Z: A flexible new technique for camera calibration. IEEE Trans. Pattern Analy. Mach. Intell 2000, 22(11):13301334. http://dx.doi.org/10.1109/34.888718 10.1109/34.888718
 33.
Duda RO, Hart PE: Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM 1972, 15: 1115. 10.1145/361237.361242
 34.
Swaminathan R, Grossberg MD, Nayar SK: Nonsingle viewpoint catadioptric cameras: geometry and analysis. Int. J. Comput. Vis 2001, 66(3):211229.
 35.
Hicks RA, Bajcsy R: Reflective surfaces as computational sensors. Image Vis. Comput 2001, 19(11):773777. http://linkinghub.elsevier.com/retrieve/pii/S0262885600001049 10.1016/S02628856(00)001049
 36.
Harris C, Stephens M: A combined corner and edge detector. In Proceedings of Fourth Alvey Vision Conference. Manchester, UK; 1988:147151.
 37.
Hartley RI, Zisserman A: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge; 2004.
Acknowledgements
This research project is funded by the Ministry of Higher Education (MOHE), Malaysia, under a Fundamental Research Grant Scheme (FRGS/2/2010/TK/SWIN/03/02). N. S. Chong also thanks the Swinburne University of Technology (Sarawak Campus) for his Ph.D. studentship.
Author information
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
About this article
Received
Accepted
Published
DOI
Keywords
 Omnidirectional view sensor
 Spherical mirror
 Nonsingle viewpoint
 Omnidirectional view image unwrapping