Efficient robust image interpolation and surface properties using polynomial texture mapping

Abstract

Polynomial texture mapping (PTM) uses simple polynomial regression to interpolate and re-light image sets taken from a fixed camera but under different illumination directions. PTM is an extension of the classical photometric stereo (PST), replacing the simple Lambertian model employed by the latter with a polynomial one. The advantage and hence wide use of PTM is that it provides some effectiveness in interpolating appearance including more complex phenomena such as interreflections, specularities and shadowing. In addition, PTM provides estimates of surface properties, i.e., chromaticity, albedo and surface normals. The most accurate model to date utilizes multivariate Least Median of Squares (LMS) robust regression to generate a basic matte model, followed by radial basis function (RBF) interpolation to give accurate interpolants of appearance. However, robust multivariate modelling is slow. Here we show that the robust regression can find acceptably accurate inlier sets using a much less burdensome 1D LMS robust regression (or ‘mode-finder’). We also show that one can produce good quality appearance interpolants, plus accurate surface properties using PTM before the additional RBF stage, provided one increases the dimensionality beyond 6D and still uses robust regression. Moreover, we model luminance and chromaticity separately, with dimensions 16 and 9 respectively. It is this separation of colour channels that allows us to maintain a relatively low dimensionality for the modelling. Another observation we show here is that in contrast to current thinking, using the original idea of polynomial terms in the lighting direction outperforms the use of hemispherical harmonics (HSH) for matte appearance modelling. For the RBF stage, we use Tikhonov regularization, which makes a substantial difference in performance. The radial functions used here are Gaussians; however, to date the Gaussian dispersion width and the value of the Tikhonov parameter have been fixed. Here we show that one can extend a theorem from graphics that generates a very fast error measure for an otherwise difficult leave-one-out error analysis. Using our extension of the theorem, we can optimize on both the Gaussian width and the Tikhonov parameter.

1 Introduction

Polynomial texture mapping (PTM) [1] uses a single fixed digital camera at constant exposure, with a set of n images captured using lighting from different directions. A typical rig would consist of a hemisphere of xenon flash lamps imaging an object, where directions to each light is known (Figure 1a). The basic idea in PTM is to improve on a simple Lambertian model for matte content, whereby the three components of the light direction are mapped to luminance, by extending the model to include a low-order polynomial of lighting-direction components. The strength of PTM, in comparison to a simple Lambertian photometric stereo (PST) [2] is that PTM can better model real radiance and to some extent grasp intricate dependencies due to self-shadowing and interreflections. Usually, some 40 to 80 images are captured. The better capture of details is the driving force behind the interest in this technique evinced by many museum professionals, with the original least squares (LS)-based PTM method already in use at major museums in the USA, including the Smithsonian, the Museum of Modern Art and the Fine Arts Museums of San Francisco, and is planned for the Metropolitan and the Louvre (M. Mudge, personal communication, Cultural Heritage Imaging). As well, some work has involved applying PTM in situ for such applications as imaging palaeolithic rock art [3]. In such situations, one has to recover lighting directions from the specular patch on a reflective sphere [4]; such a ‘highlight’ method [5] can also be applied to museum capture of small objects or to microscopic image capture.

Figure 1

A typical PTM rig and an example dataset. (a) A 40-light rig for capturing PTM datasets (courteously supplied by Cultural Heritage Imaging). (b) A PTM dataset of 50 images (courteously supplied by Tom Malzbender, Hewlett-Packard).

PTM generates a matte model for the surface, where luminance (or RGB) is modelled at each pixel via a polynomial regression from light-direction components to luminance. Say, e.g. there are n=50 images, with n known normalized light-direction three-vectors a. Then in the original embodiment, a six-term polynomial model is fitted at each pixel separately, regressing onto that pixel’s n luminance values using LS regression. The main objectives of PTM are the ability to re-light pixels using the regression parameters obtained, as well as the recovery of surface properties: surface normal, colour and albedo. For re-lighting, the idea is simply that if the regression from the n in-sample light-directions a to n luminance values, L is known then substituting a new a will generate a new L, thus yielding a simple interpolation scheme for new, out-of-sample, light directions a.

In [6], we extend PTM in three ways: First, the six-term polynomial is changed so as to allow purely linear terms to model purely linear luminance exactly. Secondly, the LS regression for the underlying matte model is replaced by a robust regression, the least median of squares (LMS) method [7]. This means that only a majority of the n pixel values obtained at each pixel need be actually matte, with specularities and shadows automatically identified as outliers. With correctly identified matte pixels in hand, surface normals, albedos and pixel chromaticity are more accurately recovered. Thirdly, authors in [6] further add an additional interpolation level by modelling the part of in-sample pixel values that is not completely explained by the matte PTM model via a radial basis function (RBF) interpolation. The RBF model does a much better job of modelling features such as specularities that depend strongly on the lighting direction. As well, the RBF approach can make use of any shadow information to help model interpolated shadows, which change abruptly with lighting direction. The interpolation is still local to each pixel and, thus, does not attempt to bridge non-specular locales as in reflectance sharing, for example [8, 9]. In reflectance sharing, a known surface geometry is assumed, as opposed to the present paper. Here, we rely on the idea that there is at least a small contribution to specularity at any pixel, e.g. the sheen on skin or paintwork, so that we need not share across neighbouring pixels and can employ the RBF approach from [6]. For cast shadows, a more difficult feature to model, at each pixel the RBF model will utilize whatever shadow content is actually present across the whole set of n images from n lights.

The current study is aimed at further refining and improving the PTM + RBF pipeline as was employed in [6], as well as exploring different combinations of basis functions. The main contributions of this work are threefold:

• 1. We introduce a more efficient, ‘mode-finder’ regression method to replace the computationally intensive multivariate LMS regression in the matte modelling stage. Compared to the 6D LMS regression, the mode-finder effectively reduces the number of unknowns from 6 to 1 and thus greatly reduces the processing time from O(n ⁶ logn) to O(n logn) ([7], p. 206). We found that this simplification introduces little reduction of accuracy. Although technically the mode-finder regression approach can be applied to the mode of either luminance or any colour components, we show that the mode of luminance provides the highest accuracy.

How a robust mode-finder works is simple: from the n luminance values at the current pixel, select one randomly; continue and adopt as the best estimate of the ‘mode’ that luminance which delivers the least median of squared residuals. What makes the LMS method powerful is that it provides strong mathematical guarantees on the performance given by choosing a much smaller subset than a simple exhaustive search and it also delivers an inlier band, automatically, thus classifying luminance values as usable or not. The multivariate version of LMS is similar: for 6D LMS, e.g., we randomly select six luminances and find residuals for a polynomial regression. Again, the number of selections is tremendously smaller than an exhaustive search, but nonetheless is very slow compared to a 1D search.

• 2. We explore different combinations of basis functions for PTM. Firstly, we extend the classical polynomial models from 6D to 16D. Moreover, due to another observation we made that luminance reconstruction has a far greater impact on the re-lighted image quality than the reconstruction of chromaticity, we can reduce the dimension for chromaticity modelling with little loss of accuracy. Reducing the number of float regression coefficients makes a difference, when we multiply by millions of pixels. We found that 16D for luminance + 9D for chromaticity is a good balance between dimensionality and accuracy. Secondly, we compared the performance of the hemiSpherical harmonics (HSH) basis against the polynomial basis of the same order and found that, surprisingly, the polynomial model outperforms HSH in terms of the quality of appearance reconstruction, especially at large incident angles.

• 3. We adopt a method to mathematically determine the optimal parameters used for the RBF interpolation stage. Previously in [6], we made use of an RBF network consisting of Gaussian radial functions to model the non-Lambertian contribution. The parameters in this model, including the Gaussian dispersion σ and Tikhonov regularization coefficient τ, were taken heuristically and remained constant across all pixels. In this work, we start off from a theorem that minimizes error in a leave-one-out analysis by optimizing the Gaussian dispersion parameter. Such a theorem is not new, but here we extend its use to whole images and three-channel colour. More importantly, however, we also extend the theorem to optimize over the Tikhonov regularization. For the fairly large size matrices being inverted, these optimizations matter and make a substantial difference to results obtained.

Note that contributions 1 and 3 are direct improvements over the methodology of the PTM + RBF pipeline: Contribution 1 is aimed at increasing the efficiency of the first stage - matte modelling; contribution 3 is devoted to the optimization of the second stage - RBF interpolation. On the other hand, the goal of contribution 2 is to find an optimal set of basis functions. The discoveries made in contribution 2 can be applied to the matte modelling stage of PTM + RBF, as well as regular PTM with no RBF interpolation.

This paper is organized as follows: In Section 2 we review previous work in this area, and in Section 3 we provide a brief recapitulation of the PTM method. In Section 4 we introduce the notion of our contribution 1 - using a robust mode-finder instead of a full multivariate robust regression and explicate how we use the mode-finder and trimmed LS to realize outlier detection and recover surface properties. In Section 5, focusing on contribution 2, we test the appearance reconstruction with PTM separately applied to luminance and chromaticity and compare the reconstructed matte appearance for PTM and for HSH. In Section 6 we describe our contribution 3, i.e. how to use an optimized version of the RBF framework to interpolate specularity and shadows on reconstructed images. Finally, Section 7 presents concluding remarks.

2 Related work

Many methods for detecting outlier pixels in photometric methods have been proposed. Early examples include a four-light PST approach in which the values yielding significantly differing albedos are excluded [10–12]. In a similar five-light PST method [13], the highest and the lowest values, presumably corresponding to highlights and shadows, are simply discarded. Another four-light method [14] explicitly includes ambient illumination and surface integrability and adopts an iterative strategy using current surface estimates to accept or reject each additional light based on a threshold indicating a shadowed value. The problem with these methods is that they rely on throwing away only a small number of outlier pixel values, whereas our robust methods in the current and previous studies allow up to 50% of the pixel values discarded as outliers.

More recently, Willems et al. [15] used an iterative method to estimate normals. Initially, the pixel values within a certain range (10 to 240 out of 255) were used to estimate an initial normal map. In each of the following iterations, error residuals in normals for all lighting directions are computed and the normals are updated based only on those directions with small residuals. Sun et al. [16] showed that at least six light sources are needed to guarantee that every location on the surface is illuminated by at least three lights. They proposed a decision algorithm to discard only doubtful pixels, rather than throwing away all pixel values that lie outside a certain range. However, the validity of their method is based on the assumption that out of the six values for each pixel, there is at most one highlight pixels and two shadowed pixels. Julia et al. [17] utilized a factorization technique to decompose the luminance matrix into surface and light source matrices. The shadow and highlight pixels are considered as missing data, with the objective of reducing their influence on the result. Wu et al. [18] formulated the problem of surface normal recovery as a rank minimization problem, which can be solved via convex optimization. Their method is able to handle specularities and shadows as well as other non-Lambertian deviations. Compared to these methods, the algorithm proposed here is a good deal simpler, while producing excellent results.

A small number of recent studies utilize probability models as a mechanism to try to incorporate handling shadows and highlights into the PST formulation. Tang et al. [19] model normal orientations and discontinuities with two coupled Markov random fields (MRFs). They proposed a tensorial belief propagation method to solve the maximum a posteriori problem in the Markov network. Chandraker et al. [20] formulate PST as a shadow labelling problem where the labels of each pixel’s neighbours are taken into consideration, enforcing the smoothness of the shadowed region, and approximate the solution via a fast iterative graph-cut method. Another study [21] employs a maximum-likelihood (ML) imaging model for PST. In their method, an inlier map modelled via MRF is included in the ML model. However, the initial values of the inlier map would directly influence the final result, whereas our methods do not depend on the choice of any prior.

Yang et al. [22] include a dichromatic reflection model into PST and associated method for both estimating surface normals as well as separating the diffuse and specular components, based on a surface chromaticity invariant. Their method is able to reduce the specular effect even when the specular-free observability assumption (that is, each pixel is diffuse in at least one input image) is violated. However, this method does not address shadows and fails on surfaces that mix their own colours into the reflected highlights, such as metallic materials. Moreover, their method also requires knowledge of the lighting chromaticity - they suggest a simple white-patch estimator - whereas in our method, we have no such requirement. Kherada et al. [23] proposed a component-based mapping method. They decompose the captured images into direct and global components - single bounce of light from a surface, as opposed to illumination onto a point that is interreflected from all other points of the scene. They then model matte, shadow and specularity separately within each component. Their method is stated to provide a better appearance reconstruction than the original PTM [1], although at the cost of a much heavier computational load, but depends on a training phase and requires accurate disambiguation of direct and global contributions.

Aside from the polynomial basis, it is possible to use other types of basis function in PTM, as long as they provide a good approximation of the light-reflectance interaction. Spherical harmonics (SH), the angular portion of a set of solutions to the Laplace’s equations defined on a sphere, appear to be a good candidate for this purpose. Due to their appealing mathematical properties, they have been extensively applied in a great variety of topics in computer graphics, such as the modelling of BRDFs [24], early work on image-based rendering and re-lighting [25, 26], BRDF shading [27], irradiance environment maps [28], precomputed radiance transfer [29, 30], distant lighting [31, 32] and lighting-invariant object recognition [33]. However, in the context of PTM, we note that the incoming and outgoing lights are defined only on the upper hemisphere. Therefore, representation of such a hemispherical function using basis functions defined over the full spherical domain introduces discontinuities at the boundary of the hemisphere and requires a large number of coefficients [34]. Thus, it is more natural to map these functions to a basis set defined only over the upper hemisphere. In [34], a HSH basis derived from SH using shifted associated Legendre polynomials was proposed. This basis has been applied in surface modelling under distant illumination [35] and in shape description and reconstruction of surfaces [36]. Recent progress on HSH includes a HSH-based Helmholtz bidirectional reflectance basis [37] and noise-resistant Eigen hemispherical harmonics. In this study, we incorporate the HSH basis as proposed in [34] into the framework of PTM and compare its performance with the polynomial basis.

PTM and other similar reflectance transformation imaging (RTI) methods have found extensive applications in cultural heritage imaging and art conservation. Earl et al. use PTM to capture and visually examine a great variety of ancient artefacts, including bronze busts, coins, paintings, ceramics and cuneiform inscriptions [38–41]. Duffy [42] employed a highlighted RTI method to record the prehistoric rock inscriptions and carvings at the Roughting Linn rock site, UK. Padfield et al. [43] adopted PTM to digitally capture paintings in order to monitor their physical changes during conservation. These applications demonstrate the ability of PTM to visually enhance the captured images via different display modes, most notably specular enhancement and diffuse gain, allowing for inspection of features such as fingerprints and erasure marks that are otherwise much less visually prominent in regular images.

3 Matte modelling using PTM

3.1 Luminance

PTM models smooth dependence of images on lighting direction via polynomial regression. Here we briefly recapitulate PTM as amended by [6]: Suppose n images of a scene are taken with a fixed-position camera and lighting from i=1..n different lighting directions aⁱ=(uⁱ, vⁱ, wⁱ)^T. Let each RGB image acquired be denoted ρⁱ, and we also make use of luminance images, $L^i=\sum _{k=1}^3{\rho }_k^i$. Colour is re-inserted later, as is described in Section 3.4. It is also possible to ‘multiplex’ illumination by combining several lights at once in order to decrease noise [44], but here we simply use one light at a time.

In [6] we use a 6D vector polynomial p for each normalized light direction three-vector a as follows:

$$\mathit{p}\left(\mathit{a}\right)=(u,v,w,u^2,\mathit{\text{uv}},1),\text{where}w=\sqrt{1-u^2-v^2}$$

(1)

This differs from the original PTM formulation [1] in that originally the polynomial used had been (u,v,u²,v²,u v,1), which unfortunately does not model a true Lambertian (linear) surface well since it must warp a non-linear model to suit linear data.

Then at each pixel (x,y) separately, we can seek a polynomial regression six-vector of coefficients c(x,y) in a simple model, regressing lighting directions onto luminance:

$$\left[\begin{array}{c}\mathit{p}\left({\mathit{a}}^1\right)\\ \mathit{p}\left({\mathit{a}}^2\right)\\ \dots \\ \mathit{p}\left({\mathit{a}}^n\right)\end{array}\right]\mathit{c}(x,y)=\left[\begin{array}{c}{L}^1(x,y)\\ L^2(x,y)\\ \dots \\ L^n(x,y)\end{array}\right]$$

(2)

E.g. if n=50, then we could write this as

$$\begin{array}{cccc}\mathit{P}& \mathit{c}(x,y)& =& \mathit{L}(x,y)\\ 50\times 6& 6\times 1& 50\times 1\end{array}$$

(3)

An example dataset (code named Barb) for PTM is displayed in Figure 1b, which was captured with a 50-light dome (i.e. n=50) similar to the one shown in Figure 1a. The dataset Barb has large specular and shadowed regions, which cannot be well addressed by the classical PTM model, and such datasets have typically been avoided. Thus, we find Barb an ideal representative dataset to test the accuracy and/or robustness of a re-lighting method. On other such difficult datasets we have tried, very similar results were found (see [6] for depictions of shiny and shadowed datasets).

3.2 Robust 6D regression

In our recent version of PTM [6], we solve Equation 3 using a robust LMS regression [7]. The purpose of robust regression is to (1) isolate the matte and specular/shadow components and allow the latter to be more cleanly modelled with an additional RBF interpolation stage and (2) identify the non-matte outliers so that more accurate surface normals as well as other reflectance properties can be obtained with LS. The LMS algorithm as applied in [6] is summarized as follows [7]:

While the 6D LMS regression is slow, it is guaranteed to omit distracting features such as specularities and shadows. Due to the 50% breakdown point of LMS, it requires that at least half plus 1 of the luminance observations belong to a base matte reflectance that can be sufficiently addressed by a polynomial model. Fortunately, this requirement is satisfied for most pixels in real-world datasets. This regression method will be referred to Method:LMS in the following text.

3.3 Re-lighting

The re-lighting of images for PTM is fairly straightforward. Given a new light direction a^′ and estimated polynomial coefficients c(x,y), the approximated luminance can be expressed as:

$$L^{\prime }(x,y)=max\left[\mathit{p}\right({\mathit{a}}^{\prime }\left)\mathit{c}\right(x,y),0]$$

(10)

Note that with Method:LMS, c(x,y) was obtained from a trimmed LS where only the matte observations are used. Therefore, the resulting L^′(x,y) is expected to show matte-only contents as well, and non-matte components can be later addressed by other methods (such as the RBF interpolation we will describe in Section 6). This contrasts the robust methods with Method:LS, which uses only PTM to capture both the matte and non-matte components (to some degree) at the same time. Also note that in Equation 4 only luminance is recovered. Colour would be re-introduced by multiplying the chromaticity and the albedo as in Equation 5 as discussed next.

3.4 Colour, normals and albedo

The luminance L consists of the sum of colour components: L=R+G+B. Luminance is given by the shading s (e.g. this could in the simplest case be Lambertian shading, meaning surface normal dotted into light direction) times albedo α: i.e. L=s α. The chromaticity χ is defined as RGB colour ρ, made independent of intensity by dividing by the L₁ norm:

$$\mathit{\rho }=L\mathit{\chi },L=s\alpha ,\mathit{\chi }\equiv \{R,G,B\}/(R+G+B)$$

(11)

Suppose our robust regression below delivers binary weights ω, with ω=0 for outliers. As in [6], once inliers are identified we recover a robust estimate of chromaticity χ as the median of inlier values, for k=1..3:

$${\chi }_k=\underset{i\in (\omega \equiv 1)}{\text{median}}\left({\rho }_k^i/L^i\right)$$

(12)

In addition, an estimate of surface normal n is given by a trimmed PST: with the collection of directions a stored in the n×3 matrix A, suppose ω⁰ is an index variable giving the inlier subset of light directions: ω⁰=(ω≡1). Using just the inlier subset, a trimmed version of PST gives an estimate of normalized surface normal $\hat{\mathit{n}}$ and albedo α via

$$\tilde{\mathit{n}}={\left(\mathit{A}\left({\omega }^0\right)\right)}^{†}\mathit{L}\left({\omega }^0\right);\alpha =\parallel \tilde{\mathit{n}}\parallel ,\hat{\mathit{n}}=\tilde{\mathit{n}}/\alpha$$

(13)

where A^† is the Moore-Penrose pseudoinverse. Other weighting functions are also possible, such as the triangular function used by Method:QUANTILE which we will briefly describe in Section 4.1.

With chromaticity χ in hand, Equation 5 gives RGB pixel values ρ for the interpolated luminance L, and (7) above also gives us the properties albedo α and surface normal $\hat{\mathit{n}}$ intrinsic to the surface.

Institutional users of the PTM approach are indeed interested in appearance modelling for re-lighting, but they are also separately interested in surface properties, especially accurate surface normals, which carry much of the shape information.

4 Robust chromaticity/luminance modes

In this section, we present our first main contribution. As we mentioned in Section 3.2, despite its high robustness LMS can be very slow. Therefore, it is necessary to find a less computationally expensive robust method. Here, we suggest a simplified form of LMS - the mode-finder approach.

4.1 Robust mode-finder algorithm

The basic idea of a mode-finder is first to identify a central value of either luminance or chromaticity, termed ‘mode’ across all the observations at every pixel then perform trimmed LS only using the observations that are with a certain range around the mode. This is a far simpler problem than LMS. For reference, we call this new method Method:MODE, which can be achieved with the following algorithm [7]:

The rationale of Method:MODE is that non-matte outlier observations usually take extreme values in luminance (for instance, shadowed and specular pixels may have an intensity close to 0 and 1, respectively), or their chromaticity may deviate from other matte observations (for instance, specular observations are usually more desaturated whereas shadowed regions appear darker).

Method:MODE may seem to be merely another example of previous thresholding methods. In a typical method of this type [45], the top 10% and the bottom 50% of luminance observations are simply discarded. Then, coefficient values sought are found using a triangular function to weight lighting directions in the resulting range. As in [6], we refer to this simple method as Method:QUANTILE and denote the original PTM method as Method:LS. However, Method:MODE is different from Method:QUANTILE in that the inlier range is calculated based on the distribution of the observation values rather than the empirical values and heuristic triangle functions previously employed. Simply put, Method:MODE lets the data itself dictate what values are in- and outliers.

4.2 Mode-finder versus LMS

In essence, both Method:LMS and Method:MODE attempt to fit a mathematical model to as many data points as possible by minimizing the median of residuals and then identify an inlier range around the fitted model. All observations that fall outside of this range are deemed outliers. The only difference between the two methods is the mathematical model used: Method:LMS fits the data with a 6D polynomial model, whereas Method:MODE approximates the observations with one single scalar constant, i.e. a 1D mathematical model.

To see how the outlier identification works in the two methods, we study a particular pixel in the Barb dataset (marked by a yellow cross in Figure 2a). In Figure 2b,c, the actual luminance observations at this pixel location from 50 lighting conditions are represented as either black solid dots (if they are identified as inliers) or red crosses (for outliers) and are sorted in ascending order. For comparison, the approximated luminance values are shown as blue circles. An observation is classified as outlier if (1) its value is outside the inlier band, marked with green shade enclosed by blue dashed lines or (2) its approximated value (blue circle) is negative. Note that the major difference between Method:LMS (Figure 2b) and Method:Mode (Figure 2c) is that the 6D polynomial model in LMS generates an inlier band that closely approximates the actual data curve, whereas the 1D constant model in Method:MODE creates a wider, horizontal band. Despite this seemingly crucial difference, Method:MODE as a matter of fact correctly captures most of the outliers identified by Method:LMS. Although Method:MODE may throw away more data points than necessary, it would not negatively affect the accuracy of estimated polynomial coefficients since these unnecessarily excluded data points are matte anyway and a robust method is not affected by the sum of squared residuals as in LS.

Figure 2

Comparison of outlier detection with LMS and with mode-finding approach. (a) One original image; consider pixel at yellow ‘x’. (b) Outlier detection with 6D to 1D LMS regression. Here, the pixels are displayed in ascending order sorted by luminance. The approximated values are shown as blue circles. Inlier pixel values (at this 1 pixel, over the set of 50 lights) with estimated $\widehat{L}$ values that fall within the green inlier band are displayed as black dots and outlier measured luminances, including values with negative luminance estimates that fall below the horizontal line at L=0, as black dots with red crosses. The blue dashed lines indicate the boundaries of the inlier band automatically identified by LMS. (c) Outlier detection with luminance-mode finder. Here the blue solid line shows the location of the (scalar) mode, bracketed by a horizontal inlier band; inliers also exclude negative-$\widehat{L}$ lights. (d) Red vs. green chromaticity, with outliers for green mode showing red circles (see Section 4.3).

Figure 3 shows a more detailed comparison on outlier estimation and surface property recovery using LMS and mode-finder. Since there is no ground truth data for these properties available, we simply adopt the results obtained with the full 6D LMS method as our ‘gold standard’ [6] and compare the relative performance of mode-finder against it. Figure 3a displays accuracy of outlier detection in terms of precision, recall and f-statistic, and shows that as long as we use modes for luminance we can achieve a very accurate set of outliers. Results using luminance are shown using white bars. The black bars represent the results obtained by the chromaticity mode, which will be covered in Section 4.3. Figure 3b shows the results for recovered surface normal vectors using outlier detection based on the simpler mode-finder, compared to Method:LMS: the median angular error is 3.03°, which is quite small. Figure 3c shows error in three-vector chromaticity, again measured in terms of angle: the median error is 5.93°, which is quite acceptable. Figure 3d shows errors in albedo - the median is only 0.0037 (where the maximum correct albedo is 1.5855). Such small differences are quite reasonable as a tradeoff with having a much less complex algorithm.

Figure 3

Surface properties recovered with mode-finder compared with 6D LMS. (a) Accuracy of outlier detection of mode-finder compared to 6D LMS. (b,c,d) The deviation in surface normal, chromaticity and albedo, respectively.

4.3 Luminance versus chromaticity modes

As mentioned earlier in Subsection 4.1, the mode-finder can be applied on luminance but as well could be applied to colour components, since non-matte observations tend to have an altered chromaticity. For example, in Figure 2c, we have shown the outliers identified by Method:MODE on luminance. In Figure 2d, we apply mode-finder on green chromaticity only and find that the observations with outlying green components (red circles) tend to have outlying red chromaticities as well. In addition, the chromaticity outliers are also expected to largely overlap with the luminance outliers.

It is also possible to combine outliers obtained from different chromaticities or even mix luminance/chromaticity outliers in the hope of getting a more accurate outlier estimation. For example, we can estimate outliers using green chromaticity (this subset of outlier indices are denoted c_green) and red chromaticity (c_red) at the same time, and then take the outliers c that appear in both c_green and c_red, i.e. c=c_green∩c_red. We refer to such a combined method as ‘green & red’.

Now the question is: which combination of modalities gives the best approximated appearance? We found [46] that in terms of peak signal-to-noise ratio (PSNR) accuracy of the reconstructed appearance for Method:MODE, we have an ordering:

$$\begin{array}{ll}\text{Lum}& >\left(\text{green \& red \& lum}\right)>\text{green}>\left(\text{green \& red}\right)\\ >\text{lum}\left(\text{Method:QUANTILE}\right)\end{array}$$

where ‘ >’ means better accuracy; using luminance alone is always best, (green & red) seems to be slightly worse than green only, and (green & red & lum) is between green and luminance. In comparison, using luminance with Method:QUANTILE has the worst performance.

5 Higher-dimensional LS-based PTM and hemispherical harmonics

In this section, we present our second contribution. First, we investigate what can be gained by increasing the dimensionality of the classical PTM model above 6D without including robust regression. In addition, we apply PTM with different dimensions to model luminance and chromaticity separately. The objective of this part of the investigation is to show that one can, in fact, go quite a long way towards accuracy of appearance modelling using only high-dimensional smooth regression, without the final step of RBF modelling, provided we separate modelling of luminance and chrominance.

Secondly, aside from polynomials, other sets of basis functions can be used to model lighting-surface interaction. One notable example is HSH [34] - it has also been suggested that one could replace a PTM polynomial basis by HSH instead [47]. HSH is mathematically very similar to SH which have already been extensively employed in computer graphics. The key difference between HSH and SH is that HSH is only defined for light directions that live on an upper hemisphere, making it more appropriate for our experimental setup.

The conclusions we reach are that (1) a higher dimension does indeed substantially improve the quality of the reconstructed appearance; (2) if we split the problem into modelling luminance and chrominance separately, rather than applying PTM to each component of colour, then we can reduce the dimensionality for chrominance, compared to that for luminance - we find that 16D for luminance and 9D for chrominance work well; and (3) surprisingly, PTM works better than HSH. Note that every dataset we tried behaved this same way.

5.1 Separation of luminance and chromaticity using LS-based PTM

Our first observation is that the quality of the reconstructed images has a positive correlation with the dimensionality of PTM. Suppose we model luminance only, using an LS-based simple PTM. Figure 4a shows accuracy of the approximated input image set, in terms of PSNR, for different dimensionalities d. In order to calculate the overall PSNR between the original and the approximated set of images, we make the individual images into collages, as the one shown in Figure 1b, and compute the similarity between the original and approximated collages. Here we traverse d values 1, 4, 6, 9 and 16. We see that the reconstructed image quality improves steadily as dimensionality increases for both PTM and HSH (which will be covered in Section 5.2), and in fact PTM produces an acceptable (chosen to be PSNR ≥ 30 dB) reconstruction at d=16. Second, we also investigate modelling the luminance and chromaticity separately, using different dimensionalities for each. (Note that only two of the components of χ need be modelled, since $\sum _{k=1}^3{\chi }_k\equiv 1$). Figure 4b shows results for dimension of luminance versus chrominance, for HSH (coloured surface) and PTM (black circles). We see that while a higher dimension for luminance is important (as in Figure 4a), the accuracy of approximation of chrominance is only mildly dependent upon dimension. The actual PSNR values plotted in Figure 4b are shown in Table 1.

Figure 4

Quality of entire image set reconstructed with non-robust, LS-based PTM and HSH over range of dimensionalities. (a) PSNRs for PTM (black curve) and HSH (red, dashed curve), for luminance images, over values of the basis set dimension; the horizontal blue dashed line indicates PSNR = 30. Here the PSNR value displayed is for the entire set of input images compared to the approximated set. (b) PSNRs for PTM (scattered circles) and HSH (surface) in RGB images versus the dimensions for luminance and chromaticity. (c,d) PSNRs for approximated images for each of the lighting direction images, for PTM and HSH, respectively. PSNR is plotted as against the x and y components of the lighting direction. Blue circles indicate PSNRs for individual reconstructed image in the dataset, and the coloured surface shows an interpolation surface. Here, 16D for luminance and 9D for chromaticity are used.

Table 1 Comparison of PTM and HSH over various dimensionalities

Due to the two observations made above, we conclude that the quality of the reconstructed images is mainly determined by the luminance, rather than the chromaticities. Hence, in order to achieve a high PSNR with a given dimensionality, it is reasonable to assign a higher dimensionality for luminance and a relatively lower dimensionality for chromaticities. Here we adopt d=16 for luminance and d=9 for chromaticities, making the total number of dimensions 16+9×2=34.

5.2 Comparison of higher-dimension PTM and HSH

Using the LS-based approach, we use either a polynomial matrix P or an HSH equivalent, which we denote as S. When we solve Equation 3, we also prudently include some Tikhonov regularization [48] in solving for c. The solution of Equation 3 is thus

$$\mathit{c}={\mathit{P}}^{†}\mathit{L}\text{or}\mathit{c}={\mathit{S}}^{†}\mathit{L}$$

(17)

where ^† indicates forming a pseudoinverse using a small amount of regularization, with Tikhonov parameter (denoted τ) of, say, τ=10⁻³.

We relegate the definition of HSH to Appendix 1. There we list explicitly the definition of the first 16 HSH basis functions, along with the first 16 PTM polynomials.

Recall that in Figure 4a, HSH is consistently outperformed by PTM of the same dimension. Even at a high dimension d=16, HSH still cannot produce an acceptable result. Similar results are shown in Figure 4b and Table 1.

We further compare the PSNR for each individual image in the dataset. Figure 4c,d shows PSNR for approximation of each image in the colour image set, using PTM and HSH, respectively. Here, as described in Section 5.1, d=16 for luminance and d=9 for chrominance are used. We see that as well as producing higher PSNR values, PTM also does not lose too much accuracy for lighting directions with large incident angles (lights low to the object), whereas HSH does very poorly at these boundary points.

In Table 2 we summarize statistics for PSNRs in Figure 4c,d and as well include results for applying PTM or HSH to each component of RGB separately: to be comparable with dimensionality of 16 for luminance and 9 for chromaticity (for each of two components), making a total of 34 dimensions, here we model R,G,B with 11D each. For comparison, we also include results for the RBF modelling in Section 6 below: the PSNR values are not (machine-) infinite because Tikhonov regularization moves the approximation slightly away from exactly reproducing input images.

Table 2 Comparison of modelling luminance + chromaticity and RGB

6 Specularities and shadows: RBF modelling

Following [6] we adopt an RBF network approach for the remaining luminance not explained by the matte model Equation 3. For N-pixel images, the ‘excursion’ H is defined as the set of (N×3×n) non-matte colour values not explained by the R_matte given by the basic PTM matte Equation 3, now extended to functions of the colour channel as well: the approximated colour matte image is given by

$${\mathit{R}}_{\text{matte}}=\mathit{P}\mathit{C}\mathit{\chi },$$

(18)

where C is the collection of all luminance-regression slopes. Since we include colour, all RBF quantities become functions of the colour channel as well. Throughout, we use the mode-finder efficient robust outlier finder to determine coefficients C.

Then a set of non-matte excursion colour values H is defined for our input set of colour images, via H = R − R_matte where R is the (N×3×n) set of input images. We follow [6] in carrying out RBF interpolation for interpolant light directions. But here we use the much faster luminance-mode approach Method:MODE for generating matte images and also for recovering the surface chromaticity, surface normal and albedo.

For a particular input dataset, the RBF network models the interpolated excursion solely based on the direction to a new light a^′: an estimate is given by $\widehat{\eta }=\text{RBF}\left({\mathit{a}}^{\prime }\right)$. Thus, one arrives at an overall interpolant

$$\widehat{R}={\widehat{R}}_{\text{matte}}\left({\mathit{a}}^{\prime }\right)+\widehat{\eta }\left({\mathit{a}}^{\prime }\right)$$

(19)

Since in general we do not possess ground-truth data for acquired image sets, we can characterize the accuracy of appearance-interpolation methods by a leave-one-out analysis. In this approach, we carry out the entire image modelling task but omit, in turn, each of the input set images, thus yielding a modelling dimensionality decreased by 1. Since we know the left-out image’s appearance, we can generate an error characteristic by comparing the interpolated image with the actual one.

We will summarize how to use RBF interpolation and appearance reconstruction in Sections 6.1 and 6.2, respectively. Then in Section 6.3, we present a method to optimize the parameters of the radial Gaussian function, which serves as the third contribution in this work.

6.1 RBF

A brief recapitulation of the RBF calculation is in order, so as to explain the mechanism of developing a leave-one-out error measurement below.

As in [6], we first generate a matte interpolation structure from in-sample input images and then use RBF to model the excursion H, for the part of the input image which cannot be explained by a matte model. So first we model the luminance L, using either PTM or HSH. E.g. if we decide to use a 16D polynomial p(A), then luminance for in-sample images is modelled by L_matte = C (p(A))^†, where C is the set of polynomial coefficients. If there are N pixels and n lights, then L_matte is N×n and C is N×16, and the polynomial term above is 16×n.

We obtain an N×3 set of chromaticities as in Equation 6 from which we can generate a matte colour image model for in-sample images R_matte, for each if the i=1..n lighting directions, via

$${\mathit{R}}_{\text{matte}}^i=\text{diag}\left({\mathit{L}}_{\text{matte}}^i\right)\mathit{\chi },i=1\mathrm{..n}$$

(20)

The dimensionality of R_matte is N×3×n. The set of excursions for all the input images H has this same dimensionality, and H=R−R_matte. Because the RBF modelling adopted in [6] includes a so-called polynomial term (actually, linear here), we have to extend H with a set of N×3×4 zeros. Call this extended excursion H^′.

For interpolation, we need a set of RBF coefficients Ψ^′, with dimensionality N×3×(n+4). We adopt Gaussian RBF basis functions ϕ(∥a−aⁱ∥),i=1..n (although of course other functions might be tried, such as multiquadric or inverse-multiquadric). We call the set ϕ(∥aⁱ−a^j∥) matrix Φ. Then Φ is extended into an (n+4)×(n+4) matrix Φ^′ as in [6].

Then we calculate and store the RBF coefficients Ψ^′ over all the input lights as follows:

$${\mathbf{\Psi }}^{\prime }={\mathit{H}}^{\prime }{\left({\mathbf{\Phi }}^{\prime }\right)}^{†}$$

(21)

where the ^† means the Moore-Penrose pseudoinverse, guarding against reduced rank.

However, here we also extend the pseudoinverse to include some Tikhonov regularization:

$${\left({\mathbf{\Phi }}^{\prime }\right)}^{†}=\left({{\mathbf{\Phi }}^{\prime }}^T{\mathbf{\Phi }}^{\prime }+\tau {\mathit{I}}_{(n+4)}\right){{\mathbf{\Phi }}^{\prime }}^T$$

(22)

with Tikhonov parameter τ. Below, we mean to optimize this parameter using a clever mathematical theorem borrowed for this work.

6.2 Appearance reconstruction

Given a novel lighting direction a, appearance reconstruction from PTM coefficients C and RBF coefficients Ψ^′ is quite straightforward: we generate a matte image by multiplying PTM coefficient matrix C by its corresponding combination of polynomial p(a) and then use recovered chromaticity χ to form a colour matte image. Then we form a new Gaussian function ϕ from new lighting direction a and simply multiply ϕ times the prestored RBF excursion coefficient set Ψ^′ to generate a single-image excursion value η. The Gaussian radial basis function has the explicit form ϕ(a _i ,a _j ,σ)= exp(−r²/σ²), with radius r for light-direction vectors a _i and a _j given by r=∥a _i −a _j ∥.

6.3 Optimization of dispersion σ and of Tikhonov parameter τ

In this subsection, we describe our third contribution, i.e. finding the best values for the Gaussian dispersion σ and the Tikhonov coefficient τ so as to optimize the reconstructed appearance. Since we have no ground truth for real input image sets, we test the accuracy of appearance modelling by simply leaving out one of the n input images at a time and attempting to reconstruct the left-out image.

To this end, here we borrow the work in [49] in determining a best value of the Gaussian dispersion parameter σ to minimize the leave-one-out error. However, here we mean to apply the method given in [49] to a whole image at once and include colour, extend RBF modelling to include the additional polynomial term and, finally and importantly, extend [49] to include Tihkonov regularization and its optimization.

The work [49] defines the optimum σ as that yielding the smallest error in reconstructing a leave-one-out image, using only the information from the other images. E.g. if the input set consists of 50 images, then we follow through matte and then RBF modelling using only 49 images and attempt to reconstruct the 50th image, and then repeat for each of the 50 light directions.

Modelling on the theorem given in [49] in Appendix 2, we generalize the theorem, which is aimed at optimizing RBF over the dispersion parameter σ, to also optimize over Tikhonov parameter τ. The resulting calculation from this theorem is so fast that it is simple to run any unconstrained non-linear optimizer such as the subspace trust-region method [50].

We find that an approximate colour image reconstruction, for the k th leave-one-out image, is simply as follows:

$$\begin{array}{l}\mathit{E}={\mathbf{\Psi }}^{\prime }/\mathit{v}\\ \widehat{\mathit{R}}=\mathit{R}-\mathit{E}\end{array}$$

(23)

where the error image E is simply formed from the RBF coefficients Ψ^′, and a vector v generated as the solution to the following simple equation in terms of the (n+4)×(n+4) identity matrix I:

$${\mathbf{\Phi }}^{\prime }\mathit{v}=\mathit{I}$$

(24)

This theorem means that one can very rapidly assess the error generated in a leave-one-out analysis of RBF modelling. Figure 5a shows the PSNR between the actual input image set and the result of matte plus RBF modelling, for an optimal choice of σ and τ. Unsurprisingly, we see that RBF interpolation does best in the center of the cluster of lighting directions and worse when there is less supporting information, near the boundary of the cluster of light directions. We take as the optimum dispersion σ and Tihkonov parameter value τ as those which deliver the highest leave-one-out median PSNR over the set overall. Table 3 shows PSNR statistics for this leave-out-out RBF test. In comparison, we show in Figure 5b and also in the second line of Table 3 the results of a leave-one-out test using PTM matte modelling alone for dimensions 16 and 9 for luminance and chrominance, with no RBF stage. We notice that in a challenging leave-one-out test for interpolation, PTM does reasonably well. To put these plots in perspective, in Figure 5c, we also show the results for PTM + RBF in a leave-all-in setting: of course, the PSNR for PTM + RBF for leave-all-in is by far the best accuracy. In Figure 5d we show the in-sample correct image closest to the mean value of PSNR values for all leave-one-out RBF modelling, and in Figure 5e,f, we show the interpolants from using PTM + RBF and from using just PTM, respectively. Clearly, RBF provides a substantial boost in visual appearance, although PTM itself (with no RBF stage), with the higher dimensions we have specified, does produce a reasonable image. Nevertheless, qualitatively, using RBF does much better in that, without RBF, specularities are not well modelled and the shadows are wrong.

Figure 5

Leave-one-out test. (a) PSNRs for PTM + RBF. (b) PSNRs for PTM. (c) PSNRs for PTM + RBF, non-leave-one-out test, for comparison. (d) Correct interpolant for lighting direction (e) PTM + RBF interpolant, PSNR = 30.803 (note camera flare from other images in the set). (f) PTM interpolant: the PSNR = 30.763, which is acceptable, but not using RBF results in poor modelling of specular content and wrong shadows.

Table 3 PSNR statistics for leave-one-out test, using PTM + RBF, and using only PTM

7 Conclusions

In this paper, we have set out tests and conclusions that improve PTM modelling for appearance interpolation and surface property recovery. We found that increasing PTM dimensionality has a substantial effect on accuracy, more for the luminance channel than for colour. We found that a dimension of 16 for luminance and 9 for chromaticity, modelling luminance and chromaticity separately, delivered good performance. We found that for determining outliers, we could have almost as good accuracy using a much less burdensome robust 1D ‘location finder’ as in a more accurate but slower robust multivariate processing.

A second stage of modelling using RBF interpolation provides a large boost in accuracy of appearance modelling. Here we showed that Tikhonov regularization in calculating RBF coefficients was important, since we are inverting large matrices; and moreover we incorporated optimizing the Tikhonov parameter into an optimization theorem that had been initially aimed at only generating a best choice of Gaussian dispersion parameter for radial basis function networks.

Future work will include developing a real-time viewer including the new insights gained here.

Appendix 1: hemispherical harmonics

HSH are derived from spherical harmonics (SH) as an alternative set of basis functions on the unit sphere that are particularly aimed at non-negative function values. The familiar SH are defined as [51]

$$Y_l^m(\theta ,\varphi )=K_l^m{e}^{\mathrm{im\varphi }}{P}_l^{\left|m\right|}cos\left(\theta \right),l\in N,-l\le m\le l$$

(25)

where θ∈[0,π] is the altitude angle, and ϕ∈[0,2π] the azimuth angle. $P_l^m$ are the associated Legendre polynomials, orthogonal polynomial basis functions over [−1,+1], and $K_l^m$ are the normalization factors for these.

$$\begin{array}{l}{P}_l^m\left(x\right)=\frac{{(-1)}^m}{2^{l}l!}\sqrt{{(1-x^2)}^m}\frac{d^{(l+m)}}{d{x}^{(l+m)}}{(x^2-1)}^l\\ K_l^m=\sqrt{\frac{(2l+1)(l-|m\left|\right)!}{4\pi (l+|m\left|\right)!}}\end{array}$$

(26)

In the context of computer graphics, real-valued functions as follows are often preferred:

$$Y_l^m=\left\{\begin{array}{ll}\sqrt{2}{K}_l^{m}cos\left(\mathrm{m\varphi }\right)\underset{l}{\overset{m}{P}}(cos\theta ),& m>0\\ \sqrt{2}{K}_l^{m}sin(-\mathrm{m\varphi })\underset{l}{\overset{-m}{P}}(cos\theta ),& m<0\\ K_l^0{P}_l^0(cos\theta ),& m=0\end{array}\right.$$

(27)

However, since in graphics the incident and reflected lights are all distributed on an upper hemisphere, it requires a large number of coefficients to handle the discontinuities at the boundary of the hemisphere when the mapping is represented with basis defined on a full sphere [34]. Thus, it is more natural to use an HSH basis instead. In this study, we used the HSH model proposed in [34]n:

$$H_l^m=\left\{\begin{array}{ll}\sqrt{2}{\tilde{K}}_l^{m}cos\left(\mathrm{m\varphi }\right)\underset{l}{\overset{m}{\tilde{P}}}(cos\theta )& m>0\\ \sqrt{2}{\tilde{K}}_l^{m}sin(-\mathrm{m\varphi })\underset{l}{\overset{-m}{\tilde{P}}}(cos\theta )& m<0\\ {\tilde{K}}_l^0{\tilde{P}}_l^0(cos\theta )& m=0\end{array}\right.$$

(28)

where ${\tilde{P}}_l^m$ and ${\tilde{K}}_l^m$ are the ‘shifted’ associated Legendre polynomials and the hemispherical normalization factors, respectively, defined as follows:

$$\begin{array}{l}{\tilde{P}}_l^m\left(x\right)=P_l^m(2x-1)\\ K_l^m=\sqrt{\frac{(2l+1)(l-|m\left|\right)!}{2\pi (l+|m\left|\right)!}}\end{array}$$

(29)

Now the hemispherical functions are defined only over the upper hemisphere, θ∈[0,π/2],ϕ∈[0,2π].

Figure 6 shows the first three ‘bands’ of the HSH, i.e. l=0..2, and the first 16 functions are stated explicitly in Equation 22.

Figure 6

Visualization of the first three bands of hemispherical harmonics. (a)$H_0^0$. (b)$H_1^{-1}$. (c)$H_1^0$. (d)$H_1^1$. (e)$H_2^{-2}$. (f)$H_2^{-1}$. (g)$H_2^0$. (h)$H_2^1$. (i)$H_2^2$. The distance r from the origin to any point (θ,ϕ,r) on the plot surface is proportional to the value of $H_l^m$ at direction (θ,ϕ), with cyan indicating positive values and purple negative. Red, green and blue indicate x, y and z axes, respectively.

Similarly, we can also consider the polynomial basis in Equation 1 as a set of functions defined on the hemisphere by representing the lighting direction (u,v,w) with spherical polar coordinates: u= sinθ cosϕ, v= sinθ sinϕ, w= cosθ, so e.g. the PTM basis functions in Equation 1 are given by

$$\begin{array}{c}(sin\theta cos\varphi ,sin\theta sin\varphi ,cos\theta ,\stackrel{2}{sin}\theta \stackrel{2}{cos}\varphi ,\stackrel{2}{sin}\theta cos\varphi sin\varphi ,1)\end{array}$$

(30)

For comparison, a selection of nine polynomial terms are visualized as surface plots in Figure 7, and the first 16 polynomial terms are listed in Equation 23

$$\begin{array}{l}{H}_i=H_l^m;i=\left(\right(l+1)l-m)+1;\text{Order}=(l+1):\\ \text{Order 1:}\\ H_1(\theta ,\varphi )=1/\sqrt{(}2\pi )\\ \text{Order 2:}\\ H_2(\theta ,\varphi )=\sqrt{(}6/\pi \left)\right(cos\left(\varphi \right)\sqrt{(}cos\left(\theta \right)-cos{\left(\theta \right)}^2\left)\right)\\ H_3(\theta ,\varphi )=\sqrt{(}3/\left(2\pi \right)\left)\right(-1+2cos\left(\theta \right))\\ H_4(\theta ,\varphi )=\sqrt{(}6/\pi \left)\right(sin\left(\varphi \right)\sqrt{(}cos\left(\theta \right)-cos{\left(\theta \right)}^2\left)\right)\\ \text{Order 3:}\\ H_5(\theta ,\varphi )=\sqrt{(}30/\pi \left)\right(cos\left(2\varphi \right)(-cos(\theta )+cos{\left(\theta \right)}^2))\\ H_6(\theta ,\varphi )=\sqrt{(}30/\pi \left)\right(cos\left(\varphi \right)(-1+2cos(\theta \left)\right)\\ \sqrt{(}cos\left(\theta \right)-cos{\left(\theta \right)}^2\left)\right)\\ H_7(\theta ,\varphi )=\sqrt{(}5/\left(2\pi \right)\left)\right(1-6cos\left(\theta \right)+6cos{\left(\theta \right)}^2)\\ H_8(\theta ,\varphi )=\sqrt{(}30/\pi \left)\right(sin\left(\varphi \right)(-1+2cos(\theta \left)\right)\\ \sqrt{(}cos\left(\theta \right)-cos{\left(\theta \right)}^2\left)\right)\\ H_9(\theta ,\varphi )=\sqrt{(}30/\pi \left)\right((-cos(\theta )+cos{\left(\theta \right)}^2)sin\left(2\varphi \right))\end{array}$$

(31)

Figure 7

Visualization of selected polynomial basis functions. (a) 1, (b) u, (c) v, (d) w, (e) uv, (f) u², (g) uvw, (h) u², (i) u³. The distance r from the origin to any point (θ,ϕ,r) on the plot surface is proportional to the value of the polynomial term P at direction (θ,ϕ), with cyan indicating positive values and purple negative. Red, green, blue indicate x, y, z axes, respectively.

$$\begin{array}{l}\text{Order 4:}\\ H_{10}(\theta ,\varphi )=2\sqrt{(}35/\pi )cos(3\varphi ){(cos(\theta )-cos{\left(\theta \right)}^2)}^{3/2}\\ H_{11}(\theta ,\varphi )=\sqrt{(}210/\pi )cos(2\varphi )\\ (-1+2cos(\theta \left)\right)(-cos(\theta )+cos{\left(\theta \right)}^2)\\ H_{12}(\theta ,\varphi )=2\sqrt{(}21/\pi )cos(\varphi )\sqrt{(}cos(\theta )-cos{\left(\theta \right)}^2)\\ (1-5cos(\theta )+5cos{\left(\theta \right)}^2)\\ H_{13}(\theta ,\varphi )=\sqrt{(}7/\left(2\pi \right)\left)\right(-1+12cos\left(\theta \right)-30\\ cos{\left(\theta \right)}^2+20cos{\left(\theta \right)}^3)\\ H_{14}(\theta ,\varphi )=2\sqrt{(}21/\pi )sin(\varphi )\sqrt{(}cos(\theta )-cos{\left(\theta \right)}^2)\\ (1-5cos(\theta )+5cos{\left(\theta \right)}^2)\\ H_{15}(\theta ,\varphi )=\sqrt{(}210/\pi \left)\right(-1+2cos\left(\theta \right)\left)\right(-cos\left(\theta \right)+\\ cos{\left(\theta \right)}^2)sin(2\varphi )\\ H_{16}(\theta ,\varphi )=2\sqrt{(}35/\pi )sin(3\varphi ){(cos(\theta )-cos{\left(\theta \right)}^2)}^{3/2}\end{array}$$

$$\begin{array}{l}\text{Constant term:}\\ P_1=1\\ \text{Linear terms:}\\ P_2=u=sin\left(\theta \right)cos\left(\varphi \right)\\ P_3=v=sin\left(\theta \right)sin\left(\varphi \right)\\ P_4=w=cos\left(\varphi \right)\\ \text{Quadratic terms:}\\ P_5=u^2=\stackrel{2}{sin}\left(\theta \right)\stackrel{2}{cos}\left(\varphi \right)\\ P_6=\mathit{\text{uw}}=sin\left(\theta \right)\stackrel{2}{cos}\left(\varphi \right)\\ P_7=\mathit{\text{uv}}=\stackrel{2}{sin}\left(\theta \right)cos\left(\varphi \right)sin\left(\varphi \right)\\ P_8=\mathit{\text{vw}}=sin\left(\theta \right)cos\left(\varphi \right)sin\left(\varphi \right)\\ P_9=v^2=\stackrel{2}{cos}\left(\varphi \right)\end{array}$$

(32)

$$\begin{array}{l}\text{Cubic terms:}\\ P_{10}=u^3=\stackrel{3}{sin}\left(\theta \right)\stackrel{3}{cos}\left(\varphi \right)\\ P_{11}=u^{2}v=\stackrel{3}{sin}\left(\theta \right)\stackrel{2}{cos}\left(\varphi \right)sin\left(\varphi \right)\\ P_{12}=u^{2}w=\stackrel{2}{sin}\left(\theta \right)\stackrel{3}{cos}\left(\varphi \right)\\ P_{13}=\mathit{\text{uvw}}=\stackrel{2}{sin}\left(\theta \right)\stackrel{2}{cos}\left(\varphi \right)sin\left(\varphi \right)\\ P_{14}=v^{2}u=\stackrel{3}{sin}\left(\theta \right)\stackrel{2}{sin}\left(\varphi \right)cos\left(\varphi \right)\\ P_{15}=v^{2}w=\stackrel{2}{sin}\left(\theta \right)\stackrel{2}{sin}\left(\varphi \right)cos\left(\varphi \right)\\ P_{16}=v^3=\stackrel{3}{sin}\left(\theta \right)\stackrel{3}{sin}\left(\varphi \right)\end{array}$$

Appendix 2: leave-one-out optimization in RBF

It is useful to state explicitly how the optimization theorem in [49] goes over to the situation when Tikhonov regularization comes into play.

Firstly, we utilize three-band colour image data, rather than scalar data, and process whole images at once using vectorized programming in Matlab. However for clarity, below we state matters as they pertain to a single pixel and in one colour band.

Suppose there are n lights and n input values at a pixel, e.g. for our exemplar dataset n=50. Then we make (n+4)×(n+4) matrix Φ(σ), where here we are explicitly including dependence on a variable dispersion value σ. For the (n+4) vector of excursion values H (extended by four zeros to include the ‘polynomial’ RBF part), we begin by solving for the (n+4) vector set of RBF coefficients ψ, which is the vector solution for the modelling equation

$$\mathit{H}=\mathbf{\Phi }\mathit{\psi }$$

However, instead of simply using a matrix inverse in order to guard against numerical instability, we make use of the Tikhonov regularized inverse from Equation 13:

$$\mathit{\psi }=\mathbf{\Phi }{(\sigma ,\tau )}^{†}\mathit{H}$$

so that in fact we generate only approximate, not exact, approximations $\widehat{\mathit{H}}$ for in-sample lighting directions:

$$\widehat{\mathit{H}}=\mathbf{\Phi }\mathit{\psi }=\mathbf{\Phi }{\mathbf{\Phi }}^{†}\mathit{H}$$

Then the main task is interpolation to any new light a^′ via

$${\eta }^{\prime }=\sum _{j=1}^{(n+4)}{\psi }_j^T\varphi ({\mathit{a}}_j-{\mathit{a}}^{\prime })$$

where η^′ is the scalar value of interpolated excursion (for this pixel and colour channel).

Now we mean to consider the leave-one-out problem, meaning that all the matrices and vectors have extent (n + 3) because the k th input-image case has been omitted. Suppose we denote this case using superscript (k). That is, we aim for a solution ψ^(k) of

$${\mathit{H}}^{\left(k\right)}={\mathbf{\Phi }}^{\left(k\right)}{\mathit{\psi }}^{\left(k\right)}$$

(33)

Firstly, consider the following Lemma: if vector v has v _k =0, then

$$\text{if}\mathit{A}\mathit{v}=\mathit{b},\text{then}{\mathit{A}}^{\left(k\right)}{\mathit{v}}^{\left(k\right)}={\mathit{b}}^{\left(k\right)}$$

That is, if we know the not-reduced-dimension equation holds, then for the special situation in which v _k =0, we can simply omit whatever value b _k may take on, for the reduced-dimension problem indicated by (k).

Now consider an auxiliary full-dimension vector v defined such that

$$\mathit{v}={\mathbf{\Phi }}^{+}{\mathit{e}}_k$$

where e _k is the k th column of the unit matrix.

Now define a new vector

$$\mathit{\beta }=\mathit{\psi }-({\psi }_k/v_k)\mathit{v}$$

Notice that the k th component of β is zero.

Now evaluate Φ β:

$$\mathbf{\Phi }\mathit{\beta }=\mathbf{\Phi }\mathit{\psi }-({\psi }_k/v_k)\mathbf{\Phi }\mathit{v}=\widehat{\mathit{\eta }}-({\psi }_k/v_k){\mathit{e}}_k$$

Hence, by our lemma, β is the sought solution for the leave-one-out set of coefficients ψ^(k); however, this statement is approximate and not exact because $\widehat{\mathit{\eta }}$ is only approximately (but very close to being equal to) η.

So in order to optimize on σ and τ, we need only to generate the error estimate E _k for the k th case,

$$E_k=-({\psi }_k/v_k)$$

for each of the k=1..n left-out lights, and apply some appropriate error measure such as median (E _k ) for choosing the least-error solution:

$$\underset{\{\sigma ,\tau \}}{min}\underset{k=1}{\overset{n}{\text{median}}}{E}_k(\sigma ,\tau )$$

(34)

In practise, we found that utilizing this leave-one-out calculation is very fast and generates smaller interpolation errors when the resulting solution pair {σ, τ} is used for general interpolation for the dataset being optimized for by this leave-one-out procedure.