Surface Detail

Applying any of the shading models described to planar or bi-cubic surfaces produces smooth, uniform surfaces - in marked contrast to most of the real surfaces we see and feel.  In these notes a variety of methods are discussed for simulate this missing surface detail.

Surface Detail Polygons

The simplest approach adds gross detail through the use of surface detail polygons to show features (such as doors, windows, and lettering) on a base polygon (such as the side of a building).  Each surface detail polygon is coplanar with its base polygon, and is flagged so that it does not need to be compared with other polygons during visible-surface determination.  When the base polygon is shaded, its surface detail polygons and their material properties take precedence for those parts of the base polygon that they cover.

Texture Mapping

As detail becomes finer and more intricate, explicit modeling with polygons or other geometric primitives becomes less practical.  An alternative is to map an image, either digitized or synthesized, onto a surface, a technique pioneered by Catmull [1974] and refined by Blinn and Newell [1976].  This approach is known as texture mapping or pattern mapping; the image is called a texture map, and its individual elements are often called texels.  The rectangular texture map resides in its own (u, v) texture coordinate space.  Alternatively, the texture may be defined by a procedure.  At each rendered pixel, selected texels. are used either to substitute for or to scale one or more of the surface's material properties, such as its diffuse color components.  One pixel is often covered by a number of texels.  To avoid aliasing problems, we must consider all relevant texels.

As shown in the figure below, texture mapping can be accomplished in two steps.  A simple approach starts by mapping the four corners of the pixel onto the surface.  For a bi-cubic patch, this mapping naturally defines a set of points in the surface's (s,t) coordinate space.  Next, the pixel's corner points in the surface's (s, t) coordinate space are mapped into the texture's (u, v) coordinate space.  The four (u, v) points in the texture map define a quadrilateral that approximates the more complex shape into which the pixel may actually map due to surface curvature.  We compute a value for the pixel by summing all texels that lie within the quadrilateral, weighting each by the fraction of the texel that lies within the quadrilateral.  If a transformed point in (u, v) space falls outside of the texture map, the texture map may be thought of as being replicated,.

Rather than always use the identity mapping between (s, t) and (u, v), we can define a correspondence between the four comers of the 0-to-1 (s, t) rectangle and a quadrilateral in (u, v).  When the surface is a polygon, it is common to assign texture map coordinates directly to its vertices.  Since, as we have seen, linearly interpolating values across arbitrary polygons is orientation dependent, polygons may be decomposed into triangles first.  Even after triangulation, however, linear interpolation will cause distortion in the case of perspective projection.  This distortion will be more noticeable than that caused when interpolating other shading information, since texture features will not be correctly foreshortened.  We can obtain an approximate solution to this problem by decomposing polygons into smaller ones, or an exact solution, at greater cost, by performing the perspective division while interpolating.

The approach just described assumes square pixel geometry and simple box filtering.  It also fails to take into account pixels that map to only part of a surface.  Feibush, Levoy, and Cook [1980] address these problems for texture mapping polygons.  Think of the square pixel in the figure above as the bounding rectangle of the support of an arbitrary filter centered at a pixel.  Pixels and texels can then be treated as points.  In effect, all texels that lie within the mapped intersection of the transformed bounding rectangle and polygon are selected, and these texels' coordinates are transformed into the coordinate system of the bounding rectangle.  Each texel's transformed coordinates are used to index into a filter table to determine the texel's weighting, and the weighted average of the texel intensities is computed.  This weighted average must in turn be weighted by the percentage contribution that the polygon makes to the pixel's intensity.  The process is repeated for each polygon whose projection intersects the pixel, and the values are summed.

The Feibush, Levoy, and Cook algorithm can be quite inefficient.  Consider mapping a checkerboard pattern onto an infinite ground plane.  An extremely large number of texels may have to be weighted and summed just to texture a single distant ground plane pixel. One solution to this problem is to pre filter the texture and to store the results in a way that is space efficient and that allows quick determination of the weighted average of texels mapping to a pixel.  Algorithms by Williams [1983], Crow [1984], Glassner [1986], and Heckbert [1986] that take this approach.  Catmull and Smith developed an efficient technique [1980] for mapping an entire texture map directly to a surface.

Bump Mapping

Texture mapping affects a surface's shading, but the surface continues to appear geometrically smooth.  If the texture map is a photograph of a rough surface, the surface being shaded will not look quite right, because the direction to the light source used to create the texture map is typically different from the direction to the light source illuminating the surface.  Blinn [1978] developed a way to provide the appearance of modified surface geometry that avoids explicit geometrical modeling.  His approach involves perturbing the surface normal before it is used in the illumination model, just as slight roughness in a surface would perturb the surface normal.  This method is known as bump mapping, and is based on texture mapping.

A bump map is an array of displacements, each of which can be used to simulate displacing a point on a surface a little above or below that point's actual position.  Let us represent a point on a surface by a vector P, where P = [x(s, t), y(s, t), z(s, t)].  We call the partial derivatives of the surface at P with respect to the surface's s and t parameterization axes, P_s and P_t.  Since each is tangent to the surface, their cross product forms the (unnormalized) surface normal at P.  Thus,

            N = P_s x P_t

We can displace point P by adding to it the normalized normal scaled by a selected  bump map value B. The new point is

           P' = P +  B N / |N|

Blinn shows that a good approximation to the new (unnormalized) normal N' is

            N' = N + ( B_u (N x P_t) - B_v (N x P_s)) / |N|

where B_u and B_v are the partial derivatives of the selected bump map entry B with respect to the bump map parameterization axes, u and v.  N' is then normalized and substituted for the surface normal in the illumination equation.  Note that only the partial derivatives of the bump map are used, not its values.  Bilinear interpolation can be used to derive bump map values for specified (u, v) positions, and finite differences can be used to compute B_u and B_v.

The results of bump mapping can be quite convincing.  Viewers often fail to notice that an object's texture does not affect its silhouette edges.  The following image is an example of bump mapping.

Unlike texture mapping, aliasing cannot be dealt with by filtering values from the bump map, since these values do not correspond linearly to intensities; filtering the bump map just smoothes out the bumps.  Instead, sub pixel intensities may be computed and filtered for each pixel, or some pre filtering may be performed on the bump map to improve gross aliasing.

Other Mapping Techniques

Although 2D mapping can be effective in many situations, it often fails to produce convincing results.  Textures frequently betray their 2D origins when mapped onto curved surfaces, and problems are encountered at texture "seams." For example, when a wood-grain texture is mapped onto the surface of a curved object, the object will look as if it were painted with the texture.  Peachey [1985] and Perlin [1985] have investigated the use of solid or 3D textures for proper rendering of objects "carved" of wood or marble, as exemplified in the images below. In this approach,  the texture is a 3D function of its position in the object.

Other surface properties can be mapped as well.  For example, Gardner [1984] has used transparency mapping to make impressionistic trees and clouds from otherwise simple shapes, such as objects formed from groups of quadric surfaces.

Cook has implemented displacement mapping, in which the actual surface is displaced, instead of only the surface normals [1984]; this process, must be carried out before visible surface determination.

We have made the tacit assumption that the process of shading a point on an object is unaffected by the rest of that object or by any other object.  But an object might in fact be shadowed by another object between it and a light source; might transmit light, allowing another object to be seen through it; or might reflect other objects, allowing another object to be seen because of it.

More Notes on Mapping Techniques

Texture mapping was one of the first developments towards making images of three-dimensional objects more interesting and apparently more complex.  The original motivation for texture mapping was to diminish the 'shiny plastic' effects produced by using the simple Phong reflection model and to enable different objects to exhibit different surface properties (apart from the trivial distinction of color).

There are three major considerations in texture mapping:

 (1)  What attribute or parameter of the model or object is to be modulated to produce the desired textural effect?

 (2)  How is the texture mapping to be carried out? Given that a texture is defined in a texture domain and an object exists as world space data, we need to define a mapping between these domains.

 (3)  Texture mapping requires special anti-aliasing treatment because it tends to produce worse aliasing artifacts than other techniques associated with image synthesis.

These three considerations are dealt with in order.

Object attributes modulated for texture

In a review paper Heckbert (1986) categorizes the parameters that can be modulated to provide a textural impression.  A modified version of this categorization is as follows:

Surface color (diffuse reflection coefficient(s)):  This is the most commonly used parameter for texture mapping and involves, for example, in a simple reflection model, modulating the diffuse coefficient.  The first example of this was Catmull (1974).

Specular and diffuse reflection (environment mapping):  This method is now known as environment mapping and has come to be regarded as a separate technique rather than as a category of texture mapping.  It was first developed by Blinn and Newell (1976).

Normal vector perturbation: This is an elegant device, developed again by Blinn (1978) that 'tricks' a simple reflection model into producing what appears to be a textured surface.

Specularity:  Although this effect was reported by Blinn (1978) it does not seem to have been used to any great extent.  The attribute modulated is the surface roughness function.  An example of a surface with variable shininess is an object painted with textured paint.

Transparency: An example of the modulation of transparency is given in Gardener (1985).  This particular example is interesting in that it is not strictly a method of applying texture to an object but a method of generating a complete object - a cloud - using a mathematical texture function to modulate the transparency of an ellipsoid.

One can further distinguish among these methods by noting that all the methods except environment mapping 'cement' a texture onto the object and the texture then becomes a 'permanent' part of the object, invariable under object transformations and viewing conditions.  Environment mapping, in contrast, depends on where the object is placed in an environment and this is a further justification for considering environment mapping as a technique distinct from texture mapping.

Texture mapping: general

Texture mapping is the process of transforming a texture onto the surface of a three-dimensional object.  The possibilities of texture mapping are summarized in the figure below, where it is indicated that the texture domain can be one dimensional, two dimensional or three dimensional.  Although two-dimensional texture domains are the most popular (they can be constructed, for example, from scanned images, thus offering an infinite variety of texture patterns), mapping from one-dimensional or
three-dimensional domains is generally considerably easier.

This figure shows that two major implementations are possible:

(1)    mapping from texture space to object space (usually a two-dimensional to three-dimensional transformation) and then
        performing the viewing projection to screen space;
(2)    mapping directly from texture to screen space (usually a two-dimensional to two-dimensional transformation).

Most attention has been paid to the second method and an inverse mapping is commonly employed, where the screen space is uniformly sampled and the inverse or pre-image of a pixel in texture space is formed.  The area thus produced in the texture domain is then sampled and filtered and a texture value returned to the pixel.  Filtering methods, dealt with below, are easier to implement for inverse mapping.

The difficulties that occur in texture mapping arise from the situation shown in the figure below. The transformation from the screen domain to the texture domain is non-linear and in general this results in a pixel 'pre-image' that is a curvilinear quadrilateral whose shape varies as the pixel position changes.

As well as the aliasing artifacts that occur, specifying a mapping for a polygon mesh object or non-parametric object is difficult.  One-dimensional and three-dimensional texture domains have the advantage that the mapping is trivial, but the textural effects that can be obtained are generally much more restricted than in two-dimensional texture domains where any two-dimensional image can be used as a texture pattern.

Bump Mapping

This is an important technique because it simulates a wrinkled or dimpled surface rather than modulating the color of a flat surface.  Apparent surface textures are produced by 'tricking' the reflection model with a perturbed surface normal.  Its single disadvantage is that the surface perturbations do not appear on the silhouette edge.  Blinn (1978) first noted that 'the effect of wrinkles on the perceived intensity is primarily due to their effect on the direction of the surface normal (and thus the light reflected) rather than their effect on the position of the surface'. This observation led to the development of a technique, known as bump mapping, that modifies the surface normal N by a perturbation vector D prior to its use in intensity calculations:

       N'= N + D

N' is derived from a texture function in the following way.  If O(u,v) is a parameterized function representing an object surface where O(u,v) is the position vector for a point on the surface, then

is the normal at a point on the surface. (The partial derivatives lie in the tangent plane to the surface and their cross-products define the surface normal.) T(u, v) is a two-dimensional texture function known as a bump map.  This function is the source texture function that is used to control the perturbation of the original surface normal.  The relationship between D and T(u, v) is now derived.

First we use T(u, v) to add a small increment to O:

     O'(u,v) = O (u,v) + T(u,v) N / |N|

This displaces a surface point in the direction of its surface normal.  The
normal vector to the perturbed surface is given by taking the cross-
products of the partial derivatives of O':

If T is small (that is, the height of the surface irregularities is small
compared with the spatial extent of the surface) the last term in each
equation can be ignored, and

The first term is just the normal to the unperturbed surface and the last
term is always zero, so we have

or
        N'= N + D

The process is normally implemented by pre calculation in a look-up table.  A graphic representation of the process is shown in the figure below for a function of a single parameter interacting with a one-dimensional bump map.

A one-dimensional example of the stages involved in bump mapping

Anti-allasing and texture mapping

Aliasing artifacts are extremely problematic in texture mapping. Artifacts are highly noticeable, particularly in texture that exhibits coherence or periodicity, and originates from point sampling.  For example, unless inverse mapping is accurately performed then neither the shape nor the area of the mapped pixel area in the texture domain is known.  Accurate mapping produces, in general, a curvilinear quadrilateral.  In extreme cases point sampling without filtering will lead to extreme errors.  A large number, perhaps thousands, of texels (an element that is the smallest square area in the texture domain) will map into a single pixel. This case can arise when a number of texels map onto a surface, but the projection of the surface in screen space is small, either because of its depth or because of its orientation with respect to the viewing direction.

A number of points arise from these considerations.' Correct filtering of non linearly mapped areas requires space variant filters, that is filters whose shape and area change as they move across the texture domain.  Because of the prominence of aliasing artifacts that arise from texture mapping, it is important to address the aliasing problem 'correctly'.  Approximate general anti-aliasing solutions, such as super- sampling, may not be effective.

Approximations to space variant filters

There are a number of empirical approaches to the space variant filtering problem.  In the previously mentioned subdivision patch renderer, Catmull (1974) computes an unweighted average of the texels corresponding to a screen pixel.  The shape of the area mapped in the texture domain is deemed to be a quadrilateral.  Blinn and Newell.(1976) use a similar approach involving a pyramid weighting function with the pyramid distorted to fit the approximating quadrilateral.  A method that is claimed to be both economical and accurate is the elliptical weighted average (EWA) filter proposed by Greene and Heckbert (1986).  The basis of this method is to approximate square pixel areas in screen space with a circle.

Pre filtering techniques

The computational cost of the previously discussed filtering techniques is always some function of the area of the inverse mapped pixel in texture space.  Using such methods in a context where, for example, large texture areas are mapped into small screen areas is costly.  Pre filtering techniques are approaches where cost does not grow in proportion to mapped texture area.

An elegant pre filtering technique that has achieved some popularity is Williams (1983) 'mip-mapping' scheme.  Instead of a texture domain comprising a single image, Williams uses many images, all derived by averaging down the original image to successively lower resolutions.  Each image in the sequence is at exactly half the resolution of the previous.  A possible texture domain layout is shown in the figure below and is effectively a three-dimensional database with a third parameter, D, used to select the particular image resolution required.  Plate 13 shows a mipmap that was used in conjunction with Catmull's mapping method, described previously, to produce Plate 13 (top left).

The term 'mip' is derived from the latin multum in parvo or many things in a small place.  In a low resolution version of the image each texel represents the average of a number of texels.  By a suitable choice of D, an image at appropriate resolution is selected and the filtering cost remains constant - the many texels to one pixel cost problem being avoided.  In this way the original texture is filtered and, to avoid discontinuities between the images at varying resolutions, different levels are also blended.  Blending between levels occurs when D is selected.  The images are discontinuous in resolution but D is a continuous parameter. Linear interpolation is carried out from the two nearest levels. Williams selects D from:

This method of pre filtering is equivalent to approximating a texture quadrilateral with a square.  The figure below graphically highlights the approximation implicit in this technique, and compares this method with the more accurate methods described in the previous section.  Anti-aliasing is always a compromise.

Another implication of square filtering is that it presupposes that the required compression is symmetric.  Since, in this scheme, the addressing is based on the axis of maximum compression, for surfaces that curve away from the viewer fuzziness may result (although Williams claims that in animated sequences where areas collapse into long thin edges the quality of the mapping is not noticeably degraded).

Crow (1984) proposes a generalization of Williams technique that uses a single table from which a virtually continuous range of texture densities can be drawn.  It is pointed out by Crow that his method is equivalent to mip-mapping where each texture intensity is replaced by a value representing the sum of the intensities of all texels contained in the rectangle defined by the texel of interest.  This table is sampled in four places for any rectangle and a sum of texel intensities found from a sum and two differences taken from the table.

Environment mapping

This term refers to the process of reflecting the surrounding environment in a shiny object.  It tends to be classified as a texture technique, although it is distinguished from 'normal' texture mapping in that the pattern seen on an object is a function of the view vector V. A particular detail in the environment will move across the object as V changes.  En-vironment mapping can also be considered as a simplification of ray tracing, where only the reflected ray is traced and the process is terminated at a depth of two.

The principle of the technique is shown in the following figure.

For a perfect mirror object the view ray V is reflected from the surface of the object, giving a reflected view ray:

     V_r= V + 2N cos (theta)

where theta is the angle between the surface normal N and V. The intersection of this ray with a surface, such as the interior of a sphere that contains an image of the environment to be reflected in the object, gives the shading attributes for the point 0 on the object surface.

In practice four rays through the pixel point define a reflection 'cone' with a quadrilateral cross-section. The region that subtends the environment map is then filtered to give a single shading attribute for the pixel.

Environment maps are usually pre filtered and transformed into two-dimensional images.  Here we are effectively reducing a three-dimensional problem to a two-dimensional one.  This reduction is pre computed and stored as an environment map.  The illumination as seen from a point on an object surface is then given by a single indexing operation into the map rather than as a three-dimensional object space calculation - as in ray tracing.  Environment mapping can also be seen as a technique that extends the limited illumination possibilities in the n (small integer) point light source Phong shading technique.  An environment can be viewed as an infinity of light sources and a map can represent any arbitrary geometry of light sources such as an indoor or outdoor scene.  From this point of view geometrically distributed light sources such as strip lights are just rectangles of high intensity white values in the environment map.

The first use of environment mapping was by Blinn and Newell (1976) wherein the sphere interior technique was used and a latitude-longitude map created.  The practical difficulty with this approach is in the production of the environment map.  Pre distorting an environment to fit the interior surface of a sphere is difficult.  Another disadvantage of this technique is that it is only geometrically accurate for small objects based on the center of the sphere.  As the object size becomes large with respect to the environment sphere, or the object is positioned a long way from this center, the geometric distortion increases.

Using the reflected view vector, V,, it is a simple matter to index into a latitude-longitude environment map.  If E(i, j) (0=< i =< 1, 0 =< j =< 1) is a point on the map and the reflected view vector is:

   Vr = (V_rx, V_ry, V_rz)

then

        i= 0.5 [1 + (1/pi) tan^-1(V_rx/V_rz)]

        j = 0.5 + 0.5V_ry

Recent developments in environment mapping (for example, Miller and Hoffman (1984) and Greene (1984)) treat the environment as a set of two-dimensional projections.  Using the interior of a cube as an environment map is more convenient than using a sphere.  It means, for example, that a map can be created from a real environment by photographing or frame grabbing the environment in six directions using a flat field lens with a 90' view field.  Alternatively, six views of a simulated environment can be generated by a rendering system.  Miller and Hoffman use the cube projection as an intermediate step in creating a single latitude-longitude map.  The trade-off between using a latitude-longitude map and a six component cube map are concerned with the nature of the shapes that are projected by the reflection cone in each domain.  These are fully discussed by Greene.

Greene points out that, although environment mapping is a less versatile technique than ray tracing, it eliminates certain ray tracing problems.  It is far less expensive, handles diffuse illumination 'correctly', and can use texture filtering to overcome the ray tracing aliasing artifacts that arise from point sampling.  In particular, diffuse and specular surface illumination is found by pre filtering regions of two-dimensional diffuse and specular environment maps.  Integration over a two-dimensional  environment map projection, rather than over a three-dimensional environment (which is what anti-aliasing operations in ray tracing do), can produce gross errors, but Greene points out that 'the subjective quality of reality cues produced by environment mapping may be superior to those produced by a naive ray tracer'.  An obvious disadvantage of environment mapping is that an object will not normally be able to contain reflections of itself, since the object will not appear in the environment map.

The above discussion applies to specular illumination environment maps.  Miller and Hoffman have shown how to create a diffuse illumination environment map.  The shading attributes obtained for a point on the surface then consist of both a diffuse and a specular component and the diffuse component gives an approximate solution to the global interaction of diffuse light that is not catered for in ray tracing. The specular-diffuse surface attributes or coefficients then select the proportion of the contribution from each map.  A perfect diffuse surface would only have a contribution from the diffuse map and a perfect mirror object would only access the specular map:

        I = k_d D(N) + k_sS(V_r)

where D is the contribution from the diffuse map, indexed by the surface normal, and S is the contribution from the specular map, indexed by the reflected view vector.

A diffuse environment map is effectively a highly blurred version of the specular environment map.  It is created by convolving an illumination map with a Lambert's law cosine function.  Each point on the diffuse map is a weighted sum of all illumination on the surface of a hemisphere centred at that point.  The diffuse map is indexed by the surface normal N. A useful attribute of the diffuse map is that it contains no high frequency components and can thus be stored at extremely low resolution.