### 12.2 BACKGROUND AND PREVIOUS WORK

Lighting and appearance have been studied in many forms almost since the beginning of research in computer vision and graphics, as well as in a number of other areas. Horn's classic text [27] provides background on work in vision. In this section, we only survey the most relevant previous work, which relates to the theoretical developments in this chapter. This will also be a vehicle to introduce some of the basic concepts on which we will build.

#### 12.2.1 Environment Maps in Computer Graphics

A common approximation in computer graphics, especially for interactive hardware rendering, is to assume distant illumination. The lighting can then be represented as a function of direction, known in the literature as an environment map. Practically, environment maps can be acquired by photographing a chrome-steel or mirror sphere, which simply reflects the incident lighting. Environment mapping corresponds exactly to the distant-illumination assumption we make in this chapter. Another common assumption, which we also make, is to neglect *cast shadows* from one part of the object on another. These should be distinguished from *attached shadows* when a point is in shadow because the light source is behind the surface (below the horizon). We will explicitly take attached shadows into account.

In terms of previous work, Blinn and Newell [7] first used environment maps to efficiently find reflections of distant objects. This technique, known as reflection mapping, is still widely in use today for interactive computer-graphic rendering. The method was greatly generalized by Miller and Hoffman [46] (and later Greene [21]). They introduced the idea that one could acquire a real environment by photographing a mirror sphere. They also precomputed diffuse and specular reflection maps, for the corresponding components of surface reflection. Cabral et al. [9] later extended this general method to computing reflections from bump-mapped surfaces, and to computing environment-mapped images with more general reflective properties [10] (technically, the bidirectional reflectance distribution function or BRDF [52]). Similar results for more general materials were also obtained by Kautz et al. [32, 33], building on previous work by Heidrich and Seidel [25]. It should be noted that both Miller and Hoffman [46], and Cabral et al. [9, 10] qualitatively described the reflection maps as obtained by convolving the lighting with the reflective properties of the surface. Kautz et al. [33] actually used convolution to implement reflection, but on somewhat distorted planar projections of the environment map, and without full theoretical justification. In this chapter, we will formalize these ideas, making the notion of convolution precise, and derive analytic formulae.

#### 12.2.2 Lighting-Insensitive Recognition in Computer Vision

Another important area of research is in computer vision, where there has been much work on modeling the variation of appearance with lighting for robust lighting-insensitive recognition algorithms. Some of this prior work is discussed in detail in the excellent Chapter 11, *on the effect of illumination and face recognition,* by Ho and Kriegman in this volume. Illumination modeling is also important in many other vision problems, such as photometric stereo and structure from motion.

The work in this area has taken two directions. One approach is to apply an image-processing operator that is relatively insensitive to the lighting. Work in this domain includes image gradients [8], the direction of the gradient [11], and Gabor jets [37]. While these methods reduce the effects of lighting variation, they operate without knowledge of the scene geometry, and so are inherently limited in the ability to model illumination variation. A number of empirical studies have suggested the difficulties of pure image-processing methods, in terms of achieving lighting insensitivity [47]. Our approach is more related to a second line of work that seeks to explicitly model illumination variation, analyzing the effects of the lighting on a 3D model of the object (here, a face).

##### Linear 3D Lighting Model Without Shadows

Within this category, a first important result (Shashua [69], Murase and Nayar [48], and others) is that, for Lambertian objects, in the absence of *both* attached and cast shadows, the images under all possible illumination conditions lie in a three-dimensional subspace. To obtain this result, let us first consider a model for the illumination from a single directional source on a Lambertian object.

(1)$B=\rho L(\omega \cdot n)=L\cdot N,$

where *L* in the first relation corresponds to the intensity of the illumination in direction ω, ** n** is the surface normal at the point, ρ is the surface albedo and

*B*is the outgoing radiosity (radiant exitance) or reflected light. For a nomenclature of radiometric quantities, see a textbook, like McCluney [45] or Chapter 2 of Cohen and Wallance [12].

^{1}It is sometimes useful to incorporate the albedo into the surface normal, defining vectors

**= ρ**

*N***and correspondingly**

*n***=**

*L**Lω, so*we can simply write (for Lambertian surfaces in the absence of all shadows, attached or cast)

*B*=

**.**

*L. N*Now, consider illumination from a variety of light sources *L*_{1}, *L*_{2} and so on. It is straightforward to use the linearity of light transport to write the net reflected light as

(2)$B=\sum _{i}{L}_{i}\cdot N=\left(\sum _{i}Li\right)\cdot N=\stackrel{\u02c6}{L}\cdot N,$

where $\stackrel{\u02c6}{L}=i{\sum}_{i}{L}_{i}.$. But this has essentially the same form as Equation 1. Thus, in the absence of all shadows, there is a very simple linear model for lighting in Lambertian surfaces, where we can replace a complex lighting distribution by the weighted sum of the individual light sources. We can then treat the object as if lit by a *single* effective light source $\stackrel{\u02c6}{L}$.

Finally, it is easy to see that images under all possible lighting conditions lie in a 3D subspace, being linear combinations of the Cartesian components *N _{x}*,

*N*,

_{y}*N*, with $B={\stackrel{\u02c6}{L}}_{x}{N}_{x}+{\stackrel{\u02c6}{L}}_{y}{N}_{y}+{\stackrel{\u02c6}{L}}_{z}{N}_{z}$. Note that this entire analysis applies to all points on the object. The basis images of the 3D lighting subspace are simply the Cartesian components of the surface normals over the objects, scaled by the albedos at those surface locations.

_{z}^{2}

The 3D linear subspace has been used in a number of works on recognition, as well as other areas. For instance, Hayakawa [24] used factorization based on the 3D subspace to build models using photometric stereo. Koenderink and van Doorn [35] added an extra ambient term to make this a 4D subspace. The extra term corresponds to perfectly diffuse lighting over the whole sphere of incoming directions. This corresponds to adding the albedos over the surface themselves to the previous 3D subspace, i.e., adding $\rho =\Vert N\Vert =\sqrt{{N}_{x}^{2}+{N}_{y}^{2}+{N}_{z}^{2}}$.

##### Empirical Low-Dimensional Models

These theoretical results have inspired a number of authors to develop empirical models for lighting variability. As described in the introduction, one takes a number of images with different lighting directions, and then uses standard dimensionality-reduction methods like principal-component analysis. PCA-based methods were pioneered for faces by Kirby and Sirovich [34, 72], and for face recognition by Turk and Pentland [78], but these authors did not account for variations due to illumination.

The effects of lighting alone were studied in a series of experiments by Hallinan [23], Epstein et al. [15], and Yuille et al. [80]. They found that for human faces, and other diffuse objects like basketballs, a 5D subspace sufficed to approximate lighting variability very well. That is, with a linear combination of a mean and 5 basis images, we can accurately predict appearance under arbitrary illumination. Furthermore, the form of these basis functions, and even the amount of variance accounted for, were largely consistent across human faces. For instance, Epstein et al. [15] report that for images of a human face, three basis images capture 90% of image variance, while five basis images account for 94%.

At the time however, these results had no complete theoretical explanation. Furthermore, they indicate that the 3D subspace given above is inadequate. This is not difficult to understand. If we consider the appearance of a face in outdoor lighting from the entire sky, there will often be attached shadows or regions in the environment that are not visible to a given surface point (these correspond to lighting below the horizon for that point, where · ω ** n** < 0).

##### Theoretical Models

The above discussion indicates the value of developing an analytic model to account for lighting variability. Theoretical results can give new insights, and also lead to simpler and more efficient and robust algorithms.

Belhumeur and Kriegman [6] have taken a first step in this direction, developing the illumination-cone representation. Under very mild assumptions, the images of an object under arbitrary lighting form a convex cone in image space. In a sense, this follows directly from the linearity of light transport. Any image can be scaled by a positive value simply by scaling the illumination. The convexity of the cone is because one can add two images, simply by adding the corresponding lighting. Formally, even for a Lambertian object with only attached shadows, the dimension of the illumination cone can be infinite (this grows as *O*(*n*^{2}), where *n* is the number of distinct surface normals visible in the image). Georghiades et al. [19, 20] have developed recognition algorithms based on the illumination cone. One approach is to sample the cone using extremal rays, corresponding to rendering or imaging the face using directional light sources. It should be noted that exact recognition using the illumination cone involves a slow complex optimization (constrained optimization must be used to enforce a nonnegative linear combination of the basis images), and methods using low-dimensional linear subspaces and unconstrained optimization (which essentially reduces to a simple linear system) are more efficient and usually required for practical applications.

Another approach is to try to analytically construct the principal-component decomposition, analogous to what was done experimentally. Numerical PCA techniques could be biased by the specific lighting conditions used, so an explicit analytic method is helpful. It is only recently that there has been progress in analytic methods for extending PCA from a discrete set of images to a continuous sampling [40, 83]. These approaches demonstrate better generalization properties than purely empirical techniques. In fact, Zhao and Yang [83] have analytically constructed the covariance matrix for PCA of lighting variability, but under the assumption of no shadows.

##### Summary

To summarize, prior to the workreported in this chapter, the illumination variability could be described theoretically by the illumination cone. It was known from numerical and real experiments that the illumination cone lay close to a linear low-dimensional space for Lambertian objects with attached shadows. However, a theoretical explanation of these results was not available. In this chapter, we develop a simple linear lighting model using spherical harmonics. These theoretical results were first introduced for Lambertian surfaces simultaneously by Basri and Jacobs [2, 4] and Ramamoorthi and Hanrahan [64]. Much of the work of Basri and Jacobs is also summarized in an excellent book chapter on illumination modeling for face recognition [5].

#### 12.2.3 Frequency-Space Representations: Spherical Harmonics

We show reflection to be a convolution and analyze it in frequency space. We will primarily be concerned with analyzing quantities like the BRDF and distant lighting, which can be parametrized as functions on the unit sphere. Hence, the appropriate frequency-space representations are spherical harmonics [28, 29, 42]. Spherical harmonics can be thought of as signal-processing tools on the unit sphere, analogous to the Fourier series or sines and cosines on the line or circle. They can be written either as trigonometric functions of the spherical coordinates, or as simple polynomials of the Cartesian components. They form an orthonormal basis on the unit sphere, in terms of which quantities like the lighting or BRDF can be expanded and analyzed.

The use of spherical harmonics to represent the illumination and BRDF was pioneered in computer graphics by Cabral et al. [9]. In perception, D'Zmura [14] analyzed reflection as a linear operator in terms of spherical harmonics, and discussed some resulting ambiguities between reflectance and illumination. Our use of spherical harmonics to represent the lighting is also similar in some respects to previous methods such as that of Nimeroff et al. [56] that use steerable linear basis functions. Spherical harmonics have also been used before in computer graphics for representing BRDFs by a number of other authors [70, 79].

The results described in this chapter are based on a number of papers by us. This includes theoretical work in the planar 2D case or flatland [62], on the analysis of the appearance of a Lambertian surface using spherical harmonics [64], the theory for the general 3D case with isotropic BRDFs [65], and a comprehensive account including a unified view of 2D and 3D cases including anisotropic materials [67]. More details can also be found in the PhD thesis of the author [61]. Recently, we have also linked the convolution approach using spherical harmonics with principal component analysis, quantitatively explaining previous empirical results on lighting variability [60].

## FAQs

### What is interactive computer graphics in computer graphics? ›

An interactive graphic is **a way to present data to users who visit a page containing animations and customizations, creating a unique experience for those who wish to check specific information**. Therefore, instead of just presenting a fixed frame, you can let each user interact with the image in any way they want.

**What is the example of interactive computer graphics? ›**

(b) Interactive Computer Graphics:

One example of it is the **ping-pong game**. Interactive Computer Graphics require two-way communication between the computer and the user. A User can see the image and make any change by sending his command with an input device.

**What are basic components of interactive computer graphics? ›**

Abhishek Mishra. Answer: (d) All of the above Explanation: Interactive computer graphics consists of three components that are: **Frame Buffer or Digital Memory** A Monitor likes a home T.V.

**What is the advantage of interactive computer graphics? ›**

Advantages of interactive graphics

In interactive Computer Graphics user have some controls over the picture, i.e., the user can make any change in the produced image. Interactive Computer Graphics require two-way communication between the computer and the user.

**Where are interactive graphics used? ›**

Interactive graphics can be **applications on their own or be embedded within applications**. They may contain multiple forms of images, such as photography, video, and illustration, and need to incorporate principles of successful image design as well as design and presentation of appropriate controls.

**What is the importance of computer graphics? ›**

Computer graphics also are **essential to scientific visualization**, a discipline that uses images and colours to model complex phenomena such as air currents and electric fields, and to computer-aided engineering and design, in which objects are drawn and analyzed in computer programs.

**What are 5 examples of graphics? ›**

Examples of graphics include maps, photographs, designs and patterns, family trees, diagrams, architectural or engineering blueprints, bar charts and pie charts, typography, schematics, line art, flowcharts, and many other image forms.

**What are 5 interactive devices? ›**

**In this chapter, we will discuss on some new and old interactive devices.**

- Touch Screen. The touch screen concept was prophesized decades ago, however the platform was acquired recently. ...
- Gesture Recognition. ...
- Speech Recognition. ...
- Keyboard. ...
- Response Time.

**What are the 2 types of computer graphics? ›**

Computer graphics can be separated into two different categories: **raster graphics and vector graphics**. While both in essence set out to achieve the same goal (a high-quality digital image), they use different techniques and therefore have different strengths and weaknesses.

**What are the three types of computer graphics? ›**

Computer-generated imagery can be categorized into several different types: **two dimensional (2D), three dimensional (3D), and animated graphics**.

### What are the main three tasks of computer graphics? ›

Computer graphics is an art of drawing pictures on computer screens with the help of programming. It involves **computations, creation, and manipulation of data**.

**What are the main features of interactive systems? ›**

Main characteristics of interactive systems. **Input -- events triggered by users by means of input devices.** Action by users must trigger an output provided by the system. Input and output are interleaved.

**Why computer graphics is important for students? ›**

In recent years, the use of various types of computer graphics programs in teaching most subjects in the educational system has become widely popular. Computer graphics programs **greatly help the teacher in organizing the educational process, and the students in mastering the subject**.

**What are the uses of computer graphics in different? ›**

Computer-generated imagery is used for movie making, video game and computer program development, scientific modeling, and design for catalogs and other commercial art. Some people even make computer graphics as art. We can classify applications of computer graphics into four main areas: **Display of information**.

**What is interactive advantage? ›**

One of the biggest advantages of interactive technology displays is the **power to change them quickly**. You don't have to worry about ordering new materials or taking down signs. Interactive displays let you make changes at the right time for any circumstances.

**Which tool can be used to create interactive graphics? ›**

**Tableau Public**

You can easily create interactive graphs, maps, and live dashboards in just minutes.

**What is the most common use of computer graphics today? ›**

7. Entertainment: Computer Graphics are now commonly used in **making motion pictures, music videos and television shows**.

**What is the main function of graphics? ›**

Graphics are visual elements often used to **point readers and viewers to particular information**. They are also used to supplement text in an effort to aid readers in their understanding of a particular concept or make the concept more clear or interesting.

**How do graphics help in daily life? ›**

Graphic design is incorporated into almost every product humans interact with from reference manuals to billboards and road signs. Even a book – one of the simplest and ancient of human inventions – benefits from graphic design, which **enhances readability through attractive visual presentation of text**.

**What is interactive and passive graphics in computer graphics? ›**

The main difference between interactive and passive graphics is what they do when the user does something. **In passive graphics, the graphic does not do anything special when the user tries to interact with it.** **In interactive graphics, the graphics responds to what the user does to it**.

### What is interactive and non interactive computer graphics? ›

Communication between user and computer is in one way. **You can change images in an interactive system.** **You cannot change images in a non-interactive system**. Users have authority and control over all the systems. Users don't have authority and control over non-interactive systems.

**What does interactive mean in computer terms? ›**

Interactivity refers to **the communication between people and digital devices or content**. It is the ability of a computer, program, or other content to respond to the actions of the person that's using it. In a nutshell, it is what allows you to navigate a website, use social media, or play your favorite video game.

**What is interactive mode in computer? ›**

Interactive mode **allows you to rapidly analyze and visualize data using concise and efficient single-line commands**. In interactive mode, commands typed at the IDL prompt are executed when the Enter key is pressed.

**What are the 4 elements of computer graphics? ›**

**Computer graphics elements include animation, interaction, rendering, and modeling.**

- Animation is the simulation of temporal changes in the environment. ...
- Interaction is the process of generating responses to the system's external inputs.
- Rendering is the process of creating an image from a 2D or 3D model.

**What are major problems in computer graphics? ›**

There is a vast amount of open problems in real-time graphics: **shadows, aliasing, reflections, global illumination, transparencies (blending order and lighting)** etc. Save this answer.

**What is the difference between graphics and computer graphics? ›**

Graphics are defined as any sketch or a drawing or a special network that pictorially represents some meaningful information. **Computer Graphics is used where a set of images needs to be manipulated or the creation of the image in the form of pixels and is drawn on the computer**.

**What is the difference between image and graphics? ›**

**Graphics Def: computer generated or drawn by you.** **Image: scanned, captured, take photograph or an graphic file not generated by you**.

**What are the examples of interactive computer system? ›**

Interactive systems are computers that accept human input. Humans provide commands or data to computers by typing or by gestures. **MS Word and spreadsheets** are two examples of interactive systems.

**Is interactive design the same as graphic design? ›**

Graphic design is focusing on color, typography, font performance and mostly based on print media. **Interaction design emphasizes on the framework, logic, feedback, overall structure and process, more importantly, the user experience**.

**What are the five types of interactive digital media? ›**

Examples of interactive media include **web sites, user-generated content, interactive television, gaming, interactive advertising, blogs and mobile telephony**.