Eigenface

Some eigenfaces from AT&T Laboratories Cambridge

Eigenfaces are a set of eigenvectors used in the computer vision problem of human face recognition. The approach of using eigenfaces for recognition was developed by Sirovich and Kirby (1987) and used by Matthew Turk and Alex Pentland in face classification. It is considered the first successful example of facial recognition technology. These eigenvectors are derived from the covariance matrix of the probability distribution of the high-dimensional vector space of possible faces of human beings.

Eigenface generation

A set of eigenfaces can be generated by performing a mathematical process called principal component analysis (PCA) on a large set of images depicting different human faces. Informally, eigenfaces can be considered a set of "standardized face ingredients", derived from statistical analysis of many pictures of faces. Any human face can be considered to be a combination of these standard faces. For example, one's face might be composed of the average face plus 10% from eigenface 1, 55% from eigenface 2, and even -3% from eigenface 3. Remarkably, it does not take many eigenfaces combined together to achieve a fair approximation of most faces. Also, because a person's face is not recorded by a digital photograph, but instead as just a list of values (one value for each eigenface in the database used), much less space is taken for each person's face.

The eigenfaces that are created will appear as light and dark areas that are arranged in a specific pattern. This pattern is how different features of a face are singled out to be evaluated and scored. There will be a pattern to evaluate symmetry, if there is any style of facial hair, where the hairline is, or evaluate the size of the nose or mouth. Other eigenfaces have patterns that are less simple to identify, and the image of the eigenface may look very little like a face.

The technique used in creating eigenfaces and using them for recognition is also used outside of facial recognition. This technique is also used for handwriting analysis, lip reading, voice recognition, sign language/hand gestures interpretation and medical imaging analysis. Therefore, some do not use the term eigenface, but prefer to use 'eigenimage'.

Practical implementation

To create a set of eigenfaces, one must:

1. Prepare a training set of face images. The pictures constituting the training set should have been taken under the same lighting conditions, and must be normalized to have the eyes and mouths aligned across all images. They must also be all resampled to a common pixel resolution (r × c). Each image is treated as one vector, simply by concatenating the rows of pixels in the original image, resulting in a single row with r × c elements. For this implementation, it is assumed that all images of the training set are stored in a single matrix T, where each column of the matrix is an image.
2. Subtract the mean. The average image a has to be calculated and then subtracted from each original image in T.
3. Calculate the eigenvectors and eigenvalues of the covariance matrix S. Each eigenvector has the same dimensionality (number of components) as the original images, and thus can itself be seen as an image. The eigenvectors of this covariance matrix are therefore called eigenfaces. They are the directions in which the images differ from the mean image. Usually this will be a computationally expensive step (if at all possible), but the practical applicability of eigenfaces stems from the possibility to compute the eigenvectors of S efficiently, without ever computing S explicitly, as detailed below.
4. Choose the principal components. The D x D covariance matrix will result in D eigenvectors, each representing a direction in the r × c-dimensional image space. The eigenvectors (eigenfaces) with largest associated eigenvalue are kept.

These eigenfaces can now be used to represent both existing and new faces: we can project a new (mean-subtracted) image on the eigenfaces and thereby record how that new face differs from the mean face. The eigenvalues associated with each eigenface represent how much the images in the training set vary from the mean image in that direction. We lose information by projecting the image on a subset of the eigenvectors, but we minimize this loss by keeping those eigenfaces with the largest eigenvalues. For instance, if we are working with a 100 x 100 image, then we will obtain 10,000 eigenvectors. In practical applications, most faces can typically be identified using a projection on between 100 and 150 eigenfaces, so that most of the 10,000 eigenvectors can be discarded.

Computing the eigenvectors

Performing PCA directly on the covariance matrix of the images is often computationally infeasible. If small, say 100 × 100, greyscale images are used, each image is a point in a 10,000-dimensional space and the covariance matrix S is a matrix of 10,000 × 10,000 = 10⁸ elements. However the rank of the covariance matrix is limited by the number of training examples: if there are N training examples, there will be at most N − 1 eigenvectors with non-zero eigenvalues. If the number of training examples is smaller than the dimensionality of the images, the principal components can be computed more easily as follows.

Let T be the matrix of preprocessed training examples, where each column contains one mean-subtracted image. The covariance matrix can then be computed as S = TT^T and the eigenvector decomposition of S is given by

$\mathbf {Sv} _{i}=\mathbf {T} \mathbf {T} ^{T}\mathbf {v} _{i}=\lambda _{i}\mathbf {v} _{i}$

However TT^T is a large matrix, and if instead we take the eigenvalue decomposition of

$\mathbf {T} ^{T}\mathbf {T} \mathbf {u} _{i}=\lambda _{i}\mathbf {u} _{i}$

then we notice that by pre-multiplying both sides of the equation with T, we obtain

$\mathbf {T} \mathbf {T} ^{T}\mathbf {T} \mathbf {u} _{i}=\lambda _{i}\mathbf {T} \mathbf {u} _{i}$

Meaning that, if u_i is an eigenvector of T^TT, then v_i = Tu_i is an eigenvector of S. If we have a training set of 300 images of 100 × 100 pixels, the matrix T^TT is a 300 × 300 matrix, which is much more manageable than the 10,000 × 10,000 covariance matrix. Notice however that the resulting vectors v_i are not normalised; if normalisation is required it should be applied as an extra step.

Use in facial recognition

Facial recognition was the source of motivation behind the creation of eigenfaces. For this use, eigenfaces have advantages over other techniques available, such as the system's speed and efficiency. Using eigenfaces is very fast, and able to functionally operate on lots of faces in very little time. Unfortunately, this type of facial recognition does have a drawback to consider: trouble recognizing faces when they are viewed with different levels of light or angles. For the system to work well, the faces need to be seen from a frontal view under similar lighting. Face recognition using eigenfaces has been shown to be quite accurate. By experimenting with the system to test it under variations of certain conditions, the following correct recognitions were found: an average of 96% with light variation, 85% with orientation variation, and 64% with size variation. (Turk & Pentland 1991, p. 590)

To complement eigenfaces, another approach has been developed called eigenfeatures. This combines facial metrics (measuring distance between facial features) with the eigenface approach. Another method, which is competing with the eigenface technique uses 'fisherfaces'. This method for facial recognition is less sensitive to variation in lighting and pose of the face than the method using eigenfaces.

A more modern alternative to eigenfaces and fisherfaces is the active appearance model, which decouples the face's shape from its texture: it does an eigenface decomposition of the face after warping it to mean shape. This allows it to perform better on different projections of the face, and when the face is tilted.

See also

• Craniofacial anthropometry
• Human appearance
• Pattern recognition

References

• L. Sirovich and M. Kirby (1987). "Low-dimensional procedure for the characterization of human faces". Journal of the Optical Society of America A. 4 (3): 519–524. doi:10.1364/JOSAA.4.000519.
• M. Kirby and L. Sirovich (1990). "Application of the Karhunen-Loeve procedure for the characterization of human faces". IEEE Transactions on Pattern analysis and Machine Intelligence. 12 (1): 103–108. doi:10.1109/34.41390.
• M. Turk and A. Pentland (1991). "Face recognition using eigenfaces" (PDF). Proc. IEEE Conference on Computer Vision and Pattern Recognition. pp. 586–591. {{cite conference}}: Unknown parameter |booktitle= ignored (|book-title= suggested) (help)
• M. Turk and A. Pentland (1991). "Eigenfaces for recognition" (PDF). Journal of Cognitive Neuroscience. 3 (1): 71–86. doi:10.1162/jocn.1991.3.1.71.
• A. Pentland, B. Moghaddam, T. Starner, O. Oliyide, and M. Turk. (1993). "View-based and modular Eigenspaces for face recognition". Technical Report 245, M.I.T Media Lab.
• P. Belhumeur, J. Hespanha, and D. Kriegman (1997). "Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection". IEEE Transactions on pattern analysis and machine intelligence. 19 (7): 711. doi:10.1109/34.598228. {{cite journal}}: Unknown parameter |month= ignored (help)CS1 maint: multiple names: authors list (link)
• M. H. Yang (2000). "Face recognition using kernel eigenfaces". Proceedings International Conference on Image Processing. Vol. 1. pp. 37–40. {{cite conference}}: Unknown parameter |booktitle= ignored (|book-title= suggested) (help)
• R. Cendrillon, B. Lovell (2000). "Real-time face recognition using eigenfaces" (PDF). Visual Communications and Image Processing. pp. 269–276. doi:10.1117/12.386642. {{cite conference}}: Unknown parameter |booktitle= ignored (|book-title= suggested) (help)
• T. Heseltine, N. Pears, J. Austin, Z. Chen (2003). "Face Recognition: A Comparison of Appearance-Based Approaches". Proc. VIIth Digital Image Computing: Techniques and Applications, vol 1. 59-68.
• D. Pissarenko (2003). Eigenface-based facial recognition.
• F. Tsalakanidoua, D. Tzovarasb, M.G. Strintzisa (2003). "Use of depth and colour eigenfaces for face recognition". Pattern Recognition Letters (9): 1427–1435. {{cite journal}}: Unknown parameter |vol= ignored (|volume= suggested) (help)CS1 maint: multiple names: authors list (link)
• Delac, K., Grgic, M., Liatsis, P. (2005). "Appearance-based Statistical Methods for Face Recognition". Proceedings of the 47th International Symposium ELMAR-2005 focused on Multimedia Systems and Applications, Zadar, Croatia, 08-10 June 2005, pp. 151–158

External links

• Face Recognition Homepage
• PCA on the FERET Dataset
• Developing Intelligence Eigenfaces and the Fusiform Face Area
• A Tutorial on Face Recognition Using Eigenfaces and Distance Classifiers
• Matlab example code for eigenfaces
• OpenCV + C++Builder6 implementation of PCA
• Java Applet demonstration of eigenfaces