A whitening transformation is a decorrelation transformation that transforms a set of random variables having a known covariance matrix into a set of new random variables whose covariance is the identity matrix (meaning that they are uncorrelated and all have variance 1).
The transformation is called "whitening" because it changes the input vector into a white noise vector. It differs from a general decorrelation transformation in that the latter only makes the covariances equal to zero, so that the correlation matrix may be any diagonal matrix.
The inverse coloring transformation transforms a vector of uncorrelated variables (a white random vector) into a vector with a specified covariance matrix.
Suppose is a random (column) vector with covariance matrix and mean . Typically (when is not singular) whitening means multiplying by .
The matrix can be written as the expected value of the outer product of with itself, namely:
When is symmetric and positive semidefinite (i.e. not singular), it has a square root , a matrix(not necessarily symmetric) such that . Since is positive definite, is invertible, and the vector has covariance matrix:
and is therefore a white random vector.
Since the square root of a matrix is not unique, the whitening transformation is not unique either.
If is singular (and hence not positive definite), then is not invertible, and it is impossible to map to a white vector with the same number of components. In that case the vector can still be mapped to a smaller white vector with elements, where is the number of non-zero eigenvalues of .
- The ZCA whitening transformation. Appendix A of Learning Multiple Layers of Features from Tiny Images by A. Krizhevsky.