Law of total variance
Some writers on probability call this the "conditional variance formula". In language perhaps better known to statisticians than to probabilists, the two terms are the "unexplained" and the "explained" components of the variance (cf. fraction of variance unexplained, explained variation).
There is a general variance decomposition formula for c ≥ 2 components (see below). For example, with two conditioning random variables:
which follows from the law of total conditional variance:
Note that the conditional expected value E( Y | X ) is a random variable in its own right, whose value depends on the value of X. Notice that the conditional expected value of Y given the event X = x is a function of x (this is where adherence to the conventional and rigidly case-sensitive notation of probability theory becomes important!). If we write E( Y | X = x ) = g(x) then the random variable E( Y | X ) is just g(X). Similar comments apply to the conditional variance.
from the definition of variance. Then we apply the law of total expectation to each term by conditioning on the random variable X:
Now we rewrite the conditional second moment of Y in terms of its variance and first moment:
Since the expectation of a sum is the sum of expectations, the terms can now be regrouped:
Finally, we recognize the terms in parentheses as the variance of the conditional expectation E[Y|X]:
General variance decomposition applicable to dynamic systems
The following formula shows how to apply the general, measure theoretic variance decomposition formula  to stochastic dynamic systems. Let Y(t) be the value of a system variable at time t. Suppose we have the internal histories (natural filtrations) , each one corresponding to the history (trajectory) of a different collection of system variables. The collections need not be disjoint. The variance of Y(t) can be decomposed, for all times t, into c ≥ 2 components as follows:
The decomposition is not unique. It depends on the order of the conditioning in the sequential decomposition.
The square of the correlation and explained (or informational) variation
In cases where (Y, X) are such that the conditional expected value is linear; i.e., in cases where
it follows from the bilinearity of Cov(-,-) that
and the explained component of the variance divided by the total variance is just the square of the correlation between Y and X; i.e., in such cases,
One example of this situation is when (X, Y) have a bivariate normal (Gaussian) distribution.
More generally, when the conditional expectation E( Y | X ) is a non-linear function of X
which can be estimated as the R squared from a non-linear regression of Y on X, using data drawn from the joint distribution of (X,Y). When E( Y | X ) has a Gaussian distribution (and is an invertible function of X), or Y itself has a (marginal) Gaussian distribution, this explained component of variation sets a lower bound on the mutual information:
A similar law for the third central moment μ3 says
- Law of total covariance, a generalization
- Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 385–386.
- Bowsher, C.G. and P.S. Swain, Proc Natl Acad Sci USA, 2012: 109, E1320–29.
- Neil A. Weiss, A Course in Probability, Addison–Wesley, 2005, pages 380–383.