Cochran's theorem

From Wikipedia, the free encyclopedia

In statistics, Cochran's theorem, devised by William G. Cochran,[1] is a theorem used to justify results relating to the probability distributions of statistics that are used in the analysis of variance.[2]

Statement[edit]

Let U1, ..., UN be i.i.d. standard normally distributed random variables, and . Let be symmetric matrices. Define ri to be the rank of . Define , so that the Qi are quadratic forms. Further assume .

Cochran's theorem states that the following are equivalent:

Often it's stated as , where is idempotent, and is replaced by . But after an orthogonal transform, , and so we reduce to the above theorem.

Proof[edit]

Claim: Let be a standard Gaussian in , then for any symmetric matrices , if and have the same distribution, then have the same eigenvalues (up to multiplicity).

Proof: Let the eigenvalues of be , then calculate the characteristic function of . It comes out to be

(To calculate it, first diagonalize , change into that frame, then use the fact that the characteristic function of the sum of independent variables is the product of their characteristic functions.)

For and to be equal, their characteristic functions must be equal, so have the same eigenvalues (up to multiplicity).

Claim: .

Proof: . Since is symmetric, and , by the previous claim, has the same eigenvalues as 0.

Lemma: If , all symmetric, and have eigenvalues 0, 1, then they are simultaneously diagonalizable.

Fix i, and consider the eigenvectors v of such that . Then we have , so all . Thus we obtain a split of into , such that V is the 1-eigenspace of , and in the 0-eigenspaces of all other . Now induct by moving into .

Case: All are independent

Fix some , define , and diagonalize by an orthogonal transform . Then consider . It is diagonalized as well.

Let , then it is also standard Gaussian. Then we have

Inspect their diagonal entries, to see that implies that their nonzero diagonal entries are disjoint.

Thus all eigenvalues of are 0, 1, so is a dist with degrees of freedom.

Case: Each is a distribution.

Fix any , diagonalize it by orthogonal transform , and reindex, so that . Then for some , a spherical rotation of .

Since , we get all . So all , and have eigenvalues .

So diagonalize them simultaneously, add them up, to find .

Case: We first show that the matrices B(i) can be simultaneously diagonalized by an orthogonal matrix and that their non-zero eigenvalues are all equal to +1. Once that's shown, take this orthogonal transform to this simultaneous eigenbasis, in which the random vector becomes , but all are still independent and standard Gaussian. Then the result follows.

Each of the matrices B(i) has rank ri and thus ri non-zero eigenvalues. For each i, the sum has at most rank . Since , it follows that C(i) has exactly rank N − ri.

Therefore B(i) and C(i) can be simultaneously diagonalized. This can be shown by first diagonalizing B(i), by the spectral theorem. In this basis, it is of the form:

Thus the lower rows are zero. Since , it follows that these rows in C(i) in this basis contain a right block which is a unit matrix, with zeros in the rest of these rows. But since C(i) has rank N − ri, it must be zero elsewhere. Thus it is diagonal in this basis as well. It follows that all the non-zero eigenvalues of both B(i) and C(i) are +1. This argument applies for all i, thus all B(i) are positive semidefinite.

Moreover, the above analysis can be repeated in the diagonal basis for . In this basis is the identity of an vector space, so it follows that both B(2) and are simultaneously diagonalizable in this vector space (and hence also together with B(1)). By iteration it follows that all B-s are simultaneously diagonalizable.

Thus there exists an orthogonal matrix such that for all , is diagonal, where any entry with indices , , is equal to 1, while any entry with other indices is equal to 0.

Examples[edit]

Sample mean and sample variance[edit]

If X1, ..., Xn are independent normally distributed random variables with mean μ and standard deviation σ then

is standard normal for each i. Note that the total Q is equal to sum of squared Us as shown here:

which stems from the original assumption that . So instead we will calculate this quantity and later separate it into Qi's. It is possible to write

(here is the sample mean). To see this identity, multiply throughout by and note that

and expand to give

The third term is zero because it is equal to a constant times

and the second term has just n identical terms added together. Thus

and hence

Now with the matrix of ones which has rank 1. In turn given that . This expression can be also obtained by expanding in matrix notation. It can be shown that the rank of is as the addition of all its rows is equal to zero. Thus the conditions for Cochran's theorem are met.

Cochran's theorem then states that Q1 and Q2 are independent, with chi-squared distributions with n − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property characterizes the normal distribution – for no other distribution are the sample mean and sample variance independent.[4]

Distributions[edit]

The result for the distributions is written symbolically as

Both these random variables are proportional to the true but unknown variance σ2. Thus their ratio does not depend on σ2 and, because they are statistically independent. The distribution of their ratio is given by

where F1,n − 1 is the F-distribution with 1 and n − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the definition of a random variable having the F-distribution.

Estimation of variance[edit]

To estimate the variance σ2, one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution

Cochran's theorem shows that

and the properties of the chi-squared distribution show that

Alternative formulation[edit]

The following version is often seen when considering linear regression.[5] Suppose that is a standard multivariate normal random vector (here denotes the n-by-n identity matrix), and if are all n-by-n symmetric matrices with . Then, on defining , any one of the following conditions implies the other two:

  • (thus the are positive semidefinite)
  • is independent of for

See also[edit]

References[edit]

  1. ^ a b Cochran, W. G. (April 1934). "The distribution of quadratic forms in a normal system, with applications to the analysis of covariance". Mathematical Proceedings of the Cambridge Philosophical Society. 30 (2): 178–191. doi:10.1017/S0305004100016595.
  2. ^ Bapat, R. B. (2000). Linear Algebra and Linear Models (Second ed.). Springer. ISBN 978-0-387-98871-9.
  3. ^ "Cochran's theorem", A Dictionary of Statistics, Oxford University Press, 2008-01-01, doi:10.1093/acref/9780199541454.001.0001/acref-9780199541454-e-294, ISBN 978-0-19-954145-4, retrieved 2022-05-18
  4. ^ Geary, R.C. (1936). "The Distribution of "Student's" Ratio for Non-Normal Samples". Supplement to the Journal of the Royal Statistical Society. 3 (2): 178–184. doi:10.2307/2983669. JFM 63.1090.03. JSTOR 2983669.
  5. ^ "Cochran's Theorem (A quick tutorial)" (PDF).