User:Bruguiea/sandbox

Playground

$y=X\beta +\epsilon$

$(n;1)$

$(n;p)$

$(p;1)$

${\hat {\beta }}=(X'X)^{-1}Xy$

${\hat {\epsilon }}=\left(I-(X'X)^{-1}X\right)y$

$n-r$

$t_{n-r}={\frac {\hat {\beta _{i}}}{{\hat {\epsilon }}'{\hat {\epsilon }}}}={\frac {\hat {\beta _{i}}}{y'\left(I-(X'X)^{-1}X\right)'\left(I-(X'X)^{-1}X\right)y}}$

$t_{n-r}\rightarrow p$

$U=\operatorname {ER} -\alpha \operatorname {Risk}$

Special case of covariance matrices

A covariance matrix $\Sigma$ can be represented as the product $A'A$. Its eigenvalues are positive:

$\Sigma v_{i}=\lambda _{i}v_{i}$

$A'Av_{i}=\lambda _{i}v_{i}$

$v_{i}'A'Av_{i}=v_{i}'\lambda _{i}v_{i}$

$\left\|Av_{i}\right\|=\lambda _{i}\left\|v_{i}\right\|$

$\lambda _{i}={\frac {\left\|v_{i}\right\|}{\left\|Av_{i}\right\|}}\geq 0$

The eigenvectors are orthogonal to one another:

$A'Av_{i}=\lambda _{i}v_{i}$

$v_{j}'A'Av_{i}=\lambda _{i}v_{j}'v_{i}$

$(A'Av_{j})'v_{i}=\lambda _{i}v_{j}'v_{i}$

$\lambda _{j}v_{j}'v_{i}=\lambda _{i}v_{j}'v_{i}$

$(\lambda _{j}-\lambda _{i})v_{j}'v_{i}=0$

$v_{j}'v_{i}=0$ (different eigenvalues, in case of multiplicity, the basis can be orthogonalized)

The Rayleigh quotient can be expressed as a function of the eigenvalues by decomposing any vector $x$ on the basis of eigenvectors:

$x=\sum _{i=1}^{n}\alpha _{i}v_{i}$

$\rho ={\frac {x'A'Ax}{x'x}}$

$\rho ={\frac {(\sum _{j=1}^{n}\alpha _{j}v_{j})'A'A(\sum _{i=1}^{n}\alpha _{i}v_{i})}{(\sum _{j=1}^{n}\alpha _{j}v_{j})'(\sum _{i=1}^{n}\alpha _{i}v_{i})}}$

Which, by orthogonality of the eigenvectors, becomes:

$\rho ={\frac {\sum _{i=1}^{n}\alpha _{i}^{2}\lambda _{i}}{\sum _{i=1}^{n}\alpha _{i}^{2}}}$

If a vector $x$ maximizes $\rho$, then any vector $k.x$ (for $k\neq 0$) also maximizes it, one can reduce to the Lagrange problem of maximizing $\sum _{i=1}^{n}\alpha _{i}^{2}\lambda _{i}$ under the constrainst that $\sum _{i=1}^{n}\alpha _{i}^{2}=1$.

Since all the eingenvalues are non-negative, the problem is convex and the maximum occurs on the edges of the domain, namely when $\alpha _{1}=1$ and $\forall i>1\alpha _{i}=0$ (when the eigenvalues are ordered in decreasing magnitude).

This property is the basis for principal components analysis and canonical correlation.