Partition of sums of squares

This article is about sums of squares in statistics. For the sum of squares of consecutive integers, see square pyramidal number. For representing an integer as a sum of squares of integers, see Lagrange's four-square theorem.

Sum of squares is a concept that permeates much of inferential statistics and descriptive statistics. More properly, it is "the sum of the squared deviations". Mathematically, it is an unscaled, or unadjusted measure of dispersion (also called variability). When scaled for the number of degrees of freedom, it estimates the variance, or spread of the observations about their mean value.

The distance from any point in a collection of data, to the mean of the data, is the deviation. This can be written as $X_{i}-{\overline {X}}$, where $X_{i}$ is the ith data point, and ${\overline {X}}$ is the estimate of the mean. If all such deviations are squared, then summed, as in $\sum _{i=1}^{n}\left(X_{i}-{\overline {X}}\,\right)^{2}$, we have the "sum of squares" for these data.

When more data is added to the collection the sum of squares will increase, except in unlikely cases such as the new data being equal to the mean. So usually, the sum of squares will grow with the size of the data collection. That is a manifestation of the fact that it is unscaled.

In many cases, the number of degrees of freedom is simply the number of data in the collection, minus one. We write this as n − 1, where n is the number of data.

Scaling (also known as normalizing) means adjusting the sum of squares so that it does not grow as the size of the data collection grows. This is important when we want to compare samples of different sizes, such as a sample of 100 people compared to a sample of 20 people. If the sum of squares was not normalized, its value would always be larger for the sample of 100 people than for the sample of 20 people. To scale the sum of squares, we divide it by the degrees of freedom, i.e., calculate the sum of squares per degree of freedom, or variance. Standard deviation, in turn, is the square root of the variance.

The above information is how sum of squares is used in descriptive statistics; see the article on total sum of squares for an application of this broad principle to inferential statistics.

Partitioning the sum of squares in linear regression

Theorem. Given a linear regression model $y_{i}=\beta _{0}+\beta _{1}x_{i1}+\cdots +\beta _{p}x_{ip}+\varepsilon _{i}$ including a constant based on a sample $(y_{i},x_{i1},\ldots ,x_{ip}),\,i=1,\ldots ,n$ containing n observations, the total sum of squares $\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}$ can be partitioned equivalently as follows:

${\begin{aligned}\left\|y-{\bar {y}}\iota \right\|^{2}&=\left\|{\hat {\varepsilon }}\right\|^{2}+\left\|{\hat {y}}-{\bar {y}}\iota \right\|^{2},\quad \iota =(1,1,\ldots ,1)\\\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})+\sum _{i=1}^{n}(y_{i}-{\hat {y}}_{i})^{2}\\\sum _{i=1}^{n}(y_{i}-{\bar {y}})^{2}&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}\\\mathrm {TSS} &=\mathrm {ESS} +\mathrm {RSS} \end{aligned}}$

Proof.

${\begin{aligned}\sum _{i=1}^{n}(y_{i}-{\overline {y}})^{2}&=\sum _{i=1}^{n}(y_{i}-{\overline {y}}+{\hat {y}}_{i}-{\hat {y}}_{i})^{2}=\sum _{i=1}^{n}(({\hat {y}}_{i}-{\bar {y}})+\underbrace {(y_{i}-{\hat {y}}_{i})} _{{\hat {\varepsilon }}_{i}})^{2}\\&=\sum _{i=1}^{n}(({\hat {y}}_{i}-{\bar {y}})^{2}+2{\hat {\varepsilon }}_{i}({\hat {y}}_{i}-{\bar {y}})+{\hat {\varepsilon }}_{i}^{2})\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}+2\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}({\hat {y}}_{i}-{\bar {y}})\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}+2\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}({\hat {\beta }}_{0}+{\hat {\beta }}_{1}x_{i1}+\cdots +{\hat {\beta }}_{p}x_{ip}-{\overline {y}})\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})^{2}+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}+2({\hat {\beta }}_{0}-{\overline {y}})\underbrace {\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}} _{0}+2{\hat {\beta }}_{1}\underbrace {\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}x_{i1}} _{0}+\cdots +2{\hat {\beta }}_{p}\underbrace {\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}x_{ip}} _{0}\\&=\sum _{i=1}^{n}({\hat {y}}_{i}-{\bar {y}})+\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}^{2}=\mathrm {ESS} +\mathrm {RSS} \\\end{aligned}}$

The requirement that the model includes a constant or equivalently that the design matrix contains a column of ones ensures that $\sum _{i=1}^{n}{\hat {\varepsilon }}_{i}=0$.

See also

• Explained sum of squares
• Squared deviations
• Polynomial SOS
• Residual sum of squares
• Mean squared error
• Lack-of-fit sum of squares