Quantile regression

Quantile regression is a type of regression analysis used in statistics. Whereas the method of least squares results in estimates that approximate the conditional mean of the response variable given certain values of the predictor variables, quantile regression results in estimates approximating either the median or other quantiles of the response variable.

Advantages and applications

Quantile regression is used when an estimate of the various quantiles (such as the median) of a population is desired. One advantage of using quantile regression to estimate the median, rather than ordinary least squares regression to estimate the mean, is that quantile regression will be more robust in response to large outliers. Quantile regression can be seen as a natural analogue in regression analysis to the practice of using different measures of central tendency and statistical dispersion to obtain a more comprehensive and robust analysis. Another advantage to quantile regression is the fact that any quantile can be estimated.

In ecology, quantile regression has been proposed and used as a way to discover more useful predictive relationships between variables in cases where there is no relationship or only a weak relationship between the means of such variables. The need for and success of quantile regression in ecology has been attributed to the complexity of interactions between different factors leading to data with unequal variation of one variable for different ranges of another variable.

Mathematics

The mathematical forms arising from quantile regression are distinct from those arising in the method of least squares. The method of least squares leads to a consideration of problems in an inner product space, involving projection onto subspaces, and thus the problem of minimizing the squared errors can be reduced to a problem in numerical linear algebra. Quantile regression does not have this structure, and instead leads to problems in linear programming that can be solved by the simplex method. The fact that the algorithms of linear programming appear more esoteric to users may explain why quantile regression is not as widely used as the method of least squares.

Quantiles

Let $Y$ be a real valued random variable with distribution function $F_{Y}(y)=P(Y\leq y)$. The $\tau$th quantile of Y is given by

$Q_{Y}(\tau )=F_{Y}^{-1}(\tau )=\inf \left\{y:F_{Y}(y)\geq \tau \right\}$

where $\tau \in [0,1].$

Define the loss function as $\rho _{\tau }(y)=y(\tau -I(y<0))$. A specific quantile can be found by minimizing the expected loss of $Y-u$ with respect to $u$:

${\underset {u}{\min }}E(\rho _{\tau }(Y-u))={\underset {u}{\min }}(\tau -1)\int _{-\infty }^{u}(y-u)dF_{Y}(y)+\tau \int _{u}^{\infty }(y-u)dF_{Y}(y).$

This can be shown by setting the derivative of the expected loss function to 0 and letting $q_{\tau }$ be the solution of

$0=(1-\tau )\int _{-\infty }^{q_{\tau }}dF_{Y}(y)-\tau \int _{q_{\tau }}^{\infty }dF_{Y}(y).$

This equation reduces to

$0=F_{Y}(q_{\tau })-\tau ,$

and then to

$F_{Y}(q_{\tau })=\tau .$

Hence $q_{\tau }$ is $\tau$th quantile of the random variable Y.

Example

Let $Y$ be a discrete random variable that takes values 1,2,..,9 with equal probabilities. The task is to find the median of Y, and hence the value $\tau =0.5$ is chosen. The expected loss, L(u), is

$L(u)={\frac {(\tau -1)}{9}}\sum _{y_{i}<u}(y_{i}-u)+{\frac {\tau }{9}}\sum _{y_{i}\geq u}(y_{i}-u).$

Since ${0.5/9}$ is a constant, it can be taken out of the expected loss function (this is only true if $\tau =0.5$). Then, at u=3,

$L(3)\propto \sum _{i=1}^{2}-(i-3)+\sum _{i=3}^{9}(i-3)=[(2+1)+(0+1+2+...+6)]=24.$

Suppose that u is increased by 1 unit. Then the expected loss will be changed by $(3)-(6)=-3$ on changing u to 4. If , u=5, the expected loss is

$L(5)\propto \sum _{i=1}^{4}i+\sum _{i=0}^{4}i=20,$

and any change in u will increase the expected loss. Thus u=5 is the median. The Table below shows the expected loss (divided by ${0.5/9}$) for different values of u.

u	1	2	3	4	5	6	7	8	9
Expected loss	36	29	24	21	20	21	24	29	36

Intuition

Consider $\tau =0.5$ and let q be an initial guess for $q_{\tau }$. The expected loss evaluated at q is

$-0.5\int _{-\infty }^{q}(y-q)dF_{Y}(y)+0.5\int _{q}^{\infty }(y-q)dF_{Y}(y).$

In order to miminize the expected loss, we move the value of q a little bit to see whether the expect loss will rise or fall. Suppose we increase q by 1 unit. Then the change of expected loss would be

$\int _{-\infty }^{q}1dF_{Y}(y)-\int _{q}^{\infty }1dF_{Y}(y).$

The first term of the equation is $F_{Y}(q)$ and second term of the equation is $1-F_{Y}(q)$. Therefore the change of expected loss function is negative if and only if $F_{Y}(q)<0.5$, that is if and only if q is smaller than the median. Similarly, if we reduce q by 1 unit, the change of expected loss function is negative if and only if q is larger than the median.

In order to minimize the expected loss function, we would increase (decrease) L(q) if q is smaller (larger) than the median, until q reaches the median. The idea behind the minimization is to count the number of points (weighted with the density) that are larger or smaller than q and then move q to a point where q is larger than $\tau$% of the points.

Sample quantile

The $\tau$ sample quantile can be obtained by solving the following minimization problem

${\hat {q}}_{\tau }={\underset {q\in R}{\mbox{arg min}}}\sum _{i=1}^{n}\rho _{\tau }(y_{i}-q),$

$={\underset {q\in R}{\mbox{arg min}}}\left[(\tau -1)\sum _{y_{i}<q}(y_{i}-q)+\tau \sum _{y_{i}\geq q}(y_{i}-q)\right].$

The intuition is the same as for the population quantile.

Conditional Quantile and Quantile Regression

Suppose the $\tau$th conditional quantile function is $Q_{Y|X}(\tau )=X\beta _{\tau }$. Given the distribution function of $Y$, $\beta _{\tau }$ can be obtained by solving

$\beta _{\tau }={\underset {\beta \in R^{k}}{\mbox{arg min}}}E(\rho _{\tau }(Y-X\beta )).$

Solving the sample analog gives the estimator of $\beta$.

${\hat {\beta _{\tau }}}={\underset {\beta \in R^{k}}{\mbox{arg min}}}\sum _{i=1}^{n}(\rho _{\tau }(Y_{i}-X\beta )).$

Computation

The minimization problem can be reformulated as a linear programming problem

${\underset {\beta ^{+},\beta ^{-},u^{+},u^{-}\in R^{2k}\times R_{+}^{2n}}{\min }}\left\{\tau 1_{n}^{'}u^{+}+(1-\tau )1_{n}^{'}u^{-}|X(\beta ^{+}-\beta ^{-})+u^{+}-u^{-}=Y\right\},$

where

$\beta _{j}^{+}=\max(\beta _{j},0)$,    $\beta _{j}^{-}=-\min(\beta _{j},0)$,    $u_{j}^{+}=\max(u_{j},0)$ ,    $u_{j}^{-}=-\min(u_{j},0).$

Simplex methods or interior point methods can be applied to solve the linear programming problem.

Asymptotic properties

For $\tau \in (0,1)$, under some regularity conditions, ${\hat {\beta }}_{\tau }$ is asymptotically normal:

${\sqrt {n}}({\hat {\beta }}_{\tau }-\beta _{\tau }){\overset {d}{\rightarrow }}N(0,\tau (1-\tau )D^{-1}\Omega _{x}D),$

where

$D=E(f_{Y}(X\beta )XX^{\prime })$ and $\Omega _{x}=E(X^{\prime }X).$

Equivariance

See invariant estimator for background on invariance and equivariance.

Scale equivariance

For any $a>0$ and $\tau \in [0,1]$

${\hat {\beta }}(\tau ;aY,X)=a{\hat {\beta }}(\tau ;Y,X),$

${\hat {\beta }}(\tau ;-aY,X)=-a{\hat {\beta }}(1-\tau ;Y,X).$

Shift equivariance

For any $\gamma \in R^{k}$ and $\tau \in [0,1]$

${\hat {\beta }}(\tau ;Y+X\gamma ,X)={\hat {\beta }}(\tau ;Y,X)+\gamma .$

Equivariance to reparameterization of design

Let $A$ be any $p\times p$ nonsingular matrix and $\tau \in [0,1]$

${\hat {\beta }}(\tau ;Y,XA)=A^{-1}{\hat {\beta }}(\tau ;Y,X).$

Invariance to monotone transformations

If $h$ is a nondecreasing function on 'R, the following invariance property applies:

$h(Q_{Y|X}(\tau ))\equiv Q_{h(Y)|X}(\tau ).$

Example 1

Let $Y=\ln(W)$ and $Q_{Y|X}(\tau )=X\beta _{\tau }$ , then $Q_{W|X}(\tau )=\exp(X\beta _{\tau })$. The mean regression does not have the same property since $\operatorname {E} (\ln(Y)\neq \ln(\operatorname {E} (Y)).$

Example 2

Let $Y^{c}=\max(0,Y)$ and $Q_{Y|X}=X\beta _{\tau }$, then $Q_{Y^{c}|X}(\tau )=\max(0,X\beta _{\tau })$. This is the censored quantile regression model: estimated values can be obtained without making any distributional assumptions, but at the cost of computational difficulty, some of which can be avoided by using a simple three step censored quantile regression procedure as an approximation.

Implementations

Some statistics packages, such as R, Eviews (ver. 6), Stata (via qreg), gretl, SAS through proc quantreg (ver. 9.2), and RATS include implementations of quantile regression. R implements it through Roger Koenker's quantreg package.

References

• Koenker, Roger (2005) Quantile Regression, Cambridge University Press. ISBN 0-521-60827-9

External links

• Quantile LOWESS – A method to perform Local Quantile regression (with R code)