Jump to content

Leverage (statistics)

From Wikipedia, the free encyclopedia

In statistics and in particular in regression analysis, leverage is a measure of how far away the independent variable values of an observation are from those of the other observations. High-leverage points, if any, are outliers with respect to the independent variables. That is, high-leverage points have no neighboring points in space, where is the number of independent variables in a regression model. This makes the fitted model likely to pass close to a high leverage observation.[1] Hence high-leverage points have the potential to cause large changes in the parameter estimates when they are deleted i.e., to be influential points. Although an influential point will typically have high leverage, a high leverage point is not necessarily an influential point. The leverage is typically defined as the diagonal elements of the hat matrix.

Definition and interpretations

[edit]

Consider the linear regression model , . That is, , where, is the design matrix whose rows correspond to the observations and whose columns correspond to the independent or explanatory variables. The leverage score for the independent observation is given as:

, the diagonal element of the ortho-projection matrix (a.k.a hat matrix) .

Thus the leverage score can be viewed as the 'weighted' distance between to the mean of 's (see its relation with Mahalanobis distance). It can also be interpreted as the degree by which the measured (dependent) value (i.e., ) influences the fitted (predicted) value (i.e., ): mathematically,

.

Hence, the leverage score is also known as the observation self-sensitivity or self-influence.[2] Using the fact that (i.e., the prediction is ortho-projection of onto range space of ) in the above expression, we get . Note that this leverage depends on the values of the explanatory variables of all observations but not on any of the values of the dependent variables .

Properties

[edit]
  1. The leverage   is a number between 0 and 1,
    Proof: Note that is idempotent matrix () and symmetric (). Thus, by using the fact that , we have . Since we know that , we have .
  2. Sum of leverages is equal to the number of parameters in (including the intercept).
    Proof: .

Determination of outliers in X using leverages

[edit]

Large leverage corresponds to an that is extreme. A common rule is to identify whose leverage value is more than 2 times larger than the mean leverage (see property 2 above). That is, if , shall be considered an outlier. Some statisticians prefer the threshold of instead of .

Relation to Mahalanobis distance

[edit]

Leverage is closely related to the Mahalanobis distance (proof[3]). Specifically, for some matrix , the squared Mahalanobis distance of (where is row of ) from the vector of mean of length , is , where is the estimated covariance matrix of 's. This is related to the leverage of the hat matrix of after appending a column vector of 1's to it. The relationship between the two is:

This relationship enables us to decompose leverage into meaningful components so that some sources of high leverage can be investigated analytically.[4]

Relation to influence functions

[edit]

In a regression context, we combine leverage and influence functions to compute the degree to which estimated coefficients would change if we removed a single data point. Denoting the regression residuals as , one can compare the estimated coefficient to the leave-one-out estimated coefficient using the formula [5][6]

Young (2019) uses a version of this formula after residualizing controls.[7] To gain intuition for this formula, note that captures the potential for an observation to affect the regression parameters, and therefore captures the actual influence of that observations' deviations from its fitted value on the regression parameters. The formula then divides by to account for the fact that we remove the observation rather than adjusting its value, reflecting the fact that removal changes the distribution of covariates more when applied to high-leverage observations (i.e. with outlier covariate values). Similar formulas arise when applying general formulas for statistical influences functions in the regression context.[8][9]

Effect on residual variance

[edit]

If we are in an ordinary least squares setting with fixed and homoscedastic regression errors , then the regression residual, has variance

.

In other words, an observation's leverage score determines the degree of noise in the model's misprediction of that observation, with higher leverage leading to less noise. This follows from the fact that is idempotent and symmetric and , hence, .

The corresponding studentized residual—the residual adjusted for its observation-specific estimated residual variance—is then

where is an appropriate estimate of .

Partial leverage

[edit]

Partial leverage (PL) is a measure of the contribution of the individual independent variables to the total leverage of each observation. That is, PL is a measure of how changes as a variable is added to the regression model. It is computed as:

where is the index of independent variable, is the index of observation and are the residuals from regressing against the remaining independent variables. Note that the partial leverage is the leverage of the point in the partial regression plot for the variable. Data points with large partial leverage for an independent variable can exert undue influence on the selection of that variable in automatic regression model building procedures.

Software implementations

[edit]

Many programs and statistics packages, such as R, Python, etc., include implementations of Leverage.

Language/Program Function Notes
R hat(x, intercept = TRUE) or hatvalues(model, ...) See [1]
Python (x * np.linalg.pinv(x).T).sum(-1) See [2]

See also

[edit]

References

[edit]
  1. ^ Everitt, B. S. (2002). Cambridge Dictionary of Statistics. Cambridge University Press. ISBN 0-521-81099-X.
  2. ^ Cardinali, C. (June 2013). "Data Assimilation: Observation influence diagnostic of a data assimilation system" (PDF).
  3. ^ Prove the relation between Mahalanobis distance and Leverage?
  4. ^ Kim, M. G. (2004). "Sources of high leverage in linear regression model (Journal of Applied Mathematics and Computing, Vol 16, 509–513)". arXiv:2006.04024 [math.ST].
  5. ^ Miller, Rupert G. (September 1974). "An Unbalanced Jackknife". Annals of Statistics. 2 (5): 880–891. doi:10.1214/aos/1176342811. ISSN 0090-5364.
  6. ^ Hiyashi, Fumio (2000). Econometrics. Princeton University Press. p. 21.
  7. ^ Young, Alwyn (2019). "Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results". The Quarterly Journal of Economics. 134 (2): 567. doi:10.1093/qje/qjy029.
  8. ^ Chatterjee, Samprit; Hadi, Ali S. (August 1986). "Influential Observations, High Leverage Points, and Outliers in Linear Regression". Statistical Science. 1 (3): 379–393. doi:10.1214/ss/1177013622. ISSN 0883-4237.
  9. ^ "regression - Influence functions and OLS". Cross Validated. Retrieved 2020-12-06.