Partial least squares regression

Partial least squares regression (PLS regression) is a statistical method that bears some relation to principal components regression; instead of finding hyperplanes of maximum variance between the response and independent variables, it finds a linear regression model by projecting the predicted variables and the observable variables to a new space. Because both the X and Y data are projected to new spaces, the PLS family of methods are known as bilinear factor models. Partial least squares Discriminant Analysis (PLS-DA) is a variant used when the Y is binary.

PLS is used to find the fundamental relations between two matrices (X and Y), i.e. a latent variable approach to modeling the covariance structures in these two spaces. A PLS model will try to find the multidimensional direction in the X space that explains the maximum multidimensional variance direction in the Y space. PLS regression is particularly suited when the matrix of predictors has more variables than observations, and when there is multicollinearity among X values. By contrast, standard regression will fail in these cases.

The PLS algorithm is employed in PLS path modelling, a method of modeling a causal network of latent variables. This technique is a form of structural equation modeling, distinguished from the classical method by being component-based rather than covariance-based.

Partial least squares was introduced by the Swedish statistician Herman Wold, who then developed it with his son, Svante Wold, a professor of chemometrics at Umeå University. An alternative term for PLS (and more correct according to Svante Wold) is projection to latent structures, but the term partial least squares is still dominant in many areas. Although the original applications were in the social sciences, PLS regression is today most widely used in chemometrics and related areas. It is also used in bioinformatics, sensometrics, neuroscience and anthropology. In contrast, PLS path modeling is most often used in social sciences, econometrics, marketing and strategic management.

Underlying model

The general underlying model of multivariate PLS is

$\begin{align} X &= T P^{\top} + E\\ Y &= U Q^{\top} + F, \end{align}$

where X is an $n \times m$ matrix of predictors, Y is an $n \times p$ matrix of responses; T and U are $n \times l$ matrices that are, respectively, projections of X (the X score, component or factor matrix) and projections of Y (the Y scores); P and Q are, respectively, $m \times l$ and $p \times l$ orthogonal loading matrices; and matrices E and F are the error terms, assumed to be i.i.d. normal. The decompositions of X and Y are made so as to maximise the covariance of T and U.

Algorithms

A number of variants of PLS exist for estimating the factor and loading matrices T,P and Q. Most of them construct estimates of the linear regression between X and Y as $Y = X \tilde{B} + \tilde{B}_0$. Some PLS algorithms are only appropriate for the case where Y is a column vector, while others deal with the general case of a matrix Y. Algorithms also differ on whether they estimate the factor matrix T as an orthogonal, an orthonormal matrix or not. The final prediction will be the same for all these varieties of PLS, but the components will differ.

PLS1

PLS1 is a widely used algorithm appropriate for the vector Y case. It estimates T as an orthonormal matrix. In pseudocode it may be expressed as:

 1  function PLS1(X,y,l)
 2  $X^{(0)} \gets X$
 3  $W^{(0)} \gets X^T Y/||X^TY||$, an initial estimate of w.
 4  $T^{(0)} \gets X W^{(0)}$
 5  for k = 0 to l
 6      $t_k \gets {T^{(k)}}^T T^{(k)}$
 7      $T^{(k)} \gets T^{(k)} / t_k$
 8      $P^{(k)} \gets {X^{(k)}}^T T^{(k)}$
 9      $q_k \gets {y}^T T^{(k)}$
10      if qk = 0
11          $l \gets k$, break the for loop
12      if k < l
13          $X^{(k+1)} \gets X^{(k)} - t_k T^{(k)} {P^{(k)}}^T$
14          $W^{(k+1)} \gets {X^{(k+1)}}^T y$
15          $T^{(k+1)} \gets X^{(k+1)}W^{(k+1)}$
16  end for
17  define W to be the matrix with columns W(0),W(1),...,W(l − 1). Similarly define P,q
18  $B \gets W {(P^T W)}^{-1} q$
19  $B_0 \gets q_0 - {P^{(0)}}^T B$
20  return B,B0

This form of the algorithm does not require centering of the input X and Y, as this is performed implicitly by the algorithm. This algorithm features 'deflation' of the matrix X (subtraction of $T^{(k)} {P^{(k)}}^T$), but deflation of the vector y is not performed, as it is not necessary (it can be proved that deflating y yields the same results as not deflating.). The user-supplied variable l is the limit on the number of latent factors in the regression; if it equals the rank of the matrix X, the algorithm will yield the least squares regression estimates for B and B₀

Extensions

In 2002 a new preprocessing method was published called orthogonal projections to latent structures (OPLS). In OPLS systematic variation in the X matrix not correlated to Y is removed. This improves the interpretability (but not the predictivity) of the PLS models. L-PLS extends PLS regression to 3 connected data blocks.

Software implementation

Most major statistical software packages offer PLS regression, including SAS, JMP, SPSS and Statistica. The following PLS path modeling software tools implement PLS regression or variations of it: PLS-Graph, SmartPLS, and WarpPLS. There is a PLS regression module in XLSTAT, an add-in for Microsoft Excel. SIMCA-P+ from Umetrics and The Unscrambler are popular amongst chemometricians and sensometricians. PLS regression routines are available for MATLAB, the popular open source language R and open source data mining software tool Orange.

See also

• Feature extraction
• Data mining
• Machine learning
• Regression analysis

Further reading

• R. Kramer, Chemometric Techniques for Quantitative Analysis, (1998). Marcel-Dekker, ISBN 0-8247-0198-4.
• Frank, Ildiko and Jerome Friedman (1993). A Statistical View of Some Chemometrics Regression Tools, Technometrics, 35(2), pp 109–148. 
• Haenlein, Michael and Andreas M. Kaplan (2004). A Beginner's Guide to Partial Least Squares Analysis, Understanding Statistics, 3(4), 283–297. 
• Henseler, Joerg and Georg Fassott (2005). Testing Moderating Effects in PLS Path Models. An Illustration of Available Procedures. 
• Lingjærde, Ole-Christian and Nils Christophersen (2000). Shrinkage Structure of Partial Least Squares, Scandinavian Journal of Statistics, 27(3), pp 459–473. 
• Tenenhaus Michel (1998). La Regression PLS: Theorie et Pratique. Paris: Technip.. 
• Rosipal, Roman and Nicole Kramer (2006) (2006). Overview and Recent Advances in Partial Least Squares, in Subspace, Latent Structure and Feature Selection Techniques, pp 34–51. 
• Helland Inge S. (1990). PLS regression and statisticalmodels. Scandivian Journal of Statistics, 17, 97–114.
• Wold, Herman. (1966). Estimation of principal components and related models by iterative least squares. In P.R. Krishnaiaah (Ed.). Multivariate Analysis. (pp. 391–420) New York: Academic Press.
• Wold, Herman. (1981). The fix-point approach to interdependent systems. Amsterdam: North Holland.
• Wold, Herman. (1985). Partial least squares, pp. 581–591 in Samuel Kotz and Norman L. Johnson, eds., Encyclopedia of statistical sciences, Vol. 6, New York: Wiley, 1985.
• Svante Wold, Axel Ruhe, Herman Wold, and W.J. Dunn. "The collinearity problem in linear regression. the partial least squares (PLS) approach to generalized inverses." SIAM J. Sci. Stat. Comp., 5:735-743, 1984.
• Garthwaite, Paul H. (1994) "An Interpretation of Partial Least Squares", Journal of the American Statistical Association, 89, 122–127 JSTOR 2291207
• Vinzi, V., Chin, W.W., Henseler, J., Wang, H. (eds) (2010). Handbook of Partial Least Squares. ISBN 978-3-540-32825-4
• Stone, M. and Brooks, R. J. (1990) "Continuum Regression: Cross-Validated Sequentially Constructed Prediction embracing Ordinary Least Squares, Partial Least Squares and Principal Components Regression", Journal of the Royal Statistical Society, Series B, 52 (2), 237–269 JSTOR 2345437

External links

• imDEV free Excel add-in for PLS and PLS-DA
• Algorithm description
• PLS and regression tutorial
• PLS in Brain Imaging
• on-line PLS regression (PLSR) at Virtual Computational Chemistry Laboratory
• Uncertainty estimation for PLS
• A short introduction to PLS regression and its history