Representer theorem

From Wikipedia, the free encyclopedia
Jump to: navigation, search

In statistical learning theory, a representer theorem is any of several related results stating that a minimizer of a regularized empirical risk function defined over a reproducing kernel Hilbert space can be represented as a finite linear combination of kernel products evaluated on the input points in the training set data.

Formal Statement[edit]

The following Representer Theorem and its proof are due to Schölkopf, Herbrich, and Smola:

Theorem: Let be a nonempty set and a positive-definite real-valued kernel on with corresponding reproducing kernel Hilbert space . Given a training sample , a strictly monotonically increasing real-valued function , and an arbitrary empirical risk function , then for any satisfying

admits a representation of the form:

where for all .

Proof: Define a mapping

(so that is itself a map ). Since is a reproducing kernel, then

where is the inner product on .

Given any , one can use orthogonal projection to decompose any into a sum of two functions, one lying in , and the other lying in the orthogonal complement:

where for all .

The above orthogonal decomposition and the reproducing property together show that applying to any training point produces

which we observe is independent of . Consequently, the value of the empirical risk in (*) is likewise independent of . For the second term (the regularization term), since is orthogonal to and is strictly monotonic, we have

Therefore setting does not affect the first term of (*), while it strictly decreasing the second term. Consequently, any minimizer in (*) must have , i.e., it must be of the form

which is the desired result.

Generalizations[edit]

The Theorem stated above is a particular example of a family of results that are collectively referred to as "representer theorems"; here we describe several such.

The first statement of a representer theorem was due to Kimeldorf and Wahba for the special case in which

for . Schölkopf, Herbrich, and Smola generalized this result by relaxing the assumption of the squared-loss cost and allowing the regularizer to be any strictly monotonically increasing function of the Hilbert space norm.

It is possible to generalize further by augmenting the regularized empirical risk function through the addition of unpenalized offset terms. For example, Schölkopf, Herbrich, and Smola also consider the minimization

i.e., we consider functions of the form , where and is an unpenalized function lying in the span of a finite set of real-valued functions . Under the assumption that the matrix has rank , they show that the minimizer in admits a representation of the form

where and the are all uniquely determined.

The conditions under which a representer theorem exists were investigated by Argyriou, Miccheli, and Pontil, who proved the following:

Theorem: Let be a nonempty set, a positive-definite real-valued kernel on with corresponding reproducing kernel Hilbert space , and let be a differentiable regularization function. Then given a training sample and an arbitrary empirical risk function , a minimizer

of the regularized empirical risk minimization problem admits a representation of the form

where for all , if and only if there exists a nondecreasing function for which

Effectively, this result provides a necessary and sufficient condition on a differentiable regularizer under which the corresponding regularized empirical risk minimization will have a representer theorem. In particular, this shows that a broad class of regularized risk minimizations (much broader than those originally considered by Kimeldorf and Wahba) have representer theorems.

Applications[edit]

Representer theorems are useful from a practical standpoint because they dramatically simplify the regularized empirical risk minimization problem . In most interesting applications, the search domain for the minimization will be an infinite-dimensional subspace of , and therefore the search (as written) does not admit implementation on finite-memory and finite-precision computers. In contrast, the representation of afforded by a representer theorem reduces the original (infinite-dimensional) minimization problem to a search for the optimal -dimensional vector of coefficients ; can then be obtained by applying any standard function minimization algorithm. Consequently, representer theorems provide the theoretical basis for the reduction of the general machine learning problem to algorithms that can actually be implemented on computers in practice.

See also[edit]

References[edit]

  • Argyriou, Andreas; Micchelli, Charles A.; Pontil, Massimiliano (2009). "When Is There a Representer Theorem? Vector Versus Matrix Regularizers". Journal of Machine Learning Research. 10 (Dec): 2507–2529. 
  • Cucker, Felipe; Smale, Steve (2002). "On the Mathematical Foundations of Learning". Bulletin of the American Mathematical Society. 39 (1): 1–49. doi:10.1090/S0273-0979-01-00923-5. MR 1864085. 
  • Kimeldorf, George S.; Wahba, Grace (1970). "A correspondence between Bayesian estimation on stochastic processes and smoothing by splines". The Annals of Mathematical Statistics. 41 (2): 495–502. doi:10.1214/aoms/1177697089. 
  • Schölkopf, Bernhard; Herbrich, Ralf; Smola, Alex J. (2001). "A Generalized Representer Theorem". Computational Learning Theory. Lecture Notes in Computer Science. 2111: 416–426. doi:10.1007/3-540-44581-1_27. ISBN 978-3-540-42343-0.