Jump to content

Quadratic classifier

From Wikipedia, the free encyclopedia

In statistics, a quadratic classifier is a statistical classifier that uses a quadratic decision surface to separate measurements of two or more classes of objects or events. It is a more general version of the linear classifier.

The classification problem

[edit]

Statistical classification considers a set of vectors of observations x of an object or event, each of which has a known type y. This set is referred to as the training set. The problem is then to determine, for a given new observation vector, what the best class should be. For a quadratic classifier, the correct solution is assumed to be quadratic in the measurements, so y will be decided based on 𝐱𝐓𝐀𝐱+𝐛𝐓𝐱+c

In the special case where each observation consists of two measurements, this means that the surfaces separating the classes will be conic sections (i.e., either a line, a circle or ellipse, a parabola or a hyperbola). In this sense, we can state that a quadratic model is a generalization of the linear model, and its use is justified by the desire to extend the classifier's ability to represent more complex separating surfaces.

Quadratic discriminant analysis

[edit]

Quadratic discriminant analysis (QDA) is closely related to linear discriminant analysis (LDA), where it is assumed that the measurements from each class are normally distributed.[1] Unlike LDA however, in QDA there is no assumption that the covariance of each of the classes is identical.[2] When the normality assumption is true, the best possible test for the hypothesis that a given measurement is from a given class is the likelihood ratio test. Suppose there are only two groups, with means μ0,μ1 and covariance matrices Σ0,Σ1 corresponding to y=0 and y=1 respectively. Then the likelihood ratio is given by Likelihood ratio=|2πΣ1|−1exp⁡(−12(𝐱−𝝁1)TΣ1−1(𝐱−𝝁1))|2πΣ0|−1exp⁡(−12(𝐱−𝝁0)TΣ0−1(𝐱−𝝁0))<t for some threshold t. After some rearrangement, it can be shown that the resulting separating surface between the classes is a quadratic. The sample estimates of the mean vector and variance-covariance matrices will substitute the population quantities in this formula.

Other

[edit]

While QDA is the most commonly used method for obtaining a classifier, other methods are also possible. One such method is to create a longer measurement vector from the old one by adding all pairwise products of individual measurements. For instance, the vector [x1,x2,x3] would become [x1,x2,x3,x12,x1x2,x1x3,x22,x2x3,x32].

Finding a quadratic classifier for the original measurements would then become the same as finding a linear classifier based on the expanded measurement vector. This observation has been used in extending neural network models;[3] the "circular" case, which corresponds to introducing only the sum of pure quadratic terms x12+x22+x32+⋯ with no mixed products (x1x2,x1x3,…), has been proven to be the optimal compromise between extending the classifier's representation power and controlling the risk of overfitting (Vapnik-Chervonenkis dimension).[4]

For linear classifiers based only on dot products, these expanded measurements do not have to be actually computed, since the dot product in the higher-dimensional space is simply related to that in the original space. This is an example of the so-called kernel trick, which can be applied to linear discriminant analysis as well as the support vector machine.

References

[edit]

Citations

[edit]
  1. ↑ Tharwat, Alaa (2016). "Linear vs. quadratic discriminant analysis classifier: a tutorial". International Journal of Applied Pattern Recognition. 3 (2): 145. doi:10.1504/IJAPR.2016.079050. ISSN 2049-887X.
  2. ↑ "Linear & Quadratic Discriminant Analysis · UC Business Analytics R Programming Guide". uc-r.github.io. Retrieved 2020-03-29.
  3. ↑ Cover TM (1965). "Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition". IEEE Transactions on Electronic Computers. EC-14 (3): 326–334. doi:10.1109/pgec.1965.264137.
  4. ↑ Ridella S, Rovetta S, Zunino R (1997). "Circular backpropagation networks for classification". IEEE Transactions on Neural Networks. 8 (1): 84–97. doi:10.1109/72.554194. PMID 18255613. href IEEE: .

General references

[edit]