In machine learning, pattern recognition and in image processing, feature extraction starts from an initial set of measured data and builds derived values (features) intended to be informative, non redundant, facilitating the subsequent learning and generalization steps, in some cases leading to better human interpretations. Feature extraction is related to dimensionality reduction.
When the input data to an algorithm is too large to be processed and it is suspected to be redundant (e.g. the same measurement in both feet and meters, or the repetitiveness of images presented as pixels), then it can be transformed into a reduced set of features (also named features vector). This process is called feature extraction. The extracted features are expected to contain the relevant information from the input data, so that the desired task can be performed by using this reduced representation instead of the complete initial data.
Feature extraction involves reducing the amount of resources required to describe a large set of data. When performing analysis of complex data one of the major problems stems from the number of variables involved. Analysis with a large number of variables generally requires a large amount of memory and computation power or a classification algorithm which overfits the training sample and generalizes poorly to new samples. Feature extraction is a general term for methods of constructing combinations of the variables to get around these problems while still describing the data with sufficient accuracy.
The best results are achieved when an expert constructs a set of application-dependent features. Nevertheless, if no such expert knowledge is available, general dimensionality reduction techniques may help. These include:
- Principal component analysis
- Semidefinite embedding
- Multifactor dimensionality reduction
- Multilinear subspace learning
- Nonlinear dimensionality reduction
- Kernel PCA
- Multilinear PCA
- Latent semantic analysis
- Partial least squares
- Independent component analysis
One very important area of application is image processing, in which algorithms are used to detect and isolate various desired portions or shapes (features) of a digitized image or video stream. It is particularly important in the area of optical character recognition.
- Edge direction, changing intensity, autocorrelation.
- Blob extraction
- Template matching
- Hough transform
- Arbitrary shapes (generalized Hough transform)
- Works with any parameterizable feature (class variables, cluster detection, etc..)
- Deformable, parameterized shapes
- Active contours (snakes)
Feature extraction in software
Many data analysis software packages provide for feature extraction and dimension reduction. Common numerical programming environments such as MATLAB, SciLab, NumPy and the R language provide some of the simpler feature extraction techniques (e.g. principal component analysis) via built-in commands. More specific algorithms are often available as publicly available scripts or third-party add-ons.
- Cluster analysis
- Dimensionality reduction
- Feature detection
- Feature selection
- Data mining
- Connected-component labeling
- Segmentation (image processing)
- JMLR Special Issue on Variable and Feature Selection
- IMMI - Rapidminer Image Mining Extension - open-source tool for image mining