Non-negative matrix factorization

From Wikipedia, the free encyclopedia

Jump to: navigation, search
NMF redirects here. For the bridge convention, see new minor forcing.

Non-negative matrix factorization (NMF) is a group of algorithms in multivariate analysis and linear algebra where a matrix, \mathbf{X}, is factorized into (usually) two matrices, \mathbf{W} and \mathbf{H} : \operatorname{nmf}(\mathbf{X}) \rightarrow \mathbf{WH}

Factorization of matrices is generally non-unique, and a number of different methods of doing so have been developed (e.g. principal component analysis and singular value decomposition) by incorporating different constraints; non-negative matrix factorization differs from these methods in that it enforces the constraint that the factors W and H must be non-negative, i.e., all elements must be equal to or greater than zero.

Contents

[edit] History

In chemometrics non-negative matrix factorization has a long history under the name "self modeling curve resolution".[1] In this framework the vectors in the right matrix are continuous curves rather than discrete vectors. Also early work on non-negative matrix factorizations was performed by a Finnish group of researchers in the middle of the 1990s under the name positive matrix factorization.[2][3] It became more widely known as non-negative matrix factorization after Lee and Seung investigated the properties of the algorithm and published some simple and useful algorithms for two types of factorizations.[4][5]

[edit] Types

[edit] Approximative non-negative matrix factorization

Usually the number of columns of W and the number of rows of H in NMF are selected so the product WH will become an approximation to X (it has been suggested that the NMF model should be called nonnegative matrix approximation instead). The full decomposition of X then amounts to the two non-negative matrices W and H as well as a residual U, such that: X = WH + U. The elements of the residual matrix can either be negative or positive.

When W and H are smaller than X they become easier to store and manipulate.

[edit] Different cost functions and regularizations

There are different types of non-negative matrix factorizations. The different types arise from using different cost functions for measuring the divergence between X and WH and possibly by regularization of the W and/or H matrices.[6]

Two simple divergence functions studied by Lee and Seung are the squared error (or Frobenius norm) and an extension of the Kullback-Leibler divergence to positive matrices (the original Kullback-Leibler divergence is defined on probability distributions). Each divergence leads to a different NMF algorithm, usually minimizing the divergence using iterative update rules.

The factorization problem in the squared error version of NMF may be stated as: Given a nonnegative matrix \mathbf{X} find nonnegative matrices W and H that minimize the function

F(\mathbf{W},\mathbf{H}) = ||\mathbf{X} - \mathbf{WH}||^2_F

Another type of NMF for images is based on the total variation norm.[7]

[edit] Algorithms

There are several ways in which the W and H may be found: Lee and Seung's updates are usually referred to as the multiplicative update method, while others have suggested gradient descent algorithms and so-called alternating non-negative least squares and "projected gradient".[8][9]

The algorithms may be less than ideal, however, as they typically can only be guaranteed to find local minima, rather than a global minimum of the cost function but in many data mining applications a local minimum may still be enough to be useful.

[edit] Relation to other techniques

The initial paper by Lee & Seung proposed NMF mainly for parts-based decomposition of images. It compares NMF to vector quantization and principal component analysis, and shows that although the three techniques may be written as factorizations, they implement different constraints and therefore produce different results.

It was later shown that some types of NMF are an instance of a more general probabilistic model called "multinomial PCA".[10] When NMF is obtained by minimizing the Kullback–Leibler divergence, it is in fact equivalent to another instance of multinomial PCA, probabilistic latent semantic analysis,[11] trained by maximum likelihood estimation. That method is commonly used for analyzing and clustering textual data and is also related to the latent class model.

It was also shown[12][13] that when the Frobenius norm is used as a divergence, NMF is equivalent to a relaxed form of K-means clustering: matrix factor W contains cluster centroids and H contains cluster membership indicators. This also justifies the use of NMF for data clustering.

NMF extends beyond matrices to tensors of arbitrary order.[14][15] This extension may be viewed as a non-negative version of, e.g., the PARAFAC model.

[edit] Uniqueness

The factorization is not unique: A matrix and its inverse can be used to transform the two factorization matrices by, e.g.,[16]

\mathbf{WH} = \mathbf{WBB}^{-1}\mathbf{H}

If the two new matrices \mathbf{\tilde{W} = WB} and \mathbf{\tilde{H}}=\mathbf{B}^{-1}\mathbf{H} are non-negative they form another parametrization of the factorization.

The non-negativity of \mathbf{\tilde{W}} and \mathbf{\tilde{H}} applies at least if B is a non-negative monomial matrix. In this simple case it will just correspond to a scaling and a permutation.

More control over the non-uniqueness of NMF is obtained with sparsity constraints.[17]

[edit] Applications

[edit] Text mining

NMF can be used for text mining applications. In this process, a document-term matrix is constructed with the weights of various terms (typically weighted word frequency information) from a set of documents. This matrix is factored into a term-feature and a feature-document matrix. The features are derived from the contents of the documents, and the feature-document matrix describes data clusters of related documents.

One specific application used hierarchical NMF on a small subset of scientific abstracts from PubMed.[18] Another research group clustered parts of the Enron email dataset[19] with 65,033 messages and 91,133 terms into 50 clusters.[20] NMF has also been applied to citations data, with one example clustering Wikipedia articles and scientific journals based on the outbound scientific citations in Wikipedia.[21]

[edit] Spectral data analysis

NMF is also used to analyze spectral data; one such use is in the classification of space objects and debris.[22]

[edit] Current research

Current research in approximate nonnegative matrix factorization includes searching for techniques to initialize the factor matrices for various algorithms, algorithms to find global minima for the factors, and efficient ways to update the factors when new data is added to the matrix.

[edit] See also

  • Online NMF (Online non-negative matrix factorization)

[edit] Sources and external links

[edit] Notes

  1. ^ William H. Lawton; Edward A. Sylvestre (August 1971). "Self modeling curve resolution". Technometrics 13 (3): 617+. 
  2. ^ P. Paatero, U. Tapper (1994). "Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values". Environmetrics 5: 111–126. doi:10.1002/env.3170050203. 
  3. ^ Pia Anttila, Pentti Paatero, Unto Tapper, Olli Järvinen (1995). "Source identification of bulk wet deposition in Finland by positive matrix factorization". Atmospheric Environment 29 (14): 1705–1718. doi:10.1016/1352-2310(94)00367-T. 
  4. ^ Daniel D. Lee and H. Sebastian Seung (1999). "Learning the parts of objects by non-negative matrix factorization". Nature 401 (6755): 788–791. doi:10.1038/44565. 
  5. ^ Daniel D. Lee and H. Sebastian Seung (2001). "Algorithms for Non-negative Matrix Factorization". Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference. MIT Press. pp. 556–562. http://www.nips.cc/Web/Groups/NIPS/NIPS2000/00papers-pub-on-web/LeeSeung.ps.gz. 
  6. ^ Inderjit S. Dhillon, Suvrit Sra (2005). "Generalized Nonnegative Matrix Approximations with Bregman Divergences" (PDF). NIPS. http://books.nips.cc/papers/files/nips18/NIPS2005_0203.pdf. 
  7. ^ Taiping Zhanga, Bin Fang, Weining Liu, Yuan Yan Tang, Guanghui He and Jing Wen (June 2008). "Total variation norm-based nonnegative matrix factorization for identifying discriminant representation of image patterns". Neurocomputing 71 (10–12): 1824–1831. doi:10.1016/j.neucom.2008.01.022. 
  8. ^ Chih-Jen Lin (October 2007). "Projected Gradient Methods for Non-negative Matrix Factorization" (PDF). Neural Computation 19 (10). http://www.csie.ntu.edu.tw/~cjlin/papers/pgradnmf.pdf. 
  9. ^ Chih-Jen Lin (November 2007). "On the Convergence of Multiplicative Update Algorithms for Nonnegative Matrix Factorization". IEEE Transactions on Neural Networks 18 (6): 1589–1596. doi:10.1109/TNN.2007.895831. 
  10. ^ Wray Buntine (2002). "Extensions to EM and Multinomial PCA" (PDF). Proc. European Conference on Machine Learning (ECML-02). 2430. pp. 23–34. http://cosco.hiit.fi/Articles/ecml02.pdf. 
  11. ^ Eric Gaussier and Cyril Goutte (2005). "Relation between PLSA and NMF and Implications" (PDF). Proc. 28th international ACM SIGIR conference on Research and development in information retrieval (SIGIR-05). pp. 601–602. http://eprints.pascal-network.org/archive/00000971/01/39-gaussier.pdf. 
  12. ^ Ron Zass and Amnon Shashua (2005). "A Unifying Approach to Hard and Probabilistic Clustering". International Conference on Computer Vision (ICCV) Beijing, China, Oct., 2005.
  13. ^ Chris Ding, Xiaofeng He, and Horst D. Simon (2005). "On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering". Proc. SIAM Int'l Conf. Data Mining, pp. 606-610.
  14. ^ Pentti Paatero (1999). "The Multilinear Engine: A Table-Driven, Least Squares Program for Solving Multilinear Problems, including the n-Way Parallel Factor Analysis Model". Journal of Computational and Graphical Statistics 8 (4): 854–888. doi:10.2307/1390831. 
  15. ^ Max Welling and Markus Weber (2001). "Positive Tensor Factorization". Pattern Recognition Letters 22 (12): 1255–1261. doi:10.1016/S0167-8655(01)00070-8. 
  16. ^ Wei Xu, Xin Liu & Yihong Gong (2003). "Document clustering based on non-negative matrix factorization". Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval. New York: Association for Computing Machinery. pp. 267–273. http://portal.acm.org/citation.cfm?id=860485. 
  17. ^ Julian Eggert, Edgar Körner, "Sparse coding and NMF", Proceedings. 2004 IEEE International Joint Conference on Neural Networks, 2004., pp. 2529-2533, 2004.
  18. ^ Finn Årup Nielsen, Daniela Balslev & Lars Kai Hansen (September 2005). "Mining the posterior cingulate: segregation between memory and pain components". NeuroImage 27 (3): 520–522. doi:10.1016/j.neuroimage.2005.04.034. PMID 15946864. 
  19. ^ William W. Cohen (2005-04-04). "Enron Email Dataset". http://www.cs.cmu.edu/~enron/. Retrieved 2008-08-26. 
  20. ^ Michael W. Berry & Murray Browne (October 2005). "Email Surveillance Using Non-negative Matrix Factorization". Computational & Mathematical Organization Theory 11 (3): 249–264. doi:10.1007/s10588-005-5380-5. 
  21. ^ Finn Årup Nielsen (2008). "Clustering of scientific citations in Wikipedia". Wikimania. http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=5666. 
  22. ^ Michael W. Berry, et al. (June 2006). Algorithms and Applications for Approximate Nonnegative Matrix Factorization. 

[edit] Others

[edit] Software

Retrieved from "http://en.wikipedia.org/wiki/Non-negative_matrix_factorization"