Jump to content

MNIST database: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous editNext edit →
Content deleted Content added
VisualWikitext
AnomieBOT (talk | contribs)
Bots
6,502,471 edits
Rescuing orphaned refs ("Matuzas" from rev 946970357)
This particular error rate is no longer the lowest one and therefore it is no longer relevant to mention it exclusively in the introduction section.
Line 3: Line 3:


The MNIST database contains 60,000 training images and 10,000 testing images.<ref>{{cite journal|last=Kussul|first=Ernst|author2=Tatiana Baidyk|title=Improved method of handwritten digit recognition tested on MNIST database|journal=Image and Vision Computing|year=2004|volume=22|issue=12|pages=971–981|doi=10.1016/j.imavis.2004.03.008}}</ref> Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.<ref>{{cite journal|last=Zhang|first=Bin|author2=Sargur N. Srihari|title=Fast k -Nearest Neighbor Classification Using Cluster-Based Trees|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|year=2004|volume=26|issue=4|pages=525–528|url=http://mleg.cse.sc.edu/edu/csce822/uploads/Main.ReadingList/KNN_fastbyClustering.pdf|accessdate=18 August 2013|doi=10.1109/TPAMI.2004.1265868|pmid=15382657}}</ref>
The MNIST database contains 60,000 training images and 10,000 testing images.<ref>{{cite journal|last=Kussul|first=Ernst|author2=Tatiana Baidyk|title=Improved method of handwritten digit recognition tested on MNIST database|journal=Image and Vision Computing|year=2004|volume=22|issue=12|pages=971–981|doi=10.1016/j.imavis.2004.03.008}}</ref> Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.<ref>{{cite journal|last=Zhang|first=Bin|author2=Sargur N. Srihari|title=Fast k -Nearest Neighbor Classification Using Cluster-Based Trees|journal=IEEE Transactions on Pattern Analysis and Machine Intelligence|year=2004|volume=26|issue=4|pages=525–528|url=http://mleg.cse.sc.edu/edu/csce822/uploads/Main.ReadingList/KNN_fastbyClustering.pdf|accessdate=18 August 2013|doi=10.1109/TPAMI.2004.1265868|pmid=15382657}}</ref>
There have been a number of [[Academic publishing|scientific papers]] on attempts to achieve the lowest error rate; one paper, using a hierarchical system of [[convolutional neural network]]s, manages to get an [[Computer performance|error rate]] on the MNIST database of 0.23%.<ref name="Multideep">{{cite book|last=Cires¸an|first=Dan|author2=Ueli Meier |author3=Jürgen Schmidhuber |title=Multi-column deep neural networks for image classification|journal=2012 IEEE Conference on Computer Vision and Pattern Recognition|year=2012|pages=3642–3649|doi=10.1109/CVPR.2012.6248110|arxiv=1202.2745|isbn=978-1-4673-1228-8|url=http://repository.supsi.ch/5145/1/IDSIA-04-12.pdf|citeseerx=10.1.1.300.3283}}</ref> The original creators of the database keep a list of some of the methods tested on it.<ref name="LeCun" /> In their original paper, they use a [[support-vector machine]] to get an error rate of 0.8%.<ref name="Gradient">{{cite journal|last=LeCun|first=Yann|author2=Léon Bottou |author3=Yoshua Bengio |author4=Patrick Haffner |title=Gradient-Based Learning Applied to Document Recognition|journal=Proceedings of the IEEE |year=1998|volume=86|issue=11|pages=2278–2324|url=http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf|accessdate=18 August 2013|doi=10.1109/5.726791}}</ref> An extended dataset similar to MNIST called EMNIST has been published in 2017, which contains 240,000 training images, and 40,000 testing images of handwritten digits and characters.<ref>{{cite arxiv|last=Cohen|first=Gregory|last2=Afshar|first2=Saeed|last3=Tapson|first3=Jonathan|last4=van Schaik|first4=André|date=2017-02-17|title=EMNIST: an extension of MNIST to handwritten letters|eprint=1702.05373|class=cs.CV}}</ref>
The original creators of the database keep a list of some of the methods tested on it.<ref name="LeCun" /> In their original paper, they use a [[support-vector machine]] to get an error rate of 0.8%.<ref name="Gradient">{{cite journal|last=LeCun|first=Yann|author2=Léon Bottou |author3=Yoshua Bengio |author4=Patrick Haffner |title=Gradient-Based Learning Applied to Document Recognition|journal=Proceedings of the IEEE |year=1998|volume=86|issue=11|pages=2278–2324|url=http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf|accessdate=18 August 2013|doi=10.1109/5.726791}}</ref> An extended dataset similar to MNIST called EMNIST has been published in 2017, which contains 240,000 training images, and 40,000 testing images of handwritten digits and characters.<ref>{{cite arxiv|last=Cohen|first=Gregory|last2=Afshar|first2=Saeed|last3=Tapson|first3=Jonathan|last4=van Schaik|first4=André|date=2017-02-17|title=EMNIST: an extension of MNIST to handwritten letters|eprint=1702.05373|class=cs.CV}}</ref>


== Dataset ==
== Dataset ==
Line 9: Line 9:


== Performance ==
== Performance ==
Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks.<ref name="Multideep" /> The highest error rate listed<ref name="LeCun" /> on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing.<ref name="Gradient" />
Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks.<ref name="Multideep">{{cite book|last=Cires¸an|first=Dan|url=http://repository.supsi.ch/5145/1/IDSIA-04-12.pdf|title=Multi-column deep neural networks for image classification|author2=Ueli Meier|author3=Jürgen Schmidhuber|journal=2012 IEEE Conference on Computer Vision and Pattern Recognition|year=2012|isbn=978-1-4673-1228-8|pages=3642–3649|arxiv=1202.2745|citeseerx=10.1.1.300.3283|doi=10.1109/CVPR.2012.6248110}}</ref> The highest error rate listed<ref name="LeCun" /> on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing.<ref name="Gradient" />


In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles.<ref>{{cite journal|last=Kussul|first=Ernst|author2=Tatiana Baidyk|title=Improved method of handwritten digit recognition tested on MNIST database|journal=Image and Vision Computing|year=2004|volume=22|issue=12|pages=971–981|doi=10.1016/j.imavis.2004.03.008|url=https://vlabdownload.googlecode.com/files/Image_VisionComputing.pdf|accessdate=20 September 2013|archive-url=https://web.archive.org/web/20130921060416/https://vlabdownload.googlecode.com/files/Image_VisionComputing.pdf|archive-date=21 September 2013|url-status=dead}}</ref>
In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles.<ref>{{cite journal|last=Kussul|first=Ernst|author2=Tatiana Baidyk|title=Improved method of handwritten digit recognition tested on MNIST database|journal=Image and Vision Computing|year=2004|volume=22|issue=12|pages=971–981|doi=10.1016/j.imavis.2004.03.008|url=https://vlabdownload.googlecode.com/files/Image_VisionComputing.pdf|accessdate=20 September 2013|archive-url=https://web.archive.org/web/20130921060416/https://vlabdownload.googlecode.com/files/Image_VisionComputing.pdf|archive-date=21 September 2013|url-status=dead}}</ref>

Revision as of 20:48, 28 March 2020

The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems.[1][2] The database is also widely used for training and testing in the field of machine learning.[3][4] It was created by "re-mixing" the samples from NIST's original datasets. The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments.[5] Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels.[5]

MNIST sample images
Sample images from MNIST test dataset

The MNIST database contains 60,000 training images and 10,000 testing images.[6] Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.[7] The original creators of the database keep a list of some of the methods tested on it.[5] In their original paper, they use a support-vector machine to get an error rate of 0.8%.[8] An extended dataset similar to MNIST called EMNIST has been published in 2017, which contains 240,000 training images, and 40,000 testing images of handwritten digits and characters.[9]

Dataset

The set of images in the MNIST database is a combination of two of NIST's databases: Special Database 1 and Special Database 3. Special Database 1 and Special Database 3 consist of digits written by high school students and employees of the United States Census Bureau, respectively.[5]

Performance

Some researchers have achieved "near-human performance" on the MNIST database, using a committee of neural networks; in the same paper, the authors achieve performance double that of humans on other recognition tasks.[10] The highest error rate listed[5] on the original website of the database is 12 percent, which is achieved using a simple linear classifier with no preprocessing.[8]

In 2004, a best-case error rate of 0.42 percent was achieved on the database by researchers using a new classifier called the LIRA, which is a neural classifier with three neuron layers based on Rosenblatt's perceptron principles.[11]

Some researchers have tested artificial intelligence systems using the database put under random distortions. The systems in these cases are usually neural networks and the distortions used tend to be either affine distortions or elastic distortions.[5] Sometimes, these systems can be very successful; one such system achieved an error rate on the database of 0.39 percent.[12]

In 2011, an error rate of 0.27 percent, improving on the previous best result, was reported by researchers using a similar system of neural networks.[13] In 2013, an approach based on regularization of neural networks using DropConnect has been claimed to achieve a 0.21 percent error rate.[14] In 2016 the single convolutional neural network best performance was 0.31 percent error rate.[15] As of August 2018, the best performance of a single convolutional neural network trained on MNIST training data using realtime data augmentation is 0.26 percent error rate.[16] Also, the Parallel Computing Center (Khmelnitskiy, Ukraine) obtained an ensemble of only 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate.[17][18] Some images in the testing dataset are barely readable and may prevent reaching test error rates of 0%.[16] In 2018 researchers from Department of System and Information Engineering, University of Virginia announced 0.18% error with simultaneous stacked three kind of neural networks (fully connected, recurrent and convolution neural networks) [19].

Classifiers

This is a table of some of the machine learning methods used on the database and their error rates, by type of classifier:

Type Classifier Distortion Preprocessing Error rate (%)
Linear classifier Pairwise linear classifier None Deskewing 7.6[8]
Decision stream with Extremely randomized trees Single model (depth > 400 levels) None None 2.7[20]
K-Nearest Neighbors K-NN with non-linear deformation (P2DHMDM) None Shiftable edges 0.52[21]
Boosted Stumps Product of stumps on Haar features None Haar features 0.87[22]
Non-linear classifier 40 PCA + quadratic classifier None None 3.3[8]
Random Forest Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC)[23] None Simple statistical pixel importance 2.8[24]
Support-vector machine (SVM) Virtual SVM, deg-9 poly, 2-pixel jittered None Deskewing 0.56[25]
Deep neural network (DNN) 2-layer 784-800-10 None None 1.6[26]
Deep neural network 2-layer 784-800-10 Elastic distortions None 0.7[26]
Deep neural network 6-layer 784-2500-2000-1500-1000-500-10 Elastic distortions None 0.35[27]
Convolutional neural network (CNN) 6-layer 784-40-80-500-1000-2000-10 None Expansion of the training data 0.31[15]
Convolutional neural network 6-layer 784-50-100-500-1000-10-10 None Expansion of the training data 0.27[28]
Convolutional neural network Committee of 35 CNNs, 1-20-P-40-P-150-10 Elastic distortions Width normalizations 0.23[10]
Convolutional neural network Committee of 5 CNNs, 6-layer 784-50-100-500-1000-10-10 None Expansion of the training data 0.21[17][18]
Random Multimodel Deep Learning (RMDL) 10 NN-10 RNN - 10 CNN None None 0.18[19]
Convolutional neural network Committee of 20 CNNS with Squeeze-and-Excitation Networks[29] None Data augmentation 0.17[30]

See also

References

  1. ^ "Support vector machines speed pattern recognition - Vision Systems Design". Vision Systems Design. Retrieved 17 August 2013.
  2. ^ Gangaputra, Sachin. "Handwritten digit database". Retrieved 17 August 2013.
  3. ^ Qiao, Yu (2007). "THE MNIST DATABASE of handwritten digits". Retrieved 18 August 2013.
  4. ^ Platt, John C. (1999). "Using analytic QP and sparseness to speed training of support vector machines" (PDF). Advances in Neural Information Processing Systems: 557–563. Archived from the original (PDF) on 4 March 2016. Retrieved 18 August 2013.
  5. ^ a b c d e f LeCun, Yann; Corinna Cortes; Christopher J.C. Burges. "MNIST handwritten digit database, Yann LeCun, Corinna Cortes and Chris Burges". Retrieved 17 August 2013.
  6. ^ Kussul, Ernst; Tatiana Baidyk (2004). "Improved method of handwritten digit recognition tested on MNIST database". Image and Vision Computing. 22 (12): 971–981. doi:10.1016/j.imavis.2004.03.008.
  7. ^ Zhang, Bin; Sargur N. Srihari (2004). "Fast k -Nearest Neighbor Classification Using Cluster-Based Trees" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence. 26 (4): 525–528. doi:10.1109/TPAMI.2004.1265868. PMID 15382657. Retrieved 18 August 2013.
  8. ^ a b c d LeCun, Yann; Léon Bottou; Yoshua Bengio; Patrick Haffner (1998). "Gradient-Based Learning Applied to Document Recognition" (PDF). Proceedings of the IEEE. 86 (11): 2278–2324. doi:10.1109/5.726791. Retrieved 18 August 2013.
  9. ^ Cohen, Gregory; Afshar, Saeed; Tapson, Jonathan; van Schaik, André (2017-02-17). "EMNIST: an extension of MNIST to handwritten letters". arXiv:1702.05373 [cs.CV].
  10. ^ a b Cires¸an, Dan; Ueli Meier; Jürgen Schmidhuber (2012). Multi-column deep neural networks for image classification (PDF). pp. 3642–3649. arXiv:1202.2745. CiteSeerX 10.1.1.300.3283. doi:10.1109/CVPR.2012.6248110. ISBN 978-1-4673-1228-8. {{cite book}}: |journal= ignored (help)
  11. ^ Kussul, Ernst; Tatiana Baidyk (2004). "Improved method of handwritten digit recognition tested on MNIST database" (PDF). Image and Vision Computing. 22 (12): 971–981. doi:10.1016/j.imavis.2004.03.008. Archived from the original (PDF) on 21 September 2013. Retrieved 20 September 2013.
  12. ^ Ranzato, Marc’Aurelio; Christopher Poultney; Sumit Chopra; Yann LeCun (2006). "Efficient Learning of Sparse Representations with an Energy-Based Model" (PDF). Advances in Neural Information Processing Systems. 19: 1137–1144. Retrieved 20 September 2013.
  13. ^ Ciresan, Dan Claudiu; Ueli Meier; Luca Maria Gambardella; Jürgen Schmidhuber (2011). Convolutional neural network committees for handwritten character classification (PDF). pp. 1135–1139. CiteSeerX 10.1.1.465.2138. doi:10.1109/ICDAR.2011.229. ISBN 978-1-4577-1350-7. Archived from the original (PDF) on 22 February 2016. Retrieved 20 September 2013. {{cite book}}: |journal= ignored (help)
  14. ^ Wan, Li; Matthew Zeiler; Sixin Zhang; Yann LeCun; Rob Fergus (2013). Regularization of Neural Network using DropConnect. International Conference on Machine Learning(ICML).
  15. ^ a b Romanuke, Vadim. "The single convolutional neural network best performance in 18 epochs on the expanded training data at Parallel Computing Center, Khmelnitskiy, Ukraine". Retrieved 16 November 2016.
  16. ^ a b MNIST classifier, GitHub. "Classify MNIST digits using Convolutional Neural Networks". Retrieved 3 August 2018.
  17. ^ a b Romanuke, Vadim. "Parallel Computing Center (Khmelnitskiy, Ukraine) represents an ensemble of 5 convolutional neural networks which performs on MNIST at 0.21 percent error rate". Retrieved 24 November 2016.
  18. ^ a b Romanuke, Vadim (2016). "Training data expansion and boosting of convolutional neural networks for reducing the MNIST dataset error rate". Research Bulletin of NTUU "Kyiv Polytechnic Institute". 6 (6): 29–34. doi:10.20535/1810-0546.2016.6.84115.
  19. ^ a b Kowsari, Kamran; Heidarysafa, Mojtaba; Brown, Donald E.; Meimandi, Kiana Jafari; Barnes, Laura E. (2018-05-03). "RMDL: Random Multimodel Deep Learning for Classification". Proceedings of the 2018 International Conference on Information System and Data Mining. arXiv:1805.01890. Bibcode:2018arXiv180501890K. doi:10.1145/3206098.3206111.
  20. ^ Ignatov, D.Yu.; Ignatov, A.D. (2017). "Decision Stream: Cultivating Deep Decision Trees". IEEE ICTAI: 905–912. arXiv:1704.07657. doi:10.1109/ICTAI.2017.00140.
  21. ^ Keysers, Daniel; Thomas Deselaers; Christian Gollan; Hermann Ney (August 2007). "Deformation models for image recognition". IEEE Transactions on Pattern Analysis and Machine Intelligence. 29 (8): 1422–1435. CiteSeerX 10.1.1.106.3963. doi:10.1109/TPAMI.2007.1153. PMID 17568145.
  22. ^ Kégl, Balázs; Róbert Busa-Fekete (2009). "Boosting products of base classifiers" (PDF). Proceedings of the 26th Annual International Conference on Machine Learning: 497–504. Retrieved 27 August 2013.
  23. ^ https://cran.r-project.org/web/packages/randomForestSRC/
  24. ^ https://gitlab.com/m.mahmoudian/mnist-with-randomforest
  25. ^ DeCoste and Scholkopf, MLJ 2002
  26. ^ a b Patrice Y. Simard; Dave Steinkraus; John C. Platt (2003). Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis. Vol. 1. IEEE. p. 958. doi:10.1109/ICDAR.2003.1227801. ISBN 978-0-7695-1960-9. {{cite book}}: |journal= ignored (help)
  27. ^ Ciresan, Claudiu Dan; Ueli Meier; Luca Maria Gambardella; Juergen Schmidhuber (December 2010). "Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition". Neural Computation. 22 (12): 3207–20. arXiv:1003.0358. Bibcode:2010arXiv1003.0358C. doi:10.1162/NECO_a_00052. PMID 20858131.
  28. ^ Romanuke, Vadim. "Parallel Computing Center (Khmelnitskiy, Ukraine) gives a single convolutional neural network performing on MNIST at 0.27 percent error rate". Retrieved 24 November 2016.
  29. ^ Hu, Jie; Shen, Li; Albanie, Samuel; Sun, Gang; Wu, Enhua (2017). "Squeeze-and-Excitation Networks". arXiv:1709.01507 [cs.CV].
  30. ^ https://github.com/Matuzas77/MNIST-0.17.git

Further reading

Retrieved from "https://en.wikipedia.org/w/index.php?title=MNIST_database&oldid=947846949"