Jump to content

User:Themumblingprophet/sandbox

From Wikipedia, the free encyclopedia

Exmaple Algorithm Formatting

[edit]
Algorithm Ford–Fulkerson
Inputs Given a Network with flow capacity c, a source node s, and a sink node t
Output Compute a flow f from s to t of maximum value
  1. for all edges
  2. While there is a path p from s to t in , such that for all edges :
    1. Find
    2. For each edge
      1. (Send flow along the path)
      2. (The flow might be "returned" later)
  • "←" denotes assignment. For instance, "largest ← item" means that the value of largest changes to the value of item.
  • "return" terminates the algorithm and outputs the following value.

From Ford–Fulkerson algorithm found via Articles with example pseudocode

Here is a draft of the history section of the Long short-term memory page.


About Apple's Quicktype: The sourced articles are all secondary sources, tech news blogs. And both of them (three including the Wired article) said that apple would soon deploy LSTMs in quicktype. It had not yet. They were all reporting on a single talk by Apple's Vice President of software engineering, Craig Federighi in a 2016 Apple developers conference. In part of the talk he said that Apple would use LSTMs to improve in Siri and Quicktype. The blog post announcing the Apple's use of LSTMs in Quicktype came out in September of 2018. And the blog post about LSTMs in Siri came out in August of 2017. The paper about the model they used as published earlier in 2017. Also, Apple uses the LSTM for language identification (2019).

History

[edit]

1997: LSTM was proposed by Sepp Hochreiter and Jürgen Schmidhuber.[1] By introducing Constant Error Carousel (CEC) units, LSTM deals with the vanishing gradient problem. The initial version of LSTM block included cells, input and output gates.[2]

1999: Felix Gers and his advisor Jürgen Schmidhuber and Fred Cummins introduced the forget gate (also called “keep gate”) into LSTM architecture,[3] enabling the LSTM to reset its own state.[2]

2000: Gers & Schmidhuber & Cummins added peephole connections (connections from the cell to the gates) into the architecture.[4] Additionally, the output activation function was omitted.[2]

2009: An LSTM based model won the ICDAR connected handwriting recognition competition. Three such models were submitted by a team lead by Alex Graves.[5] One was the most accurate model in the competition and another was the fasted.[6]

2013: LSTM networks were a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset.[7]

2014: Kyunghyun Cho et al. put forward a simplified variant called Gated recurrent unit (GRU).[8]

2015: Google started using an LSTM for speech recognition on Google Voice.[9][10] According to the official blog post, the new model cut transcription errors by 49%. [11]

2016: Google started using an LSTM to suggest messages in the Allo conversation app.[12] In the same year, Google released the Google Neural Machine Translation system for Google Translate which used LSTMs to reduce translation errors by 60%.[13][14][15]

Apple announced in its Worldwide Developers Conference that it would start using the LSTM for quicktype[16][17][18] in the iPhone and for Siri.[19][20]

Amazon released Polly, which generates the voices behind Alexa, using a bidirectional LSTM for the text-to-speech technology.[21]

2017: Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks.[22]

Researchers from Michigan State University, IBM Research, and Cornell University published a study in the Knowledge Discovery and Data Mining (KDD) conference.[23][24][25] Their study describes a novel neural network that performs better on certain data sets than the widely used long short-term memory neural network.

Microsoft reported reaching 95.1% recognition accuracy on the Switchboard corpus, incorporating a vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory".[26]

2019: Researchers from the University of Waterloo proposed a related RNN architecture which represents continuous windows of time. It was derived using the Legendre polynomials and outperforms the LSTM on some memory-related benchmarks.[27]

An LSTM model climbed to third place on the in Large Text Compression Benchmark.[28][29]

References

[edit]
  1. ^ Sepp Hochreiter; Jürgen Schmidhuber (1997). "Long short-term memory". Neural Computation. 9 (8): 1735–1780. doi:10.1162/neco.1997.9.8.1735. PMID 9377276. S2CID 1915014.
  2. ^ a b c Cite error: The named reference ASearchSpaceOdyssey was invoked but never defined (see the help page).
  3. ^ Gers, F.A. (1999). "Learning to forget: Continual prediction with LSTM". 9th International Conference on Artificial Neural Networks: ICANN '99. Vol. 1999. pp. 850–855. doi:10.1049/cp:19991218. ISBN 0-85296-721-7.
  4. ^ Cite error: The named reference lstm2000 was invoked but never defined (see the help page).
  5. ^ Graves, A.; Liwicki, M.; Fernández, S.; Bertolami, R.; Bunke, H.; Schmidhuber, J. (May 2009). "A Novel Connectionist System for Unconstrained Handwriting Recognition". IEEE Transactions on Pattern Analysis and Machine Intelligence. 31 (5): 855–868. CiteSeerX 10.1.1.139.4502. doi:10.1109/tpami.2008.137. ISSN 0162-8828. PMID 19299860. S2CID 14635907.
  6. ^ Märgner, Volker; Abed, Haikal El (July 2009). "ICDAR 2009 Arabic Handwriting Recognition Competition". 2009 10th International Conference on Document Analysis and Recognition: 1383–1387. doi:10.1109/ICDAR.2009.256. ISBN 978-1-4244-4500-4. S2CID 52851337.
  7. ^ Graves, Alex; Mohamed, Abdel-rahman; Hinton, Geoffrey (2013-03-22). "Speech Recognition with Deep Recurrent Neural Networks". arXiv:1303.5778 [cs.NE].
  8. ^ Cho, Kyunghyun; van Merrienboer, Bart; Gulcehre, Caglar; Bahdanau, Dzmitry; Bougares, Fethi; Schwenk, Holger; Bengio, Yoshua (2014). "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation". arXiv:1406.1078 [cs.CL].
  9. ^ Beaufays, Françoise (August 11, 2015). "The neural networks behind Google Voice transcription". Research Blog. Retrieved 2017-06-27.
  10. ^ Sak, Haşim; Senior, Andrew; Rao, Kanishka; Beaufays, Françoise; Schalkwyk, Johan (September 24, 2015). "Google voice search: faster and more accurate". Research Blog. Retrieved 2017-06-27.
  11. ^ "Neon prescription... or rather, New transcription for Google Voice". Official Google Blog. Retrieved 2020-04-25.
  12. ^ Khaitan, Pranav (May 18, 2016). "Chat Smarter with Allo". Research Blog. Retrieved 2017-06-27.
  13. ^ Wu, Yonghui; Schuster, Mike; Chen, Zhifeng; Le, Quoc V.; Norouzi, Mohammad; Macherey, Wolfgang; Krikun, Maxim; Cao, Yuan; Gao, Qin (2016-09-26). "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation". arXiv:1609.08144 [cs.CL].
  14. ^ Metz, Cade (September 27, 2016). "An Infusion of AI Makes Google Translate More Powerful Than Ever | WIRED". Wired. Retrieved 2017-06-27.
  15. ^ "A Neural Network for Machine Translation, at Production Scale". Google AI Blog. Retrieved 2020-04-25.
  16. ^ Efrati, Amir (June 13, 2016). "Apple's Machines Can Learn Too". The Information. Retrieved 2017-06-27.
  17. ^ Ranger, Steve (June 14, 2016). "iPhone, AI and big data: Here's how Apple plans to protect your privacy | ZDNet". ZDNet. Retrieved 2017-06-27.
  18. ^ "Can Global Semantic Context Improve Neural Language Models? - Apple". Apple Machine Learning Journal. Retrieved 2020-04-30.
  19. ^ Smith, Chris (2016-06-13). "iOS 10: Siri now works in third-party apps, comes with extra AI features". BGR. Retrieved 2017-06-27.
  20. ^ Capes, Tim; Coles, Paul; Conkie, Alistair; Golipour, Ladan; Hadjitarkhani, Abie; Hu, Qiong; Huddleston, Nancy; Hunt, Melvyn; Li, Jiangchuan; Neeracher, Matthias; Prahallad, Kishore (2017-08-20). "Siri On-Device Deep Learning-Guided Unit Selection Text-to-Speech System". Interspeech 2017. ISCA: 4011–4015. doi:10.21437/Interspeech.2017-1798.
  21. ^ Vogels, Werner (30 November 2016). "Bringing the Magic of Amazon AI and Alexa to Apps on AWS. – All Things Distributed". www.allthingsdistributed.com. Retrieved 2017-06-27.
  22. ^ Ong, Thuy (4 August 2017). "Facebook's translations are now powered completely by AI". www.allthingsdistributed.com. Retrieved 2019-02-15.
  23. ^ "Patient Subtyping via Time-Aware LSTM Networks" (PDF). msu.edu. Retrieved 21 Nov 2018.
  24. ^ "Patient Subtyping via Time-Aware LSTM Networks". Kdd.org. Retrieved 24 May 2018.
  25. ^ "SIGKDD". Kdd.org. Retrieved 24 May 2018.
  26. ^ Haridy, Rich (August 21, 2017). "Microsoft's speech recognition system is now as good as a human". newatlas.com. Retrieved 2017-08-27.
  27. ^ Voelker, Aaron R.; Kajić, Ivana; Eliasmith, Chris (2019). Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks (PDF). Advances in Neural Information Processing Systems.
  28. ^ "The Large Text Compression Benchmark". Retrieved 2017-01-13.
  29. ^ Fabrice Bellard (2019), "Lossless Data Compression with Neural Networks"
Retrieved from "https://en.wikipedia.org/w/index.php?title=User:Themumblingprophet/sandbox&oldid=1118565405"