Long short-term memory

Long short-term memory (LSTM) units (or blocks) are a building unit for layers of a recurrent neural network (RNN). An RNN composed of LSTM units is often called an LSTM network. A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell is responsible for "remembering" values over arbitrary time intervals; hence the word "memory" in LSTM. Each of the three gates can be thought of as a "conventional" artificial neuron, as in a multi-layer (or feedforward) neural network: that is, they compute an activation (using an activation function) of a weighted sum. Intuitively, they can be thought as regulators of the flow of values that goes through the connections of the LSTM; hence the denotation "gate". There are connections between these gates and the cell.

The expression long short-term refers to the fact that LSTM is a model for the short-term memory which can last for a long period of time. An LSTM is well-suited to classify, process and predict time series given time lags of unknown size and duration between important events. LSTMs were developed to deal with the exploding and vanishing gradient problem when training traditional RNNs. Relative insensitivity to gap length gives an advantage to LSTM over alternative RNNs, hidden Markov models and other sequence learning methods in numerous applications .

History

LSTM was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber and improved in 2000 by Felix Gers' team.

Among other successes, LSTM achieved record results in natural language text compression, unsegmented connected handwriting recognition and won the ICDAR handwriting competition (2009). LSTM networks were a major component of a network that achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset (2013).

As of 2016, major technology companies including Google, Apple, and Microsoft were using LSTM as fundamental components in new products. For example, Google used LSTM for speech recognition on the smartphone, for the smart assistant Allo and for Google Translate. Apple uses LSTM for the "Quicktype" function on the iPhone and for Siri. Amazon uses LSTM for Amazon Alexa.

In 2017 Microsoft reported reaching 95.1% recognition accuracy on the Switchboard corpus, incorporating a vocabulary of 165,000 words. The approach used "dialog session-based long-short-term memory".

Architectures

There are several architectures of LSTM units. A common architecture is composed of a memory cell, an input gate, an output gate and a forget gate.

An LSTM (memory) cell stores a value (or state), for either long or short time periods. This is achieved by using an identity (or no) activation function for the memory cell. In this way, when an LSTM network (that is an RNN composed of LSTM units) is trained with backpropagation through time, the gradient does not tend to vanish.

The LSTM gates compute an activation, often using the logistic function. Intuitively, the input gate controls the extent to which a new value flows into the cell, the forget gate controls the extent to which a value remains in the cell and the output gate controls the extent to which the value in the cell is used to compute the output activation of the LSTM unit.

There are connections into and out of these gates. A few connections are recurrent. The weights of these connections, which need to be learned during training, of an LSTM unit are used to direct the operation of the gates. Each of the gates has its own parameters, that is weights and biases, from possibly other units outside the LSTM unit.

Variants

In the equations below, each variable in lowercase italics represents a vector. Matrices $W_{q}$ and $U_{q}$ collect respectively the weights of the input and recurrent connections, where $q$ can either be the input gate $i$, output gate $o$, the forget gate $f$ or the memory cell $c$, depending on the activation being calculated.

LSTM with a forget gate

Compact form of the equations for the forward pass of a LSTM unit with a forget gate.

${\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}$

where the initial values are $c_{0}=0$ and $h_{0}=0$ and the operator $\circ$ denotes the Hadamard product (entry-wise product). The subscripts $_{t}$ refer to the time step.

Variables

• $x_{t}\in R^{d}$: input vector to the LSTM unit
• $f_{t}\in R^{h}$: forget gate's activation vector
• $i_{t}\in R^{h}$: input gate's activation vector
• $o_{t}\in R^{h}$: output gate's activation vector
• $h_{t}\in R^{h}$: output vector of the LSTM unit
• $c_{t}\in R^{h}$: cell state vector
• $W\in R^{h\times d}$, $U\in R^{h\times h}$ and $b\in R^{h}$: weight matrices and bias vector parameters which need to be learned during training

Activation functions

• $\sigma _{g}$: sigmoid function.
• $\sigma _{c}$: hyperbolic tangent function.
• $\sigma _{h}$: hyperbolic tangent function or, as the peephole LSTM paper suggests, $\sigma _{h}(x)=x$.

Peephole LSTM

A peephole LSTM unit with input (i.e. $i$), output (i.e. $o$), and forget (i.e. $f$) gates. Each of these gates can be thought as a "standard" neuron in a feed-forward (or multi-layer) neural network: that is, they compute an activation (using an activation function) of a weighted sum. $i_{t},o_{t}$ and $f_{t}$ represent the activations of respectively the input, output and forget gates, at time step $t$. The 3 exit arrows from the memory cell $c$ to the 3 gates $i,o$ and $f$ represent the peephole connections. These peephole connections actually denote the contributions of the activation of the memory cell $c$ at time step $t-1$, i.e. the contribution of $c_{t-1}$ (and not $c_{t}$, as the picture may suggest). In other words, the gates $i,o$ and $f$ calculate their activations at time step $t$ (i.e., respectively, $i_{t},o_{t}$ and $f_{t}$) also considering the activation of the memory cell $c$ at time step $t-1$, i.e. $c_{t-1}$. The single left-to-right arrow exiting the memory cell is not a peephole connection and denotes $c_{t}$. The little circles containing a $\times$ symbol represent an element-wise multiplication between its inputs. The big circles containing an S-like curve represent the application of a differentiable function (like the sigmoid function) to a weighted sum. There are many other kinds of LSTMs as well.

The figure on the right is a graphical representation of a LSTM unit with peephole connections (i.e. a peephole LSTM). Peephole connections allow the gates to access the constant error carousel (CEC), whose activation is the cell state. $h_{t-1}$ is not used, $c_{t-1}$ is used instead in most places.

${\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}$

Convolutional LSTM

Convolutional LSTM. $*$ denotes the convolution operator.

${\begin{aligned}f_{t}&=\sigma _{g}(W_{f}*x_{t}+U_{f}*h_{t-1}+V_{f}\circ c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}*x_{t}+U_{i}*h_{t-1}+V_{i}\circ c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}*x_{t}+U_{o}*h_{t-1}+V_{o}\circ c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}*x_{t}+U_{c}*h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}$

Training

To minimize LSTM's total error on a set of training sequences, iterative gradient descent such as backpropagation through time can be used to change each weight in proportion to its derivative with respect to the error. A problem with using gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events. This is due to $\lim _{n\to \infty }W^{n}=0$ if the spectral radius of $W$ is smaller than 1. With LSTM units, however, when error values are back-propagated from the output, the error remains in the unit's memory. This "error carousel" continuously feeds error back to each of the gates until they learn to cut off the value. Thus, regular backpropagation is effective at training an LSTM unit to remember values for long durations.

LSTM can also be trained by a combination of artificial evolution for weights to the hidden units, and pseudo-inverse or support vector machines for weights to the output units. In reinforcement learning applications LSTM can be trained by policy gradient methods, evolution strategies or genetic algorithms.

CTC score function

Many applications use stacks of LSTM RNNs and train them by connectionist temporal classification (CTC) to find an RNN weight matrix that maximizes the probability of the label sequences in a training set, given the corresponding input sequences. CTC achieves both alignment and recognition.

Applications

Applications of LSTM include:

• Robot control
• Time series prediction
• Speech recognition
• Rhythm learning
• Music composition
• Grammar learning
• Handwriting recognition
• Human action recognition
• Protein Homology Detection
• Predicting subcellular localization of proteins
• Several prediction tasks in the area of business process management
• Prediction in medical care pathways

LSTM has Turing completeness in the sense that given enough network units it can compute any result that a conventional computer can compute, provided it has the proper weight matrix, which may be viewed as its program.

See also

• Differentiable neural computer
• Gated recurrent unit
• Long-term potentiation
• Prefrontal cortex basal ganglia working memory
• Recurrent neural network
• Time series

External links

• Recurrent Neural Networks with over 30 LSTM papers by Jürgen Schmidhuber's group at IDSIA
• Gers PhD thesis on LSTM networks.
• Fraud detection paper with two chapters devoted to explaining recurrent neural networks, especially LSTM.
• Paper on a high-performing extension of LSTM that has been simplified to a single node type and can train arbitrary architectures.
• Tutorial: How to implement LSTM in Python with Theano
• A Beginner’s Guide to Recurrent Networks and LSTMs