Long short-term memory

A peephole LSTM block with input, output, and forget gates. The exit arrows from the c_t node actually denote exit arrows from c_t-1 except for the single left-to-right arrow. There are many other kinds of LSTMs as well.

Long short-term memory (LSTM) is a recurrent neural network (RNN) architecture (an artificial neural network) proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber. Like most RNNs, a LSTM network is universal in the sense that given enough network units it can compute anything a conventional computer can compute, provided it has the proper weight matrix, which may be viewed as its program. Unlike traditional RNNs, a LSTM network is well-suited to learn from experience to classify, process and predict time series when there are time lags of unknown size and bound between important events. Relative insensitivity to gap length gives an advantage to LSTM over alternative RNNs and hidden Markov models and other sequence learning methods in numerous applications. Among other successes, LSTM achieved the best known results in natural language text compression, unsegmented connected handwriting recognition, and in 2009 won the ICDAR handwriting competition. LSTM networks have also been used for automatic speech recognition, and were a major component of a network that in 2013 achieved a record 17.7% phoneme error rate on the classic TIMIT natural speech dataset. As of 2016, major technology companies including Google, Apple, Microsoft, and Baidu are using LSTM networks as fundamental components in new products.

Architecture

A LSTM network is an artificial neural network that contains LSTM units instead of, or in addition to, other network units. A LSTM unit is a recurrent network unit that excels at remembering values for either long or short durations of time. The key to this ability is that it uses no activation function within its recurrent components. Thus, the stored value is not iteratively squashed over time, and the gradient or blame term does not tend to vanish when Backpropagation through time is applied to train it.

LSTM units are often implemented in "blocks" containing several LSTM units. This design is typical with "deep" multi-layered neural networks, and facilitates implementations with parallel hardware. In the equations below, each variable in lowercase italics represents a vector with a size equal to the number of LSTM units in the block.

LSTM blocks contain three or four "gates" that they use to control the flow of information into or out of their memory. These gates are implemented using the logistic function to compute a value between 0 and 1. Multiplication is applied with this value to partially allow or deny information to flow into or out of the memory. For example, an "input gate" controls the extent to which a new value flows into the memory. A "forget gate" controls the extent to which a value remains in memory. And, an "output gate" controls the extent to which the value in memory is used to compute the output activation of the block. (In some implementations, the input gate and forget gate are combined into a single gate. The intuition for combining them is that the time to forget is when a new value worth remembering becomes available.)

The only weights in a LSTM block ($W$ and $U$) are used to direct the operation of the gates. These weights occur between the values that feed into the block (including the input vector $x_{t}$, and the output from the previous time step $h_{t-1}$) and each of the gates. Thus, the LSTM block determines how to maintain its memory as a function of those values, and training its weights causes the LSTM block to learn the function that minimizes loss. LSTM blocks are usually trained with Backpropagation through time.

Traditional LSTM

Traditional LSTM with forget gates. $c_{0}=0$ and $h_{0}=0$. $\circ$ denotes the Hadamard product (entry-wise product).

${\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}$

Variables

• $x_{t}$: input vector
• $h_{t}$: output vector
• $c_{t}$: cell state vector
• $W$, $U$ and $b$: parameter matrices and vector (W is for weight, U is for update?, and b is for bias?)
• $f_{t}$, $i_{t}$ and $o_{t}$: gate vectors
  • $f_{t}$: Forget gate vector. Weight of remembering old information.
  • $i_{t}$: Input gate vector. Weight of acquiring new information.
  • $o_{t}$: Output gate vector. Output candidate.

Activation functions

• $\sigma _{g}$: The original is a sigmoid function.
• $\sigma _{c}$: The original is a hyperbolic tangent.
• $\sigma _{h}$: The original is a hyperbolic tangent, but the peephole LSTM paper suggests $\sigma _{h}(x)=x$.

Peephole LSTM

Peephole LSTM with forget gates. $h_{t-1}$ is not used, $c_{t-1}$ is used instead in most places.

${\begin{aligned}f_{t}&=\sigma _{g}(W_{f}x_{t}+U_{f}c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}x_{t}+U_{i}c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}x_{t}+U_{o}c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}x_{t}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}$

Convolutional LSTM

Convolutional LSTM. $*$ denotes the convolution operator.

${\begin{aligned}f_{t}&=\sigma _{g}(W_{f}*x_{t}+U_{f}*h_{t-1}+V_{f}\circ c_{t-1}+b_{f})\\i_{t}&=\sigma _{g}(W_{i}*x_{t}+U_{i}*h_{t-1}+V_{i}\circ c_{t-1}+b_{i})\\o_{t}&=\sigma _{g}(W_{o}*x_{t}+U_{o}*h_{t-1}+V_{o}\circ c_{t-1}+b_{o})\\c_{t}&=f_{t}\circ c_{t-1}+i_{t}\circ \sigma _{c}(W_{c}*x_{t}+U_{c}*h_{t-1}+b_{c})\\h_{t}&=o_{t}\circ \sigma _{h}(c_{t})\end{aligned}}$

Training

To minimize LSTM's total error on a set of training sequences, iterative gradient descent such as backpropagation through time can be used to change each weight in proportion to its derivative with respect to the error. A major problem with gradient descent for standard RNNs is that error gradients vanish exponentially quickly with the size of the time lag between important events, as first realized in 1991. With LSTM blocks, however, when error values are back-propagated from the output, the error becomes trapped in the memory portion of the block. This is referred to as an "error carousel", which continuously feeds error back to each of the gates until they become trained to cut off the value. Thus, regular backpropagation is effective at training a LSTM block to remember values for very long durations.

LSTM can also be trained by a combination of artificial evolution for weights to the hidden units, and pseudo-inverse or support vector machines for weights to the output units. In reinforcement learning applications LSTM can be trained by policy gradient methods, evolution strategies or genetic algorithms.

Applications

Applications of LSTM include:

• Robot control
• Time series prediction
• Speech recognition
• Rhythm learning
• Music composition
• Grammar learning
• Handwriting recognition
• Human action recognition
• Protein Homology Detection

See also

• Differentiable neural computer
• Artificial neural network
• Prefrontal Cortex Basal Ganglia Working Memory (PBWM)
• Recurrent neural network
• Gated recurrent unit
• Time series
• Long-term potentiation

External links

• Recurrent Neural Networks with over 30 LSTM papers by Jürgen Schmidhuber's group at IDSIA
• Gers PhD thesis on LSTM networks.
• Fraud detection paper with two chapters devoted to explaining recurrent neural networks, especially LSTM.
• Paper on a high-performing extension of LSTM that has been simplified to a single node type and can train arbitrary architectures.
• Tutorial: How to implement LSTM in python with theano