Residual neural network

Canonical form of a residual neural network. A layer ℓ  − 1 is skipped over activation from ℓ − 2.

A residual neural network (ResNet) is an artificial neural network (ANN). It is a gateless or open-gated variant of the HighwayNet, the first working very deep feedforward neural network with hundreds of layers, much deeper than previous neural networks. Skip connections or shortcuts are used to jump over some layers (HighwayNets may also learn the skip weights themselves through an additional weight matrix for their gates). Typical ResNet models are implemented with double- or triple- layer skips that contain nonlinearities (ReLU) and batch normalization in between. Models with several parallel skips are referred to as DenseNets. In the context of residual neural networks, a non-residual network may be described as a plain network.

Like in the case of Long Short-Term Memory recurrent neural networks there are two main reasons to add skip connections: to avoid the problem of vanishing gradients, thus leading to easier optimization of neural networks, where the gating mechanisms facilitate information flow across many layers ("information highways"), or to mitigate the Degradation (accuracy saturation) problem; where adding more layers to a suitably deep model leads to higher training error. During training, the weights adapt to mute the upstream layer, and amplify the previously-skipped layer. In the simplest case, only the weights for the adjacent layer's connection are adapted, with no explicit weights for the upstream layer. This works best when a single nonlinear layer is stepped over, or when the intermediate layers are all linear. If not, then an explicit weight matrix should be learned for the skipped connection (a HighwayNet should be used).

Skipping effectively simplifies the network, using fewer layers in the initial training stages. This speeds learning by reducing the impact of vanishing gradients, as there are fewer layers to propagate through. The network then gradually restores the skipped layers as it learns the feature space. Towards the end of training, when all layers are expanded, it stays closer to the manifold and thus learns faster. A neural network without residual parts explores more of the feature space. This makes it more vulnerable to perturbations that cause it to leave the manifold, and necessitates extra training data to recover.

A residual neural network was used to win the ImageNet 2015 competition, and has become the most cited neural network of the 21st century.

Forward propagation

Given a weight matrix ${\textstyle W^{\ell -1,\ell }}$ for connection weights from layer ${\textstyle \ell -1}$ to ${\textstyle \ell }$, and a weight matrix ${\textstyle W^{\ell -2,\ell }}$ for connection weights from layer ${\textstyle \ell -2}$ to ${\textstyle \ell }$, then the forward propagation through the activation function would be (aka HighwayNets)

${\begin{aligned}a^{\ell }&:=\mathbf {g} (W^{\ell -1,\ell }\cdot a^{\ell -1}+b^{\ell }+W^{\ell -2,\ell }\cdot a^{\ell -2})\\&:=\mathbf {g} (Z^{\ell }+W^{\ell -2,\ell }\cdot a^{\ell -2})\end{aligned}}$

where

${\textstyle a^{\ell }}$ the activations (outputs) of neurons in layer ${\textstyle \ell }$,

${\textstyle \mathbf {g} }$ the activation function for layer ${\textstyle \ell }$,

${\textstyle W^{\ell -1,\ell }}$ the weight matrix for neurons between layer ${\textstyle \ell -1}$ and ${\textstyle \ell }$, and

${\textstyle Z^{\ell }=W^{\ell -1,\ell }\cdot a^{\ell -1}+b^{\ell }}$

If the number of vertices on layer ${\textstyle \ell -2}$ equals the number of vertices on layer ${\textstyle \ell }$ and if ${\textstyle W^{\ell -2,\ell }}$ is the identity matrix, then forward propagation through the activation function simplifies to $a^{\ell }:=\mathbf {g} (Z^{\ell }+a^{\ell -2}).$ In this case, the connection between layers ${\textstyle \ell -2}$ and ${\textstyle \ell }$ is called an identity block.

In the cerebral cortex such forward skips are done for several layers. Usually all forward skips start from the same layer, and successively connect to later layers. In the general case this will be expressed as (aka DenseNets)

$a^{\ell }:=\mathbf {g} \left(Z^{\ell }+\sum _{k=2}^{K}W^{\ell -k,\ell }\cdot a^{\ell -k}\right)$.

Backward propagation

During backpropagation learning for the normal path

$\Delta w^{\ell -1,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -1,\ell }}}=-\eta a^{\ell -1}\cdot \delta ^{\ell }$

and for the skip paths (nearly identical)

$\Delta w^{\ell -2,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -2,\ell }}}=-\eta a^{\ell -2}\cdot \delta ^{\ell }$.

In both cases

${\textstyle \eta }$ a learning rate (${\textstyle \eta <0)}$,

${\textstyle \delta ^{\ell }}$ the error signal of neurons at layer ${\textstyle \ell }$, and

${\textstyle a_{i}^{\ell }}$ the activation of neurons at layer ${\textstyle \ell }$.

If the skip path has fixed weights (e.g. the identity matrix, as above), then they are not updated. If they can be updated, the rule is an ordinary backpropagation update rule.

In the general case there can be ${\textstyle K}$ skip path weight matrices, thus

$\Delta w^{\ell -k,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -k,\ell }}}=-\eta a^{\ell -k}\cdot \delta ^{\ell }$

As the learning rules are similar, the weight matrices can be merged and learned in the same step.