Neural Style Transfer
Neural Style Transfer (NST) refers to a class of software algorithms that manipulate digital images, or videos, to adopt the appearance or visual style of another image. NST algorithms are characterized by their use of deep neural networks in order to perform the image transformation. Common uses for NST are the creation of artificial artwork from photographs, for example by transferring the appearance of famous paintings to user supplied photographs. Several notable mobile apps use NST techniques for this purpose, including DeepArt and Prisma.
NST is an example of image stylization, a problem studied for over two decades within the field of non-photorealistic rendering. Prior to NST, the transfer of image style was performed using machine learning techniques based on image analogy. Given a training pair of images–a photo and an artwork depicting that photo–a transformation could be learned and then applied to create a new artwork from a new photo, by analogy. The drawback of this method is that such a training pair rarely exists in practice. For example original source material (photos) are rarely available for famous artworks.
NST requires no such pairing; only a single example of artwork is needed for the algorithm to transfer its style.
NST was first published in the paper "A Neural Algorithm of Artistic Style" by Gatys et al, originally released to ArXiv 2015, and subsequently accepted by the peer-reviewed Computer Vision and Pattern Recognition (CVPR) in 2016.
The core innovation of NST is the use of deep learning to disentangle the representation of the content (structure) of an image, from the appearance (style) in which it is depicted. The original paper used a convolutional neural network (CNN) VGG-19 architecture that has been pre-trained to perform object recognition using the ImageNet dataset.
The process of NST assumes an input image and an example style image .
The image is fed through the CNN, and network activations are sampled at a late convolution layer of the VGG-19 architecture. Let be the resulting output sample, called the 'content' of the input .
The style image is then fed through the same CNN, and network activations are sampled at the early to middle layers of the CNN. These activations are encoded into a Gramian matrix representation, call it to denote the 'style' of .
The goal of NST is to synthesize an output image that exhibits the content of applied with the style of , i.e. and .
An iterative optimization (usually gradient descent) then gradually updates to minimize the loss function error:
where is the L2 distance. The constant controls the level of the stylization effect.
Image is initially approximated by adding a small amount of white noise to input image and feeding it through the CNN. Then we successively backpropagate this loss through the network with the CNN weights fixed in order to update the pixels of . After several thousand epochs of training, an (hopefully) emerges that matches the style of and the content of .
Algorithms are typically implemented for GPUs, so that training takes a few minutes.
NST has also been extended to videos.
Subsequent work improved the speed of NST for images.
In a paper by Fei-Fei Li et al. adopted a different regularized loss metric and accelerated method for training to produce results in real time (three times faster than Gatys). Their idea was to use not the pixel-based loss defined above but rather a 'perceptual loss' measuring the differences between higher level layers within the CNN. They used a symmetric encoder-decoder CNN. Training uses a similar loss function to the basic NST method but also regularizes the output for smoothness using a total variation (TV) loss. Once trained, the network may be used to transform an image into the style used during training, using a single feed-forward pass of the network. However the network is restricted to the single style in which it has been trained.
Most recently, feature transform based NST methods have been explored for fast stylization that are not coupled to single specific style and enable user-controllable blending of styles, for example the Whitening and Coloring Transform (WCT).
- "Image Analogies" (PDF). Mrl.nyu.edu. 2001. Retrieved 13 February 2019.
- Gatys, Leon A.; Ecker, Alexander S.; Bethge, Matthias (26 August 2015). "A Neural Algorithm of Artistic Style". arXiv:1508.06576 [cs.CV].
- Bethge, Matthias; Ecker, Alexander S.; Gatys, Leon A. (2016). "Image Style Transfer Using Convolutional Neural Networks". Cv-foundation.org. pp. 2414–2423. Retrieved 13 February 2019.
- "Very Deep CNNS for Large-Scale Visual Recognition". Robots.ox.ac.uk. 2014. Retrieved 13 February 2019.
- Johnson, Justin; Alahi, Alexandre; Li, Fei-Fei (2016). "Perceptual Losses for Real-Time Style Transfer and Super-Resolution". arXiv:1603.08155 [cs.CV].
- Chen, Dongdong; Liao, Jing; Yuan, Lu; Yu, Nenghai; Hua, Gang (2017). "Coherent Online Video Style Transfer". arXiv:1703.09211 [cs.CV].
- Li, Yijun; Fang, Chen; Yang, Jimei; Wang, Zhaowen; Lu, Xin; Yang, Ming-Hsuan (2017). "Universal Style Transfer via Feature Transforms". arXiv:1705.08086 [cs.CV].