# StyleGAN

An image generated by a StyleGAN that looks deceptively like a portrait of a young woman. This image was generated by an artificial intelligence based on an analysis of portraits.

StyleGAN is a generative adversarial network (GAN) introduced by Nvidia researchers in December 2018,[1] and made source available in February 2019.[2][3]

StyleGAN depends on Nvidia's CUDA software, GPUs, and Google's TensorFlow,[4] or Meta AI's PyTorch, which supersedes TensorFlow as the official implementation library in later StyleGAN versions.[5] The second version of StyleGAN, called StyleGAN2, was published on February 5, 2020. It removes some of the characteristic artifacts and improves the image quality.[6][7] On October 12, 2021, Nvidia released StyleGAN3, described as an "alias-free" version.[8]

## History

In December 2018, Nvidia researchers distributed a preprint with accompanying software introducing StyleGAN, a GAN for producing an unlimited number of (often convincing) portraits of fake human faces. StyleGAN was able to run on Nvidia's commodity GPU processors.

In February 2019, Uber engineer Phillip Wang used the software to create This Person Does Not Exist, which displayed a new face on each web page reload.[9][10] Wang himself has expressed amazement, given that humans are evolved to specifically understand human faces, that nevertheless StyleGAN can competitively "pick apart all the relevant features (of human faces) and recompose them in a way that's coherent."[11]

In September 2019, a website called Generated Photos published 100,000 images as a collection of stock photos.[12] The collection was made using a private dataset shot in a controlled environment with similar light and angles.[13]

Similarly, two faculty at the University of Washington's Information School used StyleGAN to create Which Face is Real?, which challenged visitors to differentiate between a fake and a real face side by side.[10] The faculty stated the intention was to "educate the public" about the existence of this technology so they could be wary of it, "just like eventually most people were made aware that you can Photoshop an image".[14]

The second version of StyleGAN, called StyleGAN2, was published on February 5, 2020. It removes some of the characteristic artifacts and improves the image quality.[6][7]

In 2021, a third version was released, improving consistency between fine and coarse details in the generator. Dubbed "alias-free", this version was implemented with pytorch.[15]

## Architecture

### Progressive GAN

Progressive GAN[16] is a method for training GAN for large-scale image generation stably, by growing a GAN generator from small to large scale in a pyramidal fashion. Like SinGAN, it decomposes the generator as${\displaystyle G=G_{1}\circ G_{2}\circ \cdots \circ G_{N}}$, and the discriminator as ${\displaystyle D=D_{1}\circ D_{2}\circ \cdots \circ D_{N}}$.

During training, at first only ${\displaystyle G_{N},D_{N}}$ are used in a GAN game to generate 4x4 images. Then ${\displaystyle G_{N-1},D_{N-1}}$ are added to reach the second stage of GAN game, to generate 8x8 images, and so on, until we reach a GAN game to generate 1024x1024 images.

To avoid shock between stages of the GAN game, each new layer is "blended in" (Figure 2 of the paper[16]). For example, this is how the second stage GAN game starts:

• Just before, the GAN game consists of the pair ${\displaystyle G_{N},D_{N}}$ generating and discriminating 4x4 images.
• Just after, the GAN game consists of the pair ${\displaystyle ((1-\alpha )+\alpha \cdot G_{N-1})\circ u\circ G_{N},D_{N}\circ d\circ ((1-\alpha )+\alpha \cdot D_{N-1})}$ generating and discriminating 8x8 images. Here, the functions ${\displaystyle u,d}$ are image up- and down-sampling functions, and ${\displaystyle \alpha }$ is a blend-in factor (much like an alpha in image composing) that smoothly glides from 0 to 1.

### StyleGAN-1

The main architecture of StyleGAN-1 and StyleGAN-2

StyleGAN-1 is designed as a combination of Progressive GAN with neural style transfer.[17]

The key architectural choice of StyleGAN-1 is a progressive growth mechanism, similar to Progressive GAN. Each generated image starts as a constant ${\displaystyle 4\times 4\times 512}$ array, and repeatedly passed through style blocks. Each style block applies a "style latent vector" via affine transform ("adaptive instance normalization"), similar to how neural style transfer uses Gramian matrix. It then adds noise, and normalize (subtract the mean, then divide by the variance).

At training time, usually only one style latent vector is used per image generated, but sometimes two ("mixing regularization") in order to encourage each style block to independently perform its stylization without expecting help from other style blocks (since they might receive an entirely different style latent vector).

After training, multiple style latent vectors can be fed into each style block. Those fed to the lower layers control the large-scale styles, and those fed to the higher layers control the fine-detail styles.

Style-mixing between two images ${\displaystyle x,x'}$ can be performed as well. First, run a gradient descent to find ${\displaystyle z,z'}$ such that ${\displaystyle G(z)\approx x,G(z')\approx x'}$. This is called "projecting an image back to style latent space". Then, ${\displaystyle z}$ can be fed to the lower style blocks, and ${\displaystyle z'}$ to the higher style blocks, to generate a composite image that has the large-scale style of ${\displaystyle x}$, and the fine-detail style of ${\displaystyle x'}$. Multiple images can also be composed this way.

### StyleGAN-2

StyleGAN-2 improves upon StyleGAN-1, by using the style latent vector to transform the convolution layer's weights instead, thus solving the "blob" problem.[18]

This was updated by the StyleGAN-2-ADA ("ADA" stands for "adaptive"),[19] which uses invertible data augmentation. It also tunes the amount of data augmentation applied by starting at zero, and gradually increasing it until an "overfitting heuristic" reaches a target level, thus the name "adaptive".

### StyleGAN-3

StyleGAN-3[20] improves upon StyleGAN-2 by solving the "texture sticking" problem, which can be seen in the official videos.[21] They analyzed the problem by the Nyquist–Shannon sampling theorem, and argued that the layers in the generator learned to exploit the high-frequency signal in the pixels they operate upon.

To solve this, they proposed imposing strict lowpass filters between each generator's layers, so that the generator is forced to operate on the pixels in a way faithful to the continuous signals they represent, rather than operate on them as merely discrete signals. They further imposed rotational and translational invariance by using more signal filters. The resulting StyleGAN-3 is able to solve the texture sticking problem, as well as generating images that rotate and translate smoothly.

## Related issues

The technology has drawn comparison with deep fakes[22] and its potential usage for sinister purposes has been debated.[by whom?][citation needed]

In December 2019, Facebook took down a network of accounts with false identities, and mentioned that some of them had used profile pictures created with artificial intelligence.[23]

## References

1. ^ "GAN 2.0: NVIDIA's Hyperrealistic Face Generator". SyncedReview.com. December 14, 2018. Retrieved October 3, 2019.
2. ^ "NVIDIA Open-Sources Hyper-Realistic Face Generator StyleGAN". Medium.com. February 9, 2019. Retrieved October 3, 2019.
3. ^ Beschizza, Rob (February 15, 2019). "This Person Does Not Exist". Boing-Boing. Retrieved February 16, 2019.
4. ^ Larabel, Michael (February 10, 2019). "NVIDIA Opens Up The Code To StyleGAN - Create Your Own AI Family Portraits". Phoronix.com. Retrieved October 3, 2019.
5. ^ "Looking for the PyTorch version? - Stylegan2". github.com. October 28, 2021. Retrieved August 5, 2022.
6. ^ a b "Synthesizing High-Resolution Images with StyleGAN2 – NVIDIA Developer News Center". news.developer.nvidia.com. Retrieved August 11, 2020.
7. ^ a b NVlabs/stylegan2, NVIDIA Research Projects, August 11, 2020, retrieved August 11, 2020
8. ^ Kakkar, Shobha (October 13, 2021). "NVIDIA AI Releases StyleGAN3: Alias-Free Generative Adversarial Networks". MarkTechPost. Retrieved October 14, 2021.
9. ^ msmash, n/a (February 14, 2019). "'This Person Does Not Exist' Website Uses AI To Create Realistic Yet Horrifying Faces". Slashdot. Retrieved February 16, 2019.
10. ^ a b Fleishman, Glenn (April 30, 2019). "How to spot the realistic fake people creeping into your timelines". Fast Company. Retrieved June 7, 2020.
11. ^ Bishop, Katie (February 7, 2020). "AI in the adult industry: porn may soon feature people who don't exist". The Guardian. Retrieved June 8, 2020.
12. ^ Porter, Jon (September 20, 2019). "100,000 free AI-generated headshots put stock photo companies on notice". The Verge. Retrieved August 4, 2020.
13. ^ Timmins, Jane Wakefield and Beth (February 29, 2020). "Could deepfakes be used to train office workers?". BBC News. Retrieved August 4, 2020.
14. ^ Vincent, James (March 3, 2019). "Can you tell the difference between a real face and an AI-generated fake?". The Verge. Retrieved June 8, 2020.
15. ^ NVlabs/stylegan3, NVIDIA Research Projects, October 11, 2021
16. ^ a b Karras, Tero; Aila, Timo; Laine, Samuli; Lehtinen, Jaakko (October 1, 2017). "Progressive Growing of GANs for Improved Quality, Stability, and Variation". {{cite journal}}: Cite journal requires |journal= (help)
17. ^ Karras, Tero; Laine, Samuli; Aila, Timo (June 2019). "A Style-Based Generator Architecture for Generative Adversarial Networks". 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. doi:10.1109/cvpr.2019.00453.
18. ^ Karras, Tero; Laine, Samuli; Aittala, Miika; Hellsten, Janne; Lehtinen, Jaakko; Aila, Timo (June 2020). "Analyzing and Improving the Image Quality of StyleGAN". 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. doi:10.1109/cvpr42600.2020.00813.
19. ^ Tero, Karras; Miika, Aittala; Janne, Hellsten; Samuli, Laine; Jaakko, Lehtinen; Timo, Aila (2020). "Training Generative Adversarial Networks with Limited Data". Advances in Neural Information Processing Systems. 33.
20. ^ Timo, Karras, Tero Aittala, Miika Laine, Samuli Härkönen, Erik Hellsten, Janne Lehtinen, Jaakko Aila (June 23, 2021). Alias-Free Generative Adversarial Networks. OCLC 1269560084.
21. ^ Karras 1, Tero; Aittala 1, Miika; Laine 1, Samuli; Erik Härkönen 2, 1; Hellsten 1, Janne; Jaakko Lehtinen 1, 2; Aila 1, Timo; Nvidia, 1; University, 2 Aalto. "Alias-Free Generative Adversarial Networks (StyleGAN3)". nvlabs.github.io. Retrieved July 16, 2022.{{cite web}}: CS1 maint: numeric names: authors list (link)
22. ^ Paez, Danny (February 13, 2019). "This Person Does Not Exist Is the Best One-Off Website of 2019". Inverse. Retrieved February 16, 2019.
23. ^ "Facebook's latest takedown has a twist -- AI-generated profile pictures". ABC News. Retrieved August 4, 2020.