Generative adversarial network
|Machine learning and|
A generative adversarial network (GAN) is a class of machine learning systems invented by Ian Goodfellow and his colleagues in 2014. Two neural networks contest with each other in a game (in the sense of game theory, often but not always in the form of a zero-sum game). Given a training set, this technique learns to generate new data with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that look at least superficially authentic to human observers, having many realistic characteristics. Though originally proposed as a form of generative model for unsupervised learning, GANs have also proven useful for semi-supervised learning, fully supervised learning, and reinforcement learning. In a 2016 seminar, Yann LeCun described GANs as "the coolest idea in machine learning in the last twenty years".
The generative network generates candidates while the discriminative network evaluates them. The contest operates in terms of data distributions. Typically, the generative network learns to map from a latent space to a data distribution of interest, while the discriminative network distinguishes candidates produced by the generator from the true data distribution. The generative network's training objective is to increase the error rate of the discriminative network (i.e., "fool" the discriminator network by producing novel candidates that the discriminator thinks are not synthesized (are part of the true data distribution)).
A known dataset serves as the initial training data for the discriminator. Training it involves presenting it with samples from the training dataset, until it achieves acceptable accuracy. The generator trains based on whether it succeeds in fooling the discriminator. Typically the generator is seeded with randomized input that is sampled from a predefined latent space (e.g. a multivariate normal distribution). Thereafter, candidates synthesized by the generator are evaluated by the discriminator. Backpropagation is applied in both networks so that the generator produces better images, while the discriminator becomes more skilled at flagging synthetic images. The generator is typically a deconvolutional neural network, and the discriminator is a convolutional neural network.
GAN applications have increased rapidly.
Fashion, art and advertising
GANs can be used to create photos of imaginary fashion models, with no need to hire a model, photographer, makeup artist, or pay for a studio and transportation. GANs can be used to create fashion advertising campaigns including more diverse groups of models, which may increase intent to buy among people resembling the models. GANs can also be used to create portraits, landscapes and album covers.
GANs can improve astronomical images and simulate gravitational lensing for dark matter research. They were used in 2019 to successfully model the distribution of dark matter in a particular direction in space and to predict the gravitational lensing that will occur.
GANs have been proposed as a fast and accurate way of modeling high energy jet formation and modeling showers through calorimeters of high-energy physics experiments. GANs have also been trained to accurately approximate bottlenecks in computationally expensive simulations of particle physics experiments. Applications in the context of present and proposed CERN experiments have demonstrated the potential of these methods for accelerating simulation and/or improving simulation fidelity.
In 2018, GANs reached the video game modding community, as a method of up-scaling low-resolution 2D textures in old video games by recreating them in 4k or higher resolutions via image training, and then down-sampling them to fit the game's native resolution (with results resembling the supersampling method of anti-aliasing). With proper training, GANs provide a clearer and sharper 2D texture image magnitudes higher in quality than the original, while fully retaining the original's level of details, colors, etc. Known examples of extensive GAN usage include Final Fantasy VIII, Final Fantasy IX, Resident Evil REmake HD Remaster, and Max Payne.
Concerns about malicious applications
Concerns have been raised about the potential use of GAN-based human image synthesis for sinister purposes, e.g., to produce fake and/or incriminating photographs and videos. GANs can be used to generate unique, realistic profile photos of people who do not exist, in order to automate creation of fake social media profiles.
In 2019 the state of California considered and passed on 3 October 2019 the bill AB-602 that bans the use of human image synthesis technologies to make fake pornography without the consent of the people depicted and bill AB-730, which prohibits distribution of manipulated videos of a political candidate within 60 days of an election were signed by Governor Gavin Newsom. Both bills were authored by Assembly member Marc Berman. The laws will come into effect in 2020.
DARPA's Media Forensics program studies ways to counteract fake media, including fake media produced using GANs.
GANs that produce photorealistic images can be used to visualize interior design, industrial design, shoes, bags, and clothing items or items for computer games' scenes. Such networks were reported to be used by Facebook.
GANs can be used to age face photographs to show how an individual's appearance might change with age.
A variation of the GANs are used in training a network to generate optimal control inputs to nonlinear dynamical systems. Where the discriminatory network is known as a critic that checks the optimality of the solution and the generative network is known as an Adaptive network that generates the optimal control. The critic and adaptive network train each other to approximate a nonlinear optimal control.
GANs have been used to visualize the effect that climate change will have on specific houses.
A GAN model called Speech2Face can reconstruct an image of a person's face after listening to their voice.
In 2016 GANs were used to generate new molecules for a variety of protein targets implicated in cancer, inflammation, and fibrosis. In 2019 GAN-generated molecules were validated experimentally all the way into mice.
The general idea of learning via competition between players dates back to at least 1959 with the influential work of Arthur Samuel, demonstrating that algorithms could learn to play checkers via adversarial self-play.
Ian Goodfellow is recognized by several sources as having invented GANs in 2014. This paper included the first working implementation of a generative model based on adversarial networks, as well as game theoretic analysis establishing that the method is sound. However, there are several independent debates over who invented which aspects of GANs.
Some review papers about GANs from academic sources, such as assign credit for the idea to Ian Goodfellow and make no mention of Juergen Schmidhuber. Ian Goodfellow's own peer-reviewed GAN paper mentions Schmidhuber's unsupervised adversarial technique called predictability minimization (PM), claiming that PM is not a minimax game. Schmidhuber disputes this claim, citing Equation 2 of the 1996 paper on PM. He also formulates GANs as special cases of his Adversarial Curiosity (1990). Schmidhuber interrupted a presentation by Goodfellow in 2016 demanding more credit for his earlier work. Goodfellow himself states that the most direct inspiration for GANs was noise-contrastive estimation, which uses the same loss function as GANs and which Goodfellow studied during his PhD in 2010-2014.
Other people had similar ideas but did not develop them similarly. An idea involving adversarial networks was published in a 2010 blog post by Olli Niemitalo. This idea was never implemented and did not involve stochasticity in the generator and thus was not a generative model. It is now known as a conditional GAN or cGAN. An idea similar to GANs was used to model animal behavior by Li, Gauci and Gross in 2013.
Adversarial machine learning has other uses besides generative modeling and can be applied to models other than neural networks.
In 2017, a GAN was used for image enhancement focusing on realistic textures rather than pixel-accuracy, producing a higher image quality at high magnification. In 2017, the first faces were generated. These were exhibited in February 2018 at the Grand Palais. Faces generated by StyleGAN in 2019 drew comparisons with deepfakes.
Beginning in 2017, GAN technology began to make its presence felt in the fine arts arena with the appearance of a newly developed implementation which was said to have crossed the threshold of being able to generate unique and appealing abstract paintings, and thus dubbed a "CAN", for "stupid". A GAN system was used to create the 2018 painting Edmond de Belamy, which sold for US$432,500. An early 2019 article by members of the original CAN team discussed further progress with that system, and gave consideration as well to the overall prospects for an AI-enabled art.
In May 2019, researchers at Samsung demonstrated a GAN-based system that produces videos of a person speaking given only a single photo of that person.
In August 2019, A large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment was created for neural melody generation from lyrics using conditional GAN-LSTM (refer to sources at GitHub AI Melody Generation from Lyrics).
Bidirectional GAN(BiGAN) aims to introduce a generator model to act as the discriminator, whereby the discriminator naturally considers the entire translation space so that the inadequate training problem can be alleviated. To satisfy this property, generator and discriminator are both designed to model the joint probability of sentence pairs, with the difference that, the generator decomposes the joint probability with a source language model and a source-to-target translation model, while the discriminator is formulated as a target language model and a target-to-source translation model. To further leverage the symmetry of them, an auxiliary GAN is introduced and adopts generator and discriminator models of original one as its own discriminator and generator respectively. Two GANs are alternately trained to update the parameters. The resulting learned feature representation is useful for auxiliary supervised discrimination tasks, competitive with contemporary approaches to unsupervised and self-supervised feature learning.
- Goodfellow, Ian; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua (2014). Generative Adversarial Networks (PDF). Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014). pp. 2672–2680.
- Salimans, Tim; Goodfellow, Ian; Zaremba, Wojciech; Cheung, Vicki; Radford, Alec; Chen, Xi (2016). "Improved Techniques for Training GANs". arXiv:1606.03498 [cs.LG].
- Isola, Phillip; Zhu, Jun-Yan; Zhou, Tinghui; Efros, Alexei (2017). "Image-to-Image Translation with Conditional Adversarial Nets". Computer Vision and Pattern Recognition.
- Ho, Jonathon; Ermon, Stefano (2016). "Generative Adversarial Imitation Learning". Advances in Neural Information Processing Systems: 4565–4573. arXiv:1606.03476. Bibcode:2016arXiv160603476H.
- LeCun, Yann. "RL Seminar: The Next Frontier in AI: Unsupervised Learning".
- Luc, Pauline; Couprie, Camille; Chintala, Soumith; Verbeek, Jakob (2016-11-25). "Semantic Segmentation using Adversarial Networks". NIPS Workshop on Adversarial Training, Dec, Barcelona, Spain. 2016. arXiv:1611.08408. Bibcode:2016arXiv161108408L.
- Andrej Karpathy; Pieter Abbeel; Greg Brockman; Peter Chen; Vicki Cheung; Rocky Duan; Ian Goodfellow; Durk Kingma; Jonathan Ho; Rein Houthooft; Tim Salimans; John Schulman; Ilya Sutskever; Wojciech Zaremba, Generative Models, OpenAI, retrieved April 7, 2016
- Caesar, Holger (2019-03-01), A list of papers on Generative Adversarial (Neural) Networks: nightrome/really-awesome-gan, retrieved 2019-03-02
- Wong, Ceecee. "The Rise of AI Supermodels". CDO Trends.
- Dietmar, Julia. "GANs and Deepfakes Could Revolutionize The Fashion Industry". Forbes.
- Schawinski, Kevin; Zhang, Ce; Zhang, Hantian; Fowler, Lucas; Santhanam, Gokula Krishnan (2017-02-01). "Generative Adversarial Networks recover features in astrophysical images of galaxies beyond the deconvolution limit". Monthly Notices of the Royal Astronomical Society: Letters. 467 (1): L110–L114. arXiv:1702.00403. Bibcode:2017MNRAS.467L.110S. doi:10.1093/mnrasl/slx008.
- Kincade, Kathy. "Researchers Train a Neural Network to Study Dark Matter". R&D Magazine.
- Kincade, Kathy (May 16, 2019). "CosmoGAN: Training a neural network to study dark matter". Phys.org.
- "Training a neural network to study dark matter". Science Daily. May 16, 2019.
- at 06:13, Katyanna Quach 20 May 2019. "Cosmoboffins use neural networks to build dark matter maps the easy way". www.theregister.co.uk. Retrieved 2019-05-20.
- Mustafa, Mustafa; Bard, Deborah; Bhimji, Wahid; Lukić, Zarija; Al-Rfou, Rami; Kratochvil, Jan M. (2019-05-06). "CosmoGAN: creating high-fidelity weak lensing convergence maps using Generative Adversarial Networks". Computational Astrophysics and Cosmology. 6 (1): 1. arXiv:1706.02390. Bibcode:2019ComAC...6....1M. doi:10.1186/s40668-019-0029-9. ISSN 2197-7909.
- Paganini, Michela; de Oliveira, Luke; Nachman, Benjamin (2017). "Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis". Computing and Software for Big Science. 1: 4. arXiv:1701.05927. doi:10.1007/s41781-017-0004-6.
- Paganini, Michela; de Oliveira, Luke; Nachman, Benjamin (2018). "Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multi-Layer Calorimeters". Phys. Rev. Letters. 120 (4): 042003. arXiv:1705.02355. doi:10.1103/PhysRevLett.120.042003. PMID 29437460.
- Paganini, Michela; de Oliveira, Luke; Nachman, Benjamin (2018). "CaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks". Phys. Rev. D. 97: 014021. arXiv:1712.10321. doi:10.1103/PhysRevD.97.014021.
- Erdmann, Martin; Glombitza, Jonas; Quast, Thorben (2019). "Precise Simulation of Electromagnetic Calorimeter Showers Using a Wasserstein Generative Adversarial Network". Computing and Software for Big Science. 3: 4. arXiv:1807.01954. doi:10.1007/s41781-018-0019-7.
- Musella, Pasquale; Pandolfi, Francesco (2018). "Fast and Accurate Simulation of Particle Detectors Using Generative Adversarial Networks". Computing and Software for Big Science. 2: 8. arXiv:1805.00850. Bibcode:2018arXiv180500850M. doi:10.1007/s41781-018-0015-y.
- ATLAS, Collaboration (2018). "Deep generative models for fast shower simulation in ATLAS".
- SHiP, Collaboration (2019). "Fast simulation of muons produced at the SHiP experiment using Generative Adversarial Networks". arXiv:1909.04451 [physics.ins-det].
- Tang, Xiaoou; Qiao, Yu; Loy, Chen Change; Dong, Chao; Liu, Yihao; Gu, Jinjin; Wu, Shixiang; Yu, Ke; Wang, Xintao (2018-09-01). "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks". arXiv:1809.00219. Bibcode:2018arXiv180900219W.
- msmash (2019-02-14). "'This Person Does Not Exist' Website Uses AI To Create Realistic Yet Horrifying Faces". Slashdot. Retrieved 2019-02-16.
- Doyle, Michael (May 16, 2019). "John Beasley lives on Saddlehorse Drive in Evansville. Or does he?". Courier and Press.
- Targett, Ed (May 16, 2019). "California moves closer to making deepfake pornography illegal". Computer Business Review.
- Mihalcik, Carrie (2019-10-04). "California laws seek to crack down on deepfakes in politics and porn". cnet.com. CNET. Retrieved 2019-10-13.
- Knight, Will (Aug 7, 2018). "The Defense Department has produced the first tools for catching deepfakes". MIT Technology Review.
- Wei, Jerry (2019-07-03). "Generating Shoe Designs with Machine Learning". Medium. Retrieved 2019-11-06.
- Greenemeier, Larry (June 20, 2016). "When Will Computers Have Common Sense? Ask Facebook". Scientific American. Retrieved July 31, 2016.
- "3D Generative Adversarial Network". 3dgan.csail.mit.edu.
- Vondrick, Carl; Pirsiavash, Hamed; Torralba, Antonio (2016). "Generating Videos with Scene Dynamics". carlvondrick.com. arXiv:1609.02612. Bibcode:2016arXiv160902612V.
- Antipov, Grigory; Baccouche, Moez; Dugelay, Jean-Luc (2017). "Face Aging With Conditional Generative Adversarial Networks". arXiv:1702.01983 [cs.CV].
- Kang, Yuhao; Gao, Song; Roth, Rob (2019). "Transferring Multiscale Map Styles Using Generative Adversarial Networks". International Journal of Cartography. 5 (2–3): 115–141. arXiv:1905.02200. Bibcode:2019arXiv190502200K. doi:10.1080/23729333.2019.1615729.
- Wijnands, Jasper; Nice, Kerry; Thompson, Jason; Zhao, Haifeng; Stevenson, Mark (2019). "Streetscape augmentation using generative adversarial networks: Insights related to health and wellbeing". Sustainable Cities and Society. 49: 101602. arXiv:1905.06464. Bibcode:2019arXiv190506464W. doi:10.1016/j.scs.2019.101602.
- Padhi, Radhakant; Unnikrishnan, Nishant (2006). "A single network adaptive critic (SNAC) architecture for optimal control synthesis for a class of nonlinear systems". Neural Networks. 19 (10): 1648–1660. doi:10.1016/j.neunet.2006.08.010. PMID 17045458.
- "AI can show us the ravages of climate change". MIT Technology Review. May 16, 2019.
- Christian, Jon (May 28, 2019). "ASTOUNDING AI GUESSES WHAT YOU LOOK LIKE BASED ON YOUR VOICE". Futurism.
- Zhavoronkov, Alex (2019). "Deep learning enables rapid identification of potent DDR1 kinase inhibitors". Nature Biotechnology. 37 (9): 1038–1040. doi:10.1038/s41587-019-0224-x. PMID 31477924.
- Gregory, Barber. "A Molecule Designed By AI Exhibits 'Druglike' Qualities". Wired.
- Samuel, Arthur (July 1959). "Some Studies in Machine Learning Using the Game of Checkers". IBM Journal of Research and Development.
- Klok, Christie (February 21, 2018). "The GANfather: The man who's given machines the gift of imagination". MIT Technology Review. Retrieved 14 May 2019.
- "Invented a way for neural networks to get better by working together". Innovators under 35. MIT Technology Review.
- Novet, Jordan (April 4, 2019). "One of Google's top A.I. people has joined Apple". CNBC.
- Giles, Martin. "The GANfather: The man who's given machines the gift of imagination". MIT Technology Review. Retrieved 2019-06-21.
- Steven Van Vaerenbergh (2018-01-18), Ian Goodfellow: Generative Adversarial Networks (NIPS 2016 tutorial), retrieved 2019-06-21
- "GANs were invented in 2010?". reddit r/MachineLearning. 2019. Retrieved 2019-05-28.
- "[R] [1906.04493] Unsupervised Minimax: Adversarial Curiosity, Generative Adversarial Networks, and Predictability Minimization (Schmidhuber)". reddit r/MachineLearning. 2019. Retrieved 2019-06-12.
- Cresswell, Antonia; White, Tom; Dumoulin, Vincent; Arulkumaran, Kai; Sengupta, Biswa; Bharath, Anil (Jan 1, 2018). "Generative Adversarial Networks: An Overview". IEEE Signal Processing Magazine. 35 (1): 53–65. arXiv:1710.07035. Bibcode:2018ISPM...35...53C. doi:10.1109/MSP.2017.2765202.
- Schmidhuber, Juergen (2019). "Unsupervised Minimax: Adversarial Curiosity, Generative Adversarial Networks, and Predictability Minimization". arXiv:1906.04493 [cs.NE].
- Schmidhuber, Jürgen; Eldracher, Martin; Foltin, Bernhard (1996). "Semilinear predictability minimzation produces well-known feature detectors". Neural Computation. 8 (4): 773–786. doi:10.1162/neco.19184.108.40.2063.
- Schmidhuber, Jürgen (1990). "Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments" (PDF). TR FKI-126-90. Tech. Univ. Munich.
- Schmidhuber, Jürgen (1991). "A possibility for implementing curiosity and boredom in model-building neural controllers". Proc. SAB'1991. MIT Press/Bradford Books. pp. 222–227.
- Vance, Ashlee (May 15, 2018). "Quote: It was another way of saying, Hey, kid, you didn't invent this". Bloomberg Business Week. Retrieved 2019-01-16.
- Goodfellow, Ian. "Generative Adversarial Networks" (PDF).
- Gutmann, Michael; Hyvärinen, Aapo. "Noise-Contrastive Estimation" (PDF). International Conference on AI and Statistics.
- Niemitalo, Olli (February 24, 2010). "A method for training artificial neural networks to generate missing data within a variable context". Internet Archive (Wayback Machine). Retrieved February 22, 2019.
- Li, Wei; Gauci, Melvin; Gross, Roderich (July 6, 2013). "A Coevolutionary Approach to Learn Animal Behavior Through Controlled Interaction". Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation (GECCO 2013). Amsterdam, The Netherlands: ACM. pp. 223–230. doi:10.1145/2463372.2465801.
- Sajjadi, Mehdi S. M.; Schölkopf, Bernhard; Hirsch, Michael (2016-12-23). "EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis". arXiv:1612.07919 [cs.CV].
- "This Person Does Not Exist: Neither Will Anything Eventually with AI". March 20, 2019.
- "ARTificial Intelligence enters the History of Art". December 28, 2018.
- Tom Février (2019-02-17). "Le scandale de l'intelligence ARTificielle".
- "StyleGAN: Official TensorFlow Implementation". March 2, 2019 – via GitHub.
- Paez, Danny (2019-02-13). "This Person Does Not Exist Is the Best One-Off Website of 2019". Retrieved 2019-02-16.
- BESCHIZZA, ROB (2019-02-15). "This Person Does Not Exist". Boing-Boing. Retrieved 2019-02-16.
- Horev, Rani (2018-12-26). "Style-based GANs – Generating and Tuning Realistic Artificial Faces". Lyrn.AI. Retrieved 2019-02-16.
- Elgammal, Ahmed; Liu, Bingchen; Elhoseiny, Mohamed; Mazzone, Marian (2017). "CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Norms". arXiv:1706.07068 [cs.AI].
- Cohn, Gabe (2018-10-25). "AI Art at Christie's Sells for $432,500". The New York Times.
- Mazzone, Marian; Ahmed Elgammal (21 February 2019). "Art, Creativity, and the Potential of Artificial Intelligence". Arts. 8: 26. doi:10.3390/arts8010026.
- Kulp, Patrick (May 23, 2019). "Samsung's AI Lab Can Create Fake Video Footage From a Single Headshot". AdWeek.
- Yu, Yi; Canales, Simon (August 15, 2019). "Conditional LSTM-GAN for Melody Generation from Lyrics". arXiv:1908.05551 [cs.AI].
- Zhirui Zhang; Shujie Liu; Mu Li; Ming Zhou; Enhong Chen (October 2018). "Bidirectional Generative Adversarial Networks for Neural Machine Translation" (PDF). pp. 190–199.
- Knight, Will. "5 Big Predictions for Artificial Intelligence in 2017". MIT Technology Review. Retrieved 2017-01-05.
- A Style-Based Generator Architecture for Generative Adversarial Networks
- This Person Does Not Exist – photorealistic images of people who do not exist, generated by StyleGAN
- Generative Adversarial Networks: A Survey and Taxonomy