Human image synthesis

From Wikipedia, the free encyclopedia
Jump to navigation Jump to search

Human image synthesis can be applied to make believable and even photorealistic renditions[1][2] of human-likenesses, moving or still. This has effectively been the situation since the early 2000s. Many films using computer generated imagery have featured synthetic images of human-like characters digitally composited onto the real or other simulated film material.

Bidirectional scattering distribution function (BSDF) for human skin likeness requires both BRDF and special case of BTDF where light enters the skin, is transmitted and exits the skin.

Timeline of human image synthesis[edit]

BRDF vs. subsurface scattering inclusive BSSRDF i.e. Bidirectional scattering-surface reflectance distribution function

Key breakthrough to photorealism: reflectance capture[edit]

ESPER LightCage - 3D face scanning rig

In SIGGRAPH 2000 Paul Debevec et al. of USC presented their method to capture the reflectance field with their extremely simple light stage.[4]

The scientific breakthrough required finding the subsurface light component (the simulation models are glowing from within slightly) which can be found using knowledge that light that is reflected from the oil-to-air layer retains its polarization and the subsurface light loses its polarization. So equipped only with a movable light source, movable video camera, 2 polarizers and a computer program doing extremely simple math and the last piece required to reach photorealism was acquired.[4]

For a believable result both light reflected from skin (BRDF) and within the skin (a special case of BTDF) which together make up the BSDF must be captured and simulated.



The whole process of making digital look-alikes i.e. characters so lifelike and realistic that they can be passed off as pictures of humans is a very complex task as it requires photorealistically modeling, animating, cross-mapping, and rendering the soft body dynamics of the human appearance.

Synthesis with an actor and suitable algorithms is applied using powerful computers. The actor's part in the synthesis is to take care of mimicking human expressions in still picture synthesizing and also human movement in motion picture synthesizing. Algorithms are needed to simulate laws of physics and physiology and to map the models and their appearance, movements and interaction accordingly.

Often both physics/physiology based (i.e. skeletal animation) and image-based modeling and rendering are employed in the synthesis part. Hybrid models employing both approaches have shown best results in realism and ease-of-use.

Using displacement mapping plays an important part in getting a realistic result with fine detail of skin such as pores and wrinkles as small as 100 µm.


Main applications fall within the domains of virtual cinematography, computer and video games and covert disinformation attacks.

Furthermore, some research suggests that it can have therapeutic effects as "psychologists and counselors have also begun using avatars to deliver therapy to clients who have phobias, a history of trauma, addictions, Asperger’s syndrome or social anxiety."[19] The strong memory imprint and brain activation effects caused by watching a digital look-alike avatar of yourself is dubbed the doppelgänger effect.[19] The doppelgänger effect can heal when covert disinformation attack is exposed as such to the targets of the attack.

Related issues[edit]

The speech synthesis is verging on being completely indistinguishable from a real human's voice with the 2016 introduction of audio generation software Adobe Voco, a prototype slated to be a part of the Adobe Creative Suite and DeepMind WaveNet, a prototype from Google.[20] Ability to steal and manipulate other peoples voices raises obvious ethical concerns. [21]

This coupled with the fact that (as of 2016) techniques which allow near real-time counterfeiting of facial expressions in existing 2D video have been believably demonstrated increases the stress on the disinformation situation.[11]

See also[edit]


  1. ^ Physics-based muscle model for mouth shape control on IEEE Explore (requires membership)
  2. ^ Realistic 3D facial animation in virtual space teleconferencing on IEEE Explore (requires membership)
  3. ^ "Images de synthèse : palme de la longévité pour l'ombrage de Gouraud".
  4. ^ a b c Debevec, Paul (2000). "Acquiring the reflectance field of a human face". Proceedings of the 27th annual conference on Computer graphics and interactive techniques - SIGGRAPH '00. ACM. pp. 145–156. doi:10.1145/344779.344855. ISBN 978-1581132083. Retrieved 2017-05-24.
  5. ^ Pighin, Frédéric. "Siggraph 2005 Digital Face Cloning Course Notes" (PDF). Retrieved 2017-05-24.
  6. ^ In this TED talk video at 00:04:59 you can see two clips, one with the real Emily shot with a real camera and one with a digital look-alike of Emily, shot with a simulation of a camera - Which is which is difficult to tell. Bruce Lawmen was scanned using USC light stage 6 in still position and also recorded running there on a treadmill. Many, many digital look-alikes of Bruce are seen running fluently and natural looking at the ending sequence of the TED talk video.
  7. ^ ReForm - Hollywood's Creating Digital Clones (youtube). The Creators Project. 2017-05-24.
  8. ^ Debevec, Paul. "Digital Ira SIGGRAPH 2013 Real-Time Live". Retrieved 2017-05-24.
  9. ^ "Scanning and printing a 3D portrait of President Barack Obama". University of Southern California. 2013. Retrieved 2017-05-24.
  10. ^ Giardina, Carolyn (2015-03-25). "'Furious 7' and How Peter Jackson's Weta Created Digital Paul Walker". The Hollywood Reporter. Retrieved 2017-05-24.
  11. ^ a b Thies, Justus (2016). "Face2Face: Real-time Face Capture and Reenactment of RGB Videos". Proc. Computer Vision and Pattern Recognition (CVPR), IEEE. Retrieved 2017-05-24.
  12. ^ Suwajanakorn, Supasorn; Seitz, Steven; Kemelmacher-Shlizerman, Ira (2017), Synthesizing Obama: Learning Lip Sync from Audio, University of Washington, retrieved 2018-03-02
  13. ^ Roettgers, Janko (2018-02-21). "Porn Producers Offer to Help Hollywood Take Down Deepfake Videos". Variety. Retrieved 2018-02-28.
  14. ^ Takahashi, Dean (2018-03-21). "Epic Games shows off amazing real-time digital human with Siren demo". VentureBeat. Retrieved 2018-09-10.
  15. ^ Kuo, Lily (2018-11-09). "World's first AI news anchor unveiled in China". Retrieved 2018-11-09.
  16. ^ Hamilton, Isobel Asher (2018-11-09). "China created what it claims is the first AI news anchor — watch it in action here". Retrieved 2018-11-09.
  17. ^ Harwell, Drew (2018-12-30). "Fake-porn videos are being weaponized to harass and humiliate women: 'Everybody is a potential target'". The Washington Post. Retrieved 2019-03-14. In September [of 2018], Google added “involuntary synthetic pornographic imagery” to its ban list
  18. ^ a b Paez, Danny (2019-02-13). "This Person Does Not Exist Is the Best One-Off Website of 2019". Inverse (website). Retrieved 2018-03-05.
  19. ^ a b Murphy, Samantha (2011). "Scientific American: Your Avatar, Your Guide" (.pdf). Scientific American / Uni of Stanford. Retrieved 2013-06-29.
  20. ^ "WaveNet: A Generative Model for Raw Audio". 2016-09-08. Retrieved 2017-05-24.
  21. ^ "Adobe Voco 'Photoshop-for-voice' causes concern". BBC. 2016-11-07. Retrieved 2016-07-05.