Jump to content

2D to 3D conversion

Add links
From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by 3dconversion (talk | contribs) at 02:12, 29 February 2012 (→‎Conversion Vendors). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

2D to 3D video conversion is the process of transforming the original 2D video to a 3D form, which in almost all cases is stereo, so it is the process of creating imagery for each eye from one 2D image. That is why the transformation is also called 2D to stereo 3D conversion, or stereo conversion.

Overview

2D-to-3D conversion adds the binocular disparity depth cue to digital images perceived by the brain, thus, if done properly, greatly improving the immersive effect while viewing stereo video in comparison to 2D video. However, in order to be thus successful, the conversion should be done with sufficient accuracy and correctness: the quality of the original 2D images should not deteriorate, and the introduced disparity cue should not contradict to other cues used by the brain for depth perception. If done properly and thoroughly, the conversion produces stereo video of similar quality to “native” stereo video which is shot in stereo and accurately adjusted and aligned in post-production.[1] But, again, “it has to be done right” as James Cameron, director of Avatar, has said.[2]

Two approaches to stereo conversion can be loosely defined: quality semiautomatic conversion for cinema and high quality 3DTV, and low quality automatic conversion for cheap 3DTV, VOD and similar applications.

Importance and Applicability

2D to 3D conversion is widespread nowadays. The majority of non-CGI stereo 3D blockbusters were converted fully or at least partially from 2D footage. Even “Avatar” contains several scenes shot in 2D and converted to stereo in post-production.[3] The reasons of shooting in 2D instead of stereo are financial, technical and sometimes artistic:[4][1]

  • Stereo post-production workflow is much more complex and not as well-established as the 2D workflow, requires more work and rendering
  • Professional stereoscopic rigs are much more expensive and bulky than customary monocular cameras. Some shots, particularly action scenes, can be only shot with relatively small 2D cameras
  • Stereo cameras introduce various mismatches in stereo image – such as vertical parallax, tilt, color shift, reflections and glares in different positions - that should be fixed in post-production anyway because they ruin 3D effect. This correction sometimes may have complexity comparable to stereo conversion

With the lack of stereo content, 2D to 3D conversion is the only way to meet demands of the rapidly growing 3DTV industry. Stereo conversion is needed for converting older popular films – such as the Star Wars series, Titanic, The Matrix franchise and so on.

Even in case of stereo shooting the conversion can frequently be necessary. Besides the mentioned hard-to-shoot scenes, there are situations when mismatches in stereo views are too big to adjust, and it is simpler to perform 2D to stereo conversion treating one of the views as the original 2D video.

General Problems

Without respect to particular algorithms, all conversion workflows should solve the following tasks: [5][4]

  1. Allocation of “depth budget” – defining the range of permitted disparity or depth, what depth value corresponds to the screen position (so-called “convergence point” position), the permitted distance ranges for out-of-the-screen effects and behind-the-screen background objects. If an object in stereo pair is in exactly the same spot for both eyes, then it will appear on the screen surface and it will be in zero parallax. Objects in front of the screen are called to be in negative parallax, and background imagery behind the screen is in positive parallax. There are the corresponding negative or positive offsets in object positions for left and right eye images.
  2. Control of comfortable disparity depending on scene type and motion – too big parallax or conflicting depth cues may cause eye-strain and nausea effects
  3. Filling of uncovered areas – left or right view images show a scene from a different angle, and parts of objects or entire objects covered by the foreground in the original 2D image should become visible in a stereo pair. Sometimes the background surfaces are known or can be estimated, so they should be used for filling uncovered areas. Otherwise the unknown areas should be guesstimated and painted in, since the exact reconstruction is not possible.

High quality conversion methods should also deal with many typical problems including:

  • Translucent objects
  • Reflections
  • Fuzzy semitransparent object borders – such as hair, fur, foreground out-of-focus objects, thin objects
  • Film grain (real or artificial) and similar noise effects
  • Scenes with fast erratic motion
  • Small particles – rain, snow, explosions and so on.

Quality Semiautomatic Conversion

Depth-based Conversion

Most semiautomatic methods of stereo conversion use depth maps and depth-image-based rendering.[5][4]

The idea is that a separate auxiliary picture known as the "depth map" is created for each frame or for a series of homogenous frames to indicate depths of objects present in the scene. The depth map is a separate grayscale image having the same dimensions as the original 2D image, with various shades of gray to indicate the depth of every part of the frame. While depth mapping can produce a fairly potent illusion of 3D objects in the video, it inherently does not support semi-transparent objects or areas, nor does it allow explicit use of occlusion, so these and other similar issues should be dealt with in a specific way.

An example of depth map

The major steps of depth-based conversion methods are:

  1. Depth budget allocation – how much total depth in the scene and where the screen plane will be
  2. Image segmentation, creation of mattes or masks, usually by rotoscoping. Each important surface should be isolated. The level of detail depends on the required conversion quality and budget.
  3. Depth map creation. Each isolated surface should be assigned with its depth map. Those separate depth maps should be composed into scene depth map. This is an iterative process requiring adjusting of objects, shapes, depth, and visualization of intermediate results in stereo. Depth micro-relief, 3D shape is added to most important surfaces to prevent the “cardboard” effect when stereo imagery looks like a combination of flat images just set at different depth.
  4. Stereo generation based on 2D+Depth with any supplemental information like clean plates, restored background, transparency maps, etc. There is a left and right image at the end. Usually the original 2D image is treated as the center image, so both stereo views are generated. Some methods propose to use the original image as one eye and generate only the other eye image to minimize the conversion cost.[4] During stereo generation pixels of the original image is shifted to the left or to the right depending on depth map, maximum selected parallax and screen surface position.
  5. Reconstruction and painting of any uncovered areas if remained unfilled by the stereo generator.

Stereo can be presented in any format for preview purposes, including anaglyph.

Time-consuming steps are image segmentation/rotoscoping, depth map creation and uncovered area filling. The latter is especially important for the highest quality conversion.

There are various automation techniques for depth map creation and background reconstruction. For example, automatic depth estimation can be used to generate initial depth maps for certain frames and shots.[6]

Multi-layering

A development on depth mapping, multi-layering works around the limitations of depth mapping by introducing several layers of grayscale depth masks to implement limited semi-transparency. Similar to a simple technique,[7] multi-layering involves applying a depth map to more than one "slice" of the flat image, resulting in a much better approximation of depth and protrusion. The more layers are processed separately per frame, the higher the quality of 3D illusion tends to be.

Other Approaches

3D reconstruction and re-projection may be used for stereo conversion. It involves scene 3D model creation, extraction of original image surfaces as textures for 3D objects and, finally, rendering the 3D scene from two virtual cameras to acquire stereo video. The approach works well enough in case of scenes with static rigid objects like urban shots with buildings, interior shots, but has problems with non-rigid bodies and soft fuzzy edges.[3]

Conversion Vendors

There are a lot of companies offering 2D to 3D conversion. However, there are not so many vendors of high quality stereo conversion for blockbuster theatrical releases. Considering the list of films converting, the following companies are the key players in this market:

Cost

The price on high quality stereo conversion varies in the range of $25,000..100,000 per minute according to several estimates.[5][2][9] For lower quality semiautomatic conversion, especially for 3DTV application, it is much lower.

Automatic Conversion

Depth from Motion

It is possible to automatically estimate depth using different types of motion. In case of camera motion depth map of the entire scene can be calculated. Also, object motion can be detected and moving areas can be assigned with smaller depth values than the background. Besides, occlusions provide information on relative position of moving surfaces. [10][11]

Depth from Focus

Approaches of this type are also called “depth from defocus” and “depth from blur”.[12] [10] The depth information is estimated based on the amount of blur of the object. It is usually assumed that sharper surfaces are closer.

If the sky is detected in the processed image, it can also be taken into account that more distant objects, besides being hazy, should be more desaturated and more bluish because of a thick air layer.[12]

Depth from Perspective

The idea of the method is based on the fact that parallel lines, such as railroad tracks and roadsides, appear to converge with distance, eventually reaching a vanishing point at the horizon. Finding this vanishing point gives the farthest point of the whole image.[12][10]

The more the lines converge, the farther away they appear to be. So, for depth map, the area between two neighboring vanishing lines can be approximated with a gradient plane.

Software

Conversion Studios Software

Many vendors of high quality stereo 3D conversion announce development and usage of their own proprietary software. For example:

  • Prime Focus: View-D software[3]
  • In-Three (Digital Domain): Dimensionalization software and process[3]
  • Legend 3D declares use of its own unnamed software and unique conversion method[13]

It is assumed, however, that many companies usually employ customized The Foundry NUKE video compositing software with the set of plugins named Ocula and their own scripts and plugins.[3]

2D to 3D Conversion Software Released on the Market

Gimpel3D

Gimpel3D[14] uses a proprietary projective modeling system for 3D scene reconstruction and stereo generation by re-projection from a different viewpoint. The user works in true proportional space where the scene can be viewed from any location. The scene is edited geometrically in space using tools specifically designed to work with the perspective projection of the image. There are tools for relief extraction, depth painting and others.

The free fully functional version of the software is available.

Philips BlueBox (Discontinued)

Philips developed a 3D content creation software suite named BlueBox[15] which includes tools and a service for semi-automated conversion of 2D content into 2D-plus-Depth format and automatic generation of 2D-plus-Depth from stereo. The resulting 3D videos can be shown on Philips autostereoscopic 3D displays of any type.

In 2009 Philips suddenly announced closure of its 3D solutions venture.[16]

The BlueBox was offered as a service, coming with multiuser client tools and a dedicated BlueBox hardware.

The Foundry NUKE and OCULA

A lot of companies performing 2D-to-3D conversion and related works use NUKE software as a framework and a “Swiss Army Knife” collection of video processing plugins.[3] There is a collection of plugins named OCULA[17] designed for stereo post-production. Some OCULA plugins can be usable for stereo conversion tasks, especially quality control.

NUKE can be enough for stereo conversion even without any other tools.[18]

YUVsoft 2D to 3D Suite

The company YUVsoft offers tools for depth-based 2D-to-stereo 3D conversion.[19] The main product, called 2D to 3D Suite, is a set of plugins for Adobe After Effects or The Foundry NUKE. The software provides automation for depth map creation, editing and stereo generation. There are tools for estimating initial depth maps based on motion, focus and other cues. The Depth Propagation tool automates scene depth map creation and editing by propagating a given depth map from one or two frames to the entire scene. The resulting depth can be improved using roto mattes.

The software is intended for professional use and offered to companies only. It is claimed that the software was used for converting a few full-length movies and commercials.

G´MIC plugin for GIMP

G´MIC[20] plugin for GIMP[21] is a free open source filter package which contains filters for converting 2D still images and 2D videos into stereoscopic 3D based on fully automatic or operator supported depth map estimation models. The software supports direct conversion into a broad range of stereoscopic output formats. It can also produce multiple viewpoints from single images suitable for animated GIF´s and autostereoscopic presentations. It is claimed that the software can be used for automatic high-quality conversions of full-length movies as well as for still images. The filters are supported as well by step-by-step tutorials.[22]

See also

Wikimedia Commons has media related to 2D to 3D conversion.

References

  1. ^ a b Barry Sandrew. “2D – 3D Conversion Can Be Better Than Native 3D”
  2. ^ a b James Cameron Releases Scenes From Titanic 3D
  3. ^ a b c d e f Mike Seymour. Art of Stereo conversion: 2D to 3D
  4. ^ a b c d Scott Squires. 2D to 3D Conversions
  5. ^ a b c Jon Karafin. State-of-the-Art 2D to 3D Conversion and Stereo VFX International 3D Society University. Presentation from the October 21, 2011 3DU-Japan event in Tokyo.
  6. ^ YUVsoft. 2D–to–Stereo 3D Conversion Process
  7. ^ Creating multiple layers for 3D effect in Photoshop
  8. ^ Stereo D, LLC
  9. ^ Rebecca Keegan. 3-D makeover coming to aging Hollywood blockbusters Los Angeles Times, September 25, 2011
  10. ^ a b c Dr. Lai-Man Po. Automatic 2D-to-3D Video Conversion Techniques for 3DTV Department of Electronic Engineering, City University of Hong Kong. 13 April 2010
  11. ^ Automatic 2D to 2D-plus-Depth conversion sample for a camera motion scene
  12. ^ a b c Qingqing We. Converting 2D to 3D: A Survey Faculty of Electrical Engineering, Mathematics and Computer Science Delft University of Technology, the Netherlands
  13. ^ Legend 3D How 2D to 3D Conversion Works
  14. ^ Gimpel3D software
  15. ^ Philips BlueBox
  16. ^ Philips Decides to Shut Down 3D Operation March 27th, 2009
  17. ^ OCULA plug-in tools
  18. ^ Stereo Conversion in Nuke
  19. ^ YUVsoft 2D to 3D Suite
  20. ^ G´MIC[1]
  21. ^ GIMP[2]
  22. ^ TK Filter[3]

20. ^ 2D to 3D Converter for Mac

Retrieved from "https://en.wikipedia.org/w/index.php?title=2D_to_3D_conversion&oldid=479402219"