Free viewpoint television

Introduction

Free Viewpoint Television is a 3D information system that allows image acquisition, processing and displaying. This system has been designed by Masayuki Tanimoto, from Nagoya University (Japan). It’s based on “Ray-Based Image Engineering”, using the Ray-space method, and Ray-space Representation.

FTV as well as TV should be transparent. It means that FTV should work well for any scene and content. FTV acquires, stores and transmits all visual information of the 3D space.

FTV tries to build a new epoch in television history because it allows viewers to see a real 3D world, just changing their point of view, just like they were in the acquired scene. Viewers can make their choice with what they want to see from a scene.

A stage is captured by several cameras connected to a server. Watchers, who are in front of the FTV display, will be able to see the acquired images, depending on where they are watching. The information taken out from the cameras built-in the FTV displays will manage to get what image is to be displayed on the television. If the user moves to right, he or she will be able to see whatever is on that side of the stage and vice-versa.

Acquisition of FTV Signal

Signal acquisition is made using cameras connected to a server, put in front of the scene to film, at the same height. There are two main position setups for the cameras: linear & half-round. The ray space is composed by many view images. A cross-section of the ray space gives a view image.

File:Ftv1.jpg

Figure 1 Basic scheme of FTV storage and reproduction system

Added values of FTV

FTV can acquire, store and transmit all visual information of the 3D space. Furthermore, the function of FTV is not only to generate free viewpoint images but also to transport the 3D visual space. FTV should be a platform of various kinds of 3D space information. FTV signal is a new frontier of signal processing since it has higher dimension than standard video signal.

Processing of FTV Signal

The quality of the generated views depends mainly of the interpolation made, because we can’t use infinite cameras to build a scene. If the user is set in a position where a camera is not able to reach the needed image, FTV interpolates the images from the nearest cameras.

File:Ftv2.jpg

Figure 2. Variation between images according to point of view

Multi-Camera Alignment

Preprocessing of multi-view images is strongly needed for efficient coding and interpolation. In practical multi-camera systems, it is difficult to align many cameras with the same performance at the desired places precisely.

Ray-Space Interpolation

We proposed several interpolation schemes of the ray-space. SNR of the interpolated image is shown as a function of MDD (Maximum Disparity Difference) for various interpolation schemes

Ray-Space Coding

A new coding method of the ray-space is proposed for the circular camera arrangement. It subsamples the multi-views and the deleted views are reconstructed by interpolation. The interpolation is performed by considering the sinusoidal structure of the horizontal cross section of the ray-space. The subsampled views are encoded by JPEG.

Two types of interpolation are examined. One is conventional nonhierarchical interpolation and another is hierarchical one.

Display of FTV Signal

FTV offers a high-grade detail and very high definition even recording very complex scenes, like for example tiny objects as fishes, bubbles and light reflections from an aquarium glass, but FTV needs a new user interface to display free viewpoint images. By now, this TV system is developed using 2D and automultiscopic displays with a head tracking system.

100-Camera System

This system has been designed to be able to acquire large scenes. It’s based on a server and 100 clients (called nodes) equipped with cameras PULNiX TM-1400CL. The server generates a sync signal and it’s distributed to all nodes. This system is not only able to capture high-resolution video at 30fps but it can record analogue signals up to 96 kHz.

Conclusion

FTV has been realized based on the ray-space method. FTV makes it possible the 3D space communication. FTV is transparent and user-centered. The users can select their viewpoints freely by themselves. The number of cameras to generate FTV signal is greatly reduced by ray space interpolation. The amount of FTV raw data is very large. However, it can be highly compressed because the ray-space is very redundant.