Project SpatialAV

A research project of the Fraunhofer Digital Cinema Alliance

SpatialAV – experience the future of recording and production technology

Development of an intelligent, modular, multi-sensory recording and production system for immersive audiovisual media

Producing a high-quality feature film, particularly in 3D, is a time-consuming and expensive proposition. The obvious example is »Avatar«, which cost 237 million dollars and took four years to make. Productions with less than ample budgets have no way of achieving the same high standard and so are practically doomed to fail at the box office, which is one reason why there are so few successful high-quality 3D films. To be able to create practical, marketable technology solutions for the future, scientists at the four member institutes of the Fraunhofer Digital Cinema Alliance, pooling their experience and expertise in video and audio technology, have launched the »SpatialAV« development project. During this project an intelligent, modular, multi-sensory recording and production system for immersive audio-visual media will be developed, which allows a feeling like being part of the scene for the spectator.

The goal of the SpatialAV project is automating workflows, which supports high quality and helps cut technical production costs. Using intelligent, networked cameras, microphone arrays and additional sensors providing, for example, positional, inertial or depth data, the system should be able to automatically adjust recording parameters, supply additional supporting data for post-production and so be sufficiently versatile to be used in a wide range of production and reproduction scenarios. The plan is to use recorded data to derive visualization and auralization data optimized for a specific type of reproduction: 2D, 3D, free-viewpoint or panoramic video. The recording and production systems to be developed are to be suitable both for classic 2D and 3D reproduction on planar (e.g. theater and television) screens and for three-dimensional screens, as used in panoramic and dome theaters.

Fraunhofer-Institutes involved in the project:

Fraunhofer IIS
self-calibrating cameras, use of new recording methods based on light
field and HDR technologies; sensor fusion; post-production and distribution formats

Fraunhofer HHI
system design and equipment for production of 3D and panoramic content; algorithms for processing and cleaning up data for reproduction

Fraunhofer IDMT
3D audio, object-based audio coding, methods and tools for audio-visual
coherence, portable 3D audio reproduction equipment

Fraunhofer FOKUS
3D dome projection and methods for adapting content to panoramic
or spherical projection screens