The future of spatial audio: a localization in immersive space
by Johannes Scherzer | December 11, 2024"Zauberei auf dem Sender" is regarded as the beginning of a media-specific art form, the radio play. Hans Flesch used the technical means available at his time to break new ground. Exactly 100 years after the broadcast of Hans Flesch's "Zauberei auf dem Sender" on 24 October 2024, the Berlin Radio Drama Festival (BHF) celebrated at the TiME Lab (Tomorrow's immersive Media Experience Lab) of the Fraunhofer Heinrich Hertz Institute (HHI) with four prize-winning plays from the festival program of previous years. The aim was to experience what the audio drama of the future might sound like - using the technical means of our time: spatial audio.
"Zauberei auf dem Sender" is regarded as the beginning of a media-specific art form, the radio play. Hans Flesch used the technical means available at his time to break new ground. Exactly 100 years after the broadcast of Hans Flesch's "Zauberei auf dem Sender" on 24 October 2024, the Berlin Radio Drama Festival (BHF) celebrated at the TiME Lab (Tomorrow's immersive Media Experience Lab) of the Fraunhofer Heinrich Hertz Institute (HHI) with four prize-winning plays from the festival program of previous years. The aim was to experience what the audio drama of the future might sound like - using the technical means of our time: spatial audio.
The TiME Lab is equipped with a sophisticated spatial audio technology called wave field synthesis. It's a form of holophony that makes it possible to position sounds around the audience via hundreds of loudspeakers but also at a much greater distance than the physical space would allow, even amid the listening audience. In the TiME Lab, this wave field synthesis technology is supplemented by additional loudspeakers above the audience and at the bottom of the screen so that the mixing engineer can position sounds also in the third dimension, i.e., above and below. BHF and HHI presented three short audio plays and an excerpt from a 50-minute piece for the anniversary. These were "optically supported" on the vast 180° screen of the TiME-Lab by visuals created explicitly for the pieces.
And indeed, sounds could be heard from all directions. But something crucial was missing. First, the sounds seemed to stick to the loudspeakers in the TiME-Lab room; there was hardly a trace of the spatial depth that can be created with wave field synthesis. But it couldn't be down to the technology; there were fine Neumann loudspeakers and some truly capable subwoofers. What became more and more puzzling with every piece performed, however, was the motivation for the spatial positioning and movement of the sounds: why one voice here, the other voice there? The spatial arrangement of the sounds, atmospheres, and musical elements seemed equally arbitrary. What was the intention, the idea, the concept here? What was the thematic reference?
Read: What is spatial audio? An in-depth conversation with Ann, one of the most recognized experts today.
As it turned out after the presentation, the radio plays had not been conceived originally for spatial audio, using the spatial design tools of the specific medium. Instead, all were re-mixes of stereophonic radio plays, and the spatialization was subsequently applied to the ready-made products. In addition, the creators of the radio plays were apparently not involved in the re-mix, as some experienced the new mix for the first time during the presentation, including the visuals.
Indeed, the future conjured up at the TiME Lab on its anniversary certainly looked interesting. It began around 15 years ago when students of screenwriting and sound at the Film University Potsdam-Babelsberg experimented artistically in a research project on the art of media-specific narration. It was about spatial audio, and the first step was to explore the 5.1 surround format in collaboration with the broadcasting studios of the RBB, where the pieces were mixed and presented. The next step was to experiment with wave field synthesis in partnership with the then-newly established TiME Lab at the HHI. In searching for new qualities, forms, and narrative strategies in radio drama, the students examined the specific creative means of the respective medium through artistic research. They explored the qualitative possibilities both technically and in terms of content and design. That was particularly interesting in wave field synthesis due to its innovative technological features. Plots were developed, and scripts were written, changed, and adapted. Various spatial recording methods for dialogues, sound, and music were tested in the recording studio and the city. New editing and narrative techniques were being explored in editing and sound design. After months of work, the students finally mixed their pieces in the presentation room of the then-manufacturer IOSONO, the only studio far and wide where wave field synthesis could be used for production.
Exciting questions arose: How do you stage the movement of characters in space? What relationships occur between the audience and the performing characters when the audience is at the center of the medium, in the middle of the stage? How do we describe places? Do we need a different pacing? Can more events co-occur when the acoustic stage is so large? Does the performance venue take on a different role? Also, what difference does its (visual) design make? How do we perceive tiny spaces that are greatly enlarged, e.g., when an audience of 20 people witnesses a conversation inside a car? What can be meaningfully told with so-called focused sound sources, the sounds that appear in the middle of the room among the audience? The results were multifaceted and enlightening; they revealed some possible paths into the future of audio play, and the students took their first steps in these directions. I was one of them.
The future of a medium, the "added experience value" of a technology, is not always determined by engineers and marketing teams; they can also be shaped by artists who learn to use these technologies as creative tools. In the case of wave field synthesis, artists have started exploring how this technology can reveal new ways of perception and redefine the relationship between the audience and the medium by innovating narrative strategies. However, apart from this research project, the budgets allocated for this essential groundwork are generally too small or nonexistent. As a result, fundamental questions have not yet been sufficiently addressed, which is one reason wave field synthesis has never become relevant within the media landscape. While the future of wave field synthesis seemed promising, IOSONO eventually filed for bankruptcy. The technology is now available from ENCIRCLED audio solutions.
The "added experience value" relates not only to the audio play or wave field synthesis but also to spatial audio as a whole. Dolby's marketing introduced Dolby Atmos to cinemas with the promise of a completely new, three-dimensional experience — essentially creating a virtual reality for the ears. However, while the technology has significant potential, scriptwriters, directors, and rerecording mixers in Hollywood seem to preserve a mental concept of space attached to a rather traditional, front-oriented surround sound aesthetic. Still, the stories unfold in a single spatial dimension. Action-carrying elements, such as dialogue, emanate from the center speaker behind the screen, while the other speakers are chiefly used for decorative effects. Consequently, the innovation of the medium's narrative space since "Star Wars" has been marginal. And if you've browsed through streaming service catalogs or experienced environments like the Sphere in Las Vegas, you might find parallels to the current state of spatial audio in music. Too often, the concrete experience with spatial audio reveals little more than a hollow re-mix of old ideas — a marketing promise that has largely gone unfulfilled.
Read: Experiencing spatiality in electronic music at the SPATIAL festival
So, is the future a re-mix? Flashy gimmickry? What is the fundamental prospect of spatial audio other than the expanded 360° narrative space? If the seminar at the Film University mentioned earlier had taken place every year, where would we be today, after 15 years? It didn't take place. Instead of tasting the future at the celebrated anniversary, we seem to have fallen into a state of disorientation. Let's reflect on what happened at the TiME Lab. Once again, the HHI wanted to present its interesting technology, which was fair enough. The artists, whose pieces were re-mixed for wave field synthesis, presumably provided the commercial research institute with free demo material, which too often is the bitter truth. And the collaboration with a renowned festival, wasn't it all about discovering the future of audio play as an art form? The sheer knowledge about the process of how the works would find their way into the TiME Lab should have raised questions — at least among those with substantial experience. As a result, the listening experience directed our ears not to the future. What instead could be sensed by the ear was the phenomenon of disorientation costly staged through a formerly promising but already vanished spatial audio technology. Even in the past, we have already been further into the future.
If you think in stereo, you design in stereo. Except for spatial thinking, neither a re-mix nor any spatial audio technology will make a difference. A spatial turn requires our lively spatial imagination; technology is of subordinate concern here. It requires our sensibility in listening, capacity to verbalize spatial phenomena, ability to discuss phenomena, and curiosity to experiment. It requires artistic research not only to explore new technical tools but also to challenge familiar aesthetics and narrative strategies.
Yes, it's about money. Nevertheless, as artists, creatives, and producers, we should consider whether we want to make the buzzy marketing language our own. Do we want to remain satisfied with producing attractive images, writing promising announcement texts, and fueling great expectations? Do we only want to talk about those exciting experiences in a form that suggests we have actually realized it? We fake it and never make it? Or do we cultivate a curiosity to explore genuinely new dimensions in spatial audio, redefine the spatiality of auditory media, and tell stories, compose music differently?
Do we only want to describe spatial audio as we wish to experience it in the future, or do we want to actually make that future?
< table of contents
