MIRAGES: Modeling and Immersive Rendering of Archaeoacoustics in Grotto Environments

MIRAGES is an associate team between Emeraude and the Center for Computer Research in Music and Acoustics (CCRMA) at Stanford University (USA). It was launched in Spring 2025.

Context

As computers grew more powerful in the 2000s, the prospect of creating immersive virtual acoustic experiences and soundscapes became a reality [O8]. While making accurate mathematical models of acoustical spaces (e.g., concert halls, churches, caves, etc.) remains a difficult task [O7], it is now possible to create immersive experiences that faithfully reproduce the acoustics of existing spaces by using the well established Impulse Response (IR) measurement and convolution technique [O4, O8]. Overall, interest around virtual acoustics has been booming in recent years [O3], reinforced by the needs of Virtual Reality (VR) [O5], and countless concert venues [O6], museums [O2], and academic institutions [O1] are looking to provide engaging, immersive experiences.

Rendering virtual acoustics using IRs is well understood and documented through an extensive literature [O4, O8, O7]. For example, there are numerous audio plug-ins available, both commercial and open source, to implement convolutional reverberation in stereo within Digital Audio Workstations (DAW) [M1]. On the other hand, much improvement is needed to effectively implement immersive and collective virtual acoustics experiences in real-time (i.e., using dense networks of speakers), largely due to the cost and complexity, and the computational requirements of current systems.

The Chauvet Cave constitutes a formidable case study in this context [M8] thanks to the large amount of data already acquired as part of the What Did They Hear? project (see §1.2) and the acoustical model of the cave that resulted from it. The Chauvet cave is also a very unique space hosting some of the oldest paintings of humanity (see Figure 2). Immediately after its discovery in 1994, its access was restricted to scientists and conservation teams, creating the need for a replica (“Chauvet 2”), which opened to the public in 2015. Despite its importance to human experience, this replica completely omits the acoustics of the original cave, a limiting factor that MIRAGES will help fix.

Objectives

The objectives of MIRAGES are:

1. Improving virtual acoustics rendering techniques to make them more immersive.
2. Investigating the use of Field-Programmable Gate Array (FPGA) platforms to improve the performance and the quality of collective real-time virtual acoustics rendering systems.
3. Implementing an accurate, immersive, and interactive simulation of Chauvet Cave using techniques resulting from work toward the objectives above.

Thanks to the FAST ANR project (ANR-20-CE38-0001) on “Fast Audio Signal-processing Technologies on FPGA,” much progress has been made in the field of real-time audio DSP on FPGA [M6]. It also demonstrated the potential of FPGA platforms in the context of spatial audio by allowing us to manage hundreds of audio channels in parallel at a very low cost [M3,M5]. In parallel with this, thanks to the work carried out as part of PLASMA, the prospect of distributing computation in the context of high-density (hundreds of speakers) spatial audio systems has started to be explored [M7]. One of the aims of MIRAGES is to combine the findings of these two projects to provide scalable real-time virtual acoustics rendering systems with unparalleled computational power and level of immersion, at a very low cost. This system will allow us to manage hundreds (potentially thousands) of speakers and will hence be suited for advanced spatial audio techniques such as Wave-Field Synthesis (WFS) [O7].

Beyond cave acoustics modeling and its auralization, the breakthroughs realized as part of MIRAGES will significantly impact the field of acoustics and spatial audio, both for specialized (i.e., automotive, VR platforms, movie theaters, etc.) and general public applications (i.e., home cinema, etc.). It could also lead to larger scale projects/collaborations on this topic.

Work-Program

Virtual Acoustics and Spatial Audio Techniques on FPGA

This work package focuses on optimizing state-of-the-art virtual acoustics techniques as well as the models that will be designed as part of the previous work package for FPGAs. FPGAs can provide significant performance gains compared to traditional platforms used for real-time audio processing such as CPUs. However, these don’t usually come “out of the box” and various techniques must be used to optimize code to fully take advantage of the parallelization and pipelining capabilities of FPGAs. Two elements must be considered here: (i) our ability to run virtual acoustics models in real-time and (ii) “deploying” the result on a spatial audio system using various sound spatialization techniques such as WFS or Ambisonics [O8]. Since rooms are linear systems, real-time rendering of the acoustics of a room (whether it is modeled or measured) can always be done through a convolution of its impulse response with an input signal. Since convolutions are usually much cheaper from a computational standpoint than running an acoustical physical model, they’re usually preferred for real-time rendering [O8]. It is possible that this might be less true in the context of FPGAs, hence this is something that we would like to consider as well. To reach these goals, this work package will rely on the work carried out as part of the FAST ANR project on facilitating the programming of FPGAs for real-time audio DSP applications. Eventually, we hope to be able to run the model of the Chauvet cave that was designed as part of the What Did They Hear? project on FPGAs.

FPGA-Based Auralization and Spatial Audio System

This work package aims at deploying the techniques developed in A on a network of FPGA boards to target a large number of speakers while providing an extended amount of computational power. The main idea is therefore to combine the power of distributed spatial audio systems [M7] with that of FPGAs. Since a single FPGA can be used to manage and produce hundreds of audio channels in parallel [M5], the scope and size of our kind of system will be much bigger than what’s currently possible. Additionally, the combination of these two approaches should help us further reduce the “cost per channel” of such system.

A regular computer (i.e., laptop) will be responsible both for transmitting unprocessed audio streams (the sound sources) to the system and for controlling it (the dynamic positioning of the sources). Both audio streams and control messages will be transmitted through Ethernet [M2]. The main difficulty for reaching this goal is to achieve perfect synchronicity (both in terms of clock and of latency) among all the FPGAs in the network. For this, we plan to adapt the work that Thomas Rushton is currently pursuing as part of his PhD within Emeraude on deploying distributed spatial audio systems using networks of microcontrollers.

References

[M1] Nima Farzaneh et al. “Exploring the past with virtual acoustics and virtual reality”. In: Proceedings of the 2023 Immersive 3D Audio: from Architecture to Automotive Workshop (I3DA-23). Bologna, Italy, 2023.

[M2] Fernando Lopez-Lezcano. “From Jack to UDP packets to sound, and back”. In: Proceedings of the 2012 Linux Audio Conference (LAC-12). Stanford University, USA, 2012.

[M3] Romain Michon et al. “Making Frugal Spatial Audio Systems Using Field-Programmable Gate Arrays”. In: Proceedings of the 2023 New Interfaces for Musical Expression Conference (NIME-23). Mexico City, Mexico, May 2023.

[M4] Yann Orlarey, Stéphane Letz, and Dominique Fober. “New Computational Paradigms for Computer Music”. In: ed. by Gerard Assayag. Paris, France: Delatour, 2009. Chap. Faust: an Efficient Functional Approach to DSP Programming.

[M5] Maxime Popoff, Romain Michon, and Tanguy Risset. “Enabling affordable and scalable audio spatialization with multichannel audio expansion boards for FPGA”. In: Proceedings of the 2024 Sound and Music Computing Conference (SMC-24). Porto, Portugal, 2024.

[M6] Maxime Popoff et al. “Towards an FPGA-Based Compilation Flow for Ultra-Low Latency Audio Signal Processing”. In: Proceedings of the 2022 Sound and Music Computing Conference (SMC-22). Saint-Étienne, France, June 2022.

[M7] Thomas Albert Rushton, Romain Michon, and Stéphane Letz. “A Microcontroller-Based Network Client Towards Distributed Spatial Audio”. In: Proceedings of the 2023 Sound and Music Computing Conference (SMC-23). Stockholm, Sweden, June 2023.

[M8] Luna Valentin, Miriam Kolar, and Philippe Monteil. “Speleoacoustics in Southern Ardèche for Auralizations and Music Experimentation”. In: Proceedings of the 2022 Sound and Music Computing Conference (SMC-22). Saint-Étienne, France, 2022.

[O1] AlloSphere at UCSB. Project Website. https://allosphere.ucsb.edu/.

[O2] Immersive Sound System at Paris Philharmonie. Project Website. https://philharmoniedeparis.fr/fr/philharmoniedesenfants/activite/plein-les-oreilles.

[O3] Notre Dame Acoustics. Article Website. https:/ /www.nytimes.com/interactive /2023/03/03/magazine/notre-dame-cathedral-acoustics-sound.html. 2023.

[O4] Bissera Pentcheva and Jonathan Abel. “Icons of Sound: Auralizing the Lost Voice of Hagia Sophia”. In: Speculum 92.S1 (2017), pp. 336–360.

[O5] Anders Riddershom Bargum et al. “Spatial Audio Mixing in Virtual Reality”. In: (2022), pp. 269–302.

[O6] Satosphere in Montreal. Project Website. https://sat.qc.ca/en/.

[O7] Lauri Savioja and Peter Svensson. “Overview of Geometrical Room Acoustic Modeling
Techniques”. In: The Journal of the Acoustical Society of America 138.708 (2015).

[O8] Michael Vorländer. Auralization: Fundamentals of Acoustics, Modelling, Simulation, Algorithms and Acoustic Virtual Reality. Springer Nature, 2020.