SoundSpaces 2.0: Realistic Audio Rendering

• SoundSpaces 2.0 is a simulation platform enabling on-the-fly, geometry-based RIR synthesis in arbitrary 3D environments for realistic acoustics.
• It employs a Monte Carlo bidirectional path tracing algorithm to capture frequency-dependent effects and generate accurate, dynamic audio simulations.
• The platform offers configurable microphone arrays, high-speed and high-fidelity modes, and validated performance against real-world acoustics.

SoundSpaces 2.0 (SSv2) is a simulation platform enabling on-the-fly, geometry-based audio rendering within 3D visual environments, designed to advance audio-visual learning in embodied AI. Given a 3D mesh of a real-world environment, SSv2 generates physically realistic acoustics for arbitrary sounds and microphone locations, supporting research tasks such as audio-visual navigation, mapping, source localization, separation, acoustic matching, and far-field automatic speech recognition. SSv2 introduces continuous spatial sampling, instantaneous generalization to novel environments, and configurable microphone and material properties. It operates with computational efficiency suitable for embodied learning and offers high fidelity in room acoustic simulation compared to both prior precomputed and simplified analytic approaches (Chen et al., 2022).

1. Motivation and Primary Contributions

SSv2 was motivated by the challenges in prior acoustic simulators: reliance on precomputed impulse responses (IRs) on coarse grids (as in SoundSpaces 1.0), inflexibility with respect to environmental or hardware parameters, and limited support for real-world geometric complexity (e.g., only "shoebox" rooms in ThreeDWorld). As a response, SSv2 provides:

• On-the-fly, geometry-based computation of room impulse responses (RIRs) in arbitrary 3D meshes.
• Continuous spatial sampling, enabling both source and listener to occupy any position with acoustic continuity along agent trajectories.
• Configurable materials with frequency-dependent acoustic properties and customizable microphone arrays (monaural, binaural, ambisonic, user-defined).
• A high-speed simulation mode (33.5 FPS on 5 CPU threads) alongside a high-fidelity mode with sub-percent RT60 error.
• Public release of the platform, real-world RIR measurements, and the SoundSpaces-PanoIR dataset (10 million image-IR samples).

Supported tasks span audio-visual navigation, embodied mapping, source localization and separation, (de)reverberation, acoustic matching, and far-field ASR (Chen et al., 2022).

2. Acoustic Rendering via Monte Carlo Path Tracing

The core of SSv2 is a Monte Carlo bidirectional path tracing algorithm [Cao et al. ’16] for simulating sound propagation and generating accurate RIRs. For a sound source at position $\mathbf{x}_\mathrm{src}$ and a microphone at $\mathbf{x}_\mathrm{mic}$, the time-domain RIR is represented as:

$$h(\mathbf{x}_\mathrm{mic}, \mathbf{x}_\mathrm{src}, t) = \sum_{p \in \mathcal{P}} A_p \, \delta(t - \tau_p)$$

where each path $p$ contributes an amplitude $A_p$ and arrival time $\tau_p$. Paths are summed across $M$ logarithmically spaced frequency bands, with energy mapped to pressure, stochastic phase injected, and spatialization performed using spherical harmonics, yielding ambisonic or binaural IRs.

The amplitude $A_p(f)$ for each path accounts for:

• Geometric spreading: $1/(4\pi d^2)$ per path segment of distance $d$.
• Material absorption: Frequency-dependent $\alpha(f)$ at each reflection.
• Scattering: Phong scattering with coefficient $s(f)$, dividing energy between specular and diffuse components.
• Transmission: Frequency-dependent $\tau(f)$ applied at each thin barrier crossing.
• Air absorption: Exponential decay $\exp(-\beta(f) d)$ per segment, using the Bass model.

Combined, for a path with $N$ interactions and $N_t$ transmissions across total distance $L_p$:

$$A_p(f) \propto \prod_{i=1}^{N} [\alpha_i(f) (1 - s_i(f))] \times \prod_{j=1}^{N_t}\tau_j(f) \times \exp(-\beta(f) L_p) \big(4\pi L_p^2\big)^{-1/2}$$

This rendering approach supports accurate indirect sound, frequency-dependent effects, and extensibility to arbitrary 3D meshes (Chen et al., 2022).

3. Continuous Sampling and Generalization

Unlike SoundSpaces 1.0's grid-based IR precomputation, SSv2 dynamically synthesizes RIRs for arbitrary source-listener configurations $(s, r)$ and headings $\theta$, returning $h(s, r, \theta)$. As agents traverse environments, SSv2 renders IRs per timestep $\Delta t$ and convolves audio using sliding windows of length $T$, with cross-fading to preserve acoustic continuity and prevent artifacts such as popping.

This design enables:

• Continuous audio-visual feedback for navigation and interaction tasks.
• Instantaneous generalization to any environment, from large-scale scans (Gibson, Replica, HM3D, Ego4D, user-scanned scenes) to analytically simple configurations.

A plausible implication is that SSv2 reduces the gap between simulated and real-world perception by supporting seamless transitions and dynamic scenarios in high-fidelity acoustic spaces (Chen et al., 2022).

4. Configurable Microphone and Material Models

SSv2 exposes extensive configurability to match a wide variety of experimental designs:

• Microphone Arrays: Supports mono, binaural (with custom HRTFs), higher-order ambisonics, surround 5.1/7.1, and arbitrary point-based arrays.
• Material Library: 29 built-in materials (e.g., wood, concrete, carpet, glass), expressed as $\alpha(f)$ (absorption), $s(f)$ (scattering), $\tau(f)$ (transmission) with frequency-coefficient pairs, interpolated linearly between specified values.
• Air Absorption: Follows the Bass model, introducing frequency-dependent attenuation in dB/m.
• Acoustic Randomization: Allows assignment of multiple candidate materials per surface type (e.g., "floor," "wall") with additive Gaussian noise ($\mathcal{N}(0, 0.1)$) for coefficients, promoting domain robustness.

This extensive configuration supports both precise experimental control and statistical coverage of real-world variability (Chen et al., 2022).

5. Implementation Architecture and Computational Performance

SSv2 is architected in C++ with a multi-threaded RLR-Audio-Propagation engine integrated with Habitat-Sim for visual rendering. The platform leverages bounding volume hierarchies (BVH) for efficient ray–triangle intersection and employs temporal coherence for IR reuse in high-speed simulation mode. Output is CPU-only, ensuring broad accessibility.

Performance benchmarks on a Xeon Gold 6230 (2.1 GHz):

Mode	FPS (1 thread)	FPS (5 threads)	Relative RT60 Error
High-quality	0.9	4.0	Baseline
High-speed	7.7	33.5	9.5%

By comparison, SoundSpaces 1.0 precomputes at 500 FPS (fixed grid/materials), and ThreeDWorld achieves approximately 60 FPS but only for shoebox geometries. The high-speed mode in SSv2 trades a minor increase in RT60 error for a substantial increase in throughput (Chen et al., 2022).

6. Empirical Validation: Comparison with Real-World Acoustics

SSv2's simulated acoustics were benchmarked against real RIR measurements in the Replica FRL apartment at seven spatial positions, acquired via exponential sine-sweep (100 Hz–8 kHz) and B&K/Earthworks equipment. Metrics revealed:

• RT60 error: Both SSv1 and SSv2 show approximately 12.4% relative error.
• Direct-to-Reverberant Ratio (DRR) error: Improved from 11.0 dB (SSv1) to 0.98 dB (SSv2).
• Energy-decay curve matching (250–4000 Hz): SSv2 aligns more closely to ground truth.

This demonstrates substantially improved physical realism for models of reverberation and direct/reflected sound balances (Chen et al., 2022).

7. Applications, Empirical Results, and Research Directions

Downstream Tasks:

• Continuous Audio-Visual Navigation: Agents navigate to a sounding target, evaluated with success rate (SR), success weighted by path length (SPL), and distance-to-goal (DTG). Training on SSv2 (continuous space and sound) increased SR from 0.9% (SSv1 grid) to 64.7%, and SPL from 0.9% to 49.3%. Baseline discrete-grid navigation achieved 64% SR but only 27.5% SPL when transferred to continuous mode.
• Far-field Automatic Speech Recognition: A transformer-based SpeechBrain ASR model trained with synthetic RIR augmentation from SSv2 achieved a test WER of 12.48% (or 12.04% with acoustic randomization), outperforming models finetuned on real IRs (13.32%), Pyroomacoustics (16.24%), and SSv1-generated RIRs (18.48%).

Model/Finetuning	WER (%)
Pretrained (no augmentation)	29.10
Real IRs (BUT ReverbDB)	13.32
Pyroomacoustics (shoebox)	16.24
SoundSpaces 1.0	18.48
SoundSpaces 2.0	12.48
+Acoustic randomization	12.04

Limitations and Future Directions

• Current limitations: Requires high-quality, watertight meshes; geometric acoustics does not capture low-frequency (sub-100 Hz) modes precisely; real-world material properties may require on-site measurement for further fidelity.
• Opportunities: Integration of neural acoustic fields conditioned on visual and geometric inputs; further sim-to-real benchmarking (audio-visual SLAM, sound separation, AR matching); extension to dynamic scenes and moving microphones; hybridization with wave-based simulation for low-frequency phenomena; utilization of the PanoIR dataset for self-supervised visual-acoustic learning (Chen et al., 2022).

A plausible implication is that SSv2 serves as a comprehensive testbed for unified audio-visual perception, promoting research into perceptual AI capable of grounding decisions in both spatial vision and acoustics.