DynFOA: Conditional FOA from 360° Video

• DynFOA is a framework for synthesizing four-channel first-order ambisonics by integrating dynamic scene reconstruction with conditional diffusion to capture spatially coherent audio cues.
• It conditions audio generation on 3D geometry, material properties, and acoustic context derived from 360° video, overcoming limitations of visual-only approaches.
• Empirical evaluations on benchmarks like Sphere360 and M2G-360 demonstrate significant gains in spatial accuracy and acoustic fidelity compared to existing methods.

DynFOA is a framework for generating first-order ambisonics (FOA) from dynamic 360-degree video by combining dynamic scene reconstruction with conditional diffusion. Its central premise is that spatial audio in immersive panoramic media should not be inferred from visual appearance alone, because perceived sound depends not only on source location but also on scene geometry, surface materials, occlusion, reflections, reverberation, and listener viewpoint. In the published formulation, DynFOA reconstructs a 3D scene from video, derives physically grounded acoustic context, and conditions a latent diffusion model to synthesize four-channel FOA that remains spatially coherent under head rotation (Luo et al., 3 Apr 2026).

1. Problem formulation and target representation

DynFOA is posed as a response to a specific limitation in prior 360-degree video-to-audio systems: methods such as OmniAudio, ViSAGe, and related diffusion-based or video-conditioned approaches are described as relying mainly on global 2D visual cues, source-location heuristics, or simple distance-based attenuation, and therefore struggling in dynamic and acoustically complex scenes (Luo et al., 3 Apr 2026). The motivating failure cases include moving sound sources, multiple overlapping sources, view-dependent occlusion, reflective boundaries, reverberant rooms, and material-dependent propagation.

The target representation is first-order ambisonics. In the paper’s formulation, FOA uses four channels, typically $W, X, Y, Z$: one omnidirectional component and three directional components aligned with orthogonal axes. This representation is used because it is compact, can be rotated with head motion, and can later be rendered binaurally through HRTFs. DynFOA therefore treats FOA not as generic synchronized audio, but as a spherical-harmonic description of a 3D sound field that must remain spatially consistent under listener head rotation.

The conditioning extracted from the input video $V$ is written as

$c(V)=\{G(V), M(V), R(V)\},$

where $G(V)$ denotes reconstructed 3D geometry, $M(V)$ per-surface material properties, and $R(V)$ reverberation and reflection parameters. The intended mapping is written as

$f_\theta: (V, c(V)) \mapsto {S}_{\text{4D},$

although the printed equation is truncated in the manuscript. An earlier arXiv version presents a closely related formulation as

$f_\theta: (V, G, M, R, \mathbf{o}) \mapsto S_{\text{4D},$

explicitly including listener head orientation $\mathbf{o}$ and describing the system as based on “dynamic acoustic perception and conditional diffusion” (Luo et al., 6 Feb 2026).

2. Architectural organization

The later formulation organizes DynFOA as a three-part pipeline: a Video Encoder, a FOA Latent Encoder, and a Conditional Diffusion Generator (Luo et al., 3 Apr 2026). During training, ground-truth FOA is encoded into a latent target, while the video stream is processed to infer source activity, depth, semantics, geometry, materials, and acoustic context. At inference, only the video-derived conditioning is used to sample a latent FOA representation, which is then decoded into waveform audio.

The Video Encoder begins with dynamic sound-source detection and localization. For each candidate source $i$, the model predicts a bounding box $V$0 and an activity score $V$1, using

$V$2

where $V$3 is the ground-truth source box, $V$4 the predicted box, $V$5 the true activity label, and $V$6 the predicted activity. The paper describes this objective as jointly supporting localization and identification of whether an object is currently sounding.

Depth is estimated from the 360-degree imagery and back-projected into 3D by

$V$7

where $V$8 are image coordinates, $V$9 is the estimated depth, and $c(V)=\{G(V), M(V), R(V)\},$0 are elevation and azimuth. In the panoramic setting, this converts directional image measurements into 3D points that can later support geometric visibility checks and propagation reasoning.

An earlier version describes the system as consisting of a Video Encoder, an Audio Encoder, and a Conditional Diffusion Generator, and adds that the audio branch captures spatial motion and temporal 4D sound-source trajectories to fine-tune the diffusion generator (Luo et al., 6 Feb 2026). This suggests a stable core architecture across versions, with changes in module naming and emphasis rather than a complete reformulation.

3. Scene reconstruction and acoustic context

A defining component of DynFOA is its use of a reconstructed 3D scene as an acoustic prior rather than as a purely visual rendering asset. The paper states that large-scale structure is represented with a hybrid of TSDF and 3D Gaussian Splatting, with 3DGS used for fine details (Luo et al., 3 Apr 2026). The role of 3DGS is to provide explicit 3D spatial structure and surface support onto which semantics and material labels can be aggregated, enabling depth-consistent visibility testing, geometric path estimation, and material assignment.

Semantic segmentation is applied to scene elements such as walls, floors, and furniture, and each semantic class is mapped through a fixed lookup table to frequency-dependent absorption properties. The method therefore does not estimate true acoustic impedance directly. Instead, it uses semantics as a proxy for acoustic material class. The authors explicitly note that this approximation is imperfect, but it permits material-aware distinctions between surfaces with different reflective or absorptive tendencies.

The physically grounded features derived from the reconstructed scene include visibility and occlusion features, reflection cues, and reverberation cues. Occlusion masks are obtained by checking visibility between estimated source directions and the listener through the reconstructed geometry, for example with depth-consistent ray casting. Early reflections are estimated by tracing geometric paths in the scene. Late reverberation is encoded with frequency-dependent $c(V)=\{G(V), M(V), R(V)\},$1 curves. These features are intended to distinguish free-field propagation from propagation altered by room shape, barriers, and material damping.

The manuscript presents a path-gain expression as

$c(V)=\{G(V), M(V), R(V)\},$2

which is visibly malformed in the paper. The surrounding explanation makes the intended meaning clear: $c(V)=\{G(V), M(V), R(V)\},$3 is the absorption coefficient of material $c(V)=\{G(V), M(V), R(V)\},$4 along the path, $c(V)=\{G(V), M(V), R(V)\},$5 is propagation distance, and $c(V)=\{G(V), M(V), R(V)\},$6 is an air-attenuation factor. In prose, the path amplitude is reduced multiplicatively by encountered materials and exponentially by travel distance. This is the paper’s main explicit acoustic attenuation relation and the principal formal link between reconstructed geometry/materials and FOA generation.

4. Latent FOA modeling and conditional diffusion

The FOA Latent Encoder is used only during training. It takes the four FOA channels $c(V)=\{G(V), M(V), R(V)\},$7, applies channel-wise z-score normalization, and encodes them into a latent target representation for diffusion learning (Luo et al., 3 Apr 2026). The encoder is described as CNN-based over the time-frequency domain and is intended to capture both harmonic content and inter-channel directional structure. The paper also states that FOA features are projected onto a spherical harmonic basis to strengthen spatial alignment, although the basis equations themselves are not written out.

An attention-based acoustic-prior injection module is introduced through the printed expression

$c(V)=\{G(V), M(V), R(V)\},$8

which is again malformed in the manuscript. The documented semantics are that a learned gate uses encoded FOA features and a visual prior to emphasize perceptually salient acoustic information.

The core generator is a latent conditional diffusion model. Let $c(V)=\{G(V), M(V), R(V)\},$9 be the latent encoding of ground-truth FOA. DynFOA learns $G(V)$0 by denoising noisy latent variables $G(V)$1 with the standard noise-prediction objective

$G(V)$2

whose braces are also incompletely printed in the paper. Architecturally, the denoiser is a Multi-Conditional U-Net with conditioning projections, modulation layers, and cross-attention. Geometry, materials, source features, occlusion masks, early reflection cues, and late reverberation profiles are projected into a shared latent space and injected into the reverse process.

At inference, the FOA Latent Encoder is not used. The Video Encoder produces scene, source, material, and acoustic features; the diffusion model samples a latent FOA representation conditioned on those features; and a pretrained VAE decoder converts the latent back into 16 kHz four-channel FOA waveform audio. To accelerate generation, the implementation uses DPM-Solver++ and reduces sampling from a nominal 1000-step reverse process to 50 denoising steps. The generated FOA can then be rotated according to the listener’s head orientation in the spherical harmonic domain and rendered binaurally with a fixed CIPIC HRTF set converted to SOFA format. The paper further notes bilinear interpolation across discrete HRTF directions, diffuse-field equalization, and headphone compensation for Sennheiser HD 650 playback. No subject-specific HRTF personalization is used.

The reported training setup is correspondingly heavy. The system is implemented in PyTorch and trained on 8 A100 GPUs with 80 GB memory each. Because reconstruction and audio generation together are memory intensive, the authors do not train the full system end to end. Video Encoder features and FOA latent targets are precomputed offline and frozen, and only the Multi-Conditional Encoder and conditioning projections inside the diffusion generator are optimized during the main stage. The conditional U-Net is trained for 500,000 steps with effective global batch size 128 using AdamW, linear warm-up, cosine annealing, automatic mixed precision, and exponential moving average (Luo et al., 3 Apr 2026).

5. Evaluation protocols and reported performance

The later paper introduces M2G-360, a dataset of 600 real-world 360-degree clips designed as a robustness benchmark for physically complex acoustic conditions (Luo et al., 3 Apr 2026). Each clip is normalized to 10 seconds, H.264 encoded, at least 720p resolution, 30 FPS, and paired with 4-channel FOA audio at 16 kHz and 16-bit depth. The dataset is divided into MoveSources with 128 clips, Multi-Source with 107 clips, and Geometry with 365 clips. The authors state that main training follows the official Sphere360 partition, while evaluation uses the Sphere360 test remainder and all of M2G-360.

Baselines are ViSAGe, Diff-SAGe, MMAudio plus a spherical-harmonic spatialization module, and OmniAudio. Evaluation covers spatial accuracy by DOA error, acoustic fidelity by SNR and EDT, distribution matching by Fréchet Distance, KL divergence, STFT error, and SI-SDR, and human perceptual quality by MOS-SQ and MOS-AF. The perceptual study uses 24 participants in randomized double-blind evaluation on a Meta Quest 3 with HD 650 headphones.

On Sphere360, the paper reports that DynFOA outperforms all baselines on all reported metrics. Relative to OmniAudio, DOA error is reduced from 0.19 to 0.14, SNR improves from 16.85 to 18.52, EDT decreases from 0.06 to 0.04, FD decreases from 0.14 to 0.10, KL from 0.31 to 0.21, STFT error from 0.21 to 0.14, and SI-SDR increases from 12.68 to 14.85. Subjective results improve from $G(V)$3 to $G(V)$4 on MOS-SQ and from $G(V)$5 to $G(V)$6 on MOS-AF.

The M2G-360 results are presented as especially diagnostic. On MoveSources, DynFOA reduces DOA from 0.15 to 0.08 relative to OmniAudio, improves SNR, FD, KL, STFT, SI-SDR, and both MOS scores, and is reported as achieving a 46.7% DOA improvement. On Multi-Source, DOA decreases from 0.18 to 0.12, FD from 0.12 to 0.08, KL from 0.26 to 0.19, STFT from 0.19 to 0.12, and SI-SDR rises from 12.87 to 14.47. On Geometry, EDT decreases from 0.05 to 0.03, SI-SDR rises from 12.34 to 15.02, and subjective spatial quality increases from $G(V)$7 to $G(V)$8. The paper interprets these results as evidence that explicit geometry and material conditioning improve room-acoustic realism in cases where visual-only conditioning is insufficient.

The ablation studies localize the reported gains. A scene-information ablation progresses from audio-only conditioning to audio plus visual detection, then depth, then full geometry, with DOA dropping from 0.26 to 0.22 to 0.18 to 0.14, EDT from 0.09 to 0.07 to 0.05 to 0.04, and FD from 0.22 to 0.18 to 0.14 to 0.10. A diffusion ablation compares regression, simple diffusion, conditional diffusion, and conditional diffusion with more sampling steps; the full model reaches DOA 0.14, SNR 18.52, EDT 0.04, and FD 0.10. A propagation-modeling ablation starts from base geometry under free-field assumptions and then adds occlusion masks, early reflections, and late reverberation, with the full system reaching DOA 0.14, SNR 18.52, EDT 0.04, and FD 0.10. The stated interpretation is that diffusion improves temporal and spectral coherence, but geometry-aware propagation modeling is central to the strongest spatial and acoustic gains.

6. Limitations, manuscript caveats, and arXiv versioning

DynFOA is not presented as a full physical acoustic simulator. The authors explicitly note that material property estimation is only approximate because it comes from semantic segmentation and a lookup table rather than direct acoustic measurement (Luo et al., 3 Apr 2026). Reconstruction quality is therefore a bottleneck, and errors in depth, semantics, or source localization can propagate into the acoustic conditioning. The method also appears to depend on visible video evidence for source detection and scene understanding, so fully hidden or ambiguous sources remain difficult.

The framework is computationally heavy. The published implementation relies on offline feature precomputation, frozen encoders, and large-scale GPU resources, and is not trained end to end. This is not merely an engineering detail: it indicates that DynFOA’s acoustic realism is coupled to the quality and stability of several upstream perception modules. A plausible implication is that deployment quality may depend as much on reconstruction fidelity as on the diffusion generator itself.

There are also manuscript-level caveats. Several displayed equations are visibly malformed or truncated, including the mapping $G(V)$9, the path-attenuation equation, the acoustic-prior injection equation, and the diffusion loss. These typesetting issues do not erase the conceptual structure, but they do complicate exact reproduction from the text alone. A further methodological caveat is that Sphere360 evaluation uses “proxy FOA” supervision where the original audio track is projected onto FOA basis functions for relative comparison when ground truth is partially absent; the authors state that this is intended for relative benchmarking rather than absolute accuracy. The paper itself suggests that the strongest evidence for DynFOA’s advantages comes from the M2G-360 robustness benchmark and the perceptual study.

The arXiv record also shows a meaningful version distinction. The February 2026 version describes DynFOA as based on “dynamic acoustic perception and conditional diffusion,” includes an explicit Audio Encoder, and names the 600-clip benchmark Dyn360 (Luo et al., 6 Feb 2026). The April 2026 version presents M2G-360, gives a more detailed account of the hybrid TSDF–3DGS reconstruction, the conditioning variables $M(V)$0, the implementation pipeline, and the robustness-oriented evaluation (Luo et al., 3 Apr 2026). Across both versions, however, the stable conceptual core is the same: FOA generation from 360-degree video is treated as a geometry- and material-conditioned generative problem rather than a purely appearance-driven audiovisual translation task.