---
title: 'SwanSphere: Unified Spatial Audio Framework'
url: https://www.emergentmind.com/topics/swansphere
type: topic
---

# SwanSphere: Unified Spatial Audio Framework

SwanSphere is a unified framework for streaming synchronized spatial audio generation from panoramic videos and text prompts. Its output is First-Order Ambisonics (FOA), a 4-channel spatial audio representation \(\mathbf{a}\in\mathbb{R}^{C\times L}\) with channels \(W,X,Y,Z\). The framework addresses a specific systems problem in immersive audio generation: diffusion models can provide high fidelity but incur high inference latency, whereas autoregressive or token-based methods can reduce delay but often lose audio fidelity or spatial precision. SwanSphere is formulated to reduce this quality–latency tradeoff while improving the capture of direction, motion, depth, and temporal synchronization in \(360^\circ\) scenes [2605.30940].

## 1. Problem setting and representation

SwanSphere is designed for two conditional generation regimes: video-to-spatial audio and text-to-spatial audio. In both cases, the target is synchronized FOA rather than monaural audio followed by a separate spatialization stage. This distinction is central because FOA must preserve not only semantic event content but also where the sound comes from, how its orientation evolves, and whether temporal changes across \(W,X,Y,Z\) remain physically plausible [2605.30940].

The framework uses a continuous FOA VAE rather than a discrete codebook. For input audio \(\mathbf{a}\), the latent sequence is defined as
\[
\mathbf{z} = \mathcal{E}(\mathbf{a}), \qquad \mathbf{z}\in\mathbb{R}^{d\times l}.
\]
The VAE is based on Stable Audio VAE, adapted for 4-channel FOA. The stated motivation is to avoid quantization errors and phase loss from discrete tokenization. Generation is then factorized patchwise as
\[
p(\mathbf{z}_{1:T}\mid c) = \prod_{t=1}^{T} p(\mathbf{z}_t \mid \mathbf{z}_{<t}, c),
\]
where the condition \(c\) may include panoramic video features, text caption embeddings, or both [2605.30940].

The underlying difficulty of streaming FOA generation is presented as a three-part constraint. First, the model must reconcile long-range event structure with local waveform or latent detail. Second, it must maintain spatial consistency across FOA channels. Third, it must overcome weak cross-modal alignment in generic video encoders that often miss geometry and acoustic cues in panoramic video. SwanSphere addresses these constraints by separating semantic planning from local rendering and by explicitly aligning audio and panoramic-video representations [2605.30940].

## 2. Causal autoregressive diffusion transformer

The core architecture is a causal autoregressive diffusion transformer organized as a two-stage patchwise process. In Stage 1, an autoregressive “spatial LM” predicts a semantic condition \(h_t\) for the current patch from previous audio context and multimodal input. In Stage 2, a Local Diffusion Transformer (LocDiT) renders the actual FOA latent patch conditioned on \(h_t\), recent history, and boundary context [2605.30940].

| Component | Role |
|---|---|
| AR “spatial LM” | Patchwise semantic planning |
| LocDiT | High-fidelity FOA patch generation |
| History encoder | Summarizes generated audio context |
| Boundary context | Supplies preceding patches for continuity |

Streaming is implemented with fixed-size latent patches. The reported configuration is a patch size of **4 latent frames**, temporal stride **4**, causal context window of **2 patches** \(=8\) latent frames, and **20 diffusion steps per patch** for LocDiT. The architecture only needs enough context to generate the current chunk, which permits playback once the first patch is produced [2605.30940].

Synchronization is maintained through four explicit mechanisms. The current semantic prediction \(h_t\) is conditioned on aligned video tokens and a historical spatial-audio summary from the history encoder. LocDiT receives the **two preceding patches** as boundary context. Generation terminates when the video feature sequence ends rather than through a stop token, tying audio duration directly to input-video duration. For text-only generation, FLAN-T5 embeddings are injected via cross-attention, while missing modalities are represented by learnable null embeddings so that a single architecture supports video only, text only, and video + text conditioning [2605.30940].

The first-chunk latency is broken down into **0.03 s** for the spatial LM, **0.14 s** for LocDiT denoising, and **0.04 s** for video encoding and audio decoding, yielding a total **time-to-first-chunk = 0.21 s**. The total inference time reported for the full system is **9.13 s** [2605.30940].

## 3. Spatial Video-Audio Contrastive learning

Spatial Video-Audio Contrastive learning (SVAC) is the representation-alignment component of SwanSphere. It aligns panoramic video features and FOA audio features in a shared space so that the video encoder becomes acoustically and spatially aware rather than merely semantically informative. The video encoder is **VideoMAE-V2**, selected because it preserves spatial structure better than CLIP-style global representations, and the audio encoder is **AudioMAE**. Because audio has finer temporal resolution than video, video features are aligned to the audio timeline by **nearest-neighbor replication** rather than interpolation [2605.30940].

The training objective is a symmetric InfoNCE loss:
\[
\mathcal{L}_{\text{total}} = \frac{1}{2N} \sum_{i=1}^N \left( \mathcal{L}_{\text{NCE}}(v_i, a_i, \mathcal{N}_i^{a}) + \mathcal{L}_{\text{NCE}}(a_i, v_i, \mathcal{N}_i^{v}) \right),
\]
with
\[
\mathcal{L}_{\text{NCE}}(q, k^+, \mathcal{M}) =
-\log \frac{\exp(\text{s}(q,k^+)/\tau)}
{\exp(\text{s}(q,k^+)/\tau)+\sum_{m\in \mathcal{M}} \exp(\text{s}(q,m)/\tau)}.
\]
Here \(\text{s}(\cdot)\) is temporally aligned cosine similarity, \(\tau\) is a temperature, and \(N\) is batch size [2605.30940].

The distinctive feature of SVAC is its physics-aware negative construction. Four contrastive views are used: instance exchange, temporal offset, audio rotation, and video rotation. The negative sets are
\[
\mathcal{N}_i^a = \{a_j\}_{j\neq i} \cup \{\tilde a_i^{time}\} \cup \{\tilde a_i^{spat}\},
\]
\[
\mathcal{N}_i^v = \{v_j\}_{j\neq i} \cup \{\tilde v_i^{spat}\},
\]
where \(\tilde a_i^{time}\) is temporally shifted audio, \(\tilde a_i^{spat}\) is spatially rotated FOA audio, and \(\tilde v_i^{spat}\) is horizontally rotated video. This design teaches the encoders to distinguish matched from mismatched semantics, aligned from time-shifted events, and original from rotated spatial layouts [2605.30940].

The ablation data attribute measurable gains to these physics-aware negatives.

| Variant | FD | KL | Angular |
|---|---:|---:|---:|
| Full SVAC | 120.28 | 1.36 | 1.03 |
| Semantic-only | 127.12 | 1.41 | 1.12 |
| CLIP baseline | 140.28 | 1.44 | 1.34 |

These results indicate that semantic-only contrastive learning helps, but the temporal and spatial hard negatives contribute additional gains in both audio quality and localization accuracy [2605.30940].

## 4. Online Direct Preference Optimization

After pretraining, SwanSphere is refined with multi-objective online Direct Preference Optimization (ODPO). The stated aim is to bias generation toward outputs that are spatially accurate, semantically consistent, and perceptually clean or artifact-free. The paper adopts ODPO rather than RL methods such as GRPO because the sampling-and-ranking pipeline naturally yields preference pairs and is described as more stable and lightweight [2605.30940].

For each input, the model generates **8 candidate audios in parallel**. These are ranked with a weighted reward
\[
R = \lambda_{\text{spatial}} R_{\text{spatial}} + \lambda_{\text{semantic}} R_{\text{semantic}} + \lambda_{\text{fidelity}} R_{\text{fidelity}},
\]
using weights \(\lambda_{\text{spatial}}=0.4\), \(\lambda_{\text{semantic}}=0.4\), and \(\lambda_{\text{fidelity}}=0.2\). The spatial reward is based on azimuth, elevation, and spatial-angle errors between generated and reference sound fields; the semantic reward is computed from ImageBind similarity between generated audio and video or text embeddings; the fidelity reward uses Audiobox Aesthetics. All reward components are normalized to \([0,1]\) [2605.30940].

ODPO contributes substantially to final quality. The reported ablation for SwanSphere-L gives **FD 133.91** and **angular 1.22** without ODPO, versus **FD 120.28** and **angular 1.03** for the full model. Independent spatial evaluation with weighted cosine similarity (wCS) from PSELDNets shows the same pattern: **0.41** for OmniAudio, **0.52** for the model without ODPO, and **0.63** for the full system [2605.30940].

A common simplification is to treat preference optimization here as a generic perceptual fine-tuning stage. The reported reward design indicates a narrower role: ODPO is explicitly multi-objective, and spatial accuracy is weighted equally with semantic alignment rather than subordinated to fidelity. This suggests that preference tuning in SwanSphere is part of the localization stack rather than a purely cosmetic post-process.

## 5. Spatial caption annotation, datasets, and training

SwanSphere includes an automated spatial caption annotation pipeline motivated by the scarcity of spatial audio–text datasets. The pipeline has three stages. First, it extracts acoustic spatial features from FOA intensity vectors in the STFT domain:
\[
I_x = \text{Re}\{W^* \cdot X\}, \quad
I_y = \text{Re}\{W^* \cdot Y\}, \quad
I_z = \text{Re}\{W^* \cdot Z\}.
\]
These are aggregated into a spatial vector \(\vec{V}=[V_x,V_y,V_z]^T\) using an energy-weighted average over the **500–8000 Hz** band. Azimuth and elevation are then computed as
\[
\text{Azimuth} = \arctan2(V_y, V_x),
\]
\[
\text{Elevation} = \arcsin\left(\frac{V_z}{\sqrt{V_x^2+V_y^2+V_z^2}}\right).
\]
Second, the angular trajectory is smoothed by converting angles to unit vectors and applying a moving average with window size **3**. Third, the structured trajectory data, panoramic video, and audio are sent to **Gemini 2.5 Pro** to generate captions describing source content, location, timing, and movement or dynamics. The prompt explicitly encodes the coordinate system, including \(0^\circ\) = front, positive azimuth = left, negative azimuth = right, and \(\pm180^\circ\) = rear [2605.30940].

The captioned subset contains about **3,100 valid captioned samples**, with **300** reserved for evaluation. The main dataset contains **165,000 video-audio pairs**, about **458 hours**, in **10-second clips**, with **5%** held out as a test set with no overlapping video IDs. Its sources are Sphere360, YT-Ambigen, newly collected YouTube FOA clips, and additional web-crawled and filtered data. For curriculum learning, the framework also uses a **1M-sample non-spatial audio pretraining set** built from AudioCaps, VGGSound, WavText5k, and AudioSet, converted into a pseudo-FOA format for early training [2605.30940].

The training recipe is explicitly staged. The FOA VAE uses batch size **80** on **2 H800 GPUs**, with **200k steps** for encoder or initial training and then **300k steps** for the decoder, using weighted multi-resolution STFT, KL, and adversarial or discrimination losses, with the \(L_1\) loss weight set to zero. SVAC trains only projection layers for **100k steps** on **2 H800 GPUs** at learning rate \(1\times 10^{-5}\), while keeping VideoMAE-V2 and AudioMAE frozen. The SwanSphere generation model is trained with AdamW at learning rate \(1\times 10^{-5}\) for **600k steps** on **8 H800 GPUs**. ODPO then runs for **3 rounds** of online fine-tuning [2605.30940].

## 6. Evaluation, baselines, and empirical results

The evaluation protocol combines objective and subjective measures. Non-spatial audio quality is measured by **FD** on OpenL3 embeddings and **KL divergence** over AudioSet label distributions from a PaSST classifier. Spatial accuracy is measured by mean absolute azimuth error \(\Delta_{abs}\theta\), mean absolute elevation error \(\Delta_{abs}\phi\), aggregate angular error \(\Delta_{angular}\), and **wCS** from a pretrained SELD model (PSELDNets). Subjective evaluation uses **MOS-SQ** for spatial audio quality and **MOS-AF** for alignment faithfulness to video or text [2605.30940].

For video-to-spatial audio, the baselines are MMAudio + audio spatialization, Diff-Foley + audio spatialization, ViSAGe, and OmniAudio. For text-to-spatial audio, the baselines are MMAudio + AS, AudioLDM-2 + AS, Tango2 + AS, and OmniAudio(text). The “+AS” baselines are cascaded systems with an external spatialization step [2605.30940].

The main video-conditioned comparison against OmniAudio is as follows.

| Model | FD | KL | Angular | MOS-SQ | MOS-AF |
|---|---:|---:|---:|---:|---:|
| SwanSphere | 120.28 | 1.36 | 1.03 | 4.32 | 4.44 |
| OmniAudio | 157.67 | 1.93 | 1.27 | 4.12 | 4.27 |

For text-to-spatial audio, SwanSphere also outperforms OmniAudio(text), with **FD 142.80 vs 174.13**, **KL 1.43 vs 1.83**, **MOS-SQ 4.31 vs 4.11**, and **MOS-AF 4.43 vs 4.16**. On inference speed, SwanSphere reports **0.21 s first chunk** and **9.13 s total**, compared with **20.19 s** for ViSAGe, **0.85 s** for OmniAudio with no streaming, and **2.76 s** for MMAudio + AS [2605.30940].

Several secondary findings sharpen the interpretation of these results. On the caption test subset, adding captions to video conditioning improves **FD 120.28 → 118.31** and **angular error 1.03 → 0.96**, indicating that captions provide complementary spatial cues. On YT360-Test, the reported out-of-distribution performance remains strongest, with **FD 145.83**, **KL 1.60**, and **angular error 1.13**. The ablations further report that removing history worsens FD and KL, smaller models degrade performance consistently, and a comparable-size full-sequence DiT has **6.47 s** inference, worse first-response latency, and slightly weaker spatial performance. These outcomes support the patchwise autoregressive streaming design rather than a monolithic full-sequence transformer [2605.30940].

## 7. Scope, limitations, and name disambiguation

SwanSphere is positioned for **VR/AR**, **immersive 360° video**, **metaverse audio generation**, and **spatially grounded multimedia creation**. Its low first-chunk latency makes it suitable for interactive settings in which audio must begin quickly while remaining synchronized to panoramic video or text-guided scene structure [2605.30940].

The framework also has explicit limitations. The spatial captions mostly describe **dominant sound sources**. Complex multi-source scenes, such as concerts with many simultaneous instruments, are not fully disentangled. The paper also notes the need for better generalization to unseen recording setups and richer multi-source spatial datasets. These constraints are important for interpreting the reported gains: the system is strong on unified multimodal FOA generation, but it is not presented as a full solution to fine-grained source separation in cluttered scenes [2605.30940].

The name should be distinguished from unrelated arXiv usage. “Active swarms on a sphere” studies a minimal model of polar, self-propelled soft particles on a sphere and analyzes curvature-induced states such as a polar vortex and a circulating band [1407.8516]. “Joint Segment Activation and Antenna Placement for Uplink SWAN Systems” studies multiuser uplink segmented waveguide-enabled pinching-antenna systems, deriving an achievable sum-rate upper bound and proposing HSS/A schemes for segment activation and PA placement [2605.13580]. Neither concerns streaming FOA generation. In current arXiv usage, SwanSphere refers specifically to the streaming synchronized spatial-audio framework based on causal patchwise generation, local diffusion rendering, SVAC alignment, ODPO preference tuning, and automated spatial captioning [2605.30940].

Source: https://www.emergentmind.com/topics/swansphere