SwanSphere: Unified Spatial Audio Framework

• SwanSphere is a unified framework that streams synchronized 4-channel FOA spatial audio from panoramic videos and text prompts, preserving spatial direction and temporal cues.
• It uses a two-stage autoregressive diffusion transformer for semantic planning and local patch generation, achieving a rapid first-chunk latency of 0.21 seconds.
• The framework integrates spatial video-audio contrastive learning and online direct preference optimization to enhance alignment, audio fidelity, and spatial accuracy.

SwanSphere is a unified framework for streaming synchronized spatial audio generation from panoramic videos and text prompts. Its output is First-Order Ambisonics (FOA), a 4-channel spatial audio representation $\mathbf{a}\in\mathbb{R}^{C\times L}$ with channels $W,X,Y,Z$. The framework addresses a specific systems problem in immersive audio generation: diffusion models can provide high fidelity but incur high inference latency, whereas autoregressive or token-based methods can reduce delay but often lose audio fidelity or spatial precision. SwanSphere is formulated to reduce this quality–latency tradeoff while improving the capture of direction, motion, depth, and temporal synchronization in $360^\circ$ scenes (Lei et al., 29 May 2026).

1. Problem setting and representation

SwanSphere is designed for two conditional generation regimes: video-to-spatial audio and text-to-spatial audio. In both cases, the target is synchronized FOA rather than monaural audio followed by a separate spatialization stage. This distinction is central because FOA must preserve not only semantic event content but also where the sound comes from, how its orientation evolves, and whether temporal changes across $W,X,Y,Z$ remain physically plausible (Lei et al., 29 May 2026).

The framework uses a continuous FOA VAE rather than a discrete codebook. For input audio $\mathbf{a}$, the latent sequence is defined as

$$\mathbf{z} = \mathcal{E}(\mathbf{a}), \qquad \mathbf{z}\in\mathbb{R}^{d\times l}.$$

The VAE is based on Stable Audio VAE, adapted for 4-channel FOA. The stated motivation is to avoid quantization errors and phase loss from discrete tokenization. Generation is then factorized patchwise as

$$p(\mathbf{z}_{1:T}\mid c) = \prod_{t=1}^{T} p(\mathbf{z}_t \mid \mathbf{z}_{<t}, c),$$

where the condition $c$ may include panoramic video features, text caption embeddings, or both (Lei et al., 29 May 2026).

The underlying difficulty of streaming FOA generation is presented as a three-part constraint. First, the model must reconcile long-range event structure with local waveform or latent detail. Second, it must maintain spatial consistency across FOA channels. Third, it must overcome weak cross-modal alignment in generic video encoders that often miss geometry and acoustic cues in panoramic video. SwanSphere addresses these constraints by separating semantic planning from local rendering and by explicitly aligning audio and panoramic-video representations (Lei et al., 29 May 2026).

2. Causal autoregressive diffusion transformer

The core architecture is a causal autoregressive diffusion transformer organized as a two-stage patchwise process. In Stage 1, an autoregressive “spatial LM” predicts a semantic condition $h_t$ for the current patch from previous audio context and multimodal input. In Stage 2, a Local Diffusion Transformer (LocDiT) renders the actual FOA latent patch conditioned on $h_t$, recent history, and boundary context (Lei et al., 29 May 2026).

Component	Role
AR “spatial LM”	Patchwise semantic planning
LocDiT	High-fidelity FOA patch generation
History encoder	Summarizes generated audio context
Boundary context	Supplies preceding patches for continuity

Streaming is implemented with fixed-size latent patches. The reported configuration is a patch size of 4 latent frames, temporal stride 4, causal context window of 2 patches $W,X,Y,Z$0 latent frames, and 20 diffusion steps per patch for LocDiT. The architecture only needs enough context to generate the current chunk, which permits playback once the first patch is produced (Lei et al., 29 May 2026).

Synchronization is maintained through four explicit mechanisms. The current semantic prediction $W,X,Y,Z$1 is conditioned on aligned video tokens and a historical spatial-audio summary from the history encoder. LocDiT receives the two preceding patches as boundary context. Generation terminates when the video feature sequence ends rather than through a stop token, tying audio duration directly to input-video duration. For text-only generation, FLAN-T5 embeddings are injected via cross-attention, while missing modalities are represented by learnable null embeddings so that a single architecture supports video only, text only, and video + text conditioning (Lei et al., 29 May 2026).

The first-chunk latency is broken down into 0.03 s for the spatial LM, 0.14 s for LocDiT denoising, and 0.04 s for video encoding and audio decoding, yielding a total time-to-first-chunk = 0.21 s. The total inference time reported for the full system is 9.13 s (Lei et al., 29 May 2026).

3. Spatial Video-Audio Contrastive learning

Spatial Video-Audio Contrastive learning (SVAC) is the representation-alignment component of SwanSphere. It aligns panoramic video features and FOA audio features in a shared space so that the video encoder becomes acoustically and spatially aware rather than merely semantically informative. The video encoder is VideoMAE-V2, selected because it preserves spatial structure better than CLIP-style global representations, and the audio encoder is AudioMAE. Because audio has finer temporal resolution than video, video features are aligned to the audio timeline by nearest-neighbor replication rather than interpolation (Lei et al., 29 May 2026).

The training objective is a symmetric InfoNCE loss: $W,X,Y,Z$2 with

$W,X,Y,Z$3

Here $W,X,Y,Z$4 is temporally aligned cosine similarity, $W,X,Y,Z$5 is a temperature, and $W,X,Y,Z$6 is batch size (Lei et al., 29 May 2026).

The distinctive feature of SVAC is its physics-aware negative construction. Four contrastive views are used: instance exchange, temporal offset, audio rotation, and video rotation. The negative sets are

$W,X,Y,Z$7

$W,X,Y,Z$8

where $W,X,Y,Z$9 is temporally shifted audio, $360^\circ$0 is spatially rotated FOA audio, and $360^\circ$1 is horizontally rotated video. This design teaches the encoders to distinguish matched from mismatched semantics, aligned from time-shifted events, and original from rotated spatial layouts (Lei et al., 29 May 2026).

The ablation data attribute measurable gains to these physics-aware negatives.

Variant	FD	KL	Angular
Full SVAC	120.28	1.36	1.03
Semantic-only	127.12	1.41	1.12
CLIP baseline	140.28	1.44	1.34

These results indicate that semantic-only contrastive learning helps, but the temporal and spatial hard negatives contribute additional gains in both audio quality and localization accuracy (Lei et al., 29 May 2026).

4. Online Direct Preference Optimization

After pretraining, SwanSphere is refined with multi-objective online Direct Preference Optimization (ODPO). The stated aim is to bias generation toward outputs that are spatially accurate, semantically consistent, and perceptually clean or artifact-free. The paper adopts ODPO rather than RL methods such as GRPO because the sampling-and-ranking pipeline naturally yields preference pairs and is described as more stable and lightweight (Lei et al., 29 May 2026).

For each input, the model generates 8 candidate audios in parallel. These are ranked with a weighted reward

$360^\circ$2

using weights $360^\circ$3, $360^\circ$4, and $360^\circ$5. The spatial reward is based on azimuth, elevation, and spatial-angle errors between generated and reference sound fields; the semantic reward is computed from ImageBind similarity between generated audio and video or text embeddings; the fidelity reward uses Audiobox Aesthetics. All reward components are normalized to $360^\circ$6 (Lei et al., 29 May 2026).

ODPO contributes substantially to final quality. The reported ablation for SwanSphere-L gives FD 133.91 and angular 1.22 without ODPO, versus FD 120.28 and angular 1.03 for the full model. Independent spatial evaluation with weighted cosine similarity (wCS) from PSELDNets shows the same pattern: 0.41 for OmniAudio, 0.52 for the model without ODPO, and 0.63 for the full system (Lei et al., 29 May 2026).

A common simplification is to treat preference optimization here as a generic perceptual fine-tuning stage. The reported reward design indicates a narrower role: ODPO is explicitly multi-objective, and spatial accuracy is weighted equally with semantic alignment rather than subordinated to fidelity. This suggests that preference tuning in SwanSphere is part of the localization stack rather than a purely cosmetic post-process.

5. Spatial caption annotation, datasets, and training

SwanSphere includes an automated spatial caption annotation pipeline motivated by the scarcity of spatial audio–text datasets. The pipeline has three stages. First, it extracts acoustic spatial features from FOA intensity vectors in the STFT domain: $360^\circ$7 These are aggregated into a spatial vector $360^\circ$8 using an energy-weighted average over the 500–8000 Hz band. Azimuth and elevation are then computed as

$360^\circ$9

$W,X,Y,Z$0

Second, the angular trajectory is smoothed by converting angles to unit vectors and applying a moving average with window size 3. Third, the structured trajectory data, panoramic video, and audio are sent to Gemini 2.5 Pro to generate captions describing source content, location, timing, and movement or dynamics. The prompt explicitly encodes the coordinate system, including $W,X,Y,Z$1 = front, positive azimuth = left, negative azimuth = right, and $W,X,Y,Z$2 = rear (Lei et al., 29 May 2026).

The captioned subset contains about 3,100 valid captioned samples, with 300 reserved for evaluation. The main dataset contains 165,000 video-audio pairs, about 458 hours, in 10-second clips, with 5% held out as a test set with no overlapping video IDs. Its sources are Sphere360, YT-Ambigen, newly collected YouTube FOA clips, and additional web-crawled and filtered data. For curriculum learning, the framework also uses a 1M-sample non-spatial audio pretraining set built from AudioCaps, VGGSound, WavText5k, and AudioSet, converted into a pseudo-FOA format for early training (Lei et al., 29 May 2026).

The training recipe is explicitly staged. The FOA VAE uses batch size 80 on 2 H800 GPUs, with 200k steps for encoder or initial training and then 300k steps for the decoder, using weighted multi-resolution STFT, KL, and adversarial or discrimination losses, with the $W,X,Y,Z$3 loss weight set to zero. SVAC trains only projection layers for 100k steps on 2 H800 GPUs at learning rate $W,X,Y,Z$4, while keeping VideoMAE-V2 and AudioMAE frozen. The SwanSphere generation model is trained with AdamW at learning rate $W,X,Y,Z$5 for 600k steps on 8 H800 GPUs. ODPO then runs for 3 rounds of online fine-tuning (Lei et al., 29 May 2026).

6. Evaluation, baselines, and empirical results

The evaluation protocol combines objective and subjective measures. Non-spatial audio quality is measured by FD on OpenL3 embeddings and KL divergence over AudioSet label distributions from a PaSST classifier. Spatial accuracy is measured by mean absolute azimuth error $W,X,Y,Z$6, mean absolute elevation error $W,X,Y,Z$7, aggregate angular error $W,X,Y,Z$8, and wCS from a pretrained SELD model (PSELDNets). Subjective evaluation uses MOS-SQ for spatial audio quality and MOS-AF for alignment faithfulness to video or text (Lei et al., 29 May 2026).

For video-to-spatial audio, the baselines are MMAudio + audio spatialization, Diff-Foley + audio spatialization, ViSAGe, and OmniAudio. For text-to-spatial audio, the baselines are MMAudio + AS, AudioLDM-2 + AS, Tango2 + AS, and OmniAudio(text). The “+AS” baselines are cascaded systems with an external spatialization step (Lei et al., 29 May 2026).

The main video-conditioned comparison against OmniAudio is as follows.

Model	FD	KL	Angular	MOS-SQ	MOS-AF
SwanSphere	120.28	1.36	1.03	4.32	4.44
OmniAudio	157.67	1.93	1.27	4.12	4.27

For text-to-spatial audio, SwanSphere also outperforms OmniAudio(text), with FD 142.80 vs 174.13, KL 1.43 vs 1.83, MOS-SQ 4.31 vs 4.11, and MOS-AF 4.43 vs 4.16. On inference speed, SwanSphere reports 0.21 s first chunk and 9.13 s total, compared with 20.19 s for ViSAGe, 0.85 s for OmniAudio with no streaming, and 2.76 s for MMAudio + AS (Lei et al., 29 May 2026).

Several secondary findings sharpen the interpretation of these results. On the caption test subset, adding captions to video conditioning improves FD 120.28 → 118.31 and angular error 1.03 → 0.96, indicating that captions provide complementary spatial cues. On YT360-Test, the reported out-of-distribution performance remains strongest, with FD 145.83, KL 1.60, and angular error 1.13. The ablations further report that removing history worsens FD and KL, smaller models degrade performance consistently, and a comparable-size full-sequence DiT has 6.47 s inference, worse first-response latency, and slightly weaker spatial performance. These outcomes support the patchwise autoregressive streaming design rather than a monolithic full-sequence transformer (Lei et al., 29 May 2026).

7. Scope, limitations, and name disambiguation

SwanSphere is positioned for VR/AR, immersive 360° video, metaverse audio generation, and spatially grounded multimedia creation. Its low first-chunk latency makes it suitable for interactive settings in which audio must begin quickly while remaining synchronized to panoramic video or text-guided scene structure (Lei et al., 29 May 2026).

The framework also has explicit limitations. The spatial captions mostly describe dominant sound sources. Complex multi-source scenes, such as concerts with many simultaneous instruments, are not fully disentangled. The paper also notes the need for better generalization to unseen recording setups and richer multi-source spatial datasets. These constraints are important for interpreting the reported gains: the system is strong on unified multimodal FOA generation, but it is not presented as a full solution to fine-grained source separation in cluttered scenes (Lei et al., 29 May 2026).

The name should be distinguished from unrelated arXiv usage. “Active swarms on a sphere” studies a minimal model of polar, self-propelled soft particles on a sphere and analyzes curvature-induced states such as a polar vortex and a circulating band (Sknepnek et al., 2014). “Joint Segment Activation and Antenna Placement for Uplink SWAN Systems” studies multiuser uplink segmented waveguide-enabled pinching-antenna systems, deriving an achievable sum-rate upper bound and proposing HSS/A schemes for segment activation and PA placement (Gu et al., 13 May 2026). Neither concerns streaming FOA generation. In current arXiv usage, SwanSphere refers specifically to the streaming synchronized spatial-audio framework based on causal patchwise generation, local diffusion rendering, SVAC alignment, ODPO preference tuning, and automated spatial captioning (Lei et al., 29 May 2026).