---
title: 'SonicMotion: Dynamic Spatial Audio Soundscapes'
url: https://www.emergentmind.com/topics/sonicmotion
type: topic
---

# SonicMotion: Dynamic Spatial Audio Soundscapes

SonicMotion is an end-to-end latent diffusion model for generating dynamic spatial audio soundscapes in first-order Ambisonics (FOA), introduced in "SonicMotion: Dynamic Spatial Audio Soundscapes with Latent Diffusion Models" [2507.07318]. It targets text-guided generation of 3D audio scenes with moving sound sources, extending recent FOA generative systems that either produced non-spatial audio, offered only coarse stereo control, or assumed static source positions. The framework is organized around a spatial autoencoder pretransform, conditioning modules, a diffusion transformer, two user-control variants with different localization granularity, and a simulated dataset of FOA audio-caption pairs that includes both static and dynamic sources.

## 1. Problem formulation and historical position

SonicMotion addresses dynamic spatial audio generation in FOA, rather than mono or stereo generation. The task combines semantic control over the sound event with temporal control over source motion in 3D space. In the paper’s formulation, this fills a specific gap in generative audio research: SpatialSonic provides stereo output with coarse spatial control but cannot represent full 3D cues such as behind or above; Diff-SAGe provides FOA with azimuth and elevation control but requires the source to remain static; and ImmerseDiffusion provides FOA with text-based spatial guidance but does not support moving sources [2507.07318].

The choice of FOA is central rather than incidental. SonicMotion treats dynamic source movement as a native property of the generated representation, not as a post hoc spatialization layer applied to monaural audio. This places the method in a lineage of immersive-audio generation systems, but with a stronger emphasis on temporally varying directionality. A plausible implication is that the model is designed for settings where the spatial trajectory is part of the content specification itself, rather than a downstream rendering parameter.

## 2. FOA representation and spatial formalism

FOA is the paper’s target output format. It uses four channels: \(W\) as the omnidirectional channel and \(X\), \(Y\), \(Z\) as directional components. The paper states that FOA provides “precise control over the direction of arrival of sound sources in the X, Y, and Z directions,” preserving spatial cues that stereo cannot express [2507.07318].

For spatial analysis, SonicMotion derives intensity vectors from the FOA channels as
\[
I_x = W \cdot X, \quad I_y = W \cdot Y, \quad I_z = W \cdot Z.
\]
Direction of arrival is then estimated through
\[
\theta = \tan^{-1} \left( \frac{I_y}{I_x} \right), \quad
\phi = \tan^{-1} \left( \frac{I_z}{I_x^2 + I_y^2} \right).
\]
The paper further evaluates spatial preservation using a spherical distance:
\[
a = \sin^2\left( \frac{\Delta\phi}{2} \right) + \cos(\phi) \cdot \cos(\hat{\phi}) \cdot \sin^2\left( \frac{\Delta\theta}{2} \right),
\]
\[
\Delta_{\text{Spatial-Angle}} = 2 \cdot \arctan2\left( \sqrt{a}, \sqrt{1 - a} \right).
\]

These definitions are not only evaluation machinery. They also clarify the paper’s notion of correctness: a generated sample should preserve both semantic plausibility and spatial directionality over time. In that sense, SonicMotion is not merely a text-to-audio model with extra channels; it is a model whose fidelity is partly measured by whether FOA structure remains physically interpretable under source motion.

## 3. Latent diffusion architecture

SonicMotion uses latent diffusion rather than direct waveform diffusion. Its pipeline comprises four stages: FOA audio is encoded into a latent representation with a spatial autoencoder, conditioning signals are injected, a diffusion transformer generates latent audio, and the latent is decoded back into FOA audio [2507.07318].

The spatial autoencoder is a modified version of the Descript Audio Codec (DAC) used as a pretransform. Two modifications are explicit. First, the Residual Vector Quantization bottleneck is replaced with a continuous Variational Autoencoder bottleneck to avoid information loss from quantization. Second, the final \( \tanh() \) activation is removed from the decoder to prevent harmonic distortion. The codec compresses a 4-channel FOA signal into 64 channels with a downsampling ratio of 2048, giving an overall compression factor of 128. Its loss combines MRSTFT losses for each FOA channel with KL divergence, adversarial loss, and feature matching loss.

The generative backbone is the continuous diffusion transformer from the Stable Audio and latent diffusion line of work. Its blocks include self-attention, cross-attention, MLP gates, layer normalization, and skip connections. All conditioning inputs are projected into a shared token dimension of 768. Training minimizes the Mean Squared Error between the ground-truth latent and the predicted latent at a given timestep, conditioned on the input controls. This architecture situates SonicMotion among latent generative systems optimized for long-form audio efficiency, while adapting that paradigm to a multichannel spatial representation.

## 4. Conditioning design and model variants

SonicMotion is presented in two variants that differ in how spatial motion is specified. The descriptive model uses text only, relying on a spatially descriptive caption such as a language description of movement. The parametric model augments text with explicit spatial parameters: azimuth, elevation, initial position, final position, movement direction, movement speed, start time, and total duration [2507.07318].

Both variants use a text prompt embedding from a pretrained T5 encoder. The paper notes that T5 performed better than CLAP for spatial retrieval in prior work. The parametric model adds a position conditioner based on a state matrix that tracks source position over time. Relative to earlier work, SonicMotion expands azimuth to the full \(360^\circ\), supports both clockwise and counter-clockwise motion, and adds support for initial and final elevation. The time-varying position is defined by linear interpolation from start to end position over a duration \(T\), and the resulting azimuth and elevation state matrices are concatenated. Fusion with the text embedding is performed by a cross-attention module with 4 layers and 4 heads. Two additional cross-attention conditioners encode start time and total time.

The distinction between the two variants is empirical as well as conceptual. The paper explicitly attributes the lower precision of the descriptive model to the ambiguity of natural-language spatial descriptions.

| Variant | Conditioning | Reported spatial errors |
|---|---|---|
| Descriptive | Text prompt only | \(L1(\theta)=21.20^\circ\), \(L1(\phi)=16.04^\circ\), \(\Delta_{\text{angle}}=29.22^\circ\) |
| Parametric | Text prompt + explicit spatial parameters | \(L1(\theta)=13.17^\circ\), \(L1(\phi)=4.01^\circ\), \(\Delta_{\text{angle}}=14.32^\circ\) |

This division makes SonicMotion notable among generative spatial-audio systems because it supports both natural-language control and structured control, rather than treating them as interchangeable. The results suggest that explicit control signals are materially advantageous when localization fidelity is a primary requirement.

## 5. Simulated spatial audio-caption dataset

Because no public dataset contained spatial audio-caption pairs with moving sources, the authors constructed a new simulated dataset by spatially augmenting Clotho, AudioCaps, and FreeSound [2507.07318]. Each original mono sample is resampled to 16 kHz, stripped of leading and trailing silence, truncated or looped to exactly 10 seconds, and then converted into one static spatial version and one dynamic spatial version.

Dynamic source generation follows an explicit procedure. For each moving source, the system randomly chooses whether azimuth, elevation, or both change; samples a final position with minimum separation of at least \(45^\circ\) for azimuth and at least \(30^\circ\) for elevation; assigns one of three speed categories—fast, moderate, or slow; randomly samples movement start and end times based on speed; linearly interpolates position over the movement window; and encodes the time-varying position into FOA. For azimuth motion, the direction is randomly set to clockwise or counter-clockwise. Each sample records initial azimuth, final azimuth, initial elevation, final elevation, clockwise flag, speed, movement start time, and movement end time.

The captions are rewritten with Gemma-3 12B using the relevant spatial parameters. The paper includes a mapping from numerical spatial bins to natural-language labels such as front, front-left, left, back-left, back, back-right, right, front-right, as well as up, down, and fast, moderate, slow. The dataset sizes are 1,018,430 training samples, 3,078 validation samples, and 4,040 test samples.

Training follows the model decomposition. The autoencoder is trained on random 2-second crops from the training split using 4×H100 GPUs, batch size 128, for 400K steps, with AdamW, generator learning rate \(1 \times 10^{-4}\), discriminator learning rate \(2 \times 10^{-4}\), momentum \(0.8, 0.99\), and weight decay \(1 \times 10^{-3}\). The diffusion model is trained on full 10-second samples using 4×H100 GPUs, batch size 1024, for 100K steps, with the autoencoder weights and pretrained T5 encoder frozen, the position encoder’s cross-attention trainable, AdamW, learning rate \(1 \times 10^{-4}\), momentum \(0.9, 0.99\), weight decay \(1 \times 10^{-3}\), position encoder cross-attention learning rate \(2 \times 10^{-4}\), and an inverse learning-rate scheduler with inverse gamma \(=10^6\), power \(=0.5\), and warmup \(=0.99\). The descriptive model is trained only on spatial captions, whereas the parametric model is trained for 50k steps on original non-spatial captions and 50k steps on spatial captions.

## 6. Evaluation methodology and reported performance

SonicMotion is evaluated at two levels: codec reconstruction quality and generative performance. Autoencoder evaluation uses STFT distance, Mel spectrogram distance, \(L1(\theta)\), \(L1(\phi)\), and \(\Delta_{\text{angle}}\), computed with auraloss defaults. The reported reconstruction scores are STFT \(1.35\), MEL \(0.95\), \(L1(\theta)=3.32^\circ\), \(L1(\phi)=1.43^\circ\), and \(\Delta_{\text{angle}}=3.72^\circ\), indicating that the codec preserves both signal fidelity and spatial directionality [2507.07318].

For the diffusion model, the paper evaluates CLAP score, Fréchet Distance between CLAP audio embeddings, Fréchet Audio Distance based on VGGish embeddings, KL divergence between CLAP embedding distributions, and the same spatial metrics. Semantic evaluation uses only the \(W\) channel of generated FOA audio and compares it against the original non-spatial caption, thereby isolating audio quality and semantic content from the spatial annotations. The baselines are AudioLDM 2 and Stable Audio Open, used only for semantic and audio-quality comparison because they do not generate FOA spatial soundscapes.

The reported semantic and audio-quality results place SonicMotion near strong text-to-audio baselines. AudioLDM 2 obtains CLAP 0.22, FD 0.55, FAD 1.78, and KL 0.27. Stable Audio Open obtains CLAP 0.19, FD 1.18, FAD 7.90, and KL 0.59. SonicMotion achieves CLAP 0.23 for both variants; FD 0.52 and 0.57 for the descriptive and parametric variants; FAD 2.97 and 3.00; and KL 0.26 and 0.28. The paper characterizes these results as competitive with or better than the baselines while simultaneously modeling dynamic FOA soundscapes.

The main performance distinction lies in spatial accuracy. The descriptive model yields \(\Delta_{\text{angle}}=29.22^\circ\), whereas the parametric model reaches \(\Delta_{\text{angle}}=14.32^\circ\). The paper attributes this difference to ambiguity in natural-language spatial descriptions; a term such as “front” can refer to a wide directional range, while explicit parameterization removes that ambiguity.

## 7. Significance, interpretation, and relation to adjacent systems

SonicMotion’s principal contribution is to extend latent diffusion-based audio generation from static or non-spatial settings to dynamic spatial audio in full 3D ambisonics [2507.07318]. It combines a spatial audio codec, text conditioning, and optional explicit motion-state conditioning in a single end-to-end generator. The descriptive variant shows that dynamic source motion can be prompted in language, while the parametric variant demonstrates that explicit motion metadata materially improves localization precision.

A common misconception is to treat SonicMotion as a spatialization postprocessor or a physics-based simulator. It is neither. Unlike Sonic4D, which is a training-free modular system that reconstructs a dynamic 4D scene from monocular video, localizes a source trajectory, and synthesizes viewpoint-adaptive binaural audio through physics-based simulation [2506.15759], SonicMotion generates FOA soundscapes directly with latent diffusion. It is also distinct from moving-reverberation simulators such as "Fast Algorithm for Moving Sound Source," which reformulates moving-source reverberation as a sampled-reconstructed time-varying image-source process [2508.03065], and from SonicSim, a Habitat-sim-based toolkit for customizable moving-source speech simulation and benchmark construction [2410.01481]. The similarly named "Sound Sparks Motion" is a training-free framework for motion-centric video editing in a frozen audio-visual generator, not a spatial-audio generator [2605.15307].

The clearest limitation identified within SonicMotion itself is the ambiguity of natural-language spatial descriptions. The descriptive model remains viable, but its localization error is substantially higher than that of the parametric model. This suggests that, within the paper’s own evidence, free-form language and precise spatial control are not equivalent interfaces. A plausible implication is that future work on dynamic FOA generation may require stronger structured-conditioning schemes, better spatial language grounding, or hybrid controllers that preserve natural promptability without sacrificing directional accuracy.

Source: https://www.emergentmind.com/topics/sonicmotion