Stable Audio 3: Audio Diffusion Model

• Stable Audio 3 is a text-conditioned latent diffusion model for generating and editing multi-minute, high-fidelity audio with native variable-length synthesis.
• It integrates a semantic–acoustic autoencoder, differential attention, and mask-based inpainting to ensure precise control and preservation of audio quality.
• Open-weight small and medium models enable fast inference on consumer-grade hardware, achieving rapid generation with superior prompt adherence and efficiency.

Stable Audio 3 is a family of fast, text-conditioned latent diffusion models (LDMs) for multi-minute, high-fidelity, and controllable audio generation and editing, supporting both instrumental music and sound effects. Distinctive architectural features include native variable-length synthesis, semantic-acoustic latent modeling, mask-based inpainting, differential attention mechanisms, and an adversarial post-training pipeline for rapid few-step inference. Open-weight versions for small and medium scales enable deployment on consumer-grade hardware, setting new standards in prompt adherence, audio quality, and inference efficiency (Evans et al., 18 May 2026).

1. Key Innovations and Model Capabilities

Stable Audio 3 delivers a series of methodological advancements over previous audio diffusion models:

• Variable-Length Generation: Native handling of arbitrary output durations from seconds to several minutes is achieved by allocating $L = \lceil (d + d_{sp}) f_s / r \rceil$ latent frames, where $d_{sp} \approx 6$ s (silence padding), $f_s = 44,\!100$ Hz (sampling rate), and $r = 4096$ (autoencoder downsampling ratio). This avoids the inefficiency of full-length computation for short generations.
• Inpainting and Continuation: Flexible mask-based latent-space editing enables single-region, multi-region, and prefix-based (causal) inpainting for local edits and seamless continuation. The inpainting protocol injects the mask and masked-latent at each transformer block via additive local operations, affording precise control.
• Semantic-Acoustic Autoencoder: The backbone autoencoder (SAME) achieves high downsampling (4096×) and yields compact 256-dimensional latents at $\sim$10.76 Hz. Its loss formulation blends multi-resolution STFT reconstruction, relativistic GAN objectives, cross-modal contrastive alignment, and semantic regressions (chroma, interaural level difference) to preserve both perceptual fidelity and semantic content.
• Latent Diffusion Transformer: A flow-matching formulation enables the diffusion model to predict the continuous “velocity” field $v_t(x_t, t)$, with integrated text conditioning (frozen T5Gemma embeddings), duration embedding, and inpainting signal injection. Differential Attention, employed in medium/large models, leverages dual $(Q, K)$ pairs with subtracted context heads for expressive context modeling.
• Adversarial Post-Training: A hybrid training pipeline, progressing through flow-matching, one-step distillation, and adversarial finetuning (relativistic GAN, CLAP-based alignment), enables rapid inference with as few as 8 “ping-pong” steps, matching or exceeding the fidelity of 50-step Euler samplers.
• Open Model Accessibility: Pretrained weights for small and medium variants are released, compatible with CPU (e.g., MacBook Pro M4) and affordable GPUs (8 GB VRAM), along with training/inference code and pipeline.

2. Semantic–Acoustic Autoencoder Architecture

The autoencoder (SAME) serves as a high-capacity, frozen backbone for compressing audio waveforms into condensed, semantically structured latents:

• Patch Tokenization and Downsampling: Stereo audio is split into non-overlapping 256-sample blocks per channel, then processed through Transformer Resampling Blocks (TRB), which further contract the representation by a 16× factor via learnable embedding interleaving and RoPE. The result is a compression factor of 4096× down to 256-dimensional latent sequences.
• Soft-Normalization Bottleneck: Latent representations undergo a learnable affine scaling with running standard deviation tracking, yielding stable and deterministic encodings.
• Loss Stack:

1. Multi-resolution STFT loss (FFT sizes 32–2048), combining log-magnitude $L_1$, spectral contrast, instantaneous frequency phase loss (IFGD), and K-weighting pre-emphasis. 2. Relativistic GAN loss with multi-scale convolutional discriminators operating on raw waveforms. 3. Diffusion alignment loss (flow-matching with a compact transformer critiquing latent geometry). 4. Semantic regression losses predicting chroma and ILD. 5. Contrastive latent alignment via a critic transformer trained to distinguish matched/mismatched triplets (latent, wavelet feature, text embedding).

• Variants: SAME-S (108 M parameters, suitable for CPU) and SAME-L (852 M parameters), both operating with 256-dim latents.

Variant	Parameters	Latent Dim.	Compression
SAME-S	108 M	256	4096×
SAME-L	852 M	256	4096×

3. Latent Diffusion Transformer Design

The diffusion backbone operates on SAME latents, employing a multi-pathway conditioning and advanced attention mechanisms:

• Dataflow: $x_0$ (44.1 kHz stereo) → SAME encoder → $z_0 \in \mathbb{R}^{256 \times T}$ → linear projection → prepend 64 memory embeddings → $d_{sp} \approx 6$0 stacked transformer blocks → project to 256 → SAME decoder → $d_{sp} \approx 6$1.
• Conditioning Pathways:

1. Diffusion Timestep & Duration: $d_{sp} \approx 6$2 and duration $d_{sp} \approx 6$3 are Fourier-encoded and injected via adaptive LayerNorm (AdaLN) in both attention and feed-forward pathways. 2. Text: Up to 256 T5Gemma-embedded tokens plus duration embedding condition all cross-attention layers. 3. Inpainting: Mask $d_{sp} \approx 6$4 concatenated with $d_{sp} \approx 6$5, projected through MLP and additively injected at each block.

• Transformer Blocks:
  • Self-Attn: RMSNorm, AdaLN, multi-head QK-RMSNorm, RoPE on partial dims.
  • Cross-Attn: RMSNorm, QK-RMSNorm.
  • Feed-forward: SwiGLU (SiLU gate), AdaLN.
  • Differential Attention: For medium/large, the transformer computes dual attention heads $d_{sp} \approx 6$6 and $d_{sp} \approx 6$7—outputting the difference, with shared $d_{sp} \approx 6$8, to capture finer context interactions.
• Model Scaling:

Model	Latent Dim. ($d_{sp} \approx 6$9)	Blocks ($f_s = 44,\!100$0)	Heads ($f_s = 44,\!100$1)	Max Length ($f_s = 44,\!100$2)	Params
small	512	12	8	2 min	459 M
medium	1024	24	16	6 m 20 s	1.4 B
large	1536	32	24	6 m 20 s	2.7 B

4. Training Pipeline and Optimization

The training pipeline consists of three distinct phases, all operating on SAME-encoded latents with variable-length sequences:

• Variable-Length Schema: Batches contain right-padded sequences of heterogeneous duration $f_s = 44,\!100$3; FlashAttention masks out padding; losses are computed solely over unmasked positions. Silence augmentation, via exponential-length runs of silence latents, improves output termination and “ending” fidelity. A per-example timestep shift (Equation 3: $f_s = 44,\!100$4; $f_s = 44,\!100$5) biases longer sequences toward higher-noise regions to mitigate redundancy.
• Flow-Matching Pre-Training: Latents are diffused as $f_s = 44,\!100$6, $f_s = 44,\!100$7, with loss $f_s = 44,\!100$8. Inpainting masks are sampled to provide a blend of unconditional, random-region, and continuation setups. Minibatch optimal-transport permutes $f_s = 44,\!100$9 for minimal $r = 4096$0 cost.
• Distillation Warmup: A frozen teacher (flow-matching model) generates reference trajectories; the student is trained for one-step mapping using cached $r = 4096$1 pairs, minimizing $r = 4096$2.
• Adversarial Post-Training: The distilled generator is adversarially tuned against a transformer-based discriminator, both conditioned on text, duration, mask, and individual timesteps, with a relativistic GAN loss and CLAP-based prompt-alignment loss. The generator minimizes $r = 4096$3 with hybrid Muon/AdamW optimizers and learning-rate schedule $r = 4096$4. This phase enables the few-step "ping-pong" inference procedure.

5. Inference Procedure and Hardware Performance

• Few-Step “Ping-Pong” Sampling: The adversarially post-trained generator provides a near one-step $r = 4096$5 mapping, but highest fidelity is recovered with $r = 4096$6 bidirectional steps in log-SNR space, alternating denoising ($r = 4096$7) and re-noising ($r = 4096$8).
• Classifier-Free Guidance: No extra CFG inference pass—guidance is baked into the post-trained generator, halving the computational cost relative to classifier-free schemes of prior LDMs.
• Latency and Memory: On an NVIDIA H200 GPU, the medium model can generate 6 m 20 s (stereo 44.1 kHz) audio in under 2 s; the small model delivers similar outputs in less than 0.5 s, fitting consumer-grade hardware. MacBook Pro M4 (CPU) inference for 120 s is $r = 4096$96 s; CoreML achieves $\sim$0 s for the same. Peak VRAM for medium is $\sim$16.5 GB at 120 s.

Hardware	Model	Duration	Latency
H200 (PyTorch)	small	5 s	0.41 s
H200 (PyTorch)	medium	30 s	0.65 s
H200 (TensorRT)	small	120 s	0.044 s
MacBook Pro M4 CPU	small	120 s	5.92 s

6. Empirical Results: Audio Quality, Adherence, and Editing

Performance is quantitatively and subjectively benchmarked against prior open and proprietary systems (Stable Audio 2.5/Open, ACE-Step 1.5, DiffRhythm 2, Woosh Flow/DFlow, TangoFlux) using FAD (Fréchet Audio Distance, LAION-CLAP embeddings), CLAP Score (cosine similarity), and subjective Mean Opinion Scores (MOS).

• Music Generation: At 120 s, medium attains FAD ≈ 0.107, CLAP ≈ 0.390, MOS_OVL = 4.2/5 in <0.8 s; at 190 s, FAD/CLAP remain tightly bounded (0.116/0.362, 0.100/0.373), exceeding Stable Audio 2.5 at matched latency. The small variant remains competitive at reduced parameter count.
• Sound Effects: At 5 s, large achieves FAD ≈ 0.358, CLAP ≈ 0.370, MOS_OVL = 3.6/5, outperforming Woosh Flow (FAD ≈ 0.580) and TangoFlux. trends persist up to 120 s.
• Variable-Length Fidelity: Across durations (20–380 s), FAD/CLAP remain stable for variable-length-capable models. By contrast, prior fixed-length models degrade sharply (FAD > 0.7) at off-nominal lengths.
• Inpainting and Continuation: On 120 s music, large achieves FAD_full ≈ 0.047, FAD_inpaint ≈ 0.040, CLAP_{text-gen} ≈ 0.289, CLAP_{gen-orig} ≈ 0.878 for single-region edits. Even complex edits (multi-segment, causal) yield only marginal FAD increases.
• Adversarial Post-Training: Reduces inference time from ≈3.9 s (50-step Euler) to ≈0.8 s (8-step ping-pong), with FAD improvements (to 0.101–0.107 for music, 0.259–0.266 for SFX). One-step inference is possible but FAD/CLAP degrade.
• Memory/Latency: small model remains below 2.5 GB VRAM at 120 s, medium at 6.5 GB; TensorRT-accelerated H200 inference achieves sub-0.2 s generation at all tested durations.

7. Significance and Practical Implications

Stable Audio 3 substantiates the efficacy of combining a high-downsampling semantic-acoustic autoencoder with a transformer-based, variable-length latent diffusion process tuned via flow matching, knowledge distillation, and adversarial alignment. The result is multi-minute, high-quality, text-aligned generation and editing on consumer hardware (CPU/GPU), with editing flexibility, prompt adherence, and inference speed significantly exceeding previous models (Evans et al., 18 May 2026). Open access to models and pipelines further supports research and adoption in controllable generative audio synthesis.