SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation

Abstract: Diffusion-based text-to-audio (TTA) models achieve impressive synthesis quality but suffer from high inference latency due to iterative multi-step denoising. Existing one-step approaches alleviate this issue but still rely on paired text--audio data during distillation. To address these limitations, we propose SwiftAudio, a one-step TTA framework that performs audio-free distillation from a pretrained diffusion teacher using only text captions. Specifically, we adapt Variational Score Distillation (VSD) to the audio domain and introduce a temporal smoothness regularization objective to encourage coherent latent audio representations. This design enables the student model to inherit the teacher's generative prior without requiring paired audio supervision and allows effective training with only approximately 45K captions. Experiments on AudioCaps and Clotho demonstrate that SwiftAudio achieves state-of-the-art performance among strict one-step methods and substantially narrows the gap to multi-step diffusion systems. Project page: https://swiftaudio.org/

• The paper introduces caption-only distillation, transferring a multi-step teacher diffusion model to a one-step student without using paired audio data.
• The approach employs Variational Score Distillation with temporal total variation regularization to ensure coherent audio synthesis in a single inference step.
• Experimental results on AudioCaps and Clotho demonstrate competitive performance, matching multi-step methods while using 200X fewer denoising queries.

SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion Generation

Introduction and Motivation

Text-to-audio (TTA) generation using diffusion models has set the standard in high-quality audio synthesis from textual descriptions, but the reliance on slow, multi-step denoising limits practical deployment, especially for latency-sensitive applications. Most accelerated approaches—such as Consistency Models and progressive distillation—still require large paired text–audio datasets and encounter quality degradation under strict one-step inference. The scarcity and high curation cost of paired audio–caption corpora exacerbate this problem when compared to the abundance of caption-only datasets available via LLM and VLM advances.

SwiftAudio introduces a new inference- and data-efficient text-to-audio generation paradigm: caption-only distillation. This is achieved by transferring the generative prior of a strong teacher diffusion model to a one-step student using only text captions and without paired audio during distillation, simultaneously eliminating the step-wise sampling bottleneck and the need for paired data.

Figure 1: Conceptual illustration of existing TTA paradigms with respect to inference efficiency and data requirements. SwiftAudio advances along both dimensions by enabling one-step generation while eliminating the need for paired text--audio data during distillation.

Methodology

Variational Score Distillation in the Audio Domain

SwiftAudio adapts Variational Score Distillation (VSD), previously successful in vision and text-to-image domains, for the audio modality. The essential training setup consists of a one-step student, a frozen multi-step diffusion teacher, and a LoRA-adapted online teacher. The student learns to map Gaussian noise, conditioned on a text prompt, directly to a clean audio latent in a single pass. VSD aligns the student’s implicit distribution with the teacher by minimizing the KL divergence where the score function is approximated via LoRA.

Figure 2: Overview of the proposed SwiftAudio framework. The student $f_\theta$ is trained using the total joint loss $\mathcal{L}$, integrating the VSD guidance from the teacher models and the temporal smoothness constraint on the synthesized latents. The LoRA teacher is alternately updated to estimate the student score.

Temporal Smoothness Regularization

Unlike images, audio signals demand structured temporal coherence. SwiftAudio introduces a temporal total variation (TV) regularization loss, penalizing unnecessary frame-to-frame fluctuations in the generated latent trajectory but allowing abrupt transitions when required by the text prompt (e.g., sound onsets). The $\mathcal{L}_{temp}$ term applies an $L_1$ norm over temporal differences in the latent space, emphasizing piecewise-smoothness without oversmoothing transients.

Figure 3: Conceptual illustration of temporal regularization. Compared to VSD-only generation, adding temporal TV encourages a piecewise-smooth latent trajectory by suppressing spurious frame-to-frame fluctuations, while still allowing localized temporal changes when required by abrupt acoustic events.

Training Procedure

The training alternates between two operations:

• Student Update: Optimize the student with a total loss of $$\mathcal{L}_{\text{total}} = \lambda \mathcal{L}_{\text{temp}} + \mathcal{L}_{\text{VSD}}$$ using only text-caption inputs and synthetic latent samples.
• LoRA Teacher Update: Adapt the small-rank LoRA branch to track student distribution and approximate its score.

At inference, the entire generation process is collapsed to a single deterministic mapping from noise and text to audio with no iterative denoising, providing dramatic latency advantages.

Experimental Results

Synthesis Quality and Efficiency

SwiftAudio is evaluated on AudioCaps and the out-of-domain Clotho benchmark. Remarkably, with $\sim$45K captions (no paired audio used in distillation), SwiftAudio achieves:

• FD 22.73, FAD 2.25, IS 9.13 on AudioCaps (one-step),
• Outperforms AudioLCM and ConsistencyTTA on all objectives,
• Nearly matches its multi-step teacher (Auffusion: FD 22.49) while using 200$\times$ fewer denoising queries,
• Surpasses AudioLDM2 on FD despite the latter using $\sim$15$\times$ more supervised data,
• Retains strong MOS audio quality and relevance scores in subjective evaluation.

Out-of-Domain Robustness

On Clotho, SwiftAudio maintains substantially better transfer performance than prior one-step models—demonstrating that caption-only distillation learns a more abstract, generalizable semantic prior rather than overfitting to specific training domains.

Visualizations and Qualitative Analysis

Mel-spectrogram visualizations demonstrate precise alignment between text prompts and generated audio features in a single inference step, covering a range of acoustic phenomena from transient to continuous events.

Figure 4: Qualitative results of the SwiftAudio framework. The figure illustrates mel-spectrograms of generated audio samples for diverse text prompts, highlighting detailed and varied acoustic patterns created in a single inference step.

Data Efficiency and Ablation

SwiftAudio’s data efficiency is highlighted by:

• Achieving near-SOTA one-step generation using 45K captions (30$\times$ fewer than image domain studies).
• Further improvement as caption count increases, but high-quality results even with modest data scale.

Ablation studies confirm that:

• The diffusion-style student parameterization is crucial,
• LoRA teacher’s rank impacts score approximation accuracy,
• TV regularization is optimal for temporal structure (L1 outperforms L2),
• Removing temporal regularization substantially degrades perceptual quality, especially in FAD.

Semantic Control and Disentanglement

SwiftAudio preserves fine-grained semantic control properties, including:

• Word swapping: Editing keywords (e.g., source, action) precisely alters target content without affecting background.
• Attention reweighting: Scaling token importance modulates spectral and temporal intensity in a continuous fashion.
• Prompt refinement: Appending phrases compositionally augments the auditory scene without disrupting core content.

  Figure 5: Semantic Controllability and Latent Disentanglement in SwiftAudio. Right: Word swapping modifies specific acoustic properties while maintaining scene consistency. Middle: Attention reweighting modulates sound intensity and density. Left: Word refinement compositionally integrates new semantic elements.

Limitations and Future Directions

Current constraints include the fixed-length output (10s) and the lack of linguistic control over explicit speech content, since semantic supervision at the phonetic/lexical level is unavailable. Promising avenues for future research include:

• One-step semantic audio editing for fine-grained modification of existing audio scenes,
• Scaling the approach to longer or more complex soundscapes,
• Integrating more expressive caption corpora and multi-modal supervision.

Conclusion

SwiftAudio demonstrates that efficient, high-fidelity, and highly controllable text-to-audio generation is achievable without the need for paired audio–caption data. By leveraging VSD, LoRA-based online teacher adaptation, and temporal smoothness priors, the approach delivers competitive objective and perceptual quality at unprecedented speed and data efficiency. Caption-only distillation establishes a practical and scalable paradigm for TTA, enabling wider deployment and versatility in generative audio models.

---

Additional Figures

Figure 6: Visualizing semantic control via word swapping. SwiftAudio precisely modifies sound events and acoustic properties while maintaining scene consistency in a single inference step.

Figure 7: Effect of attention reweighting. Increasing token weights enhances intensity and presence of the target sound.

Figure 8: Word refinement in one-step TTA. Additional semantic phrases yield localized, consistent mel-spectrogram changes; primary sound events remain intact while refined concepts are integrated compositionally.