DreamAudio: Customizable Audio Synthesis

• DreamAudio is a customizable text-to-audio system that uses latent diffusion models and user-provided references to achieve fine-grained acoustic control.
• It integrates multi-modal encoding with rectified flow matching to blend global semantic intent with precise auditory features.
• The framework demonstrates superior performance with lower FAD/KL metrics and higher CLAP scores, supporting applications in film, gaming, and accessibility.

DreamAudio refers to a class of generative, customizable text-to-audio synthesis systems built on latent diffusion models with explicit mechanisms for user-driven control of fine-grained sound characteristics. These systems move beyond prior text-to-audio approaches by incorporating user reference concepts (paired audio-caption samples) to guide the generative process, enabling the creation of audio outputs that are semantically aligned with a text prompt and also match detailed, personalized acoustic features. DreamAudio is formally introduced in "DreamAudio: Customized Text-to-Audio Generation with Diffusion Models" (Yuan et al., 7 Sep 2025), and the term also serves as a conceptual label for a broader paradigm shift toward highly controllable, modular, and interactive audio generation in both academic and applied settings.

1. Customized Text-to-Audio Generation: Motivation and Problem Definition

Traditional text-to-audio generation models primarily focus on producing audio samples that are globally semantically consistent with a given text description. However, these models fail to accommodate scenarios where users require specific auditory features—such as reproducing the timbre, rhythm, or idiosyncratic “event” characteristics—of custom or rare sounds. This limitation is especially salient in fields such as film production, game audio design, and accessible multimedia where one often needs to generate sounds that must satisfy both descriptive semantics and user-imposed acoustic constraints.

DreamAudio addresses the customized text-to-audio (CTTA) task, where new audio must be generated that (i) adheres to a target text prompt and (ii) incorporates fine-grained features identified from a set of user-provided reference audio-caption pairs. The system is designed to generalize across both synthetic benchmarks and real-world CTTA use cases.

2. Core Architecture and Conditioning Mechanisms

DreamAudio is built on a latent diffusion generative model, enhanced through both algorithmic and architectural innovations to facilitate customized generation.

2.1. Multi-Modal Encoding

• Language Conditioning: A pretrained FLAN-T5 encoder processes the text prompt (“target prompt”) as well as the captions corresponding to the reference audio samples. These embeddings capture global semantic intent and context.
• Audio Conditioning: A VAE encoder processes each user-provided reference audio, producing a latent representation that encodes fine-grained auditory details.
• Condition Fusion: Both sets of embeddings (reference and target) participate as conditioning vectors in the main generative process.

2.2. Diffusion with Rectified Flow Matching (RFM)

Classical discrete-time diffusion models, such as DDPM, progress through a fixed sequence of noise steps and train a denoising network with step-dependent targets. DreamAudio instead employs a rectified flow matching scheme in the continuous interval λ ∈ [0, 1], improving numerical smoothness and allowing cross-modal information to be blended seamlessly at all timescales.

Let $z_0$ denote pure noise and $z_1$ denote the clean target latent. The noisy latent at interpolation “time” λ is:

$z_\lambda = (1 - \beta\lambda) z_0 + \lambda z_1$

with stability parameter β = 1 – σ. Rather than predicting stepwise noise, the model regresses to a velocity vector $v = z_1 - \beta z_0$ that is independent of λ. The RFM loss is:

$$\mathcal{L}_{\text{RFM}}(\theta) = \mathbb{E}_{\lambda, z_1, z_0} \big\| \mu(z_\lambda, R, \lambda, E, C) - v \big\|^2$$

where:

• $R$ = latent audio representations from the reference clips
• $E$ = concatenated reference caption embeddings
• $C$ = embedding of the target text prompt
• $\mu$ = neural network predicting the velocity field

This continuous paradigm yields higher stability and expressive flexibility compared to discrete-time diffusion.

2.3. Multi-Reference Customization Module (MRC)

To maximize compositional control, DreamAudio employs a two-branch UNet architecture:

• Branch 1: Processes $z_\lambda$ with target prompt conditioning.
• Branch 2: Independently encodes and transforms the stacked reference features $(R, E)$ using specialized encoder blocks.
• Up-sampling Fusions: Dedicated cross-attention layers and external linking modules fuse these branches during up-sampling, ensuring that both global semantics and specific reference-driven features direct synthesis at all stages.

This design is inspired by modular conditioning strategies (e.g., ControlNet in image generation) but is adapted for audio and multi-reference inputs.

3. Training Methodology and Datasets

3.1. Dataset Construction

Two complementary datasets were constructed for robust training and evaluation of the CTTA task:

Dataset Name	Construction Method	Purpose
Customized-Concatenation	Concatenates several short (≤5s) clips into a 10s sample; references are grouped and multi-captioned	Benchmarks sequential event reproduction and order-following
Customized-Overlay	Overlays short event clips onto a base track with random SNR mixing	Emulates natural mixed-audio scenarios; tests model robustness to overlaying

Additionally, a human-curated Customized‑Fantasy dataset was created with challenging sound events (e.g., monster fights, laser guns) for real-world evaluation.

3.2. Evaluation Metrics

• Fréchet Audio Distance (FAD)
• Kullback-Leibler (KL) divergence
• CLAP / CLAP_A scores: Evaluate semantic and feature alignment between generated and reference audio.
• AudioBox Aesthetics: Automated and human-judged production quality and content usefulness.
• Subjective ratings: Overall impression and audio-text relation from human raters.

DreamAudio achieves significantly lower FAD/KL and higher CLAP metrics than retrieval-based and baseline generation approaches, demonstrating effective alignment of synthesized audio to both semantic and customization constraints.

4. Customization and Fine-Grained Control Capabilities

DreamAudio provides explicit, tuning-free integration of user-specified sound features:

• Feature Extraction: VAE and language embeddings recover detailed auditory and semantic information from reference audio-caption pairs.
• Conditioned Synthesis: The UNet and cross-attention structure ensure that generated audio is modulated both globally (semantic intent) and locally (specific reference properties such as timbre, rhythm, event type).
• Temporal and Overlay Control: By training on customized-concatenation and overlay datasets, DreamAudio can control both which events appear and their temporal arrangement or overlap, a requirement for many multimedia or post-production tasks.

5. Broader Applications and Benchmarks

DreamAudio’s CTTA capabilities underpin applications that demand custom and rare event synthesis:

• Film/game/advertising audio design: Rapid generation of unique, fantasy, or branded sound effects.
• Virtual environments and educational platforms: Creation of immersive, event-specific soundscapes that require high semantic fidelity and precise acoustic features.
• Accessibility: Synthetic but personalized audio cues for assistive technologies.

Evaluation on human-involved datasets shows that DreamAudio produces samples with high subjective quality and maintains generalization ability for non-customized TTA tasks, indicating broad domain applicability.

6. Relationship with Prior and Emerging Paradigms

DreamAudio extends earlier work on audio diffusion, instruction-following modification (e.g., AUDIT (Wang et al., 2023)), and external control in image synthesis, adapting these advances for customized audio generation. Distinctions include:

• Reference-based Customization: Enables specific user-driven control over output, distinct from prompt-only systems.
• Rectified Flow Matching: Provides smooth, non-discrete transformation, improving stability and flexibility compared to discrete DDPM frameworks.
• Modular Conditioning: The split UNet and customization module generalizes modular control principles emerging in image and multimodal generative modeling.

A plausible implication is that similar frameworks could be adapted for video-to-audio, co-separation, or interactive audio editing applications with further conditioning signal integration or additional interactive elements.

7. Future Directions and Limitations

Current DreamAudio systems are distinguished by their ability to integrate reference examples in an end-to-end manner without requiring additional tuning or retraining for each customization task. Nevertheless, challenges remain:

• Scaling to complex, longer, or layered real-world scenes may require further innovations in data representation and model capacity.
• Reference diversity and coverage affect fidelity; selecting or augmenting references could improve generalizability.
• Computational resource demands for large-scale, low-latency deployment remain an area for active optimization.

Recent progress in audio-LLMs (e.g., DeSTA2.5-Audio (Lu et al., 3 Jul 2025), Audio Flamingo 3 (Goel et al., 10 Jul 2025)) and benchmarks for fine-grained control and interpretability suggest that future iterations may enhance DreamAudio with compositional language understanding, model-based control of non-textual fine structure, and interactivity across modalities.

---

DreamAudio defines a notable advancement in controllable, reference-guided text-to-audio generation, combining novel diffusion-based architectures, explicit customization modules, and engineered training benchmarks to produce user-driven, semantically coherent, and acoustically precise audio outputs (Yuan et al., 7 Sep 2025).