- The paper introduces a unified framework that employs layer-wise deep LLM fusion to seamlessly integrate text-to-audio, TTS, and editing tasks using a shared VAE and DiT backbone.
- It achieves state-of-the-art performance on benchmarks like AudioCaps with impressive metrics (e.g., FAD: 1.558, CLAP: 0.503) and low error rates in TTS without specialized phoneme modules.
- Its unified architecture simplifies deployment by replacing multiple specialized modules with a shared latent space and task-specific mask channels, enabling robust cross-modal audio editing.
UNISON: A Unified Framework for Sound Generation and Editing via Deep LLM Fusion
UNISON proposes an architecture that unifies a broad array of audio tasks, including text-to-audio (T2A) generation, text-to-speech (TTS) (with and without zero-shot speaker cloning), mixed speech-and-sound generation, scene-level audio editing, speech-in-scene editing, and temporally controlled composition, with a single set of model weights. Existing approaches predominantly address such tasks with specialized architectures, typically resulting in isolated latent spaces and bespoke conditioning pipelines which hinder cross-task generalization and complicate deployment. Notably, prior "unified" systems often rely on task-specific auxiliary modules (e.g., mel encoders, phoneme front-ends) and shallow text conditioning mechanisms for generative backbones, limiting representational alignment and compositional instruction following.
UNISON addresses these with two core design innovations:
- Layer-wise Deep LLM Fusion: Hidden states from multiple layers of a frozen LLM are precisely mapped into corresponding MM-DiT blocks via linear projections, yielding depth-matched modality conditioning.
- Architecture Unification: All tasks route through a single VAE and DiT backbone, with task identity encoded by a mask channel and source audio provided as VAE-encoded latents via channel concatenation.

Figure 1: Overview of UNISON. A flow-matching model handles text-to-audio, zero-shot TTS, gender control, scene editing, and timed composition with identical weights, only differentiated by a task mask channel and (optional) source latent concatenation.
Architectural Contributions
DeepFusion MM-DiT with Layer-wise LLM Injection
UNISON utilizes a double-stream MM-DiT backbone. For each DiT block, it injects the Qwen2.5-Omni-7B frozen LLM's hidden states from layer-matched projections, aligning shallow LLM layers (capturing lexical/syntactic structure) with early DiT blocks and deeper layers (abstract semantics) with later blocks. Only ephemeral text tokens are used per block, and text tokens are not propagated to subsequent blocks, obviating redundant or persistent cross-modal conditioning.
Channel-wise task and reference information are encoded through channel concatenation: [zt∥zs∥m], where zt is a noise-perturbed target latent, zs the reference/source latent (if any), and m a per-frame mask for task identity (generation, editing, zero-shot TTS). All inputs, including reference and target, are encoded with the same frozen VAE, further constraining the latent space to be fully shared.

Figure 2: UNISON architecture—layer-wise deep LLM fusion introduces per-layer Qwen features into matched DiT blocks, and double-stream blocks perform joint attention; channel-concatenated inputs encode generation/editing conditions directly.
Unified Multi-task Training and Data Synthesis
UNISON implements an online, GPU-resident pipeline constructing multi-task data on-the-fly from raw corpora—including open-source speech and sound datasets. Editing and composition samples are synthesized by RMS-normalized mixing, fade-ins/outs, and template-based instruction generation. Multi-task curriculum learning is employed: initial training focuses exclusively on generation-style tasks, and editing tasks are introduced later to avoid gradient contention. Homogeneous batching (one task type per batch) further mitigates conflicting gradients.
Empirical Results and Analysis
Text-to-Audio and Text-to-Speech SOTA Performance
UNISON achieves strong performance on all tested domains. On AudioCaps T2A, the D24 variant (16 kHz, 732M params) obtains best-in-class FAD (1.558) and CLAP (0.503), while the D20 variant (44.1 kHz, 621M params) yields lowest FD (15.82) and highest IS (12.04). Both outperform the much larger Audio-Omni (3.05B params) and MMAudio-L (1.03B params) (2605.31530).
In TTS, UNISON (D24) achieves lowest error rates on both English and Chinese (pure WER 1.27%, CER 0.92%; zero-shot WER 1.50%, CER 0.89%), surpassing all baselines including specialist systems. Remarkably, UNISON accomplishes this without any G2P or phoneme front-end.
Flexible Editing and Mixed-Modal Generation
UNISON introduces multimodal editing within its unified architectural regime:
- Audio editing (add, remove, replace events) reaches FD as low as 12.38 and CLAP 0.364, substantially outperforming MMEDIT and Audio-Omni.
- Speech-in-scene editing enables inserting/removing/rewriting speech with high speech removal in the "delete" task (>99%) and maintained intelligibility (WER ≤1.35%) in "insert/rewrite" conditions.

Figure 3: Audio editing: source, UNISON output, and ground truth comparisons highlight accurate event manipulation with preservation of non-edited content.

Figure 4: Speech-in-scene editing: UNISON inserts, deletes, or rewrites speech over complex backgrounds, accurately executing specified instructions.
Timed Audio Generation
Temporal composition (segments assigned to natural-language timestamps) is solved entirely by text instruction parsing. Observed segment-level CLAPs reach 0.308--0.311, and overall scene alignment is even higher (0.345--0.405).

Figure 5: Time-conditioned generation: mel spectrograms visibly align spectral changes with boundaries set by the prompt, showcasing effective audio scheduling from text.
Ablation Studies
Depth-matched layer-wise LLM fusion is essential: broadcasting only the penultimate LLM layer (as in prior art) significantly degrades semantic alignment and instruction adherence (CLAP down, WER up). Double-stream design is required for modality specialization—single-stream models erode both speech and non-speech performance. Increasing LLM size further improves compositional following and speech quality.
Theoretical and Practical Implications
UNISON empirically substantiates that explicit architectural unification—the use of a single encoder, shared latent spaces, and layer-aware text fusion—enables multi-domain, multi-task generalization without incurring the typical degeneration in task-specific performance seen in previous architectures. Its design is especially notable for:
- Removing the dependence on phonemic or mel auxiliary modules for TTS and editing.
- Allowing seamless, cross-modal audio scene manipulations (text-sound-speech mixing, editing, temporal composition) within a single model instance.
- Demonstrating that with sufficient LLM capacity and deep fusion, generic text instructions suffice for both instruction following and control (e.g., gender, timing, speaker) in speech generation.
Practical implications are substantial for deployment: a single 621M–732M parameter checkpoint yields state-of-the-art generation, editing, and composition, dramatically lowering system complexity and hardware requirements relative to specialized ensembles.
Limitations and Future Directions
The primary bottleneck lies in the VAE backbone, which (trained for general environmental sounds) bottlenecks high-fidelity speech timbre, especially for zero-shot speaker identity features. This points to future work on unified VAEs with resolution capable of capturing subtle prosodic and spectral details. Data realism in editing (current algorithmic mixes are less naturalistic than true audio edits) is also a limitation, suggesting more robust or physically-motivated synthesis. Scaling model and data sizes, as well as expanding language and music coverage, are natural and viable directions.
Conclusion
UNISON establishes a scalable, practically deployable architecture for unified sound generation and editing, leveraging deep LLM fusion over a shared latent space and DiT backbone. It demonstrates that instruction alignment and generalization across all major text-audio tasks are feasible without resorting to specialist pathways, and that high-level compositional and temporal control can be realized with architectural minimalism. This facilitates the emergence of general-purpose, instruction-following audio systems and lowers the barriers to broad industrial and research deployment.