Papers
Topics
Authors
Recent
Search
2000 character limit reached

Unified Synthesis of Compositional Speech and Sound from Free-Form Text Prompts

Published 27 May 2026 in cs.SD, cs.AI, and cs.MM | (2605.28063v1)

Abstract: Audio generation has made significant progress, yet synthesizing unified audio where speech and sounds are naturally composited remains a challenge. Current methods either rely on disjoint pipelines, which fail to capture fine-grained interactions, or require structured inputs and external text rewriting, which limits the flexibility of free-form text prompts. In this paper, we introduce a new task: Free-Form-Text-Prompt-to-Unified-Audio generation, which aims to directly synthesize unified audio containing speech, sound, and their composites from unconstrained natural language. To address this task, we propose PlanAudio, a unified, autoregressive LLM-based framework. First, it simplifies the model architecture by leveraging intrinsic LLM reasoning capability instead of traditional text encoders. Second, it introduces a semantic latent chain-of-thought mechanism, an implicit planning mechanism that bridges high-level semantic understanding and low-level acoustic synthesis. Furthermore, we create PlanAudio-Bench, a specialized benchmark for evaluating composite audio scenarios. We perform evaluations in the scenarios of speech, sound, and their composites. The results demonstrate that PlanAudio generally outperforms the existing pipeline and unified baselines, while staying competitive with models designed for a single scenario. Our analysis further reveals the superiority of semantic latent CoT over other CoT mechanisms and highlights the importance of continuous multi-scenario training curricula.

Summary

  • The paper introduces PlanAudio, an autoregressive LLM that uses a six-step semantic latent plan to generate speech, non-speech sounds, and composite audio directly from free-form prompts.
  • On PlanAudio-Bench, PlanAudio achieves a composite Fréchet Distance of 8.52 versus 14.3 for AudioLDM2 and 22.9–25.2 for VoiceLDM, while human evaluations favor its authenticity, timing, and semantic alignment.
  • The paper finds that semantic latent planning outperforms natural-language and acoustic chain-of-thought alternatives, improves semantic coverage, and enables positive transfer between composite, speech, and sound generation tasks.

Motivation and task definition

Text-to-Speech (TTS) and Text-to-Sound (T2S) systems have advanced considerably, but real-world audio frequently interleaves speech and non-speech events whose temporal and semantic relationships are naturally expressed in a single free-form sentence, e.g., "Upbeat music starts playing, Trevor Noah says 'This is great!', followed by loud applause." Existing approaches handle such prompts poorly: pipeline methods rewrite text into separate TTS/T2S tracks and merge outputs, losing fine-grained timing and acoustic interaction; unified models such as VoiceLDM, AudioBox, VoiceEdit, and ControlAudio require structured multi-input conditions or external LLM rewriting, adding system complexity and cascading error risk.

The paper formalizes Free-Form-Text-Prompt-to-Unified-Audio generation: synthesizing a waveform directly from one unconstrained prompt, covering three scenarios — Sound (non-linguistic constraints only), Speech (exact transcriptions), and Composite (both). The authors identify three challenges: the modality gap between high-level semantics and low-level acoustics, the modeling burden of explicit rewriting pipelines, and the scarcity of joint speech-sound annotations in existing corpora (LibriTTS, AudioCaps, WavCaps).

The PlanAudio framework

PlanAudio is an autoregressive LLM initialized from Qwen2.5-1.5B that eliminates both external text encoders and text-rewriting modules; raw prompt tokens serve as the conditional prefix, exploiting the LLM's intrinsic language understanding. Generation is factorized into two phases mediated by a continuous latent sequence z\mathbf{z}:

P(y∣x)=P(z∣x)⋅P(y∣x,z)P(\mathbf{y} \mid \mathbf{x}) = P(\mathbf{z} \mid \mathbf{x}) \cdot P(\mathbf{y} \mid \mathbf{x}, \mathbf{z})

The first phase is a Semantic Latent Chain-of-Thought: over a fixed horizon K=6K=6 steps, the model autoregressively produces continuous features supervised by segment-wise mean-pooled embeddings from the Audio Flamingo 3 encoder, trained with an MSE-plus-cosine-similarity loss. This latent plan acts as an implicit layout specifying event orchestration and prosodic orientation before any acoustic commitment. The second phase predicts hierarchical discrete audio tokens from the AudioCraft codec across QQ codebooks using delayed interleaving, conditioned on both prompt and plan, optimized with cross-entropy. All modalities are serialized into a single sequence delimited by special tokens (<∣sot∣>,<∣sol∣>,<∣soa∣>,<∣eoa∣><|sot|>, <|sol|>, <|soa|>, <|eoa|>), enabling end-to-end training on one Transformer backbone and streaming-compatible inference.

Training uses 1.27M clips (371k composite, 451k sound, 354k speech), each paired with five augmented annotations, fully fine-tuned for 70 epochs on 8×A800 GPUs (~10 days).

PlanAudio-Bench

To address annotation scarcity, the authors construct PlanAudio-Bench from raw AudioSet via a decoupled strategy: Whisper provides transcriptions and Gemini-2.5 Pro generates non-linguistic descriptions; after filtering to intelligible single-speaker samples, 4,500 instances judged to exhibit strong speech-sound interactions form the benchmark, with the remaining 371k used for training. This is a practical but notable dependency: composite ground truth is LLM-annotated rather than human-curated, so benchmark quality inherits captioner biases.

Main results

Evaluation spans objective metrics (FD/KL/IS/CLAP for sound; WER/UTMOS for speech) plus a 50-sample subjective study on four criteria.

Composite scenario. On PlanAudio-Bench, PlanAudio achieves FDPANNs_{\text{PANNs}} of 8.52 versus 14.3 for the AudioLDM2 pipeline and 22.9–25.2 for VoiceLDM, with KL and CLAP also leading among generated outputs. Human raters place it first on all four dimensions (e.g., Authenticity 3.47 vs. 2.93 for VoiceLDM-m), particularly on Temporal Correctness and Semantic Alignment — direct evidence that the latent CoT improves orchestration of interacting events. Spectrogram case studies show correct temporal ordering (drum → cymbal → speech) where baselines fail. The paper concedes that WER (0.41) trails VoiceLDM-m (0.09), attributing this to the codec's reconstruction ceiling and to training on real-world AudioSet recordings rather than VoiceLDM's clean synthetic composites — a trade-off it argues favors authenticity, supported by the subjective scores.

Single-scenario generalization. With frozen weights and no task indicators, PlanAudio beats VoiceLDM on all sound metrics (FDPANNs_{\text{PANNs}} 24.7 vs. 55.8) and all speech metrics (WER 0.11, UTMOS 3.11), remaining competitive with specialists such as Tango and PromptTTS++. Notably, its text-encoder-free design matches encoder-based baselines on CLAP, indicating minimal loss of semantic grounding from removing the text encoder.

Analysis of CoT mechanisms

Ablations against three alternatives — no CoT, Explicit CoT (Gemini-generated natural-language reasoning chains), and Acoustic CoT (codec vectors as supervision) — support four claims. First, semantic latent CoT wins consistently across all benchmarks. Second, Explicit CoT's benefit diminishes in Composite scenarios because natural language cannot expressively capture dense overlapping interactions. Third, Acoustic CoT performs worst overall, exposing a reconstruction-semantic mismatch: codec features optimized for fidelity lack the semantic depth needed for planning. Fourth, the authors introduce the Semantic Coverage Factor (SCF), a confidence-weighted measure of whether prompted events appear in the output; PlanAudio attains SCF 0.34 versus 0.09–0.20 for variants, indicating the latent plan functions as a robust anchor against semantic omission. These ablations use a reduced configuration (50 epochs, one-third data), so absolute numbers are not comparable to the main tables.

Data curriculum findings

Comparing Constant, Gradual, and Disjoint sampling schedules reveals that uniform mixing outperforms staged curricula, and that abrupt shifts to exclusive composite training cause catastrophic forgetting of single-scenario skills. A more interesting result is synergistic transfer: Constant and Gradual beat Disjoint on sound/speech even when Disjoint devotes early training exclusively to those scenarios, suggesting that learning to decouple compositional prompts forces more discriminative representations that accelerate single-scenario proficiency. Practically, this means the framework requires no meticulous ratio tuning.

Limitations and open questions

The paper is candid about several constraints. Speech intelligibility in composite settings remains behind specialized models, bounded by the AudioCraft codec's reconstruction quality; stronger codecs are identified as future work. The optimal balance between semantic planning and acoustic fidelity (the λ1/λ2\lambda_1/\lambda_2 trade-off) is left unexplored. Benchmark annotations are machine-generated, and the fixed planning horizon K=6K=6 is set empirically without analysis of sensitivity or of how latent plans scale with prompt complexity. The authors also acknowledge misuse risk for deceptive audio synthesis without proposing mitigations.

Conclusion

PlanAudio demonstrates that a single autoregressive LLM, equipped with a semantically supervised latent chain-of-thought and trained under a constant multi-scenario curriculum, can synthesize speech, sound, and their composites directly from free-form text — outperforming pipeline and unified baselines in composite generation while staying competitive with specialists. The central empirical contributions are the superiority of semantic latents over textual and acoustic intermediate representations, quantified partly through the new SCF metric, and evidence of positive transfer from composite to single-scenario training. Open questions include codec-limited speech fidelity, planning-horizon scaling, and the reliability of LLM-annotated composite benchmarks.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.