FreeAudio: Controllable Long-Form T2A

Updated 4 July 2026

FreeAudio is a training-free framework for controllable long-form text-to-audio synthesis that overcomes timing precision issues and extends generation beyond short clips.
It employs an LLM-based planning stage to convert global captions into non-overlapping local windows, ensuring explicit timing and enhanced event sequence control.
The method integrates decoupled and aggregated attention manipulation with latent composition across overlapping segments to achieve coherent and high-fidelity audio outputs.

Searching arXiv for the specified paper and closely related work on training-free audio generation/editing and open audio instruction corpora. FreeAudio is a training-free framework for controllable long-form text-to-audio generation that targets two failure modes of contemporary T2A systems: weak obedience to explicit timing constraints and degradation beyond the native short-clip horizon of the backbone model. It is presented as the first training-free method to jointly support timing-conditioned and long-form T2A synthesis. The framework operates by converting a global caption and timestamped event prompts into non-overlapping, recaptioned local windows, then imposing those plans on a pretrained DiT-based latent diffusion model through inference-time attention manipulation, latent composition across overlapping segments, and reference-guided consistency control (Jiang et al., 11 Jul 2025).

1. Problem formulation and system assumptions

FreeAudio assumes an overall scene caption $y^c$ , a target duration $M$ , and timing prompts

$y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$

where $p_i^c$ is a fine-grained event description and $[s_i,e_i]$ is its intended time interval. The central requirement is not merely semantic plausibility, but explicit control over when events occur, including prompts of the form “owl hooted at 2.4s–5.2s and crickets chirping at 0s–24s” (Jiang et al., 11 Jul 2025).

The method is built on a pretrained DiT-based latent diffusion T2A backbone with an audio VAE, a text encoder, and a diffusion transformer. In the reported implementation, the text encoder is Flan-T5; audio is 16 kHz mono; the VAE is based on Oobleck; the diffusion backbone has 24 layers, 24 attention heads, embedding dimension 1536, internal token dimension 64, conditional token dimension 768, global condition embedding dimension 1536, and latent output dimension 64. The backbone is trained for variable-length generation up to $10$ seconds, so the native maximum duration is treated as $M_{\max}=10$ s (Jiang et al., 11 Jul 2025).

A recurring misconception is that “training-free” implies the absence of any pretrained model. In FreeAudio, training-free refers only to the control layer: no new weights are learned for timing planning or long-form generation, and no test-time optimization over model parameters is introduced. The backbone itself is pretrained on public datasets including AudioCaps, AudioSet, WavCaps, FSD50k, VGG-Sound, MTT, MSD, and FMA, all standardized to 16 kHz mono with maximum length 10 s (Jiang et al., 11 Jul 2025).

2. LLM-based planning and recaptioned temporal windows

The first stage converts overlapping or incomplete user specifications into a sequence of non-overlapping windows,

$y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$

with

$0 = t_1 < t_2 < \cdots < t_{k+1} = M.$

FreeAudio constructs these windows by collecting all unique start and end times, sorting them, and then assigning to each elementary window the event prompts whose intervals cover it. If intervals are uncovered, the LLM uses the overall caption, nearby assigned events, and scene-level common sense to infer plausible content. Each window is then recaptioned into a natural-language prompt better matched to the training distribution of the T2A backbone (Jiang et al., 11 Jul 2025).

This planning stage resolves overlap in the text domain rather than by directly mixing multiple timing conditions inside the diffusion model. The paper’s rationale is that direct overlap handling in attention space is unstable, whereas non-overlapping windows let the text encoder operate on coherent local descriptions. An input such as a list of fragments can therefore be rewritten as a natural local caption, e.g., a window-level description that jointly expresses frying, dog barking, and background water (Jiang et al., 11 Jul 2025).

The planner also serves as an interface normalizer. The appendix example described in the paper shows that noisy formatting and misspellings can be regularized into clean windowed descriptions. This suggests that FreeAudio is not merely a timing-conditioned sampler, but a planning-and-generation pipeline in which prompt canonicalization is structurally important to performance (Jiang et al., 11 Jul 2025).

3. Decoupling and aggregating attention control

The timing-control mechanism operates on a batch of size $1+k$: one base latent conditioned on the full-scene prompt $M$ 0 and $M$ 1 auxiliary sub-latents conditioned on the local prompts $M$ 2, with durations

$M$ 3

Only the denoised base latent is retained as the final sample; the sub-latents are auxiliary guides (Jiang et al., 11 Jul 2025).

The unmodified attention operator is

$M$ 4

FreeAudio alters this operator in two complementary ways. In Decoupling Attention Control, the base query latent in cross-attention is segmented according to the planned windows, producing $M$ 5. For the $M$ 6-th sub-latent, the first $M$ 7-duration positions are replaced by the corresponding segment $M$ 8, while the remaining positions are left unchanged. The paper reports that applying this decoupling to self-attention hurt timing precision, so it is restricted to cross-attention (Jiang et al., 11 Jul 2025).

In Aggregating Attention Control, the active output segment from each sub-latent is extracted as $M$ 9 and concatenated in temporal order:

$y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 0

This timing-guided composite is then fused back into the base output:

$y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 1

Aggregating Attention Control is applied in both cross-attention and self-attention. The ablations treat $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 2 as the cross-attention fusion ratio and $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 3 as the analogous self-attention fusion strength. Lower $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 4 and higher $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 5 improve temporal alignment but can degrade fidelity, whereas higher $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 6 preserves more global coherence from the base prompt (Jiang et al., 11 Jul 2025).

A useful interpretation is that decoupling gives each temporal region access to its local textual condition without abandoning the global latent trajectory, while aggregation reimposes the locally aligned outputs onto a single full-length sample. This suggests that FreeAudio is not segment-wise concatenation disguised as diffusion control; it is a joint denoising procedure with explicit intra-batch temporal factorization (Jiang et al., 11 Jul 2025).

4. Long-form generation beyond the backbone horizon

For durations $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 7, FreeAudio introduces three additional mechanisms: Contextual Latent Composition, Reference Guidance, and Contextual Trim-and-Concatenate. The long-form clip is divided into overlapping segments, typically of length $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 8 with overlap $y^t = \{(p_i^c, s_i, e_i)\}_{i=1}^n,$ 9, and neighboring segments share latent content during denoising:

$p_i^c$ 0

This is intended to smooth local transitions by mixing half-overlap regions in latent space before decoding rather than relying only on waveform postprocessing (Jiang et al., 11 Jul 2025).

Reference Guidance addresses global drift. For segment $p_i^c$ 1, if $p_i^c$ 2 is the self-attention query, $p_i^c$ 3 the original self-attention output, and $p_i^c$ 4 reference key-value features, the guided output is

$p_i^c$ 5

The reference may come from input reference audio or synchronously generated audio features. Increasing $p_i^c$ 6 improves FAD, KL, FD, Intra-LPIPS, and Intra-CLAP on the reported AudioCaps 26 s ablation, but excessively large $p_i^c$ 7 makes the output too uniform and monotonous; the paper recommends $p_i^c$ 8 (Jiang et al., 11 Jul 2025).

After denoising, segments are decoded by the pretrained VAE, overlap regions are trimmed, and the waveforms are concatenated. The significance of this design is that local continuity is enforced in latent space while global consistency is encouraged by self-attention to reference features. A plausible implication is that FreeAudio distributes the long-form problem across three scales: planning at the caption/window level, temporal control inside the diffusion process, and coherence regularization across overlapping segment boundaries (Jiang et al., 11 Jul 2025).

5. Empirical evaluation

For timing-controlled generation, the benchmark is the AudioCondition test split: 1,110 samples from AudioSet-Strong with frame-level timestamps and 456 event classes. Metrics include event-based measure (Eb), clip-level macro F1 (At), FAD, KL, CLAP, and subjective scores for temporal alignment and overall quality. FreeAudio reports

Eb $p_i^c$ 9
At $[s_i,e_i]$ 0
FAD $[s_i,e_i]$ 1
KL $[s_i,e_i]$ 2
CLAP $[s_i,e_i]$ 3
subjective temporal $[s_i,e_i]$ 4
subjective overall $[s_i,e_i]$ 5

The reported comparison highlights that FreeAudio is essentially tied with AudioComposer-L on Eb ( $[s_i,e_i]$ 6 versus $[s_i,e_i]$ 7), achieves the best At ( $[s_i,e_i]$ 8 versus $[s_i,e_i]$ 9 for AudioComposer-L), and obtains the best or near-best quality metrics, including the best FAD ($10$0, slightly better than Stable Audio’s $10$1) and best KL ($10$2) (Jiang et al., 11 Jul 2025).

For long-form generation, evaluation is conducted on AudioCaps and MusicCaps at 10 s, 26 s, and 90 s. On AudioCaps, FreeAudio reports:

10 s: FAD $10$3, KL $10$4, FD $10$5, CLAP $10$6
26 s: FAD $10$7, KL $10$8, FD $10$9, CLAP $M_{\max}=10$ 0
90 s: FAD $M_{\max}=10$ 1, KL $M_{\max}=10$ 2, FD $M_{\max}=10$ 3, CLAP $M_{\max}=10$ 4

These are described as the best among listed methods on all four AudioCaps metrics at 10 s and 26 s, and still strongest at 90 s. On MusicCaps, FreeAudio remains robust but Stable Audio is competitive or stronger on some metrics, especially CLAP and sometimes KL or IS. FreeAudio’s reported MusicCaps results are:

10 s: FAD $M_{\max}=10$ 5, KL $M_{\max}=10$ 6, IS $M_{\max}=10$ 7, CLAP $M_{\max}=10$ 8
26 s: FAD $M_{\max}=10$ 9, KL $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 0, IS $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 1, CLAP $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 2
90 s: FAD $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 3, KL $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 4, IS $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 5, CLAP $y^p = \{(p_j^c, t_j, t_{j+1})\}_{j=1}^k,$ 6 (Jiang et al., 11 Jul 2025).

Subjective long-form evaluation reports the following scores.

Dataset	Quality	Consistency	Coherence
AudioCaps	4.08	4.12	4.20
MusicCaps	3.88	4.36	4.34

These are described as the best among compared methods on both datasets. Category-wise timing evaluation further indicates stronger performance on stable long-duration sounds such as Toothbrush, Frying, and Vacuum Cleaner, and weaker alignment on Cat, Dog, and Dishes, which the paper characterizes as shorter, weaker, or label-ambiguous events (Jiang et al., 11 Jul 2025).

FreeAudio occupies a specific niche within open audio research: training-free controllable generation rather than supervised instruction tuning, audio-language reasoning, or source-conditioned editing. Audio-FLAN, for example, is a large-scale instruction-tuning corpus with 108.5 million instances across speech, music, and sound, but it introduces neither a T2A architecture nor a canonical audio tokenization pipeline; its role is a supervision and formatting layer for unified audio-LLMs rather than inference-time control over a pretrained generator (Xue et al., 23 Feb 2025). Audio Flamingo 3 is a fully open large audio-LLM focused on understanding and reasoning across speech, sound, and music, with long-audio reasoning up to 10 minutes and multi-turn, multi-audio chat, but it is not a timing-planned T2A system (Goel et al., 10 Jul 2025).

Within training-free editing, FreeAudio is adjacent but not equivalent to AudioEditor and FreeSonic. AudioEditor is a diffusion-inversion framework for addition, deletion, and replacement based on a pretrained TTA model, using Null-text Inversion, EOT-suppression, and attention loss to balance editability and preservation (Jia et al., 2024). FreeSonic, by contrast, is a training-free source-conditioned editor built on Rectified Flow-based TangoFlux, using attention-derived temporal masks, scheduled attention decoupling, and masked task-oriented noise injection for addition, removal, and replacement (Jiang et al., 13 Jun 2026). FreeAudio differs from both in that it addresses timestamped prompt planning and long-form synthesis rather than source-audio editing.

The paper also implies several limitations. First, the framework depends on the pretrained T2A backbone; if the base model renders an event poorly, inference-time control does not correct the underlying generative deficiency. Second, prompt decomposition quality depends on the LLM planner. Third, overlap is resolved by converting simultaneous events into a single local caption over a non-overlapping window, which simplifies genuinely polyphonic structure. Fourth, temporal precision remains bounded by latent temporal discretization, diffusion dynamics, and the sound event detector used for evaluation. Fifth, very long durations require more windows and segments, so planning complexity and generation cost increase with duration. Sixth, excessive Reference Guidance reduces local diversity. Finally, long-form evaluation is constrained by the fact that AudioCaps and MusicCaps captions correspond to 10-second clips, so the semantics of 26 s and 90 s generation are only indirectly supervised and evaluated (Jiang et al., 11 Jul 2025).

Taken together, these properties make FreeAudio important less as a universal audio foundation model than as a methodological demonstration that fine-grained temporal control and long-horizon synthesis can be added to an open pretrained T2A backbone without retraining. In that respect, it complements semantic-space generation approaches such as SemanticAudio, which emphasize semantic planning and training-free text-guided editing in semantic space but are trained on AudioCaps and short 10-second clips, and it complements open long-context audiovisual generation work such as LD-LAudio-V1, which extends short-form video-to-audio generation to long-form synthesis through dual lightweight adapters rather than training-free attention control (Dai et al., 29 Jan 2026, Zhang et al., 14 Aug 2025).