PicoAudio2: Temporally-Controlled TTA Framework

• PicoAudio2 is a temporally-controlled text-to-audio generation framework that fuses semantic and timestamp encoding to precisely align audio events.
• It employs a three-module architecture—using a VAE, Flan-T5 text encoder, and a timestamp encoder—to ensure high-fidelity synthesis and detailed event timing.
• The framework supports applications in VR soundscapes, film post-processing, and gaming by providing robust audio quality with precise temporal control.

PicoAudio2 is a temporally-controllable text-to-audio generation (TTA) framework designed to address limitations in sound event category coverage and generalization stemming from exclusive use of simulated data. The system integrates annotated real audio-text data—providing temporally precise alignment—with a timestamp matrix encoding that decouples coarse semantic description from fine-grained event timing, facilitating improved fidelity and controllability in audio synthesis based on natural language instructions.

1. Model Architecture

PicoAudio2 is constructed upon a three-module architecture comprising a Variational Autoencoder (VAE), a text encoder, and a timestamp encoder. The VAE, based on the EzAudio architecture, compresses audio waveforms into latent representations ($A$) for subsequent reconstruction. Textual semantics are captured using a frozen Flan-T5 encoder, which encodes temporally coarse captions (TCC) into semantic features ($C$). The key novelty is the timestamp encoder, which transforms free-form event descriptions from Temporally Detailed Captions (TDC) into a timestamp matrix ($T$), aligned at a fine temporal resolution.

The generative backbone is a Diffusion Transformer (DiT) architecture, consisting of sequential DiT blocks with standard self-attention, cross-attention (fusing text and timestamp features), and feed-forward components. Adaptive Layer Normalization (AdaLN) integrates diffusion step data within each block. Concatenation of the timestamp matrix with audio latents prior to cross-attention enforces explicit alignment of descriptive temporal cues with audio synthesis, improving event placement. This architecture flexibly extends controllability to arbitrary free-form sound events rather than fixed categories.

2. Data Processing Pipeline

To overcome constraints of prior frameworks—specifically fixed sound categories and restricted simulated data—PicoAudio2 employs a bifurcated data processing pipeline incorporating both simulation-generated and real-world annotated data:

• Simulated Data: Adopted from AudioTime. Single-event captions, originally category labels (“barking”), are expanded via LLMs into multiple free-text descriptions. Candidates are evaluated and ranked by CLAP similarity scores and manual assessments, producing a mapping from categories to diverse text descriptors, all coupled with precise timestamp annotations.
• Real Data: Using datasets such as AudioCaps and WavCaps, long captions are segmented into event-level descriptions by LLMs. The Text-to-Audio Grounding (TAG) model generates temporal event annotations. As the TAG model may miss overlapping or unclear events, a filtering phase ensures only samples with non-overlapping, well-defined timestamps are used. The result is a collection of temporally strong audio–TCC–TDC triplets suitable for fine-grained supervision.

This pipeline enables PicoAudio2 to simultaneously train on synthetic precision and naturalistic content, enhancing alignment and generalization.

3. Timestamp Matrix Encoding

Temporal information in PicoAudio2 is formalized via the timestamp matrix: $T \in \mathbb{R}^{T \times C}$ where $T$ is the temporal resolution (20 ms per timestep) and $C$ denotes the feature dimension. For each timestep $t$,

$$T_t = \begin{cases} \sum_i a_i & \text{if event } i \text{ occurs at } t \ 0 & \text{otherwise} \end{cases}$$

Here, each event description is encoded (via Flan-T5) into a feature vector ($a_i$), which are summed whenever multiple events coincide. The matrix is constructed such that for each time slice, the relevant summed features directly correspond to ongoing events. This mechanism enables direct fusion of high-resolution temporal patterns and global semantic information within the diffusion transformer, facilitating nuanced, temporally-aligned audio synthesis.

4. Training Data Regime

PicoAudio2 leverages a mixed training regime:

• Simulated Data: ~64,000 synthetic clips (≤10 sec, 1–4 events per clip), optimized for timestamp and event precision through synthetic diversity.
• Real Data: After data processing, approximately 49,000 “strong” samples (audio–TCC–TDC triplets) and 106,000 “weak” samples (audio–TCC pairs) sourced from AudioCaps and WavCaps; sampling during training employs a 1:2 ratio of weak to strong data, mixing simulated and real modalities.

The inclusion of temporally annotated real-world samples addresses the domain distribution gap, improving both quality and event alignment. This mixed strategy exploits the strengths of simulated and annotated data, contributing to enhanced synthesis fidelity and robustness.

5. Experimental Evaluation

Performance assessment reveals superior outcomes for PicoAudio2 on metrics of both general audio quality and temporal controllability. Standard quality evaluations include Frechet Distance (FD), Kullback–Leibler divergence (KL), Inception Score (IS), CLAP score, and user-rated MOS-Q. Temporal controllability is quantified using Segment-$F_1$ (Seg-$F_1$) and its multi-event variant (Seg-$F_1$-ME). PicoAudio2 consistently outperforms previous models, such as AudioComposer, in aligning sound events to input instructions.

Ablation studies demonstrate that removal of the timestamp matrix degrades temporal accuracy while maintaining moderate audio quality, establishing the necessity of explicit timestamp encoding for fine-grained control. Comparisons of training regimes indicate that incorporating real temporally-annotated data increases both fidelity and alignment performance relative to simulation-only approaches.

6. Application Scope and Implications

PicoAudio2 enables temporally-controllable generation of audio tracks from rich natural language descriptions, supporting use cases requiring precise event placement. These include virtual and augmented reality soundscapes, film and video post-processing, interactive media, content creation for social platforms, assistive audio technology, automated restoration or transformation, and gaming sound design. Its design—decoupling global semantics from event timing—allows for detailed, flexible user control over generated audio. This suggests that PicoAudio2 broadens the applicability of TTA systems to workflows where both audio quality and precise synchronization are required. A plausible implication is the advancement of text-to-audio methodologies in multimedia, offering a foundation for future research aimed at robust, context-aware, event-aligned audio synthesis from free-form language inputs.