Make-An-Audio 2: Advanced Text-to-Audio Generation

• Make-An-Audio 2 is a latent diffusion-based text-to-audio generation model that uses temporal parsing and dual text encoders to improve semantic alignment and variable-length audio synthesis.
• It replaces conventional 2D U-Nets with a temporal transformer-based denoiser, enhancing temporal consistency and reducing artifacts in generated audio.
• The model leverages LLM-guided data augmentation to expand training samples, achieving superior objective and subjective performance on standard T2A benchmarks.

Make-An-Audio 2 is a latent diffusion-based text-to-audio (T2A) generation model designed to address shortcomings in semantic alignment and temporal consistency that affect prior T2A systems. Traditional approaches, including those relying on 2D spatial structures (e.g., 2D U-Nets), often suffer from misaligned semantics and poor handling of variable-length audio, with limited modeling of temporal information. Make-An-Audio 2 introduces temporal parsing, dual text encoders, a transformer-based denoiser focused on the temporal dimension, and LLM-powered data augmentation to achieve state-of-the-art results in both objective and subjective metrics on standard T2A benchmarks (Huang et al., 2023).

1. Latent Diffusion Framework for T2A Generation

Make-An-Audio 2 models audio synthesis as a latent-space denoising diffusion process over mel-spectrogram representations. Given an input mel-spectrogram $x \in \mathbb{R}^{C_a \times T}$, a 1D-convolutional variational autoencoder (VAE) encoder $E$ compresses $x$ into a latent code $z = E(x) \in \mathbb{R}^{d \times L}$, where $L$ reflects audio duration.

During training, diffusion operates in this latent space over $T$ discrete steps, with a fixed noise schedule $\{\beta_t\} \subset (0,1)$:

• Forward (Noising) Process: At step $t$,

$$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$

By induction, $$q(z_t \mid z_0) = \mathcal{N}\left(\sqrt{\bar{\alpha}_t} z_0,\, (1-\bar{\alpha}_t) I\right)$$ with $E$0.

• Reverse (Denoising) Process: A neural network $E$1 parameterizes the denoising step as

$E$2

where typically

$E$3

and $E$4.

• Training Objective: The loss is a mean squared error in noise space:

$E$5

with $E$6, $E$7.

• Classifier-Free Guidance: To control tradeoffs between generation diversity and conditioning faithfulness, classifier-free guidance is used:

$E$8

with $E$9 as optimal guidance scale.

This architecture prioritizes efficient learning in the more tractable latent space and allows variable-length audio generation (Huang et al., 2023).

2. Temporal Transformer-Based Diffusion Denoiser

Make-An-Audio 2 replaces the 2D U-Net backbone of earlier methods with a feed-forward transformer acting solely along the temporal axis of the latent variable $x$0. This temporal modeling design includes:

• Initial 1D-convolution for channel mixing: $x$1,
• Addition of timestep embeddings $x$2 and conditional text embedding $x$3 using FiLM or simple addition: $x$4,
• Temporal self-attention (“FFT block”): $x$5, followed by scaled dot-product attention and residual connections,
• Position-wise feedforward layers with GELU activation and layer normalization.

The model stacks $x$6 FFT blocks with $x$7 heads and hidden dimensionality $x$8. Attention cost scales as $x$9, supporting variable-length audio inputs. This structure improves handling of true temporal relationships, avoiding artifacts from 2D “image-like” architectures that ignore audio’s sequential structure (Huang et al., 2023).

3. Temporal Parsing and Dual Text Encoder Fusion

Key to Make-An-Audio 2’s improvements in semantic and temporal alignment is temporal parsing with a dual text encoder system. Natural-language captions $z = E(x) \in \mathbb{R}^{d \times L}$0 are transformed via an LLM (e.g., GPT-4) to produce structured representations:

$z = E(x) \in \mathbb{R}^{d \times L}$1

This enables explicit modeling of event order (e.g., $z = E(x) \in \mathbb{R}^{d \times L}$2car door slamming$z = E(x) \in \mathbb{R}^{d \times L}$3paper_content$z = E(x) \in \mathbb{R}^{d \times L}$4start$z = E(x) \in \mathbb{R}^{d \times L}$5@$z = E(x) \in \mathbb{R}^{d \times L}$6footsteps$z = E(x) \in \mathbb{R}^{d \times L}$7paper_content$z = E(x) \in \mathbb{R}^{d \times L}$8mid$z = E(x) \in \mathbb{R}^{d \times L}$9...). Extraction is performed by prompting the LLM with a fixed template.

This structured text $L$0 is encoded by a fine-tuned T5 “temporal encoder” $L$1. The original textual caption $L$2 is also encoded via a frozen CLAP model $L$3. The two embeddings are fused:

$L$4

with $L$5 a learned linear layer and $L$6 denoting vector concatenation. This dual-encoder scheme assigns temporal-reasoning to the LLM+T5 stack while CLAP provides fidelity to semantic style and detail, significantly enhancing alignment for complex temporally-structured prompts (Huang et al., 2023).

4. Data Augmentation via LLM-Guided Synthesis

To address data scarcity for temporally-annotated T2A supervision, Make-An-Audio 2 synthesizes approximately 61k extra training samples using LLM-guided augmentation:

1. From a database $L$7 of single-label audio clips $L$8, $L$9 samples are chosen and temporally concatenated (with optional overlap).
2. Each event is labeled by temporal position (“start”, “mid”, “end”, “all”).
3. The resultant structured caption $T$0 is generated.
4. An LLM rewrites $T$1 into coherent natural-language caption $T$2.

Pseudocode for this augmentation:

$$q(z_t \mid z_0) = \mathcal{N}\left(\sqrt{\bar{\alpha}_t} z_0,\, (1-\bar{\alpha}_t) I\right)$$0

This strategy yields a large, high-quality, temporally-diverse training set supporting robust learning for variable-length conditional generations (Huang et al., 2023).

5. Training Protocol and Loss Functions

The training corpus comprises 0.92M audio-text pairs (≈3.7k hours), pooled from AudioCaps, WavCaps, AudioSet, ESC-50, FSD50K, TUT, EpidemicSound, AdobeStock, UrbanSound, etc., augmented by the 61k LLM-built pairs and pseudo-prompts.

• VAE Training: Loss combines
  • $T$3 reconstruction: $T$4
  • GAN realism penalty: $T$5
  • Latent KL-penalty: $T$6

$T$7

• Diffusion Training: Mean-squared noise regression loss $T$8, using either learnable schedules $T$9 or cosine schedules, and classifier-free guidance with $\{\beta_t\} \subset (0,1)$0.
• Text Encoder Training: CLAP is frozen, T5 is fine-tuned on structured captions, $\{\beta_t\} \subset (0,1)$1 is trained end-to-end.
• Optimization: AdamW optimizer, $\{\beta_t\} \subset (0,1)$2 for diffusion, batch size $\{\beta_t\} \subset (0,1)$3 GPUs, total $\{\beta_t\} \subset (0,1)$4M steps. VAE trained for $\{\beta_t\} \subset (0,1)$5k steps ($\{\beta_t\} \subset (0,1)$6).

A table summarizing the training configuration:

Component	Architecture	Key Hyperparameters
Denoiser	FFT transformer	8 blocks, 8 heads, $\{\beta_t\} \subset (0,1)$7
Conv1D	1D kernel	size=7, padding=3
VAE optimizer	AdamW	lr=$\{\beta_t\} \subset (0,1)$8
Diffusion opt.	AdamW	lr=$\{\beta_t\} \subset (0,1)$9
GPUs	-	$t$0

6. Experimental Evaluation and Comparative Results

Make-An-Audio 2 is evaluated on AudioCaps (test) and in zero-shot settings on Clotho and AudioCaps-short. Metrics include Frechet Distance (FD), Inception Score (IS), KL divergence (KL), Frechet Audio Distance (FAD), CLAP-score, and Mean Opinion Scores for quality/faithfulness (MOS-Q/MOS-F).

• On AudioCaps (100 DDIM steps, 937M parameters): $t$1, $t$2, $t$3, $t$4, $t$5, $t$6-$t$7, $t$8-$t$9.
• By comparison, previous SOTA (TANGO, 1.21B params): $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$0, $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$1, $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$2, $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$3, $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$4, $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$5-$$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$6.
• Predecessor Make-An-Audio: $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$7, $$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$8-$$q(z_t \mid z_{t-1}) = \mathcal{N}\left(\sqrt{1-\beta_t} z_{t-1},\, \beta_t I\right)$$9.

Zero-shot results on Clotho and short AudioCaps exhibit strong gains over all baselines, particularly in temporal and semantic metrics as well as on variable-length audio synthesis.

Ablations show:

• Addition of structured T5 to CLAP yields substantial FD/IS improvements.
• Dual-encoder fusion outperforms single T5 on all alignment criteria.
• Replacement of 2D-U-Net with FFT backbone restores and exceeds performance for variable-length and temporally-complex data.

In summary, the innovations of Make-An-Audio 2—temporal parsing, dual-encoder fusion, temporal transformer-based diffusion, and augmented training data—are collectively responsible for advances in T2A generation quality and flexibility, establishing new state-of-the-art results for the field (Huang et al., 2023).