Generative Audio Compression (GAC)

• Generative Audio Compression (GAC) is a paradigm that employs deep generative models to reconstruct high-fidelity audio from highly compressed, semantically rich latent representations.
• It leverages conditional normalizing flows, diffusion models, and GANs to achieve ultra-low-bitrate transmission and enables generative editing across diverse audio domains such as speech, music, and sound effects.
• GAC shifts the traditional rate–distortion trade-off by exchanging higher model capacity for reduced bitrate, with performance validated through objective metrics and subjective listening tests.

Generative Audio Compression (GAC) refers to a paradigm in audio coding that leverages expressive deep generative models to synthesize high-fidelity audio from highly compressed—or semantically factorized—representations. In contrast to traditional codecs, which prioritize signal fidelity under algorithmic transformations and quantization, GAC explicitly utilizes statistical priors on audio, enabling reconstruction of perceptually meaningful signals from extremely compact codes. Recent GAC architectures integrate conditional normalizing flows, diffusion models, GANs, and semantic bottlenecks, enabling ultra-low-bitrate transmission, task-aligned encoding, and generative editing capabilities across general audio domains including speech, music, and sound effects.

1. Theoretical Foundations and Rate–Distortion Tradeoffs

The defining theoretical framework of GAC is the explicit exchange between bitrate and generative model capacity, shifting the classic Shannon rate–distortion paradigm by introducing powerful priors at the decoder. The Law of Information Capacity (IC-1) formalizes this as:

$\eta N = D (H - L)$

where $H$ is source entropy, $L$ is cross-entropy loss (or residual coding rate), $N$ is the number of model parameters, $D$ is the effective data magnitude, and $\eta$ the information capacity per parameter. In practical GAC systems, as model capacity $N$ increases, the required channel bitrate $R$ for a fixed reconstruction quality can be reduced, encapsulating the "More Computation, Less Bandwidth" strategy (Ma et al., 31 Jan 2026). This computation–bandwidth trade-off is fundamental to enabling compression ratios unattainable by conventional codecs.

The compression pipeline is typically decomposed into an encoder extracting semantically or perceptually meaningful latents or tokens (potentially at rates as low as 0.1–7 kbps or frame rates as low as 7.8 Hz), and a decoder comprising a large-scale generative model that reconstructs the full-band audio, leveraging priors trained on vast audio corpora (Ma et al., 31 Jan 2026, Braun et al., 8 Oct 2025, Pia et al., 2024).

2. Architectural Paradigms: Tokenizers, Quantization, and Latent Spaces

GAC systems universally adopt a modular pipeline with (1) a front-end audio encoder, (2) a quantization bottleneck, and (3) a generative decoder.

• Front-End Encoders may utilize time-domain convolutional stacks (e.g., DAC, Improved RVQGAN, QinCodec), frequency-domain representations (e.g., mel-spectrograms, MFCCs in MelCap and MFCC-GAN), semantic encoders (pretrained MAEs, contrastive distillation in SALAD-VAE), or autoregressive content–timbre factorization (semantic codecs) (Li et al., 2 Oct 2025, Hasanabadi, 2023, Braun et al., 8 Oct 2025, Collette et al., 18 Sep 2025).
• Quantization Bottlenecks include:
  • Residual Vector Quantization (RVQ): Cascaded codebooks, quantize encoder outputs in stages (e.g., DAC, Gull, Improved RVQGAN, QinCodec, FlowMAC), supporting scalable bitrates and smooth quality–rate trade-offs (Lahrichi et al., 19 Mar 2025, Pia et al., 2024).
  • Vector Quantization with Implicit Neural Codebooks: Qinco2 and iRVQ decouple offline clustering from autoencoder pretraining, enabling flexible codec design and re-quantization (Lahrichi et al., 19 Mar 2025).
  • Spherical VQ and SRVQ: Exploit L2 normalization and Householder rotations for efficient high-dimensional quantization (Gull) (Luo et al., 2024).
  • Continuous Latents: (e.g., Music2Latent, Music2Latent2, SALAD-VAE) use trainable autoencoder manifolds with either explicit or implicit quantization, benefiting generative modeling and MIR tasks (Pasini et al., 2024, Pasini et al., 29 Jan 2025, Braun et al., 8 Oct 2025).
  • Semantic and Task-Aware Tokens: Learned content, style, and speaker tokens (Vevo, AVCC), or multi-modal latent alignment (AVCC) (Collette et al., 18 Sep 2025, Xu et al., 17 Dec 2025).
• Rate–Distortion Controls: Bitrate is determined by codebook sizes, number of codebooks, token/frame rates, vector dimensions, and, when present, entropy coding schemes (Pia et al., 2024, Li et al., 2 Oct 2025, Xu et al., 17 Dec 2025). Semantic codecs achieve bitrate savings by amortizing infrequent "who" (timbre) information over many utterances (Collette et al., 18 Sep 2025).

3. Generative Decoding: Flows, Diffusion, GANs, and Consistency Models

The core innovation of GAC lies in the generative decoder, capable of synthesizing plausible, perceptually aligned audio from underdetermined or semantically reduced codes.

• Conditional Normalizing Flows / ODE Solvers: FlowMAC and GAC employ ODE-based flow models, parameterized by U-Net/Transformer architectures, integrating conditional vector fields to reverse a learned stochastic interpolation between prior noise and the target waveform. Flow matching objectives train the flow field to match optimal transport between prior and conditional distributions (Pia et al., 2024, Ma et al., 31 Jan 2026).
• Diffusion Models and Consistency Models: Many recent codecs (AVCC, Music2Latent, Music2Latent2) utilize iterative or (in advances such as Music2Latent) single-step denoising of noisy latent variables conditioned on encoder features, attaining both high fidelity and fast inference (Xu et al., 17 Dec 2025, Pasini et al., 2024, Pasini et al., 29 Jan 2025). Consistency learning allows direct map from noise to latent, enabling real-time feasible deployability.
• Adversarial Generation (GAN Decoding): GAN-based decoders enforce distributional matching between generated and reference audio by adversarial training. Multi-resolution discriminators (MPD, MRD) and feature-matching objectives (L1, multi-scale STFT, perceptual embeddings) are ubiquitous, yielding high perceptual quality at low bitrates and restoring lost high-frequency content. Examples include MFCC-GAN, Improved RVQGAN, Penguins, Gull, DAC, and MelCap (Hasanabadi, 2023, Kumar et al., 2023, Liu et al., 2023, Luo et al., 2024, Li et al., 2 Oct 2025).
• Semantic and Multi-modal Generation: AVCC demonstrates cross-modal decoding via a joint audio–video diffusion process, where audio and video latents participate in multi-head cross-attention inside denoising blocks; this enables conditional synthesis and even cross-modal reconstruction at ultra-low rates (Xu et al., 17 Dec 2025).

4. Evaluation Methodologies, Metrics, and Subjective Testing

GAC models are evaluated across objective, perceptual, and application-level benchmarks.

Objective and Perceptual Metrics:

• Fréchet Audio Distance (FAD): Quantifies dissimilarity of distributions in an embedding space, strongly correlates with human judgements; preferred over MMD (Biswas et al., 23 Sep 2025).
• VISQOL, SI-SDR, Mel/Log Spectral Distances: Standard spectral-domain metrics to assess fidelity to the original audio (Kumar et al., 2023, Li et al., 2 Oct 2025, Luo et al., 2024).
• Automatic Speech Recognition, Classification (ASR, MIR tasks): Used for semantic codecs and consistency models to probe intelligibility and semantic retention in the latent codes (Collette et al., 18 Sep 2025, Pasini et al., 29 Jan 2025, Pasini et al., 2024).
• Codebook Perplexity, Token Match Rate, Idempotence: Metrics for quantizer health and repeatability, critical in GAC pipelines employing token-based generation or iterative model chaining (O'Reilly et al., 2024).

Subjective Listening Tests:

• P.808 DCR, MUSHRA: Naïve and expert listening tests, following ITU standards and diverse content (speech, music, mixed); used for direct perceptual A/B against conventional codecs (Opus, USAC, EVS, EnCodec, DAC, and neural baselines) (Pia et al., 2024, Kumar et al., 2023, Li et al., 2 Oct 2025, Luo et al., 2024).
• Downstream Task Benchmarks: WER, speaker verification, emotion and music tagging, CLAP/OpenL3-based perceptual metrics (Collette et al., 18 Sep 2025, Braun et al., 8 Oct 2025, Biswas et al., 23 Sep 2025, Pasini et al., 29 Jan 2025).

Table: Example Bitrates and Perceptual Quality

Model/Codec	Bitrate (kbps)	MUSHRA MOS / FAD	Notable Result
FlowMAC	3	≃DAC/EnCodec 6 kbps	Outperforms USAC 8 kbps at half the bitrate (Pia et al., 2024)
GAC (AI Flow framework)	0.275	MOS 4.1–4.2	≈3000× compression, outperforms semantic/wave codecs (Ma et al., 31 Jan 2026)
Vevo (semantic)	0.65–0.9	NISQA 4.2	Matches/Exceeds Encodec @3 kbps on task/perceptual (Collette et al., 18 Sep 2025)
Improved RVQGAN	8	ViSQOL 4.18	Outperforms Opus/Encodec at same rate (Kumar et al., 2023)
AVCC (low-rate)	0.36	VISQOL 2.9	Joint audio–video, better than neural baselines (Xu et al., 17 Dec 2025)

5. Applications, Limitations, and Future Directions

Applications:

• Ultra-low-bitrate transmission: GAC systems, especially semantic and flow/diffusion-based, can operate at <1 kbps and still support intelligible communication, real-time dialog, and high perceptual quality (Ma et al., 31 Jan 2026, Collette et al., 18 Sep 2025).
• Generative Editing and Model Chaining: Stable, token-based codecs with idempotent encoding enable iterative generation, cross-model handoff, and robust multi-step editing workstreams (O'Reilly et al., 2024).
• Cross-modal/Multi-modal Compression: AVCC demonstrates cross-modal generative pipelines supporting strong lip-sync and mutual reconstruction between audio and video modalities (Xu et al., 17 Dec 2025).
• Downstream Usage: GAC latents are proven to preserve semantic and instrumental structure for MIR, ASR, and classification—enabling joint generative-compression and discriminative tasks (Pasini et al., 2024, Pasini et al., 29 Jan 2025, Braun et al., 8 Oct 2025).

Limitations:

• Inference Cost: Decoder complexity, especially for flow/ODE and diffusion models, can approach billions of parameters and substantial compute—posing challenges for on-device or real-time mobile deployment (Ma et al., 31 Jan 2026, Pia et al., 2024).
• Vocoder Upper-bounds: Final perceptual quality is often bottlenecked by the vocoder (e.g., BigVGAN), rather than upstream coding (Pia et al., 2024).
• Domain Generalization and OOD Robustness: Performance may degrade for out-of-distribution content. Robustness and generalization improvements are ongoing challenges (Pia et al., 2024, O'Reilly et al., 2024).
• Latency and Algorithmic Delay: Some architectures (especially those relying on large-context or ODE/diffusion sampling) may preclude sub-100 ms streaming or synchronous telecom usage (Pia et al., 2024).

Open Problems and Future Work:

• End-to-End Semantic–Acoustic Joint Training: Closer coupling of semantic factorization and acoustic reconstruction, including joint flow–vocoder optimization, hierarchical compression, and adaptive bitrate control (Collette et al., 18 Sep 2025, Pia et al., 2024, Ma et al., 31 Jan 2026).
• Improved Quantization and Compression: Entropy modeling, hybrid continuous/discrete latents, and auto-quantized consistency models to reach <1 kbps with high MOS (Lahrichi et al., 19 Mar 2025, Pasini et al., 2024).
• Real-time Fast Sampling: Faster decoders, single-step consistency models, and model distillation targeting embedded/CPU deployment (Pasini et al., 2024, Pia et al., 2024).
• Multi-modal and Cross-modal Expansion: Audio/video, audio/text, or audio/gesture unified generative codecs for truly joint communication (Xu et al., 17 Dec 2025, Ma et al., 31 Jan 2026).
• Semantic Control and Editing: Token-based GACs with explicit structure (content, style, timbre) enabling semantic editing and generative control over the decoded signal (Collette et al., 18 Sep 2025, Pasini et al., 29 Jan 2025).

6. Comparison of Representative Architectures

Model	Encoder Domain	Quantization	Generative Decoder	Bitrate (kbps)	Core Innovation	Reference
FlowMAC	Mel-Spectrogram	8×RVQ, 256	Conditional Flow (ODE)	1.5–6	CFM-based decoder, real-time	(Pia et al., 2024)
GAC, AI Flow	Tokenizer, VQ	Discrete tokens	Large flow-matching ODE	0.175–0.275	VIB semantic bottleneck	(Ma et al., 31 Jan 2026)
Improved RVQGAN	Waveform	9×RVQ	GAN (multi-scale disc.)	8	Universal, 90× compress	(Kumar et al., 2023)
DAC/DACe	Conv waveform	16–32×RVQ	GAN	8–32	Balanced music/speech tokens	(Biswas et al., 23 Sep 2025)
Music2Latent	STFT	Continuous	Consistency, 1-step UNet	–	Single-step, 64× compression	(Pasini et al., 2024)
Music2Latent2	STFT	Summary emb.	Autoregressive Consistency	–	Unordered chunk-level latents	(Pasini et al., 29 Jan 2025)
Gull	STFT-subband	Spherical RVQ	Elastic RNN/Adversarial	2–24	User-tunable complexity	(Luo et al., 2024)
AVCC	Dual encoder	k-means	Joint AV Diffusion	0.36–1.4	Cross-modal tokenization	(Xu et al., 17 Dec 2025)
MelCap	Mel-spectrogram	2D VQ (1 code)	GAN vocoder	~2–6	One-codebook for all audio	(Li et al., 2 Oct 2025)
MFCC-GAN	Handcrafted MFCC	Scalar Q	GAN	8–36	DSP front, GAN upsampler	(Hasanabadi, 2023)
SALAD-VAE	STFT	VAE (cont.)	Adversarial VAE	~2–16	Semantic contrastive/CLAP	(Braun et al., 8 Oct 2025)
Semantic Codec(Vevo)	Transformer	Token/Embed	Flow-matching, TTS style	0.65–0.9	Content/timbre factorization	(Collette et al., 18 Sep 2025)

7. Idempotence, Stability, and Generative Editing

Idempotent neural codecs, preserving token/code-streams under multiple rounds of encoding/decoding, are critical for stable generative workflows. Adding a latent quantized idempotence loss ensures long-range token stability, improves robustness against phase perturbations, and enables safe iterative generative editing, generative model chaining, and robust transcoding (O'Reilly et al., 2024). High codebook entropy and faithful phase encoding are necessary for this property in GAC pipelines.

---

GAC has thus rapidly advanced to the forefront of audio compression research, coupling deep generative models with domain-aware encoding, attaining bitrates previously exclusive to extreme parametric coders, while delivering perceptually high-quality, semantically meaningful, and generatively flexible audio across domains. Current challenges include efficient real-time deployment, domain adaptation, multi-modal factorization, comprehensive OOD robustness, and integrating joint task-driven objectives in end-to-end training.