Neural Audio Codec

• Neural audio codec is a data-driven compression system that converts audio waveforms into discrete tokens for high-fidelity reconstruction at low bitrates.
• It employs several quantization methods such as RVQ, FSQ, and NDVQ to optimize code utilization and ensure robustness against transmission errors.
• This technology underpins applications including speech and music compression, TTS, audio restoration, and generative audio modeling.

A neural audio codec is a data-driven, end-to-end compression system that encodes audio waveforms into discrete, compact representations optimized via neural networks, enabling high-fidelity reconstruction at low bitrate. The rapid maturation of neural audio codecs since 2022 has shifted benchmarks in audio compression, with such systems now matching or surpassing traditional codecs (e.g., Opus, EVS) at fractions of the classical audio bitrate and providing unique properties for downstream generative modeling, speech processing, and multimodal LLMs.

1. Foundations and Quantization Approaches

Neural audio codecs transform audio $x(t)$ into compressed discrete tokens through an encoder–quantizer–decoder pipeline. The encoder (typically convolutional, residual, or transformer-based) generates continuous latent vectors that are quantized to discrete codes via either residual vector quantization (RVQ) (Défossez et al., 2022, Ai et al., 2024, Wu et al., 2023), finite scalar quantization (FSQ) (Julia et al., 11 Sep 2025), or, more recently, distributional quantizers such as Normal Distribution-based VQ (NDVQ) (Niu et al., 2024). The decoder reconstructs the waveform from the quantized latents.

Quantizer structures:

• Residual Vector Quantization (RVQ): Stacks of codebooks quantizing successive reconstruction errors (Défossez et al., 2022). Each layer projects to discrete indices, composing the bitstream.
• FSQ (Finite Scalar Quantization): Uniform scalar quantization of each latent dimension, yielding code indices that are robust to bit errors (Julia et al., 11 Sep 2025).
• Distributional VQ (e.g., NDVQ): Models each codebook entry as a Gaussian, quantizing by selecting the most probable distribution under the latent, benefiting codebook utilization and robustness at low bitrates (Niu et al., 2024).
• Sparse/Expert Quantization: SwitchCodec employs residual expert VQ, routing tokens through adaptively selected codebooks to expand discrete representational capacity at very low bitrates (Wang et al., 30 May 2025).

The codec design is typically modular: architectures such as SEANet (stacked residual convolutions + LSTM), Wave-U-Net (variance-constrained for deep models), and ConvNeXt v2 (used for STFT-domain models) appear frequently (Niu et al., 2024, Ahn et al., 2024, Ai et al., 2024).

2. Training Objectives and Loss Functions

Neural audio codecs are trained with composite multi-objective losses to ensure perceptual and spectral fidelity:

• Reconstruction loss: Weighted combination of time-domain ($L_1$ or $L_2$) and spectral-domain (multi-scale Mel-spectrogram, STFT) distances (Défossez et al., 2022).
• Quantization losses: Codebook commitment, embedding, and (for NDVQ) distributional losses to guarantee code diversity and codebook utilization (Niu et al., 2024, Ai et al., 2024).
• GAN-based adversarial losses: Multi-period and/or multi-scale STFT discriminators regularize the outputs and enforce perceptual realism (Défossez et al., 2022, Niu et al., 2024).
• Feature-matching: $L_1$ distances between intermediate discriminator activations for stable generator learning (Défossez et al., 2022).
• Semantic distillation: For specialized tokenization (e.g., HybridCodec), semantic stream codes are forced to match frozen self-supervised (SSL) embeddings (Gangwar et al., 4 Jun 2026).

Training may also employ staged paradigms, e.g., first jointly training encoder–quantizer–decoder–discriminator, then freezing encoder/quantizer and retraining the decoder/discriminator for better fidelity (APCodec+ two-stage recipe) (Du et al., 2024).

3. Architectures and Discrete Representation Design

The discrete representation produced by neural codecs underpins both compression efficacy and downstream utility.

• Tokenization: Most codecs output a sequence of discrete indices, with RVQ-based systems producing hierarchical multi-stream tokens amenable to autoregressive or transformer-based sequence modeling (Défossez et al., 2022, Niu et al., 2024).
• Semantic–acoustic disentanglement: Architectural variants such as dual-stream (separate VQ bottlenecks for semantic and acoustic content) or distillation into early VQ layers accentuate linguistic content separation, boosting controllability and compatibility with speech LLMs and text-to-speech (Gangwar et al., 4 Jun 2026).
• Domain/source separation: By designating domain-specific codebooks or conditioning on prompts (e.g., SD-Codec, SUNAC), codecs can yield explicit latent disentanglement across speech, music, and environmental sounds, enabling source-controllable resynthesis and better interpretability (Bie et al., 2024, Aihara et al., 20 Nov 2025).
• Channel-scalable codecs: VCNAC supports mono, stereo, and surround with a single parameterization, adding positional embeddings, cross-channel attention, and compatibility losses for format-agnostic token streams (Grötschla et al., 21 Jan 2026).

Table: Representative Neural Audio Codecs and Key Features

Model	Quantizer	Specialization	Bitrate Range	Notable Features
EnCodec	RVQ	General audio	1.5–24 kbps	Modular, streaming & entropy coding
NDVQ	Distributional	Robust at low bitrate	1.5–24 kbps	Codebook variance margins
HybridCodec	RVQ (dual)	Semantic/acoustic	2–16 kbps	Dual-stream, SSL distillation
VCNAC	RVQ	Multichannel	~8 kbps	Mono, stereo, surround in one model
SD-Codec	RVQ (domain)	Source disentangle	6–18 kbps	Domain-codebook separation
SUNAC	RVQ (prompt)	Source prompt control	6 kbps	Prompt-conditioned extraction
SwitchCodec	Sparse RVQ	Ultra low bitrate	<3 kbps	Residual experts, STFT disc
APCodec+	RVQ (STFT-dom)	Phase-preserving	3–6 kbps	Amplitude/phase joint coding
NeuCodec	FSQ	Robust, LLM-compat	2–8 kbps	Bit-flip robustness, simple tokens

4. Efficiency, Robustness, and Deployment Considerations

Neural codecs balance computational efficiency, robustness, and fidelity:

• Real-time and streaming: Architectures are designed for parallel, causal processing, with practical implementations achieving $<15$ ms latency (e.g., HILCodec RTF=1.4, AudioDec GPU decode $<$6 ms) (Ahn et al., 2024, Wu et al., 2023).
• Model size and complexity: High-fidelity lightweight models such as HILCodec and LDCodec demonstrate state-of-the-art scores with $<10$ M parameters and $<0.3$ GMACs for decoding, suitable for on-device or mobile deployment (Ahn et al., 2024, Jiang et al., 17 Oct 2025).
• Bitrate control and scalability: RVQ dropout, codebook selection, or per-band allocation (PQMF) are used to tune bitrate dynamically across modalities (music, speech, multi-channel audio) (Niu et al., 2024, He et al., 2 Mar 2026, Grötschla et al., 21 Jan 2026).
• Robustness: Distributional quantization and scalar quantization (FSQ) afford increased resilience to bit-flip or transmission error, as neighboring code indices result in small latent perturbations, in contrast to conventional RVQ where index errors may cause catastrophic jumps (Niu et al., 2024, Julia et al., 11 Sep 2025).
• Hybrid classical/neural: Some codecs (e.g., Penguins) combine neural coding for low frequencies with classical MDCT/MDCT-based bandwidth extension for high bands, leveraging strengths of both paradigms at ultra-low bitrate (Liu et al., 2023).

5. Applications and Downstream Tasks

Neural audio codecs serve both as advanced compression tools and as backbone representations for generative and analysis systems:

• Speech and general audio compression: EnCodec, HILCodec, and related models outperform Opus and EVS at 3–9 kbps for both speech and music (Défossez et al., 2022, Ahn et al., 2024).
• TTS and audio LLM tokenization: Discrete codec tokens are used for autoregressive generation in TTS (e.g., VALL-E, VALL-E X) and as input for multimodal LLMs, with specialized token streams (semantic/acoustic separation) enabling controllable synthesis (Gangwar et al., 4 Jun 2026).
• Source separation and controllable resynthesis: SD-Codec and SUNAC embed source/domain information into the token space, making it possible to resynthesize, mute, or edit individual sources (speech, music, SFX) post-compression (Bie et al., 2024, Aihara et al., 20 Nov 2025).
• Audio denoising and restoration: Latent-space U-Nets and GAN-based post-filtering, as in ADNAC, leverage pre-trained codec latents for high-fidelity audio denoising, outperforming classical U-Net approaches in subjective tests (Jimon et al., 3 Nov 2025).

6. Evaluation Metrics and Empirical Results

Assessment of neural audio codecs leverages both objective and subjective metrics:

• Objective: ViSQOL, PESQ, SI-SDR, Mel/STFT distance, Log-Spectral Distance (LSD), MCD, and STOI. Notably, NDVQ achieves PESQ = 3.17 and SI-SDR = 7.25 dB at 6 kbps, exceeding EnCodec (Niu et al., 2024). VCNAC attains PESQ = 4.16 and SI-SDR = 11.3 dB on speech at 7.9 kbps, outperforming EnCodec and DAC (Grötschla et al., 21 Jan 2026).
• Subjective: MUSHRA and MOS, with HILCodec scoring $\sim85$ at 9 kbps and LDCodec surpassing Opus at half the bitrate (Ahn et al., 2024, Jiang et al., 17 Oct 2025).
• Specialized tasks: Semantic accuracy (word error rate on tokenized streams), zero-shot synthesis Quality/Error Rate, and cross-lingual generalization are reported for HybridCodec and others (Gangwar et al., 4 Jun 2026).

7. Future Directions

Key open research avenues include:

• Adaptive bitrate and content-aware allocation: Dynamic allocation between semantic and acoustic streams, or between subbands for perceptual optimization (Gangwar et al., 4 Jun 2026, He et al., 2 Mar 2026).
• Interpretable and disentangled representations: Further advances in semantic/acoustic/pitch disentanglement, interpretable token design, and integration with audio understanding or editing pipelines (Sadok et al., 4 Jun 2025, Bie et al., 2024).
• Source-aware and prompt-driven coding: Prompt-conditioned extraction and manipulation (SUNAC, HybridCodec), scaling to more granular source types and greater source counts (Aihara et al., 20 Nov 2025, Bie et al., 2024).
• Scalability to high-rate, multichannel formats: Efficient support for surround, multi-track, or immersive audio within a single model (Grötschla et al., 21 Jan 2026, Li et al., 7 Aug 2025).
• Decoding and deployment at extreme resource constraints: Further reducing model and compute complexity for ubiquitous mobile and embedded deployment (Ahn et al., 2024, Jiang et al., 17 Oct 2025).
• Downstream integration: Training codecs as front-ends for end-to-end LLMs, content-based retrieval, and controllable synthesis conditioned on high-level semantic attributes.

Neural audio codecs have thereby established themselves as not only a disruptive audio compression technology but also a crucial representation substrate for modern neural audio processing pipelines and generative systems.