Neural Audio Codec Architecture

• Neural audio codec architecture is a learned, end-to-end system that compresses and reconstructs audio using specialized encoder, quantizer, and decoder modules.
• It employs advanced quantization techniques like residual vector quantization and multi-stage codebook design to balance compression efficiency and audio fidelity.
• Practical applications include real-time communication, generative audio modeling, and spatial audio processing, offering low latency and high perceptual quality.

A neural audio codec architecture is a learned, end-to-end system that efficiently compresses and reconstructs audio via neural networks, replacing or augmenting classical signal processing with deep learning techniques. Modern neural codecs ingest waveform or spectral input, produce compact representations (typically discrete token streams via quantization), and reconstruct perceptually high-quality audio while operating at low bitrates and often with real-time or streaming constraints. Architecturally, these systems typically integrate specialized encoders, quantizers (often multi-stage and residual), and decoders, with training objectives that combine signal fidelity, perceptual, and adversarial criteria.

1. Architectural Components and Encoding Pipeline

Neural audio codecs are primarily structured as encoder–quantizer–decoder pipelines, where:

• Encoder: Transforms the input audio (either time-domain waveform or spectral representation) into a lower-dimensional latent embedding. For waveform-based systems, this is typically realized using stacked convolutional blocks with residual units, dilated convolutions, and downsampling operations (as in SoundStream (Zeghidour et al., 2021)). For spectral codecs (APCodec (Ai et al., 16 Feb 2024), STFTCodec (Feng et al., 21 Mar 2025)), audio is converted into time–frequency features (e.g., STFT magnitude and phase) and processed via deep feature extraction (frequently involving 1D or 2D ConvNeXt v2 or similar blocks).
• Quantizer: The latent representation is discretized, commonly by residual vector quantization (RVQ). RVQ cascades multiple independent or partially-shared codebooks, where each stage quantizes the residual error from previous stages, yielding an additive sequence of quantized vectors that enables progressive refinement without the combinatorial explosion of a single massive codebook. Some codecs implement group-wise or beam-search variants to reduce quantization noise and improve codebook utilization (CBRC (Xu et al., 2 Feb 2024)). Recent codecs introduce implicit neural codebooks (QINCODEC (Lahrichi et al., 19 Mar 2025)) or even employ scalar–vector hybrid quantization for efficient codebook exploitation and low-latency causal operation (StreamCodec (Jiang et al., 9 Apr 2025)).
• Decoder: Mirrors the encoder, reconstructing the audio from the quantized latent. Decoders may use transposed convolutions, upsampling blocks, and dedicated neural architectures for waveform or spectral domain restoration. Some systems (APCodec (Ai et al., 16 Feb 2024), APCodec+ (Du et al., 30 Oct 2024), STFTCodec (Feng et al., 21 Mar 2025)) implement parallel reconstruction of amplitude and phase, while others leverage Mel-spectrogram intermediate supervision (HH-Codec (Xue et al., 25 Jul 2025)).
• Discriminator: For perceptual quality, adversarial discriminators are frequently incorporated, ranging from multi-period waveform to multi-scale STFT-based designs.

2. Quantization and Bitrate Control Strategies

The compression efficiency and bitrate flexibility of neural codecs are primarily governed by quantization design:

• Residual Vector Quantization (RVQ): Most state-of-the-art codecs leverage multi-stage RVQ, where each quantizer in the cascade captures increasingly fine detail. This architecture allows straightforward adjustment of bitrate by varying the number of quantization layers engaged at inference—a paradigm known as quantizer dropout (SoundStream (Zeghidour et al., 2021)), supporting a range from 3 kbps to 18 kbps without architectural change.
• Single Quantizer Approaches: Some designs (SQCodec (Zhai et al., 7 Apr 2025), HH-Codec (Xue et al., 25 Jul 2025)) demonstrate that a single, large codebook quantizer can achieve quality comparable to multi-quantizer approaches, reducing computational overhead and inference complexity.
• Group-wise, Beam Search, Scalar–Vector Hybrid: CBRC (Xu et al., 2 Feb 2024) splits the embedding into groups for independent quantization and combines this with beam-search to optimize codeword selection. StreamCodec (Jiang et al., 9 Apr 2025) sequentially applies scalar and vector quantizers in a residual fashion (RSVQ), coarsely quantizing structure before refining detail.
• Frame Rate and Temporal Adaptivity: Codecs such as FlexiCodec (Li et al., 1 Oct 2025) operate at extremely low and dynamically adaptive frame rates (down to 3 Hz). Merging of semantically similar frames leverages ASR-derived features, enabling aggressive token reduction in information-sparse regions without degrading semantic fidelity.
• Codebook Disentanglement: SD-Codec (Bie et al., 17 Sep 2024) partitions the latent space into orthogonal domains, each with dedicated codebooks for speech, music, or environmental effects, supporting source separation and improved controllability.

3. Training Methodologies and Losses

Neural codec training uses losses engineered to balance objective fidelity and perceptual quality:

• Reconstruction Losses: Multi-scale spectral losses compare mel-spectrograms at diverse window sizes and compute $L_1$ or $L_2$ distances. Waveform $L_1$ or $L_2$ losses may be included.
• Perceptual and Feature Losses: Intermediate layer activations from discriminators or auxiliary networks serve as feature match spaces to promote perceptual realism (SoundStream (Zeghidour et al., 2021), EnCodec (Défossez et al., 2022)).
• Adversarial Losses: Multiple discriminators (e.g., waveform-domain, STFT-domain, multi-period) are optimized to enforce audio realism. Losses are often hinge-type or least-squares (as in APCodec (Ai et al., 16 Feb 2024) and EnCodec (Défossez et al., 2022)).
• Quantizer Commitment and Balance Losses: These penalize large deviations between latent representations and quantized codebook entries, or enforce codebook usage entropy constraints to prevent dead codewords (StreamCodec (Jiang et al., 9 Apr 2025)).
• Spectral Phase Losses: Phase-aware and anti-wrapping losses (AW-IP, AW-GD, AW-IAF) ensure not just magnitude but also temporal structure is faithfully captured (APCodec (Ai et al., 16 Feb 2024), APCodec+ (Du et al., 30 Oct 2024), STFTCodec (Feng et al., 21 Mar 2025)).
• Staged Training Paradigms: APCodec+ (Du et al., 30 Oct 2024) introduces two-phase training: first joint optimization of all modules, followed by individual decoder/discriminator fine-tuning on fixed quantized features, which improves convergence and final audio fidelity at lower bitrates.

4. Architectural and Quantization Innovations

Significant architectural advances in neural audio codecs include:

• Spectral Domain Processing: APCodec (Ai et al., 16 Feb 2024), STFTCodec (Feng et al., 21 Mar 2025), SpectroStream (Li et al., 7 Aug 2025) process and quantize amplitude and phase spectra directly, with parallel branches and indirect phase reconstruction (e.g., predicting real and imaginary parts, then arctan2).
• Multi-Scale Quantization: SNAC (Siuzdak et al., 18 Oct 2024) applies RVQ hierarchically at variable temporal resolutions, each quantizer operating at different frame rates to capture long-term structure and local fine detail. This hierarchical, multi-scale design yields higher quality at reduced bitrates.
• Delayed Fusion for Multi-Channel Audio: SpectroStream (Li et al., 7 Aug 2025) processes multi-channel audio by initially separating and then gradually fusing channels in the encoder. This addresses the trade-off between per-channel fidelity and cross-channel phase alignment, crucial in high-fidelity stereo music coding.
• Dual-Stream and Feature-Assisted Architectures: FlexiCodec (Li et al., 1 Oct 2025) uses ASR-based semantic token streams alongside residual acoustic streams, with Transformer modules for merging/unmerging, to preserve semantics at low token rates.
• Lightweight/Real-Time Streaming: Extensive use of causal convolutions (SoundStream (Zeghidour et al., 2021), StreamCodec (Jiang et al., 9 Apr 2025)) or transformer-based streaming setups (TS3-Codec (Wu et al., 27 Nov 2024)) ensures short end-to-end latency (down to 13–20 ms), efficiency on constrained hardware, and direct support for real-time applications.

5. Empirical Performance and Evaluation

Neural audio codecs are benchmarked with both objective and subjective metrics:

• Objective Metrics: ViSQOL, PESQ, STOI, SI-SDR, Log-Spectral Distance, Mel Cepstral Distortion, UTMOS, and codebook utilization rates.
• Subjective Metrics: MUSHRA-style listening tests and crowd-sourced evaluations frequently demonstrate that neural codecs operating at 3–6 kbps can surpass the quality of legacy codecs (Opus, EVS) running at much higher bitrates (e.g., SoundStream (Zeghidour et al., 2021), CBRC (Xu et al., 2 Feb 2024)).
• Compression–Quality and Latency Tradeoffs: Modern codecs demonstrate real-time operation on CPUs at low GHz and parameter footprints (e.g., StreamCodec (Jiang et al., 9 Apr 2025): 7M parameters, ViSQOL 4.30 at 1.5 kbps, 20× real-time CPU generation).

Codec/Architecture	Bitrate Range (kbps)	Key Metric(s)	Subjective Outperf.
SoundStream (Zeghidour et al., 2021)	3–18	ViSQOL, MUSHRA	Opus (3 kbps vs 12 kbps)
EnCodec (Défossez et al., 2022)	1.5–12	SI-SNR, ViSQOL, MUSHRA	Lyra-v2, Opus, EVS
CBRC (Xu et al., 2 Feb 2024)	3–6	ViSQOL, PESQ, codebook utilization	Opus (3 vs 12 kbps)
HH-Codec (Xue et al., 25 Jul 2025)	~0.3	UTMOS, STOI, SIM	Multi-quantizer baselines
FlexiCodec (Li et al., 1 Oct 2025)	3–12.5 Hz frame rate	WER, PESQ, MCD, SIM, UTMOS	DualCodec, LFR baselines

6. Practical Applications and Downstream Utility

Neural audio codecs are now integral to:

• Real-time Communication: Their streaming capabilities and low latency make them suitable as drop-in replacements for VoIP and teleconferencing codecs, delivering higher quality at lower bandwidth.
• Generative and LLMing: Tokenized representations enable audio-LLMs (e.g., speech-to-text, speech-to-speech translation) to operate efficiently with short discrete sequences (FlexiCodec (Li et al., 1 Oct 2025), HH-Codec (Xue et al., 25 Jul 2025)).
• Multichannel and Spatial Audio: BANC (Ratnarajah et al., 2023) and SpectroStream (Li et al., 7 Aug 2025) bring architectural advances for spatial audio and binaural encoding, crucial in immersive media contexts (VR/AR).
• Source Separation and Editing: SD-Codec (Bie et al., 17 Sep 2024) and CodecSep (Banerjee et al., 15 Sep 2025) extend codec architectures to learn disentangled, source-specific representations, enabling advanced editing and separation tasks.
• Extreme Compression: HH-Codec (Xue et al., 25 Jul 2025) optimizes for ultra-low bitrates and minimal token rates tailored for spoken LLMing, enabling light-weight audio representations for LLMs with minimal information loss.

7. Current Trends and Research Directions

Recent advances orient towards:

• Spectrum-based Coding: More codecs now operate in the spectral domain, employing amplitude/phase decoupling (APCodec+ (Du et al., 30 Oct 2024), STFTCodec (Feng et al., 21 Mar 2025)) and flexible time-frequency parameterization for bitrate/resolution adaptation.
• Single-Quantizer Architectures: Demonstrated by SQCodec (Zhai et al., 7 Apr 2025) and HH-Codec (Xue et al., 25 Jul 2025), single quantizer designs are challenging the necessity of multi-quantizer cascades for extreme compression and better integration with downstream models.
• Staged and Progressive Training: Training strategies that decouple encoder–quantizer learning from decoder/adversarial optimization (APCodec+ (Du et al., 30 Oct 2024)), or that involve progressive supervision (HH-Codec (Xue et al., 25 Jul 2025)), yield improvements in convergence and ultimate perceptual quality, especially at the lowest bitrates.
• Flexible and Dynamic Rate Coding: FlexiCodec (Li et al., 1 Oct 2025) and SNAC (Siuzdak et al., 18 Oct 2024) illustrate the advantages of dynamic token merging, frame rate control, and multi-scale quantization, permitting adaptive allocation of coding bandwidth as content demands.
• Open Source and Benchmarks: There is a trend toward public release of code and models (EnCodec (Défossez et al., 2022), SNAC (Siuzdak et al., 18 Oct 2024), HH-Codec (Xue et al., 25 Jul 2025), SQCodec (Zhai et al., 7 Apr 2025)), supporting reproducibility and wider application.

In summary, neural audio codec architecture has evolved rapidly from basic end-to-end autoencoders to sophisticated, domain-adaptive, multi-scale, and real-time systems that are now competitive or superior to classical codecs over multiple tasks, sampling rates, and application domains. The field remains active, with research pushing toward even lower bitrates, more robust and disentangled representations, and seamless integration with large-scale audio–LLMing pipelines.