NeuCodec: Robust Neural Audio Coding via FSQ

• NeuCodec is a neural audio codec architecture that uses Finite Scalar Quantization to discretize latent dimensions independently, yielding high entropy and inherent redundancy.
• It demonstrates encoder distillation and interoperability by achieving comparable perceptual quality from diverse encoder designs even with only ~2% index matching.
• NeuCodec maintains robust transmission under noisy conditions, gracefully handling up to 10% bit-flip errors while supporting integration with Large Language Models for speech tasks.

NeuCodec defines a neural audio codec architecture centered on Finite Scalar Quantization (FSQ), diverging from the predominating Residual Vector Quantization (RVQ) approaches in neural audio coding. FSQ discretizes each latent dimension output by the encoder independently, providing a high-entropy, locally smooth, and redundant representation. These features impart unique robustness and transmission reliability at very low bitrates, while simplifying codebook management and reducing code collapse. Experimental results demonstrate NeuCodec’s superiority over RVQ-based codecs in the context of noisy channel transmission, encoder interoperability, and rate-distortion trade-offs, while supporting seamless integration with LLMs for downstream tasks in speech processing and generation (Julia et al., 11 Sep 2025).

1. Architectural Principles: Finite Scalar Quantization in NeuCodec

NeuCodec leverages FSQ to discretize the encoder’s output, wherein each dimension is quantized independently into $n_i$ equidistant levels in a bounded projection space (typically $[-1, 1]$). This process yields a codebook of size

$C = \prod_{i=1}^d n_i$

where $d$ is the latent dimension. Each quantized dimension functions as an implicit codebook; the encoder output $z \in \mathbb{R}^d$ is mapped to a vector of discrete indices, each corresponding to one quantization level per dimension. Unlike RVQ—which stacks multiple codebooks and requires auxiliary losses to enforce codebook utilization—FSQ ensures near-complete codebook usage without extra regularization.

Feature	RVQ	FSQ (NeuCodec)
Codebook Structure	Hierarchical, stacked	Single, per-dimension
Auxiliary Losses	Required for usage	Not required
Encoding Redundancy	Minimal	Baked-in, high

This simplification of codebook and quantization dynamics leads to more robust training and predictable behavior under perturbations.

2. Encoder Distillation and Interoperability

An explicit experiment in the paper demonstrates that two distinct encoder architectures (original full-capacity and distilled compact) can produce vastly different code sequences for the same input audio. When paired with the same FSQ quantizer and decoder, both systems reconstruct the audio at comparable perceptual quality. Only $\approx2\%$ of the quantization indices matched between encoder outputs, yet the cosine similarity between the quantizer outputs averaged $\approx 0.73$ and $93\%$ of levels were identical or off by only one scalar level. This inherent redundancy means that divergent code sequences still encode overlapping latent regions.

A plausible implication is that NeuCodec supports flexible and modular encoder designs for domain adaptation, without the risk of losing perceptual quality due to encoder mismatch.

3. Robustness to Channel Noise and Bit Perturbations

NeuCodec’s FSQ coding is transmission-robust. Each bit in a scalar-quantized dimension corresponds to a localized change in the latent and decoded audio space. When simulating transmission errors—specifically, bit-flips in the discrete code—the reconstructed audio quality degrades gracefully even with up to $10\%$ of bits perturbed. The objective quality, measured via STOI, PESQ, and SI-SDR, decays slowly rather than precipitously. In contrast, RVQ-based systems suffer sharp drops in quality with $>1\%$ bit-flip probability.

Metric	FSQ (NeuCodec): Bit-flip $<10\%$	RVQ: Bit-flip $>1\%$
STOI	High	Severe drop
PESQ	Maintained	Severe drop
SI-SDR	Degrades slowly	Rapid decay

This suggests NeuCodec is suitable for noisy, low-resource communication environments, such as wireless transmission with unreliable links.

4. Redundancy, Local Smoothness, and Downstream Application

The "baked-in" redundancy of FSQ (Editor’s term) arises because each dimension is quantized independently. Perturbations in any single dimension cause only localized changes in the reconstructed audio, and the overall representation is distributed across the entire code sequence.

Such redundancy means the discrete codes can be treated as domain-specific tokens, compatible with LLM modeling for automatic speech recognition (ASR), text-to-speech (TTS), and full-duplex speech modeling. NeuCodec’s code sequence structure enables seamless integration into audio generation frameworks reliant on LLMs, since each code has consistent local semantics and entropy (Julia et al., 11 Sep 2025).

5. Rate-Distortion Performance and Practical Deployment

NeuCodec achieves competitive rate-distortion performance at very low bitrates, while supporting robust speech reconstruction across encoder variants and transmission conditions. The streamlined FSQ quantization not only simplifies training and deployment but enhances deployment flexibility. The model is well-suited to tasks requiring robustness against packet loss, such as telephony, streaming, and on-device speech synthesis. Its transmission stability and interoperability with LLMs position NeuCodec as a promising solution for next-generation neural audio coding applications.

6. Comparison to RVQ Systems and Design Implications

Compared to RVQ-based neural codecs:

• FSQ-based NeuCodec exhibits stable training and codebook utilization without auxiliary correction losses.
• Error propagation is minimized due to local smoothness in the quantized latent space.
• Encoder architecture can be flexibly swapped or distilled so long as the quantizer and decoder remain fixed.
• Transmission errors induce only localized, minor artifacts rather than catastrophic failure.

A plausible implication is that future neural codec designs may increasingly favor scalar quantization architectures similar to NeuCodec for scenarios demanding rate-distortion resilience, training simplicity, and compatibility with symbolic downstream models.

7. Summary and Outlook

NeuCodec is defined by its adoption of Finite Scalar Quantization for neural audio coding, yielding high code entropy, redundancy, and transmission robustness. Empirical findings show NeuCodec is resilient to channel noise, tolerant to encoder architecture changes, and conducive to integration with LLMs for generative and speech processing tasks. Its streamlined design and error-tolerant code sequences distinguish it as an effective neural codec paradigm, likely to influence future developments in robust audio compression and synthesis systems (Julia et al., 11 Sep 2025).