An Ultra-Low-Bitrate Neural Speech Codec with Plain-to-Pseudo Synergistic Vector Quantization

Abstract: Most neural speech codecs use residual vector quantization (RVQ), in which later VQs contribute less but consume the same bitrate, leading to inefficiency. We propose P2PSynCodec, an ultra-low-bitrate neural speech codec with a plain-to-pseudo synergistic vector quantizer (P2PSVQ). P2PSVQ consists of one plain VQ and multiple pseudo VQs. The plain VQ produces basic tokens by quantization, while the pseudo VQs generate auxiliary tokens by neural prediction and incur zero transmitted bitrate. Thus, speech is decoded from the plain-VQ tokens together with predicted pseudo-VQ tokens, greatly reducing bitrate. Experiments show that P2PSynCodec achieves speech reconstruction quality comparable to competing codecs at 2.0 kbps while operating at only 0.5 kbps, demonstrating high efficiency for ultra-low-bitrate speech coding.

• The paper introduces a plain-to-pseudo synergistic VQ that decouples bitrate and expressiveness, enabling a 75% bitrate reduction without compromising quality.
• It employs lightweight convolutional networks and MDCT representations to achieve state-of-the-art speech compression at 0.5 kbps versus 2.0 kbps in traditional models.
• Ablation studies confirm that using three pseudo VQs optimizes the balance between decoding quality and effective token transmission.

An Ultra-Low-Bitrate Neural Speech Codec with Plain-to-Pseudo Synergistic Vector Quantization

Introduction and Motivation

Ultra-low-bitrate speech coding is central to applications subject to stringent bandwidth constraints, such as satellite communication, on-device voice storage, and IoT voice interfaces. While neural speech codecs have surpassed traditional methods in rate-distortion performance, they exhibit severe degradation below 2 kbps, primarily due to inefficient information allocation in the quantization stages. Residual vector quantization (RVQ), a backbone of current neural codecs, yields diminishing returns as more quantizers are added, with each consuming equal bitrate irrespective of their incremental contribution to quality. Other approaches, such as aggressive scalar quantization (FSQ), sacrifice expressiveness for low bitrate, while simply scaling up model capacity, as in BigCodec and WavTokenizer, incurs major compute and deployment penalties.

The proposed P2PSynCodec with its plain-to-pseudo synergistic vector quantizer (P2PSVQ) directly addresses these limitations by fundamentally reformulating how neural quantization allocates bitrate and expressiveness in speech compression.

Architectural Overview

P2PSynCodec is built upon a highly efficient architecture that processes MDCT representations of speech using lightweight convolutional encoder and decoder networks. The quantization stage, which is the architectural core, is realized as a cascade where a single plain VQ produces basic tokens for transmission, and several pseudo VQs predict auxiliary tokens at zero additional bitrate by leveraging neural token predictors at the decoder.

Figure 1: Overview of the P2PSynCodec architecture and the pseudo-VQ training process, highlighting the single plain VQ and three pseudo VQs.

The encoder transforms waveform input to compact MDCT spectra using a ConvNeXt v2-based architecture, followed by quantization. Crucially, only the plain VQ's output is entropy coded and transmitted, drastically minimizing bitrate to 0.5 kbps (16 kHz). The pseudo VQ stack, parameterized by neural networks combining Conformer blocks and BiLSTMs, is applied only at the decoder side. During training, the pseudo VQs are guided by targets from an RVQ-based teacher model using teacher forcing and cross-entropy objectives, forming a knowledge distillation regime that enables accurate token prediction from heavily lossy plain VQ codes.

Synergistic Plain-to-Pseudo Vector Quantization

The innovation within P2PSVQ is the decoupling of bitrate and expressive power of the quantization mechanism. Whereas conventional RVQ distributes the information load uniformly (with later stages contributing less but costing equally in bitrate), P2PSVQ concentrates transmitted information in a single plain VQ and leverages pseudo VQs for conditional expressiveness. The auxiliary tokens predicted by pseudo VQs are dependent on the higher-entropy basic tokens and are learned to mimic the behavior of full RVQ tokens as in the teacher network.

This results in an additive quantization scheme,

$$\hat{\mathbf{e}} = \mathbb{L}(\mathbb{W}_{pl}, d_{pl}) + \sum_{n=1}^N \mathbb{L}(\mathbb{W}_{ps}^{(n)}, \hat{d}_{ps}^{(n)}),$$

where only $d_{pl}$ is transmitted, and the $\hat{d}_{ps}^{(n)}$ are noiselessly generated at the receiver.

Empirical Results

Objective and Subjective Quality

P2PSynCodec was evaluated on LibriTTS (16 kHz) and VCTK (48 kHz), being configured to three pseudo VQs and a single plain VQ with shared codebook sizes. The bitrate is 0.5~kbps and 1.5~kbps respectively, drastically less than prior art.

Subjective listening tests (MUSHRA) on both datasets show that P2PSynCodec matches or exceeds state-of-the-art neural codecs at the same bitrate and achieves quality comparable to competing codecs at four times higher bitrate (2.0 kbps).

Figure 2: Subjective MUSHRA results at 16 and 48 kHz, including the hidden reference and anchor, with error bars reflecting 95\% confidence intervals.

Objective metrics (UTMOS, SIGMOS, ViSQOL, STOI) confirm that P2PSynCodec achieves a notable margin over RVQ-based codecs (e.g., MDCTCodec and DAC) at equal ultra-low bitrates, despite a lower model complexity than BigCodec and WavTokenizer, evidenced by significantly reduced FLOPs and parameter counts.

In paired ABX preference tests, listeners show no significant preference between P2PSynCodec at 0.5 kbps and competing codecs at 2.0 kbps, as indicated by non-significant $p$-values.

Figure 3: Average preference scores (\%) in ABX tests of P2PSynCodec at 0.5 kbps versus other codecs at high bitrates on LibriTTS (16 kHz).

Impact of Pseudo VQ Number

Ablation studies examining the number of pseudo VQs reveal a tradeoff: increasing $N$ first improves decoding quality, but excessive pseudo VQs lead to degraded performance due to attenuation of information in the transmitted plain token, which hinders accurate auxiliary token prediction. An intermediate setting ($N=3$) achieves the optimal complexity-quality balance.

Implications and Future Work

P2PSynCodec demonstrates that the synergy of quantized transmission and neural token prediction enables a paradigm shift in low-bitrate speech coding. By decoupling bit allocation from decoder-side expressiveness, it is possible to deliver a 75% bitrate reduction for equivalent subjective quality. This approach opens the avenue for neural codecs to target previously unattainable deployment regimes: embedded device storage, distributed voice control, and limited-bandwidth streaming.

Theoretically, the architecture provides a template for broader quantization frameworks employing neural synthesis of codebook tokens, which could extend to other signal domains (audio, vision) and foster advances in neural compression, tokenization for generative models, and joint semantic-signal coding.

Possible next directions include adaptation to causal inference for real-time communication, scaling the token prediction regime to integrate semantic compression, and the extension to multilingual or zero-resource scenarios, where predictive token synthesis may compensate for language or speaker mismatch.

Conclusion

The P2PSynCodec effectively bridges the ultra-low-bitrate speech coding quality gap by introducing a plain-to-pseudo synergistic vector quantization scheme, leveraging neural token prediction to enable full exploitation of model capacity at a fraction of the bitrate. The system's performance at 0.5 kbps is empirically on par with state-of-the-art codecs at up to 2.0 kbps in both subjective and objective metrics, at moderate compute cost. The framework supports practical low-bitrate deployment and lays the groundwork for future research in neural quantization and generative compression (2606.05876).