P2PSynCodec: Ultra-Low-Bitrate Neural Speech Codec

• The paper introduces a novel plain-to-pseudo synergistic vector quantization method that shifts residual coding to a neural prediction process, enabling ultra-low-bitrate speech reconstruction.
• It employs a fully convolutional encoder–decoder operating on MDCT spectra and integrates one transmitted plain VQ with multiple zero-bitrate pseudo VQs for efficient coding.
• Experiments show that at 0.5 kbps, P2PSynCodec matches or exceeds the quality of codecs at higher bitrates while significantly reducing computational complexity and model size.

Searching arXiv for the cited P2PSynCodec paper and closely related codec papers to ground the article in current literature. P2PSynCodec is an ultra-low-bitrate neural speech codec introduced in "An Ultra-Low-Bitrate Neural Speech Codec with Plain-to-Pseudo Synergistic Vector Quantization" (Jiang et al., 4 Jun 2026). It is defined by a quantization scheme called plain-to-pseudo synergistic vector quantization (P2PSVQ), in which one plain vector quantizer produces the only transmitted tokens and multiple pseudo vector quantizers generate auxiliary tokens by neural prediction at zero transmitted bitrate. The codec operates on MDCT spectra rather than waveforms, uses a fully convolutional encoder–decoder with a modified ConvNeXt v2 backbone, and is designed to address a stated inefficiency of residual vector quantization (RVQ): later VQs contribute less information yet still consume the same bitrate. Experiments reported in the paper state that P2PSynCodec achieves speech reconstruction quality comparable to competing codecs at 2.0 kbps while operating at only 0.5 kbps (Jiang et al., 4 Jun 2026).

1. Conceptual basis and problem setting

P2PSynCodec is motivated by a limitation attributed to standard RVQ-based neural speech codecs. In RVQ, the encoder output is quantized by a sequence of codebooks, where the first quantizer captures the largest part of the signal and later quantizers encode residuals. The paper states that this design is inefficient because later VQs contribute less information yet still consume the same bitrate (Jiang et al., 4 Jun 2026). At ultra-low bitrates such as 0.5 kbps, this inefficiency becomes especially consequential, because reducing the number of transmitted quantizers tends to degrade reconstruction quality unless the model is made substantially larger or more complex.

The central design move in P2PSynCodec is therefore not a larger backbone or a more elaborate waveform generator, but a change in the quantization strategy itself. The proposed P2PSVQ uses one transmitted plain VQ and several zero-bitrate pseudo VQs. The plain VQ produces a basic token by direct quantization, while the pseudo VQs generate auxiliary tokens by neural prediction. Speech is then decoded from the plain-VQ tokens together with predicted pseudo-VQ tokens. This arrangement preserves part of the representational effect of a multi-stage quantizer while making the bitrate depend only on the plain VQ (Jiang et al., 4 Jun 2026).

This suggests that P2PSynCodec should be understood less as a variant of ordinary RVQ than as a hybrid between transmitted quantization and decoder-side latent reconstruction. A plausible implication is that its main contribution lies in relocating part of the latent-description burden from the communication channel to a learned prediction process at the receiver.

2. Architecture and signal path

The codec has three main parts: an encoder, P2PSVQ, and a decoder (Jiang et al., 4 Jun 2026). It operates on MDCT spectra rather than waveforms, and this design is described as being inspired by MDCTCodec. The encoder is fully convolutional and compresses the spectra using a modified ConvNeXt v2 backbone. Each residual block contains a 1D depthwise convolution, LayerNorm, a linear layer, GRN (Global Response Normalization), and GELU activation. Two extra 1D convolution layers are used at input and output to adjust dimensionality, and a 1D downsampling layer compresses the time axis (Jiang et al., 4 Jun 2026).

The decoder mirrors the encoder. It uses upsampling instead of downsampling, reconstructs MDCT spectra, and then applies IMDCT to recover the waveform. The symmetry between encoder and decoder places the main novelty of the system not in an asymmetric generative back end, but in the quantization module positioned between them (Jiang et al., 4 Jun 2026).

The following table summarizes the pipeline exactly as described in the source:

Component	Role	Reported details
Encoder	Compress MDCT spectra	Modified ConvNeXt v2; residual blocks with 1D depthwise convolution, LayerNorm, linear layer, GRN, GELU; extra 1D convolutions; 1D downsampling
P2PSVQ	Quantize latent representation	1 plain VQ and $N$ pseudo VQs
Decoder	Reconstruct speech	Mirrors encoder; uses upsampling; reconstructs MDCT spectra; IMDCT converts spectra back to waveform

Within this architecture, the transmitted representation is intentionally minimal. The decoder reconstructs a richer latent not because more bits are sent, but because pseudo tokens are generated locally and their codebook vectors are combined with the plain-VQ codeword before decoding (Jiang et al., 4 Jun 2026).

3. Plain-to-pseudo synergistic vector quantization

The defining component of P2PSynCodec is P2PSVQ. Let the encoder output be a frame-level vector

$\mathbf{e}\in\mathbb{R}^K .$

The plain VQ codebook is

$$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$

and the plain token is obtained by nearest-neighbor lookup: $$d_{pl} = \arg\min_m \|\mathbf{e} - \mathbf{w}_m\|_2 .$$ The paper characterizes this token as the basic token because it is directly quantized from the encoder output, is the most information-rich token, and serves as the foundation for predicting pseudo tokens (Jiang et al., 4 Jun 2026).

Only the plain VQ contributes to bitrate. The bitrate is given as

$\text{Bitrate} = \frac{f_s}{D}\cdot \log_2 M_{pl},$

where $f_s$ is the sampling rate, $D$ is the encoder/decoder downsampling rate, and $M_{pl}$ is the plain codebook size (Jiang et al., 4 Jun 2026). This formula makes explicit that the pseudo VQs are zero-bitrate: they do not contribute to the transmitted token rate.

The pseudo VQs differ from ordinary quantizers in that they do not directly quantize the encoder output. Instead, each pseudo VQ predicts an auxiliary token using a neural network, conditioned on the plain token and previously predicted pseudo tokens. Each pseudo VQ $Q_{ps}^{(n)}$ has 3 Conformer blocks and 2 BiLSTM layers, which the paper says are used because they capture local spectral patterns and long-range temporal dependencies (Jiang et al., 4 Jun 2026).

For the $n$-th pseudo VQ, the prediction input is defined as

$\mathbf{e}\in\mathbb{R}^K .$0

The auxiliary token is then selected by

$\mathbf{e}\in\mathbb{R}^K .$1

The synergistic combination of plain and pseudo tokens is expressed by

$\mathbf{e}\in\mathbb{R}^K .$2

This is the paper’s central reconstruction mechanism: the transmitted basic token provides the base representation, and predicted pseudo tokens enrich that representation without increasing bitrate (Jiang et al., 4 Jun 2026).

4. Training methodology

The training procedure is divided into two stages (Jiang et al., 4 Jun 2026). In the first stage, all pseudo VQs are replaced with plain VQs, so the system becomes a standard RVQ codec. This teacher model is stated to be equivalent to MDCTCodec and is trained with adversarial loss, codebook loss, and spectral-level loss. The function of this stage is to establish the high-bitrate target behavior that the later pseudo-VQ predictors will attempt to approximate.

In the second stage, the encoder, plain VQ, and decoder are inherited from the teacher, while the pseudo VQs are trained with other modules fixed. The codebooks of pseudo VQs are inherited from the corresponding teacher codebooks according to

$\mathbf{e}\in\mathbb{R}^K .$3

Training then uses teacher forcing: inputs are the teacher RVQ tokens, and outputs are trained to predict the next token (Jiang et al., 4 Jun 2026).

For the $\mathbf{e}\in\mathbb{R}^K .$4-th pseudo VQ, the paper gives

$\mathbf{e}\in\mathbb{R}^K .$5

and the loss is cross-entropy: $\mathbf{e}\in\mathbb{R}^K .$6

The training logic is explicit: the teacher codec provides the target multi-stage token structure, and the pseudo VQs learn to reproduce later-stage RVQ behavior through prediction rather than transmission (Jiang et al., 4 Jun 2026). This suggests that the codec’s efficiency depends not only on the expressiveness of the plain token, but also on how predictable the teacher’s later quantizer decisions are from earlier quantized information.

5. Experimental configuration and reported performance

The reported evaluations use LibriTTS at 16 kHz and VCTK at 48 kHz (Jiang et al., 4 Jun 2026). For LibriTTS, the paper lists train-clean-100 and train-clean-360 for training, dev-clean for validation, and test-clean for evaluation. For VCTK, it reports 40,936 training utterances and 2,937 test utterances.

The P2PSynCodec configuration uses $\mathbf{e}\in\mathbb{R}^K .$7 pseudo VQs, codebook sizes

$\mathbf{e}\in\mathbb{R}^K .$8

code vector dimension

$\mathbf{e}\in\mathbb{R}^K .$9

Conformer modules with 256 channels and 8 attention heads, BiLSTM with 256 channels, and downsampling/upsampling rate

$$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$0

Using the bitrate formula above, the paper reports 0.5 kbps for $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$1 and 1.5 kbps for $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$2 (Jiang et al., 4 Jun 2026).

Objective metrics are UTMOS at 16 kHz, SIGMOS at 48 kHz, STOI, and ViSQOL. Complexity metrics are FLOPs and parameter count. Subjective evaluation uses MUSHRA and ABX preference tests on Amazon Mechanical Turk, with at least 25 native English listeners, natural speech as hidden reference, and a 3.5-kHz low-pass anchor in MUSHRA (Jiang et al., 4 Jun 2026).

At 0.5 kbps / 16 kHz and 1.5 kbps / 48 kHz, the paper reports the following objective and complexity comparison:

Codec	UTMOS / SIGMOS	FLOPs / Params
MDCTCodec	2.670 / 2.846	2.32G / 6.75M
DAC	2.725 / 2.971	55.53G / 73.87M
BigCodec	3.939 / 3.277	61.03G / 159.32M
WavTokenizer	3.269 / 3.232	4.21G / 71.65M
P2PSynCodec	3.947 / 3.305	3.31G / 22.99M

The paper states that P2PSynCodec is best or near-best in perceived quality, matches or exceeds BigCodec’s objective quality at a much smaller scale, uses about 5% of BigCodec’s FLOPs, and about 14% of BigCodec’s parameters (Jiang et al., 4 Jun 2026).

In a further ABX comparison against higher-bitrate codecs, the paper reports that P2PSynCodec at 0.5 kbps is judged comparable to MDCTCodec, DAC, and WavTokenizer at 2 kbps and SQCodec at 1.5 kbps, with $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$3 (Jiang et al., 4 Jun 2026). The paper interprets this as support for the claim that 0.5 kbps P2PSynCodec can match the perceptual quality of codecs running at 2.0 kbps.

6. Design tradeoffs, ablations, and relation to adjacent codec research

A key ablation varies the number of pseudo VQs, testing $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$4 (Jiang et al., 4 Jun 2026). The reported finding is that increasing $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$5 does not monotonically improve final speech quality and that the best tradeoff is at $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$6. The explanation given is structural: as $$\mathbb{W}_{pl}=\{\mathbf{w}_m\in\mathbb{R}^K \mid m=1,\dots,M_{pl}\},$$7 grows, the plain VQ carries less information, auxiliary tokens become harder to predict from the basic token, the distinction between basic and auxiliary roles weakens, and model complexity increases. The paper further states that although the teacher codec may improve with more VQs, the plain VQ alone becomes less informative, making pseudo-token prediction harder (Jiang et al., 4 Jun 2026).

This is an important corrective to a possible misconception. P2PSynCodec does not assert that more pseudo stages are intrinsically beneficial. Rather, its performance depends on an equilibrium between the informativeness of the transmitted token and the predictability of the non-transmitted tokens. The plain-to-pseudo decomposition therefore introduces a distinct optimization problem absent from standard RVQ: later latent structure must be both useful and inferable.

The paper also identifies explicit limitations. The system is not causal and is not yet designed for real-time streaming. Increasing pseudo VQs too much hurts token predictability and quality. Robustness under domain shift is not fully addressed, because pseudo-token generation depends on a learned decoder-side prediction process (Jiang et al., 4 Jun 2026). These limitations frame the current method as a highly efficient ultra-low-bitrate codec rather than a general solution to all deployment regimes.

Within the broader codec landscape, P2PSynCodec occupies a different design position from low-bitrate semantic codecs such as SimWhisper-Codec (Zhang et al., 23 Oct 2025). SimWhisper-Codec is built around a frozen, simplified Whisper encoder and targets the semantic–acoustic conflict through semantic-first representation design, whereas P2PSynCodec addresses bitrate inefficiency through a transmitted-plus-predicted quantization mechanism (Zhang et al., 23 Oct 2025, Jiang et al., 4 Jun 2026). This suggests that P2PSynCodec is best categorized as a quantization-centered efficiency method rather than a semantics-supervised or semantic-first codec.

7. Significance and interpretation

The main contributions claimed for P2PSynCodec are fivefold: P2PSynCodec as an ultra-low-bitrate neural speech codec based on MDCT spectra, P2PSVQ as a quantization scheme with one transmitted plain VQ and multiple zero-bit pseudo VQs, a teacher-forcing training framework using an RVQ teacher codec, demonstration that 0.5 kbps operation can achieve quality comparable to 2.0 kbps codecs, and strong efficiency in both quality and model size (Jiang et al., 4 Jun 2026).

Its broader implication is stated in direct terms: ultra-low-bitrate speech coding does not necessarily require bigger models; smarter use of quantization can recover much of the lost expressiveness (Jiang et al., 4 Jun 2026). The “plain-to-pseudo” paradigm therefore points toward a more general compression strategy in which only a minimal token stream is transmitted while richer latent structure is reconstructed locally by learned prediction.

A plausible implication is that the P2PSVQ formulation could be relevant beyond speech, wherever latent decompositions exhibit a strong asymmetry between indispensable coarse structure and predictable fine structure. Within the paper itself, however, the demonstrated application domain remains ultra-low-bitrate speech coding, with prospective relevance to satellite communication, IoT voice interfaces, low-storage voice archiving, and bandwidth-constrained speech transmission (Jiang et al., 4 Jun 2026).