P2PSVQ: Plain-to-Pseudo Vector Quantizer

• The paper introduces P2PSVQ as a quantization method that combines one plain VQ with multiple pseudo VQs to enable ultra-low-bitrate transmission.
• It uses neural prediction to generate auxiliary tokens without extra transmission, preserving the multi-codebook reconstruction structure of RVQ.
• Empirical results show that P2PSVQ achieves competitive perceptual quality at significantly reduced bitrates and computational cost compared to traditional codecs.

Plain-to-Pseudo Synergistic Vector Quantizer (P2PSVQ) is a quantization scheme introduced within P2PSynCodec, an ultra-low-bitrate neural speech codec, to decouple representational richness from transmitted bitrate. In the formulation reported in "An Ultra-Low-Bitrate Neural Speech Codec with Plain-to-Pseudo Synergistic Vector Quantization" (Jiang et al., 4 Jun 2026), P2PSVQ consists of one plain VQ and multiple pseudo VQs. The plain VQ produces basic tokens by quantization, while the pseudo VQs generate auxiliary tokens by neural prediction and incur zero transmitted bitrate. Speech is then decoded from the plain-VQ tokens together with predicted pseudo-VQ tokens. Within the same data block, UniSVQ in "UniSVQ: 2-bit Unified Scalar-Vector Quantization" (Wang et al., 9 Jun 2026) is described as conceptually close to a P2PSVQ-style design in LLM quantization, and "Stochastic Vector Quantisers" (Luttrell, 2010) provides a technical precursor through multi-index stochastic encoding and superposed reconstruction.

1. Definition and conceptual position

P2PSVQ was proposed to address a specific inefficiency of residual vector quantization (RVQ): later VQ stages often contribute less but consume the same bitrate. In ordinary multi-stage RVQ, early stages capture coarse content and later stages refine residual errors, but every stage index must be transmitted. At ultra-low bitrates such as $0.5$ kbps, the paper identifies this as a rate-efficiency problem, because later-stage refinements may provide only incremental gains while still requiring a full token stream (Jiang et al., 4 Jun 2026).

The terminology in the source is precise. The single plain VQ $Q_{pl}$ produces the basic token $d_{pl}$, and the $N$ pseudo VQs $Q_{ps}^{(n)}$ produce auxiliary tokens $\hat d_{ps}^{(n)}$. The plain VQ is “plain” because it is a normal vector quantizer applied directly to the encoder output. The pseudo VQs are “pseudo” because they mimic later VQ stages in a multi-stage quantizer, but their discrete outputs are generated by neural prediction rather than by explicit quantization of the encoder latent and explicit transmission (Jiang et al., 4 Jun 2026).

This construction preserves the additive multi-codebook reconstruction structure of RVQ while transmitting only the first-stage token stream. Relative to predictive coding, the prediction target is not waveform samples or continuous residuals but discrete codebook tokens. Relative to latent token generation methods, the pseudo networks are described as stage-conditioned token predictors anchored to codebooks inherited from an RVQ teacher (Jiang et al., 4 Jun 2026). A plausible implication is that P2PSVQ occupies an intermediate position between conventional multi-stage vector quantization and decoder-side predictive reconstruction.

2. Codec architecture and signal path

The P2PSVQ mechanism was introduced in P2PSynCodec, a lightweight MDCT-domain neural codec (Jiang et al., 4 Jun 2026). The codec first converts the input waveform into MDCT spectra, then uses a fully convolutional encoder to compress them into latent frame-level representations. The encoder backbone is a modified ConvNeXt v2, and its residual blocks contain 1D depthwise convolution, layer normalization, a linear layer, GRN, and GELU. Temporal compression is performed by a 1D downsampling layer, while the decoder mirrors the encoder with upsampling and reconstructs MDCT spectra before IMDCT returns the waveform (Jiang et al., 4 Jun 2026).

The distinctive component is inserted between encoder and decoder. The encoder outputs a compressed latent vector for each frame, and the plain VQ quantizes this latent and produces the only token transmitted over the channel or stored in the bitstream. At the decoder, that received token is embedded through the plain codebook and then used as the conditioning signal for a sequence of pseudo VQs. These pseudo VQs do not quantize the encoder latent directly during deployment; instead, they infer auxiliary tokens from the plain token and, recursively, from previously predicted pseudo tokens (Jiang et al., 4 Jun 2026).

All transmitted and predicted tokens are then looked up in their respective codebooks and summed into a reconstructed quantized latent. This final latent is decoded back to speech. The architecture therefore retains a multi-codebook latent representation even though the bitstream carries only the basic token stream. The paper’s target application is ultra-low-bitrate speech coding for bandwidth-constrained scenarios such as satellite communication, IoT voice interfaces, or on-device storage (Jiang et al., 4 Jun 2026).

3. Mathematical formulation

Let the encoder output a frame-level vector be

$\bm e \in \mathbb R^K,$

where $K$ is the code vector dimension. The plain quantization stage uses codebook

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$

and the basic token is selected by nearest-neighbor search: $d_{pl} = \arg\min_m \| \bm e - \bm w_m \|_2.$ This basic token is the only truly transmitted token stream in the codec (Jiang et al., 4 Jun 2026).

The bitrate depends only on the frame rate after downsampling and on the plain-codebook size: $Q_{pl}$0 where $Q_{pl}$1 is the sampling rate and $Q_{pl}$2 is the encoder/decoder downsampling and upsampling rate. With $Q_{pl}$3 and $Q_{pl}$4, the paper reports $Q_{pl}$5 bits per token. Thus at $Q_{pl}$6, the rate is $Q_{pl}$7, and at $Q_{pl}$8, the rate is $Q_{pl}$9 (Jiang et al., 4 Jun 2026).

For each pseudo stage $d_{pl}$0, the pseudo VQ $d_{pl}$1 has its own codebook $d_{pl}$2 and a neural predictor $d_{pl}$3 composed of three Conformer blocks and two BiLSTM layers. The predictor outputs logits

$d_{pl}$4

conditioned on the plain token and, for later pseudo stages, on previously predicted pseudo-stage embeddings: $d_{pl}$5 The predicted auxiliary token is

$d_{pl}$6

The final quantized latent is then

$d_{pl}$7

This is the paper’s explicit synergistic mechanism: the plain token provides the indispensable coarse representation, while the pseudo tokens act as learned refinements without increasing transmitted rate (Jiang et al., 4 Jun 2026).

4. Training procedure and teacher-student structure

The paper uses a two-stage training strategy (Jiang et al., 4 Jun 2026). In the first stage, the whole quantizer stack is converted into an ordinary $d_{pl}$8-stage RVQ teacher model by temporarily replacing the pseudo VQs with normal plain VQs, yielding $d_{pl}$9 with codebooks $N$0. This teacher is equivalent to an MDCTCodec-style RVQ codec and is trained using the same recipe as MDCTCodec, namely generative adversarial loss, codebook loss, and spectral-level loss. The paper does not reproduce the exact formulas for these losses in the main text (Jiang et al., 4 Jun 2026).

In the second stage, the encoder, decoder, and first-stage plain VQ are frozen from the teacher, and the pseudo VQs are trained to imitate the later RVQ stages. The mapping is

$N$1

and

$N$2

Thus each pseudo VQ reuses the corresponding later-stage teacher codebook (Jiang et al., 4 Jun 2026).

Let the teacher RVQ quantize $N$3 into tokens $N$4. The pseudo VQs are trained with teacher forcing and are described as being “actually trained independently.” For the $N$5-th pseudo VQ, the input is

$N$6

and the target distribution $N$7 is the one-hot encoding of the teacher token. The loss is cross-entropy: $N$8 This makes the pseudo VQs stage-specific classifiers that predict later-stage RVQ indices from earlier-stage quantized information (Jiang et al., 4 Jun 2026).

The distinction between training and deployment is important. During pseudo-VQ training, teacher-forcing inputs derived from true teacher RVQ tokens are used. During inference, those teacher tokens are unavailable, so the decoder receives only $N$9 and generates

$Q_{ps}^{(n)}$0

recursively using its own previous predictions. The paper identifies this as a train–test gap typical of teacher-forced predictors, while reporting that with a moderate number of pseudo VQs the quality remains strong (Jiang et al., 4 Jun 2026).

5. Bitrate efficiency, operating regimes, and empirical results

The reported system uses

$Q_{ps}^{(n)}$1

so there is one plain VQ and three pseudo VQs. All codebooks have size

$Q_{ps}^{(n)}$2

the code vector dimension is

$Q_{ps}^{(n)}$3

and the temporal downsampling factor is

$Q_{ps}^{(n)}$4

Each pseudo VQ predictor uses Conformer blocks with 256 channels and 8 attention heads, plus BiLSTMs with 256 channels. The design is non-causal (Jiang et al., 4 Jun 2026).

The codec was evaluated on LibriTTS at 16 kHz and VCTK at 48 kHz. Objective metrics include UTMOS and SIGMOS as non-intrusive quality estimators, STOI and ViSQOL as intrusive metrics, complexity in FLOPs and parameter count, and subjective tests including MUSHRA and ABX preference (Jiang et al., 4 Jun 2026).

On LibriTTS at $Q_{ps}^{(n)}$5 kbps, the reported objective results are as follows.

Method	Objective scores	Complexity
MDCTCodec	UTMOS 2.670, STOI 0.844, ViSQOL 3.631	2.32G FLOPs, 6.75M params
DAC	UTMOS 2.725, STOI 0.818, ViSQOL 3.386	55.53G FLOPs, 73.87M params
BigCodec	UTMOS 3.939, STOI 0.872, ViSQOL 3.682	61.03G FLOPs, 159.32M params
WavTokenizer	UTMOS 3.269, STOI 0.834, ViSQOL 3.484	4.21G FLOPs, 71.65M params
P2PSynCodec	UTMOS 3.947, STOI 0.823, ViSQOL 3.476	3.31G FLOPs, 22.99M params

At $Q_{ps}^{(n)}$6 kHz / $Q_{ps}^{(n)}$7 kbps, P2PSynCodec gives the best SIGMOS, reported as $Q_{ps}^{(n)}$8, among the listed methods (Jiang et al., 4 Jun 2026). In subjective MUSHRA, it outperforms MDCTCodec and DAC and is competitive with BigCodec while using only about $Q_{ps}^{(n)}$9 of BigCodec’s FLOPs and $\hat d_{ps}^{(n)}$0 of its parameters (Jiang et al., 4 Jun 2026).

The most direct rate-efficiency comparison is against a four-stage RVQ teacher with $\hat d_{ps}^{(n)}$1. If each stage has codebook size $\hat d_{ps}^{(n)}$2, then a conventional four-stage RVQ requires $\hat d_{ps}^{(n)}$3 bits per frame. At $\hat d_{ps}^{(n)}$4 frames/s, that is $\hat d_{ps}^{(n)}$5 bit/s, or $\hat d_{ps}^{(n)}$6 kbps. P2PSynCodec keeps the representational structure of four summed codebooks but transmits only one stage, yielding $\hat d_{ps}^{(n)}$7 bits/frame $\hat d_{ps}^{(n)}$8 frames/s $\hat d_{ps}^{(n)}$9 kbps. The paper describes this as a $\bm e \in \mathbb R^K,$0 bitrate reduction relative to a 4-stage RVQ teacher such as MDCTCodec at 2.0 kbps (Jiang et al., 4 Jun 2026).

In ABX tests on 16 kHz LibriTTS, P2PSynCodec at $\bm e \in \mathbb R^K,$1 kbps is reported to achieve perceptual quality comparable to MDCTCodec, DAC, and WavTokenizer at $\bm e \in \mathbb R^K,$2 kbps, as well as SQCodec at $\bm e \in \mathbb R^K,$3 kbps, with no significant preference difference $\bm e \in \mathbb R^K,$4 (Jiang et al., 4 Jun 2026). This is the empirical result most directly associated with the intended role of P2PSVQ.

6. Design sensitivity, limitations, and interpretive context

The ablation on the number of pseudo VQs identifies a central operating constraint. The authors vary $\bm e \in \mathbb R^K,$5. For the final system using all VQs, UTMOS peaks around $\bm e \in \mathbb R^K,$6 to $\bm e \in \mathbb R^K,$7, with $\bm e \in \mathbb R^K,$8 giving $\bm e \in \mathbb R^K,$9 and $K$0 giving $K$1, but other metrics degrade for too many pseudo VQs. The “Only Plain VQ” decoding results show why: as $K$2 increases, quality obtainable from the plain token alone collapses. The reported UTMOS values using only the plain VQ are $K$3 at $K$4, $K$5 at $K$6, $K$7 at $K$8, and $K$9 at $$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$0 (Jiang et al., 4 Jun 2026).

The paper interprets this as follows: when the teacher has many RVQ stages, the first stage carries less and less information because the representation burden is spread across later stages. That weakens the basic token, making it harder for pseudo VQs to infer the omitted auxiliary tokens. This is why the authors state that the number of pseudo VQs should be moderate and select $$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$1 as the best trade-off between quality and complexity (Jiang et al., 4 Jun 2026).

Several limitations are stated directly. The current design is non-causal because the pseudo predictors use BiLSTM-based architectures. Performance depends on the predictability of later-stage tokens from the first-stage token and preceding pseudo stages; if prediction errors accumulate, quality degrades. Too many pseudo VQs can be counterproductive because the basic token becomes too weak. In addition, pseudo-token training uses teacher forcing, whereas deployment conditions on predicted pseudo tokens rather than ground-truth teacher tokens (Jiang et al., 4 Jun 2026).

The paper also argues that non-intrusive perceptual metrics and subjective results are more appropriate indicators in this setting, because generative reconstruction can be penalized unfairly by intrusive reference-based measures (Jiang et al., 4 Jun 2026). This is not a rejection of intrusive metrics, but a statement about metric interpretation under generative decoding.

7. Related formulations and broader significance

Within the supplied research context, two additional papers situate P2PSVQ conceptually. "Stochastic Vector Quantisers" (Luttrell, 2010) presents a stochastic generalisation of vector quantization in which an input vector is encoded by multiple sampled code indices,

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$2

with

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$3

and reconstruction is formed by superposition,

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$4

Its upper-bound objective

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$5

balances single-component fidelity and composite reconstruction fidelity. The paper shows that such a system can self-organize into factorial or subspace-specialized codes (Luttrell, 2010). This suggests a theoretical precursor to P2PSVQ’s synergistic use of multiple discrete components, even though that work does not use the term P2PSVQ.

"UniSVQ: 2-bit Unified Scalar-Vector Quantization" (Wang et al., 9 Jun 2026) addresses a different problem domain—post-training quantization for LLMs—but is described in the supplied details as highly aligned conceptually with a P2PSVQ-style design. UniSVQ parameterizes codewords as an affine transform of integer lattices,

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$6

thereby introducing vector-level expressiveness while preserving compatibility with optimized integer kernels (Wang et al., 9 Jun 2026). The supplied interpretation characterizes this as retaining a “plain” integer-coded execution path while adding “pseudo-vector” structure through a shared affine lattice warp. This suggests that the plain-to-pseudo idea is not restricted to speech codecs; it can also describe structured transitions between scalar-like deployment and vector-like representational capacity.

Across these papers, the recurring theme is that a representation need not be limited to a single explicitly transmitted or explicitly stored codeword in order to preserve vector-quantized expressiveness. In P2PSVQ proper, that principle appears in the specific form

$$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$7

where only $$\mathbb W_{pl} = \{\bm w_m \in \mathbb R^K \mid m=1,\dots,M_{pl}\},$$8 is transmitted (Jiang et al., 4 Jun 2026). The resulting significance of P2PSVQ lies in replacing the standard assumption that every useful discrete latent refinement must be carried in the bitstream.