PURE Codec: Progressive Residual Entropy

• PURE Codec is a neural speech compression framework that decomposes speech signals into a progressive hierarchy of entropy, enabling efficient multi-stage quantization.
• It integrates a pre-trained speech enhancer to anchor low-entropy, denoised structures while subsequent stages capture high-entropy residual details.
• PURE Codec achieves robust performance across varied conditions, enhancing downstream synthesis quality and overall compression efficiency.

The PURE Codec (Progressive Unfolding of Residual Entropy) is a neural speech compression framework that introduces a pre-trained speech enhancement model for guiding residual vector quantization (RVQ). Its core innovation addresses instability and redundancy in conventional multi-stage quantization by decomposing the embedded speech signal into a hierarchy reflecting perceptual entropy, improving training robustness and rate-distortion performance, especially under noisy or limited data conditions. PURE Codec achieves these properties by organizing quantization stages such that the first stage captures low-entropy, denoised structure, with residual, higher-entropy detail captured progressively in subsequent stages (Shi et al., 27 Nov 2025).

1. Motivations and Core Contributions

Neural speech codecs leveraging multi-stream RVQ architectures are attractive for low-bitrate, high-quality compression, but are hindered by two primary deficiencies: late-stage quantizers often degenerate (collapse) or become highly redundant, and the apportionment of information across codebooks is typically suboptimal without explicit guidance. PURE Codec addresses these deficiencies through enhancement-informed RVQ.

The main technical contributions are:

• Integration of a pre-trained, frozen speech enhancer to produce low-entropy embeddings for anchoring the first quantization stream.
• A two-stage training procedure combining variational autoencoding (VAE) pretraining and stochastic enhancement supervision.
• Demonstrated robustness and improved rate–distortion and downstream speech LLM-based synthesis performance across clean and noisy datasets.

This design enforces an information hierarchy on the quantization stages, assigning the most compressible (perceptually salient) structure to earlier streams (Shi et al., 27 Nov 2025).

2. Architectural Details: Enhancement-Guided Multi-Stage RVQ

The framework adopts the conventional encoder–quantizer–decoder topology. Let $S$ denote the input waveform, with encoder output $$Q = Enc(S) = [q_1,\dots,q_T] \in \mathbb{R}^{D \times T}$$. A pre-trained enhancement model produces a denoised waveform $S^{\mathrm{enh}}$, yielding low-entropy embeddings $\tilde{Q} = Enc(S^{\mathrm{enh}})$. Empirically, these enhanced embeddings exhibit approximately 58% lower perceptual entropy than their raw counterparts.

Quantization proceeds in $L$ sequential streams (codebooks):

• Stage 1: Forced alignment to the denoised embeddings

$$c_t^1 = \arg\min_j \|\tilde{q}_t - b_j^1\|_2^2,\quad \hat{q}_t^{(1)} = b_{c_t^1}^1,\quad r_t^1 = q_t - \hat{q}_t^{(1)}$$

• Stages $l > 1$: Residual refinement

$$c_t^l = \arg\min_j \|r_t^{l-1} - b_j^l\|_2^2,\quad \hat{q}_t^{(l)} = b_{c_t^l}^l,\quad r_t^l = r_t^{l-1} - \hat{q}_t^{(l)}$$

• Reconstruction

$$\hat{q}_t = \sum_{l=1}^L \hat{q}_t^{(l)},\quad \hat{Q} = [\hat{q}_1,\dots,\hat{q}_T],\quad \hat{S} = Dec(\hat{Q})$$

Anchoring the first stream to low-entropy content ensures that the initial codebook captures easily compressible speech structure, while later streams focus on residual high-entropy detail. This staged decomposition enables more effective bitrate allocation and improves downstream representation quality.

3. Mathematical Underpinnings and Information Decomposition

The quantization structure of PURE Codec emphasizes entropy-aware information allocation:

• Total embedded information is approximately decomposed as:

$$H(Q) \approx H(\hat{Q}^{(1)}) + \sum_{l=2}^L H(r^l)$$

where $H(\cdot)$ denotes differential or perceptual entropy.

• The first stage’s codebook captures low-entropy, semantically pertinent components, with each residual $r^l$ encoding successively more complex or noisy detail. This progressive partitioning structurally differentiates PURE from standard RVQ, which lacks explicit entropy alignment.
• Training is staged: First a VAE pretraining phase (no quantization), with reconstruction ($L_1$) and KL regularization, then quantization with adversarial and enhancement-guidance terms.

Letting $\mathcal{L}_{\mathcal{G}}$ denote the generator loss: $$\mathcal{L}_{\mathcal{G}} = \lambda_{\mathrm{enh}}\,\mathcal{L}_{\mathrm{enh}} + \lambda_{\mathrm{rec}}\,\mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{vq}}\,\mathcal{L}_{\mathrm{vq}} + \lambda_{\mathrm{adv}}\,\mathcal{L}_{\mathrm{adv}}^{\mathcal{G}}$$ where

• $\mathcal{L}_{\mathrm{enh}}$ is an $\ell_2$ loss on the first stream’s embedding to the denoised reference,
• $\mathcal{L}_{\mathrm{rec}}$ combines waveform $L_1$ and mel-spectrogram differences,
• $\mathcal{L}_{\mathrm{vq}}$ is the vector quantization codebook loss,
• $\mathcal{L}_{\mathrm{adv}}^{\mathcal{G}}$ is the GAN loss.

Stochastic supervision of stage 1 using the enhanced embedding, probability $p_{\mathrm{enh}}$, additionally controls regularization strength. Empirical ablations indicate $p_{\mathrm{enh}} \approx 0.25–0.5$ is optimal (Shi et al., 27 Nov 2025).

4. Training Protocols and Experimental Regimen

PURE Codec training is performed on 16 kHz datasets (OWSM-v3.2, CommonVoice V13, and URGENT ‘24), using encoders with hidden dimension $D = 512$, $L = 8$ streams, and codebooks of 1,024 bins (per stream). The training schedule comprises:

1. VAE pretraining: 180 epochs, batch size 64, KL weight 0.1.
2. Quantization + adversarial training: 180–360 epochs, batch size 32, using AdamW with learning rate $2 \times 10^{-4}$, weight decay $10^{-2}$, and exponential LR decay ($\gamma = 0.999875$). Encoder is frozen in the second stage; enhancement regularization with dropout/probability $p_{\mathrm{enh}} = 0.5$.

Loss weights are empirically set as $$\lambda_{\mathrm{enh}} = 0.1, \lambda_{\mathrm{rec}} = 1.0, \lambda_{\mathrm{vq}} = 0.25, \lambda_{\mathrm{adv}} = 1.0$$.

This protocol, especially the separated VAE pretraining and stochastic enhancement scheduling, is critical for optimization stability and for mitigating RVQ codebook collapse in late stages (Shi et al., 27 Nov 2025).

5. Empirical Results: Reconstruction, Robustness, and Downstream Quality

The PURE Codec demonstrates improved performance across reconstruction, conversational robustness, and downstream utility as measured by speech-LLM-based text-to-speech synthesis. Key metrics include signal to distortion ratio (SDR), wideband PESQ, UTMOS, DNSMOS, VISQOL, word error rate (WER), and speaker similarity (SPK-SIM). The following table summarizes representative results:

Training Set	Model	SDR↑	PESQ↑	UTMOS↑	DNSMOS↑	VISQOL↑	WER↓	SPK-SIM↑
OWSM (40k h)	DAC baseline	4.01	2.37	3.42	3.17	4.34	2.26 %	0.64
	PURE Codec	2.17	2.62	3.64	3.21	4.41	2.05 %	0.71
CommonVoice (16k h)	DAC baseline	-5.21	1.36	1.45	2.13	4.07	3.00 %	0.31
	PURE Codec	2.70	2.70	3.65	3.18	4.39	2.14 %	0.76
URGENT (2k h)	DAC baseline	-6.79	1.32	1.31	1.97	4.12	3.67 %	0.38
	PURE Codec	1.35	2.50	3.41	3.09	4.36	2.07 %	0.54

These results indicate that PURE Codec provides substantial stability even as data quality degrades; the baseline model collapses under hard conditions, but PURE remains robust (Shi et al., 27 Nov 2025).

In downstream SpeechLM-based TTS (LibriSpeech-960h), PURE achieves lower WER (10.5% vs 10.8%), similar or improved SPK-SIM, and a notable increase in UTMOS (3.95 vs 3.68) compared to baseline RVQ.

6. Design Considerations, Ablations, and Limitations

Critical findings from ablations include:

• Encoder weights should remain frozen during quantization training.
• Stochastic enhancement injection in the first stream, with $p_{\mathrm{enh}} \approx 0.25–0.50$, is required to balance structure guidance and model flexibility.
• Supervising only the first stream provides optimal performance; extending supervision degrades rate–distortion tradeoff.
• Larger speech enhancer models provide small but measurable improvements.

The principal limitation is the reliance on a frozen speech enhancement model, restricting direct applicability to general audio. The $p_{\mathrm{enh}}$ hyperparameter is also set manually. Future research directions include joint enhancer+codec optimization, entropy-adaptive scheduling, semantically-motivated quantizer stages, and generalization beyond speech.

7. Significance and Impact

PURE Codec demonstrates that anchoring multi-stage vector quantization on entropy-aware priors derived from speech enhancement models enforces a robust inductive bias for neural speech coding. This yields stable, high-fidelity, and efficient codecs that better serve both reconstruction and semantic downstream generative modeling. The progressive unfolding of residual entropy, as instantiated in the PURE design, overcomes typical RVQ failure modes and opens avenues for more principled information allocation in neural audio coding pipelines (Shi et al., 27 Nov 2025).