Residual Activation Module with SimVQ

• The paper introduces a residual activation module that quantizes the acoustic residual using a global reparameterized SimVQ codebook, enhancing speech fidelity.
• It decouples semantic and acoustic information by subtracting the upstream embedding from the encoder output, enabling precise quantization of fine details.
• Experimental results show that a trade-off exists between codebook size and bitrate, with marginal quality gains from increased codebook sizes.

The Residual Activation Module with SimVQ is a quantization architecture implemented in SACodec, a neural speech codec optimized for high-fidelity and semantic preservation at extremely low bitrates. Situated as the acoustic branch (Q₂) of a dual-pathway codec, this module operates by explicitly modeling the acoustic residual—defined as the difference between the frame-wise encoder output and a semantics-driven embedding—then quantizing this residual with a specially parameterized codebook. The core innovation consists of a SimVQ-based (Simple Vector Quantization) codebook constructed through a global reparameterization, ensuring full code utilization and parameter efficiency. This enables accurate recovery of fine-grained acoustic details, critical for perceptual reconstruction quality, especially under bandwidth constraints (Dong et al., 24 Dec 2025).

1. Architectural Overview and Inputs

The module receives the frame-wise acoustic residual vector: $\mathbf{r}_t = \mathbf{h}_t - \mathbf{e}_{1,t}$ where $\mathbf{h}_t \in \mathbb{R}^D$ is the encoder output at time $t$ and $\mathbf{e}_{1,t}$ represents the semantic embedding sourced from an upstream quantizer (Q₁). This decouples semantic and acoustic information, with the residual $\mathbf{r}_t$ containing non-semantic, fine-structure acoustic details necessary for high-fidelity reconstruction.

2. SimVQ Codebook Construction

The effective residual codebook, denoted $\mathcal{C}_2$, is generated using a global reparameterization: $$\mathcal{C}_2 = \mathbf{C}_{\mathrm{coeff}} \, \mathbf{W}_{\mathrm{basis}}$$ Here, $$\mathbf{C}_{\mathrm{coeff}} \in \mathbb{R}^{K_2\times d}$$ is a frozen, randomly-initialized coefficient matrix, and $$\mathbf{W}_{\mathrm{basis}} \in \mathbb{R}^{d\times D}$$ is a learnable basis. Only $\mathbf{W}_{\mathrm{basis}}$ receives gradient updates, assuring that each code-vector $\mathbf{c}_{2,k}$ within the codebook is updated globally, avoiding the local update limitations of traditional vector quantization approaches.

Key Codebook Hyper-parameters

Hyper-parameter	Value(s) Used	Observed Effect
Residual codebook size $K_2$	$1024$ (baseline); $2048$ (ablation)	Larger $K_2$ offers marginal UTMOS gain (+0.0029) at higher bitrate cost
Latent basis dimension $d$	$64$, $128$	Parameter efficiency; $d\ll D$
Commitment weight $\lambda_{c2}$	$5.0$	Balances quantization stability with fidelity

3. Residual Quantization and Reconstruction

Each residual vector is quantized via nearest-neighbor lookup in the SimVQ-reparameterized codebook: $$i_{2,t} = \arg\min_{k \in \{1, \dots, K_2\}} \|\mathbf{r}_t - \mathbf{c}_{2,k}\|^2, \qquad \mathbf{e}_{2,t} = \mathbf{c}_{2,i_{2,t}}$$ The quantized residual embedding $\mathbf{e}_{2,t}$ is then fused element-wise with the semantic embedding: $$\mathbf{e}_{\mathrm{final},t} = \mathbf{e}_{1,t} + \mathbf{e}_{2,t}$$ This composite embedding is subsequently fed to a ConvNeXt-Attention decoder followed by inverse STFT to reconstruct the waveform.

4. Training Objectives and Loss Functions

Optimization employs a multi-term generator loss: $$\mathcal{L}_G = \lambda_{\mathrm{rec}} \mathcal{L}_{\mathrm{rec}} + \lambda_{\mathrm{adv}} \mathcal{L}_{\mathrm{adv}} + \lambda_{\mathrm{feat}} \mathcal{L}_{\mathrm{feat}} + \lambda_{c1} \mathcal{L}_{\mathrm{com,1}} + \lambda_{c2} \mathcal{L}_{\mathrm{com,2}}$$ Main components:

• $\mathcal{L}_{\mathrm{rec}}$: Multi-scale mel-spectrogram loss for reconstruction fidelity.
• $\mathcal{L}_{\mathrm{adv}}$: Adversarial loss from MPD and MS-STFT discriminators, promoting perceptual quality.
• $\mathcal{L}_{\mathrm{feat}}$: Feature-matching loss to align internal feature representations.
• $\mathcal{L}_{\mathrm{com,2}}$: Commitment loss specific to Q₂, enforcing stability in code-vector usage:

$$\mathcal{L}_{\mathrm{com,2}} = \mathbb{E}_t \big\| \mathbf{r}_t - \mathrm{sg}(\mathbf{e}_{2,t}) \big\|_2^2$$

with $\mathrm{sg}(\cdot)$ as the stop-gradient operator.

Hyper-parameters are set to $\lambda_{\mathrm{rec}} = 45.0$, $\lambda_{\mathrm{adv}} = 1.0$, $\lambda_{\mathrm{feat}} = 1.0$, $\lambda_{c1} = 25.0$, $\lambda_{c2} = 5.0$.

5. Ablations and Quantitative Impact

Removal of the residual activation module ("Semantic-only" configuration) demonstrates significant degradations: Perceptual Evaluation of Speech Quality (PESQ) drops from $2.69$ to $2.35$ ($-12.8\%$), and UTMOS decreases from $4.04$ to $3.95$. This illustrates that the Q₂ acoustic pathway is essential for recovering fine acoustic details, complementing the semantics-only pathway and preventing collapse of perceptual quality.

Additionally, increasing the codebook size to $K_2 = 2048$ yields only a marginal increase in UTMOS (+0.0029) at the expense of $+1$ kbps overhead, establishing $1024$ as a practical trade-off between bitrate and quality.

6. Significance, Limitations, and Broader Implications

The residual activation module with SimVQ in SACodec achieves efficient and expressive acoustic modeling at extreme compression rates (1.5 kbps), maintaining high subjective and objective fidelity. The SimVQ formulation ensures all codebook entries remain useful throughout training, circumventing codebook collapse and maximizing representational capacity. A plausible implication is that the decoupled design and codebook parameterization strategies found in SACodec can inform future developments in neural compression, not only for speech but potentially for other modalities where separating semantic and non-semantic components is desirable (Dong et al., 24 Dec 2025). Limitations regarding codebook size scaling and bitrate trade-offs are partially addressed through ablations but warrant further investigation for broader deployment scenarios.