RFSQ: Robust Residual Finite Scalar Quantization

• RFSQ is a quantization framework that overcomes residual magnitude decay by conditioning the signals via learnable scaling or invertible LayerNorm.
• It preserves the additive residual decomposition, enabling efficient autoencoder integration and achieving lower L1 error and improved perceptual metrics in neural compression.
• In AffectCodec, Block-Diagonal RFSQ enforces explicit bit allocation between emotion and acoustic subspaces, preserving emotion-relevant information at low bitrates.

Robust Residual Finite Scalar Quantization (RFSQ) is a residual quantization framework built on Finite Scalar Quantization (FSQ) that is designed to keep multi-stage scalar quantization effective when a naive residual stack would otherwise suffer from progressively weaker residual signals. In the formulation introduced for neural compression, RFSQ conditions each residual stage with either learnable scaling factors or invertible layer normalization, while preserving the additive residual decomposition $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k$ and end-to-end training with straight-through gradient estimators (Zhu, 20 Aug 2025). In later speech codec work, AffectCodec realizes a structurally constrained variant, Block-Diagonal RFSQ (BD-RFSQ), in which emotion and acoustic subspaces are quantized through block-diagonal input and output projections, yielding explicit and structurally guaranteed bit allocation together with a flat token interface for downstream speech LLMs (Meng et al., 22 May 2026).

1. Origins in finite scalar and residual quantization

FSQ was introduced as a simpler, more stable alternative to classical Vector Quantization (VQ). Rather than learning a codebook, FSQ quantizes each dimension independently to a fixed number of levels. For a scalar $z_i$, the per-dimension quantizer is

$$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$

so that the overall code-rate is $\sum_i \log_2 L_i$ bits (Zhu, 20 Aug 2025).

The central motivation for RFSQ is the residual magnitude decay problem. Residual quantization repeatedly quantizes the error from the previous stage, but under naive residual FSQ the residual after the first step becomes small enough that subsequent FSQ layers receive vanishingly small signals and contribute little refinement. The generic RFSQ framework was proposed to restore effective multi-stage FSQ while retaining the simplicity of scalar quantization, lookup-free operation, and the absence of learned codebook maintenance (Zhu, 20 Aug 2025).

This original notion of robustness is therefore tied to the stability and efficacy of residual FSQ itself rather than to noisy-channel robustness or attribute preservation. That distinction becomes important in later literature, where the same acronym is adapted to different failure modes and application domains.

2. Formal definition of the generic RFSQ framework

In the generic formulation, RFSQ takes an input feature $\mathbf{z}\in\mathbb{R}^d$ and maintains residuals $\{\mathbf{r}_k\}_{k=0}^K$ with $\mathbf{r}_0=\mathbf{z}$. The overall quantized representation is the sum of the stagewise outputs,

$\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$

At each stage, the residual is conditioned, passed through FSQ, and then subtracted from the running residual so that exact reconstruction across stages is preserved (Zhu, 20 Aug 2025).

Two conditioning strategies define the canonical RFSQ variants. The first uses a learned positive scalar $\alpha_k$ at each stage:

$$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$

The second uses invertible LayerNorm. In that variant, the residual is normalized by a per-channel affine LayerNorm, quantized, and then mapped back through the inverse transform using the stored statistics, after which the residual is updated. The paper emphasizes that because the normalization statistics are stored, the LayerNorm transform is perfectly invertible (Zhu, 20 Aug 2025).

The corresponding pipeline is straightforward. Starting from $z_i$0 and $z_i$1, each stage applies one of the conditioning strategies, performs an FSQ step, updates $z_i$2, and accumulates $z_i$3. Each FSQ call also emits discrete indices for entropy coding. The training loss is applied through a shared decoder with straight-through estimators, and no extra regularization on $z_i$4 or the LayerNorm parameters is required (Zhu, 20 Aug 2025).

A practical implication is that RFSQ can be inserted into an autoencoder by replacing a single quantizer with a $z_i$5-stage residual FSQ module, adding conditioning before each FSQ stage, and summing the outputs. In the generic image-compression setting, this replacement does not require extra losses or codebook maintenance.

3. Empirical profile in neural compression

The initial RFSQ study evaluates the method on ImageNet 1K with images resized to $z_i$6 and normalized to $z_i$7. The encoder uses three convolutional layers with channels $z_i$8 and output $z_i$9, while the decoder is a symmetric transposed-convolution network that upsamples back to $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$0. All methods are compared at the same code rate of $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$1 bits and total codebook size $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$2 (Zhu, 20 Aug 2025).

Optimization is also fixed across methods: Adam with $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$3, weight decay $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$4, initial learning rate $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$5 with linear decay to zero, warmup over $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$6 of steps, batch size $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$7, training for $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$8 epochs, and gradient clipping at $$\mathrm{FSQ}_i(z_i) = \mathrm{round}\Bigl(\frac{z_i(L_i-1)}{2}\Bigr)\,\frac{2}{L_i-1},$$9. The reconstruction objective is

$\sum_i \log_2 L_i$0

Under these conditions, the best-performing reported variant is RFSQ-4$\sum_i \log_2 L_i$11024-LayerNorm, which achieves $\sum_i \log_2 L_i$2, perceptual loss $\sum_i \log_2 L_i$3, and $\sum_i \log_2 L_i$4 dB, compared with FSQ at $\sum_i \log_2 L_i$5, perceptual loss $\sum_i \log_2 L_i$6, and $\sum_i \log_2 L_i$7 dB (Zhu, 20 Aug 2025).

Method	$\sum_i \log_2 L_i$8 / Perceptual	PSNR (dB)
FSQ	0.143 / 0.182	20.3
LFQ	0.241 / 0.361	16.0
VQ-EMA	0.355 / 0.489	12.7
RFSQ-4×1024-LayerNorm	0.102 / 0.100	22.9
RFSQ-4×1024-Scale	0.103 / 0.101	22.9
RFSQ-4×1024-None	0.113 / 0.121	22.2

Relative to FSQ, RFSQ-4$\sum_i \log_2 L_i$91024-LayerNorm yields a $\mathbf{z}\in\mathbb{R}^d$0 reduction in $\mathbf{z}\in\mathbb{R}^d$1 error and a $\mathbf{z}\in\mathbb{R}^d$2 improvement in perceptual loss. The ablation results further report that LayerNorm adds approximately $\mathbf{z}\in\mathbb{R}^d$3 relative $\mathbf{z}\in\mathbb{R}^d$4 gain, $\mathbf{z}\in\mathbb{R}^d$5 perceptual gain, and $\mathbf{z}\in\mathbb{R}^d$6 dB PSNR over vanilla RFSQ, while scaling adds approximately $\mathbf{z}\in\mathbb{R}^d$7 $\mathbf{z}\in\mathbb{R}^d$8 gain and $\mathbf{z}\in\mathbb{R}^d$9 perceptual gain. Even the “None” variant, which uses pure residual FSQ without explicit conditioning, outperforms single-stage FSQ, supporting the claim that hierarchical decomposition is useful even before the additional conditioning mechanisms are introduced (Zhu, 20 Aug 2025).

The paper also reports faster convergence, more stable training curves, and that four-stage architectures uniformly outperform two-stage ones, with $\{\mathbf{r}_k\}_{k=0}^K$0–$\{\mathbf{r}_k\}_{k=0}^K$1 extra $\{\mathbf{r}_k\}_{k=0}^K$2 reduction. These observations place RFSQ within a broader line of work that treats residual quantization not merely as a rate-allocation device but as a hierarchical approximation scheme whose later stages require active conditioning to remain useful.

4. Block-Diagonal RFSQ in AffectCodec

AffectCodec adapts residual FSQ to an emotion-preserving neural speech codec by imposing structural separation between emotion and acoustic subspaces. Let $\{\mathbf{r}_k\}_{k=0}^K$3 be the concatenated codec latent and set $\{\mathbf{r}_k\}_{k=0}^K$4. BD-RFSQ runs $\{\mathbf{r}_k\}_{k=0}^K$5 stages, and at stage $\{\mathbf{r}_k\}_{k=0}^K$6 it applies a block-diagonal input projection, affine normalization and scalar quantization, inverse-affine transformation, a block-diagonal back-projection, and residual subtraction:

$\{\mathbf{r}_k\}_{k=0}^K$7

with

$\{\mathbf{r}_k\}_{k=0}^K$8

followed by

$\{\mathbf{r}_k\}_{k=0}^K$9

$\mathbf{r}_0=\mathbf{z}$0

$\mathbf{r}_0=\mathbf{z}$1

and the final quantized latent is

$\mathbf{r}_0=\mathbf{z}$2

At inference, one may truncate $\mathbf{r}_0=\mathbf{z}$3 stages to operate at lower bitrate (Meng et al., 22 May 2026).

The defining architectural constraint is the block-diagonal structure of the projections. Emotion and acoustic latents are first projected into partitions $\mathbf{r}_0=\mathbf{z}$4 and $\mathbf{r}_0=\mathbf{z}$5 with $\mathbf{r}_0=\mathbf{z}$6. Each $\mathbf{r}_0=\mathbf{z}$7 and $\mathbf{r}_0=\mathbf{z}$8 is implemented as a $\mathbf{r}_0=\mathbf{z}$9 convolution with weight matrix

$\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$0

so that emotion dimensions only read and write emotion channels and acoustic dimensions only read and write acoustic channels. The paper states that this enforces the invariant in Proposition B.1: no cross-stream “leakage” can occur inside the quantizer (Meng et al., 22 May 2026).

A second defining property is explicit bit allocation. Each stage uses $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$1 scalar FSQ dimensions with level counts

$\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$2

The per-stage codebook size is therefore

$\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$3

and each stage emits a $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$4-bit index, decomposed as $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$5 bits for emotion and $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$6 bits for acoustics. At a frame rate of $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$7 Hz, $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$8 active stages yield bitrate $\hat{\mathbf{z}}=\sum_{k=1}^K \mathbf{q}_k.$9 bps, so $\alpha_k$0 corresponds to $\alpha_k$1 kbps. The paper emphasizes that this reserves a fixed $\alpha_k$2 of bits for emotion at every stage, in contrast to standard RVQ or FSQ codecs where bit allocation emerges implicitly from the loss weighting (Meng et al., 22 May 2026).

5. Training objectives, rate control, and affect preservation

In AffectCodec, BD-RFSQ is embedded in an end-to-end pipeline that first computes acoustic features $\alpha_k$3 and frozen emotion features $\alpha_k$4, then applies Coarse-granularity Emotion Modulation (CEM), projects the acoustic and emotion branches into $\alpha_k$5 and $\alpha_k$6, concatenates $\alpha_k$7, runs the $\alpha_k$8-stage BD-RFSQ forward pass, emits the token sequence $\alpha_k$9, and decodes both the acoustic reconstruction $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$0 and the emotion slice $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$1 (Meng et al., 22 May 2026).

The CEM module uses a global emotion embedding $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$2 and FiLM on the acoustic pathway:

$$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$3

with $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$4 and $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$5. The paper describes this as multi-granularity conditioning that augments frame-level emotion signals in the dual-path front end (Meng et al., 22 May 2026).

The overall optimization objective is

$$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$6

Here $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$7 consists of multi-scale mel $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$8, time-domain $$(\mathbf{q}_k',\_)=\mathrm{FSQ}_k(\alpha_k\,\mathbf{r}_{k-1}),\quad \mathbf{q}_k=\frac{\mathbf{q}_k'}{\alpha_k},\quad \mathbf{r}_k=\mathbf{r}_{k-1}-\mathbf{q}_k.$$9, and multi-scale STFT adversarial plus feature-matching losses; $z_i$00 is the commitment loss $z_i$01; $z_i$02 is an $z_i$03 loss between the reserved emotion partition decoded back into emotion2vec space and the frozen teacher features; $z_i$04 is a cosine distance between emotion2vec embeddings of the original and reconstructed waveform; and the multi-rate loss is

$z_i$05

This loss supervises intermediate quantization depths so as to ensure emotion and acoustic quality at low $z_i$06 (Meng et al., 22 May 2026).

The empirical motivation for this design is low-bitrate emotion preservation. On IEMOCAP at $z_i$07 kbps, Emotion Degradation Rate (EDR) improves from $z_i$08 for DAC (RVQ) and $z_i$09 for X-Codec to $z_i$10 for BD-RFSQ. Across IEMOCAP, CREMA-D, and ESD at $z_i$11 kbps, BD-RFSQ achieves the lowest Macro-EDR in $z_i$12 conditions and the second-best result in the remaining two conditions. The paper also reports that valence/arousal/dominance MSE corroborates the categorical EDR drops (Meng et al., 22 May 2026).

The ablation results isolate the contributions of the main components at $z_i$13 kbps on IEMOCAP. Replacing BD-RFSQ with RVQ changes MEDR from $z_i$14 to $z_i$15; removing multi-rate training yields MEDR $z_i$16; and removing CEM yields MEDR $z_i$17. The paper interprets these results as showing that block-diagonal quantization, multi-rate supervision, and multi-granularity emotion conditioning each contribute to robustness at low bitrates (Meng et al., 22 May 2026).

6. Terminological scope and relation to other robustness claims

The term “robust” in RFSQ is not uniform across the surrounding literature. In the original neural compression paper, robustness refers to overcoming residual magnitude decay in multi-stage FSQ through conditioning mechanisms that keep later residual stages effective (Zhu, 20 Aug 2025). In AffectCodec, the structurally constrained BD-RFSQ variant is robust in the sense of preserving emotion-relevant information under low-bitrate compression by guaranteeing stream-aware partitioning and preventing cross-stream leakage (Meng et al., 22 May 2026).

A distinct line of work on FSQ-based neural audio codecs studies transmission robustness. In NeuCodec, FSQ is described as naturally redundant and locally smooth because adjacent quantization levels differ by only $z_i$18 in embedding space, and the paper reports that FSQ codecs degrade gracefully under a binary-symmetric channel while RVQ codecs collapse for $z_i$19; specifically, NeuCodec’s STOI remains $z_i$20 up to $z_i$21, and intelligibility persists even at $z_i$22 (Julia et al., 11 Sep 2025). That paper also includes a final section titled “Toward a Robust-Residual FSQ (RFSQ) framework,” but it explicitly presents residual FSQ quantization, Gray-coded indices, a channel-aware training loss, parity or ECC bits, and an “error-corrector” network as possible modifications and loss terms, with preliminary experiments left for future work rather than as an implemented benchmarked method (Julia et al., 11 Sep 2025).

This suggests that RFSQ should be treated as a family of residual-FSQ designs rather than as a single canonical architecture. One instantiation is the generic conditioning framework for neural compression; another is the block-diagonal, attribute-aware realization in AffectCodec; and a third is an outlined but not yet established direction toward channel-aware residual FSQ. A common misconception is therefore to read all uses of “RFSQ” as interchangeable. The published record instead distinguishes between residual-conditioning robustness, structurally protected attribute preservation, and prospective noisy-channel hardening.

7. Significance and open directions

Within neural compression, RFSQ occupies a specific position between single-stage FSQ and more elaborate codebook-based residual quantizers. The generic framework preserves FSQ’s advantages—no learnable codebook, stable training, fewer hyperparameters, and lookup-free quantization—while making residual refinement practical through lightweight conditioning (Zhu, 20 Aug 2025). In speech coding, BD-RFSQ extends that logic by turning bit allocation from an implicit optimization outcome into an explicit architectural constraint, while maintaining a flat token interface that is compatible with downstream speech LLMs (Meng et al., 22 May 2026).

The two most developed formulations also point to complementary design principles. The first is residual conditioning: later quantization stages require normalization or scaling if they are to operate on non-degenerate signals. The second is structural partitioning: when a codec latent carries multiple attribute types, quantization can be organized so that specific dimensions are reserved for specific information streams. AffectCodec’s results suggest that structurally protected quantization is an effective principle for preserving emotion-relevant information and may provide a general route toward attribute-aware neural speech compression (Meng et al., 22 May 2026).

The open directions already articulated in the literature are narrowly specified. One is the extension of residual FSQ toward explicit noisy-channel training, including Gray coding, parity or ECC bits, and a reconstruction loss under random bit flips (Julia et al., 11 Sep 2025). Another is broader deployment of block-diagonal residual quantizers in settings where a latent has multiple semantically distinct partitions, such as emotion and acoustics. A plausible implication is that future RFSQ research will continue to separate the questions of how to stabilize residual scalar quantization, how to allocate bits across structured subspaces, and how to harden discrete representations against transmission errors, even when those questions are discussed under a shared acronym.