QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

Abstract: Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) LLMs, motivating low-precision rollout acceleration such as FP8. As an emerging low-precision format, NVFP4 combines fine-grained scaling for accuracy preservation with native W4A4 FP4 GEMMs for higher throughput than FP8. However, we find that directly applying NVFP4 to MoE RL rollout is impractical. NVFP4 rollout with BF16 training collapses after roughly 150 steps, accompanied by rapidly growing rollout-trainer log-probability gaps. Through training-inference error analysis and controlled ablations, we identify activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid. Therefore, to stabilize NVFP4 RL for MoE, we propose QUantization-error Alignment across Dual Sides (QUADS). On the trainer side, we introduce Asymmetric Quantization-Aware Training fake-quantizing weights while keeping activations unquantized for better alignment. On the rollout side, Residual Activation Compensation corrects high-error activation channels while preserving native W4A4 GEMMs. In our MoE RL experiments on several benchmarks, QUADS achieves BF16-level accuracy, improves average pass@1 by 21.49 points over naive NVFP4 RL, and delivers ~16% higher rollout throughput than FP8.

• The paper introduces QUADS, an asymmetric alignment method combining trainer-side W4A16 quantization-aware training with rollout-side residual activation compensation to stabilize NVFP4 GRPO.
• QUADS recovers average benchmark accuracy to 72.86% versus 51.37% for naive NVFP4 RL and 73.15% for BF16, while improving throughput by roughly 16% over FP8.
• The results show activation quantization drives policy divergence more than weight quantization, making matched weight QDQ and targeted correction of high-error activation channels essential for reliable FP4 RL.

Overview

QUADS addresses a specific systems problem in reinforcement learning for LLMs: stabilizing GRPO training when rollout inference runs at NVFP4 precision on NVIDIA Blackwell hardware. The paper, from the Qwen team at Alibaba, demonstrates that directly combining native W4A4 FP4 rollout with BF16 training causes reward collapse after roughly 150 optimization steps, and proposes a dual-side alignment strategy that restores BF16-level accuracy while retaining most of the FP4 throughput advantage (2607.15810).

Motivation and failure diagnosis

Rollout generation dominates end-to-end RL wall-clock time—often exceeding 70% of each training step—making low-precision rollout acceleration attractive. FP8 rollout is an established baseline whose training–inference mismatch can be controlled via truncated importance sampling (TIS), adaptive clipping, or unified precision flows. NVFP4 offers roughly 2× higher GEMM throughput than FP8 by using an E2M1 core with only eight positive representable levels per sign (32× coarser than E4M3), but this coarseness makes it far less forgiving.

The paper's central empirical finding is that naive NVFP4 W4A4 rollout with BF16 training is not merely degraded but unstable: reward and held-out scores peak near step 150 and then collapse, while the per-token log-probability gap grows monotonically until importance ratios exit the trust region. A controlled ablation isolates the cause. Comparing W4A16 (FP4 weights, BF16 activations) against W16A4 (BF16 weights, FP4 activations) reveals a strong operand asymmetry: W16A4 closely tracks the failure mode of full W4A4, whereas W4A16 remains much closer to stable behavior. Activation quantization—not weight quantization—is the dominant driver of divergence.

The paper grounds this asymmetry analytically. Weights are synchronized bit-exactly between trainer and rollout engine, so applying a matched quantization–dequantization (QDQ) operator on both sides drives weight error to zero. Activations, however, are recomputed online through different kernel paths, so even before FP4 quantization the two engines see different tensors; applying activation QDQ to already-different activations does not guarantee—and empirically does not achieve—mismatch reduction.

Method

Asymmetric W4A16 QAT (trainer side). The learner fake-quantizes weights to NVFP4 using a QDQ operator matched to the rollout kernel, keeps activations in BF16, and uses a straight-through estimator for gradients. Because $$\mathbf{W}_{\mathrm{infer}} = \mathbf{W}_{\mathrm{train}}$$ and the quantizers match, the weight error term vanishes exactly, and the mismatch provably shrinks relative to no alignment. Notably, symmetric W4A4 QAT performs worse than asymmetric W4A16 QAT: coarse E2M1 activation fake-quantization amplifies engine drift rather than reducing it.

Residual activation compensation (rollout side). Trainer-side alignment alone avoids collapse but leaves a residual gap to BF16. The paper observes that FP4 activation quantization error is highly structured across hidden dimensions—a small set of high-magnitude channels forms persistent outlier bands. QUADS therefore applies a second-pass FP4 correction to the top-$k$\% channels ranked by residual norm $s_j = \|\Delta \mathbf{X}_q[:,j]\|_2$, computed online per forward pass since activation statistics are non-stationary under RL. Channels outside the selected set follow the standard W4A4 path, so the main GEMM remains on FP4 Tensor Cores. With $k=50$, the module adds roughly 10% FLOPs overhead, and fused Triton kernels keep launch overhead low.

Main results

The evaluation uses full-parameter GRPO on a 25B-A2.8B MoE model trained for 260 steps on Blackwell GPUs, with Megatron-LM as trainer and SGLang as rollout engine, evaluated on AIME 2024/25, HMMT 2025, and LiveCodeBench.

Configuration	LCB	HMMT 2025	AIME 2024	AIME 2025	Average
BF16 RL	63.05	66.04	83.54	80.00	73.15
FP8 RL	65.27	63.54	82.91	79.79	72.88
Naive NVFP4 RL	24.25	48.33	66.67	66.25	51.37
QUADS	64.79	62.92	83.33	80.42	72.86

Three results stand out. First, naive NVFP4 RL degrades average pass@1 by nearly 22 points relative to BF16, confirming that unmitigated W4A4 rollout is incompatible with stable policy-gradient training. Second, QUADS recovers accuracy to within the variance of the BF16 baseline across all four benchmarks (72.86% vs. 73.15%), a gain of 21.49 points over naive NVFP4. Third, throughput analysis shows plain NVFP4 delivers ~1.3× FP8 decode throughput; residual compensation costs only ~5–9%, leaving the full pipeline ~16% faster than FP8.

The log-probability gap metric corroborates these findings: naive NVFP4 sustains a maximum gap of ~1.3 throughout training, QUADS reduces it to ~0.86, and the BF16 baseline sits at ~0.74—an irreducible floor caused by kernel-level differences between Megatron-LM and SGLang that persists even without any quantization.

Ablations

The component ablation confirms independent contributions: naive NVFP4 averages 51.4%, W4A16 QAT alone recovers to 71.1%, and adding residual compensation reaches 72.9%. The channel-selection ablation shows monotonic improvement in the maximum log-probability gap from 0.923 at 12.5% selected channels to 0.759 at 100%, though larger ratios incur proportionally higher compute cost—the chosen $k=50$ balances accuracy recovery against throughput.

Limitations and open questions

Several caveats bear on the generality of the results. The evaluation covers a single MoE model size (25B-A2.8B), one RL algorithm (GRPO), and math/code reasoning tasks over 260 steps; whether the activation-dominated failure mode and its remedy transfer to dense models, other algorithms, or longer training horizons is not established. The residual channel ratio $k=50$ was profiled specifically for the W13 activation site, and the paper notes that optimal selection must balance accuracy against compute, leaving adaptive per-layer or per-step selection as an open question. The method also depends on exact weight synchronization and matched QDQ operators between engines—an assumption that may be fragile across heterogeneous serving stacks. Finally, the ~0.74 BF16 log-probability gap represents irreducible engine drift that no precision scheme can eliminate, bounding how closely any quantized pipeline can approach full-precision behavior.

Conclusion

QUADS reframes NVFP4 RL stability as a quantization-error alignment problem and resolves it asymmetrically: weight error is eliminated by matched QDQ paths on the trainer, while activation error—which cannot be aligned by symmetric fake quantization—is reduced on the rollout side through targeted second-pass compensation of high-residual channels. The result is stable full-parameter MoE GRPO with native W4A4 FP4 GEMMs at BF16-level accuracy and a ~16% rollout throughput advantage over FP8, establishing a practical path to FP4-tier RL acceleration on Blackwell-class hardware.