SOAR: Scale Optimization for Accurate Reconstruction

• The paper introduces a reconstruction-error minimization framework that optimizes scale selection for NVFP4 quantization of large language models.
• It employs Closed-form Joint Scale Optimization (CJSO) and Decoupled Scale Search (DSS) to improve reconstruction fidelity compared to max-based heuristics.
• SOAR maintains standard NVFP4 hardware compatibility and memory layout while integrating with complementary quantization techniques for enhanced performance.

Searching arXiv for the primary SOAR paper and a few related NVFP4/PTQ baselines for grounding. Search query: SOAR Scale Optimization for Accurate Reconstruction NVFP4 Quantization Scale Optimization for Accurate Reconstruction (SOAR) is a post-training quantization framework for LLMs that targets NVFP4, a 4-bit microscaling floating-point format with native hardware support on NVIDIA GPUs. It is designed to improve the reconstruction fidelity of FP16/FP32 weights under the NVFP4 storage and execution constraints by optimizing how scales are chosen and represented. In the formulation introduced in "SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization" (Bao et al., 12 May 2026), the method addresses two specific limitations of prior NVFP4 workflows: inflexible scale selection and the coupled use of a single FP8 block scale for both quantization and dequantization. Its two core components are Closed-form Joint Scale Optimization (CJSO), which derives analytical updates for global and block-wise scales under fixed FP4 codes, and Decoupled Scale Search (DSS), which separates the high-precision quantization scale from the hardware-constrained dequantization scale.

1. Definition and problem formulation

SOAR is specialized for NVFP4 quantization of LLM weights, where the objective is to reconstruct original full-precision tensors as accurately as possible while preserving the exact storage format and inference-time hardware path of NVFP4 (Bao et al., 12 May 2026). The method operates on pretrained FP16/FP32 weight tensors, partitioned into blocks of 16 elements, and is explicitly calibration-free in its core form: scale optimization is performed directly on weights rather than through activation-aware objectives.

The underlying optimization target is a squared reconstruction loss over quantized weights. Let a tensor $W$ be partitioned into blocks $W_i$, with global scale $\alpha$, block-wise scales $\Delta_i$, FP4 codes $Q_i$, and reconstruction $\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$. SOAR minimizes

$$\mathcal{L}(\alpha, \{\Delta_i\}) = \sum_i \left\| W_i - Q_i \cdot (\alpha \Delta_i) \right\|_2^2,$$

with the complication that $Q_i$ depends discretely on the scales through FP4 quantization (Bao et al., 12 May 2026). The framework therefore alternates between code assignment and scale fitting.

The motivation for this formulation is specific to NVFP4. In the reference workflow, the scale-selection rules are heuristic and max-based, whereas the practical objective of interest is reconstruction error. SOAR redefines scale determination as an optimization problem rather than a fixed rule, and this shift is the central methodological contribution.

2. NVFP4 numerical structure and sources of error

NVFP4 is a hierarchical low-precision format in which weights are stored as FP4 (E2M1) values, scaled by a tensor-wise global scale $\alpha$ in FP32 and block-wise scales $\Delta_i$ in FP8 (E4M3), with one block scale per 16-element block (Bao et al., 12 May 2026). For a full-precision tensor $W_i$0, the reference quantization process uses a heuristic global scale,

$W_i$1

where $W_i$2 and $W_i$3, followed by heuristic block-wise scaling,

$W_i$4

and FP4 code assignment through a piecewise rule mapping $W_i$5 into the representable FP4 set in $W_i$6 (Bao et al., 12 May 2026). Dequantization then reconstructs

$W_i$7

The paper identifies three sources of difficulty in applying NVFP4 to LLMs. First, LLM weights exhibit heavy tails and irregular local structure, which makes max-based tensor-wise and block-wise scale selection poorly matched to the actual distribution of reconstruction error. Second, prior NVFP4 methods such as standard NVFP4, 4over6, MR-GPTQ, and RaZeR do not solve a continuous optimization problem for scales; instead, they rely on max-based formulas or coarse range choices (Bao et al., 12 May 2026). Third, the standard workflow uses the same FP8 block scale in two roles: to define the FP4 quantization grid and to define the dequantization multiplier. Because the block scale must lie on the discrete E4M3 grid, scale quantization error affects both the FP4 rounding boundaries and the reconstructed magnitude simultaneously.

That coupling is especially consequential because the FP8 constraint is only required for the stored dequantization scale used at inference. The offline computation that determines FP4 codes does not, in principle, require the same restriction. SOAR exploits this asymmetry directly.

3. Overall architecture of SOAR

SOAR is a layer-wise or tensor-wise PTQ procedure that leaves the final NVFP4 representation unchanged: FP4 weights, one FP32 global scale, and FP8 block-wise dequantization scales (Bao et al., 12 May 2026). The algorithm begins from the standard NVFP4 max-based initialization, then alternates between two stages.

The first stage is CJSO, which treats the global scale $W_i$8 and the block-wise scales $W_i$9 as optimization variables and updates them analytically under fixed FP4 assignments. The second stage is DSS, which refines each block by separately selecting a high-precision quantization scale $\alpha$0 and a stored FP8 dequantization scale $\alpha$1 through local discrete search. The quantization scales are not stored; only the FP8 dequantization scales are kept in the final model (Bao et al., 12 May 2026).

Per tensor, the pipeline is: initialize $\alpha$2 and $\alpha$3 using standard NVFP4 rules; iteratively apply CJSO and then DSS; stop after a maximum number of iterations or once the relative MSE improvement falls below $\alpha$4; and store the resulting FP4 weights $\alpha$5, FP32 global scale $\alpha$6, and FP8 dequantization scales $\alpha$7 (Bao et al., 12 May 2026). The paper reports up to 15 iterations, with early stopping.

A notable property of this design is that it changes only the scale selection procedure. Inference uses standard NVFP4 kernels and the same memory layout as standard NVFP4, so the method introduces no additional hardware overhead (Bao et al., 12 May 2026). Memory analysis in the paper is reported to confirm an identical footprint.

4. Closed-form Joint Scale Optimization

CJSO addresses the scale-selection problem by alternating between discrete code assignment and continuous least-squares fitting. With fixed FP4 codes $\alpha$8, the reconstruction loss is quadratic in the global scale and in each block-wise scale, so exact minimizers can be written in closed form (Bao et al., 12 May 2026).

For fixed $\alpha$9 and $\Delta_i$0, differentiation with respect to $\Delta_i$1 yields the global update

$\Delta_i$2

which is the least-squares scalar aligning $\Delta_i$3 to $\Delta_i$4 across the entire tensor (Bao et al., 12 May 2026).

For fixed $\Delta_i$5 and $\Delta_i$6, each block-wise scale has the local update

$\Delta_i$7

followed by projection to the nearest FP8 (E4M3) value,

$\Delta_i$8

optionally clipped to the FP8 range (Bao et al., 12 May 2026).

The iterative CJSO loop is initialized from the standard NVFP4 scales. At each iteration, the method computes FP4 codes,

$\Delta_i$9

updates $Q_i$0 using the global closed-form expression, updates each $Q_i$1 using the block-wise closed-form expression and projects to FP8, and then recomputes the FP4 codes (Bao et al., 12 May 2026). This alternating scheme is an approximation because $Q_i$2 is treated as fixed while solving for the scales, even though the codes depend on those same scales. The paper presents this as a coordinate-wise treatment of a mixed discrete-continuous problem.

The significance of CJSO lies in its exactness under fixed codes. Given $Q_i$3, the scale updates are not heuristic but optimal for the corresponding quadratic objective. This is the formal basis for the claim that SOAR replaces max-based scale heuristics with reconstruction-error minimization.

5. Decoupled Scale Search

DSS addresses the second limitation identified by the paper: the coupling of quantization and dequantization scales in conventional NVFP4 pipelines. In standard NVFP4, a single FP8 block scale $Q_i$4 is used both to compute the FP4 code,

$Q_i$5

and to reconstruct the block,

$Q_i$6

so the discretization error of the E4M3 scale perturbs both the code boundaries and the reconstruction multiplier (Bao et al., 12 May 2026).

SOAR instead introduces two block-wise scales. The quantization scale $Q_i$7 is high precision, used only offline to compute FP4 codes, and never stored. The dequantization scale $Q_i$8 is the hardware-constrained value stored for inference. The block-wise optimization becomes

$Q_i$9

This decoupling permits good FP4 code selection without forcing the code boundaries to use the same discretized scale that must later be stored (Bao et al., 12 May 2026).

DSS is executed after each CJSO update. For each block, it constructs a dequantization candidate set from the two nearest E4M3 neighbors of the current $\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$0, and a quantization candidate set

$\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$1

a 101-point multiplicative grid (Bao et al., 12 May 2026). For each pair $\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$2 in the Cartesian product of these sets, DSS computes the block-wise reconstruction MSE,

$\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$3

and selects the minimizing pair (Bao et al., 12 May 2026).

The practical consequence is that FP8 scale quantization no longer constrains FP4 code selection. The dequantization scale remains hardware-compatible, while the quantization scale acts as an unconstrained helper variable. The paper further reports that DSS improves MXFP4 as well, which suggests that the decoupling principle is not specific to NVFP4 but to microscaling formats more generally (Bao et al., 12 May 2026).

6. Empirical behavior, constraints, and scope

The experimental evaluation covers LLaMA-3.1-8B-Instruct, LLaMA-3.2-1B and 3B-Instruct, and Qwen3-4B and 8B, under W4A4 and W4A16 regimes, with benchmarks including WinoGrande, PIQA, HellaSwag, ARC-Easy, ARC-Challenge, MMLU, and GSM8K (Bao et al., 12 May 2026). Activations are quantized with standard NVFP4 in the reported experiments, except that RaZeR follows its own activation quantization.

The reported results show consistent gains over standard NVFP4 and other baselines at the same memory footprint. On Qwen3-8B under NVFP4 W4A4, the average score is 68.75 for the NVFP4 baseline, 69.15 for 4over6, 70.12 for RaZeR, and 70.68 for SOAR, compared with 71.53 in FP16 (Bao et al., 12 May 2026). On LLaMA-3.2-3B-Instruct, the corresponding averages are 65.02 for NVFP4, 65.19 for RaZeR, and 66.00 for SOAR, with FP16 at 66.83. On reasoning benchmarks, Qwen3-8B reaches 77.02 with SOAR versus 75.00 for NVFP4 and 76.86 for RaZeR, while LLaMA-3.2-3B-Instruct reaches 66.43 with SOAR versus 65.13 for NVFP4 and 66.11 for RaZeR (Bao et al., 12 May 2026).

Ablation results indicate that both major components contribute independently. On LLaMA-3.2-3B-Instruct under NVFP4 W4A4, the baseline average is 65.02, CJSO only gives 65.64, DSS only gives 65.50, and the combined SOAR method gives 66.00 (Bao et al., 12 May 2026). The iteration study on Qwen3-8B reports 70.39 at one iteration, 70.55 at five iterations, and 70.68 at 15 iterations with early stopping, indicating diminishing but persistent gains across additional iterations.

SOAR is also reported to integrate with GPTQ. For LLaMA-3.1-8B-Instruct, GPTQ yields an average of 72.95 and SOAR+GPTQ yields 73.18 with essentially identical perplexity (Bao et al., 12 May 2026). This places SOAR as a scale-optimization layer that can complement existing reconstruction-aware quantization schemes rather than replace them outright.

The computational cost is described as moderate: approximately 7 minutes for LLaMA-3.2-1B, 17 minutes for LLaMA-3.2-3B, and 36 minutes for LLaMA-3.1-8B and Qwen3-8B on A800-80GB (Bao et al., 12 May 2026). Complexity is linear in the number of weights, with a small per-block DSS search and a limited number of global iterations. Because the final representation remains standard NVFP4, inference incurs no additional runtime cost.

The framework is nonetheless constrained by its objective. It is designed around weight reconstruction, not activation-aware end-to-end error, and the paper explicitly notes as future work an extension using $\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$4 with calibration inputs $\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$5 (Bao et al., 12 May 2026). This suggests that SOAR is most directly applicable where weight quantization error is the dominant bottleneck and where calibration-free deployment is operationally desirable.

7. Interpretation within low-bit microscaling quantization

Within the NVFP4 setting, SOAR formalizes two methodological claims. First, when FP4 codes are fixed, scale fitting should be treated as a regression problem with exact minimizers rather than as a max-based heuristic. Second, the stored scale and the code-selection scale need not be the same variable, because only the former is constrained by hardware representation (Bao et al., 12 May 2026). These claims jointly explain why the method can improve reconstruction MSE and downstream accuracy without altering the inference kernel or memory layout.

A common misconception would be to interpret SOAR as changing the NVFP4 format itself. The paper does not do so. The stored representation remains FP4 weights with FP8 block scales and a global FP32 scale; only the offline procedure that determines those values is modified (Bao et al., 12 May 2026). Another possible misunderstanding is to regard DSS as adding an inference-time parameterization. It does not: the quantization scale $\hat{W}_i = Q_i \cdot (\alpha \Delta_i)$6 exists only during offline quantization and is discarded afterward.

The broader significance of SOAR lies in its treatment of scale quantization as an optimization bottleneck rather than an unavoidable implementation detail. The reported transfer of DSS to MXFP4 supports the view that constrained stored scales and unconstrained helper scales can be separated in other microscaling formats as well (Bao et al., 12 May 2026). A plausible implication is that future low-bit PTQ systems may increasingly distinguish offline quantizer design variables from the numerically constrained values that must survive into deployment.