ISO: An RLVR-Native Optimization Stack

Abstract: Reinforcement learning with verifiable rewards (RLVR) is rapidly advancing the reasoning capabilities of LLMs, yet the optimization layer that converts reward feedback into weight-space updates remains poorly understood. Building on our prior analysis (Zhu et al., 2025), we study this missing layer through the singular structure of model weights and identify spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior through changes in the associated input and output singular frames. We operationalize spectral inheritance as Isospectral Optimization (ISO), an RLVR-native, fixed-spectrum optimization framework with complementary offline and online instantiations. Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model, requiring no post-merge data, rollouts, gradient updates, or on-policy distillation (OPD). It recovers complementary specialist capabilities and achieves the strongest aggregate performance among the compared data-free merging methods. Online, ISO-Optimizer applies a chosen base optimizer, including AdamW and Muon, to the frame variables while keeping the base spectra fixed. Across reasoning and coding tasks ranging from 1.5B to 8B parameters, ISO-Optimizer improves accuracy in the reported runs and reaches matched scores with substantially fewer training steps. On Qwen3-8B-Base, AdamW reaches an aggregate accuracy of 0.495 after 270 training steps. ISO-AdamW reaches the same accuracy after only 100 training steps and improves further to 0.509 after 210 training steps. Together, ISO offers a concrete answer to RLVR's missing optimization layer: rather than inheriting pre-training optimization wholesale, design post-training around the structure of reward-driven adaptation: inherit the spectrum, optimize the frames.

• The paper introduces ISO, a framework that preserves a model’s singular-value spectrum while adapting both singular frames, with experiments showing RLVR changes behavior primarily through frame updates rather than spectrum changes.
• ISO-Merger combines shared-base RL experts without data, rollouts, or retraining, reaching aggregate scores of 63.80 versus 62.88 for the strongest baseline on Qwen2.5-7B-Instruct and improving worst-case performance.
• ISO-Optimizer trains fixed-spectrum models through polar-retracted frame updates, achieving up to 2.7× fewer training steps and higher final scores than AdamW across math reasoning and coding tasks from 1.5B to 8B models.

Motivation and scope

Reinforcement learning with verifiable rewards (RLVR) has become a central scaling axis for reasoning models, yet the optimization layer that converts reward feedback into weight-space updates is largely inherited from pre-training. This paper, building on the authors' prior analysis of RLVR optimization dynamics (Zhu et al., 11 Nov 2025), studies that layer through the singular-value structure of weight matrices and arrives at a design principle the authors call spectral inheritance: RLVR can reuse the base model's weight spectra while acquiring new behavior through changes in the associated input and output singular frames. The paper operationalizes this principle as Isospectral Optimization (ISO), an RLVR-native stack with two instantiations: ISO-Merger for checkpoint-only composition of shared-base RL experts, and ISO-Optimizer for online fixed-spectrum training.

Spectra stay: near-isospectrality and its calibration

For a base matrix $W_0 = U_0\Sigma_0V_0^\top$, the paper defines the fixed-spectrum family $\mathcal{F}(W_0)$ of all matrices sharing $\Sigma_0$ and proves (via von Neumann's trace inequality) that the Frobenius distance to this family equals $\|\sigma(W)-\sigma(W_0)\|_2$, with $U\Sigma_0V^\top$ a nearest representative. Empirically, across unconstrained RLVR runs—including a long-horizon endpoint on DeepSeek-R1-Distill-Qwen-1.5B trained over 3,000 updates—the learned checkpoints remain close to their base fixed-spectrum families: layer-wise spectral distance $\delta_\Sigma$ is on the order of $10^{-2}\%$ relative to base-weight scale, and the mean relative spectral residual $\rho_\Sigma$ is approximately 3% of the total checkpoint displacement, versus roughly 35% for SFT. Trajectory-level measurements on Qwen3-8B-Base confirm stability persists during training, not only at endpoints.

The authors are careful not to overclaim: because only $q = \min(d_{\text{out}}, d_{\text{in}})$ first-order coordinates change singular values, high dimensionality alone predicts small spectral drift. A dimension-aware calibration statistic $\kappa_{\text{spec}}$—normalized so that isotropic motion yields expectation one—shows RLVR values between roughly 1.02 and 1.35, indicating no strong intrinsic preference for spectrum-preserving updates beyond dimensional effects. The contrast with SFT remains sharp, however: SFT yields $\mathcal{F}(W_0)$0 values between 89 and 1364, two to three orders of magnitude above reference. A same-base case study using two public 14B checkpoints derived from Qwen2.5-14B reproduces this contrast while controlling for backbone differences.

Functional evidence: spectral inheritance

The decisive question is whether the small spectral changes are functionally necessary. Two interventions answer negatively. First, restoring the base spectrum after unconstrained RLVR while retaining the learned frames preserves most acquired performance; conversely, substituting the RL spectrum into SFT frames does not transfer behavior. Second, keeping $\mathcal{F}(W_0)$1 fixed throughout training and optimizing only the frames still supports strong gains—in fact outperforming AdamW in the studied run—whereas a spectrum-only control (frozen frames, trainable diagonal spectrum) achieves little, though its failure reflects restricted capacity ($\mathcal{F}(W_0)$2 degrees of freedom per matrix) rather than a general impossibility claim.

A further structural analysis asks which variables must remain adaptable. Using rank-$\mathcal{F}(W_0)$3 truncations ($\mathcal{F}(W_0)$4), the paper computes unexplained-update ratios $\mathcal{F}(W_0)$5 for four reconstruction classes: remixing within both incoming subspaces, retaining one incoming subspace, or fixing the incoming spectrum while adapting both frames. For the cumulative transition in a two-stage vision-language-action pipeline, subspace remixing leaves a median 87% of the update unexplained, one-sided restrictions leave 45% and 42%, whereas the fixed-spectrum/two-frame class leaves only 1.8%. The same ordering recurs at the objective-shift transition $\mathcal{F}(W_0)$6 where the incoming checkpoint is already RL-trained, and across truncation ranks down to $\mathcal{F}(W_0)$7. The conclusion is that among tested restrictions, the spectrum can be fixed but both frames must remain adaptable—directly motivating the parameterization $\mathcal{F}(W_0)$8.

ISO-Merger: data-free expert composition

ISO-Merger composes $\mathcal{F}(W_0)$9 shared-base RL specialists by representing each expert as $\Sigma_0$0, projecting frame displacements onto Stiefel tangent spaces at the shared base, masking trailing modes (keep ratio 0.9), solving a ridge-stabilized Gram system targeting unit self-retention of each expert's first-order effect proxy, and retracting via polar decomposition. It requires no post-merge data, rollouts, gradient updates, or on-policy distillation. On Qwen2.5-7B-Instruct merging three experts (coding, tool use, long-context memory), it reaches an aggregate average of 63.80 versus 62.88 for the strongest training-free baseline (RAM); on DeepSeek-R1-Distill-Qwen-1.5B merging coding and math experts, it reaches 44.38 versus 43.52 (TIES). Distributional metrics show it essentially matches best@4 baselines while improving worst@4 by 1.62 and 1.36 points, suggesting more consistent capability recovery. The evaluation is limited to shared-base specialists; merging broadly trained generalists is explicitly out of scope.

ISO-Optimizer: online fixed-spectrum RLVR

ISO-Optimizer applies a chosen base optimizer directly to the frame variables $\Sigma_0$1 with factor gradients $\Sigma_0$2 and $\Sigma_0$3, followed by polar retraction. The construction provably preserves the spectrum to first order along feasible frame motion and exactly after retraction. Importantly, the method is not projection of weight-space updates onto $\Sigma_0$4: optimizer states live in frame coordinates, and the feasible model class is strictly smaller than unconstrained weight space, so gains cannot be attributed to added capacity.

Results span math reasoning (Qwen3-1.7B/4B/8B-Base on DeepMath-103K) and competitive coding (DS-1.5B on ArcherCodeR). On Qwen3-4B-Base, ISO-AdamW reaches 43.46 aggregate versus 41.69 for tuned AdamW and 42.23 for Muon, matching the strongest AdamW run's final accuracy with roughly $\Sigma_0$5 fewer steps. On Qwen3-8B-Base without further tuning, ISO-AdamW reaches 0.509 at step 210 while AdamW plateaus at 0.495 even after 60 extra steps; ISO-AdamW attains that level by step 100, a $\Sigma_0$6 reduction in training-step count. On coding, ISO-AdamW peaks at 0.268 within 220 steps, a level extended AdamW baselines do not reach within 330 steps. ISO-Muon transfers the same benefit (0.428 vs. 0.422 final). The polar retraction implemented in FP64 GPU SVD adds about 86 seconds per optimizer step but only ~7% of end-to-end RL step time, which is dominated by rollout generation—a cost profile amenable to overlap in asynchronous systems.

Limitations and open questions

The paper concedes several boundaries. Near-isospectrality is descriptive, not a proven optimization preference; the calibration shows no strong suppression beyond dimensionality, and ISO treats spectral inheritance as an inductive bias rather than an exact update law. The spectrum-only control's failure is capacity-confounded. Frame-adaptability conclusions are stated only "among the transformation classes tested," and reconstruction classes have different dimensions, so residual orderings are not complexity-normalized model selection. Merging results cover only shared-base specialists. At 8B, ISO produces longer responses requiring response-length cap adjustments, and neither method was tuned at that scale. Systematic comparison against alternative spectrum-preserving optimizers formulated directly in weight space (e.g., Pion (Shi et al., 12 May 2026), POET) is deferred, as are geometry-aware adapter initialization, training, and composition within the same framework.

Conclusion

The paper establishes spectral inheritance as a recurring, functionally validated structure in RLVR—base spectra remain reusable while behavior changes through both singular frames—and converts it into a coherent optimization stack. ISO-Merger delivers the strongest data-free composition of shared-base RL experts among compared methods, and ISO-Optimizer improves accuracy and convergence speed across model scales from 1.5B to 8B under a strictly constrained parameterization. The central message is that post-training can be designed around reward-driven adaptation structure itself: inherit the spectrum, optimize the frames.