Low-Rank and Linear RLVR Dynamics

• The paper demonstrates that low-rank RLVR update dynamics reveal a dominant rank-1 structure aligning with geometric and trust-region constraints, essential for stable deep RL.
• Methodologies like GeoRA leverage geometry-aware spectral masking to confine updates to compressible, off-principal subspaces, thereby mitigating instability.
• Empirical results show that these low-rank strategies yield significant RL step reductions and maintain sufficient expressivity, optimizing performance across language and control tasks.

Low-rank and linear RLVR (Reinforcement Learning with Verifiable Rewards) update dynamics characterize the geometry, expressivity, and stability properties of parameter-efficient adaptation strategies in large-scale deep RL for language and control. These settings demand mechanisms that both exploit the strong empirical compressibility of RLVR updates and preserve alignment with the geometric structure and trust-region constraints critical for reliable optimization.

1. Fundamental Structure of RLVR Update Dynamics

RLVR induces update dynamics that are both highly compressible and geometrically structured. The gradient of the reward objective for a linear parameter block $$W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$$ takes the form $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$. By the chain rule, this decomposes at the sample level into rank-1 increments $\delta x^\top$, so the mini-batch policy gradient is a sum of rank-1 matrices weighted by advantages. In practice, the top singular vector ($u_1 v_1^\top$) of the observed update dominates, with heavy-tailed power-law decay in the remainder of the spectrum (Ye et al., 7 May 2026). The left singular vector $u_1$ (output space) typically aligns with the reinforced reasoning direction, whereas the input-side $v_1$ is less constrained, as quantified by principal angle metrics.

These properties are observable across linear layers in LLMs, multi-task RL, and system control settings, enforcing a strong inductive bias toward power-law, low-rank update subspaces (Ye et al., 7 May 2026, Bai et al., 3 Mar 2025, 2011.01568, Zhang et al., 14 Jan 2026). Most of the reasoning improvement under RLVR is captured by the principal rank-1 update component, while sub-leading singular modes represent task-specific or "overfit" noise that does not transfer out-of-domain (Ye et al., 7 May 2026).

2. Geometry-Induced Failure Modes in Low-Rank Adaptation

Standard parameter-efficient fine-tuning (PEFT) methods such as LoRA, PiSSA, and MiLoRA, originally developed for supervised fine-tuning (SFT), fail to align with the RLVR-specific update geometry and constraints (Zhang et al., 14 Jan 2026, Yin et al., 29 Dec 2025). SVD-informed methods that initialize adapters in the top-$r$ principal directions of pretrained weights impose large updates along high-curvature subspaces. This violates the RLVR trust-region (typically a KL-divergence constraint), leading to spectral collapse and optimization instability. For example, PiSSA-adapted models exhibit KL spikes and lose all reward progress under RLVR (Zhang et al., 14 Jan 2026). Conversely, methods focusing on minor singular directions (e.g., MiLoRA) yield adapters dominated by stochastic drift back to the principal axis, failing to exploit the RL signal's off-principal structure.

The underlying mechanism is that RLVR gradients inhabit low-curvature, off-principal manifolds; forcing updates into principal directions directly contradicts the trust-region requirements of RL optimization. Empirically, SVD-informed adapters revert to principal component updates and quickly collapse (Yin et al., 29 Dec 2025).

3. Geometry-Aware Low-Rank Adaptation Mechanisms

To address these instabilities, geometric-aware schemes restrict adaptation to the true compressible RL update subspace while explicitly preserving pretrained geometry. The GeoRA (Geometry-Aware Low-Rank Adaptation) method parameterizes the update as follows (Zhang et al., 14 Jan 2026):

• A geometry-constrained mask $M_\mathrm{Geo}$ is defined over the weight $W$ using spectral and Euclidean quantiles.
• The masked weights $W_\mathrm{Geo}=W\odot M_\mathrm{Geo}$ are decomposed by SVD: $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$0.
• The adaptation is parameterized as $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$1 with $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$2 and $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$3.
• The updated weight is $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$4, with the residual leash $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$5 frozen throughout training.

This architecture localizes adaptation in the true RL-relevant subspace, maintains spectral proximity ($$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$6), and enforces bounded KL drift by anchoring the orthogonal complement—a critical requirement for reinforcement learning stability. All updates are performed in dense, low-rank form, achieving high hardware efficiency and avoiding bottlenecks of unstructured sparsity (Zhang et al., 14 Jan 2026).

4. Empirical and Theoretical Implications for Expressivity and Stability

The low-rank constraint must not reduce expressivity below the "reasoning capacity floor" demanded by the task. Empirical ablations consistently find that LoRA, DoRA, and AdaLoRA with adapter ranks $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$7–$$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$8 yield stable convergence and high final rewards, while Rank-1, VeRA, and other extreme reductions exhibit severe bottlenecks (Yin et al., 29 Dec 2025, Ye et al., 7 May 2026). Spectrally, the optimal RLVR update exhibits a heavy-tailed singular value spectrum—full fine-tuning distributes energy broadly, while SVD-based adapters collapse it into a spike (Yin et al., 29 Dec 2025). Structural PEFT variants that decouple magnitude and direction (DoRA) or shard low-rank factors (MiSS) further enhance alignment with RLVR geometry.

From a stability perspective, the constrained, geometry-aware form allows for explicit spectral shift and KL bounds. For suitable choices of scaling $$\nabla_W J = \mathbb{E}[\,\hat{A}\cdot \nabla_W \log \pi(o|q,o_{<t})\,]$$9, learning rate $\delta x^\top$0, and rank $\delta x^\top$1, the spectral shift and the induced KL divergence can be made arbitrarily small, preserving trust-region adherence (Zhang et al., 14 Jan 2026).

5. Trajectory Modeling: Linearity, Nonlinearity, and Acceleration

Empirical studies on LLM RLVR demonstrate that the parameter-delta trajectories are extremely low-rank, with the rank-1 subspace overwhelmingly capturing the downstream performance improvement (Wei et al., 20 May 2026, Chen et al., 13 Apr 2026). In the RELEX framework, per-tensor SVD reveals that the top singular direction of the delta trajectory grows almost perfectly linearly with training steps ($\delta x^\top$2), allowing for highly accurate linear regression-based extrapolation of future model checkpoints. Extrapolating on this one-dimensional subspace enables recovery of state-of-the-art RLVR performance across in-domain and out-of-domain tasks with only 15–20% of RLVR step cost (Table 1) (Wei et al., 20 May 2026).

However, further analysis reveals this linearity is not universal: in some tasks and layers, the evolution of the dominant subspace exhibits significant nonlinearity, especially under LoRA training. The NExt (Nonlinear Extrapolation) framework addresses this by learning a neural mapping from the extracted sequence of low-rank SVD components to future update increments. This approach achieves robust performance with up to 37.5% reduction in RLVR steps, indicating that while the rank-1 geometry is dominant, its trajectory frequently requires nonlinear modeling for accurate extrapolation in later or more complex RLVR stages (Chen et al., 13 Apr 2026).

Framework	Extrapolation Model	Empirical Gain (Acc/Cost)
RELEX	Linear regression	15–20% RLVR steps; matched SOTA
NExt	Learned nonlinear	37.5% step reduction; robust OOD

6. Linear RLVR Update Variants and Classical Control Connections

Linear (full-rank) RLVR variants serve as ablations or baselines: parameterizing the update as $\delta x^\top$3 with a dense matrix $\delta x^\top$4 recovers the family of full fine-tuning approaches. These offer no low-rank constraint and match the instability profiles of unconstrained optimization under aggressive learning rates (Zhang et al., 14 Jan 2026). In multi-task and control-theoretic RL contexts, explicit use of low-rank projections (e.g., truncated SVD in multi-task TD learning (Bai et al., 3 Mar 2025), low-rank transition models in LQR (2011.01568)) accelerates convergence within the true signal subspace and yields regret/sample complexity bounds proportional to rank, not ambient dimension.

From the perspective of dynamical systems, the progression of low-rank and linear RLVR models can be formalized using matrix ODEs confined to low-rank manifolds; variants such as predictor-corrector DLRA maintain adaptivity and efficiently handle transients and nonstationarities by dynamically projecting and correcting along the evolving low-rank trajectory (Hauck et al., 2022).

7. Broader Implications and Best Practices

Low-rank and linear dynamics are pervasive in RLVR, large-scale LLM reasoning, multi-task RL, and system identification. Key guidance derived from the current state of the literature includes:

• Use geometry-aware low-rank parameterizations to align with RLVR's off-principal, compressible update subspaces (Zhang et al., 14 Jan 2026).
• Avoid SVD-based initializations in the principal singular directions, as they induce spectral collapse and instability (Yin et al., 29 Dec 2025).
• Enforce sufficient adapter rank to meet the task's expressivity threshold, while leveraging dense low-rank updates for hardware efficiency (Yin et al., 29 Dec 2025, Zhang et al., 14 Jan 2026).
• When possible, exploit the near-linear rank-1 evolution for extrapolation and acceleration, but employ nonlinear trajectory models (such as NExt) to avoid drift in non-stationary regimes (Wei et al., 20 May 2026, Chen et al., 13 Apr 2026).
• Periodically project onto the dominant singular subspace and regularize the spectral tail to prevent reward overfitting and encourage robust generalization (Ye et al., 7 May 2026).

This geometry-centric, low-rank approach unifies the optimization and architectural strategies now prevailing in RLVR and related reinforcement learning paradigms for large-scale reasoning, control, and continuous adaptation.