Gradient Reuse: Methods and Applications

• Gradient Reuse is a design pattern that repurposes previously computed gradients to improve optimization efficiency, reduce variance, and accelerate convergence.
• It encompasses diverse methods such as multi-update reuse in RL, retrieval-conditioned reuse in language models, and structural reuse in tree training.
• While offering efficiency gains, gradient reuse demands careful bias, variance, and dependency management to prevent performance degradation.

Gradient reuse denotes a family of procedures that extract additional utility from gradients, gradient surrogates, or gradient-bearing data beyond a single forward–backward consumption. In the recent literature, the phrase is used for multi-update reuse of rollout batches in reinforcement learning with verifiable rewards, storage and retrieval of document-specific gradients as parametric knowledge units, reuse of shared-prefix backward signals in tree-structured large-language-model training, control-variate reuse in finite-sum optimization, repeated reuse of data in multi-pass stochastic gradient descent, reuse of perturbation directions in online evolution strategies, and reuse of inner-loop adaptation gradients by outer-loop meta-learning [2605.19425][2606.15734][1511.01942][2304.12180][2212.11726].

1. Scope and principal forms

Taken together, the cited works suggest that gradient reuse is not a single algorithmic primitive but a recurrent design pattern: learning systems often benefit when previously computed gradient information, or data that induces gradients, is repurposed under carefully controlled statistical or structural conditions.

Mechanism	Reused object	Representative setting
Multi-update reuse	One rollout batch reused for multiple optimizer steps	RLVR with Dynamic Gradient Gating [2605.19425]
Retrieval-conditioned adaptation	Per-document gradients stored and reapplied	ReGrad and its Gradient Bank [2606.15734]
Off-policy reuse	Replay-buffer trajectories or transitions reweighted by IS/MLR	BIRIS, VRER, PGPE [2209.07074]
Control-variate reuse	Snapshot gradients or stale component gradients	SVRG [1511.01942]
Structural reuse	Shared-prefix activations and aggregated branch gradients	Tree Training [2511.00413]
Perturbation reuse	The same Gaussian perturbation reused across windows	NRES [2304.12180]
Meta-gradient reuse	Inner-loop controller gradients reused by an outer loop	Reusable options via meta-learning [2212.11726]
Data-pass reuse	Repeated passes over the same samples in SGD	Multi-pass SGD scaling laws [2506.08415]

The common technical issue is that reuse changes the dependency structure of optimization. In some settings, reuse lowers variance, increases sample efficiency, or amortizes expensive computation. In others, reuse creates bias, amplifies policy shift, or invalidates naive assumptions of independence. The modern literature therefore treats reuse not as a free gain, but as a mechanism that must be paired with either reweighting, gating, variance control, architectural accounting, or higher-order differentiation.

2. Sample reuse in reinforcement learning and policy optimization

In RLVR for reasoning LLMs, gradient reuse means reusing a single rollout batch for multiple optimizer steps before collecting a new batch. The motivating optimizer is Group Relative Policy Optimization, with clipped surrogate
$$
\mathcal{L}{\mathrm{GRPO}}(\theta) = \frac{1}{T} \sum{i=1}^{T} \min \left( r_i(\theta)\,\hat{A}i,\; \mathrm{clip}\bigl(r_i(\theta),\, 1-\epsilon_c,\, 1+\epsilon_c\bigr)\,\hat{A}_i \right),
$$
and raw gradient in the unclipped region
$$
\nabla\theta \mathcal{L}{\mathrm{GRPO}}(\theta) = \frac{1}{T} \sum{i=1}^{T} r_i(\theta)\,\hat{A}i\,\nabla\theta \log \pi_\theta(a_i).
$$
The difficulty is policy-shift amplification: in a large action space, reuse causes $r_i=\pi_\theta(a_i)/\pi_{\mathrm{old}}(a_i)$ to inflate rapidly as the policy drifts, concentrating harmful gradients on tail tokens and inducing catastrophic collapse. The paper identifies the Disproportionate Weight Divergence phenomenon, in which degradation is synchronized with an abrupt surge in the lm_head weight change while intermediate transformer layers remain stable; it further proves that harmful gradients concentrate at the lm_head, and that the lm_head gradient norm lower-bounds empirical $\chi^2$ policy divergence [2605.19425].

This analysis motivates Dynamic Gradient Gating. DGG monitors
$$
g_t=|G_t^{\mathrm{lm}}|_F^2,\qquad \Delta g_t=g_t-g_{t-1},\qquad z_t=\frac{\Delta g_t-\mu_t}{\sigma_t+\epsilon},
$$
using a sliding window of the $W=20$ most recent increments, $\epsilon=1\mathrm{e}{-8}$, and a threshold $\tau\in{0.1,0.5,1.0}$. When the window is warm, the reuse index satisfies $k>1$, and $z_t>\tau$, gradients are zeroed before the optimizer step and the reuse loop terminates. On Qwen3-4B-Instruct-2507 and Qwen2.5-7B-Instruct, across math reasoning, ALFWorld, WebShop, and search-augmented QA, DGG matches or exceeds the converged performance of the single-use baseline across all 16 reported settings and achieves up to $2.93\times$ rollout efficiency and $2.14\times$ wall-clock speedup [2605.19425].

A different but closely related problem appears in off-policy reinforcement learning. There, importance sampling is unbiased only when the target policy is independent of the replay buffer. Once the same buffer is reused both to optimize and to evaluate the policy, the estimator acquires Reuse Bias, formalized through
$$
\epsilon_{\mathrm{RE}}(O,\pi_0,D)=\hat J_{\mathrm{IS}}(O(\pi_0,D);D)-J(O(\pi_0,D)),
$$
with expectation $\epsilon_{\mathrm{RB}}(O,\pi_0)$. The analysis shows overestimation for an optimizer that maximizes the IS estimate on the reused buffer, gives a high-probability upper bound involving a KL term and a data-dependent mismatch term, and derives a stability-based control criterion. Bias-Regularized Importance Sampling operationalizes this by penalizing
$$
L_{\mathrm{BR}}(\pi,D)=\mathbb{E}_{(s,a)\sim D}\left[\left|\frac{\pi(a\mid s)}{\mu(a\mid s)}-1\right|\right],
$$
yielding empirical gains in MiniGrid, SAC, and TD3 settings [2209.07074].

Earlier policy-gradient work treated reuse primarily as an importance-weighting problem. In PGPE, past trajectories are reused through weights
$$
w(\theta)=\frac{p(\theta\mid \rho)}{p(\theta\mid \rho_{\mathrm{old}})},
$$
together with an optimal constant baseline
$$
b^*=\frac{\mathbb{E}[R(h)\,w(\theta)^2|s(\theta)|^2]}{\mathbb{E}[w(\theta)^2|s(\theta)|^2]},
$$
which reduces variance without losing unbiasedness under support overlap [1301.3966]. In infinite-horizon policy-gradient optimization, partial trajectory reuse replaces episode-level reuse with per-step reuse, and Variance Reduction Experience Replay uses a mixture likelihood ratio estimator with a variance-based selection rule to obtain
$$
\mathrm{Tr}\,\mathrm{Var}!\left[\hat\nabla J_k^{\mathrm{MLR}}\mid M_k\right]\le \frac{c}{|U_k|}\,\mathrm{Tr}\,\mathrm{Var}!\left[\hat\nabla J_k^{\mathrm{PG}}\mid M_k\right],
$$
thereby accelerating actor–critic and PPO on CartPole, Acrobot, and a fermentation control task [2205.02976].

3. Retrieval-conditioned and structural reuse in large-language-model systems

In continual post-training and retrieval-augmented adaptation, gradient reuse has been recast as an inference-time knowledge mechanism. ReGrad computes unsupervised document-level language-modeling gradients in a LoRA subspace, stores them in an indexed Gradient Bank, retrieves top-$K$ documents by BM25 via Elasticsearch, and aggregates their stored gradients by
$$
g(q)=\sum_{k=1}^{K} g_{i_k},\qquad \theta_q=\theta^-\alpha^\odot g(q).
$$
The backbone parameters $W$ remain frozen, the temporary update is confined to the low-rank subspace $\theta$, and after answer generation $\theta_q$ is discarded and $\theta^*$ restored. The outer bi-level objective differentiates through the inner gradient step so that reconstruction gradients become query-useful adaptation signals rather than mere document-reconstruction updates [2606.15734].

This formulation distinguishes ReGrad from both continual post-training and standard RAG. CPT accumulates persistent weight drift in shared parameters; RAG avoids drift but leaves knowledge external to the model. ReGrad instead performs reversible parametric injection. Across LLaMA-1B, 3B, and 8B, ReGrad yields the best average among non-hybrid methods, while ReGrad+ICL gives the best overall averages. In the continual injection experiment on LLaMA-8B, CPT improves initially but degrades at 10K injected documents, whereas ReGrad improves monotonically and reaches an average of 44.05 at 10K. The ablations are decisive: raw document LM gradients without meta-learning underperform Direct Generation, and replacing retrieved gradients with random gradients yields only marginal gains [2606.15734].

A structurally different use of gradient reuse appears in Tree Training for agentic LLMs. Here the computation graph is tree-structured because one rollout can branch through memory retrieval, concurrent tool calls, or planning. Standard sequence packing flattens each branch into an independent sequence, recomputing shared prefixes in both forward and backward passes. Tree Training instead computes each shared prefix once and restores exact gradient equivalence by aggregating branch contributions at shared positions. For a prefix token representation $h_p$ and total loss $L=\sum_i L_i$ over leaves,
$$
\nabla_{\theta_p} L=\sum_{i\in \mathcal{L}(u)} \frac{\partial L_i}{\partial h_p}\cdot \frac{\partial h_p}{\partial \theta_p}.
$$
Gradient Restoration implements this by scaling upstream gradients at shared prefix positions by their reuse counts, while original position ids and a shared-prefix causal mask preserve the local Jacobians needed for equivalence [2511.00413].

The significance is twofold. First, simple KV reuse is insufficient for training, because prefix gradients depend on suffix branches through attention; forward-only caching does not restore the branch-dependent backward signal. Second, exact equivalence to naive per-branch training can coexist with large systems gains. On synthetic data, full-tree fitting in memory gives up to $5.7\times$ speedup and memory-constrained Tree Packing yields $4.5$–$5.3\times$ acceleration; on real agentic RL rollouts, end-to-end training time is reduced by up to $3.9\times$, with typical SFT and RL settings showing up to $2.85\times$ throughput improvement [2511.00413].

4. Variance reduction, control variates, and repeated passes

In finite-sum optimization, gradient reuse is often synonymous with controlled staleness. SVRG computes a snapshot gradient
$$
\tilde\mu=\nabla F(\tilde w)=\frac{1}{n}\sum_{i=1}^{n}\nabla f_i(\tilde w)
$$
and reuses it across an inner loop through the update
$$
w_{t+1}=w_t-\eta\bigl(\nabla f_{i_t}(w_t)-\nabla f_{i_t}(\tilde w)+\tilde\mu\bigr).
$$
The reused term acts as a control variate, preserving unbiasedness while sharply reducing variance. The practical analysis shows that convergence is maintained even when $\tilde\mu$ is inexact, provided the control-variate error decreases geometrically; this yields growing-batch strategies for the snapshot gradient, support-vector screening for zero-gradient examples, and a regularized iteration in which the smoothness constant improves from $L$ to $L_m=\max{L_h,L_g}$ in composite objectives [1511.01942].

Online evolution strategies adopt a different reuse variable: the perturbation itself. Noise-Reuse Evolution Strategies reuses the same Gaussian perturbation across consecutive truncation windows of an episode, in contrast to estimators that resample fresh perturbations per window. Within the unbiased GPES$_K$ family, the variance analysis identifies the $K=T$ case—NRES—as the minimum-variance member. The result is an online estimator that is unbiased for the same smoothed objective as FullES but avoids the additional variance from accumulating independent noises. Empirically, NRES outperforms AD and existing ES baselines on chaotic dynamical systems, meta-training learned optimizers, and Mujoco RL; for longer-horizon learned-optimizer settings, the reported wall-clock speedups are $3.8\times$ and $2.1\times$ over PES and about $9.7\times$ and $3.5\times$ over FullES to reach comparable losses [2304.12180].

In sketched linear regression, reuse is implemented at the level of repeated data passes. Multi-pass SGD updates
$$
v_t=v_{t-1}-\gamma_t\,Sx_{i_t}\bigl(x_{i_t}^T S^T v_{t-1}-y_{i_t}\bigr),
$$
with an effective number of steps $L_{\mathrm{eff}}=\lfloor T/\log T\rfloor$. Under a power-law covariance spectrum $\lambda_i(H)\asymp i^{-a}$ and aligned source condition $\mathbb{E}[\lambda_i\langle v_i,w^*\rangle^2]\asymp i^{-b}$, the main scaling result for $a\ge b>1$ is
$$
\mathbb{E}[\mathrm{Risk}M(v_T)]=\sigma^2+\Theta(1/M^{b-1})+\Theta!\left(1/(L{\mathrm{eff}}\gamma)^{(b-1)/a}\right),
$$
whereas one-pass SGD gives
$$
\mathbb{E}[\mathrm{Risk}M(v_N^o)]=\sigma^2+\Theta(1/M^{b-1})+\Theta!\left(1/(N{\mathrm{eff}}\gamma)^{(b-1)/a}\right),
$$
with $N_{\mathrm{eff}}=N/\log N$. Choosing $L_{\mathrm{eff}}\lesssim N^{a/b}/\gamma$ improves the excess-risk scaling from $N^{(1-b)/a}$ to $N^{(1-b)/b}$ in the data-constrained regime [2506.08415].

Across these works, the repeated pattern is explicit reuse of information that would otherwise be discarded: stale gradients in SVRG, shared perturbations in NRES, or repeated passes over the same samples in multi-pass SGD. The gains arise from variance reduction, stronger implicit spectral filtering, or reduced latency; the costs appear as additional bookkeeping, snapshot error control, or step-size restrictions.

5. Meta-gradient reuse and the problem of information reuse

In hierarchical reinforcement learning, gradient reuse becomes a higher-order phenomenon. The method for reusable options formulates option learning as gradient-based meta-learning. A high-level controller $\pi^\Omega(\omega\mid s;\theta)$ adapts within a task for $L$ inner steps,
$$
\theta_{i+1}=\theta_i-\alpha_{\mathrm{in}}\nabla_\theta L_T(\theta_i,\phi),
$$
while shared option parameters $\phi$ are optimized for post-adaptation return:
$$
J(\phi,\theta_0)=\mathbb{E}{T\sim p(T)}\bigl[\mathbb{E}{\tau\sim p(\tau\mid \pi(\theta_L(T,\phi),\phi))}[G_T(\tau)]\bigr].
$$
The outer gradient differentiates through the entire inner update path, so the gradients used to adapt the controller are themselves reused to shape the option policies and terminations. DiCE and Loaded DiCE are employed to obtain correct higher-order policy gradients in the RL setting, and the inferred option policy gradient supplies responsibilities over latent options [2212.11726].

This meta-gradient interpretation directly addresses limitations of earlier reusable-option methods. MLSH passes no gradient signal between the high-level controller and the options; fixed-duration terminations are brittle when option lengths should be state-dependent. By contrast, the meta-objective explicitly rewards options that allow the high-level controller to improve in a few steps. On Taxi and AntMaze, the method learns transferable components that accelerate adaptation and outperform MLSH and several non-hierarchical baselines in the reported settings [2212.11726].

The most stringent formulation of information reuse appears in quantum machine learning, where the question is whether backpropagation-like scaling can survive quantum measurement collapse. The paper formalizes backpropagation scaling by requiring
$$
\mathrm{TIME}(F'(\theta))\le c_t\,\mathrm{TIME}(F(\theta)),\qquad
\mathrm{MEMORY}(F'(\theta))\le c_m\,\mathrm{MEMORY}(F(\theta)),
$$
with $c_t,c_m=O(\log M)$. It then proves that in the quantum-data setting, with single-copy access and no quantum memory, such scaling is impossible in general. With multiple copies and gentle measurements, however, a shadow-tomography-based algorithm can estimate all gradient components of a broad quantum neural network class using
$$
m=O!\left(\frac{n\log^2 M}{\epsilon^4}\right)
$$
copies and $\tilde O(mM)$ quantum operations, while a special case attains $\mathrm{TIME}(F'(\theta))=O(\log M)\cdot \mathrm{TIME}(F(\theta))$ [2305.13362].

The quantum result clarifies a limiting case of gradient reuse: unlike classical backpropagation, one cannot generally reuse a single quantum state across many gradient coordinates because measurement collapse destroys the intermediate information. Reuse becomes possible only by replacing single-shot access with multiple copies, gentle measurements, and a classical hypothesis state updated online.

6. Failure modes, diagnostics, and open questions

A persistent misconception is that more reuse is automatically better. The recent literature does not support that conclusion. In RLVR, naive multi-update reuse collapses after an initial acceleration, and the onset is not reliably detected by KL divergence, clip ratio, or global gradient norm; the lm_head gradient energy is the sharp real-time signal in the reported experiments. In off-policy RL, reusing the replay buffer for both optimization and evaluation produces an overoptimistic estimator by construction, and the resulting Reuse Bias can be arbitrarily severe [2605.19425][2209.07074].

A second misconception is that any stored or shared gradient-like object is directly reusable. ReGrad shows that raw document LM gradients are not sufficient: the Raw Gradient baseline consistently underperforms Direct Generation, and irrelevant retrieved gradients produce gradient interference, especially as $K$ grows. Tree Training shows the analogous point on the systems side: prefix caching alone is a forward-pass optimization and does not make backward gradients correct unless the shared prefix receives the sum of upstream branch contributions [2606.15734][2511.00413].

The stabilizing conditions are domain-specific. PGPE requires support overlap and, in high-dimensional settings, may need a reuse window restricted to recent iterations and weight truncation such as $w\le 2$ [1301.3966]. VRER depends on a variance-based admissibility test for reused transitions [2205.02976]. Practical SVRG relies on decreasing control-variate error, growing-batch schedules, and, in some models, support-vector detection [1511.01942]. NRES assumes a structured online ES setting and notes hysteresis from parameter changes between windows as a source of bias not covered by the main theory [2304.12180]. Multi-pass SGD improves scaling only under explicit constraints on $\gamma$ and $L_{\mathrm{eff}}$; beyond the stated regime, fluctuation terms can dominate [2506.08415]. In quantum settings, multiple copies and quantum memory are required, and the remaining classical overhead is tied to open problems in shadow tomography [2305.13362].

The open problems therefore mirror the diversity of the field. RLVR work points to extending the analysis from GRPO to standard PPO and value-head dynamics, and to alternative real-time divergence surrogates such as natural gradient norms or explicit $\chi^2$ control [2605.19425]. ReGrad raises questions about weak or synthetic supervision, retrieval robustness, and privacy of stored gradients [2606.15734]. Tree Training leaves online packing, irregular branching, and distributed implementation complexity as practical scaling questions [2511.00413]. The quantum literature frames the strongest unresolved issue: whether the classical overhead of shadow-tomography-style gradient estimation can be reduced for poly-time observables without violating known lower bounds [2305.13362].

In this sense, gradient reuse is best understood not as a single optimization trick but as a broad research program about when prior gradient information can be reused safely, exactly, or approximately. The answer depends on what is being reused, how dependence is controlled, and whether the reuse mechanism changes the estimator’s bias, variance, or computational graph in a way that must itself be modeled.