A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

Abstract: Reinforcement learning (RL) post-training improves LLMs on individual domains such as mathematical reasoning, code generation, question answering, and creative writing (CW), but training on one domain often degrades performance on others. Existing explanations based on catastrophic forgetting or global gradient conflict are incomplete: substantial interference can occur even when full-model gradients are nearly orthogonal. We show that single-domain RL produces sparse, small-magnitude parameter edits with weak overlap among top-changed neurons, while different domains still share substantial active computation routes on which update directions determine whether they act synergistically or conflict. Guided by this observation, we prove under a local perturbation model of multi-domain RL that later-domain training harms an earlier domain mainly through a second-order damage term, which under the observed sparse route structure concentrates in a low-dimensional shared conflict subspace. Moreover, a short domain refresh contracts the harmful component on this subspace, enabling selective recovery with limited collateral damage. Consistent with the theory, a brief Re-Math refresh after Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW recovers Math from 57.66 to 66.04 while largely preserving performance on the other domains, yielding the best average score of 66.39. Beyond refresh, a training-free rollback on a sparse proxy conflict coordinate set for the Math-QA pair partially restores Math, providing direct proxy-level evidence for localized damage. These results provide a localized mechanistic account of interference and recovery in multi-domain RL.

• The paper presents a local perturbation theory that characterizes interference as a second-order, route-localized phenomenon.
• It demonstrates that interference arises from localized gradient conflicts in key MLP and attention modules during sequential training.
• A short, targeted RL refresh is shown to effectively recover degraded domain performance with minimal collateral effects.

A Local Perturbation Theory of Cross-Domain Interference in Multi-Domain RL

Introduction

This paper provides a rigorous analysis of the origins, structure, and recovery of selective cross-domain interference in reinforcement learning (RL) post-training for LLMs. The authors analyze the non-uniform and domain-specific performance degradation observed during sequential multi-domain RL and challenge the sufficiency of classical explanations grounded in catastrophic forgetting or global gradient conflict. Instead, they develop a local perturbation theory which formalizes interference as a second-order, route-localized phenomenon and empirically validate this theory with module-level, neuron-level, and targeted-intervention analyses.

Structural Localization of Cross-Domain Interference

Global Gradient Orthogonality and Localized Conflict

The investigation starts by empirically measuring the pairwise cosine similarity of domain-wise gradients during joint training. Despite substantial degradation of domain-specific performance after sequential domain RL, the global cosine similarity between domain gradients (e.g., Math vs. QA) remains near zero, suggesting that interference is not well explained by antagonistic global update directions. However, decomposing the gradient along the model’s modular structure reveals local hotspots of strong conflict and synergy, particularly in selected MLP and attention modules.

Figure 1: Global gradient cosine between Math and QA during joint training shows near-zero (orthogonal) full-model gradients in spite of observed performance interference.

Figure 2: Attention-module heatmap of pairwise gradient cosine exposes localized regions of domain antagonism and alignment.

Figure 3: Top six attention and MLP modules, ranked by conflict magnitude, demonstrate that not all locations in the model contribute equally to interference.

Parameter Edit Sparsity and Neuron Edit Overlap

Analysis of RL-induced parameter edits shows that single-domain experts exhibit highly sparse and low-magnitude changes relative to the base initialization: upwards of 77–89% of parameters change less than $10^{-7}$ in absolute terms. Sequential training preserves this local sparsity, with the vast majority of parameters unperturbed at each incremental domain addition.

Figure 4: Absolute parameter change distributions demonstrate the sparsity of RL-induced edits per domain.

Despite this sparsity, top-changed neurons between domains, defined over all MLP layers, share only weak overlap, with average Jaccard coefficients below 0.19. This rules out direct co-editing of the same parameter subsets as the main cause of interference and motivates a functional analysis of computation routes.

Shared Active Routes and Directionality

When ranking neurons not by edit magnitude but by inference-time activation, there is considerable overlap in the most active neurons across reasoning-related domains (Math, Code, QA). Thus, even if different domains edit largely non-overlapping parameters, the effect of the edits can be functionally coupled due to route reuse. Critically, directional alignment or conflict between domain update vectors on these shared active units distinguishes between synergistic and interfering transfer.

Figure 5: Layer-wise average directional cosine on shared top-changed neurons: Code-Math edits are mostly aligned, while Math-QA shows layerwise split in directionality.

Local Perturbation Theory of Interference and Recovery

Formal Model

The paper formalizes interference in a local Taylor expansion framework. After training on domain $A$, post-training on domain $B$ yields displacement of the parameters. Under locality, stationarity, and effective sparsity assumptions, the first-order effect on the $A$-domain objective is minimal; global gradient orthogonality ensures negligible drift from first-order terms. The dominant interference term is thus the second-order component

$$\Delta_{A\leftarrow B} \approx \frac{1}{2} \delta_B^\top H_A \delta_B$$

where $\delta_B$ is the update vector induced by domain $B$ RL, and $H_A$ is the Hessian of the $A$-domain loss at its local optimum.

Moreover, leveraging the observed empirical structure, the theory postulates that the harmful component is concentrated in a low-dimensional shared conflict subspace $S_{A,B}$—spanned by the intersection of active routes and parameter update support. Crucially, the main source of damage is not the global norm of the update, but its projection onto $A$0.

Selective and Fast Recovery via Refresh

A central claim is that a short, targeted RL refresh on the previously damaged domain effectively contracts the projection of parameters onto $A$1—thus selectively reversing performance degradation without substantial collateral damage to other domains, provided global gradient near-orthogonality holds. This is formalized as a geometric contraction in norm over $A$2 under positive curvature, explaining the empirical observation of rapid recovery with minimal non-target drift.

Empirical Validation

Task-level results validate the central theoretical predictions. Sequential RL in the order Code→Math→QA→CW results in a pronounced drop in Math performance after the introduction of QA and CW updates (from 66.49 to 57.66), while the performance of Code and QA remain stable, highlighting selective, asymmetric interference. A short Math refresh recovers performance from 57.66 to 66.04, restoring Math to domain-expert levels without significant degradation to other domains and producing the best average cross-domain score.

Figure 6: Validation dynamics during Re-Math refresh reveal rapid, monotonic recovery of Math with limited effect on other domains.

Direct Subspace Intervention

To directly test the subspace-localization hypothesis, the paper introduces a weight-space rollback protocol, reverting only a sparse set of MLP neuron coordinates (proxy-conflict subspace) where Math and QA updates are both large, active, and conflicting. Reverting only 2% of neurons using a composite score recovers 20.4% of the Math loss induced by QA training, far exceeding random or single-factor selection. Expanding to MLP+Attention coordinates and increasing intervention budget shows a dose-response curve in recoverable performance, saturating below the full Math expert but strongly supporting the claim that interference is both localized and causally recoverable.

Figure 7: Layer-wise neuron selection analysis indicates that the conflict factor dominates the layerwise intervention distribution and that composite scores maximize overlap with underlying conflict subspaces.

Implications and Theoretical Significance

The results and analysis represent a significant advance in understanding of multi-domain RL and post-training interference. The formal and empirical localization of interference to shared active computation routes—not the full parameter space or an explicit large-overlap subset—renders classical solutions like naive regularization or global gradient decorrelation insufficient. Instead, local, route-aware monitoring and correction become feasible. This has direct implications for practical multi-domain RL: rapid, targeted refreshes or route-constrained projection steps can ensure robust cross-domain performance without complex replay schedules or static data mixtures.

From a theoretical point of view, this work draws a clear distinction between full-model and route-local gradient geometry, demonstrates the limitations of one-dimensional global metrics, and motivates more granular approaches to multi-objective RL optimization in high-capacity neural systems.

Conclusion

This paper establishes that cross-domain interference in multi-domain RL for LLMs is concentrated in sparse, low-dimensional, locally-active subspaces determined by shared computation routes. The dominant mechanism is a second-order, directionally-sensitive displacement, which can be selectively and efficiently mitigated by short refresh steps or targeted subspace rollback. These insights motivate a new generation of route-aware, robust multi-domain RL strategies and advance mechanistic understanding of post-training interference and recovery dynamics.