Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

Abstract: Continual post-training aims to extend LLMs with new knowledge, skills, and behaviors, yet it remains unclear when sequential updates enable capability transfer and when they cause catastrophic forgetting. Existing methods mitigate forgetting through sequential fine-tuning, replay, regularization, or model merging, but offer limited criteria for determining when incorporating new updates is beneficial or harmful. In this work, we study LLM continual post-training through three questions: What drives forgetting? When do sequentially acquired capabilities transfer or interfere? How can compatibility be used to control update integration? We address these questions through task geometry: we represent each post-training task by its parameter update and study the covariance geometry induced by the update. Our central finding is that: forgetting can be considered as a state-relative update-integration failure, it arises when the covariance geometries induced by tasks misalign with the geometry of the evolving model state. Sequential updates transfer when they remain compatible with the model state shaped by previous updates, and interfere when state-relative geometry conflict becomes high. Motivated by this finding, we propose Geometry-Conflict Wasserstein Merging (GCWM), a data-free update-integration method that constructs a shared Wasserstein metric via Gaussian Wasserstein barycenters and uses geometry conflict to gate geometry-aware correction. Across Qwen3 0.6B--14B on domain-continual and capability-continual settings, GCWM consistently outperforms data-free baselines, improving retention and final performance without replay data. These results identify geometry conflict as both an explanatory signal for forgetting and a practical control signal for LLM continual post-training.

• The paper presents Geometry-Conflict Wasserstein Merging (GCWM), demonstrating that state-relative geometry conflict effectively predicts and controls forgetting in LLM continual post-training.
• The methodology leverages layer-wise covariance geometries and normalized Bures–Wasserstein distance to provide a robust compatibility signal beyond traditional update norms.
• Empirical results reveal that GCWM improves domain and capability retention across various model scales while theoretical bounds link geometry conflict to loss changes for practical adaptation.

Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training

Problem Formulation and Motivation

Continual post-training for LLMs involves sequentially adapting a base model to multiple domains or capabilities, where each post-training task is incorporated via model parameter updates. Catastrophic forgetting—a sharp degradation in prior capabilities after integrating new task updates—remains a central challenge. Existing approaches (sequential SFT, replay, regularization, merging) mitigate forgetting but lack principled criteria for when update integration is beneficial versus harmful. Task heterogeneity, scale, and complex post-training objectives exacerbate this, particularly for LLMs.

This work introduces a detailed task-geometry analytic framework, representing each post-training task via its parameter update and characterizing the induced layer-wise covariance geometry. Geometry conflict—a normalized Bures–Wasserstein discrepancy between task-induced covariance geometries—is defined as the primary compatibility signal. The central claim is that forgetting manifests as state-relative update-integration failure: it occurs when task-induced covariance geometries are misaligned with the geometry of the evolving model state, and transfer is achieved when new updates remain compatible with prior state-induced geometry.

Task Geometry, Compatibility Signals, and Diagnostic Analysis

Each task's update is analyzed by its layer-wise covariance geometry, capturing not just the update norm (magnitude), but also the subspace projection and spectral structure. Geometry conflict employs the normalized Bures–Wasserstein distance over task-induced covariance matrices projected into a shared basis.

The quantitative analysis demonstrates that:

• Update Norm is Insufficient: Parameter drift (update norm) only weakly explains retention loss. State-relative geometry conflict, in contrast, exhibits stronger correlation with forgetting, especially as model scale increases.

  Figure 1: State-relative geometry tracks forgetting across continual steps and scales.

• Geometry Conflict Refines Subspace Compatibility: Subspace Alignment Ratio (SAR) is non-redundant with geometry conflict; task pairs with similar SAR can have highly divergent geometry conflict. Pairwise compatibility is informative for regime diagnosis but fails to predict forgetting in isolation.
• State-Relative Geometry is Predictive: Incoming update compatibility with the current model state, rather than isolated task pairs, best predicts forgetting. This is robust across LLM scales and continual methods.
• Complementary Failure Modes: Geometry conflict and gradient conflict (cosine similarity between gradients) identify distinct failure loci in the model. Geometry conflict is strongest in MLP and value projections, while gradient conflict concentrates on attention key/query projections.

  Figure 2: Pairwise compatibility and conflict complementarity; SAR and geometry conflict stratify transfer regimes, but pairwise conflict alone weakly predicts forgetting. Top-layer share and heatmaps reveal complementary mechanisms.

Geometry-Conflict Wasserstein Merging (GCWM): Mechanism and Theoretical Guarantees

The findings motivate Geometry-Conflict Wasserstein Merging (GCWM)—a data-free, compatibility-controlled update integration algorithm for continual LLM post-training. GCWM constructs layer-wise, task-induced covariance geometries, aggregates them via Gaussian Wasserstein barycenters to define shared metrics, and gates geometry-aware correction based on conflict scores. The incremental update applied at each step is the compatibility-controlled difference from the previous merged state.

Theoretical results demonstrate that the additional loss induced by GCWM relative to a plain merge is bounded by layer-wise geometry conflict and the gated merge displacement:

$$\mathcal L_u(\Theta_{\rm gcwm}) - \mathcal L_u(\Theta_{\rm plain}) \leq \eta \sum_\ell c_{u}^{(\ell)} g^{(\ell)} + \frac{\eta^2}{2} \sum_\ell d_{u}^{(\ell)} \| \Delta_{\rm merge}^{(\ell)} - \Delta_{\rm plain}^{(\ell)} \|_{\widetilde B^{(\ell)}}^2,$$

This establishes geometry conflict as both an explanatory and actionable signal for forgetting and compatibility-driven update integration.

Empirical Results: Domain and Capability Continual Post-Training

GCWM is evaluated as a data-free update-integration method across Qwen3 models from 0.6B to 14B on broad domain-continual (MMLU-Pro) and capability-continual (math/code/knowledge) benchmarks. Data-free baselines include AIMMerging, Localize-and-Stitch, OPCM; reference pipelines (Seq. SFT, EWC, FOREVER) are also reported.

GCWM consistently delivers the highest non-MTL accuracy across all scales and benchmarks:

• Domain-Continual: GCWM achieves up to $+1.6$ pp higher mean accuracy for Qwen3-1.7B vs the best baseline, and $+1.2$ pp for 14B. Gains are realized across 9–12 of 14 subdomains per scale.
• Capability-Continual: GCWM improves over the best data-free baseline by $+1.4$ pp at 1.7B, $+4.3$ pp at 14B, leading on all knowledge, math, and code sub-benchmarks.
• Ablations: Removing the conflict gate or the Wasserstein barycenter degrades aggregate performance, especially in economic, math, psychology, and business domains.

  Figure 3: Final performance across model scales and continual post-training methods.

  

  Figure 4: Capability ablation breakdown confirms necessity of conflict gating and Wasserstein barycenter for optimal data-free integration.

Runtime and memory profiling confirm offline feasibility at large scales; merge-time overheads are linear in the union rank and independent of full model dimension due to projected low-rank SPD operations.

Practical and Theoretical Implications

The geometry conflict signal provides both diagnostic and mechanistic insight into continual forgetting. For practical continual LLM adaptation, GCWM offers a principled, data-free merge strategy that is robust across domain and capability shifts, and easily extendable to multi-stage and memory policy variants.

The theoretical contribution is a quantitative bound linking loss change to geometry conflict, validating the control signal basis. Practically, GCWM provides a tool for compatibility-aware adaptation in scenarios where replay is infeasible or undesirable (e.g., privacy-sensitive, distributed training).

GCWM establishes a new analytic baseline for understanding sequential update-integration in large neural models, and sets the stage for future methods leveraging geometric signals—possibly informing hybrid replay+geometry controllers, task-aware merge optimization, and more granular layer-wise gating mechanisms.

Future Directions

• Causal Analysis of Geometry Signals: Extend the explanatory framework for state-relative geometry beyond empirical association; investigate causal mediation between geometry conflict and forgetting under broader distributional shifts.
• Scalable Gating and Metric Aggregation: Explore adaptive, hierarchical gating across layers/modules based on local geometry conflict, and compressive metric aggregation for ultra-large models.
• Hybrid Replay–Geometry Methods: Combine replay and geometric signals, balancing data efficiency and compatibility for continual adaptation.
• Safety and Governance: Use geometry conflict diagnostics for monitoring dangerous or adversarial capability transfer during LLM post-training.

  Figure 5: GCWM merge-time runtime and memory profiling establishes practical feasibility at large scales.

  

  Figure 6: GCWM hyperparameter sensitivity on Qwen3-8B; moderate gate and energy settings yield stable performance.

Conclusion

Task-induced covariance geometry is a central mechanism governing forgetting in continual LLM post-training. State-relative geometry conflict outperforms raw parameter drift and isolated pairwise compatibility as an explanatory and control signal. Geometry-Conflict Wasserstein Merging provides principled, data-free update integration, consistently improving retention and transfer. This analytic and algorithmic framework advances the understanding and control of catastrophic forgetting, opening new avenues for compatibility-driven continual adaptation in large models.

---

Reference:

"Geometry Conflict: Explaining and Controlling Forgetting in LLM Continual Post-Training" (2605.09608)