HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning

Abstract: Lifelong model editing (LME) aims to sequentially rectify outdated or inaccurate knowledge in deployed LLMs while minimizing side effects on unrelated inputs. However, existing approaches typically apply parameter perturbations to a static and dense set of LLM layers for all editing instances. This practice is counter-intuitive, as we hypothesize that different pieces of knowledge are stored in distinct layers of the model. Neglecting this layer-wise specificity can impede adaptability in integrating new knowledge and result in catastrophic forgetting for both general and previously edited knowledge. To address this, we propose HiEdit, a hierarchical reinforcement learning framework that adaptively identifies the most knowledge-relevant layers for each editing instance. By enabling dynamic, instance-aware layer selection and incorporating an intrinsic reward for sparsity, HiEdit achieves precise, localized updates. Experiments on various LLMs show that HiEdit boosts the performance of the competitive RLEdit by an average of 8.48% with perturbing only half of the layers per edit. Our code is available at: https://github.com/yangfanww/hiedit.

• The paper presents a novel hierarchical RL mechanism that decomposes model editing into high-level layer selection and low-level parameter updates.
• It achieves 8-12% improvements in edit efficacy, generalization, and retention while minimizing interference with existing knowledge.
• HiEdit scales to 20,000 sequential edits, effectively preserving general language capabilities and reducing catastrophic forgetting compared to baselines.

HiEdit: Hierarchical Reinforcement Learning for Adaptive Lifelong Model Editing

Motivation and Problem Statement

The critical challenge addressed in "HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning" (2604.11214) is the development of scalable, adaptive, and robust lifelong model editing (LME) for LLMs. LME demands the sequential rectification of erroneous or obsolete knowledge in deployed models, with the explicit goal of mitigating catastrophic forgetting and minimizing unintended effects on unrelated knowledge and general capabilities of LLMs.

Conventional approaches—both closed-form editing solutions (e.g., ROME, MEMIT) and hypernetwork-based methods (e.g., MEND, MALMEN, RLEdit)—uniformly apply parameter perturbations across fixed and often extensive layer subsets, disregarding the empirical observation that distinct knowledge items are manifested in disparate model layers. This universal editing strategy introduces excessive interference, impeding targeted knowledge incorporation and exacerbating forgetting of prior edits and general skills.

Methodology: Hierarchical RL for Localized Parameter Updates

HiEdit advances LME via hierarchical reinforcement learning (HRL) to explicitly decompose the editing trajectory into:

1. Layer Selection (High-Level Policy): Instance-aware determination of which subset of layers to perturb for the specific knowledge update.
2. Parameter Updating (Low-Level Policy): Generation of the layer-wise parameter update for the selected component set.

The high-level policy (a hypernetwork $\pi_\phi$) adapts to each instance, producing a sparse binary mask over the "influential layer range" (selected per experimental setting), specifying the layers for localization. Gradient signals from standard fine-tuning for the edit input are subject to low-rank decomposition; features from the decomposed vectors are concatenated and processed through a shared encoder, gated, and TopK-selected for binary masking. The low-level policy (hypernetworks $\mathcal{H}_\theta$) then generates update tensors only for the chosen layers.

Figure 1: HiEdit decouples lifelong model editing into hierarchical high-level (where to edit) and low-level (how to edit) stages, coordinated via HRL.

Intrinsic reward design is pivotal: a relative advantage signal, based on the benefit of sparse (partial-layer) over dense (full-layer) updates, incentivizes minimal, efficient intervention. This is accomplished by contrasting the loss/reward for selective perturbation with that for all-layer editing.

Experimental Protocols and Benchmarks

HiEdit is evaluated on competitive settings involving Llama-3-8B and Gemma-2-9B models, with sequential edits drawn from ZsRE and CounterFact—benchmarks targeting both factual knowledge updating and out-of-distribution generalization. The evaluation regime is stricter than prior works, employing timely and long-range streaming of 8,000 to 20,000 edits, moving far beyond isolated batch or short-range sequential settings.

Metrics include:

• Efficacy: Edit success for target facts.
• Generalization: Propagation to paraphrased/related instances.
• Specificity: Preservation of unrelated knowledge.
• Edited Retention: New metric for prior-edited knowledge recall.
• General Retention: Average F1 on GLUE tasks for overall language competence.

Results: Robustness, Scalability, and Efficiency

HiEdit achieves statistically significant improvements over state-of-the-art baselines.

Quantitatively: Compared to RLEdit and AlphaEdit, HiEdit delivers mean performance gains of 8-12% on Efficacy, Generalization, and Retention metrics, while requiring only half as many layers per edit. Performance on GLUE tasks is less degraded by large-scale sequential editing, evidencing superior catastrophic forgetting mitigation.

Figure 2: HiEdit outperforms prior LME methods by combining adaptive sparse updates and retention of previous knowledge.

Figure 3: After up to 20k sequential edits, HiEdit preserves general capabilities on six GLUE benchmarks, unlike baselines that rapidly degrade.

Figure 4: HiEdit maintains high Efficacy, Generalization, and Specificity on the first 500 sequentially edited knowledge instances, preventing knowledge erosion.

Ablation and random-layer baselines show sharp performance drops when the layer selection policy is replaced with arbitrary or zero-gradient regimes, supporting the necessity of the HRL-based adaptive selection. Furthermore, analyses of layer selection distributions show that HiEdit reliably targets semantically relevant layers for different types of knowledge, with domain-specific patterns emerging in selection statistics.

Figure 5: Visualization of HiEdit's layer selection confirms domain- and instance-specific targeting, with frequent selection of key layers for different knowledge types.

Scalability analysis demonstrates minimal degradation even at 20,000 edits—a regime where all non-adaptive or flat editing models collapse.

Figure 6: HiEdit alone maintains high Specificity, General Retention, and Edited Retention across orders-of-magnitude increases in edit count.

Computational Efficiency and Case Study

HiEdit introduces negligible parameter overhead (about 3% relative to baseline hypernetwork size, and <2% with respect to the LLM), but delivers editing speedups due to reduced layer count per edit. Case studies confirm the qualitative preservation of previously and recently edited knowledge, where competitive baselines experience output collapse or semantic drift.

Figure 7: Editing time per instance is significantly reduced by HiEdit, due to adaptive layer selection versus dense-update baselines.

Figure 8: Edited instances after long sequential editing periods: only HiEdit continues to reliably update both old and new facts.

Implications and Future Directions

The demonstration that targeted, HRL-based sparse updates yield increased edit efficacy and superior retention properties redefines the paradigm for LLM editing. From a practical perspective, HiEdit provides a candidate solution for deploying continually updated LLMs in production environments, where both factual correctness and general-purpose capabilities must be simultaneously maintained. The theoretical implication is the empirical validation of distributed, instance-specific knowledge localization within transformer layers, supporting recent findings on knowledge circuits.

Possible next steps include:

• Dynamic (rather than fixed-TopK) layer count selection via e.g., Top-p or learnable sparsity control.
• Extension to multi-domain or unstructured factual editing.
• Exploring alternative or more expressive HRL algorithms for subgoal discovery and meta-control in model editing.

Conclusion

HiEdit introduces a rigorous hierarchical RL framework for lifelong LLM editing, enabling adaptive, data-driven instance-specific selection of parameter subsets for editing. This hierarchical structure combines efficiency, scalability, and robustness, achieving strong gains over state-of-the-art LME approaches. The empirical evidence supports the importance of targeted, reward-driven sparse updates, and opens directions for more generalizable, interpretable, and sustainable model updating schemes.