When to Write and When to Suppress: Route-Specialized Dual Adapters for Memory-Assisted Knowledge Editing

Abstract: Knowledge editing systems must update selected facts while preserving nearby but irrelevant behavior. This paper studies this problem in a memory-assisted setting where an edit memory is retrieved at inference time and a parameter-efficient adapter corrects the model's object preference. We argue that the central design question is not only how to write an edit, but also when to suppress it. We introduce \method{}, a route-specialized dual-adapter editor. A relevance router first decides whether a prompt should receive an edit memory. Routed prompts use an edit adapter trained to prefer the new object over the original object; unrouted non-direct prompts use a separate locality adapter trained to preserve or restore the original-object preference. We evaluate \method{} on three 1,000-case protocols, \cf{}, \zsre{}, and \mquake{}, under the same memory protocol and two 7B/8B base models. On Llama-3.1-8B-Instruct, \method{} obtains the best overall probability-preference accuracy on all three benchmarks: 0.8180 on \cf{}, 0.8946 on \zsre{}, and 0.9922 on \mquake{}. The same trend holds on Qwen3-8B. Router ablations show that the relevant memory boundary differs across datasets: a lexical neural router is safest on \cf{}, while BGE embedding routing is better on \zsre{} and \mquake{}. Component and module ablations show that the gain mainly comes from separating edit injection from off-route suppression rather than from simply increasing LoRA capacity.

• The paper introduces RRDA, a dual-adapter framework that distinctly writes new knowledge and suppresses existing facts via route specialization.
• It employs a learned relevance router to activate either the edit or locality adapter, enhancing precision across various prompt types.
• Experiments demonstrate RRDA's superior probability-preference accuracy on benchmarks like CounterFact, ZsRE, and MQuAKE-CF.

Route-Specialized Dual Adapters for Memory-Assisted Knowledge Editing: An In-Depth Analysis

Problem Formulation and Motivation

The knowledge editing paradigm for LLMs demands precision in updating specific factual associations without polluting neighboring knowledge. Classical approaches either recursively fine-tune model weights or introduce explicit external memory banks. However, both techniques suffer from locality failures: parametric edits may generalize inappropriately, and memory-based edits may leak if retrieval is indiscriminately applied.

This work proposes reframing knowledge editing as a routed probability-preference problem: the critical question is not merely how to inject a new fact but whether the context necessitates memory activation. This recognition leads to a dual-path architecture, where route-specialized adapters—one for edit injection, one for edit suppression—are mediated by a learned relevance router.

Figure 1: Route-specialized memory-assisted editing. A relevance router decides whether to retrieve edit memory. Routed prompts use the edit adapter and are optimized to prefer the new object. Unrouted non-direct prompts use the locality adapter and are optimized to prefer the protected original object.

Methodological Contributions

RRDA Architecture

The core method, RRDA (Relevance-Routed Dual Adapters), employs a two-stage decision process on each prompt: (1) a relevance router selects whether to retrieve the edit memory, and (2) the system activates either an edit-oriented or locality-oriented LoRA adapter for probability preference correction. Key architectural features include:

• Memory Bank: Constructed from direct edit records, containing explicit subject-relation-object triples for each intended edit.
• Router: Validated across four variants (subject, subject+relation, neural MLP, BGE-based semantic routing), selecting the most dataset-appropriate.
• Edit Adapter: Activated for routed prompts, trained with cross-entropy and ranking objectives to favor the new object.
• Locality Adapter: Activated for off-route, non-direct prompts, trained exclusively to preserve the original factual association (locality).
• Deterministic Inference Path: Fixed policies for routing and adapter activation, enforcing auditability and reducing stochasticity in deployment.

Probabilistic Evaluation Metric

The paper evaluates systems using the micro-averaged probability-preference accuracy over direct, paraphrase, and locality prompts—a stricter measure than generation-based metrics as it detects even silent probability shifts toward the edited object.

Experimental Results

Extensive evaluations are conducted across three knowledge editing benchmarks—CounterFact, ZsRE, and MQuAKE-CF—using two contemporary 7B/8B class models (Llama-3.1-8B-Instruct, Qwen3-8B).

Figure 2: Overall probability-preference accuracy on two base models. RRDA is best on all three benchmarks for both Llama-3.1-8B and Qwen3-8B.

RRDA achieves top overall accuracy on all benchmarks and models under the memory-assisted editing regime:

• CounterFact: RRDA reaches 0.8180 on Llama, a relative gain over strong LocFT-style and PRUNE-style PEFT baselines, primarily from substantial improvements in locality.
• ZsRE: RRDA achieves 0.8946 accuracy, notably improving locality where baseline routers perform poorly.
• MQuAKE-CF: Nearly saturating at 0.9922, emphasizing robust generalization in multi-hop scenarios.

Notably, gains are not uniform: direct prompt accuracy is already high for all methods, but RRDA excels where off-route leaks would penalize locality.

Route Specialization: Router Ablations

The empirical investigation demonstrates that relevance boundaries are benchmark-dependent:

Figure 3: Different relevance boundaries are needed across benchmarks. CounterFact selects NN because it preserves no-route locality, while ZsRE and MQuAKE-CF select BGE semantic routing.

• CounterFact: Lexical/subject-based routers are safer—neural MLP routing outperforms semantic embeddings due to adverse over-routing by the latter.
• ZsRE/MQuAKE-CF: Semantic routing (BGE+MiniLM) is essential, as locality prompts can lexically overlap in subjects but differ relationally.

This suggests that future editors should consider hybrid or context-sensitive router calibration.

Adapter Ablation and Target Module Analysis

Component ablations confirm that RRDA’s performance is not simply due to increased LoRA capacity. The dual-adapter approach demonstrably segregates strengths: edit adapters excel on edit tasks, suppressor adapters safeguard locality.

Figure 4: Locality in the component ablation. The locality adapter is most informative on CounterFact and ZsRE, where off-route prompts are present and nontrivial.

Further, module target sweeps reveal minimal benefit in expanding LoRA target modules beyond $q/v$, reinforcing that architectural bifurcation, rather than brute parameter increase, drives the observed improvements.

Figure 5: More target modules do not materially improve CounterFact accuracy under the rank-8, 500-step setting.

Implications and Future Directions

This study substantiates the thesis that reliable knowledge editing in LLMs requires explicit route-specialization: edit injection and suppression must be disjoint, and routers must be empirically validated for the target domain.

Practical implications include:

• Auditability and Determinism: Decomposed, route-explicit design enables explicit provenance tracking and post-edit validation.
• Deployment Suitability: The low parameter count and auditable paths support application in controlled settings where edit implications are sensitive.

Potential extensions include:

• Hybrid/Adaptive Routing: Combining semantic retrieval with supervised lexical calibration to further adapt to domain-specific risks.
• Generalization to Larger Models/Longer Edits: Scaling up to high-capacity models and longer edit sequences, where new failure modes may surface.
• Integration with Official Benchmarks: Adapting the protocol for direct comparability with generation-based standards (e.g., MQuAKE-CF exact-match).

Conclusion

RRDA represents an explicit, deterministic architecture for knowledge editing in LLMs, separating the act of writing from off-route suppression via route-specialized dual adapters and learned relevance routing. This yields strong probability-preference locality and accuracy across benchmarks and model architectures, with ablation studies confirming the essential role of route specialization over capacity scaling. The framework sets a new standard for auditable, memory-assisted knowledge editing in LLMs, with clear implications for the design of safe and precise model editors.

Reference: "When to Write and When to Suppress: Route-Specialized Dual Adapters for Memory-Assisted Knowledge Editing" (2606.14668).