Can a Language Model Learn Facts Continually in Its Weights?
Abstract: Continual learning promises a LLM that keeps acquiring knowledge after training, with each new fact written into its weights. Whether weight writes can support accumulation remains undecided. We follow invented facts written into Qwen3 models from creation through sequences of twenty to one hundred later writes, using held-out questions of five types, with the original model given the fact in its prompt as the reference. Across these experiments, the breadth of the training data determines the kind of knowledge created. Bare-statement training produces recitation, while diverse restatements reduce the recitation-to-use gap from 27.4 to 5.4 points without showing the model a conclusion. This difference carries into later writes: after twenty sequential writes, bare-statement facts retain 1% accuracy while facts written from broad study data retain 46%. We also find that facts can be behaviourally forgotten without being erased. Forgotten facts keep most of the log-probability added by their write, and under bare-statement training 70% of wrong answers about them contain the most recently written fact. The same writes barely degrade the model's use of facts in context, and a forgotten study fact supplied in the prompt recovers to 77-80% on its questions. These results describe knowledge that is stored but question-keyed: later writes redirect the questions that reached it. Damage to unrelated abilities tracks KL divergence from the original model, and the later writes cause interference regardless of how the earlier fact was stored. Broad data can create usable knowledge, and a frozen reference can preserve capability, but no intervention we tested, including those built on accurate local measurements of each write, keeps earlier facts reachable. When facts must be composed or survive later writes, the reliable channel is context rather than the weights.
Summary
- The paper demonstrates that while bare-statement training achieves high recall, it severely limits compositional and counterfactual accuracy compared to study training.
- The paper shows that sequential writes cause catastrophic forgetting, with study-trained facts retaining up to 46% after 20 updates versus near-zero retention for bare statements.
- The paper establishes that using frozen-teacher distillation and KL-penalized training can mitigate general capability loss, though current methods fail to robustly support compositional use of stored facts.
Can a LLM Learn Facts Continually in Its Weights?
Introduction and Experimental Setup
This paper rigorously interrogates the parametric continual learning capacity of LLMs, specifically studying how (or if) new factual knowledge can be sequentially installed and robustly retained in model weights without catastrophic forgetting or interference. The experiments utilize Qwen3 models and synthetic, invented facts designed to force disambiguation between prior/world knowledge and injected content. Each fact is introduced through two possible mechanisms: (1) written into the weights via LoRA adapters and merged, or (2) placed directly in the prompt context; both routes face an identical battery of held-out question types evaluating recall, inference, and composition.
Figure 1: The experimental paradigm contrasts two knowledge channels: parameter-writing and context placement, both assessed across a suite of question types.
This dual-channel measurement, with carefully certified and adversarially audited automatic evaluation, sets up strong positive and negative controls (in-context ceiling, model floor) for all knowledge evaluations.
Breadth of Training Data Determines Utility of Installed Knowledge
A central empirical finding is that the spectrum of training data breadth sharply partitions the effectiveness of written knowledge. Narrow, bare-statement training only supports recitation but fails at compositional or inferential usage: e.g., the model can quote the new "fact" but cannot reliably apply it or override its prior. In contrast, training with generated paraphrases, implications, and contrasts ("study training") substantially closes this recitation-to-use gap—quantified as the "entailment gap" between lenient (supports entailed answers) and strict (requires question-conclusion statement) accuracy metrics.
Figure 2: Study training robustly closes the gap between recitation and genuine use, especially for compositional and counterfactual questions.
Figure 3: The entailment gap quantifies how much bare-statement training leaves knowledge stranded at recitation, while broad-data methods almost eliminate the gap.
Empirically, bare-statement training achieves 97% recall but only 21–23% counterfactual accuracy and 39–41% composition; study training achieves 88% recall, but dramatically higher (45–50% counterfactual, 59–60% composition), with gaps of 18–29 points across use cases and a 27.4-point entailment gap reduced to 5.4 points. These effects are robust across model scales and optimization algorithms.
Catastrophic Forgetting and Its Mechanisms
When facts are written sequentially, retention rates are dominated by the kind of knowledge created. After 20 sequential writes, only 1% of bare-statement facts survive, compared to 46% of study-trained facts—a difference that persists to 100 writes, where a plateau of 25–28% is observed for "study facts" and near-zero for bare statements.
Figure 4: Retention drops precipitously for bare-statement facts during sequential writes, while study-trained facts retain a substantial fraction even after 20.
Figure 5: Beyond 20 sequential writes, periodic consolidation does NOT restore lost retention; the decay plateaus but never recovers.
The study parses this decay as a failure of access, not physical erasure: behavioral forgetting occludes the retrieval routes to written facts, but storage (as measured by internal likelihood) remains high. Overwritten facts retain most (57–67%) of their write-induced log-probability lift, but the correct answers become unreachable in normal inference. Furthermore, re-insertion of the fact into the context immediately restores in-context question performance to near ceiling.
Figure 6: Even after behavioral forgetting, the underlying parameters retain most of the statistical lift conferred at write time.
Composition and Joint Use Failures
A notable, consistent limitation is the inability to compose or jointly use multiple written facts. Success rates for questions demanding two facts (with both written into weights) are low (32%), while context placement achieves over 90%. Self-retrieval or compositional queries are not robustly supported by written memory.
Figure 7: Written facts, even if individually usable, cannot reliably be composed to support multi-hop queries; context placement outperforms weight storage.
Model Capabilities, Drift, and Mitigation Approaches
General capability preservation correlates strongly with KL divergence from the original model distribution ( across objectives/conditions). Batch distillation (teacher-forcing from a frozen model) is markedly less damaging than sequential self-distillation, which accumulates drift and rapidly erodes general ability and retention.
Figure 8: KL-divergence from the base model predicts loss of general capability and fact retention across a wide grid of training regimes.
Figure 9: Distillation from a frozen teacher, not an accumulated, self-distilled model, is critical for preserving both retention and general abilities.
A penalty applied during training to KL-divergence from the original model dramatically mitigates catastrophic forgetting without necessarily reducing observed KL, demonstrating that general ability loss and retention are orthogonal and must be managed separately.
Figure 10: Penalty on drift stabilizes capability and fact retention in sequential fine-tuning, but the observed KL from the original model does not systematically decrease, indicating non-causal alignment.
Mechanistic and Causal Analysis
Factorized experiments elucidate the source of interference. The fate of earlier facts is governed almost entirely by the nature of later writes (incoming update) not by the method of initial storage. Local, gradient-based predictors can anticipate immediate effects but not long-term forgetting dynamics. Direct interventions such as parameter-level gradient projection, bridge data, and activation-based mitigation do not outperform random baselines.
Figure 11: Diverse recitation is necessary for robust 'use', but local gradient-based controls do not yield strong protection against eventual forgetting.
Figure 12: Crossed experiments confirm incoming writes (not stored-write method) determine the long-run fate of previously written facts.
Implications, Limitations, and Directions
The conclusions generalize across update methods (SFT, distillation, RL) and model sizes. The findings carry significant implications:
- Practical—To reliably accumulate new knowledge, broad, diverse data for each write is mandatory, and a frozen reference model should be used for distillation; when facts must be composed or survive many updates, context-based storage is essential.
- Theoretical—Current LLMs do not implement a true address-based parametric memory. Written content is stored but only accessible via narrow, question-keyed routes, and general-purpose retrieval or composition is not supported. Local measures are insufficient to mechanistically control long-horizon interference.
- Future Research—Scaling, iterative architectural changes, or differentiated objectives that explicitly model addressing may be required to enable robust continual parametric knowledge accumulation. Connections with addressable (non-parametric) memory architectures and routing-enhanced transformer variants warrant investigation.
Conclusion
This study demonstrates that LLMs trained with standard adapters/fine-tuning cannot continually accumulate and robustly retain new factual knowledge in their weights. While broad, diverse training data can create genuinely usable knowledge and capability loss can be mitigated (e.g., via frozen-teacher distillation or explicit KL penalization), earlier facts remain highly susceptible to access interference by subsequent updates. Forgetting in this setting is operationalized as an irreversible loss of access, not erasure of stored content, and reliable composition or joint use of multiple written facts is not supported. As a result, for complex continual learning use cases requiring robust, compositional, and retrievable parametric memory, current LLM weights are inadequate as a principal channel—context remains essential, and research must focus on mechanisms that build true addressable and compositional knowledge storage in the parametric core of LLMs.
(2607.11020)
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper asks a simple question with big consequences: can a LLM keep learning new facts after it’s trained, and actually use those facts later? The authors compare two ways a model can “know” something:
- In its context: like a sticky note you attach to the question each time.
- In its weights: like writing the fact into the model’s long‑term memory so it should remember without a reminder.
They test whether the model can keep adding new facts to its weights, use them in different ways, and still keep its general skills.
What questions did the researchers ask?
They focused on five kid‑friendly questions about “teaching a model new facts”:
- What kind of knowledge do different training styles create—does the model just repeat the fact or can it use it to solve problems?
- After you teach many facts one after another, how many of the earlier facts does the model still know and use?
- When a fact seems “forgotten,” is it really erased, or is it still stored but hard to find?
- Do repeated updates hurt the model’s general skills (like writing quality or reasoning), and can we prevent that?
- Is it more reliable to put facts in the prompt (context) or to write them into the model’s memory (weights), especially when facts must be combined or outlast many updates?
How did they test it?
Think of teaching a student in two ways:
- Sticky note mode (context): You hand the student a note with the fact every time you ask a question.
- Memory mode (weights): You practice the fact with the student so they’re supposed to remember it without a note.
To avoid mixing with things the model already knows, the authors made up new, fake facts (like “Zorvathine is a metal that melts when cooled below −10°C”). Then they tested five kinds of questions for each fact:
- Recall: repeat the fact.
- Paraphrase: say the same idea in different words.
- Application: use the fact to solve a simple task.
- Composition: combine the fact with other world knowledge.
- Counterfactual: choose the fact over a common default (e.g., “Does it melt or freeze when cooled?”).
They tried different ways to teach facts into the model’s weights:
- Bare‑statement training: show only the fact sentence (repeated in a few simple frames).
- Study training: show lots of varied material—paraphrases, Q&As, worked examples, and contrasts with the default the fact breaks.
- Distillation: have the model mimic a “teacher model” that has the fact in its prompt. Sometimes the teacher is a frozen copy of the original (stable), and sometimes the “teacher” moves along with the student’s updates (unstable).
They taught facts one after another (like 20 in a row, and even 100) and checked:
- Can the model still use earlier facts?
- Are general skills harmed?
- Do forgotten facts come back if you put the fact back into the prompt?
To make the tests fair, an automatic system plus separate AI “judges” scored answers, and the authors verified that the scoring wasn’t biased and that the tests really measured use, not lucky guesses.
What did they find?
Here are the main results, written simply:
- Narrow practice makes parrots; broad practice makes problem‑solvers.
- If you train only on the bare statement (e.g., “Zorvathine melts when cooled…”), the model gets great at repeating it but much worse at using it to answer “what would happen” or “how to apply it.” The authors call this gap between “can recite” and “can use” the entailment gap, and it was large for bare‑statement training.
- With diverse “study” data (paraphrases, Q&As, worked implications), the gap shrank a lot. The model didn’t just repeat; it could draw the right conclusion.
- After many new facts are added, what survives depends on how you taught it.
- After 20 sequential writes:
- Bare‑statement facts: ~1% retained (almost all lost in practice).
- Study‑trained facts: ~46% retained (about half survive).
- After 100 writes, study‑trained facts level off around 25–28% retained, not zero.
- “Forgetting” is often an access problem, not a storage problem.
- Even when the model failed questions about a fact, the model’s inner probabilities still “favored” the fact’s statement—suggesting the fact was still stored in its weights.
- Wrong answers often repeated the most recently learned fact instead (like new material overwrote the “address” to reach the old fact).
- If you put the “forgotten” fact back into the prompt (context), performance bounced back strongly (around 77–80% for study‑trained facts). So the fact wasn’t gone—it was just hard to reach.
- Two written facts are hard to use together.
- When both facts were in the model’s weights, it struggled to answer questions requiring both at once.
- The model also struggled to “retrieve” its own stored facts on demand.
- Supplying the true statements in the prompt helped a lot, showing the knowledge was there but the model lacked a reliable way to fetch and combine it from memory.
- Context is the reliable channel under pressure.
- Using facts from the prompt stayed strong even after many writes, and it decayed more slowly than general skills did. In other words, if you need facts to be combined or to survive many updates, giving them in the prompt is more dependable than writing them into weights.
- Keeping general abilities is possible—if you don’t drift too far.
- The more the model’s behavior drifted from the original (measured as “how different the next‑word predictions are”), the more its general skills dropped. This tracked closely across many settings.
- Distilling from a frozen original teacher preserved general skills well and still wrote facts effectively.
- Distilling from a teacher that moves with the model across writes was the worst—it amplified drift and hurt skills the most.
- An “online” style of distillation (where the student practices on its own outputs) helped retention compared to a fixed sampled dataset, but stability depended on using a stable (frozen) teacher.
- Surprising twist: facts that contradicted the model’s usual expectations sometimes survived better.
- In one set, “weird” facts (that break a default) were kept better than “neutral” facts. One idea is that unusual facts stand out more and collide less with later updates.
Why does this matter?
- For updating AI safely and usefully: If you want a model to keep learning from new information over time, simply “writing” statements into its memory isn’t enough. You need rich, varied training data so the model learns how to use facts, not just repeat them.
- For long‑term knowledge: Even when facts are stored in the model’s weights, later updates can hijack the paths that questions use to find those facts. This means:
- Facts may be “forgotten” behaviorally even though they’re still in there.
- Giving facts in the prompt is often the safer, more reliable way, especially when you must combine multiple facts or survive many future updates.
- For maintaining quality: You can preserve general abilities by keeping the model’s behavior close to the original and by distilling from a frozen teacher. This keeps damage low while still adding knowledge.
- For future design: Models need better “addressing” or retrieval mechanisms for their written memories—ways to reliably find and combine stored facts on demand. Until that’s solved, prompts (context) remain the dependable tool for complex or long‑lived knowledge use.
Bottom line
- Broad, realistic practice creates knowledge the model can actually use.
- Many sequential “writes” make older facts hard to reach, even if they’re still stored.
- Context (putting facts in the prompt) stays reliable when you need to combine facts or keep them usable through many updates.
- To keep skills intact, anchor learning to a stable reference (a frozen teacher) and avoid drifting too far from the original model.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper uncovers important phenomena but leaves several concrete questions unresolved that future work could address:
- Generality across architectures and scales: Do the findings (entailment gap, retention patterns, access vs storage) hold for other model families (e.g., Llama/Mistral/GPT), larger scales, and different tokenizers?
- Beyond LoRA/full FT: How do other parameter-efficient methods (IA3, adapters, prefix/soft prompts, residual adapters, sparse/orthogonal adapters, MoE with frozen experts) affect write usability, reachability, and interference?
- Real-world vs synthetic facts: Do results transfer from invented, single-sentence facts to real facts drawn from dynamic domains (news, scientific updates), multi-sentence passages, and long-form sources?
- Scale of the knowledge base: What happens when writing thousands to millions of facts (orders of magnitude beyond 100)? Does the retention plateau (25–28%) change with scale or capacity?
- Compute/throughput confounds: Many method comparisons did not match token throughput or effective updates; can conclusions be replicated with strictly controlled compute and sample budgets?
- Study data attribution: Which specific components of “study” data (paraphrases vs QA vs worked implications vs contrasts) drive the reduction in the entailment gap and improved retention?
- Why study hurts recitation on recall/paraphrase: What mechanisms cause study-trained models to underperform bare-statement training on recall/paraphrase despite better use?
- Composition ceiling gap: Why do weight-based methods still trail the in-prompt ceiling on composition questions, and can targeted objectives close this gap without degrading capability?
- RL-based writing: The attempted GRPO variant failed to install facts; what RL objectives/reward designs (e.g., outcome-supervision, contrastive rewards, DPO-style losses) can reliably write new knowledge?
- On-policy distillation stability: Online distillation showed retention benefits but suffered degeneration (looping outputs). What stabilizers (e.g., conservative policy iteration, trust-region constraints, entropy regularization) prevent collapse?
- Cause of looping degeneration: Why do certain objectives (e.g., reverse-KL with LoRA on bare statements) induce repetitive looping? Is this due to sampling settings, EOS handling, reward hacking, or mode collapse, and how can it be prevented?
- Causal mechanism of capability loss: KL drift correlates with capability loss but is not a sufficient mechanism (penalty reduced damage without lowering KL). What representation-level metrics (e.g., FIM, layer-wise drift, subspace overlap) better explain and control damage?
- Address capture mechanism: The paper hypothesizes “question-keyed” failures and address capture, but provides no mechanistic localization. Which layers/heads/neurons implement the routing from questions to stored facts, and how do later writes redirect these routes?
- Interventions for reachability: No tested intervention preserved access to earlier facts. Can targeted constraints (sparse updates, orthogonal gradient projection, parameter partitioning, modular routing, key–value addressing, learned retrievers) prevent question-key capture?
- Consolidation failure for retention: Batch re-distillation into the original model preserved capability but not reachability. Why doesn’t re-distilling facts from their statements re-key access paths, and can consolidation with explicit query–key supervision fix this?
- Spaced revisiting: Does periodically revisiting earlier facts (spaced repetition of writes) improve reachability compared to one-pass sequential writing?
- Joint use binding problem: Two written facts could not be used together; self-retrieval of written facts was unreliable. Can auxiliary objectives (self-retrieval tasks, contrastive bindings, associative key supervision) or architectural changes enable robust internal retrieval and composition?
- Storage probe validity: Using the statement’s log-probability as a storage proxy may not capture the full written object. Can causal tracing or probe-based interpretability validate that the broader knowledge (implications, use) remains stored after forgetting?
- Counterfactual cue sensitivity: Counterfactual items were excluded from some endpoints due to cue sensitivity under test. How robust are conclusions across adversarial/cued phrasings, and can evaluation protocols be hardened?
- Distinctiveness vs prior conflict: Inverting facts survived better than neutral ones, but confounds (question difficulty) remain. Does distinctiveness/rarity causally protect retention when controlled for difficulty and writing strength?
- Relearning dynamics: Slow relearning of forgotten facts was observed on a tiny sample (n=3). Do larger studies confirm that address capture, not trace decay, drives relearning cost, and can “uncapture” procedures accelerate recovery?
- In-context robustness breadth: In-context use degraded less than held-out capability on fresh facts; does this hold for deeper multi-hop reasoning, multi-fact chains, and longer contexts?
- Hybrid memory strategies: Given context’s reliability, what are effective hybrids (external retrieval, memory buffers, tool calls) that enable composition and persistence while minimizing prompt length and latency?
- Safety and conflict handling: How do weight writes interact with pre-existing knowledge, especially when injected facts contradict world knowledge? Are there safe mechanisms for conflict resolution and rollback?
- Scheduling and curriculum: How do ordering, topical clustering, and curriculum design (e.g., interleaving related/unrelated facts) affect interference and retention?
- Capacity limits and measurement: Is the retention plateau due to model capacity, routing bottlenecks, or objective mismatch? Can capacity expansion (e.g., adding experts/parameters) or routing modularity lift the plateau?
- Alternative drift constraints: Can local penalties targeted at sensitive subspaces (e.g., layer-wise orthogonality, feature drift limits) preserve both capability and reachability better than global KL penalties?
- Broader evaluations: The capability suite (~100 rule-scored items) and main fact set (247 items) may not capture diverse skills/domains. Do results replicate on broader, standardized continual-learning benchmarks?
- Judge dependence and noise: Despite certification, judgments rely on a specific judge configuration (GPT-5.4-mini, dual policy). How sensitive are results to different judges, temperatures, and rubric variants, and can inter-annotator reliability be improved?
- Tokenization and prompt format effects: Do different tokenizers, prompt scaffolds, and instruction formats materially change the entailment gap or retention?
- Write integration methods: Adapter merging was used; are alternative integrations (e.g., additive merges, weight averaging with guards, low-rank fusion, gating) less prone to address capture?
- Long-horizon consolidation schedules: Only every-20-writes consolidation was tested. Do different consolidation frequencies or adaptive triggers improve retention without sacrificing capability?
- Minimal data for usability: What is the least study breadth needed to close the entailment gap for various fact types, and how does this trade off with interference risk?
- Realistic update streams: How do findings change under naturally correlated update streams (e.g., topical bursts over time) versus iid invented facts?
Practical Applications
Immediate Applications
Below are actionable use cases that can be deployed now, drawing directly from the paper’s findings on training-data breadth, retention, interference, and evaluation.
- Deploy retrieval over weight writes for dynamic facts
- Sectors: software, healthcare, finance, customer support, policy
- What to do: Prefer retrieval-augmented generation (RAG), prompts, or external memory for updates to facts (guidelines, regulations, prices) rather than continual fine-tuning.
- Why: In-context facts remain usable and compose, while weight-written facts lose reachability under later writes and don’t compose reliably.
- Tools/workflows: Knowledge base with embeddings; prompt templates that surface exact statements; cache critical facts per session.
- Assumptions/dependencies: Requires a maintained KB; latency budget for retrieval; access control for sensitive data.
- If you must fine-tune facts, use “study” data, not bare statements
- Sectors: software, healthcare, education, safety/compliance
- What to do: Build write datasets with diverse paraphrases, Q&A, worked implications, and contrasts (study data). Explicitly avoid training solely on bare statements.
- Why: Study data shrinks the recitation-to-use (entailment) gap and yields much higher retention after sequential writes than bare statements.
- Tools/workflows: Data generation pipelines that create paraphrases and implication chains per fact; small LoRA adapters; paired strict/lenient grading to verify use over recitation.
- Assumptions/dependencies: Synthetic data quality; prompt hygiene to avoid leakage from training phrasing.
- Anchor capability with a frozen teacher during distillation
- Sectors: software (foundation models), MLOps
- What to do: For batch or sequential updates, distill from a frozen copy of the original model (teacher) instead of a student that contains accumulated writes.
- Why: Preserves general capabilities (low KL drift), avoids degeneracy/looping, and maintains higher retention versus self-distillation through own merges.
- Tools/workflows: “Frozen-teacher” context distillation pipeline; scheduled teacher rebase and evaluation gates.
- Assumptions/dependencies: Access to the original model; compute for teacher inference; clear update cadence.
- Monitor KL drift as a leading indicator of capability loss
- Sectors: MLOps, software
- What to do: Track token-level KL divergence to the original model over a fixed held-out prompt set; set alert thresholds tied to capability degradation.
- Why: Capability loss correlates strongly with divergence from the original model; drift is predictive across objectives and update methods.
- Tools/workflows: Drift dashboards; CI/CD gating on KL and capability tests; rollback on exceedance.
- Assumptions/dependencies: Reliable held-out prompt pool; reproducible measurement; acceptance thresholds tuned to application risk.
- Use dual grading to separate recitation from use in evaluation
- Sectors: academia, MLOps, evaluation vendors, education
- What to do: Score with strict and lenient policies; track the entailment gap as an artifact-level property of a write.
- Why: A large gap indicates knowledge that stops at restating premises instead of producing conclusions; correlates with poor survival.
- Tools/workflows: Test sets covering recall/paraphrase/application/composition/counterfactual; multi-judge pipelines; pass/floor/ceiling certification.
- Assumptions/dependencies: Reliable judging; leakage audits; robust answer normalization (negation/units/numeric signs).
- Design updates around context-led composition of multiple facts
- Sectors: software, healthcare, finance, education
- What to do: When tasks require combining several facts, deliver all relevant facts in context. Do not expect the model to self-retrieve written facts.
- Why: Two weight-written facts rarely compose; self-retrieval fails; providing true statements in context restores accuracy.
- Tools/workflows: Multi-hop retrievers; prompt assembly that enumerates facts; composition-aware testing.
- Assumptions/dependencies: Retriever must find all relevant facts; prompt length limits; privacy constraints for sensitive facts.
- Recover “forgotten” facts by prompting with the statement
- Sectors: support, enterprise assistants, education
- What to do: If a previously written fact fails, include the statement explicitly in the prompt or memory window.
- Why: Behavioral forgetting reflects lost access, not erasure; supplying the statement restores performance to near-ceiling.
- Tools/workflows: Auto-remediation: on suspected overwrite, surface the fact; per-entity “fact cards” injected into prompts.
- Assumptions/dependencies: The system can identify the relevant fact to surface; UI budget for citations or tooltips.
- Safer fine-tuning defaults: cap optimization, detect loops, and use on-policy sampling
- Sectors: MLOps, software
- What to do: Use moderate steps; detect repetitive/looping outputs; prefer on-policy context distillation over offline at equal budgets.
- Why: Over-optimizing narrow statements yields recitation and instability; on-policy helps retention at similar strength.
- Tools/workflows: Early stopping on loop rate; decoding monitors; seed variance checks.
- Assumptions/dependencies: Decoding budget; instrumentation in inference harness.
- Productize “Fact Vault” and “Drift Guard”
- Sectors: software tooling
- What to do: Offer external memory “Fact Vault” (RAG) and “Drift Guard” (KL/capability monitors) as turnkey modules.
- Why: Immediate uplift from context-first knowledge and proactive capability protection; matches paper’s conclusions.
- Tools/workflows: APIs for insert/update/query; red-team prompts; periodic certification runs.
- Assumptions/dependencies: Customer data governance; secure storage; PII handling.
- Curriculum design and assessment for educational LMs
- Sectors: education
- What to do: Teach via “study” data (worked implications and contrasts), and assess with strict vs lenient scoring to track application, not just recall.
- Why: Mirrors improved “use” outcomes and better transfer than bare-statement exposure.
- Tools/workflows: Item banks spanning application/composition/counterfactual; analytics on entailment gap per concept.
- Assumptions/dependencies: Content coverage; fairness audits.
- Compliance/safety updates via context, with verifiable audits
- Sectors: healthcare, finance, policy, legal
- What to do: Publish compliance/safety changes as signed, retrievable facts; inject at inference with traceable provenance; avoid weight writes for time-critical updates.
- Why: Context channel is reliable under continued updates, and provenance supports audits.
- Tools/workflows: Signed KB entries; immutable logs; evaluation gates on counterfactual questions against prior defaults.
- Assumptions/dependencies: Organizational process; key management; audit trail policy.
- MLOps patterns for adapters without merges
- Sectors: software
- What to do: Mount per-domain/per-release adapters at inference via router/gating instead of merging; keep the base untouched.
- Why: Merging accumulates interference; non-merged routing limits cross-talk and preserves access via context.
- Tools/workflows: Adapter registries; request-time routing; fallback to base+RAG.
- Assumptions/dependencies: Latency/memory budget; clean routing signals (metadata, user intent).
Long-Term Applications
These require further research, scaling, or development to reach robust deployment.
- Addressable weight memory with explicit keys
- Sectors: software, research
- Idea: Architectures that bind facts to stable, queryable addresses so later writes cannot capture their routes (e.g., learned key-value slots, routing indices).
- Dependencies: New training objectives for key stability; interference-resistant routing; evaluation on multi-hop retention.
- Interference-aware write schedulers and per-fact isolation
- Sectors: MLOps, software
- Idea: Plan writes with collision risk estimates; dedicate modules/expert shards per fact/topic; use gating/MoE to prevent address capture.
- Dependencies: Reliable collision predictors; dynamic gating; cost of additional parameters.
- Retention-preserving consolidation that restores reachability
- Sectors: research, software
- Idea: Consolidation passes that do more than restore output distributions—explicitly re-key older facts to their questions post-write.
- Dependencies: New losses/constraints; supervision from retrieval traces; scalable consolidation.
- Causal/local-control methods that truly prevent overwrite
- Sectors: research
- Idea: Methods that use accurate local measurements to constrain updates without sacrificing later performance (beyond current linearized predictors).
- Dependencies: Better local-to-global behavior predictors; efficient constraints; proofs of stability.
- Industry-wide evaluation standards for continual knowledge writing
- Sectors: academia, standards bodies, policy
- Idea: Benchmarks with dual grading, five question types, floor/ceiling certification, leak audits, and sequential-write retention tracks.
- Dependencies: Community curation; judge transparency; standardized reporting (KL, entailment gap, retention over k writes).
- Secure update channels and governance policies
- Sectors: policy, security, enterprise
- Idea: Regulate update mechanisms (weights vs context), require drift/retention reporting, and mandate provenance for critical facts.
- Dependencies: Regulatory adoption; interoperable audit formats; secure KB infra.
- Safety against adversarial “address capture”
- Sectors: security, safety
- Idea: Detect and mitigate writes that hijack question routes (poisoning); quarantine and roll back harmful adapters.
- Dependencies: Poisoning detectors tied to wrong-answer content bleed; sandboxed update pipelines.
- Personalized assistants with private external memory
- Sectors: consumer software, education
- Idea: Keep user facts in encrypted, per-user memory and rely on context injection instead of weight writes; portable across base-model updates.
- Dependencies: Privacy-preserving retrieval; device/on-prem storage; consent and retention policies.
- Healthcare-grade update pipelines
- Sectors: healthcare
- Idea: Clinically validated pipelines where guideline changes are context-fed with strong provenance; periodic frozen-teacher distillation for domain style but not facts; continuous KL/capability monitoring.
- Dependencies: Clinical validation; liability frameworks; integration with EHRs and formularies.
- Finance/regulatory copilots with auditable fact channels
- Sectors: finance, legal
- Idea: Systems where regulatory changes are treated as signed, queryable facts; composition tested via strict evaluation; explainable citations alongside responses.
- Dependencies: Regulator-backed schemas; audit automation; real-time updates.
- Robotics and embedded LMs: operational updates via context
- Sectors: robotics, IoT
- Idea: Deliver new procedures/constraints as context packets rather than firmware-like weight writes; schedule frozen-teacher style updates for style/format only.
- Dependencies: Reliable on-device memory; bandwidth constraints; safety certs.
- Tooling products: Write-Safe SDK
- Sectors: developer tools
- Idea: SDK that unifies Fact Vault (RAG), Frozen-Teacher Updater, KL Drift Monitor, Entailment Gap Evaluator, Looping Detector, and Retention Runner (k-write tests).
- Dependencies: Vendor integrations; standardized model interfaces; cost management.
Key Assumptions and Dependencies (cross-cutting)
- Findings are measured on Qwen3-4B/8B with LoRA and specific objectives; behavior may differ at larger scales or with different architectures.
- Evaluation integrity depends on dual grading, judge reliability, and leakage/truncation controls; porting requires replicating certification steps.
- Frozen-teacher availability and compute budgets matter for distillation strategies.
- Organizational readiness for MLOps gating, monitoring (KL/capability), and provenance is required to realize benefits.
Glossary
- adapter: A small trainable module added to a fixed base model to implement updates without modifying all parameters. "Written into the weights, each fact trains an adapter that is merged before the next fact arrives"
- adapter rank: The low-rank dimensionality used in LoRA adapters that controls their capacity. "Each method writes the same 40 certified facts into a fresh rank-16 adapter for 192 steps"
- address capture: The phenomenon where later updates redirect the queries that previously reached stored knowledge, making earlier facts hard to retrieve. "and it is what address capture predicts: retraining must displace the newer content that now owns the fact's questions, not restrengthen a faded trace."
- batch distillation: Training a student model on many facts at once using a fixed teacher to match its output distribution. "Batch distillation uses the original model as teacher, while these sequential conditions use the current merged model"
- bootstrap confidence interval: A statistical uncertainty estimate computed by resampling the data. "Uncertainty is a bootstrap 95\% confidence interval over facts"
- catastrophic forgetting: The abrupt loss of previously learned knowledge after training on new data. "This turns catastrophic forgetting into a property of the written object"
- composition (question type): Tasks that require combining a learned fact with unstated background knowledge. "composition combines the fact with unstated world knowledge"
- context distillation: Training a student to imitate a teacher model that sees extra context (e.g., a fact in the prompt), so the student can perform without that context at inference. "We therefore compare bare-statement and study training with offline context distillation and online context distillation in both KL directions."
- counterfactual (question type): Questions that force choosing between a newly written fact and the model’s default belief. "counterfactual questions force a choice between the written fact and the model's default."
- difference-in-differences: A comparative statistic measuring how much one change differs from another between two conditions. "the difference-in-differences is points"
- distributional drift: A shift in the model’s output distribution away from its original behavior, often measured by KL divergence. "distributional drift predicts forgetting under further training"
- drift control: A baseline measurement used to subtract general drift effects when probing storage, isolating the specific impact of a write. "A drift control, the same quantity computed on never-written statements, corrects for the elevation that any statement receives from the surrounding writes."
- dual grading policy: An evaluation scheme that scores answers under both strict and lenient criteria to distinguish recitation from use. "The dual grading policy and entailment gap"
- entailment gap: The difference between lenient and strict accuracy, quantifying how often a model states a premise without drawing the required conclusion. "The entailment gap is 22--42 points on bare-statement use questions"
- forward KL: The KL divergence evaluating how well the student covers the teacher’s distribution, . "in the forward direction ()"
- frozen teacher: A fixed reference model used during distillation to stabilize training and preserve capabilities. "A frozen teacher writes facts at near-zero cost"
- GRPO: A reinforcement-learning optimization method akin to PPO used here with programmatic rewards. "A GRPO condition with a programmatic answer-match reward received almost no signal"
- KL divergence: A measure of difference between probability distributions, used here to quantify drift from the original model. "Damage to unrelated abilities tracks KL divergence from the original model"
- linearised predictor: A model that approximates the immediate effect of parameter updates linearly, used to predict interference. "a linearised predictor captures the immediate effect of an update () but not the full trajectory"
- log-probability lift: The increase in log-probability assigned to a statement after writing it into the model’s weights. "keep 57--67\% of their write's drift-corrected log-probability lift"
- LoRA: Low-Rank Adaptation, a parameter-efficient fine-tuning method that inserts low-rank adapters into a frozen base model. "All experiments use Qwen3-4B with a frozen base and LoRA adapters unless stated otherwise"
- mode-seeking reverse KL: Using , which tends to concentrate on high-probability modes of the teacher; hypothesized (and here refuted) to create more committed knowledge. "Our prediction that mode-seeking reverse KL would create more committed knowledge fails"
- offline distillation: Matching the teacher on sequences sampled once from the teacher, without on-policy sampling by the student. "Offline distillation matches the student to this distribution on sequences sampled once from the teacher"
- on-policy distillation: Distillation where the student samples its own outputs and is trained using KL divergence to the teacher, often in reverse or forward directions. "The reverse direction is the on-policy-distillation objective of \citet{lu2025onpolicy}."
- online distillation: Distillation that samples from the student during training, using token-level KL to the teacher at each step. "Online distillation samples and uses the same token-level divergence in the forward direction"
- prior-conflict evaluation: An assessment using facts that either align with or contradict the model’s prior beliefs to test retention. "Retention on the prior-conflict evaluation."
- prior-inverting (fact): A fact that contradicts the model’s prior, used to test distinctiveness and retention. "prior-neutral and prior-inverting facts"
- reverse KL: The KL divergence , emphasizing matching the student’s mass to the teacher’s peaks. "or reverse direction ()"
- self-distillation: Distilling a model into itself (or its updated versions), which can compound drift and destabilize training. "frozen teachers are stable and self-distillation diverges"
- sequential writing: Writing facts into the model one after another, assessing how later writes affect earlier ones. "after twenty sequential writes"
- Spearman ρ: A rank correlation coefficient used to measure monotonic relationships between variables. "Across 45 such pairs, Spearman "
- student-sampled sequences: Outputs generated by the student during online distillation, which can improve performance over fixed teacher samples. "Student-sampled sequences buy seven points over fixed teacher samples"
- supervised fine-tuning (SFT): Training that maximizes the likelihood of target text given input prompts, using labeled examples. "Writing a fact with supervised fine-tuning maximises the likelihood of target text given prompt "
- token-level divergence: KL divergence computed at each token step between teacher and student distributions during distillation. "uses the same token-level divergence in the forward direction"
