--- title: 'FACTS Parametric: Factual Memory in LLMs' url: https://www.emergentmind.com/topics/facts-parametric type: topic --- # FACTS Parametric: Factual Memory in LLMs A “parametric” fact, or “parametric knowledge,” refers to explicit factual information that is stored directly inside a model’s parameter tensors as a consequence of pre-training or supervised fine-tuning, rather than in an external memory or via retrieval mechanisms. In natural language modeling, parametric recall is evaluated by presenting the model closed-book factoid questions: if it can produce the correct answer purely from inference over its weights—without any access to search engines, databases, or tool-augmented context—then the fact is deemed stored parametrically. The evaluation of parametric factuality has become critical for understanding the internal world knowledge of language models and for benchmarking their ability to recall such knowledge reliably. Distinguishing parametric from in-context or retrieval-based knowledge is foundational in both NLP evaluation and neural knowledge systems. ## 1. Definition and Conceptual Distinctions Parametric knowledge is defined as factual information (such as the capital of a country, an individual’s date of birth, or a scientific constant) that has been directly encoded in the persistent weights of a model through self-supervised pre-training, supervised learning, or related processes. In contrast, non-parametric (retrieval-based) knowledge is stored outside the model (e.g., in a database or document corpus) and is only accessible at inference time through explicit lookup or retrieval modules [2508.09494], [2406.12277]. In formal evaluation settings, a fact ⟨subject, relation, object⟩ is classified as parametrically stored if the model, when presented with a factoid prompt in the absence of any external retrieval, produces the correct object with high accuracy. Typical queries are cloze-style (e.g., “[X] is the capital of [MASK]”) or direct questions (“who is the president of France?”) [2406.12277]. Parametric memory therefore refers to the set of all facts and regularities that are implicitly encoded into the model parameter space and can be recalled purely through the forward pass of the network without augmentation or search [2508.09494], [2512.10791]. ## 2. Benchmarks and Empirical Measurement Parametric factuality is systematically measured by specialized benchmarks designed to stress-test the closed-book knowledge of large language models (LLMs). The most prominent of these is FACTS Parametric, a sub-benchmark in the FACTS Leaderboard suite, which evaluates LLMs' ability to answer atomic, single-fact questions covering diverse topics relying solely on parameters [2512.10791]. ### FACTS Parametric Benchmark | Size | 2,104 QA pairs (split evenly public+private) | |--------------|----------------------------------------------------------| | Source | User traffic, adversarial filter, Wikipedia-verified | | Coverage | Politics, sports, science, arts, “long tail” topics | | Judging | Gemini 2.5 Pro, three evaluations per item, closed-book | Each response is automatically labeled as correct, incorrect, not-attempted (“I don’t know”), or unknown. Primary metrics include accuracy, hedging rate, attempted accuracy, and F1-score: - $\text{Accuracy} = \frac{\# \text{correct}}{N} \times 100\%$ - $\text{HedgingRate} = \frac{\# \text{not-attempted}}{N} \times 100\%$ - $\text{AttemptedAccuracy} = \frac{\# \text{correct}}{\# \text{correct} + \# \text{incorrect}} \times 100\%$ - $F_1 = \frac{2 \cdot \text{Accuracy} \cdot \text{AttemptedAccuracy}}{\text{Accuracy} + \text{AttemptedAccuracy}}$ FACTS Parametric’s adversarial curation ensures that questions answered correctly by any open-weight model are filtered out, focusing on the genuinely challenging “long tail” of world knowledge [2512.10791]. Other notable multi-faceted parametric knowledge probes include BELIEF and BELIEF-ICL [2406.12277], which assess accuracy, consistency, and confidence calibration of the parametric recall using highly diversified, multi-prompt templates. ## 3. Dynamics of Parametric Knowledge Acquisition Large-scale studies on factual learning dynamics in LLMs have shown that encoding facts into parametric memory follows a multi-phase trend [2503.21676]. During pre-training or fine-tuning: 1. **Phase I (Statistics learning):** The model rapidly captures distributional statistics of attributes (e.g., common locations, frequent professions). 2. **Phase II (Plateau):** Learning stalls at this plateau as attention-based extraction circuits assemble, but individualized fact recall accuracy remains negligible. 3. **Phase III (Knowledge emergence):** Once circuits form, the model rapidly acquires specific entity-relation bindings, and fact recall surges. The length of Phase II—the “parametric bottleneck”—is primarily determined by the distribution of fact frequencies in the training data. Highly imbalanced (“Zipfian”) data lead to faster acquisition for head entities but slow for tail entities. Curriculum strategies, such as two-stage warmup focusing on subsets, reduce plateau length and improve tail fact learning. A plausible implication is that data scheduling and diversity are as important as model capacity for robust parametric factuality [2503.21676], [2508.09494]. ## 4. Model Scale, Data, and Training Methodology Parametric factual recall increases nearly monotonically with model parameter count and training data volume, but both scaling and data quality matter. Experimental evidence from BELIEF-ICL [2406.12277] shows that: - Larger models (e.g., Llama3-70B) exhibit higher Acc@1 and consistency in parametric recall relative to smaller counterparts (e.g., Llama2-7B). - Pretraining on larger, more diverse, and educationally rich corpora enhances fact encoding, especially for low-frequency facts. For example, the Phi3-mini (3.8B), trained on textbook-style data, surpasses Llama2-7B despite its smaller size. - Training strategies (e.g., whole-word masking in BERT_wwm) and prompt diversity are also significant factors. Active Reading, a two-stage data augmentation pipeline (strategy generation + synthetic example instantiation), substantially improves parameteric knowledge absorption relative to vanilla document repetition or shallow augmentation. Task-specific Active Reading yields up to +793% relative accuracy gains over base finetuning on SimpleWikiQA and +160% on FinanceBench [2508.09494]. At pre-training scale, applying Active Reading to 6 million Wikipedia articles (1 trillion tokens), produces the Meta WikiExpert-8B, which outperforms models with orders-of-magnitude more parameters on closed-book QA [2508.09494]. ## 5. Editing, Hallucinations, and Robustness Parametric memory is notably difficult to edit. Fine-tuning to inject new facts into an LLM’s weights typically causes **parametric memory corruption**, erasing unrelated facts (catastrophic forgetting), and is accompanied by a “slow acquisition” effect for new facts [2503.21676]. Mechanistically, corruption affects model feed-forward (MLP) “key–value” memories, while attention patterns remain relatively stable. Stubborn knowledge—facts for which the model has high pre-training confidence—tends to resist “in-context editing” (ICE) approaches: even if ICE boosts the new fact’s probability, the original fact remains overconfident, hampering successful editing [2405.11613]. The DeCK (Decoding by Contrasting Knowledge) framework addresses this by jointly comparing ICE-augmented and unedited logits at each decoding step, contrastively down-weighting stubborn original parametric knowledge. DeCK improves stubborn fact editability by up to 219% on standard benchmarks and shifts probability mass for edited facts from <0.2 to >0.8 [2405.11613]. Hallucinations in parametric models—confident predictions for unseen or fabricated inputs—arise simultaneously with the ability to recall learned facts. Specifically, as the attention extraction circuit forms during training, hallucinations on held-out entities increase [2503.21676]. This suggests a fundamental limitation: robust parametric recall is inherently accompanied by hallucination risk for out-of-distribution queries. ## 6. Prompt Engineering and Probing Methodologies Prompt diversity and context are critical for eliciting parametric facts. BELIEF-ICL demonstrates that using multiple, diverse templates for each fact (prompt ensembling) and providing in-context exemplars (especially from the same relation/template) dramatically increase recall accuracy (e.g., Llama3-8B: 0.34 zero-shot → 0.73 with 4-template ICL) [2406.12277]. There is a pronounced “coverage gap”: no universal prompt elicits all parametric knowledge, and fact recall can increase by over 30–66 percentage points if the best prompt for each specific fact is selected (oracle accuracy). Prompt-based probing demands large-scale prompt sets to avoid underestimating model knowledge [2406.12277]. This motivates automated prompt diversification (e.g., MyriadLAMA: ~100 templates per relation) and recommends clustering or optimization to achieve robust, tractable evaluation. ## 7. Limitations, Extensions, and Hybrid Approaches The principal limitations of parametric factual memory are: - **Catastrophic Overwrite:** Fine-tuning for new facts degrades previously stored facts without replay/interleaving, due to feed-forward memory interference [2503.21676]. - **Hallucination Risk:** Hallucinations increase for entities/facts not present in pre-training data, especially as attention circuits become specialized [2503.21676]. - **Stubbornness:** Highly confident (frequent) facts are resistant to editing via context; advanced contrastive decoding (e.g., DeCK) provides partial remediation [2405.11613]. - **Prompt Sensitivity:** Factual recall is highly sensitive to prompt design; covering the full knowledge base requires extensive prompt engineering [2406.12277]. Hybrid strategies, such as integrating external retrieval with a stable attention backbone or using retrieval-augmented ICE, are suggested to overcome the limitations of purely parametric knowledge—especially for dynamic, long-tail, or time-sensitive facts [2508.09494], [2406.12277], [2503.21676]. Advanced data scheduling, lifelong/active reading frameworks, and dynamic parametric-retrieval blending represent emerging directions for robust factual knowledge modeling. --- **References:** - [2512.10791] The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality - [2406.12277] What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models - [2503.21676] How do language models learn facts? Dynamics, curricula and hallucinations - [2508.09494] Learning Facts at Scale with Active Reading - [2405.11613] Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts Source: https://www.emergentmind.com/topics/facts-parametric