INFUSER: Factual Knowledge Infusion in LLMs

• INFUSER is a document-centric method that infuses factual knowledge into LLMs via fine-tuning on prompts augmented with retrieved entity-centric sentences.
• It integrates unstructured text evidence without modifying base architectures, offering an alternative to large-scale pre-training or adapter modules.
• Experimental results show marked improvements (e.g., Hits@1 up to 0.914) and highlight its effectiveness in low-resource and domain-specific settings.

INFUSER, short for INfusing FactUal conteXt via Sentence pRompting of Entities, is a knowledge-infusion method for LLMs that injects domain-specific or up-to-date factual information directly from unstructured text rather than from knowledge graphs or large-scale additional pre-training. The method constructs contextual prompts by retrieving sentences surrounding entities mentioned in an input instance, prepending those sentences during fine-tuning, and then updating the model with a standard sequence-to-sequence objective. In the formulation introduced for Flan-T5-based experiments, retrieval is used during training but omitted at inference, so the factual content is expected to be absorbed into the model parameters rather than supplied through a runtime retrieval pipeline (Vasisht et al., 2024).

1. Definition and conceptual position

INFUSER is presented as a simple, general-purpose recipe for endowing pre-trained LLMs with factual knowledge from document corpora. Its central claim is methodological rather than architectural: knowledge infusion can be accomplished by ordinary fine-tuning on prompts augmented with retrieved natural-language context, without adapters, additional attention modules, or knowledge-graph-specific preprocessing (Vasisht et al., 2024).

This design places INFUSER in contrast with two common strategies for domain adaptation. One is additional pre-training over large corpora or graph-derived triples; the other is prompt-based or soft-prompt-based knowledge injection. INFUSER instead treats relevant sentences from a domain corpus as explicit training-time evidence. This makes the approach applicable when structured knowledge graphs are unavailable, incomplete, or poorly suited to the entities of interest. The paper explicitly emphasizes that this is useful for entities that are usually not found in any knowledge graph.

A common misconception is to interpret INFUSER as a retrieval-augmented generation system in the usual inference-time sense. The workflow described for INFUSER does not retrieve context at inference time. Retrieval is used to build training examples, after which the model is queried using only the instruction and task text. This suggests that the method is closer to factual parameter updating through supervised fine-tuning than to classical test-time RAG.

2. Prompt construction and training objective

The method begins from a base task prompt $x=(I,T)$, where $I$ is a brief instruction and $T$ is the instance text. For triple completion, $T$ may be a subject–predicate pair with the object masked. From $T$, the system identifies the set of mentioned entities $E=\{e_1,\ldots,e_m\}$ and applies a retrieval function $R(e)$ that returns a small set of sentences or paragraphs from a pre-indexed domain corpus.

The retrieved evidence is concatenated into a single context string $C=[c_1 \,\unicode{x29FA}\, \ldots \,\unicode{x29FA}\, c_k]$, and the INFUSER prompt is formed by simple concatenation:

$P(x)=I \,\unicode{x29FA}\, T \,\unicode{x29FA}\, C.$

The paper also gives the equivalent notation

$P(x) \;=\; [\,I; \;T; \;\bigcup_{e\in E}R(e)\,].$

Training then uses the ordinary sequence-to-sequence objective. If $I$0 denotes the target output, the loss is

$I$1

Gradients flow through both the original prompt tokens and the retrieved context tokens. In the authors’ interpretation, this is the mechanism by which new factual signals are infused into the model weights (Vasisht et al., 2024).

The retrieval layer is intentionally generic. The paper states that any off-the-shelf search index may be used, with ElasticSearch, FAISS, and Lucene given as examples. Because the approach relies on retrieved natural-language evidence rather than canonicalized triples, it can operate over corpora such as legal judgments, product manuals, or biomedical papers, provided those corpora contain relevant entity-centric statements.

3. Architectural characteristics

INFUSER leaves the base LLM architecture unchanged and performs full fine-tuning. The described instantiation uses Flan-T5, but the method is framed as applicable to “any autoregressive/encoder-decoder LLM.” The architecture receives no dedicated knowledge-fusion head, no new attention layers, and no adapters (Vasisht et al., 2024).

The paper examines one optional variant: a per-entity soft-prompt matrix $I$2 trained in an adapter-like fashion. This variant operates in embedding space rather than through discrete retrieved text. Empirically, however, it is reported to be less potent than appending real sentences. That contrast is important because it identifies the core operative choice in INFUSER: discrete contextual evidence is not merely a convenience for conditioning, but the preferred mechanism for factual infusion.

This architectural minimalism has two consequences. First, INFUSER is operationally easy to reuse because it reduces the intervention to prompt construction and standard fine-tuning. Second, its gains must be interpreted as arising from supervision design rather than from additional representational capacity. A plausible implication is that the method tests how far prompt-level factual exposure can substitute for more specialized knowledge-integration modules when the downstream task is sufficiently aligned with the retrieved evidence.

4. Experimental evaluation

The evaluation spans relation prediction, closed-book question answering, and a domain-specific legal setting. For relation or tail prediction, the paper uses KELM-TEKGEN, described as 1 M samples from WikiData, TACRED, and Re-TACRED. For closed-book QA, it uses TriviaQA on the rc-wikipedia split, with fine-tuning first on KELM triples. For domain specificity, it evaluates on a small legal KG built from Indian court judgments (Dhani et al. 2021) (Vasisht et al., 2024).

For triple completion, the reported metrics are Hits@K for $I$3, Mean Reciprocal Rank (MRR), and Approximate Edit Distance (AED) between predicted and gold graphs. For question answering, the metric is Exact-Match (EM) percentage.

The headline quantitative results are summarized below.

Setting	Baseline	With INFUSER
Flan-T5-Small Hits@1	0.528	0.800
Flan-T5-Base Hits@1	0.514	0.825
MRR	$I$4	$I$5
AED	$I$6	2.75
TriviaQA EM	16.7%	20.0%

The paper also reports an ablation on soft prompts, where learned continuous prompts achieve 0.176 Hits@1, while discrete retrieved context achieves 0.710 Hits@1. A context-length ablation shows that longer contexts further improve Hits@1, from 0.798 to 0.836 for Flan-T5-Small and from 0.846 to 0.914 for Flan-T5-Base. In the legal-domain test, the gain is smaller but consistent, with Hits@10 increasing from 0.245 to 0.290 and MRR from 0.200 to 0.217 (Vasisht et al., 2024).

These numbers establish that the method is strongest on factual completion settings closely tied to retrieved entity evidence, while still yielding a modest improvement in closed-book QA. The legal-domain result is particularly significant because it uses a specialized corpus rather than a broad public benchmark.

5. Empirical interpretation and error profile

The ablation findings indicate that inserting more sentences per entity steadily improves performance. The authors interpret this as evidence that the model is genuinely learning facts rather than merely overfitting to a prompt template (Vasisht et al., 2024). Because performance rises with additional retrieved evidence, the mechanism appears sensitive to the informational density of the contextual prompt rather than only to the presence of any auxiliary text.

The soft-prompt comparison reinforces this reading. Since continuous prompts underperform discrete retrieved sentences, the experiments suggest that semantically explicit textual evidence is more effective than learned entity-conditioned embedding perturbations in this setting. This is an empirical observation from the reported ablation, not a universal claim about all prompt-learning methods.

The authors also note that low-resource or tail entities benefit disproportionately, especially those missing from standard knowledge graphs. This suggests that INFUSER is particularly well matched to factual regimes where curation lags behind real-world documentation. A plausible implication is that the method is most useful when the authoritative source of truth exists primarily in documents rather than in a normalized triple store.

Another important interpretive point concerns inference. Because no retrieval is supplied at test time, the evaluation probes whether the fine-tuned model has internalized the infused information. The paper explicitly states that the fine-tuned LLMs are evaluated by probing. This makes the framework relevant to research questions about parametric knowledge acquisition, retention, and transfer, rather than only to system-level retrieval performance.

6. Limitations, operational considerations, and reuse

The paper identifies several limitations. It studies full fine-tuning only, with no parameter-efficient fine-tuning adapters. It does not provide an end-to-end evaluation of knowledge propagation or catastrophic forgetting, and retrieval quality remains a single-point dependency (Vasisht et al., 2024). These constraints are consequential. If retrieval fails to surface the right sentences, the training examples will inject weak or misleading evidence; if full fine-tuning is impractical, the reported recipe does not directly establish that the same gains transfer to PEFT settings.

The practical recommendations are correspondingly conservative. Corpus preparation should focus on assembling a document collection that covers the relevant entities or concepts. Retrieval should map entity mentions to surrounding sentences, with precision more important than recall. Instructions should remain brief and consistently formatted, and the retrieved context should follow the task text. The reported heuristic is that 1–3 sentences per entity strikes a good balance. When context length exceeds model limits, the paper suggests chunking or moving to soft-prompt embeddings. If a partial knowledge graph exists, hybrid retrieval (KG lookup + document sentences) may yield further gains.

The resource profile is also intentionally modest. The paper states that INFUSER runs on a single mid-range GPU (e.g. A4000/16 GB) using standard mixed precision and gradient checkpointing. This operational footprint is consistent with the method’s broader characterization as a lightweight alternative to graph construction, large-scale re-pretraining, or specialized prompt-engineering pipelines.

In aggregate, INFUSER defines a document-centric form of knowledge infusion in which factual learning is induced by concatenating retrieved entity-level evidence to ordinary supervised examples during fine-tuning. Its main significance lies in showing that a standard encoder-decoder LLM can absorb substantial new factual content from unstructured text with no architectural modification, while achieving strong improvements on relation prediction and measurable gains in closed-book QA and legal-domain knowledge tasks (Vasisht et al., 2024).