Knowledge-Infused Multimodal Models

• Knowledge-infused multimodal models are systems that integrate external structured knowledge with visual and linguistic data to enhance reasoning and reduce hallucination.
• They employ techniques such as graph-based infusion, retrieval augmentation, and reinforcement learning to fuse multimodal data and dynamic context effectively.
• These models demonstrate improved performance in tasks like VQA, analogical reasoning, and time-sensitive knowledge retention across various specialized domains.

Knowledge-infused large multimodal models constitute a paradigm in which deep neural architectures are equipped with explicit, structured, or external sources of knowledge to enhance reasoning, grounding, and generalization across visual and language modalities. This approach addresses two persistent limitations observed in large (multimodal) LLMs (LMMs, MLLMs): hallucination—confident but incorrect generation in unfamiliar or knowledge-intensive contexts—and static, outdated parametric knowledge representations. By tightly integrating structured facts, multimodal knowledge graphs, policy-level constraints, or nonparametric retrieval with language–vision pipelines, researchers systematically expand the range, reliability, and interpretability of such systems.

1. Architectures and Mechanisms for Knowledge Infusion

A spectrum of techniques underpins knowledge-infused large multimodal modeling:

• Explicit Graph-based Infusion: Methods like MR-MKG incorporate Multimodal Knowledge Graphs (MMKGs) via a Relation Graph Attention Network (RGAT) and lightweight adapters. MMKGs encode entities, images, and attributes; at runtime, relevant subgraphs are retrieved and fused with visual and linguistic embeddings to form an augmented prompt for the (frozen) LLM, with cross-modal alignment enforced through triplet losses (Lee et al., 2024). The architecture is illustrated by:

$\text{prompt} = [ H'_K \oplus H'_I \oplus H_T ]$

where $H'_K$ are knowledge-adapted KG embeddings, $H'_I$ are visual-adapted image features, and $H_T$ are token embeddings.

• Retrieval-augmented Generation (RAG): Both multimodal retrieval (e.g., text/image retrieval, as in AKGP-LVLM (Perry et al., 15 Jan 2025) and CaMML (Chen et al., 2024)) and specialized MMKG-RAG pipelines supply contextually selected external facts to LMMs, avoiding parametric forgetting while improving generalizability.
• Policy-level Knowledge Infusion via RL: Reinforcement learning frameworks such as Vision-EKIPL expand the exploration boundary by integrating policy rollouts not just from the target MLLM, but also from high-quality external "expert" models (e.g., GPT-4o, Gemini-1.5-Pro). These methods apply group-level advantage estimation and gradient updates over pooled action groups, yielding notably faster convergence and higher ceilings in reasoning benchmarks (Wang et al., 7 Jun 2025).
• Domain-invariant and Constraint-based Fine-tuning: Domain knowledge is encoded at the optimization level in RL through invariant-support distributions and policy-level constraints (e.g., requiring output invariance to geometric transformations—rotation, symmetry, etc.—in remote sensing or medical MLLMs) (Cao et al., 23 Jan 2026).
• Parametric Memory and Modality-aware Routing: Vision-Enhancing LLMs such as MKS2 store visual knowledge internally using modular visual memory, injected as lightweight FFN blocks at each transformer layer. A soft mixture-of-experts routing selects between visual and linguistic processing per-token, thus enabling the model to reason about unseen queries via stored, not just retrieved, visual knowledge (Li et al., 2023).

2. Knowledge Graphs and Multimodal Representations

Multimodal Knowledge Graphs (MMKGs), which extend traditional textual knowledge graphs by incorporating real images, attributes, and cross-modal semantic relations, are foundational for knowledge infusion:

• Graph Construction: MMKGs are derived from aggregating facts in sources such as FreeBase, DBpedia, or YAGO, with image nodes contributed via large-scale web scraping and linking entities to ∼36 images each (Lee et al., 2024). For specialized domains (e.g., agriculture (Wang et al., 2024), time series (Sun et al., 13 Aug 2025)), custom ontologies and task-specific attributes are incorporated.
• Encoding and Fusion: RGATs, cross-modal transformers, and dynamic gating adaptively project KG node embeddings into the LMM's feature space. Fine-grained adapters (e.g., FKA in CVLM (Li et al., 2024)) extract region-specific knowledge, while contrastive or alignment losses (triplet, InfoNCE) enforce multimodal correspondence.

The resultant architectures enable fusion of grounded subgraph knowledge with both visual and linguistic streams, resulting in improved factual consistency and analogical reasoning.

3. Knowledge Injection Strategies: Adaptation, Retention, and Continual Learning

Practical knowledge infusion necessitates careful design to balance adaptation to novel knowledge and retention of previous capabilities:

• Supervised Fine-tuning (SFT), LoRA, and Adapter-based Updates: Empirically, naively fine-tuning all model parameters (Full-FT) or adopting parameter-efficient adapters without constraints leads to catastrophic forgetting, especially in instruction-following and dialogue skills (Jiang et al., 30 May 2025). Mixture-of-Experts LoRA (MoELoRA) and replay mechanisms—where small buffers of pretraining data are interleaved—significantly mitigate forgetting in evolving-knowledge settings.
• Structured Augmentation (KORE): Knowledge-oriented augmentations—dialogue-style expansion, multimodal captioning, and VQA generation for each new knowledge item—maximize knowledge coverage. Null-space projection of LoRA adapters, guided by covariance statistics of old data, constrains weight updates to directions orthogonal to previously stored knowledge, yielding superior adaptation-retention tradeoffs (Jiang et al., 22 Oct 2025).
• Retrieval-Augmentation and Dynamic Context: Approaches such as MM-RAG (both text and image-based) index evolving knowledge pools, returning the most relevant items as external context at inference-time. Dynamic selection (rather than fixed-context size) has been shown to outperform fixed-top-K fetch, especially for knowledge-based VQA (Jhalani et al., 2024).

4. Robustness, Benchmarking, and Application Domains

Sophisticated benchmarks have been introduced, precisely measuring the capability of knowledge-infused MLLMs:

• Dynamic and Temporal Knowledge: EVOKE focuses on evolving knowledge adaptation and retention in real-world conditions, demonstrating no method currently exceeds ∼56% accuracy and that catastrophic forgetting remains pervasive outside of continual/replay hybrid solutions (Jiang et al., 30 May 2025). MINED probes time-sensitive knowledge, temporal reasoning, and robustness, providing a multi-axis diagnostic and a suite of editing techniques (in-context, parameter-editing, memory-based) for updating stale facts (Jiang et al., 22 Oct 2025).
• Amodal Completion and Causal Reasoning: Models such as AmodalCG explicitly fuse MLLM-derived knowledge about physical object continuity—invoked only for high-occlusion cases—with diffusion-based inpainting, substantially increasing accuracy on amodal segmentation tasks (Yun et al., 30 Mar 2026). In time series, TimeMKG aligns textual variable semantics, a constructed multivariate knowledge graph, and raw numerical patterns via a cross-modal transformer, elevating forecasting and interpretability (Sun et al., 13 Aug 2025).
• Specialized Domains: Agri-LLaVA demonstrates significant improvements in pest/disease diagnosis by incorporating staged feature-alignment and dialogue training grounded in agricultural knowledge bases (Wang et al., 2024). Domain-aware RL constraints outperform prompt-based domain knowledge injection in scientific imaging (Cao et al., 23 Jan 2026).

5. Quantitative Evaluation and Ablation Insights

Knowledge-infused MLLMs consistently outperform their non-knowledge-augmented counterparts in knowledge-centric benchmarks:

Model/Method	Benchmark/Task	Performance Gain	Citation
MR-MKG (FLAN-T5-11B)	ScienceQA	92.78% (+1.10pp vs UnifiedQA-L)	(Lee et al., 2024)
MR-MKG (Visual-LLaMA-2 7B)	MARS (Hits@1)	0.405 (+10.4pp vs SOTA)	(Lee et al., 2024)
CaMML-13B	ScienceQA	92.03% (+1.13pts over LLaVA-13B)	(Chen et al., 2024)
CVLM (7B)	Knowledge-VQA avg	57.8% (+4.8pts vs LLaVA-1.5)	(Li et al., 2024)
AKGP-LVLM	OK-VQA	41.82% (+1.47pts vs GKN)	(Perry et al., 15 Jan 2025)
ELMM	FB15k-237-IMG Hits@1	34.1 (+7.2pts over LAFA)	(Huang et al., 19 Oct 2025)
KORE (LLaVA-7B, rank=235)	EVOKE CEM	30.65 (+15.4pts over LoRA)	(Jiang et al., 22 Oct 2025)
MKS2-Llama-2-7b	Avg. text reasoning	54% (+7pts over Llama-2-7b-chat)	(Li et al., 2023)

Ablation studies universally show that omission of knowledge-aligned adapters, fusion mechanisms, or external context yields significant drops in performance (e.g., in CVLM, −1.1 to −3.2pts per module (Li et al., 2024); in ELMM, MVTC removal yields −6.2 Hits@1 (Huang et al., 19 Oct 2025)). Optimal triplet count for MMKG is ∼10–20, with diminishing or negative returns at higher cardinality due to noise (Lee et al., 2024).

6. Challenges, Limitations, and Future Directions

Despite advances, significant challenges persist:

• Catastrophic Forgetting and Scalability: SFT and naive adapter-based tuning rapidly degrade instruction and dialogue capabilities; scalable replay and null-space projection strategies are required for real-world evolving knowledge (Jiang et al., 30 May 2025, Jiang et al., 22 Oct 2025).
• Dynamic and Time-sensitive Knowledge: MINED reveals persistent deficits in updating and retaining time-varying facts, with parameter-preserving editing (SERAC, IKE) emerging as the most robust solution for continual updates (Jiang et al., 22 Oct 2025).
• Coverage and Efficiency: Generating comprehensive multimodal augmentations for every fact (as in KOA) is costly; storing and updating covariance statistics for all layers is computationally expensive (Jiang et al., 22 Oct 2025). Online alignment of KBs, increased reliance on high-quality web search, and scalable retrieval/adapter mechanisms are active research areas.
• Domain Adaptation and Invariance: Explicit domain constraints in the RL objective yield measurable gains, but their design requires task-specific expertise regarding which invariances matter (e.g., rotation, scale, symmetry) (Cao et al., 23 Jan 2026).
• Self-contained Generalization: Parametric memory-based approaches (MKS2) enable zero-shot use of visual knowledge but may incur fixed capacity bottlenecks and risk overwriting visual priors during language fine-tuning (Li et al., 2023).

Advances in learned context weighting, hybrid retrieval-memorizations, improved expert routing, and temporally adaptive pretraining are central to bridging these gaps. The field continues to progress toward dynamic, continually-updating, and robust knowledge-infused LMMs.

---

References:

(Lee et al., 2024, Wang et al., 7 Jun 2025, Jiang et al., 30 May 2025, Zhang et al., 2024, Perry et al., 15 Jan 2025, Cao et al., 23 Jan 2026, Wang et al., 2024, Yun et al., 30 Mar 2026, Li et al., 2024, Jiang et al., 22 Oct 2025, Chen et al., 2024, Jiang et al., 22 Oct 2025, Huang et al., 19 Oct 2025, Jhalani et al., 2024, Li et al., 2023, Sun et al., 13 Aug 2025)