Med-GRIM: Multimodal GraphRAG for Medical VQA

• The paper introduces Med-GRIM, a novel framework that combines dense multimodal encoding, graph-based retrieval, and advanced prompt engineering to enable zero-shot medical VQA.
• It employs a BIND encoder with a True Transformation Layer and a two-stage retrieval process that fuses image-text embeddings with structured medical knowledge graphs.
• Empirical evaluations demonstrate significant accuracy and efficiency improvements over conventional and fine-tuned vision-language models on biomedical VQA benchmarks.

Med-GRIM (Multimodal GraphRAG for Visual Question Answering, VQA) is a computational framework that integrates dense multimodal encoding, graph-based retrieval-augmentation, and advanced prompt engineering to achieve precise zero-shot medical VQA. Med-GRIM distinguishes itself by delivering the performance of large vision-LLMs (VLMs) at reduced computational cost, leveraging structured medical knowledge graphs and modular agents operating over image and text modalities. The system has demonstrated statistically significant improvements over conventional and fine-tuned VLMs on both standard and biomedical VQA benchmarks (Madavan et al., 20 Jul 2025).

1. Core Architecture: BIND Encoder and Embedding Space

At the heart of Med-GRIM is the BIND encoder (BLIVA Integrated with Dense Encoding), which utilizes pretrained unimodal backbones—such as a ViT or CNN for images ($x_v\in\mathbb{R}^{d_v}$), and BERT or RoBERTa for text ($x_t\in\mathbb{R}^{d_t}$). These modality-specific features are processed through a frozen Q-Former (from BLIVA), projecting both into a small set of query tokens:

• Visual modality: $q^{(v)}_i = W_q\,\mathrm{QForm}(x_v)_i$
• Text modality: $q^{(t)}_j = W_q'\,\mathrm{QForm}(x_t)_j$

A True Transformation Layer (TTL), implemented as a modality-agnostic MLP with residual connection, fuses query tokens: $$z_k = \mathrm{TTL}(q^{(v)}_k + q^{(t)}_k) = \mathrm{MLP}(q^{(v)}_k + q^{(t)}_k) + (q^{(v)}_k + q^{(t)}_k)$$ The joint embedding for any image-text pair is: $$f_{\text{enc}}(I_{\rm img}, I_{\rm text}) = \frac{1}{m}\sum_{k=1}^m z_k \in \mathbb{R}^d$$

Contrastive pretraining employs a CLIP-style symmetric objective across $N$ samples: $$\mathcal{L}_{\rm contrastive} = \frac{1}{2N}\sum_{i=1}^N \left[\ell_{i\to *} + \ell_{*\to i}\right]$$ where $$\ell_{i\to j} = -\log \frac{\exp(\mathrm{sim}(v_i, t_i)/\tau)}{\sum_{k=1}^N \exp(\mathrm{sim}(v_i, t_k)/\tau)}$$, and $\mathrm{sim}(u, w)=\frac{u^\top w}{\|u\|\|w\|}$ is cosine similarity. Downstream VQA training uses standard cross-entropy loss, yielding final objective $$\mathcal{L} = \mathcal{L}_{\rm contrastive} + \alpha\mathcal{L}_{\rm CE}$$ (empirical $\alpha\approx1$, $\tau\approx0.07$) (Madavan et al., 20 Jul 2025).

2. Graph-RAG Knowledge Retrieval and Filtering

Med-GRIM's retrieval pipeline utilizes a structured medical knowledge graph $\mathcal{G}=(V,E)$ constructed from the DermaGraph dataset. Each condition node $v\in V$ contains both a multimodal image-text embedding ($e_v^{\rm mm}$, from BIND) and an independent text embedding ($e_v^{\rm text}$), with hierarchical child nodes for symptoms, treatments, and preventive measures.

Edges are:

1. Intra-condition (linking to child attributes)
2. Inter-condition (between conditions with high text similarity)

Retrieval is performed in two stages: A. Hybrid Encoding-Based Filtering Input $I=(I_{\rm img}, I_{\rm text})$ yields embeddings $h_{\rm mm}$ and $h_{\rm text}$. Condition node similarity is computed as: $$S(I,v) = \lambda \cos(h_{\rm text}, e_v^{\rm text}) + (1-\lambda)\cos(h_{\rm mm}, e_v^{\rm mm})$$ with empirical $\lambda=0.4$ (Madavan et al., 20 Jul 2025). Top-$K$ nodes above a dynamic threshold (typically $0.95$ times the max score) and their 1-hop graph neighbors are selected (Algorithm 1).

B. SLM-Based Response Filtering Small LLMs (SLMs, e.g., Phi-3.8B) generate targeted follow-up questions for each retained condition. User input informs SLM-based likelihood scoring over candidates, retaining only those with likelihood $\pi_v>0.5$ (Algorithm 2).

3. Modular Agent Workflow and Prompt Injection

Med-GRIM’s workflow is modularized into agent functions:

• Encoder Agent: BIND encoder for multimodal embedding generation.
• Retriever Agent: Graph traversal, similarity filtering, and question proposal.
• SLM QA Agent: Follow-up question generation via SLM.
• SLM Reasoner Agent: Likelihood assignment and final differential diagnosis composition (e.g., via Mistral-7B).

The prompt passed to the response LLM is dynamically assembled, infusing structured graph-derived knowledge:

You are a medical assistant. A patient shows the following image: <Image> Their description: ‘{user_text}’ Based on our database, the most likely conditions are: {#1} {condition₁} – key symptoms: {symptoms₁}; treatments: {treatments₁}. ... Please provide a detailed differential diagnosis, listing alternatives, uncertainties, and recommended follow-up questions.

Agents communicate strictly through text and embeddings; the workflow involves clear roles and handoff steps, with response generation strictly based on injected graph-based context (Madavan et al., 20 Jul 2025).

4. Domain-Specific Knowledge Graphs: DermaGraph and Extensions

The DermaGraph dataset anchors Med-GRIM’s graph-retrieval capabilities:

• 50 dermatological conditions harvested from the NHS website, with symptoms, treatments, and prevention.
• Each node encodes averaged image embeddings ($e_v^{\rm mm}$, 10–15 samples per condition) and independent text embeddings ($e_v^{\rm text}$).
• Child nodes cover Symptoms, Clinical Treatments, Home Remedies, and Preventive Measures.
• Inter-condition edges connect conditions with textual similarity above 0.8 (e.g., psoriasis–eczema).

DermaGraph supports both unimodal and multimodal retrieval, and an open-ended evaluation split is crafted for zero-shot testing (Madavan et al., 20 Jul 2025).

Extensions in related graph-based frameworks (e.g. mKG-RAG) generalize the knowledge graph construction to include textual entities, visual regions, and modality-crossing relationships ($G=(V,E,X,M)$). Extraction involves automated scene graph segmentation, textual entity recognition (often LLM-driven), and cross-modal matching using multimodal LLMs (Yuan et al., 7 Aug 2025). Such frameworks facilitate the migration of Med-GRIM concepts to any medical domain with robust document/image corpora.

5. Multimodal Graph Construction and Reasoning

Several graph-centric VQA systems inform Med-GRIM's reasoning paradigm:

• Cross-modal feature graphing in computed tomography VQA organizes slices and question tokens as graph nodes with spatial and cross-modal connectivity, leveraging attentive graph convolutional networks (A-GCN) for feature fusion (Tian et al., 6 Jul 2025).
• Multi-modal relationship graphs in chest-X-ray VQA encode spatial, semantic (domain expert knowledge), and implicit (data-driven) edge relationships among image regions and semantic labels, processed through relation-aware graph attention (ReGAT) (Hu et al., 2023).

A plausible implication is that the choice of graph structure (e.g., anatomical knowledge, co-occurrence statistics, slice continuity) significantly impacts the fidelity and interpretability of answers to clinical queries, as the topological pathways in the graph correspond to reasoning chains employed by domain experts.

6. Empirical Evaluation and Performance Metrics

Med-GRIM’s performance is validated across multiple axes:

• Zero-Shot VQA (Table 1): BIND (Flan-T5-XXL) achieves 23.38% accuracy, outperforming BLIVA (+2.9%) and InstructBLIP (+4.48%) (Madavan et al., 20 Jul 2025).
• Biomedical VQA (Table 2): On VQA-RAD and PathVQA, BIND outperforms fine-tuned medical VLMs: 87.5% and 95.8% closed-form accuracy.
• DermaGraph Evaluation (Table 3): Full Med-GRIM pipeline reaches 83.33% accuracy, Semantic-BERT coherence 0.81; this surpasses LLaVA-Med (76.7%, 0.63), MUMC, Med-Flamingo, and RULE (vanilla RAG). Paired t-tests confirm significance ($p<0.01$) over RULE.

Efficiency is notable: a complete run on Ryzen 5 4600HS CPU requires 17.5s/6.7GB RAM (Mistral-7B), versus >60s/20GB for fine-tuning monolithic VLMs; FLOPs savings exceed 3× (Madavan et al., 20 Jul 2025).

Multi-modal graph VQA frameworks (e.g. mKG-RAG) demonstrate state-of-the-art accuracy (e.g. +7.1 points over EchoSight and ReflectiVA on InfoSeek when fine-tuned) (Yuan et al., 7 Aug 2025). In clinical imaging VQA, graph-based models (A-GCN, ReGAT) yield 0.84 BLEU and 0.74 BERT-Score improvements over vanilla multimodal LLMs (Tian et al., 6 Jul 2025), while semantic graph fusion achieves micro/macro AUC of 0.996/0.964 versus 0.981/0.948 for meta-learning baselines (Hu et al., 2023).

7. Interpretability, Error Analysis, and Limitations

Interpretability is intrinsic to graph-reasoned VQA:

• Activated ROIs: Visualization of top-nodes by attention weights shows correspondence with clinical regions of interest (e.g., heart bounding boxes for cardiomegaly).
• Reasoning Paths: Directed attention trails in the graph reveal diagnostic workflows analogous to expert reasoning, enabling verification of model faithfulness.

Error modes in Med-GRIM cluster in cases of visually similar conditions (e.g., mild eczema vs. contact dermatitis), requiring additional Q&A iterations for clarification. This suggests that increased granularity in graph attribute modeling and finer question templates may further improve robustness (Madavan et al., 20 Jul 2025). Graph-based VQA models face scaling challenges when graph size is quadratic in slice/token number, motivating further research into dynamic or hierarchical pruning (Tian et al., 6 Jul 2025).

Summary Table: Med-GRIM Components

Component	Technical Details	Empirical Impact
BIND Encoder	Dense query-token embedding, True Transformation Layer	+2.9% zero-shot gain over BLIVA; outperforming MUMC, K-PathVQA
Graph-RAG Module	Hybrid similarity, 2-stage retrieval, prompt injection	83.33% DermaGraph accuracy; >3× efficiency increase vs. VLM finetune
DermaGraph	50 condition KG, child nodes, multimodal support	Supports multimodal/unimodal queries; source of interpretable context
Modular Workflow	Agents for encoding, retrieval, reasoning, QA	Enables dynamic, robust, modular VQA diagnostics
Interpretability	Attention-based ROI/graph reasoning path tracing	Faithful, expert-aligned diagnostic rationale exposure

Med-GRIM establishes a paradigm for efficient, interpretable, and highly accurate medical VQA through its amalgamation of dense multimodal encoding, explicit domain knowledge graphs, modular reasoning workflows, and dynamic prompt engineering, with strong empirical grounding on rigorous zero-shot and biomedical benchmarks (Madavan et al., 20 Jul 2025, Yuan et al., 7 Aug 2025, Tian et al., 6 Jul 2025, Hu et al., 2023).