Patho-AgenticRAG: Multimodal RAG in Pathology

• Patho-AgenticRAG is a multimodal framework integrating joint text-image search with RL-driven agent routing for pathology diagnosis.
• It uses a page-level textbook database and specialized fusion scoring (Patho-Fusion) to retrieve localized diagnostic cues effectively.
• The design couples active evidence synthesis with controlled retrieval policy to reduce hallucinations and boost classification accuracy.

Patho-AgenticRAG is a multimodal Retrieval-Augmented Generation framework for pathology vision–LLMs that replaces text-only retrieval with joint text–image search over page-level embeddings from authoritative pathology textbooks and couples that retrieval layer to agentic reasoning, task decomposition, multi-turn search, and reinforcement-learning-based routing (Zhang et al., 4 Aug 2025). It is motivated by a pathology-specific failure mode: ultra-high-resolution slides, localized diagnostic cues, and nuanced clinical semantics make pathology VLMs prone to hallucinations, while conventional text-only RAG discards the visual evidence carried by textbook figures and atlas pages. In this formulation, retrieval is not merely evidential support appended after the fact; it is an active component of diagnostic policy, deciding whether to retrieve, how to rewrite the query, whether to restrict search to tissue-specific partitions, and how to synthesize visual and textual evidence into a grounded answer.

1. Architectural definition

Patho-AgenticRAG is organized as a modular agentic stack with four principal components: a multimodal pathology knowledge base, an Agent Router, a VRAG Agent for retrieval and evidence synthesis, and a core pathology VLM used for final inference (Zhang et al., 4 Aug 2025). The central design claim is that pathology diagnosis should be treated as a multimodal agentic RAG problem rather than as either a purely parametric VLM problem or a text-only retrieval problem.

The system’s “agentic” character lies in explicit control decisions. The Agent Router takes the original query and decides whether RAG should be invoked at all, how many query rewrites to perform, whether a tissue-specific classifier should be used, and which organ or tissue partition should constrain retrieval. If retrieval is invoked, the VRAG Agent executes multi-turn search over the pathology textbook database, applies multimodal fusion and re-ranking, iteratively refines evidence, and summarizes the most relevant support for the core VLM. The VLM, instantiated as Patho-R1-7B, then produces the final diagnosis or answer conditioned on the original image–text query and the retrieved evidence summaries (Zhang et al., 4 Aug 2025).

Component	Role	Key implementation
Knowledge base	Multimodal evidence source	Page-level textbook embeddings
Agent Router	Retrieval policy control	RL-trained discrete decisions
VRAG Agent	Retrieval and summarization	Multi-turn search and fusion
Core VLM	Final reasoning engine	Patho-R1-7B

A common misconception is that Patho-AgenticRAG is simply a pathology search engine with a VLM attached. Its defining feature is instead the coupling of retrieval policy, multimodal evidence fusion, and diagnostic reasoning. Another common misconception is that the method is equivalent to generic multimodal RAG. The system is specifically shaped by pathology’s need for localized visual grounding: the database is built at the page level so that figures, captions, and surrounding descriptive text remain co-located rather than being flattened into disjoint textual fragments (Zhang et al., 4 Aug 2025).

2. Multimodal textbook database and retrieval layer

The knowledge base is built from more than 600 authoritative pathology textbooks, initially yielding roughly 300k pages and then, after filtering, more than 200k high-quality pages (Zhang et al., 4 Aug 2025). Each page is treated as a multimodal document containing OCR or digital-native text together with one or more pathology figures, micrographs, or diagrams. This page-level granularity is important because pathology textbooks encode diagnostic knowledge through the co-occurrence of image, caption, and explanatory prose.

For representation learning, Patho-AgenticRAG uses ColPali/ColQwen2-style encoders to place text tokens, image patches, and document tokens into a joint vector space. These page-level features are stored in Milvus and indexed with HNSW for approximate nearest-neighbor search. This enables a query containing image evidence, text, or both to retrieve pages that are jointly relevant in visual and textual terms rather than only lexically similar (Zhang et al., 4 Aug 2025).

The retrieval model distinguishes among several fusion strategies. CoPaLi text-only retrieval is already competitive, with Rec@1 of 0.640 and Rec@5 of 0.900, whereas image-only CoPaLi is weak, with Rec@1 of 0.060 and Rec@5 of 0.220. WeiMoCIR, a weighted modality-fusion baseline, remains similarly weak, with Rec@1 of 0.060 and Rec@5 of 0.200. The proposed Patho-Fusion re-ranking function improves top-ranked retrieval, achieving Rec@1 of 0.720, MRR@5 of 0.777, and NDCG@5 of 0.824 (Zhang et al., 4 Aug 2025).

The Patho-Fusion score is defined over token- and patch-level similarity statistics rather than simple average similarity:

$$\begin{aligned} \text{score}(Q, D) \;=\; &\; \text{std}\bigl(\text{std}(S_t[i,:])\bigr) \times \bigl[\text{mean}\bigl(\kappa(S_t[i,:])\bigr)\bigr]^2 \ &\times \text{mean}\bigl(\kappa(S_v[i,:])\bigr) + \text{mean}\bigl(\max(S_t[i,:])\bigr), \end{aligned}$$

where $S_t$ and $S_v$ are text–document and image–document similarity matrices, $\text{std}$ is standard deviation, and $\kappa$ is kurtosis (Zhang et al., 4 Aug 2025). The intended effect is to reward pages in which relevance is concentrated in diagnostically specific subregions or terms rather than diffusely spread across the page. In pathology terms, localized strong matches are often more meaningful than uniformly moderate matches, because decisive evidence may reside in a small cluster of nuclei, a specific glandular architecture, or a narrowly described morphologic criterion.

This design directly rejects the assumption that stronger pathology RAG can be obtained by adding image embeddings to a standard text retriever. The empirical weakness of image-only retrieval and naïve weighted fusion indicates that pathology retrieval requires modality interaction structured around sparse, localized diagnostic evidence rather than coarse global similarity (Zhang et al., 4 Aug 2025).

3. Agent Router, VRAG, and reinforcement learning

The Agent Router formalizes retrieval as a sequence of discrete policy decisions. The first decision is whether to invoke RAG. If the answer is negative, the system follows a non-retrieval path. If the answer is positive, the router selects a rewrite count, decides whether a tissue-specific classifier should constrain the search space, and, if so, selects a partition $C_j$ from a set of organ- or tissue-specific partitions (Zhang et al., 4 Aug 2025). This means the system’s control flow is not fixed; retrieval is conditional, possibly rewritten, and optionally routed through organ-specific subindices.

The router is optimized by reinforcement learning rather than by prompt engineering alone. Given an original query $Q_{\text{orig}}$, the objective is

$$\max_{\pi} \mathbb{E}_{P \sim \pi(Q_{\text{orig}})} \left[ R_{\text{final}}(P, P_{\text{gt}}) \right],$$

where $P$ is the chosen decision path and $P_{\text{gt}}$ is the target path (Zhang et al., 4 Aug 2025). Rewards are hierarchical. A wrong top-level decision on whether to use RAG yields zero reward. Correct non-RAG routing yields $S_t$0. If RAG is invoked, the system accumulates partial credit for correct rewrite count, correct classifier usage, and correct partition choice, with rewards in $S_t$1 (Zhang et al., 4 Aug 2025).

Optimization uses GRPO. For a query $S_t$2, the agent samples a group of paths $S_t$3, computes group-wise normalized advantages,

$S_t$4

and updates the policy with a PPO-like clipped objective plus KL regularization against a reference model (Zhang et al., 4 Aug 2025). A critical implementation finding is that routing policy should be trained, but the core pathology VLM should not be directly subjected to the same RL regime. The paper explicitly frames direct end-to-end RL fine-tuning of large VLMs in medicine as risky and instead concentrates RL on routing and tool usage, which is easier to audit and control.

Training combines a small supervised cold start with RL. Pure GRPO without SFT converges poorly, while large SFT before GRPO makes the policy overly rigid. The best trade-off is SFT400 followed by GRPO4k. On Path-VQA this raises accuracy from 77.51% under GRPO4k alone to 80.34%, and on Quilt-VQA from 60.93% to 75.80% (Zhang et al., 4 Aug 2025). The router therefore embodies a specific philosophy of agentic training: supervise just enough to stabilize tool use, then let RL optimize the retrieval policy.

4. Benchmark performance and empirical profile

Patho-AgenticRAG is evaluated on retrieval benchmarks and on closed-ended pathology QA tasks, with particularly strong gains in knowledge-intensive and classification-style settings (Zhang et al., 4 Aug 2025). On retrieval, the key result is that pathology-specific multimodal fusion improves the probability that the top retrieved page is genuinely relevant. Patho-Fusion outperforms text-only CoPaLi at Rec@1 by 0.080 and dramatically outperforms image-only retrieval and coarse weighted fusion, establishing that the database and fusion design are not interchangeable with generic multimodal retrieval.

On PathMMU-test, the main comparison is against Patho-R1-7B, the prior pathology-specialized baseline. Patho-AgenticRAG improves Atlas from 75.34 to 78.32, EduContent from 66.43 to 70.96, PathCLS from 45.40 to 53.16, PubMed from 66.06 to 69.69, and SocialPath from 67.93 to 71.06. On PathMMU-test-tiny, it raises PathCLS from 44.63 to 57.22 and SocialPath from 67.89 to 74.70, although Atlas declines slightly from 81.73 to 79.33 (Zhang et al., 4 Aug 2025). The largest improvements occur where classification and cross-source evidence synthesis are central.

The strongest absolute gains appear on Y/N and knowledge-intensive benchmarks. On Quilt-VQA YORN, Patho-R1-7B reaches 64.72, whereas Patho-AgenticRAG reaches 75.80. On Path-VQA YORN, Patho-R1-7B reaches 46.97, while Patho-AgenticRAG reaches 80.34. On the pathology subset of MedXpertQA, performance rises from 22.00 with Patho-R1-7B to 60.00 with Patho-AgenticRAG. On OmniMedVQA Bright, it rises from 70.79 to 90.11 (Zhang et al., 4 Aug 2025). These numbers indicate that the method’s largest advantages emerge when the answer depends on external knowledge, differential comparison, or nontrivial grounding rather than on local visual recognition alone.

The ablations clarify what drives those gains. Patho-Fusion is essential: replacing it with generic multimodal fusion weakens retrieval quality. Likewise, the retrieval policy itself matters: the principal improvement over Patho-R1 comes not from replacing the reasoning VLM, but from equipping it with an RL-trained router and VRAG-style multi-turn evidence acquisition (Zhang et al., 4 Aug 2025). This sharply distinguishes Patho-AgenticRAG from both static-RAG pipelines and pure-RL pathology VLMs.

5. Position within the broader pathology and clinical agentic-RAG landscape

Patho-AgenticRAG belongs to a broader family of systems that use retrieval not merely to supplement generation, but to structure diagnostic, biomarker, or extraction workflows. In digital pathology VQA, Path-RAG uses HistoCartography to retrieve relevant image regions and improves LLaVA-Med on PathVQA-Open from 38% to 47%, with a notable gain of 28% for H&E-stained images and longer-form gains of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books (Naeem et al., 2024). Relative to that lineage, Patho-AgenticRAG moves the retrieval target from image patches to textbook pages while adding explicit RL-based control over when and how retrieval is used.

In computational pathology biomarker discovery, SAGE demonstrates a different but related instantiation of pathology-oriented agentic retrieval: a literature-derived knowledge graph of 41,053 nodes and 56,338 edges supports path generation, novelty critique, feasibility assessment, and automated multimodal validation. Its FABP5–TLS example yields log-rank $S_t$5 and $S_t$6 in TCGA-BLCA, showing how agentic retrieval can support interpretable, biologically grounded pathology hypotheses rather than only question answering (Nasser et al., 1 Feb 2026).

Outside pathology but within clinical diagnosis, RareCollab provides a hybrid pattern in which a deterministic Diagnostic Engine is combined with LLM-based specialist “Labs” for RNA, phenotype, database, and literature interpretation, achieving 46% top-1 and 77% top-5 diagnostic accuracy on 131 UDN probands (Qi et al., 3 Feb 2026). Deep-DxSearch extends agentic retrieval further by treating the medical corpus as an RL environment and learning policies over actions such as <lookup>, <match>, and <search>, with large gains over prompt-engineered and training-free RAG baselines in both common and rare disease diagnosis (Zheng et al., 21 Aug 2025). ACIE shows that an on-premise agentic RAG pipeline can reason over complete patient contexts and ground every extraction in cited passages, reaching 96.5% clinician acceptance across 7,326 judgments in a lymphoma registry study (Çinar-Koraş et al., 17 Jun 2026). Taken together, these systems suggest that Patho-AgenticRAG is best understood not as an isolated pathology VQA method, but as one member of an increasingly coherent class of clinically grounded agentic RAG architectures.

6. Limitations, misconceptions, and future directions

Patho-AgenticRAG does not eliminate pathology hallucination; it changes the evidential substrate and the routing policy through which hallucination is managed (Zhang et al., 4 Aug 2025). The paper identifies several residual failure modes. If the visual encoder misreads a subtle lesion, retrieval may return visually similar but semantically wrong pages. If the tissue-specific router misclassifies the organ context, evidence may be drawn from the wrong partition. Some textbook pages are noisy or generic enough to generate false positives. Rare entities, uncommon stains, and underrepresented subspecialties remain limited by database coverage. These limitations are especially significant because the method’s strongest claims rely on authoritative textbooks as the principal external memory.

Another misconception is that the method proves multimodal retrieval alone is sufficient for trustworthy pathology assistance. The empirical story is more specific. Image-only retrieval is weak; naïve fusion is weak; static routing is insufficient; and gains are largest when retrieval, policy control, and final reasoning are tightly coupled (Zhang et al., 4 Aug 2025). A plausible implication is that future pathology systems will need even finer evidence localization, for example at the figure-region rather than page level, and broader integration across radiology, genomics, and clinical text. The paper explicitly lists scaling the knowledge base, adding more modalities, improving RL and agentic frameworks, supporting online feedback, and moving from page-level to region-level evidence localization as future directions (Zhang et al., 4 Aug 2025).

In a broader technical sense, Patho-AgenticRAG suggests a transition from “retrieve then read” pathology systems toward “route, retrieve, compare, and justify” systems. That transition is visible across adjacent work: key-region retrieval in pathology VQA, literature-grounded biomarker discovery, deterministic-plus-agentic rare-disease diagnosis, RL-trained diagnostic trajectories, and whole-patient agentic extraction (Naeem et al., 2024, Nasser et al., 1 Feb 2026, Qi et al., 3 Feb 2026, Zheng et al., 21 Aug 2025, Çinar-Koraş et al., 17 Jun 2026). The shared architectural theme is not simply the presence of retrieval, but the treatment of retrieval as a controllable, inspectable, and clinically meaningful component of reasoning.