WSI-Agents: Collaborative Slide Analysis

• WSI-Agents is a specialized collaborative multi-agent framework that orchestrates patch- and slide-level models to perform comprehensive multimodal pathology analysis.
• It integrates task allocation, multi-layer verification, and summary synthesis to enhance diagnostic accuracy and mitigate model hallucinations.
• The system leverages a diverse model zoo and consensus mechanisms for sequential, tool-mediated reasoning, outperforming monolithic WSI models on key diagnostic tasks.

WSI-Agents is a collaborative multi-agent system for multi-modal whole slide image (WSI) analysis that combines specialized functional agents, task allocation, multi-layer verification, and summary synthesis to answer natural-language questions about pathology slides, perform morphological analysis, produce diagnoses, propose treatment plans, and generate pathology-style reports (Lyu et al., 19 Jul 2025). In current computational pathology, the term also sits within a broader agentic shift from isolated patch processing toward slide-level perception, navigation, memory, and reasoning. Recent work on WSI-level pre-training, multi-magnification navigation, morphology-aware multimodal modeling, and scalable slide retrieval indicates that whole-slide analysis is increasingly being treated as a sequential, tool-using, and explicitly structured decision process rather than a single monolithic classifier (Wu et al., 2024).

1. Definition, motivation, and scope

WSI-Agents was introduced to address a central tension in digital pathology: recent WSI-level multimodal LLMs such as WSI-LLaVA, SlideChat, and WSI-VQA provide multi-task natural-language interaction, but they remain single unified models and lag 15–30% behind specialized foundation models on key diagnostic tasks, while also exhibiting hallucinations and factual errors in domain-specific reasoning (Lyu et al., 19 Jul 2025). The motivating question is therefore how to preserve the versatility of multimodal LLMs while recovering the accuracy and reliability of task-specific WSI models.

The system answers this by replacing a single generalist model with a collaborative multi-agent design specialized for pathology. Its stated components are a task allocation module that assigns tasks to expert agents using a model zoo of patch- and WSI-level multimodal LLMs, a verification mechanism that performs internal consistency checks and external validation against pathology knowledge bases and domain-specific models, and a summary module that synthesizes the final answer together with visual interpretation maps (Lyu et al., 19 Jul 2025). The framework is pathology-specific rather than a generic medical agent.

This pathology specialization responds to a second empirical observation: general medical multi-agent systems such as MedAgents and MDAgents are not tailored to WSIs and achieve only ~40–50% accuracy on pathology-specific tasks versus 75–80% on broader medical tasks (Lyu et al., 19 Jul 2025). The distinctive difficulties are the gigapixel scale of WSIs, the need to integrate local histomorphology with slide-level architecture, and the requirement to reason over diagnostic criteria such as grading, staging, molecular subtype, and treatment implications using pathology-specific terminology.

A broader implication suggested by adjacent work is that “WSI-agents” can denote more than one named framework. Slide-level representation models such as PAMA treat the WSI as a structured object with anchors, distances, and orientations; navigation models such as MMNavAgent explicitly choose where and at what magnification to inspect next; and multimodal systems such as WSI-LLaVA convert whole-slide evidence into dialogue-oriented outputs (Wu et al., 2024). This suggests an emerging category of agentic WSI systems defined by whole-slide perception, sequential evidence accumulation, and tool-mediated reasoning.

2. System architecture and functional agents

At a high level, WSI-Agents processes a WSI $I$ and a natural-language query $x$ through three modules: Task Allocation Module (TAM), Verification Mechanism, and Summary Module (Lyu et al., 19 Jul 2025). The implementation is built with AutoGen, and for models with limited input size, WSIs are resized into $1024 \times 1024$ thumbnails for WSI multimodal LLMs, while patch-level foundation models are used for attention maps and classifier outputs (Lyu et al., 19 Jul 2025).

Module	Principal components	Function
Task allocation	Task agent, expert agents, WSI MLLM model zoo	Route the query and generate candidate answers
Verification	Logic agent, fact agent, consensus agent, memory module	Score internal consistency, factuality, and model agreement
Summary	Summarizing agent, reasoning agents, visual interpretation maps	Select, revise, and present the final answer

The task agent first determines whether the query is actually WSI-related and then classifies it into task types such as morphological analysis, diagnosis, treatment planning, or report generation (Lyu et al., 19 Jul 2025). It delegates to one or more expert agents: a Morphology Expert, Diagnosis Expert, Treatment Planning Expert, and Report Generation Expert. These expert agents select appropriate models from a model zoo of five WSI analysis multimodal LLMs: WSI-VQA, Quilt-LLaVA, WSI-LLaVA, WSI-Caption / MI-Gen, and HistGen (Lyu et al., 19 Jul 2025). Each selected model produces an initial response $\{y_i\}_{i=1}^M$.

The architecture is therefore not a new monolithic visual-LLM. It is an orchestration layer over existing WSI multimodal LLMs and WSI foundation models. This distinction is important because it clarifies both the system’s strength and its constraint: versatility comes from coordinated specialization, while performance depends on the coverage and quality of the external model zoo.

The task decomposition also mirrors the structure of the evaluation benchmark. Morphological analysis includes Global Morphological Description, Key Diagnostic Description, Regional Structure Description, and Specific Feature Description; diagnosis includes Histological Typing, Grading, Molecular Subtyping, and Staging; treatment planning includes Treatment Recommendations and Prognosis; and report generation is treated as a separate task (Lyu et al., 19 Jul 2025). This alignment between agent roles and benchmark tasks gives the framework a relatively explicit functional ontology.

3. Verification, consensus, and summary formation

The most distinctive subsystem in WSI-Agents is the verification pipeline, which is explicitly multi-layered. It comprises Internal Consistency Verification (ICV) and External Knowledge Verification (EKV), and all verification logs and scores are stored in a memory module (Lyu et al., 19 Jul 2025).

The logic agent performs internal consistency verification by extracting claims $\{C_i\}_{i=1}^N$ and evidence $\{E_i\}$ from candidate responses, constructing a compatibility matrix $G \in \mathbb{R}^{N \times N}$, and computing

$$\phi_g = \frac{\sum_{i=1}^{N}\sum_{j=i+1}^{N} G_{ij}}{N(N-1)/2}, \qquad \phi_e = \frac{\sum_{i=1}^{N} V_i}{N}, \qquad \phi_l = \frac{\phi_g + \phi_e}{2}.$$

Here $G_{ij} \in [0,1]$ measures whether claim $C_i$ can coexist with $x$0, and $x$1 scores whether the extracted evidence supports claim $x$2 (Lyu et al., 19 Jul 2025). The system therefore penalizes internally contradictory or weakly supported outputs even before consulting external references.

The fact agent then verifies claims against a pathology knowledge base built from PathologyOutlines and similar websites and WHO tumour classification books for digestive and breast systems etc. Documents are chunked, embedded, and indexed for similarity search; retrieved chunks are summarized into reference information $x$3; and each claim receives a factual score $x$4, aggregated as

$x$5

This converts pathology reference material into a retrieval-backed factuality prior (Lyu et al., 19 Jul 2025).

The consensus agent compares a candidate diagnosis against a foundation model zoo consisting of TITAN, CONCH, and Prism. If $x$6 denotes the extracted cancer type and $x$7 the classifier predictions, then agreement with the multimodal response is

$x$8

inter-classifier agreement is

$x$9

and the final classifier verification score is

$1024 \times 1024$0

This design gives low weight both to disagreements between the multimodal output and the classifiers and to unstable situations in which the classifiers disagree among themselves (Lyu et al., 19 Jul 2025).

The summarizing agent combines all three signals using

$1024 \times 1024$1

selects the best-scoring response as the basis for the answer, integrates supporting content from other responses, and generates an initial summary (Lyu et al., 19 Jul 2025). Multiple reasoning agents then critique that summary; the summarizing agent revises it; and the process iterates until more than half of the reasoning agents endorse the draft (Lyu et al., 19 Jul 2025). In parallel, the system aggregates attention maps from multiple patch-level WSI models into a single comprehensive interpretation map overlaid on the WSI thumbnail.

A frequent misconception is that the framework “solves” reliability through prompting alone. The architecture is more specific than that: reliability is mediated by formalized internal compatibility scoring, knowledge-base validation, and classifier consensus, each of which measurably contributes to performance in ablation studies (Lyu et al., 19 Jul 2025).

4. Relation to slide-level perception, navigation, and supporting infrastructure

The named WSI-Agents system emerged alongside a broader shift toward whole-slide agentic computation. A central enabling trend is the move from patch-level to slide-level representation learning. PAMA introduces a slide-level Masked Image Modeling proxy task for WSIs, a position-aware representation $1024 \times 1024$2, and a Position-aware Cross-Attention (PACA) module with complexity $1024 \times 1024$3 rather than $1024 \times 1024$4; it is explicitly framed as a WSI-level foundation model that can serve as a perception or representation core for WSI-agents (Wu et al., 2024). Its Kernel Reorientation (KRO) and Anchor Dropout (AD) mechanisms address rotational ambiguity and robustness, while pan-cancer pre-training improves both in-domain and out-of-domain classification performance.

A second strand is explicit sequential navigation. MMNavAgent models WSI diagnosis as an iterative process beginning from a thumbnail $1024 \times 1024$5 and a text prompt $1024 \times 1024$6, then alternating between a Cross-Magnification navigation Tool (CMT) and a Magnification Selection Tool (MST) that chooses move, zoom, or stop based on memory and visual evidence (Xu et al., 2 Mar 2026). CMT uses a Magnification-Aware Block (MAB) and a Cross-Magnification Block (CMB) to fuse adjacent magnifications, while MST stores navigation history and VLM-generated descriptions in an explicit memory bank $1024 \times 1024$7. On a public skin WSI dataset, MMNavAgent improves over a non-agent baseline by +1.45% AUC and +2.93% BACC, illustrating that adaptive magnification control and memory-driven navigation can be operationalized as a WSI-agent (Xu et al., 2 Mar 2026).

A third strand is morphology-aware multimodal reasoning. WSI-LLaVA introduces WSI-Bench, a benchmark with 179,569 WSI-question-answer pairs from 9,850 WSIs across 30 cancer types, and trains a slide-level multimodal LLM through WSI-text alignment, feature space alignment, and task-specific instruction tuning (Liang et al., 2024). The benchmark’s task taxonomy—global morphology, key diagnostic features, histological typing, grading, staging, treatment recommendations, prognosis, and report generation—closely matches the functional decomposition later used by WSI-Agents, and its pathology-specific metrics WSI-Precision and WSI-Relevance provide a natural evaluation interface for agent outputs (Liang et al., 2024).

Related infrastructure addresses other capabilities that are relevant for agentic WSI systems. SeqShort performs attention-based sequence shortening so that thousands of patch embeddings can be reduced to a fixed short sequence and processed by a text-pretrained transformer while fine-tuning less than 0.1% of transformer parameters, offering an efficient slide-level reasoning substrate (Pisula et al., 2022). HoloHisto shifts WSI segmentation to end-to-end gigapixel processing with 4K resolution sequential tokenization and direct WSI I/O, suggesting a dense-prediction perception module for agents that require full-slide masks rather than only global answers (Tang et al., 2024). Scalable memory mechanisms are represented by hierarchical GCN-based WSI retrieval, which reports mean average precision above 0.857 on ACDC-LungHP and above 0.864 on Camelyon16 with average retrieval time 0.802 ms from a database within 120 WSIs, indicating a practical region-retrieval substrate for case-based or memory-augmented agents (Zheng et al., 2021).

Single-vector slide embeddings and long-term adaptation are also relevant. Aggregation studies for WSI retrieval show that GMM Fisher Vector, Deep Sparse Fisher Vector, and Deep Binary Fisher Vector provide strong WSI search performance, which is useful when an agent must index or retrieve many slides efficiently (Hemati et al., 29 Jan 2025). For continual deployment across hospitals and cohorts, AGLR-CL uses Gaussian mixture models to synthesize WSI representations and patch count distributions without storing raw data, and is explicitly positioned as a privacy-aware continual learning mechanism for WSI-agent backbones (Kumari et al., 13 May 2025). Domain robustness can be further improved either through semi-weakly supervised pseudo-bag learning with AdaPse and MergeUp for WSI classification (Ouyang et al., 2024) or through WSI-level and patch-level contrastive learning over pseudo-domains derived from non-tumor regions, which improves segmentation macro-F1 to 0.7675 in cross-hospital evaluation (Shigeyasu et al., 11 Aug 2025).

5. Benchmarks, empirical performance, and ablations

WSI-Agents is evaluated primarily on WSI-Bench and WSI-VQA. On WSI-Bench, the reported average WSI-Precision across task groups is 0.703 overall for WSI-Agents, compared with 0.610 for WSI-LLaVA, 0.607 for Quilt-LLaVA, 0.541 for WSI-VQA, 0.528 for Med-Agents, and 0.326 for MDAgents (Lyu et al., 19 Jul 2025). On WSI-VQA, WSI-Agents achieves 0.600 accuracy, versus 0.550 for WSI-LLaVA, 0.470 for WSI-VQA, 0.208 for MDAgents, and 0.183 for Med-Agents (Lyu et al., 19 Jul 2025).

System	WSI-Bench overall average	WSI-VQA accuracy
WSI-LLaVA	0.610	0.550
Quilt-LLaVA	0.607	0.130
WSI-VQA	0.541	0.470
Med-Agents	0.528	0.183
MDAgents	0.326	0.208
WSI-Agents	0.703	0.600

The task-group breakdown is also informative. WSI-Agents reaches 0.568 average WSI-Precision on morphological analysis, 0.714 on diagnosis, and 0.827 on treatment planning (Lyu et al., 19 Jul 2025). The diagnosis improvement is particularly notable because it is the capability most directly tied to the verification stack: candidate outputs are not only generated but also checked against logic, reference pathology knowledge, and foundation-model agreement.

Report generation results emphasize the distinction between surface-text similarity and pathology-grounded correctness. WSI-Agents attains BLEU-1 = 0.4443, METEOR = 0.4719, and WSI-Precision (“Acc”) = 0.440, with the last value reported as best by 6% over WSI-LLaVA (Lyu et al., 19 Jul 2025). In other words, the system’s strongest advantage is not merely stylistic fluency but verified report content.

Ablation studies show that every major component contributes. In the reported average score, removing the task agent reduces performance to 0.391, removing expert agents to 0.531, removing the logic agent to 0.607, removing the fact agent to 0.603, removing the consensus agent to 0.601, removing the summarizing agent to 0.536, and removing reasoning agents to 0.561, versus 0.637 for the full system (Lyu et al., 19 Jul 2025). The largest degradations arise from disrupting task/expert coordination and the summary/reasoning loop, but verification components each contribute several points.

A qualitative case study further illustrates the system’s behavior. For the question “What is the histological classification based on your examination of the slide?”, with ground truth adenocarcinoma, Quilt-LLaVA predicts seminoma, WSI-VQA predicts invasive squamous cell carcinoma, WSI-LLaVA predicts high-grade urothelial carcinoma, and WSI-Agents predicts adenocarcinoma, explicitly citing glandular structures and the absence of keratinization or intercellular bridges to rule out squamous carcinoma (Lyu et al., 19 Jul 2025). The significance of this example is less the single label than the verified differential reasoning pattern.

6. Limitations, misconceptions, and future directions

WSI-Agents remains subject to several explicit limitations. The framework is pathology-specific, so extension to other imaging domains would require a different model zoo and knowledge infrastructure (Lyu et al., 19 Jul 2025). For multimodal LLMs, WSIs are resized to $1024 \times 1024$8 thumbnails, which can omit fine detail critical for some diagnoses, so the system still depends on patch-level foundation models and attention maps for finer-grained validation and interpretation (Lyu et al., 19 Jul 2025). The pathology knowledge base is built from selected websites and WHO classification texts, so rare entities and very recent guideline changes may be underrepresented (Lyu et al., 19 Jul 2025).

A second misconception is to treat WSI-Agents as a fully autonomous navigation system. The named framework is primarily a coordination-and-verification architecture over existing WSI MLLMs and classifiers, not an explicit reinforcement-learning agent that decides where to look next. By contrast, MMNavAgent includes an explicit memory-driven navigation policy over magnification and movement but, even there, the policy is prompt-driven rather than reward-trained (Xu et al., 2 Mar 2026). This indicates that “agent” in current WSI research spans several levels of agency: orchestration, navigation, and potentially future closed-loop planning.

Several research directions follow directly from adjacent literature. Multimodal integration of histology with genomics, transcriptomics, or clinical metadata is repeatedly identified as a next step for slide-level representation models and would naturally expand agent state spaces (Wu et al., 2024). Active region selection and navigation remain a natural frontier, since PAMA already yields interpretable attention maps and MMNavAgent already implements move/zoom/stop decisions (Wu et al., 2024). Continual and privacy-preserving learning is another practical requirement for clinical deployment across centers; AGLR-CL shows one route via GMM-based latent replay without storing raw WSIs (Kumari et al., 13 May 2025). Online domain detection and adaptation are also likely to matter, particularly where intra-hospital or cross-hospital shifts affect morphology or stain appearance (Shigeyasu et al., 11 Aug 2025). For dense perception, pathology-specific tokenizers could strengthen end-to-end slide segmentation beyond the natural-image VQGAN tokenizer used in HoloHisto (Tang et al., 2024).

A plausible synthesis is that WSI-Agents names both a specific collaborative framework and a research program. In its current form, the system demonstrates that coordinating a model zoo through task routing, logic checking, external pathology references, and foundation-model consensus can surpass single WSI multimodal models on multi-task slide analysis (Lyu et al., 19 Jul 2025). The surrounding literature suggests the next generation will couple that verified multimodal reasoning with native slide-level representation learning, active navigation, scalable retrieval, lifelong adaptation, and richer multimodal evidence streams. The source code for the named framework has been publicly released at https://github.com/XinhengLyu/WSI-Agents (Lyu et al., 19 Jul 2025).