PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA

Abstract: Whole-slide image visual question answering (WSI-VQA) frames pathology as an extreme-context search problem: to answer a free-form clinical query, a system must first navigate a gigapixel slide under a strict inspection budget to locate sparse, high-resolution evidence. Existing approaches largely fall into two paradigms: i) supervised pathology multimodal LLMs (MLLMs) and agents can absorb localization and reasoning into learned modules, but they often couple navigation to task-specific supervision and retraining, limiting their practicality; ii) training-free pathology agents avoid this cost by keeping core models frozen, but often follow a question-first design, constructing the initial candidate set mainly from query-conditioned relevance. This can miss decisive morphology that is not named in the question, and force heavier inference-time scaffolding. To address this challenge, we introduce PathNavigate, a training-free pathology agent built around a scan-search-readout routine. Before question matching, PathNavigate scans the current slide at low magnification with a shared online memory module over frozen pathology features, producing a slide-specific surprise field that marks an abnormal-region pool. It then applies question-conditioned PLIP relevance only within this pool to select high-magnification search targets. Finally, it extracts local high-magnification evidence and answers with a frozen perceptor-adjudicator stack, using the same online memory as slide-level context. Experiments on WSI-VQA and SlideBench-BCNB show that the proposed scan-search-readout design improves answer accuracy and yields more interpretable evidence-selection trajectories with higher efficiency.The code is available online.

• The paper introduces a training-free Scan-Search-Readout agent that uses slide-specific surprise to build diverse regions of interest, then applies question relevance and high-magnification evidence readout.
• PathNavigate achieves 59.42% overall accuracy on SlideBench-BCNB and 56.34 total on WSI-VQA, with especially large open-ended gains over PathAgent of 61.00% versus 13.91%.
• The shared online memory reduces peak token use to about 17K versus 30K for PathAgent and offline storage to under 6 MB, while ablations show structured ROI selection is the main source of accuracy gains.

Motivation and problem framing

Whole-slide image visual question answering (WSI-VQA) requires a system to allocate a small inspection budget over a gigapixel slide before answering a free-form clinical question. The paper frames this as an extreme-context visual search problem and identifies two existing paradigms: supervised pathology MLLMs and agents (SlideChat, WSI-LLaVA, CPathAgent, PathFinder), which absorb localization into learned modules but require task-specific retraining; and training-free agents such as PathAgent, which keep backbones frozen but typically adopt a question-first design, constructing the initial candidate set from query-conditioned relevance. The authors argue this design has two costs: incomplete navigation — pathology questions often underspecify the decisive morphology (e.g., invasive ductal architecture or a small high-grade focus is not named in the question), so a query-conditioned scan can miss it — and heavier inference machinery — weak scan signals force reliance on longer trajectories, reflection rounds, caption caches, and external context, inflating latency and token cost.

The motivating observation comes from human pathology practice: broader scanning behavior correlates with diagnostic accuracy more than zooming alone (2605.23559). This motivates the paper's central proposal, a three-stage Scan-Search-Readout routine in which the agent first builds a slide-specific content prior independent of the question, then applies question relevance only within that prior's candidate set, and finally reuses the same scan state as answer-time slide memory.

Method

PathNavigate instantiates the routine with four modules over frozen backbones (CONCH v1.5 features, PLIP text-image similarity, Patho-R1-7B perceptor, Qwen3.5-4B adjudicator).

Shared online memory. A per-slide two-layer GELU reconstruction MLP ($\mathbb{R}^{768}\to\mathbb{R}^{768}$) is freshly initialized per slide and adapted at test time via self-supervised Huber reconstruction on the tile stream, following test-time training principles. A tile's surprise score $\sigma_i$ is the pre-update gradient norm of the reconstruction loss — measuring how strongly the tile would change the current state under earlier slide context. After a warm-up window of 100 tiles, only tiles exceeding a threshold derived from warm-up statistics trigger updates; others apply multiplicative decay. Because the memory is reinitialized per slide, surprise is a within-slide signal rather than a cohort-level comparison.

Surprise-guided scan. Surprise scores over all low-magnification tiles form a slide-wide surprise field. Thresholding plus distance-based non-maximum suppression yields a spatially diverse ROI pool $\mathcal{R}(q)$. The question acts here only as a lightweight router adjusting NMS spacing and search budget (e.g., denser coverage for morphology questions, wider spacing for clinical questions); it does not rank candidate semantics.

Task-conditioned search. Within the pool, PLIP cosine similarity to the question is computed and fused with min-max-normalized surprise via $$f_\alpha = \alpha\hat r_{\mathrm{PLIP}} + (1-\alpha)\hat\sigma$$, with a fixed symmetric default $\alpha=0.5$. Top-$K$ targets are selected for high-magnification follow-up. The role split is deliberately asymmetric: surprise builds the pool; PLIP only reranks within it.

Evidence readout and QA. Each target is refined at high magnification using a fresh local online memory (so micro-feature ranking does not overwrite global slide state), keeping top-$T=2$ patches per ROI under a global perceptor cap of 15 calls. An optional reference archive retrieves $k=3$ similar cases from a ~4 MB in-domain index. The adjudicator answers from regional evidence strings plus a compact Navigation Summary (surprise mean, standard deviation, high-surprise fraction, candidate count) read out from the same scan state — no separate learned store is introduced.

Main results

On SlideBench-BCNB (7,274 multiple-choice questions across seven diagnostic sub-tasks), PathNavigate achieves the best Overall accuracy at 59.42%, versus 55.72% for PathAgent under the same controlled evaluation stack and 54.07% for supervised SlideChat. Relative to its own frozen perceptor used alone (Patho-R1, 37.26%), the agent loop adds +22.16 points Overall, including +46.12 on Tumor and +22.78 on merged HER2. It is best on ER (76.09), HER2 (48.40), and Molecular Subtype (32.33), runner-up on Tumor and PR, but not dominant on Grading (52.58 vs. Qwen3.5-9B's 59.15).

On WSI-VQA (735 TCGA-BRCA questions), PathNavigate reaches 56.34 Total, 61.00 open-ended accuracy, 60.92 BLEU-1, and 63.53 ROUGE-L — the largest margins over PathAgent appear precisely on open-ended answering (61.00 vs. 13.91) rather than MCQ (52.21 vs. 51.54). Since both agents share the identical perceptor/adjudicator stack, this contrast indicates the gains derive from scan-derived context and high-magnification evidence coverage rather than decoder substitution. On patch-level PathMMU transfer, PathNavigate scores 53.60, marginally above PathAgent's 52.84.

Component analysis

The ablations support a division of labor. Replacing structured ROI proposal with random ROIs causes the largest drop (−6.84 Total on WSI-VQA), identifying the first-pass pool as the primary failure point; removing PLIP costs less (−2.92) but hurts MCQ specifically, where task wording discriminates among plausible regions. On BCNB, replacing the surprise pool with a PLIP-built pool costs −1.81 Overall and −7.98 on Grading, while a random pool costs −5.26 Overall and −12.38 on Grading. Holding targets fixed and varying the readout shows fresh local high-magnification memory outperforms global reuse, random local sampling, and low-magnification-only evidence (BCNB 59.42 vs. 56.33/57.50/52.37), confirming that selected ROIs still require nucleus-resolution morphology. An MLP architecture sweep finds the default two-layer equal-width configuration optimal; collapsing to one linear layer hurts most, while added width or depth does not help. The $\alpha$ sweep peaks at the default 0.5, with both endpoints degraded.

Efficiency

Against PathAgent on identical hardware, PathNavigate maintains comparable latency (~84 s/question median) while reducing peak token consumption (~17K vs. ~30K per question) and offline footprint (a 4 MB case archive plus a <2 MB online-memory MLP versus PathAgent's 277 MB caption cache). Disabling the archive cuts latency to ~28 s/question at a modest accuracy cost. The efficiency claim is explicitly scoped to storage and token budget against training-free agents, not GPU memory parity with trained slide MLLMs.

Limitations and open questions

The paper concedes several boundaries. Tasks depending on information outside visible H&E morphology remain hard even after improved routing: the merged HER2 column stays compressed relative to Tumor/ER because expression level is not directly recoverable from H&E, and Grading remains competitive rather than dominant when cues are diffuse at the selected scale. The question router is a keyword heuristic rather than a learned component, and its routing values are launch-config heuristics whose sensitivity is not fully characterized. Two documented negative results — norm-preserving consolidation of low-surprise patches and episodic trace memory prepended to the adjudicator prompt — did not yield gains and were removed. Single-pass adjudication matched multi-round variants, so reflection was disabled in main results; whether reflection helps under different budgets remains untested. Finally, the surprise mechanism depends on frozen CONCH features; whether the same gradient-norm surprise transfers to other pathology encoders is left open.

Conclusion

PathNavigate demonstrates that reordering navigation — scanning for slide-specific surprise before question matching, reranking within the resulting pool, and reading out the same scan state as answer-time memory — improves WSI-VQA accuracy and evidence-selection interpretability without any task-specific training, while reducing token and storage overhead relative to the strongest training-free baseline. The results position test-time navigation as a first-class design variable for extreme-context visual search with frozen perception and language backbones, while leaving modality-aware evidence beyond H&E as the principal unresolved constraint.