WISE-FUSE: Adaptive WSI Encoding

• WISE-FUSE is an adaptive WSI encoding framework that reduces high-resolution patch usage to 10% while maintaining diagnostic performance.
• It uses a coarse-to-fine pipeline with Cross-Scale Visual Prompt Tuning to transfer fine-grained details from high to low resolutions.
• The framework fuses VLM-derived visual features with LLM-generated morphological cues to achieve over threefold reduction in encoding time.

Searching arXiv for the WISE-FUSE paper and a few directly related pathology VLM papers for grounding. arXiv search: "WISE-FUSE (Shin et al., 20 Aug 2025)" WISE-FUSE is an adaptive whole slide image (WSI) encoding framework for computational pathology that addresses the gigapixel-scale cost of slide processing by selectively encoding diagnostically relevant regions rather than exhaustively processing tens to hundreds of thousands of high-resolution patches per slide (Shin et al., 20 Aug 2025). It combines pathology-domain vision-LLMs (VLMs) and LLMs in a coarse-to-fine pipeline: low-resolution patches are screened against class-specific textual descriptions, a small subset of informative regions is retained, the corresponding high-resolution patches are encoded, and visual and textual features are fused to reinforce diagnostic context (Shin et al., 20 Aug 2025). The framework is presented as a scalable and practical response to the fact that WSI encoding is often the dominant bottleneck in real-world deployment, with preprocessing and training times extending to days or even weeks (Shin et al., 20 Aug 2025).

1. Problem formulation and computational motivation

WISE-FUSE is situated in computational pathology under the specific constraint that WSIs are gigapixel images whose standard processing pipelines often require exhaustive extraction and encoding of high-resolution patches (Shin et al., 20 Aug 2025). The central problem is therefore not only predictive accuracy, but also the prohibitive computational cost of high-resolution feature extraction.

The framework targets this bottleneck through selective processing. The paper states that WISE-FUSE first processes low-resolution patches to quickly discard diagnostically irrelevant regions, then focuses computation on the most informative high-resolution regions (Shin et al., 20 Aug 2025). This establishes a coarse-to-fine selection regime in which early-stage filtering is intended to preserve downstream diagnostic performance while materially reducing encoding cost.

A key claim is that WISE-FUSE reduces high-res patch usage to 10% relative to exhaustive approaches and reduces encoding time by over fourfold, with the example that TCGA-NSCLC processing is reduced from two weeks to under three days (Shin et al., 20 Aug 2025). The abstract states the runtime reduction more conservatively as over threefold, while also reporting diagnostic performance comparable to or surpassing exhaustive patch processing (Shin et al., 20 Aug 2025). This suggests that the method is designed to alter the computational profile of WSI analysis without requiring region-of-interest annotations.

2. System architecture and coarse-to-fine pipeline

The pipeline begins with representative WSI selection and low-resolution patch extraction. For each class, representative WSIs are selected based on similarity of pathology reports encoded via the VLM text encoder. For a dataset $\mathcal{D}_c$ for class $c$, and each report $r_i$, the score is

$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$

and the top $n$ slides, with the example $n=5$, are selected per class for distillation (Shin et al., 20 Aug 2025).

Low-resolution patch representations are then compared against class-specific textual descriptions. For each class $c$, the text embedding is defined as

$$E_{\text{text}, c} = \frac{1}{2} \left( E_{\text{class}, c} + E_{\text{morph}, c} \right),$$

where the formulation explicitly averages class-name text features and morphology-oriented text features derived from LLMs (Shin et al., 20 Aug 2025). For each low-resolution patch $x_i^{(l)}$, the prompt-tuned VLM produces

$E^{(l)}_i = V_{\text{VLM}}(x_i^{(l)}, P),$

and class-wise cosine similarity is computed as

$c$0

Patch selection uses two statistics derived from these scores: the mean similarity $c$1 and the standard deviation $c$2 (Shin et al., 20 Aug 2025). The framework selects the top $c$3 patches by mean similarity and the top $c$4 among the remaining patches by standard deviation. In the terminology of the paper, the first criterion captures high diagnostic relevance for any class, while the second captures class-discriminativeness (Shin et al., 20 Aug 2025).

After this coarse screening, the corresponding 20$c$5 high-resolution regions are extracted for downstream analysis (Shin et al., 20 Aug 2025). The resulting design is explicitly coarse-to-fine: low-resolution inference is used for elimination, and high-resolution encoding is reserved for a reduced candidate set.

3. Cross-Scale Visual Prompt Tuning and knowledge distillation

A central methodological component is Cross-Scale Visual Prompt Tuning (CS-VPT), described as a mechanism for knowledge distillation that enables low-resolution patch representations to retain fine-grained diagnostic features otherwise visible only at high magnification (Shin et al., 20 Aug 2025). Its stated goal is to transfer fine-grained information from 20$c$6 high-res patches to 5$c$7 low-res embeddings.

For each low-resolution patch $c$8, the method collects the corresponding set $c$9 of high-resolution patches and their embeddings $r_i$0. A global embedding is formed by averaging:

$r_i$1

The prompt-tuned low-resolution embedding is

$r_i$2

where $r_i$3 is a set of learnable prompts (Shin et al., 20 Aug 2025).

Training uses two losses. The global alignment objective is a KL divergence between softened global high-resolution and low-resolution representations:

$r_i$4

The local discrimination term is a binary cross-entropy objective over a discriminator $r_i$5 that determines whether a high-resolution patch belongs to the same region:

$r_i$6

The total objective is

$r_i$7

Only the prompts $r_i$8 and the projection layer are trained; the rest of the VLM is frozen (Shin et al., 20 Aug 2025).

Within the paper’s framing, CS-VPT is what makes coarse selection diagnostically credible rather than merely efficient. The ablation summary reports that CS-VPT brings appreciable accuracy gains over vanilla selection (Shin et al., 20 Aug 2025). A plausible implication is that WISE-FUSE depends not only on sparse region selection, but also on scale-bridging supervision that prevents low-resolution screening from collapsing into purely texture-level filtering.

4. VLM-LLM knowledge fusion

WISE-FUSE uses what the paper calls vision-language and LLM knowledge fusion (Shin et al., 20 Aug 2025). Domain-adapted VLMs, including CONCH, PLIP, and BiomedCLIP, provide patch-text alignment, while LLMs such as GPT-4o and Gemini contribute fine-grained morphological descriptions that enhance diagnostic context (Shin et al., 20 Aug 2025).

The LLM component is used to generate concise, diagnostic morphological features for each class as observed at high magnification (Shin et al., 20 Aug 2025). These descriptions are encoded with the VLM text encoder and incorporated into the class text representation. Thus, the text side is not restricted to class labels; it includes morphology-aware prompts intended to better align with pathology semantics.

At the feature-fusion stage, patch-specific text features are computed as a similarity-weighted mixture over class text embeddings:

$r_i$9

The high-resolution visual embedding for a selected patch, $$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$0, is then concatenated with the patch-specific text representation:

$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$1

The paper states that this enriched feature set is then used for downstream multiple instance learning (MIL) tasks such as classification and survival prediction (Shin et al., 20 Aug 2025).

This fusion scheme has two distinct roles. First, text embeddings guide region selection through low-resolution similarity scoring. Second, text features remain in the representation after patch selection through explicit concatenation with high-resolution visual features (Shin et al., 20 Aug 2025). This suggests that language supervision is not merely a retrieval aid, but an active component of the final slide encoding.

5. Empirical results and ablation findings

The reported evaluations cover TCGA-BRCA, NSCLC, RCC, COAD, and STAD datasets, and include subtyping, survival, and other tasks (Shin et al., 20 Aug 2025). Across these settings, WISE-FUSE is reported to match or outperform “20$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$2 all” baselines in accuracy, weighted F1, and AUC, despite using only 10% of high-resolution patches (Shin et al., 20 Aug 2025). It also consistently outperforms random 10% patch sampling, denoted “20$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$3 10%” (Shin et al., 20 Aug 2025).

The computational gains are central to the empirical claim set. The method reduces high-resolution patch usage to 10% and reduces encoding time over fourfold, while the low-resolution processing and CS-VPT overhead are described as negligible relative to the high-resolution savings (Shin et al., 20 Aug 2025). The abstract summarizes the efficiency result as reducing WSI encoding time by over threefold (Shin et al., 20 Aug 2025). Read together, these statements indicate substantial runtime savings, with the stricter numerical example supplied in the detailed results and the broader claim in the abstract.

The ablation findings isolate three contributing factors. CS-VPT improves over vanilla selection; adding LLM morphological descriptors to patch prompts further improves selection; and full knowledge fusion via visual-text concatenation provides the largest single boost, with the example that BRCA subtyping accuracy rises from 86.3% to 89.3% (Shin et al., 20 Aug 2025). Qualitative analysis further reports that selected patches closely align with human-annotated diagnostic or attention regions, and that focus improves after CS-VPT (Shin et al., 20 Aug 2025).

The following summary organizes the empirical claims reported for WISE-FUSE.

Aspect	Reported finding	Source
High-res patch usage	Reduced to 10%	(Shin et al., 20 Aug 2025)
Encoding time	Reduced over fourfold; abstract states over threefold	(Shin et al., 20 Aug 2025)
Baseline comparison	Matches or outperforms “20$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$4 all”; outperforms “20$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$5 10%”	(Shin et al., 20 Aug 2025)
Ablation	CS-VPT helps; LLM descriptors help; full fusion gives the largest single boost	(Shin et al., 20 Aug 2025)
Example gain	BRCA subtyping accuracy from 86.3% to 89.3%	(Shin et al., 20 Aug 2025)

6. Scope, properties, and limitations

The paper characterizes WISE-FUSE as both task-agnostic and model-agnostic (Shin et al., 20 Aug 2025). It can be applied to any CLIP-like VLM, is compatible with various downstream tasks, and does not require labor-intensive region-of-interest annotations (Shin et al., 20 Aug 2025). In the reported experiments, it is also described as working well across different VLM backbones, specifically CONCH, PLIP, and BiomedCLIP (Shin et al., 20 Aug 2025).

This model-agnostic framing does not imply independence from foundation-model quality. The paper explicitly notes that performance is inherently limited by the capabilities and domain adaptation of the underlying VLMs and LLMs (Shin et al., 20 Aug 2025). It also identifies a lower bound on how aggressive coarse sampling can be: using extremely low resolution such as 1.25$$C_i = \text{Softmax}\left(\sum_{j \in \mathcal{D}_c} \frac{r_i \cdot r_j}{\|r_i\|\|r_j\|}\right)$$6 for initial selection can compromise accuracy (Shin et al., 20 Aug 2025). In zero-shot settings, some normal tissue may still be selected, which the paper describes as expected (Shin et al., 20 Aug 2025).

These limitations clarify a possible misconception. WISE-FUSE is not presented as eliminating the need for high-resolution pathology evidence; rather, it reallocates high-resolution computation to a smaller subset of regions selected using low-resolution screening and text-guided priors (Shin et al., 20 Aug 2025). A plausible implication is that its efficiency gains depend on the assumption that diagnostically salient structure can be made sufficiently visible in coarse representations once cross-scale distillation has been applied.

7. Position within WSI representation learning

Within the terms used in the paper, WISE-FUSE is an efficient WSI encoding framework built around selective region processing, cross-scale distillation, and cross-modal knowledge fusion (Shin et al., 20 Aug 2025). Its contribution is not a single module but a composition of three mechanisms: coarse-to-fine adaptive patch selection, CS-VPT for transferring fine-grained information into low-resolution selection features, and fusion of VLM-derived visual features with LLM-derived morphological cues (Shin et al., 20 Aug 2025).

The paper’s practical framing is explicit. The motivation is that gigapixel WSIs make exhaustive feature extraction prohibitively slow and memory-intensive, while selective focus on a small but highly informative subset of regions can maintain or improve diagnostic accuracy (Shin et al., 20 Aug 2025). The claimed result is a scalable pathway toward real-world deployment in computational pathology, with compatibility with future advances in VLMs and LLMs (Shin et al., 20 Aug 2025).

Future directions named in the paper include improved VLM/LLM adaptation to pathology, semi-supervised or active learning for ROI refinement, and broader clinical validation (Shin et al., 20 Aug 2025). This suggests an emerging line of work in which WSI encoding is increasingly treated as a retrieval-and-fusion problem over heterogeneous knowledge sources, rather than solely as exhaustive image tiling followed by downstream aggregation.