Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoning

Abstract: The processing of gigapixel whole slide images within vision LLMs faces a major difficulty due to an excessive number of visual tokens. Existing solutions typically rely on spatial downsampling or heuristic pruning strategies that operate without training, and these methods often discard subtle but clinically meaningful patterns because pathological evidence is scattered irregularly across the tissue. To overcome this limitation, we reformulate token reduction in whole slide images as a trainable sparsification problem, allowing the model to learn an optimal selection strategy instead of following fixed heuristics. We propose a decoupled routing architecture. To enable gradient propagation through the nondifferentiable pruning operation during training, we introduce a component called SparseLearn. This component uses a variance-preserving noise gate that regulates the information flow of each patch via a differentiable Soft Top-K operator, together with a diagonal attention denoiser that recovers perturbed representations without leaking spatial information. At inference time, the SparseLearn module is entirely discarded, and the trained scorer applies a deterministic Hard Top-K operator to keep only the highest scoring 32 tokens, incurring no extra computation. By compressing the visual sequence down to a sparse set of just 32 tokens, which represents as little as 0.78% of the original length, our framework achieves 73.32% overall accuracy on SlideBench (TCGA), consistently surpassing sampling-based baselines and general-purpose vision LLMs. It also demonstrates strong zero shot generalization on SlideBench (BCNB) and WSI VQA*. By resolving the visual context bottleneck and preventing the dilution of sparse diagnostic evidence, this work provides a highly efficient paradigm for end to end gigapixel whole slide image reasoning.

• The paper presents a fully learnable token sparsification method that reduces gigapixel WSI tokens by up to 128x while maintaining high diagnostic fidelity.
• The methodology features a decoupled routing architecture with continuous variance-preserving noise gating and diagonal denoising, ensuring end-to-end differentiability.
• Experimental results on multiple benchmarks confirm robust cross-cohort generalization and stable performance across varying slide scales.

Learnable Token Sparsification for Efficient Gigapixel Whole Slide Image Reasoning

Introduction

Efficient and accurate analysis of gigapixel whole slide images (WSIs) is severely constrained by the memory and computational bottlenecks imposed by the excessive length of visual token sequences in vision-LLMs (VLMs) and multimodal LLMs. Traditional approaches—aggressive spatial downsampling, training-free token sampling, or attention-based heuristics—often fail to retain the minute but critical morphological signals required for robust diagnostic performance. This work recontextualizes the visual token reduction problem as a learnable sparsity optimization, introducing a fully trainable, context-aware pruning paradigm that circumvents both semantic dilution and the non-differentiability of discrete selection.

Figure 1: Illustration of WSI sparsity and pruning methods. The left panel highlights critical tumor regions submerged within redundant tissue; the right panel contrasts training-free sampling and learnable pruning using loss feedback.

Methodology

The proposed framework is grounded in a decoupled routing architecture. After feature extraction by frozen vision and text encoders, a learnable Scorer estimates per-token diagnostic importance. Pruning is not performed at training time to preserve gradient flow; instead, a Soft Top-K operator assigns continuous importance weights under a strict sparsity budget. To approximate hard selection while maintaining differentiability, a variance-preserving (VP) noise gate stochastically perturbs tokens in proportion to their scored importance, and a diagonal-attention denoiser stabilizes perturbed features, strictly preventing cross-token semantic leakage. During inference, both noise and denoising modules are detached, and a deterministic Hard Top-K operator selects the top $K$ tokens for the LLM.

Figure 2: Framework overview. Training uses VP noise injection and diagonal-only denoising to enable end-to-end gradient flow while learning optimal token retention; inference uses deterministic scoring.

This construct ensures that the pruning process is optimized via standard downstream losses (e.g., next-token prediction) and does not require architectural modifications or auxiliary objectives. The use of a fixed-length compact set, e.g., $K=32$ tokens, dramatically reduces the sequence length (by up to $128\times$) without impairing the reasoning capacity of the overarching LLM.

Experimental Results

Diagnostic Accuracy and Token Efficiency

The approach is evaluated on SlideBench (TCGA), SlideBench (BCNB), and WSI-VQA*. Under an aggressive 32-token budget (representing 0.78% of the original sequence length), the model attains 73.32% overall accuracy on SlideBench (TCGA), outperforming all spatial sampling and non-learnable pruning baselines. On the Diagnosis subset—requiring nuanced multimodal reasoning—the method retains over 96% of the uncompressed upper bound performance (70.09% vs 73.27%), while using only 1.3% of the computational FLOPs.

Figure 3: Accuracy and token efficiency. High diagnostic accuracy is retained with 32 tokens—128 times fewer than the full SlideChat baseline and significantly compressed versus standard VLMs.

Zero-shot generalization to SlideBench (BCNB) and WSI-VQA* benchmarks substantiate the framework's robust cross-cohort transfer and confirm that learned semantic-aware selection reliably extracts domain-invariant features, mitigating spatial overfitting.

Scale-Invariant Performance

A central claim is the insensitivity of diagnostic accuracy to raw input sequence length. Across WSIs with patch counts ranging from 64 to over 10,000, the interval accuracy remains stably aligned to the global average, demonstrating that the model’s high-level reasoning is governed by the sparsity-constrained selection module and not by the physical scale or background redundancy.

Figure 4: Performance stability. The diagnostic accuracy remains consistent across exponentially varying patch counts, confirming immunity to gross input scale.

Topological Analysis of Selected Features

UMAP projections of latent token embeddings illustrate that pruned tokens form compact clusters along topological extremities, signifying that the selection mechanism robustly isolates morphologically discriminative outliers. Central dense manifolds are composed almost entirely of discarded, semantically redundant normal tissue.

Figure 5: UMAP visualization. Retained tokens cluster at feature-space boundaries, representing diagnostic outliers without semantic blurring.

Ablation Studies

Ablation reveals the necessity of both VP noise gating and diagonal-only denoising. Substituting these for scale gating or standard global self-attention degrades accuracy (~3% and ~1.7%, respectively). This quantitatively validates that stochastic, variance-preserving noise more faithfully simulates hard removal for frozen LLMs than naive scaling and that diagonal isolation strictly prevents bypassing the intended information bottleneck.

Theoretical and Practical Implications

This work demonstrates that differentiable, end-to-end learnable token sparsification enables LLMs to operate efficiently and robustly on gigapixel WSIs, achieving compression ratios of up to $128\times$ with minimal loss of diagnostic fidelity. The decoupled Soft/Hard Top-K approach, supported by VP noise and diagonal denoising, provides a mathematically principled substitute for non-differentiable selection—resolving longstanding limitations of surrogate gradients and hard gating in weakly supervised dense prediction.

The implications are substantial for practical deployment of VLM-driven pathology assistants: models can streamline reasoning, minimize memory/compute footprints, and still preserve high-fidelity, diagnosis-critical information, enabling routine clinic-scale integration and interactive slide-level analysis.

Limitations and Future Directions

A core architectural constraint is the isolation of cross-token communication within the scorer prior to selection; this may preclude modeling long-range or diffuse morphological dependencies, potentially omitting higher-order spatial features. The enforced static token budget further constrains adaptability to slides with vastly varying diagnostic densities. Future extensions could involve dynamic sparsity allocation conditioned on global slide context and incorporation of explicit graph-structural priors in cross-token scoring. Efficient, learnable summarization for foundation models in digital pathology remains an open avenue.

Conclusion

This study introduces a fully learnable, end-to-end differentiable token sparsification architecture tailored for high-resolution WSI reasoning. By exploiting continuous VP noise gating during optimization and deterministic Top-K selection at inference, the paradigm achieves unprecedented computational efficiency without sacrificing diagnostic accuracy or generalization. This offers a scalable blueprint for future multimodal LLM systems in computational pathology and related domains.

(2606.08641)