TCP-LLaVA: Token Compression for WSI-VQA

• The paper introduces TCP-LLaVA, the first MLLM to use token compression for WSI-VQA by condensing thousands of image patches into around 100 trainable tokens.
• It employs a frozen pretrained visual encoder with linear projections and a text-guided compression module that attends jointly over visual and textual tokens to reduce sequence length.
• This approach significantly cuts computational cost and memory usage while achieving competitive accuracy across various tumor subtypes on TCGA-derived benchmarks.

Token Compression Pathology LLaVA (TCP-LLaVA) is a LLaVA-style multimodal LLM architecture for whole-slide image visual question answering that introduces an explicit token compression stage between patch encoding and language-model decoding. In "Efficient Whole Slide Pathology VQA via Token Compression" (Lyu et al., 19 Jul 2025), TCP-LLaVA is presented as the first MLLM architecture to perform WSI VQA via token compression: a frozen pretrained visual encoder embeds thousands to tens of thousands of WSI patches, a modality compression module distills the resulting visual tokens together with textual context into a compact set of trainable compression tokens, and only the compressed tokens are forwarded to the LLM for answer generation. The acronym also intersects with a broader LLaVA literature on token-compression-induced degradation, where aggressive reduction of visual tokens can impair perception, OCR, grounding, and reasoning (Wang et al., 2024).

1. Problem setting: whole-slide image VQA and sequence-length pressure

Whole-slide images in pathology can reach up to $10{,}000 \times 10{,}000$ pixels, and WSI-VQA pairs a gigapixel WSI with a natural-language question, sometimes with multiple-choice answers (A–D). A WSI is tiled into non-overlapping $224 \times 224$ patches, yielding thousands to tens of thousands of patches per slide. If each patch contributes a visual token to an LLM, the sequence length can easily exceed 10,000 tokens per slide, which stresses context-length limits and induces quadratic attention costs in both memory and compute (Lyu et al., 19 Jul 2025).

The long-sequence bottleneck is formalized by letting $L = L_v + L_t$, where $L_v$ denotes visual tokens and $L_t$ denotes text tokens. Standard decoder self-attention then scales as $O(L^2)$ in time and $O(L^2)$ in memory. In WSI-VQA, $L_v > 10{,}000$ and typical $L_t \ll L_v$, so the visual side dominates the computational profile. This is the central systems problem that TCP-LLaVA addresses (Lyu et al., 19 Jul 2025).

Prior pathology pipelines illustrate the trade-off. CLIP+MIL models compress patch features into a single slide descriptor and work well for classification, but they lack the generative capacity and dialog-style reasoning required for VQA. WSI-level MLLMs such as SlideChat directly feed thousands of patch tokens to an LLM to enable VQA, but this is resource-heavy and scales poorly with WSI size. TCP-LLaVA is positioned between these regimes: it preserves the generative advantages of MLLMs while replacing the raw visual-token stream with a compact learned representation (Lyu et al., 19 Jul 2025).

2. Architecture and modality compression mechanism

TCP-LLaVA consists of four stages: WSI preprocessing and patch encoding, text tokenization, modality compression, and LLM decoding. A frozen pretrained visual encoder initialized from CONCH embeds each patch, and a linear projection aligns visual features to the LLM hidden size $d$, producing visual tokens $224 \times 224$0. The question, and optionally multiple-choice options, are tokenized by the LLM tokenizer to produce textual embeddings $224 \times 224$1. The language backbone is instantiated from Qwen2.5-7B-Instruct in a LLaVA-style integration (Lyu et al., 19 Jul 2025).

The compression stage maintains $224 \times 224$2 trainable compression tokens $224 \times 224$3, with $224 \times 224$4 as the recommended default. These tokens jointly attend over the concatenated multimodal sequence $224 \times 224$5. In the paper’s single-head formulation,

$224 \times 224$6

$224 \times 224$7

$224 \times 224$8

With residual, normalization, and feed-forward structure, one compression layer is written as

$224 \times 224$9

In practice, the modality compression module is initialized from the early layers of the target LLM, so its blocks already contain multi-head attention, residual connections, layer norms, and MLPs (Lyu et al., 19 Jul 2025).

Conceptually, each compression token functions as a learnable $L = L_v + L_t$0-like query over visual and textual context. Only the compressed tokens are forwarded as the “image-side” evidence to the LLM, while the question prompt itself is still provided to the LLM as ordinary text tokens for generation. The model therefore replaces thousands of visual tokens with approximately 100 compressed tokens while keeping the question text intact (Lyu et al., 19 Jul 2025).

This joint compression is text-conditioned rather than purely visual. Appendix t-SNE visualizations show that the compressed tokens cluster tightly and lie close to the text token cluster, while the uncompressed visual tokens are spread. The paper interprets this as evidence that compression is heavily text-guided: the module distills image evidence most relevant to the question, rather than producing a generic slide summary (Lyu et al., 19 Jul 2025).

3. Complexity, efficiency, and training protocol

TCP-LLaVA changes the computational profile of WSI-VQA in two places. First, the compression module performs cross-attention from $L = L_v + L_t$1 queries to $L = L_v + L_t$2 keys and values, which costs $L = L_v + L_t$3 per layer when suppressing constant factors. Second, the LLM no longer receives $L = L_v + L_t$4 raw visual tokens; instead, its effective sequence is $L = L_v + L_t$5, so the per-layer self-attention cost becomes $L = L_v + L_t$6 instead of $L = L_v + L_t$7. The dominant complexity therefore switches from

$L = L_v + L_t$8

to

$L = L_v + L_t$9

With $L_v$0 and $L_v$1, this yields large savings (Lyu et al., 19 Jul 2025).

The reported empirical gains are consistent with that analysis. Replacing $L_v$2 with $L_v$3 reduces the visual-token dimension by more than 99%. The paper reports a $L_v$4 reduction in LLM input length on the visual side; training TFLOPS increases from 2.35 for SlideChat to 10.87 for TCP-LLaVA on a single NVIDIA A6000 (48 GB) under batch size 1 and no gradient accumulation; training throughput improves from 0.42 to 179.46 samples/s; inference throughput improves from 0.58 to 3.33 samples/s; and end-to-end wall-clock for 10,000 samples is 0.67 hours for TCP-LLaVA versus 1 hour for SlideChat under partial fine-tuning, a $L_v$5 reduction in training time (Lyu et al., 19 Jul 2025).

The training setup is deliberately narrow in scope. The visual encoder and LLM are frozen; only the projector and the compression module are trained. Optimization uses AdamW, linear warmup plus cosine decay, learning rate $L_v$6, 2 epochs, batch size 1, 8 gradient-accumulation steps, and FP16 mixed precision. The objective is standard autoregressive language modeling over the answer tokens $L_v$7: $L_v$8 where $L_v$9 denotes the final compressed tokens and $L_t$0 the question prompt (Lyu et al., 19 Jul 2025).

4. Benchmark results on TCGA-derived WSI-VQA

The evaluation dataset is a TCGA-derived WSI-VQA benchmark with ten tumor subtypes: BLCA, BRCA, COAD, GBM, HNSC, LGG, LUAD, LUSC, READ, and SKCM. At the QA level, the dataset contains tens of thousands of pairs per subtype, including BRCA 37,564; LGG 30,074; GBM 21,009; LUAD 17,969; LUSC 16,438; BLCA 15,294; COAD 14,481; HNSC 13,615; READ 5,287; and SKCM 4,066. The WSI split is 8:1:1 train/val/test per tumor, with no slide overlap across splits. Questions are multiple-choice with four options, and the model generates a free-form answer string that includes the chosen letter, which is parsed to compute accuracy (Lyu et al., 19 Jul 2025).

Average accuracy favors TCP-LLaVA over both generic MLLMs and WSI-oriented baselines.

Model	Setting	Average accuracy
LLaVA-v1.6-Vicuna-7B	30-patch setting per slide	34.63%
LLaVA-Med	30-patch setting per slide	44.35%
Quilt-LLaVA	30-patch setting per slide	38.99%
SlideChat	WSI-level MLLM	77.23%
SlideChat + DivPrune	token pruning	77.18%
TCP-LLaVA	token compression	78.57%

Representative per-subtype comparisons show mixed but generally favorable behavior relative to SlideChat. TCP-LLaVA reaches BLCA 83.54 vs 81.65, COAD 76.43 vs 74.52, GBM 80.00 vs 75.00, LUAD 76.44 vs 74.87, READ 87.84 vs 85.14, and SKCM 82.35 vs 79.41. It ties on HNSC at 78.79 and LGG at 79.34, while BRCA is 67.56 vs 69.57 and LUSC is 73.45 vs 74.01. The paper’s stated takeaways are that MLLMs specialized for WSI outperform generic LLaVA variants on this task, direct token pruning preserves SlideChat’s performance but does not improve it, and compression-based TCP-LLaVA slightly improves accuracy over the strongest WSI baseline while cutting cost (Lyu et al., 19 Jul 2025).

Ablation studies further constrain the interpretation. Replacing token compression with ACMIL inside the SlideChat pipeline yields comparable or lower accuracy: on BRCA, both methods achieve 67.56%; on LUAD, TCP-LLaVA improves from 74.35% for ACMIL+SlideChat to 75.39%. Varying the number of compression tokens from 100 to 4,000 produces small performance variation on BRCA, approximately 67.1–67.8%, and modest gains on LUAD, 74.35% to 75.92%. This indicates diminishing returns beyond a few hundred tokens, with $L_t$1 already capturing most of the benefit at a fraction of the cost (Lyu et al., 19 Jul 2025).

5. Relation to token compression pathology in the broader LLaVA literature

The name TCP-LLaVA belongs to a pathology-domain model in (Lyu et al., 19 Jul 2025), but the broader LLaVA literature uses closely related analyses to describe failure modes that occur when visual tokens are compressed too aggressively. "LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information" (Wang et al., 2024) diagnoses visual token overload and performance degradation under fixed or naive compression; "LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models" (Shang et al., 2024) frames the issue in terms of detail loss, semantic drift, hallucination, degraded spatial reasoning, and instability across inputs; and "LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token" (Zhang et al., 7 Jan 2025) attributes the brittleness of direct compression to the fact that many vision tokens are crucial in the early layers of the LLM backbone.

The quantitative pattern in general-purpose LLaVA is severe. In LLaVA-Zip, static compression at inference degrades LLaVA-1.5 from 576 to 144 to 64 visual tokens with benchmark drops such as GQA 62.78 $L_t$2 56.63 $L_t$3 52.55, MM-Vet 33.62 $L_t$4 24.77 $L_t$5 19.49, and TextVQA 46.82 $L_t$6 27.08 $L_t$7 16.38. The paper describes this as strong empirical evidence that static compression harms tasks that depend on spatial detail and textual grounding, with TextVQA suffering most (Wang et al., 2024).

Different mitigation strategies follow different diagnoses.

Paper	Core mechanism	Representative evidence
LLaVA-Zip (Wang et al., 2024)	Dynamic Feature Map Reduction using $L_t$8 and threshold $L_t$9	At 64 tokens, DFMR gives TextVQA 32.03 vs 16.38 for static LLaVA-1.5
LLaVA-PruMerge (Shang et al., 2024)	IQR-based adaptive token selection from $O(L^2)$0 patch attention plus key-based KNN merging	$O(L^2)$1 average compression; FP16 prefill FLOPs 9.3 $O(L^2)$2 0.91
LLaVA-Mini (Zhang et al., 7 Jan 2025)	Modality pre-fusion outside the LLM plus query-based compression to one token	1 vision token instead of 576; FLOPs reduced by 77%

In LLaVA-Zip, DFMR is inserted between the CLIP vision encoder and the projector, computes the mean of patch standard deviations $O(L^2)$3, selects a compression factor $O(L^2)$4 according to threshold $O(L^2)$5, and then applies $O(L^2)$6 average pooling. Its 64-token results improve substantially over static compression: GQA 59.72 vs 52.55, LLaVA-Bench 62.30 vs 47.50, and TextVQA 32.03 vs 16.38 (Wang et al., 2024).

In PruMerge, token importance is derived from sparse attention between the class token and visual tokens. Outliers are selected by interquartile-range rules, retained tokens become cluster centers, and pruned information is merged back into them via key-similarity KNN and attention-weighted averaging. The paper reports 18× average compression in the abstract, 5.5% tokens retained across six tasks in the detailed analysis, FP16 prefill FLOPs 9.3 $O(L^2)$7 0.91, prefill time 88.6 ms $O(L^2)$8 15.3 ms, and activation memory 4.60 GB $O(L^2)$9 0.28 GB on V100 with Vicuna-7B (Shang et al., 2024).

In LLaVA-Mini, the central claim is that most vision tokens only play a crucial role in the early layers of the LLM backbone, where they mainly fuse visual information into text tokens. The model therefore moves this fusion stage outside the LLM with $O(L^2)$0 Transformer blocks and then compresses the visual stream with learnable queries. Under the extreme setting $O(L^2)$1, one image is represented by a single token. The paper reports that LLaVA-Mini outperforms LLaVA-v1.5 with just 1 vision token instead of 576, reduces FLOPs by 77%, delivers low-latency responses within 40 milliseconds, and can process over 10,000 frames of video on 24 GB of GPU memory (Zhang et al., 7 Jan 2025).

Taken together, these works indicate that token compression in LLaVA-like MLLMs is not a single technique but a design space. TCP-LLaVA in pathology uses trainable compression tokens over $O(L^2)$2; DFMR uses intrinsic image variability; PruMerge uses attention sparsity plus merging; and LLaVA-Mini uses pre-fusion plus query-based compression. A plausible implication is that the pathology-domain TCP-LLaVA belongs to the same broader family of attempts to replace naive visual-token reduction with adaptive, information-preserving compression.

6. Limitations, failure modes, and open directions

TCP-LLaVA remains a lossy compression system. The pathology paper explicitly notes potential information loss: subtle but diagnostically crucial signals, such as rare tumor features occupying few patches, may be discarded. Although average accuracy is strong and GBM improves by 5.00 percentage points relative to SlideChat, BRCA and LUSC show small regressions, which is consistent with possible over-compression or sampling sensitivity. The model also depends on non-overlapping $O(L^2)$3 tiling, so fine-grained or boundary features may be missed, and results depend on the tiling scale and number of extracted patches (Lyu et al., 19 Jul 2025).

The current empirical scope is limited to TCGA-derived slides and question templates refined from SlideBench and WSI-VQA. External generalization, domain shifts such as scanner and staining variation, and artifact robustness were not reported. The supervision target is VQA rather than long-form pathology report generation, and the paper identifies extension to report generation and rationale extraction as future work (Lyu et al., 19 Jul 2025).

Related LLaVA studies show that compression pathologies are mitigated rather than eliminated. LLaVA-Zip reports that extreme compression still hurts detail-sensitive tasks even when DFMR is used, for example TextVQA 47.13 at 576 tokens versus 32.03 at 64 tokens. LLaVA-Mini likewise reports residual deficits on some metrics relative to LLaVA-v1.5, including POPE $O(L^2)$4, TextVQA $O(L^2)$5, and MME $O(L^2)$6, despite strong overall averages (Wang et al., 2024, Zhang et al., 7 Jan 2025). This suggests that aggressively shortened visual prefixes remain vulnerable when tasks require OCR, counting, or highly localized evidence.

The future directions listed for TCP-LLaVA follow naturally from those limits: adaptive or hierarchical compression, multi-scale WSIs, pathology priors such as nuclei or tissue segmentation cues, retrieval-augmented patch selection, and extensions beyond QA. In the broader LLaVA compression literature, corresponding open questions concern how to choose compression ratios dynamically, how to preserve rare but important signals, and how to distribute compute between fusion and compression stages without reintroducing the quadratic cost that token compression is intended to avoid (Lyu et al., 19 Jul 2025, Zhang et al., 7 Jan 2025).