--- title: 'TCP-LLaVA: Token Compression for WSI-VQA' url: https://www.emergentmind.com/topics/token-compression-pathology-llava-tcp-llava type: topic --- # TCP-LLaVA: Token Compression for WSI-VQA Token Compression Pathology LLaVA (TCP-LLaVA) is a LLaVA-style multimodal large language model architecture for whole-slide image visual question answering that introduces an explicit token compression stage between patch encoding and language-model decoding. In "Efficient Whole Slide Pathology VQA via Token Compression" [2507.14497], TCP-LLaVA is presented as the first MLLM architecture to perform WSI VQA via token compression: a frozen pretrained visual encoder embeds thousands to tens of thousands of WSI patches, a modality compression module distills the resulting visual tokens together with textual context into a compact set of trainable compression tokens, and only the compressed tokens are forwarded to the LLM for answer generation. The acronym also intersects with a broader LLaVA literature on token-compression-induced degradation, where aggressive reduction of visual tokens can impair perception, OCR, grounding, and reasoning [2412.08771]. ## 1. Problem setting: whole-slide image VQA and sequence-length pressure Whole-slide images in pathology can reach up to \(10{,}000 \times 10{,}000\) pixels, and WSI-VQA pairs a gigapixel WSI with a natural-language question, sometimes with multiple-choice answers (A–D). A WSI is tiled into non-overlapping \(224 \times 224\) patches, yielding thousands to tens of thousands of patches per slide. If each patch contributes a visual token to an LLM, the sequence length can easily exceed 10,000 tokens per slide, which stresses context-length limits and induces quadratic attention costs in both memory and compute [2507.14497]. The long-sequence bottleneck is formalized by letting \(L = L_v + L_t\), where \(L_v\) denotes visual tokens and \(L_t\) denotes text tokens. Standard decoder self-attention then scales as \(O(L^2)\) in time and \(O(L^2)\) in memory. In WSI-VQA, \(L_v > 10{,}000\) and typical \(L_t \ll L_v\), so the visual side dominates the computational profile. This is the central systems problem that TCP-LLaVA addresses [2507.14497]. Prior pathology pipelines illustrate the trade-off. CLIP+MIL models compress patch features into a single slide descriptor and work well for classification, but they lack the generative capacity and dialog-style reasoning required for VQA. WSI-level MLLMs such as SlideChat directly feed thousands of patch tokens to an LLM to enable VQA, but this is resource-heavy and scales poorly with WSI size. TCP-LLaVA is positioned between these regimes: it preserves the generative advantages of MLLMs while replacing the raw visual-token stream with a compact learned representation [2507.14497]. ## 2. Architecture and modality compression mechanism TCP-LLaVA consists of four stages: WSI preprocessing and patch encoding, text tokenization, modality compression, and LLM decoding. A frozen pretrained visual encoder initialized from CONCH embeds each patch, and a linear projection aligns visual features to the LLM hidden size \(d\), producing visual tokens \(X \in \mathbb{R}^{L_v \times d}\). The question, and optionally multiple-choice options, are tokenized by the LLM tokenizer to produce textual embeddings \(T \in \mathbb{R}^{L_t \times d}\). The language backbone is instantiated from Qwen2.5-7B-Instruct in a LLaVA-style integration [2507.14497]. The compression stage maintains \(K\) trainable compression tokens \(C \in \mathbb{R}^{K \times d}\), with \(K = 100\) as the recommended default. These tokens jointly attend over the concatenated multimodal sequence \([X; T]\). In the paper’s single-head formulation, \[ Q_c = C W_q, \qquad K_{\text{all}} = [X; T] W_k, \qquad V_{\text{all}} = [X; T] W_v, \] \[ A = \operatorname{softmax}\!\left(\frac{Q_c K_{\text{all}}^\top}{\sqrt{d}}\right) \in \mathbb{R}^{K \times (L_v + L_t)}, \] \[ Z = A V_{\text{all}} \in \mathbb{R}^{K \times d}. \] With residual, normalization, and feed-forward structure, one compression layer is written as \[ \tilde{Z} = \operatorname{LN}(C + Z), \qquad \hat{Z} = \operatorname{LN}(\tilde{Z} + \operatorname{MLP}(\tilde{Z})). \] In practice, the modality compression module is initialized from the early layers of the target LLM, so its blocks already contain multi-head attention, residual connections, layer norms, and MLPs [2507.14497]. Conceptually, each compression token functions as a learnable \([CLS]\)-like query over visual and textual context. Only the compressed tokens are forwarded as the “image-side” evidence to the LLM, while the question prompt itself is still provided to the LLM as ordinary text tokens for generation. The model therefore replaces thousands of visual tokens with approximately 100 compressed tokens while keeping the question text intact [2507.14497]. This joint compression is text-conditioned rather than purely visual. Appendix t-SNE visualizations show that the compressed tokens cluster tightly and lie close to the text token cluster, while the uncompressed visual tokens are spread. The paper interprets this as evidence that compression is heavily text-guided: the module distills image evidence most relevant to the question, rather than producing a generic slide summary [2507.14497]. ## 3. Complexity, efficiency, and training protocol TCP-LLaVA changes the computational profile of WSI-VQA in two places. First, the compression module performs cross-attention from \(K\) queries to \(L_v + L_t\) keys and values, which costs \(O(K(L_v + L_t))\) per layer when suppressing constant factors. Second, the LLM no longer receives \(L_v\) raw visual tokens; instead, its effective sequence is \(L_t + K\), so the per-layer self-attention cost becomes \(O((L_t + K)^2)\) instead of \(O((L_t + L_v)^2)\). The dominant complexity therefore switches from \[ O((L_t + L_v)^2) \] to \[ O(K(L_v + L_t)) + O((L_t + K)^2). \] With \(L_v \gg L_t\) and \(K \ll L_v\), this yields large savings [2507.14497]. The reported empirical gains are consistent with that analysis. Replacing \(L_v \approx 10{,}000\) with \(K = 100\) reduces the visual-token dimension by more than 99%. The paper reports a \(>99\%\) reduction in LLM input length on the visual side; training TFLOPS increases from 2.35 for SlideChat to 10.87 for TCP-LLaVA on a single NVIDIA A6000 (48 GB) under batch size 1 and no gradient accumulation; training throughput improves from 0.42 to 179.46 samples/s; inference throughput improves from 0.58 to 3.33 samples/s; and end-to-end wall-clock for 10,000 samples is 0.67 hours for TCP-LLaVA versus 1 hour for SlideChat under partial fine-tuning, a \(\approx 33\%\) reduction in training time [2507.14497]. The training setup is deliberately narrow in scope. The visual encoder and LLM are frozen; only the projector and the compression module are trained. Optimization uses AdamW, linear warmup plus cosine decay, learning rate \(1.5 \times 10^{-5}\), 2 epochs, batch size 1, 8 gradient-accumulation steps, and FP16 mixed precision. The objective is standard autoregressive language modeling over the answer tokens \(A = (a_1, \ldots, a_T)\): \[ \mathcal{L}_{\text{VQA}} = -\sum_{t=1}^{T} \log P(a_t \mid a_{