Histo-TransCLIP: Transductive Histopathology

• Histo-TransCLIP is a transductive zero-shot framework for histopathology classification that leverages frozen vision-language models and text-derived pseudo-labels.
• It employs affinity graph construction and Laplacian regularization to jointly reason over patch embeddings, yielding significant accuracy gains over inductive methods.
• Experimental results across diverse histopathology datasets demonstrate efficient convergence and robust performance using Gaussian mixture modeling for class assignment.

Histo-TransCLIP is a transductive inference framework for histopathology classification leveraging vision-LLMs (VLMs) in a zero-shot setting. Unlike existing inductive approaches—which classify each image patch independently—Histo-TransCLIP jointly reasons over the affinity structure of test patch embeddings and zero-shot pseudo-labels derived from text prompts, operating entirely in the representation space of frozen VLMs without requiring any model weights or additional supervision (Zanella et al., 2024).

1. Vision-LLM Foundations and Zero-Shot Setup

Given a set $Q = \{1, \ldots, N\}$ of index-labeled patches sampled from one or more whole-slide images, each patch $i$ is encoded by a frozen vision backbone to obtain $f_i \in \mathbb{R}^d$. For each of $K$ possible tissue classes, a natural-language prompt (e.g., “a pathology tissue showing [class]”) is processed by the frozen text encoder to yield class prototype $t_k \in \mathbb{R}^d$.

Zero-shot classification assigns to each patch $i$ the pseudo-label $\hat{y}_i \in \Delta_K$ (the K-simplex), given by softmaxed cosine similarities: $$\hat{y}_{i,k} = \frac{\exp \bigl( \tau\,f_i^\top t_k \bigr)}{\sum_{j=1}^K \exp \bigl( \tau\,f_i^\top t_j \bigr)}$$ where $\tau > 0$ is the temperature parameter inherited from the VLM.

2. Affinity Graph Construction and Laplacian Regularization

The transductive component uses the affinity structure among all patch embeddings. The unnormalized affinity for patches $i$, $i$0 is calculated as

$i$1

which is equivalent to cosine similarity if $i$2, $i$3 are $i$4-normalized. To ensure computational tractability, each patch $i$5 retains only its top-$i$6 nearest neighbors (with $i$7 in all experiments), rendering the graph sparse with $i$8 nonzero affinities.

These affinities are incorporated in the overall objective via a Laplacian regularization term,

$i$9

which encourages similar patches to share similar class assignment vectors $f_i \in \mathbb{R}^d$0.

3. Joint Transductive Inference via Gaussian Mixture Modeling

Histo-TransCLIP infers the assignments $f_i \in \mathbb{R}^d$1, class means $f_i \in \mathbb{R}^d$2, and shared diagonal covariance $f_i \in \mathbb{R}^d$3 of a $f_i \in \mathbb{R}^d$4-component Gaussian mixture in embedding space, minimizing the aggregate cost: $f_i \in \mathbb{R}^d$5 where $f_i \in \mathbb{R}^d$6 and

$f_i \in \mathbb{R}^d$7

The optimization proceeds via block-coordinate updates:

• E-step: Each $f_i \in \mathbb{R}^d$8 is updated in parallel by

$f_i \in \mathbb{R}^d$9

where $K$0 are the $K$1 nearest neighbors of $K$2.

• M-step: Means and covariance are re-estimated in closed form:

$K$3

This is repeated until convergence (typically 20–30 iterations). Each iteration is $K$4; for $K$5 patches with $K$6, convergence is achieved in approximately 6 s on a single RTX3090.

4. Experimental Design: Datasets and VLM Backbones

Experiments encompass four histopathology datasets, all processed into $K$7 patches:

• NCT-CRC: 100,000 colorectal patches, $K$8 tissue classes.
• SICAP-MIL: Prostate cancer grading, $K$9 Gleason classes.
• SKINCANCER: $t_k \in \mathbb{R}^d$0 anatomical skin structures.
• LC25000 (Lung): $t_k \in \mathbb{R}^d$1 lung cancer subtypes.

Five pretrained/frozen vision-language backbones are used:

• CLIP (ViT-B/16)
• Quilt-B16
• Quilt-B32 (two scales of the 1M histopathology VLM)
• PLIP
• CONCH

Text prompts per class are taken from the template pool of CONCH (22 variants, averaged). All model embeddings are held fixed at inference.

5. Empirical Evaluation: Classification Gains and Ablations

Comparative results demonstrate consistent improvements of Histo-TransCLIP (transductive) over standard zero-shot (inductive) classification. The following table summarizes top-1 accuracy (%) across models and datasets:

Dataset / Model	CLIP	Quilt-B16	Quilt-B32	PLIP	CONCH
SICAP-MIL	29.85→24.72	40.44→58.49	35.04→28.18	46.84→53.23	27.71→32.58
LC(Lung)	31.46→25.62	43.00→50.53	76.24→93.93	84.96→93.80	84.81→96.29
SKINCANCER	4.20→11.46	15.38→33.33	39.71→48.80	22.90→36.72	58.53→66.22
NCT-CRC	25.39→39.61	29.61→48.40	53.73→58.13	63.17→77.53	66.27→70.36
Average	22.73→25.35 (+2.62)	32.10→47.69 (+15.59)	51.18→57.26 (+6.08)	54.47→65.32 (+10.85)	59.33→66.36 (+7.03)

Ablation studies confirm the necessity of affinity-based Laplacian regularization; removing the Laplacian (r=0) results in <1% gain over zero-shot, indicating text-only regularization is insufficient.

6. Computational Aspects and Black-Box Constraints

All hyperparameters, including temperature $t_k \in \mathbb{R}^d$2, are inherited directly from the VLM; no tuning is performed. The number of neighbors is fixed at $t_k \in \mathbb{R}^d$3.

Efficiency metrics on RTX3090 (Quilt-B16 backbone):

$t_k \in \mathbb{R}^d$4 (patches)	Feature extraction	Histo-TransCLIP
$t_k \in \mathbb{R}^d$5	1 s	0.1 s
$t_k \in \mathbb{R}^d$6	4 s	0.2 s
$t_k \in \mathbb{R}^d$7	28 s	0.4 s
$t_k \in \mathbb{R}^d$8	5 min	6 s

Memory usage is dominated by storage of $t_k \in \mathbb{R}^d$9 patch embeddings and $i$0 affinity edges, not model parameters. The framework operates entirely in black-box regime, requiring no access to model weights.

7. Synthesis and Methodological Significance

Histo-TransCLIP integrates zero-shot text supervision, expressed as a KL divergence penalty to the VLM’s softmax outputs, with unsupervised exploration of the intrinsic structure of test patch embeddings via sparse graph Laplacian and GMM. The transductive inference paradigm supports large-scale histopathological classification with consistent and substantial accuracy improvements (up to +15.6%), at negligible additional cost. This suggests that exploiting affinities among test instances—rather than treating patches independently—can unlock latent discriminative signal even in zero-shot, label-free scenarios (Zanella et al., 2024). A plausible implication is broader applicability across other domains where patch-level contextual dependencies are salient and model weights remain inaccessible.