GNN-ViTCap: MIL with ViT & GNN for Histopathology

• The paper introduces GNN-ViTCap, a MIL framework that integrates pretrained ViT embeddings, deep embedded clustering, and graph attention networks for effective WSI analysis.
• The architecture combines redundancy reduction, attention-based representative selection, and GNN context aggregation to enable both diagnosis and clinically faithful caption generation.
• Empirical results demonstrate significant performance gains over prior MIL and captioning methods on BreakHis and PatchGastric datasets, with notable improvements in F1, AUC, BLEU-4, and METEOR scores.

GNN-ViTCap is a multimodal, multiple instance learning (MIL)-based framework for histopathology whole slide image (WSI) analysis that jointly addresses slide-level classification and automatic pathology captioning. It was introduced in “GNN-ViTCap: GNN-Enhanced Multiple Instance Learning with Vision Transformers for Whole Slide Image Classification and Captioning” (Raju et al., 9 Jul 2025). The framework is designed for microscopy-driven WSIs in which patch captures may be redundant, absolute patch coordinates may be unavailable, and text generation must remain clinically faithful. Its core pipeline combines a pretrained Vision Transformer (ViT) for patch embedding, Deep Embedded Clustering (DEC) and scalar dot attention for redundancy reduction, a Graph Attention Network (GAT) for contextual aggregation under an MIL formulation, and a learned projection that conditions biomedical LLMs for caption generation (Raju et al., 9 Jul 2025).

1. Problem setting and motivation

The method is motivated by three intertwined difficulties in histopathology image analysis. First, microscopy workflows can produce many overlapping or redundant patches because captures are subjective. Second, unlike scanner-generated WSIs, microscope-acquired WSIs may not provide absolute coordinates for patches, which makes explicit spatial modeling non-trivial. Third, automatic pathology captioning must express lesion subtype, grade, and tissue architecture in appropriate medical language, yet small datasets and long-range visual-language dependencies make this difficult for conventional RNN/LSTM captioners (Raju et al., 9 Jul 2025).

MIL provides the formal setting for the classification problem. A slide is treated as a bag of instances, and the binary MIL label assignment is specified as

$$Y_i = 0 \text{ if } \sum_{j=1}^{n_i} y_{i,j} = 0; \quad 1 \text{ otherwise.}$$

A bag-level prediction is given by a permutation-invariant pooling operator,

$$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$

The paper argues that classical MIL, although natural for WSI classification, treats patches as independent and ignores contextual relationships that pathologists use when reasoning over tissue morphology and inter-region structure (Raju et al., 9 Jul 2025).

A common misconception is that GNN-ViTCap reconstructs physical slide geometry. The formulation does not do so. Because absolute patch positions are unknown, inter-patch structure is inferred from embedding similarity rather than from coordinates. This distinction is central to the method’s design and to its claimed applicability to microscopy WSIs.

2. Architecture and mathematical formulation

The pipeline begins by extracting a variable number of patches from each patient’s microscopic WSI, denoted $X^{(s)}$, with patches $P^{(s)} = \{p_1, \ldots, p_{N_p}\}$. A pretrained ViT encoder $E_v$ maps each patch to an embedding:

$$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$

and the per-slide embedding matrix is

$$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$

The main backbone is ViT-B/16 pretrained on ImageNet-21k, with input resolution $224 \times 224$, internal patch size $16 \times 16$, and embedding dimension $d_v = 768$. The paper also evaluates ResNet-34 with $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$0 as a visual encoder baseline for classification. ViT internals are not explicitly modified in the method (Raju et al., 9 Jul 2025).

After feature extraction, the system removes redundancy, constructs a graph over representative patches, aggregates context with a GNN, and produces a slide-level embedding $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$1. That embedding is used in two downstream heads. For classification, it is passed to an MLP that predicts the binary label. For captioning, it is projected by a linear map $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$2 into the LLM input space to form a visual prefix $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$3, which is concatenated with caption token embeddings to condition generation (Raju et al., 9 Jul 2025).

The full data flow is explicitly defined as patch extraction, vision feature extraction, dynamic redundancy removal via DEC, representative patch selection via scalar dot attention, graph construction from pairwise similarity, GAT-based context aggregation, MIL bag representation, classification through an MLP, and captioning through LLM conditioning. The framework therefore uses the same aggregated visual summary for both diagnosis-oriented prediction and pathologist-style text generation.

3. Redundancy removal, graph construction, and MIL aggregation

Redundancy reduction is handled by Deep Embedded Clustering. Patch embeddings $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$4 are grouped into $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$5 clusters with centers $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$6, using a KL-divergence objective that sharpens soft assignments. With Student’s $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$7-distribution and $$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$8, the soft assignment is

$$\hat{Y}_i = g\big(\mathrm{AvgPool}(f(x_{i,1}), \ldots, f(x_{i,n_i}))\big).$$9

The target distribution is

$X^{(s)}$0

and the clustering loss is

$X^{(s)}$1

The number of clusters is dataset-specific: $X^{(s)}$2 for BreakHis and $X^{(s)}$3 for PatchGastric, with DEC convergence threshold $X^{(s)}$4 (Raju et al., 9 Jul 2025).

Representative patch selection is then performed within each cluster using scalar dot attention. For cluster $X^{(s)}$5, embeddings $X^{(s)}$6 are projected to queries, keys, and values:

$X^{(s)}$7

with $X^{(s)}$8. For a patch $X^{(s)}$9,

$P^{(s)} = \{p_1, \ldots, p_{N_p}\}$0

and

$P^{(s)} = \{p_1, \ldots, p_{N_p}\}$1

The top-1 patch in each cluster is retained,

$P^{(s)} = \{p_1, \ldots, p_{N_p}\}$2

forming the representative set

$P^{(s)} = \{p_1, \ldots, p_{N_p}\}$3

The paper states that this attention reduces intra-cluster redundancy and encourages inter-cluster diversity before graph construction (Raju et al., 9 Jul 2025).

A patient-specific graph is built over the $P^{(s)} = \{p_1, \ldots, p_{N_p}\}$4 representatives. Cosine similarity defines

$P^{(s)} = \{p_1, \ldots, p_{N_p}\}$5

Edge selection applies Gumbel-Softmax $P^{(s)} = \{p_1, \ldots, p_{N_p}\}$6 to discretize neighbor selection:

$P^{(s)} = \{p_1, \ldots, p_{N_p}\}$7

The paper does not specify a fixed $P^{(s)} = \{p_1, \ldots, p_{N_p}\}$8-NN value; edges are learned and discretized from similarities. The graph is then processed by a GAT with $P^{(s)} = \{p_1, \ldots, p_{N_p}\}$9 layers, hidden dimension $E_v$0, and LeakyReLU activation. The node update is

$E_v$1

with attention coefficients

$E_v$2

After $E_v$3 layers, global mean pooling yields the WSI embedding

$E_v$4

In the method’s MIL interpretation, the representative nodes are the instances and the combination of GNN message passing and global mean pooling is the MIL aggregator (Raju et al., 9 Jul 2025).

4. Captioning interface and optimization

The classification head applies an MLP to the bag embedding:

$E_v$5

Binary cross-entropy is

$E_v$6

and the total classification objective is

$E_v$7

This couples diagnostic prediction to the clustering objective that shapes the representative instance set (Raju et al., 9 Jul 2025).

For caption generation, the slide embedding is projected into the LLM input space:

$E_v$8

with $E_v$9 and $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$0. This visual prefix is concatenated with the start-of-sequence caption token embeddings to condition the LLM. The paper evaluates four LLMs: ClinicalT5-Base, BioGPT, LlamaV2-Chat, and BiomedGPT. The token-level cross-entropy captioning loss is

$$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$1

and the total captioning objective is

$$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$2

The design therefore conditions text generation on a slide-level visual summary rather than on independent patch descriptions (Raju et al., 9 Jul 2025).

The paper’s stated rationale is that dynamic clustering plus attention mitigates redundancy and noise, the GNN learns relational structure despite missing coordinates, ViT embeddings provide stronger patch-level morphology features, and the visual-prefix interface aligns slide-level context with biomedical language priors. This rationale is offered to explain the reported gains over RNN/LSTM and earlier transformer-based captioning baselines.

5. Datasets, implementation, and empirical results

The evaluation uses BreakHis for classification and PatchGastric for captioning. BreakHis contains 7,909 RGB microscopy histopathology images from 82 patients, with benign-versus-malignant labels provided at patient level only. It includes magnifications of 40×, 100×, 200×, and 400×, with image size $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$3 pixels and H&E staining. PatchGastric contains 262,777 patches of size $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$4 from 991 H&E slides at 20× magnification, with each slide unique to a patient. Its captions have a vocabulary of 344 words and maximum sentence length 47 words; patches are paired with captions and do not have absolute coordinates (Raju et al., 9 Jul 2025).

Training uses Adam with learning rate $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$5, weight decay $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$6, dropout 0.3, 100 epochs, and batch size 16 for both training and evaluation. The implementation uses PyTorch and Deep Graph Library (DGL), and the reported hardware is an NVIDIA RTX A6000 GPU with 48 GB memory. Data augmentation is not specified. The paper also includes an end-to-end pseudocode description consisting of patch encoding, DEC, attention-based representative selection, graph construction, GAT aggregation, classification, and caption generation (Raju et al., 9 Jul 2025).

For computational analysis, the paper reports per-slide ViT inference complexity $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$7, DEC complexity $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$8 per iteration for distance computation and $$f_k^{(s)} = E_v(p_k^{(s)}) \in \mathbb{R}^{1 \times d_v},$$9 overall, attention-based selection $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$0, graph construction $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$1 for cosine similarity and $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$2 for Gumbel-Softmax discretization, GAT cost $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$3, projection and classification $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$4, and LLM captioning $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$5 per generated sequence token length $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$6. The graph remains modest in size because DEC and attention reduce each slide to at most 50 representative nodes in PatchGastric (Raju et al., 9 Jul 2025).

For BreakHis classification, the paper compares against ABMIL, DSMIL, TransMIL, and DTFD-MIL.

Method	F1	AUC
ABMIL	0.900	0.871
DSMIL	0.856	0.869
TransMIL	0.886	0.862
DTFD-MIL	0.911	0.887
GNN-ViTCap (ResNet-34)	0.921	0.906
GNN-ViTCap (ViT-B/16)	0.934	0.963

The ViT-B/16 configuration also reports precision 0.926 and recall 0.942, while the ResNet-34 configuration reports precision 0.917 and recall 0.925. The paper interprets the gap between the two backbones as evidence that ViT-B/16 yields stronger visual embeddings than ResNet-34 for this task (Raju et al., 9 Jul 2025).

For PatchGastric captioning, the paper compares against PatchCap, PathM3, and SGMT, then reports several GNN-ViTCap variants using different LLMs.

Model	BLEU-4	METEOR
PathM3	0.520	0.394
SGMT	0.551	0.432
GNN-ViTCap + BioGPT	0.686	0.485
GNN-ViTCap + ClinicalT5-Base	0.753	0.526
GNN-ViTCap + LlamaV2-Chat	0.796	0.557
GNN-ViTCap + BiomedGPT	0.811	0.567

The paper further reports that PatchCap, depending on encoder choice, reaches best BLEU-4 $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$7, METEOR $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$8, ROUGE $$F_{(s)} = [f_1^{(s)}; \ldots; f_{N_p}^{(s)}] \in \mathbb{R}^{N_p \times d_v}.$$9, and CIDEr $224 \times 224$0. For the best GNN-ViTCap configuration, BiomedGPT yields BLEU-1 0.886, BLEU-2 0.851, BLEU-3 0.828, BLEU-4 0.811, METEOR 0.567, ROUGE 0.865, and CIDEr 7.42. The paper reports this as about $224 \times 224$1 BLEU-4 improvement over SGMT and PathM3, and about $224 \times 224$2 METEOR improvement over SGMT. Qualitative examples show generated captions closely matching ground-truth pathology descriptions, including “well-differentiated tubular adenocarcinoma,” with example BLEU-4 scores of 0.889 and 0.768 (Raju et al., 9 Jul 2025).

6. Limitations, interpretation, and place in the literature

The paper explicitly notes that DEC is sensitive to the number of clusters $224 \times 224$3. If $224 \times 224$4 is too small, information may be lost; if $224 \times 224$5 is too large, graph complexity increases. It also identifies end-to-end training with full LLM fine-tuning as computationally expensive and suggests parameter-efficient fine-tuning such as LoRA as a possible mitigation. A further limitation is positional uncertainty: because graph edges are inferred from embedding similarity rather than explicit coordinates, the method may benefit from positional encodings, coordinate estimation, or explicit spatial priors when such information becomes available (Raju et al., 9 Jul 2025).

Additional future directions listed in the paper include adaptive or nonparametric clustering, Graph Transformers or attention-based global aggregation, self-supervised pretraining on histopathology patches, uncertainty estimation for classification and captioning, multi-scale modeling across magnifications, stain normalization pipelines, and explicit incorporation of spatial cues in scanner WSIs. These proposals indicate that the framework is presented not as a closed solution but as a configurable architecture for microscopy WSIs with missing spatial metadata.

Within its reported comparisons, GNN-ViTCap occupies an intersection of MIL-based WSI classification and biomedical vision-language generation. Relative to ABMIL, DSMIL, TransMIL, and DTFD-MIL, it adds redundancy-aware representative selection and graph-based context modeling. Relative to PatchCap, PathM3, and SGMT, it introduces slide-level visual conditioning of biomedical LLMs through a learned projection. This suggests that the paper’s main contribution is not a single module in isolation but a coordinated treatment of instance pruning, relational aggregation, and domain-aligned caption generation for pathology workloads in which absolute coordinates are unavailable (Raju et al., 9 Jul 2025).