---
title: 'ANTONI-α: Whole-Slide VLM in Pathology'
url: https://www.emergentmind.com/topics/antoni
type: topic
---

# ANTONI-α: Whole-Slide VLM in Pathology

ANTONI-α is a whole-slide vision-language model (VLM) developed to facilitate visual-question answering (VQA) in pathology at the slide level. It natively processes gigapixel-scale hematoxylin and eosin (H&E) stained whole-slide images (WSIs), aligning detailed visual context with multimodal clinical task responses. Built from open components and trained entirely on publicly available synthetic instruction data, ANTONI-α establishes a reproducible benchmark for generalizable, clinically relevant VLMs in digital pathology, surpassing prior models such as MedGemma on key diagnostic tasks [2512.17326].

## 1. Model Architecture

ANTONI-α integrates four principal modules:

- **VIRCHOW (Tile Encoder):** A ResNet-style convolutional neural network pretrained on clinical histopathology providing 1280-dimensional embeddings per tile. Its parameters (~350M) remain frozen during instruction tuning.
- **PRISM/CoCa (Slide Aggregator):** Aggregates 1 global and 512 local tokens (all 1280-dimensional) using image–text contrastive and generative captioning objectives. Model size is ~400M parameters and weights are frozen for ANTONI-α fine-tuning.
- **Vision Projector:** A custom, single-layer cross-attention module with 256 learnable query tokens ($Q \in \mathbb{R}^{256 \times 3072}$) attending to 513 vision latents ($V \in \mathbb{R}^{513 \times 1280}$). This single layer maps visual information into the language model (LLM) input space. The projector is the principal trainable component in early-stage tuning (~2.5M parameters).
- **MedGemma-4B-IT (LLM Decoder):** A 4B parameter transformer, instruction-tuned for medical applications, with a hidden size of 3072 and 32 layers. The SigLIP input mechanism is ablated; instead, the decoder receives the 256 vision tokens from the projector.

Data flow: WSI → VIRCHOW (tile embeddings) → PRISM (aggregation: 1 global + 512 locals) → Vision projector (cross-attention, 256 tokens) → MedGemma-4B-IT → text output.

**Key Design Choices:**
- High-resolution tile embeddings without downsampling artifacts.
- Single-layer cross-modal alignment for efficient resource use.
- Two-stage finetuning: (1) projector warmup, (2) projector+LLM joint QLoRA adaptation.

## 2. Training Objectives and Loss Functions

ANTONI-α employs a standard next-token cross-entropy loss for instruction tuning:

$$
\mathcal{L}_{\mathrm{CE}} = -\frac{1}{T} \sum_{t=1}^T \log P(y_t | x_{1:t-1})
$$

where $x_{1:t-1}$ are the prompt, user, and image tokens, and $y_t$ is the ground-truth assistant token at step $t$.

**PRISM Pretraining Objectives (Contextual, Frozen in ANTONI-α):**
- **Image–text contrastive loss (CLIP-style):**
$$
\mathcal{L}_{\mathrm{contrastive}} = -\frac{1}{B} \sum_{i=1}^B \log \frac{\exp(\mathrm{sim}(v_i,t_i)/\tau)}{\sum_{j=1}^B \exp(\mathrm{sim}(v_i, t_j)/\tau)}
$$
- **Captioning loss:** As above, cross-entropy over text targets.

These objectives ensure robust multimodal alignments during pretraining of aggregator components.

## 3. Training Data and Synthetic Instruction Generation

The primary training resource is HISTAI-Instruct, derived via a synthetic pipeline over the public HISTAI archive:

- **WSIs Post-Filtering:** 24,259 total, with 22,530 curated for H&E tuning.
- **Instruction–Response Pairs:** Initially 1,118,691, filtered via LLM-based judge to 1,175,524 high-quality instances.
- **Conversational Categories:**
  - Advanced reasoning
  - Clean report
  - Detailed description
  - Differential diagnosis
  - Multi-turn conversation
  - Negative reasoning
  - Short VQA

**Polysome Pipeline:**
1. Metadata (descriptions, ICD-10, conclusions) is injected into prompt templates.
2. English instructions and corresponding responses are generated in all 7 categories.
3. Instances are machine-translated into Dutch, French, German, Italian, Polish, and Spanish.
4. “LLM-as-Judge” filtering is applied for adherence, factuality, and reasoning quality.
5. Question variation increases factual coverage and robustness.

## 4. Training Regimen and Hyperparameters

Hardware configuration utilizes 8 NVIDIA H200 GPUs, FSDP, and mixed bfloat16 precision.

- **Optimizer:** AdamW, weight decay 0.01.
- **Learning Rate:** Cosine schedule with 10% linear warmup.
- **Batch Size:** 16 sequences/GPU × 8 GPUs × 4 gradient accumulation steps = 512 effective batch.
- **Two-Stage Protocol:**
  - Stage 1 (Vision projector warmup): Only the projector is trained; MedGemma is frozen. Data: multilingual "clean report" samples; 35 epochs; learning rate $3\times10^{-4}$.
  - Stage 2 (Projector + LLM adapters): QLoRA (rank-16) is applied to MedGemma and the projector. Data: English instructions across all categories; 21 epochs; learning rate $3\times10^{-5}$.

**Data-Scaling Variants:** ANTONI-α models were additionally trained on WSI subsets (2,000, 9,000, 18,000) to interrogate scaling behavior.

## 5. Evaluation and Performance Benchmarks

Evaluation benchmarks WSI-level VQA on 317 held-out samples:

| Model                 | Organ Score   | Prec (%)         | Rec (%)        | F1 (%)         | Diff. Diag Acc (%)  |
|-----------------------|--------------|------------------|----------------|----------------|---------------------|
| MedGemma-4B           | 0.48 [0.43–0.53] | 71.4 [65.3–77.4] | 68.8 [62.7–74.9] | 70.1 [64.9–74.9] | 40.1 [34.7–45.4]    |
| MedGemma-27B          | 0.37 [0.32–0.42] | 85.5 [76.0–93.8] | 24.3 [18.7–30.1] | 37.9 [30.3–45.0] | 44.8 [39.1–50.2]    |
| ANTONI-α (Base, no ft)| 0.52 [0.47–0.58] | 60.3 [53.1–67.4] | 50.9 [44.3–57.4] | 55.2 [49.2–60.7] | 48.3 [42.9–53.6]    |
| ANTONI-α (2k WSIs)    | 0.66 [0.60–0.71] | 68.7 [63.6–73.7] | 99.5 [98.6–100] | 81.3 [77.6–84.7] | 52.7 [47.0–58.4]    |
| ANTONI-α (9k WSIs)    | 0.91 [0.88–0.94] | 70.9 [65.6–76.0] | 95.0 [91.8–97.7]| 81.2 [77.3–84.7] | 66.3 [60.9–71.3]    |
| ANTONI-α (18k WSIs)   | 0.91 [0.88–0.94] | 72.9 [67.5–78.2] | 91.3 [87.4–94.8]| 81.1 [77.1–84.7] | 68.5 [63.1–73.5]    |

**Key Empirical Findings:**
- Native WSI feature encoding improves organ identification (from 0.48 to 0.91) and differential diagnosis accuracy (from 44.8% to 68.5%) over prior models.
- Organ score scales steeply with data: from 0.66 to 0.91 when increasing from 2,000 to 9,000 slides.
- Fine-tuned ANTONI-α surpasses both MedGemma-4B and MedGemma-27B on most metrics.

## 6. Representative Model Output

A sample dialogue (paraphrased for brevity) on a dermatofibroma case demonstrates the model's clinical reasoning:

- **User:** “Which organ/tissue is shown on this slide?”
  - **ANTONI-α:** “This is a section of skin, specifically dermis with a well-circumscribed fibrohistiocytic lesion (dermatofibroma).”
- **User:** “Is there a neoplasm present?”
  - **ANTONI-α:** “Yes. The lesion is benign but neoplastic, composed of spindle-shaped fibroblasts and histiocytes without significant atypia or mitotic activity.”
- **User:** “Select most likely diagnosis: 1) Dermatofibroma, 2) Dermatofibrosarcoma protuberans, 3) Hypertrophic scar.”
  - **ANTONI-α:** “1) Dermatofibroma. The lesion’s well-circumscription, absence of storiform pattern, and lack of deep infiltration support a benign fibrohistiocytic neoplasm.”

MedGemma, when presented with a downsampled thumbnail, confuses boundaries and cellular features, yielding incorrect diagnostic output.

## 7. Codebase and Resources

All core resources are fully open-source:

- **Model code and weights:** https://github.com/computationalpathologygroup/ANTONI-Alpha
- **Polysome instruction-generation toolkit:** https://github.com/computationalpathologygroup/Polysome
- **HISTAI-Instruct dataset:** https://huggingface.co/datasets/SaltySander/HISTAI-Instruct

These resources facilitate fully reproducible experimentation, from WSI preprocessing through instruction synthesis to model training and evaluation [2512.17326].

Source: https://www.emergentmind.com/topics/antoni