CONCH: Histopathology Vision–Language Model

• CONCH is a vision–language foundation model for histopathology that leverages joint image–text representation learning from over 1.17 million image–text pairs.
• It utilizes a CoCa-style framework with ViT-B/16 and GPT-style encoders, pretraining on extensive histology images and reports for robust feature extraction.
• Empirical evaluations demonstrate strong transfer performance in 3D retrieval, weak supervision, and diagnostic tasks, while highlighting sensitivity to distribution shifts.

Searching arXiv for the original CONCH pathology model and related papers. CONCH, short for Contrastive learning from captions for histopathology, is a pathology-specific vision–language foundation model centered on joint image–text representation learning. In recent computational pathology literature, it is used in two principal ways: as a multimodal encoder for zero-shot reasoning with pathology prompts, and as a frozen visual backbone that supplies patch embeddings to downstream pipelines for retrieval, weakly supervised whole-slide analysis, grading, molecular prediction, and interpretability studies. Across these uses, CONCH is repeatedly evaluated against both classic CNNs and newer pathology foundation models, with results that show strong transfer performance, especially in histology-aligned settings, but also clear sensitivity to certain distribution shifts and task formulations (Mahbod et al., 2024, Neidlinger et al., 2024).

1. Definition, architecture, and pretraining

CONCH is a vision–language foundation model built specifically for histopathology. In the medical-image retrieval study, it is described as using the CoCa framework with a ViT-B/16 image encoder, a GPT-style text encoder, and a GPT-style text decoder for captioning. The same study states that CONCH was trained on over 1.17 million histology image–text pairs, that its image encoder was pre-trained with iBOT on 16 million in-house histology images, and that its LLM was pre-trained on more than 550k surgical histopathology reports and more than 400k PubMed abstracts about histopathology (Mahbod et al., 2024).

The training objective is multimodal. The reported description ties CONCH to a CLIP-like contrastive alignment between image and text embeddings together with a captioning objective, which is the hallmark of CoCa-style training. In the retrieval paper, the standard image–text similarity is given as cosine similarity,

$$s(z_i^I, z_j^T)=\frac{(z_i^I)^\top z_j^T}{\|z_i^I\|\,\|z_j^T\|},$$

with a CLIP-style InfoNCE loss used conceptually to align paired samples (Mahbod et al., 2024).

Subsequent papers characterize CONCH somewhat differently depending on version and deployment. The pediatric brain-tumor study and the retrieval benchmark use 512-dimensional CONCH image features, whereas the glioblastoma spatial-transcriptomics study reports CONCH v1.5 as a frozen encoder producing

$\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$

for each $224\times224$ H&E patch at $20\times$ and $0.5\,\mu\text{m/pixel}$ (Tampu et al., 2024, Srikanthan et al., 3 Jun 2026). This suggests that the encoder family is stable in concept but not necessarily identical in output dimensionality across releases and evaluation stacks.

2. Operational modes and inference patterns

A recurrent feature of the literature is that CONCH is often used without any fine-tuning of its backbone. In content-based medical image retrieval, the text encoder and decoder are discarded and only the vision encoder is used as a zero-shot feature extractor. For 2D images, inputs are resized to $224\times224$; for 3D datasets, each slice is resized to $224\times224$, encoded into a 512-D vector, and the volume feature is formed by concatenation, so a volume with $S$ slices yields a $512S$-dimensional descriptor. Retrieval is then performed with cosine similarity,

$$\operatorname{sim}(\mathbf f_q,\mathbf f_i)=\frac{\mathbf f_q^\top \mathbf f_i}{\|\mathbf f_q\|\,\|\mathbf f_i\|},$$

with the training split used as the database and the test split used as queries (Mahbod et al., 2024).

In zero-shot histopathology classification, CONCH is used as a true multimodal model. MR-PHE keeps both the image encoder $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$0 and text encoder $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$1 frozen, computes normalized patch and global embeddings, and classifies by similarity in the shared embedding space. Its final prediction score is

$\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$2

where $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$3 is a hybrid image embedding and $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$4 is a weighted class prompt embedding. The method builds $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$5 from a global CONCH image embedding and a weighted aggregation of multi-resolution patch embeddings, with patch weights derived from text–image similarities inside CONCH’s joint space (Rahaman et al., 13 Mar 2025).

For WSI-scale weak supervision, CONCH is typically reduced again to a patch encoder. The pediatric brain-tumor study extracts non-overlapping $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$6 patches at $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$7, encodes each patch into $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$8, and aggregates all case-level patch features with ABMIL or CLAM. MambaMIL+ follows the same two-stage philosophy: patches are first embedded offline with CONCH, then long-range slide modeling is performed by a Mamba-based MIL architecture rather than by the foundation model itself (Tampu et al., 2024, Zeng et al., 19 Dec 2025).

3. Empirical performance across tasks

Across published benchmarks, CONCH performs strongly but not uniformly. In content-based medical image retrieval on eight MedMNIST v2 datasets, it is reported to be the best overall model on the 3D datasets, with averaged mAP@5 = 78.86% and ACC@1 = 72.68%, while on 2D datasets it is highly competitive but slightly behind UNI overall; its averaged ACC@1 on 2D is 78.46%, compared with 79.45% for UNI and 74.07% for the best CNN baseline, DenseNet121 (Mahbod et al., 2024).

In a large external computational pathology benchmark spanning 31 weakly supervised tasks and 13 cohorts, CONCH is reported as the best or tied-best overall model, with mean AUROC = 0.71 across tasks, mean AUROC = 0.77 on morphology tasks, and mean AUROC = 0.63 on prognostic tasks. The same benchmark also reports that an ensemble combining CONCH and Virchow2 reaches 0.719 mean AUROC and is significantly better than CONCH alone in 9 of 29 binary tasks (Neidlinger et al., 2024).

In prompt-based zero-shot digestive pathology, CONCH is the strongest of the three evaluated histopathology VLMs, achieving average AUC = 0.876 across prompts and tissue types for invasiveness classification. The best invasiveness prompt for CONCH is the anatomically specific, task-focused $\mathbf h_i=f_\theta(x_i)\in\mathbb R^{768}$9 prompt with AUC = 0.935, whereas a more elaborate expert-framed prompt $224\times224$0 drops to AUC = 0.736. For dysplasia, the best-performing lexical variant uses “Precancerous”, reaching AUC = 0.904 (Sharma et al., 30 Apr 2025).

The downstream WSI literature shows a broader spread of outcomes. In pediatric brain-tumor classification, CONCH features substantially outperform ResNet50 and are close to UNI: for tumor category classification with ABMIL, CONCH reaches MCC = $224\times224$1, versus $224\times224$2 for UNI and $224\times224$3 for the best ResNet50 configuration (Tampu et al., 2024). In MAKO, a breast-cancer recurrence-risk benchmark using ROR-P as target, CONCH yields the highest ROC AUC for binary high-vs-low/medium classification in both internal and external evaluation: 0.809 on CBCS and 0.852 on TCGA BRCA (Kaczmarzyk et al., 16 Aug 2025). MambaMIL+ further reports that under CONCH features, the model attains strong average performance across 20 benchmarks, including AUC 95.3, ACC 82.2, F1 81.5 on the five diagnostic datasets and average C-index 0.720 on six survival cohorts (Zeng et al., 19 Dec 2025).

By contrast, on IHC staining assessment, a specialized architecture outperforms fine-tuned CONCH. In the HPA10M-based iSight study, fully fine-tuned CONCH reaches 70.0% accuracy for staining intensity, 75.2% for location, 70.0% for quantity, and 59.0% for tissue type, all below the corresponding iSight results (Leiby et al., 3 Feb 2026).

Setting	Use of CONCH	Representative outcome
MedMNIST CBMIR	Frozen visual feature extractor	Best overall on 3D: mAP@5 78.86%, ACC@1 72.68%
External CPath benchmark	Frozen tile encoder + transformer MIL	Mean AUROC 0.71 across 31 tasks
Digestive pathology zero-shot	Prompted VLM	Average AUC 0.876; best prompt AUC 0.935
Breast recurrence risk	Frozen patch encoder + ABMIL	Highest ROC AUC: 0.809 internal, 0.852 external

4. Representation, interpretability, and biological coherence

Several recent studies examine not only what CONCH predicts, but what its representations encode. In representational similarity analysis across six computational pathology foundation models, CONCH has mean representational similarity 0.463 to the other CPath models, the highest among the three vision–LLMs in that study. Its slide-specificity measured by Cliff’s Delta is 0.596, reported as the lowest slide-dependence among the six CPath models, while its disease-specificity is 0.135. Under Macenko stain normalization, CONCH’s slide-specificity decreases to 0.563, a change of −0.033, which is the smallest decrease among the evaluated pathology models. The same study also reports that CONCH has the steepest singular-value accumulation curve, interpreted there as the most compact representation among the models analyzed (Mishra et al., 18 Sep 2025).

A separate glioblastoma study evaluates whether attention maps derived from pathology foundation models correspond to biological structure measured by spatial transcriptomics. There, CONCH v1.5 is used as a frozen patch encoder inside a shared attention-based MIL head. On internal CPTAC validation, it achieves mean multi-task AUROC = 0.692, the best internal average across encoders; on external TCGA validation, its mean multi-task AUROC is 0.703, competitive but not top-ranked. More importantly for interpretability, CONCH’s attention maps have pathway-level mean Cohen’s $224\times224$4, among the highest in the study, while its spatial autocorrelation is the lowest among models with Moran’s $224\times224$5. The same paper highlights a distinctive encoder-specific association for CONCH with the immune-related gene HLA-B, with $224\times224$6, and argues that CONCH attends to multi-gene programs and cell states rather than isolated molecular events (Srikanthan et al., 3 Jun 2026).

Qualitative attention analyses in zero-shot digestive pathology are consistent with this emphasis on biologically structured regions. A board-certified pathologist judged CONCH’s invasiveness heatmaps to be the most accurate among the compared VLMs, with highest attention concentrated on invasive cancer, intermediate attention over precursor lesions and peritumoral tissue, and lower attention in remote non-neoplastic regions (Sharma et al., 30 Apr 2025). This suggests that, when prompts are well aligned, CONCH’s multimodal features support not only classification but spatially plausible diagnostic saliency.

5. Robustness, limitations, and adaptation

The strongest cautionary result in the surveyed literature concerns distribution shift. In prostate cancer grading on PANDA, CONCH performs well in-distribution as a frozen encoder: with ABMIL it reaches quadratically weighted kappa $224\times224$7 on mixed-site PANDA, $224\times224$8 on Karolinska-only, and $224\times224$9 on Radboud-only. Under cross-site shift, however, the degradation is severe. Training on Radboud and testing on Karolinska yields $20\times$0 kappa for CONCH–ABMIL, compared with $20\times$1 for UNI and $20\times$2 for ResNet-IN; even after equalizing label distributions, Radboud-U $20\times$3 Karolinska-U remains poor for CONCH at $20\times$4 with ABMIL. By contrast, pure label-distribution shift within Radboud is much milder, with $20\times$5 in-distribution dropping to $20\times$6 for Radboud-L $20\times$7 Radboud-R (Gustafsson et al., 2024).

This pattern indicates that CONCH’s pathology-specific pretraining does not, by itself, guarantee robustness to scanner, staining, or institutional covariate shifts. The same study reports that adding even modest fractions of target-site data to downstream training substantially improves target-site performance, which points toward multi-site supervision rather than foundation-model scale alone as the immediate lever for deployment robustness (Gustafsson et al., 2024).

At the method-development level, multiple papers respond to CONCH’s strengths and limitations by building wrappers around its frozen encoders. MR-PHE improves zero-shot histopathology classification by combining multi-resolution patches, patch weighting, and prompt selection without changing CONCH’s parameters; for example, it raises CRC100K performance from 59.0% F1 / 59.8% Acc for the original CONCH zero-shot classifier to 76.95% F1 / 80.22% Acc, and improves WSSS4LUAD from 86.5% F1 / 86.7% Acc to 89.6% F1 / 88.9% Acc (Rahaman et al., 13 Mar 2025). Annotation-free specialization through continued pretraining on retrieved image–caption pairs likewise reports that domain-adaptive pretraining is particularly effective for CONCH on BACH, MHIST, and SICAP, with the paper emphasizing qualitative gains in both zero-shot and few-shot regimes without manual annotation (Qiu et al., 11 Aug 2025).

Other limitations are task specific. In IHC, CONCH is outperformed by a purpose-built multi-task model that integrates metadata and token-level attention (Leiby et al., 3 Feb 2026). In WSI segmentation, ConStruct treats CONCH as the semantic backbone but requires structural distillation from SegFormer to improve spatial completeness and boundary coherence (Le et al., 11 Dec 2025). These results imply that CONCH’s native strengths lie in semantic and morphology-aware encoding rather than in being a universally sufficient end-to-end architecture.

6. Broader scope of the acronym

Although CONCH in current arXiv biomedical literature most often denotes the histopathology foundation model, the acronym is not unique. In astronomy, CONCH-SHELL names the “Catalog Of Nearby Cool Host-Stars for Habitable ExopLanets and Life,” an all-sky catalog of 2970 nearby bright M- or late K-type dwarfs intended for exoplanet surveys (Gaidos et al., 2014). In statistics, CONCH also abbreviates CONformal CHangepoint localization, a distribution-free method for offline changepoint localization under exchangeability assumptions (Hore et al., 5 Feb 2026, Dandapanthula et al., 1 May 2025). Other unrelated uses include “Conch” as an interactive visualization system for competitive-debate analysis (Chen et al., 19 Jul 2025) and conch maximal subrings in commutative algebra (Azarang, 2020).

Within computational pathology, however, the term has acquired a more specific technical meaning: a CoCa-style pathology vision–language foundation model whose image encoder and joint embedding space have become recurring infrastructure for zero-shot retrieval, prompt-based diagnosis, WSI-level MIL, representation analysis, and biologically grounded interpretability. The published record portrays it neither as a universally dominant model nor as a merely niche histology encoder. Rather, it occupies a distinctive position: semantically rich, often top-tier in histology-aligned tasks, unusually informative for image–text reasoning, yet still dependent on prompt design, downstream architecture, and training-domain match for reliable deployment (Neidlinger et al., 2024, Sharma et al., 30 Apr 2025, Gustafsson et al., 2024).