MedFMC: Benchmark for Medical Image Adaptation

• MedFMC is a real-world dataset and benchmark for foundation model adaptation in medical image classification, aggregating five diverse clinical tasks.
• It features heterogeneous label structures—binary, multi-label, and multi-class—to address varying clinical objectives in radiography, pathology, endoscopy, and more.
• Benchmark results show that few-shot adaptation methods, such as Visual Prompt Tuning and partial freezing, effectively mitigate overfitting in low-data regimes.

MedFMC is a real-world dataset and benchmark for foundation model adaptation in medical image classification. It was introduced to examine the overall performance of accommodating large-scale foundation models downstream on a set of diverse clinical tasks, and it aggregates five medical imaging subsets spanning radiography, pathology, endoscopy, digital photography, and retinal fundoscopy, for a total of 22,349 images (Wang et al., 2023).

1. Dataset scope and label structure

MedFMC collects five sets of medical imaging data from multiple institutes targeting a variety of clinical classification problems. The subsets are ChestDR, ColonPath, Endo, NeoJaundice, and Retino. ChestDR consists of 4,848 frontal chest X-rays, one per patient. ColonPath contains 10,009 H&E-stained pathology patches from 396 patients. Endo contains 3,865 white-light colonoscopy frames from 80 patients. NeoJaundice contains 2,235 digital camera skin photos, with three views per infant and 745 infants. Retino contains 1,392 non-mydriatic color fundus images, one per patient (Wang et al., 2023).

Subset	Images	Task type
ChestDR	4,848	Multi-label, 19 classes
ColonPath	10,009	Binary classification
Endo	3,865	Multi-label, 4 classes
NeoJaundice	2,235	Binary classification
Retino	1,392	Multi-class, 5 grades

The label structures are deliberately heterogeneous. ChestDR targets 19 thoracic disease labels in a single frontal chest X-ray; examples listed in the benchmark include cardiomegaly, pleural effusion, pneumonia, pneumothorax, and tuberculosis. ColonPath uses a binary label in which 0 denotes a normal patch and 1 denotes a patch containing malignant lesion tissue. Endo is multi-label with four lesion types: ulcer, erosion, polyp, and tumor, and an image may contain none, one, or multiple labels. NeoJaundice is binary, defined by total serum bilirubin at a threshold of $12.9$ mg/dL. Retino is a 5-grade diabetic retinopathy scale from 0, No DR, to 4, Proliferative DR (Wang et al., 2023).

This combination of binary, multi-label, and multi-class supervision is central to the benchmark’s design. A plausible implication is that MedFMC is intended not as a modality-specific corpus but as a controlled test bed for adaptation behavior across distinct prediction geometries and clinical label semantics.

2. Clinical tasks and evaluation regimes

The five MedFMC tasks are framed around concrete clinical objectives. ChestDR is defined as thoracic disease screening, with the objective of flagging any of 19 radiographic findings in a single frontal chest X-ray. ColonPath targets pathological lesion screening by determining whether a $1024\times1024$ patch sampled from a whole-slide image contains lesion(s). Endo targets lesion detection in real-world colonoscopy video frames. NeoJaundice is a noninvasive bilirubin-risk classification task based on newborn skin photos. Retino is a diabetic retinopathy grading task on fundus images (Wang et al., 2023).

The original benchmark partitions each subset by first choosing approximately $20\%$ of patients to form a few-shot pool, with at least 10 samples per class, and using the remaining images as the held-out test set. Few-shot experiments sample $k\in\{1,5,10\}$ patients per class from the few-shot pool to build a support set; evaluation is then carried out on the held-out test set. Transfer-learning experiments instead train on the entire few-shot pool and test on the held-out set. Each experiment is repeated 10 times with different random support sets, and mean $\pm$ standard deviation are reported (Wang et al., 2023).

The metric design depends on task type. For ColonPath, NeoJaundice, and Retino, the benchmark reports Accuracy and AUC, where macro-AUC is defined as area under the ROC curve per class, averaged. For ChestDR and Endo, it reports mean Average Precision and AUC, and for ChestDR it further distinguishes “head” classes with at least 100 samples from “tail” classes with fewer than 100 samples (Wang et al., 2023).

The MedFMC challenge setting described in the TransMed technical report narrows the scope to three public clinical tasks designed for few-shot adaptation of vision foundation models with no medical images in pre-training: ChestDR, ColonPath, and Endo. In that setting, the support/train, validation, and test splits are 2,140, 2,708, and 2,626 for ChestDR; 5,654, 4,355, and 10,009 for ColonPath; and 1,810, 2,055, and 2,199 for Endo. Few-shot support sets are drawn by patient at 1-shot, 5-shot, and 10-shot, and the metric is mean area under the ROC curve over classes (Zheng et al., 2023).

A recurring source of confusion is scope rather than definition: the original MedFMC benchmark contains five subsets, whereas the NeurIPS 2023 challenge report covers three public tasks. The two descriptions are compatible but not identical.

3. Baseline adaptation paradigms and benchmark findings

The benchmark evaluates both few-shot adapters and transfer-learning baselines. The listed backbones are DenseNet-121, pretrained on ImageNet in a supervised fashion, and Swin-Transformer-base, considered with both ImageNet-supervised pre-training and SimMIM self-supervised pre-training. The few-shot adapters are Meta-Baseline and Visual Prompt Tuning. Transfer-learning baselines include linear probing, in which the backbone is frozen and only the final fully connected layer is trained, and full fine-tuning of backbone plus head (Wang et al., 2023).

The 10-shot summary reported for few-shot adaptation shows a consistent advantage for VPT Swin-SL across all five subsets. The values are $0.190/0.667$ on ChestDR for mAP/AUC, $0.912/0.971$ on ColonPath for Acc/AUC, $0.667/0.727$ on NeoJaundice, $0.256/0.714$ on Endo, and $0.527/0.752$ on Retino. Under the same 10-shot protocol, Meta-Baseline Swin-SL reports $1024\times1024$0 on ChestDR, $1024\times1024$1 on ColonPath, $1024\times1024$2 on NeoJaundice, $1024\times1024$3 on Endo, and $1024\times1024$4 on Retino; Meta-Baseline Swin-SSL reports $1024\times1024$5, $1024\times1024$6, $1024\times1024$7, $1024\times1024$8, and $1024\times1024$9, respectively (Wang et al., 2023).

The benchmark also reports that, with 10-shot data, fine-tuning DenseNet, EfficientNet-B4, and Swin-base produces overall lower or comparable performance relative to few-shot adapters, and that full-network fine-tuning with only 1 to 5 samples per class leads to overfitting or underfitting. With the full few-shot pool, larger gains become possible; examples reported include Swin-base with frozen backbone plus fully connected layer on ChestDR head classes at mAP $20\%$0, AUC $20\%$1, and EfficientNet-B4 on ColonPath at Acc $20\%$2, AUC $20\%$3 (Wang et al., 2023).

Head-tail analysis on ChestDR further clarifies the long-tail regime. For 10-shot Meta-Baseline Swin-SL, the head classes obtain mAP $20\%$4 and AUC $20\%$5, whereas the tail classes obtain mAP $20\%$6 and AUC $20\%$7 (Wang et al., 2023).

These results establish the original empirical profile of MedFMC: prompt-based or few-shot-specific adaptation is favored in the extreme low-data regime, and class imbalance remains analytically important rather than incidental.

4. MedFMC challenge and the TransMed solution

TransMed is a MedFMC challenge solution for few-shot medical image classification that combines a Vision Transformer backbone with semantic supervision derived from LLMs. The backbone used in the main system is Swin-Transformer-Large pretrained on ImageNet-21K, with no architectural change except appending a final linear classification head. For comparison and ablation, ViT-Base with $20\%$8 patches is also studied (Zheng et al., 2023).

The adaptation strategy is “Partial Freezing.” The rationale stated in the report is that full fine-tuning exposes too many trainable parameters and therefore overfits in the $20\%$9-shot regime, whereas linear probing freezes too much and yields poor domain adaptation because of the natural-to-medical gap. The backbone is written as $k\in\{1,5,10\}$0, and for Swin-L the first two stages are frozen while $k\in\{1,5,10\}$1, $k\in\{1,5,10\}$2, and the final head $k\in\{1,5,10\}$3 are trained. This reduces the number of trainable parameters by approximately $k\in\{1,5,10\}$4, with the stated interpretation that shallow layers capture generic edges and textures while deeper layers adapt to medical features (Zheng et al., 2023).

TransMed’s second component is semantic supervision via LLM-contextualized labels. GPT-4 is used to generate a short diagnostic description for each class using the prompt form “How to recognize [CLASS] in X-ray? $k\in\{1,5,10\}$5 answer: key features/regions.” Each occurrence of $k\in\{1,5,10\}$6 is replaced with $k\in\{1,5,10\}$7, and the masked text is passed through PubMed BERT to extract mask-token embeddings. For class $k\in\{1,5,10\}$8, the description yields $k\in\{1,5,10\}$9 tokens at mask positions, $\pm$0. Visual-text alignment is then defined by

$\pm$1

where $\pm$2 is cosine similarity, $\pm$3, and $\pm$4 is sigmoid. The semantic loss is

$\pm$5

The report also gives the total objective over the support set $\pm$6 as

$\pm$7

and states that no additional regularization terms were needed beyond standard weight decay (Zheng et al., 2023).

The training protocol uses 1-shot, 5-shot, and 10-shot experiments, AdamW with learning rate $\pm$8, weight decay $\pm$9, batch size $0.190/0.667$0, and 20 epochs. Augmentation consists of center crop, resize to $0.190/0.667$1, random horizontal flip, and random crop. In the semantic pipeline, GPT-4 is used for generation and PubMed BERT for embeddings. During training, a random subset of mask tokens is sampled per epoch for bootstrapping; at test time, predictions are averaged over all mask tokens (Zheng et al., 2023).

The reported validation mAUC $0.190/0.667$2 standard deviation for TransMed with Swin-L is 62.95 $0.190/0.667$3 0.19, 69.14 $0.190/0.667$4 0.60, and 71.31 $0.190/0.667$5 0.37 on ChestDR at 1-shot, 5-shot, and 10-shot; 91.50 $0.190/0.667$6 2.47, 97.51 $0.190/0.667$7 1.16, and 97.85 $0.190/0.667$8 1.25 on ColonPath; and 62.51 $0.190/0.667$9 4.65, 70.13 $0.912/0.971$0 2.37, and 74.85 $0.912/0.971$1 0.61 on Endo. Under the same protocol, VPT reports 56.37 $0.912/0.971$2 1.15, 64.51 $0.912/0.971$3 1.34, and 64.29 $0.912/0.971$4 1.63 on ChestDR; 81.67 $0.912/0.971$5 3.80, 92.13 $0.912/0.971$6 3.55, and 95.42 $0.912/0.971$7 0.56 on ColonPath; and 60.13 $0.912/0.971$8 6.13, 67.26 $0.912/0.971$9 2.23, and 70.99 $0.667/0.727$0 1.31 on Endo. CITE reports 58.64 $0.667/0.727$1 1.35, 65.72 $0.667/0.727$2 0.93, and 67.35 $0.667/0.727$3 0.98 on ChestDR; 83.95 $0.667/0.727$4 5.58, 88.90 $0.667/0.727$5 1.51, and 91.29 $0.667/0.727$6 3.43 on ColonPath; and 59.54 $0.667/0.727$7 8.28, 64.55 $0.667/0.727$8 5.04, and 67.52 $0.667/0.727$9 1.88 on Endo. The reported 1-shot gains for TransMed are $0.256/0.714$0, $0.256/0.714$1, and $0.256/0.714$2 versus VPT on Chest, Colon, and Endo, and $0.256/0.714$3, $0.256/0.714$4, and $0.256/0.714$5 versus CITE (Zheng et al., 2023).

Ablation results on ChestDR 1-shot isolate the contributions of both components. For adaptation strategy with Swin-L, full-model fine-tuning yields 58.89 $0.256/0.714$6 0.74, linear probe 54.18 $0.256/0.714$7 1.19, LoRA adapter 53.43 $0.256/0.714$8 1.64, VPT 56.37 $0.256/0.714$9 1.15, and partial freezing 59.92 $0.527/0.752$0 1.29. For semantic supervision with Swin-L plus partial freezing, one-hot supervision yields 59.92 $0.527/0.752$1 1.29, class-name embedding 59.08 $0.527/0.752$2 1.22, template M-LM 57.32 $0.527/0.752$3 4.59, and LLM-contextualized supervision 62.95 $0.527/0.752$4 0.19. The report attributes the gains to training stabilization through reduced trainable parameters and to finer semantic discrimination among clinically similar diseases, and states that the solution secured first place in the MedFMC challenge (Zheng et al., 2023).

5. Evaluation of open-source vision-LLMs on MedFMC

MedFMC was later used as the evaluation dataset in a retrospective study of five open-source vision-LLMs: Qwen2.5, Phi-4, Gemma3, Llama3.2, and Mistral3.1. The study used 22,349 images from 7,461 unique patients across the same five tasks and compared three prompting configurations: visual input only, multimodal input, and chain-of-thought reasoning (Müller-Franzes et al., 1 Aug 2025).

In the visual-input-only setting, Qwen2.5 achieved the highest accuracy for ChestDR at 90.4\% $0.527/0.752$5 and Endo at 84.2\% $0.527/0.752$6, significantly outperforming the other models in both modalities. In ColonPath, Phi-4 at 69.6\% $0.527/0.752$7 and Qwen2.5 at 69.0\% $0.527/0.752$8 were statistically indistinguishable with $0.527/0.752$9, and both exceeded the other models. In NeoJaundice, Qwen2.5 at 58.3\% $1024\times1024$00 and Phi-4 at 58.1\% $1024\times1024$01 were likewise comparable with $1024\times1024$02. Retino remained difficult: Qwen2.5 and Gemma3 both achieved 18.6\%, with no difference at $1024\times1024$03, and overall performance remained below 20\% (Müller-Franzes et al., 1 Aug 2025).

The multimodal-input experiment was applied only to NeoJaundice by combining the skin photo with serum bilirubin level. All models except Phi-4 showed a statistically significant accuracy boost under multimodal versus baseline, with $1024\times1024$04. Phi-4 showed no improvement, changing from 58.1\% to 55.6\% with $1024\times1024$05. The study also reports that for Qwen2.5, Gemma3, and Llama3.2, clinical-only input outperformed multimodal input with $1024\times1024$06, suggesting that image data sometimes introduced noise when the bilirubin threshold alone sufficed (Müller-Franzes et al., 1 Aug 2025).

Chain-of-thought prompting produced model-dependent effects rather than a uniform gain. Qwen2.5 showed a significant accuracy drop in ColonPath, Endo, and Retino with $1024\times1024$07, and no effect in NeoJaundice. Gemma3 improved on Endo with $1024\times1024$08 but declined elsewhere. Llama3.2 improved on ColonPath with $1024\times1024$09 and was neutral or negative on other tasks. Phi-4 produced nearly zero valid JSON outputs and performed below baseline with $1024\times1024$10. Mistral3.1 improved consistently in ColonPath, Endo, and Retino, all with $1024\times1024$11 (Müller-Franzes et al., 1 Aug 2025).

Within the MedFMC context, these findings expand the benchmark’s role from parameter-efficient adaptation of image encoders to prompt-sensitive evaluation of open-source VLMs. They also underscore that multimodal fusion and explicit reasoning are not universally beneficial, even when the benchmark task is fixed.

6. Limitations, recommendations, and research role

The MedFMC benchmark is explicitly motivated by the scarcity of publicly accessible data and benchmarks for adapting foundation models to medical image analysis. Its design emphasizes both predictive performance and cost-effective evaluation. The benchmark records not only accuracy-type metrics but also data and computation cost, including few-shot sample count and backbone FLOPs. Fine-tuning baselines in the original study use a single NVIDIA A100 GPU for 20 epochs with batch size $1024\times1024$12, and the authors note that few-shot adaptation drastically reduces annotation cost and training time relative to full-data training (Wang et al., 2023).

The benchmark paper identifies several practical findings. Visual Prompt Tuning significantly outperforms classic meta-learning and transfer learning with extremely limited data of at most 10 patients per class. Few-shot methods show relative robustness on rare tail classes, which is presented as improving equity across disease prevalence. SimMIM self-supervised pre-training on Swin sometimes helps but does not universally beat supervised pre-training. The authors recommend evaluating new adaptation methods across diverse modalities and label types, including head-tail analysis and reporting both accuracy-type metrics and annotation or computation cost. They further recommend extending MedFMC to 3D, segmentation, and detection tasks, and state that code, data splits, and baseline scripts are to be made publicly available on GitHub and Figshare under CC0 (Wang et al., 2023).

The later VLM study adds a different set of limitations and recommendations. It is restricted to a single retrospective dataset, classification tasks only, and models of at most 30B parameters. Its recommendations include domain-specific fine-tuning on large medical imaging corpora, rigorous validation across multi-institutional datasets, exploration of adaptive prompting strategies and reasoning-enhanced training, and, before clinical deployment, integration with structured reporting systems, compliance with privacy regulations, and prospective trials (Müller-Franzes et al., 1 Aug 2025).

Taken together, these results position MedFMC as a benchmark for measuring how foundation models adapt under realistic medical heterogeneity and severe sample constraints. This suggests that its enduring value lies less in any single leaderboard and more in the fact that the same corpus supports analysis of few-shot adaptation, transfer-learning failure modes, long-tail behavior, and prompt-sensitive VLM evaluation within one consistent clinical test suite.