UltraBench: Ultrasound Representation Benchmark

• UltraBench is an open benchmark designed to evaluate learned ultrasound representations across eight clinical tasks and various organs using static frames.
• The design employs a standardized frozen-backbone linear probing protocol to fairly compare state-of-the-art ultrasound models.
• It incorporates rigorous preprocessing, few-shot analysis, and corruption robustness tests to ensure stable and transferable performance.

Searching arXiv for the specified paper to ground the article in the cited source. UltraBench is an open, frame-level classification benchmark spanning eight clinical tasks, eight organs, and a variety of pathological conditions. It was reported in the context of "US-JEPA: A Joint Embedding Predictive Architecture for Medical Ultrasound" as a public dataset benchmark for rigorous comparison of publicly available state-of-the-art ultrasound foundation models under a standardized frozen-backbone linear probing protocol (Radhachandran et al., 22 Feb 2026). In that formulation, UltraBench is designed to evaluate learned ultrasound representations rather than end-to-end task-specific systems, with all models assessed on the same downstream classification setup.

1. Benchmark scope and task coverage

UltraBench aggregates eight downstream ultrasound classification tasks. Two new public datasets, BUSBRA for breast and TN5000 for thyroid, were added, bringing the benchmark to a total of 53,213 static frames extracted from both still images and individual video frames (Radhachandran et al., 22 Feb 2026).

Organ	Dataset	Task
Liver	AUL	mass malignancy classification (3 classes – normal, benign, malignant)
Liver	Fatty Liver	steatosis detection (2 classes – normal, fatty)
Gallbladder	GBCU	lesion malignancy (3 classes)
Breast	BUSBRA	lesion malignancy (2 classes)
Ovary	MMOTU	ovarian tumor classification (8 classes)
Lung	POCUS	healthy vs. pneumonia vs. COVID–19 (3 classes)
Thyroid	TN5000	nodule malignancy (2 classes)
Multi-organ	Butterfly	anatomical region (9 classes)

The task set mixes malignancy classification, pathology detection, and anatomical region recognition. This combination is significant because it spans both organ-specific pathology-centric tasks and a multi-organ anatomical task. A plausible implication is that UltraBench was structured to probe transfer across heterogeneous semantic targets rather than performance on a single disease domain.

2. Data composition, modalities, and preprocessing

UltraBench uses B-mode 2D static images, individual frames sampled from 2D videos, and individual frames sampled from 3D volumes. The benchmark therefore treats ultrasound primarily at the frame level rather than as a sequence-modeling or volumetric-reconstruction problem (Radhachandran et al., 22 Feb 2026).

The reported preprocessing pipeline consists of grayscale conversion; inpainting of small non-anatomical overlays occupying less than 5% area; percentile intensity rescaling from the 2nd–98th percentile to the full dynamic range; and frame extraction from videos with video-level grouping to avoid split leakage. The emphasis on avoiding split leakage is methodologically important because the benchmark includes video-derived frames; without grouping at the video level, train-test contamination would be a substantial risk.

All splits are patient-independent. The reported train, validation, and test partitions are as follows.

Dataset	Classes	Train / Val / Test
AUL	3	529 / 59 / 147
Fatty Liver	2	390 / 50 / 110
GBCU	3	1,019 / 114 / 122
BUSBRA	2	765 / 86 / 213
MMOTU	8	800 / 200 / 469
POCUS	3	1,444 / 177 / 443
TN5000	2	3,500 / 500 / 1,000
Butterfly	9	28,053 / 6,272 / 6,751

The class structure is notably heterogeneous, ranging from binary tasks such as Fatty Liver and TN5000 to 9-way anatomical recognition in Butterfly. This heterogeneity matters for interpretation of cross-task averages and for understanding why macro-averaged F1, rather than plain accuracy alone, is the principal metric.

3. Evaluation protocol and benchmarking philosophy

All models in UltraBench are evaluated under a frozen-backbone linear probing protocol. Backbone encoders, including US-JEPA, URFM, USFM, and universal self-supervised learning models, are kept frozen, and a single linear classification head is trained on each downstream task (Radhachandran et al., 22 Feb 2026).

The optimization setup is fully standardized: cross-entropy loss, AdamW with learning rate $1\times10^{-3}$ and weight decay $1\times10^{-4}$, batch size 64, input size $224\times224$, cosine-annealing learning rate, and early stopping with patience of 15 epochs. Results are reported as mean $\pm$ standard deviation over five independent runs with different random seeds.

This protocol is central to the meaning of UltraBench scores. By standardizing on linear probing, UltraBench isolates the quality of the learned representation from head capacity or full fine-tuning complexities. A common misreading is to treat the benchmark as a direct ranking of fully adapted clinical models; the reported design instead evaluates representational quality under a deliberately capacity-limited readout.

4. Metrics and reported definitions

The primary metric in UltraBench is the macro-averaged F1 score across classes, with all F1 scores multiplied by 100 to report percentages (Radhachandran et al., 22 Feb 2026). Secondary metrics are described as commonly used but not explicitly reported: Accuracy, AUC, Precision, and Recall.

The reported definitions are:

$\mathrm{ACC} = \frac{TP + TN}{TP + TN + FP + FN}$

For class $c$,

$P_c = \frac{TP_c}{TP_c + FP_c}$

$R_c = \frac{TP_c}{TP_c + FN_c}$

$F1_c = 2 \cdot \frac{P_c \cdot R_c}{P_c + R_c}$

and macro-F1 is

$\text{Macro-F1} = \frac{1}{C}\sum_{c=1}^{C} F1_c$

The choice of macro-F1 is consequential in a benchmark that mixes binary, three-class, eight-class, and nine-class tasks. Macro-averaging gives equal weight to each class, which reduces the extent to which performance can be dominated by majority classes. This suggests that UltraBench was intended to emphasize balanced representation quality across disease states and anatomical categories rather than aggregate frequency-weighted correctness.

5. Comparative linear-probe results

The published comparison reports linear-probe macro-F1 for US-JEPA and USrc-JEPA against eight baselines: DINOv3, I-JEPA, UltraSAM, SAMUS, EchoCare, USF-MAE, USFM, and URFM (Radhachandran et al., 22 Feb 2026). Within that table, USrc-JEPA achieves the best macro-F1 on five of eight tasks—BUSBRA, Fatty Liver, GBCU, MMOTU, and POCUS—and second best on the remaining three.

Task	Best model	Second best model
AUL	URFM, $1\times10^{-4}$0	US-JEPA, $1\times10^{-4}$1
BUSBRA	USrc-JEPA, $1\times10^{-4}$2	USFM, $1\times10^{-4}$3
Butterfly	USFM, $1\times10^{-4}$4	URFM, $1\times10^{-4}$5
Fatty Liver	USrc-JEPA, $1\times10^{-4}$6	URFM, $1\times10^{-4}$7
GBCU	USrc-JEPA, $1\times10^{-4}$8	USFM, $1\times10^{-4}$9
MMOTU	USrc-JEPA, $224\times224$0	US-JEPA, $224\times224$1
POCUS	US-JEPA, $224\times224$2	USrc-JEPA, $224\times224$3
TN5000	URFM, $224\times224$4	USrc-JEPA, $224\times224$5

Several comparative patterns are explicitly highlighted. US-JEPA matches or improves on URFM in six tasks, despite a static (frozen) teacher and no generative pixel objective. The reported table also shows that universal vision foundation model baselines and ultrasound-specific baselines can both be competitive on some tasks, but no single baseline dominates the benchmark uniformly. This makes UltraBench useful not only for reporting an overall winner but also for exposing task-dependent inductive biases.

6. Few-shot behavior, corruption robustness, and interpretive boundaries

The UltraBench analysis extends beyond full-label linear probing. At 1%–10% label budgets, US-JEPA and USrc-JEPA degrade far more gracefully than URFM and USFM; on Fatty Liver with less than 10% labels, US-JEPA is reported to be approximately 18 percentage points higher in macro-F1 (Radhachandran et al., 22 Feb 2026). The paper interprets this as demonstrating stronger label efficiency and transferability of JEPA-learned features.

The benchmark also includes robustness analyses under domain-specific corruptions. Under increasing Gaussian blur, US-JEPA retains more than 75% macro-F1 on POCUS and more than 70% on Butterfly at the highest blur severity, whereas URFM drops to less than 50% and less than 25%, respectively. Against synthetic speckle noise, USrc-JEPA loses approximately 10 percentage points macro-F1 at peak severity on Butterfly, while URFM and USFM lose approximately 45 and 25 percentage points. Contrast reduction hurts all models, but US-JEPA and USrc-JEPA maintain parity or exceed baselines on most tasks.

Statistical stability is another reported property of the benchmark results. Across five runs, standard deviations for US-JEPA and USrc-JEPA are consistently less than or equal to 1.5 percentage points, which is presented as evidence of stable optimization. No explicit hypothesis tests, such as paired $224\times224$6-tests, were reported, although non-overlapping $224\times224$7 standard deviation intervals with URFM on key tasks including MMOTU and POCUS are noted as suggestive of significant improvements.

The qualitative analysis attributes modest gains from USrc, described as region-conditioning, to focusing the model on anatomical content and ignoring probe metadata or borders. It also states that JEPA’s latent reconstruction objective produces a representation more invariant to pixel-level noise and anatomical viewpoint changes than purely generative or contrastive baselines. Because these observations are tied to the reported evaluation design, they should be interpreted within the scope of frozen-backbone probing and corruption tests rather than as exhaustive evidence of downstream clinical deployment behavior.

Taken together, UltraBench functions as a public, standardized framework for evaluating ultrasound foundation models across heterogeneous organs, label spaces, and corruption regimes. Its central methodological contribution is not merely the aggregation of datasets, but the combination of patient-independent splits, controlled preprocessing, unified linear-probe training, macro-F1-based reporting, few-shot evaluation, and corruption stress testing in a single benchmark definition (Radhachandran et al., 22 Feb 2026).