U2-BENCH: Ultrasound LVLM Benchmark

• U2-BENCH is a public benchmark for assessing large vision-language models on ultrasound imaging, encompassing 7,241 studies across 15 anatomical regions.
• The benchmark defines 8 clinically inspired tasks, ranging from classification to text generation, to mirror real-world ultrasound workflows.
• Evaluation with metrics such as the U2-Score highlights strengths in image-level classification while exposing challenges in spatial reasoning and clinical text generation.

Searching arXiv for U2-BENCH and closely related papers to ground the article. U2-BENCH is a public benchmark for large vision-LLMs on ultrasound understanding. Introduced in 2025, it aggregates 7,241 ultrasound studies from 40 licensed public datasets, spans 15 anatomical regions, and defines 8 clinically inspired tasks across classification, detection, regression, and text generation, organized into 50 ultrasound application scenarios. Its purpose is to provide a unified, rigorous testbed for zero-shot prompted evaluation of LVLMs on ultrasound rather than a training corpus, with task formulations intended to mirror clinical workflows from view recognition and organ identification to lesion localization, clinical value estimation, and report generation (Le et al., 23 May 2025).

1. Clinical motivation and benchmark scope

U2-BENCH is motivated by the fact that ultrasound is difficult for AI in ways that differ materially from CT, MRI, and radiography. The benchmark paper emphasizes that ultrasound is noisy and artifact-prone, with speckle noise, acoustic shadowing, variable gain, and machine presets; operator-dependent and dynamic, because image acquisition depends on probe manipulation over 3D anatomy and often yields video rather than a single static frame; and anatomically and contextually complex, because correct interpretation can require standard-plane recognition, temporal reasoning, and clinically conditioned measurement or diagnosis (Le et al., 23 May 2025).

This framing is important because earlier ultrasound AI literature largely focused on narrow tasks, such as fetal plane classification or segmentation, usually with task-specific models and comparatively small datasets. By contrast, LVLMs promise a single multimodal interface for image interpretation, quantitative estimation, and report drafting. U2-BENCH was introduced to determine whether such models actually transfer to ultrasound, a modality that had been under-represented in both general LVLM benchmarks and medical LVLM benchmarks. The benchmark therefore targets not only coarse image-level recognition but also the more clinically difficult regimes of spatial reasoning, measurement, and clinical language generation (Le et al., 23 May 2025).

The benchmark is explicitly positioned as a clinically structured evaluation suite. It covers the pipeline from identifying anatomy, views, and organs through lesion localization and keypoint detection to diagnosis, numerical estimation, and report or caption generation. This design makes it closer to clinical ultrasound workflows than generic visual question answering benchmarks.

2. Benchmark composition and curation pipeline

U2-BENCH contains 7,241 ultrasound studies, covers 15 anatomical regions, and is built from 40 licensed public datasets rather than proprietary hospital collections. The 15 anatomies are fetus, thyroid, breast, heart, liver, cervix, carotid artery, musculoskeletal system, kidney, prostate, skin, lung, pancreas, brachial plexus, and colon (Le et al., 23 May 2025).

The source datasets include breast datasets such as BUSV, DBUI, UBIBC, and BrEaST; thyroid datasets such as AUITD, Thyroid, and DDTI; fetal datasets such as FETAL PLANES DB, FPUS23, HC18, ACOUSLIC-AI, AFSP, JNU-IFM, and PSFHS; cardiac datasets such as CAMUS and Unity Imaging Collaborative; liver, kidney, prostate, vascular, skin, lung, and nerve datasets including DBFLUSI, TOKUD, MUPSD, C-TRUS, CUBS, CAUCD, CCA, DUIC, COVID-BLUES, and USGRA; and CUSRD for report generation (Le et al., 23 May 2025).

The benchmark reuses annotations from original datasets rather than introducing new clinical labels. Diagnosis, view, clinical value, report, and caption labels are inherited directly. Lesion localization annotations are derived from segmentation masks or bounding boxes and converted into nine spatial grid categories. Organ detection uses segmentation labels or presence annotations to indicate which organ is visible. Keypoint detection uses original point annotations but evaluates them after discretization into a classification-style formulation (Le et al., 23 May 2025).

Quality control follows a two-stage QA process. Automated checks remove missing labels, inconsistent metadata, invalid measurements, or corrupted images. Manual verification is then performed by a team of 10 annotators; each case is checked by at least 3 annotators for image-label alignment, bounding-box correctness, measurement units and numeric plausibility, and text-report alignment, with disagreements resolved by majority vote (Le et al., 23 May 2025).

3. Task design and prompt formulation

The benchmark organizes evaluation into 8 tasks grouped under 4 capability families: classification, detection, regression, and text generation. The case counts and task-level metrics are as follows (Le et al., 23 May 2025).

Task	Cases	Metrics
DD	1,411	Acc, F1
VRA	1,588	Acc, F1
LL	503	Acc
OD	1,918	Acc
KD	500	Acc
CVE	521	RMSE, MAE, %tol
RG	600	BLEU-4, ROUGE, BERTScore
CG	200	BLEU-4, ROUGE, BERTScore

Disease Diagnosis (DD) asks the model to predict disease presence or severity from an ultrasound image. View Recognition and Assessment (VRA) asks the model to identify the standard plane or view, and in some settings assess adequacy. These are conventional classification tasks, but they are clinically central because view correctness constrains both measurement validity and downstream diagnosis (Le et al., 23 May 2025).

The three detection tasks are deliberately reformulated to fit LVLM interfaces. Lesion Localization (LL) is not evaluated with continuous bounding boxes or IoU; instead, lesions are mapped to one of nine spatial categories in a $3 \times 3$ grid: upper-left, upper-middle, upper-right, center-left, center, center-right, lower-left, lower-middle, and lower-right. Keypoint Detection (KD) follows a similar discretization from original point annotations to coarse location classification. Organ Detection (OD) is framed as selecting the visible organ from candidate classes. This design standardizes interaction with text-generating LVLMs, but it also means that U2-BENCH measures coarse spatial reasoning rather than full detection performance in the computer-vision sense (Le et al., 23 May 2025).

Clinical Value Estimation (CVE) is the regression component. It targets continuous quantities such as fetal biometry, carotid intima-media thickness, ejection fraction or volumes, and liver fat grading or percentages. Report Generation (RG) and Caption Generation (CG) are the two text-generation tasks. RG uses structured clinical reports, while CG targets short anatomical descriptions (Le et al., 23 May 2025).

Prompt construction is standardized across 50 scenarios. Each prompt contains a role definition, a task instruction, and an explicit output specification. For classification, the prompt constrains the answer to one of the listed labels. For regression, it requires a single numeric value in a specified range and unit. For report generation, it provides examples of the expected section structure. The benchmark paper reports that instruction following is high across models, so the dominant failure mode is not compliance with prompt format but perception, reasoning, and clinical grounding (Le et al., 23 May 2025).

4. Evaluation protocol and the U2-Score

U2-BENCH is a pure test benchmark. The paper evaluates models in zero-shot or prompted settings and does not fine-tune them on benchmark data. Most video datasets are reduced to selected key frames per study, a pragmatic choice that controls evaluation cost but also limits explicit testing of temporal reasoning and real-time probe-motion understanding (Le et al., 23 May 2025).

Task-specific metrics follow standard definitions. Classification and detection-style tasks use Accuracy, with macro F1 additionally reported for DD and VRA. CVE uses RMSE, MAE, and percentage within tolerance. RG and CG use BLEU-4, ROUGE, and BERTScore. The benchmark’s composite metric is the U2-Score: $$\text{U2-Score} := \sum_{t=1}^{N} w_t d_t, \qquad w_t = \frac{n_t}{\sum_j n_j},$$ where $N = 8$ tasks, $n_t$ is the sample count for task $t$, and $d_t$ is the normalized task metric. For DD, VRA, LL, OD, and KD, $d_t$ is Accuracy; for CVE, $d_t = 1 - \text{RMSE}$ after RMSE normalization to $[0,1]$; for RG and CG, $d_t$ is BLEU-4 scaled to $$\text{U2-Score} := \sum_{t=1}^{N} w_t d_t, \qquad w_t = \frac{n_t}{\sum_j n_j},$$0 (Le et al., 23 May 2025).

The task weights are sample-count based:

Task	Weight $$\text{U2-Score} := \sum_{t=1}^{N} w_t d_t, \qquad w_t = \frac{n_t}{\sum_j n_j},$$1	U2-Score metric
DD	0.20	Acc
VRA	0.20	Acc
LL	0.07	Acc
OD	0.27	Acc
KD	0.07	Acc
CVE	0.07	$$\text{U2-Score} := \sum_{t=1}^{N} w_t d_t, \qquad w_t = \frac{n_t}{\sum_j n_j},$$2
RG	0.08	BLEU-4
CG	0.04	BLEU-4

This weighting scheme makes OD the largest contributor and gives classification tasks a large aggregate share. A plausible implication is that models with strong coarse recognition can rank relatively well even when they remain weak on report generation or keypoint reasoning, although the benchmark still exposes those deficits through per-task metrics.

The benchmark dataset is publicly released on HuggingFace, and the evaluation code is provided through a framework based on VLMEvalKit. The paper reports evaluation on 32 NVIDIA A800 GPUs over roughly two weeks (Le et al., 23 May 2025).

5. Empirical findings and observed error structure

The benchmark paper evaluates 20 state-of-the-art LVLMs, including medical-specific open-source models, general open-source LVLMs, and closed-source systems. The main aggregate finding is that ultrasound is not uniformly solved by current LVLMs. Performance is relatively stronger on image-level classification, but persistent weaknesses remain in spatial reasoning, quantitative estimation, and especially clinically faithful text generation (Le et al., 23 May 2025).

On the original benchmark leaderboard reported in the paper, the highest U2-Score is achieved by Gemini-2.5-Pro-Preview at 0.2968, followed by Dolphin-V1 at 0.2841, with DeepSeek-VL2 the strongest open-source model at 0.2630. Closed-source systems generally outperform open-source systems, but the gap is modest rather than overwhelming. Within the Qwen2.5-VL family, larger models improve DD accuracy and CVE RMSE, but improvements in localization and generation are not monotonic, which suggests that scale alone does not resolve ultrasound-specific failure modes (Le et al., 23 May 2025).

The taskwise pattern is more diagnostic than the aggregate ranking. DD and VRA are the benchmark’s comparatively strong regimes. By contrast, LL and KD are notably difficult: LL peaks around 0.295 to 0.300 Accuracy, and KD remains below 0.16 Accuracy across all models. OD is also challenging, with Gemini-2.5-Pro-Preview reaching 0.2714 Accuracy. For RG, BLEU-4 scores remain very low, with DeepSeek-VL2 at 7.47% as the best reported system in the benchmark paper, while BERTScore values are higher but still consistent with semantically vague or weakly aligned reporting. These results support the paper’s central claim that image-level classification is substantially easier for present LVLMs than spatial reasoning or clinical language generation in ultrasound (Le et al., 23 May 2025).

The paper also includes a prompt-design ablation on DD using 521 breast and thyroid cases with Gemini-2.0-Pro-Exp. Adding explicit anatomy tokens to the prompt increases accuracy from 45.1% to 52.4%, a gain of 7.3 percentage points, with McNemar’s $$\text{U2-Score} := \sum_{t=1}^{N} w_t d_t, \qquad w_t = \frac{n_t}{\sum_j n_j},$$3 and $$\text{U2-Score} := \sum_{t=1}^{N} w_t d_t, \qquad w_t = \frac{n_t}{\sum_j n_j},$$4. This indicates that contextual prompt specification has a measurable causal effect on ultrasound performance, and that prompt engineering is not a superficial concern in this domain (Le et al., 23 May 2025).

6. Adoption, nomenclature, and limitations

U2-BENCH rapidly became a reference evaluation suite for ultrasound LVLMs. The "Dolphin v1.0 Technical Report" uses it as the primary external benchmark and reports a U2-score of 0.5390 for Dolphin v1.0 and 0.5835 for Dolphin R1, with the report identifying 0.2968 as the second-best score among comparison systems. In that usage, U2-BENCH functions as a standardized testbed for measuring whether ultrasound-specialized training and reasoning-oriented reinforcement learning improve multimodal ultrasound understanding across the full task set (Weng et al., 30 Sep 2025).

A common misconception arises from nomenclature. U2-BENCH is unrelated to the U2 and U2++ line of end-to-end speech-recognition models, including "Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames" and "U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF," and it is also unrelated to "Unsupervised Unfolded rPCA (U2-rPCA)" in ultrasound microvascular imaging. These works use "U2" in different senses and belong to distinct research programs in ASR, MoE scaling, and clutter filtering rather than multimodal ultrasound benchmarking (Liang et al., 2022, Song et al., 2024, Li et al., 1 Oct 2025).

The benchmark paper explicitly acknowledges several limitations. Geographic coverage, vendor diversity, and pathology coverage are constrained by what is available in public datasets. Some anatomies have relatively small sample sizes. Videos are reduced to key frames, so temporal dynamics and full cine-loop reasoning are not comprehensively tested. Detailed subgroup fairness analysis is not reported because public datasets often lack demographic metadata. Long-form contextual reasoning is also limited, because the benchmark does not integrate EHR, laboratory values, or broader clinical notes (Le et al., 23 May 2025).

These design choices define both the utility and the present boundary of U2-BENCH. It is the first comprehensive benchmark focused on LVLMs for ultrasound understanding, but it is not a complete simulation of ultrasound practice. Its most important contribution is therefore methodological: it turns ultrasound LVLM evaluation into a unified, public, multi-task problem and makes clear that progress in the field requires more than generic multimodal competence. The benchmark shows that current models can often classify, sometimes estimate, but still struggle to localize, measure, and report with the reliability that routine clinical ultrasound demands (Le et al., 23 May 2025).