TinyUSFM Evaluation Framework

• TinyUSFM-based Evaluation Framework is a modality-specific ultrasound system that employs both full-reference (TinyUSFM-uLPIPS) and no-reference (TinyUSFM-NRQ) metrics to assess diagnostic quality.
• It integrates advanced methods such as token neighborhood comparisons and worst-region aggregation to capture structural nuances and artifact detection in ultrasound images.
• The framework benchmarks compact models on organ versatility, task adaptability, and efficiency while leveraging distillation and coreset selection to optimize performance.

The TinyUSFM-based evaluation framework denotes a set of evaluation procedures built around the pretrained ultrasound foundation model TinyUSFM. In one formulation, it is a modality-aligned ultrasound quality-evaluation framework comprising TinyUSFM-uLPIPS for full-reference assessment and TinyUSFM-NRQ for no-reference assessment, with the same feature space reused as a perceptual loss for super-resolution optimization. In another formulation, it is an evaluation recipe for compact ultrasound foundation models centered on UniUS-Bench, downstream transfer across classification and segmentation, and explicit efficiency reporting. Across both formulations, the common objective is to replace generic, domain-mismatched evaluation with procedures aligned to ultrasound imaging physics, organ diversity, task adaptability, and deployment constraints (Huang et al., 21 Apr 2026, Ma et al., 22 Oct 2025).

1. Motivation and problem setting

The framework arises from the claim that ultrasound evaluation is poorly served by generic image-quality measures. In reconstruction settings, existing standards such as PSNR and VGG-LPIPS are described as inadequate because they fail to account for modality-specific physics or the structural nuances of acoustic imaging. The cited failure modes are threefold: pixel fidelity is not equivalent to diagnostic utility; natural-image perceptual metrics are domain-mismatched; and prior ultrasound-specific no-reference QA methods such as NIQE-, BRISQUE-, fetal-IQA-, or KDE-based methods are narrow and do not generalize well across organs (Huang et al., 21 Apr 2026).

The underlying rationale is explicitly ultrasound-specific. Image appearance is governed by speckle from coherent interference, view-dependent specular reflections, attenuation, acoustic shadowing, scan-conversion artifacts, and anisotropic spatial resolution. Under these conditions, global intensity agreement can over-penalize benign brightness or gain changes while missing localized failures such as shadow or clutter regions that erase diagnostically important boundaries. The framework therefore operationalizes diagnostic utility through two anchors: task-linked quality, in which image corruption should track downstream Dice-score or classification damage, and clinician preference, in which experts should prefer the higher-ranked image (Huang et al., 21 Apr 2026).

In the compact-foundation-model setting, the motivation is complementary rather than identical. The question is not only whether a small encoder performs well, but whether it preserves the organ versatility and task adaptability associated with a large ultrasound foundation model while remaining computationally deployable. The evaluation is therefore designed to assess generalization across organs, tasks, datasets, centers, and devices; organ versatility over 15 organs; task adaptability over 8 classification and 10 segmentation datasets; and efficiency through parameter count, GFLOPs, inference time, GPU memory usage, and training-time trends in coreset-size ablations (Ma et al., 22 Oct 2025).

2. Shared representation and architectural basis

Both formulations are built around TinyUSFM as a shared feature extractor. For an input image $x$, TinyUSFM produces hidden representations at multiple transformer layers,

$f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$

where $T_\ell$ is the number of spatial patch tokens and $C_\ell$ is the channel dimension. After removing the global [CLS] token, the patch-token matrix is

$F_\ell(x)\in\mathbb{R}^{T_\ell\times C_\ell}.$

For no-reference quality modeling, each layer is summarized by average pooling over patch tokens to obtain

$u_\ell(x)\in\mathbb{R}^{C_\ell},$

and pooled features from selected layers $\mathcal{L}$ are concatenated and $L_2$-normalized into a global descriptor $z(x)$ (Huang et al., 21 Apr 2026).

The selected layers are

$\mathcal{L}=\{3,5,7,11\}.$

These layers recur throughout the framework: they define the multi-layer feature space used by TinyUSFM-uLPIPS, the descriptor construction used by TinyUSFM-NRQ, and the downstream multi-scale features used for segmentation in the compact-FM benchmark (Huang et al., 21 Apr 2026, Ma et al., 22 Oct 2025).

TinyUSFM itself is introduced as a lightweight ultrasound foundation model derived from USFM through knowledge distillation with feature-gradient driven coreset selection, domain-separated masked image modeling, and consistency-driven dynamic distillation. The model is evaluated as retaining the teacher’s versatility with substantially reduced computational cost. The compactness figures reported for TinyUSFM are central to the broader evaluation framework: in classification, TinyUSFM has 5.5M parameters and 2.16 GFLOPs, versus 86.5M and 33.72 GFLOPs for USFM; in segmentation, TinyUSFM has 6.2M parameters and 3.45 GFLOPs, versus 101.2M and 53.07 GFLOPs for USFM (Ma et al., 22 Oct 2025).

3. Quality-assessment subsystem: TinyUSFM-uLPIPS and TinyUSFM-NRQ

The quality-assessment subsystem is organized around two complementary metrics, one full-reference and one no-reference.

Component	Core mechanism	Primary use
TinyUSFM-uLPIPS	multi-layer token relations and Gram matrices	benchmarking reconstruction models in paired settings
TinyUSFM-NRQ	clean-manifold modeling and worst-region aggregation	deployment-time QC without ground-truth images

TinyUSFM-uLPIPS is the full-reference metric. It compares a reference image $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$0 and a reconstructed or test image $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$1 in TinyUSFM feature space. Its distinguishing feature is that it does not only compare corresponding token embeddings directly; it also compares relations among neighboring tokens and global channel co-activation statistics. In all experiments, the layer set is $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$2, the neighborhood radius is $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$3, and the temperature is $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$4. After row-wise $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$5-normalization of token matrices, local neighborhoods are formed around the same token index $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$6 in the reference and test images, and local self-similarity matrices are compared after row-wise softmax sharpening. A Gram-matrix term

$f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$7

is added to capture global channel co-activation structure. The final TinyUSFM-uLPIPS score is the equally weighted average across selected layers, and lower values correspond to higher similarity or better reconstruction (Huang et al., 21 Apr 2026).

Implementation-wise, TinyUSFM-uLPIPS uses explicit token correspondence, because token neighborhoods are formed around the same token index in aligned reference and test images. This makes it suited to reconstruction and restoration settings in which spatial alignment is expected. The explicit aggregation sequence is: extract multi-layer token matrices; row-wise normalize tokens; build local neighborhood matrices; compute local self-similarity matrices; compare row-softmaxed relation matrices using mean absolute difference; compute global Gram matrices and compare them; sum local and Gram distances per layer; and average over layers (Huang et al., 21 Apr 2026).

TinyUSFM-NRQ is the no-reference metric. It scores a single image $f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$8 by checking whether its local patch descriptors lie on a learned clean ultrasound manifold. The method extracts overlapping patches

$f_\ell(x)\in\mathbb{R}^{(1+T_\ell)\times C_\ell},$9

of size $T_\ell$0 with stride $T_\ell$1, encodes each patch using the TinyUSFM descriptor $T_\ell$2, projects clean patch descriptors into a PCA space of dimension $T_\ell$3, and fits organ-specific diagonal-covariance Gaussian mixture models with $T_\ell$4 components. If the organ label is known, the patch score is

$T_\ell$5

If the organ label is unavailable, the score is computed using a uniform mixture over all organ models,

$T_\ell$6

Higher likelihood indicates that a patch resembles valid clean ultrasound content; lower likelihood indicates anomaly, artifact, or out-of-distribution appearance (Huang et al., 21 Apr 2026).

The defining aggregation rule of TinyUSFM-NRQ is worst-region aggregation. Let

$T_\ell$7

and let $T_\ell$8 index the $T_\ell$9 lowest-scoring patches. The final score is

$C_\ell$0

with higher values indicating better quality. This is explicitly described as a robust lower-tail pooling strategy: it is not a strict minimum, which could be unstable, and not a full average, which could miss local defects (Huang et al., 21 Apr 2026).

4. Benchmark protocol for compact ultrasound foundation models

As a broader evaluation framework, TinyUSFM is assessed through UniUS-Bench, introduced as “the largest publicly available ultrasound benchmark comprising 8 classification and 10 segmentation datasets across 15 organs.” UniUS-Bench contains 60,940 ultrasound images and 62,409 annotations, is built entirely from existing public datasets, and uses official train/validation/test splits whenever available; otherwise it uses a stratified random split of 7:1:2. All input images are resized to $C_\ell$1 (Ma et al., 22 Oct 2025).

The 8 classification datasets are CUBS, UF1990, TN3K, STMUS, AUL, BUSI, MMOTU, and Fetal Planes. The 10 segmentation datasets are Luminous, KidneyUS, GIST514, DDTI, MMOTU, BUSBRA, Ultrasound Nerve Seg., LUSS, FH-PS-AoP, and CAMUS. The benchmark spans 15 organs or structures: carotid, uterine, thyroid, skeletal muscle, liver, breast, ovarian, fetus, multifidus muscle, kidney, stomach, neck nerve, lung, pelvis, and cardiac. The task mix includes binary and multi-class classification, as well as single-target and multi-target segmentation (Ma et al., 22 Oct 2025).

The reported downstream metrics are classification accuracy and Dice score. The paper reports per-dataset mean ± standard deviation and task-family averages, but does not report AUC, F1, sensitivity, specificity, precision/recall, calibration metrics, IoU, Hausdorff distance, ASSD, or lesion-wise metrics. It also does not define a single combined score merging classification and segmentation, nor weighted averaging by dataset size (Ma et al., 22 Oct 2025).

A central part of the protocol is the baseline structure. The evaluation compares TinyUSFM with conventional models, lightweight models, foundation models, its teacher USFM, and a vanilla ViT-Tiny or ViT-T backbone of comparable size. For classification, the compared models are ResNet50, ViT-B, EfficientNet, MobileViT, ViT-T, SimMIM, USFM, and TinyUSFM. For segmentation, the compared models are TransUNet, SwinUNet, SegFormer, SeaFormer, ViT-T, SimMIM, USFM, and TinyUSFM. The paper states that all compared methods were implemented following their official repositories or original-paper settings to ensure fair comparison, although not every training detail is documented exhaustively (Ma et al., 22 Oct 2025).

Efficiency reporting is not auxiliary but constitutive. The protocol includes parameter count, GFLOPs, inference time, GPU memory usage, and training-time trends in coreset-size ablations. On RTX 2080 Ti with batch size 16, the reported deployment figures for TinyUSFM are average inference time per image below 1 ms and GPU memory usage under 1 GB (Ma et al., 22 Oct 2025).

5. Distillation-aware evaluation and empirical results

The evaluation of TinyUSFM as a compact foundation model is explicitly distillation-aware. TinyUSFM is trained by distilling from USFM using a curated coreset selected from the large-scale 3M-US database, domain-separated masks, consistency-weighted deep feature distillation, and mid-layer reconstruction supervision. The coreset-size ablation varies subset size from 10K to full 3M-US, with performance increasing up to 200K and then declining when more data are added. At 200K images, TinyUSFM achieves the best reported downstream performance—84.91 average classification accuracy and 85.78 average segmentation Dice—while using only about 10% of the full dataset and requiring less than 10% of full training time on one A100 GPU. The same 200K subset outperforms the full dataset by +1.37% in classification and +1.77% in segmentation (Ma et al., 22 Oct 2025).

The coreset-selection ablation further compares the proposed feature-gradient driven method against random sampling, feature-only clustering, and gradient-only filtering at 200K images. The reported gains are +2.54% classification and +2.67% segmentation over random sampling, +1.03% and +1.08% over feature-only clustering, and +1.87% and +1.44% over gradient-only filtering. The framework therefore evaluates not only end-task transfer but also the validity of data curation for small ultrasound encoders (Ma et al., 22 Oct 2025).

The quality-evaluation branch reports four main claims: task-linked quality, cross-organ comparability, PSNR-consistent sensitivity, and clinical utility. For full-reference evaluation, TinyUSFM-uLPIPS outperforms VGG-LPIPS in correlation with segmentation damage on DDTI and KidneyUS, and the text emphasizes that VGG-LPIPS systematically over-penalizes noise-like degradations and under-penalizes ultrasound-specific structural artifacts. A specific example is reported on DDTI at PSNR = 20: VGG-LPIPS ranks AdditiveGaussian as more severe than ClutterHaze/ROIShadow Occlusion even though the latter causes a larger Dice drop, whereas TinyUSFM-uLPIPS gives the correct ordering. For no-reference evaluation, TinyUSFM-NRQ achieves mean Spearman $C_\ell$2 over PSNR levels for within-organ monotonicity, while NIQE, BRISQUE, and KDE-UQA fail systematically on missing scanlines and specular clipping and may become numerically ill-conditioned for structural artifacts at PSNR = 25. Under domain shift to unseen organs—stomach, multifidus muscle, and neck nerve—TinyUSFM-NRQ maintains strong monotonicity and consistent severity-ranking trends (Huang et al., 21 Apr 2026).

The clinical utility results are among the most specific. In a blinded two-alternative forced choice study using PSNR-matched image pairs, TinyUSFM-NRQ predicts clinician preference with 72.8% accuracy, with 95% CI $C_\ell$3 under a Wilson interval and $C_\ell$4 under a two-sided binomial test, contrasted with a baseline preference-prediction level of 47.2%. Reader sanity checks achieve 100% accuracy and consistency. In the super-resolution optimization loop, SwinIR trained with TinyUSFM perceptual loss rather than VGG improves PSNR by 5.3% and SSIM by 3.3% relative to $C_\ell$5. Clinician pairwise preference for super-resolution outputs is 87.5% for TinyUSFM-loss outputs over $C_\ell$6, 100% for TinyUSFM-loss outputs over VGG-loss outputs, and 97.5% for $C_\ell$7 over VGG-loss outputs (Huang et al., 21 Apr 2026).

The headline compact-FM benchmark results are likewise explicit. TinyUSFM achieves 84.91% average classification accuracy and 85.78% average segmentation Dice across UniUS-Bench, compared with 85.51 and 86.46 for USFM. These results are obtained with 6.36% of the teacher’s parameters and 6.40% of the teacher’s GFLOPs in classification, and with comparable reductions in segmentation. Relative to the vanilla model, TinyUSFM improves classification by 9.45% and segmentation by 7.72%. The per-dataset results further show that TinyUSFM exceeds USFM on ovarian classification and fetal plane classification, and on neck nerve, pelvis, and cardiac segmentation, while remaining slightly behind on several other datasets (Ma et al., 22 Oct 2025).

6. Applications, limitations, and interpretive issues

The framework supports both evaluation and optimization. TinyUSFM-uLPIPS is intended for paired benchmarking of reconstruction, denoising, deblurring, super-resolution, or generative fidelity when a clean reference exists and alignment is available. TinyUSFM-NRQ is intended for deployment-time quality control, scoring real acquisitions or reconstructions without paired references, monitoring incoming images, flagging suspicious outputs, and identifying localized failure regions through the worst-scoring patches. The same feature space is also reused as a differentiable TinyUSFM perceptual loss for super-resolution training, creating an integrated assessment-and-optimization loop (Huang et al., 21 Apr 2026).

A broader practical implication is that the framework defines three simultaneous axes for compact ultrasound foundation-model evaluation: foundation-like transfer ability across multiple organs and tasks; compactness and deployability through params, GFLOPs, memory, and latency; and distillation or data-efficiency validity through curated-versus-full-data scaling and component-wise ablations. This suggests that a small ultrasound model should not be evaluated only by a single downstream accuracy number, but by a structured profile of transfer, efficiency, and training-data behavior (Ma et al., 22 Oct 2025).

Several caveats are explicit or implicit. In the quality-evaluation formulation, metric quality depends on the representational quality and coverage of the pretrained TinyUSFM model; the full-reference metric assumes alignment because it compares corresponding token neighborhoods; and TinyUSFM-NRQ requires clean patch datasets together with fitted PCA and GMM clean-manifold models. Performance may therefore depend on training-organ coverage, and full generalization across scanners, probes, pathologies, and acquisition protocols is not guaranteed. The averaging of the worst 15% of patches is effective in the reported experiments, but the paper notes that this hyperparameter may need retuning when artifacts are more diffuse or image sizes differ substantially. The system is also validated against downstream tasks and preference judgments rather than trained directly on large-scale expert MOS labels (Huang et al., 21 Apr 2026).

In the compact-FM formulation, the benchmark is heterogeneous by design and integrates datasets from different medical centers and imaging devices, but the paper does not provide dedicated per-center or per-device stratified metrics. It reports mean ± standard deviation but does not specify the number of repeated runs or seeds, confidence intervals, or formal significance tests. The downstream adaptation protocol states that classification uses TinyUSFM features plus a linear layer and segmentation uses TinyUSFM as backbone with a lightweight pyramid neck and FPN-style head, but whether the encoder is frozen or fully fine-tuned is not stated explicitly. The paper also does not explicitly state the release of code, pretrained TinyUSFM weights, exact benchmark split files, coreset indices, or training scripts (Ma et al., 22 Oct 2025).

A recurring misconception addressed by the framework is that generic fidelity metrics or natural-image perceptual metrics are sufficient for ultrasound. The reported evidence rejects that assumption in two ways: first, by showing that PSNR and VGG-LPIPS can mis-rank clinically harmful artifacts; and second, by showing that compact-model evaluation cannot be reduced to one downstream score without explicit efficiency and transfer analysis. In that sense, the TinyUSFM-based evaluation framework is not a single metric but a coordinated modality-native evaluation program spanning reconstruction quality, clinical acceptability, benchmark transfer, and deployability (Huang et al., 21 Apr 2026, Ma et al., 22 Oct 2025).