Defining Robust Ultrasound Quality Metrics via an Ultrasound Foundation Model

Abstract: Clinicians lack a principled framework to quantify diagnostic utility in ultrasound reconstructions. Existing standards like PSNR and VGG-LPIPS are inadequate, failing to account for modality-specific physics or the structural nuances of acoustic imaging. We close this gap with a TinyUSFM-based evaluation framework featuring two distinct metrics: TinyUSFM-uLPIPS, a full-reference perceptual distance based on multi-layer token relations, and TinyUSFM-NRQ, a deployable no-reference quality score utilizing clean-manifold modeling and worst-region aggregation to detect localized harmful artifacts. We demonstrate that the presented metrics have four unique advantages: 1) Task-linked quality, where TinyUSFM-uLPIPS achieves superior calibration with semantic task damage, accurately reflecting Dice-score drops in segmentation where VGG-based metrics fail; 2) Cross-organ comparability, maintaining stable scoring scales and consistent severity rankings across diverse anatomical sites and domain-shifted data; 3) PSNR-consistent sensitivity, with TinyUSFM-NRQ providing a reliable quality score without ground-truth images that remains consistent with traditional fidelity benchmarks (i.e. PSNR); and 4) Clinical utility, improving the prediction of expert preference from 47.2$\%$ to 72.8$\%$ accuracy and producing super-resolution reconstructions preferred by sonographers. By integrating these advantages into a unified assessment and optimization loop, this work establishes a modality-aligned standard that finally bridges the gap between algorithmic performance and diagnostic utility. https://github.com/sextant-fable/US-Metrics

• The paper presents TinyUSFM, a lightweight transformer model that extracts ultrasound-native features for precise quality assessment.
• It introduces two metrics, TinyUSFM-uLPIPS and TinyUSFM-NRQ, that reliably capture clinically relevant artifacts and correlate with segmentation performance.
• Experimental results demonstrate enhanced clinician concordance, cross-organ robustness, and superior performance compared to traditional quality evaluation methods.

Robust Ultrasound Quality Metrics with an Ultrasound Foundation Model

Introduction

Quantitative evaluation of ultrasound image quality remains a challenge due to a lack of modality-calibrated metrics sensitive to clinically relevant artifacts. Traditional standards such as PSNR, SSIM, and natural-image perceptual metrics (e.g., VGG-LPIPS) are inadequate for ultrasound, failing to capture modality-specific physics, domain shift across organs, and the localizable nature of diagnostic artifacts. This paper proposes a principled evaluation framework based on TinyUSFM, a compact foundation model, introducing a full-reference (FR) metric, TinyUSFM-uLPIPS, and a no-reference (NR) metric, TinyUSFM-NRQ, to bridge the gap between algorithmic reconstructions and clinical utility.

Figure 1: Overview of the TinyUSFM-based ultrasound quality evaluation framework, detailing metric motivation, FR/NR designs, and downstream diagnostic utility.

TinyUSFM Metrics and Framework

Ultrasound-Native Feature Extraction

TinyUSFM, a lightweight transformer-based ultrasound foundation model, computes multi-layer spatial patch token features. For FR comparison, token matrices after [CLS] removal are used from layers $\mathcal{L} = \{3,5,7,11\}$. NRQ pools patch tokens, forming global descriptors normalized across layers, robust to device and acquisition variability.

TinyUSFM-uLPIPS: Full-Reference Metric

TinyUSFM-uLPIPS compares test and reference images by evaluating both neighborhood token similarity and channel co-activation statistics (via Gram matrices) across the TinyUSFM feature space. By capturing token-level relations with layer-wise softmax aggregation and temperature scaling, it provides a structured distance measure that penalizes clinically harmful distortions, such as speckle or shadowing, more reliably than VGG-LPIPS or pixel-based metrics.

TinyUSFM-NRQ: No-Reference Metric

TinyUSFM-NRQ assesses a single image's quality using clean-manifold likelihood modeling and localized worst-region aggregation. Patch descriptors are projected into a PCA space and scored against organ-conditioned Gaussian mixture models trained on pristine examples. By averaging log-likelihoods of the lowest-scoring patches, NRQ becomes sensitive to localized degradations, mirroring expert review of diagnostic regions.

Experiments

Task-Linked Calibration and Diagnostic Relevance

To validate clinical relevance, the metrics were evaluated under controlled distortions, PSNR-matched, across thyroid and kidney ultrasound datasets. FR and NR metrics were tested for their correlation with Dice-score drops from segmentation models, reflecting how quality degradations interfere with task performance.

Figure 2: Calibration of FR metrics; TinyUSFM-uLPIPS demonstrates superior task-anchored rank correlation and stability across organs and PSNR anchors relative to VGG-LPIPS.

TinyUSFM-uLPIPS consistently shows higher and more stable Kendall's $\tau$ correlation with segmentation accuracy drops and correct severity ordering of artifacts. VGG-LPIPS exhibits noise bias and systematic failure to reflect local occlusion severity, contradicting clinical needs.

Cross-Organ Robustness and Baseline Comparison

Cross-organ degradation ranking consistency (Kendall's $W$) and score-scale dispersion (IQR) were evaluated over 9 organs, comparing TinyUSFM metrics to NIQE, BRISQUE, and KDE-UQA.

Figure 3: Cross-organ robustness—TinyUSFM metrics yield higher ranking consistency, lower inter-organ dispersion, and superior alignment with known degradation severity and clinician 2AFC judgments.

TinyUSFM-uLPIPS achieves uniform ranking and stable scale across anatomical sites, outperforming VGG-LPIPS, while NRQ maintains strong monotonicity (mean Spearman $\rho=0.744$) for synthetic artifact severity. Legacy NRQ baselines, especially under shadowing and scanline loss, are unreliable and poorly correlated with task degradation or clinical preference.

Clinician Utility Assessment

A blinded two-alternative forced choice (2AFC) evaluation was conducted, comparing NRQ scores to clinician preference across matched PSNR pairs from multiple organs and artifact types. TinyUSFM-NRQ predicts expert preference with 72.8% accuracy (95% CI [0.689, 0.764]), significantly outperforming chance and classical metrics.

Ultrasound Super-Resolution Optimization

TinyUSFM feature space was employed as a perceptual loss for ultrasound SR using SwinIR. Three objectives ($\mathcal{L}_1$, $\mathcal{L}_1+\mathcal{L}_{\mathrm{VGG}}$, $\mathcal{L}_1+\mathcal{L}_{\mathrm{TinyUSFM}}$) were compared on breast and echocardiography datasets.

Figure 4: SR outcome comparison—TinyUSFM perceptual loss yields reconstructions with superior perceptual and diagnostic fidelity, aligning both expert and metric preference.

$\mathcal{L}_{\mathrm{TinyUSFM}}$ improves PSNR and SSIM by 5.3% and 3.3% over VGG-based loss, avoids over-smoothing and hallucination, and generates reconstructions preferred by clinicians in 87.5–100% of pairwise evaluations. TinyUSFM metrics favor these outputs, establishing a closed-loop alignment between model optimization and evaluation.

Practical and Theoretical Implications

The presented framework provides a concrete modality-aligned answer to diagnostic utility assessment in ultrasound image analysis. By leveraging ultrasound-native foundation model features, it successfully calibrates both FR and NR metrics for task relevance, cross-organ robustness, and clinician concordance. The metrics enable principled quality control for generative and restoration pipelines, overcoming domain limitations of image-based and generic QA measures. TinyUSFM-NRQ's efficiency facilitates real-time monitoring, complementing the high-fidelity offline evaluation of TinyUSFM-uLPIPS.

Future research may extend this feature-centric approach to other foundation models, additional medical imaging modalities, or adaptive and multi-organ quality modeling. The methodology establishes a rigorous link between perceptual feature learning and practical clinical endpoints, enabling further advances in foundation model interpretability and modality-adaptive optimization.

Conclusion

This work introduces TinyUSFM-based ultrasound quality metrics, establishing a unified framework that robustly reflects clinical utility and cross-domain generalization, outperforming established baselines in both quantitative and clinician-involved evaluations. These metrics facilitate principled assessment and deployment of ultrasound reconstruction and restoration algorithms and provide a viable direction for developing modality-specific IQA/QA standards in medical imaging (2604.19512).