Boosting Ultrasound Image Classification via Attribute-Guided Dual-Branch Framework

Abstract: Ultrasound image classification is essential for computer-aided diagnosis. However, current methods often neglect clinical priors, leading to poor generalization in challenging scenarios and a lack of interpretability that limits clinical adoption. To address these issues, we aim to develop a medical-prior module that can be seamlessly integrated into existing pipelines to enhance both diagnostic performance and interpretability. In this paper, we propose an attribute-guided dual-branch framework for ultrasound classification that introduces domain-agnostic medical attribute priors, improving generalization while offering interpretable evidence. Specifically, a baseline branch follows conventional architectures and predicts image categories via a fully connected classifier. An attribute-guided branch injects domain-agnostic attributes as priors and produces human-interpretable decision cues. Finally, an adaptive decision module fuses the two branches in a data-dependent manner to yield the final prediction. Experiments across diverse ultrasound classification tasks demonstrate that our approach can be integrated into multiple backbones and state-of-the-art methods with low overhead, consistently improving accuracy and interpretability. Code is available at: https://github.com/zhaobo253-crypto/AttrGuide.

• The paper introduces AttrGuide, a dual-branch framework that integrates clinical attribute priors via CLIP embeddings to enhance interpretability and robustness.
• It achieves measurable performance gains across BUSI and fetal ultrasound benchmarks by adaptively fusing baseline and attribute-guided predictions.
• The method improves generalization and offers clinically verifiable decision rationales while incurring minimal computational overhead.

Attribute-Guided Dual-Branch Framework for Ultrasound Image Classification

Introduction

Deep learning-based ultrasound image classification remains challenging due to heterogeneity in acquisition protocols, low image quality, and the intrinsic limitations of black-box models that lack clinically grounded interpretability. Prevailing methods heavily rely on learning texture- or appearance-based discriminative features, often neglecting clinical semantics and domain knowledge, which has led to concerns regarding poor generalization—especially in out-of-distribution and hard-case scenarios. The paper “Boosting Ultrasound Image Classification via Attribute-Guided Dual-Branch Framework” (2607.01648) systematically addresses these deficits by introducing AttrGuide, an architecture-agnostic, plug-and-play module that injects domain-agnostic clinical attribute priors into deep classifiers. The overarching goals are to achieve robust performance under distributional shifts and to provide interpretable, clinically verifiable decision rationales.

Figure 1: Traditional classifiers fail on hard samples; AttrGuide adds an attribute-guided branch for semantic self-correction with negligible overhead.

Methodology

AttrGuide implements a dual-branch design consisting of a standard baseline branch and an attribute-guided semantic branch, both utilizing shared encoder features but operating in parallel and fusing outputs adaptively. The baseline branch preserves the original end-to-end image classification pipeline, while the attribute branch explicitly integrates predefined, domain-agnostic clinical attributes—serving as plug-in guidance for Class-Attribute association. Attribute priors are encoded using CLIP text embeddings, mapped to a common latent space, and cross-matched with local image features for semantic alignment.

The attribute-guided branch predicts the presence of each attribute via cosine similarity-based matching and then maps attribute activations to class scores through a fixed class-attribute incidence matrix. Supervision is applied both at the global (classification) and attribute levels with cross-entropy, BCEWithLogits, and joint regularization losses to align predicted attributes and class membership.

An adaptive fusion module, parameterized by learnable weights and temperature, interpolates baseline and attribute-guided predictions at the logit level. During training, the classifier is solely supervised via this fused output. This fusion enables low-cost, data-dependent self-correction and improved robustness to challenging cases.

Figure 2: AttrGuide seamlessly plugs into existing ultrasound classifiers by reusing feature maps; attribute-guided scores provide semantic priors for interpretable and robust decision-making.

Experimental Results

Performance

AttrGuide demonstrates consistent improvements across breast (BUSI) and fetal ultrasound classification benchmarks using diverse backbones, including ViT, ResNet50, and the high-capacity Vim-s-16 (BU-Mamba) architecture. For example, when integrated with BU-Mamba, accuracy on BUSI improves from 87.86% (baseline) to 88.72%, and similar gains are achieved on ViT-B and ResNet50 backbones. On a challenging private 7-class fetal dataset, test accuracy for ViT-B increases from 92.8% (baseline) to 94.9% with AttrGuide—corresponding to an absolute improvement of +2.1% and a macro F1 increase of +3.56%.

Computational analysis shows minimal overhead: incorporating the attribute branch results in a marginal increase in training time (e.g., +0.16 minutes for ViT-B per fold on BUSI) and a parameter increment of less than 2%.

Ablation Analysis

Ablation studies confirm the necessity of each component. The learnable fusion module is critical, yielding an improvement of +5.8% accuracy over the classifier-only baseline on the private fetal task, with naive averaging being notably inferior. Standalone attribute or classifier branches underperform compared to the synergistic dual-branch configuration, validating the design's importance.

Interpretability and Attribute Prediction

Interpretability is quantitatively assessed by measuring attribute prediction accuracy. AttrGuide achieves 87.56% on the fetal dataset and 83.29% on BUSI, indicating that attribute-guided predictions are often clinically plausible and verifiable.

Figure 3: The attribute-guided branch can predict explicit, meaningful clinical attributes, providing interpretable outputs with high accuracy.

Practical and Theoretical Implications

The plug-and-play nature of AttrGuide enables rapid adoption in existing clinical pipelines and research codebases, with minimal re-engineering. The approach's lightweight overhead ensures viability in resource-constrained environments and real-time clinical settings. Moreover, by injecting CLIP-derived domain-agnostic priors, AttrGuide facilitates semantic alignment with clinician reasoning and, crucially, produces evidence that can be independently scrutinized by experts, addressing regulatory and deployment barriers associated with opaque deep models.

The reliance on a fixed, handcrafted class-attribute matrix sidesteps the need for dense attribute annotations per image, offering practical gains in human annotation cost without sacrificing interpretability. The methodology is agnostic to backbone choice, indicating potential extensibility to other imaging modalities or multi-task medical vision pipelines.

Theoretically, the combination of domain-agnostic semantic spaces and adaptive logit fusion highlights a new paradigm for robust, self-corrective medical AI. This framework could stimulate further work on interpretable detection, concept-aligned latent space regularization, and transfer learning via shared clinical priors across tasks and datasets. It may also inspire advances in zero-shot or few-shot medical image understanding.

Conclusions

AttrGuide establishes a principled, efficient, and interpretable pathway for ultrasound image classification by fusing baseline deep representations with explicit clinical attribute guidance. Benchmark evaluation confirms consistent gains in accuracy and interpretability with negligible additional computational cost. The dual-branch attribute-guided approach, leveraging CLIP-encoded priors and a learnable decision fusion block, addresses longstanding issues of generalization and transparency in computer-aided diagnosis. Future work may generalize this framework to broader medical imaging applications, further strengthening the interface between AI-driven decision support and clinical workflows.