Mammo-CLIP Dissect Analysis

• The paper introduces Mammo-CLIP Dissect, a framework that assigns human-interpretable clinical labels to neurons using a mammography-specific vision-language model.
• It employs a four-stage pipeline—including image-text embedding and SoftWPMI scoring—to quantify neuron-concept alignment with radiological knowledge.
• Empirical findings show that mammography-specific pretraining enhances clinical concept learning, although fine-tuning can narrow the model's representational breadth.

Searching arXiv for the primary paper and closely related mammography vision-language work to ground the article in current literature. Mammo-CLIP Dissect is a concept-based explainability framework for systematically dissecting deep learning vision models trained for mammography. It uses a mammography-specific vision-LLM, Mammo-CLIP, as a “dissector” that labels neurons at specified layers with human-interpretable textual concepts and quantifies their alignment to domain knowledge. The framework was introduced to address a gap between pixel-based explainability methods and the higher-level conceptual reasoning used in radiology, and it is positioned as the first concept-based explainability framework for mammography models (Salahuddin et al., 25 Sep 2025). In the broader mammography VLM literature, it sits downstream of mammography-specific pretraining efforts such as Mammo-CLIP (Chen et al., 2024, Ghosh et al., 2024), and its findings are complementary to other interpretability analyses, including sparse-autoencoder-based probing of Mammo-CLIP latents (Nakka, 21 Jul 2025).

1. Definition, scope, and motivation

Mammo-CLIP Dissect was proposed in response to a recurrent problem in clinical AI: deep learning models for mammography screening can achieve strong results, but their “black box” character hinders deployment because it is difficult to determine whether the learned internal representations correspond to clinically meaningful reasoning. The framework targets this issue by shifting explanation from pixels alone to textual concepts that reflect mammography practice, including mass, calcification, density, findings, interpretations, and related BI-RADS-oriented descriptors (Salahuddin et al., 25 Sep 2025).

The central motivation is that pixel-based explainability methods such as Grad-CAM highlight regions but do not map model behavior to the semantic concepts central to clinical workflows. Mammo-CLIP Dissect instead asks what concepts a neuron or layer appears to encode. This emphasis on concept-level interpretation distinguishes it from earlier mammography VLM work that focused on data efficiency, robustness, classification, localization, or prompt-based prediction. Mammo-CLIP, for example, introduced a multi-modal framework to process multi-view mammograms and corresponding simple texts for malignancy prediction (Chen et al., 2024), while the paired-report Mammo-CLIP foundation model emphasized strong performance in classifying and localizing mammographic attributes and introduced Mammo-FActOR for sentence-level spatial attribution (Ghosh et al., 2024). Mammo-CLIP Dissect addresses a different question: how domain-specific knowledge is represented inside the visual model.

A plausible implication is that concept-level analysis is better suited than purely spatial saliency for auditing whether internal features align with radiologists’ workflows. The paper itself frames this as an advantage for trust, debuggability, and clinical insight (Salahuddin et al., 25 Sep 2025).

2. Framework architecture and neuron-labelling pipeline

The framework has three core elements: a dissector model $F_{\text{dissector}}$, a target model $F_{\text{target}}$, and a curated concept set $\mathcal{C}$. In the reported experiments, the dissector is Mammo-CLIP, described as a domain-specific vision-LLM with an EfficientNet-B5 image encoder and a BioClinicalBERT text encoder, pretrained on paired mammography images and reports. The target model is the deep vision model to be analyzed, and the concept set contains 763 unique concepts, co-developed with radiologists, spanning mammography-specific and non-mammography concepts (Salahuddin et al., 25 Sep 2025).

The operational pipeline proceeds in four stages. First, probing images and textual concepts are embedded with the dissector’s image and text encoders. For each image-concept pair, a similarity matrix $\mathbf{P}$ is formed with entries $$\mathbf{P}_{i,j}=\langle \mathbf{I}_i,\mathbf{T}_j\rangle$$. Second, for each neuron $k$ in a selected target-model layer, activations $A_k(\mathbf{x}_i)$ over the probe set are aggregated into an activation vector $\mathbf{q}_k$. Third, neuron–concept alignment is computed using Soft Weighted Pointwise Mutual Information (SoftWPMI), with

$$\text{sim}(t_m,\mathbf{q}_k)=\log \mathbb{E}[p(t_m \mid D^{\mathbf{q}_k}_{\text{probe}})]-\lambda \log p(t_m),$$

and the neuron is assigned the label

$\tilde{m}=\arg\max_m \text{sim}(t_m,\mathbf{q}_k).$

Fourth, an adaptive layer-wise threshold $F_{\text{target}}$0, defined as the mean similarity for that layer, is used to define “concept activation” and support layer- and model-level coverage analysis (Salahuddin et al., 25 Sep 2025).

This mechanism is conceptually related to CLIP-Dissect-style neuron description, but Mammo-CLIP Dissect specializes the idea to mammography through a mammography-trained dissector and a radiologist-informed concept ontology. The broader literature shows why such specialization is necessary. Mammography-specific VLMs were developed because generic CLIP-style models face atypical conditions in this modality, including high-resolution images, small regions of interest, limited image-report data, class imbalance, and multi-view acquisition structure (Du et al., 2024, Chen et al., 2024).

3. Concept inventory and experimental configurations

The concept inventory used in Mammo-CLIP Dissect consists of 763 unique terms or phrases. Of these, 369 are mammography-related concepts organized into 22 subcategories, and 394 are non-mammography concepts such as colors, materials, objects, and nature. The mammography-related categories include breast anatomy and structures, breast locations, findings, interpretations, and actions or follow-up. For downstream analyses, task-specific subsets were also defined: 73 mass concepts, 79 calcification concepts, and 38 density concepts (Salahuddin et al., 25 Sep 2025).

Three configurations were examined. In G-Mammo-CLIP Dissect, both the target and dissector correspond to a general model: an EfficientNet-B5 image encoder pretrained on ImageNet paired with a general biomedical text encoder, probed on VinDR-Mammo. In M-Mammo-CLIP Dissect, both target and dissector are mammography-pretrained, again probed on VinDR-Mammo. In C-Mammo-CLIP Dissect, the target is a task-specific classifier obtained by fine-tuning the mammography-pretrained image encoder on downstream tasks such as mass, calcification, density, or BI-RADS cancer classification, while the dissector remains mammography-specific (Salahuddin et al., 25 Sep 2025).

The design of these configurations allows three questions to be addressed directly: how concept learning differs between models trained on general image datasets versus mammography-specific datasets; how downstream fine-tuning alters concept specialization; and which mammography-relevant concepts remain underrepresented. This setup also situates Mammo-CLIP Dissect within a larger mammography representation-learning landscape. Mammography VLMs now span several supervision regimes: paired mammogram-report pretraining in Mammo-CLIP (Ghosh et al., 2024), multi-view prompt-based case classification in Mammo-CLIP (Chen et al., 2024), template-generated captions and symmetric local alignment in MaMA (Du et al., 2024), atlas-caption pretraining in MAM-CLIP (Gulluk et al., 19 May 2026), and synthetic-report multi-view pretraining in MV-MLM (Zheng et al., 30 Oct 2025). Mammo-CLIP Dissect can, in principle, analyze how these regimes shape internal concept representations, though the reported experiments focus on CNN-based mammography models analyzed with the Mammo-CLIP dissector (Salahuddin et al., 25 Sep 2025).

4. Principal empirical findings

The primary empirical result is that models trained on mammography data capture more clinically relevant concepts and align more closely with radiologists’ workflows than models not trained on mammography data. In the reported comparisons, M-Mammo-CLIP Dissect encoded more and more relevant mammography-specific concepts than G-Mammo-CLIP Dissect. The mean neuron-concept similarity $F_{\text{target}}$1 increased after mammography pretraining, and coverage analysis showed that the mammography-pretrained model captured more unique mammography-related concepts than non-mammography concepts (Salahuddin et al., 25 Sep 2025).

The category distribution of discovered concepts is also notable. For M-Mammo-CLIP Dissect, the top categories were Findings and Characterizations and Interpretations, both identified as central to radiologist workflow. By contrast, G-Mammo-CLIP Dissect was described as less focused and sometimes defaulted to non-mammography or environmental concepts. The paper further reports example neurons with strong selectivity for clinically meaningful features, including an implant-selective neuron and a calcification-selective neuron, which supports the claim that neuron-level representations can be interpreted in radiologically meaningful terms (Salahuddin et al., 25 Sep 2025).

These findings resonate with the broader trend that mammography-specific pretraining improves semantic relevance. In panoptic segmentation, domain-bridging with MAM-E and BiomedCLIP was introduced specifically to reduce the natural–medical domain gap and provide modality- and anatomy-specific features for lesion delineation (Zhao et al., 2024). In classification, MAM-CLIP showed that 2,313 image-caption pairs from mammography atlases improved 3-class average F1 by +1% to +14%, with the largest gains in low-label settings, suggesting that mammography-specific textual supervision can materially shape the learned visual representation (Gulluk et al., 19 May 2026). Mammo-CLIP Dissect adds an interpretability layer to this picture by showing that the resulting internal concepts are not merely more predictive, but more clinically aligned.

5. Fine-tuning, specialization, and representational trade-offs

A central result of Mammo-CLIP Dissect is that downstream fine-tuning increases alignment for target-task concepts but may reduce conceptual breadth. In C-Mammo-CLIP Dissect, fine-tuning for specific classification tasks increased alignment $F_{\text{target}}$2, particularly in deeper layers. For calcification-tuned models, the number of captured calcification morphology concepts increased, which the paper interprets as improved task-specific concept specialization (Salahuddin et al., 25 Sep 2025).

The same analysis revealed a trade-off. Calcification fine-tuned models focused more on benign calcification types while losing some coverage of broader density-related concepts and some malignant calcification subtypes. Density fine-tuned models had fewer density concepts in deeper layers than the base M-Mammo-CLIP Dissect. The paper interprets this as specialization improving detection of some findings while potentially reducing generalization or breadth (Salahuddin et al., 25 Sep 2025).

This specialization–generalization tension is echoed by complementary interpretability work. Mammo-SAE, which trains a sparse autoencoder on patch-level Mammo-CLIP features, reports that fine-tuning improves separation between class-mean activations in the latent space and makes certain neurons more distinctly activated for pathology, while the pretrained model shows less class separation and more activity on confounders or background regions (Nakka, 21 Jul 2025). Taken together, these results suggest that task adaptation sharpens certain clinically useful concepts but may also narrow the representational vocabulary available to the model.

A plausible implication is that concept coverage could become a model-selection criterion alongside AUC or F1 in mammography systems, particularly where holistic assessment is clinically important. The reported trade-off indicates that optimization for a single endpoint may not preserve the full spectrum of mammographic reasoning (Salahuddin et al., 25 Sep 2025).

6. Coverage gaps, confounding, and limitations

Mammo-CLIP Dissect does not conclude that mammography pretraining solves concept coverage. Even after mammography-specific pretraining, the framework identified 54 unique missed mammography concepts. The underrepresented set included less visually distinct findings such as architectural distortion and concepts rare or absent in mammographic images, including modality-inappropriate terms such as peri-implant fluid. Category-level analysis showed that more concepts were covered than missed in the main mammography categories, but the remaining gaps were framed as diagnostic blind spots (Salahuddin et al., 25 Sep 2025).

The framework also highlights textual ambiguity and overlap. The paper notes that concept pairs such as “amorphous calcification” and “amorphous” can be difficult to disentangle at the neuron level, especially when only single images are provided rather than temporal or multi-view information. Another reported issue is single-keyword bias, where neurons may latch on to a keyword like “calcification” without capturing the full clinical subtype structure (Salahuddin et al., 25 Sep 2025).

Related work supports the importance of separating clinically meaningful representation from confounding. Mammo-SAE identified a neuron, 15699, that was activated in both classes and tied to background areas, acting as a potential confounder; it also found that some background-related neurons remained strongly active even after fine-tuning (Nakka, 21 Jul 2025). In the mammography segmentation literature, annotation quality, imprecise masks, limited labeled data, and annotation protocol heterogeneity were explicitly noted as factors constraining performance and interpretability (Zhao et al., 2024). These observations suggest that concept-level dissection should be read not only as a semantic probe of models, but also as a probe of dataset structure and supervision quality.

7. Position within mammography VLM research and clinical significance

Mammo-CLIP Dissect is part of a broader shift from generic vision backbones toward mammography-specific foundation models that encode language, multi-view structure, and high-resolution semantics. Mammo-CLIP for multi-view case classification introduced early feature fusion over the four standard mammographic views and plug-and-play adapters that limit updates to about 1% of parameters during fine-tuning, achieving AUC 0.841 on internal evaluation and 0.837 on an external dataset (Chen et al., 2024). The paired-report Mammo-CLIP foundation model showed strong performance in classifying and localizing mammographic attributes, alongside Mammo-FActOR for sentence-level spatial attribution (Ghosh et al., 2024). MaMA extended full CLIP-style pretraining to mammography with multi-view supervision, a symmetric local alignment module, and LoRA-based parameter-efficient language adaptation, outperforming baselines on EMBED and RSNA-Mammo with only 52% model size compared with the largest baseline (Du et al., 2024). MAM-CLIP and MV-MLM further showed that atlas captions and synthetic reports can serve as effective textual supervision for BI-RADS classification, diagnosis, and risk prediction (Gulluk et al., 19 May 2026, Zheng et al., 30 Oct 2025).

Within this ecosystem, Mammo-CLIP Dissect provides an interpretability layer that is concept-centric rather than prediction-centric. Its stated advantages over pixel-level methods are alignment with radiologist reasoning, neuron-level granularity, identification of failure points, and flexibility across architectures, data regimes, and tasks (Salahuddin et al., 25 Sep 2025). The framework therefore addresses a distinct clinical question: not only whether a mammography model is accurate, but whether its learned internal concepts resemble the taxonomy and workflow used by radiologists.

This suggests a broader research program in which mammography foundation models are evaluated along three axes simultaneously: predictive performance, data efficiency and robustness, and concept coverage or alignment. The Mammo-CLIP Dissect results indicate that domain-specific training and task-specific adaptation materially shape concept learning, and that explainability in mammography can be framed as a structured comparison between learned neuronal concepts and clinical knowledge rather than as a purely visual localization problem (Salahuddin et al., 25 Sep 2025).