Mammo-SAE: Sparse Autoencoder in Breast Imaging

• Mammo-SAE is an interpretability framework using sparse autoencoders to extract mammogram patch embeddings and map them to clinical breast cancer concepts.
• It enforces an overcomplete latent space with ℓ1 regularization, enabling precise lesion localization and quantitative model performance assessment.
• The framework facilitates causal analysis by intervening on latent neuron activations to differentiate confounding factors from clinically relevant features.

Mammo-SAE is a sparse autoencoder-based interpretability framework developed for breast cancer concept learning and localization in mammogram imaging. Designed to interface with Mammo-CLIP—a vision–language foundation model pretrained on large-scale mammogram image–report pairs—Mammo-SAE provides interpretable, patch-level embeddings that reveal the internal concept structure learned by foundation models for high-stakes medical tasks such as mass and calcification detection. By enforcing a heavily overcomplete and sparse latent space, Mammo-SAE enables identification of monosemantic latent neurons whose top activations show strong alignment with lesion annotations and clinical features relevant to breast cancer diagnosis. The approach yields precise quantitative and qualitative insights into model decision-making, identifies confounding latent factors, and facilitates debugging and refinement through controlled latent interventions (Nakka, 21 Jul 2025).

1. Architecture and Data Flow

Mammo-SAE operates in conjunction with Mammo-CLIP, which utilizes an EfficientNet-B5 backbone pretrained on mammogram–report pairs. A mammogram is input to Mammo-CLIP, and local patch-level feature vectors $x_\ell^j \in \mathbb{R}^d$ are extracted from a chosen layer $\ell$ (typically the final convolutional layer, $d=2048$). Each feature vector corresponds to a spatial patch from the original image, resulting in a feature grid of size $H_\ell \times W_\ell$ with $N_\ell$ total patches. These feature vectors are then processed by the Mammo-SAE module, a two-layer sparse autoencoder comprising a linear encoder and decoder. After training, the SAE remains frozen, acting as a sparse bottleneck for interpretability analysis at the level of CLIP feature patches.

2. Encoder–Decoder Design

The encoder consists of a single linear projection $W_\mathrm{enc} \in \mathbb{R}^{d \times h}$ followed by a ReLU activation:

$$z = \phi(W_\mathrm{enc} \cdot x), \qquad \phi(\cdot) = \mathrm{ReLU}(\cdot)$$

where $d=2048$ and $h=8d=16\,384$. The decoder is also a single linear mapping $W_\mathrm{dec} \in \mathbb{R}^{h \times d}$ with no activation:

$\ell$0

This architecture transforms each $\ell$1-dimensional patch embedding into an $\ell$2-dimensional sparse code and reconstructs it back, providing a tractable pipeline for interpretability at the component and spatial level.

3. Training Objective and Sparsity Regularization

Mammo-SAE is trained to minimize both reconstruction loss and a sparsity-inducing penalty on latent activations. The objective is:

$\ell$3

where $\ell$4 is mean squared error, $\ell$5 is the $\ell$6 norm of latent activations, and $\ell$7 balances sparsity against reconstruction fidelity. Experimental grid search set $\ell$8, enforcing that ~95% of patches activate only a small fraction of neurons while maintaining less than 2% drop in area under the curve (AUC) for key clinical tasks. The latent dimension $\ell$9 is selected to promote monosemantic (concept-aligned) neurons, with sparsity via $d=2048$0 regularization:

$d=2048$1

4. Identification and Alignment of Top-Activated Neurons

To map latent neurons to clinical concepts, Mammo-SAE computes class-wise mean activations for each latent unit. For class $d=2048$2:

$d=2048$3

where $d=2048$4 is the training set for class $d=2048$5. Neurons are scored and ranked by $d=2048$6. The top-$d=2048$7 activated neurons for each concept class are selected and spatially visualized by generating heatmaps of their activations, thresholded at the 95th percentile to produce binary masks. Extracted bounding boxes from these masks are compared against ground-truth lesion annotations, enabling quantitative evaluation of concept localization through metrics such as mAP (mean average precision) at IoU = 0.25.

5. Quantitative and Qualitative Evaluation

Performance evaluation demonstrates that SAE-sparsified CLIP features retain high predictive signal. For suspicious calcification detection, AUC-ROC shifts only marginally after SAE projection and reconstruction: 0.951→0.933 for pretrained (drop 0.018), 0.978→0.979 for finetuned (no AUC drop). Mass detection behaves similarly. Ablation studies reveal that restricting representations to only the top-$d=2048$8 (with $d=2048$9) neurons for either class recovers over 98% of full-model AUC, whereas ablating these neurons causes a sharp performance decline, confirming reliance on a small feature subset.

Concept localization with mAP @ IoU = 0.25 over the ten highest-ranked positive neurons yields mean APs of ≈0.256 for suspicious calcification (finetuned, best individual neuron ≈0.278) and ≈0.187 for mass (finetuned). Visual analysis confirms that roughly 70% of top-10 calcification neurons are spatially focused within true lesion boxes. Mass neurons display weaker, more diffuse alignment.

Task	Pretrained AUC	SAE AUC	Mean mAP (finetuned)
Suspicious calcification	0.951	0.933	0.256
Mass	0.786	0.763	0.187

6. Confounding Latent Factors and Causal Analysis

Analysis of neuron activations uncovers latent confounders. Notably, neuron index 15699 displays high activity across both positive and negative classes, persistently highlighting anatomically irrelevant regions such as the chest wall or pectoral muscle. This neuron's steady activation post-finetuning indicates a spurious background feature that the model may exploit. Causal intervention—zeroing out this neuron along with top concept neurons—modifies downstream predictions, establishing its nontrivial contribution as a confounder within the learned representation.

7. Effects of Downstream Fine-Tuning

Fine-tuning the entire Mammo-CLIP backbone and its classification head on concept labels sharpens neuron–class separation in the SAE latent space. After re-training the SAE on new final-layer features, class-wise activation distributions become much more distinct, especially for suspicious calcification. For this task, neuron 13867 emerges as the principal positive indicator with spatial activations tightly co-localizing with lesion regions. For mass detection, increased specificity is observed in neurons (e.g., indices ~3450, 6789), but localization alignment is consistently weaker than for calcification.

A plausible implication is that fine-tuning aligns previously distributed latent features into more interpretable, class-specific units, while suppressing spurious background neurons. Ultimately, the combination of SAE-imposed sparsity and downstream fine-tuning results in a model whose decision logic is both highly localizable and tractable for diagnostic auditing.

Mammo-SAE thereby provides a microscope into the black box of breast imaging foundation models, supplying a foundation for clinician-facing interpretability, diagnosis support, and targeted model debugging. Causal interventions and visual evidence from sparse latent activations substantiate its claims for transparent, clinically relevant representation learning (Nakka, 21 Jul 2025).