BMC-CLIP: Interpretable Vision-Language Bottlenecks

• BMC-CLIP is a framework that integrates CLIP with human-interpretable concept bottlenecks to enable transparent image classification and fMRI-based brain decoding.
• It employs a lightweight adapter and dual-branch fusion strategy to align both high-level semantics and detailed perceptual cues, ensuring robust performance.
• The framework leverages rigorous statistical concept selection and composite loss functions to balance interpretability, biological plausibility, and diagnostic accuracy.

The BMC-CLIP framework refers to a class of models that bridge the representational power of foundation vision-LLMs, particularly CLIP, with human-interpretable or neurobiologically-informed bottlenecks. Two distinct paradigms have emerged under this term: the Concept Bottleneck integration with CLIP for explainable classification (Chowdhury et al., 2024) and the multi-layer CLIP fusion approach designed for fMRI-based brain image decoding (Xia et al., 22 Oct 2025). Both approaches are characterized by architectural innovations that prioritize interpretability and/or biological plausibility, while effectively leveraging the zero-shot power and rich feature abstractions of CLIP.

1. Architectural Foundations

1.1. CLIP-Based Concept Bottleneck Model (CBM)

The BMC-CLIP architecture for explainable image classification comprises three main components:

• Frozen CLIP Image Encoder ($f_{\text{img}}$): Maps images to high-dimensional embeddings $x \in \mathbb{R}^d$.
• Adapter Module $F(\cdot)$: A learnable, low-capacity neural network (typically 1–2 linear layers with LeakyReLU activation), transforming $x$ to $x'=F(x)$ to facilitate domain adaptation without fine-tuning the backbone.
• Concept Bottleneck Layer: Encodes a set of $K$ concept vectors $\mathcal{T} = \{t_1, ..., t_K\}$ (CLIP text embeddings), computes their cosine similarity to $x'$, and feeds them, through a masked and weighted linear transformation, to the class-prediction head.

The logit for class $i$ is computed as: $$z_i = \sum_{j=1}^K (M \odot V)_{ji} \cdot (x' \cdot t_j + \alpha'_j) + \beta_i$$ with $x \in \mathbb{R}^d$0 as the concept selection mask, $x \in \mathbb{R}^d$1 as learnable scalars, and $x \in \mathbb{R}^d$2 as learnable biases.

1.2. Multi-Layer CLIP Fusion for Brain Decoding

In the brain decoding domain, BMC-CLIP (specifically "BrainMCLIP") introduces a dual-branch network:

• fMRI-Semantic Inputs ($x \in \mathbb{R}^d$3): Voxels from high-level visual regions are mapped to both CLIP’s final text ($x \in \mathbb{R}^d$4) and visual ($x \in \mathbb{R}^d$5) embeddings.
• fMRI-Detail Inputs ($x \in \mathbb{R}^d$6): Voxels from lower-level visual areas are mapped to an averaged set of intermediate CLIP visual layers ($x \in \mathbb{R}^d$7).
• Multi-Path Encoders, Decoders, and Fusion: Each stream uses dedicated encoders and MLP backbones to predict semantic and detail embeddings, which are then fused: $x \in \mathbb{R}^d$8 The mapping is regularized by decodability constraints on $x \in \mathbb{R}^d$9 and $F(\cdot)$0.

2. Bottleneck Mechanism and Interpretability

The BMC-CLIP model, in its concept bottleneck instantiation, enforces interpretability by restricting each class’s prediction to a subset of preselected concepts:

• Concept Selection: Concepts are chosen for each class based on a Welch’s t-statistic for discriminability, and filtered for low inter-correlation ($F(\cdot)$1).
• Linear Decomposition: The model’s class logits are a sum of per-concept activations and weights, retaining the canonical CBM decomposition aligned to clinical attributes.
• Intervention and Explainability: The mask $F(\cdot)$2 allows clinical users to directly inspect or ablate contributions of individual concepts, providing granular transparency (Chowdhury et al., 2024).

In the fMRI domain, interpretability is tied to anatomical plausibility; each encoding branch mirrors the hierarchical and functionally specialized organization of the visual cortex.

3. Explicit Domain Alignment and Adaptation

3.1. Adapter Placement and Design

In explainable diagnosis, the addition of a light-weight adapter $F(\cdot)$3 after the image encoder is found to be both necessary and sufficient to reconcile domain discrepancies—without sacrificing interpretability or risking overfitting:

• 1–2 Linear Layers + LeakyReLU: Empirical results demonstrate more layers induce overfitting, while fewer are insufficient for adaptation.
• Frozen CLIP Backbone: Only adapter parameters and linear bottleneck weights are learned, preserving the pre-trained alignments and avoiding catastrophic forgetting (Chowdhury et al., 2024).

3.2. Multi-Layer Mapping in fMRI

For brain decoding, BMC-CLIP's semantic (final layer) and detail (intermediate layer average) mapping is guided by the correspondence between brain visual regions and CLIP's layerwise abstractions—a direct encoding of the brain’s functional hierarchy into model design.

4. Losses and Optimization

4.1. Multi-Granularity Alignment in BrainMCLIP

BrainMCLIP employs a composite loss designed to robustly align predicted and target CLIP embeddings at both global (CKA) and fine-grained (token-wisecosine similarity) scales: $F(\cdot)$4 with

$F(\cdot)$5

$F(\cdot)$6

This loss is applied jointly with cross-reconstruction and branch-specific MSE terms (Xia et al., 22 Oct 2025).

4.2. Classification and Geometry

In BMC-CLIP for diagnosis, the cross-entropy loss is used for classification after the concept bottleneck, and ablations confirm the key discriminative role of the cosine term over vector norms (Chowdhury et al., 2024).

5. Empirical Evaluation and Parameter Efficiency

5.1. Diagnostic Classification

On medical-imaging datasets (HAM, DR, BCCD), AdaCBM consistently matches or surpasses the performance of label-free or post-hoc CBMs and matches linear classifiers while preserving interpretability. Key findings:

• Performance robust to concept source (doctor-written vs. GPT-4 generated).
• CLIP+adapter+CBM converges faster and more stably than post-hoc fine-tuning approaches.
• Adapter after image encoder is optimal; greater network depth degrades performance.

Sample results table (accuracy, mean ± std):

Dataset	Linear CLS	CBM	AdaCBM (k=10)
HAM	80.9%	78.9%	82.8%
BCCD	74.5%	63.7%	74.1%
DR	78.0%	75.7%	78.3%

5.2. Brain Image Decoding Performance

BrainMCLIP achieves state-of-the-art results on both detail-sensitive (PixCorr, SSIM) and high-level semantic (CLIP-similarity, Inception) metrics, rivaling VAE-based architectures. Notably, it slashes parameter count by 71.7% compared to leading VAE pipelines:

Method	Params	PixCorr	SSIM	CLIP	Incep
MindEye₂ (VAE)	2.58B	0.322	0.431	93.0%	95.4%
BrainMCLIP	0.73B	0.212	0.263	95.2%	94.6%

BrainMCLIP outperforms all CLIP-only approaches on semantic metrics, nearly matches VAE pipelines, and does so with a drastically smaller model (Xia et al., 22 Oct 2025).

6. Practical Considerations and Deployment

• Concept Engineering: Use statistical utility (Welch’s t-test) and low inter-correlation to select relevant, diverse concepts.
• Adapter: 1–2-layer linear with LeakyReLU; excessive depth induces overfitting.
• Training: Precompute and cache CLIP features; train only adapter and CBM weights.
• Inference: CLIP encode → adapter → per-concept dot-product → linear CBM; no need for augmentation or ensemble inference.
• Interpretability: Provide per-concept class contributions for transparency; clinicians can selectively mask concepts at test-time to assess model behavior.
• fMRI Decoding: Train fusion and mapping networks per subject; decoded embeddings are rendered into images with a frozen Versatile Diffusion model.

7. Contextual Significance and Implications

The BMC-CLIP framework exemplifies the synthesis of foundation models with explicit bottlenecks for interpretable AI and brain-aligned decoding. In explainable medical diagnosis, this approach closes the gap between black-box accuracy and full transparency. In brain decoding, the alignment of CLIP’s layerwise features to the brain's visual hierarchy yields parameter-efficient models that retain both semantic and perceptual fidelity. These developments signal a broader movement toward integrating architectural priors (e.g., interpretability, biological plausibility) into large-scale neural systems without forfeiting the generalization power of modern foundation models (Xia et al., 22 Oct 2025, Chowdhury et al., 2024).

A plausible implication is that such frameworks may enable both scientific discovery (by revealing concept-level structure in neural or medical representations) and practical deployment in high-stakes domains where accountability and domain adaptation are paramount.