PCBM-ReD: Post-hoc Concept Bottleneck

• The paper introduces PCBM-ReD, a framework that retrofits pretrained vision models with interpretable, independent concept representations using sparse autoencoding and CLIP-based alignment.
• It employs a multi-stage pipeline with automatic sparse concept extraction, MLLM-guided labeling, and reconstruction-guided filtering to ensure high accuracy and faithful interpretability.
• Empirical results demonstrate state-of-the-art performance, improved few-shot retention, and enhanced causal faithfulness compared to existing concept bottleneck methods.

Post-hoc Concept Bottleneck Model via Representation Decomposition (PCBM-ReD) is a principled framework for retrofitting concept-based interpretability onto pretrained, opaque vision and generative models. It addresses limitations of both ante-hoc and prior post-hoc concept bottleneck models, ensuring not only high accuracy but also human-interpretable, data-driven concept representations with rigorous independence and faithfulness properties (Gong et al., 18 Jan 2026, Kulkarni et al., 11 Dec 2025, Kulkarni et al., 25 Mar 2025, Shang et al., 2024).

1. Problem Setting and Rationale

Deep neural networks for vision are high-performing yet opaque, complicating their use in critical domains where interpretability and error intervention are required. Concept Bottleneck Models (CBMs) offer interpretability by forcing classification through an intermediate layer of human-understandable concept activations $C = \{c_1, \dots, c_{N_C}\}$. Classic CBMs require manual design and annotation of concepts, often yielding labor-intensive, incomplete, or non-visual bottlenecks. Post-hoc methods try to mine latent features from trained models but typically result in polysemantic or uninterpretable units, redundancy, and suboptimal faithfulness to the model's internal reasoning. Independence among extracted concepts, crucial for robust interventions and error analysis, is rarely addressed in prior frameworks (Gong et al., 18 Jan 2026).

PCBM-ReD systematically overcomes these issues by decomposing the representations of a frozen neural encoder through a structured pipeline:

• Automatic unsupervised concept extraction via sparse autoencoding
• Multimodal LLM (MLLM) labeling and scoring for filtering out low-value and non-visual units
• Independence-enforced concept selection guided by minimum reconstruction loss
• Explicit decomposition of the encoder’s latent vectors as sparse, linear combinations of concept embeddings, utilizing CLIP’s visual–textual alignment.

2. Methodology and Pipeline

The PCBM-ReD pipeline consists of the following major steps (Gong et al., 18 Jan 2026):

Automatic Sparse Concept Extraction

Given a frozen image encoder $I$, typically CLIP, each image $x_i$ is mapped to a $d$-dimensional embedding $I_i = I(x_i) \in \mathbb{R}^d$. The encoder’s latent space is assumed to be reconstructable as a sparse linear combination of $k$ dictionary atoms, $V = [v_1,...,v_k]\in\mathbb{R}^{d\times k}$:

$I_i \approx V u_i, \quad u_i = \psi(I_i)$

with $\psi$ a small encoder enforcing sparsity on $u_i$. The dictionary atoms $v_j$ are interpreted as “proto-concepts” whose activations are measured by $u_{i,j}$. The sparse autoencoder is trained by minimizing:

$$\min_{V, \psi} \sum_{i=1}^N \|I_i - V\psi(I_i)\|_2^2 + \lambda\|\psi(I_i)\|_1$$

MLLM-Guided Concept Labeling and Filtering

Each dictionary atom is associated with a set of top-$K$ activating images. For these, a multimodal LLM (e.g., LLaMA-Vision) is prompted in a two-stage chain-of-thought process:

1. Describe visual features supporting the image’s class.
2. Summarize the features in one concise sentence (without class labels).

The resulting per-image descriptions are aggregated and the MLLM proposes candidate names (e.g., “striped pattern”). The same model rates each candidate on visual identifiability, discriminative power, and absence of spurious shortcuts (scale: 1–10); only concepts with score $\geq6$ are retained.

Independence and Task Relevance via Reconstruction-Guided Selection

To remove redundancy, PCBM-ReD selects a subset $C\subset C_0$ of $m$ concepts (from $C_0$, the filtered pool) that best reconstructs the original embedding space, i.e.:

$$\min_{C\subset C_0, |C|=m} \sum_{i=1}^N \min_{\beta_i\in \mathbb{R}^m} \|I_i - R(C)^\top \beta_i\|_2^2$$

where $R(C)$ is a stack of CLIP text embeddings of the selected concepts. The actual discrete selection uses a greedy algorithm that ensures new candidates are linearly independent from the existing set, discarding those whose embeddings fall into the prior span.

Representation Decomposition with CLIP Alignment

With the independent, filtered concept set $\{c_j \equiv T(c_j)\}$ (CLIP text-embedding for each concept), each image embedding $I_i$ is decomposed as:

$$I_i = \hat{I}_i + \varepsilon_i = \sum_{j=1}^m w_{i,j} c_j + \varepsilon_i,\quad \text{with } \|w_i\|_0 \leq s \ll m$$

Weights $w_{i,j}$ are solved using Orthogonal Matching Pursuit to enforce sparsity, yielding a fitted representation $\hat{I}_i$. The residual $\varepsilon_i$ is discarded.

Bottlenecked Model Training

A linear classifier is attached to $\hat{I}_i$, trained with cross-entropy loss. Initialization of classifier weights uses CLIP text embeddings for classes, so that zero-shot priors are maintained. Only the top linear layer is trained; the encoder, dictionary, and decomposition remain frozen.

3. Extensions: Residual, Sparse, and Generative CBMs

Several extensions generalize the PCBM-ReD methodology:

• Incremental Residual CBMs: Unexplained variance in the encoder space is modeled by a small set of optimized residual vectors, which are then incrementally converted into interpretable concepts via similarity-to-candidate-concept losses (Shang et al., 2024). This improves concept completeness and descriptive efficiency, measured by Concept Utilization Efficiency (CUE).
• Concept Bottleneck Sparse Autoencoders (CB-SAE): Starting from a sparse autoencoder, low-utility (i.e., low interpretability or steerability) neurons are pruned and replaced with a lightweight, supervised bottleneck aligned to a curated human concept set (Kulkarni et al., 11 Dec 2025). Concept and steerability alignment is enforced via CLIP-based metrics and cyclic similarity losses.
• Generative Model Interpretation: PCBM-ReD has been adapted for post-hoc concept bottlenecking of GAN and diffusion model generators (Kulkarni et al., 25 Mar 2025). In this setting, the latent code is decomposed into logits for user-interpretable concepts and residual factors, enabling controllable, interpretable generative modeling.

4. Empirical Performance and Interpretability

PCBM-ReD demonstrates:

• State-of-the-art accuracy: On 11 benchmarks (ImageNet, CIFAR, Food-101, FGVC-Aircraft, etc.), the fully supervised mode attains 86.97% average accuracy versus CLIP linear probe’s 87.38%, and surpasses all prior CBM baselines (e.g., LaBo at 85.72%, Res-CBM at 83.39%) (Gong et al., 18 Jan 2026).
• Zero/few-shot retention: The CLIP-decomposed bottleneck maintains zero-shot performance close to vanilla CLIP (69.73% vs 69.69%). In few-shot settings (1–16 per class), PCBM-ReD outperforms LaBo by +5.01% across shots.
• Interpretability and causal faithfulness: Human studies confirm higher visual identifiability, faithful description, and causal links between concepts and predictions (all $p<0.05$ versus LaBo) (Gong et al., 18 Jan 2026). Comparable or improved interpretability and steerability metrics (+32.1% and +14.5% over SAEs for vision–LLMs) are reported in large-scale evaluations (Kulkarni et al., 11 Dec 2025). In generative models, PCBM-ReD improves steerability by 25–42 pp over previous methods (Kulkarni et al., 25 Mar 2025).

5. Advantages, Limitations, and Future Directions

Key Innovations and Advantages

• Data-driven concept discovery: Concepts are discovered in accordance with both the data distribution and the encoder’s inductive biases, not restricted by hand-engineered sets.
• Independence and task-relevance: Reconstruction-guided optimization yields a minimal, independent set of concepts, avoiding redundancy and ensuring high intervention fidelity.
• CLIP-alignment and annotation-free scoring: No additional label collection is required since CLIP and MLLMs provide semantic alignment and automatic scoring; this also ensures adaptation to transfer and zero/few-shot tasks.

Limitations

• Reliance on generalist MLLMs can reduce reliability in domain-specialized settings (e.g., medical imagery).
• Overall performance is limited by the expressivity of the frozen encoder; improvements may require jointly optimizing the backbone.
• Sequential filtering and recursive greedy selection introduce modest extra computational cost (Gong et al., 18 Jan 2026, Shang et al., 2024).

Future Research

Potential directions include:

• Incorporating domain-specialized LLMs for improved visual concept labeling.
• More adaptive prompt engineering for robust MLLM scoring.
• Integrating concept discovery tightly with encoder fine-tuning and exploring compositional/hierarchical CBMs (Gong et al., 18 Jan 2026, Shang et al., 2024).

6. Related Models and Comparative Summary

Model	Approach	CBM Integration	Concept Acquisition	Reported Gains
PCBM-ReD (Gong et al., 18 Jan 2026)	Sparse autoencoding + MLLM + CLIP	Post-hoc, fully modular	Unsupervised + MLLM naming	SOTA accuracy/interpretability
CB-SAE (Kulkarni et al., 11 Dec 2025)	Sparse AE + pruning + CBM	Post-hoc, augmented	Pruned+supervised concepts	+32.1% interpretability
Res-CBM (Shang et al., 2024)	Residual vector discovery	Post-hoc, modular	Incremental, from data	High CUE, strong few-shot
CB-AE, CC (Kulkarni et al., 25 Mar 2025)	AE or controller on generator	Post-hoc on generator	Minimal supervision	4–15× faster, +25–42pp steer.

All methods leverage fixed encoders/backbones, decompose feature spaces post-hoc into interpretable and independent concept spaces, and validate both accuracy and interpretability gains through extensive benchmarks and human evaluation.