- The paper introduces a novel multimodal CBM that enforces transparency through dual concept bottleneck layers for image and text modalities.
- It leverages automatic concept generation from LLMs and open-vocabulary detection to create a shared, interpretable concept space.
- Empirical results show up to 10% higher accuracy on ImageNet and 20x sparser activations, highlighting the model's performance and interpretability.
Multimodal Concept Bottleneck Models: Architectures and Implications
Motivation and Context
Interpretability remains a central concern in deploying deep learning models, particularly in domains where transparency, auditability, and human alignment are necessary. Concept Bottleneck Models (CBMs) represent an important class of intrinsically interpretable systems, enforcing prediction via human-aligned concept layers. Despite their theoretical promise, prior CBMs suffer from restricted generalization capabilities—relying on fixed, predefined class sets—and risk information leakage through residual non-concept signals. The presented Multimodal Concept Bottleneck Model (MM-CBM) addresses these issues by extending CBMs into the multimodal domain, fundamentally rearchitecting the bottleneck mechanism to enforce both semantic alignment and modality flexibility.
Methodological Advances
The MM-CBM introduces dual Concept Bottleneck Layers (CBLs) spanning both image and text modalities. Both the image and textual inputs are independently projected into a shared concept space, enabling predictive reasoning grounded solely in interpretable concept activations. Unlike traditional CBMs, which route predictions through a final linear classifier susceptible to leakage, MM-CBM computes similarity between concept embeddings from both modalities and directly utilizes these scores for inference. This approach explicitly precludes reliance on non-conceptual cues.
Concept sets are generated automatically through LLM queries, greatly accelerating scalable annotation. Concept activation data is constructed by leveraging open-vocabulary object detection (OWLv2) and semantic similarity scoring (all-mpnet-base-v2) to supervise CBL training. Interpretability is enforced via distinct strategies: binary cross-entropy for image concepts and negative cosine similarity for textual concept associations.
The final objective integrates interpretability alignment and classification loss, with control provided via a trade-off parameter. The non-negativity constraint—applied via ReLU—removes ambiguity in activation interpretation and increases sparsity, thereby reinforcing faithfulness in prediction logic.
Empirical Evaluation
MM-CBM is evaluated across seven datasets, including general classification (CIFAR-10, CIFAR-100, ImageNet), fine-grained recognition (Food-101, CUB, Oxford-IIIT Pets), and texture categorization (DTD). Compared to CBM baselines (LF-CBM, LaBo, LM4CV, VLG-CBM) and the CLIP backbone, MM-CBM consistently achieves strong numerical results:
- On ImageNet, MM-CBM achieves up to 10% higher accuracy over non-VLM-guided CBMs and maintains performance within 5% of black-box CLIP in both zero-shot and fine-tuned scenarios.
- Accuracy improvements up to 51.26% on average are observed across four benchmarks relative to prior CBMs, indicating substantial gains in concept-grounded generalization.
- Increased interpretability is achieved without sacrificing accuracy: MM-CBM yields 20x sparser visual activations and 5x higher alignment scores relative to sigmoid-based methods, with no reduction in performance.
The ablation studies confirm the efficacy of non-negative activation and top-N concept selection for interpretability and task faithfulness.
Interpretability Results
Extensive comparison to leading vision-LLMs (VLMs) such as LLaVA and Llama-3.2 demonstrates MM-CBM's superior capacity for producing concept-centric explanations. MM-CBM explanations are preferred in 88.7% and 65.8% of cases respectively by VLM judges, underscoring its advantage in fine-grained visual concept coverage. Furthermore, MM-CBM achieves explanations and image retrieval with orders-of-magnitude higher efficiency compared to VLMs prompted for detailed responses.
MM-CBM supports flexible, open-vocabulary queries, including hybrid, out-of-distribution, and polysemous/abstract prompts. Retrieval experiments demonstrate that MM-CBM generalizes semantic alignment beyond fixed label sets and is robust to contextual variations, highlighting its practical applicability for real-world interpretability tasks.
Theoretical and Practical Implications
The MM-CBM architecture fundamentally enables interpretable, multimodal reasoning with arbitrary input and output vocabularies. The alignment enforced between modalities opens pathways for transparent zero-shot classification, controllable image retrieval, and detailed model auditing. By decoupling prediction from the classifier head and relying exclusively on concept activations, MM-CBM precludes information leakage, thus maintaining faithfulness—a critical requirement for safety-sensitive deployments.
Automated concept generation and flexible supervision strategies facilitate scalable adaptation to new domains and datasets, including unsupervised settings via knowledge distillation. The ablation results clarify the utility of sparsity, non-negativity, and NEC constraints in sustaining transparency and robustness.
Future Directions
This framework provides groundwork for further exploration in multimodal interpretability. Potential avenues include:
- Extension to more granular, context-dependent concept sets leveraging advanced LLMs.
- Integration with prompt-based generative models for concept-driven synthesis and interpolation.
- Fine-tuning strategies to enhance domain adaptation under distribution shifts, especially for datasets with concept attribution bottlenecks.
- Incorporation of human-in-the-loop interventions for real-time decision editing and auditing.
- Formal analysis of concept space topology and information-theoretic properties of bottleneck layers.
Conclusion
MM-CBM advances the interpretability frontier in multimodal learning by introducing dual concept bottleneck layers and enforcing end-to-end transparency. Empirical results substantiate both its high task performance and interpretability, demonstrating utility for zero-shot vision-language tasks and scalable deployment. The architecture's flexibility and faithfulness position it as a viable paradigm for interpretable deep learning in domains demanding semantic transparency and rigorous model auditing (2606.19882).