Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks

Abstract: Concept Bottleneck Models (CBMs) enhance interpretability by projecting learned features into a human-understandable concept space. Recent approaches leverage vision-LLMs to generate concept embeddings, reducing the need for manual concept annotations. However, these models suffer from a critical limitation: as the number of concepts approaches the embedding dimension, information leakage increases, enabling the model to exploit spurious or semantically irrelevant correlations and undermining interpretability. In this work, we propose Concept Flow Models (CFMs), which replace the flat bottleneck with a hierarchical, concept-driven decision tree. Each internal node in the hierarchy focuses on a localized subset of discriminative concepts, progressively narrowing the prediction scope. Our framework constructs decision hierarchies from visual embeddings, distributes semantic concepts at each hierarchy level, and trains differentiable concept weights through probabilistic tree traversal. Extensive experiments on diverse benchmarks demonstrate that CFMs match the predictive performance of flat CBMs, while substantially mitigating information leakage by reducing effective concept usage. Furthermore, CFMs yield stepwise decision flows that enable transparent and auditable model reasoning with hierarchical class structures.

• The paper introduces Concept Flow Models (CFMs) that enforce hierarchical sparsity, structurally mitigating information leakage in concept-based models.
• It employs CLIP-based hierarchies and LLM-guided local concept selection to create node-specific pools, ensuring semantic alignment and reducing effective concept count.
• Empirical results demonstrate that CFMs achieve competitive accuracy with far fewer active concepts per prediction, making the decision process more interpretable.

Concept Flow Models: Hierarchical Bottlenecks for Semantically Grounded Reasoning in Concept Bottleneck Architectures

Introduction and Motivation

Concept Bottleneck Models (CBMs) have emerged as a model class designed to enhance interpretability in deep classification tasks by introducing a human-interpretable concept space between learned features and final predictions. Classical CBMs rely on explicit, manually annotated concepts as intermediate variables, allowing post hoc explanations and even human intervention. Recent shifts towards post-hoc and label-free variants utilize pre-trained vision-LLMs (VLMs), such as CLIP, to generate or embed visual concepts, partially alleviating manual annotation burdens. However, these models remain fundamentally limited by information leakage: when the number of concepts approaches the feature embedding dimension, models can use large pools of semantically irrelevant or random concepts to reconstruct target labels. This phenomenon undermines the core interpretability goal, as models may deviate from human-aligned reasoning and instead exploit spurious statistical associations embedded in high-dimensional spaces.

The "Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks" (2606.19489) systematically addresses information leakage by structurally enforcing hierarchical sparsity in concept usage through a tree-structured bottleneck, termed the Concept Flow Model (CFM). Instead of projecting feature embeddings through all concepts in a flat layer, CFMs partition the concept space and route predictions through localized, node-specific subsets, enforcing path-specific reasoning and increasing reliance on task-meaningful concepts while severely limiting the capacity to exploit spurious or random features.

Figure 1: Architectural comparison of original CBMs (left), post-hoc CBMs (center), and Concept Flow Models (right); in CFMs, only path-specific concepts are used for any prediction, limiting leakage pathways.

Methodology

CFMs are instantiated as follows:

1. Hierarchical Tree Extraction: Utilizing class-level visual embeddings extracted via CLIP, agglomerative clustering builds a hierarchy among classes. Node depth is controlled and low-separation nodes are pruned, ensuring high inter-node separation per subtree, and internal nodes are labeled via LLM prompts over their descendant leaves.
2. Local Concept Selection and Allocation: For each internal node, LLM-guided concept pools are compiled by aggregating class-specific candidate concepts. Concepts containing class names or redundant expressions are removed; final selections are made by Lasso-regularized multi-class logistic regression for local discriminative capacity. Subsets of concepts are assigned to internal nodes, parameterized to balance child count and average share, enforcing an allocation budget that curtails the presence of any concept to localized subtrees.
3. Flow-Based Differentiable Model: Each image is projected via the (frozen) CLIP encoder, and, at each internal node, local concept activations are linearly combined and softmaxed (per-learned node temperature) to yield transition probabilities to children. The traversal probability for each class is the product of path transition scores, augmented by a calibrating leaf bias.
4. Structural Sparsity and Training Objective: The model is trained (end-to-end for classifier weights/per-node temperature; CLIP backbone fixed) to maximize ground-truth path probability; only concepts on the ground-truth root-to-leaf decision path are involved in the prediction for a sample, structurally enforcing sparsity.

   Figure 2: CFM pipeline: (1) extracting class hierarchies, annotating nodes; (2) generating and allocating LLM-derived candidate concepts; (3) training the model to trace predictions along concept-weighted transition paths.

The critical theoretical results demonstrated are:

• For random, non-semantic concepts, a flat bottleneck recovers separability as soon as the number of projected concepts matches the embedding dimension ($R = \Omega(d)$), enabling leakage. In contrast, CFMs, by distributing concepts over $m$ internal nodes and restricting concept usage per node, require $R = \Omega(md)$ random concepts to retain expressivity—making leakage practically unattainable for any reasonable choice of $R$.
• For task-aligned semantic concepts, CFMs can match the accuracy of flat CBMs using only the compact path-specific subsets per prediction; the expected number of concepts contributing to a prediction is reduced from $R$ (flat) to at most $\ell r'$ (path length times per-node local subset), with $\ell$ logarithmic in the number of internal nodes and $r' \approx R/m$.
• The Semantic Improvement over Random concepts (SIR) metric is defined for quantitative evaluation: the relative accuracy gain realized by semantic (as opposed to random) concepts under fixed budgets, serving as a measure of semantic reliance and interpretability.

  Figure 3: Information leakage with random concepts. In CFMs, accuracy saturates much slower with increasing random concept count, and cannot reach CBM's performance unless random concept pool size increases multiplicatively with tree size; increasing tree depth (number of internal nodes) increases leakage resistance and sparsity.

Empirical Evaluation

Experimental Setup

Benchmarks include CIFAR-10/100, UCF-101, CUB-200, and TinyImageNet, covering a range of domain complexity and granularity. CFMs, CBMs (both PCBM and LaBo variants), and random-concept baselines are evaluated under standardized concept budgets, matching total number of concepts per method and dataset.

Scenario 1: Leakage Under Random Concepts

Experiments empirically validate that, with random, non-semantic concepts, CBMs maintain high accuracy once concept count reaches the embedding dimension, affirming vulnerability to information leakage. By contrast, CFMs are robust, with accuracy curves requiring a multiplicative increase in the total number of random concepts to approach flat model accuracy. Pruning the tree (reducing internal nodes) relaxes this isolation and increases leakage, while per-node sparsity controls in CBM are less effective.

Scenario 2: Semantic Concept Utility

With curated, semantically meaningful concepts, CFMs match or surpass the accuracy of CBM baselines (PCBM and Labo), but require substantially fewer effective concepts per prediction (7 vs. 57 on CIFAR-10, 92 vs. >360 on TinyImageNet). The SIR metric is consistently higher for CFMs (e.g., 19.78 vs. 1.94 for Labo), denoting greater semantic grounding and interpretability. Notably, CFM's performance under random concepts remains suppressed, further corroborating leakage resistance.

Hierarchy and Selection Ablations

Alternative hierarchy construction methods (random trees, clustering probe network weights) are clearly outperformed by CLIP-induced hierarchies, especially for datasets with semantically consistent visual groupings. Additionally, the Lasso-based concept selection per node outpaces random, similarity-maximization, orthogonality, and submodular heuristics, achieving the highest accuracy with compact node-local pools.

Figure 4: Increasing hierarchy depth through tree pruning improves interpretability (further sparsifies possible paths) at the cost of reduced accuracy due to limited concepts per node; performance-interpretability tradeoff is tightly tunable via pruning.

Interpretability and Decision Path Visualization

CFMs enable stepwise, semantically grounded explanations: each prediction follows a unique path from root to leaf, with each intermediate decision annotated by the top-activated concepts at each node. This design makes model reasoning auditable—contrast with CBMs, which treat all concepts in a flat, global fashion, occasionally leveraging semantically irrelevant activations. The visualized CFM explanations highlight progressive class-space narrowing in decision-making, and errors tend to be associated with semantically unrelated or ambiguous node decisions, providing interpretable diagnostics.

Figure 5: Example CFM decision paths, highlighting ground-truth class path traversal and top-activated concepts at each decision point—node-wise sparsity enforces localized, human-auditable reasoning.

Theoretical and Practical Implications

CFMs represent a substantial evolution in interpretable modeling, demonstrating that structurally imposed sparsity via hierarchical bottlenecks is more effective and robust to information leakage than flat sparsity constraints or regularization, and that it preserves (and in many cases enhances) semantic alignment and human auditability.

Practical impact: The stepwise, path-localized use of concepts enables traceable decisions and systematic audit trails, desirable in safety-critical or regulatory domains. The model's resistance to spurious correlations makes it particularly suitable for high-risk applications where robust explanations are required.

Theoretical implication: The analysis contributes precise, quantifiable conditions under which hierarchical models are provably resistant to the information leakage pathologies of flat CBMs, particularly when operating with limited or noisy concepts. The SIR metric provides a rigorous framework for benchmarking semantic utility, and the architecture is compatible with current and future advances in VLMs, LLM-driven concept discovery, and hierarchical annotation.

Limitations and Future Directions

Interpretability and reasoning quality in CFMs fundamentally depend on the semantic fidelity of VLM backbones and LLM concept generators. Domain mismatch or poor grounding may cause suboptimal explanation fidelity. The current approach assumes class hierarchies exist and a tree is an appropriate structure; extending to DAGs or dynamically learned, input-adaptive hierarchies could further minimize remaining limitations. As region-level or compositional grounding improves in vision-LLMs, CFM variants leveraging local or compositional concepts may further increase both performance and explanation granularity.

Conclusion

Concept Flow Models structurally enforce path-specific concept usage via hierarchical bottlenecks, effectively mitigate information leakage, and enhance interpretability compared to prior CBMs. Under comparable concept budgets they yield equivalent or superior accuracy, require dramatically fewer effective concepts per prediction, and provide path-explainable, human-auditable reasoning. Hierarchical mechanisms for conceptual sparsity generalize the CBM paradigm, providing a robust, scalable blueprint for interpretable modeling in high-dimensional, multi-class tasks.

Figure 6: Tradeoff between per-node concept sparsity and accuracy; CFM architectures allow for fine-grained control over interpretability-performance, with diminishing returns for concept counts above node-wise discriminative minima.

Figure 7: Qualitative illustration of CFM decision path for a UCF-101 "Bowling" sample, showing high-confidence, interpretable routing based on localized concept activations at each node.

---

Cite: "Concept Flow Models: Anchoring Concept-Based Reasoning with Hierarchical Bottlenecks" (2606.19489).