---
title: Compositional Interpretability in AI Models
url: https://www.emergentmind.com/topics/compositional-interpretability
type: topic
---

# Compositional Interpretability in AI Models

Compositional interpretability denotes a family of interpretability objectives in which a model’s prediction, representation, or statistical effect is explained as a composition of simpler parts whose roles remain explicit under composition. In the recent literature, this idea appears in several technically distinct but structurally related forms: concept-based explanations that compose object-, attribute-, or prototype-level primitives; neuron- and circuit-level analyses that express behavior through logical forms or sparse subgraphs; neural-symbolic systems whose intermediate programs or trees are directly inspectable; and compositional data methods in which interpretability must respect the simplex, log-ratio structure, or aggregation geometry rather than unconstrained Euclidean coordinates [2501.06059][2406.18534][2605.08934][2509.05563].

## 1. Formal definitions and mathematical structure

A direct concept-based definition states that compositional interpretability is the property that a model’s decision can be expressed as a composition of simpler concept-level parts whose evidence can be traced to concrete regions in the input and matched to prototypical regions in training images [2501.06059]. In representation learning, the same idea is formalized by requiring additive composition of concept vectors: for concepts $c_i$ and $c_j$, a concept representation is compositional if $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$ for some positive weights, and a sample embedding is interpretable when it decomposes into one concept per attribute [2406.18534].

A more general formalization separates syntax from semantics. In the category-theoretic account, a compositional model is written as $M = (D, S, C, [[\cdot]])$, where $S$ is a monoidal category of structure, $D$ is a string diagram, $C$ is a semantics category, and $[[\cdot]]: S \to C$ is a monoidal representation functor [2605.08934]. A compositional interpretation is then a pair of mappings, $I_S: S \to H$ and $I_C: C \to H$, constrained by the commuting condition $I_S = I_C \circ [[\cdot]]$, so that structural explanations and behavioral explanations agree [2605.08934]. Closely related accounts define compositional models via symmetric monoidal categories, strong symmetric monoidal semantics functors, and natural transformations linking abstract and concrete interpretation; the point is that explanations become diagrammatic, compositional, and checkable rather than merely descriptive [2406.17583][1911.06602].

In compositional data analysis, the formal object of interpretation is different. A log-contrast regression writes coefficients under the sum-to-zero constraint $\mathbf{1}_p^\top \beta = 0$, and tree-guided compositional regression reparameterizes leaf effects by $\beta = A\gamma$, so that sparsity in $\gamma$ selects interpretable internal nodes [2605.15469]. A different line replaces log-ratio coordinates with a reduction map on the simplex, $y = Wx$, where $W$ is a nonnegative column-stochastic matrix; interpretability then lies in the fact that each reduced coordinate is a softened amalgamation of original parts, and the columns of $W$ are themselves compositional objects [2509.05563]. DeepCoDA adopts yet another formulation: it learns log-contrast bottleneck features $z_{ib}$ together with sample-specific weights $w_i$, yielding predictions of the form $y_i = w_i^\top z_i$, so the explanation is a personalized composition of log-contrasts rather than a single global coefficient vector [2006.01392].

## 2. Concept extraction, prototypes, and compositional representations

One influential representation-level formulation identifies two salient properties of compositional concept representations: cross-attribute orthogonality and intra-attribute non-orthogonality [2406.18534]. Compositional Concept Extraction (CCE) operationalizes these constraints by alternating between learning an attribute-specific subspace and discovering concepts within that subspace via spherical k-means clustering, then orthogonally rejecting that subspace before learning the next attribute. The resulting concepts are evaluated by mean average precision for predicting composed concepts from base concept scores, by a Compositionality Score measuring reconstruction from true per-attribute concepts, and by cosine alignment to ground-truth concept vectors. On controlled subsets, CCE reaches $1.000 \pm 0.000$ MAP on CLEVR and attains compositionality scores close to the ground truth, while also improving downstream classification on several datasets [2406.18534].

COMIX turns compositional interpretability into an ante hoc image-classification mechanism rather than a post hoc explanation layer [2501.06059]. It builds on B-cos networks, discovers class-defining features by mutual information, decomposes a test image into feature-level regions, retrieves nearest training exemplars for each active feature, and aggregates per-feature votes by majority vote. The explanation is therefore not an auxiliary attribution map but the actual decision path: each primitive in the test image is matched to a training primitive, and the final class is the composition of those matches. The paper states a sufficiency theorem under pseudo-label consistency and reports a C-insertion score of $0.7365$ on ImageNet versus $0.4949$ for the best post hoc baseline, a $48.82\%$ improvement, together with improved PQ sparsity across several datasets [2501.06059].

Prompt tuning and intrinsic concept extraction extend the same logic to other representational interfaces. IntCoOp decomposes prompts into instance-conditioned context tokens, an explicit attribute token, and the class token, with prompt form $p^{+}(\mathcal{I}) = [h_1,\ldots,h_M,\mathcal{A}(\mathcal{V}(\mathcal{I})), [cls]_c]$; attribute embeddings are supervised against BLIP-2-derived attribute text embeddings and regularized toward compositional templates such as “a photo of a [attribute] [class]” [2406.13683]. In a 16-shot setup it improves CoOp by $7.35\%$ in average harmonic mean across $10$ datasets. HyperExpress addresses “Compositional and Interpretable Intrinsic Concept Extraction” from a single image by combining hyperbolic contrastive learning, hyperbolic entailment learning, and Horosphere Projection so that object-level and attribute-level concepts are both hierarchically organized and linearly composable on a horospherical submanifold [2603.11795]. On UCEBench it reports SIMI $0.699$, SIMC $0.786$, ACC1 $0.504$, and ACC3 $0.736$, with higher ACC than ICE and AutoConcept [2603.11795].

## 3. Neuron, feature, and circuit-level compositional explanations

At the neuron level, compositional interpretability often means replacing single labels with structured logical descriptions. “Compositional Explanations of Neurons” defines a concept language over atomic predicates with operators And, Or, Not, and, in natural language inference, Neighbors [2006.14032]. A beam search over formulas up to length $N$ maximizes IoU between thresholded neuron masks and candidate logical forms. On the final convolutional layer of a Places365 ResNet-18, mean IoU rises from $0.059$ at $N=1$ to $0.099$ at $N=10$, a $68\%$ increase, and manual inspection of $128$ randomly sampled vision neurons found $69\%$ meaningful compositional abstractions and $31\%$ polysemantic neurons [2006.14032]. The same framework is used to localize shallow lexical heuristics in NLI and to generate “copy-paste” adversarial interventions that alter model predictions in predictable ways.

A closely related program has been adapted to deep reinforcement learning. “Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning” defines atomic concepts as binary predicates over states, composes them with $\wedge$, $\vee$, and $\neg$, binarizes neuron activations at threshold $\beta = 0$, and aligns activation vectors to concept vectors with Jaccard similarity [2502.00684]. A beam search with width $10$ and maximum formula length $5$ identifies best-match concepts for neurons in DQN and PPO networks. The method reports human-interpretable neuron concepts in Blackjack and LunarLander and validates them with targeted semantic perturbations, where flipping a concept truth value changes neuron activation and downstream action or value outputs in a manner consistent with learned weights [2502.00684].

Mechanistic circuit analysis pushes the same idea from individual neurons to sparse subgraphs. “An explainable transformer circuit for compositional generalization” identifies a compact encoder–decoder transformer circuit for compositional induction using causal ablations, path patching, and program-like reconstruction [2502.15801]. The model reaches $98\%$ exact-match accuracy on held-out compositional episodes, and the paper attributes performance to a specific Q-circuit and K-circuit involving heads such as Enc-self-0.5, Enc-self-1.1, Dec-cross-0.6, and Dec-cross-1.5. The resulting description is explicitly algorithmic: question primitives broadcast index-in-question information, function-right-hand-side tokens recover relative-index-on-LHS, and the Output Head aligns the two to emit the next token [2502.15801]. Related work on MAC networks for natural-language question answering treats each reasoning step as a composition of control and memory operations, with explicit attention over question tokens and context tokens at each step, which yields a step-wise reasoning trace rather than a single monolithic attribution [1810.12698].

## 4. Architectural compositionality and faithful reasoning traces

Some work treats compositional interpretability as an architectural property to be optimized during training. Interpretable compositional CNNs partition filters in a high-level convolutional layer into groups and optimize a correlation-based ratio loss that increases within-group similarity and decreases similarity across groups [2107.04474]. When network parameters are fixed, minimizing the group loss is equivalent to a normalized cut objective, so spectral clustering can update the partition. The model evaluates interpretability with inconsistency of visual patterns and diversity of visual patterns, and reports lower entropy at matched diversity than both traditional CNNs and ICNNs, while maintaining comparable classification accuracy across CUB-200-2011, PASCAL-Part, CelebA, and Helen [2107.04474].

Tree-structured supervision in vision-language models offers a different architectural route. 3VL expands each caption into a hierarchy of progressively refined sub-captions and targeted negatives, trains CLIP with a mixture of standard contrastive loss and tree-aware node-level alignment, and supplements this with Anchor inference and Differential Relevance (DiRe) as interpretability tools [2312.17345]. On VL-CheckList, 3VL reaches an average of $82.46$ compared with $68.87$ for CLIP and $72.43$ for NegCLIP; with Anchor Token Removal it improves relation and attribute averages further, while also improving retrieval and segmentation metrics [2312.17345]. The explanation is tied to the tree: objects, attributes, states, and relations are aligned at different levels of the loss, and DiRe compares relevancy maps for positive and negative captions to isolate discriminative evidence.

Neural-symbolic VQA systems make this compositionality explicit at the level of executable programs. “Interpretable Neural Computation for Real-World Compositional Visual Question Answering” disentangles images into scene graphs and questions into programs, executes object-returning functions symbolically, and passes the selected regions iteratively to LXMERT for answer prediction [2010.04913]. On GQA it reports $58.50\%$ accuracy together with Validity $96.87$, Plausibility $87.94$, and Distribution $2.72$, which the paper presents as substantially better interpretability-oriented metrics than prior baselines. A related line, however, shows that explicit compositional structure is not automatically faithful. “Obtaining Faithful Interpretations from Compositional Neural Networks” evaluates intermediate module outputs in neural module networks on NLVR2 and DROP and finds that module outputs often differ from expected outputs, so the architecture-implied reasoning trace is not by itself a faithful explanation [2005.00724]. Auxiliary supervision, decontextualized utterance representations, and low-expressivity module design improve module-wise faithfulness at minimal cost to end-task accuracy.

## 5. Compositional interpretability on the simplex

In statistics and biomedicine, compositional interpretability concerns how explanations respect the algebra and geometry of compositions rather than how a neural network decomposes a perceptual scene. Tree-guided log-contrast regression provides one version of this objective: effects are selected at internal nodes of a rooted tree, with $\beta = A\gamma$, so nonzeros in $\gamma$ correspond to clades or taxonomic groups rather than individual leaves [2605.15469]. “Tree-aggregated regression for compositional data with measurement errors” shows that aggregation turns leaf-level measurement error into level-dependent, correlated contamination across aggregated nodes, and that naive methods bias selection toward higher-level nodes. TARCO corrects the aggregated Gram matrix by subtracting $\Sigma_{U,A}$, stabilizes it by a tree-weighted PSD projection, and solves a constrained convex program with sparse penalties. The paper proves finite-sample prediction and estimation bounds, sign consistency under tree-sensitive conditions, and reports, in a baseline $n=100$, $p=100$ regime, MSPE $\approx 2.11$ for TARCO-n05 versus $\approx 3.93$ for TRAC-Naive and $\approx 18.9$ for COCO, together with improved aggregation-level recovery [2605.15469].

A second route avoids log-ratio preprocessing entirely. “Interpretable dimension reduction for compositional data” introduces softened amalgamation through a nonnegative column-stochastic matrix $W \in M_{K,D}$ and map $y = Wx$, so each reduced coordinate is a weighted sum of original parts that stays on a lower-dimensional simplex [2509.05563]. This framework defines the central compositional subspace as an identifiable sufficient dimension reduction target and estimates it with Compositional Kernel Dimension Reduction (CKDR) by minimizing a conditional covariance trace objective over $W$. The estimator is provably consistent, often sparse without explicit penalties, and admits dual visualization: reduced compositions and columns of $W$ can be shown on the same ternary plot. On a Crohn’s disease ileum microbiome dataset, CKDR-3 yields an interpretable ternary separation of CD and healthy subjects and competitive misclassification rates, while on a vaginal microbiome dataset it produces interpretable gradients associated with Nugent score [2509.05563].

Personalized and perturbation-based interpretability fill out the statistical picture. DeepCoDA learns multiple sparse log-contrasts $z_{ib}$ together with sample-specific weights $w_i$, so predictions decompose as sums of terms $w_{ib}z_{ib}$ and interpretations remain normalization-free and subcompositionally coherent [2006.01392]. KernelBiome extends supervised learning beyond linear log-contrast models by defining kernels directly on the simplex and introducing Compositional Feature Influence (CFI) and Compositional Feature Dependence (CPD) as perturbation-based interpretability quantities [2205.07271]. The paper proves that CFI consistently estimates an average infinitesimal perturbation effect and that CPD consistently estimates a finite perturbation effect under stated support conditions; in the linear log-contrast case, CFI reduces exactly to the coefficient $\beta_j$ [2205.07271]. Across these methods, interpretability is not merely sparsity or visualization: it is the requirement that explanation itself remain valid under closure, log-contrast constraints, perturbation on the simplex, or tree aggregation.

## 6. Evaluation criteria, recurring misconceptions, and open problems

A central pattern across the literature is that compositional interpretability is evaluated by criteria tailored to the composition law of the setting rather than by a single universal score. The table summarizes representative examples.

| Setting | Criterion | Role |
|---|---|---|
| Concept representations | MAP, Compositionality Score, cosine alignment | Predict composed concepts; reconstruct from true per-attribute concepts [2406.18534] |
| Prototype and prompt models | Sufficiency, C-insertion, C-deletion, PQ index, attribute alignment | Test faithfulness, fidelity, and sparsity of compositional explanations [2501.06059][2406.13683] |
| Neuron and circuit analysis | IoU, Jaccard similarity, path patching, targeted perturbations | Align internal units with logical forms or mechanistic circuits [2006.14032][2502.00684][2502.15801] |
| Neural-symbolic systems | Module-wise faithfulness, Validity, Plausibility, Distribution | Check whether intermediate reasoning traces match intended operations [2005.00724][2010.04913] |
| Compositional data | Sign consistency, consistency of CKDR, CFI, CPD | Quantify reliable aggregation-level selection or simplex-respecting perturbation effects [2605.15469][2509.05563][2205.07271] |

A recurring misconception is that explicit modular structure is automatically faithful. The NMN study on NLVR2 and DROP shows the opposite: intermediate outputs may deviate markedly from intended module behavior unless auxiliary supervision and carefully restricted module architectures are introduced [2005.00724]. Another misconception is that good single-concept discrimination implies compositionality. CCE explicitly argues that non-compositional concepts can still achieve high ROC-AUC, and therefore discriminative utility is insufficient as a criterion for concept composition [2406.18534].

A second recurring tension concerns hierarchy. Deeper trees or richer structures can improve interpretability by exposing clades, attributes, or reasoning steps, but they can also amplify contamination, increase optimization difficulty, or require stronger regularization. TARCO emphasizes hierarchical contamination under aggregation [2605.15469]; 3VL shows that deeper tree structure improves compositional language concept understanding [2312.17345]; CKDR notes that interpretability benefits from low-dimensional simplex structure but remains sensitive to kernel and optimization choices [2509.05563]. This suggests that compositionality is not a free by-product of adding more structure: the structure must be aligned with the mechanism or geometry one intends to explain.

Open problems are correspondingly diverse. The concept-learning literature highlights hierarchical and relational compositionality, non-linear concept composition, superposition, and robustness under domain shift [2406.18534]. Prototype and prompt methods identify pseudo-label dependency, attribute ambiguity, and dataset-specific prompting as unresolved issues [2501.06059][2406.13683]. Compositional data methods leave zero inflation beyond pseudocounts, tree uncertainty quantification, adaptive penalty design, and stabilized covariance estimation without replicates as open directions [2605.15469][2509.05563][2006.01392]. At the most abstract level, category-theoretic work reframes interpretability as a constrained optimization over faithfulness and description length, with compressive refinement as a route to simpler yet behavior-preserving explanations [2605.08934]. Across these strands, compositional interpretability is best understood not as a single method but as a requirement that explanations decompose into parts, compose according to the model or data domain, and remain faithful under that composition.

Source: https://www.emergentmind.com/topics/compositional-interpretability