Compositional Interpretability in AI Models

• Compositional interpretability is the ability to express model predictions as compositions of simpler, traceable components that correspond to specific features or concepts.
• It encompasses diverse methodologies such as concept-based extraction, neuron and circuit analyses, and compositional data techniques that provide faithful, quantifiable explanations.
• This approach enhances transparency and robustness by aligning interpretative structures with model behavior across various domains including vision, natural language, and biomedical data.

Compositional interpretability denotes a family of interpretability objectives in which a model’s prediction, representation, or statistical effect is explained as a composition of simpler parts whose roles remain explicit under composition. In the recent literature, this idea appears in several technically distinct but structurally related forms: concept-based explanations that compose object-, attribute-, or prototype-level primitives; neuron- and circuit-level analyses that express behavior through logical forms or sparse subgraphs; neural-symbolic systems whose intermediate programs or trees are directly inspectable; and compositional data methods in which interpretability must respect the simplex, log-ratio structure, or aggregation geometry rather than unconstrained Euclidean coordinates (Sivaprasad et al., 10 Jan 2025, Stein et al., 2024, Gauderis et al., 9 May 2026, Park et al., 6 Sep 2025).

1. Formal definitions and mathematical structure

A direct concept-based definition states that compositional interpretability is the property that a model’s decision can be expressed as a composition of simpler concept-level parts whose evidence can be traced to concrete regions in the input and matched to prototypical regions in training images (Sivaprasad et al., 10 Jan 2025). In representation learning, the same idea is formalized by requiring additive composition of concept vectors: for concepts $c_i$ and $c_j$, a concept representation is compositional if $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$ for some positive weights, and a sample embedding is interpretable when it decomposes into one concept per attribute (Stein et al., 2024).

A more general formalization separates syntax from semantics. In the category-theoretic account, a compositional model is written as $M = (D, S, C, [[\cdot]])$, where $S$ is a monoidal category of structure, $D$ is a string diagram, $C$ is a semantics category, and $[[\cdot]]: S \to C$ is a monoidal representation functor (Gauderis et al., 9 May 2026). A compositional interpretation is then a pair of mappings, $I_S: S \to H$ and $I_C: C \to H$, constrained by the commuting condition $c_j$0, so that structural explanations and behavioral explanations agree (Gauderis et al., 9 May 2026). Closely related accounts define compositional models via symmetric monoidal categories, strong symmetric monoidal semantics functors, and natural transformations linking abstract and concrete interpretation; the point is that explanations become diagrammatic, compositional, and checkable rather than merely descriptive (Tull et al., 2024, Smithe, 2019).

In compositional data analysis, the formal object of interpretation is different. A log-contrast regression writes coefficients under the sum-to-zero constraint $c_j$1, and tree-guided compositional regression reparameterizes leaf effects by $c_j$2, so that sparsity in $c_j$3 selects interpretable internal nodes (Li et al., 14 May 2026). A different line replaces log-ratio coordinates with a reduction map on the simplex, $c_j$4, where $c_j$5 is a nonnegative column-stochastic matrix; interpretability then lies in the fact that each reduced coordinate is a softened amalgamation of original parts, and the columns of $c_j$6 are themselves compositional objects (Park et al., 6 Sep 2025). DeepCoDA adopts yet another formulation: it learns log-contrast bottleneck features $c_j$7 together with sample-specific weights $c_j$8, yielding predictions of the form $c_j$9, so the explanation is a personalized composition of log-contrasts rather than a single global coefficient vector (Quinn et al., 2020).

2. Concept extraction, prototypes, and compositional representations

One influential representation-level formulation identifies two salient properties of compositional concept representations: cross-attribute orthogonality and intra-attribute non-orthogonality (Stein et al., 2024). Compositional Concept Extraction (CCE) operationalizes these constraints by alternating between learning an attribute-specific subspace and discovering concepts within that subspace via spherical k-means clustering, then orthogonally rejecting that subspace before learning the next attribute. The resulting concepts are evaluated by mean average precision for predicting composed concepts from base concept scores, by a Compositionality Score measuring reconstruction from true per-attribute concepts, and by cosine alignment to ground-truth concept vectors. On controlled subsets, CCE reaches $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$0 MAP on CLEVR and attains compositionality scores close to the ground truth, while also improving downstream classification on several datasets (Stein et al., 2024).

COMIX turns compositional interpretability into an ante hoc image-classification mechanism rather than a post hoc explanation layer (Sivaprasad et al., 10 Jan 2025). It builds on B-cos networks, discovers class-defining features by mutual information, decomposes a test image into feature-level regions, retrieves nearest training exemplars for each active feature, and aggregates per-feature votes by majority vote. The explanation is therefore not an auxiliary attribution map but the actual decision path: each primitive in the test image is matched to a training primitive, and the final class is the composition of those matches. The paper states a sufficiency theorem under pseudo-label consistency and reports a C-insertion score of $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$1 on ImageNet versus $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$2 for the best post hoc baseline, a $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$3 improvement, together with improved PQ sparsity across several datasets (Sivaprasad et al., 10 Jan 2025).

Prompt tuning and intrinsic concept extraction extend the same logic to other representational interfaces. IntCoOp decomposes prompts into instance-conditioned context tokens, an explicit attribute token, and the class token, with prompt form $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$4; attribute embeddings are supervised against BLIP-2-derived attribute text embeddings and regularized toward compositional templates such as “a photo of a [attribute] [class]” (Ghosal et al., 2024). In a 16-shot setup it improves CoOp by $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$5 in average harmonic mean across $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$6 datasets. HyperExpress addresses “Compositional and Interpretable Intrinsic Concept Extraction” from a single image by combining hyperbolic contrastive learning, hyperbolic entailment learning, and Horosphere Projection so that object-level and attribute-level concepts are both hierarchically organized and linearly composable on a horospherical submanifold (Shi et al., 12 Mar 2026). On UCEBench it reports SIMI $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$7, SIMC $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$8, ACC1 $R(c_i \cup c_j) = w_{c_i} R(c_i) + w_{c_j} R(c_j)$9, and ACC3 $M = (D, S, C, [[\cdot]])$0, with higher ACC than ICE and AutoConcept (Shi et al., 12 Mar 2026).

3. Neuron, feature, and circuit-level compositional explanations

At the neuron level, compositional interpretability often means replacing single labels with structured logical descriptions. “Compositional Explanations of Neurons” defines a concept language over atomic predicates with operators And, Or, Not, and, in natural language inference, Neighbors (Mu et al., 2020). A beam search over formulas up to length $M = (D, S, C, [[\cdot]])$1 maximizes IoU between thresholded neuron masks and candidate logical forms. On the final convolutional layer of a Places365 ResNet-18, mean IoU rises from $M = (D, S, C, [[\cdot]])$2 at $M = (D, S, C, [[\cdot]])$3 to $M = (D, S, C, [[\cdot]])$4 at $M = (D, S, C, [[\cdot]])$5, a $M = (D, S, C, [[\cdot]])$6 increase, and manual inspection of $M = (D, S, C, [[\cdot]])$7 randomly sampled vision neurons found $M = (D, S, C, [[\cdot]])$8 meaningful compositional abstractions and $M = (D, S, C, [[\cdot]])$9 polysemantic neurons (Mu et al., 2020). The same framework is used to localize shallow lexical heuristics in NLI and to generate “copy-paste” adversarial interventions that alter model predictions in predictable ways.

A closely related program has been adapted to deep reinforcement learning. “Compositional Concept-Based Neuron-Level Interpretability for Deep Reinforcement Learning” defines atomic concepts as binary predicates over states, composes them with $S$0, $S$1, and $S$2, binarizes neuron activations at threshold $S$3, and aligns activation vectors to concept vectors with Jaccard similarity (Jiang et al., 2 Feb 2025). A beam search with width $S$4 and maximum formula length $S$5 identifies best-match concepts for neurons in DQN and PPO networks. The method reports human-interpretable neuron concepts in Blackjack and LunarLander and validates them with targeted semantic perturbations, where flipping a concept truth value changes neuron activation and downstream action or value outputs in a manner consistent with learned weights (Jiang et al., 2 Feb 2025).

Mechanistic circuit analysis pushes the same idea from individual neurons to sparse subgraphs. “An explainable transformer circuit for compositional generalization” identifies a compact encoder–decoder transformer circuit for compositional induction using causal ablations, path patching, and program-like reconstruction (Tang et al., 19 Feb 2025). The model reaches $S$6 exact-match accuracy on held-out compositional episodes, and the paper attributes performance to a specific Q-circuit and K-circuit involving heads such as Enc-self-0.5, Enc-self-1.1, Dec-cross-0.6, and Dec-cross-1.5. The resulting description is explicitly algorithmic: question primitives broadcast index-in-question information, function-right-hand-side tokens recover relative-index-on-LHS, and the Output Head aligns the two to emit the next token (Tang et al., 19 Feb 2025). Related work on MAC networks for natural-language question answering treats each reasoning step as a composition of control and memory operations, with explicit attention over question tokens and context tokens at each step, which yields a step-wise reasoning trace rather than a single monolithic attribution (Selvakumar et al., 2018).

4. Architectural compositionality and faithful reasoning traces

Some work treats compositional interpretability as an architectural property to be optimized during training. Interpretable compositional CNNs partition filters in a high-level convolutional layer into groups and optimize a correlation-based ratio loss that increases within-group similarity and decreases similarity across groups (Shen et al., 2021). When network parameters are fixed, minimizing the group loss is equivalent to a normalized cut objective, so spectral clustering can update the partition. The model evaluates interpretability with inconsistency of visual patterns and diversity of visual patterns, and reports lower entropy at matched diversity than both traditional CNNs and ICNNs, while maintaining comparable classification accuracy across CUB-200-2011, PASCAL-Part, CelebA, and Helen (Shen et al., 2021).

Tree-structured supervision in vision-LLMs offers a different architectural route. 3VL expands each caption into a hierarchy of progressively refined sub-captions and targeted negatives, trains CLIP with a mixture of standard contrastive loss and tree-aware node-level alignment, and supplements this with Anchor inference and Differential Relevance (DiRe) as interpretability tools (Yellinek et al., 2023). On VL-CheckList, 3VL reaches an average of $S$7 compared with $S$8 for CLIP and $S$9 for NegCLIP; with Anchor Token Removal it improves relation and attribute averages further, while also improving retrieval and segmentation metrics (Yellinek et al., 2023). The explanation is tied to the tree: objects, attributes, states, and relations are aligned at different levels of the loss, and DiRe compares relevancy maps for positive and negative captions to isolate discriminative evidence.

Neural-symbolic VQA systems make this compositionality explicit at the level of executable programs. “Interpretable Neural Computation for Real-World Compositional Visual Question Answering” disentangles images into scene graphs and questions into programs, executes object-returning functions symbolically, and passes the selected regions iteratively to LXMERT for answer prediction (Tang et al., 2020). On GQA it reports $D$0 accuracy together with Validity $D$1, Plausibility $D$2, and Distribution $D$3, which the paper presents as substantially better interpretability-oriented metrics than prior baselines. A related line, however, shows that explicit compositional structure is not automatically faithful. “Obtaining Faithful Interpretations from Compositional Neural Networks” evaluates intermediate module outputs in neural module networks on NLVR2 and DROP and finds that module outputs often differ from expected outputs, so the architecture-implied reasoning trace is not by itself a faithful explanation (Subramanian et al., 2020). Auxiliary supervision, decontextualized utterance representations, and low-expressivity module design improve module-wise faithfulness at minimal cost to end-task accuracy.

5. Compositional interpretability on the simplex

In statistics and biomedicine, compositional interpretability concerns how explanations respect the algebra and geometry of compositions rather than how a neural network decomposes a perceptual scene. Tree-guided log-contrast regression provides one version of this objective: effects are selected at internal nodes of a rooted tree, with $D$4, so nonzeros in $D$5 correspond to clades or taxonomic groups rather than individual leaves (Li et al., 14 May 2026). “Tree-aggregated regression for compositional data with measurement errors” shows that aggregation turns leaf-level measurement error into level-dependent, correlated contamination across aggregated nodes, and that naive methods bias selection toward higher-level nodes. TARCO corrects the aggregated Gram matrix by subtracting $D$6, stabilizes it by a tree-weighted PSD projection, and solves a constrained convex program with sparse penalties. The paper proves finite-sample prediction and estimation bounds, sign consistency under tree-sensitive conditions, and reports, in a baseline $D$7, $D$8 regime, MSPE $D$9 for TARCO-n05 versus $C$0 for TRAC-Naive and $C$1 for COCO, together with improved aggregation-level recovery (Li et al., 14 May 2026).

A second route avoids log-ratio preprocessing entirely. “Interpretable dimension reduction for compositional data” introduces softened amalgamation through a nonnegative column-stochastic matrix $C$2 and map $C$3, so each reduced coordinate is a weighted sum of original parts that stays on a lower-dimensional simplex (Park et al., 6 Sep 2025). This framework defines the central compositional subspace as an identifiable sufficient dimension reduction target and estimates it with Compositional Kernel Dimension Reduction (CKDR) by minimizing a conditional covariance trace objective over $C$4. The estimator is provably consistent, often sparse without explicit penalties, and admits dual visualization: reduced compositions and columns of $C$5 can be shown on the same ternary plot. On a Crohn’s disease ileum microbiome dataset, CKDR-3 yields an interpretable ternary separation of CD and healthy subjects and competitive misclassification rates, while on a vaginal microbiome dataset it produces interpretable gradients associated with Nugent score (Park et al., 6 Sep 2025).

Personalized and perturbation-based interpretability fill out the statistical picture. DeepCoDA learns multiple sparse log-contrasts $C$6 together with sample-specific weights $C$7, so predictions decompose as sums of terms $C$8 and interpretations remain normalization-free and subcompositionally coherent (Quinn et al., 2020). KernelBiome extends supervised learning beyond linear log-contrast models by defining kernels directly on the simplex and introducing Compositional Feature Influence (CFI) and Compositional Feature Dependence (CPD) as perturbation-based interpretability quantities (Huang et al., 2022). The paper proves that CFI consistently estimates an average infinitesimal perturbation effect and that CPD consistently estimates a finite perturbation effect under stated support conditions; in the linear log-contrast case, CFI reduces exactly to the coefficient $C$9 (Huang et al., 2022). Across these methods, interpretability is not merely sparsity or visualization: it is the requirement that explanation itself remain valid under closure, log-contrast constraints, perturbation on the simplex, or tree aggregation.

6. Evaluation criteria, recurring misconceptions, and open problems

A central pattern across the literature is that compositional interpretability is evaluated by criteria tailored to the composition law of the setting rather than by a single universal score. The table summarizes representative examples.

Setting	Criterion	Role
Concept representations	MAP, Compositionality Score, cosine alignment	Predict composed concepts; reconstruct from true per-attribute concepts (Stein et al., 2024)
Prototype and prompt models	Sufficiency, C-insertion, C-deletion, PQ index, attribute alignment	Test faithfulness, fidelity, and sparsity of compositional explanations (Sivaprasad et al., 10 Jan 2025, Ghosal et al., 2024)
Neuron and circuit analysis	IoU, Jaccard similarity, path patching, targeted perturbations	Align internal units with logical forms or mechanistic circuits (Mu et al., 2020, Jiang et al., 2 Feb 2025, Tang et al., 19 Feb 2025)
Neural-symbolic systems	Module-wise faithfulness, Validity, Plausibility, Distribution	Check whether intermediate reasoning traces match intended operations (Subramanian et al., 2020, Tang et al., 2020)
Compositional data	Sign consistency, consistency of CKDR, CFI, CPD	Quantify reliable aggregation-level selection or simplex-respecting perturbation effects (Li et al., 14 May 2026, Park et al., 6 Sep 2025, Huang et al., 2022)

A recurring misconception is that explicit modular structure is automatically faithful. The NMN study on NLVR2 and DROP shows the opposite: intermediate outputs may deviate markedly from intended module behavior unless auxiliary supervision and carefully restricted module architectures are introduced (Subramanian et al., 2020). Another misconception is that good single-concept discrimination implies compositionality. CCE explicitly argues that non-compositional concepts can still achieve high ROC-AUC, and therefore discriminative utility is insufficient as a criterion for concept composition (Stein et al., 2024).

A second recurring tension concerns hierarchy. Deeper trees or richer structures can improve interpretability by exposing clades, attributes, or reasoning steps, but they can also amplify contamination, increase optimization difficulty, or require stronger regularization. TARCO emphasizes hierarchical contamination under aggregation (Li et al., 14 May 2026); 3VL shows that deeper tree structure improves compositional language concept understanding (Yellinek et al., 2023); CKDR notes that interpretability benefits from low-dimensional simplex structure but remains sensitive to kernel and optimization choices (Park et al., 6 Sep 2025). This suggests that compositionality is not a free by-product of adding more structure: the structure must be aligned with the mechanism or geometry one intends to explain.

Open problems are correspondingly diverse. The concept-learning literature highlights hierarchical and relational compositionality, non-linear concept composition, superposition, and robustness under domain shift (Stein et al., 2024). Prototype and prompt methods identify pseudo-label dependency, attribute ambiguity, and dataset-specific prompting as unresolved issues (Sivaprasad et al., 10 Jan 2025, Ghosal et al., 2024). Compositional data methods leave zero inflation beyond pseudocounts, tree uncertainty quantification, adaptive penalty design, and stabilized covariance estimation without replicates as open directions (Li et al., 14 May 2026, Park et al., 6 Sep 2025, Quinn et al., 2020). At the most abstract level, category-theoretic work reframes interpretability as a constrained optimization over faithfulness and description length, with compressive refinement as a route to simpler yet behavior-preserving explanations (Gauderis et al., 9 May 2026). Across these strands, compositional interpretability is best understood not as a single method but as a requirement that explanations decompose into parts, compose according to the model or data domain, and remain faithful under that composition.