Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry

Abstract: Humans cannot always intuit what scenarios are most challenging to LLMs. Hoping to capture challenging edge cases, developers either design problems to be difficult for humans or curate extensive benchmarks. What if we could instead anticipate which scenarios a model will fail on? In this paper, we use an LLM's representational geometry to predict which concept combinations it will fail on. We attribute this compositional failure to interference between salient features. In tasks that require systematic composition - toy programmatic settings, multihop reasoning, multilingual factual recall - we find that when a pair of concepts is encoded near-orthogonally, the model reliably composes them. When their linear encodings are close, producing interference, the model fails to compose them. Our method reliably anticipates failure modes across different compositional tasks, without evaluating specific inputs. These results lay the groundwork to use representational geometry to identify high-risk examples, construct targeted stress tests, and provide a scalable foundation for active learning in real-world deployment.

• The paper introduces a compositional interference (CI) metric based on vector angles to predict when LLMs will fail on complex compositional tasks.
• It outlines a methodology that retrieves atomic concept encodings and applies background correction to isolate relevant feature interactions.
• Empirical analyses in benchmarks like SCAN and real-world LLM settings confirm that higher CI correlates with significantly lower accuracy.

Predicting LLM Compositional Failures via Adversarial Concept Search and Feature Interference

Introduction and Motivation

The paper "Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry" (2606.13934) addresses the challenge of anticipating which compositional scenarios LLMs are likely to fail, without the need to enumerate and test all possible input combinations explicitly. The primary insight is that model failures in systematic compositional generalization can be predicted using the internal representational geometry of atomic concepts—specifically, the angles between their vector encodings in feature space. This work moves beyond human intuition or extensive benchmarking by leveraging the representation geometry as a proactive metric for constructing targeted stress sets and enabling active learning in high-combinatorial domains.

Theoretical Framework: Lossy Superposition and Compositional Interference

The study grounds its hypothesis in compressed sensing theory and the superposition principle for neural representations. While idealized compressed sensing allows lossless recovery of $k$-sparse feature vectors from superposed, even non-orthogonal encodings, practical LLMs learn lossy representations subject to interference. Noise and limited decoding capacity render feature recovery increasingly error-prone as the encoding directions of the active features become less orthogonal.

Crucially, the compositional interference (CI) metric extends this to practical error prediction. CI for a set of concepts is defined as: $$CI(C) = \max_{i \in \mathcal{S}(C)} \frac{1}{|\mathcal{S}(C)|} \sum_{\substack{j \in \mathcal{S}(C)}} |\cos(a_i, a_j)|$$ where $\mathcal{S}(C)$ is the set of salient (active) features for the composition $C$, and $a_k$ the vector encoding of concept $k$. A high CI (small angle, large cosine similarity) signifies high interference and, theoretically, greater likelihood of compositional failure.

Figure 1: The angle between atomic concept representations identifies the most challenging compositions, enabling failure prediction across the combinatorial space without evaluating specific inputs.

Methodology: Estimating CI from Internal Representations

The approach consists of these steps:

• Atomic Concept Extraction: Atomic concept vectors are estimated from model residual activations, using instances where each concept appears in isolation or in simple contexts.
• Salient Feature Encoding: These vectors may be simple averages (single features) or, in multilingual settings, subspaces extracted via SVD.
• Background Structure Correction: To remove irrelevant global clustering effects (e.g., prompt format, task structure), representations are mean-centered within empirically identified background clusters.

Figure 2: Large-scale structure in representation space.

Validation in Synthetic Compositional Settings

The theory is validated in the SCAN benchmark, a standard environment for compositional generalization. Transformers trained with varying data coverage and model size regimes show that increasing training coverage yields more orthogonal atomic concept representations, while low coverage (and low rank) increases CI and corresponding error rates.

Figure 3: CI between atomic concepts.

Empirically, examples with higher CI are systematically harder for models. Cumulative accuracy curves, when test sets are sorted by minimum CI, confirm that strict CI cutoffs select for more challenging examples, with performance significantly below the mean.

Figure 4: SCAN models fail on examples with high interference. Model accuracy on SCAN test sets with varying cutoffs for minimum CI.

PR-AUC analysis across model sizes and training regimes shows that CI reliably predicts compositional failures, supporting its use for adversarial challenge set selection.

Predicting Compositional Failures in Real-World LLMs

The framework is extended to Llama-3.2-3B for two naturalistic settings: multihop question answering (QA) and multilingual factual recall.

• Multihop QA: The combination of two factual queries (e.g., "author of 1984" $\rightarrow$ "birth year of George Orwell") forms a composition whose difficulty is predicted from the CI between the representations of the constituent atomic queries. Only cases where the LLM succeeds on individual queries are retained, isolating compositional errors.
• Multilingual Fact Recall: Here, compositional generalization is required to link an English fact with a non-English language context. Concept representations for the factual component are extracted from English queries, and language subspaces are estimated from typical target-language samples. CI between the two is used to anticipate cross-lingual performance breakdowns.

Numerical results show monotonic declines in accuracy as CI increases, across both settings and all evaluated languages. In multilingual fact recall, CI not only predicts per-example errors but also selects the subset of language-fact pairs that constitute the most challenging challenge sets.

Figure 5: LLMs fail on examples with high interference. (a) For multihop QA, CI provides a strong ranking of example difficulty. (b) For multilingual fact recall, PR-AUC for the CI-based error prediction outperforms a majority baseline on every language. (c) For multilingual fact recall, CI ranking chooses difficult challenge sets for each language.

Visualization of correctness across language-fact combinations further demonstrates the predictive validity of CI, with errors concentrated at high CI values.

Figure 6: LLMs fail on datasets with high interference. (a) Fact recall across languages: columns are English facts, rows are target languages, marks indicate LLM correctness on each multilingual fact, and color shows normalized CI within each language. CI is broadly predictive of whether the model answers the compositional question correctly. (b,c) Subspace-level CI correlates strongly with accuracy for most languages in multilingual fact recall; Japanese is shown here, with all languages in Appendix.

Implications and Future Directions

The primary result is that compositional interference, manifest as geometric overlap between atomic concept encodings, acts as a strong predictor of failures in compositional generalization—even for unseen scenarios and without access to specific input instances. This enables:

• Targeted Challenge Set Construction: By ranking possible compositional scenarios by CI, developers can efficiently select high-risk examples for robust evaluation.
• Active Learning: In data- or resource-constrained settings, the method enables prioritization of examples that maximize error coverage for further annotation or human review.
• Analysis of Model Systematicity: The unified error arising from lossy superposition and encoded interference supports theoretical models connecting orthogonality, compositionality, and robust generalization.

From a broader perspective, this work suggests that feature geometry provides interpretable, scalable diagnostics for model robustness. This diverges from adversarial example generation (which needs full input instantiation) and connects to mechanistic interpretability efforts by enabling proactive prediction of generalization failures.

Potential future extensions include:

• Automated de novo challenge input synthesis via combinatorial concept search.
• More granular concept extraction using causal or sparse autoencoder features.
• Positive (constructive) interference analysis and the role of co-activation structure in aiding generalization.
• Enabling scalable, geometry-aware active learning in production LLM deployment.

Conclusion

This paper establishes compositional interference, as measured by internal feature geometry, as a practical and theoretically grounded predictor of model failure in compositional scenarios. The adversarial concept search framework unifies principles from compressed sensing, interpretability, and generalization theory while delivering practical value for evaluation and active dataset curation in both synthetic and real-world LLM settings. These advances highlight the continued importance of representational geometry as a key diagnostic and steering tool for complex neural LLMs.