AgentCoMa: Commonsense & Math Benchmark

• AgentCoMa is a compositional benchmark combining commonsense filtering and math operations in agentic, real-world problems.
• It isolates mixed-type reasoning by pairing full questions with two step-isolated sub-questions to evaluate compositional performance.
• Empirical findings reveal that while LLMs excel on isolated steps, they struggle when integrating them, highlighting a significant compositionality gap.

Searching arXiv for the benchmark and closely related agentic reasoning papers. Agentic Commonsense and Math benchmark (AgentCoMa) is a compositional benchmark in which each task requires a commonsense reasoning step and a math reasoning step in a real-world scenario. It was introduced to evaluate mixed-type compositionality rather than composition within a single reasoning type, and it is presented as the first benchmark to systematically measure LLMs’ ability to compose disparate reasoning types—commonsense and math—in real-world scenarios. The benchmark pairs each full question with two step-isolated sub-questions, making it possible to compare performance on the component steps with performance on the full composition. Across 61 LLMs, the central empirical finding is that models can usually solve the two steps in isolation yet exhibit a large accuracy drop when the steps must be combined, whereas non-expert humans do not show a comparable gap (Alazraki et al., 27 Aug 2025).

1. Conceptual motivation and scope

AgentCoMa was designed for a setting in which LLM agents are expected to operate in real-world “agentic” scenarios such as shopping, travel, and household assistance. In such settings, success often depends on combining multiple reasoning types: commonsense constrains what is feasible or relevant, and math then plans, counts, prices, or optimizes over the filtered set. Prior benchmarks, as described for AgentCoMa, generally focus on either commonsense or math reasoning, and existing agentic benchmarks often emphasize tool-use, long horizons, or dynamic environments rather than isolating mixed-type compositionality itself (Alazraki et al., 27 Aug 2025).

The benchmark therefore targets a specific failure mode: not whether a model can perform commonsense reasoning alone, and not whether it can perform elementary arithmetic alone, but whether it can recombine the two when they appear in a single task. The paper characterizes this as a form of brittleness in mixed-type composition. This framing is important because strong scores on single-skill or same-type compositional benchmarks can mask failures that arise only when qualitatively different reasoning processes must be coordinated within one prompt.

2. Benchmark structure, domains, and construction

Each AgentCoMa question composes two steps in a fixed order: a commonsense reasoning step followed by a mathematical reasoning step. Questions are set in five real-world agentic scenarios: House Working, Web Shopping, Science Experiments, Smart Assistant, and Travel Agent. The benchmark contains 260 total samples, with a development set of 80 and a test set of 180, and there is no training set because the evaluation is zero-shot. The domains are balanced across the five scenarios, the four math operations $(+, -, \times, \div)$ are equally represented, and every question is matched with its two step-isolated sub-questions (Alazraki et al., 27 Aug 2025).

Property	Value
Total samples	260
Development set	80
Test set	180
Domains	5
Training set	None
Step pattern	Commonsense $\rightarrow$ Math

Construction was manual rather than synthetic. The questions were written by expert annotators, not LLM-generated, because LLMs were reported to fail at writing questions that genuinely require both reasoning types. Validation was strict: each question was checked for real-world utility, agentic framing, need for genuine commonsense knowledge, and unambiguous single-operation math. This design choice matters because it constrains the benchmark to problems whose difficulty is intended to come from composition, not from vague wording, multi-operation arithmetic, or annotation noise.

The benchmark’s examples illustrate the intended decomposition. In a web-shopping problem, a model may first need to determine which products can safely be stored at room temperature, then compute the total cost of stocking 20 such items. In a house-working problem, it may need to determine which floor can be mopped, then compute area only over the eligible surface. In a travel-agent problem, it may need to identify the relevant continents, then calculate the number of vaccines required. These examples instantiate the benchmark’s core principle: the mathematical step depends on a prior semantic filter supplied by commonsense.

3. Evaluation protocol and baselines

AgentCoMa was evaluated on 61 LLMs spanning sizes from 1.5B to 141B, multiple model families, and multiple training strategies, including instruction-tuned, supervised fine-tuning, reinforcement-learning-tuned, and mixture-of-experts models. The prompting protocol was standardized across models and used few-shot chain-of-thought prompting with two worked examples. Inference used greedy decoding, making the evaluation deterministic rather than search-based or sample-based (Alazraki et al., 27 Aug 2025).

Answer extraction and grading were split by output type. Numerical answers were compared to ground truth after regex extraction of the final value. Non-numerical commonsense answers were judged by an LLM-as-a-judge, specifically GPT-4o, which the paper reports as highly reliable with Cohen’s kappa $= 0.86$ with humans. This is an important methodological detail because AgentCoMa includes both symbolic and semantic outputs across its isolated-step decomposition, and a single grading procedure would not capture both faithfully.

The human baseline consisted of 45 crowdworkers who were high school+ educated, fluent in English, and not allowed calculators or tools. Each participant saw unique samples covering both sub-steps and full compositional questions. This baseline is not merely a sanity check; it functions as a direct comparison between human and model compositionality under matched task structure. The resulting contrast is central to the paper’s interpretation of mixed-type brittleness.

4. Main empirical findings

The benchmark reveals a marked compositionality gap. On isolated steps, LLMs average more than 85% accuracy on both commonsense and math. The median “both correct” rate—meaning that a model answers both isolated sub-questions correctly for a sample—is approximately 75%. On the full compositional question, however, median accuracy drops to approximately 42%, yielding an average compositionality gap of about 30%. Non-expert humans, by contrast, show no significant compositionality gap between answering the isolated steps and answering the full question (Alazraki et al., 27 Aug 2025).

System	Both correct	Compositional accuracy
Phi4 Mini 3.8B	66.1%	35.6%
Mixtral MoE 141B	90.6%	66.1%
Non-expert human	78.9%	82.8%

The table illustrates the benchmark’s main asymmetry. Even stronger LLMs preserve a noticeable gap between “both steps are individually available” and “the composed task is solved,” whereas humans do not. The paper therefore argues that the failure is not simply weak commonsense, weak arithmetic, or lack of exposure to either isolated skill. Rather, it is a failure of recombination.

The error breakdown reinforces that conclusion. In approximately 74% of AgentCoMa failures, the LLM got both the commonsense and the math steps correct individually. This means that a large majority of failures cannot be explained by inability to perform either constituent operation in isolation. The model often appears able to do both subproblems when explicitly separated, yet fails when required to integrate them in one trajectory.

Comparison with earlier compositional benchmarks sharpens the contrast. On Bamboogle, which targets compositional general knowledge, and on MultiArith, which targets compositional elementary math, LLMs do not show a similar compositionality gap. In those benchmarks, nearly all compositional errors arise because at least one sub-step is missed. AgentCoMa is different: the distinctive failure mode is incorrect composition despite isolated-step competence. This is the empirical basis for the claim that mixed-type composition exposes a weakness that same-type composition often does not (Alazraki et al., 27 Aug 2025).

5. Interpretability studies and failure analysis

AgentCoMa includes several interpretability analyses aimed at explaining why mixed-type composition fails. A membership-inference analysis using the Min-K%++ metric indicates that mixed-type compositions have lower similarity scores to model pretraining data than isolated math or isolated commonsense questions. The paper interprets this as evidence that LLMs rarely encounter such mixed-reasoning patterns during training (Alazraki et al., 27 Aug 2025).

A separate control addresses a common alternative explanation: that compositional questions are simply longer or more context-heavy. The benchmark authors added irrelevant math details to commonsense-only inputs to make their length more similar to full compositional inputs. Performance did decline somewhat, especially for smaller models, but the compositionality gap remained substantially larger and therefore could not be explained by added context alone. This directly challenges the view that the main difficulty is superficial prompt complexity rather than cross-type reasoning.

Attention analysis further suggests a retrieval or grounding deficit. On compositional questions, models assign slightly less weight to the original context. The reported lookback ratios are 70.75 for composition, 71.49 for commonsense, and 72.20 for math. The decrease is small, but the paper states that it supports hallucination in the compositional setting.

The most specific mechanistic evidence comes from Query-Relevant Neuron Cluster Attribution (QRNCA). The reported neuron overlap between math and commonsense steps is only 3%. Compositional questions overlap mainly with math-type neurons at 39% and almost not at all with commonsense-type neurons at 3%. The paper’s interpretation is that the model behaves as though the compositional question were a standard math question and neglects the commonsense substructure, even when commonsense filtering is indispensable. In this view, the issue is not merely that the model forgets a detail; it activates an incomplete internal circuit for the task class.

An appendix example illustrates the practical consequence. A model that answers both isolated steps correctly can still hallucinate or mangle the combined logic, for instance by double-counting or by misapplying a rule that requires commonsense filtering of relevant events. This kind of failure is consistent with the neural finding that the composition is processed more like a pure math prompt than like a hybrid reasoning problem.

6. Position within agentic reasoning research and future directions

AgentCoMa occupies a distinct position relative to neighboring benchmarks. GSM-Agent, for example, isolates agentic reasoning in a controlled tool environment: the model receives only the question, the necessary premises are hidden in a document database, and the agent must retrieve them using Search(x) and NextPage(). GSM-Agent then analyzes trajectories through an agentic reasoning graph and finds that revisit behavior is highly correlated with success (Zhu et al., 26 Sep 2025). AgentCoMa, by contrast, is not a tool-use benchmark. Its contribution is to isolate mixed-type compositionality itself, without long-horizon search, dynamic environments, or hidden-premise retrieval.

This distinction helps clarify what AgentCoMa does and does not test. It does not measure whether a model can search effectively, whether it can manage external memory, or whether it can coordinate multiple tools. Instead, it measures whether a model can combine a commonsense filter with a math operation inside a single textual reasoning problem. That is why the benchmark’s central comparison is between isolated-step competence and compositional competence rather than between tool-free and tool-augmented settings.

The paper’s implications for future development are correspondingly specific. It argues that current evaluation misses an important weakness because many benchmarks focus on either step multiplicity within one reasoning type or on agentic tool use without isolating cross-type composition. It proposes that future training should emphasize mixed-type steps rather than only many-step tasks of a single type, and it labels this target capability RRR (Reasoning Recombination Resilience). The benchmark authors also argue that interpretability tools, including neuronal and attention analysis, should remain central for understanding and mitigating such failures (Alazraki et al., 27 Aug 2025).

The benchmark’s stated limitations are equally clear. AgentCoMa questions are all two-step, follow the order commonsense $\rightarrow$ math, and are text-only. Future work could extend the setting to multimodal inputs, different step orderings, more than two interleaved step types, and richer, open-ended agentic tasks. A related methodological direction appears in AgenticMath, which presents a multi-agent pipeline for generating high-quality reasoning data and explicitly describes that pipeline as directly translatable to commonsense (AgentCoMa) and other structured reasoning challenges (Liu et al., 22 Oct 2025).

Taken together, these findings establish AgentCoMa as a benchmark for a narrow but consequential failure mode: models often possess the ingredients for success, yet fail when the reasoning recipe mixes commonsense and mathematics within the same question. That is the sense in which AgentCoMa functions as a test bed for future improvement in robust, real-world mixed-type reasoning (Alazraki et al., 27 Aug 2025).