---
title: 'GSM-Agent: LLM Agentic Reasoning Benchmark'
url: https://www.emergentmind.com/topics/gsm-agent
type: topic
---

# GSM-Agent: LLM Agentic Reasoning Benchmark

GSM-Agent is a benchmark and analysis framework for agentic reasoning in large language models. It converts grade-school math problems into tool-use tasks by presenting only the question while withholding the premises needed for solution; those premises are embedded in a controllable document environment, and the agent must proactively retrieve them through search before producing a numeric answer. The framework combines benchmark construction, a formal graph representation of agent trajectories, and tool-augmented test-time scaling methods intended to improve revisit behavior during search [2509.21998].

## 1. Definition and nomenclature

In GSM-Agent, the central object of study is **agentic reasoning**: the coupled process of deciding what information to seek, using tools to gather it, and reasoning over the retrieved material to solve a task. The benchmark is designed to separate this capability from advanced mathematics, expert knowledge, or program synthesis by using problems whose underlying reasoning is at the grade-school level, but whose required premises are omitted from the prompt and must be recovered from the environment [2509.21998].

The name is potentially ambiguous. In earlier arXiv literature, “GSM” commonly referred to unrelated domains, including Global System for Mobile Communications in localization and SMS-enabled control systems [1110.3425], [0904.3715], [2606.16468], [2004.04864], [1806.03715], and generalized spatial modulation in massive MIMO [1903.04460]. In GSM-Agent, by contrast, “GSM” refers to the grade-school math source tasks from which the benchmark is derived [2509.21998].

This distinction matters because a common misconception is to read GSM-Agent as a telecommunications or wireless-systems term. In the benchmark literature, it instead denotes a controlled evaluation setting for LLM agents. The benchmark’s motivating empirical result is that even when the underlying task is simple arithmetic or word-problem reasoning, agentic performance remains far from saturated: GPT-5 achieves 66.78% accuracy, while the best reported model, o3, reaches 68.46% [2509.21998].

## 2. Task formulation and controllable environment

Each benchmark instance is formalized as a task
$$
T = (q, (p_1,\dots,p_k), a),
$$
where $q$ is the question, $p_1,\dots,p_k$ are the premises, and $a$ is the ground-truth numeric answer. In the static version of the task, the premises are provided directly. In GSM-Agent, the agent sees only $q$ and must recover the missing premises from an environment
$$
E = \{D_1,\dots,D_m\},
$$
whose documents contain the needed facts in context-rich form [2509.21998].

The tool interface is deliberately minimal. Agents receive two tools, $F=\{\mathrm{Search}(\cdot), \mathrm{NextPage}(\cdot)\}$. A Search call returns the top 5 most relevant documents to a query, and NextPage returns the next 5 results for the current search. For each Search call, an agent may invoke NextPage up to 19 times, so a single query can expose up to the top 100 documents [2509.21998].

Environment construction proceeds in five stages. First, data preprocessing is used to avoid ambiguity in a shared database through entity detection, name assignment for generic entities, and timestamp assignment via graph coloring. Second, problems are decomposed into self-contained premises and rephrased questions. Third, a hierarchical document-generation process creates coherent documents; independence verification is used to ensure documents do not leak information across premises, and 30% of documents are randomly anonymized to discourage direct “cheat” queries. Fourth, problems are filtered to keep only those solvable when all documents are provided. Fifth, a Chroma database is built with text-embedding-3-large as the default embedding model [2509.21998].

After filtering, the dataset contains 7,323 problems. The paper also reports “with 32,315 unique problems stored in the chroma database,” and explicitly notes that this appears inconsistent with the prior count and is not clarified in the text [2509.21998]. The benchmark is provided in three difficulty-controlled sizes: GSM-Agent-Full, GSM-Agent-Medium at 25% of problems, and GSM-Agent-Small at 6.25% of problems. Appendix statistics report, for the Full setting, a mean of 4.41 documents per problem, standard deviation 1.74, minimum 1, maximum 14, and median 4; the mean problem “span,” defined as unique cluster classes per problem, is approximately 2.89, with median 3 [2509.21998].

The controllability of the environment is a defining methodological feature. By tuning database size and distractor density while keeping the underlying mathematical reasoning simple, the benchmark isolates search policy, retrieval sequencing, and information integration from raw symbolic difficulty. This suggests that GSM-Agent is intended less as a test of arithmetic competence than as a diagnostic of search-conditioned reasoning.

## 3. Agentic reasoning graph

The benchmark’s main analytical contribution is the **agentic reasoning graph**, a discrete representation of the environment and of the trajectory induced by tool use. Let $E=\{D_i\}_{i=1}^N$ be the document set, let $e_\theta(\cdot)$ be the embedding model, and let $e_i=e_\theta(D_i)\in\mathbb{R}^d$ denote document embeddings. K-means is applied with $K=250$ clusters, yielding clusters $\{C_i\}_{i=1}^K$ and centroids $\{c_k\}_{k=1}^K$, which define graph nodes $V=\{v_1,\dots,v_K\}$ [2509.21998].

The clustering objective is
$$
J = \sum_{i=1}^{K} \sum_{x \in C_i} \|x - \mu_i\|^2,
$$
where $\mu_i$ is the centroid of cluster $C_i$. A tool-use trace is then mapped to a node sequence. If the $t$-th call is $\mathrm{Search}(x)$, the node is
$$
p_t = \arg\min_{v_k \in V} \| q(x) - c_k \|_2,
$$
where $q(x)$ is the embedding of the query prompt. If the $t$-th call is $\mathrm{NextPage}(\cdot)$, then $p_t=p_{t-1}$. The resulting path is $\pi=(p_1,\dots,p_T)$ [2509.21998].

Three step types are defined over this path. **Exploration** occurs when a node is visited for the first time; **exploitation** occurs when the current step remains at the immediately previous node; **revisit** occurs when the current node was visited earlier but is not the immediately previous node. The formal revisit rate is
$$
R = \frac{\#\{ i > 1 : p_i \in \{ p_1,\dots,p_{i-1} \} \land p_i \neq p_{i-1} \}}{T - 1}.
$$
Appendix metrics also include $V$, the number of unique nodes visited per trace, and `hasRvst`, the proportion of traces containing at least one revisit step [2509.21998].

The graph is intended to make search behavior interpretable. A t-SNE visualization of nodes is reported to show semantically coherent clusters, including Node 29 (“Grocery Receipts”), Node 54 (“Fruit Counting”), and Node 100 (“Youth Sports Stats”) [2509.21998]. This provides a topology in which agent behavior can be analyzed in terms of movement among semantic regions rather than only in terms of token-level traces or raw document IDs.

## 4. Empirical performance and observed failure modes

Evaluations use a LangChain ReAct agent with temperature 0.4 and max tokens 4096. Zero-shot prompting is reported as the most stable configuration overall, although Claude-4-sonnet benefits from few-shot prompting [2509.21998].

The main performance picture is notable because the benchmark’s underlying reasoning is intentionally elementary, yet agentic accuracy remains modest. The “solvable by any model” rate is 88.00%, indicating that a large fraction of tasks are not intrinsically pathological, but that models vary sharply in how effectively they search and integrate premises [2509.21998].

| Model | Accuracy | Revisit profile |
|---|---:|---|
| o3-zeroshot | 68.46% | hasRvst 70.13%; Rvst 24.56% |
| GPT-5-zeroshot | 66.78% | hasRvst 46.47%; Rvst 16.81% |
| Grok-4-zeroshot | 53.00% | hasRvst 64.88%; Rvst 17.77% |
| GPT-4o-zeroshot | 22.67% | hasRvst 2.03%; Rvst 0.62% |
| DeepSeek-V3-zeroshot | 19.42% | hasRvst 0.00%; Rvst 0.00% |

The broader table of results adds search-process metrics. o3-zeroshot uses 13.33 search rounds on average, has duration 117.85 s, search-complete rate 53%, extra rounds after all relevant documents are found of 4.89, follow-format rate 95%, premature-total rate 0%, total generated tokens 5775.75, and 386.03 tokens per round. GPT-5-zeroshot records 9.98 search rounds, 116.00 s duration, 52% search-complete rate, 2.18 extra rounds, 100% follow-format rate, 1% premature-total rate, 7184.10 total tokens, and 615.99 tokens per round [2509.21998].

Weaker models tend to exhibit shorter and shallower search. Gemini-2.5-Pro-zeroshot reaches 38.33% accuracy with 2.93 search rounds and 25% search-complete rate. Kimi-K2-Instruct-zeroshot reaches 37.42% with 5.41 search rounds and 24% search-complete rate. Llama-4-Scout-zeroshot reaches 12.54% with 2.07 search rounds and 9% search-complete rate [2509.21998].

The reported failure modes are structured rather than random. They include premature commitment, shallow or unfocused search, lack of revisit, and post-retrieval integration errors. Interaction-time scaling alone is limited: increasing tool calls produces only marginal gains for many open models, whereas GPT-5 shows stronger scaling behavior [2509.21998]. This suggests that the bottleneck is often policy quality rather than budget alone.

## 5. Revisiting behavior and tool-augmented test-time scaling

The benchmark’s central mechanistic finding is that **revisiting previously visited nodes** is strongly associated with higher accuracy. Figure 3 reports a strong positive correlation between revisit ratio and accuracy, while exploitation correlates negatively with accuracy [2509.21998]. The contrast between high-performing and low-performing systems is pronounced: o3-zeroshot has `hasRvst` 70.13% and `Rvst` 24.56%, whereas DeepSeek-V3-zeroshot has both values at 0.00%, and GPT-4o-zeroshot has `hasRvst` 2.03% and `Rvst` 0.62% [2509.21998].

The paper interprets revisit as important because relevant documents are distributed across clusters, so an agent often needs to return to an earlier topic or query family to integrate premises across multiple parts of the problem. Without revisit, the trajectory becomes a shallow one-pass scan that is prone to stopping early or to misintegrating partial evidence [2509.21998].

To exploit this finding, GSM-Agent introduces a **tool-augmented test-time scaling** scheme. Three additional tools are proposed. `Thinking(·)` copies preceding tokens to enforce reflection. `Explore(x)` is identical to Search(x) but framed to encourage movement toward new topics. `Revisit(x)` is identical to Search(x) but framed to encourage return to prior topics with refined queries. The method operates through prompt shaping rather than through an explicit policy equation [2509.21998].

The gains are model-dependent but sometimes large. For Kimi-K2-Instruct, zero-shot yields 37.42%, `explore` 44.00%, `revisit` 45.61%, `explore+revisit` 45.67%, `think_tool` 46.00%, and CoT 46.00%. For Qwen3-235B, zero-shot yields 19.30%, CoT 31.00%, `explore` 42.21%, `revisit` 45.68%, `explore+revisit` 41.50%, and `think_tool` 37.79%; the improvement of `revisit` over zero-shot is +26.38%. Llama-4-Scout improves from 12.54% zero-shot to 19.39% with `explore+revisit` and 19.33% with `think_tool`. Llama-4-Maverick is an exception in which `think_tool` improves performance to 22.33%, while pure `revisit` falls to 11.07% and CoT to 8.67% [2509.21998].

Figure 5(b) shows a strong relationship between $\Delta(\text{revisit ratio})$ and $\Delta(\text{accuracy})$ across strategies [2509.21998]. The paper therefore treats revisit not merely as a descriptive statistic but as a mechanistically informative variable for intervention.

## 6. Positioning, limitations, and significance

GSM-Agent is positioned against broader agent benchmarks such as SWE-bench, TAU, ToolSandbox, AppWorld, and Berkeley benchmarks. Its distinguishing feature is a controllable environment paired with simple grade-school tasks, enabling closer comparison between static and agentic reasoning and avoiding confounding with expert knowledge or more complex mathematics [2509.21998].

Several limitations are explicit. The documents and premises are generated or processed by LLMs, including Claude-3.5-Sonnet during data construction, so bias or occasional inconsistency may remain despite independence verification and filtering. The default evaluation uses only Search and NextPage, leaving richer tool configurations unexplored. Evaluation requires exact numeric answers and does not assign partial credit. Search-complete rates are often low, and the ambiguity around the reported “32,315 unique problems” remains unresolved. The paper also does not provide release URLs or explicit licensing details [2509.21998].

Ablations nonetheless indicate some robustness. Replacing text-embedding-3-large with text-embedding-3-small or all-MiniLM-L6-v2 changes metrics only marginally and preserves relative model ranking; for o3, accuracy drops from 68.46% with the default embedding to 65.00% with the small embedding and 56.00% with MiniLM. Database size also matters predictably: smaller databases reduce retrieval difficulty and improve results, with o3-zeroshot moving from 68% on Full to 81% on Medium and 80% on Small, and Grok-4-zeroshot moving from 53% to 61% to 69% [2509.21998].

The benchmark’s broader significance lies in its separation of information-seeking policy from raw reasoning difficulty. This suggests that a large part of current agent failure arises not because models cannot perform the final arithmetic once premises are present, but because they do not reliably decide when to search, what to search for, and when to return to previously explored regions. The benchmark and its graph formalism are therefore aimed at future work on richer toolsets, verification mechanisms, stronger distractor controls, and training methods that explicitly optimize path quality rather than only final-answer accuracy [2509.21998].

Source: https://www.emergentmind.com/topics/gsm-agent