GSM-Agent: LLM Agentic Reasoning Benchmark

• GSM-Agent is a benchmark for agentic reasoning that converts grade-school math problems into tool-use tasks by requiring proactive retrieval of hidden premises.
• It maps tool interactions to a formal graph, analyzing exploration, exploitation, and revisit behaviors to better understand LLM search policies.
• Test-time scaling with augmented tools, such as 'revisit', improves retrieval accuracy and exposes limitations in current LLM information-seeking strategies.

GSM-Agent is a benchmark and analysis framework for agentic reasoning in LLMs. It converts grade-school math problems into tool-use tasks by presenting only the question while withholding the premises needed for solution; those premises are embedded in a controllable document environment, and the agent must proactively retrieve them through search before producing a numeric answer. The framework combines benchmark construction, a formal graph representation of agent trajectories, and tool-augmented test-time scaling methods intended to improve revisit behavior during search (Zhu et al., 26 Sep 2025).

1. Definition and nomenclature

In GSM-Agent, the central object of study is agentic reasoning: the coupled process of deciding what information to seek, using tools to gather it, and reasoning over the retrieved material to solve a task. The benchmark is designed to separate this capability from advanced mathematics, expert knowledge, or program synthesis by using problems whose underlying reasoning is at the grade-school level, but whose required premises are omitted from the prompt and must be recovered from the environment (Zhu et al., 26 Sep 2025).

The name is potentially ambiguous. In earlier arXiv literature, “GSM” commonly referred to unrelated domains, including Global System for Mobile Communications in localization and SMS-enabled control systems (Ibrahim et al., 2011, 0904.3715, Rocha et al., 15 Jun 2026, Mustafa et al., 2020, Teymourzadeh et al., 2018), and generalized spatial modulation in massive MIMO (Gecgel et al., 2019). In GSM-Agent, by contrast, “GSM” refers to the grade-school math source tasks from which the benchmark is derived (Zhu et al., 26 Sep 2025).

This distinction matters because a common misconception is to read GSM-Agent as a telecommunications or wireless-systems term. In the benchmark literature, it instead denotes a controlled evaluation setting for LLM agents. The benchmark’s motivating empirical result is that even when the underlying task is simple arithmetic or word-problem reasoning, agentic performance remains far from saturated: GPT-5 achieves 66.78% accuracy, while the best reported model, o3, reaches 68.46% (Zhu et al., 26 Sep 2025).

2. Task formulation and controllable environment

Each benchmark instance is formalized as a task

$T = (q, (p_1,\dots,p_k), a),$

where $q$ is the question, $p_1,\dots,p_k$ are the premises, and $a$ is the ground-truth numeric answer. In the static version of the task, the premises are provided directly. In GSM-Agent, the agent sees only $q$ and must recover the missing premises from an environment

$E = \{D_1,\dots,D_m\},$

whose documents contain the needed facts in context-rich form (Zhu et al., 26 Sep 2025).

The tool interface is deliberately minimal. Agents receive two tools, $$F=\{\mathrm{Search}(\cdot), \mathrm{NextPage}(\cdot)\}$$. A Search call returns the top 5 most relevant documents to a query, and NextPage returns the next 5 results for the current search. For each Search call, an agent may invoke NextPage up to 19 times, so a single query can expose up to the top 100 documents (Zhu et al., 26 Sep 2025).

Environment construction proceeds in five stages. First, data preprocessing is used to avoid ambiguity in a shared database through entity detection, name assignment for generic entities, and timestamp assignment via graph coloring. Second, problems are decomposed into self-contained premises and rephrased questions. Third, a hierarchical document-generation process creates coherent documents; independence verification is used to ensure documents do not leak information across premises, and 30% of documents are randomly anonymized to discourage direct “cheat” queries. Fourth, problems are filtered to keep only those solvable when all documents are provided. Fifth, a Chroma database is built with text-embedding-3-large as the default embedding model (Zhu et al., 26 Sep 2025).

After filtering, the dataset contains 7,323 problems. The paper also reports “with 32,315 unique problems stored in the chroma database,” and explicitly notes that this appears inconsistent with the prior count and is not clarified in the text (Zhu et al., 26 Sep 2025). The benchmark is provided in three difficulty-controlled sizes: GSM-Agent-Full, GSM-Agent-Medium at 25% of problems, and GSM-Agent-Small at 6.25% of problems. Appendix statistics report, for the Full setting, a mean of 4.41 documents per problem, standard deviation 1.74, minimum 1, maximum 14, and median 4; the mean problem “span,” defined as unique cluster classes per problem, is approximately 2.89, with median 3 (Zhu et al., 26 Sep 2025).

The controllability of the environment is a defining methodological feature. By tuning database size and distractor density while keeping the underlying mathematical reasoning simple, the benchmark isolates search policy, retrieval sequencing, and information integration from raw symbolic difficulty. This suggests that GSM-Agent is intended less as a test of arithmetic competence than as a diagnostic of search-conditioned reasoning.

3. Agentic reasoning graph

The benchmark’s main analytical contribution is the agentic reasoning graph, a discrete representation of the environment and of the trajectory induced by tool use. Let $E=\{D_i\}_{i=1}^N$ be the document set, let $e_\theta(\cdot)$ be the embedding model, and let $e_i=e_\theta(D_i)\in\mathbb{R}^d$ denote document embeddings. K-means is applied with $q$0 clusters, yielding clusters $q$1 and centroids $q$2, which define graph nodes $q$3 (Zhu et al., 26 Sep 2025).

The clustering objective is

$q$4

where $q$5 is the centroid of cluster $q$6. A tool-use trace is then mapped to a node sequence. If the $q$7-th call is $q$8, the node is

$q$9

where $p_1,\dots,p_k$0 is the embedding of the query prompt. If the $p_1,\dots,p_k$1-th call is $p_1,\dots,p_k$2, then $p_1,\dots,p_k$3. The resulting path is $p_1,\dots,p_k$4 (Zhu et al., 26 Sep 2025).

Three step types are defined over this path. Exploration occurs when a node is visited for the first time; exploitation occurs when the current step remains at the immediately previous node; revisit occurs when the current node was visited earlier but is not the immediately previous node. The formal revisit rate is

$p_1,\dots,p_k$5

Appendix metrics also include $p_1,\dots,p_k$6, the number of unique nodes visited per trace, and hasRvst, the proportion of traces containing at least one revisit step (Zhu et al., 26 Sep 2025).

The graph is intended to make search behavior interpretable. A t-SNE visualization of nodes is reported to show semantically coherent clusters, including Node 29 (“Grocery Receipts”), Node 54 (“Fruit Counting”), and Node 100 (“Youth Sports Stats”) (Zhu et al., 26 Sep 2025). This provides a topology in which agent behavior can be analyzed in terms of movement among semantic regions rather than only in terms of token-level traces or raw document IDs.

4. Empirical performance and observed failure modes

Evaluations use a LangChain ReAct agent with temperature 0.4 and max tokens 4096. Zero-shot prompting is reported as the most stable configuration overall, although Claude-4-sonnet benefits from few-shot prompting (Zhu et al., 26 Sep 2025).

The main performance picture is notable because the benchmark’s underlying reasoning is intentionally elementary, yet agentic accuracy remains modest. The “solvable by any model” rate is 88.00%, indicating that a large fraction of tasks are not intrinsically pathological, but that models vary sharply in how effectively they search and integrate premises (Zhu et al., 26 Sep 2025).

Model	Accuracy	Revisit profile
o3-zeroshot	68.46%	hasRvst 70.13%; Rvst 24.56%
GPT-5-zeroshot	66.78%	hasRvst 46.47%; Rvst 16.81%
Grok-4-zeroshot	53.00%	hasRvst 64.88%; Rvst 17.77%
GPT-4o-zeroshot	22.67%	hasRvst 2.03%; Rvst 0.62%
DeepSeek-V3-zeroshot	19.42%	hasRvst 0.00%; Rvst 0.00%

The broader table of results adds search-process metrics. o3-zeroshot uses 13.33 search rounds on average, has duration 117.85 s, search-complete rate 53%, extra rounds after all relevant documents are found of 4.89, follow-format rate 95%, premature-total rate 0%, total generated tokens 5775.75, and 386.03 tokens per round. GPT-5-zeroshot records 9.98 search rounds, 116.00 s duration, 52% search-complete rate, 2.18 extra rounds, 100% follow-format rate, 1% premature-total rate, 7184.10 total tokens, and 615.99 tokens per round (Zhu et al., 26 Sep 2025).

Weaker models tend to exhibit shorter and shallower search. Gemini-2.5-Pro-zeroshot reaches 38.33% accuracy with 2.93 search rounds and 25% search-complete rate. Kimi-K2-Instruct-zeroshot reaches 37.42% with 5.41 search rounds and 24% search-complete rate. Llama-4-Scout-zeroshot reaches 12.54% with 2.07 search rounds and 9% search-complete rate (Zhu et al., 26 Sep 2025).

The reported failure modes are structured rather than random. They include premature commitment, shallow or unfocused search, lack of revisit, and post-retrieval integration errors. Interaction-time scaling alone is limited: increasing tool calls produces only marginal gains for many open models, whereas GPT-5 shows stronger scaling behavior (Zhu et al., 26 Sep 2025). This suggests that the bottleneck is often policy quality rather than budget alone.

5. Revisiting behavior and tool-augmented test-time scaling

The benchmark’s central mechanistic finding is that revisiting previously visited nodes is strongly associated with higher accuracy. Figure 1 reports a strong positive correlation between revisit ratio and accuracy, while exploitation correlates negatively with accuracy (Zhu et al., 26 Sep 2025). The contrast between high-performing and low-performing systems is pronounced: o3-zeroshot has hasRvst 70.13% and Rvst 24.56%, whereas DeepSeek-V3-zeroshot has both values at 0.00%, and GPT-4o-zeroshot has hasRvst 2.03% and Rvst 0.62% (Zhu et al., 26 Sep 2025).

The paper interprets revisit as important because relevant documents are distributed across clusters, so an agent often needs to return to an earlier topic or query family to integrate premises across multiple parts of the problem. Without revisit, the trajectory becomes a shallow one-pass scan that is prone to stopping early or to misintegrating partial evidence (Zhu et al., 26 Sep 2025).

To exploit this finding, GSM-Agent introduces a tool-augmented test-time scaling scheme. Three additional tools are proposed. Thinking(·) copies preceding tokens to enforce reflection. Explore(x) is identical to Search(x) but framed to encourage movement toward new topics. Revisit(x) is identical to Search(x) but framed to encourage return to prior topics with refined queries. The method operates through prompt shaping rather than through an explicit policy equation (Zhu et al., 26 Sep 2025).

The gains are model-dependent but sometimes large. For Kimi-K2-Instruct, zero-shot yields 37.42%, explore 44.00%, revisit 45.61%, explore+revisit 45.67%, think_tool 46.00%, and CoT 46.00%. For Qwen3-235B, zero-shot yields 19.30%, CoT 31.00%, explore 42.21%, revisit 45.68%, explore+revisit 41.50%, and think_tool 37.79%; the improvement of revisit over zero-shot is +26.38%. Llama-4-Scout improves from 12.54% zero-shot to 19.39% with explore+revisit and 19.33% with think_tool. Llama-4-Maverick is an exception in which think_tool improves performance to 22.33%, while pure revisit falls to 11.07% and CoT to 8.67% (Zhu et al., 26 Sep 2025).

Figure 2(b) shows a strong relationship between $p_1,\dots,p_k$7 and $p_1,\dots,p_k$8 across strategies (Zhu et al., 26 Sep 2025). The paper therefore treats revisit not merely as a descriptive statistic but as a mechanistically informative variable for intervention.

6. Positioning, limitations, and significance

GSM-Agent is positioned against broader agent benchmarks such as SWE-bench, TAU, ToolSandbox, AppWorld, and Berkeley benchmarks. Its distinguishing feature is a controllable environment paired with simple grade-school tasks, enabling closer comparison between static and agentic reasoning and avoiding confounding with expert knowledge or more complex mathematics (Zhu et al., 26 Sep 2025).

Several limitations are explicit. The documents and premises are generated or processed by LLMs, including Claude-3.5-Sonnet during data construction, so bias or occasional inconsistency may remain despite independence verification and filtering. The default evaluation uses only Search and NextPage, leaving richer tool configurations unexplored. Evaluation requires exact numeric answers and does not assign partial credit. Search-complete rates are often low, and the ambiguity around the reported “32,315 unique problems” remains unresolved. The paper also does not provide release URLs or explicit licensing details (Zhu et al., 26 Sep 2025).

Ablations nonetheless indicate some robustness. Replacing text-embedding-3-large with text-embedding-3-small or all-MiniLM-L6-v2 changes metrics only marginally and preserves relative model ranking; for o3, accuracy drops from 68.46% with the default embedding to 65.00% with the small embedding and 56.00% with MiniLM. Database size also matters predictably: smaller databases reduce retrieval difficulty and improve results, with o3-zeroshot moving from 68% on Full to 81% on Medium and 80% on Small, and Grok-4-zeroshot moving from 53% to 61% to 69% (Zhu et al., 26 Sep 2025).

The benchmark’s broader significance lies in its separation of information-seeking policy from raw reasoning difficulty. This suggests that a large part of current agent failure arises not because models cannot perform the final arithmetic once premises are present, but because they do not reliably decide when to search, what to search for, and when to return to previously explored regions. The benchmark and its graph formalism are therefore aimed at future work on richer toolsets, verification mechanisms, stronger distractor controls, and training methods that explicitly optimize path quality rather than only final-answer accuracy (Zhu et al., 26 Sep 2025).