GTA-2: Hierarchical Benchmark for Tool Agents
- GTA-2 is a hierarchical benchmark that evaluates large-language-model agents on both short-horizon atomic tool use and complex, deliverable-centric workflows.
- It comprises two components—GTA-Atomic for closed-ended tasks and GTA-Workflow for open-ended productivity tasks—leveraging diverse, multimodal inputs and a wide array of tools.
- Experimental findings reveal a pronounced capability cliff, emphasizing the critical role of execution harnesses and recursive checkpoint feedback for end-to-end agent success.
GTA-2 is a hierarchical benchmark for General Tool Agents (GTA) that is designed to evaluate large-language-model-powered agents on both atomic tool use and open-ended productivity workflows, shifting assessment beyond isolated API calls or image-grounded reasoning toward end-to-end completion of realistic, long-horizon tasks (Wang et al., 17 Apr 2026). It combines two complementary tracks—GTA-Atomic and GTA-Workflow—under a framework grounded in real-world authenticity, multimodal inputs, and verifiable, deliverable-centric outcomes. The benchmark is motivated by the claim that current tool-use benchmarks remain misaligned with real-world requirements because they rely on AI-generated queries, dummy tools, and limited system-level coordination (Wang et al., 17 Apr 2026).
1. Benchmark definition and hierarchical organization
GTA-2 organizes evaluation into two levels. GTA-Atomic inherits the original GTA design and evaluates short-horizon, closed-ended tasks that require precise multi-step tool use. GTA-Workflow introduces 132 open-ended, long-horizon productivity tasks in which agents are judged on final deliverables rather than on matching a fixed tool-invocation trajectory (Wang et al., 17 Apr 2026).
This hierarchical organization is central to the benchmark’s stated purpose. GTA-Atomic targets end-to-end answer accuracy and tool-use precision under constrained conditions, whereas GTA-Workflow targets realistic end-to-end completion under tool, modality, and deliverable constraints. In the benchmark’s framing, these are complementary rather than substitutive evaluation regimes: one measures precision on atomic tool-use episodes, and the other measures system-level competence on open-ended workflows.
| Component | GTA-Atomic | GTA-Workflow |
|---|---|---|
| Task form | Short-horizon, closed-ended | Long-horizon, open-ended |
| Inputs | Real user query, often image plus question | Realistic multimodal inputs and attachments |
| Evaluation target | Ground-truth tool chain and final answer | Final deliverables and checkpoint scores |
The paper presents this as a unified testbed for measuring both the precision of atomic tool use and the competence required for complex productivity workflows. A plausible implication is that GTA-2 is intended not merely as a model benchmark, but as an evaluation framework for the coupled performance of models, planners, memories, and execution systems.
2. GTA-Atomic: task structure, tools, and evaluation focus
Each sample in GTA-Atomic consists of three elements: a real user query, often an image plus a question, drawn from human workflows; a small set of $1$–$4$ deployed tools selected from a pool of $14$ tools; and a ground-truth “tool chain” of $2$–$8$ steps recorded via a ReAct-style interaction together with a final answer (Wang et al., 17 Apr 2026). The emphasis is therefore on precise multi-step tool use under bounded action spaces.
The $14$ executable tools are organized into four categories. The Perception category includes OCR, ImageDescription, DetectGivenObject, and RegionAttributeDesc. The Operation category includes DrawBox, AddText, and GoogleSearch. The Logic category includes Calculator, Plot, Solver, MathOCR, and CountGivenObject. The Creativity category includes TextToImage and ImageStylization (Wang et al., 17 Apr 2026).
The source material for GTA-Atomic consists of real photographs, screenshots, and scanned documents from user-submitted problems. This design choice directly supports the benchmark’s claim of “real-world authenticity.” Rather than evaluating tool use on synthetic prompts or simulated interfaces, GTA-Atomic situates tool invocation in human-originated multimodal problem settings.
The reported experimental pattern is that GTA-Atomic remains difficult even for strong models. The best closed-source models, specifically GPT-4 and GPT-4o, achieve end-to-end answer accuracy AnsAcc below , and tool-selection across categories remains in the $60$– range at best (Wang et al., 17 Apr 2026). This result is presented as evidence that even short-horizon, closed-ended tool-use precision is not solved.
3. GTA-Workflow: open-ended workflows, sources, and deliverables
GTA-Workflow extends the benchmark from atomic tool use to open-ended productivity tasks. It contains $4$0 long-horizon workflows spanning business reporting, data analysis, multimedia production, travel planning, and research synthesis (Wang et al., 17 Apr 2026). In these settings, the benchmark does not require a single correct action trajectory. Instead, agents are assessed on the quality and completeness of the final deliverables.
The deliverables are explicitly artifact-centered. Examples given in the benchmark include a PDF report, an XLSX spreadsheet, a multi-slide PPTX deck, and a composite video (Wang et al., 17 Apr 2026). This deliverable-centric evaluation shifts the object of judgment from action trace correctness to end-product completion under realistic constraints of tool availability and multimodal input.
Task sourcing is broadened along two dimensions. First, the benchmark draws from agent platforms—Manus, Kortix, MiniMax Agent, Flowith, and CrewAI—for professional workflows. Second, it uses high-engagement community posts on Reddit and Stack Exchange for personal and niche needs (Wang et al., 17 Apr 2026). This sourcing strategy is intended to increase ecological validity by using tasks that arise in deployed agent ecosystems and in user communities rather than tasks synthesized exclusively for benchmarking.
The tool ecosystem is also expanded. GTA-Workflow uses $4$1 tools encompassing document I/O such as ReadDOCX, PdfFileGenerator, and PptxFileGenerator; audio/video processing such as SpeechToText, VideoObjectDetection, and AudioClipTool; structured data handling such as CsvFileGenerator and XlsxFileGenerator; and advanced logic via Prover using Z3 (Wang et al., 17 Apr 2026). Attachments include images, PDF, DOCX, PPTX, XLSX, audio clips, and video files, thereby reflecting what the paper describes as the multimodal complexity of real productivity scenarios.
A common misconception addressed implicitly by this design is that benchmark difficulty can be captured by tool invocation alone. GTA-Workflow instead encodes the requirement that the agent must transform heterogeneous inputs into externally inspectable deliverables. This suggests that file construction, cross-modal integration, and sustained state management are first-class components of agent competence rather than peripheral implementation details.
4. Recursive checkpoint-based evaluation
To evaluate open-ended deliverables, GTA-Workflow introduces a recursive checkpoint-based evaluation mechanism. Each task is decomposed into a tree of verifiable checkpoints, or sub-goals, with assigned weights. A strong model, GPT-5.2, acts as a judge that scores each leaf checkpoint $4$2 on a scale $4$3 based on the delivered artifact (Wang et al., 17 Apr 2026).
The recursive aggregation rule is
$4$4
so that the root score $4$5 reflects overall completion (Wang et al., 17 Apr 2026).
From these checkpoint scores, the benchmark derives three primary metrics. Root Success Rate (Root SR) is defined as
$4$6
typically with threshold $4$7. Leaf Success Rate (Leaf SR) is the fraction of leaf nodes with $4$8. Tool Success Rate (Tool SR) is the fraction of tool invocations that execute without error (Wang et al., 17 Apr 2026).
The paper emphasizes that this evaluation decouples benchmarking from a single “correct” trajectory. That design allows comparison across planning schemas, internal memories, and system architectures. In effect, GTA-Workflow treats planning diversity as admissible so long as the resulting artifacts satisfy the checkpoint tree. This is significant because it aligns evaluation with open-ended tasks in which multiple action sequences may be valid.
A plausible implication is that the checkpoint tree serves a dual role: it standardizes judgment for benchmarking while also exposing internal task structure that can be reused for agent refinement. The paper makes that implication explicit in later experimental analysis of feedback.
5. Experimental results and the capability cliff
The main experimental finding is a pronounced capability cliff between tool execution and full workflow completion. In GTA-Workflow, top models such as Gemini-2.5-Pro and GPT-5 maintain high Tool SR, approximately $4$9, but achieve Root SR of only $14$0 and $14$1, respectively (Wang et al., 17 Apr 2026). The excerpted main workflow results report: Gemini-2.5-Pro with Tool SR $14$2, Root SR $14$3, Root Score $14$4; GPT-5 with Tool SR $14$5, Root SR $14$6, Root Score $14$7; Qwen3-235B-A22B with Tool SR $14$8, Root SR $14$9, Root Score $2$0; and Llama-4-Scout with Tool SR $2$1, Root SR $2$2, Root Score $2$3 (Wang et al., 17 Apr 2026).
The contrast between high Tool SR and low Root SR is one of the benchmark’s central empirical claims. Smaller or older models are reported to collapse to near $2$4 success despite some correct tool calls (Wang et al., 17 Apr 2026). This directly challenges the assumption that robust tool invocation is sufficient for realistic agent performance.
On GTA-Atomic, the difficulty profile is different but still severe. Frontier systems already struggle on atomic tasks, with answer accuracy below $2$5 for the strongest closed-source models (Wang et al., 17 Apr 2026). Taken together, the two tracks indicate that failure appears at both levels: precise multi-step tool use is itself challenging, and workflow completion compounds that challenge through horizon length, multimodal state, and deliverable construction.
This empirical gap between Tool SR and Root SR is the benchmark’s most important analytical signal. It indicates that correct local actions do not automatically compose into globally successful workflows. The paper treats this as evidence that end-to-end agent reliability depends on more than model-level tool competence.
6. Execution harnesses and checkpoint-guided feedback
GTA-2 explicitly evaluates not only models but also execution frameworks, described as execution harnesses. In a controlled comparison using Claude-Sonnet-4.5 as the base LLM, Lagent, the default harness, attains Root SR $2$6 and Root Score $2$7, whereas OpenClaw raises these to Root SR $2$8 and Root Score $2$9 at moderate extra cost (Wang et al., 17 Apr 2026). In system-level settings, Manus and Kortix achieve comparable Root SR, approximately $8$0, and superior cost efficiency (Wang et al., 17 Apr 2026).
These results are used to support the paper’s claim that execution harnesses materially affect end-to-end performance. The underlying interpretation given in the benchmark is that robust planning, state tracking, error handling, and compositional orchestration are critical to complete deliverables. The benchmark therefore treats system architecture as an experimental variable rather than a hidden implementation choice.
Checkpoint-guided feedback also yields measurable gains. On a $8$1-task subset with GPT-5, coarse “try again” feedback raises Root Score by $8$2, while detailed checkpoint diagnostics improve it by $8$3 (Wang et al., 17 Apr 2026). The benchmark presents this as evidence that structured subgoal signals can guide iterative refinement.
The significance of these findings lies in the interaction between evaluation and control. Checkpoint trees are not merely passive scoring devices; they can provide actionable diagnostics. This suggests that benchmark annotations can function as a scaffold for agent improvement, especially in settings where agents must revise partial outputs rather than restart from scratch.
7. Research significance and implications for assistant design
The paper states two principal lessons for building reliable personal or professional assistants. First, “Model Capacity + System Design = End-to-End Success”: high-quality tool invocation, measured by Tool SR, is necessary but not sufficient, and advanced harnesses are critical for robust planning and orchestration (Wang et al., 17 Apr 2026). Second, “Verifiable Subgoal Structures Aid Both Evaluation and Iteration”: checkpoint trees enable scalable, consistent benchmarking of open-ended tasks and also serve as actionable feedback channels for agents to self-diagnose and refine outputs (Wang et al., 17 Apr 2026).
Within the benchmark’s own framing, GTA-2 is therefore not only a dataset but a methodology for evaluating agentic systems across two axes. One axis is the precision of atomic tool use. The other is system-level competence in long-horizon workflow execution. The benchmark argues that progress along both axes will be required to deliver reliable assistants for professional and everyday use (Wang et al., 17 Apr 2026).
A common misunderstanding that GTA-2 helps clarify is the idea that open-ended agent evaluation can be reduced either to synthetic task success or to trace-level correctness. GTA-2 instead centers real user queries, deployed tools, multimodal contexts, and deliverable-centric verification. This suggests a broader methodological position: realistic evaluation of general tool agents must jointly model authenticity of tasks, executability of tools, and verifiability of final artifacts.
The benchmark’s dataset and code are stated to be available at the OpenCompass GTA repository, and the paper positions GTA-2 as a realistic testbed for developing and comparing general-purpose agents under conditions closer to real productivity workflows than earlier tool-use benchmarks (Wang et al., 17 Apr 2026).