GTA-Workflow Benchmark Suite

• GTA-Workflow is a long-horizon evaluation framework defining tasks as composite deliverables beyond simple, short atomic tool chains.
• The benchmark employs a recursive checkpoint scoring mechanism to assess planning, state management, error recovery, and adaptive sub-goal decomposition.
• Empirical results reveal a steep capability cliff in current LLM agents, emphasizing the need for enhanced execution harnesses and detailed feedback.

GTA-Workflow denotes the long-horizon, open-ended workflow evaluation setting introduced as part of the GTA-2 benchmark suite. GTA-Workflow is intended for benchmarking general tool-using agents under realistic productivity task distributions, moving beyond short atomic tool chains to complex, composite deliverables that more closely reflect real-world applications. In this framework, a task is the production of a heterogeneous artifact (e.g., a multi-section PDF, a slide presentation, a multimedia report) from a raw set of files and a natural user request, utilizing a specified suite of API-accessible tools. GTA-Workflow is architected to evaluate not only model-level tool-use precision but also the orchestration and execution capabilities of agent frameworks—such as planning, state management, error recovery, and adaptive sub-goal decomposition—across extended task horizons (Wang et al., 17 Apr 2026).

1. Conceptual Foundations and Motivation

GTA-Workflow arises from recognition that benchmarks for general tool agents must reflect the open-ended, multi-modal, multi-step nature of real productivity workflows. Existing tool-use evaluations (cf. GTA-Atomic (Wang et al., 2024)) focus on closed-ended, “atomic” tasks—few-step tool chains with single reference outputs. However, practical AI assistants must execute dynamic plans, integrate across multimodal inputs, and synthesize diverse deliverables without fixed scripts or templates. GTA-Workflow addresses these requirements by defining a benchmark in which each task specifies a workflow objective (e.g., “prepare a ten-slide PDF synthesizing findings from five research papers,” or “extract, denoise, and summarize a 3-minute audio section from a podcast”), along with an input context, a toolset, and a recursive checkpoint tree for outcome-driven evaluation (Wang et al., 17 Apr 2026).

2. Task Definition and Tool Universe

Each GTA-Workflow task is formalized as a quadruple $(F, Q, T, \mathit{Cpt})$, where:

• $F$: a set of input files (potentially including images, videos, audio, DOCX/XLSX/PDF/PPTX).
• $Q$: a real user-authored natural language query describing the composite deliverable.
• $T$: a set of 37 deployed APIs covering perception (e.g., image/audio/video analysis), document and structured data operations (e.g., ReadPDF, CsvFileGenerator), reasoning (e.g., Prover via Z3), and creative generation (e.g., TextToImage, TextToVideoTool).
• $\mathit{Cpt}$: a hierarchical tree of verifiable, outcome-oriented checkpoints, assigning weights to each sub-goal.

Unlike atomic tasks, for which there exists a fixed ground-truth tool chain of length $m$, GTA-Workflow tasks are open-ended: the deliverable $D$ may be achieved via many valid execution paths, and success is defined extensionally by the satisfaction of the sub-goals in $\mathit{Cpt}$, not by strict stepwise matching (Wang et al., 17 Apr 2026).

3. Recursive Checkpoint-Based Evaluation

GTA-Workflow employs a recursive checkpoint scoring mechanism to assess deliverables. Each node in the checkpoint tree $\mathit{Cpt}$ is either a composite sub-goal (internal) or an atomic outcome (leaf). The evaluation proceeds as follows:

1. Traverse the tree, for each node $n$:
   • If $F$0 is a leaf, invoke a powerful LLM judge $F$1 (e.g., GPT-5.2) with $F$2; assign a score $F$3.
   • For internal nodes, recursively aggregate child scores: $F$4, with $F$5 the normalized weights and $F$6 the children.
2. The root score $F$7 is the workflow deliverable score. Success is binary: the workflow “passes” if $F$8, where $F$9 by default.

Leaf Success Rate (fraction of checkpoints with $Q$0) and Tool Success Rate (fraction of tool invocations running without execution errors) are also reported (Wang et al., 17 Apr 2026).

4. Dataset Construction and Task Diversity

The GTA-Workflow corpus includes 132 workflow tasks. Source diversity is ensured by:

• Sourcing ~50% of workflows from production agent platforms (Manus, Minimax Agent, Kortix, Flowith, CrewAI), thus capturing authentic user queries and actual business or productivity workflows.
• Curating the remainder from high-engagement Stack Exchange and Reddit queries, which are then further refined by LLMs under human supervision.
• Each workflow specifies 3–19 checkpoints, elaborating requirements on structure, correctness, and presentation of the final artifact.

Examples illustrate the spectrum: scientific reviews, data analysis with tabular visualization, multimedia extraction, document synthesis, chaining of perception, reasoning, and generation tools. Table 2 in (Wang et al., 17 Apr 2026) details the filtration and selection process, starting from 154 candidates and resulting in 132 finalized workflows.

5. Experimental Analysis and Performance Results

Empirical evaluation demonstrates a pronounced “capability cliff” in contemporary tool agents:

• While leading LLMs and frameworks (e.g., Gemini-2.5-Pro, GPT-5, Claude-Sonnet-4.5) maintain high Tool Success Rates (e.g., 91.2% for Gemini-2.5-Pro), Root Success Rates and average $Q$1 drop sharply in GTA-Workflow: maximum observed Root SR is 14.39%, with $Q$2 for Gemini-2.5-Pro and similarly low for others.
• In contrast, the same models achieve >40% end-to-end solution rates on GTA-Atomic (short, closed tasks).
• This outcome demonstrates that correct atomic tool use is necessary but insufficient for robust workflow completion, with failures driven by inadequate state tracking, error recovery, and sub-goal decomposition (Wang et al., 17 Apr 2026).

Advancements in execution harnesses (e.g., OpenClaw, Manus, Kortix) offer substantial improvement. OpenClaw increases Root SR from 0% to 50% (S_root from 2.49 to 6.82) for a fixed base LLM. Manus and Kortix achieve over 53% in subset analysis, indicating that harness design (system-level memory, persistent state, advanced error handling) is a key determinant of actual workflow success, surpassing differences in base LLM reasoning when Tool SR is saturated.

6. Feedback, Diagnostics, and Future Directions

The impact of feedback granularity is quantified:

• Generic “coarse” feedback during re-tries yields only a 4% relative gain in $Q$3, while detailed checkpoint diagnostics raise $Q$4 by 12% (from 2.83 to 3.15).
• This demonstrates that fine-grained, task-decomposed feedback is critical to guiding agent improvement on workflow-scale problems (Wang et al., 17 Apr 2026).

Recommended future directions include:

• Extending evaluation beyond deliverable quality to encompass safety and governance.
• Rigorous system-level ablations across model types and harnesses to distinguish execution and reasoning contributions.
• Refinement of checkpoint taxonomy for causal modeling of error pathways and more robust diagnostic signals.
• Releasing both raw and reformulated task pairs to control for bias in benchmark construction.

7. Significance and Implications

GTA-Workflow establishes a benchmark for the next generation of agent evaluation: its open-ended, multimodal, verifiably-scored workflows expose bottlenecks not only in LLM reasoning but, more critically, in the orchestration, planning, and execution machinery underpinning modern tool agents. The empirical gap revealed by GTA-Workflow argues that future progress in personal and professional AI assistants depends as much on the architecture of execution frameworks as on scaling model capacity per se. As such, GTA-Workflow quantifies the steep drop from atomic precision to reliable end-to-end workflows, and provides a framework for systematically closing this gap via improved harnesses, feedback granularity, and workflow planning methodologies (Wang et al., 17 Apr 2026).