WebTailBench: Web Interaction Benchmark

• WebTailBench is a long-horizon web interaction benchmark that rigorously evaluates vision-first computer use agents performing complex tasks in realistic browser environments.
• It offers a diverse suite of tasks spanning 11 categories—including transactional, informational, and explicit refusal tests—with clear success criteria and automated verification.
• The benchmark supports both native and multi-agent coordination, enabling comparative analysis of model efficiency, error recovery, and token cost reduction.

WebTailBench is a long-horizon web interaction benchmark designed for the rigorous evaluation of vision-first computer use agents (CUAs) performing complex, goal-directed tasks in realistic browser environments. Introduced to address key shortcomings in earlier web-agent benchmarks, WebTailBench provides a diverse, hand-verified suite of live-site tasks spanning transactional and informational goals, explicit refusal tests, and robust automated evaluation protocols. It has catalyzed the development and comparative analysis of native CUA models, multi-agent orchestration methods, and efficiency-driven vision-processing algorithms.

1. Motivation and Design Objectives

WebTailBench was developed to fill critical gaps observed in standard web agent benchmarks. Most prior work limited evaluation to navigation or form-filling on a few static domains, relied on DOM-based actions, or accepted ill-specified “browse”/“find” guidance, resulting in ambiguous goal completion and poor real-world coverage. WebTailBench remedies these issues by:

• Broadening coverage to 11 task categories, including shopping, flights, hotels, restaurants, activities, ticketing, real estate, jobs, multi-item lists, price comparisons, and compositional reasoning tasks.
• Emphasizing multi-site and multi-step workflow tasks to stress longer-range planning, cross-domain knowledge aggregation, and robust error recovery.
• Ensuring each task has objectively specified success criteria, supporting unambiguous automated and human verification.
• Requiring agents to engage with live, high-traffic sites, reflecting dynamically changing web environments.
• Including a refusal segment—111 tasks spanning seven safety-sensitive categories—to measure safe agent refusal on harmful or prohibited requests.
• Providing hand-verified instructions and a matched LLM-based judge pipeline aligned to human assessments (Awadallah et al., 24 Nov 2025).

2. Dataset Composition and Properties

WebTailBench consists of 609 primary evaluation tasks plus 111 “Refusals” tasks. The benchmark’s 11 core segments are distributed as follows:

Segment	Task Count
Shopping	56
Flights	51
Hotels	52
Restaurants	52
Activities	80
Ticketing	57
Real Estate	48
Jobs/Careers	50
Shopping Lists (2 items)	51
Comparison Shopping	57
Compositional Tasks	55

Tasks are derived using targeted URL prompts, agent-driven domain exploration, and template-based LLM expansion. Each scenario is checked for achievability (e.g., absence of paywalls), clarity, real-world relevance, and automatic verifiability. Human annotators validate that instructions remain actionable and site content supports the required workflow. The “Refusals” subset includes policy violations ranging from illegal activity to misinformation and is curated for coverage across potentially risky domains (Awadallah et al., 24 Nov 2025).

3. Environment, Interaction Protocol, and Success Criteria

WebTailBench is executed in the OSWorld desktop environment, simulating browser-based workflows with full-visual input. At each step, the agent obtains a full-screen screenshot (encoded as 2,769 ViT-style patches in some settings), the current URL, and a complete text history of previous actions and internal reasoning.

The agent’s action space (in both pixel-action and discrete-tokenized variants) consists of:

Action	Description
click(x, y)	Left-click at pixel coordinate (x, y)
type(str, x, y)	Enter string str at (x, y)
key_press(k)	Press keyboard key or combination
move_mouse(x, y)	Move/hover cursor to (x, y)
scroll(dir)	Scroll web page
visit_url(url)	Navigate to new URL
web_search(q)	Query a web search engine
history_back	Go back one page
memorize(fact)	Store extracted fact in agent memory
wait(sec)	Pause for specific duration
terminate	End the episode and return output

Episodes are deemed successful if, within a capped number of steps (typically 100), the agent’s trajectory results in a goal-aligned state according to a GPT-4o or o4-mini LLM judge, further corroborated by multimodal screenshot analysis. Success rate is computed as the proportion of tasks correctly completed out of attempted episodes. Both macro- (per-segment) and micro- (aggregate) means are reported over three independent runs (Awadallah et al., 24 Nov 2025, Abaskohi et al., 11 May 2026).

4. Task Structure, Complexity, and DAG-Based Decomposition

WebTailBench tasks range from atomic, single-site goals to highly compositional, multi-site information retrieval and cross-domain aggregation scenarios. Notable subtask architectures include:

• Map-Reduce: Parallel lookups across multiple domains with subsequent aggregation (e.g., price comparison).
• Retry Chains: Serialized attempts or fallbacks when workflows encounter obstacles (e.g., 404 pages).
• Compositional Pipelines: Sequential subgoals with chained dependencies (e.g., gather information from separate sites, then combine for a final answer).

The benchmark supports evaluation of multi-agent computer use (MACU) models utilizing a directed acyclic graph (DAG) decomposition, where the manager decomposes each task into subnodes of two types:

• CUA-subtask node (agent_type="cua"): Executes granular browser actions per manager instructions.
• Manager node (agent_type="manager"): Performs reasoning or aggregation over completed subtask results.

Dependencies are explicitly encoded as edges, forming a workflow that supports continuous parallel execution and runtime re-planning. A node is ready for dispatch when all its parents have completed. The manager dispatches $N$ subtasks in parallel, collects responses, and revises the DAG subject to a replan budget $B$, maintaining acyclicity (Koh et al., 1 Jun 2026).

5. Quantitative Results and Comparative Analysis

Multiple studies report on WebTailBench, spanning native CUAs, multi-agent orchestration, and visual token efficiency. The table summarizes selected findings.

Model / Setup	Success Rate (%)	Cost per Task (\$)	Key Properties
SoM Agent (GPT-5)	60.4	0.595	Frontier closed model, highest accuracy
SoM Agent (o3)	52.7	0.948
Fara-7B (7B)	38.4	0.069	Pixel-in/action-out, data from FaraGen
UI-TARS-1.5-7B (7B)	19.5	0.133	Native CUA baseline
OpenCUA (k=3)	36.2 (SR@100)	~0.12*	Specialized UI agent
Qwen2.5-VL-7B (hist=5)	10.1	n/a	No-drop baseline
ReVision (Qwen2.5-VL-7B, k=5)	40.2	—	Temporal redundancy reduction (+30.7% over base)

*: Not directly stated as cost, but implied from token/step usage.

Fara-7B outperforms other 7B-scale native CUA models (e.g., UI-TARS) by a wide margin (38.4% vs. 19.5%), at roughly half the per-task cost (Awadallah et al., 24 Nov 2025). Multi-agent orchestration (MACU) on WebTailBench-v2, using a manager model (Opus4.6), achieves a statistically significant $+8.7$ percentage point gain in success rate over single-agent baselines (29.5% vs. 20.8%, $p<0.01$), with the largest improvements observed for map-reduce tasks (+30.2 pp price comparison), flights (+20.0 pp), and compositional pipelines (+17.8 pp) (Koh et al., 1 Jun 2026). Visual token efficiency techniques (ReVision) further enable higher history horizons, raising SR@100 to 40.2% and reducing token cost by $\sim$37% (Abaskohi et al., 11 May 2026).

6. Evaluation Pipeline, Judging Alignment, and Limitations

Agents are evaluated in Playwright-driven headless browser environments under live-site conditions with robust error-handling: up to five fresh-browser retries if page-load errors occur. The average trajectory on WebTailBench (Fara-7B) comprises 41.1 $\pm$ 33.1 actions, 343k $\pm$ 323k input tokens, and 2.4k $\pm$ 1.9k output tokens. LLM judges (GPT-4o, o4-mini) are prompted with rubric-based instructions, achieving approximately 83% agreement with human raters (17% false positives, 18% false negatives) (Awadallah et al., 24 Nov 2025).

Key limitations include the high inference-time cost for multi-agent runs (LLM and VM overhead), challenging engineering requirements (state management, VM cloning), and concentrated gains in tasks with decomposable parallel structure. Tasks that are inherently serial may see decreased speed or efficiency. The benchmark abstracts away certain real-world deployment concerns (authentication, irreversible actions), suggesting future systems will need integrated safeguards, adaptive orchestration, and robust serialization (Koh et al., 1 Jun 2026).

7. Research Impact and Future Directions

WebTailBench has reshaped the methodology for evaluation and training of visually grounded web agents by introducing:

• A comprehensive, cross-domain benchmark that supports systematic studies of planning, memory, and multi-step reasoning.
• A platform for comparative analysis of pipeline efficiency (e.g., token usage, step economy), model scale, and data synthesis quality.
• Explicit challenges for multi-agent coordination, data efficiency, refusal handling, and visual redundancy reduction.

Future directions identified by primary studies include adaptive adjustment of parallelism/replanning budgets to task structure, meta-controllers for orchestrator selection, improved state/provenance tracking, and tighter coupling between agentic models and real-world safety constraints (Koh et al., 1 Jun 2026, Awadallah et al., 24 Nov 2025, Abaskohi et al., 11 May 2026).