Graph2Eval-Bench: Agent Evaluation Dataset

• Graph2Eval-Bench is a curated benchmark that uses knowledge graphs to automatically generate 1,319 multimodal tasks for assessing dynamic agent capabilities.
• It unifies document comprehension and web interaction by organizing tasks via 12 templates and 7 meta-path patterns, enabling structured evaluation.
• Empirical results indicate that while multi-agent setups show marginal gains over single-agent models on documents, web tasks remain challenging due to interaction complexities.

Searching arXiv for Graph2Eval-Bench / Graph2Eval to ground the article in the relevant paper. Graph2Eval-Bench is a curated dataset of 1,319 tasks spanning document comprehension and web interaction scenarios, instantiated from Graph2Eval, a knowledge graph-based framework that automatically generates both multimodal document comprehension tasks and web interaction tasks, enabling comprehensive evaluation of agents' reasoning, collaboration, and interactive capabilities (Chen et al., 1 Oct 2025). It is designed for settings in which evaluation based on static datasets can no longer adequately assess the capabilities of multimodal LLM-driven agents in dynamic environments and diverse tasks. In the benchmark, knowledge graphs constructed from multi-source external data serve as the task space, and the resulting evaluation covers Single-Agent, Multi-Agent, and Web Agent configurations. A related but earlier naming antecedent appears in GraphEval36K, which suggested that instruction-based decomposition modules could be standardized as “plugins” in a Graph2Eval-Bench for systematic comparison of reasoning paradigms such as CoT, SSD, and retrieval-augmented methods (Wu et al., 2024).

1. Scope and conceptual positioning

Graph2Eval-Bench is centered on automatic multimodal task generation for agents via knowledge graphs rather than on static, manually authored evaluation sets. The motivating claim is that existing LLM-based synthetic data methods are largely designed for LLM training and evaluation and thus cannot be directly applied to agent tasks that require tool use and interactive capabilities. The framework therefore targets two evaluation regimes in a unified way: multimodal document comprehension and web interaction. This unification is operational rather than merely thematic, because all underlying content—documents and web pages—is treated as one heterogeneous task graph.

The benchmark is also explicitly agent-oriented. It supports end-to-end evaluation of multiple agent types—Single-Agent, Multi-Agent, and Web Agent—and measures reasoning, collaboration, and interaction capabilities. This distinguishes it from benchmarks that focus primarily on model-only QA or code generation. A plausible implication is that Graph2Eval-Bench should be read as an infrastructure for controlled stress-testing of agent architectures, where task difficulty, modality composition, and interaction topology are all exposed as benchmark variables rather than hidden nuisance factors.

2. Corpus, task inventory, and taxonomy

The benchmark contains 1,319 tasks in total: 1,002 document comprehension tasks and 317 web interaction tasks. Its source material comprises 16 research papers and 8 live websites, yielding approximately 83.5 tasks per document and approximately 48.4 tasks per website (Chen et al., 1 Oct 2025).

Component	Quantity	Notes
Document comprehension tasks	1,002	12 template-defined types
Web interaction tasks	317	7 meta-path patterns
Source documents / websites	16 / 8	Research papers and live websites

The document-comprehension portion is organized through 12 template-defined task types, including span extraction, question answering, comparison, analysis, multi-hop reasoning, fact verification, summarization, image interpretation, and cross-referencing. Difficulty levels are Easy, Medium, Hard, and Expert. Modalities include text-only instances, defined as pure paragraph or table inputs, and image-augmented instances, defined as figure plus OCR or caption. Reasoning depth is controlled by maximum hop distance in the subgraph, making the graph distance itself part of the task construction logic.

The web-interaction portion is organized around 7 meta-path patterns: Basic Navigation, Toast handling, Content retrieval, Search + Detail, Modal interaction, Business-data navigation, and Button-only flows. These tasks admit both single-step and multi-step execution paradigms, including linear and branching meta-path chains. The benchmark therefore does not reduce web evaluation to surface clicking accuracy; it models interaction traces as structured graph paths with typed actions and execution dependencies.

Agent assignments are separated by modality and task class. Single-Agent and Multi-Agent systems are used for document comprehension. The Single-Agent setup is RAG-based, combining text retrieval and LLM answering. The Multi-Agent setup comprises Planner, Retriever, Reasoner, Verifier, and Summarizer roles working collaboratively, with inter-agent communication and extra token cost recorded. Web tasks are evaluated with Web Agent configurations, specifically SoM Agent and Agent S 2.5.

3. Knowledge-graph formalization and generation pipeline

Graph2Eval treats all underlying content as a unified, heterogeneous knowledge graph $G = (V, E, R)$, where $V$ consists of semantic units such as paragraphs, headings, table cells, and figures, together with HTML elements such as forms, buttons, and links; $R$ contains structural, semantic, co-reference, navigation, click-trigger, and layout relation types; and $E \subseteq V \times V \times R$ encodes typed edges (Chen et al., 1 Oct 2025). Each node $v_i$ has textual content $c_i^T$ and visual content $c_i^V$, with visual information converted and concatenated as

$$c_i^{(T+V)} = c_i^T \,\|\, \phi_{\text{visual}}(c_i^V),$$

and then embedded by a deep encoder: $$h_i = f_{\text{embed}}(c_i^{(T+V)}) \in \mathbb{R}^d.$$ The embeddings are indexed in a vector database for fast similarity queries.

Subgraph sampling is bifurcated into document mode and web mode. In document mode, for a task objective $g$, an embedding $V$0 is computed, and relevance is measured by

$V$1

A node is included in the task subgraph if either $V$2 or $V$3, with node types restricted to paragraph, table, and figure. The final edge set is induced over the retained nodes. The paper also presents the implicit sampling probability

$V$4

In web mode, task generation is seed-driven. Seed nodes such as search boxes, submit buttons, and key links are identified, and $V$5-hop neighbors are retrieved, restricted to WebElementNode or WebPageNode types. This establishes a task-local interaction graph from which executable sequences can be rendered.

Task instantiation proceeds through two distinct mechanisms. For documents, Graph2Eval uses a library of 12 Jinja2-style templates, each with a prompt skeleton, reference-answer skeleton, graph constraints, a difficulty tag, and an evaluation metrics list. Variables such as $V$6, $V$7, and $V$8 are extracted from the sampled subgraph, and an LLM prompt produces the final natural-language instance. For web tasks, meta-path patterns are defined in regex-like graph syntax; one example is

$V$9

Pattern priority is hierarchical: business-data patterns first, then general interaction, then basic fallback. Slot variables are bound to concrete node IDs, after which a final LLM pass renders the multi-step instruction with screenshots and DOM summaries.

Quality control is multi-stage. The pipeline applies node-reachability filtering, LLM-based quality scoring on a $R$0–$R$1 scale, similarity and novelty analysis with Maximal Marginal Relevance,

$R$2

and dimension-wise coverage constraints over node-type combinations, edge-type patterns, task templates, web meta-paths, and difficulty levels. This suggests that benchmark diversity is treated as an optimization target rather than as a by-product of large-scale generation.

4. Implementation, validation, and metrics

The benchmark implementation uses 16 publicly published high-quality research papers and 8 websites from diverse domains, including Open Library, The Guardian, and StackOverflow. Documents are parsed into semantic chunks, tables, figures, and headings. Websites are processed through automated crawling with human-like navigation scripts, combined with screenshots and DOM extraction. All page elements are converted into graph nodes with metadata such as URL and DOM path, and the resulting embeddings are vector-indexed for semantic search (Chen et al., 1 Oct 2025).

A central methodological point is that there is no manual annotation of each task. Instead, validation is distributed across rule-based heuristics, LLM judgments, and automatic consistency checks. For web tasks, reachability is verified by simulated walkthroughs, and final executability is checked through sandboxed Docker runs. For document tasks, LLM judges validate clarity and exactness. Template variables are also checked automatically against subgraph contents. A common misconception is that Graph2Eval-Bench is simply a synthetic prompt collection; the benchmark is more precisely an annotation-free pipeline with explicit executability and consistency constraints.

The evaluation protocol separates document comprehension, web interaction, collaboration, and interaction efficiency. For document comprehension, the benchmark reports Precision, Recall, and F1: $R$3

$R$4

It also reports ROUGE-L through longest-common-subsequence precision and recall: $R$5

$R$6

An LLM-as-Judge score is obtained by averaging three $R$7–$R$8 ratings: answer_quality, relevance, and completeness.

For web interaction, the primary metric is Success Rate,

$R$9

where success is determined by an LLM evaluator examining the final page state, executed action trace, and any errors. Collaboration is measured by

$E \subseteq V \times V \times R$0

together with average token consumption. Interaction efficiency is reported through average steps to completion and average execution time per task, though not as a standalone scalar metric.

The experimental setup compares Single-Agent against Multi-Agent systems on document tasks and SoM Agent against Agent S 2.5 on web tasks. LLM backbones include GPT-4o, GPT-4.1-mini, Deepseek-V3/V3.1, Qwen2.5-VL-7B/32B/72B, and Gemini-2.5-flash, with temperature fixed at 0.1. Baselines include RAG-only text-only systems versus multimodal systems, Single-Agent versus Multi-Agent document settings, and SoM Agent versus Agent S 2.5 for web interaction.

5. Empirical behavior and discriminative findings

On document comprehension, Graph2Eval-Bench reports that GPT-4o in the Single-Agent setting achieves $E \subseteq V \times V \times R$1, $E \subseteq V \times V \times R$2, and $E \subseteq V \times V \times R$3, while GPT-4o in the Multi-Agent setting achieves $E \subseteq V \times V \times R$4 with $E \subseteq V \times V \times R$5 improvement, $E \subseteq V \times V \times R$6, and $E \subseteq V \times V \times R$7 (Chen et al., 1 Oct 2025). Deepseek-V3 in the Single-Agent setting reaches $E \subseteq V \times V \times R$8 and the best judge score, $E \subseteq V \times V \times R$9. The benchmark’s summary interpretation is that Multi-Agent yields negligible $v_i$0 and incurs approximately $v_i$1 token cost. This indicates that collaboration overhead is not automatically beneficial for relatively straightforward RAG-like document tasks.

On web interaction, the contrast between agent architectures is much sharper. With Gemini-2.5-flash, SoM Agent obtains overall $v_i$2, whereas Agent S 2.5 obtains overall $v_i$3. The second-best reported open-model configuration is Qwen2.5-VL-72B plus Agent S 2.5 with $v_i$4, and GPT-4o-mini plus Agent S 2.5 reaches $v_i$5. The benchmark attributes the large gap between SoM Agent and Agent S 2.5 to reflection and procedural memory. A plausible implication is that, in dynamic web environments, agent-loop design may dominate raw backbone quality once perceptual grounding is adequate.

The benchmark also reports macro-level strengths. It generated 1,319 tasks in minutes, with approximately $v_i$6 seconds per document task and approximately $v_i$7 seconds per web task. It covers $v_i$8 document templates, $v_i$9 web patterns, and $c_i^T$0 difficulty levels, and it exposes clear performance gaps across model sizes, architectures, and agent configurations. At the same time, even the best web agent succeeds on fewer than $c_i^T$1 of web tasks, indicating that these interactions remain challenging under the benchmark’s executability and state-trace criteria.

6. Limitations, recommendations, and relation to adjacent benchmarks

The benchmark identifies several weaknesses directly. The document multi-agent design underperforms the expected synergy for RAG tasks, and the best web agent still succeeds on fewer than $c_i^T$2 of web tasks (Chen et al., 1 Oct 2025). These findings are significant because they separate two distinct failure modes: collaboration that adds coordination cost without enough task decomposition benefit, and web interaction that remains brittle even when equipped with multimodal perception.

The paper’s recommendations for future agent design are correspondingly architectural. It recommends incorporating reflection and error-attribution modules in the style of Agent S 2.5, combining graph-structured subgoal planning with tighter verification loops, extending memory management to record DOM states and intermediate page snapshots, and pursuing safety-aware agents through the safety-task generation extension described in Section A.12. This suggests that Graph2Eval-Bench is not only an evaluation dataset but also an experimental scaffold for studying how memory, planning, and verification interact in multimodal agents.

Within the broader benchmark landscape, Graph2Eval-Bench also connects to a line of work on graph-structured evaluation design. GraphEval36K had proposed that instruction-based decomposition modules could be standardized as “plugins” in a Graph2Eval-Bench to enable systematic comparison of reasoning paradigms such as CoT, SSD, and retrieval-augmented methods (Wu et al., 2024). In the 2025 formulation, that name becomes a concrete benchmark instantiated around knowledge-graph-driven task generation across documents and live websites. The continuity lies in the use of structured graph abstractions to control task diversity and evaluative granularity; the difference lies in target capability, shifting from code-based graph reasoning to end-to-end agent reasoning, collaboration, and interaction.