Graph2Eval: Dynamic Task Generation Framework

• Graph2Eval is a knowledge graph-based framework that automatically generates evaluation tasks for multimodal agents in document comprehension and web interaction scenarios.
• It employs subgraph sampling, task templates, and meta-paths to translate complex graph structures into executable tasks with fine-tuned difficulty controls.
• Empirical results indicate that while reflective, memory-equipped web agents outperform simpler models in interactive tasks, multi-agent setups may increase token usage without clear gains on document tasks.

Graph2Eval is a knowledge graph-based framework for automatic task generation and end-to-end evaluation of multimodal agents. It was introduced to generate both multimodal document comprehension tasks and web interaction tasks from multi-source external data, with the knowledge graph serving as the task space and with subgraph sampling, task templates, and meta-paths translating graph structure into executable evaluations. The framework was instantiated as Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning document comprehension and web interaction scenarios, and was used to evaluate Single-Agent, Multi-Agent, and Web Agent settings along reasoning, collaboration, and interaction dimensions (Chen et al., 1 Oct 2025).

1. Motivation, scope, and problem setting

Graph2Eval addresses a limitation of static benchmarks for multimodal LLM-driven agents. The motivating claim is that evaluation based on static datasets can no longer adequately assess agents in dynamic environments and diverse tasks, particularly when agents must browse, click, search, read PDFs, reason, and collaborate. The framework is positioned against two classes of prior evaluation practice: static, human-annotated datasets such as GAIA, Mind2Web, WebArena, and OSWorld, and LLM-synthetic data methods such as Self-Instruct and TaskCraft. The former are expensive to extend or refresh and embed a fixed set of tasks and content; the latter focus mainly on text-only training or evaluation and rarely model multi-step, multi-hop, interactive workflows in realistic web environments (Chen et al., 1 Oct 2025).

The framework’s central design choice is to treat a heterogeneous multimodal knowledge graph as a latent task space. In this view, documents, webpages, DOM elements, figures, semantic chunks, and interaction affordances are not merely context sources but structured substrates from which evaluation tasks can be sampled. This enables the generation of document comprehension tasks and web interaction tasks within a common graph-centric formalism. Graph2Eval is explicitly designed to measure reasoning, collaboration, and interaction capabilities across different agent classes rather than only answer correctness in a fixed dataset (Chen et al., 1 Oct 2025).

A recurrent misconception is that more elaborate agent scaffolding necessarily yields better evaluation performance. Graph2Eval’s experiments do not support that as a general rule: on document comprehension tasks, multi-agent collaboration increased token usage without improving performance and could slightly degrade results. By contrast, on web tasks, architectures with task-aligned reflection and multidimensional memory management showed large gains over simpler Set-of-Marks prompting. This indicates that the framework is sensitive not only to model quality but also to the interaction between task structure and agent architecture (Chen et al., 1 Oct 2025).

2. Knowledge graph formulation and data construction

The core abstraction is a heterogeneous multimodal knowledge graph

$G = (V, E, R),$

where $V$ denotes nodes, $E \subseteq V \times V \times R$ denotes labeled edges, and $R$ denotes relation types. Node types include document elements and web elements such as Paragraph, Heading, Table, Image, Figure caption, Hyperlink, Form, Button, Input box, Table cell, WebPageNode, WebElementNode, EntityNodes, and SemanticChunkNode. Each node preserves a structural path, denoted $\text{Path}(v_i)$, corresponding to DOM path or document hierarchy (Chen et al., 1 Oct 2025).

Each node $v_i$ carries textual content $c_i^T$ and visual content $c_i^V$. Visual content is converted to text through a vision-language function and concatenated with the original text: $$c_i^{T+V} = c_i^T \,||\, \phi_{\text{visual}}(c_i^V),$$ followed by embedding

$h_i = f_{\text{embed}}(c_i^{T+V}),$

with $V$0 reported as, for example, 384 using all-MiniLM-L6-v2. The resulting vectors are stored in a vector database for semantic similarity computations (Chen et al., 1 Oct 2025).

Edges are partitioned into document-oriented and web-oriented relations: $V$1 For documents, Graph2Eval constructs structural edges such as sequence, contains, and figure-caption links; semantic edges such as entity relations and semantic similarity links; and contextual edges such as figure or table context and cross-references. For web data, it constructs navigation edges, interaction edges, and layout or data-flow edges, including relations such as click button $V$2 open modal and submit form $V$3 results page (Chen et al., 1 Oct 2025).

Document parsing proceeds through semantic chunking, embedding computation, and metadata annotation. PDFs and HTML documents are split into minimal semantic units including paragraphs, headings, table cells, figures, and captions, each of which becomes a graph node. Metadata such as file path, title, author, page, section indices, and local coordinates are attached to preserve layout and hierarchical structure (Chen et al., 1 Oct 2025).

Web parsing uses URL crawling over selected sites, including Mozilla Developer, GitHub, Project Gutenberg, Open Library, OpenWeather, StackOverflow, The Guardian, and WIRED. Graph2Eval extracts DOM trees and screenshots, simulates human-like interactions to explore dynamic content such as modals, infinite scroll, and Ajax, and filters pages through heuristics and LLM-based quality checks to remove low-information or adversarial pages. The parsed DOM nodes and screenshots are then incorporated as graph nodes with multimodal embeddings (Chen et al., 1 Oct 2025).

3. Subgraph sampling and task generation mechanisms

Given a task objective $V$4, Graph2Eval samples a local subgraph

$V$5

The full graph is denoted $V$6, and the sampling logic differs by mode. In document mode, relevance is computed by cosine similarity between node and objective embeddings: $V$7 A node is included when its relevance exceeds a threshold $V$8 or when it satisfies structural constraints through StructMatch(v_i,g)=1, subject to membership in the allowed NodeTypeSet. In web mode, sampling begins from seed nodes $V$9 and expands to $E \subseteq V \times V \times R$0-hop neighbors within WebNodeSet, producing

$E \subseteq V \times V \times R$1

This is the mechanism formalized in Algorithm 1, “Workflow of Subgraph Sampling” (Chen et al., 1 Oct 2025).

For document comprehension, task generation is template-driven. Templates specify a template ID, name, description, task type, difficulty level, required capabilities, Jinja2 prompt template, reference answer template, required node types, required edge types, minimum and maximum node counts, and maximum hop distance. The reported document task types include information extraction, comprehension, summarization, multi-hop reasoning, comparative analysis, fact verification, image interpretation, and cross-referencing. Difficulty is discretized into Easy, Medium, Hard, and Expert. Subgraph sampling is constrained by the template’s graph requirements, after which variable extraction binds graph content into template slots such as comparison_items, paragraphs, and entities. An LLM then instantiates the natural-language task prompt and reference answer from the structured subgraph representation (Chen et al., 1 Oct 2025).

For web interaction, Graph2Eval uses a Seed $E \subseteq V \times V \times R$2 Subgraph $E \subseteq V \times V \times R$3 Meta-path $E \subseteq V \times V \times R$4 Task pipeline. Task seeds are operational nodes such as buttons, input boxes, forms, navigation links, search boxes, filter controls, modals, and toast notifications. Subgraph sampling around seeds captures local workflows. Meta-paths then formalize patterns of interaction as typed graph patterns. One example given in the appendix is

$E \subseteq V \times V \times R$5

The system’s Graph Regex Engine matches such patterns in sampled subgraphs, producing a MetapathPattern and a concrete MetapathInstance that binds variables to specific nodes. Priority is assigned to Business Data Patterns, then General Interaction Patterns, then Basic Interaction Patterns (Chen et al., 1 Oct 2025).

Graph2Eval’s web task generation then conditions an LLM on the meta-path instance, DOM element list, optionally Set-of-Marks annotated screenshot, and page-level context. The LLM produces a natural-language goal or mission, together with expected success conditions and a reference outcome. Reported task-chain patterns include Search + Detail and Search + Filter + Detail, and example goals include searching for a title on Open Library and summarizing author and publication year, or finding current weather information on OpenWeather and reporting minimum and maximum temperatures in Celsius (Chen et al., 1 Oct 2025).

This design suggests that Graph2Eval does not generate tasks from unstructured prompting alone. It generates them from graph-constrained local contexts whose size, hop count, node types, and edge types are explicitly controlled. A plausible implication is that task difficulty is not merely an emergent property of LLM prompting, but a directly tunable property of graph extraction and template constraints (Chen et al., 1 Oct 2025).

4. Filtering, benchmark composition, and evaluation protocol

Graph2Eval applies a multi-stage filtering pipeline to candidate tasks. The reported stages are node reachability and graph constraints, LLM-based quality scoring, and similarity analysis with MMR-style selection. Reachability and structural constraints ensure that task targets are attainable from the starting context and satisfy template-specific structural requirements. LLM-based quality scoring evaluates clarity, answerability, coherence with the supporting subgraph, and correctness of the reference answer. Similarity analysis then uses embeddings and possibly structural signatures to reduce redundancy while maximizing quality and coverage across task types, websites, node types, edge types, pattern types, page-level variety, difficulty, variable patterns, and content length (Chen et al., 1 Oct 2025).

Graph2Eval-Bench is the concrete benchmark instantiated from this pipeline.

Component	Value
Documents	16
Websites	8
Total tasks	1,319
Document tasks	1,002
Web tasks	317
Average tasks per document	~83.5
Average tasks per website	~48.4
Doc task types	12
Web task types	7

The document sources include recent technical papers such as Agent AI, AgentHarm, AI Agents Under Threat, The Dawn of GUI Agent, Data Shapley, DeepSeek-R1, Speculative Decoding, GPT-4o System Card, Learning Dynamics, LightRAG, Navigating the Risks, OpenAI o3/o4-mini System Card, OS Agents, Qwen-VL, RTBAS, and TaskCraft. The eight websites are Mozilla Developer Network, GitHub, Project Gutenberg, Open Library, OpenWeather, StackOverflow, The Guardian, and WIRED. Average generation time is reported as 34.87 seconds for document tasks and 95.51 seconds for web tasks (Chen et al., 1 Oct 2025).

The framework evaluates three agent classes. The Single Agent is a RAG pipeline that retrieves, executes, evaluates, and responds, with support for structured outputs, references, reasoning paths, and confidence scores. The Multi-Agent configuration assigns roles PLANNER, RETRIEVER, REASONER, VERIFIER, and SUMMARIZER, with each role maintaining structured ReasoningStep records linking source node, target node, edge type, logic, and confidence. Web agents include a SoM Agent that grounds interactions by Set-of-Marks prompting over screenshots and an Agent S 2.5 configuration composed of LMMAgent, WebACI, Worker, and ProceduralMemory, with task-aligned reflection after each step or failure (Chen et al., 1 Oct 2025).

For document tasks, Graph2Eval reports F1, ROUGE-L, and LLM-as-a-Judge. The F1 computation is defined through precision and recall: $E \subseteq V \times V \times R$6

$E \subseteq V \times V \times R$7

ROUGE-L is given by

$E \subseteq V \times V \times R$8

with

$E \subseteq V \times V \times R$9

and $R$0. The LLM judge scores answer_quality, relevance, and completeness in $R$1. For web tasks, the primary metric is Success Rate,

$R$2

with success determined by an LLM evaluator that examines the task prompt, action trajectory, final page URL or title, and content summary (Chen et al., 1 Oct 2025).

5. Empirical results and substantive findings

On document comprehension tasks, Graph2Eval reports results for multimodal models including GPT-4o, GPT-4.1-mini, and Qwen2.5-VL-7B, 32B, and 72B, as well as text-only models Deepseek-V3 and Deepseek-V3.1. In the Single Agent setting, GPT-4o obtained F1 0.5766, ROUGE-L 0.4874, and LLM Judge 0.7854, while Qwen2.5-VL-72B obtained F1 0.5730, ROUGE-L 0.4837, and LLM Judge 0.7094. Deepseek-V3, in text-only mode, obtained F1 0.5376, ROUGE-L 0.4518, and the highest LLM Judge score at 0.8351. The framework’s per-task-type analyses indicated that performance varied more strongly by model than by agent configuration, and that larger Qwen2.5-VL models outperformed smaller ones (Chen et al., 1 Oct 2025).

The comparison between single-agent and multi-agent document evaluation is one of the framework’s clearest empirical findings. For GPT-4o, moving from single-agent to multi-agent raised F1 from 0.5766 to 0.5916, but reduced LLM Judge from 0.7854 to 0.7623 and increased token usage from 1631.82 to 3560.92. For Deepseek-V3, F1 rose from 0.5376 to 0.5497 and ROUGE-L from 0.4518 to 0.4635, while LLM Judge fell from 0.8351 to 0.7984 and token usage roughly doubled. The authors explicitly conclude that multi-agent collaboration increases token usage without improving performance on document comprehension tasks and can even slightly degrade results (Chen et al., 1 Oct 2025).

On web interaction tasks, the results differ sharply. SoM Agent achieved its best overall Success Rate with gemini-2.5-flash, at 0.1451, followed by qwen2.5-vl-72b at 0.1388 and gpt-4o-mini at 0.0946. Agent S 2.5 substantially outperformed SoM in the same evaluation setting. With gemini-2.5-flash, Agent S 2.5 achieved overall SR 0.6920, including Search 0.6340 and Modal 1.0. With qwen2.5-vl-72b, it achieved overall SR 0.3880, including Search 0.4278, Modal 0.8333, and Button 0.5625; with gpt-4o-mini, overall SR was 0.3312. The paper attributes this gap to task-aligned reflection as a Test-Time Scaling strategy and to multidimensional memory management, both of which improved reasoning performance in web environments (Chen et al., 1 Oct 2025).

These results matter because they separate three axes that are often conflated in agent evaluation: model capability, orchestration strategy, and environment-interaction competence. Graph2Eval shows that stronger multimodal models do not automatically benefit from multi-agent decomposition on document tasks, while reflective, memory-equipped web agents can dominate simpler visually grounded agents on interactive tasks. This suggests that benchmark structure and agent architecture must be interpreted jointly rather than independently (Chen et al., 1 Oct 2025).

6. Relation to adjacent graph-centered benchmarks and systems

Graph2Eval belongs to a broader family of graph-centered evaluation and optimization systems, but its object of evaluation is distinct. GraphEval36K evaluates LLMs on graph problem-solving through code generation, using 40 graph coding problems, 36,900 test cases, per-category pass rates, and an execution harness with Structured Symbolic Decomposition. Its emphasis is graph reasoning via program synthesis rather than automatic generation of multimodal document and web tasks for agents (Wu et al., 2024).

The framework is also distinct from graph-native optimization systems such as GOpt and Geo. GOpt introduces a modular graph-native query optimization framework for Complex Graph Patterns with a unified intermediate representation, automatic type inference, heuristic rules, and cost-based optimization across multiple query languages (Lyu et al., 2024). Geo presents a programmable pattern matching query optimizer for graph pattern mining that uses e-graphs, equality saturation, canonicalization, and embedded reconstructability to manage rewrite-rule interactions and query equivalences (Yousefian et al., 25 May 2026). These systems optimize graph queries or graph mining workloads; Graph2Eval instead uses knowledge graphs as a controlled task-generation substrate for agent evaluation.

A plausible implication is that these lines of work are complementary rather than competing. Graph2Eval contributes graph-based benchmark construction and agent evaluation; GraphEval36K contributes graph-structured reasoning benchmarks for code generation; GOpt and Geo contribute graph-native optimization machinery for graph queries and graph pattern mining. The commonality is not the task domain, but the use of graph structure to make evaluation or optimization more systematic (Wu et al., 2024).

7. Limitations and future directions

The authors identify several limitations. Safety tasks are not yet fully integrated, although the appendix sketches a preliminary safety-task-generation pipeline. For documents, that pipeline would parse safety or policy documents, extract threat types, inject threats through content injection, prompt manipulation, context switching, and indirect reference, and then create safety-focused tasks. For web environments, it would use sandboxed settings to inject controlled malicious artifacts such as phishing forms or suspicious redirects and then generate tasks that test whether agents avoid unsafe actions (Chen et al., 1 Oct 2025).

A second limitation is error attribution. The current framework reports aggregate metrics, but future work is described as exploiting graph structure to attribute errors to specific subgraph patterns and to distinguish failures of language understanding, reasoning, or interaction. A third limitation arises from dynamic real-world changes: websites evolve, which can break previously generated tasks. The paper notes that this is partially mitigated by LLM-as-judge evaluation, which can reinterpret final states, but also suggests that more robust environment versioning or caching will be needed (Chen et al., 1 Oct 2025).

More broadly, the framework’s design implies several research directions already named in the paper: richer safety evaluation, better graph-structured diagnostics, and stronger handling of environmental drift. The central claim that knowledge graphs can serve as a multimodal, interactive task space remains the framework’s durable contribution. In Graph2Eval, evaluation is no longer tied to a fixed benchmark corpus; it is generated from structured relations in documents and websites, then filtered, instantiated, and executed as a living benchmark for agent reasoning, collaboration, and web interaction (Chen et al., 1 Oct 2025).