---
title: 'Graph2Eval: Dynamic Task Generation Framework'
url: https://www.emergentmind.com/topics/graph2eval
type: topic
---

# Graph2Eval: Dynamic Task Generation Framework

Graph2Eval is a knowledge graph-based framework for automatic task generation and end-to-end evaluation of multimodal agents. It was introduced to generate both multimodal document comprehension tasks and web interaction tasks from multi-source external data, with the knowledge graph serving as the task space and with subgraph sampling, task templates, and meta-paths translating graph structure into executable evaluations. The framework was instantiated as Graph2Eval-Bench, a curated dataset of 1,319 tasks spanning document comprehension and web interaction scenarios, and was used to evaluate Single-Agent, Multi-Agent, and Web Agent settings along reasoning, collaboration, and interaction dimensions [2510.00507].

## 1. Motivation, scope, and problem setting

Graph2Eval addresses a limitation of static benchmarks for multimodal LLM-driven agents. The motivating claim is that evaluation based on static datasets can no longer adequately assess agents in dynamic environments and diverse tasks, particularly when agents must browse, click, search, read PDFs, reason, and collaborate. The framework is positioned against two classes of prior evaluation practice: static, human-annotated datasets such as GAIA, Mind2Web, WebArena, and OSWorld, and LLM-synthetic data methods such as Self-Instruct and TaskCraft. The former are expensive to extend or refresh and embed a fixed set of tasks and content; the latter focus mainly on text-only training or evaluation and rarely model multi-step, multi-hop, interactive workflows in realistic web environments [2510.00507].

The framework’s central design choice is to treat a heterogeneous multimodal knowledge graph as a latent task space. In this view, documents, webpages, DOM elements, figures, semantic chunks, and interaction affordances are not merely context sources but structured substrates from which evaluation tasks can be sampled. This enables the generation of document comprehension tasks and web interaction tasks within a common graph-centric formalism. Graph2Eval is explicitly designed to measure reasoning, collaboration, and interaction capabilities across different agent classes rather than only answer correctness in a fixed dataset [2510.00507].

A recurrent misconception is that more elaborate agent scaffolding necessarily yields better evaluation performance. Graph2Eval’s experiments do not support that as a general rule: on document comprehension tasks, multi-agent collaboration increased token usage without improving performance and could slightly degrade results. By contrast, on web tasks, architectures with task-aligned reflection and multidimensional memory management showed large gains over simpler Set-of-Marks prompting. This indicates that the framework is sensitive not only to model quality but also to the interaction between task structure and agent architecture [2510.00507].

## 2. Knowledge graph formulation and data construction

The core abstraction is a heterogeneous multimodal knowledge graph
\[
G = (V, E, R),
\]
where \(V\) denotes nodes, \(E \subseteq V \times V \times R\) denotes labeled edges, and \(R\) denotes relation types. Node types include document elements and web elements such as Paragraph, Heading, Table, Image, Figure caption, Hyperlink, Form, Button, Input box, Table cell, WebPageNode, WebElementNode, EntityNodes, and SemanticChunkNode. Each node preserves a structural path, denoted \(\text{Path}(v_i)\), corresponding to DOM path or document hierarchy [2510.00507].

Each node \(v_i\) carries textual content \(c_i^T\) and visual content \(c_i^V\). Visual content is converted to text through a vision-language function and concatenated with the original text:
\[
c_i^{T+V} = c_i^T \,||\, \phi_{\text{visual}}(c_i^V),
\]
followed by embedding
\[
h_i = f_{\text{embed}}(c_i^{T+V}),
\]
with \(d\) reported as, for example, 384 using `all-MiniLM-L6-v2`. The resulting vectors are stored in a vector database for semantic similarity computations [2510.00507].

Edges are partitioned into document-oriented and web-oriented relations:
\[
E = E_{\text{text}} \cup E_{\text{web}}.
\]
For documents, Graph2Eval constructs structural edges such as sequence, contains, and figure-caption links; semantic edges such as entity relations and semantic similarity links; and contextual edges such as figure or table context and cross-references. For web data, it constructs navigation edges, interaction edges, and layout or data-flow edges, including relations such as click button \(\rightarrow\) open modal and submit form \(\rightarrow\) results page [2510.00507].

Document parsing proceeds through semantic chunking, embedding computation, and metadata annotation. PDFs and HTML documents are split into minimal semantic units including paragraphs, headings, table cells, figures, and captions, each of which becomes a graph node. Metadata such as file path, title, author, page, section indices, and local coordinates are attached to preserve layout and hierarchical structure [2510.00507].

Web parsing uses URL crawling over selected sites, including Mozilla Developer, GitHub, Project Gutenberg, Open Library, OpenWeather, StackOverflow, The Guardian, and WIRED. Graph2Eval extracts DOM trees and screenshots, simulates human-like interactions to explore dynamic content such as modals, infinite scroll, and Ajax, and filters pages through heuristics and LLM-based quality checks to remove low-information or adversarial pages. The parsed DOM nodes and screenshots are then incorporated as graph nodes with multimodal embeddings [2510.00507].

## 3. Subgraph sampling and task generation mechanisms

Given a task objective \(g\), Graph2Eval samples a local subgraph
\[
G_g = (V_g, E_g) \subseteq G.
\]
The full graph is denoted \(G=(V,E)\), and the sampling logic differs by mode. In document mode, relevance is computed by cosine similarity between node and objective embeddings:
\[
\text{Relevance}(v_i,g) = \cos(h_i,h_g).
\]
A node is included when its relevance exceeds a threshold \(\tau\) or when it satisfies structural constraints through `StructMatch(v_i,g)=1`, subject to membership in the allowed `NodeTypeSet`. In web mode, sampling begins from seed nodes \(S_{\text{seed}}(g)\) and expands to \(k\)-hop neighbors within `WebNodeSet`, producing
\[
E_g = \{ (v_i, v_j) \in E \mid v_i, v_j \in V_g \}.
\]
This is the mechanism formalized in Algorithm 1, “Workflow of Subgraph Sampling” [2510.00507].

For document comprehension, task generation is template-driven. Templates specify a template ID, name, description, task type, difficulty level, required capabilities, Jinja2 prompt template, reference answer template, required node types, required edge types, minimum and maximum node counts, and maximum hop distance. The reported document task types include information extraction, comprehension, summarization, multi-hop reasoning, comparative analysis, fact verification, image interpretation, and cross-referencing. Difficulty is discretized into Easy, Medium, Hard, and Expert. Subgraph sampling is constrained by the template’s graph requirements, after which variable extraction binds graph content into template slots such as `comparison_items`, `paragraphs`, and `entities`. An LLM then instantiates the natural-language task prompt and reference answer from the structured subgraph representation [2510.00507].

For web interaction, Graph2Eval uses a Seed \(\rightarrow\) Subgraph \(\rightarrow\) Meta-path \(\rightarrow\) Task pipeline. Task seeds are operational nodes such as buttons, input boxes, forms, navigation links, search boxes, filter controls, modals, and toast notifications. Subgraph sampling around seeds captures local workflows. Meta-paths then formalize patterns of interaction as typed graph patterns. One example given in the appendix is
\[
\text{SearchBox}(\$search) -[\text{Fills}]-> \text{BusinessData}(\$query) -[\text{Controls}]-> \text{Button}(\$submit).
\]
The system’s Graph Regex Engine matches such patterns in sampled subgraphs, producing a `MetapathPattern` and a concrete `MetapathInstance` that binds variables to specific nodes. Priority is assigned to Business Data Patterns, then General Interaction Patterns, then Basic Interaction Patterns [2510.00507].

Graph2Eval’s web task generation then conditions an LLM on the meta-path instance, DOM element list, optionally Set-of-Marks annotated screenshot, and page-level context. The LLM produces a natural-language goal or mission, together with expected success conditions and a reference outcome. Reported task-chain patterns include `Search + Detail` and `Search + Filter + Detail`, and example goals include searching for a title on Open Library and summarizing author and publication year, or finding current weather information on OpenWeather and reporting minimum and maximum temperatures in Celsius [2510.00507].

This design suggests that Graph2Eval does not generate tasks from unstructured prompting alone. It generates them from graph-constrained local contexts whose size, hop count, node types, and edge types are explicitly controlled. A plausible implication is that task difficulty is not merely an emergent property of LLM prompting, but a directly tunable property of graph extraction and template constraints [2510.00507].

## 4. Filtering, benchmark composition, and evaluation protocol

Graph2Eval applies a multi-stage filtering pipeline to candidate tasks. The reported stages are node reachability and graph constraints, LLM-based quality scoring, and similarity analysis with MMR-style selection. Reachability and structural constraints ensure that task targets are attainable from the starting context and satisfy template-specific structural requirements. LLM-based quality scoring evaluates clarity, answerability, coherence with the supporting subgraph, and correctness of the reference answer. Similarity analysis then uses embeddings and possibly structural signatures to reduce redundancy while maximizing quality and coverage across task types, websites, node types, edge types, pattern types, page-level variety, difficulty, variable patterns, and content length [2510.00507].

Graph2Eval-Bench is the concrete benchmark instantiated from this pipeline.

| Component | Value |
|---|---:|
| Documents | 16 |
| Websites | 8 |
| Total tasks | 1,319 |
| Document tasks | 1,002 |
| Web tasks | 317 |
| Average tasks per document | ~83.5 |
| Average tasks per website | ~48.4 |
| Doc task types | 12 |
| Web task types | 7 |

The document sources include recent technical papers such as *Agent AI*, *AgentHarm*, *AI Agents Under Threat*, *The Dawn of GUI Agent*, *Data Shapley*, *DeepSeek-R1*, *Speculative Decoding*, *GPT-4o System Card*, *Learning Dynamics*, *LightRAG*, *Navigating the Risks*, *OpenAI o3/o4-mini System Card*, *OS Agents*, *Qwen-VL*, *RTBAS*, and *TaskCraft*. The eight websites are Mozilla Developer Network, GitHub, Project Gutenberg, Open Library, OpenWeather, StackOverflow, The Guardian, and WIRED. Average generation time is reported as 34.87 seconds for document tasks and 95.51 seconds for web tasks [2510.00507].

The framework evaluates three agent classes. The Single Agent is a RAG pipeline that retrieves, executes, evaluates, and responds, with support for structured outputs, references, reasoning paths, and confidence scores. The Multi-Agent configuration assigns roles PLANNER, RETRIEVER, REASONER, VERIFIER, and SUMMARIZER, with each role maintaining structured `ReasoningStep` records linking source node, target node, edge type, logic, and confidence. Web agents include a SoM Agent that grounds interactions by Set-of-Marks prompting over screenshots and an Agent S 2.5 configuration composed of LMMAgent, WebACI, Worker, and ProceduralMemory, with task-aligned reflection after each step or failure [2510.00507].

For document tasks, Graph2Eval reports F1, ROUGE-L, and LLM-as-a-Judge. The F1 computation is defined through precision and recall:
\[
\text{Precision} = \frac{|P \cap G|}{|P|}, \quad
\text{Recall} = \frac{|P \cap G|}{|G|},
\]
\[
\text{F1} = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}.
\]
ROUGE-L is given by
\[
\text{ROUGE-L} = \frac{(1 + \beta^2) \cdot \text{R}_{\text{LCS}} \cdot \text{P}_{\text{LCS}}}{\text{R}_{\text{LCS}} + \beta^2 \cdot \text{P}_{\text{LCS}}},
\]
with
\[
\text{R}_{\text{LCS}} = \frac{\text{LCS}(P,G)}{|G|}, \quad
\text{P}_{\text{LCS}} = \frac{\text{LCS}(P,G)}{|P|},
\]
and \(\beta=1\). The LLM judge scores `answer_quality`, `relevance`, and `completeness` in \([0,1]\). For web tasks, the primary metric is Success Rate,
\[
\text{SR} = \frac{N_{\text{success}}}{N_{\text{total}}},
\]
with success determined by an LLM evaluator that examines the task prompt, action trajectory, final page URL or title, and content summary [2510.00507].

## 5. Empirical results and substantive findings

On document comprehension tasks, Graph2Eval reports results for multimodal models including GPT-4o, GPT-4.1-mini, and Qwen2.5-VL-7B, 32B, and 72B, as well as text-only models Deepseek-V3 and Deepseek-V3.1. In the Single Agent setting, GPT-4o obtained F1 0.5766, ROUGE-L 0.4874, and LLM Judge 0.7854, while Qwen2.5-VL-72B obtained F1 0.5730, ROUGE-L 0.4837, and LLM Judge 0.7094. Deepseek-V3, in text-only mode, obtained F1 0.5376, ROUGE-L 0.4518, and the highest LLM Judge score at 0.8351. The framework’s per-task-type analyses indicated that performance varied more strongly by model than by agent configuration, and that larger Qwen2.5-VL models outperformed smaller ones [2510.00507].

The comparison between single-agent and multi-agent document evaluation is one of the framework’s clearest empirical findings. For GPT-4o, moving from single-agent to multi-agent raised F1 from 0.5766 to 0.5916, but reduced LLM Judge from 0.7854 to 0.7623 and increased token usage from 1631.82 to 3560.92. For Deepseek-V3, F1 rose from 0.5376 to 0.5497 and ROUGE-L from 0.4518 to 0.4635, while LLM Judge fell from 0.8351 to 0.7984 and token usage roughly doubled. The authors explicitly conclude that multi-agent collaboration increases token usage without improving performance on document comprehension tasks and can even slightly degrade results [2510.00507].

On web interaction tasks, the results differ sharply. SoM Agent achieved its best overall Success Rate with `gemini-2.5-flash`, at 0.1451, followed by `qwen2.5-vl-72b` at 0.1388 and `gpt-4o-mini` at 0.0946. Agent S 2.5 substantially outperformed SoM in the same evaluation setting. With `gemini-2.5-flash`, Agent S 2.5 achieved overall SR 0.6920, including Search 0.6340 and Modal 1.0. With `qwen2.5-vl-72b`, it achieved overall SR 0.3880, including Search 0.4278, Modal 0.8333, and Button 0.5625; with `gpt-4o-mini`, overall SR was 0.3312. The paper attributes this gap to task-aligned reflection as a Test-Time Scaling strategy and to multidimensional memory management, both of which improved reasoning performance in web environments [2510.00507].

These results matter because they separate three axes that are often conflated in agent evaluation: model capability, orchestration strategy, and environment-interaction competence. Graph2Eval shows that stronger multimodal models do not automatically benefit from multi-agent decomposition on document tasks, while reflective, memory-equipped web agents can dominate simpler visually grounded agents on interactive tasks. This suggests that benchmark structure and agent architecture must be interpreted jointly rather than independently [2510.00507].

## 6. Relation to adjacent graph-centered benchmarks and systems

Graph2Eval belongs to a broader family of graph-centered evaluation and optimization systems, but its object of evaluation is distinct. GraphEval36K evaluates large language models on graph problem-solving through code generation, using 40 graph coding problems, 36,900 test cases, per-category pass rates, and an execution harness with Structured Symbolic Decomposition. Its emphasis is graph reasoning via program synthesis rather than automatic generation of multimodal document and web tasks for agents [2406.16176].

The framework is also distinct from graph-native optimization systems such as GOpt and Geo. GOpt introduces a modular graph-native query optimization framework for Complex Graph Patterns with a unified intermediate representation, automatic type inference, heuristic rules, and cost-based optimization across multiple query languages [2401.17786]. Geo presents a programmable pattern matching query optimizer for graph pattern mining that uses e-graphs, equality saturation, canonicalization, and embedded reconstructability to manage rewrite-rule interactions and query equivalences [2605.26291]. These systems optimize graph queries or graph mining workloads; Graph2Eval instead uses knowledge graphs as a controlled task-generation substrate for agent evaluation.

A plausible implication is that these lines of work are complementary rather than competing. Graph2Eval contributes graph-based benchmark construction and agent evaluation; GraphEval36K contributes graph-structured reasoning benchmarks for code generation; GOpt and Geo contribute graph-native optimization machinery for graph queries and graph pattern mining. The commonality is not the task domain, but the use of graph structure to make evaluation or optimization more systematic [2406.16176].

## 7. Limitations and future directions

The authors identify several limitations. Safety tasks are not yet fully integrated, although the appendix sketches a preliminary safety-task-generation pipeline. For documents, that pipeline would parse safety or policy documents, extract threat types, inject threats through content injection, prompt manipulation, context switching, and indirect reference, and then create safety-focused tasks. For web environments, it would use sandboxed settings to inject controlled malicious artifacts such as phishing forms or suspicious redirects and then generate tasks that test whether agents avoid unsafe actions [2510.00507].

A second limitation is error attribution. The current framework reports aggregate metrics, but future work is described as exploiting graph structure to attribute errors to specific subgraph patterns and to distinguish failures of language understanding, reasoning, or interaction. A third limitation arises from dynamic real-world changes: websites evolve, which can break previously generated tasks. The paper notes that this is partially mitigated by LLM-as-judge evaluation, which can reinterpret final states, but also suggests that more robust environment versioning or caching will be needed [2510.00507].

More broadly, the framework’s design implies several research directions already named in the paper: richer safety evaluation, better graph-structured diagnostics, and stronger handling of environmental drift. The central claim that knowledge graphs can serve as a multimodal, interactive task space remains the framework’s durable contribution. In Graph2Eval, evaluation is no longer tied to a fixed benchmark corpus; it is generated from structured relations in documents and websites, then filtered, instantiated, and executed as a living benchmark for agent reasoning, collaboration, and web interaction [2510.00507].

Source: https://www.emergentmind.com/topics/graph2eval