Generative DyTAG Benchmark (GDGB)

• GDGB is a benchmark that defines evaluation protocols for dynamic text-attributed graphs, formalizing inductive dynamic graph generation and holistic metric assessments.
• It introduces eight high-quality real-world DyTAG datasets with rich textual attributes and detailed node-edge evolution, ensuring robust performance evaluation.
• The framework leverages an LLM-based multi-agent generation model alongside comparative baselines to assess structure, text quality, and embedding fidelity.

The Generative DyTAG Benchmark (GDGB) defines the state of the art for evaluation and development of generative models over dynamic text-attributed graphs (DyTAGs)—structures that encapsulate evolving node/edge sets, temporal information, and high-fidelity textual attributes. GDGB rigorously addresses the deficiencies of prior DyTAG generation resources, which lacked semantically rich texts and standardized protocols for both transductive and inductive dynamic graph synthesis (Peng et al., 4 Jul 2025).

1. Formal Task Definition: Inductive Dynamic Graph Generation (IDGG)

GDGB centers on the Inductive Dynamic Graph Generation (IDGG) paradigm, which formalizes the expansion of a DyTAG through open-ended growth in both edges and the node set, with newly introduced nodes annotated by detailed textual profiles. Let $$G = (N, E, T; N^{\mathrm{text}}, E^{\mathrm{text}}, L)$$, where $N$ is the node set, $E \subseteq N \times N$ are timestamped edges, $T$ is the time domain, $N^{\mathrm{text}}$ and $E^{\mathrm{text}}$ are node/edge text attributes, and $L$ is the set of edge labels.

The IDGG protocol begins from a seed subgraph $G_0=(N^0,E^0)$, typically consisting of the first 1,000 edges of the observed DyTAG. At each round $k$, the process splits into three steps:

1. Node Generation: Generate $R_\mathrm{src}^k$ new source nodes and $N$0 new destination nodes with textual profiles.
2. Node Update and Activation: Update global node lists, then sample $N$1 active source nodes for edge interactions.
3. Edge Generation: For each active source node, select destination nodes (using memory/textual cues) and generate new edges (with label, text, timestamp).

The process is iterated $N$2 times to produce an expanded DyTAG $N$3, transitioning from the initial graph (e.g., after 1,000 edges) to a larger graph (e.g., 3,000 edges), respecting realistic node/edge birth and temporally coherent growth (Peng et al., 4 Jul 2025).

2. GDGB Dataset Suite

GDGB comprises eight real-world DyTAG datasets, each exhibiting high-quality textual attributes on both nodes and edges and spanning diverse structural types. Table 1 summarizes their main characteristics.

Dataset	Type	# Nodes	# Edges	Labels	Notable Texts
Sephora	Bipartite	210K users / 2.3K prod	801K	5	Full user/product profiles, reviews
Dianping	Bipartite	158K users/88K biz	1.99M	5	Rich user/business reviews
WikiRevision	Bipartite	75K users/3.2K pages	2.78M	2	Page intros, revision comments
WikiLife	Bipartite	406K persons/54K loc	1.99M	24	Person/location intros, event desc.
IMDB	Non-bipartite	125K actors	1.53M	20	Actor biographies, film/role texts
WeiboTech/Daily	Non-bipartite	20–66K users	0.11–0.35M	2	User profiles, posts, comments
Cora	Non-bipartite	48K papers	110K	5	Titles/abstracts, citing sentences

Each is split such that $N$4 is the subgraph induced by the first 1,000 edges; the model then generates an additional 2,000 edges and corresponding new nodes to evaluate IDGG protocols (Peng et al., 4 Jul 2025).

3. Holistic Evaluation Protocol

GDGB mandates comprehensive evaluation over structure, time, and text using three key categories of metrics:

• Graph Structural Metrics:
  • Degree/Spectra MMD: Maximum Mean Discrepancy on node degree distributions and adjacency spectral features (RBF kernel, lower is better).
  • Power-law Analysis: Fit degree distributions to $N$5, with power-law validity requiring Kolmogorov–Smirnov distance $N$6 and $N$7.
• Textual Quality Metrics:
  • An LLM-as-Evaluator assigns 1–5 scores (Contextual Fidelity, Personality Depth, Dynamic Adaptability, Immersive Quality, Content Richness); the final score is an average across dimensions.
• Global Graph Embedding Metric:
  • JL-Metric-derived cosine distance $N$8, comparing randomly projected per-node embeddings concatenating timestamp, edge and neighbor texts, and node text (lower $N$9 is better).

This multi-criteria design enables rigorous benchmarking of generative models' abilities to capture the joint distribution over topology, temporal evolution, and natural language phenomena (Peng et al., 4 Jul 2025).

4. GAG-General: LLM-Based Multi-Agent Generation Framework

GAG-General instantiates each (source/destination) node as an autonomous LLM agent managing a memory module of historical neighbor interactions (by default, random walks of length 10, up to 1,000 entries). Its main components are:

• Memory Reflection: Optionally prompts the LLM to summarize node memory, akin to neural message aggregation.
• Iterative Pipeline: For each step—
  • "GenerateNodes": Create new nodes and textual profiles via agent LLM prompts.
  • "Recall & Select": For each source, prompt-driven destination (neighbor) selection leveraging both current textual profiles and summarized memory.
  • "GenerateEdges": LLM composes the new edge, including label, text, and timestamp.

Prompt templates capture node descriptions, prior summaries, candidate outputs, prior examples, and request JSON-formatted responses. Generation runs with $E \subseteq N \times N$0, top-$E \subseteq N \times N$1, repetition penalty 1.1, and response truncation at 2,000 tokens, parallelizing across active agent pairs (Peng et al., 4 Jul 2025).

5. Comparative Baselines and Experimental Performance

GDGB benchmarks GAG-General against DG-Gen (deep probabilistic, continuous-time generator (Hosseini et al., 2024)) and VRDAG (bi-flow VAE with node/edge attribute support). Select metrics (on representative datasets, GAG-General “GPT” backbone):

Dataset	Metric	GAG-General	VRDAG	DG-Gen
Sephora	Deg.MMD ↓	0.370	0.795	0.422
	Spec.MMD ↓	0.189	0.847	0.274
	Power-law ✓/✗	✓	✗	✗
	JL-ρ ↓	0.661	0.011	0.228
Dianping	Deg.MMD ↓	0.150	0.887	0.167
	...	...	...	...

Textual quality is only available for GAG-General (mean scores 4.3–4.8/5), as the baseline models do not generate text. GAG-General outperforms baselines in structural fidelity (lower MMDs, more power-law conformant graphs) and global embedding similarity, and is the only model supporting integrated text–structure generative evaluation.

Key empirical observations:

• Ablating memory modules increases textual and embedding errors by 5–15%.
• Memory reflection yields 2–5% improvement in textual scores and 0.03–0.15 gain in embedding fidelity.
• GAG-General preserves power-law degree behavior in ~60% of graphs (vs. <20% for baselines).
• Scalability remains a challenge: generating 2,000 edges requires ~1 hour on an NVIDIA A800; scaling to 50K–100K edges induces substantial computational burden (Peng et al., 4 Jul 2025).

6. Principal Insights and Future Directions

• Inductive generative tasks fundamentally complicate DyTAG synthesis: the inductive regime requires models to invent plausible node semantics and integrate them structurally and textually. This is reflected in higher MMD values compared to transductive settings.
• Multi-agent LLM-centric design is critical for maintaining joint fidelity across text and structure; specialized encoder-based models fail to capture full DyTAG complexity, often producing unrealistic hub distribution or missing text–structure coupling.
• Scenario forecasting emerges as a unique affordance, as IDGG-generated nodes can plausibly simulate yet-unobserved entities, e.g., emergent bestsellers in e-commerce graphs. This suggests potential for counterfactual and future trend analysis.
• Open challenges include node-generation refinement, prompt efficiency, LLM distillation, and tighter coupling of GNN modules for structural regularization without compromising textual expressivity (Peng et al., 4 Jul 2025).

7. Significance and Relationship to Broader Research

GDGB establishes the reference benchmark for generative DyTAG evaluation, introducing high-quality text data, formal IDGG/TDGG tasks, and a holistic protocol integrating deep structural and language metrics. It forms the methodological backbone for further advances in generative modeling of real-world dynamic attributed graphs, supporting code and leaderboard infrastructure. Related frameworks such as RAG4DyG (Wu et al., 2024), DG-Gen (Hosseini et al., 2024), SEDGE (Papoudakis et al., 2017), and multi-agent LLM protocols in social simulation (Graphia (Ji et al., 28 Oct 2025)) extend specific facets of the inductive, temporal, and multi-modality regime, but GDGB uniquely provides the union of realistic text, structure, and semantic benchmarks necessary for robust model comparison and ablation in this evolving field.