Temporal Graph Reasoning Benchmark (T-GRAB)

• Temporal Graph Reasoning Benchmark (T-GRAB) is a synthetic diagnostic tool designed to assess TGNNs on fundamental temporal tasks.
• It isolates key temporal skills by employing controlled synthetic tasks on periodicity, delayed cause-and-effect, and long-range spatio-temporal reasoning.
• Empirical findings reveal diverse model strengths and expose hidden failure modes, emphasizing the need for layered temporal evaluation.

Searching arXiv for the benchmark paper and closely related temporal graph benchmark references. Temporal Graph Reasoning Benchmark (T-GRAB) is a synthetic diagnostic benchmark for temporal graph learning introduced to test whether Temporal Graph Neural Networks (TGNNs) actually capture fundamental temporal patterns rather than exploiting shortcuts present in conventional datasets (Dizaji et al., 14 Jul 2025). It is defined in the discrete-time dynamic graph setting, where a temporal graph is represented as a sequence of snapshots $\mathcal G = G_1,G_2,G_3,\cdots,G_T$, with each snapshot $G_t = (V_t, E_t, X_t)$ and, in T-GRAB, a fixed node set over time and fixed one-hot node features $X_t = I$ (Dizaji et al., 14 Jul 2025). The benchmark is organized around three synthetic task families—periodicity, delayed cause-and-effect, and long-range spatio-temporal reasoning—designed to isolate counting and memorization, delayed dependency inference, and joint spatial-temporal compositional reasoning, respectively (Dizaji et al., 14 Jul 2025). In contrast to broader temporal graph benchmarks that emphasize realistic data diversity and standardized infrastructure, T-GRAB prioritizes controlled generation, factorized difficulty, and interpretability of failure modes (Huang et al., 2023, Huang et al., 2023).

1. Concept and benchmark design

T-GRAB was introduced to address a diagnosis problem in temporal graph evaluation: real-world temporal datasets often entangle evolving graph structure, sparsity, noise, shortcut patterns, and domain-specific biases, making it difficult to attribute model failure to a specific missing temporal capability (Dizaji et al., 14 Jul 2025). Its design is explicitly synthetic and diagnostic. The benchmark provides controlled tasks where the temporal rule is known in advance, difficulty can be scaled by a small number of parameters, and specific temporal skills can be isolated (Dizaji et al., 14 Jul 2025).

The benchmark operates in the discrete-time dynamic graph setting. A temporal graph is defined as

$\mathcal G = G_1,G_2,G_3,\cdots,G_T.$

Each snapshot is

$G_t = (V_t, E_t, X_t),$

where $V_t$ is the node set, $E_t$ the edge set, and $X_t$ the node-feature matrix (Dizaji et al., 14 Jul 2025). In T-GRAB, the node set is fixed across time and node features are fixed one-hot encodings with no edge features (Dizaji et al., 14 Jul 2025). For continuous-time methods, each edge in $G_t$ is translated into timestamped events $\{(u,v,t) \mid (u,v)\in E_t\}$ (Dizaji et al., 14 Jul 2025).

All benchmark tasks are framed as dynamic link prediction (Dizaji et al., 14 Jul 2025). This places T-GRAB closer to predictive temporal graph learning than to temporal question answering or temporal logic execution. However, its difficulty stems from the need to recover latent temporal rules rather than from large-scale data heterogeneity. This suggests a complementary relationship with broader benchmarks such as TGB, which emphasizes realistic large-scale evaluation, historical negative sampling, and streaming protocols, and BenchTemp, which emphasizes standardized TGNN comparison across workloads (Huang et al., 2023, Huang et al., 2023).

2. Task families and isolated temporal skills

T-GRAB contains three benchmark components: periodicity tasks, delayed cause-and-effect tasks, and long-range spatio-temporal tasks (Dizaji et al., 14 Jul 2025). Each family targets a different temporal reasoning capability.

The periodicity family tests whether TGNNs can count repeated temporal repetitions and memorize recurring graph states (Dizaji et al., 14 Jul 2025). It distinguishes between deterministic periodicity, where fixed graph states recur, and stochastic periodicity, where graph-generating distributions recur (Dizaji et al., 14 Jul 2025). This is intended to expose whether a model genuinely tracks phase progression or merely exploits persistence.

The delayed cause-and-effect family tests whether models can infer delayed dependencies, that is, whether an event at time $G_t = (V_t, E_t, X_t)$0 causes a predictable edge pattern at time $G_t = (V_t, E_t, X_t)$1 (Dizaji et al., 14 Jul 2025). The task isolates long-term memory and delayed dependency tracing under random graph noise. Its interpretation of “cause-and-effect” is synthetic and operational rather than causal in the interventionist sense.

The long-range spatio-temporal family extends delayed dependency tasks by combining temporal delay $G_t = (V_t, E_t, X_t)$2 with spatial graph distance $G_t = (V_t, E_t, X_t)$3 (Dizaji et al., 14 Jul 2025). The target at time $G_t = (V_t, E_t, X_t)$4 depends on path endpoints that existed at time $G_t = (V_t, E_t, X_t)$5, thereby requiring both memory over time and multi-hop structural reasoning. This family is intended to separate temporal-memory weakness from spatial-range weakness.

A concise summary is useful because the benchmark’s structure is central to its interpretation.

Task family	Primary skill	Difficulty controls
Periodicity	Counting and memorization	$G_t = (V_t, E_t, X_t)$6, $G_t = (V_t, E_t, X_t)$7, deterministic vs stochastic, SBM noise
Cause-and-effect	Delayed dependency inference	lag $G_t = (V_t, E_t, X_t)$8
Long-range spatio-temporal	Joint temporal and spatial reasoning	lag $G_t = (V_t, E_t, X_t)$9, path distance $X_t = I$0

This decomposition is one of T-GRAB’s defining features. A plausible implication is that the benchmark can reveal different inductive-bias strengths across architectures even when aggregate link-prediction scores on conventional datasets appear similar.

3. Formal task constructions

The periodicity family is defined by

$X_t = I$1

In deterministic periodicity,

$X_t = I$2

so the graph alternates among $X_t = I$3 fixed graphs, each repeated for $X_t = I$4 consecutive snapshots (Dizaji et al., 14 Jul 2025). In stochastic periodicity,

$X_t = I$5

so each phase corresponds to one of $X_t = I$6 graph distributions rather than a fixed graph instance (Dizaji et al., 14 Jul 2025). For counting experiments, deterministic graphs are generated from Erdős–Rényi graphs with 100 nodes and edge probability $X_t = I$7, typically with $X_t = I$8 and varying $X_t = I$9 (Dizaji et al., 14 Jul 2025). For stochastic periodicity, the benchmark uses Stochastic Block Models with 100 nodes, 3 communities, shared inter-community probability $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$0, and intra-community probability $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$1 or $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$2 (Dizaji et al., 14 Jul 2025).

The delayed cause-and-effect family $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$3 augments an Erdős–Rényi base graph with a memory node $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$4. For $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$5,

$\mathcal G = G_1,G_2,G_3,\cdots,G_T.$6

Only edges involving the memory node are predictable; the underlying graph remains random noise (Dizaji et al., 14 Jul 2025). The benchmark evaluates lags $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$7 (Dizaji et al., 14 Jul 2025).

The long-range spatio-temporal family is defined as $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$8, with $\mathcal G = G_1,G_2,G_3,\cdots,G_T.$9 fixed in experiments (Dizaji et al., 14 Jul 2025). Each snapshot contains a source node $G_t = (V_t, E_t, X_t),$0, a target node $G_t = (V_t, E_t, X_t),$1, and $G_t = (V_t, E_t, X_t),$2 disjoint paths of length $G_t = (V_t, E_t, X_t),$3 from $G_t = (V_t, E_t, X_t),$4. For $G_t = (V_t, E_t, X_t),$5,

$G_t = (V_t, E_t, X_t),$6

where path nodes are sampled randomly without replacement (Dizaji et al., 14 Jul 2025). The benchmark studies $G_t = (V_t, E_t, X_t),$7 and $G_t = (V_t, E_t, X_t),$8 (Dizaji et al., 14 Jul 2025).

These constructions are deliberately sparse in semantics. Their purpose is not realism but identifiability of the required temporal skill. This contrasts with TGB, which uses large real-world temporal graphs and evaluates under realistic streaming conditions, and with TGB-Seq, which stresses unseen-edge prediction under low repeat ratio and sequential dynamics but remains focused on real-world future link prediction (Huang et al., 2023, Yi et al., 5 Feb 2025).

4. Experimental protocol and evaluated methods

T-GRAB evaluates 11 methods: four continuous-time dynamic graph methods (DyGFormer, CTAN, TGN, TGAT), three discrete-time dynamic graph methods (EvolveGCN, T-GCN, GC-LSTM), two static graph methods (GCN, GAT), and two simple baselines (Persistence heuristic and EdgeBank$G_t = (V_t, E_t, X_t),$9) (Dizaji et al., 14 Jul 2025).

For periodicity tasks, splits are made in whole periods: 40 full periods for training, 4 for validation, and 4 for testing (Dizaji et al., 14 Jul 2025). For cause-and-effect and long-range tasks, the benchmark uses chronological 80%/10%/10% train/validation/test splits (Dizaji et al., 14 Jul 2025). Training proceeds epoch by epoch over snapshots in temporal order, with batch size one snapshot, Adam optimizer, learning rate $V_t$0, single message-passing layer for all models, temporal neighbor sampling size 20 by default where applicable, patience 100 epochs, and early stopping based on training-loss stagnation (Dizaji et al., 14 Jul 2025). Validation $V_t$1 is computed each epoch, and reported results are averages over five random seeds (Dizaji et al., 14 Jul 2025).

The main metric is average $V_t$2, computed over all possible node pairs at each test timestep to avoid negative-sampling artifacts (Dizaji et al., 14 Jul 2025). For periodicity, $V_t$3 is computed over all test edges and all node pairs; for cause-and-effect and long-range tasks, it is computed only over edges involving the memory or target node, because only those are predictable (Dizaji et al., 14 Jul 2025). T-GRAB also reports change-point $V_t$4 for periodicity tasks, which is critical for distinguishing true counting from persistence-based heuristics (Dizaji et al., 14 Jul 2025).

Two implementation details matter for interpreting results. First, CTDG methods are evaluated on discrete-time snapshots by converting edges to timestamped events (Dizaji et al., 14 Jul 2025). Second, the benchmark includes implementation modifications: CTAN required an added activation to avoid NaNs, and TGN uses TGAT as its embedding layer instead of the default TransformerConv because the default performed substantially worse in initial experiments (Dizaji et al., 14 Jul 2025). These details make T-GRAB not only a diagnostic benchmark but also a concrete reminder that benchmark conclusions depend on engineering choices, a point echoed by DyGLib’s re-evaluation of TGB and by BenchTemp’s emphasis on unified evaluation pipelines (Yu, 2023, Huang et al., 2023).

5. Empirical findings and diagnosed failure modes

A central result is that no single model dominates across all task families (Dizaji et al., 14 Jul 2025). GC-LSTM is strongest on periodic tasks, while TGAT and TGN dominate delayed cause-and-effect and long-range spatio-temporal tasks, with DyGFormer competitive but not universally best (Dizaji et al., 14 Jul 2025). CTAN underperforms surprisingly often (Dizaji et al., 14 Jul 2025). This is one of the benchmark’s strongest claims: apparently strong methods on standard datasets can have sharply different temporal failure modes when core skills are isolated.

On periodicity counting tasks $V_t$5, as $V_t$6 increases many models maintain or even improve overall $V_t$7, but their change-point performance degrades (Dizaji et al., 14 Jul 2025). The persistence baseline improves over all timesteps as $V_t$8 grows but scores zero at change points (Dizaji et al., 14 Jul 2025). This shows that aggregate performance can mask a failure to count. T-GRAB interprets this as reliance on persistence-like heuristics rather than genuine temporal state tracking (Dizaji et al., 14 Jul 2025). Static GCN and GAT can outperform some temporal models such as EvolveGCN in this regime (Dizaji et al., 14 Jul 2025).

On periodicity memorization $V_t$9, all TGNNs degrade as $E_t$0 increases; GC-LSTM and DyGFormer are strongest, while T-GCN, TGN, and TGAT remain robust up to $E_t$1 but fall sharply at $E_t$2 (Dizaji et al., 14 Jul 2025). On stochastic periodicity, GC-LSTM remains consistently best, with TGAT second-best and T-GCN more sensitive to noise (Dizaji et al., 14 Jul 2025). This suggests that recurrent inductive bias is advantageous for tracking recurring latent states.

On delayed cause-and-effect tasks, performance drops steadily as lag $E_t$3 increases (Dizaji et al., 14 Jul 2025). GC-LSTM and T-GCN degrade sharply already at $E_t$4 and $E_t$5, while DyGFormer, TGAT, and TGN degrade more gracefully (Dizaji et al., 14 Jul 2025). At long delays such as $E_t$6 and $E_t$7, all methods approach EdgeBank-level performance (Dizaji et al., 14 Jul 2025). The benchmark reads this as evidence that long-term temporal memory remains largely unsolved.

On long-range spatio-temporal tasks, DTDG methods perform well at short lags but degrade sharply as $E_t$8 grows, whereas TGAT and TGN remain more robust at larger temporal lags (Dizaji et al., 14 Jul 2025). DyGFormer is strong when spatial dependency is short, but TGAT and TGN outperform it as spatial distance increases (Dizaji et al., 14 Jul 2025). All methods degrade substantially when $E_t$9 (Dizaji et al., 14 Jul 2025). This indicates that long-range spatial aggregation remains difficult even for architectures that handle temporal lag relatively well.

The benchmark also studies temporal neighbor sampling for CTDG models. CTAN, TGN, and TGAT often improve when more historical neighbors are sampled; DyGFormer is more stable on some tasks; on $X_t$0, DyGFormer continues improving beyond 128 neighbors while others saturate (Dizaji et al., 14 Jul 2025). This suggests that some failures arise from restricted historical access, but others reflect architectural limits.

6. Position within the temporal graph benchmark landscape

T-GRAB occupies a distinct place relative to broader temporal graph benchmarks. TGB is a collection of large-scale real-world datasets for dynamic link property prediction and node affinity prediction, with chronological 70/15/15 splits, streaming evaluation, harder historical-negative sampling, and ranking metrics such as filtered MRR and NDCG@10 (Huang et al., 2023). BenchTemp provides a standardized framework for comparing TGNNs across link prediction and node classification, transductive and inductive settings, with unified preprocessing and efficiency reporting (Huang et al., 2023). DyGLib’s empirical evaluation of TGB shows that benchmark conclusions are sensitive to unified implementation quality and broader baseline coverage (Yu, 2023). TGB-Seq stresses unseen-edge prediction and sequential dynamics by curating low-repeat-ratio datasets, showing that current temporal GNNs often fail even simple sequential generalization despite strong performance on conventional benchmarks (Yi et al., 5 Feb 2025).

T-GRAB differs from all of these in its deliberate synthetic minimalism. Rather than maximizing realism or scale, it maximizes interpretability of temporal skill evaluation (Dizaji et al., 14 Jul 2025). A plausible implication is that T-GRAB is best used as a complement to real-world benchmarks rather than a replacement. Its diagnostic tasks can identify whether a model can count periodic phases, retain delayed dependencies, or compose spatial and temporal reasoning before that model is tested on noisier real data.

This positioning resembles the role of synthetic suites such as CLEVR, bAbI, or BSuite in other subfields, a comparison that T-GRAB itself makes explicitly (Dizaji et al., 14 Jul 2025). The benchmark’s value lies in revealing failure modes hidden by conventional aggregate scores.

7. Strengths, limitations, and significance

T-GRAB’s principal strength is that it makes temporal graph evaluation skill-specific and interpretable (Dizaji et al., 14 Jul 2025). Because the generative rule is known, failure at change points, long delays, or large path distances has a precise meaning. The benchmark also exposes that different model families encode different temporal inductive biases: recurrent architectures excel at periodicity, while attention and memory mechanisms help more with delayed dependency retrieval (Dizaji et al., 14 Jul 2025).

A second strength is methodological. By computing $X_t$1 over all possible node pairs and by reporting change-point diagnostics, T-GRAB avoids some evaluation artifacts common in link-prediction benchmarks (Dizaji et al., 14 Jul 2025). This is aligned with broader concerns in temporal graph benchmarking about negative-sampling bias, inconsistent splits, and implementation drift (Huang et al., 2023, Huang et al., 2023, Yu, 2023).

Its limitations follow directly from its design. It is synthetic and transductive, and it does not test inductive generalization to new nodes or domains (Dizaji et al., 14 Jul 2025). It focuses on dynamic link prediction rather than richer task formulations such as temporal question answering, timestamp prediction, causal intervention, temporal logic, or explanation generation. Its task families isolate temporal skills, but they do not capture the semantic richness of temporal knowledge graph forecasting, temporal QA, or document-grounded dynamic retrieval (Fan et al., 29 Jan 2026, Wen et al., 23 Feb 2026, Li et al., 3 Aug 2025). It also does not include explicit efficiency leaderboards beyond supporting analyses such as neighbor-sampling sensitivity, whereas real-world benchmark suites increasingly treat efficiency as a primary axis (Huang et al., 2023, Xing et al., 1 Jan 2026).

Even with those limitations, T-GRAB is significant because it identifies weaknesses that standard leaderboards often conceal. It shows that many TGNNs do not actually count, often fail to retain delayed information over long lags, and remain weak at joint spatio-temporal composition (Dizaji et al., 14 Jul 2025). In that sense, it sharpens the notion of temporal graph reasoning from a vague claim into a set of benchmarkable capabilities.

For temporal graph research, the broader implication is that evaluation should be layered. Real-world benchmarks such as TGB and TGB-Seq test ecological validity, distribution shift, and scalability (Huang et al., 2023, Yi et al., 5 Feb 2025). Diagnostic suites such as T-GRAB test whether a model possesses the temporal skills those real tasks presuppose (Dizaji et al., 14 Jul 2025). A comprehensive temporal graph reasoning agenda likely requires both.