D3-GNN: Streaming Graph Neural Network System

• D3-GNN is a distributed, hybrid-parallel streaming GNN system that incrementally updates node embeddings in dynamic graphs without global recomputation.
• It employs incremental streaming aggregators, multi-level windowed forward-pass schemes, and distributed computation graph unrolling to optimize latency, memory, throughput, and load-balance.
• Empirical benchmarks demonstrate up to 76× higher throughput, 10× faster runtime, and 15× lower network volume in real-time graph applications.

D3-GNN is a distributed, hybrid-parallel streaming Graph Neural Network (GNN) system engineered for real-time graph workloads under an online query setting. Designed to address both algorithmic and systems-level challenges, D3-GNN enables continuous, low-latency updates of node embeddings in dynamically evolving graphs without requiring global recomputation. The system introduces incremental streaming GNN aggregators, distributed computation graph unrolling, and multi-level windowed forward-pass mechanisms to optimize latency, memory, throughput, and load-balance, achieving significant performance gains over existing frameworks (Guliyev et al., 2024).

1. Streaming Graph Neural Network Problem Formulation

D3-GNN models the input as a temporal graph $G(t) = (V(t), E(t), X_V(t), X_E(t))$ where the vertex set, edge set, and associated features evolve over time via a discrete event stream $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$. Events include edge or node insertions/deletions and feature updates. The central objective is to maintain, for each node $v$, a real-time embedding $h_v^{(L)}(t)$ that approximates the outcome of a full $L$-layer GNN forward pass on $G(t)$, updating only the minimal influenced subset $\mathcal{I} \subset V$ as graph events occur. This incremental update process seeks sublinear time complexity relative to $|V|$ and $|E|$ rather than full-graph recomputation.

Concretely, for message-passing GNNs with the layered update recursions

$$\begin{aligned} m_{u\to v}^{(l)}(t) &= \phi^{(l)}(h_u^{(l-1)}(t), h_v^{(l-1)}(t), x_{uv}(t)), \ a_v^{(l)}(t) &= \rho^{(l)}(\{m_{u\to v}^{(l)}(t) : u\in N_{in}(v, t)\}), \ h_v^{(l)}(t) &= \psi^{(l)}(h_v^{(l-1)}(t), a_v^{(l)}(t)), \end{aligned}$$

D3-GNN implements incremental propagation of only those updates attributable to a given atomic input change, relying on explicit, fine-grained dependency tracking.

2. System Architecture and Dataflow

The core architecture of D3-GNN is an Apache Flink pipeline composed of sequential stages: Source, Parser, Partitioner, Splitter, and a chain of GraphStorage operators (one per GNN layer), terminating at Output/Trainer. Key elements include:

• Partitioner: Implements streaming vertex-cut algorithms (HDRF, CLDA) assigning logical partition keys to events, promoting load-balance by distributing high-degree nodes’ incident edges.
• Splitter: Filters events to the appropriate layer for processing, filtering irrelevant state.
• GraphStorage Operator: Stores sharded subgraphs, maintains per-layer parameters, and orchestrates local GNN aggregator states.
• Unrolled Computation: Each GraphStorage instance realizes a logical layer in the GNN, with updates cascading sequentially from layer 0 to layer $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$0 via asynchronous message passing.

Fault tolerance leverages Flink’s Chandy–Lamport exactly-once checkpointing: operator/keyed state is persisted such that, upon recovery, replaying in-flight events restores pipeline consistency. Load-balance is further refined using an explosion factor $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$1 and combined intra-/inter-layer windowing (see Section 4).

3. Dynamic Streaming GNN Aggregation

Each node $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$2 and GNN layer $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$3 in D3-GNN maintains a mutable, incremental aggregator state $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$4. Incoming messages $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$5 perturb this state via efficient, operator-specific updates: reduce ($\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$6), remove ($\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$7), or replace ($\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$8), with $\mathcal{U} = \{u_i = (\tau_i, op_i, elem_i)\}$9 defined by the combinator $v$0 (sum, mean, max, etc.).

As soon as $v$1 is modified, $v$2 is recomputed and propagated upward through the layer chain—immediately triggering downstream updates via remote method invocation. This structure reduces the per-edge update cost from $v$3 for naïve approaches to $v$4, facilitating real-time responsiveness to graph events.

4. Multi-Level Windowed Forward-Pass Schemes

To address issues of neighborhood explosion, skew, and burstiness induced by high-degree nodes or workload irregularity, D3-GNN introduces complementary intra- and inter-layer windowing. These mechanisms buffer, coalesce, and batch updates before triggering forward passes, trading potentially higher single-event latency for dramatically improved throughput and message efficiency.

• Intra-layer window $v$5: Accumulates multiple forward calls per vertex $v$6 over a range $v$7, merging them into a single update per window interval.
• Inter-layer window $v$8: Aggregates reduce operations to the same node destination over $v$9, applying local batching before network transmission.

Window eviction policies include fixed-interval tumbling windows, inactivity-based session windows, and adaptively-sized windows using exponential means (CountMinSketch-based).

Empirically, windowing decreases running time by up to $h_v^{(L)}(t)$0 and network message volumes by up to $h_v^{(L)}(t)$1, with at most $h_v^{(L)}(t)$2 added latency per window, especially pronounced at higher degrees of parallelism and skew.

5. Distributed Computation Graph Unrolling

D3-GNN inherently "unrolls" the $h_v^{(L)}(t)$3 GNN layers into a pipeline of chained GraphStorage operators, with each operator representing a distinct logical layer and managing a share of the relevant subgraph. Events propagate asynchronously from the source through the $h_v^{(L)}(t)$4-layer pipeline, with state and updates keyed by logical vertex partitioning.

Event-driven scheduling eliminates barriers and straggler-induced delays: as soon as all dependencies for a reduce or forward operation are satisfied, execution proceeds immediately, maximizing pipeline utilization and concurrency. This approach obviates the need for explicit subgraph materialization at query time and supports both streaming inference and synchronous, stale-free GNN training.

6. Implementation and Performance

D3-GNN is implemented atop Apache Flink 1.14 employing the DJL/PyTorch backend for GNN model execution. Key engineering solutions include:

• Multi-threaded partitioning with dynamic logical-to-physical key hashing for online resource scaling.
• Custom serialization and selective broadcast to minimize RMI overhead.
• In-memory single-producer-single-consumer queues to support iteration-intensive operations (training gradients, termination signaling) with consistent checkpointing.
• Per-thread tensor cleanup routines to reduce JVM garbage collection latency.
• CountMinSketch-based structures for adaptive windowing.

Extensive benchmarks on 1–10 machines (each 20 cores/40 threads, 64GB RAM) used real-world graph streams (sx-superuser, reddit-hyperlink, stackoverflow, ogb-products, wikikg90Mv2; up to 601M edges). In comparison to DGL:

• Streaming inference throughput is up to $h_v^{(L)}(t)$5 greater.
• Windowed D3-GNN reduces runtime by $h_v^{(L)}(t)$6 and network message volume by up to $h_v^{(L)}(t)$7.
• Load imbalance factor reduces from $h_v^{(L)}(t)$8 (streaming) to $h_v^{(L)}(t)$9 (windowed) for highly skewed graphs.
• Training throughput matches or exceeds DGL at larger cluster scales.
• Tuning the explosion factor $L$0 optimizes pipeline parallelism, with windowed D3-GNN rendering performance less sensitive to $L$1 and partitioner choice.

A tabular summary of empirical results:

Aspect	D3-GNN	DGL (best)
Streaming Throughput	$L$2 higher	Baseline
Runtime (windowed)	Up to $L$3 faster	Baseline
Network Volume	Up to $L$4 lower	Baseline
Load Imbalance Factor	$L$5	$L$6

7. Applications, Limitations, and Future Work

D3-GNN provides a unified system for real-time graph learning where full-graph recomputation is infeasible—enabling applications in fraud detection, recommendation, and social monitoring on large-scale, rapidly changing graphs. Its architectural strengths are in incremental update propagation, distributed unrolled computation, and systematic mitigation of data/compute skew.

Limitations include current focus on message-passing GNNs (MPGNNs); extension to memory-augmented temporal models (GRU/LSTM variants), heterogeneous multi-relation graphs, and benchmarked edge deletions is pending. Future directions encompass automatic window and explosion-factor tuning, and application of approximate sketching for very high-degree nodes. These enhancements aim to generalize D3-GNN’s applicability to broader GNN and streaming graph learning domains while maintaining deterministic, low-latency semantics (Guliyev et al., 2024).