Understanding and Reducing Metadata-Driven Host Overheads in Sampling-Based GNN Training

Abstract: Modern deep learning workloads increasingly exhibit dynamic, metadata-driven execution, where runtime-generated information determines memory provisioning and kernel launch decisions. In sampling-based graph neural network (GNN) training, this behavior places the CPU on the critical path, introducing persistent host-device orchestration overhead and frequent GPU-CPU synchronization, which dominate end-to-end runtime when GPU computation is small. Existing approaches, including CUDA Graphs and GPU dynamic parallelism, fail to address this problem because the metadata-driven control loop remains host-mediated, and execution structure varies across iterations. We present ZEROGNN, a system that removes the host from the metadata-driven control loop and enables fully GPU-resident execution under dynamic behavior. ZEROGNN keeps runtime metadata on-device, mediates dynamic execution within a fixed launch structure, and provisions a conservative yet tight execution envelope to restore CUDA Graph replayability. Experiments on sampling-based GNN workloads show that ZEROGNN achieves up to 5.28 x end-to-end speedup, near 100% GPU execution fraction, and memory efficiency comparable to ideal metadata-informed allocation, while enabling strong multi-GPU scaling by eliminating host-side bottlenecks.

• The paper introduces ZeroGNN, which eliminates host-device orchestration by enabling fully device-resident execution for metadata-driven dynamic control in sampling-based GNN training.
• It leverages novel techniques including Device-Resident Metadata Buffer, Device-Side Launch Mediation, and Metadata-Free Dispatcher to bypass host mediation and improve GPU utilization.
• ZeroGNN achieves up to 5.28× end-to-end speedup, near-100% GPU execution, and substantial memory efficiency improvements compared to conventional frameworks.

Understanding and Reducing Metadata-Driven Host Overheads in Sampling-Based GNN Training

Introduction and Problem Statement

Sampling-based Graph Neural Network (GNN) training pipelines on GPU clusters are increasingly bottlenecked by metadata-driven host-device orchestration overheads rather than by raw GPU compute. This bottleneck originates from dynamic subgraph construction: every mini-batch requires runtime generation of metadata (e.g., sampled node/edge counts) on GPU, which must immediately drive memory provisioning, kernel launch parameters, and pipeline scheduling. Persistently, existing frameworks enforce a host-mediated control flow for these decisions, resulting in frequent GPU-to-CPU synchronizations, intrinsic loss of asynchrony, and severe under-utilization of the GPU. Classic pipeline optimizations and even CUDA Graphs fail to address this, as they cannot replay dynamic, metadata-driven workloads.

A key technical question addressed is: Can we completely eliminate host-device orchestration and enable fully GPU-resident execution for dynamic, metadata-driven GNN training workloads with efficient memory and launch resource provisioning?

Analysis of Host-Device Overhead Bottlenecks

End-to-End Bottleneck Characterization

The empirical analysis identifies that the two dominant contributors to end-to-end training time in sampling-based GNNs are:

• Subgraph sampling (stage 1)
• GNN training on the sampled subgraph (stage 3)

  Figure 1: Stage-wise breakdown of end-to-end training time, showing the dominance of sampling and training stages.

Moreover, the GPU execution fraction—the ratio of active GPU time to total training time—is low, particularly at realistic batch sizes (substantially below 50% for moderate batch sizes in DGL). This is because host-device orchestration overhead (HDOO) remains constant per iteration, irrespective of subgraph scale, while GPU kernels become fast for small batches, shifting the critical path to the host.

Figure 2: GPU execution fraction for DGL GraphSAGE stages across batch sizes, highlighting low GPU utilization for small and moderate batches due to host-side synchronization overheads.

Attempts to trim framework code and use C++ modules (e.g., the lightweight "Gong et al." baseline) yield limited improvements; GPU utilization remains far below saturation even with aggressive software optimizations.

Figure 3: End-to-end runtime and GPU utilization for DGL vs. Gong et al. across batch sizes—host overheads remain a bottleneck.

Root Cause: Host-Mediated Dependency Barriers

The source of the overhead is a host-mediated dependency barrier (HMDB). Each sampled subgraph produces dynamic metadata (counts, adjacency structures) that are immediately required by subsequent pipeline stages. Existing systems require GPU → CPU → GPU round-trip: the GPU computes the metadata, the CPU uses it to size memory/kernel launches, and finally GPU computation resumes. This sequentializes the pipeline and breaks replayability, as CUDA Graphs cannot capture varying memory addresses or kernel launches. The issue compounds for multi-hop sampling and across multi-GPU setups, where orchestration doesn't parallelize and limits scalability.

Figure 4: Execution flow with host-device coordination in sampling-based GNNs, revealing the repeated round-trip serialization imposed by metadata-driven dependencies.

ZeroGNN Architecture

ZeroGNN addresses these bottlenecks by enforcing fully device-resident execution for metadata-driven dynamic control, with three key system-level contributions:

1. Device-Resident Metadata Buffer (DRMB): Keeps all dynamic metadata on device as pointers, enabling downstream kernel specialization and eliminating GPU→CPU materialization.
2. Device-Side Launch Mediation (DLM): All kernel launches are mediated on the GPU using conservative upper bounds; runtime sizes are checked against DRMB, and any over-allocated threads perform early exit.
3. Metadata-Free Dispatcher (MFD): Tight, yet conservative, upper bounds on subgraph sizes are derived using statistical properties of multi-hop sampled graphs, allowing for single-shot allocation and kernel launch structure that satisfy CUDA Graph replayability. Overflow cases (rare) are handled with safe fallbacks, maintaining correctness.

   Figure 5: ZeroGNN architecture overview, ensuring metadata-driven replayable execution with only device-side mediation.

This design fully removes HMDB, restores CUDA Graph replay, and enables the host to entirely decouple from iteration-to-iteration dynamic control, recovering asynchrony and memory safety without per-iteration host logic.

Memory Management and Efficiency

A naive solution allocates for the theoretical maximum subgraph (product of batch size and per-layer fanouts), leading to exponential memory blow-up. ZeroGNN instead leverages the high empirical stability of deduplicated sampled subgraph sizes—formally modelled as a Poisson-binomial distribution with tight concentration—using this to define a high-confidence execution envelope.

Figure 6: SpMM kernel runtime is largely insensitive to over-allocation, justifying conservative static upper bounds for kernel resource provisioning.

Figure 7: ZeroGNN achieves memory usage comparable to dynamic, optimal allocation approaches, outperforming DGL and eliminating the exponential overhead of naive strategies.

Figure 8: Memory usage efficiency, log₂ scale versus the naive maximum allocation baseline, demonstrating sustained 10.84× improvement across sampling depths.

Numerical Results and Empirical Validation

ZeroGNN yields substantial improvements over DGL, Gong et al., and dynamic parallelism-based baselines (CU-DPI):

• End-to-end training speedup: Up to 5.28× over DGL; 2.92× over Gong et al.; 2.33× over CU-DPI.
• Sampling phase speedup: Up to 17.69× over DGL (sampling only).
• Memory efficiency: Up to 3.41× less than DGL; 10.84× less than naive maximal.
• GPU execution fraction: Near-100% for both sampling and end-to-end training (unique among all frameworks).
• Multi-GPU scaling: Preserves speedup and strong scaling even for small batch sizes, indicating elimination of per-worker host orchestration bottleneck.

  Figure 9: ZeroGNN delivers accuracy on par with DGL across labeled datasets.

  

Figure 10: Sampling runtime speedup over DGL and Gong et al. across datasets, with ZeroGNN sustaining significant accelerations.

Figure 11: End-to-end runtime speedup for ZeroGNN across different datasets.

Figure 12: GPU execution fraction during sampling, showing ZeroGNN at full GPU saturation compared to suboptimal utilization for other frameworks.

Figure 13: End-to-end runtime speedup for ZeroGNN in multi-GPU settings, validating preserved scaling.

ZeroGNN also demonstrates robust performance on large graphs (e.g., OGBN-papers100M), achieving 2.31–2.70× speedup over Gong et al. with consistent memory efficiency.

Implications and Future Directions

ZeroGNN's approach cleanly decouples the execution pipeline from host-mediated bottlenecks, restoring GPU asynchronicity and enabling pipeline replayability for dynamic, metadata-driven workloads that previously were intractable for CUDA Graphs. This opens several further research avenues:

• Integration with Operator-level GPU Optimizations: By removing the host from the pipeline, further kernel- or operator-level optimizations for GNN computation can realize their full potential.
• Generality to Other Dynamic Workloads: The design pattern—tight envelope allocation, device-side control, and replayable skeletons—has generality to non-GNN dynamic workloads in deep learning, such as adaptive routing or reinforcement learning pipelines.
• Reproducible, Deterministic Execution: Fixed launch and allocation structure, along with fallback mechanisms, open paths to reproducible and deterministic training even for stochastic dataflow.
• Scalability Beyond Multi-GPU: With strong-scaling host independence, adaptation to multi-node environments and large-scale distributed DL is facilitated.

Conclusion

ZeroGNN demonstrates that the central bottleneck in sampling-based GNN training is not GPU compute but rather persistent host-device orchestration induced by runtime metadata-driven dynamic control. By colocating all control decisions, metadata, and memory provisioning on the device and employing tight probabilistic bounds for resource allocation, ZeroGNN eliminates the host from the critical path, enabling highly efficient, fully GPU-resident execution with near-perfect GPU utilization, strong multi-GPU scaling, and substantial memory and performance gains. The principles introduced are broadly applicable and signal a shift toward system architectures where dynamic, data-dependent machine learning workloads can finally escape the inefficiencies of CPU-GPU roundtrips.