CUDA Graphs: Definition & Performance

• CUDA Graphs are a GPU execution mechanism that captures a fixed directed acyclic graph (DAG) of operations to minimize host-side overhead.
• They are implemented via manual graph creation or stream capture and are used in tasks like machine learning inference and GPU communications.
• Performance improvements are achieved through optimal iteration batching and careful graph instantiation, though dynamic control flow remains challenging.

to=arxiv_search 天天中彩票公众号json {"query":"CUDA Graphs kernel launch overhead PyTorch UCX LLM inference cold start", "max_results": 10} to=arxiv_search 彩神争霸安卓json {"query":"CUDA Graphs kernel launch overhead PyTorch UCX LLM inference cold start", "max_results": 10} CUDA Graphs are a GPU execution mechanism and task-graph execution model for CUDA in which a fixed sequence of GPU operations is captured once as a directed acyclic graph (DAG) and then replayed with very little host-side involvement. In this model, graph nodes represent operations such as kernels or memory copies, and edges encode execution dependencies. The primary motivation is the reduction of repeated CPU-side launch overheads—Python execution, CUDA driver dispatch, API setup, synchronization, and CPU–GPU coordination—that become increasingly prominent when applications execute many short kernels or repeated iterative steps. Across machine learning, iterative scientific computing, GPU communication, and LLM serving, CUDA Graphs are used to consolidate repeated GPU work into reusable execution objects; their effectiveness, however, is conditioned by static graph structure, deterministic control flow, and stable execution context (Ekelund et al., 16 Jan 2025, Ghosh et al., 25 Mar 2025, Yadav et al., 25 Apr 2026, Liu et al., 8 Apr 2026).

1. Execution semantics and performance rationale

CUDA Graphs capture a sequence of GPU tasks as a static DAG and launch that DAG as a single unit. This differs from ordinary execution in which kernels are launched one by one from the CPU. The consolidation is operationally significant because the cost of a kernel launch is often dominated not by the kernel itself, but by fixed host-side overheads including driver interaction, OS/runtime work, API setup, and CPU–GPU coordination. As GPUs become faster and kernels shorter, those fixed costs occupy a larger fraction of end-to-end time. In ML workloads, a model may launch hundreds or thousands of kernels, plus memcpys and other GPU tasks; each CPU launch adds roughly 5–10 microseconds of latency, creating inter-kernel gaps that lower GPU utilization (Ekelund et al., 16 Jan 2025, Ghosh et al., 25 Mar 2025).

Two construction modes are described in the literature summarized here. One is manual graph creation, in which the program explicitly constructs graph nodes and dependencies. The other is stream capture, in which a region bracketed by cudaStreamBeginCapture and cudaStreamEndCapture is recorded into a graph, then instantiated with cudaGraphInstantiate and launched with cudaGraphLaunch. Manual creation provides precise control over node construction and dependency wiring, whereas stream capture fits higher-level frameworks such as PyTorch more naturally (Ekelund et al., 16 Jan 2025, Ghosh et al., 25 Mar 2025).

The central benefit of replay is that the GPU runtime executes the recorded DAG directly rather than requiring the CPU to relaunch each operation individually. In the LLM setting, this has been framed as avoiding repeated Python execution, repeated CUDA driver/kernel launch dispatch, and excess host–device synchronization, while making execution more deterministic and stable. In communication runtimes, the same mechanism bundles many staged copies and synchronization constraints into one reusable GPU-executable workflow, reducing CPU orchestration costs for recurring schedules (Yadav et al., 25 Apr 2026, Sojoodi et al., 24 Apr 2026).

2. Staticity, replay semantics, and graph lifecycle

A CUDA Graph is essentially fixed after capture. Its structure must remain unchanged, and many recorded values are fixed as well. This staticity imposes three recurring requirements: fixed tensor shapes, deterministic control flow, and stable memory allocation. These constraints make CUDA Graphs well suited to static, repetitive GPU work, but problematic for input-dependent control flow, shape changes, stochastic behavior, or runtime allocation patterns that conflict with replay (Yadav et al., 25 Apr 2026, Ghosh et al., 25 Mar 2025).

One correctness hazard is that kernel parameters are captured by value. If a parameter changes later, replay can still use the stale captured value, which can silently corrupt outputs or crash the program. A related issue is that captured CPU tensor addresses may become invalid. In LLM serving, the problem extends beyond topology: graph nodes may embed device pointers to model weights, KV cache, activations, and intermediate buffers; kernel nodes also reference kernel function handles resolved from modules or libraries lazily loaded during warmup. A graph is therefore not self-contained. Saving only topology is insufficient because topology alone cannot reconstruct the execution context needed for correct replay in a fresh process (Ghosh et al., 25 Mar 2025, Liu et al., 8 Apr 2026).

The graph lifecycle itself has measurable cost. In the UCX integration study, the lifecycle is separated into four phases: creation, construction, instantiation, and launch. The first execution is particularly expensive because these phases are all paid once, with instantiation identified as the heaviest first-time cost. For a 34-node graph corresponding to a 512 MB message under dual-path communication, instantiation can reach about 3 ms. Subsequent executions are much cheaper because cached graph reuse skips creation and construction, leaving launch overhead as the main recurring cost, although even launch overhead still scales with node count (Sojoodi et al., 24 Apr 2026).

Topology and parameterization must also be distinguished carefully. In Foundry, topology includes node types, ordering, dependencies, and some topology-stable node attributes, while per-node parameters include kernel arguments, gridDim, blockDim, and related launch settings. The paper further notes that kernel-node attributes set via cuGraphKernelNodeSetAttribute can remain stale and must therefore be treated as part of topology rather than as freely mutable parameters (Liu et al., 8 Apr 2026).

3. Structuring repeated computation and performance modeling

Iterative workloads are an especially natural fit for CUDA Graphs because they repeat the same kernel pattern many times. One explicit strategy is iteration batch unrolling: group a fixed number of loop iterations into a batch, unroll that batch into a graph, instantiate and upload the graph, and then launch that graph repeatedly. The cited work presents this as analogous to loop unrolling in compiler optimization and as a generalized template for converting iterative solvers to CUDA Graphs (Ekelund et al., 16 Jan 2025).

The performance model separates total time into graph creation overhead and execution time:

$T = T_C + T_E.$

Creation time is modeled as growing linearly with batch size $S$:

$T_C = k_c S + b_c.$

Execution time is simplified to

$T_E = \frac{a}{S} + b.$

This yields an optimal batch size

$S^* = \arg\min_S (T_C + T_E).$

The model’s key condition is that performance gains are obtained when the time to switch to the next kernel inside the graph, $T_i$, is lower than the time to switch from one launched kernel to the next launched kernel, $T_a$. In that regime, increasing $S$ reduces the amortized transition cost represented by the $a/S$ term (Ekelund et al., 16 Jan 2025).

Empirically, the optimal iteration batch size is reported as platform-dependent but workload-independent, with best batch sizes around 50–100 kernels and approximately 100 nodes recommended for the tested system. The same study reports that the optimal batch size gives more than 1.4x speed-up in the skeleton application, that the graph version starts to outperform the baseline after about two to three iteration batches, and that similar speed-up can be gained in Hotspot, Hotspot3D, and an FDTD Maxwell solver. Larger graphs increase memory usage linearly, and graph creation overhead grows linearly with batch size up to about 2,500 nodes, beyond which measured execution behavior becomes less favorable (Ekelund et al., 16 Jan 2025).

A plausible implication is that graph construction must be treated as an optimization problem rather than as a purely mechanical replacement of ordinary launches. The same source explicitly argues against naively putting every iteration into one huge graph, because graph creation cost and memory footprint eventually offset replay benefits (Ekelund et al., 16 Jan 2025).

4. Compiler integration and hybrid runtimes for machine learning

Machine-learning deployments expose both the benefits and the limitations of CUDA Graphs. In transformer inference, a useful partition is between graph-safe static operations and runtime-dependent dynamic operations. The hybrid LLM runtime literature classifies linear projections, layer normalization, matrix multiplications, attention score computation, fused attention kernels, and transformer layer compute as static operations with fixed shapes, deterministic control flow, and stable allocation patterns; token sampling, KV-cache updates, positional embedding extension, shape adaptation logic, and stochastic masking or randomized components remain dynamic (Yadav et al., 25 Apr 2026).

Execution region	Operations
Captured in CUDA Graph	matrix multiplications; fused attention kernels; normalization layers; linear projections; attention score computation; other deterministic transformer compute
Kept dynamic / JIT	stochastic sampling; KV-cache management; shape normalization/adaptation; preprocessing; token decoding logic; runtime-dependent control flow

The hybrid JIT–CUDA Graph runtime splits inference into a static domain executed by CUDA Graph replay and a dynamic domain executed by JIT-compiled code. It uses two CUDA streams—$\mathsf{S_{cap}}$ for capture and $S$0 for replay—and a rolling CUDA Graph buffer $S$1. In Algorithm 1, the system pre-captures short-sequence graphs for $S$2 during warm-up, preprocesses the current context tensor $S$3, and then replays a matching graph $S$4 if one exists; otherwise, it executes the static segment via JIT, asynchronously captures a new graph on $S$5, inserts it into the rolling buffer, and returns the resulting hidden state $S$6 for token decoding through $S$7. The design also uses a two-process architecture consisting of a JIT Context Generator and a CUDA Graph Generator, with device-pointer-based communication, asynchronous graph generation threads, explicit CUDA event synchronization, and least-used eviction in the rolling cache. Practical integration points include FlashAttention v2, nvFuser LayerNorm, and paged KV-cache kernels from vLLM inside static graph regions (Yadav et al., 25 Apr 2026).

On LLaMA-2 7B with a single NVIDIA H100 GPU, batch size 1, FP16 precision, and prompt lengths from 10 to 500 tokens, this hybrid runtime reports the lowest TTFT among HuggingFace Transformers / PyTorch eager, TensorRT–LLM, and Hybrid JIT + CUDA Graph. At 10 tokens, the reported TTFTs are 24.65 ms, 16 ms, and 13.36 ms respectively; at 500 tokens, they are 48.96 ms, 88 ms, and 17.79 ms. The reported overall TTFT speedups are 1.02× to 5.90× versus PyTorch Eager and 1.04× to 5.42× versus TensorRT–LLM, with up to 66.0% TTFT reduction. For P99 per-token latency, the hybrid runtime reports a 20.2% average reduction relative to TensorRT–LLM and a 43.5% average reduction relative to PyTorch Eager (Yadav et al., 25 Apr 2026).

Compiler support is equally important because naïve deployment can be counterproductive. PyGraph, integrated into the PyTorch2 compilation toolchain, is motivated by three observations: PyTorch2 rejects some graphifiable cases that can be repaired; nearly 60% of CUDA Graph overhead can be attributed to parameter data copies; and blind deployment is frequently harmful. Across 416 CUDA Graphs from 183 applications, 143 hurt performance, 82 gave no meaningful speedup, and only 191 improved performance by more than 2%. PyGraph therefore combines CUDA-Graph-aware Data Placement, Kernel Parameter Indirection, and selective deployment. Data Placement rewrites CPU-resident scalars or host-resident tensors so more kernels become graph-capturable; coverage rises, for example, from 4.66% to 78.08% for Speech Transformer, from 76.51% to 99.14% for DALLE, from 1.27% to 98.58% for XLNet-T, and from 0% to 100% for XLNet-I. Kernel Parameter Indirection converts expensive parameter data copies into 8-byte pointer copies by passing pointer-to-pointer arguments in Triton-generated kernels, often reducing copied bytes by around 99%. Selective deployment profiles three choices—no graph, graph without indirection, and graph with indirection—and caches the best-performing option (Ghosh et al., 25 Mar 2025).

A recurrent misconception is therefore that more graphification is always better. The PyGraph study shows explicitly that CUDA Graphs can hurt performance in many cases, including worst-case slowdown of 36% on EOS under naïve PyTorch2-CG, while the communication and iterative-computing studies similarly show that payoff depends on graph size, repetition, and amortization (Ghosh et al., 25 Mar 2025, Sojoodi et al., 24 Apr 2026, Ekelund et al., 16 Jan 2025).

5. Communication schedules and GPU-to-GPU transfer workflows

CUDA Graphs are not restricted to kernel-heavy computation; they are also used to encode repeated communication schedules. In the UCX integration work, the graph-based execution object represents a multi-path intra-node GPU-to-GPU communication workflow inside uct_cuda, specifically around cuda_ipc. The system extends standard UCT::CUDA-IPC with a multi-path communication handler, a 2-D pipelining engine, a CUDA Graph engine, and a graph cache. The 2-D pipelining engine partitions a message horizontally across paths and vertically into chunks, with per-path chunk sizes computed as

$S$8

Graph construction then chooses one of three transfer modes for each path: direct peer-to-peer copy for a direct GPU path, host-staged copy for PCIe-through-host transfers, or staging-GPU copy through an intermediate GPU (Sojoodi et al., 24 Apr 2026).

In this setting, graph nodes are primarily memcpy nodes, and dependencies encode staging and completion ordering. PeerToPeerCopy creates device-to-device memcpy nodes, StageHostCopy creates device-to-host followed by host-to-device nodes with a dependency, and StageGPUCopy creates the analogous sequence using an intermediate GPU. The graph instance is cached by communication configuration using a least-recently-used eviction policy in a fixed-size hash table, tunable via environment variables. Launch then becomes a graph lookup followed by replay on the proper stream, with CUDA events used so UCX can synchronize with the input stream (Sojoodi et al., 24 Apr 2026).

The performance results show that graphified multi-path schedules can substantially reduce CPU-side communication overhead. On four-GPU Beluga and Narval nodes, the reported headline improvement is up to 2.95x bandwidth over the standard single-path UCT::CUDA-IPC baseline at message sizes up to 512MB. In UCX Put bandwidth, using three GPU paths and optionally a host-staging path yields up to 2.85× improvement on Beluga and 2.75× on Narval for messages larger than 32 MB; enabling CUDA Graphs raises this to as much as 2.95× on Beluga and 2.85× on Narval. For Jacobi with four MPI ranks and 1000 iterations, runtime improves by up to 1.26× on Beluga and 1.15× on Narval with two concurrent paths, and to 1.28× and 1.16× respectively with CUDA Graphs added (Sojoodi et al., 24 Apr 2026).

The same study also defines clear boundaries. Small messages below about 8 MB are a poor fit because the graph contains too few nodes and launch overhead can dominate. Adding a host-staging path contributes only marginally, typically up to about 15%, and in bidirectional bandwidth tests can hurt performance because both directions contend for the same PCIe path through the host. This suggests that, in communication systems as in ML inference, graph replay is most valuable when the schedule is repeated, fine-grained, and large enough for one-time setup and launch costs to be amortized (Sojoodi et al., 24 Apr 2026).

6. Persistence, cold start, and execution-context materialization

CUDA Graph usage in production LLM serving introduces a distinct systems problem: cold-start latency. Recent serving systems can load model weights in seconds, but graph capture itself can take tens of seconds to minutes and often dominates startup. Foundry addresses this by treating persistence as template-based CUDA graph context materialization rather than as simple graph serialization. The core claim is that cold start is dominated by reconstructing execution context, not merely graph topology (Liu et al., 8 Apr 2026).

Foundry’s SAVE phase performs normal warmup and graph capture once while intercepting CUDA driver calls, recording serialized graph metadata, deterministic memory-layout information, and kernel binaries or module-loading information. Its LOAD phase reconstructs executable graphs in a fresh process without rerunning warmup or capture. Two context components are materialized explicitly. First, deterministic memory layout is enforced by interposing on CUDA virtual memory management APIs, reserving a large virtual address region at a fixed base address, redirecting allocations into that region, and replaying capture-window allocations during LOAD so that embedded device pointers remain valid even though the LOAD path skips graph capture. Second, kernel binaries are extracted and reloaded by intercepting cuModuleLoad and cuLibraryLoad, recording binary payloads, load APIs, options, and a catalog keyed by binary hash and mangled kernel name; LOAD then resolves kernel function handles from this catalog (Liu et al., 8 Apr 2026).

Foundry also exploits topology regularity through templating. Graphs are grouped by a topology key derived from node types, node order, dependency structure, and certain topology-affecting attributes. One template graph is built per unique topology, and CUDA graph-executable update APIs such as cuGraphExecUpdate are used to specialize non-template graphs by updating kernel function, argument buffer, and launch dimensions in place. For Qwen3-14B across batch sizes 1–512, 512 captured graphs reduce to only 22 unique topologies. Across 512 captured graphs in the broader evaluation, unique templates range from 12 to 25, so 95–98% of graphs are served by on-demand parameter update rather than full graph construction. Reported per-graph costs are about 59.7–198.6 ms for stream capture, 31.1–69.5 ms for driver-API template construction, and 0.98–2.89 ms for in-place update (Liu et al., 8 Apr 2026).

The system further extends single-GPU offline capture to multi-GPU distributed serving for SPMD-style DP, TP, and EP by using a stub layer over communication libraries such as NCCL and NVSHMEM during SAVE, then patching rank-dependent communication state during LOAD. Pipeline parallelism is explicitly excluded because topology is not invariant across ranks (Liu et al., 8 Apr 2026).

The operational impact is large. For dense models, Foundry reduces Qwen3-14B initialization from 36–48 s to 1.7–1.8 s, Llama3-8B from about 28 s to 1.3 s, and Gemma3-12B from about 45 s to 2.0 s. For MoE models in BF16, Qwen3-30B-A3B drops from 112–154 s to 2.7–2.8 s, and Qwen3-235B-A22B EP8 from 650 s to 3.9 s, which is the reported headline reduction of about 99%. Throughput is preserved: using TPOT, Foundry and native vLLM graphs overlap almost perfectly across data parallelism scales, dense and MoE models, BF16 and FP8, and different GPU generations, and generated tokens are identical between Foundry and vLLM (Liu et al., 8 Apr 2026).

Taken together, these results delimit the regimes in which CUDA Graphs are most effective. They are a strong fit for repeated decoding steps, mostly stable transformer compute, many short iterative kernels, and repeated fine-grained communication schedules. They are less attractive when control flow is highly dynamic, when graphs are too small for launch savings to dominate, when parameter-copy overhead is large relative to useful work, or when communication or execution patterns are sparse and irregular. The literature surveyed here therefore presents CUDA Graphs not as a universal accelerator, but as a reusable execution substrate whose value depends on graph-safe structure, amortizable repetition, and accurate management of execution context (Yadav et al., 25 Apr 2026, Ekelund et al., 16 Jan 2025, Ghosh et al., 25 Mar 2025, Sojoodi et al., 24 Apr 2026, Liu et al., 8 Apr 2026).