---
title: 'CUDA Graphs: Definition & Performance'
url: https://www.emergentmind.com/topics/cuda-graphs
type: topic
---

# CUDA Graphs: Definition & Performance

to=arxiv_search  天天中彩票公众号json
{"query":"CUDA Graphs kernel launch overhead PyTorch UCX LLM inference cold start", "max_results": 10}
to=arxiv_search  彩神争霸安卓json
{"query":"CUDA Graphs kernel launch overhead PyTorch UCX LLM inference cold start", "max_results": 10}
CUDA Graphs are a GPU execution mechanism and task-graph execution model for CUDA in which a fixed sequence of GPU operations is captured once as a directed acyclic graph (DAG) and then replayed with very little host-side involvement. In this model, graph nodes represent operations such as kernels or memory copies, and edges encode execution dependencies. The primary motivation is the reduction of repeated CPU-side launch overheads—Python execution, CUDA driver dispatch, API setup, synchronization, and CPU–GPU coordination—that become increasingly prominent when applications execute many short kernels or repeated iterative steps. Across machine learning, iterative scientific computing, GPU communication, and LLM serving, CUDA Graphs are used to consolidate repeated GPU work into reusable execution objects; their effectiveness, however, is conditioned by static graph structure, deterministic control flow, and stable execution context [2501.09398] [2503.19779] [2604.23467] [2604.06664].

## 1. Execution semantics and performance rationale

CUDA Graphs capture a sequence of GPU tasks as a static DAG and launch that DAG as a single unit. This differs from ordinary execution in which kernels are launched one by one from the CPU. The consolidation is operationally significant because the cost of a kernel launch is often dominated not by the kernel itself, but by fixed host-side overheads including driver interaction, OS/runtime work, API setup, and CPU–GPU coordination. As GPUs become faster and kernels shorter, those fixed costs occupy a larger fraction of end-to-end time. In ML workloads, a model may launch hundreds or thousands of kernels, plus memcpys and other GPU tasks; each CPU launch adds roughly 5–10 microseconds of latency, creating inter-kernel gaps that lower GPU utilization [2501.09398] [2503.19779].

Two construction modes are described in the literature summarized here. One is **manual graph creation**, in which the program explicitly constructs graph nodes and dependencies. The other is **stream capture**, in which a region bracketed by `cudaStreamBeginCapture` and `cudaStreamEndCapture` is recorded into a graph, then instantiated with `cudaGraphInstantiate` and launched with `cudaGraphLaunch`. Manual creation provides precise control over node construction and dependency wiring, whereas stream capture fits higher-level frameworks such as PyTorch more naturally [2501.09398] [2503.19779].

The central benefit of replay is that the GPU runtime executes the recorded DAG directly rather than requiring the CPU to relaunch each operation individually. In the LLM setting, this has been framed as avoiding repeated Python execution, repeated CUDA driver/kernel launch dispatch, and excess host–device synchronization, while making execution more deterministic and stable. In communication runtimes, the same mechanism bundles many staged copies and synchronization constraints into one reusable GPU-executable workflow, reducing CPU orchestration costs for recurring schedules [2604.23467] [2604.22228].

## 2. Staticity, replay semantics, and graph lifecycle

A CUDA Graph is essentially fixed after capture. Its structure must remain unchanged, and many recorded values are fixed as well. This staticity imposes three recurring requirements: fixed tensor shapes, deterministic control flow, and stable memory allocation. These constraints make CUDA Graphs well suited to static, repetitive GPU work, but problematic for input-dependent control flow, shape changes, stochastic behavior, or runtime allocation patterns that conflict with replay [2604.23467] [2503.19779].

One correctness hazard is that kernel parameters are captured by value. If a parameter changes later, replay can still use the stale captured value, which can silently corrupt outputs or crash the program. A related issue is that captured CPU tensor addresses may become invalid. In LLM serving, the problem extends beyond topology: graph nodes may embed device pointers to model weights, KV cache, activations, and intermediate buffers; kernel nodes also reference kernel function handles resolved from modules or libraries lazily loaded during warmup. A graph is therefore not self-contained. Saving only topology is insufficient because topology alone cannot reconstruct the execution context needed for correct replay in a fresh process [2503.19779] [2604.06664].

The graph lifecycle itself has measurable cost. In the UCX integration study, the lifecycle is separated into four phases: creation, construction, instantiation, and launch. The first execution is particularly expensive because these phases are all paid once, with instantiation identified as the heaviest first-time cost. For a 34-node graph corresponding to a 512 MB message under dual-path communication, instantiation can reach about 3 ms. Subsequent executions are much cheaper because cached graph reuse skips creation and construction, leaving launch overhead as the main recurring cost, although even launch overhead still scales with node count [2604.22228].

Topology and parameterization must also be distinguished carefully. In Foundry, topology includes node types, ordering, dependencies, and some topology-stable node attributes, while per-node parameters include kernel arguments, `gridDim`, `blockDim`, and related launch settings. The paper further notes that kernel-node attributes set via `cuGraphKernelNodeSetAttribute` can remain stale and must therefore be treated as part of topology rather than as freely mutable parameters [2604.06664].

## 3. Structuring repeated computation and performance modeling

Iterative workloads are an especially natural fit for CUDA Graphs because they repeat the same kernel pattern many times. One explicit strategy is **iteration batch unrolling**: group a fixed number of loop iterations into a batch, unroll that batch into a graph, instantiate and upload the graph, and then launch that graph repeatedly. The cited work presents this as analogous to loop unrolling in compiler optimization and as a generalized template for converting iterative solvers to CUDA Graphs [2501.09398].

The performance model separates total time into graph creation overhead and execution time:
$$
T = T_C + T_E.
$$
Creation time is modeled as growing linearly with batch size $S$:
$$
T_C = k_c S + b_c.
$$
Execution time is simplified to
$$
T_E = \frac{a}{S} + b.
$$
This yields an optimal batch size
$$
S^* = \arg\min_S (T_C + T_E).
$$
The model’s key condition is that performance gains are obtained when the time to switch to the next kernel inside the graph, $T_i$, is lower than the time to switch from one launched kernel to the next launched kernel, $T_a$. In that regime, increasing $S$ reduces the amortized transition cost represented by the $a/S$ term [2501.09398].

Empirically, the optimal iteration batch size is reported as platform-dependent but workload-independent, with best batch sizes around 50–100 kernels and approximately 100 nodes recommended for the tested system. The same study reports that the optimal batch size gives more than 1.4x speed-up in the skeleton application, that the graph version starts to outperform the baseline after about two to three iteration batches, and that similar speed-up can be gained in Hotspot, Hotspot3D, and an FDTD Maxwell solver. Larger graphs increase memory usage linearly, and graph creation overhead grows linearly with batch size up to about 2,500 nodes, beyond which measured execution behavior becomes less favorable [2501.09398].

A plausible implication is that graph construction must be treated as an optimization problem rather than as a purely mechanical replacement of ordinary launches. The same source explicitly argues against naively putting every iteration into one huge graph, because graph creation cost and memory footprint eventually offset replay benefits [2501.09398].

## 4. Compiler integration and hybrid runtimes for machine learning

Machine-learning deployments expose both the benefits and the limitations of CUDA Graphs. In transformer inference, a useful partition is between graph-safe static operations and runtime-dependent dynamic operations. The hybrid LLM runtime literature classifies linear projections, layer normalization, matrix multiplications, attention score computation, fused attention kernels, and transformer layer compute as static operations with fixed shapes, deterministic control flow, and stable allocation patterns; token sampling, KV-cache updates, positional embedding extension, shape adaptation logic, and stochastic masking or randomized components remain dynamic [2604.23467].

| Execution region | Operations |
|---|---|
| Captured in CUDA Graph | matrix multiplications; fused attention kernels; normalization layers; linear projections; attention score computation; other deterministic transformer compute |
| Kept dynamic / JIT | stochastic sampling; KV-cache management; shape normalization/adaptation; preprocessing; token decoding logic; runtime-dependent control flow |

The hybrid JIT–CUDA Graph runtime splits inference into a **static domain** executed by CUDA Graph replay and a **dynamic domain** executed by JIT-compiled code. It uses two CUDA streams—$\mathsf{S_{cap}}$ for capture and $\mathsf{S_{rep}}$ for replay—and a rolling CUDA Graph buffer $\mathcal{G}$. In Algorithm 1, the system pre-captures short-sequence graphs for $\ell \in [1, 50]$ during warm-up, preprocesses the current context tensor $\mathbf{x}_i$, and then replays a matching graph $G_\ell \in \mathcal{G}$ if one exists; otherwise, it executes the static segment via JIT, asynchronously captures a new graph on $\mathsf{S_{cap}}$, inserts it into the rolling buffer, and returns the resulting hidden state $\mathbf{h}_i$ for token decoding through $f_{\text{decode}}(\cdot)$. The design also uses a two-process architecture consisting of a JIT Context Generator and a CUDA Graph Generator, with device-pointer-based communication, asynchronous graph generation threads, explicit CUDA event synchronization, and least-used eviction in the rolling cache. Practical integration points include FlashAttention v2, nvFuser LayerNorm, and paged KV-cache kernels from vLLM inside static graph regions [2604.23467].

On LLaMA-2 7B with a single NVIDIA H100 GPU, batch size 1, FP16 precision, and prompt lengths from 10 to 500 tokens, this hybrid runtime reports the lowest TTFT among HuggingFace Transformers / PyTorch eager, TensorRT–LLM, and Hybrid JIT + CUDA Graph. At 10 tokens, the reported TTFTs are 24.65 ms, 16 ms, and 13.36 ms respectively; at 500 tokens, they are 48.96 ms, 88 ms, and 17.79 ms. The reported overall TTFT speedups are 1.02× to 5.90× versus PyTorch Eager and 1.04× to 5.42× versus TensorRT–LLM, with up to 66.0% TTFT reduction. For P99 per-token latency, the hybrid runtime reports a 20.2% average reduction relative to TensorRT–LLM and a 43.5% average reduction relative to PyTorch Eager [2604.23467].

Compiler support is equally important because naïve deployment can be counterproductive. PyGraph, integrated into the PyTorch2 compilation toolchain, is motivated by three observations: PyTorch2 rejects some graphifiable cases that can be repaired; nearly 60% of CUDA Graph overhead can be attributed to parameter data copies; and blind deployment is frequently harmful. Across 416 CUDA Graphs from 183 applications, 143 hurt performance, 82 gave no meaningful speedup, and only 191 improved performance by more than 2%. PyGraph therefore combines CUDA-Graph-aware Data Placement, Kernel Parameter Indirection, and selective deployment. Data Placement rewrites CPU-resident scalars or host-resident tensors so more kernels become graph-capturable; coverage rises, for example, from 4.66% to 78.08% for Speech Transformer, from 76.51% to 99.14% for DALLE, from 1.27% to 98.58% for XLNet-T, and from 0% to 100% for XLNet-I. Kernel Parameter Indirection converts expensive parameter data copies into 8-byte pointer copies by passing pointer-to-pointer arguments in Triton-generated kernels, often reducing copied bytes by around 99%. Selective deployment profiles three choices—no graph, graph without indirection, and graph with indirection—and caches the best-performing option [2503.19779].

A recurrent misconception is therefore that more graphification is always better. The PyGraph study shows explicitly that CUDA Graphs can hurt performance in many cases, including worst-case slowdown of 36% on EOS under naïve PyTorch2-CG, while the communication and iterative-computing studies similarly show that payoff depends on graph size, repetition, and amortization [2503.19779] [2604.22228] [2501.09398].

## 5. Communication schedules and GPU-to-GPU transfer workflows

CUDA Graphs are not restricted to kernel-heavy computation; they are also used to encode repeated communication schedules. In the UCX integration work, the graph-based execution object represents a multi-path intra-node GPU-to-GPU communication workflow inside `uct_cuda`, specifically around `cuda_ipc`. The system extends standard UCT::CUDA-IPC with a multi-path communication handler, a 2-D pipelining engine, a CUDA Graph engine, and a graph cache. The 2-D pipelining engine partitions a message horizontally across paths and vertically into chunks, with per-path chunk sizes computed as
$$
chunk\_size[p] \leftarrow \left\lceil \frac{size \times share[p]}{max\_chunks} \right\rceil .
$$
Graph construction then chooses one of three transfer modes for each path: direct peer-to-peer copy for a direct GPU path, host-staged copy for PCIe-through-host transfers, or staging-GPU copy through an intermediate GPU [2604.22228].

In this setting, graph nodes are primarily memcpy nodes, and dependencies encode staging and completion ordering. `PeerToPeerCopy` creates device-to-device memcpy nodes, `StageHostCopy` creates device-to-host followed by host-to-device nodes with a dependency, and `StageGPUCopy` creates the analogous sequence using an intermediate GPU. The graph instance is cached by communication configuration using a least-recently-used eviction policy in a fixed-size hash table, tunable via environment variables. Launch then becomes a graph lookup followed by replay on the proper stream, with CUDA events used so UCX can synchronize with the input stream [2604.22228].

The performance results show that graphified multi-path schedules can substantially reduce CPU-side communication overhead. On four-GPU Beluga and Narval nodes, the reported headline improvement is up to 2.95x bandwidth over the standard single-path UCT::CUDA-IPC baseline at message sizes up to 512MB. In UCX Put bandwidth, using three GPU paths and optionally a host-staging path yields up to 2.85× improvement on Beluga and 2.75× on Narval for messages larger than 32 MB; enabling CUDA Graphs raises this to as much as 2.95× on Beluga and 2.85× on Narval. For Jacobi with four MPI ranks and 1000 iterations, runtime improves by up to 1.26× on Beluga and 1.15× on Narval with two concurrent paths, and to 1.28× and 1.16× respectively with CUDA Graphs added [2604.22228].

The same study also defines clear boundaries. Small messages below about 8 MB are a poor fit because the graph contains too few nodes and launch overhead can dominate. Adding a host-staging path contributes only marginally, typically up to about 15%, and in bidirectional bandwidth tests can hurt performance because both directions contend for the same PCIe path through the host. This suggests that, in communication systems as in ML inference, graph replay is most valuable when the schedule is repeated, fine-grained, and large enough for one-time setup and launch costs to be amortized [2604.22228].

## 6. Persistence, cold start, and execution-context materialization

CUDA Graph usage in production LLM serving introduces a distinct systems problem: cold-start latency. Recent serving systems can load model weights in seconds, but graph capture itself can take tens of seconds to minutes and often dominates startup. Foundry addresses this by treating persistence as **template-based CUDA graph context materialization** rather than as simple graph serialization. The core claim is that cold start is dominated by reconstructing execution context, not merely graph topology [2604.06664].

Foundry’s SAVE phase performs normal warmup and graph capture once while intercepting CUDA driver calls, recording serialized graph metadata, deterministic memory-layout information, and kernel binaries or module-loading information. Its LOAD phase reconstructs executable graphs in a fresh process without rerunning warmup or capture. Two context components are materialized explicitly. First, deterministic memory layout is enforced by interposing on CUDA virtual memory management APIs, reserving a large virtual address region at a fixed base address, redirecting allocations into that region, and replaying capture-window allocations during LOAD so that embedded device pointers remain valid even though the LOAD path skips graph capture. Second, kernel binaries are extracted and reloaded by intercepting `cuModuleLoad` and `cuLibraryLoad`, recording binary payloads, load APIs, options, and a catalog keyed by binary hash and mangled kernel name; LOAD then resolves kernel function handles from this catalog [2604.06664].

Foundry also exploits topology regularity through templating. Graphs are grouped by a topology key derived from node types, node order, dependency structure, and certain topology-affecting attributes. One template graph is built per unique topology, and CUDA graph-executable update APIs such as `cuGraphExecUpdate` are used to specialize non-template graphs by updating kernel function, argument buffer, and launch dimensions in place. For Qwen3-14B across batch sizes 1–512, 512 captured graphs reduce to only 22 unique topologies. Across 512 captured graphs in the broader evaluation, unique templates range from 12 to 25, so 95–98% of graphs are served by on-demand parameter update rather than full graph construction. Reported per-graph costs are about 59.7–198.6 ms for stream capture, 31.1–69.5 ms for driver-API template construction, and 0.98–2.89 ms for in-place update [2604.06664].

The system further extends single-GPU offline capture to multi-GPU distributed serving for SPMD-style DP, TP, and EP by using a stub layer over communication libraries such as NCCL and NVSHMEM during SAVE, then patching rank-dependent communication state during LOAD. Pipeline parallelism is explicitly excluded because topology is not invariant across ranks [2604.06664].

The operational impact is large. For dense models, Foundry reduces Qwen3-14B initialization from 36–48 s to 1.7–1.8 s, Llama3-8B from about 28 s to 1.3 s, and Gemma3-12B from about 45 s to 2.0 s. For MoE models in BF16, Qwen3-30B-A3B drops from 112–154 s to 2.7–2.8 s, and Qwen3-235B-A22B EP8 from 650 s to 3.9 s, which is the reported headline reduction of about 99%. Throughput is preserved: using TPOT, Foundry and native vLLM graphs overlap almost perfectly across data parallelism scales, dense and MoE models, BF16 and FP8, and different GPU generations, and generated tokens are identical between Foundry and vLLM [2604.06664].

Taken together, these results delimit the regimes in which CUDA Graphs are most effective. They are a strong fit for repeated decoding steps, mostly stable transformer compute, many short iterative kernels, and repeated fine-grained communication schedules. They are less attractive when control flow is highly dynamic, when graphs are too small for launch savings to dominate, when parameter-copy overhead is large relative to useful work, or when communication or execution patterns are sparse and irregular. The literature surveyed here therefore presents CUDA Graphs not as a universal accelerator, but as a reusable execution substrate whose value depends on graph-safe structure, amortizable repetition, and accurate management of execution context [2604.23467] [2501.09398] [2503.19779] [2604.22228] [2604.06664].

Source: https://www.emergentmind.com/topics/cuda-graphs