CUDA Driver Command Streams

• CUDA driver command streams are sequences of low-level FIFO instructions in pushbuffers that convert high-level CUDA API calls into GPU operations.
• The streams integrate pushbuffers, GPFIFO rings, and doorbell writes to enable precise performance attribution, latency-hiding middleware, and efficient scheduling.
• Advanced capture techniques using kernel-level instrumentation reveal detailed command submission behavior, helping optimize DMA performance and overall concurrency.

CUDA driver command streams are the sequence of low-level, hardware-targeted instructions produced by the closed-source CUDA userspace driver to effectuate high-level CUDA API calls (e.g., kernel launches, memory copies, event operations) on NVIDIA GPUs. These command streams form the essential substrate through which work is described, queued, and scheduled on GPU engines, mediating all data movement, compute dispatch, and synchronization operations. The structure, composition, and performance behavior of these streams are critical for both precise software–hardware performance attribution and for the development of advanced, latency-hiding middleware such as PETSc and graph-based execution models (Faibussowitsch et al., 2023, Yan et al., 29 Apr 2026).

1. Architecture and Format of CUDA Driver Command Streams

At the lowest visible abstraction, a CUDA driver command stream within the userspace driver is a FIFO of "pushbuffer" entries, each a 32-bit word encoding a discrete hardware command. Pushbuffers, resident in contiguous host memory, are segmented and referenced by descriptors in the GPU-resident GPFIFO ring buffer. Submission of work is finalized by a doorbell MMIO write, signaling the GPU PBDMA front end to process queued pushbuffer segments (Yan et al., 29 Apr 2026).

Key components:

• Pushbuffer: Host-allocated buffer (4 B/entry), sequentially packed with header and payload words specifying opcodes, addresses, flags, subchannels, and parameters.
• GPFIFO: An on-GPU ring (8 B/entry) of descriptors ⟨GPU virtual address, length⟩, enabling indirect, segmented submission.
• Doorbell: 32-bit global MMIO interface, with a write committing a particular GPFIFO range to the PBDMA for execution.

Principal command types include:

Command Type	Description	Example Payload Structure
Kernel Dispatch	Launch of a compute grid	Kernel params, grid/block dims
DMA Transfer (Inline/Direct)	Host↔Device memory copy (≤24 KiB / >24 KiB)	Source/dest addresses, length
Semaphore Operation	Synchronization, timing, release/acquire	Addr + value, optional timestamp
Doorbell Write	Submission notification to GPU	Context/channel ID

The structure of pushbuffer entries closely mirrors 32-bit hardware register writes, with tightly packed bitfields for engine, operation, and addressing. This architecture enables fine-grained and efficient command encoding but can obscure performance attribution absent low-level visibility (Yan et al., 29 Apr 2026).

2. Submission Pipeline: From API Call to GPU Engine

Each CUDA API call (e.g., cuLaunchKernel, cuMemcpyAsync, cuEventRecord) maps to one or more pushbuffer commands, subsequently submitted to the GPFIFO through user–kernel interaction. The hardware submission pipeline is as follows:

1. Pushbuffer Build-up: The userspace driver serializes high-level API calls into pushbuffer entries.
2. GPFIFO Enqueue: On command batch finalization, corresponding GPFIFO descriptors are updated on the GPU.
3. Doorbell MMIO Write: Submission is finalized by a doorbell write, which notifies the GPU's PBDMA to process the new pushbuffer segment.
4. GPU Scheduling and Execution: Commands are routed by PBDMA to the appropriate engine (compute, copy, semaphore, timestamp), with the GPU scheduler interleaving execution from all non-empty streams based on dependency satisfaction and engine availability (Faibussowitsch et al., 2023, Yan et al., 29 Apr 2026).

This architectural separation enables massive concurrency across multiple streams and efficient dependency-driven scheduling.

3. Command Stream Capture and Visibility Techniques

Obtaining full-fidelity traces of closed-source CUDA driver command streams requires invasive kernel-level instrumentation. The methodology introduced by Yan et al. (Yan et al., 29 Apr 2026) leverages:

• Intercepting nv_mmap in the kernel driver to substitute a shadow doorbell page in RAM when userspace maps the real MMIO register.
• Employing x86 debug-register hardware watchpoints (DR0–DR3) on the shadow doorbell.
• On each doorbell write: reconstructing GPFIFO and pushbuffer submission state by mapping channel metadata, performing VA→PA translation, and extracting raw command words prior to GPU consumption.

This approach ensures atomic, artifact-free capture of every command package as seen by hardware. The approach enables detailed decoding and timing analysis by correlating captured commands with execution events and associated device-side timestamps.

Watchpoint Trap Handler Sketch

$M$1 As the handler completes before any subsequent pushbuffer writes, snapshot integrity is guaranteed for each submission round (Yan et al., 29 Apr 2026).

4. Stream Types, Synchronization, and PETSc Abstractions

At the CUDA driver level, streams are strictly-ordered FIFO queues. Work in a single stream is totally ordered; work across streams is only partially ordered, constrained by explicit synchronization via events or memory dependencies. Each stream is mapped to one or more hardware engines. Execution progresses whenever all inter-command dependencies are satisfied, enabling extensive overlap between computation, memory copies, and event signaling (Faibussowitsch et al., 2023).

PETSc introduces the PetscDeviceContext (PDC) abstraction to manage CUDA streams, intent annotations, and event pools, encapsulating contexts for asynchronous execution. The main properties and API features include:

• Creation and Stream-Type Selection:
  • Creation via PetscDeviceContextCreate, internally mapping to cuStreamCreate.
  • Selection between global blocking, default blocking, and global non-blocking stream types, corresponding to different CUDA stream handles and synchronization semantics.
• Fork/Join (Nested Scopes):
  • PetscDeviceContextFork and PetscDeviceContextJoin encapsulate sub-workflows on shared or inherited streams. During join, PETSc inspects recorded memory intents and injects cuStreamWaitEvent as needed for dependency enforcement.
• Command Submission:
  • Asynchronous variants of linear algebra routines (e.g., VecScaleAsync) propagate work- and memory-intent information, enqueue CUDA kernels or cuMemcpyAsync calls, and mark memory intents with pooled events for downstream synchronization as needed.
• Event-Based Synchronization and Dependencies:
  • Each memory read/write intent is associated with a pooled cudaEvent_t. Explicit event insertion (cuEventRecord) and stream waits (cuStreamWaitEvent) guarantee inter-stream data-dependence correctness without globally stalling execution (Faibussowitsch et al., 2023).

This model enables safe, seamless, and scalable integration of asynchronous GPU streams at the library level, minimizing the risk of user error or excessive API invasiveness.

5. Latency Hiding, Pipeline Throughput, and DMA Performance

Command stream-based submission is central to latency hiding and performance optimization on CUDA platforms. PETSc employs a simple pipeline throughput model to describe ideal concurrency of compute and memory operations:

$$T_{\mathrm{sequential}} = T_\mathrm{mem} + T_\mathrm{comp}$$

$$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$

For $M$ independent chunks,

$$T_\mathrm{total} = (M-1)\,\max\{T_\mathrm{mem}, T_\mathrm{comp}\} + (T_\mathrm{mem} + T_\mathrm{comp})$$

Efficient scheduling hides kernel launch and data movement latency, with overhead $\epsilon$ minimized by event pooling and streamlined stream/event destruction (Faibussowitsch et al., 2023).

DMA submission in command streams bifurcates by payload size:

• Inline DMA (compute engine, $<$24 KiB): Data is embedded directly into the pushbuffer for low-latency, small transfers (startup ≈ 24 ns, saturates near 17.5 GiB/s at 8 KiB).
• Direct DMA (copy engine, $\geq$24 KiB): Source/destination VAs and metadata only, offloaded to a dedicated copy engine. Higher startup (≈ 500 ns), bandwidth asymptote ≈ 22 GiB/s at $\geq$1 MiB (Yan et al., 29 Apr 2026).

Empirical device-side bandwidth and latency measurements confirm that, for large transfers, GPU-side data movement ($L_\mathrm{GPU}$) dominates and submission overhead ($L_\mathrm{driver} + L_\mathrm{DMA}$) is negligible. In contrast, small, frequent transfers can be bottlenecked by software submission and interconnect latency.

Transfer Size	Nsight Time (ns)	Raw Device (ns)	Driver+PBDMA Overhead (%)
8 KiB	1,924.75	448.00	76.7
32 KiB	3,780	1,900	49.9
512 KiB	22,800	22,060	3.3
2 MiB	87,890	87,110	0.9

As size increases, the overhead fraction drops sharply, affirming the model's validity at scale (Yan et al., 29 Apr 2026).

6. CUDA Graphs and Optimized Submission

CUDA Graphs decouple repeated execution of static compute DAGs from per-kernel API overhead by enabling their instantiation as command stream batches. The command-stream footprint and submission pattern directly influence launch latency and scalability.

• In CUDA 11.8, launching a graph of $$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$0 kernels exhibits linear growth in both pushbuffer size ($$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$1) and doorbell writes $$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$2, with launch overhead scaling as $$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$3, $$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$4.
• CUDA 13.0 achieves constant $$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$5 and sublinear $$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$6 via pushbuffer batching, doubling effective "submission bandwidth" (≈ 450 MiB/s) and yielding sub-5 µs launches for up to 2000 kernels.

This highlights the performance and efficiency dividends of low-level batching and submission abstraction improvements in the driver (Yan et al., 29 Apr 2026).

7. Performance Attribution, Middleware, and Hardware–Software Co-design

Precise command stream reconstruction enables the decomposition of observed performance into host driver ($$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$7), submission path ($$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$8), and engine-side ($$T_{\mathrm{streamed}} = \max\{T_{\mathrm{mem}}, T_{\mathrm{comp}}\}$$9) contributions. For middleware developers, this allows for:

• Tuning batch sizes and DMA mode thresholds based on published and empirically observed command traces.
• Identifying when performance limitations emerge from host software bottlenecks instead of true hardware saturation.
• Adapting library designs (e.g., PETSc's event pooling, fork/join) to minimize overhead $M$0 and exploit maximal concurrency.

For hardware/software co-design, recommended strategies include harmonizing pushbuffer/GPFIFO locality, exposing command-stream introspection APIs to user code, and exposing tunable heuristics for DMA and batching within the driver (Yan et al., 29 Apr 2026).

A plausible implication is that future GPU architectures could provide a more programmable and introspectable command submission interface and finer control over DMA engine and submission path, enhancing both transparency for research and efficiency for production workloads.

---

References

(Faibussowitsch et al., 2023): Safe, Seamless, And Scalable Integration Of Asynchronous GPU Streams In PETSc (Yan et al., 29 Apr 2026): Revealing NVIDIA Closed-Source Driver Command Streams for CPU-GPU Runtime Behavior Insight