GPUOS: GPU Operating System Integration

Updated 3 July 2026

GPUOS is a GPU operating system that provides OS-like primitives such as resource multiplexing, dynamic operator injection, and transparent checkpoint/restore.
It employs a persistent kernel design that dispatches micro-operations via an in-kernel work queue, reducing coordination overhead by up to 50×.
GPUOS bridges deep learning pipelines and general-purpose OS paradigms through unified virtual memory, direct system call capabilities, and enhanced fault tolerance.

A GPU Operating System (GPUOS) encompasses the foundational primitives, abstractions, and mechanisms required to enable the GPU to serve as a first-class computational citizen with OS-like features, including resource multiplexing, dynamic operator injection, generalized virtual memory, direct system-call capabilities, robust fault tolerance, and transparent checkpoint/restore. GPUOS integrates these services in the context of emerging DNN-heavy workloads—where microkernel launch overheads, heterogeneity in resource requirements, and fine-grained operation fusion can be severe bottlenecks—and aims to bridge the performance and programmability gap between the bulk-synchronous deep learning pipeline and general-purpose operating system paradigms.

1. Motivation and Historical Context

The increasing computational significance of GPUs in high-performance workloads, particularly deep learning, has exposed fundamental mismatches between traditional bulk-synchronous GPU programming models and the demands of modern tensorized micro-batch pipelines. Kernel launch overhead on contemporary platforms (3–7 μs per launch) often exceeds the compute time for small tensor operations, resulting in prohibitively high orchestration costs in regimes dominated by attention patterns or micro-batched inference (Yang et al., 20 Apr 2026). The necessity for performant, transparent resource management on the GPU motivated early explorations into OS-level integration (e.g., glinux for in-kernel services (Sun et al., 2013)) and has since evolved into robust systems that deeply virtualize GPU control paths, memory, and compute abstractions.

Concurrently, efforts to abstract unified virtual address spaces (e.g., UVM) revealed the OS/CPU bottleneck for memory oversubscription. This prompted a shift towards GPU-resident resource management, including virtual memory (GPUVM (Nazaraliyev et al., 2024)), direct system call invocation from GPU threads (GENESYS (Veselý et al., 2017)), and transparent checkpoint/restore for AI job fault tolerance and migration (PARALLELGPUOS (Huang et al., 2024)).

2. Persistent Kernel and Transparent Operation Fusion

GPUOS introduces a persistent kernel paradigm where a single, long-lived worker kernel is launched and remains active, polling a device-visible work queue for task descriptors written by the host (Yang et al., 20 Apr 2026). Rather than launching a new kernel per operation, each micro-operation is dispatched in-kernel via a device function pointer table. The host enqueues TaskDescriptors—containing operation IDs, memory pointers, dimensions, and other metadata—into a ring buffer, which is atomically consumed by streaming multiprocessor (SM) resident warps.

The fundamental cost model is:

Classic: $T_\text{total} = N \cdot (L_\text{launch} + T_\text{compute})$ .
GPUOS: $T_\text{total GPUOS} \approx N \cdot (L_\text{dispatch} + T_\text{compute})$ ,

where $L_\text{launch}\sim5\mu$ s and $L_\text{dispatch}\sim0.1\mu$ s. For $N=100$ operations, coordination cost decreases from $500\mu$ s to $10\mu$ s (up to $50\times$ reduction).

This persistent kernel design supports transparent fusion of fine-grained operations, eliminates kernel launch gaps, and enables dynamic extension of operator sets through in-kernel function pointer indirection.

3. Dynamic Operator Injection and Table Aliasing

GPUOS implements hot-loading of new device operators using NVIDIA's NVRTC JIT compiler. At runtime, the system generates device PTX code, dynamically downloads, and injects function pointers into the in-kernel operator table. The table is structured as a dual-slot aliasing scheme: two versions (slots) are maintained, and an atomic version bit determines active selection (Yang et al., 20 Apr 2026).

The update protocol ensures race-freedom:

Install new operator pointer into the inactive slot.
Atomically update the version counter.
Device-side dispatches read the version before fetching the corresponding pointer.

This mechanism allows concurrent updates to the operator set with zero kernel downtime or partial installation hazards. Dispatches always observe either the old complete table or the newly installed one, never a partially updated state.

4. Unified Virtual Memory and GPU-Driven Page Management

GPUOS as a vision includes GPU-driven virtual memory (as exemplified by GPUVM) to eliminate OS/CPU involvement in memory oversubscription scenarios. GPUVM constructs unified address spaces where device-resident page tables, eviction queues, and page fault handling logic are implemented entirely on the GPU, with one-sided RDMA-capable network cards providing direct access to backing CPU memory (Nazaraliyev et al., 2024).

Each device load/store probes in-GPU page tables and, upon page fault, a warp leader coordinates on-demand paging: allocates/ejects victim frames, issues RDMA requests via device-mapped send queues, and updates entry metadata upon completion. This results in:

Up to $4\times$ speedup over UVM on latency-bound kernels.
Near-constant overhead under oversubscription (contrasted to superlinear UVM slowdown).
Programming abstraction: $gpuvm<T>$ type that exposes managed arrays with fully unified addressability, matching the CPU-side experience.

This system sets the stage for a GPUOS that includes robust, autonomic memory management, on-demand paging, and hardware-accelerated protection.

5. OS-Level System Call Abstractions

Mechanisms such as GENESYS expose POSIX system calls directly to GPU programs. GENESYS leverages a shared-memory syscall area, GPU-to-CPU interrupts, and kernel work queues to execute generic Linux system calls—including I/O, memory management, and network socket operations—issued from arbitrary GPU execution contexts (Veselý et al., 2017).

Invocation granularity (thread, work-group, kernel), ordering (strong/relaxed), and blocking semantics are programmably selectable. This model supports efficient batching, coalescing, and overlap of syscall execution to maximize throughput and CPU-GPU concurrency. A majority (~79%) of Linux syscalls are "useful and implementable" in the GPU context; some remain impractical due to lacking thread context. GENESYS embodies an OS service interface for the accelerator, extending resource autonomy and programmability.

6. Fault Tolerance, Migration, and Checkpoint/Restore

PARALLELGPUOS enables transparent, concurrent checkpoint and restore (C/R) for GPU processes independent of application cooperation (Huang et al., 2024). By speculatively reconstructing per-kernel buffer read/write sets from launch arguments—and validating via offline instrumentation—PARALLELGPUOS dynamically builds a "kernel DAG" of dependencies. This supports the application of classic C/R methodologies (copy-on-write, dirty-bit, on-demand paging) to the GPU context, enabling:

Fault tolerance (periodic C) with up to $T_\text{total GPUOS} \approx N \cdot (L_\text{dispatch} + T_\text{compute})$ 0 faster throughput than stop-the-world baselines.
Live migration with sub-second downtime (e.g., $T_\text{total GPUOS} \approx N \cdot (L_\text{dispatch} + T_\text{compute})$ 1 for Llama-2).
Cold start acceleration for serverless AI with up to $T_\text{total GPUOS} \approx N \cdot (L_\text{dispatch} + T_\text{compute})$ 2 speedup.

Copy-on-write and dirty-bit mechanisms are coordinated to avoid write-access conflicts with in-flight checkpointing. Restore employs a prebuilt GPU context pool to eliminate allocation stalls.

7. OS Kernel Services Offload and Broader Integration

Early work (e.g., glinux (Sun et al., 2013)) demonstrated acceleration of kernel services (cryptography, packet processing, program analysis) through persistent CUDA kernels and shared memory buffers coordinated from in-kernel modules and user-space helpers. While PCIe bottlenecks and kernel launch overheads limit small-task acceleration, throughput-oriented workloads achieve $T_\text{total GPUOS} \approx N \cdot (L_\text{dispatch} + T_\text{compute})$ 3 speedup, confirming the utility of OS-integrated GPU co-processors.

Modern GPUOS concepts generalize this: persistent kernels, direct memory management, hot-pluggable compute/service routines, and seamless integration with established frameworks (such as PyTorch via TorchDispatch (Yang et al., 20 Apr 2026)) are converging into an operating-system abstraction on the GPU.

GPUOS, as articulated by recent literature, advances the GPU from a programmable coprocessor to an autonomic hardware resource with capabilities for persistent runtime execution, dynamic operator and memory management, direct OS service invocation, and robust fault tolerance. The integration of these primitives closes the semantic and performance gap between classical OS resource multiplexing and the high-throughput, fine-grained computational landscape of modern accelerators (Yang et al., 20 Apr 2026, Nazaraliyev et al., 2024, Veselý et al., 2017, Huang et al., 2024, Sun et al., 2013).