Unified GPU Memory Pool Overview

Updated 8 December 2025

Unified GPU memory pools are system abstractions that present GPU and host memories as one logical address space for transparent allocation and migration.
They enhance data-intensive applications by dynamically managing memory allocations, reducing fragmentation, and optimizing performance across heterogeneous platforms.
Modern implementations combine software runtime techniques, hardware prefetching, and architectural codesign to support multi-GPU, APU, and disaggregated cloud environments.

A unified GPU memory pool is a system abstraction and underlying implementation that presents GPU and host physical (and possibly secondary/storage) memories as a single logical address space, with transparent or semi-transparent management of allocations, migrations, and overflows. Unified memory pools are critical in scaling GPU workloads beyond local physical RAM, improving programmability for data-intensive applications, and maximizing real hardware efficiency across heterogeneous platforms. Modern approaches span software runtime techniques, OS/hardware cooperative protocols, and architectural codesign, with implementations tailored to diverse environments including single- and multi-GPU systems, CPU-GPU APUs, and disaggregated cloud clusters.

1. Logical and Physical Foundations of Unified GPU Memory Pools

Unified GPU memory pools expose a single virtual address space across GPU and host DRAM (and, in some systems, NVM/SSD or remote DRAM), enabling pointers to be transparently dereferenced by both CPU and GPU contexts. The classic instance is CUDA Unified Memory (UM), which defines a single 49–64-bit address space mapping to host and device physical pages, handled via demand paging and dynamic migration on first-touch or explicit prefetch (Chien et al., 2019, Gu et al., 2020).

In NVidia's UVM and AMD's SVM, GPUs manage multi-level page tables (with 4–64 KB granularity), mirroring CPU-native PTs. A page may reside on either side, and the runtime moves data at page- or chunk-granularity based on access patterns, eviction policies such as LRU or range-based LRF, and optional programmer hints. Physical layouts, such as interleaving and banking (for bandwidth/latency optimization), are used in hardware-supported architectures (e.g., MGPU-TSM (Mojumder et al., 2020)).

Unified pools can be further extended: G10 generalizes the address space to span GPU HBM, host DRAM, and SSD, tracked using augmented page table entries with location tags and page-based DMA or direct storage access (DSA) (Zhang et al., 2023).

2. Memory Management Strategies and Pool Operations

Memory allocations in a unified GPU memory pool are mapped to contiguous or stitched regions of device and host memory. Management includes splitting, coalescing, and defragmentation mechanisms, varying by implementation.

Key management methods:

Best-Fit Cached Pools: PyTorch and TensorFlow use a "best-fit with coalescing" allocator (BFC), subdividing a reserved memory pool and merging freed blocks (Guo et al., 16 Jan 2024). This is fast but causes fragmentation under irregular allocation/deallocation.
Virtual Memory Stitching: GMLake implements virtual address stitching using CUDA VMM APIs, allowing non-contiguous physical chunks to back single logical tensors, significantly reducing both internal and external fragmentation. The stitched pool (sPool) and chunk pool (pPool) enable large tensor allocations without OOM, even under highly irregular reuse (Guo et al., 16 Jan 2024).
Heuristic and Optimal Allocation: For deep learning workloads, SmartPool uses interval graph coloring and best-fit assignment to minimize fragmentation, with extensions to automatic swapping (AutoSwap) that leverages variable lifetime and access patterns to move tensors between GPU and CPU (Zhang et al., 2019).
Block/Region List and Defragmentation: In serverless LLM systems (Tangram), each tensor or KV-cache block is tracked within a unified Region List (double-linked), and fragmentation is managed using eviction/merge heuristics and minimal-cost defragmentation, ensuring contiguous physical allocation for bandwidth optimization (Zhu et al., 1 Dec 2025).

3. Data Movement, Prefetching, and Oversubscription

Unified memory pools must manage data migration between fast device memory and larger, slower pools (host DRAM, remote DRAM, SSD). Key techniques include:

On-Demand Paging: Upon first access, a non-resident page incurs a page fault, triggering a migration—either via host OS/driver (traditional UVM/SVM) or, for improved efficiency, via GPU-driven RDMA or direct device logic (e.g., GPUVM eliminates all OS/CPU handling, moving page tables and migration logic to GPU and RNIC) (Nazaraliyev et al., 8 Nov 2024).
Hardware and Software Prefetching: cudaMemPrefetchAsync (UM) and similar routines permit bulk, nonblocking migration overlapped with computation, substantially reducing exposed stalls for PCIe platforms but less impactful in high-bandwidth (NVLink) environments (Chien et al., 2019, Gu et al., 2020).
Eviction and Thrashing Control: LRU and range-based eviction strategies dominate. Aggressive prefetch (as in AMD SVM) works well under non-oversubscription but results in excessive thrashing for irregular or dense compute, requiring careful tuning (window sizes, hybrid eviction with LFU, parallel eviction/migration threads) (Cooper et al., 10 May 2024).
Compression: Buddy Compression transparently increases effective capacity by compressing DRAM data and using a disaggregated buddy region for incompressible spillover, with fine-grained metadata and hit-tracking (Choukse et al., 2019).
Cross-Tier Scheduling: G10 leverages static compiler analysis and dynamic benefit/cost-based migration scheduling, proactively migrating tensors to host/SSD when inactive, and overlapping transfers under compute (Zhang et al., 2023).

4. Extensions to Multi-GPU, APU, and Disaggregated Architectures

Unified pools originally focused on accelerating single-GPU workloads, but modern practice extends to multi-GPU and CPU–GPU APU configurations.

True Shared Memory Across Multiple GPUs: MGPU-TSM provides one uniform HBM address space with timestamp/directory coherence and interleave-based bank allocation, removing “remote” vs “local” distinction and approaching linear M× speedup (3.9× with 4 GPUs) (Mojumder et al., 2020).
APU and Integrated Architectures: AMD MI300A features unified physical memory (UPM), with the CPU, GPU, and caches (Infinity Cache) all accessing the same address space, managed by a coherent interconnect and single set of hardware page tables (Tandon et al., 1 May 2024, Wahlgren et al., 18 Aug 2025). On such APUs, all conventional pitfalls of software-driven migration (faulting, driver copies) vanish, and performance approaches the sum of raw device and host bandwidths.
Integrated CPU-GPU Superchips: Nvidia's Grace Hopper uses a system-wide page table, hardware-level cache coherence via NVLink-C2C, and a hybrid of system and device-exclusive page tables. The system supports both first-touch and delayed migration, with page size tuning and access-aware migration thresholds to optimize bandwidth and fault cost under varying locality (Schieffer et al., 10 Jul 2024).

5. Programming Abstractions, APIs, and Integration Patterns

Unified GPU memory pools are exposed via several paradigms:

cudaMallocManaged / hipMallocManaged: Allocates pointers in the shared space, with automatic (or hint-driven) migration (Chien et al., 2019, Wahlgren et al., 18 Aug 2025).
OpenMP 5.2 Unified Shared Memory: Host and device share a global address space; any pointer, including C++ STL allocations or arrays, is valid on both; page tables and translations are hardware-synchronized (Tandon et al., 1 May 2024).
Custom allocators/hooks: GMLake substitutes for PyTorch or other runtime allocators with no code changes, permitting automatic application of fragmentation-avoiding strategies (Guo et al., 16 Jan 2024).
High-level APIs: MemServe offers alloc/free, index, match, and cross-instance transfer operations for distributed LLM context management (Hu et al., 25 Jun 2024). Tangram provides GPU heap, reuse store, and affinity-aware scheduling primitives for serverless LLM inference (Zhu et al., 1 Dec 2025).
Migration Event Hooks: G10 uses instrumented alloc/pre-evict/prefetch calls inserted by a DNN compiler pass, handled by migration queues and arbiters (Zhang et al., 2023).

6. Performance Analysis, Limitations, and Design Trade-Offs

Unified GPU memory pools enable programmability and solve the essential memory capacity bottleneck but introduce performance and management trade-offs:

Performance Under Oversubscription: Page-fault rates and migration bandwidth become the limiting factors. Empirically, UVM on PCIe incurs up to 50% slowdowns at r~1.5, whereas NVLink and APU-style UPM see graceful, sometimes negligible drops (Chien et al., 2019, Tandon et al., 1 May 2024).
Latency and Bandwidth Costs: OS-involved fault handling (NVidia/AMD UVM/SVM) typically adds hundreds of microsends per page fault; direct GPU-driven pagers (GPUVM) reduce this by 4–7× and maximize PCIe link utilization (Nazaraliyev et al., 8 Nov 2024).
Fragmentation: Classic BFC pools can suffer 20–30% fragmentation under irregular DNN fine-tuning; GMLake drops this to 5–10% with VMS (Guo et al., 16 Jan 2024).
Workload Sensitivity: Range-based prefetch (AMD SVM) leads to near-ideal performance for streaming access but severe collapse for stencil/dense linear algebra under oversubscription (throughput drops from 0.85 to near zero) (Cooper et al., 10 May 2024).
Architectural Overheads and Scalability: Hardware switch radix (for multi-GPU UMA), coherence network traffic, and cross-GPU TLB pressure scale with device count; per-GPU bandwidth can become bottlenecked in extreme configurations (Mojumder et al., 2020).

Programmer and system designer choices (e.g., when to supply memory advises, how to chunk & prefetch, what eviction heuristics to enable/tune) are fundamental. On modern APU and unified-memory-APU systems, the programming model dramatically simplifies, and code porting often reduces to allocator replacement (Tandon et al., 1 May 2024, Wahlgren et al., 18 Aug 2025). However, multi-tier pools (including SSD) or disaggregated clusters require advanced runtime orchestration and careful migration scheduling to approach raw hardware performance (Zhang et al., 2023, Hu et al., 25 Jun 2024).

7. Empirical Outcomes and Best Practices

A unified memory pool, appropriately optimized, can deliver both practical capacity and near-raw bandwidth, extending the reach of both training and inference workloads:

In DNN training: UTP+recomputation reduces AlexNet's peak memory by 59.5%, enables training of ResNet-1920 on 12 GB GPUs (vs. 592 layers with standard allocators), and reduces iteration communication volume by an order of magnitude (Wang et al., 2018).
In LLM serving: Systems like Tangram and MemServe cut model load and TTFT by factors up to 6× and halve tail latency through block-wise memory pooling and prompt-tree-aware reuse (Zhu et al., 1 Dec 2025, Hu et al., 25 Jun 2024).
On MI300A APU: Unified memory mode delivers 4–5× speedup over discrete GPU/host systems with similarly sized problems (e.g., OpenFOAM motorbike mesh), with up to 44% lower memory footprint (Wahlgren et al., 18 Aug 2025, Tandon et al., 1 May 2024).

Recommended best practices include:

Employ software/hardware prefetch and explicit locality hints for predictable workloads.
Integrate memory-pool-aware allocators to reduce fragmentation and overhead.
For serverless/multi-tenant scenarios, use fine-grained block/shared pool abstractions with global scheduling for maximal cache reuse.
Analyze and tune eviction and prefetch strategies in line with access pattern regularity and workload oversubscription.

A unified GPU memory pool constitutes the key enabling abstraction for high-utilization, programmer-friendly, and scalable GPU computing across disciplines from science to LLMs, in both vertically integrated supercomputers and cloud–edge pools.

References:

(Chien et al., 2019, Mojumder et al., 2020, Choukse et al., 2019, Nazaraliyev et al., 8 Nov 2024, Guo et al., 16 Jan 2024, Zhu et al., 1 Dec 2025, Wahlgren et al., 18 Aug 2025, Tandon et al., 1 May 2024, Schieffer et al., 10 Jul 2024, Zhang et al., 2023, Zhang et al., 2019, Wang et al., 2018, Hu et al., 25 Jun 2024, Gu et al., 2020, Cooper et al., 10 May 2024)