Elastic Memory Pooling

• Elastic memory pooling is a dynamic technique that creates a unified memory pool across devices and workloads to improve efficiency and performance.
• It employs virtual address decoupling, elastic page tables, and adaptive scheduling to enable flexible data movement and resource sharing.
• Applications in LLM inference, cloud consolidation, and data-parallel scheduling demonstrate significant throughput gains and reduced latency.

Elastic memory pooling is a class of system-level and application-level techniques that enable the flexible, dynamic provisioning of memory resources across compute instances, devices, or tasks. Rather than statically partitioning DRAM, HBM, or other memory hierarchies, elastic memory pooling creates a unified memory substrate that can be reallocated or shared across workloads, instances, or physical boundaries with minimal disruption and overhead. These mechanisms are central to efficient multi-tenant datacenter operation, high-throughput LLM inference, disaggregated cloud architectures, and dynamic scheduling in data-parallel clusters.

1. Fundamental Principles and Models

Elastic memory pooling abstracts a set of memory resources—across hosts, accelerators, or devices—as a dynamic pool from which allocation and reclamation can occur in response to application or system demands. Typical implementations virtualize the address space of compute nodes or applications and provide page- or block-level mechanisms for moving data, adjusting mappings, or offloading cold regions.

Foundational models include:

• Virtual Address Decoupling: Memory is virtualized such that virtual address space is decoupled from the physical device or node holding the data, allowing dynamic re-mapping and on-demand page migration (Xu et al., 18 Jun 2025, Xu et al., 2024).
• Elastic Page Tables/Block Tables: Page or block tables are extended with remote or tier-indirection entries to support seamless page movement across contexts and devices (Ababneh et al., 2018, Xu et al., 18 Jun 2025, Singh et al., 2 Oct 2025).
• Unified Pool Management: Global or distributed management layers expose APIs for allocation, offload, and migration, supporting fine-grained elasticity without application rewrites (Li et al., 2022, Shen et al., 2023, Hu et al., 2024).

Mathematically, models often balance performance penalties due to remote or slow-tier access against the benefit of improved queueing or higher system utilization. For example, for a data-parallel reducer, expected time with limited memory is $T(M) = T_{id} + \frac{B(M)}{R}$, where $B(M)$ is the amount spilled and $R$ is disk bandwidth (Iorgulescu et al., 2017).

2. System Architectures and Pooling Substrates

Several classes of elastic memory pooling systems have emerged:

• OS-Level Pooling: Modified kernels (e.g., ElasticOS) stretch process address spaces and execution across physical nodes with primitives such as stretch (address space extension), push/pull (page transfer), and jump (execution migration) (Ababneh et al., 2018).
• Device and Interconnect Pooling: Pond and Octopus layer pooling over persistent-memory fabrics (CXL) via external memory controllers (EMCs), using small pools of 8–16 nodes, replication, and block-based allocation for DRAM efficiency and low-latency access (Li et al., 2022, Berger et al., 15 Jan 2025).
• Compute-Accelerator Pooling: GPU-CPU or HBM-DRAM pooling, as applied to LLM inference, expose memory on CPUs or disaggregated DRAM as overflow/extension for GPU KV-caches via tightly coupled transfer controllers and adaptive policies (Xu et al., 2024, Xu et al., 18 Jun 2025, Singh et al., 2 Oct 2025).
• Disaggregated Memory Systems: Systems like Ditto extend key-value stores and in-memory caches to utilize elastic memory pools across networked DRAM via RDMA, utilizing distributed client scheduling, adaptive caching, and regret-minimization across policies (Shen et al., 2023).

Key architectural elements:

Mechanism	Primary Context	Example Systems
Virtual memory abstraction	Kernel/process memory	ElasticOS (Ababneh et al., 2018), eLLM (Xu et al., 18 Jun 2025)
CXL memory pooling	Cloud DRAM sharing	Pond (Li et al., 2022), Octopus (Berger et al., 15 Jan 2025)
GPU-CPU/HBM-DRAM pooling	LLM inference, MoE serving	Pie (Xu et al., 2024), eLLM (Xu et al., 18 Jun 2025), ElasticMoE (Singh et al., 2 Oct 2025)
Disaggregated remote memory	Cache, in-memory KV stores	Ditto (Shen et al., 2023)
Cross-GPU prefix/KV cache pooling	LLM context management	MemServe/MemPool (Hu et al., 2024), TokenLake (Wu et al., 24 Aug 2025)

3. Algorithms and APIs for Elastic Pooling

Elastic memory pooling systems define abstractions and algorithms to orchestrate data movement, allocation, and scheduling:

• API Surfaces: Typical primitives include memory block alloc/free, index/insert/delete (for KV or tensor blocks), swap_out/swap_in (tier migration), transfer (remote copy with optional insertion), and declarative planning (compute with data placement) (Hu et al., 2024, Wu et al., 24 Aug 2025, Singh et al., 2 Oct 2025).
• Locality-Aware Placement: Many systems use hierarchical or tree-based indices (e.g., token-based radix trees, global prompt trees) to maximize locality and reuse, with match/longest-common-prefix logic for cache lookup (Hu et al., 2024).
• Heavy-Hitter Replication and Load Balancing: Segment-level deduplication and O(N log N) selective replication are used in distributed pools to balance bandwidth and hit rates with bounded overhead (Wu et al., 24 Aug 2025).
• Adaptive Scheduling and Cost Models: Decisions about moving data or computation (e.g., push/pull vs. jump) are made using empirical or analytical models balancing memory, compute, and network cost, with thresholds adapted online (Ababneh et al., 2018, Hu et al., 2024, Xu et al., 18 Jun 2025).
• Elastic Ballooning and Virtualization: The virtual tensor abstraction and page-table-based ballooning (runtime inflation/deflation) allow dynamic rebalance between activation/KV usage or GPU/CPU tiers, supporting SLO-compliant scheduling (Xu et al., 18 Jun 2025).

4. Applications: LLM Inference, Cloud, and Data-Parallel Workloads

Elastic memory pooling addresses memory bottlenecks, cost, and resource utilization in diverse workloads:

• LLM Serving: Pools CPU DRAM and GPU HBM transparently to expand effective KV-cache and batch size, supporting both inter-request (context caching) and intra-request (prefill/decode disaggregation) reuse (Hu et al., 2024, Xu et al., 2024, Xu et al., 18 Jun 2025).
• Data-Parallel Cluster Scheduling: Empirical models of “memory elasticity” (quantified slowdown under undersized allocations) are integrated into cluster schedulers to trade-off queueing time vs. per-task runtime, yielding up to 60% lower job completion times (Iorgulescu et al., 2017).
• Cloud Memory Consolidation: Pond and Octopus enable DRAM savings of 7–22% at cluster scale with <5% SLO penalty, using ML models to predict VM latency-insensitivity and frigid memory, and BIBD-based topology designs for efficient pool connectivity (Li et al., 2022, Berger et al., 15 Jan 2025).
• In-Memory and Disaggregated Caching: Systems like Ditto achieve instant cache resizing and up to 9× throughput over VM-based caching, using multi-armed-bandit adaptive replacement and CPU-bypass one-sided RDMA (Shen et al., 2023).
• MoE LLMs: ElasticMoE executes zero-downtime scaling, redistributing expert weights across devices via page-table remapping with only pointer flips, sub-10s scale-up, and up to 2× throughput better than cold vertical scaling (Singh et al., 2 Oct 2025).

5. Performance, Scalability, and Empirical Results

Empirical evaluation demonstrates that elastic memory pooling yields significant improvements in throughput, latency, and system utilization:

• LLM Inference Pools: Pie attains 1.9× higher throughput and 2× lower per-token latency over baseline vLLM, delivering equivalent performance with up to 1.67× less GPU memory required; eLLM achieves 2.32× decoding throughput, 1.82× total throughput, and 3× larger batch sizes for 128K-token context (Xu et al., 2024, Xu et al., 18 Jun 2025).
• Pooling in Cloud Platforms: Pond saves 7–10% DRAM fleet-wide, maintaining SLOs for ≥98% of VMs; Octopus reduces TCO by 17% and DRAM by up to 22% compared to monolithic pools (Li et al., 2022, Berger et al., 15 Jan 2025).
• Cache Pooling for LLM Serving: TokenLake delivers up to 2.6× higher goodput, 4.6× higher throughput at equal latency, and 2× or better cache hit rates compared to both router-based and cache-centric baselines (Wu et al., 24 Aug 2025).
• Disaggregated and Kernel-Level Approaches: ElasticOS provides up to 10× speedup over network swapping, with 2–5× reduction in network traffic for large applications (Ababneh et al., 2018).
• Data-Parallel Scheduling: YARN-ME increases memory utilization from ~77% to 95%, reduces average job runtime by 60%, and achieves 39–48% improvement in job completion times under mixed workloads (Iorgulescu et al., 2017).

6. Limitations, Open Challenges, and Research Directions

Current elastic memory pooling systems encounter several limitations and open areas for research:

• Network and Interconnect Constraints: Most schemes rely on high-bandwidth (NVLink, CXL, IB) interconnect; extending pooling to slower or hierarchically tiered fabrics introduces new orchestration and scheduling complexities (Xu et al., 2024, Berger et al., 15 Jan 2025).
• Security and Isolation: Most deployments lack authenticated or encrypted transfers for pooled memory; trust boundaries and policy isolation are insufficiently addressed (Ababneh et al., 2018, Shen et al., 2023).
• Heterogeneity and Scalability: Handling pools with heterogeneous node latency/bandwidth or extremely large scale remains challenging, requiring weighted placement and adaptive policies (Li et al., 2022, Berger et al., 15 Jan 2025).
• Fragmentation and Defragmentation: Despite advances in virtual tensor abstractions, cross-type fragmentation and optimal scheduling remain open (Xu et al., 18 Jun 2025, Wu et al., 24 Aug 2025).
• Proactive/Adaptive Scheduling: Most controllers use reactive, threshold-based adaptation; learning-based or burst-aware scheduling offers further potential to optimize trade-offs and further cut SLO violations (Hu et al., 2024, Singh et al., 2 Oct 2025).
• Extending Beyond Memory: Several efforts propose extending pooling and disaggregation to I/O, compute (CPU, GPU), and even in-network compute, enabling joint resource pooling (Ababneh et al., 2018).

Elastic memory pooling constitutes a critical enabling substrate for high-efficiency, high-utilization, low-latency computing in modern large-scale, heterogeneous, and dynamic environments. Ongoing research continues to improve its efficiency, security, and transparency across system stacks and workloads.