- The paper demonstrates an eBPF-based framework that dynamically controls page migration admission in tiered memory systems.
- It employs contention-aware mTHP splitting and architecture-aware filtering to optimize throughput and reduce migration failures.
- Evaluations show geomean throughput gains up to 17.7% and migration failure reductions up to 777× across diverse, memory-intensive benchmarks.
TierBPF: Page Migration Admission Control for Tiered Memory via eBPF
Problem Statement and Motivation
Tiered memory architectures, incorporating fast but capacity-limited DRAM with slower, higher-capacity devices (e.g., CXL memory or Intel Optane PMEM), have become mainstream due to the mismatch between CPU core scaling and DRAM provisioning. Existing tiered memory management systems—such as AutoNUMA, TPP, Colloid, and MEMTIS—primarily base page migration policies on access frequency or recency. This abstraction ignores two critical and orthogonal dimensions: page migration granularity (especially in the presence of Transparent Huge Pages, THPs, and Linux 6.8’s multi-size THP, mTHP) and heterogeneous characteristics of underlying memory topologies. As a result, these systems either waste valuable migration bandwidth and fast-tier capacity or incur high TLB miss overheads, and fail to optimize migration policies for deployment-specific hardware properties.
System Design Overview
TierBPF introduces an extensible, eBPF-based policy layer for page migration admission control. Its architectural placement is between the page migration decision logic of existing tiered memory systems and the actual migration mechanism, exerting fine-grained admission control on a per-migration basis. Three custom eBPF hooks are injected into the kernel’s NUMA migration path: one to select the mTHP split size, one to classify subpages as hot or cold, and one for migration admission (potentially incorporating runtime traffic properties).
The key architectural insights are:
- Contention-aware THP splitting: Migration granularity (mTHP size) is determined dynamically, based on access heat density within huge pages and current memory subsystem contention.
- Architecture-aware migration filtering: Migration decisions are customized for the deployed hardware’s duplex mode (e.g., full-duplex CXL links versus half-duplex DDR/PMEM), selectively favoring hot page migration only when it matches the dominant traffic direction.
- Profiling scalability and efficiency: Lightweight global subpage histograms and dual blocked counting Bloom filters (bCBF), implemented in eBPF handlers directly on hardware event samples, alleviate user-space context switch and aggregation overhead and maintain scalability regardless of working set size.
Technical Contributions
- Migration-time page size and architecture awareness: TierBPF’s filtering mechanisms account for both the intra-THP access skew (allowing for migration at intermediate mTHP sizes, e.g., 64–512 KB), and underlying hardware heterogeneity (CXL, Optane, etc.), yielding more robust and workload/device-agnostic policies.
- Lightweight, in-kernel profiling: By aggregating hardware sample events in-kernel via eBPF, TierBPF avoids the high per-sample copy, context switch, and cache pollution overheads of traditional user-space profilers. The fixed-size global histogram (4 KB/application) and on-demand dual bCBF enable effective yet low-overhead hot subregion extraction and read-write classification.
- Clean separation of mechanism and policy: The hook design and policy implementation as eBPF programs cleanly decouple mechanisms (THP splitting, migration admission) from operator- or deployment-specific policy logic, facilitating rapid adaptation to emerging hardware or workload profiles.
Evaluation and Results
TierBPF was integrated with three OS-level tiered memory subsystems (AutoNUMA, TPP, Colloid) and evaluated on both CXL and PMEM platforms across 17 memory-intensive benchmarks, including GAPBS graph workloads, NAS Parallel Benchmarks, and Silo-YCSB.
Strong claims and findings include:
- Geomean throughput gains ranging from 7.4% (Colloid) to 17.7% (TPP) under high memory contention, with individual workloads (e.g., certain graph analytics on power-law graphs) exhibiting up to 75% improvement.
- Migration failure reduction: Page migration failures (due to contiguous region allocation) drop up to 777× by reducing large THP migration requests to smaller, easier-to-allocate mTHP splits; successful migrations scale by 1–3.1×.
- Comparison to MEMTIS: On PMEM, TierBPF outperforms MEMTIS by up to 26%, especially in cases where MEMTIS’s 2 MB/4 KB all-or-nothing split induces excessive TLB misses that TierBPF’s mTHP-based selection avoids.
- Adaptivity: No single optimal mTHP size exists; TierBPF’s dynamic policy chooses granularities based on current contention and workload phase, extracting hot subregions as small as 64 KB when the access pattern is highly skewed.
Notably, traffic-balancing migration policies leveraging full-duplex CXL only produce measurable benefit under moderate contention; under severe contention, maximizing hot page placement dominates system throughput, and admission deferrals are detrimental.
Theoretical and Practical Implications
TierBPF demonstrates that page migration decision logic in tiered systems must embrace both page-granularity (exploiting mTHP for hot region migration precision without TLB penalty) and device/topology-aware adaptation. The notion of encoding migration policy adaptable to runtime bandwidth pressure, workload variability, and non-uniform memory traffic within eBPF logic, rather than static heuristics or user-space daemons, facilitates robust deployment on diverse or evolving hardware. The extensibility provided by kernel eBPF hooks paves the way for further dynamic, context-sensitive tiered memory management, such as multi-tier (beyond two-level) configurations or adaptation to ultra-fast CXL-attached SSDs.
Potential for Future Work
TierBPF’s framework explicitly supports rapid policy innovation. Future directions include policy composition (multi-objective admission), integration with application hints for data semantics, augmentation for three or more memory tiers, or cross-node CXL-pooling deployments. As hardware architectures diversify, the eBPF-based, plug-and-play policy mechanism can underpin research and deployment of new memory migration strategies without wholesale kernel modification.
Conclusion
TierBPF provides a robust, performance-portable framework for overcoming key limitations of existing tiered memory migration systems, namely static migration granularity and hardware-agnostic policies. By leveraging safe, in-kernel eBPF hooks and lightweight access profiling, TierBPF achieves strong throughput gains across diverse workloads and devices, bridging the gap between TLB efficiency and migration selectivity. Its architecture and experimental validation establish a new baseline for deployable, runtime-adaptive migration policies in heterogeneous, tiered-memory systems.
Reference:
"TierBPF: Page Migration Admission Control for Tiered Memory via eBPF" (2604.12300)