---
title: NUMA-Aware Resource Scheduling
url: https://www.emergentmind.com/topics/numa-aware-resource-scheduling
type: topic
---

# NUMA-Aware Resource Scheduling

Non-Uniform Memory Access (NUMA)-Aware Resource Scheduling refers to a class of methodologies and systems that explicitly account for the spatial heterogeneity and memory-access locality characteristics of modern many-core, multiprocessor, and disaggregated hardware during the allocation, placement, and dynamic control of computational and memory resources. NUMA-aware scheduling seeks to minimize access latencies, maximize local bandwidth, and reduce harmful remote memory traffic, cache-thrashing, and resource contention, thereby improving end-to-end application throughput, latency, and system scalability in environments where memory and compute locality are non-uniformly distributed.

## 1. NUMA Architectures and Performance Implications

NUMA architectures are characterized by sets of cores and memory banks grouped into nodes (“NUMA domains”) that communicate with lower latency and higher bandwidth within the domain than across it. Typical topologies include SMP systems with multiple sockets (each with local DRAM controllers), modern server CPUs with integrated memory controllers per socket, multi-chiplet accelerators, and disaggregated datacenter hardware where remote resources are accessible only with high penalty. Memory-access cost between domain $i$ and $j$ is typically captured as $L_{i,j}$ (latency) and $B_{i,j}$ (bandwidth), with $L_{i,i} \ll L_{i,j}$, $B_{i,i} \gg B_{i,j}$ for $i \neq j$ [1908.01860, 2411.02933, 2501.01356, 2511.19832, 2511.02132].

The negative impact of NUMA-unaware scheduling includes increased memory-access latency, lower cache-coherence efficiency, bandwidth imbalance, reduced throughput, and higher tail-latency for remote-heavy workloads [2511.19832, 2411.02933].

## 2. Fundamental Principles in NUMA-Aware Scheduling

NUMA-aware resource scheduling is guided by the following central principles:

- **Locality Maximization**: Tasks/threads/containers/virtual machines should be placed, whenever feasible, near the physical memory holding their dominant working set, to maximize the ratio of local to remote accesses (LAR) [1908.01860, 2411.01460, 2101.09284].
- **Bandwidth Balancing**: Hot data and compute regions (“slices,” “clusters”) should be distributed across NUMA nodes to prevent overloading individual memory controllers or network links [2411.02933].
- **Contention Avoidance**: Scheduling must recognize and mitigate resource contention, both for compute (core/core and socket/socket) and for bandwidth (DRAM/IMC or interconnect link) [2501.01356].
- **Adaptive Response**: Workload and interference profiles are dynamic; reactive or learning-based approaches are favored for runtime adaptation [1803.00355, 2411.01460, 2112.09509].
- **Hardware-Awareness**: The scheduler must discover or be provisioned with the actual hardware topology: the mapping of cores to sockets/nodes, interconnect graph, per-node memory/caches/bandwidth [2511.02132, 2511.19832, 1411.7131].

## 3. Algorithmic and System Approaches

A spectrum of approaches has been developed and deployed:

### 3.1. Heuristic and Rule-Based Scheduling

- **Thread and Memory Binding**: Explicitly pins threads to cores and memory pages to nodes based on detected or specified access patterns [1908.01860, 2101.09284]. Thread placement can follow dense or sparse policies (packing vs spreading across nodes).
- **User-Level and Kernel-Level Schedulers**: User-space solutions monitor per-process NUMA statistics (/proc, /sys) and conduct migration or rebinding without kernel modification; kernel-level extensions (e.g., Phoenix) integrate CPU scheduling with page allocation and direct page-table migration/replication for further optimization [2502.10923].
- **Smart Thread and Task Scheduling in Runtimes**: Extensions to OpenMP/NANOS and work-stealing runtimes employ topology-aware thread binding, per-core priority equations, and NUMA-priority work-stealing victim lists [1411.7131, 1806.11128]. These target co-location of tasks and data, and mitigate “work inflation” [1806.11128].

### 3.2. Learning- and Metric-Driven Scheduling

- **Hardware Counter–Driven Policies**: Low-level performance monitoring units (PMU) statistics are used to drive decisions, including instructions-per-cycle, stall ratios, local/remote bandwidth, and DTLB miss rates [2411.02933, 2411.01460, 1803.00355]. MAO (Baidu) applies a lightweight PMU-based monitoring and uses an XGBoost regression model (“NUMA Sensitivity Model”) for workload placement and dynamic rebinding [2411.01460].
- **Reinforcement Learning & Transformative Models**: P-MOSS frames NUMA scheduling for DBMS index “slices” as an MDP, training a Decision Transformer on PMU trajectories to jointly decide compute and memory placement [2411.02933]. SPANE recasts cloud VM allocation as an MDP over multi-NUMA PMs, enforcing $S_m$-permutation invariance via symmetry-aware dueling DQN to optimize wait time across dynamic workloads [2504.14946].

### 3.3. Cost- and Model-Based Optimization

- **Mathematical Formulation**: Scheduling problems are cast as mixed-integer programs (MIPs) or ILPs, with binary assignment variables tracking thread/VM/core/memory placements, and cost functions that aggregate local/remote access penalties, contention measures, and migration or fragmentation terms [2501.01356, 2404.15246].
- **Multi-level and Moldable Schedulers**: ARMS models per-task performance as a function of moldable resource partition size and NUMA domain, assigning tasks at runtime to local or wide domains to minimize total CPU×time cost, dynamically adapting to data locality and parallelism [2112.09509].

## 4. Implementation in Operating Systems, Runtimes, and Data Systems

NUMA-aware scheduling has been realized across several layers:

- **Linux Kernel Extensions**: Phoenix (Linux LKM) directly coordinates thread placement, allocates page-table pages on the home node, and triggers on-demand (coherent) replication of page-tables based on real-time PMU-detected page-walk cycles and memory traffic [2502.10923].
- **Runtime Libraries and Task Schedulers**: NUMA extensions to task-parallel runtimes (NUMA-WS, OpenMP/NANOS) maintain per-node “place” abstractions, NUMA-aware work-stealing protocols (biasing/restricted victim selection), and integrate task/data annotation via API [1806.11128, 1411.7131].
- **Database and Data Analytics Systems**: NUMA-aware scheduling for in-memory analytics often leverages black-box strategies—thread binding, memory interleaving, alternative allocators—and may further include spatial scheduling of data and compute, exploiting dynamic PMU data or learning models (e.g., P-MOSS) [1908.01860, 2411.02933].
- **Cloud Orchestration**: At scale, per-container or per-VM placement decisions are fed back to higher-level schedulers via monitoring, profiling, and formal sensitivity models, e.g., MAO’s integration with Matrix scheduler at Baidu, or SPANE’s symmetry-preserving policy learning in dynamic cloud traces [2411.01460, 2504.14946].

## 5. Theoretical Guarantees, Limitations, and Performance Results

Rigorous analysis and empirical evaluation confirm that NUMA-aware scheduling can guarantee:

- **Existence of Equilibrium and Bounded Sub-optimality**: Dynamic pinning schemes and NUMA-aware work-stealing admit pure-strategy Nash equilibria, which are within a tight bound ($\leq 2m/(m+1)$ of global optimal makespan for $m$ cores) [1803.00355].
- **Provable Scalability**: NUMA-WS, by adhering to the work-first principle and biasing steals, matches the O($T_1/P + O(T_{\infty})$) time bound, while mitigating work inflation that can otherwise degrade parallel efficiency by 2–5$\times$ [1806.11128].
- **Empirical Gains**:

  | System/Method         | Experimental Setting                        | Max Observed Gain        |
  |-----------------------|---------------------------------------------|--------------------------|
  | P-MOSS                | B-Tree index, Milan, YCSB-A                 | Up to $5.3\times$        |
  | MAO                   | Feed (Baidu, 12,700+ prod servers)          | $12.1$% latency, $9.8$% CPU saved |
  | Phoenix               | Real servers, page-table replication        | $2.09\times$ fewer CPU cycles, $1.58\times$ fewer page-walk cycles |
  | User-Level Scheduler  | PARSEC suite, Dell PE-R910, 40 cores        | $25\%$ speedup           |
  | ARMS                  | MatMul/Stencil, dual-socket Xeon            | up to $3.5\times$ speedup|
  | nFlows                | HEFT/Min-Min, HPC node                      | Avoided 20$\mu$s–100$\mu$s extra makespan in micro-benchmarks    |

  NUMA-aware algorithms especially dominate in resource-constrained, memory-intensive, or interference-prone scenarios; their benefit is smaller but non-negative when resources are ample and workload is compute-bound [1803.00355, 2501.01356, 1908.01860, 2511.19832]. However, page migration and overly frequent binding can cause performance regressions in some dynamic contexts—careful thresholding and hybrid heuristics are essential [2411.01460, 2502.10923].

## 6. Advanced Topics and Heterogeneous Environments

NUMA-effect complexity extends to modern GPUs (chiplet-based), scientific workflows, and distributed clouds:

- **GPU Chiplet NUMA**: AMD MI300X provides private L2 caches per chiplet (XCD); swizzled head-first mapping for attention kernels aligns compute for “heads” with NUMA domains, raising cache hit rates from 1% (naive) to 80-97%, and offering up to 50% throughput gain over conventional scheduling [2511.02132].
- **Disaggregated and Multi-Tier NUMA**: VM placement and memory mapping in multi-level, disaggregated clusters require algorithms aware of multiple latency bands across interconnects and node hierarchies; migration and isolation policies avoid cross-server remote hot-spots [2501.01356, 2511.19832, 2504.14946].
- **Workflow Scheduling**: Scientific workloads modeled as DAGs (nFlows) require extension of classic heuristics (Min-Min, HEFT) to account for NUMA-specific latency and bandwidth in their cost functions, shifting scheduling outcomes in both simulation and real-machine runs [2511.19832, 2404.15246].

## 7. Best Practices and Open Challenges

NUMA-aware resource scheduling best practices, as distilled from the surveyed literature [1908.01860, 2411.01460, 2501.01356, 2502.10923, 2511.02132, 2411.02933, 2511.19832], include:

- Pin threads/containers/VMs to cores and local memory regions, balancing for both load and bandwidth.
- Monitor PMU (hardware performance) counters continuously, using key statistics (memory bandwidth, stall ratio, locality ratio, DTLB misses) to guide adaptation.
- Integrate NUMA affinity and performance feedback into global orchestration (scheduler) frameworks.
- For short-lived or bandwidth-bound workloads, prefer binding over page-migration.
- Employ lightweight or learning-based cost models, retrained periodically for changing workload mixtures.
- Explicitly handle page-table data placement, replicate only when page-walk overhead or remote-walk fraction exceeds cost thresholds (e.g., $\alpha_t>10\%$ [2502.10923]).
- Develop placement policies and scheduling algorithms that are robust to topology size and asymmetry.

Open research directions include managing heterogeneity across nodes, integrating storage and network as first-class NUMA resources, supporting live migration and preemption, enforcing multi-tenant fairness, and robust adaptation to non-stationary or unpredictable workload shifts [2504.14946, 2511.19832].

---
**Key References:**  
[1803.00355], [2411.02933], [2411.01460], [2511.02132], [2502.10923], [2501.01356], [2511.19832], [1411.7131], [2112.09509], [2511.19832], [1908.01860], [2101.09284], [1806.11128], [2404.15246], [2504.14946]

Source: https://www.emergentmind.com/topics/numa-aware-resource-scheduling