Memory-Centric Coroutines

• Memory-centric coroutines are a class of techniques that restructure computation to overlap long-latency memory accesses using explicit yield points.
• They employ C++20 stackless coroutines, software prefetching, batch scheduling, and custom allocators to reduce memory footprint and contention.
• Empirical evaluations demonstrate significant speedups and efficient memory usage in in-memory databases, HPC workloads, and graph analytics.

Memory-centric coroutines are a class of program structuring techniques and run-time scheduling models that specifically target the bottlenecks imposed by long-latency memory accesses, especially in pointer-rich, irregular workloads. Unlike classical compute-centric coroutines, which interleave tasks awaiting I/O or direct user events, memory-centric coroutines structure computation and scheduling to maximize hardware overlap of memory requests and to minimize memory footprint and contention. Their design typically leverages C++20 stackless coroutines, software prefetching decisions, batch scheduling, and custom user-level allocators or stacks to enable both micro- and macro-parallelism beyond traditional threading or event-driven models. These methods have profound implications for parallel programming, memory-bandwidth utilization, and real-world performance in in-memory databases, graph analytics, and fine-grained HPC workloads.

1. Conceptual Foundations and Motivation

The motivation for memory-centric coroutines derives from the observation that pointer-rich, branch-heavy algorithms, such as B-trees, skip lists, or version chains in main-memory databases, are frequently bottlenecked by unpredictable, long-latency memory accesses. Contemporary out-of-order CPUs can have tens of concurrent misses in flight; however, ordinary code, which processes one pointer dereference or load at a time, leaves the majority of this potential memory-level parallelism (MLP) unrealized. Memory-centric coroutines expose this hidden MLP by:

• Treating each logical request (database lookup, transaction, etc.) as a coroutine that explicitly yields at long-latency operations, typically after issuing a software prefetch or memory load.
• Organizing schedulers that continuously interleave suspended coroutines, so that the CPU always presents a multitude of outstanding memory operations to the memory subsystem.
• Avoiding global stalls by design: when one coroutine suspends due to a cache miss, others are scheduled onto the same hardware thread, keeping functional units, out-of-order buffers, and memory interleaving saturated (Kiriansky et al., 2018, He et al., 2020).

The key distinction is a shift in focus: from merely tolerating memory stalls as unavoidable, to programmatically structuring execution—at the granularity of algorithmic operations—to overlap these stalls for aggregate throughput.

2. Stackless Coroutine Infrastructure and Memory Management

C++20 introduced stackless coroutines, wherein the compiler emits a small coroutine frame at suspension points. These frames store only the local variables and return addresses needed across suspends and can be individually allocated and deallocated, usually on the heap or using a custom allocator. This stands in contrast to stackful coroutines (fibers or green threads), which require a full, contiguous stack per coroutine and significant context-switch overhead (Williams et al., 28 Feb 2024).

Libfork demonstrates an advanced memory-centric coroutine substrate by introducing user-space, geometric segmented stacks—“stacklets”—to back these coroutine frames. Each segmented stack comprises contiguous “stacklets,” each with a metadata header (size $c=48$ B) and a data region for coroutine frames. Stacklets grow geometrically (typically doubling in size) as needed, and allocation within the current stacklet is a fast pointer increment. Metadata and FILO allocation order enforce fragmentation bounds:

• Total memory for $M$ bytes of active coroutine frames is bounded by $$M' \leq O(c) + c \cdot \lceil \log_2(M) \rceil + 4M$$, i.e., at most $3M$ additional bytes plus $O(c \log M)$ metadata.
• The memory consumption across $P$ workers with a serial max stack usage $M_1$ each is $M_P \leq (2c+3)\cdot P\cdot M_1$, nearly matching the theoretical lower bound $O(P\cdot M_1)$ (Williams et al., 28 Feb 2024).

This segmentation avoids heap thrashing, supports fast stacklet reuse, and enables efficient continuation stealing for fine-grained, dynamic parallelism.

3. Scheduling Models and Execution Semantics

Memory-centric coroutines require a scheduler that can rapidly switch between coroutines at annotated yield points, typically at memory or branch operations expected to incur high latency. The core principles in scheduler design are:

• Yield points are placed immediately after issuing software prefetch or unpredictable memory load/store operations.
• The scheduler maintains a pool or batch of active coroutines; as one suspends, another is resumed, ensuring $W$ independent memory requests are always outstanding (where $W$ targets observed hardware memory parallelism, e.g., 40–50 on modern CPUs) (Kiriansky et al., 2018, He et al., 2020).
• In Libfork, each worker thread maintains a segmented stack, a lock-free Chase–Lev work-stealing deque for suspended frames, and links continuation frames to enable continuation stealing—workers can resume suspended parents as their children complete, progressing along the DAG of dependencies (Williams et al., 28 Feb 2024).

In main-memory databases, the “coroutine-to-transaction” paradigm is adopted: each transaction is a coroutine; long-latency operations (pointer dereferences in index or version chain traversal) trigger a prefetch and immediate suspend, allowing inter-transaction batching without requiring API changes. A round-robin scheduler resumes transactions in batches, ensuring that as one awaits memory, others make progress (He et al., 2020).

4. Compiler and DSL Support

Memory-centric coroutine frameworks, such as Cimple, introduce embedded DSLs and compile-time transformations to facilitate marking and managing yield points:

• Coroutines can be written in C++ using DSL primitives, e.g., Prefetch(address), Load(expr), Yield(), indicating memory operations or branches expected to incur latency. These become explicit scheduler-managed suspension points.
• Struct-of-Arrays (SoA) and batch-oriented representations are automatically generated for static batches, enabling automatic vectorization and software pipelining across the W-wide batch.
• Batch schedulers can operate in a stage-wise manner (all coroutines issue a prefetch, then all perform dependent loads, etc.), supporting SoA/AVX2/AVX-512 vectorization, or dynamically refill as coroutines complete, maximizing throughput for variable-latency tasks (Kiriansky et al., 2018).

For heterogeneous tasks or dynamically-shaped operations, push/pull and hybrid schedulers alternate between batch execution and dynamic refill for tail events, striking a balance between static efficiency and dynamic responsiveness.

5. NUMA Awareness and Macro-Parallelism

On multi-socket and multi-core machines with non-uniform memory access (NUMA) architectures, memory-centric coroutine schedulers can exploit topology-aware victim selection to minimize cross-node memory traffic. Libfork’s scheduler, for example, builds a NUMA topology using hwloc, pins workers to cores, and selects steal victims based on hop distance and core count:

• Stealing probability is weighted as $w_{ij} = 1 / (n_{ij} \cdot r_{ij}^2)$, where $r_{ij}$ is NUMA hop distance and $n_{ij}$ the number of candidate workers at that distance.
• Two strategies exist: “Busy” spinning for maximum responsiveness, and adaptive “Lazy” per-NUMA-node idling, to reduce unnecessary interconnect usage and cut idle CPU overhead (Williams et al., 28 Feb 2024).

On memory-bound workloads, NUMA-local awareness enables near-ideal bandwidth scaling matching busy-waiting schedulers, with significant reductions in remote memory fetches and improved effective memory bandwidth.

6. Empirical Evaluation and Performance Characteristics

Research on memory-centric coroutines demonstrates compelling empirical results:

• Libfork achieves near-linear speedups out to 56 cores on benchmarks such as recursive Fibonacci, matrix multiplication, and n-queens; on fine-grained tasks, it is up to 7.5× faster than Intel TBB, 24× than OpenMP, and up to 100× faster than taskflow (Williams et al., 28 Feb 2024).
• Peak memory consumption fits the model $MRSS(P) \approx a + b \cdot M_1 \cdot P^n$ with exponent $n < 1$, indicating low fragmentation and efficient stack sharing. Competing frameworks (OpenMP, TBB) show $n \approx 1–1.3$, consuming up to 10× (OpenMP) or 6.2× (TBB) more memory.
• Cimple, in single-thread workloads, delivers up to 6.4× speedups (e.g., binary search trees), raising MLP from 1.2 to 4.3, and instructions-per-cycle (IPC) from 0.10 to 0.70, substantially increasing both throughput and hardware utilization (Kiriansky et al., 2018).
• In CoroBase, treating transactions as coroutines without altering the transaction/record API achieves 1.8–2× speedups for read-intensive YCSB and TPC-C–variant workloads. The sweet spot for batch sizes aligns with the number of available hardware MSHRs, typically 10. Moderate latency increases (~10%) are observed but remain acceptable for pipelined transactional workloads (He et al., 2020).

7. Limitations and Generalizations

The benefits of memory-centric coroutines are most pronounced on pointer-rich, memory-bound workloads. For data sizes resident in the last-level cache, or predominantly compute-bound operations, the user-level scheduling overhead can outweigh benefits (as seen in CoroBase, with up to a 10% penalty for cache-resident datasets). Write-heavy or highly skewed workloads with bottlenecks outside memory latency offer diminished gains. Nevertheless, the architectural techniques generalize to key-value stores, in-process graph analytics, network stacks, and any computation with significant pointer-chasing behavior. Shared-nothing and intra-transaction parallelism models can further exploit coroutine-based overlap by decomposing sub-transactions or sub-operations into coroutine fragments (He et al., 2020).

A plausible implication is that, as C++20 stackless coroutines and custom scheduler/allocator designs mature, memory-centric coroutine approaches will become increasingly foundational in high-performance systems aiming for both latency tolerance and memory efficiency.