Inter-Batch Work Stealing Algorithms

• Inter-batch work stealing is a scheduling algorithm that redistributes batches of tasks between worker threads, enhancing load balancing in parallel systems.
• It leverages a lock-free design with a singly linked list and atomic operations to support bulk operations efficiently, irrespective of batch size.
• Empirical benchmarks reveal constant push/pop latencies and linear scalability, making it suitable for high-performance applications like decision diagram-based solvers.

Inter-batch work stealing is a class of scheduling and load-balancing algorithms in parallel runtime systems whereby tasks are redistributed in batches between worker threads, rather than via repeated single-element operations. A state-of-the-art instantiation is presented in "A Lock-Free Work-Stealing Algorithm for Bulk Operations" (Kataru et al., 5 Mar 2026), which targets scenarios such as mixed-integer programming optimization solvers parallelized via decision diagrams. This approach is built on lock-free principles, assumes at most one queue owner and a single concurrent stealer, and natively supports bulk operations that remain efficient independent of batch size, offering both constant-latency behavior and linear scalability for large workloads.

1. Algorithmic Structure and Data Organization

The core data structure is an unbounded singly linked list representing per-worker queues. Each queue maintains two atomic fields:

• head: Pointer to the head node (top).
• size: Integer count of accessible nodes.

Nodes are dynamically allocated from the heap, each equipped with a pointer to the next and padding to mitigate false sharing. The fundamental invariant is that the queue consists of the head-owned prefix and, if a stealing operation is in progress, a single suffix destined for transfer. There is no use of fixed arrays or ring buffers, eliminating resizing overhead and supporting unbounded growth subject only to system memory.

Bulk operations are exposed natively:

• Bulk Push (Owner Only): The owner thread can prepend a segment $L$ of $B$ nodes atomically:

$$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$

where $L.\text{end} \rightarrow \text{next} \gets h$.

• Bulk Steal (Single Stealer): The stealer retrieves a suffix roughly representing a $p$-fraction of all tasks:
  1. Atomically load $s_0 := \text{size}$.
  2. Abort if $s_0 < 2$.
  3. $k := \lfloor (1-p) s_0 \rfloor$; traverse $k-1$ next-pointers to the cut-point $C$.
  4. Abort if $B$0 on a second $B$1 read.
  5. Sever $B$2 (linearization point).
  6. Second traversal to determine batch size, then atomically decrement size and return the suffix.

This method ensures that all access patterns are either owned by the local worker or in transit as a contiguous, single-batch steal.

2. Concurrency Constraints and Synchronization Semantics

The design leverages strict concurrency restrictions:

• At most two threads access each queue: the owner (local push/pop) and a single stealer.
• Only the owner writes to $B$3; both parties atomically manipulate $B$4 using fetch-add/sub with acquire-release semantics.

This model obviates the need for locks or compare-and-swap (CAS) loops controlling $B$5, allowing simple atomic store and RMW (read-modify-write) operations. The critical linearization points for operations—guaranteeing sequential consistency—are:

Operation	Linearization Point
push(L)	Release-store of $B$6
pop()	Atomic advance of $B$7 to $B$8
steal_p()	Severing at $B$9

Acquire-release barriers on $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$0 enforce visibility and order between head and queue size for all operations.

3. Progress Guarantees and Correctness Properties

The algorithm achieves lock-freedom:

• Failed steal attempts (e.g., rapid queue drain) abort immediately, enabling progress for the owner.
• Owner operations (push and pop) are wait-free, as only a single writer exists.
• At least one thread makes progress at all times.

Safety is enforced via structural rules:

• The owner only prepends or pops single nodes from $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$1 and never modifies internal next pointers.
• The stealer severs only at $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$2, returning an exclusive, contiguous suffix—no node duplication or loss arises.

Flooding and lost-update races are avoided by invariants:

1. $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$3 always forms a valid list.
2. $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$4 accurately reflects the reachable node count post-operation.
3. Severing operations atomically partition the queue.

4. Bulk Operation Performance and Latency Trends

The microbenchmark results demonstrate the efficacy of inter-batch work stealing in relevant regimes:

• Push Latency: Native bulk push remains approximately 300–500 ns for batches ($$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$5) up to 1024, regardless of $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$6. By contrast, C++ Taskflow’s queue latency grows from 300 ns ($$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$7) to 5 μs ($$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$8).
• Pop Latency: Consistent across all implementations at ≈ 215 ns.
• Bulk Steal: Steal latency for a $$\text{push}(L):\quad (h, z) \to (L.\text{start},\, z + B)$$9-fraction of a 10 000-node queue remains flat (39–41 μs for $L.\text{end} \rightarrow \text{next} \gets h$0). Taskflow’s deque rises nearly linearly from 16 μs to 112 μs as $L.\text{end} \rightarrow \text{next} \gets h$1 grows.
• Optimized Steal Variant: If the owner does not interleave updates, $L.\text{end} \rightarrow \text{next} \gets h$2 can be inferred without a second traversal, yielding a 3x speedup at large $L.\text{end} \rightarrow \text{next} \gets h$3 (down to 12.5 μs at $L.\text{end} \rightarrow \text{next} \gets h$4).

Operations amortize $L.\text{end} \rightarrow \text{next} \gets h$5 or $L.\text{end} \rightarrow \text{next} \gets h$6 traversals into a single, fast action, avoiding per-node CAS-induced overhead (Kataru et al., 5 Mar 2026).

5. Scaling and Workload Adaptivity

The inter-batch work stealing paradigm exhibits robust scaling properties:

• In pseudo large-graph exploration (DAGs of 2.5M and 300M nodes, half-queue stolen per starvation event), execution times decrease proportionally to the number of threads—approaching ideal speedup (slope ≈ –1 on log–log plot) with thread counts up to 128.
• For workloads with highly variable node processing times (the norm in real-world parallel solvers due to factors like LP solves, cut generation, and pruning), batch-wise stealing reduces the frequency and unpredictability of contention, avoiding the high variance and overheads inherent to repeated single-node steals.

A plausible implication is that in domains characterized by workload irregularity, inter-batch work-stealing preserves throughput and fairness with markedly lower synchronization costs than single-element deques.

6. Implications and Generalizations for Parallel Frameworks

Native support for bulk operations in the lock-free queue delivers several key advantages:

• The cost of stealing a batch of $L.\text{end} \rightarrow \text{next} \gets h$7 nodes is $L.\text{end} \rightarrow \text{next} \gets h$8, not $L.\text{end} \rightarrow \text{next} \gets h$9, where $p$0 is per-node transaction cost as in single-node CAS-based algorithms.
• Unbounded linked lists avoid resizing and lock contention associated with bounded or array-based structures.
• Single-stealer semantics collapse general synchronization into single-store "handoff" (linearization) points on each operation.
• Acquire-release barriers on $p$1 fields suffice for correctness, sidestepping the need for universal memory barriers.

General lessons:

• Scheduler restrictions on concurrency greatly simplify synchronization.
• APIs exposing bulk operations better align with coarse-grained, batched parallel computation.
• Leveraging domain- or workload-specific knowledge (e.g., allowing the master to select batch sizes) avoids heuristic over-stealing/under-stealing and tailors queue mechanics to practical demands.
• For workloads with unpredictable or high peak queue sizes, linked structures with unbounded growth potential are superior to bounded dynamic arrays.

The design trades the universality of multi-thief, general-purpose deques (e.g., Chase-Lev) for the performance and predictability of a specialized, single-owner/single-stealer framework. Where scheduling policies and workloads permit, this approach enables dramatically simpler, high-performance bulk scheduling and load balancing (Kataru et al., 5 Mar 2026).