---
title: 'Stream-K++ Scheduling: Unified Adaptive Strategies'
url: https://www.emergentmind.com/topics/stream-k-scheduling
type: topic
---

# Stream-K++ Scheduling: Unified Adaptive Strategies

Stream-K++ Scheduling is a unified term for two distinct, state-of-the-art resource management frameworks targeting high-throughput parallel computing: a joint coded scheduling system for heterogeneous distributed clusters [2103.01921], and an adaptive kernel selection methodology for GPU-based general matrix multiplication (GEMM) [2408.11417]. Despite different architectural foci, both share the principle of adaptively partitioning and mapping computational workloads—leveraging heterogeneity-aware scheduling policies, modern coding techniques, and rapid selection/pruning mechanisms—to achieve near-optimal job latency or kernel throughput in highly variable, large-scale environments.

## 1. Distributed Coded Computing: Stream-K++ Scheduling Framework

The distributed Stream-K++ framework addresses the challenge of stream job execution across heterogeneous clusters, characterized by stochastic job arrivals and variable worker compute/communication capabilities [2103.01921]. Jobs arrive at the master node according to a Poisson process, with per-step arrival rate $\lambda$. Each job $j$ is a non-iterative function $f(X_1(j), ..., X_m(j))$, potentially encoded via an $(K\Omega,K)$ MDS code that expands into $K\Omega$ tasks, any $K$ of which suffice for decoding.

Worker nodes $p$ are associated with distinct mean job times $E[T_p]$, communication rates $c_p$, and maintain independent M/G/1 queues with stability per $\lambda \phi_p \le \mu_p = 1/E[T_p]$. The core decision variable is the job split $\phi_p$ ($\phi_p \ge 0$, $\sum_p \phi_p = 1$), denoting the proportion of work routed to each worker. The system enforces a "no‐bottleneck" constraint on per-worker communication and encoding/decoding throughput, relative to $\mu_p$.

The scheduling-coding interaction is formalized as an optimization of average in-order execution delay,
\[
E[D] = E[T_{\rm enc}] + D_{\rm comp} + D_{\rm comm} + E[T_{\rm dec}],
\]
with $D_{\rm comp}$ governed by Pollaczek–Khinchin queueing, and $D_{\rm comm}$ reflecting aggregate communication times. The joint selection of the code family $\mathcal{C}$ (e.g., PolyDot) and optimal worker subset $\mathcal{P}$, coupled with closed-form convex optimization for $\boldsymbol\phi$, constitutes the Stream-K++ approach.

## 2. GPU Kernel Autotuning: Stream-K++ for GEMM Scheduling

On modern GPUs, Stream-K++ refers to an adaptive methodology for GEMM kernel scheduling and selection, implemented in the Open-sieve C++ library and integrated with the AMD Composable Kernel framework [2408.11417]. The fundamental objective is to maximize GFLOPS for a triple $(M,N,K)$ of matrix sizes by adaptively choosing from a pool of tightly parameterized scheduling policies, each orchestrating the mix between Stream-K (persistent-kernel, atomic-enabled) and traditional data-parallel (DP) execution modes.

Stream-K++ extends the original three Stream-K schedules to seven, indexed by $r\in\{0,1,2,3,4,5,6\}$; in addition, the “all_StreamK” policy ($r\rightarrow\infty$) disables DP entirely. Each policy executes $r$ batches of Stream-K iterations before transitioning to DP for the remainder, with concrete mapping determined at the granularity of workgroups. The best policy is non-trivial to predict for each tuple $(M,N,K)$, necessitating rapid policy selection.

## 3. Policy Pruning and Selection via Bloom Filters

Stream-K++ leverages a set of Bloom filters—one per scheduling policy—to rapidly eliminate configurations unsuitable for a given GEMM problem size prior to benchmarking or cost-model evaluation. Filters are constructed offline using empirical wins across $\sim 923$ fp16 GEMM sizes, with a false-positive rate $P_{\mathrm{fp}} \lesssim 0.05$ and 100% true-negative guarantee. Runtime selection entails, for each policy $pr$:
- If $\text{BloomFilter}[pr].\text{may\_contain}(M,N,K)$ is false, skip $pr$ (safe negative).
- If true, perform empirical timing (e.g., using ckProfiler) and select $pr^*$ minimizing observed latency.

This mechanism prunes approximately $95.8\%$ of policy candidates on average, resulting in $<0.4$ μs overhead per selection and a tuning speedup approaching $20\times$.

## 4. Algorithmic Outline and Optimization

### Distributed Cluster Scheduling [2103.01921]

1. **Worker and Code Selection**
   - For each candidate code in $\mathcal{C}$, assemble a valid worker pool $\mathcal{P}$ meeting the redundancy criterion $\sum_{p\in\mathcal{P}}\mu_p \ge \lambda(1+\Theta)$.
2. **Optimal Load Split**
   - Solve the convex program:
     \[
     \min_{\{\phi_p\}}\;\;E[D],\quad
     \text{s.t. } \sum_p \phi_p = 1,\;\phi_{\min} \le \phi_p \le \beta_p
     \]
     where $\beta_p = 1/(\lambda E[T_p])$, $a_p = \lambda E[T_p^2]/(2E[T_p])$.
   - The optimal split $\phi_p^*$ is derived in closed form:
     \[
     \phi_p^* =
     \begin{cases}
       1 - \sqrt{a_p/(\eta - \xi_p)}, & \text{if } \eta \ge a_p/\beta_p^2 + \xi_p \\
       \beta_p, & \text{otherwise}
     \end{cases}
     \]
     where $\xi_p = 1/\beta_p - 2/\lambda$, $\eta$ enforces $\sum_p \phi_p = 1$.

### GPU Scheduling Policy Selection [2408.11417]

1. **Host-Side Selection**
   - For each unpruned policy $pr$, measure kernel execution time $t_{pr}$ (optional: use a cost model).
   - Select $pr^* = \arg\min t_{pr}$.
2. **Device Execution**
   - Use persistent kernel parameterized by $pr^*$ for the launch.
   - Each workgroup dynamically interleaves Stream-K and DP iterations as dictated by $r$.

## 5. Performance and Empirical Insights

### Distributed Clusters

- Simulation with $P=200$, worker compute and communication uniformly distributed, and PolyDot codes at moderate redundancy $(\Omega\approx1.2)$ shows Stream-K++ achieves mean delay within $5\%$ of a non-causal, idealized scheduler.
- Uniform splitting of load is substantially suboptimal, exhibiting 3–5$\times$ greater mean delay and instability (queue divergence) for $\Omega$ near unity.
- The optimal split exploits heterogeneity by heavily loading the fastest nodes.

### GPU Kernel Scheduling

- Benchmarked on AMD Instinct MI250X, Stream-K++ attains up to $43\%$ GFLOPS gain over the best pure-DP kernel for certain $(M,N,K)$.
- Pure DP ($pr=0$) wins in $\sim87\%$ of cases at zero tolerance; permitting $20\%$ slowdown, Stream-K policies are within that envelope for $97.6\%$ of sizes.
- For cases where Stream-K policies prevail, median gains approach $40\%$, with maximal outliers at $+43\%$.
- Bloom-filter selection ensures minimal policy search without compromising optimality for observed problem configurations.

## 6. Flexibility, Extensibility, and Future Directions

Both instantiations of Stream-K++ are architected for extension. In the GPU schedule-selector, incorporating new $(M,N,K)$ sizes, precision types, or additional tuning parameters (such as tile sizes or vector widths) requires only rebuilding the corresponding Bloom filters and header-only library; additional policies (e.g., “7_batch_then_DP”) are accommodated by training and addition of new filters. The distributed framework similarly generalizes over code families and worker selection, adapting to new hardware capabilities or evolving job characteristics.

A plausible implication is that O(1) policy pruning via filter-based lookup will remain beneficial as tuning parameter spaces expand, supporting future GPU autotuning needs and distributed computing environments with more intricate heterogeneity profiles. For both domains, the Stream-K++ approach exemplifies unified, data-driven, and low-overhead adaptivity in high-dimensional resource allocation.

---

**References:**
- "Stream Distributed Coded Computing" [2103.01921]
- "Stream-K++: Adaptive GPU GEMM Kernel Scheduling and Selection using Bloom Filters" [2408.11417]

Source: https://www.emergentmind.com/topics/stream-k-scheduling