Bubble Scheduling in Parallel Systems

• Bubble scheduling is a set of strategies that identify and fill idle intervals ('bubbles') in distributed computations, reducing synchronization delays and enhancing resource utilization.
• It employs methods like bubble filling, auxiliary task harvesting, and affinity-driven allocation to optimize execution and achieve significant throughput improvements.
• Advanced techniques such as Optimus and Zero-Bubble scheduling demonstrate up to 85% bubble reduction and notable speedups in large-scale, parallel deep learning pipelines.

Bubble scheduling refers to a family of scheduling strategies and abstractions that explicitly recognize and exploit "bubbles"—intervals of idleness or under-utilization—in parallel and distributed computation pipelines. In modern large-scale compute systems, bubbles frequently arise due to synchronization, communication, memory, or data dependencies. Bubble scheduling methodologies identify these idle periods and implement software, algorithmic, or system-level interventions to either fill, minimize, or systematically assign auxiliary work to these intervals, with the aim of improving hardware utilization, reducing end-to-end time-to-solution, and increasing process throughput. This article provides a comprehensive synthesis of the theory, mechanisms, and empirical achievements of bubble scheduling across distributed deep learning, numerical parallelism, and system software.

1. Definition and Origins of Bubbles

A "bubble" is technically defined as a temporal interval during which a computational resource—classically a GPU or CPU core—is allocated to a workflow but sits idle, due to pipeline imbalances, communication latency, synchronization, or resource contention. These bubbles are prevalent in pipelined distributed training of deep neural networks, particularly in configurations involving multi-dimensional parallelism (e.g., pipeline, tensor, data parallelism) (Feng et al., 2024). In formal notation, the bubble size for device $i$, $\mathrm{Bubble}_i$, is

$$\mathrm{Bubble}_i = E_i - \sum_{\text{ops}} T_{\text{op}}$$

where $E_i$ is the wall-clock time the device is allocated, and $\sum_{\text{ops}} T_{\text{op}}$ is the sum of useful operation runtimes. The bubble rate—fraction of idle device time—is then $\frac{\max_i \mathrm{Bubble}_i}{\max_i E_i}$ (Qi et al., 2023).

Bubbles also appear at the operating system/software runtime level, such as in hierarchical OpenMP threading, where thread affinity and cache/memory locality issues cause CPUs to idle while waiting for remote memory accesses or synchronization (0706.2073).

2. Bubble Scheduling Principles

Bubble scheduling aims to methodically fill or eliminate bubbles, rather than simply minimizing them as a side effect. Core principles include:

• Bubble-Filling: Identifying bubble intervals and actively scheduling auxiliary or reorganized primary computation to occupy these periods, thus reducing aggregate makespan (Feng et al., 2024, Zhang et al., 2024).
• Bubble-Free Scheduling: Defining pipelined or distributed execution plans whose resource allocation leaves no significant intervals of idleness; i.e., achieving $\mathrm{Bubble}_i \approx 0$ for all devices (Wang et al., 15 Aug 2025, Qi et al., 2023).
• Auxiliary Task Harvesting: Opportunistically assigning "side tasks" (data preparation, model training, analytics, etc.) to bubbles, while respecting resource and interference constraints (Zhang et al., 2024).
• Affinity-Driven Allocation: Informing bubble-aware placement and scheduling with application-level knowledge of thread/data affinities, particularly in hierarchical multiprocessor systems (0706.2073).

These schemes leverage formal models of pipeline structure, data dependency graphs, and kernel execution characteristics. Bubble scheduling thus often requires both system-level instrumentation (to detect and characterize bubbles) and optimization-based scheduling logic (to assign work).

3. Methods and Algorithms in Modern Bubble Scheduling

Optimus: Fine-Grained Kernel Packing in Multimodal LLM Training

"Optimus" introduces a principled bubble scheduling algorithm designed for multimodal LLM training under 3D parallelism (data, pipeline, tensor). Optimus decomposes auxiliary encoder computation into kernels, times each offline, and packs them into LLM bubbles—fine-grained idle slots ranging from $\sim300\,\mu\text{s}$ (TP) to several milliseconds (PP/DP) (Feng et al., 2024).

Algorithmically, Optimus proceeds in two phases:

• Model Planning: Independently determines optimal parallelism plans for encoder and LLM.
• Bubble Scheduling: For each microbatch, iteratively decomposes encoder work and greedily assigns kernels to bubble intervals, subject to data dependency constraints (ensuring, e.g., scheduled encoder completion before LLM consumption points).

Key abstracted pseudocode elements:

• findCritical: Identifies encoder pipeline sitting on the schedule's critical path.
• ScheduleKernels: Fits kernels into scheduled bubbles without violating duration constraints.
• CheckEncLLMDep: Verifies that forward/backward dependency constraints are met.

This fine-grained, kernel-level approach is required because coarse-grain assignment (entire layers) would vastly under-fill the briefest bubbles.

Zero-Bubble Schedules and Backward Splitting

"Zero Bubble Pipeline Parallelism" (ZB-PP) realizes bubble-free schedules in traditional synchronous deep learning pipelines. ZB-PP’s core innovation is to split the backward pass into “input-gradient” (B) and “parameter-gradient” (W) operations, allowing W to be delayed and filling pipeline gaps without additional memory (Qi et al., 2023). This enables the construction of parallelogram-shaped execution diagrams—eliminating the long-presumed necessary pipeline bubbles of canonical 1F1B schedules.

Both heuristic and ILP-based search algorithms assign forward, backward-input, and backward-weight ops to pipeline stages, explicitly minimizing bubble width under memory constraints.

Bubble Harvesting for Side-Task Parallelism

"FreeRide" formalizes bubble detection and harvest, enabling generic side tasks (e.g., other training jobs, analytics, image processing) to be scheduled into pipeline bubbles. The system defines each bubble formally as a [$t_\text{start}, t_\text{end}$] interval of low occupancy; implements memory/time resource controls; and exposes a state-machine API for pausable side tasks (Zhang et al., 2024).

FreeRide’s scheduler and resource controls maintain negligible training cost overhead ($\approx1\%$), corresponding to an average GPU cost savings of $\mathrm{Bubble}_i$0 by co-locating side work.

Bubble Scheduling in Hierarchical Multiprocessors: Thread Affinity

"BubbleSched" (Marcel threading library) and "bubble_spread" algorithms capture explicit application-level grouping/affinity as "bubbles"—hierarchical containers of threads—and recursively map them to processor hierarchies using a greedy load-balancing strategy. The algorithm honors NUMA and cache-locality, assigning large bubbles to NUMA nodes and smaller sub-bubbles to cores or processors, thus reducing cache misses and remote memory accesses (0706.2073).

4. Mathematical Formulations and Scheduling Constraints

The formulation of bubble scheduling depends on the context:

• Pipeline Parallelism: Scheduling is posed as a makespan minimization with constraints:
  • Data dependencies (e.g., $\mathrm{Bubble}_i$1, $\mathrm{Bubble}_i$2), ensuring correctness of upstream/downstream operation order.
  • Memory constraints (total active model/activation size $\mathrm{Bubble}_i$3 per-device memory).
  • Bubble assignment constraints ($\mathrm{Bubble}_i$4 for kernels assigned to bubble $\mathrm{Bubble}_i$5) (Feng et al., 2024, Li et al., 6 Oct 2025).
• MILP/ILP Schedulers: For generalized pipeline bubble minimization, scheduling variables encode op precedence, offloading/reuse of activations, and device assignment. The MILP formally captures trade-offs between parallel overlap, memory footprint, and bubble duration, with constraints on activation reload/offload and topology-aware bandwidth limits (Li et al., 6 Oct 2025).

For classic hierarchical thread allocation:

• The cost function combines processor load, affinity weights (e.g., NUMA distance penalties), and bubble cohesion costs. The assignment minimizes this sum while exploding/breaking large bubbles only as needed for balance (0706.2073).

5. Empirical Impacts and Quantitative Results

Bubble scheduling techniques have achieved substantial empirical improvements in a variety of domains:

System	Domain	Bubble Reduction	Throughput/Speedup	Memory Overhead
Optimus	MLLM training (ViT+GPT)	Up to 85% (fine)	20.5–21.3% at 3072 GPUs	≤12%
ZB-PP	GPT-3 LLMs (8–32 GPUs)	To <1% (ZB-2, ZB-V)	23–31% (memory-relaxed)	Up to 2×
FreeRide	LLM + side-tasks	Utilization >80%	Cost saving up to 11.8%,	Negligible
OptPipe	LLM pipeline/offload	Bubble ↓ 50%	Time ↓ 36–39%	Enables larger
BubbleSched	OpenMP/NUMA	Not explicit	Speedup +25–30% over plain	—

Performance validation is typically reported in terms of overall speedup, bubble rate reduction ($\mathrm{Bubble}_i$6), memory utilization (fraction of device memory actively used), and slowdown/cost metrics for auxiliary workloads (Feng et al., 2024, Qi et al., 2023, Zhang et al., 2024, Li et al., 6 Oct 2025, 0706.2073).

6. Generalization, Applications, and Current Trends

Bubble scheduling has broad applicability far beyond its origins in LLM pipeline training:

• Any system exhibiting periodic or irregular resource idleness—including data-processing pipelines, agentic RL rollouts, hierarchical runtime threading, and even asynchronous optimization or inference workloads—can leverage bubble scheduling abstractions and algorithms (Feng et al., 2024, Wang et al., 15 Aug 2025).
• Tag-based resource multiplexing, as in SeamlessFlow, achieves provably bubble-free scheduling in heterogeneously tagged clusters (e.g., mixed rollout/train pools) for multi-agent RL (Wang et al., 15 Aug 2025).
• Activation offloading and reuse—explicitly scheduled with bubble-aware constraints—enables larger models to be trained within fixed memory budgets by packing forward work into otherwise-idle time slots (Li et al., 6 Oct 2025).
• OpenMP and NUMA Multiprocessor Architectures—scheduling bubbles promotes cache and memory locality, improves synchronization efficiency, and yields sizable speedup in irregular scientific workloads (0706.2073).

A plausible implication is that bubble scheduling will continue to grow in relevance as hardware parallelism deepens, orchestration frameworks become more programmable, and auxiliary workloads (e.g., multi-modal encoders, second-order optimizers) proliferate in distributed machine learning.

7. Practical Considerations and Future Directions

• Overhead and Memory Tradeoff: Systems such as Optimus and ZB-PP demonstrate that fine-grained kernel scheduling and optimal schedule computation can be performed as a one-time or infrequent offline process with minimal runtime impact, but memory colocation of auxiliary and primary workloads must be managed carefully (Feng et al., 2024, Qi et al., 2023).
• Scheduler Complexity: Efficient heuristics, hybrid optimization (heuristics plus MILP/ILP), and caching of prior schedules are important for scaling to large P×M regimes (Li et al., 6 Oct 2025).
• Domain and System Extensibility: Bubble scheduling frameworks require clear APIs or abstractions to express dependency relationships, auxiliary workload decomposition, and resource constraints. Programming model advances (in DeepSpeed, Marcel/BubbleSched, or dedicated side-task APIs) are enabling integration with contemporary ML and systems pipelines (0706.2073, Zhang et al., 2024).
• Automated Affinity Detection: Automation of affinity and data-locality-driven bubble structuring in hierarchical scheduling remains an open topic in the system software literature (0706.2073).

In summary, bubble scheduling encapsulates an evolving research direction at the intersection of systems, algorithms, and workload modeling, targeting explicit, programmatic exploitation or suppression of resource idleness in complex parallel systems. It covers a spectrum from kernel-level packing in multi-modal distributed training to hierarchical thread grouping for HPC, and is now a key driver of practical efficiency at scale (Feng et al., 2024, Qi et al., 2023, Zhang et al., 2024, Li et al., 6 Oct 2025, Wang et al., 15 Aug 2025, 0706.2073).