Hierarchical DNN Partitioning Techniques

• Hierarchical DNN Partitioning is a technique that integrates bubble scheduling to manage idle intervals in distributed deep learning, boosting overall efficiency.
• The methodology employs bubble-filling and bubble-elimination strategies to optimize GPU utilization and reduce compute idle time in pipeline-parallel training.
• Key implementations like Optimus, ZB-PP, and OptPipe demonstrate significant speedup and improved throughput across multiprocessor and NUMA systems.

Bubble scheduling denotes a family of scheduling methodologies that precisely identify, characterize, and reclaim idle intervals—known as "bubbles"—in parallel and distributed computational workloads. Originating in the context of hierarchical thread and data placement for multiprocessors, the term now encompasses a broader spectrum in distributed deep learning, pipeline parallelism, multi-modal training, and agentic RL, where bubble scheduling enables practical and provable improvements in hardware utilization, throughput, and wall-clock training time.

1. Definitions: Bubbles and Bubble Scheduling

A "bubble" is a maximal interval of compute resource idleness induced by synchronizations, data dependencies, or work imbalance. In the context of pipeline-parallel large-scale model training, bubbles arise due to warm-up/drain phases, communication stalls, or microbatch imbalances, resulting in significant GPU under-utilization. Bubble scheduling refers to the explicit management and exploitation of such intervals: either by advancing work from other tasks into bubbles (bubble-filling) or by re-factoring the execution schedule so that stages never enter idle (bubble-elimination).

Context	Bubble Definition/example	Main Optimization Target
GPU pipeline parallelism	Idle GPU cycles due to 1F1B	Wall-clock time, MFU
Thread scheduling (NUMA)	Thread groups awaiting sync	NUMA/cache affinity, load balance
RL pipelines	Idle GPUs between rollout/train	End-to-end throughput

Bubble rates (the fraction of total time spent in bubbles) reach up to 48% in 3072-GPU MLLM runs (Feng et al., 2024), ≈40% in moderately sized LLMs (Zhang et al., 2024), or even more on poorly tuned OpenMP code on NUMA (0706.2073). Bubble scheduling methodologies seek to reduce these rates toward zero by means of precise mapping between available compute windows and schedulable work.

2. Sources and Formalization of Bubbles

Bubbles arise from temporal and resource misalignments between data, computation, or communication stages. In large-scale DNN training ("3D parallelism": data, pipeline, tensor parallel), bubbles have three principal sources (Feng et al., 2024):

• Data parallel (DP): Bubbles during all-gather/reduce-scatter that stall forward or backward compute.
• Pipeline parallel (PP): 1F1B schedule creates bubbles during warm-up and drain, and in imbalanced partitions.
• Tensor parallel (TP): Sub-millisecond idle times (e.g., ≈300μs) due to all-gather of layer-partitioned activations.

For each resource $i$, the bubble size is formalized as: $$\mathrm{Bubble}_i = E_i - \text{ideal non-idle work}$$ where $E_i$ is total elapsed time and "ideal" is the sum of actual compute/communication. The bubble rate is: $$\mathrm{BubbleRate}_i = \frac{\mathrm{Bubble}_i}{E_i}$$ In NUMA-aware OpenMP threading (0706.2073), "bubbles" denote groups ("bubbles" as an abstraction) of threads or tasks that share cache/memory affinity and must be dynamically mapped to hardware in a way that minimizes remote accesses and balances load.

3. Bubble Scheduling Algorithms and Implementations

3.1 Bubble Filling in Distributed Training: Optimus

The "Optimus" system (Feng et al., 2024) exemplifies modern bubble scheduling in pipeline-parallel MLLM training. Its algorithm:

1. Bubble Identification: Detect idle intervals ("bubbles") in the LLM pipeline, classifying them into pre-LLM, post-LLM, and fine TP bubbles.
2. Work Decomposition: Decompose auxiliary tasks (e.g., encoder compute) into microbatch-level and further into individual CUDA kernels.
3. Critical Path Search: Iteratively identify the encoder pipeline that is limiting the end-to-end makespan.
4. Kernel Packing: Schedule as many kernels as fit into the identified bubble, subject to duration and dependency constraints.
5. Dependency and Resource Checks: Enforce $EF_i \leq F_i$ and $EB_i \geq B_i$ per microbatch, and ensure per-GPU memory ($$DP_{enc}\cdot\phi_{enc} + DP_{LLM}\cdot\phi_{LLM} \leq n_{gpu} \cdot M_{gpu}$$).

The scheduler exploits up to 85% of encoder FLOPs into bubbles at large scales, delivering 20–21% speedup for ViT+GPT runs over 3072 GPUs (Feng et al., 2024).

3.2 Bubble Elimination: Zero Bubble Pipeline Parallelism

"Zero-Bubble Pipeline Parallelism" (ZB-PP) (Qi et al., 2023) aims for complete eradication of bubbles in synchronous pipeline training. By decomposing backward passes into input-gradient and parameter-gradient components (B/W split), it constructs schedules where

$(\text{bubble size}) = (p-1)(T_F + T_B - 2T_W)$

can actually reach zero for balanced compute times ($T_F = T_B = T_W$).

ZB-PP schedules both handcrafted and ILP-optimized, and further introduces post-update validation to avoid synchronization bottlenecks on the optimizer step, allowing full parallelogram-shaped schedules with no idle time under ideal conditions.

3.3 Fine-grained Joint Scheduling: OptPipe

"OptPipe" (Li et al., 6 Oct 2025) generalizes bubble scheduling to optimize memory, activation, and offload/reload events via a MILP that minimizes pipeline bubble time subject to memory constraints. The approach distinguishes itself by jointly scheduling all operations (F, B, W) with explicit offload decision variables and memory events, yielding up to 50% fewer idle bubbles and better memory utilization than previous heuristics.

3.4 Agentic and RL Contexts: Spatiotemporal Multiplexing

"SeamlessFlow" (Wang et al., 15 Aug 2025) models each resource by dynamic capability tags and uses preemptive, tag-driven scheduling to multiplex rollout and training tasks. Its spatiotemporal scheme reassociates idle training/rollout nodes between incompatible stages, ensuring zero bubbles ($\beta_k = 0$ at all times), delivering 92–100% throughput increases and $$\mathrm{Bubble}_i = E_i - \text{ideal non-idle work}$$0 utilization on multi-GPU RL training runs.

3.5 Side-task Bubble Harvesting: FreeRide

"FreeRide" (Zhang et al., 2024) characterizes pipeline-parallel training bubbles (types A/B/C) and schedules auxiliary low-priority side-tasks (deep learning, image processing, graph analytics) into those intervals, with tight runtime and memory controls (via MPS, API, and OS signals). This system achieves cost-savings of $$\mathrm{Bubble}_i = E_i - \text{ideal non-idle work}$$1 with sub-1.5% training time overhead.

4. Bubble Abstraction in Hierarchical Thread Schedulers

The bubble abstraction was first formalized for hierarchical multiprocessor/thread scheduling in BubbleSched (0706.2073). Here, a bubble is a container of threads and/or sub-bubbles, carrying affinity and load metadata. The scheduler recursively maps a tree of bubbles (mirroring, e.g., nested OpenMP teams) onto machine runqueues, seeking to keep affinity groups together, preferring local caches and NUMA, and minimizes cache/memory traffic.

The "bubble_spread" algorithm recursively assigns bubbles to runqueues, "exploding" bubbles if their load exceeds a tunable threshold $$\mathrm{Bubble}_i = E_i - \text{ideal non-idle work}$$2, and recursing into finer granularity.

Experimental results on the NAS BT-MZ benchmark showed up to 30% speedup over baseline threading by remapping thread groups onto hardware with bubble affinity.

5. Empirical Results and Efficiency Metrics

Empirical studies demonstrate the following bubble scheduling effects:

• End-to-end DNN Training: Optimus yields 20.5–21.3% reduction in iteration time at 3072-GPU scale; coarse exploitation alone covers 34% of available encoder FLOPs, fine-grained exploitation raises this to 57–85% (Feng et al., 2024).
• Zero-bubble PP: ZB-PP achieves 20–30% higher throughput under fixed memory, up to 31% if more memory is available, with bubble rates <1% (Qi et al., 2023).
• Memory-constrained Scheduling: OptPipe reduces idle pipeline time by up to 50% for large LLMs, sometimes enabling larger models per-GPU (Li et al., 6 Oct 2025).
• RL Agentic Pipelines: SeamlessFlow delivers $$\mathrm{Bubble}_i = E_i - \text{ideal non-idle work}$$3 end-to-end throughput and 98% utilization by provably eliminating bubbles (Wang et al., 15 Aug 2025).
• Opportunistic Colocation: FreeRide schedules side-tasks during real pipeline bubbles, providing up to 11.8% cost savings for certain auxiliary jobs with less than 1.5% overhead (Zhang et al., 2024).
• NUMA Threading: BubbleSched improves speedup in nested OpenMP programs by 25–30% over traditional schedulers (0706.2073).

6. Generalization, Limitations, and Future Directions

While bubble scheduling originated in thread-affinity and pipeline parallel LLM contexts, the methodology generalizes to any system with:

• A dominant workload whose execution leaves idle intervals due to communication/synchronization.
• An auxiliary workload decomposable into small enough schedulable units (kernels, tasks).
• Explicit data-dependency constraints between main and auxiliary work.

Bubble scheduling's two-stage approach—identify bubbles, then fit auxiliary tasks subject to dependencies and resource constraints—can be applied to custom pipeline topologies, multi-branch or agentic computation graphs, asynchronous inference, and even system-level thread/data placement.

Limitations and ongoing challenges include:

• Optimal bubble-filling schedules are NP-hard; practical solvers use heuristics, MILPs, or ILPs, and must blend warm-starts and caching (Li et al., 6 Oct 2025, Qi et al., 2023).
• Fine-grained kernel scheduling introduces nontrivial memory overhead (e.g., up to 12% more GPU memory in Optimus (Feng et al., 2024)).
• Dynamic and adaptive rescheduling is an open problem, especially in heterogeneous or bursty environments.
• For NUMA/threading bubbles, precise load and affinity estimation remains a challenge, and "bubble explosion" sensitivity must be application-tuned (0706.2073).

A plausible implication is that as model and hardware scales grow, advanced bubble scheduling—whether via kernel-level packing, tag-driven preemptive multiplexing, or NUMA-aware affinity mapping—will be an essential substrate for scientific and industrial distributed compute platforms.