---
title: 'Hanayo: Wave-Like Pipeline Parallelism'
url: https://www.emergentmind.com/topics/hanayo
type: topic
---

# Hanayo: Wave-Like Pipeline Parallelism

Searching arXiv for Hanayo and related pipeline-parallel scheduling papers.
Hanayo is a pipeline parallelism framework and scheduling method for large-model training whose defining idea is a **wave-like pipeline structure** that seeks to mitigate pipeline bubbles and excessive memory consumption **without duplicating model weights** as in Chimera. In the original presentation, Hanayo is both a scheduling strategy and a runtime system for expressing and executing pipeline schedules; in a later communication-aware reevaluation, it is framed more narrowly as a **wave-like synchronous pipeline-parallel schedule** whose strongest behavior appears in a restricted operating point, especially the \((S,B)=(8,8)\) setting where the number of pipeline stages equals the number of microbatches [2308.15762] [2605.24006].

## 1. Conceptual basis and design objective

The motivating problem is large-model training under accelerator memory constraints. The original Hanayo paper frames this through four “walls”: **Memory wall**, **Scaling wall**, **Computational wall**, and **Development wall**. Within that framing, pipeline parallelism is emphasized as a practical compromise because it splits layers across devices and primarily incurs point-to-point activation and gradient transfer rather than the heavier communication associated with tensor parallelism [2308.15762].

Hanayo is positioned against three prior synchronous baselines. **GPipe** performs all forward passes first and all backward passes after a flush; this is simple but incurs a **high bubble ratio** and **high activation memory consumption**. **DAPPLE / 1F1B** interleaves forward and backward passes, lowering memory and improving utilization, but the paper notes that memory usage is still **uneven across devices** and bubble ratio remains significant. **Chimera** reduces bubbles through a **bidirectional pipeline**, but it does so by requiring **model replicas**, which roughly doubles model-weight memory overhead relative to ordinary pipeline methods [2308.15762].

Hanayo’s central insight is that one can obtain the scheduling benefit associated with bidirectional execution **without duplicating model weights**, by making a single pipeline **change direction** during execution. This produces the method’s characteristic **wave-like** execution. The original paper describes this as a way to preserve the low-bubble behavior associated with Chimera while keeping weight memory comparable to ordinary pipeline-parallel schemes. A later study adopts a more restrictive interpretation: it treats Hanayo as a specialized schedule family from Liu et al. that is effective in its intended regime but not a universal replacement for GPipe, 1F1B, or Chimera across all communication environments [2308.15762] [2605.24006].

## 2. Wave-like structure, notation, and schedule representation

Hanayo defines **wave-like execution** as a schedule in which the pipeline direction changes within the same iteration, forming multiple “V” shapes in the computation timeline. The paper defines the number of “V” patterns in forward/backward execution as the **number of waves**. More waves generally reduce bubbles, although they also increase communication frequency [2308.15762].

The notation standardized in the original paper is as follows.

| Symbol | Meaning | Note |
|---|---|---|
| \(S\) | number of pipeline stages | original Hanayo notation |
| \(B\) | number of micro-batches in one iteration | used in later restricted regime \((S,B)=(8,8)\) |
| \(D\) | number of replicated pipelines | parallelism parameter |
| \(P\) | number of workers in the pipeline | worker count |
| \(W\) | number of waves in one forward/backward iteration | \(W=\frac{S}{2P}\) |
| \(M_w\) | memory consumption for weights of one stage | weight memory |
| \(M_a\) | memory consumption for activations of one stage | activation memory |
| \(T_F\) | time cost of a complete forward pass, divided by \(P\) | compute term |
| \(T_B\) | time cost of a complete backward pass, divided by \(P\) | compute term |
| \(T_C\) | time cost for a single P2P communication | communication term |

A key structural claim in the original work is that Chimera’s bidirectional behavior can be reinterpreted in wave form. The paper states that a 4-stage Chimera pipeline can be transformed into two one-wave pipelines plus data-parallel replication **without extra overhead**, so the extra “replica” effect can be interpreted as ordinary data parallelism rather than duplicated pipeline weights. This is presented as the bridge from Chimera’s two-directional execution to Hanayo’s single-pipeline wave restructuring [2308.15762].

The later reevaluation introduces a **tabular schedule abstraction** for schedule comparison:
\[
S \in \left( \mathcal{M} \times \mathcal{P} \;\cup\; \{\emptyset\} \right)^{W \times T}
\]
with
\[
\mathcal{P} = \{fwd, agrad, wgrad, opt, recomp\}.
\]
This representation is explicitly **structural rather than temporal**: a table cell specifies what runs on a worker in a slot, but not how long that slot lasts on hardware. The abstraction makes visible idle slots, fill and drain behavior, phase ordering, and activation-retention intervals. This later framing is important because it recasts Hanayo not just as a scheduling idea, but as an object of comparison across multiple abstraction levels [2605.24006].

## 3. Bubble-ratio theory and memory semantics

The original Hanayo paper organizes its efficiency analysis around bubbles. It identifies **four bubble types** in Hanayo: **Zone A bubbles**, caused mainly by waiting for forward activations and forward communication; **Zone B bubbles**, caused by mismatch between forward and backward overheads; **Zone C bubbles**, caused by backward propagation and its communication; and **Cross-communication bubbles**, which arise because NCCL batching requires cross-communication to be grouped to avoid deadlock [2308.15762].

For these regions, the paper gives the following bubble sizes:
\[
\frac{T_F}{2W} + T_C
\]
for Zone A,
\[
\frac{P-LR}{2W}(T_B - T_F) + 2T_C
\]
for Zone B, where \(LR\) is the local rank, and
\[
T_B + 2T_C \quad \text{and} \quad T_B + T_C
\]
for Zone C. The resulting overall bubble ratio is
\[
\frac{\frac{1}{W}T_B + \left(1+2W+\frac{2}{P}+\frac{P-2}{3}\right)T_C}{\frac{P}{P-1}T_F + \left(\frac{1}{2W} + \frac{P}{P-1}\right)T_B + \left(\frac{P-2}{2} + 4W\right)T_C}.
\]
Under the simplifying assumption \(T_B = 2T_F\) and ignoring \(T_C\), this becomes
\[
\frac{2P-2}{3PW + P - 1}.
\]
The paper states that this decreases as \(W\) increases, which is the theoretical basis for the claim that more waves reduce bubble ratio [2308.15762].

Memory behavior is a coequal concern. The original paper distinguishes model weights \(M_w\) from activations \(M_a\), and emphasizes peak memory and balance across devices. Its reported findings are: **GPipe** has the highest activation memory because it stores all micro-batch activations; **DAPPLE** is lower than GPipe but has uneven balance; **Chimera** has lower peak memory than GPipe and DAPPLE and is fairly balanced, but requires duplicated weights; **Hanayo** has similar peak memory to Chimera, low variance, and no weight duplication. The reported average memory variance values are **1.33** for GPipe, **16.85** for DAPPLE, **2.86** for Chimera, and **1.44** for Hanayo [2308.15762].

A later comparison refines the interpretation of structural metrics. It reports that **GPipe and 1F1B are equivalent in bubble ratio**, while **Chimera generally has lower bubble ratio than GPipe/1F1B**. It also shows that formula-based bubble estimates can be more optimistic than instantiated schedule tables: for Chimera at \((S,B)=(8,16)\), the bubble ratio is **16%** from formula and **26%** from the table; at \((S,B)=(4,16)\), it is **6%** from formula and **13%** from the table. This suggests that bubble-ratio analysis alone is insufficient for ranking schedules once exact phase ordering and communication structure are instantiated [2605.24006].

## 4. Runtime system and communication handling

Hanayo is presented not merely as a schedule but as a **runtime system** whose design goal is to decouple the **pipeline scheduling algorithm** from the **pipeline execution engine**. The system uses an **action list** inspired by DeepSpeed’s instruction-based scheduler. Rather than relying on coarse commands such as ForwardPass, BackwardPass, and SendActivation, Hanayo breaks execution into finer-grained instructions that include **target device rank** and **local module rank**. The **master scheduler** generates the action list for a given pipeline scheme, and each worker interprets and executes its local actions [2308.15762].

The communication layer includes explicit overlap mechanisms. A key optimization is **prefetching**: before executing the current computation slice, a worker looks ahead, checks the next receive instruction, and launches the next receive request early so that communication can overlap with current computation. The implementation uses **NCCL**, **PyTorch Distributed**, and **batch_isend_irecv** to avoid deadlock in cross-communication. This is especially relevant because increasing the number of waves raises communication frequency, making overlap a central performance determinant [2308.15762].

The later communication-aware study formalizes this perspective by translating schedule tables into execution graphs. In that translation, each non-empty table cell becomes a compute block, consecutive cells on the same worker induce precedence edges, and cross-worker dependencies generate explicit send/receive communication nodes. The simulation then models explicit send/receive dependencies, compute cost, communication cost, overlap, and bottlenecks using Graphculon. This multi-abstraction pipeline is the basis for the claim that schedule rankings are **not abstraction-invariant** [2605.24006].

## 5. Experimental results, scaling behavior, and hardware dependence

The original evaluation covers **four clusters** and both **GPT-like** and **BERT-like** architectures. The four environments are **TACC Lonestar6**, **Tencent CVM cloud server**, **Local cluster A**, and **Local cluster B**. The models are a **BERT-style model** with **64 layers**, **64 attention heads**, and **hidden size 2560**, and a **GPT-style model** with **128 layers**, **16 attention heads**, and **hidden size 1024**. The implemented baselines are **GPipe**, **DAPPLE**, **Chimera**, and **Hanayo**; Chimera is evaluated in a **wave-transformed form** called **Chimera-wave** so that the comparison uses only one set of model weights [2308.15762].

| Environment | GPUs | Connectivity / note |
|---|---|---|
| TACC Lonestar6 | NVIDIA A100, 40 GB HBM, 3 GPUs per node | GPU 0 on socket 0; GPU 1 and 2 on socket 1 |
| Tencent CVM cloud server | 8 NVIDIA V100, 32 GB each | NVLink-connected |
| Local cluster A | 8 NVIDIA A100, 80 GB each | 0–1, 2–3, 4–5, 6–7 via NVLink |
| Local cluster B | 8 NVIDIA A100, 80 GB each | fully NVLink-connected |

The primary metrics are **Peak memory consumption**, **Memory variance across GPUs**, **Throughput**, **Weak scaling efficiency**, **Strong scaling speedup**, and **Adaptability across cluster topologies and GPU types**. The abstract reports **up to a 30.4 \% increase in throughput compared to the state-of-the-art approach**. On eight environment/setting combinations, Hanayo outperforms Chimera by **15.7%**, **30.4%**, **23.2%**, **29.9%**, **8.2%**, **17.1%**, **24.6%**, and **28.0%** [2308.15762].

The paper emphasizes that the optimal wave count depends on the communication environment. In stronger NVLink or better-topology settings, more waves can help; in weaker interconnects, fewer waves may be optimal because additional communication becomes costly. This is corroborated by the configuration search in the **32-GPU BERT** setting on TACC, where the considered pipeline/data-parallel combinations are \((8,4)\), \((16,2)\), and \((32,1)\). The selected best configuration is **data parallelism \(D=4\), pipeline parallelism \(P=8\)**, and for Hanayo the best wave setting is **2 waves** [2308.15762].

The scaling experiments are also explicit. In **weak scaling**, device count and batch size increase proportionally: **8 → 16 → 32** devices and **2 → 4 → 8** batch size. Hanayo outperforms Chimera by **8.19%**, **8.11%**, and **8.13%**; DAPPLE by **33.7%**, **33.2%**, and **33.1%**; and GPipe by **33.4%**, **33.3%**, and **33.3%**. Reported parallel efficiencies are **100.1%** and **99.8%**, with efficiencies above 100% attributed to a GPU batching effect. In **strong scaling**, batch size is fixed at **4** while GPUs scale **8 → 16 → 32**; Hanayo is the top performer, outperforming GPipe and DAPPLE by about **33.3%** and **33.8%**, and Chimera by **8.8%**, **8.1%**, and **8.7%**. The reported speedup values are **188.4%** and **337.5%** [2308.15762].

## 6. Reevaluation, restricted regime, and interpretation

The later study substantially narrows the scope in which Hanayo is evaluated. It states that Hanayo is **designed for the restricted \((S,B)=(8,8)\) regime**, is “specifically designed for \(S = B\) configuration,” and is therefore **excluded from broader microbatch sweeps**. In that paper’s framing, Hanayo is not treated as a general-purpose pipeline schedule; it is treated mainly as a **specialized two-wave schedule** whose performance is meaningful only in that narrow regime [2605.24006].

Within that intended operating point, the reported results are favorable. In the restricted \((S,B)=(8,8)\) setting, Hanayo is faster than Chimera in **8 of 9 modeled systems**. The paper reports that Hanayo typically improves runtime over Chimera by about **11%–14%** in fast- and medium-network regimes. On the baseline system, **Chimera: \(59.32\,\mathrm{s}\)**, **Hanayo: \(51.79\,\mathrm{s}\)**, yielding an improvement of **12.69%**. Idle time also drops from **34.51\%** for Chimera to **24.99\%** for Hanayo [2605.24006].

The same study, however, stresses that the benefit is not robust under communication degradation. In **slow\_nw\_mid\_cp**, Hanayo improves runtime by only **2.33%**. In the most communication-constrained **slow\_nw\_fast\_cp** regime, Hanayo becomes **12.32% slower** than Chimera and has a higher idle ratio. The paper summarizes the point directly: **Hanayo is effective in its intended restricted operating point, but remains sensitive to communication bottlenecks** [2605.24006].

This reevaluation also relocates Hanayo within a broader methodological claim. The comparison is carried out at three levels—**formula-based reasoning**, **idealized tabular schedule representation**, and **communication-aware execution simulation**—and concludes that **schedule rankings are not abstraction-invariant**. Under the modeled assumptions, **GPipe and 1F1B are runtime-equivalent**, but **1F1B achieves a lower activation-memory peak**; **Chimera** is advantageous mainly at low microbatch counts and in communication-favorable regimes; and **Hanayo** is effective in its intended restricted operating point but cannot be ranked independently of the modeled execution environment. A plausible implication is that Hanayo’s status is best understood not as “universally best,” but as a schedule whose quality depends on the interaction between phase ordering, activation retention, and network characteristics [2605.24006].

Source: https://www.emergentmind.com/topics/hanayo