Hanayo: Wave-Like Pipeline Parallelism

• Hanayo is a pipeline parallelism framework that employs a wave-like execution schedule to minimize bubbles and optimize memory use in large-model training.
• It avoids model weight duplication unlike Chimera, achieving balanced memory consumption and effective utilization of accelerator resources.
• Experimental results show up to a 30% throughput improvement over existing methods, particularly in the restricted (S,B)=(8,8) operating regime.

Searching arXiv for Hanayo and related pipeline-parallel scheduling papers. Hanayo is a pipeline parallelism framework and scheduling method for large-model training whose defining idea is a wave-like pipeline structure that seeks to mitigate pipeline bubbles and excessive memory consumption without duplicating model weights as in Chimera. In the original presentation, Hanayo is both a scheduling strategy and a runtime system for expressing and executing pipeline schedules; in a later communication-aware reevaluation, it is framed more narrowly as a wave-like synchronous pipeline-parallel schedule whose strongest behavior appears in a restricted operating point, especially the $(S,B)=(8,8)$ setting where the number of pipeline stages equals the number of microbatches (Liu et al., 2023, Barley et al., 19 May 2026).

1. Conceptual basis and design objective

The motivating problem is large-model training under accelerator memory constraints. The original Hanayo paper frames this through four “walls”: Memory wall, Scaling wall, Computational wall, and Development wall. Within that framing, pipeline parallelism is emphasized as a practical compromise because it splits layers across devices and primarily incurs point-to-point activation and gradient transfer rather than the heavier communication associated with tensor parallelism (Liu et al., 2023).

Hanayo is positioned against three prior synchronous baselines. GPipe performs all forward passes first and all backward passes after a flush; this is simple but incurs a high bubble ratio and high activation memory consumption. DAPPLE / 1F1B interleaves forward and backward passes, lowering memory and improving utilization, but the paper notes that memory usage is still uneven across devices and bubble ratio remains significant. Chimera reduces bubbles through a bidirectional pipeline, but it does so by requiring model replicas, which roughly doubles model-weight memory overhead relative to ordinary pipeline methods (Liu et al., 2023).

Hanayo’s central insight is that one can obtain the scheduling benefit associated with bidirectional execution without duplicating model weights, by making a single pipeline change direction during execution. This produces the method’s characteristic wave-like execution. The original paper describes this as a way to preserve the low-bubble behavior associated with Chimera while keeping weight memory comparable to ordinary pipeline-parallel schemes. A later study adopts a more restrictive interpretation: it treats Hanayo as a specialized schedule family from Liu et al. that is effective in its intended regime but not a universal replacement for GPipe, 1F1B, or Chimera across all communication environments (Liu et al., 2023, Barley et al., 19 May 2026).

2. Wave-like structure, notation, and schedule representation

Hanayo defines wave-like execution as a schedule in which the pipeline direction changes within the same iteration, forming multiple “V” shapes in the computation timeline. The paper defines the number of “V” patterns in forward/backward execution as the number of waves. More waves generally reduce bubbles, although they also increase communication frequency (Liu et al., 2023).

The notation standardized in the original paper is as follows.

Symbol	Meaning	Note
$S$	number of pipeline stages	original Hanayo notation
$B$	number of micro-batches in one iteration	used in later restricted regime $(S,B)=(8,8)$
$D$	number of replicated pipelines	parallelism parameter
$P$	number of workers in the pipeline	worker count
$W$	number of waves in one forward/backward iteration	$W=\frac{S}{2P}$
$M_w$	memory consumption for weights of one stage	weight memory
$M_a$	memory consumption for activations of one stage	activation memory
$S$0	time cost of a complete forward pass, divided by $S$1	compute term
$S$2	time cost of a complete backward pass, divided by $S$3	compute term
$S$4	time cost for a single P2P communication	communication term

A key structural claim in the original work is that Chimera’s bidirectional behavior can be reinterpreted in wave form. The paper states that a 4-stage Chimera pipeline can be transformed into two one-wave pipelines plus data-parallel replication without extra overhead, so the extra “replica” effect can be interpreted as ordinary data parallelism rather than duplicated pipeline weights. This is presented as the bridge from Chimera’s two-directional execution to Hanayo’s single-pipeline wave restructuring (Liu et al., 2023).

The later reevaluation introduces a tabular schedule abstraction for schedule comparison: $S$5 with

$S$6

This representation is explicitly structural rather than temporal: a table cell specifies what runs on a worker in a slot, but not how long that slot lasts on hardware. The abstraction makes visible idle slots, fill and drain behavior, phase ordering, and activation-retention intervals. This later framing is important because it recasts Hanayo not just as a scheduling idea, but as an object of comparison across multiple abstraction levels (Barley et al., 19 May 2026).

3. Bubble-ratio theory and memory semantics

The original Hanayo paper organizes its efficiency analysis around bubbles. It identifies four bubble types in Hanayo: Zone A bubbles, caused mainly by waiting for forward activations and forward communication; Zone B bubbles, caused by mismatch between forward and backward overheads; Zone C bubbles, caused by backward propagation and its communication; and Cross-communication bubbles, which arise because NCCL batching requires cross-communication to be grouped to avoid deadlock (Liu et al., 2023).

For these regions, the paper gives the following bubble sizes: $S$7 for Zone A,

$S$8

for Zone B, where $S$9 is the local rank, and

$B$0

for Zone C. The resulting overall bubble ratio is

$B$1

Under the simplifying assumption $B$2 and ignoring $B$3, this becomes

$B$4

The paper states that this decreases as $B$5 increases, which is the theoretical basis for the claim that more waves reduce bubble ratio (Liu et al., 2023).

Memory behavior is a coequal concern. The original paper distinguishes model weights $B$6 from activations $B$7, and emphasizes peak memory and balance across devices. Its reported findings are: GPipe has the highest activation memory because it stores all micro-batch activations; DAPPLE is lower than GPipe but has uneven balance; Chimera has lower peak memory than GPipe and DAPPLE and is fairly balanced, but requires duplicated weights; Hanayo has similar peak memory to Chimera, low variance, and no weight duplication. The reported average memory variance values are 1.33 for GPipe, 16.85 for DAPPLE, 2.86 for Chimera, and 1.44 for Hanayo (Liu et al., 2023).

A later comparison refines the interpretation of structural metrics. It reports that GPipe and 1F1B are equivalent in bubble ratio, while Chimera generally has lower bubble ratio than GPipe/1F1B. It also shows that formula-based bubble estimates can be more optimistic than instantiated schedule tables: for Chimera at $B$8, the bubble ratio is 16% from formula and 26% from the table; at $B$9, it is 6% from formula and 13% from the table. This suggests that bubble-ratio analysis alone is insufficient for ranking schedules once exact phase ordering and communication structure are instantiated (Barley et al., 19 May 2026).

4. Runtime system and communication handling

Hanayo is presented not merely as a schedule but as a runtime system whose design goal is to decouple the pipeline scheduling algorithm from the pipeline execution engine. The system uses an action list inspired by DeepSpeed’s instruction-based scheduler. Rather than relying on coarse commands such as ForwardPass, BackwardPass, and SendActivation, Hanayo breaks execution into finer-grained instructions that include target device rank and local module rank. The master scheduler generates the action list for a given pipeline scheme, and each worker interprets and executes its local actions (Liu et al., 2023).

The communication layer includes explicit overlap mechanisms. A key optimization is prefetching: before executing the current computation slice, a worker looks ahead, checks the next receive instruction, and launches the next receive request early so that communication can overlap with current computation. The implementation uses NCCL, PyTorch Distributed, and batch_isend_irecv to avoid deadlock in cross-communication. This is especially relevant because increasing the number of waves raises communication frequency, making overlap a central performance determinant (Liu et al., 2023).

The later communication-aware study formalizes this perspective by translating schedule tables into execution graphs. In that translation, each non-empty table cell becomes a compute block, consecutive cells on the same worker induce precedence edges, and cross-worker dependencies generate explicit send/receive communication nodes. The simulation then models explicit send/receive dependencies, compute cost, communication cost, overlap, and bottlenecks using Graphculon. This multi-abstraction pipeline is the basis for the claim that schedule rankings are not abstraction-invariant (Barley et al., 19 May 2026).

5. Experimental results, scaling behavior, and hardware dependence

The original evaluation covers four clusters and both GPT-like and BERT-like architectures. The four environments are TACC Lonestar6, Tencent CVM cloud server, Local cluster A, and Local cluster B. The models are a BERT-style model with 64 layers, 64 attention heads, and hidden size 2560, and a GPT-style model with 128 layers, 16 attention heads, and hidden size 1024. The implemented baselines are GPipe, DAPPLE, Chimera, and Hanayo; Chimera is evaluated in a wave-transformed form called Chimera-wave so that the comparison uses only one set of model weights (Liu et al., 2023).

Environment	GPUs	Connectivity / note
TACC Lonestar6	NVIDIA A100, 40 GB HBM, 3 GPUs per node	GPU 0 on socket 0; GPU 1 and 2 on socket 1
Tencent CVM cloud server	8 NVIDIA V100, 32 GB each	NVLink-connected
Local cluster A	8 NVIDIA A100, 80 GB each	0–1, 2–3, 4–5, 6–7 via NVLink
Local cluster B	8 NVIDIA A100, 80 GB each	fully NVLink-connected

The primary metrics are Peak memory consumption, Memory variance across GPUs, Throughput, Weak scaling efficiency, Strong scaling speedup, and Adaptability across cluster topologies and GPU types. The abstract reports up to a 30.4 \% increase in throughput compared to the state-of-the-art approach. On eight environment/setting combinations, Hanayo outperforms Chimera by 15.7%, 30.4%, 23.2%, 29.9%, 8.2%, 17.1%, 24.6%, and 28.0% (Liu et al., 2023).

The paper emphasizes that the optimal wave count depends on the communication environment. In stronger NVLink or better-topology settings, more waves can help; in weaker interconnects, fewer waves may be optimal because additional communication becomes costly. This is corroborated by the configuration search in the 32-GPU BERT setting on TACC, where the considered pipeline/data-parallel combinations are $(S,B)=(8,8)$0, $(S,B)=(8,8)$1, and $(S,B)=(8,8)$2. The selected best configuration is data parallelism $(S,B)=(8,8)$3, pipeline parallelism $(S,B)=(8,8)$4, and for Hanayo the best wave setting is 2 waves (Liu et al., 2023).

The scaling experiments are also explicit. In weak scaling, device count and batch size increase proportionally: 8 → 16 → 32 devices and 2 → 4 → 8 batch size. Hanayo outperforms Chimera by 8.19%, 8.11%, and 8.13%; DAPPLE by 33.7%, 33.2%, and 33.1%; and GPipe by 33.4%, 33.3%, and 33.3%. Reported parallel efficiencies are 100.1% and 99.8%, with efficiencies above 100% attributed to a GPU batching effect. In strong scaling, batch size is fixed at 4 while GPUs scale 8 → 16 → 32; Hanayo is the top performer, outperforming GPipe and DAPPLE by about 33.3% and 33.8%, and Chimera by 8.8%, 8.1%, and 8.7%. The reported speedup values are 188.4% and 337.5% (Liu et al., 2023).

6. Reevaluation, restricted regime, and interpretation

The later study substantially narrows the scope in which Hanayo is evaluated. It states that Hanayo is designed for the restricted $(S,B)=(8,8)$5 regime, is “specifically designed for $(S,B)=(8,8)$6 configuration,” and is therefore excluded from broader microbatch sweeps. In that paper’s framing, Hanayo is not treated as a general-purpose pipeline schedule; it is treated mainly as a specialized two-wave schedule whose performance is meaningful only in that narrow regime (Barley et al., 19 May 2026).

Within that intended operating point, the reported results are favorable. In the restricted $(S,B)=(8,8)$7 setting, Hanayo is faster than Chimera in 8 of 9 modeled systems. The paper reports that Hanayo typically improves runtime over Chimera by about 11%–14% in fast- and medium-network regimes. On the baseline system, Chimera: $(S,B)=(8,8)$8, Hanayo: $(S,B)=(8,8)$9, yielding an improvement of 12.69%. Idle time also drops from 34.51\% for Chimera to 24.99\% for Hanayo (Barley et al., 19 May 2026).

The same study, however, stresses that the benefit is not robust under communication degradation. In slow_nw_mid_cp, Hanayo improves runtime by only 2.33%. In the most communication-constrained slow_nw_fast_cp regime, Hanayo becomes 12.32% slower than Chimera and has a higher idle ratio. The paper summarizes the point directly: Hanayo is effective in its intended restricted operating point, but remains sensitive to communication bottlenecks (Barley et al., 19 May 2026).

This reevaluation also relocates Hanayo within a broader methodological claim. The comparison is carried out at three levels—formula-based reasoning, idealized tabular schedule representation, and communication-aware execution simulation—and concludes that schedule rankings are not abstraction-invariant. Under the modeled assumptions, GPipe and 1F1B are runtime-equivalent, but 1F1B achieves a lower activation-memory peak; Chimera is advantageous mainly at low microbatch counts and in communication-favorable regimes; and Hanayo is effective in its intended restricted operating point but cannot be ranked independently of the modeled execution environment. A plausible implication is that Hanayo’s status is best understood not as “universally best,” but as a schedule whose quality depends on the interaction between phase ordering, activation retention, and network characteristics (Barley et al., 19 May 2026).