Block-Quantized Collectives (ZeRO++)

• The paper introduces block-quantized collectives that partition tensors into blocks and independently quantize them, reducing communication volume by up to 4× with negligible convergence impact.
• It integrates these techniques within the Zero Redundancy Optimizer paradigm to optimize All-Gather, Reduce-Scatter, and gradient reduction operations in distributed deep learning.
• Empirical results on GPT-style models demonstrate significant speedups and maintained model quality, even under varying bandwidth conditions.

Block-Quantized Collectives (ZeRO++) are advanced communication primitives for distributed deep learning that employ block-wise quantization to reduce the communication volume of collective operations such as All-Gather and Reduce-Scatter, enabling highly efficient LLM training at scale. Distinct from traditional floating-point collectives, block-quantized schemes partition tensors into small contiguous blocks, each quantized independently, and communicate only quantized values and block-specific scaling metadata. ZeRO++ integrates these techniques within the Zero Redundancy Optimizer (ZeRO) paradigm, achieving up to 4× reduction in communication volume with negligible convergence impact in both low- and high-bandwidth environments (Wang et al., 2023).

1. Block-Wise Quantization Fundamentals

Block-wise quantization divides each tensor into contiguous blocks of fixed size $B$ (typically $128 \leq B \leq 512$) and independently quantizes each block to $b_q$ bits, commonly 8. For a block $m$ with elements $\{x_i\}_{i=1}^{B}$, a scaling factor $\alpha_m = \max_{i} |x_i|$ is computed, and elements are quantized as: $$Q(x_i) = \mathrm{clip}\left( \lfloor x_i\,s_m + 0.5 \rfloor, -2^{b_q-1}, 2^{b_q-1}-1 \right), \quad s_m = \frac{2^{b_q-1}-1}{\alpha_m}$$ Dequantization reverses this via

$$\widehat{x}_i = \alpha_m\,\frac{Q(x_i)}{2^{b_q-1}-1}$$

Per-block scaling isolates outlier values, maintaining numerical fidelity and lowering worst-case and mean squared quantization error. Metadata overhead is kept small—e.g., with $B=128$, $b_s=16$, the per-element overhead is $128 \leq B \leq 512$0 bits ($128 \leq B \leq 512$1 bits).

2. ZeRO++ Block-Quantized Collectives: Algorithms and Pseudocode

ZeRO++ targets all major bandwidth-bound collectives in ZeRO-3:

• qwZ: Block-Quantized All-Gather for weights in the forward pass.
• hpZ: Hierarchical partitioning eliminates redundant backward All-Gather.
• qgZ: All-to-All Block-Quantized Gradient Reduction as a Reduce-Scatter replacement.

The block-quantized All-Gather algorithm follows:

$b_q$5

Each rank communicates only quantized values and scales, with total bit volume per rank $128 \leq B \leq 512$2 (Wang et al., 2023).

3. Communication Complexity and Performance Gains

Block-quantized collectives substantially reduce communication volume:

Operation	Volume Reduction Factor
Forward All-Gather	$128 \leq B \leq 512$3 (qwZ)
Backward All-Gather	$128 \leq B \leq 512$4 (eliminated by hpZ)
Grad Reduce-Scatter	$128 \leq B \leq 512$5 (qgZ)

The aggregate reduction factor for ZeRO++ is $128 \leq B \leq 512$6 over ZeRO-3. End-to-end throughput measurements on GPT-style models (18B–138B) with 384 GPUs and 100 Gbps InfiniBand demonstrate speedups up to $128 \leq B \leq 512$7, scaling to 45% of GPU peak TFLOPs. On high-bandwidth clusters, speedup ranges from $128 \leq B \leq 512$8 to $128 \leq B \leq 512$9 (Wang et al., 2023).

4. Numerical Error Analysis and Convergence

Block quantization introduces bounded error per block:

• Worst-case error: $b_q$0
• MSE: proportional to $b_q$1

Decorrelation of quantization noise across blocks and averaging during distributed Reduce-Scatter further suppresses error. Empirical validation on GPT-350M trained over 30 billion tokens shows that ZeRO++ with full block quantization (qwZ+hpZ+qgZ) increases validation loss by only $b_q$2, while omitting gradient quantization eliminates almost all loss increase. Partial quantization strategies provide intermediate tradeoffs (Wang et al., 2023).

Method	Validation Loss
ZeRO-3 baseline	2.121762
ZeRO++ (full quantization)	2.165584 (+1.97%)
ZeRO++ (no grad quantization)	2.121653 (-0.005%)
ZeRO++ (partial grad quant)	2.134013 (+0.58%)

This confirms that block-quantized collectives, when coupled with hierarchical and all-to-all schemes, yield negligible convergence degradation.

5. Interactions with Hardware and Compiler Optimizations

Block-quantized collective methods benefit from hardware and compiler-aware optimizations. EQuARX (Ahmed et al., 21 Jun 2025) demonstrates the integration of block-quantized AllReduce operations directly into the XLA compiler for TPUs, leveraging per-block int8 quantization, pipelined communication, and accelerator-register alignment to maximize overlap and minimize transformations. Pipelined “microsharding” and tight hardware coupling enable nearly optimal communication hiding and a $b_q$3 speedup relative to baseline BF16 collectives.

Critical details include:

• Per-block symmetric quantization matching device register shapes (e.g., $b_q$4 tiles on TPU)
• Deep pipelining of quantize-send-receive-dequantize-add stages across microshards
• Semi-loop ring variants to minimize quantization error in high device-count settings

EQuARX achieves two orders of magnitude less quantization MSE than naive FP8 collectives while preserving throughput gains and negligible model quality loss.

6. Limitations, Extensions, and Applicability

Limitations of block-quantized collectives include:

• Metadata overhead increases with finer block granularity
• Accumulated error scales with communication hop count, which is partially mitigated by semi-loop or hierarchical variants
• Integer overflow risk appears if quantization ranges are underestimated and requires dynamic adaptation or periodic higher-precision correction
• Hardware and software support for fusing quantized collectives impacts usability beyond targeted accelerators (e.g., TPUs vs. general-purpose GPUs)

EQuARX identifies that block-quantized rings, deep pipelining, and register-aware block layouts can be transplanted into ZeRO++ pipelines, potentially improving collective efficiency further for future distributed LLM training (Ahmed et al., 21 Jun 2025).

7. Relationship to Other Quantization in Distributed Learning

Block-quantized collectives are distinguished from classic quantization by their focus on reducing communication in distributed data-parallel training rather than limiting model storage or inference compute. Unlike naive quantized AllReduce, block-quantized designs avoid error accumulation by using high-precision accumulation interleaved with communication and exploit block-local statistics for robustness to outliers. These methods are orthogonal to model, gradient, or activation quantization applied for memory or inference, and are complementary in end-to-end distributed system design (Wang et al., 2023, Ahmed et al., 21 Jun 2025).