---
title: Block-Quantized Collectives (ZeRO++)
url: https://www.emergentmind.com/topics/block-quantized-collectives-zero
type: topic
---

# Block-Quantized Collectives (ZeRO++)

Block-Quantized Collectives (ZeRO++) are advanced communication primitives for distributed deep learning that employ block-wise quantization to reduce the communication volume of collective operations such as All-Gather and Reduce-Scatter, enabling highly efficient large language model (LLM) training at scale. Distinct from traditional floating-point collectives, block-quantized schemes partition tensors into small contiguous blocks, each quantized independently, and communicate only quantized values and block-specific scaling metadata. ZeRO++ integrates these techniques within the Zero Redundancy Optimizer (ZeRO) paradigm, achieving up to 4× reduction in communication volume with negligible convergence impact in both low- and high-bandwidth environments [2306.10209].

## 1. Block-Wise Quantization Fundamentals

Block-wise quantization divides each tensor into contiguous blocks of fixed size $B$ (typically $128 \leq B \leq 512$) and independently quantizes each block to $b_q$ bits, commonly 8. For a block $m$ with elements $\{x_i\}_{i=1}^{B}$, a scaling factor $\alpha_m = \max_{i} |x_i|$ is computed, and elements are quantized as:
\[
Q(x_i) = \mathrm{clip}\left( \lfloor x_i\,s_m + 0.5 \rfloor, -2^{b_q-1}, 2^{b_q-1}-1 \right), \quad s_m = \frac{2^{b_q-1}-1}{\alpha_m}
\]
Dequantization reverses this via
\[
\widehat{x}_i = \alpha_m\,\frac{Q(x_i)}{2^{b_q-1}-1}
\]
Per-block scaling isolates outlier values, maintaining numerical fidelity and lowering worst-case and mean squared quantization error. Metadata overhead is kept small—e.g., with $B=128$, $b_s=16$, the per-element overhead is $16/128$ bits ($0.125$ bits).

## 2. ZeRO++ Block-Quantized Collectives: Algorithms and Pseudocode

ZeRO++ targets all major bandwidth-bound collectives in ZeRO-3:

- **qwZ:** Block-Quantized All-Gather for weights in the forward pass.
- **hpZ:** Hierarchical partitioning eliminates redundant backward All-Gather.
- **qgZ:** All-to-All Block-Quantized Gradient Reduction as a Reduce-Scatter replacement.

The block-quantized All-Gather algorithm follows:

```python
function block_quantized_allgather(x_local: FP16[N]) -> x_full: FP16[N*R]
    // Partition into M blocks of B elements
    for m in 0 .. M-1:
        block = x_local[m*B : (m+1)*B]
        alpha[m] = max(abs(block))
        s[m] = (2^{b_q-1}-1) / alpha[m]
        for i in 0 .. B-1:
            q[m][i] = clip(round(block[i] * s[m]), -2^{b_q-1}, 2^{b_q-1}-1)
    send_buf = pack(q, alpha)
    allgather(send_buf, recv_buf, group=R)
    for rank r in 0..R-1:
        (q_r, alpha_r) = unpack(recv_buf[r])
        for m in 0..M-1, i in 0..B-1:
            idx = r*N + m*B + i
            x_full[idx] = alpha_r[m] * q_r[m][i] / (2^{b_q-1}-1)
    return x_full
end function
```

Each rank communicates only quantized values and scales, with total bit volume per rank $N\,b_q + (N/B)\,b_s$ [2306.10209].

## 3. Communication Complexity and Performance Gains

Block-quantized collectives substantially reduce communication volume:

| Operation            | Volume Reduction Factor |
|----------------------|-----------------------|
| Forward All-Gather   | $2\times$ (qwZ)       |
| Backward All-Gather  | $0$ (eliminated by hpZ) |
| Grad Reduce-Scatter  | $4\times$ (qgZ)       |

The aggregate reduction factor for ZeRO++ is $4\times$ over ZeRO-3. End-to-end throughput measurements on GPT-style models (18B–138B) with 384 GPUs and 100 Gbps InfiniBand demonstrate speedups up to $2.16\times$, scaling to 45% of GPU peak TFLOPs. On high-bandwidth clusters, speedup ranges from $1.13\times$ to $1.30\times$ [2306.10209].

## 4. Numerical Error Analysis and Convergence

Block quantization introduces bounded error per block:

- Worst-case error: $|\Delta x| \leq \alpha_m / 2^{b_q}$
- MSE: proportional to $\alpha_m^2 / 2^{2(b_q-1)}$

Decorrelation of quantization noise across blocks and averaging during distributed Reduce-Scatter further suppresses error. Empirical validation on GPT-350M trained over 30 billion tokens shows that ZeRO++ with full block quantization (qwZ+hpZ+qgZ) increases validation loss by only $+1.97\%$, while omitting gradient quantization eliminates almost all loss increase. Partial quantization strategies provide intermediate tradeoffs [2306.10209]. 

| Method                         | Validation Loss   |
|---------------------------------|------------------|
| ZeRO-3 baseline                 | 2.121762         |
| ZeRO++ (full quantization)      | 2.165584 (+1.97%)|
| ZeRO++ (no grad quantization)   | 2.121653 (-0.005%)|
| ZeRO++ (partial grad quant)     | 2.134013 (+0.58%)|

This confirms that block-quantized collectives, when coupled with hierarchical and all-to-all schemes, yield negligible convergence degradation.

## 5. Interactions with Hardware and Compiler Optimizations

Block-quantized collective methods benefit from hardware and compiler-aware optimizations. EQuARX [2506.17615] demonstrates the integration of block-quantized AllReduce operations directly into the XLA compiler for TPUs, leveraging per-block int8 quantization, pipelined communication, and accelerator-register alignment to maximize overlap and minimize transformations. Pipelined “microsharding” and tight hardware coupling enable nearly optimal communication hiding and a $1.8\times$ speedup relative to baseline BF16 collectives.

Critical details include:

- Per-block symmetric quantization matching device register shapes (e.g., $8\times128$ tiles on TPU)
- Deep pipelining of quantize-send-receive-dequantize-add stages across microshards
- Semi-loop ring variants to minimize quantization error in high device-count settings

EQuARX achieves two orders of magnitude less quantization MSE than naive FP8 collectives while preserving throughput gains and negligible model quality loss.

## 6. Limitations, Extensions, and Applicability

Limitations of block-quantized collectives include:

- Metadata overhead increases with finer block granularity
- Accumulated error scales with communication hop count, which is partially mitigated by semi-loop or hierarchical variants
- Integer overflow risk appears if quantization ranges are underestimated and requires dynamic adaptation or periodic higher-precision correction
- Hardware and software support for fusing quantized collectives impacts usability beyond targeted accelerators (e.g., TPUs vs. general-purpose GPUs)

EQuARX identifies that block-quantized rings, deep pipelining, and register-aware block layouts can be transplanted into ZeRO++ pipelines, potentially improving collective efficiency further for future distributed LLM training [2506.17615].

## 7. Relationship to Other Quantization in Distributed Learning

Block-quantized collectives are distinguished from classic quantization by their focus on reducing communication in distributed data-parallel training rather than limiting model storage or inference compute. Unlike naive quantized AllReduce, block-quantized designs avoid error accumulation by using high-precision accumulation interleaved with communication and exploit block-local statistics for robustness to outliers. These methods are orthogonal to model, gradient, or activation quantization applied for memory or inference, and are complementary in end-to-end distributed system design [2306.10209], [2506.17615].

Source: https://www.emergentmind.com/topics/block-quantized-collectives-zero