FlashCommunication V2: Arbitrary-Bit LLM Communication

• The paper introduces a novel communication framework that treats quantization as a first-class systems challenge, enabling arbitrary integer bit widths through bit splitting and spike reserving.
• It employs bit splitting to decompose irregular bit widths into hardware-friendly slices, overcoming the limitations of fixed SIMD support and misaligned memory accesses.
• Spike reserving combined with fused CUDA kernels minimizes quantization error in low-bit regimes, achieving significant speedups in collective operations such as AllReduce and All2All.

FlashCommunication V2 is a software–hardware co-designed communication framework for distributed LLM training and inference that enables cross-GPU transmission at arbitrary integer bit widths, including irregular widths such as 5-bit and 3-bit, while retaining acceptable accuracy at extremely low precision. Its central contribution is to treat communication-only quantization as a first-class systems problem rather than as a by-product of model quantization, combining arbitrary-bit packing, outlier-aware quantization, fused CUDA kernels, and topology-aware collectives for AllReduce and All2All on both NVLink-based and PCIe-based systems (Li et al., 4 Aug 2025).

1. Problem setting and conceptual scope

The framework targets the communication bottlenecks that arise in three recurring regimes of modern LLM systems: tensor parallelism, where activations or gradients must be exchanged with AllReduce; expert parallelism in MoE models, where token dispatch and gather are implemented with All2All; and serving disaggregation, where KV-cache or activation transfers occur between prefill and decoding pools. In these settings, communication can dominate runtime even on NVLink systems, and it is often the principal bottleneck on PCIe-only nodes such as L40 servers (Li et al., 4 Aug 2025).

The paper identifies two obstacles to low-bit communication. The first is hardware granularity. GPUs and interconnects are optimized for byte-aligned loads and stores and for fixed, power-of-two formats such as INT8, INT4, FP8, and BF16. There is no native SIMD support for 3-bit or 5-bit integers, so naive implementations incur misaligned memory accesses, heavy bitwise overhead, and reduced warp efficiency. The second obstacle is quantization error under extreme compression. The activation distributions of contemporary LLMs contain large outliers, and simple asymmetric round-to-nearest quantization can fail severely below 4-bit. For AllReduce, the paper reports that naive 2-bit communication for Llama-3-8B can drive C4 perplexity to $7 \times 10^5$, which makes aggressive compression unusable without additional outlier handling (Li et al., 4 Aug 2025).

FlashCommunication V2 is therefore designed as a communication path replacement rather than as a model reparameterization method. Tensors remain in BF16 or FP16 for computation, but the inter-GPU payload is quantized, packed, transmitted, unpacked, and dequantized inside fused kernels. This separation is important: the framework is intended to be integrated into existing tensor-parallel and expert-parallel pipelines without requiring low-bit matmul kernels or low-bit model weights.

2. Arbitrary-bit transmission via bit splitting

The mechanism that enables “any bit communication” is bit splitting. Instead of demanding native support for every target bit width $b$, the framework decomposes a $b$-bit integer into hardware-friendly slices, typically 4-bit, 2-bit, and optionally 1-bit components, which are then packed into separate contiguous buffers. This allows irregular widths such as 5-bit, 6-bit, and 3-bit to be implemented using GPU-efficient layouts (Li et al., 4 Aug 2025).

For a 5-bit code $q_i \in \{0,\dots,31\}$, the decomposition is

$q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$

where the regular part is the lower nibble,

$q_i^{(4)} = q_i \bmod 2^4,$

and the extra part is the highest bit,

$$q_i^{(1)} = \left\lfloor \frac{q_i}{2^4} \right\rfloor \in \{0,1\}.$$

The 4-bit components are packed with the same nibble-aligned routines used in FlashCommunication V1, while the 1-bit components are stored in a bitmask buffer. At the receiver, the two streams are unpacked and reassembled as

$\hat{q}_i = q_i^{(4)} + 16 \cdot q_i^{(1)}.$

The same principle generalizes to arbitrary widths by expressing

$$b = k_4 \cdot 4 + k_2 \cdot 2 + r,\quad r \in \{0,1\}.$$

A 6-bit format can therefore be represented as $4+2$, and a 3-bit format as $b$0. The communication substrate remains indifferent to the logical precision: it only transmits several aligned integer buffers plus metadata. The significance of this design is not merely representational. It decouples the logical communication precision from the set of formats natively supported by the GPU, which is what makes arbitrary-bit quantization practical without new hardware.

3. Spike reserving and the low-bit quantization scheme

Bit splitting solves irregular packing, but it does not solve the collapse in accuracy that appears at 3-bit and especially 2-bit. FlashCommunication V2 addresses this with spike reserving, an outlier-aware quantization scheme that stores the groupwise minimum and maximum values as floating-point metadata and quantizes only the remaining values within a reduced dynamic range (Li et al., 4 Aug 2025).

For a group of values $b$1, the method first finds the two spikes,

$b$2

$b$3

These two values are reserved in BF16 or FP16, and their indices are stored as INT8. Quantization is then performed only on the residual set $b$4, with range

$b$5

The scale for $b$6-bit asymmetric quantization becomes

$b$7

and the residuals are quantized as

$b$8

At dequantization time, the residuals are reconstructed and the spikes are restored exactly at their stored indices. The result is a large reduction in effective range for the values that are actually quantized, which is what makes 2-bit communication feasible.

The framework uses asymmetric fine-grained RTN as its base quantizer. For a group without spike reserving, the standard form is

$b$9

$b$0

with dequantization

$b$1

Group size depends on the target precision. INT8, INT6, and INT5 use group size $b$2, whereas INT4, INT3, and INT2 use group size $b$3. The smaller group size at lower precision reduces quantization error.

Metadata overhead is also addressed explicitly. The paper introduces compressed integer scales,

$b$4

with approximate recovery

$b$5

For a 4096-element block, the paper reports that original BF16 activations occupy $b$6 bytes, while spike reserving with 2-bit quantization reduces the total to $b$7 bytes, or to $b$8 bytes with integer scales. This is $b$9 or $q_i \in \{0,\dots,31\}$0 of the original payload, respectively (Li et al., 4 Aug 2025).

4. Collective communication design and implementation

The quantization scheme is embedded into fused communication kernels. Each CUDA block processes $q_i \in \{0,\dots,31\}$1 BF16 values with $q_i \in \{0,\dots,31\}$2 threads, so each thread handles $q_i \in \{0,\dots,31\}$3 values. Within that block, values are partitioned into quantization groups, quantized, bit-split, packed, transmitted, unpacked, and dequantized. The first four warps are used for vectorized access to metadata such as scales, zeros, and spike indices, which improves coalescing and reduces divergence (Li et al., 4 Aug 2025).

For tensor parallelism, FlashCommunication V2 replaces standard BF16 ncclAllReduce with a fused AllReduce that operates on compressed buffers. For expert parallelism, it compresses the dispatch activations used in MoE All2All, following the strategy of quantizing only the dispatch volume. In both cases, the computation outside communication remains in standard precision, which preserves compatibility with established model implementations.

A major systems contribution is the distinction between NVLink and PCIe/NUMA deployment. On NVLink systems, the main objective is higher raw throughput from reduced communication volume and efficient packing. On PCIe/NUMA systems such as 8×L40, the framework adds a hierarchical AllReduce. The sequence is intra-NUMA ReduceScatter, cross-NUMA reduction of the smaller partial sums, and intra-NUMA AllGather. For a per-GPU payload of size $q_i \in \{0,\dots,31\}$4, the paper states that the cross-NUMA volume is reduced from $q_i \in \{0,\dots,31\}$5 in the two-step scheme to $q_i \in \{0,\dots,31\}$6 in the hierarchical scheme (Li et al., 4 Aug 2025).

Pipeline parallelism is then layered on top of the hierarchical design. The payload is divided into micro-chunks, and the system overlaps intra-NUMA ReduceScatter for chunk $q_i \in \{0,\dots,31\}$7, cross-NUMA reduction for chunk $q_i \in \{0,\dots,31\}$8, and intra-NUMA AllGather for chunk $q_i \in \{0,\dots,31\}$9. This reduces idle link time and yields approximately $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$0 additional time saving over serial hierarchical execution. The broader implication is that arbitrary-bit communication is not only a quantization problem; its benefit depends on how quantization is matched to interconnect topology and to the scheduling of the collective itself.

5. Empirical behavior, accuracy regimes, and performance

The reported evaluations cover dense models including Llama-3-8B, Llama-3-70B, Qwen-3-8B, and Qwen-3-32B, and MoE models including Qwen3-30B-A3B and Qwen1.5-MoE-A2.7B. Accuracy is measured with C4 perplexity and LMEval tasks such as PIQA, ARC-C, ARC-E, HellaSwag, and WinoGrande, while systems performance is measured through algorithmic bandwidth and TTFT (Li et al., 4 Aug 2025).

A consistent pattern emerges. INT8, INT6, and INT5 remain close to BF16 under naive RTN, while INT3 and INT2 degrade sharply unless spike reserving is used. For Llama-3-8B in AllReduce at INT2 with group size $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$1, RTN gives C4 perplexity $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$2, whereas SpikeReserving INT2 gives $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$3. For Llama-3-70B, the corresponding numbers are $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$4 and $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$5. On LMEval, Llama-3-8B has BF16 average $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$6, INT5 average $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$7, RTN INT2 average $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$8, and SpikeReserving INT2 average $q_i = q_i^{(4)} + 2^4 \cdot q_i^{(1)},$9. These results place INT4 and INT5 as the principal throughput–accuracy trade-off points, while INT2 becomes viable only with explicit spike handling.

The same trend holds for MoE All2All. For Qwen3-30B-A3B at INT2 and group size $q_i^{(4)} = q_i \bmod 2^4,$0, RTN gives perplexity $q_i^{(4)} = q_i \bmod 2^4,$1, while SpikeReserving gives $q_i^{(4)} = q_i \bmod 2^4,$2. For Qwen1.5-MoE-A2.7B, the same comparison is $q_i^{(4)} = q_i \bmod 2^4,$3 versus $q_i^{(4)} = q_i \bmod 2^4,$4. This shows that the outlier problem is not restricted to tensor-parallel reductions; it also affects dispatch activations in expert routing.

The headline systems results are summarized below.

Scenario	Baseline	Best reported result
AllReduce on L40	NCCL BF16: 10.43 GB/s	HierPP + INT4: 30.84 GB/s
AllReduce on H800	BF16: 94.18 GB/s	INT4: 185.27 GB/s
All2All on H800	BF16: 169.76 GB/s	INT4: 341.87 GB/s

The paper states a maximum $q_i^{(4)} = q_i \bmod 2^4,$5 speedup in AllReduce and $q_i^{(4)} = q_i \bmod 2^4,$6 in All2All. For TTFT with Llama-3-8B at tensor parallel degree $q_i^{(4)} = q_i \bmod 2^4,$7, low-bit communication reduces TTFT by $q_i^{(4)} = q_i \bmod 2^4,$8 on L40, up to $q_i^{(4)} = q_i \bmod 2^4,$9 on A100, and up to $$q_i^{(1)} = \left\lfloor \frac{q_i}{2^4} \right\rfloor \in \{0,1\}.$$0 on H800. On H20, improvements are limited because the device is bandwidth-rich but compute-weak, so the overhead of quantization and packing offsets much of the communication gain (Li et al., 4 Aug 2025).

A common misconception is that the lowest possible precision must also be the fastest or the most useful. The empirical data does not support that interpretation. INT2 can be made workable with spike reserving, but the additional metadata and processing overhead reduce its systems advantage. The paper therefore identifies INT4 and INT5 as the operational sweet spots for many workloads.

6. Relation to adjacent “Flash” research, limitations, and future directions

FlashCommunication V2 belongs to a specific line of work on communication-efficient distributed LLM execution and should not be conflated with other contemporary uses of “Flash” in machine learning systems. “FlashMotion: Few-Step Controllable Video Generation with Trajectory Guidance” studies the re-alignment of a trajectory adapter and a distilled video generator, where “communication” is effectively the interface between a control module and a DiT backbone rather than cross-GPU transmission (Li et al., 12 Mar 2026). “Flash-VL 2B: Optimizing Vision-LLM Performance for Ultra-Low Latency and High Throughput” focuses on vision–language efficiency through fixed-resolution design, pixel-shuffle compression, and Implicit Semantic Stitching (Zhang et al., 14 May 2025). “FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models” is an inference-only front-end that reduces visual token count before the LLM by fusing intrinsic saliency and extrinsic query relevance (Cai et al., 23 Dec 2025). The much earlier “Flash-based Audio and Video Communication in the Cloud” addresses browser-embedded voice and video communication via Flash Player, RTMP, RTMFP, and SIP interoperation, in a historical web multimedia context (Singh et al., 2011). This suggests that the term “FlashCommunication” has acquired several surrounding associations in the literature, but in (Li et al., 4 Aug 2025) it denotes arbitrary-bit inter-GPU communication for collective operations.

The limitations stated in the paper are equally specific. The evaluation is single-node only, even though it covers up to 8 GPUs. Multi-node RDMA, InfiniBand, and SHARP-like offload are not studied. Compression is implemented on GPU SMs, so the method consumes compute resources that are not free on bandwidth-rich devices such as H20. The paper does not provide long-run training convergence guarantees under communication-induced noise. It also does not integrate its All2All path with specialized libraries such as DeepEP. These constraints are important because they define the current applicability of the method: it is a practical intra-node communication framework, not yet a full-stack distributed training substrate.

The future directions proposed in the paper follow directly from these limits. They include deeper hardware co-design, evaluation on multi-node clusters, integration with high-end communication stacks such as InfiniBand plus SHARP, and more adaptive spike selection schemes if their metadata cost remains acceptable. Within the present scope, however, FlashCommunication V2 establishes a clear systems result: communication-only quantization can be pushed beyond conventional INT8 and INT4 formats, down to arbitrary bit widths and even INT2, provided that packing layout, outlier handling, and collective scheduling are co-designed rather than optimized in isolation.