Distributed Generative Inference of LLM at Internet Scales with Multi-Dimensional Communication Optimization

Abstract: Decentralized LLM inference distributes computation among heterogeneous nodes across the internet, offering a performant and cost-efficient solution, alternative to traditional centralized inference. However, the low cross-node network bandwidth makes communication the primary bottleneck. In this paper, we introduce BloomBee, an internet-scale distributed LLM inference framework. BloomBee integrates LLM-layer assignment, micro-batching and tensor offloading to optimize communication from multiple dimensions. Additionally, BloomBee formulates the coordination of these techniques as an optimization problem and solves it using dynamic programming. BloomBee also customizes lossless compression and speculative decoding according to low-bandwidth network settings to reduce communication overhead. We evaluate BloomBee across a spectrum of network environments and show that it improves service throughput by up to 1.76x. It also reduces average latency by up to 43.20% compared to state-of-the-art decentralized LLM inference systems. BloomBee is open-sourced.

• The paper demonstrates that communication overhead is the key bottleneck in distributed LLM inference, shifting focus from compute to network optimizations.
• It introduces BloomBee's joint optimization of pipeline parallelism, micro-batching, and lossless activation compression to enhance throughput and lower latency.
• Empirical results show BloomBee achieves up to 1.76× higher throughput and 43.2% reduced latency, while cutting GPU costs through effective offloading.

Distributed Generative Inference of LLMs at Internet Scales: Multi-Dimensional Communication Optimization

Introduction and Motivation

This work introduces BloomBee, a distributed LLM inference system that targets the primary bottleneck in decentralized, geo-distributed inference: communication overhead across heterogeneous, low-bandwidth internet links. While prior distributed inference approaches have prioritized compute or memory efficiency, the paper demonstrates through empirical analysis that internet-scale deployments fundamentally shift the bottleneck to communication—both in hop count and per-hop data volume. Communication dominates step latency, especially as model activations must traverse slow, variable internet paths, with activation transfer times vastly exceeding per-block GPU computation.

The design of BloomBee is motivated by the need to uphold throughput and latency guarantees for generative LLM inference, with use cases encompassing batch document processing, report generation, and other offline applications. The authors highlight the inadequacy of traditional cluster inference techniques by profiling per-layer decoding time, emphasizing disproportionate NIC-to-NIC communication costs on the internet versus data center intraconnects.

Figure 2: Per-step decoding-time breakdown shows communication overwhelmingly dominates latency outside intradata center scenarios.

System Architecture and Optimization Formulation

BloomBee orchestrates a pipeline-parallel deployment of transformer blocks across a dynamic set of heterogeneous nodes, focusing on minimizing critical-path communication. The assignment of layers (pipeline stages), GPU/CPU KV cache offloading, micro-batch pipelining, lossless activation compression, and speculative decoding are simultaneously optimized. Placement and scheduling are posed as a constrained dynamic programming problem, which enumerates all feasible layer assignments and micro-batch sizes to maximize throughput under per-node compute, memory, and network constraints.

Figure 1: BloomBee's system architecture; node inclusion, layer assignment, offloading, micro-batching, compression, and speculative decoding are jointly coordinated.

Layer assignment is optimized to avoid including slow or weak nodes, with offloading enabling deeper block consolidation within GPU memory limits. Micro-batching exposes communication-compute overlap, which is critical given CPU mediation and the absence of end-to-end GPUDirect. Throughput becomes a function of batch size, hop count, per-hop bandwidth, and micro-batch concurrency.

Figure 5: Micro-batch pipelining in BloomBee; compressed CPU-side activations enable overlapping network transfer with GPU computation for subsequent micro-batches.

Activation Compression: Data Layout and Entropy Analysis

Lossless activation compression is central to BloomBee's design, contrasting with weight-focused approaches in prior work. The authors analyze FP16 activation entropy distribution under various serialization schemes, finding that conventional byte serialization obscures underlying structure, while exponent/mantissa and most-significant-byte (MSB) separation exposes redundancy amenable to LZ/entropy coding.

BloomBee adopts a byte-split layout exploiting both exponent and MSB regularity, leading to marked entropy reduction (by 40% for the exponent stream compared to the raw FP16 stream). This enables more effective application of ZSTD's entropy coding, outperforming both ZipNN (which separates only exponent and mantissa) and generic compression approaches.

Figure 3: Comparison of FP16 activation layouts—the novel byte split yields lower-entropy streams than prior exponent/mantissa separations.

Figure 4: Entropy profile per component stream supports empirical superiority of BloomBee's separation for lossless compression of activations.

Speculative Decoding under Bandwidth Constraints

BloomBee adapts speculative decoding (SD) for distributed inference by introducing a pruning mechanism to reduce the volume of candidate tokens transferred for verification in the target model. While SD accelerates local inference by parallelizing verifications, it increases inter-node transfer volume in decentralized contexts. BloomBee employs a lightweight classifier to prune draft tokens before network transfer, preserving high token acceptance rates with significantly reduced bandwidth requirements.

Empirical Evaluation

Extensive experiments compare BloomBee to leading systems (Petals, Helix) across varied network conditions, batch sizes, and LLM model sizes (LLaMA, Falcon, Mixtral). Throughput and cost efficiency are evaluated for up to 65B-parameter models on heterogenous, geo-distributed clusters.

• Throughput: BloomBee achieves up to 1.76× higher service throughput and up to 43.2% lower latency compared to Helix and Petals in representative low-bandwidth scenarios.
• Micro-batching and Compression: In low bandwidth (20 Mbps) settings, combined micro-batching and compression improve throughput by 76.3% over baseline; gains are additive and occasionally super-additive.
• Offloading: CPU offloading reduces needed GPUs, yielding up to 1.82× improved throughput per GPU dollar in cost-constrained deployments.

  Figure 6: Aggregate performance across multiple environments and model sizes highlight BloomBee's throughput advantage as bandwidth becomes a limiting factor.

  

  Figure 7: Cost efficiency (throughput per GPU dollar) rises substantially with offloading, especially as network bandwidth degrades.

  

  Figure 11: Micro-batching delivers monotonic throughput improvements as bandwidth tightens.

  

  Figure 8: Compression amplifies the impact of micro-batching, with their combination achieving the highest throughput in severe network constraints.

  

  Figure 9: End-to-end throughput demonstrated for realistic, heterogeneous clusters across the open internet, with BloomBee robustly outperforming baselines.

Contradictory Findings and Non-Trivial Claims

One of the bold claims of the paper is the counterintuitive finding that speculative decoding, while highly effective in tightly-coupled (intra-datacenter) pipelines, can become detrimental under WAN constraints unless aggressive pruning is used: unpruned SD increases network transfer and reduces throughput. The joint optimization of micro-batching and lossless compression is shown to have synergistic benefits, sometimes yielding more throughput gain than the arithmetic sum of individual techniques.

Implications and Future Directions

The implications of this system-level co-design are immediate for practitioners seeking to deploy large LLMs using fragmented, heterogeneous global compute resources:

• Democratization: BloomBee provides an architecture that facilitates the pooling of idle and underutilized consumer or cloud GPUs for high-throughput LLM inference, obviating single-vendor dependence.
• Extensibility: The modular optimization formulations and entropy-based activation compression could be transferred to other DNN workloads (e.g., vision transformers or multi-modal models).
• Broader Network-Aware Optimization: The findings underline the criticality of network topology-and-bandwidth-aware model placement and scheduling for any future distributed AI system, both for inference and potentially training.

Future developments could include dynamically adaptive pipeline topologies, bandwidth-oblivious model partitioning based on real-time network measurements, and further advances in structure-preserving, domain-specific compression.

Conclusion

BloomBee establishes that decentralized generative LLM inference at internet scale requires a fundamentally communication-centric approach. Its integration of multi-dimensional communication optimization yields substantial throughput and efficiency improvements over prior systems, especially in bandwidth-limited, heterogeneous environments (2604.21072). The adoption of entropy-minimizing, structure-aware compression, dynamic programming-based scheduling, CPU offloading, and WAN-robust speculative decoding together represent a state-of-the-art baseline for scalable, open distributed LLM serving.