Prime Collective Communications Library -- Technical Report (2505.14065v1)

Abstract: This report presents the Prime Collective Communications Library (PCCL), a novel fault-tolerant collective communication library designed for distributed ML workloads over the public internet. PCCL introduces a new programming model that enables dynamic peer joining and failure recovery. The library implements efficient collective operations like all-reduce while providing robust fault tolerance mechanisms that allow the system to continue operating even when peers fail or join during ongoing operations. We demonstrate that PCCL's design enables practical solutions to dynamic membership challenges in workloads with repeated operations and deterministic state advancement. Our implementation passes extensive stress tests across all major operating systems, showing reliable operation even under rapid peer churn and concurrent collective operations. By dispatching to multiple connections, we can efficiently utilize cross-continental long-fat-pipe TCP WAN links, in our experiments achieving up to 45 Gbit/s of bandwidth utilization across Europe and 25 Gbit/s across North America and Europe. PCCL's architecture enables easy implementation of distributed low-communication optimization strategies like DiLoCo, which significantly reduce communication frequency. Combined with quantization, this leads to a significant reduction in the bandwidth required for distributed training workloads. PCCL also allows for concurrent collective operations, which enables optimization strategies like async DiLoCo, which can completely hide communication overhead by implementing one-step delayed parameter updates. PCCL can facilitate exact bit-parity of the shared state across peers in all cases induced by graceful or abrupt peer churn. While PCCL exposes a C99 API, Python bindings are available which are compatible with PyTorch alongside FSDP. PCCL is available under the open source MIT license.

Overview of the Prime Collective Communications Library

The paper presents the Prime Collective Communications Library (PCCL), a novel fault-tolerant library designed for facilitating collective communication in distributed machine learning tasks, particularly over the public internet. Unlike traditional libraries such as MPI or NCCL, which are tailored for high-performance computing clusters with consistent network characteristics and static node configurations, PCCL addresses the challenges that arise in more dynamic environments marked by fluctuating network conditions and the potential for nodes to join or leave unexpectedly.

PCCL introduces a new programming model allowing for dynamic peer membership and robust failure recovery, while still supporting efficient collective operations such as all-reduce. This functionality is facilitated through a master-client architecture and a state machine that allows for ultrafast micro-consensus validation among peers, ensuring the bit-level determinism critical for maintaining consistent model synchronization across nodes, even during peer churn.

Key Features and Results

PCCL's architecture strategically employs a master-client model that simplifies dynamic membership and ensures fault tolerance through micro-consensus steps during collective operations. This design allows for topology optimizations utilizing bandwidth metrics and solves essential operational logistics via a custom asymmetric traveling-salesman problem solver. This strategy resulted in significant improvements in bandwidth utilization, achieving up to 45 Gbit/s across Europe and 25 Gbit/s between North America and Europe.

The library supports multiple concurrent collective operations across distributed frameworks, facilitating optimization strategies like DiLoCo and its asynchronous variant, significantly decreasing communication frequency and hiding communication overhead. PCCL provides Python bindings compatible with PyTorch, allowing easy integration into existing machine learning pipelines.

Numerous benchmarks have been conducted, highlighting the improved performance of PCCL over traditional libraries. The research details a 4.15% improvement in reduce time across North America and Europe compared to Gloo and substantial bandwidth utilization improvements through increased parallel connection pooling.

Practical and Theoretical Implications

The implications of PCCL intersect multiple dimensions of distributed machine learning research and practice. Practically, this library provides a robust solution for leveraging unstable resources, such as spot instances, for cost-effective training without risking data consistency or necessitating prolonged recovery times. PCCL also presents potential for democratizing AI training resources across variably subscribed cloud infrastructures, effectively utilizing otherwise idle compute power even across different regions.

Theoretically, by maintaining a bit-parity among peers through intelligent state synchronization and network topology optimizations, PCCL tackles critical issues in distributed computing, such as ensuring consistent convergence amid network variations and peer fluctuations. This contributes to the broader discourse on scalable collective communication strategies that go beyond the limitations imposed by synchronized start and static architecture configurations in current HPC-focused designs.

Future Developments

Looking towards future developments, PCCL can serve as a foundation for adaptive communication strategies tightly integrated with real-time workload telemetry. This could optimize inter-peer communication further based on dynamic network conditions and compute loads, paving the way for more sophisticated distributed learning methodologies.

In closing, the PCCL framework represents a significant step forward in addressing modern requirements of distributed machine learning, proposing a resilient and efficient model for training large-scale AI systems across heterogeneous and distributed environments.