PCCL: Process Group-Aware Scalable and Generic Collective Algorithm Synthesizer

Abstract: Distributed machine learning has become increasingly important due to the massive scale of large-scale generative models. Both model parameters and data are distributed across many compute devices, which requires frequent collective communications to synchronize activations and parameter updates. Such collective communications have become a major bottleneck. While the performance of the collective algorithm depends on the physical network topology, the baseline collective algorithms in collective communication libraries are largely topology-agnostic. Collective algorithm synthesizers address this inefficiency by automatically generating topology-aware collective algorithms. However, prior works have largely overlooked that collective communication typically occurs only among a subset of devices, known as process groups. Additionally, most existing synthesizers are limited in the range of target collective patterns they can generate. We propose PCCL, a scalable and generic framework for synthesizing topology-aware collective algorithms. PCCL is process group-aware and capable of generating near-optimal collective algorithms even when only a subset of devices participates in collective operations. PCCL synthesizes arbitrary collective patterns, including 512-NPU All-to-All synthesis in 11.68 minutes.

• The paper introduces a process group-aware synthesizer that leverages a BFS-based time-expanded network to rapidly generate congestion-free collective algorithms.
• It employs dynamic chunk-based scheduling to optimize time slot allocation and pathfinding, achieving up to a 3× bandwidth improvement in large-scale clusters.
• Experimental results demonstrate scalability to 1,000 NPUs and offer a robust framework for integrating adaptive, hardware-aware collective communications in distributed ML.

PCCL: A Scalable, Process Group-Aware Synthesizer for Generic Collective Algorithms

Introduction

Scaling large generative models has imposed stringent requirements on collective communication efficiency in distributed machine learning clusters. As model and data parallelism proliferate across heterogeneous, asymmetric topologies comprising hundreds to thousands of compute devices, collective communication—specifically the synchronization of activations and gradient updates—emerges as the principal system bottleneck. While state-of-the-art collective communication libraries offer some topology-aware optimizations, most existing algorithm synthesizers remain inadequate along four practical axes: scalability, generic topology support, extensibility to all collective patterns (including All-to-All), and process group awareness. "PCCL: Process Group-Aware Scalable and Generic Collective Algorithm Synthesizer" (2606.07019) introduces a synthesizer explicitly designed to meet all these requirements concurrently.

Core Contributions and Methodology

PCCL leverages a time-expanded network (TEN) representation, unifying spatial and temporal network aspects, and introduces an efficient BFS-based pathfinding mechanism for collective synthesis. The algorithm operates chunk-centrically: each collective pattern is decomposed into a set of chunk transfer conditions specifying source-destination requirements. By propagating chunk routes over the TEN and dynamically allocating time slots and paths, PCCL produces non-conflicting, congestion-aware, topology-adaptive collective algorithms. The condition-based representation and BFS search are critical in maintaining scalability and flexibility.

Process group awareness is integral to PCCL: the synthesizer can optimize collective algorithms not just across the global cluster but for arbitrary, dynamic subsets of devices (process groups), inherently supporting concurrent multi-job workloads typical in contemporary clusters. Notably, PCCL exploits idle network resources outside the process group to improve collective bandwidth, an optimization impractical with naive subgraph isolation or legacy synthesizers.

Reduction collectives (e.g., All-Reduce, Reduce-Scatter) are synthesized by leveraging the non-reduction pathfinding results and reversing dataflow directions, composing them with reduction semantics without re-deriving procedures for each pattern. Native support for switch modeling and heterogeneous link characteristics is achieved by incorporating explicit switch nodes and α–β network models into the TEN, allowing fine-grained control over switch buffers, multicast, and link bandwidth asymmetries.

Numerical Results and Technical Claims

PCCL demonstrates strong empirical results:

• Synthesis Scalability: For All-to-All collectives on a 512-NPU topology, PCCL synthesizes algorithms in 11.68 minutes—over three orders of magnitude faster than state-of-the-art optimizer-based synthesizers such as TE-CCL [liu2024teccl]. The complexity scales as $O(n^3)$ for $n$ NPUs, with robust performance demonstrated on clusters up to 1,000 NPUs.
• Bandwidth and Utilization Gains: In process group scenarios, PCCL-synthesized All-to-All algorithms achieve 2.33–3.03× bandwidth improvement (2.68× on average) over baseline pairwise algorithms currently used in CCLs for both homogeneous and heterogeneous topologies. The improvements are attributed to network-wide utilization enabled by process group-aware synthesis, as indicated by detailed link utilization heatmaps.
• Generic Collective and Topology Support: PCCL supports arbitrary topology heterogeneity, including switch-centric fabrics, and arbitrary collective patterns (Reduce-Scatter, All-Gather, All-Reduce, All-to-All, All-to-Allv), sustaining synthesis times within practical limits even as collective size increases.
• Congestion-Free Synthesis: BFS-based scheduling ensures no temporal link overlaps for simultaneous chunk transfers, with priority scheduling for longest paths to maximize parallelism and resource utilization.

These results position PCCL as the most scalable and generic collective synthesizer in published literature to date, eclipsing existing tree-based, optimizer-based, and greedy approaches [(2606.07019), won2024tacos, shah2023taccl, cai2021sccl, zhao2025forestcoll].

Implications for Distributed Learning Systems

The principal practical implication of PCCL is its capability to automatically synthesize and provision optimized collective algorithms for diverse distributed ML jobs, irrespective of network and collective complexity, without requiring manual expert intervention. In particular, efficient All-to-All communication is essential for Mixture-of-Experts and MoE-based LLMs [fedus2022switchtransformer, chowdhery2023palm], where All-to-All traffic can dominate runtime as system scale grows. By supporting dynamic process groups and leveraging otherwise idle bandwidth, PCCL enables more efficient cluster utilization, crucial in multi-job production environments.

Theoretically, PCCL's BFS over TEN abstraction unifies spatial, temporal, and collective conditions in a scalable synthesis pipeline, extending the generalizability limits of algorithmic synthesis in collective communication. The capacity to operate directly on real-world topologies and switch characteristics permits rapid adaptation to evolving hardware (e.g., optical interconnects, hierarchical switch fabrics), with immediate applicability to both research and production clusters.

By providing a chunk-centric algorithmic abstraction, PCCL also facilitates downstream integration with hardware communication DSLs (e.g., MSCCL, MSCCL++) and simulation/validation platforms, allowing synthesized collectives to be readily executed and profiled without synthesizer-specific tuning.

Future Directions

PCCL's framework opens several research directions:

• Dynamic and Adaptive Scheduling: Integration with workload-aware or online-adaptive schedulers to respond to time-varying network conditions and dynamic process group reshaping.
• End-to-End System Optimization: Joint optimization of collective algorithms with computation-communication overlap, scheduler-driven co-design with parallelism strategies, or model layer partitioning.
• Robustness to Network Failures: Synthesis of collectives under partial link/switch availability or targeted failover support for reliability in hyperscale deployments.
• Hardware Specialization: Further exploration of PCCL's compatibility with emerging hardware fabrics (e.g., photonic switching, in-network processing) and leveraging custom DSL backends for direct deployment on heterogeneous accelerator clusters.

Conclusion

PCCL provides a unified and scalable solution for collective algorithm synthesis, uniquely supporting process-group granularity and generic collectives across heterogeneous, real-world topologies. Its BFS-over-TEN methodology, robust empirical performance, and seamless extensibility establish a new baseline for automated collective algorithm synthesis in distributed deep learning systems (2606.07019). The practical and theoretical ramifications of this work underline the need for process group-aware, topology-adaptive algorithm synthesis as a foundational primitive for future AI and HPC infrastructure.