GC3: An Optimizing Compiler for GPU Collective Communication (2201.11840v3)

Abstract: Machine learning models made up of millions or billions of parameters are trained and served on large multi-GPU systems. As models grow in size and execute on more GPUs, the collective communications used in these applications become a bottleneck. Custom collective algorithms optimized for both particular network topologies and application specific communication patterns can alleviate this bottleneck and help these applications scale. However, correctly and efficiently implementing custom algorithms is challenging. This paper introduces GC3, a system for programmable GPU communication. GC3 provides a domain specific language for writing collective communication algorithms and an optimizing compiler for lowering them to an executable form, which can be executed efficiently and flexibly in an interpreter based runtime. We used GC3 to write novel collective algorithms for AllReduce and AllToAll that are up to $1.9\times$ and $1.3\times$ faster than hand-optimized implementations, respectively.

• The paper introduces GC3, an innovative system that uses a domain-specific language and compiler to optimize GPU collective communication algorithms.
• The paper demonstrates up to 1.9x improvement in AllReduce and 1.3x in AllToAll compared to traditional NCCL implementations.
• The paper leverages advanced optimizations like chunk parallelization and instruction fusion, achieving up to 14.5x speedup in hierarchical AllReduce on GPU clusters.

Overview of GC3: An Optimizing Compiler for GPU Collective Communication

The paper presents GC3, a sophisticated system for programmable GPU communication, targeting the challenges associated with large-scale machine learning models leveraging multi-GPU environments. As model sizes escalate, the communication overhead during training and inference becomes a significant bottleneck, undermining overall system efficiency. Addressing this, the authors introduce GC3, which integrates a domain-specific language (DSL) and an optimizing compiler to facilitate the efficient execution of custom communication algorithms.

Key Contributions

1. GC3 Design: GC3's architecture comprises a DSL for specifying collective communication algorithms, a compiler for translating these high-level descriptions into efficient executables, and a runtime compatible with NCCL, which is widely used in ML workloads. This design ensures a high degree of flexibility and performance optimization without requiring extensive manual coding.
2. Algorithmic Flexibility: GC3 allows researchers to specify custom algorithms, diverging from the fixed algorithms provided by vendor libraries like NCCL. By utilizing GC3's DSL, developers can write new algorithms, such as AllReduce and AllToAll, in a manner that significantly reduces communication overhead, achieving speeds up to 1.9x and 1.3x faster than hand-optimized counterparts, respectively.
3. Optimization Techniques: The system supports various optimizations, including chunk parallelization, instruction fusion, and sophisticated scheduling strategies. These optimizations ameliorate resource utilization, enhance link saturation, and reduce execution time, crucial for complex ML models spanning large GPU clusters.
4. Evaluation and Results: GC3's efficacy is demonstrated through evaluations on A100 and V100 GPU clusters. The system's implementations match or surpass existing solutions for both standard and custom collectives. Specifically, GC3 achieves notable speed-ups for hierarchical AllReduce and showcases its capability to efficiently execute novel collectives like AllToNext, with improvements reaching 14.5x in certain scenarios.

Implications and Future Directions

The introduction of GC3 expands the horizon for optimizing collective communications in GPU clusters. By providing a framework to explore novel algorithms with reduced overhead and increased performance, GC3 empowers researchers to tackle specific network topologies and application needs. This adaptability is critical as ML workloads become increasingly complex and distributed.

For future developments, GC3's model could inspire more dynamic integration of computation and communication, potentially exploring overlapping these processes to further alleviate bottlenecks. Additionally, extending GC3 to accommodate a broader range of interconnection technologies and improving the runtime's adaptability to varying workloads could enhance its applicability across diverse computational environments.

In summary, GC3 stands as a valuable tool in the quest to optimize distributed machine learning processes, with its careful blend of flexibility, efficiency, and user-friendliness positioning it as a catalytic force in the continued evolution of GPU-based computation frameworks.