IOS: Inter-Operator Scheduler for CNN Acceleration (2011.01302v2)

Abstract: To accelerate CNN inference, existing deep learning frameworks focus on optimizing intra-operator parallelization. However, a single operator can no longer fully utilize the available parallelism given the rapid advances in high-performance hardware, resulting in a large gap between the peak performance and the real performance. This performance gap is more severe under smaller batch sizes. In this work, we extensively study the parallelism between operators and propose Inter-Operator Scheduler (IOS) to automatically schedule multiple operators' parallel execution through a novel dynamic programming algorithm. IOS consistently outperforms state-of-the-art libraries (e.g., TensorRT) by 1.1 to 1.5x on modern CNN benchmarks. The code to reproduce each experiment is available at: https://github.com/mit-han-lab/inter-operator-scheduler.

• The paper introduces IOS, a dynamic programming-based scheduler that enhances CNN performance by exploiting inter-operator parallelism.
• It employs a novel methodology combining operator merging and concurrent execution strategies to improve hardware utilization on GPUs.
• Empirical results demonstrate a 1.1 to 1.5x speedup over state-of-the-art libraries, underscoring its potential for diverse CNN architectures.

An Expert Overview of the Inter-Operator Scheduler for CNN Acceleration

The paper "IOS: Inter-Operator Scheduler for CNN Acceleration" presents a novel approach to optimizing the execution of Convolutional Neural Networks (CNNs) by leveraging both intra- and inter-operator parallelism. This work specifically addresses the inefficiencies observed in modern deep learning frameworks when running CNN inferences on advanced hardware architectures such as GPUs.

Background and Challenges

Current frameworks predominantly utilize intra-operator parallelism, focusing on parallelizing computation within a single operator. However, the increasing computational capabilities of modern hardware have outpaced these optimizations, leaving substantial resources underutilized. This problem is exacerbated by recent CNN architectures that favor multiple branches of small operators over a single monolithic operator, further diminishing the opportunities for adequate resource utilization.

Inter-Operator Scheduling Approach

The authors propose the Inter-Operator Scheduler (IOS), a dynamic programming-based method designed to maximize hardware utilization by automatically determining an optimized schedule for operator execution. IOS concurrently explores operator merging and concurrent execution strategies, directly addressing the bottleneck of the existing execution model.

Key Methodological Innovations

• Dynamic Programming Algorithm: IOS leverages dynamic programming to efficiently explore the scheduling space. It recognizes common sub-schedules across different operator sequences, reducing the computational overhead associated with exhaustive search.
• Parallelization Strategies: IOS evaluates both operator merge and concurrent execution strategies to find the optimal configuration for a given hardware and workload. Merging operators streamlines memory access patterns, while concurrent execution utilizes hardware capabilities for running operations in parallel on different CUDA streams.
• Hardware and Configuration Adaptability: The algorithm customizes schedules based on different hardware environments and inference configurations, such as batch sizes, thus providing nuanced performance improvements across varying scenarios.

Results and Implications

The paper provides compelling numerical results showing that IOS achieves between 1.1 to 1.5 times the speedup on modern CNN architectures over state-of-the-art libraries like TensorRT. The empirical evaluation was conducted across several popular CNNs, such as Inception-V3, RandWire, NasNet-A, and SqueezeNet, demonstrating consistent improvements.

Practical Implications

These findings suggest that integrating inter-operator scheduling into existing deep learning frameworks could significantly enhance performance, particularly in cloud and edge computing applications where resource efficiency is critical. The paper also highlights the potential for dynamic scheduling algorithms to become a staple augmentation for the deployment of CNNs across diverse hardware platforms.

Future Directions

The concept of inter-operator scheduling opens several avenues for future research and development. One potential area is the integration of IOS with other optimization frameworks, such as those utilizing neural architecture search, to further compound performance gains. Additionally, exploring the adaptation of similar dynamic scheduling approaches to other deep learning paradigms and models beyond CNNs could broaden the impact of these techniques.

In conclusion, the introduction of the IOS framework provides a substantial step forward in CNN acceleration, enabling a higher degree of resource utilization and flexibility in adapting to modern hardware capabilities. As deep learning models continue to evolve, such adaptive scheduling algorithms will likely play a pivotal role in bridging the gap between theoretical peak performance and practical deployment efficacy.