Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools (1903.11314v2)

Abstract: Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.

• The paper systematically reviews parallelization techniques—data, model, and pipeline parallelism—to scale deep learning across distributed systems.
• It presents optimization strategies like gradient quantization and asynchronous updates to mitigate communication overhead and ensure efficient training.
• The survey highlights resource scheduling and data management practices that are crucial for maintaining consistency and scalability in distributed deep learning.

Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools

The paper, "Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools," presents a comprehensive survey of the methods and issues associated with scaling deep learning (DL) over distributed systems. The authors, Ruben Mayer and Hans-Arno Jacobsen, systematically dissect the multi-faceted problem of scaling DL, covering foundational concepts, current practices, and potential future directions.

Core Insights and Techniques

Deep learning has gained prominence due to its proficiency across domains such as image recognition and natural language processing. This success is partly attributed to advancements in model scale and the availability of vast datasets. However, scaling DL efficiently on distributed infrastructures poses significant challenges involving computation, communication, and data management.

1. Distributed Infrastructure and Parallelization:
   • The paper categorizes parallelization methods into data, model, and pipeline parallelism, each with unique benefits and limitations. Data parallelism involves distributing data across multiple workers, each maintaining a copy of the model to independently compute updates. Conversely, model parallelism partitions the model itself across different workers to handle cases where a single worker's memory is insufficient for the entire model.
   • Pipeline parallelism combines aspects of both data and model parallelism to balance workloads and manage resource utilization systematically.
2. Optimizations for Data Parallelism:
   • The synchronization of model parameters is a critical concern. Approaches vary from synchronous updates, which offer simplicity but can cause delays due to stragglers, to asynchronous updates that provide more flexibility at the expense of added complexity in convergence guarantees.
3. Communication Management:
   • Techniques to optimize communication, such as gradient quantization and sparsification, are emphasized to mitigate the bandwidth-intensive nature of model updates. For instance, training with reduced precision can decrease the communication load without substantially affecting model performance.
4. Resource Scheduling:
   • The survey highlights scheduling challenges at both single-tenant and multi-tenant levels. Single-tenant scheduling involves efficient use of resources allocated to a specific job, while multi-tenant scheduling must balance fairness and resource allocation across multiple GPUs or nodes in a shared environment.
5. Data Management:
   • Effective management of training data and model data is critical. This involves maintaining consistency and efficiency in the handling and storage of large datasets and model checkpoints. The emergence of federated learning and on-device training introduces new paradigms for data and model management without sacrificing privacy.

Frameworks and Practical Considerations

Mayer and Jacobsen review and compare open-source DL frameworks such as TensorFlow, PyTorch, and MXNet, with particular attention to how they implement the methods and techniques discussed. Each framework offers varying levels of support for distributed training, differentiated by the extensibility of APIs, ease of customization, and community activity.

Future Directions

The authors identify future research trends, emphasizing the importance of:

• Enhancing data management to cope with ever-growing datasets.
• Developing tools and frameworks adapted to decentralized architectures such as those used in federated learning.
• Addressing the identified challenges in scheduling and parallelism to better leverage heterogeneous hardware environments.

In conclusion, this survey serves as a critical resource for researchers and practitioners in DL, providing an articulate synthesis of the state-of-the-art methodologies and a roadmap for addressing the intricate challenges faced when scaling DL models over distributed infrastructures. The paper underscores the significance of interdisciplinary collaboration to drive both theoretical and practical advancements in scalable DL systems.