Benchmarking State-of-the-Art Deep Learning Software Tools (1608.07249v7)

Abstract: Deep learning has been shown as a successful machine learning method for a variety of tasks, and its popularity results in numerous open-source deep learning software tools. Training a deep network is usually a very time-consuming process. To address the computational challenge in deep learning, many tools exploit hardware features such as multi-core CPUs and many-core GPUs to shorten the training time. However, different tools exhibit different features and running performance when training different types of deep networks on different hardware platforms, which makes it difficult for end users to select an appropriate pair of software and hardware. In this paper, we aim to make a comparative study of the state-of-the-art GPU-accelerated deep learning software tools, including Caffe, CNTK, MXNet, TensorFlow, and Torch. We first benchmark the running performance of these tools with three popular types of neural networks on two CPU platforms and three GPU platforms. We then benchmark some distributed versions on multiple GPUs. Our contribution is two-fold. First, for end users of deep learning tools, our benchmarking results can serve as a guide to selecting appropriate hardware platforms and software tools. Second, for software developers of deep learning tools, our in-depth analysis points out possible future directions to further optimize the running performance.

Benchmarking State-of-the-Art Deep Learning Software Tools

The paper provides a comprehensive comparative evaluation of several prominent deep learning software tools, namely Caffe, CNTK, MXNet, TensorFlow, and Torch, with a particular focus on their performance across various GPU-accelerated platforms. The authors establish a methodology to benchmark these tools using three widespread neural network architectures—Fully Connected Neural Networks (FCNs), Convolutional Neural Networks (CNNs), and Recurrent Neural Networks (RNNs)—on different CPU and GPU configurations.

The primary objectives are two-fold: firstly, to guide end users in selecting suitable hardware-software pairings for efficient deep learning tasks, and secondly, to inform software developers about potential areas for optimization in tool performance.

Experimental Methodology

The evaluation framework makes a distinction between performance metrics based on synthetic and real-world data. The authors benchmark on two types of multi-core CPUs and three NVIDIA GPU platforms (GTX 980, GTX 1080, and Tesla K80), exploring both single and multiple GPU scenarios. Moreover, performance measurement uses specific network configurations, like a large Fully Connected Network (FCN-S) with around 55 million parameters and the canonical CNNs, AlexNet and ResNet-50, as representatives of different networks.

Key Findings

1. CPU Performance: The scalability on many-core CPUs is limited. With 32 threads, TensorFlow generally performs better due to its efficient utilization of Eigen and SIMD operations.
2. Single GPU Performance: On single GPUs, performance varies significantly by task. For FCNs, Caffe, CNTK, and Torch are typically superior, but for large CNNs like ResNet-50, MXNet often delivers better results. CNTK shows a pronounced advantage in RNNs using LSTM units.
3. Multi-GPU Performance: The paper highlights the efficacy of GPU utilization across multiple units, with CNTK and MXNet showing substantial scaling benefits. The use of techniques like 1-bit stochastic gradient descent in CNTK dramatically reduces overhead from GPU-to-CPU data transfers.

Implications and Future Work

Numerical results demonstrate the potential for further optimizations; for example, performance enhancements in network training can hinge on reducing PCIe data transfer overhead and better CUDA API utilization. Practically, these insights could influence future architecture design, specifically in balancing computation across hardware resources.

From a theoretical standpoint, understanding these performance constraints offers insights into the optimization boundaries set by current hardware and software paradigms. Moving forward, the authors plan to integrate additional tools and extend their evaluations to include other hardware, such as AMD GPUs and Intel's Xeon Phi processors.

This work provides a robust foundational comparison of deep learning frameworks, crucial for both users seeking efficiency and developers aiming to innovate. As performance constraints evolve, such benchmarking studies will continue to elucidate paths for technology advancement in artificial intelligence.