Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices (1603.07341v1)

Abstract: In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We identify the RPU device and system specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30,000X compared to state-of-the-art microprocessors while providing power efficiency of 84,000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisted of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration and analysis of multimodal sensory data flows from massive number of IoT (Internet of Things) sensors.

• The paper demonstrates that a novel RPU-based design minimizes data movement, achieving up to 30,000x faster training than current processors.
• It employs stochastic computing for weight updates, achieving competitive classification accuracy on the MNIST dataset with simplified operations.
• The study outlines an integrated on-chip system with RPU arrays, promising significant gains in power efficiency and scalability for DNNs.

Essay on "Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices"

The paper by Gokmen and Vlasov represents a detailed exploration into the utilization of resistive processing units (RPUs) to significantly enhance the efficiency of deep neural network (DNN) training. The core premise revolves around addressing the heavy computational and time constraints typical of DNN training tasks, by leveraging the locality and parallelism inherent in the backpropagation algorithm through novel nano-electronic device concepts.

The authors propose an innovative RPU architecture, which presents a paradigm shift from the prevalent digital memory systems used in contemporary hardware accelerators like GPUs and FPGAs. The approach posits that training can be accelerated by minimizing data movement between memory and processing units, which is achieved by storing and updating weights locally at the cross-point devices of a 2D crossbar array. This architecture aims to reach an unprecedented acceleration factor of 30,000 times over current microprocessors, while offering a power efficiency of 84,000 G-Ops/s/W.

Technical Contributions and Numerical Results

The paper explores several technical innovations:

1. Stochastic Computing for Weight Updates: By employing stochastic computing techniques, the weight update operation simplifies to an AND operation, which reduces computational intensity and allows for a scalable hardware system. This stochastic model demonstrates competitive classification accuracy, validated on the MNIST dataset, achieving similar classification errors to a conventional model albeit with probabilistic updates.
2. Device Specifications and Design: Through a series of stress tests, the paper outlines specific RPU device specifications allowing for practical implementation. These include parameters like incremental conductance change, storage requirements, and noise tolerances, which are fine-tuned to maintain acceptable error penalties.
3. System-Level Integration: A comprehensive integration strategy details how RPU tiles can be assembled into a system-on-chip architecture, supported by peripheral circuits, ADCs, and potential interconnects like coherent buses or networks-on-chip. The performance metrics are notable, suggesting efficiencies surpassing current GPU architectures by several orders of magnitude.

Theoretical and Practical Implications

The proposed RPU device concept reveals several implications for both AI research and hardware design:

• Theoretical Implications: The paper presents a rigorous examination of how contemporary neural algorithms can be mapped to analog and neuromorphic computing paradigms, challenging the hegemony of digital logic in DNN accelerators. The exploration into noise tolerance and non-ideal device characteristics prompts further inquiry into stochastic computing and probabilistic algorithm design.
• Practical Implications: On a practical front, this architecture offers a path towards deploying large-scale DNN systems with reduced computational overhead and power consumption, thereby democratizing access to advanced AI capabilities which are traditionally resource-intensive.

Speculation on Future Developments

This research could serve as a cornerstone for the development of neuromorphic processors capable of handling multi-trillion parameter models efficiently. As resistive switching materials evolve, and fabrication techniques improve, the practical realization of such systems in commercial hardware accelerators could revolutionize domains like real-time IoT data processing and complex pattern recognition applications without the thermal and power constraints of current systems.

In conclusion, Gokmen and Vlasov's paper provides a comprehensive framework for understanding the potential of resistive processing units in accelerating neural network training. Its contributions lay the groundwork for further research into alternative computing paradigms, suggesting that the next leap in AI hardware may well be driven by advances in material sciences and unconventional computing architectures.