Fathom: Reference Workloads for Modern Deep Learning Methods (1608.06581v1)

Abstract: Deep learning has been popularized by its recent successes on challenging artificial intelligence problems. One of the reasons for its dominance is also an ongoing challenge: the need for immense amounts of computational power. Hardware architects have responded by proposing a wide array of promising ideas, but to date, the majority of the work has focused on specific algorithms in somewhat narrow application domains. While their specificity does not diminish these approaches, there is a clear need for more flexible solutions. We believe the first step is to examine the characteristics of cutting edge models from across the deep learning community. Consequently, we have assembled Fathom: a collection of eight archetypal deep learning workloads for study. Each of these models comes from a seminal work in the deep learning community, ranging from the familiar deep convolutional neural network of Krizhevsky et al., to the more exotic memory networks from Facebook's AI research group. Fathom has been released online, and this paper focuses on understanding the fundamental performance characteristics of each model. We use a set of application-level modeling tools built around the TensorFlow deep learning framework in order to analyze the behavior of the Fathom workloads. We present a breakdown of where time is spent, the similarities between the performance profiles of our models, an analysis of behavior in inference and training, and the effects of parallelism on scaling.

• The paper presents Fathom, a benchmark suite of eight deep learning models covering tasks in vision, language, and reinforcement learning.
• It analyzes performance using TensorFlow tools to compare training and inference times across varied neural architectures.
• The authors highlight the need for general-purpose hardware solutions that efficiently support the computational demands of modern deep learning models.

Fathom: Reference Workloads for Modern Deep Learning Methods

The paper "Fathom: Reference Workloads for Modern Deep Learning Methods" explores an essential facet of deep learning, namely the computational intensity associated with state-of-the-art algorithms. The authors, Robert Adolf, Saketh Rama, Brandon Reagen, Gu-Yeon Wei, and David Brooks from Harvard University, underscore the need for generalizable hardware solutions that surpass the status quo of algorithm-specific optimizations. This endeavor materializes in the form of Fathom—an assemblage of eight canonical deep learning models.

Overview of Fathom Workloads

The Fathom suite encapsulates seminal models that are integral to modern deep learning. These include Krizhevsky et al.'s deep convolutional network, variational autoencoders, memory networks from Facebook's AI unit, and others like seq2seq for language translation. Each model in Fathom serves a benchmark function, representing diverse neural styles, tasks, and application domains such as vision, speech recognition, and reinforcement learning. The models range from convolutional architectures like AlexNet and VGG to recurrent models used in natural language processing and memory networks indicative of novel topologies.

Key Findings in Performance Analysis

The paper explores the performance characteristics of these models using tools built around TensorFlow, breaking down time expenditures in inference and training processes. A comparative analysis is provided, highlighting performance profile similarities despite architectural differences. An important focus is placed on parallelism's role in scaling, revealing how these models leverage computational resources.

Theoretical and Practical Implications

The research has significant theoretical implications, highlighting the necessity for hardware that accommodates the diverse computational demands of deep learning models. Practically, it provides a comprehensive framework for evaluating and refining deep learning workloads, essential for architects aiming to design efficient AI systems. Importantly, it delineates the gaps in current architecture research, advocating for exploration beyond narrowly focused methods.

Future Prospects

Fathom sets a precedent for creating comprehensive testbeds that reflect the evolving landscape of deep learning. Future work might involve expanding the suite to cover emerging neural architectures and learning paradigms, fostering broader hardware-software co-design pursuits. Moreover, as AI applications continue to integrate into more complex tasks, a detailed evaluation of energy efficiency, real-time processing capabilities, and adaptability will become indispensable.

In conclusion, the Fathom project represents a methodical advance towards understanding and optimizing the computational nuances of deep learning workloads. By bridging the gap between current and potential architectures, Fathom crystalizes a path forward in creating universally applicable hardware models that cater to the broad spectrum of deep learning models.