ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA (1612.00694v2)

Abstract: Long Short-Term Memory (LSTM) is widely used in speech recognition. In order to achieve higher prediction accuracy, machine learning scientists have built larger and larger models. Such large model is both computation intensive and memory intensive. Deploying such bulky model results in high power consumption and leads to high total cost of ownership (TCO) of a data center. In order to speedup the prediction and make it energy efficient, we first propose a load-balance-aware pruning method that can compress the LSTM model size by 20x (10x from pruning and 2x from quantization) with negligible loss of the prediction accuracy. The pruned model is friendly for parallel processing. Next, we propose scheduler that encodes and partitions the compressed model to each PE for parallelism, and schedule the complicated LSTM data flow. Finally, we design the hardware architecture, named Efficient Speech Recognition Engine (ESE) that works directly on the compressed model. Implemented on Xilinx XCKU060 FPGA running at 200MHz, ESE has a performance of 282 GOPS working directly on the compressed LSTM network, corresponding to 2.52 TOPS on the uncompressed one, and processes a full LSTM for speech recognition with a power dissipation of 41 Watts. Evaluated on the LSTM for speech recognition benchmark, ESE is 43x and 3x faster than Core i7 5930k CPU and Pascal Titan X GPU implementations. It achieves 40x and 11.5x higher energy efficiency compared with the CPU and GPU respectively.

• The paper introduces a 20x LSTM compression method combining pruning and quantization to enable efficient FPGA implementation.
• The FPGA design attains 282 GOPS with a novel load-balance-aware scheduler, delivering a 43x speedup over conventional CPUs.
• The engine achieves 40x higher energy efficiency, demonstrating significant performance and cost benefits for scalable speech recognition.

Overview of "ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA"

The paper presents an innovative approach to accelerating speech recognition, specifically through the efficient deployment of Long Short-Term Memory (LSTM) models on FPGA hardware. The authors focus on the substantial computational and memory demands of large LSTM models, proposing compression techniques to address these challenges.

Model Compression and Algorithm Optimization

The authors introduce a method to compress the LSTM model by a factor of 20, combining pruning and quantization techniques. Load-balance-aware pruning ensures uniform distribution of non-zero weights across processing elements (PEs), which is crucial for maintaining high hardware utilization. The pruning method achieves a 10% non-zero weight density, coupled with quantization to 12-bit precision, resulting in minimal accuracy loss.

Hardware Architecture and Implementation

The Efficient Speech Recognition Engine (ESE) is designed specifically to exploit the sparsity of the pruned LSTM model. Implemented on a Xilinx XCKU060 FPGA, the architecture achieves 282 GOPS, equivalent to 2.52 TOPS on a dense model. The design includes a novel scheduler for efficiently managing the LSTM’s complex data flows and dependencies, thereby optimizing computational parallelism.

Performance Evaluation

The ESE’s performance is rigorously evaluated against traditional CPU and GPU implementations, demonstrating a speed-up of 43x over an i7 CPU and 3x over a Pascal Titan X GPU. Furthermore, energy efficiency comparisons reveal that ESE is 40x more efficient than the CPU and 11.5x more efficient than the GPU. These results are particularly significant given the hardware's 41-watt power consumption.

Implications and Future Directions

The research provides a compelling case for deploying sparse LSTM models on FPGA platforms, offering substantial improvements in both speed and energy efficiency over conventional hardware. The proposed load-balance-aware pruning method and dataflow scheduling strategies could extend to other types of neural networks beyond LSTM.

The implications suggest a move towards more energy-efficient and cost-effective speech recognition systems, with the potential to reduce the total cost of ownership for data centers. Future developments may explore extensions to other recurrent neural network architectures or further optimizations in FPGA resource utilization.

This work contributes to the growing field of AI hardware acceleration, providing practical insights and methodologies that can be adapted to a wide range of applications in machine learning and signal processing. As neural network models continue to grow in size and complexity, approaches like ESE will be crucial for maintaining scalable and sustainable AI solutions.