Bit Error Robustness for Energy-Efficient DNN Accelerators (2006.13977v3)

Abstract: Deep neural network (DNN) accelerators received considerable attention in past years due to saved energy compared to mainstream hardware. Low-voltage operation of DNN accelerators allows to further reduce energy consumption significantly, however, causes bit-level failures in the memory storing the quantized DNN weights. In this paper, we show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors in (quantized) DNN weights significantly. This leads to high energy savings from both low-voltage operation as well as low-precision quantization. Our approach generalizes across operating voltages and accelerators, as demonstrated on bit errors from profiled SRAM arrays. We also discuss why weight clipping alone is already a quite effective way to achieve robustness against bit errors. Moreover, we specifically discuss the involved trade-offs regarding accuracy, robustness and precision: Without losing more than 1% in accuracy compared to a normally trained 8-bit DNN, we can reduce energy consumption on CIFAR-10 by 20%. Higher energy savings of, e.g., 30%, are possible at the cost of 2.5% accuracy, even for 4-bit DNNs.

Bit Error Robustness for Energy-Efficient DNN Accelerators

The paper "Bit Error Robustness for Energy-Efficient DNN Accelerators" addresses the challenge of designing deep neural network (DNN) accelerators that are robust to random bit errors arising from low-voltage operations. The primary motivation for such a design is to achieve energy efficiency, which is critical for reducing the carbon footprint of DNN-driven applications and enabling their deployment in edge computing scenarios.

Key Contributions

1. Robust Quantization Techniques: The paper introduces a robust fixed-point quantization scheme that significantly enhances bit error robustness without sacrificing accuracy. The quantization strategy leverages per-layer asymmetric quantization into unsigned integers with proper rounding, as opposed to the traditional methods that employ either global or symmetric quantization.
2. Weight Clipping as Regularization: Weight clipping is utilized as a regularization technique during training to limit weights within a range of $[-w_{\text{max}}, w_{\text{max}}]$. This approach promotes redundancy in the weight distributions, which mitigates the impact of bit errors by making the network less susceptible to changes induced by these errors.
3. Random Bit Error Training (RandBET): The paper presents a training strategy that involves injecting random bit errors during the training process. This method fosters the development of DNN models whose robustness to bit errors generalizes well across different chips and varying voltages without relying on specific memory profiling or intervention at the hardware level.

Numerical Results and Trade-offs

The empirical results indicate significant energy savings through the methods proposed:

• On the CIFAR10 dataset, DNN accelerators achieved around 20% energy savings with only a 1% drop in accuracy compared to standard 8-bit networks, and up to 30% energy savings with a 2.5% accuracy loss when using 4-bit quantization.
• The robustness of these models is experimentally verified under random and profiled SRAM bit error patterns, demonstrating their generalizability.

Implications and Future Directions

This work holds substantial practical implications for the design of DNN accelerators, particularly where energy efficiency and operational reliability are paramount. The suggested training techniques provide a software-based alternative to traditional hardware-centric approaches for error correction, such as ECCs or voltage boosting, which often involve additional energy or spatial costs.

Theoretically, this paper opens avenues for further exploration into robust quantization schemes and regularization techniques that can improve the statistical fault tolerance of DNNs. Potential future developments may include extending the robustness strategies to other types of memory errors and investigating their applicability in more diverse hardware configurations and applications.

In the broader context of AI, enhancing the fault tolerance of DNN models against bit errors is crucial for deploying AI systems in environments where resource constraints and energy efficiency are critical concerns, such as portable devices and remote sensing. This not only enhances the reliability and longevity of these devices but also accelerates the adoption of AI technologies across various sectors.