ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers (1812.11677v1)

Abstract: To facilitate efficient embedded and hardware implementations of deep neural networks (DNNs), two important categories of DNN model compression techniques: weight pruning and weight quantization are investigated. The former leverages the redundancy in the number of weights, whereas the latter leverages the redundancy in bit representation of weights. However, there lacks a systematic framework of joint weight pruning and quantization of DNNs, thereby limiting the available model compression ratio. Moreover, the computation reduction, energy efficiency improvement, and hardware performance overhead need to be accounted for besides simply model size reduction. To address these limitations, we present ADMM-NN, the first algorithm-hardware co-optimization framework of DNNs using Alternating Direction Method of Multipliers (ADMM), a powerful technique to deal with non-convex optimization problems with possibly combinatorial constraints. The first part of ADMM-NN is a systematic, joint framework of DNN weight pruning and quantization using ADMM. It can be understood as a smart regularization technique with regularization target dynamically updated in each ADMM iteration, thereby resulting in higher performance in model compression than prior work. The second part is hardware-aware DNN optimizations to facilitate hardware-level implementations. Without accuracy loss, we can achieve 85$\times$ and 24$\times$ pruning on LeNet-5 and AlexNet models, respectively, significantly higher than prior work. The improvement becomes more significant when focusing on computation reductions. Combining weight pruning and quantization, we achieve 1,910$\times$ and 231$\times$ reductions in overall model size on these two benchmarks, when focusing on data storage. Highly promising results are also observed on other representative DNNs such as VGGNet and ResNet-50.

Overview of ADMM-NN: An Algorithm-Hardware Co-Design Framework for DNNs Using ADMM

The paper presents ADMM-NN, an advanced framework aimed at optimizing deep neural networks (DNNs) through model compression, specifically leveraging the Alternating Direction Method of Multipliers (ADMM). This comprehensive approach addresses the challenges in simultaneously minimizing storage size and enhancing computation speeds without sacrificing accuracy, crucial for embedded systems and IoT applications requiring efficient power-budgeted deployments.

Main Contributions

ADMM-NN is divided into two notable components:

1. Algorithm-Level Co-Optimization:

The primary focus involves a systematic integration of weight pruning and quantization techniques via ADMM. These are elaborated as follows: - Weight Pruning: The framework highlights the ability to discard redundant weights in DNN models, significantly reducing the model size. - Weight Quantization: By adjusting the bit representation for DNN weight storage, computational efficiency gains are realized.

Through ADMM-based methods, both weight pruning and quantization are addressed as non-convex optimization issues, effectively decomposing the problem into manageable subproblems. For instance, optimal Euclidean projections onto constrained sets in the ADMM iterations seamlessly lead to effective parameter adjustments.

1. Hardware-Level Optimization: This innovation accounts for hardware performance considerations by optimizing for computation reduction and energy efficiency, while mitigating overheads associated with irregular sparsity—known burdens in conventional pruning approaches. The concept of a break-even pruning ratio aids in determining the minimal pruning thresholds that avoid performance degradation in hardware implementations.

Numerical Results and Implications

Empirical evaluation illustrates substantial benefits derived from the ADMM-NN framework:

• Model Compression Achievements: Noteworthy compression statistics include a 1,910$\times$ reduction in LeNet-5 and a 231$\times$ reduction in AlexNet model sizes, achieved through combined pruning and quantization strategies.
• Computational Efficiency: Without incurring accuracy losses, computations are significantly scaled down by 3.6$\times$, underscoring the framework's contribution to operational cost efficiencies.

These results are remarkable across several DNN architectures such as VGGNet and ResNet-50, indicating the broad applicability and scalability of ADMM-NN.

Broader Impacts and Future Directions

The concept of ADMM-based pruning and quantization introduces a potential paradigm shift in DNN deployment, especially within resource-constrained environments. The ability to store high-redundancy models like AlexNet entirely on-chip paves the way for more efficient portable applications without compromising performance. Such advances open avenues to rethink hardware designs, accommodating increasingly sophisticated DNN models in various real-time applications.

Future work could explore extending the ADMM framework's capabilities to newer neural architectures or integrating more nuanced hardware-awareness mechanisms, thus further bridging the gap between theory and practical deployment efficacy in AI. The interplay between increased model complexity and hardware resource availability continues to offer intriguing challenges and opportunities for innovation in AI system design.