Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions (1806.09228v1)

Abstract: The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on mobile devices. To this end, we proposed a simple yet effective scheme for compressing convolutions though applying k-means clustering on the weights, compression is achieved through weight-sharing, by only recording $K$ cluster centers and weight assignment indexes. We then introduced a novel spectrally relaxed $k$-means regularization, which tends to make hard assignments of convolutional layer weights to $K$ learned cluster centers during re-training. We additionally propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We finally evaluated Deep $k$-Means across several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss. The code is available at https://github.com/Sandbox3aster/Deep-K-Means

• The paper introduces a novel spectrally-relaxed k-means regularization that promotes hard cluster assignments for effective CNN weight compression.
• It employs weight sharing via cluster centroids, maintaining or improving accuracy while reducing computation and energy consumption.
• The method scales across models such as Wide ResNet and GoogleNet, offering a practical solution for on-device deployment and energy efficiency.

Deep k-Means: Re-Training and Parameter Sharing for Compressing Deep Convolutions

Deep convolutional neural networks (CNNs) have epitomized a paradigm shift in machine learning by excelling in various recognition tasks. However, their extensive computational requirements underscore the need for efficient compression techniques to deploy them on resource-constrained devices. The work "Deep k-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions" addresses this precise challenge by introducing an innovative approach that leverages clustering for compression of CNNs.

Overview of the Approach

The paper introduces a two-step compression process named Deep k-Means aimed primarily at convolutional layers. The first step involves the application of a novel spectrally-relaxed k-means regularization during the re-training of CNNs, which encourages weights to form tight clusters efficiently. The second step encompasses weight sharing, accomplished through recording the centroids of these clusters and using them to represent the weights. This method focuses on regularizing weights to favor k-means clustering directly, yielding hard cluster assignments that align naturally with parameter sharing.

Key Contributions

1. Spectrally Relaxed k-means Regularization: The proposed regularization term stitches the clustering process into re-training, seamlessly promoting a clustered weight structure. This regularization bypasses the complexity introduced by Bayesian methods, providing a computationally light alternative that is adaptable to large-scale models.
2. Energy-Aware Metrics for Evaluation: Alongside compression ratio metrics, the authors propose a set of energy-aware metrics that correlate strongly with real-world energy estimates. This is critical as reducing the model size or number of operations does not inherently translate to energy savings.
3. Scalability and Simplicity: The method's minimal complexity and minimal parameter tuning demand allow it to scale effectively to models such as Wide ResNet and GoogleNet, optimizing them without imposing a significant accuracy cost.

Results and Evaluations

Evaluated across various state-of-the-art models, Deep k-Means demonstrates promising outcomes, maintaining or even enhancing the accuracy post-compression. On Wide ResNet, it achieves comparable or better compression ratios and accuracy than previous methods, reflecting its robustness in handling modern CNN architectures. When assessed with the proposed metrics, the compressed models display favorable energy efficiency—a significant consideration for mobile and embedded devices.

Deep k-Means showcases an efficacy that stems from its alignment with hardware considerations. The paper verifies that its energy estimation metrics are consistent with those extrapolated from hardware-measured tools, substantiating its practical relevance. The results reveal the potential gains in both the weight representational cost and computational expenditure, underscoring its comprehensive advantage.

Implications and Future Directions

The research is an important step toward bridging algorithmic and practical trade-offs in deploying CNNs. It not only provides a method with tangible performance and energy benefits but also posits a framework for future developments in CNN compression. Moving forward, an emphasis on adaptive cluster rates could enhance the granularity and efficiency of the compression process. Additionally, incorporating more energy-specific regularizations to further minimize energy consumption remains an open avenue of potential.

The implications are manifold, promising more sustainable and broad deployments of CNNs in edge computing scenarios. As the demand for on-device intelligence continues to grow, methods like Deep k-Means that prioritize computational and power efficiency will become instrumental in advancing practical applications of machine learning.