Model Compression with Adversarial Robustness: A Unified Optimization Framework (1902.03538v3)

Abstract: Deep model compression has been extensively studied, and state-of-the-art methods can now achieve high compression ratios with minimal accuracy loss. This paper studies model compression through a different lens: could we compress models without hurting their robustness to adversarial attacks, in addition to maintaining accuracy? Previous literature suggested that the goals of robustness and compactness might sometimes contradict. We propose a novel Adversarially Trained Model Compression (ATMC) framework. ATMC constructs a unified constrained optimization formulation, where existing compression means (pruning, factorization, quantization) are all integrated into the constraints. An efficient algorithm is then developed. An extensive group of experiments are presented, demonstrating that ATMC obtains remarkably more favorable trade-off among model size, accuracy and robustness, over currently available alternatives in various settings. The codes are publicly available at: https://github.com/shupenggui/ATMC.

Model Compression with Adversarial Robustness: A Unified Optimization Framework

The paper "Model Compression with Adversarial Robustness: A Unified Optimization Framework" presents a novel approach to deep model compression that preserves adversarial robustness while maintaining accuracy. This work is foundational due to its consideration of adversarial robustness in conjunction with model compactness, two objectives that have been traditionally seen as contradictory.

Framework and Methodology

The authors introduce the Adversarially Trained Model Compression (ATMC) framework, a constrained optimization problem that integrates adversarial training with model compression techniques such as pruning, factorization, and quantization. The optimization problem is formulated as a min-max problem where the goal is to minimize the adversarial risk under compression constraints. Specifically, constraints are introduced through novel structured sparsity involving additive and multiplicative decompositions and non-uniform quantization, thus allowing flexibility in achieving model compactness while sustaining robustness.

Experimental Setup and Results

The framework is tested across multiple datasets and models, including MNIST with LeNet, CIFAR-10 and CIFAR-100 with ResNet34, and SVHN with WideResNet. The evaluation addresses robustness using PGD attacks while compression ratios vary by modifying the sparsity parameter $k$ and bit precision $b$. The proposed ATMC method consistently demonstrates superior trade-offs between test accuracy in benign conditions and adversarial robustness across compression ratios compared to several benchmark methods such as Non-Adversarial Pruning (NAP) and Adversarial Pruning (AP).

Key findings include the observation that naive compression can severely degrade adversarial robustness—a notion supported empirically by comparing ATMC against non-defensive compression strategies. Furthermore, ATMC’s incorporation of quantization allows for achieving high compression ratios while maintaining competitive accuracy and robustness levels.

Practical and Theoretical Implications

From a practical standpoint, the ATMC framework provides a pathway to deploy resource-intensive neural networks in environments characterized by limited computational resources, such as Internet-of-Things devices, without compromising security against adversarial attacks. The flexibility to experiment with different compression strategies and adversarial training techniques means that the ATMC framework can be adapted to a variety of application contexts where model size constraints are critical.

Theoretically, the paper challenges the notion that robustness and model compactness are inherently incompatible by providing empirical evidence to the contrary. By introducing structured sparsity and joint optimization, ATMC opens avenues for expanded research into optimization techniques that balance these two objectives. Subsequently, it provokes inquiries into the nature of robustness in compressed models and encourages further exploration into adversarial defense mechanisms tailored for lightweight models.

Future Directions

Looking forward, improvements in deep model compression by enhancing the optimization algorithm to accommodate broader forms of adversarial training are promising. Additionally, investigating the potential of reinforcement learning agents to dynamically adjust compression practices based on real-time adversarial environments may contribute significantly to advancing adaptive model compression and defense strategies.

Overall, the paper profoundly influences model compression approaches by factoring robustness into the process and paves the way for future work in this intersectional research area.