Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks (2505.08022v1)

Abstract: Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing clean accuracy. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94% compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks

The paper "Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks" presents a method for compressing deep neural networks while preserving their robustness against adversarial attacks. It introduces a novel training regime that combines dynamical low-rank compression alongside a spectral regularizer designed to maintain the condition number of the low-rank factor matrices dynamically during training. This approach aims to balance model compression and adversarial robustness, two objectives often seen as conflicting.

Methodology

1. Compression via Low-Rank Factorization: The authors leverage a low-rank matrix factorization technique to compress neural network weights. Each layer's weight matrix is expressed in a low-rank form as $W = USV^T$, where $U$ and $V$ are orthonormal matrices, and $S$ is the rank-reducing coefficient matrix. This approach reduces memory and computational demand without requiring the storage of full-rank weight matrices.
2. Adversarial Robustness with Spectral Regularization: To address the susceptibilities of compressed networks to adversarial perturbations, the authors introduce a spectral regularizer that controls the condition number $\kappa(S)$ of the coefficient matrix $S$. The regularizer aims to make $S$ well-conditioned by penalizing deviations from an ideal singular value distribution, thus enhancing robustness against adversarial inputs.
3. Rank-Adaptivity: The proposed methodology supports automatic rank adaptivity, where the rank $(r)$ of each layer's factorization can increase or decrease based on model requirements during training, reducing manual hyperparameter tuning and optimizing resource allocation.
4. Empirical Validation: Extensive experiments validate the method across diverse standard architectures and datasets. The results demonstrate that regularized networks achieve over 94% compression while retaining clean and adversarial accuracy comparable to uncompressed models.

Numerical Results

The paper claims strong numerical results, with regularized networks achieving significant robustness against adversarial attacks and maintaining competitive accuracy. For instance, the proposed method allowed for a VGG16 network to be compressed by up to 95% while outperforming adversarial accuracy compared to a baseline network under $\ell^2$-FGSM attacks. Similar improvements were observed in alternative architectures, such as VGG11 and ViT-32b, ensuring the method's applicability across standard model families.

Implications and Future Directions

The practical implications of this research are profound for deploying deep learning models on devices with constraints—such as UAVs or mobile platforms—where computational efficiency and model reliability are critical. By effectively compressing models without sacrificing adversarial robustness, the authors provide a pathway for scalable AI applications in resource-limited environments.

On the theoretical side, this work expands on the understanding of low-rank neural networks, intertwining tensor decomposition theories with machine learning robustness strategies. It lays a foundation for future exploration into dynamic adaptations of model architectures, potentially influencing how networks self-optimize during different phases of learning and deployment.

Future research might focus on refining rank adaptivity protocols, examining the interaction of spectral regularization with various adversarial defense mechanisms, or extending this framework to incorporate federated learning and data privacy aspirations. Additionally, exploring the application of the spectral regularizer in contexts beyond adversarial robustness—such as model explainability or fairness—could yield valuable insights into deep learning's broader impacts.

In summary, the paper offers a robust, adaptable framework for neural network compression complemented with adversarial defense, making it a strong candidate for facilitating efficient AI in practical and theoretical spheres.