Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Lyapunov-Stable Deep Equilibrium Models (2304.12707v3)

Published 25 Apr 2023 in cs.LG, cs.CR, and cs.CV

Abstract: Deep equilibrium (DEQ) models have emerged as a promising class of implicit layer models, which abandon traditional depth by solving for the fixed points of a single nonlinear layer. Despite their success, the stability of the fixed points for these models remains poorly understood. By considering DEQ models as nonlinear dynamic systems, we propose a robust DEQ model named LyaDEQ with guaranteed provable stability via Lyapunov theory. The crux of our method is ensuring the Lyapunov stability of the DEQ model's fixed points, which enables the proposed model to resist minor initial perturbations. To avoid poor adversarial defense due to Lyapunov-stable fixed points being located near each other, we orthogonalize the layers after the Lyapunov stability module to separate different fixed points. We evaluate LyaDEQ models under well-known adversarial attacks, and experimental results demonstrate significant improvement in robustness. Furthermore, we show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better adversarial robustness.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Input convex neural networks. In International Conference on Machine Learning, 146–155. PMLR.
  2. Anderson, D. G. 1965. Iterative procedures for nonlinear integral equations. Journal of the ACM (JACM), 12(4): 547–560.
  3. Deep Equilibrium Optical Flow Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 620–630.
  4. Deep equilibrium models. Advances in Neural Information Processing Systems, 32.
  5. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 33: 5238–5250.
  6. Neural deep equilibrium solvers. In International Conference on Learning Representations.
  7. Stabilizing equilibrium models by jacobian regularization. arXiv preprint arXiv:2106.14342.
  8. Convex optimization. Cambridge university press.
  9. Broyden, C. G. 1965. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation, 19(92): 577–593.
  10. AntisymmetricRNN: A dynamical system view on recurrent neural networks. arXiv preprint arXiv:1902.09689.
  11. Neural ordinary differential equations. In Advances in neural information processing systems, 6571–6583.
  12. Augmented neural odes. Advances in Neural Information Processing Systems, 32.
  13. Review on computational methods for Lyapunov functions. Discrete & Continuous Dynamical Systems-B, 20(8): 2291.
  14. Deep equilibrium architectures for inverse problems in imaging. IEEE Transactions on Computational Imaging, 7: 1123–1133.
  15. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  17. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32.
  18. Stable neural ode with lyapunov-stable equilibrium points for defending against adversarial attacks. Advances in Neural Information Processing Systems, 34: 14925–14937.
  19. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  20. Learning stable deep dynamics models. Advances in neural information processing systems, 32.
  21. Learning multiple layers of features from tiny images.
  22. Adversarial examples in the physical world. In Artificial intelligence safety and security, 99–112. Chapman and Hall/CRC.
  23. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11): 2278–2324.
  24. Cerdeq: Certifiable deep equilibrium model. In International Conference on Machine Learning, 12998–13013. PMLR.
  25. Optimization inspired Multi-Branch Equilibrium Models. In International Conference on Learning Representations.
  26. Black-box attacks against log anomaly detection with adversarial examples. Information Sciences, 619: 249–262.
  27. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
  28. Stable neural flows. arXiv preprint arXiv:2003.08063.
  29. Reading digits in natural images with unsupervised feature learning.
  30. Automatic differentiation in pytorch.
  31. The lyapunov neural network: Adaptive stability certification for safe learning of dynamical systems. In Conference on Robot Learning, 466–476. PMLR.
  32. LyaNet: A Lyapunov framework for training neural ODEs. In International Conference on Machine Learning, 18687–18703. PMLR.
  33. Learning Stable Deep Dynamics Models for Partially Observed or Delayed Dynamical Systems. Advances in Neural Information Processing Systems, 34: 11870–11882.
  34. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  35. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199.
  36. Learning dynamics models with stable invariant sets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 9782–9790.
  37. Deep equilibrium models as estimators for continuous latent variables. In International Conference on Artificial Intelligence and Statistics, 1646–1671. PMLR.
  38. Declarative nets that are equilibrium models. In International Conference on Learning Representations.
  39. Certified robustness for deep equilibrium models via interval bound propagation. In International Conference on Learning Representations.
  40. Monotone operator equilibrium networks. Advances in neural information processing systems, 33: 10718–10728.
  41. Improving Adversarial Robustness of Deep Equilibrium Models with Explicit Regulations Along the Neural Dynamics.
  42. A Closer Look at the Adversarial Robustness of Deep Equilibrium Models. Advances in Neural Information Processing Systems, 35: 10448–10461.
  43. Wide residual networks. arXiv preprint arXiv:1605.07146.
  44. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, 7472–7482. PMLR.
  45. Improving the invisibility of adversarial examples with perceptually adaptive perturbation. Information Sciences.
  46. Deep Equilibrium Models for Snapshot Compressive Imaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 3642–3650.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now