Papers
Topics
Authors
Recent
2000 character limit reached

STR-Cert: Robustness Certification for Deep Text Recognition on Deep Learning Pipelines and Vision Transformers (2401.05338v1)

Published 28 Nov 2023 in cs.CV and cs.LG

Abstract: Robustness certification, which aims to formally certify the predictions of neural networks against adversarial inputs, has become an integral part of important tool for safety-critical applications. Despite considerable progress, existing certification methods are limited to elementary architectures, such as convolutional networks, recurrent networks and recently Transformers, on benchmark datasets such as MNIST. In this paper, we focus on the robustness certification of scene text recognition (STR), which is a complex and extensively deployed image-based sequence prediction problem. We tackle three types of STR model architectures, including the standard STR pipelines and the Vision Transformer. We propose STR-Cert, the first certification method for STR models, by significantly extending the DeepPoly polyhedral verification framework via deriving novel polyhedral bounds and algorithms for key STR model components. Finally, we certify and compare STR models on six datasets, demonstrating the efficiency and scalability of robustness certification, particularly for the Vision Transformer.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Verification of RNN-based neural agent-environment systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6006–6013, 2019.
  2. Rowel Atienza. Vision transformer for fast and efficient scene text recognition. Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, 12821 LNCS:319–334, 5 2021.
  3. Verifying reinforcement learning up to infinity. Proceedings of the International Joint Conference on Artificial Intelligence, 2021.
  4. What is wrong with scene text recognition model comparisons? dataset and model analysis. Proceedings of the IEEE International Conference on Computer Vision, 2019-October:4714–4722, 4 2019.
  5. Fast and precise certification of transformers. International Conference on Programming Language Design and Implementation, 2021.
  6. CNN-Cert: An efficient framework for certifying robustness of convolutional neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33:3240–3247, 2019.
  7. Certified adversarial robustness via randomized smoothing. 36th International Conference on Machine Learning, ICML 2019, 2019-June:2323–2356, 2 2019.
  8. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021 - 9th International Conference on Learning Representations, 10 2021.
  9. A dual approach to scalable verification of deep networks. In UAI, volume 1, page 3, 2018.
  10. Scalable certified segmentation via randomized smoothing. Proceedings of the International Conference on Machine Learning, 139:3340–3351, 7 2021.
  11. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR, 2015.
  12. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ACM International Conference Proceeding Series, volume 148, pages 369–376, 2006.
  13. Synthetic data for text localisation in natural images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016-December:2315–2324, 4 2016.
  14. Synthetic data and artificial neural networks for natural scene text recognition. In Workshop on Deep Learning, NIPS, 6 2014.
  15. Spatial transformer networks. Advances in Neural Information Processing Systems, 2015-January:2017–2025, 6 2015.
  16. Exactly computing the local Lipschitz constant of ReLU networks. Conference on Neural Information Processing Systems, 2020.
  17. ICDAR 2015 competition on robust reading. Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, 2015-November:1156–1160, 11 2015.
  18. ICDAR 2013 robust reading competition. In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, 2013.
  19. Reluplex: An efficient SMT solver for verifying deep neural networks. Computer Aided Verification, 10426 LNCS:97–117, 2017.
  20. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR, 2015.
  21. POPQORN: Quantifying robustness of recurrent neural networks. In 36th International Conference on Machine Learning, ICML, volume 2019-June, pages 6031–6087, 2019.
  22. SoK: Certified robustness for deep neural networks. Proceedings - IEEE Symposium on Security and Privacy, 2023-May:1289–1310, 9 2023.
  23. Deep text classification can be fooled. In IJCAI International Joint Conference on Artificial Intelligence, 2018.
  24. Are attention networks more robust? towards exact robustness verification for attention networks. In Computer Safety, Reliability, and Security: 41st International Conference, 2 2022.
  25. STAR-Net: A spatial attention residue network for scene text recognition. The British Machine Vision Conference, 2016.
  26. Towards deep learning models resistant to adversarial attacks. 6th International Conference on Learning Representations, 6 2018.
  27. Scene text recognition using higher order language priors. The British Machine Vision Association, 2012.
  28. When adversarial training meets vision transformers: Recipes from training to architecture. Advances in Neural Information Processing Systems, 35, 10 2022.
  29. Scaling polyhedral neural network verification on gpus. Proceedings of the 4 th MLSys Conference, 7 2021.
  30. PRIMA: General and precise neural network certification via scalable convex hull approximations. Proceedings of the ACM on Programming Languages, 6, 3 2021.
  31. Sequential randomized smoothing for adversarially robust speech recognition. EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 6372–6386, 11 2021.
  32. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 12 2019.
  33. Recognizing text with perspective distortion in natural scenes. Proceedings of the IEEE International Conference on Computer Vision, pages 569–576, 2013.
  34. Semidefinite relaxations for certifying robustness to adversarial examples. Advances in Neural Information Processing Systems, 2018-December:10877–10887, 11 2018.
  35. A robust arbitrary text detection system for natural scene images. Expert Systems with Applications, 41:8027–8048, 12 2014.
  36. Scalable polyhedral verification of recurrent neural networks. Computer Aided Verification, 12759 LNCS:225–248, 2020.
  37. A convex relaxation barrier to tight robustness verification of neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  38. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. In Advances in Neural Information Processing Systems, volume 2018-Decem, pages 3835–3844. Neural information processing systems foundation, 2018.
  39. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(11):2298–2304, 2017.
  40. Robust scene text recognition with automatic rectification. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December:4168–4176, 3 2016.
  41. Formal verification for neural networks with general nonlinearities via branch-and-bound. The second Workshop on Formal Verification of Machine Learning, ICML, 2023.
  42. Robustness verification for transformers. Proceedings of the International Conference on Learning Representations, 2 2020.
  43. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages, 3(POPL):1–30, 2019.
  44. Reachability is np-complete even for the simplest neural networks. International Conference on Reachability Problems, 13035 LNCS:149–164, 8 2021.
  45. Evaluating Robustness of Neural Networks with Mixed Integer Programming. 7th International Conference on Learning Representations, ICLR 2019, 11 2017.
  46. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  47. End-to-end scene text recognition. Proceedings of the IEEE International Conference on Computer Vision, pages 1457–1464, 2011.
  48. Efficient formal safety analysis of neural networks. Advances in neural information processing systems, 31, 2018.
  49. Beta-CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete neural network nobustness verification. Conference on Neural Information Processing Systems, 3 2021.
  50. Convex bounds on the Softmax function with applications to robustness verification. International Conference on Artificial Intelligence and Statistics, 2023.
  51. Towards fast computation of certified robustness for relu networks. In International Conference on Machine Learning, pages 5276–5285. PMLR, 2018.
  52. Attention-based extraction of structured information from street view imagery. In Proceedings of the International Conference on Document Analysis and Recognition, ICDAR, 2017.
  53. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning, pages 5286–5295. PMLR, 2018.
  54. Robustness guarantees for deep neural networks on videos. Proceedings of the IEEE, 2020.
  55. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 2020-December, 2 2020.
  56. Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. 9th International Conference on Learning Representations, 11 2021.
  57. Adaptive adversarial attack on scene text recognition. Conference on Computer Communications Workshops, pages 358–363, 7 2020.
  58. Efficient neural network robustness certification with general activation functions. In Advances in Neural Information Processing Systems, volume 2018-, pages 4939–4948. Neural information processing systems foundation, 2018.
  59. Recurjac: An efficient recursive algorithm for bounding jacobian matrix of neural networks and its applications. Proceedings of the Thirty-Third AAAI Conference, 2019.
  60. Adversarial attacks on deep-learning models in natural language processing. ACM Transactions on Intelligent Systems and Technology, 11(3):1–41, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Paper to Video (Beta)

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up