Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering (2403.09288v1)
Abstract: Scene-Text Visual Question Answering (ST-VQA) aims to understand scene text in images and answer questions related to the text content. Most existing methods heavily rely on the accuracy of Optical Character Recognition (OCR) systems, and aggressive fine-tuning based on limited spatial location information and erroneous OCR text information often leads to inevitable overfitting. In this paper, we propose a multimodal adversarial training architecture with spatial awareness capabilities. Specifically, we introduce an Adversarial OCR Enhancement (AOE) module, which leverages adversarial training in the embedding space of OCR modality to enhance fault-tolerant representation of OCR texts, thereby reducing noise caused by OCR errors. Simultaneously, We add a Spatial-Aware Self-Attention (SASA) mechanism to help the model better capture the spatial relationships among OCR tokens. Various experiments demonstrate that our method achieves significant performance improvements on both the ST-VQA and TextVQA datasets and provides a novel paradigm for multimodal adversarial training.
- “Vqa: Visual question answering,” in ICCV, 2015, pp. 2425–2433.
- “Towards vqa models that can read,” in CVPR, 2019, pp. 8317–8326.
- “Tap: Text-aware pre-training for text-vqa and text-caption,” in CVPR, 2021, pp. 8751–8761.
- “Layoutlmv2: Multi-modal pre-training for visually-rich document understanding,” arXiv preprint arXiv:2012.14740, 2020.
- “Latr: Layout-aware transformer for scene-text vqa,” in CVPR, 2022, pp. 16548–16558.
- “Attention is all you need,” NeurIPS, vol. 30, 2017.
- “Two-stage multimodality fusion for high-performance text-based visual question answering,” in ACCV, 2022, pp. 4143–4159.
- “From token to word: Ocr token evolution via contrastive learning and semantic matching for text-vqa,” in ACM MM, 2022, pp. 4564–4572.
- “Beyond ocr+ vqa: Involving ocr into the flow for robust and accurate textvqa,” in ACM MM, 2021, pp. 376–385.
- “What is wrong with scene text recognition model comparisons? dataset and model analysis,” in ICCV, 2019, pp. 4715–4723.
- “Large-scale adversarial training for vision-and-language representation learning,” NeurIPS, vol. 33, pp. 6616–6628, 2020.
- “Localize, group, and select: Boosting text-vqa by scene text modeling,” in ICCV, 2021, pp. 2631–2639.
- “Charbert: character-aware pre-trained language model,” arXiv preprint arXiv:2011.01513, 2020.
- “Iterative answer prediction with pointer-augmented multimodal transformers for textvqa,” in CVPR, 2020, pp. 9992–10002.
- “Spatially aware multimodal transformers for textvqa,” in ECCV. Springer, 2020, pp. 715–732.
- “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017.
- Zhixuan Shen (9 papers)
- Haonan Luo (2 papers)
- Sijia Li (33 papers)
- Tianrui Li (84 papers)