Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Object Attribute Matters in Visual Question Answering (2401.09442v1)

Published 20 Dec 2023 in cs.CV and cs.AI

Abstract: Visual question answering is a multimodal task that requires the joint comprehension of visual and textual information. However, integrating visual and textual semantics solely through attention layers is insufficient to comprehensively understand and align information from both modalities. Intuitively, object attributes can naturally serve as a bridge to unify them, which has been overlooked in previous research. In this paper, we propose a novel VQA approach from the perspective of utilizing object attribute, aiming to achieve better object-level visual-language alignment and multimodal scene understanding. Specifically, we design an attribute fusion module and a contrastive knowledge distillation module. The attribute fusion module constructs a multimodal graph neural network to fuse attributes and visual features through message passing. The enhanced object-level visual features contribute to solving fine-grained problem like counting-question. The better object-level visual-language alignment aids in understanding multimodal scenes, thereby improving the model's robustness. Furthermore, to augment scene understanding and the out-of-distribution performance, the contrastive knowledge distillation module introduces a series of implicit knowledge. We distill knowledge into attributes through contrastive loss, which further strengthens the representation learning of attribute features and facilitates visual-linguistic alignment. Intensive experiments on six datasets, COCO-QA, VQAv2, VQA-CPv2, VQA-CPv1, VQAvs and TDIUC, show the superiority of the proposed method.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Don’t just assume; look and answer: Overcoming priors for visual question answering. In CVPR.
  2. Compound Tokens: Channel Fusion for Vision-Language Representation Learning. arXiv preprint arXiv:2212.01447.
  3. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
  4. Neural module networks. In CVPR.
  5. Block: Bilinear superdiagonal fusion for visual question answering and visual relationship detection. In AAAI.
  6. Murel: Multimodal relational reasoning for visual question answering. In CVPR.
  7. Rubi: Reducing unimodal biases for visual question answering. NeurIPS.
  8. Counterfactual samples synthesizing for robust visual question answering. In CVPR.
  9. Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases. In EMNLP-IJCNLP.
  10. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.
  11. LRC-BERT: latent-representation contrastive knowledge distillation for natural language understanding. In AAAI.
  12. Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. In EMNLP.
  13. Multi-modal graph neural network for joint reasoning on vision and scene text. In CVPR.
  14. Transform-retrieve-generate: Natural language-centric outside-knowledge visual question answering. In CVPR.
  15. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In CVPR.
  16. Multi-modality latent interaction network for visual question answering. In ICCV.
  17. MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering. In EMNLP.
  18. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR.
  19. KAT: A Knowledge Augmented Transformer for Vision-and-Language. In NAACL.
  20. AdaVQA: Overcoming Language Priors with Adapted Margin Cosine Loss. In IJCAI.
  21. Loss re-scaling VQA: Revisiting the language prior problem from a class-imbalance view. IEEE Transactions on Image Processing.
  22. An analysis of visual question answering algorithms. In ICCV.
  23. Bilinear attention networks. NeurIPS.
  24. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
  25. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML. PMLR.
  26. Relation-aware graph attention network for visual question answering. In ICCV.
  27. Dynamic key-value memory enhanced multi-step graph reasoning for knowledge-based visual question answering. In AAAI.
  28. Visual question answering with question representation update (QRU). NeurIPS.
  29. A Multi-modal Debiasing Model with Dynamical Constraint for Robust Visual Question Answering. In Findings of ACL.
  30. Learning to contrast the counterfactual samples for robust visual question answering. In EMNLP.
  31. Densely connected attention flow for visual question answering. In IJCAI.
  32. ALSA: Adversarial Learning of Supervised Attentions for Visual Question Answering. IEEE transactions on cybernetics.
  33. Hierarchical question-image co-attention for visual question answering. NeurIPS.
  34. Co-attending free-form regions and detections with multi-modal multiplicative feature embedding for visual question answering. In AAAI.
  35. Positional attention guided transformer-like architecture for visual question answering. IEEE Transactions on Multimedia.
  36. Coarse-to-fine reasoning for visual question answering. In CVPR.
  37. Counterfactual vqa: A cause-effect look at language bias. In CVPR.
  38. MRA-Net: Improving VQA Via Multi-Modal Relation Attention Network. IEEE TPAMI.
  39. Answer Again: Improving VQA With Cascaded-Answering Model. IEEE Transactions on Knowledge and Data Engineering.
  40. Glove: Global vectors for word representation. In EMNLP.
  41. Overcoming language priors in visual question answering with adversarial regularization. NeurIPS.
  42. Exploring models and data for image question answering. NeurIPS.
  43. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS.
  44. The graph neural network model. IEEE transactions on neural networks.
  45. Taking a hint: Leveraging explanations to make vision and language models more grounded. In ICCV.
  46. Question type guided attention in visual question answering. In ECCV.
  47. Check It Again: Progressive Visual Question Answering via Visual Entailment. In ACL-IJCNLP.
  48. Compressing and Debiasing Vision-Language Pre-Trained Models for Visual Question Answering. In EMNLP.
  49. Towards Robust Visual Question Answering: Making the Most of Biased Samples via Contrastive Learning. In Findings of EMNLP.
  50. Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA. In Findings of EMNLP.
  51. Combo of Thinking and Observing for Outside-Knowledge VQA. In ACL.
  52. From pixels to objects: cubic visual attention for visual question answering. In IJCAI.
  53. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In EMNLP-IJCNLP.
  54. On the value of out-of-distribution testing: An example of goodhart’s law. NeurIPS.
  55. MIRTT: learning multimodal interaction representations from trilinear transformers for visual question answering. In Findings of EMNLP.
  56. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML.
  57. Chain of reasoning for visual question answering. NeurIPS.
  58. Object-difference attention: A simple relational attention for visual question answering. In Proceedings of the 26th ACM international conference on Multimedia.
  59. Self-critical reasoning for robust visual question answering. NeurIPS.
  60. Stacked attention networks for image question answering. In CVPR.
  61. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178.
  62. Deep modular co-attention networks for visual question answering. In CVPR.
  63. Vinvl: Revisiting visual representations in vision-language models. In CVPR.
  64. Learning to Count Objects in Natural Images for Visual Question Answering. In ICLR.
  65. Overcoming language priors with self-supervised learning for visual question answering. In IJCAI.
  66. Mucko: multi-layer cross-modal knowledge reasoning for fact-based visual question answering. In IJCAI.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Peize Li (7 papers)
  2. Qingyi Si (23 papers)
  3. Peng Fu (43 papers)
  4. Zheng Lin (104 papers)
  5. Yan Wang (733 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets