Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Beyond Image-Text Matching: Verb Understanding in Multimodal Transformers Using Guided Masking (2401.16575v1)

Published 29 Jan 2024 in cs.CL and cs.CV

Abstract: The dominant probing approaches rely on the zero-shot performance of image-text matching tasks to gain a finer-grained understanding of the representations learned by recent multimodal image-language transformer models. The evaluation is carried out on carefully curated datasets focusing on counting, relations, attributes, and others. This work introduces an alternative probing strategy called guided masking. The proposed approach ablates different modalities using masking and assesses the model's ability to predict the masked word with high accuracy. We focus on studying multimodal models that consider regions of interest (ROI) features obtained by object detectors as input tokens. We probe the understanding of verbs using guided masking on ViLBERT, LXMERT, UNITER, and VisualBERT and show that these models can predict the correct verb with high accuracy. This contrasts with previous conclusions drawn from image-text matching probing techniques that frequently fail in situations requiring verb understanding. The code for all experiments will be publicly available https://github.com/ivana-13/guided_masking.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21406–21415.
  2. Vl-match: Enhancing vision-language pretraining with token-level and instance-level matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2584–2593.
  3. Multimodal pretraining unmasked: A meta-analysis and a unified framework of vision-and-language berts. Transactions of the Association for Computational Linguistics, 9:978–994.
  4. Measuring progress in fine-grained vision-and-language understanding. arXiv preprint arXiv:2305.07558.
  5. Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In European Conference on Computer Vision, pages 565–580. Springer.
  6. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397–406.
  7. gscorecam: What objects is clip looking at? In Proceedings of the Asian Conference on Computer Vision, pages 1959–1975.
  8. Uniter: Learning universal image-text representations.
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  10. Vision-and-language or vision-for-language? on cross-modal influence in multimodal transformers. arXiv preprint arXiv:2109.04448.
  11. Saurabh Gupta and Jitendra Malik. 2015. Visual semantic role labeling. arXiv preprint arXiv:1505.04474.
  12. Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb understanding. arXiv preprint arXiv:2106.09141.
  13. Incorporating structured representations into pretrained vision & language models using scene graphs. arXiv preprint arXiv:2305.06343.
  14. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR.
  15. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705.
  16. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
  17. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014, pages 740–755. Springer.
  18. Visual spatial reasoning. arXiv preprint arXiv:2205.00363.
  19. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32.
  20. Seeing past words: Testing the cross-modal capabilities of pretrained v&l models on counting tasks. arXiv preprint arXiv:2012.12352.
  21. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR.
  22. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
  23. Are vision-language transformers learning multimodal representations? a probing perspective. In AAAI 2022.
  24. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565.
  25. Foil it! find one mismatch between image and language caption. arXiv preprint arXiv:1705.01359.
  26. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638–15650.
  27. Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490.
  28. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248.
  29. Trankit: A light-weight transformer-based toolkit for multilingual natural language processing. arXiv preprint arXiv:2101.03289.
  30. Attention is all you need. Advances in neural information processing systems, 30.
  31. Omnivl: One foundation model for image-language and video-language tasks. Advances in neural information processing systems, 35:5696–5710.
  32. Improving visual grounding by encouraging consistent gradient-based explanations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19165–19174.
  33. What you see is what you read? improving text-image alignment evaluation. arXiv preprint arXiv:2305.10400.
  34. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv e-prints, pages arXiv–2210.
  35. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Ivana Beňová (3 papers)
  2. Michal Gregor (11 papers)
  3. Martin Tamajka (4 papers)
  4. Marcel Veselý (1 paper)
  5. Marián Šimko (10 papers)
  6. Jana Košecká (4 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets