Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (2404.06511v1)

Published 9 Apr 2024 in cs.CV, cs.AI, and cs.LG

Abstract: This paper addresses the task of video question answering (videoQA) via a decomposed multi-stage, modular reasoning framework. Previous modular methods have shown promise with a single planning stage ungrounded in visual content. However, through a simple and effective baseline, we find that such systems can lead to brittle behavior in practice for challenging videoQA settings. Thus, unlike traditional single-stage planning methods, we propose a multi-stage system consisting of an event parser, a grounding stage, and a final reasoning stage in conjunction with an external memory. All stages are training-free, and performed using few-shot prompting of large models, creating interpretable intermediate outputs at each stage. By decomposing the underlying planning and task complexity, our method, MoReVQA, improves over prior work on standard videoQA benchmarks (NExT-QA, iVQA, EgoSchema, ActivityNet-QA) with state-of-the-art results, and extensions to related tasks (grounded videoQA, paragraph captioning).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (82)
  1. Evaluating clip: towards characterization of broader capabilities and downstream implications. arXiv preprint arXiv:2108.02818, 2021.
  2. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  3. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39–48, 2016.
  4. ViViT: A Video Vision Transformer. In ICCV, 2021.
  5. Test of time: Instilling video-language models with a sense of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2503–2516, 2023.
  6. JAX: composable transformations of Python+NumPy programs, 2018.
  7. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023.
  8. Revisiting the “video” in video-language understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2917–2927, 2022.
  9. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  10. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794, 2022.
  11. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023a.
  12. Pali-3 vision language models: Smaller, faster, stronger. arXiv preprint arXiv:2310.09199, 2023b.
  13. Visual programming for text-to-image generation and evaluation. arXiv preprint arXiv:2305.15328, 2023.
  14. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  15. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2305.06500, 2023.
  16. Scenic: A jax library for computer vision research and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21393–21398, 2022.
  17. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  18. EVA: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
  19. Violet : End-to-end video-language transformers with masked visual-token modeling. arXiv:2111.12681, 2021.
  20. Env-qa: A video question answering benchmark for comprehensive understanding of dynamic environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1675–1685, 2021.
  21. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14773–14783, 2023.
  22. ImageBind: One embedding space to bind them all. In CVPR, 2023.
  23. Palm 2 technical report, 2023.
  24. Ego4d: Around the world in 3,000 hours of egocentric video, 2022.
  25. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11287–11297, 2021.
  26. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953–14962, 2023.
  27. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015.
  28. Avis: Autonomous visual information seeking with large language model agent. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  29. Finding “it”: Weakly-supervised, reference-aware visual grounding in instructional videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  30. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651–4664. PMLR, 2021.
  31. Reasoning with heterogeneous graph alignment for video question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11109–11116, 2020.
  32. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference on computer vision, pages 2989–2998, 2017.
  33. Analyzing modular approaches for visual question decomposition. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
  34. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706–715, 2017.
  35. Revealing single frame bias for video-and-language learning. arXiv preprint arXiv:2206.03428, 2022.
  36. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  37. Videochat: Chat-centric video understanding, 2024.
  38. Invariant grounding for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2928–2937, 2022.
  39. Towards fast adaptation of pretrained contrastive models for multi-channel video-language retrieval. arXiv preprint arXiv:2206.02082, 2023.
  40. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2023.
  41. Egoschema: A diagnostic benchmark for very long-form video language understanding. arXiv preprint arXiv:2308.09126, 2023.
  42. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV, 2019.
  43. Simple open-vocabulary object detection with vision transformers. European Conference on Computer Vision (ECCV), 2022.
  44. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019.
  45. Verbs in action: Improving verb understanding in video-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15579–15591, 2023.
  46. Gpt-4 technical report, 2024.
  47. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames, 2023.
  48. Learning transferable visual models from natural language supervision. In icml, 2021.
  49. End-to-end generative pretraining for multimodal video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17959–17968, 2022.
  50. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023.
  51. Modular visual question answering via code generation. arXiv preprint arXiv:2306.05392, 2023.
  52. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023.
  53. Gemini: A family of highly capable multimodal models, 2023.
  54. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  55. All in one: Exploring unified video-language pre-training, 2022a.
  56. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022b.
  57. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6847–6857, 2021.
  58. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022c.
  59. Language models with image descriptors are strong few-shot video-language learners. Advances in Neural Information Processing Systems, 35:8483–8497, 2022d.
  60. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS), 2021.
  61. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777–9786, 2021.
  62. Video graph transformer for video question answering. In European Conference on Computer Vision, pages 39–58. Springer, 2022.
  63. Can i trust your answer? visually grounded video question answering. arXiv preprint arXiv:2309.01327, 2023.
  64. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
  65. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979, 2022.
  66. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1686–1697, 2021.
  67. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124–141, 2022.
  68. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In CVPR, 2023a.
  69. The dawn of lmms: Preliminary explorations with gpt-4v(ision), 2023b.
  70. Hitea: Hierarchical temporal-aware video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15405–15416, 2023a.
  71. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023b.
  72. Keunwoo Peter Yu. VideoBLIP, 2023.
  73. Self-chained image-language model for video localization and question answering. In NeurIPS, 2023.
  74. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 9127–9134, 2019.
  75. Socratic models: Composing zero-shot multimodal reasoning with language. ICLR, 2023.
  76. Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276, 2021.
  77. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In EMNLP 2023 Demo, 2023.
  78. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. ICLR, 2024.
  79. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  80. Video question answering: Datasets, algorithms and challenges. arXiv preprint arXiv:2203.01225, 2022.
  81. End-to-end dense video captioning with masked transformer. In CVPR, 2018.
  82. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now