Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Vamos: Versatile Action Models for Video Understanding (2311.13627v3)

Published 22 Nov 2023 in cs.CV and cs.AI

Abstract: What makes good representations for video understanding, such as anticipating future activities, or answering video-conditioned questions? While earlier approaches focus on end-to-end learning directly from video pixels, we propose to revisit text-based representations, such as general-purpose video captions, which are interpretable and can be directly consumed by LLMs. Intuitively, different video understanding tasks may require representations that are complementary and at different granularity. To this end, we propose versatile action models (Vamos), a learning framework powered by a LLM as the ``reasoner'', and can flexibly leverage visual embedding and free-form text descriptions as its input. To interpret the important text evidence for question answering, we generalize the concept bottleneck model to work with tokens and nonlinear models, which uses hard attention to select a small subset of tokens from the free-form text as inputs to the LLM reasoner. We evaluate Vamos on five complementary benchmarks, Ego4D, NeXT-QA, IntentQA, Spacewalk-18, and EgoSchema, on its capability to model temporal dynamics, encode visual history, and perform reasoning. Surprisingly, we observe that text-based representations consistently achieve competitive performance on all benchmarks, and that visual embeddings provide marginal or no performance improvement, demonstrating the effectiveness of text-based video representation in the LLM era. We also demonstrate that our token bottleneck model is able to select relevant evidence from free-form text, support test-time intervention, and achieves nearly 5 times inference speedup while keeping a competitive question answering performance. Code and models are publicly released at https://brown-palm.github.io/Vamos/

PDF HTML Abstract

Versatile Action Models for Video Understanding: A Text-Based Representation Approach

Introduction to Versatile Action Models (Vamos)

The quest for enhanced video understanding capabilities has led to the conceptualization of Versatile Action Models (Vamos). This framework diverges from traditional methodologies that rely heavily on visual embeddings, by reintroducing the concept of text-based representations. Through integrating discrete action labels and free-form text descriptions with LLMs, Vamos presents a novel pathway to action modeling. Essentially, it leverages the interpretability and flexibility of textual information, assessing its effectiveness across various tasks like activity anticipation and video question answering.

Theoretical and Practical Implications

Vamos operates on the hypothesis that different video understanding tasks could benefit from representations of varying granularity and form. The model caters to this need by accommodating visual embeddings, action labels, and textual descriptions within a unified framework. This multifaceted approach posits several implications:

  1. Textual Representation's Competitiveness: Across benchmarks, text-based representations not only held their ground but demonstrated superior or competitive performance to visual embeddings. This finding raises questions about the relative efficiency and utility of directly interpretable representations in harnessing LLMs for video understanding.
  2. Marginal Utility of Visual Embeddings: The incremental benefit provided by visual embeddings was documented as marginal. This observation could potentially shift the focus of future research towards optimizing text-based video representations, exploring their limits and capabilities.
  3. Interpretability and Intervention Capabilities: The readability of text-based representations provides an added advantage of interpretability. Vamos showcases the capability to intervene and correct representations, emphasizing the model's flexibility and adaptability.

Future Directions in AI Research

The insights drawn from Vamos open multiple avenues for further exploration:

  • Optimizing Text-Based Representations: The effectiveness of text descriptions prompts an investigation into refining these representations. Future work could explore the granularity of descriptions, the optimal combination of action labels and free-text, and methods to enhance their descriptive accuracy.
  • LLMs as Reasoners for Complex Tasks: Vamos demonstrates the prowess of LLMs in understanding and processing complex video data through text. This capability could be extended to more nuanced tasks, examining the outer limits of text-based reasoning in video understanding.
  • Visual and Textual Fusion Models: Despite the highlighted efficiency of text-based representations, integrating visual information could enrich model understanding. Exploring innovative methods to blend these modalities without compromising the benefits of interpretability and flexibility warrants investigation.
  • Efficiency in Representation: The paper also touches upon the compression of text descriptions and the selective emphasis on crucial tokens, indicating potential for efficiency improvements. Future research could delve into methods for optimizing input representation without loss of essential information, directly contributing to computational efficiency.

Concluding Thoughts

Vamos presents a compelling case for re-evaluating text-based representations in video understanding tasks. By demonstrating the competitiveness of free-form text descriptions and leveraging the reasoning capabilities of LLMs, it sets a foundational step towards a new direction in video understanding research. The blend of interpretability, flexibility, and performance underscores the potential of text-based approaches, inviting a renaissance in how we model and interpret complex visual data. As we stand on this threshold, the future of video understanding seems poised to embrace the versatility and depth offered by textual representations, heralding a new era in generative AI research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. Library of actions: Implementing a generic robot execution framework by using manipulation action semantics. The International Journal of Robotics Research, 2019.
  2. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
  3. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305.13673, 2023.
  4. When can transformers reason with abstract symbols? arXiv preprint arXiv:2310.09753, 2023.
  5. Revisiting the “video” in video-language understanding. In CVPR, 2022.
  6. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
  7. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
  8. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292, 2023a.
  9. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In CVPR, 2022.
  10. Atm: Action temporality modeling for video question answering. In ACM Multimedia, 2023b.
  11. Uniter: Universal image-text representation learning. In ECCV, 2020.
  12. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018.
  13. Language models show human-like content effects on reasoning. arXiv preprint arXiv:2207.07051, 2022.
  14. Attention over learned object embeddings enables complex visual reasoning. In NeurIPS, 2021.
  15. Learning temporal dynamics from cycles in narrated video. In ICCV, 2021.
  16. Slowfast networks for video recognition. In ICCV, 2019.
  17. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint arXiv:2111.12681, 2021.
  18. An empirical study of end-to-end video-language transformers with masked visual modeling. In CVPR, 2023.
  19. Cloob: Modern hopfield networks with infoloob outperform clip. In NeurIPS, 2022.
  20. Mist: Multi-modal iterative spatial-temporal transformer for long-form video question answering. In CVPR, 2023.
  21. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.
  22. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
  23. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018.
  24. Visual programming: Compositional visual reasoning without training. In CVPR, 2023.
  25. Video-based event recognition: activity representation and probabilistic recognition methods. Computer Vision and Image Understanding, 2004.
  26. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  27. Avis: Autonomous visual information seeking with large language models. arXiv preprint arXiv:2306.08129, 2023.
  28. Palm: Predicting actions through language models@ ego4d long-term action anticipation challenge 2023. arXiv preprint arXiv:2306.16545, 2023.
  29. Technical report for ego4d long term action anticipation challenge 2023. arXiv preprint arXiv:2307.01467, 2023.
  30. Recognition of visual activities and interactions by stochastic parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000.
  31. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.
  32. Time-agnostic prediction: Predicting predictable video frames. arXiv preprint arXiv:1808.07784, 2018.
  33. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
  34. Reasoning with heterogeneous graph alignment for video question answering. In AAAI, 2020.
  35. High-level event recognition in unconstrained videos. International journal of multimedia information retrieval, 2013.
  36. Action-gpt: Leveraging large-scale language models for improved and generalized zero shot action generation. arXiv preprint arXiv:2211.15603, 2022.
  37. Event detection in crowded videos. In ICCV, 2007.
  38. Large language models are temporal and causal reasoners for video question answering. In EMNLP, 2023.
  39. Concept bottleneck models. In ICML, 2020.
  40. A hybrid discriminative/generative approach for modeling human activities. In IJCAI, 2005.
  41. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
  42. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023b.
  43. Intentqa: Context-aware video intent reasoning. In ICCV, 2023c.
  44. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.
  45. Ai choreographer: Music conditioned 3d dance generation with aist++. In ICCV, 2021.
  46. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020.
  47. Egocentric video-language pretraining. In NeurIPS, 2022.
  48. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023.
  49. Forecasting human-object interaction: joint prediction of motor attention and actions in first person video. In ECCV, 2020.
  50. Egoschema: A diagnostic benchmark for very long-form video language understanding. arXiv preprint arXiv:2308.09126, 2023.
  51. Unsupervised learning of object structure and dynamics from videos. In NeurIPS, 2019.
  52. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721, 2023.
  53. Anymal: An efficient and scalable any-modality augmented language model. arXiv preprint arXiv:2309.16058, 2023.
  54. An ontology for video event representation. In CVPR Workshop, 2004.
  55. Training language models to follow instructions with human feedback, 2022. URL https://arxiv. org/abs/2203.02155, 13, 2022.
  56. The minimalist grammar of action. Philosophical Transactions of the Royal Society B: Biological Sciences, 2012.
  57. Parsing video events with goal inference and intent prediction. In ICCV, 2011.
  58. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021.
  59. Action bank: A high-level representation of activity in video. In CVPR, 2012.
  60. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  61. Flava: A foundational language and vision alignment model. In CVPR, 2022.
  62. Videobert: A joint model for video and language representation learning. In ICCV, 2019.
  63. Vipergpt: Visual inference via python execution for reasoning. In ICCV, 2023.
  64. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  65. Generating videos with scene dynamics. In NeurIPS, 2016.
  66. All in one: Exploring unified video-language pre-training. In CVPR, 2023.
  67. Actions~ transformations. In CVPR, 2016.
  68. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022.
  69. De-diffusion makes text a strong cross-modal interface. arXiv preprint arXiv:2311.00618, 2023.
  70. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
  71. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, 2021.
  72. Video as conditional graph hierarchy for multi-granular question answering. In AAAI, 2022a.
  73. Video graph transformer for video question answering. In ECCV, 2022b.
  74. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021.
  75. Zero-shot video question answering via frozen bidirectional language models. In NeurIPS, 2022.
  76. Hitea: Hierarchical temporal-aware video-language pre-training. In ICCV, 2023a.
  77. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023b.
  78. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
  79. Self-chained image-language model for video localization and question answering. arXiv preprint arXiv:2305.06988, 2023.
  80. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480, 2022.
  81. Merlot: Multimodal neural script knowledge models. In NeurIPS, 2021.
  82. Merlot reserve: Neural script knowledge through vision and language and sound. In CVPR, 2022.
  83. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
  84. Antgpt: Can large language models help long-term action anticipation from videos? arXiv preprint arXiv:2307.16368, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Shijie Wang (62 papers)
  2. Qi Zhao (181 papers)
  3. Minh Quan Do (2 papers)
  4. Nakul Agarwal (16 papers)
  5. Kwonjoon Lee (23 papers)
  6. Chen Sun (187 papers)
Citations (12)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. Vamos: Versatile Action Models for Video Understanding