Open-Vocabulary Action Localization with Iterative Visual Prompting (2408.17422v5)

Abstract: Video action localization aims to find the timings of specific actions from a long video. Although existing learning-based approaches have been successful, they require annotating videos, which comes with a considerable labor cost. This paper proposes a training-free, open-vocabulary approach based on emerging off-the-shelf vision-LLMs (VLMs). The challenge stems from the fact that VLMs are neither designed to process long videos nor tailored for finding actions. We overcome these problems by extending an iterative visual prompting technique. Specifically, we sample video frames and create a concatenated image with frame index labels, allowing a VLM to identify the frames that most likely correspond to the start and end of the action. By iteratively narrowing the sampling window around the selected frames, the estimation gradually converges to more precise temporal boundaries. We demonstrate that this technique yields reasonable performance, achieving results comparable to state-of-the-art zero-shot action localization. These results support the use of VLMs as a practical tool for understanding videos. Sample code is available at https://microsoft.github.io/VLM-Video-Action-Localization/

• The paper introduces the T-PIVOT method, which iteratively refines sampling windows using VLMs to accurately identify action start and end frames.
• It achieves over 60% mean-over-frame accuracy on challenging datasets, showcasing effective open-vocabulary performance without model retraining.
• The approach reduces annotation efforts and provides adaptable, on-the-fly solutions for complex video understanding tasks.

Overview of Open-vocabulary Temporal Action Localization using VLMs

The paper "Open-vocabulary Temporal Action Localization using VLMs" tackles the problem of temporal action localization in videos through an innovative learning-free approach that leverages Vision-LLMs (VLMs). The method specifically addresses the challenge of identifying the start and end frames of an action in long videos without the extensive data annotation typically required by learning-based approaches.

Methodology

The approach, termed Temporal PIVOT (T-PIVOT), innovatively utilizes VLMs for temporal action localization by iteratively narrowing down the sampling time window to pinpoint actions. The process begins by sampling frames and forming a concatenated image containing index labels, which is analyzed to locate frames corresponding to the start or end of a specified action. This iterative sampling and querying of VLMs allow the precise localization of actions by continuously refining the time window around the identified frames.

The paper’s methodology extends the PIVOT framework, originally designed for iterative visual question answering, to the temporal domain, enabling the application of VLMs to a task they were not initially designed for.

Experimental Evaluation

Using OpenAI's GPT-4o as the primary VLM, the authors evaluated their method on two datasets: the third-person Breakfast Dataset and a manually annotated first-person Fine-grained Breakfast dataset. The experimental results, specifically the over-60% accuracy in mean-over-frame (MoF) measures, suggest that the T-PIVOT method produces reasonable action localization results despite not reaching the performance levels of the latest learning-based methods. However, it stands out with its ability to handle open-vocabulary tasks, thereby accommodating a broader range of action labels without retraining requirements.

The paper conducts a thorough exploration of the optimal sampling strategy, taking into account the number of frames sampled and their impact on spatial and temporal resolution. It was found that a 5x5 grid yielded favorable results on the Breakfast Dataset, while smaller grids, such as 3x3 and 4x4, performed better on the Fine-grained dataset.

Implications and Future Directions

This novel exploration highlights the burgeoning potential for VLMs in tasks beyond their conventional applications, such as natural language processing and visual question answering. The absence of model training and dataset-specific learning in T-PIVOT may substantially lower the barrier to entry for researchers and practitioners who need adaptable, on-the-fly action localization tools.

Moreover, by supporting open-vocabulary queries, this approach aligns well with dynamic applications such as robot teaching, where the sequences of complex tasks can be grounded against demonstration videos without exhaustive pre-labeling.

There remains ample scope for future work, particularly in refining visual prompting techniques and exploring alternative methods to improve performance for videos with numerous sequential actions. Advancements in VLM technology and better integration with specific application needs could lead to significant improvements in adaptive, context-aware action localization systems.

In conclusion, while the paper demonstrates an innovative application of VLMs in open-vocabulary action localization, it also sets the stage for future developments in leveraging these models further for a variety of complex video understanding tasks. The flexibility and adaptability of this approach suggest promising improvements to come in video processing and analysis domains.