ViSTa Dataset: Do vision-language models understand sequential tasks? (2411.13211v2)

Abstract: Using vision-LLMs (VLMs) as reward models in reinforcement learning holds promise for reducing costs and improving safety. So far, VLM reward models have only been used for goal-oriented tasks, where the agent must reach a particular final outcome. We explore VLMs' potential to supervise tasks that cannot be scored by the final state alone. To this end, we introduce ViSTa, a dataset for evaluating Vision-based understanding of Sequential Tasks. ViSTa comprises over 4,000 videos with step-by-step descriptions in virtual home, Minecraft, and real-world environments. Its novel hierarchical structure -- basic single-step tasks composed into more and more complex sequential tasks -- allows a fine-grained understanding of how well VLMs can judge tasks with varying complexity. To illustrate this, we use ViSTa to evaluate state-of-the-art VLMs, including CLIP, ViCLIP, and GPT-4o. We find that, while they are all good at object recognition, they fail to understand sequential tasks, with only GPT-4o achieving non-trivial performance.

• The paper introduces the ViSTa dataset to assess vision-language models' ability to interpret multi-step sequential tasks.
• It employs over 4,000 videos from diverse virtual and real-world settings to test model performance on tasks of increasing complexity.
• Results indicate that while VLMs excel at static object recognition, they struggle with comprehending complex action sequences.

Evaluation of Vision-LLMs on Sequential Task Understanding

The paper under review presents an investigation into the capabilities of vision-LLMs (VLMs) when applied to supervise sequential tasks within reinforcement learning frameworks. By introducing the ViSTa dataset, the authors aim to extend the role of VLMs beyond simple goal-oriented task assessment to more intricate evaluations that require understanding task sequences. ViSTa encompasses a hierarchical dataset featuring over 4,000 videos recorded across diverse environments, including virtual home scenarios, Minecraft, and real-world settings. The dataset serves as a thorough measure of VLMs' aptitude for judging tasks that progress through various complexities in sequential contexts.

Objectives and Dataset Composition

The ViSTa dataset has been crafted to test the performance of state-of-the-art VLMs, such as CLIP, ViCLIP, and GPT-4o, across different task domains. The unique hierarchical structure of the dataset integrates basic, single-step tasks which are incrementally composed into more complex, multi-step tasks. This setup poses a challenge: can VLMs move beyond just recognizing objects to accurately gauging the order and execution of sequences?

Task sequences in ViSTa are categorized into different levels of complexity, with tasks being divided into levels 1 (single-action tasks) to levels 2 through 8 (multiple-action tasks). Single-action tasks challenge the model's understanding of basic actions, while multiple-action tasks require the understanding of action sequences and dynamics. Furthermore, the dataset is designed to test specific capabilities such as reaction to object properties, order of actions, and effective handling of complex scenarios in unfamiliar or simulated environments.

Methodology and Evaluation

ViSTa provides a testbed not only for VLM evaluation but also for refining the models to transition from single-outcome assessments to understanding entire action trajectories. Each video in ViSTa is paired with stepwise descriptions, and models are tasked with matching the video to a set of possible task descriptions, thereby evaluating the model’s comprehension of tasks and order of actions.

Three VLMs were evaluated: CLIP, which uses static image inputs, ViCLIP with native video support albeit limited frames, and the high-performing GPT-4o. The evaluation methodology involved contrasting video scene understanding against textual descriptions, and contained provisions for systematically examining video and text embeddings, thereby gaining a nuanced perspective into model comprehension capabilities.

Findings and Interpretation

The results reveal significant shortcomings in current VLM implementations concerning their ability to aptly supervise multi-step tasks. While all tested VLMs excelled in tasks requiring simple object recognition, their performance dramatically declines as tasks require understanding sequential actions or more intricate object properties. Among the tested models, GPT-4o exhibited superior comprehension abilities; yet, its performance also dropped considerably with increased task complexity level, highlighting fundamental challenges in sequence understanding.

The evaluation demonstrated that model performance tends to deteriorate notably in simulated and unfamiliar environments, such as virtual home scenarios, compared to real-world tasks. This, coupled with an observed reliance on static object recognition without deeper temporal comprehension, stresses the level of task representational ability yet to be achieved by current VLMs.

Implications and Future Directions

The implications of these findings are profound for the continued integration of VLMs into more sophisticated AI systems, specifically within RL frameworks that necessitate multi-step task comprehension over simplified goal recognition. The ViSTa dataset opens opportunities for further research into model architecture improvements aimed at enhancing sequential understanding. This suggests potential pathways in leveraging fused vision-language data to enhance temporal reasoning capabilities.

In conclusion, the development of ViSTa and its outcomes pressure the future of VLM design toward more refined models able to understand and execute complex tasks over a trajectory, calling for further interdisciplinary research bridging reinforced AI systems with nuanced vision-language integrations.