LongVILBench: Benchmark for Long-Horizon Imitation

• LongVILBench is a benchmark for long-horizon visual imitation learning that pairs real human demonstration videos with structured action plans to test sustained planning and error recovery.
• It systematically evaluates temporal and spatial reasoning over extended action sequences, addressing challenges like cascading mistakes and variable visual conditions.
• The dataset consists of 300 videos across 150 tasks and multiple difficulty levels, providing actionable metrics (EMA, FSA, SMS) to diagnose planning and code synthesis failures.

LongVILBench is a benchmark for long-horizon visual imitation learning introduced alongside the LongVIL agent framework in “Long-Horizon Visual Imitation Learning via Plan and Code Reflection” (Chen et al., 4 Sep 2025). It is designed to evaluate imitation from real human demonstration videos when success depends on sustained planning, temporal ordering, spatial object relations, and robustness to early errors. The benchmark targets a gap identified by the paper: many existing visual imitation learning systems perform adequately on short “atomic” tasks of roughly 1–5 steps, but degrade when demonstrations require longer action chains, object-relational reasoning, and recovery from cascading mistakes. LongVILBench therefore functions both as a dataset and as a stress test for long-horizon VIL, pairing demonstration videos with structured action and program targets (Chen et al., 4 Sep 2025).

1. Motivation and problem setting

LongVILBench is explicitly motivated by the mismatch between existing VLM-based visual imitation learning evaluations and the requirements of real-world imitation. The paper argues that most existing benchmarks and methods are built around short, relatively simple demonstrations, whereas real-world imitation often requires reasoning over long action chains with nontrivial temporal order and spatial relations (Chen et al., 4 Sep 2025). In such settings, a single early mistake can invalidate later steps, and models also tend to hallucinate actions or misread spatial dependencies. The benchmark is intended to make these failures measurable.

This design emphasis distinguishes LongVILBench from benchmarks that primarily test short-horizon recognition or low-complexity execution. The benchmark stresses temporal and spatial complexity across multiple task types, and the paper frames it as a more realistic testbed for manipulation workflows in which action dependencies accumulate over time (Chen et al., 4 Sep 2025). A plausible implication is that LongVILBench is not merely an annotation resource for video understanding; it is structured to expose failure modes in planning and execution pipelines that might remain hidden on short demonstrations.

2. Dataset composition and formal representation

The benchmark contains 300 real human demonstration videos spanning 150 tasks, with 2,332 annotated atomic actions in total, or 7.8 actions per video on average (Chen et al., 4 Sep 2025). Action sequences range from 1 to 18 steps. Tasks are organized into three difficulty levels based on sequence length: Level 1 short tasks have 1–4 actions, Level 2 medium tasks have 5–8 actions, and Level 3 long tasks have 9–18 actions (Chen et al., 4 Sep 2025).

The benchmark covers three representative manipulation categories: block manipulation, tabletop cleanup, and vegetable sorting. Across these tasks, there are 14 unique objects, four atomic operations—pick, place, open, and close—and six spatial relations: left, right, front, behind, on top of, and into (Chen et al., 4 Sep 2025).

Aspect	Specification
Demonstrations	300 real human demonstration videos
Tasks	150 tasks
Atomic actions	2,332 annotated atomic actions
Sequence length	1 to 18 steps
Difficulty levels	Level 1: 1–4, Level 2: 5–8, Level 3: 9–18
Categories	Block manipulation, tabletop cleanup, vegetable sorting
Operations	pick, place, open, close
Spatial relations	left, right, front, behind, on top of, into

Each task instance is formalized as

$$\mathcal{D} = (\mathcal{O}, \mathcal{P}, \mathcal{V}, \mathcal{A}, \boldsymbol{\Pi}),$$

where $\mathcal{O}$ is the object set, $\mathcal{P}$ the initial object positions, $\mathcal{V}$ the demonstration video frames, $\mathcal{A}$ the ordered action sequence, and $\boldsymbol{\Pi}$ the executable symbolic program (Chen et al., 4 Sep 2025). Each atomic action is represented as

$a_t = [\alpha, o, d],$

with $\alpha$ as the operation, $o$ the object, and $d$ the spatial relation (Chen et al., 4 Sep 2025).

This representation makes LongVILBench more than a video corpus. It is a paired dataset of visual demonstrations and structured execution targets, intended to evaluate not only action recognition but also planning and code synthesis. That choice is central to the benchmark’s identity, because the paper evaluates systems that transform demonstration videos into executable robot programs rather than into textual descriptions alone (Chen et al., 4 Sep 2025).

3. Collection pipeline and visual conditions

The benchmark was collected through a multi-stage pipeline intended to ensure semantic validity, physical feasibility, and visual robustness. Candidate task plans are first generated with GPT-4o using task-specific prompt templates, and are then manually checked for semantic and logical validity (Chen et al., 4 Sep 2025). Those plans are next tested in PyBullet with a UR5e robot to ensure that they are physically feasible. Only then are they enacted by human demonstrators in a real tabletop setup (Chen et al., 4 Sep 2025).

Each demonstration is recorded twice: once under a clean, controlled visual condition with fixed lighting, camera, and background, and once under a more complex condition with variable lighting and camera viewpoint (Chen et al., 4 Sep 2025). The paper emphasizes that this dual recording is meant to probe robustness not only to long sequences but also to visual variability.

This collection protocol gives LongVILBench a dual role. First, it constrains task design through physical feasibility checks in simulation. Second, it introduces controlled distribution shift through paired clean and complex recordings. The latter is especially important because the paper reports that all methods degrade under complex conditions with varying lighting and camera viewpoints, suggesting that the benchmark measures both long-horizon reasoning and robustness to visual disturbance (Chen et al., 4 Sep 2025).

4. Evaluation protocol and benchmark metrics

The evaluation setting compares prior video-to-code methods with the LongVIL framework. The main baselines are SeeDo and GPT-4VforRobots, both re-implemented with GPT-4o as the backbone for fairness (Chen et al., 4 Sep 2025). The paper also reports results for the authors’ framework using Qwen-VL-Max and GPT-4o, and compares “Base” versions with variants augmented by reflection modules (Chen et al., 4 Sep 2025). This setup isolates not only method-level differences but also the contribution of reflection.

Performance is measured using three metrics. Exact Match Accuracy (EMA) is defined as

$\mathcal{O}$0

Final State Accuracy (FSA) as

$\mathcal{O}$1

and Step-wise Matching Score (SMS) as

$\mathcal{O}$2

where $\mathcal{O}$3 is the matching prefix length between predicted and ground-truth action sequences (Chen et al., 4 Sep 2025).

The choice of prefix-based evaluation is explicit: long-horizon tasks are brittle, so once an early action fails, later actions are often no longer meaningful (Chen et al., 4 Sep 2025). This evaluation logic is closely aligned with the benchmark’s stated motivation. Rather than rewarding unordered overlap or partial end-state coincidence alone, the benchmark emphasizes early-step correctness and execution validity under sequential dependence.

5. LongVIL and the benchmark’s intended use

LongVILBench is introduced together with LongVIL, the paper’s agent framework for transforming a human demonstration video into executable robot code (Chen et al., 4 Sep 2025). The overall pipeline is

$\mathcal{O}$4

where $\mathcal{O}$5 is the input video and $\mathcal{O}$6 is the final verified program (Chen et al., 4 Sep 2025).

The first stage, $\mathcal{O}$7, extracts keyframes and generates an initial action plan. Keyframes are selected using hand motion: the authors estimate 3D hand positions $\mathcal{O}$8 with MediaPipe and compute velocity

$\mathcal{O}$9

then select local minima of the smoothed velocity curve as keyframes:

$\mathcal{P}$0

To avoid missing rapid actions, the method adds a “keyframe completion” step that inserts additional candidate frames at roughly one-third and two-thirds of distant intervals, then snaps them to nearby frames with visible hands, producing a denser set $\mathcal{P}$1 (Chen et al., 4 Sep 2025).

Given those keyframes and detected objects, a VLM generates a structured action plan

$\mathcal{P}$2

where $\mathcal{P}$3 is a natural-language action description, $\mathcal{P}$4 is the segment span, and $\mathcal{P}$5 is an explanation (Chen et al., 4 Sep 2025). The plan reflection module $\mathcal{P}$6 then checks temporal coherence and spatial alignment using segment-level temporal verification and frame-level spatial verification, invoking CorrectPlan when inconsistencies are detected. The code generation stage $\mathcal{P}$7 maps each validated action to a code snippet over seven predefined robot motion primitives, and the code reflection module $\mathcal{P}$8 verifies semantic alignment between code and plan, invoking CorrectCode when needed (Chen et al., 4 Sep 2025).

Within the article’s scope, LongVILBench is therefore inseparable from a particular evaluation philosophy: imitation is treated as a loop of planning, verification, and correction rather than as a one-shot prediction problem. This suggests that the benchmark is deliberately structured to support diagnosis of errors at both the plan and code levels, rather than only measuring final task success.

6. Empirical results, ablations, and robustness

The reported results show that LongVILBench is difficult for existing approaches. On the full benchmark, GPT4forRobots obtains 0.21 EMA/FSA and 0.5760 SMS, while SeeDo obtains 0.1367 EMA/FSA and 0.2919 SMS (Chen et al., 4 Sep 2025). The best reported configuration, Ours-Reflection-GPT4o, reaches 0.4867 EMA, 0.49 FSA, and 0.6611 SMS overall (Chen et al., 4 Sep 2025). The paper explicitly states that the overall EMA and FSA are more than double those of GPT4forRobots, and that SMS improves by about 18%.

The performance gap is especially pronounced on Level 3 long tasks. There, the best method achieves 0.25 EMA and 0.26 FSA, compared with 0.15/0.15 for GPT4forRobots and 0.00/0.00 EMA/FSA for SeeDo (Chen et al., 4 Sep 2025). The paper also reports that, within both GPT-4o and Qwen-VL-Max backbones, adding reflection consistently improves performance over the Base variant. For GPT-4o, reflection gives about 15% relative improvement in EMA and 17% in FSA (Chen et al., 4 Sep 2025).

The ablation study attributes these gains to multiple components. Starting from the Base system, adding keyframe completion improves completeness and reduces missed actions. Adding visual reflection $\mathcal{P}$9 improves spatiotemporal alignment, and adding code reflection $\mathcal{V}$0 yields the strongest final performance (Chen et al., 4 Sep 2025). The largest gains appear on the hardest tasks, reinforcing the benchmark’s role as a stress test for long-horizon imitation rather than a simple recognition benchmark.

Robustness experiments under visual disturbance show that the best method performs better under clean conditions than under complex conditions with varying lighting and camera viewpoints, but all methods degrade (Chen et al., 4 Sep 2025). This confirms that sequence length is not the only challenge captured by LongVILBench. The benchmark also measures sensitivity to visual variability, which the paper presents as evidence of realistic variability.

7. Relation to prior benchmarks and nomenclature

The paper positions LongVILBench as challenging relative to prior visual imitation and manipulation benchmarks for several reasons. Compared with Imitrob and FetchBench, it has much longer action sequences and richer relational structure; compared with RLBench, it adds explicit difficulty stratification and more systematic spatiotemporal organization; compared with SeeDo, it has more diverse spatial relations and stronger hierarchical complexity (Chen et al., 4 Sep 2025). The comparison table highlights that LongVILBench uses videos, supports 1–18 steps, includes 6 spatial relation types, 150 tasks, and three difficulty levels (Chen et al., 4 Sep 2025).

A naming clarification is important because similarly named benchmarks appear in adjacent literature. “LongBench v2” is a long-context language benchmark and is explicitly not presented as “LongVILBench” in that paper (Bai et al., 2024). Separately, “LVBench” is a benchmark for extreme long-video understanding, and one source notes that it is “also referred to in the query as LongVILBench,” but LVBench is a different benchmark centered on long-video question answering rather than visual imitation learning (Wang et al., 2024). In the context of “Long-Horizon Visual Imitation Learning via Plan and Code Reflection,” LongVILBench denotes the benchmark of 300 human demonstration videos for long-horizon VIL (Chen et al., 4 Sep 2025).

In that sense, LongVILBench occupies a distinct position among long-context and long-video benchmarks. It is neither a text long-context reasoning benchmark nor a generic long-video QA benchmark. Its defining feature is the coupling of long-horizon visual demonstrations with structured action sequences and executable programs, enabling evaluation of imitation, planning, and code synthesis in a single benchmark formulation (Chen et al., 4 Sep 2025).