---
title: 'LongVILBench: Benchmark for Long-Horizon Imitation'
url: https://www.emergentmind.com/topics/longvilbench
type: topic
---

# LongVILBench: Benchmark for Long-Horizon Imitation

LongVILBench is a benchmark for long-horizon visual imitation learning introduced alongside the LongVIL agent framework in “Long-Horizon Visual Imitation Learning via Plan and Code Reflection” [2509.05368]. It is designed to evaluate imitation from real human demonstration videos when success depends on sustained planning, temporal ordering, spatial object relations, and robustness to early errors. The benchmark targets a gap identified by the paper: many existing visual imitation learning systems perform adequately on short “atomic” tasks of roughly 1–5 steps, but degrade when demonstrations require longer action chains, object-relational reasoning, and recovery from cascading mistakes. LongVILBench therefore functions both as a dataset and as a stress test for long-horizon VIL, pairing demonstration videos with structured action and program targets [2509.05368].

## 1. Motivation and problem setting

LongVILBench is explicitly motivated by the mismatch between existing VLM-based visual imitation learning evaluations and the requirements of real-world imitation. The paper argues that most existing benchmarks and methods are built around short, relatively simple demonstrations, whereas real-world imitation often requires reasoning over long action chains with nontrivial temporal order and spatial relations [2509.05368]. In such settings, a single early mistake can invalidate later steps, and models also tend to hallucinate actions or misread spatial dependencies. The benchmark is intended to make these failures measurable.

This design emphasis distinguishes LongVILBench from benchmarks that primarily test short-horizon recognition or low-complexity execution. The benchmark stresses temporal and spatial complexity across multiple task types, and the paper frames it as a more realistic testbed for manipulation workflows in which action dependencies accumulate over time [2509.05368]. A plausible implication is that LongVILBench is not merely an annotation resource for video understanding; it is structured to expose failure modes in planning and execution pipelines that might remain hidden on short demonstrations.

## 2. Dataset composition and formal representation

The benchmark contains 300 real human demonstration videos spanning 150 tasks, with 2,332 annotated atomic actions in total, or 7.8 actions per video on average [2509.05368]. Action sequences range from 1 to 18 steps. Tasks are organized into three difficulty levels based on sequence length: Level 1 short tasks have 1–4 actions, Level 2 medium tasks have 5–8 actions, and Level 3 long tasks have 9–18 actions [2509.05368].

The benchmark covers three representative manipulation categories: block manipulation, tabletop cleanup, and vegetable sorting. Across these tasks, there are 14 unique objects, four atomic operations—pick, place, open, and close—and six spatial relations: left, right, front, behind, on top of, and into [2509.05368].

| Aspect | Specification |
|---|---|
| Demonstrations | 300 real human demonstration videos |
| Tasks | 150 tasks |
| Atomic actions | 2,332 annotated atomic actions |
| Sequence length | 1 to 18 steps |
| Difficulty levels | Level 1: 1–4, Level 2: 5–8, Level 3: 9–18 |
| Categories | Block manipulation, tabletop cleanup, vegetable sorting |
| Operations | pick, place, open, close |
| Spatial relations | left, right, front, behind, on top of, into |

Each task instance is formalized as
$$
\mathcal{D} = (\mathcal{O}, \mathcal{P}, \mathcal{V}, \mathcal{A}, \boldsymbol{\Pi}),
$$
where \(\mathcal{O}\) is the object set, \(\mathcal{P}\) the initial object positions, \(\mathcal{V}\) the demonstration video frames, \(\mathcal{A}\) the ordered action sequence, and \(\boldsymbol{\Pi}\) the executable symbolic program [2509.05368]. Each atomic action is represented as
$$
a_t = [\alpha, o, d],
$$
with \(\alpha\) as the operation, \(o\) the object, and \(d\) the spatial relation [2509.05368].

This representation makes LongVILBench more than a video corpus. It is a paired dataset of visual demonstrations and structured execution targets, intended to evaluate not only action recognition but also planning and code synthesis. That choice is central to the benchmark’s identity, because the paper evaluates systems that transform demonstration videos into executable robot programs rather than into textual descriptions alone [2509.05368].

## 3. Collection pipeline and visual conditions

The benchmark was collected through a multi-stage pipeline intended to ensure semantic validity, physical feasibility, and visual robustness. Candidate task plans are first generated with GPT-4o using task-specific prompt templates, and are then manually checked for semantic and logical validity [2509.05368]. Those plans are next tested in PyBullet with a UR5e robot to ensure that they are physically feasible. Only then are they enacted by human demonstrators in a real tabletop setup [2509.05368].

Each demonstration is recorded twice: once under a clean, controlled visual condition with fixed lighting, camera, and background, and once under a more complex condition with variable lighting and camera viewpoint [2509.05368]. The paper emphasizes that this dual recording is meant to probe robustness not only to long sequences but also to visual variability.

This collection protocol gives LongVILBench a dual role. First, it constrains task design through physical feasibility checks in simulation. Second, it introduces controlled distribution shift through paired clean and complex recordings. The latter is especially important because the paper reports that all methods degrade under complex conditions with varying lighting and camera viewpoints, suggesting that the benchmark measures both long-horizon reasoning and robustness to visual disturbance [2509.05368].

## 4. Evaluation protocol and benchmark metrics

The evaluation setting compares prior video-to-code methods with the LongVIL framework. The main baselines are SeeDo and GPT-4VforRobots, both re-implemented with GPT-4o as the backbone for fairness [2509.05368]. The paper also reports results for the authors’ framework using Qwen-VL-Max and GPT-4o, and compares “Base” versions with variants augmented by reflection modules [2509.05368]. This setup isolates not only method-level differences but also the contribution of reflection.

Performance is measured using three metrics. Exact Match Accuracy (EMA) is defined as
$$
\text{EMA} = \frac{1}{N}\sum_{i=1}^N \mathbb{I}[\hat{\mathcal{A}_i} = \mathcal{A}_i],
$$
Final State Accuracy (FSA) as
$$
\text{FSA} = \frac{1}{N}\sum_{i=1}^N \mathbb{I}[\hat{\mathcal{S}_i} = \mathcal{S}_i],
$$
and Step-wise Matching Score (SMS) as
$$
\text{SMS} = \frac{1}{N}\sum_{i=1}^N \frac{l_i}{T_i},
$$
where \(l_i\) is the matching prefix length between predicted and ground-truth action sequences [2509.05368].

The choice of prefix-based evaluation is explicit: long-horizon tasks are brittle, so once an early action fails, later actions are often no longer meaningful [2509.05368]. This evaluation logic is closely aligned with the benchmark’s stated motivation. Rather than rewarding unordered overlap or partial end-state coincidence alone, the benchmark emphasizes early-step correctness and execution validity under sequential dependence.

## 5. LongVIL and the benchmark’s intended use

LongVILBench is introduced together with LongVIL, the paper’s agent framework for transforming a human demonstration video into executable robot code [2509.05368]. The overall pipeline is
$$
\Pi = \mathcal{R}_{\rm code}\big(\mathcal{G}_{\rm code}(\mathcal{R}_{\rm plan}(\mathcal{G}_{\rm plan}(V)))\big),
$$
where \(V\) is the input video and \(\Pi\) is the final verified program [2509.05368].

The first stage, \(\mathcal{G}_{\rm plan}\), extracts keyframes and generates an initial action plan. Keyframes are selected using hand motion: the authors estimate 3D hand positions \(h_t\) with MediaPipe and compute velocity
$$
v_t = \|h_{t+1} - h_t\|_2,
$$
then select local minima of the smoothed velocity curve as keyframes:
$$
\mathcal{K} = \{f_t \mid v_t < v_{t-1},\ v_t < v_{t+1}\}.
$$
To avoid missing rapid actions, the method adds a “keyframe completion” step that inserts additional candidate frames at roughly one-third and two-thirds of distant intervals, then snaps them to nearby frames with visible hands, producing a denser set \(\mathcal{K}^*\) [2509.05368].

Given those keyframes and detected objects, a VLM generates a structured action plan
$$
\mathcal{A} = [\alpha_1,\dots,\alpha_T], \qquad \alpha_i = \langle a_i, S_i, e_i \rangle,
$$
where \(a_i\) is a natural-language action description, \(S_i\) is the segment span, and \(e_i\) is an explanation [2509.05368]. The plan reflection module \(\mathcal{R}_{\rm plan}\) then checks temporal coherence and spatial alignment using segment-level temporal verification and frame-level spatial verification, invoking CorrectPlan when inconsistencies are detected. The code generation stage \(\mathcal{G}_{\rm code}\) maps each validated action to a code snippet over seven predefined robot motion primitives, and the code reflection module \(\mathcal{R}_{\rm code}\) verifies semantic alignment between code and plan, invoking CorrectCode when needed [2509.05368].

Within the article’s scope, LongVILBench is therefore inseparable from a particular evaluation philosophy: imitation is treated as a loop of planning, verification, and correction rather than as a one-shot prediction problem. This suggests that the benchmark is deliberately structured to support diagnosis of errors at both the plan and code levels, rather than only measuring final task success.

## 6. Empirical results, ablations, and robustness

The reported results show that LongVILBench is difficult for existing approaches. On the full benchmark, GPT4forRobots obtains 0.21 EMA/FSA and 0.5760 SMS, while SeeDo obtains 0.1367 EMA/FSA and 0.2919 SMS [2509.05368]. The best reported configuration, Ours-Reflection-GPT4o, reaches 0.4867 EMA, 0.49 FSA, and 0.6611 SMS overall [2509.05368]. The paper explicitly states that the overall EMA and FSA are more than double those of GPT4forRobots, and that SMS improves by about 18%.

The performance gap is especially pronounced on Level 3 long tasks. There, the best method achieves 0.25 EMA and 0.26 FSA, compared with 0.15/0.15 for GPT4forRobots and 0.00/0.00 EMA/FSA for SeeDo [2509.05368]. The paper also reports that, within both GPT-4o and Qwen-VL-Max backbones, adding reflection consistently improves performance over the Base variant. For GPT-4o, reflection gives about 15% relative improvement in EMA and 17% in FSA [2509.05368].

The ablation study attributes these gains to multiple components. Starting from the Base system, adding keyframe completion improves completeness and reduces missed actions. Adding visual reflection \((\mathcal{R}_v)\) improves spatiotemporal alignment, and adding code reflection \((\mathcal{R}_c)\) yields the strongest final performance [2509.05368]. The largest gains appear on the hardest tasks, reinforcing the benchmark’s role as a stress test for long-horizon imitation rather than a simple recognition benchmark.

Robustness experiments under visual disturbance show that the best method performs better under clean conditions than under complex conditions with varying lighting and camera viewpoints, but all methods degrade [2509.05368]. This confirms that sequence length is not the only challenge captured by LongVILBench. The benchmark also measures sensitivity to visual variability, which the paper presents as evidence of realistic variability.

## 7. Relation to prior benchmarks and nomenclature

The paper positions LongVILBench as challenging relative to prior visual imitation and manipulation benchmarks for several reasons. Compared with Imitrob and FetchBench, it has much longer action sequences and richer relational structure; compared with RLBench, it adds explicit difficulty stratification and more systematic spatiotemporal organization; compared with SeeDo, it has more diverse spatial relations and stronger hierarchical complexity [2509.05368]. The comparison table highlights that LongVILBench uses videos, supports 1–18 steps, includes 6 spatial relation types, 150 tasks, and three difficulty levels [2509.05368].

A naming clarification is important because similarly named benchmarks appear in adjacent literature. “LongBench v2” is a long-context language benchmark and is explicitly not presented as “LongVILBench” in that paper [2412.15204]. Separately, “LVBench” is a benchmark for extreme long-video understanding, and one source notes that it is “also referred to in the query as LongVILBench,” but LVBench is a different benchmark centered on long-video question answering rather than visual imitation learning [2406.08035]. In the context of “Long-Horizon Visual Imitation Learning via Plan and Code Reflection,” LongVILBench denotes the benchmark of 300 human demonstration videos for long-horizon VIL [2509.05368].

In that sense, LongVILBench occupies a distinct position among long-context and long-video benchmarks. It is neither a text long-context reasoning benchmark nor a generic long-video QA benchmark. Its defining feature is the coupling of long-horizon visual demonstrations with structured action sequences and executable programs, enabling evaluation of imitation, planning, and code synthesis in a single benchmark formulation [2509.05368].

Source: https://www.emergentmind.com/topics/longvilbench