FineVLA-Data: Fine-Grained VLA Dataset

• FineVLA-Data is a human-verified dataset for fine-grained supervision of vision-language-action models, unifying 47K trajectories across 10 robotic datasets.
• It employs a rigorous pipeline that standardizes, filters, and clusters trajectories using dynamic time warping to ensure action–state consistency and execution diversity.
• The dataset offers step-level annotations across ten dimensions, enhancing robot video understanding and steerable policy training with detailed execution parameters.

Searching arXiv for the FineVLA paper and closely related VLA/robot-dataset references to ground the article. FineVLA-Data is a human-verified dataset for fine-grained supervision of Vision-Language-Action (VLA) models, introduced within the FineVLA framework to address a specific deficiency of existing robot corpora: trajectories are often paired only with coarse goal-level language, leaving execution-critical variables such as active arm, approach direction, contact region, and object-specific disambiguation unspecified. In FineVLA, 972,247 trajectories spanning approximately 85K tasks from 10 open-source robot datasets are consolidated into a unified format, filtered for action–state consistency, clustered for execution diversity, and reduced to 47,159 annotated trajectories comprising 220,606 step-level annotations across ten fine-grained dimensions. These annotations are used both as a standalone resource, FineVLA-Data, and as supervision for a robotics-specialized VLM annotator and steerable VLA policy training (Hu et al., 26 May 2026).

1. Definition and scope

FineVLA-Data is the curated data component of the FineVLA framework, which is described as an open framework for action-aligned fine-grained VLA supervision. Its central objective is to supply instruction signals that specify not only what goal should be achieved, but how the robot should execute the behavior. The motivating observation is that coarse goal-level language is insufficient for steerable policy learning and robotic video understanding when execution depends on latent variables such as which arm should act, from which direction the manipulator should approach, or which region of an object should be contacted (Hu et al., 26 May 2026).

The source pool begins with $N_0 = 972,247$ trajectories spanning $T_0 \approx 85\text{K}$ tasks from 10 open-source robot datasets: BridgeData-V2, BC-Z, RT-1, Galaxea, RoboMIND-V1, RoboMIND-V2, RoboCOIN, RH20T, RDT-1B, and DROID. Each trajectory is converted into a unified “LeRobot” format with RGB frames at 4 fps, up to 1,024 frames per view, and canonical state/action fields comprising absolute positions, quaternion orientations in $xyzw$ convention, and gripper state (Hu et al., 26 May 2026).

This scope places FineVLA-Data at the intersection of robot demonstration aggregation, language grounding, and controllable policy learning. A plausible implication is that the dataset is intended not merely as an annotation layer over existing corpora, but as an intermediate representation for transferring heterogeneous robot logs into a common supervision substrate.

2. Construction pipeline and trajectory selection

The construction procedure combines normalization, filtering, clustering, and representative selection. First, trajectories are standardized into the unified schema. Second, an action–state consistency filter is applied using a dynamic time warping objective

$$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$

and trajectories with

$$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$

are dropped. A corresponding description later states the same statistical filter as dropping trajectories when

$D_{\mathrm{DTW}(i)} > \tau_d.$

Third, within each task, the remaining trajectories are clustered by DTW on canonical action sequences, and 2–3 representatives per cluster are selected using medoid proximity and action smoothness. This reduces 972,247 trajectories to 47,159 samples, approximately $4.8\%$ of the original pool, while preserving execution diversity (Hu et al., 26 May 2026).

A further statistical constraint, cluster-size gating, removes clusters smaller than a minimum size $s_{\min}$ to avoid spurious outliers. In aggregate, this establishes a selection strategy that emphasizes temporal consistency and within-task behavioral diversity rather than uniform random subsampling.

The resulting dataset statistics reported for FineVLA-Data are summarized below.

Quantity	Value
Selected trajectories	47,159
Total steps	220,606
Avg coarse words	9.3
Avg FG words	96.8
Density increase	10.4×

The source-specific summary table in the technical overview includes, for example, BridgeData-V2 with 4,958 trajectories, 21,554 steps, average coarse instruction length 10.1 words, average fine-grained instruction length 61.7 words, and density $6.1\times$, while the overall total is 47,159 trajectories and 220,606 steps with instruction length increased by $10.4\times$ (Hu et al., 26 May 2026).

3. Annotation schema and semantic structure

Each selected trajectory is annotated step-by-step using a unified ten-dimensional fine-grained schema. For trajectory $T_0 \approx 85\text{K}$0 with steps $T_0 \approx 85\text{K}$1, the annotation is formalized as

$T_0 \approx 85\text{K}$2

The ten fields are:

1. Action Sequence ($T_0 \approx 85\text{K}$3): primitive verb + gripper state
2. Active Actor ($T_0 \approx 85\text{K}$4): e.g. left/right arm, finger
3. Target Object ($T_0 \approx 85\text{K}$5): category + disambiguation (color/positional)
4. Initial Config. ($T_0 \approx 85\text{K}$6): pre-action pose/state
5. Final Config. ($T_0 \approx 85\text{K}$7): post-action pose/state
6. Contact & Approach ($T_0 \approx 85\text{K}$8): contact region + approach direction
7. Trajectory & Orientation ($T_0 \approx 85\text{K}$9): path shape + rotation dir
8. Object Interaction ($xyzw$0): secondary collisions, induced motions
9. Failure & Recovery ($xyzw$1): visible retries, slips
10. Body Motion ($xyzw$2): base/torso/camera movement (Hu et al., 26 May 2026)

The formulation of $xyzw$3 is given explicitly as

$xyzw$4

Within the benchmark analysis, fact categories are aligned with these ten fields. The held-out benchmark contains 10,816 atomic facts over 500 videos, with averages of 4.3 steps per trajectory and 21.6 facts per trajectory. Example category counts include 218 “Action Sequence” facts and 155 “Contact & Approach” facts. The distribution metric is defined by

$xyzw$5

where $xyzw$6 is the count of atomic facts of type $xyzw$7 and $xyzw$8 is the total number of facts; one example is $xyzw$9 (Hu et al., 26 May 2026).

This schema operationalizes execution semantics at a granularity that coarse task labels do not encode. This suggests that FineVLA-Data functions simultaneously as an instruction dataset and as a structured event representation for robotic video understanding.

4. Human verification and quality control

The annotation workflow is model-assisted but explicitly human-verified. Qwen3.5-Plus first fills the ten fields for each step automatically. Human reviewers then verify temporal ordering, object identity, actor identity, contact region and approach, trajectory/orientation and state transitions, and the absence of hallucinated events. The stated verification criteria are:

• temporal ordering: step $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$0 corresponds to the correct video segment
• object identity: $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$1 is consistent
• actor identity: $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$2 matches the observed effector
• contact region and approach: $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$3 is faithful
• trajectory/orientation and state transitions: $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$4 and $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$5 are correct
• hallucination removal: no unsupported events remain (Hu et al., 26 May 2026)

The use of human verification is significant because fine-grained language supervision is particularly vulnerable to plausible but unsupported model-generated detail. The explicit requirement to remove hallucinations indicates that the dataset is intended to preserve event-level factuality rather than merely fluency or descriptive richness.

A common misconception would be to equate fine-grained annotation with automatically expanded captions. FineVLA-Data does not fit that description. The workflow is model-assisted, but the final dataset is described as human-verified, and verification targets action-aligned, temporally localized fields rather than generic caption elaboration (Hu et al., 26 May 2026).

5. Benchmarking interface and evaluation formats

FineVLA includes a held-out benchmark, RoboFine-Bench, consisting of 500 held-out videos, 10,816 human-reviewed atomic facts, and 1,030 VQA questions. The split is 50 videos per source dataset. VQA entries are stored as JSON objects containing fields such as sample_id, question_id, mode, capability, question, options, answer, and reference_text. An example question asks: “From which side does the gripper approach the cup?” with options ["A. above","B. side","C. front","D. below"], answer "A", and reference text "Contact region: handle; approach direction: above" (Hu et al., 26 May 2026).

Answers are scored by exact matching. The benchmark also contains a caption track with two settings: Easy, which includes the original coarse instruction, and Hard, which uses video only. Caption evaluation uses three per-trajectory metrics: $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$6

$$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$7

$$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$8

These formats make the benchmark suitable for evaluating both discriminative understanding, through fact and VQA tasks, and generative faithfulness, through captioning metrics that jointly assess factual alignment and hallucination behavior. A plausible implication is that FineVLA-Data supports not only policy conditioning but also robotics-specialized video-language evaluation, especially for execution-sensitive phenomena such as contact and approach.

6. Robotics-specialized VLM annotator and downstream policy use

FineVLA-Data supervises a robotics-specialized VLM annotator, RoboFine-VLM. The model base is Qwen3.5-397B-A17B. It is supervised fine-tuned on the 47,159 human-verified trajectories. The input is sampled multi-view frames together with the prompt “Decompose into fine-grained steps,” and the output is a JSON structure of $$D_{\mathrm{DTW}(i)} = \min_{\pi} \sum_{(t,u)\in\pi} c\bigl(a^i_t,\,s^i_u\bigr),$$9 with the ten fields $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$0 (Hu et al., 26 May 2026).

The training procedure is given in pseudocode form: for each trajectory $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$1, sample frames, construct the prompt, obtain predicted instructions from Qwen3.5, compute $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$2, and update parameters via AdamW. Although the pseudocode is compact, it establishes the intended learning objective as sequence-level supervised generation against the human-verified structured annotations.

For steerable VLA policy training, two parallel corpora are formed from the same trajectories: $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$3 for goal-level instructions and $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$4 for fine-grained instructions. Their sampling ratio is defined as

$$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$5

Seven settings are compared: Raw-only ($$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$6), FG:Raw $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$7 ($$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$8), $$D_{\mathrm{DTW}(i)} > \tau_{\text{dataset\_specific}}$$9, $D_{\mathrm{DTW}(i)} > \tau_d.$0, $D_{\mathrm{DTW}(i)} > \tau_d.$1, $D_{\mathrm{DTW}(i)} > \tau_d.$2, and FG-only ($D_{\mathrm{DTW}(i)} > \tau_d.$3) (Hu et al., 26 May 2026).

This formulation is notable because it treats fine-grained language not as a replacement for raw instructions by default, but as a controllable supervisory component whose optimal proportion is empirically determined.

7. Empirical findings and significance

The reported empirical pattern is a consistent inverted-U performance curve over the instruction mixture ratio $D_{\mathrm{DTW}(i)} > \tau_d.$4. Across three dataset–architecture settings—RDT-OFT, RDT-GR00T, and AlohaMix-OFT—and in both Easy and Hard splits of RoboTwin, success rate $D_{\mathrm{DTW}(i)} > \tau_d.$5 increases as $D_{\mathrm{DTW}(i)} > \tau_d.$6 moves from 0 toward 1 and decreases as $D_{\mathrm{DTW}(i)} > \tau_d.$7 grows beyond 1, with the peak occurring near $D_{\mathrm{DTW}(i)} > \tau_d.$8. The paper states this as

$D_{\mathrm{DTW}(i)} > \tau_d.$9

For AlohaMix-OFT on the Easy split, the reported values are: $4.8\%$0 (Hu et al., 26 May 2026)

In simulation on RoboTwin, FG-only improves over Raw-only by $4.8\%$1 to $4.8\%$2 success-rate points across architectures, and mixed supervision at $4.8\%$3 yields the best success rate, including 86.8% on the Easy split and a $4.8\%$4-point gain over raw supervision in the cited example. In real-world dual-arm manipulation on 7 in-domain paired tasks—Color, Pose, Approach, Rotate, Arm, plus 2 general tasks—the comparison between Raw-only and FG:Raw $4.8\%$5 is reported as follows (Hu et al., 26 May 2026):

Supervision	Pose	Color	Approach	Overall
Raw-only	24	22	60	49.9
FG:Raw = 1:1	47	40	78	62.7

The largest per-factor gains over Raw-only are Pose $4.8\%$6, Color $4.8\%$7, and Approach $4.8\%$8. The abstract additionally reports that the best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation, versus 49.9 for Raw-only (Hu et al., 26 May 2026).

These findings are significant because the largest gains occur precisely in factors for which goal-level instructions provide no guidance. This suggests that FineVLA-Data is most consequential not when the task objective is ambiguous, but when the objective is fixed and the execution mode must be controlled. In that sense, FineVLA-Data formalizes a distinction between goal specification and execution specification, and provides empirical evidence that the latter can improve both steerability and task success when mixed appropriately with the former.