ProcCorpus-60M: Dense Procedural Supervision

• ProcCorpus-60M is a large-scale procedure-annotated corpus built from 30 embodied datasets, offering near frame-wise annotations for robotic reasoning and progress estimation.
• It employs a standardized, high-throughput annotation pipeline that delivers Embodied Chain-of-Thought supervision for both reward modeling in ProcVLM and cross-embodiment pretraining in ZR-0.
• Empirical findings suggest that dense procedural supervision improves transfer learning and progress perception, though challenges remain with data imbalance and annotation noise.

Searching arXiv for papers on ProcCorpus-60M and its use in ProcVLM/ZR-0. ProcCorpus-60M is a large-scale procedure-annotated embodied manipulation corpus used to supply dense supervisory signals for robotic reasoning, progress estimation, and cross-embodiment vision-language-action pretraining. In "ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulation," it is introduced as a corpus built from 30 embodied datasets with approximately 60 million annotated frames for procedure-grounded progress learning; in "Training Vision-Language-Action Models with Dense Embodied Chain-of-Thought Supervision," it is the pretraining corpus that provides dense Embodied Chain-of-Thought supervision for ZR-0 (Feng et al., 9 May 2026, Li et al., 29 Jun 2026). Across these two framings, ProcCorpus-60M is characterized by mixed real-robot and simulation data, about 400K trajectories, nearly frame-wise annotation coverage, and structured supervision that targets shared manipulation cognition rather than only low-level state-action alignment.

1. Definition and conceptual role

ProcCorpus-60M is not presented as a generic robot video archive. Its defining property is that raw manipulation trajectories are automatically enriched with procedural or reasoning-centric annotations that describe what stage a task is in, what has already been accomplished, what remains to be done, and, optionally, which objects are relevant. In the ProcVLM framing, the corpus exists to support dense reward modeling for long-horizon manipulation, where progress should reflect procedural advancement rather than elapsed time or terminal success alone (Feng et al., 9 May 2026).

In the ZR-0 framing, the same corpus serves a different but closely related purpose: it provides dense Embodied Chain-of-Thought supervision that aligns high-level manipulation representations across heterogeneous robot embodiments. The motivating claim is that low-level state and action spaces differ fundamentally across robots, whereas scene perception, object identification, task planning, and sub-task decomposition are largely shared. ProcCorpus-60M is therefore used as a semantic alignment substrate rather than merely as a source of action trajectories (Li et al., 29 Jun 2026).

This dual role is central to understanding the resource. In one line of work, it underwrites procedure-grounded progress estimation and dense reward modeling; in another, it underwrites embodiment-agnostic representation learning inside a vision-LLM while still allowing embodiment-specific continuous control.

2. Corpus composition and construction

ProcCorpus-60M is constructed by aggregating heterogeneous embodied manipulation datasets rather than by collecting a new monolithic teleoperation corpus. The ProcVLM paper states that it is built from 30 embodied manipulation datasets spanning real-robot and simulation sources. The appendix enumerates 26 real-robot sources and 4 simulation sources: DROID, BridgeData V2, Fractal, RH20T, Table30, OXE / Austin BUDS, OXE / Austin SAILOR, OXE / Austin SIRIUS, OXE / Berkeley AUTOLab UR5, OXE / Berkeley FANUC Manipulation, OXE / CMU Play Fusion, OXE / Columbia PushT, OXE / DLR EDAN Shared Control, OXE / DLR SARA Pour, OXE / Dobb-E, OXE / FMB, OXE / IAMLab CMU Pickup-Insert, OXE / Jaco Play, OXE / NYU Door Opening, OXE / QUT Dexterous Manipulation, OXE / Stanford HYDRA, OXE / TOTO, OXE / UCSD Kitchen, OXE / UT Austin Mutex, Internal / JAKA Teleop, Internal-xArm / Real-world Tabletop, GR00T-Teleop-Sim, Internal-xArm / Simulation Tabletop, LIBERO v2.1, and RoboTwin 2.0 (Feng et al., 9 May 2026).

The exact appendix-level statistics are more precise than the rounded figures used elsewhere:

Split	Frames / trajectories	Annotated frames / rate
Overall	60,466,354 / 412,404	58,531,267 / 96.80%
Real-robot	46,358,797 / 315,791	44,454,222 / 95.89%
Simulation	14,107,557 / 96,613	14,077,045 / 99.78%

The ZR-0 paper reports the same corpus at a rounded scale as approximately 60 million frames, approximately 1,000 hours, and over 400K trajectories, with dense ECoT annotations covering 96.8% of all frames (Li et al., 29 Jun 2026).

Construction is mediated by a standardized automatic annotation pipeline. ProcVLM describes a queue-based system with asynchronous data reading, CPU-side multimodal preprocessing, GPU-side VLM inference, post-processing, and output into standardized JSONL annotations. Reported throughput is up to 4M keyframes/day on 8 H100 GPUs. The appendix identifies Qwen3-VL-235B-A22B-Instruct as the annotator for video-level planning and subtask temporal localization, and InternVL3.5-38B for frame-level reasoning and grounding annotations (Feng et al., 9 May 2026).

3. Annotation schema and procedural semantics

The corpus’s core novelty lies in its annotation schema. ProcVLM describes enriched frame-level annotations that may include task-centric scene reasoning, completion-state assignment, current subtask, global subtask structure of the trajectory, remaining actions, and optional 2D target-object bounding boxes. The appendix explicitly refers to these enriched annotations as Embodied Chain-of-Thought annotations (Feng et al., 9 May 2026).

ZR-0 provides the most explicit six-part decomposition of the per-frame ECoT structure. Each annotated frame receives: Scene Description, Progress Assessment, Future Plan, To-Do Actions, Target Objects, and Discrete Actions. Scene Description is a textual depiction of the current visual scene. Progress Assessment is a brief reasoning passage followed by a binary completion indicator, Yes/No. Future Plan is a free-form natural language description of what remains to be done. To-Do Actions decompose the future plan into atomic sub-tasks written in the template Verb + Object [+ Prepositional Phrase]. Target Objects are bounding boxes in standard JSON format, with the explicit example {"blue plate": [120, 85, 340, 260]}. Discrete Actions are embodiment-specific discrete action tokens produced by the FAST tokenizer (Li et al., 29 Jun 2026).

ProcVLM formalizes the procedural substrate through atomic actions. An atomic action is defined as an executable subtask that can be described by a single explicit verb-level action. Valid decompositions need not be unique, but they should be temporally coherent and verb-maximal. The plan-generation prompt constrains subtasks to one of several patterns: Grasp [specific object], Place [specific object] onto / into [specific location], Push [specific object] [forward / backward / left / right], Tilt the gripper, Hang [specific object] on / above [specific location], Press [specific object], Open [specific object], Close [specific object], and Rotate [specific object] (Feng et al., 9 May 2026).

Temporal segmentation is also rule-governed. For each planned subtask, the annotator outputs start_frame and complete_frame; unfinished subtasks use complete_frame=null, absent subtasks use start_frame=null, complete_frame=null, and notes="not present". The segmentation heuristics instruct the model to use changes in effector pose and object motion, to pick the final success when retries occur, and to use the last video frame as reference for overall notes. This means the corpus favors final successful realization when assigning completion boundaries (Feng et al., 9 May 2026).

4. Progress formalization and dense procedural supervision

ProcCorpus-60M is unusual among robot datasets because it does not stop at symbolic subtask labels. In ProcVLM, the corpus supports continuous progress targets derived from subtask structure plus intra-subtask visual change. With trajectory length $T$, number of valid subtasks $K$, active subtask index $k(t)$, and temporal span $[s_k,e_k]$, task progress is defined as

$$p(t)= \frac{\int_{0}^{t} w(\tau) r(\tau)\,d\tau} {\int_{0}^{T} w(\tau) r(\tau)\,d\tau}.$$

The subtask budget and local progress rate are

$$w(\tau) = \operatorname{clip}\left( \frac{K\left(e_{k(\tau)}-s_{k(\tau)}\right)}{T}, 0.75,1.25 \right),$$

$$r(\tau) = \frac{\|\dot{\phi}(\tau)\|} {\int_{s_{k(\tau)}}^{e_{k(\tau)}} \|\dot{\phi}(u)\|\,du}.$$

The interpretation given in the paper is explicit: each subtask receives a progress budget according to relative duration, but clipping to $[0.75,1.25]$ acts as a soft anchor around an equal-subtask prior; within a subtask, more perceptual change receives more progress mass, while stagnation receives less (Feng et al., 9 May 2026).

The corpus also distinguishes completion states textually. Frame-level reasoning labels separate unfinished, finished, and give-up/failed states. For unfinished frames, the model is given remaining subtasks and asked to explain why the task is unfinished and what remains. For finished frames, it explains why the task is complete; for failed or give-up frames, it explains why it remains unfinished (Feng et al., 9 May 2026).

A closely related design choice is the reasoning-before-estimation formulation used downstream. ProcVQA, derived from ProcCorpus, asks the model to first infer the remaining atomic actions and then estimate the current completion percentage, ending with <progress>~p~\%</progress>. This ties scalar progress estimation to explicit procedural grounding rather than direct regression from image frames alone (Feng et al., 9 May 2026).

5. Use in ProcVLM and ZR-0

ProcCorpus-60M feeds two different model pipelines. In ProcVLM, it is transformed into ProcVQA, a multitask instruction-tuning corpus with three task families: action segmentation, future planning, and task progress estimation. Action segmentation outputs JSON lists with "action_description", "start_frame", and "end_frame". Future planning includes immediate next-step prediction and remaining-step prediction. Progress estimation uses recent observations plus the task instruction and requires reasoning over remaining atomic actions before outputting a progress tag (Feng et al., 9 May 2026).

In ZR-0, ProcCorpus-60M is the primary robotic pretraining corpus for a dual-stream architecture consisting of a pre-trained VLM and a Diffusion Transformer-based action expert. At time $t$, the model receives task instruction $l$, image observations $K$0, and robot state $K$1. The VLM is trained to generate the ECoT reasoning sequence $K$2, while the action expert is trained to predict an action chunk $K$3. The VLM takes the instruction and resized images, with all input images resized to $K$4; the action expert takes VLM features $K$5, robot state $K$6, noisy action chunk, and flow timestep $K$7 (Li et al., 29 Jun 2026).

The corpus is essential because it supplies both sides of this multitask objective: dense reasoning labels for ECoT next-token prediction and state-action supervision for action flow matching. ZR-0 further reports that state and action dimensions are padded to 64 with zeros, the loss on padded dimensions is masked out, and each state/action dimension is min-max normalized using the 1st and 99th percentiles of the training data, which is a format-level accommodation for heterogeneous embodiments layered on top of the semantic alignment provided by ECoT (Li et al., 29 Jun 2026).

A distinctive architectural decision is the cross-attention mask. During training, the action expert is restricted to attend only to the VLM’s features corresponding to the input prompt, excluding the ECoT tokens. The practical consequence is that ECoT generation can be entirely skipped at inference without any performance loss. ProcCorpus-60M therefore affects internal representation learning during training rather than imposing runtime reasoning generation (Li et al., 29 Jun 2026).

6. Empirical significance, bias profile, and limitations

ProcCorpus-60M is significant less because it is the largest robot corpus by raw hours than because it couples heterogeneous trajectories with dense procedural supervision. ZR-0 explicitly notes that its pretraining corpus is approximately 1,000 hours, an order of magnitude below leading VLA models such as $K$8 (over 10,000 hours), LingBot-VLA (around 20,000 hours), and Qwen-RoboManip (over 30,000 hours). The paper’s claim is therefore not scale leadership, but that dense structured supervision can make a smaller corpus highly effective (Li et al., 29 Jun 2026).

Two empirical results are particularly relevant to the corpus design. First, on LIBERO, removing ECoT supervision while keeping the same architecture and action loss reduces average success from 97.8 to 95.7, with the largest drop on LIBERO-10 from 96.4 to 92.6. This is evidence that ProcCorpus-derived reasoning labels contribute materially beyond action-only supervision. Second, ProcVLM reports that removing procedure-aware pretraining causes the largest decline in RoboFAC one-shot transfer: full ProcVLM obtains VOC$K$9 0.9137, MAE$k(t)$0 0.1241, and MCC 0.7918, whereas the w/o Pretrain ablation yields 0.8281, 0.2343, and 0.5949. This supports the view that ProcCorpus-scale procedure-aware supervision is important for transferable progress perception (Feng et al., 9 May 2026).

The corpus also exhibits identifiable biases. ZR-0 states that pick-and-place is the most prevalent manipulation primitive in the pre-training corpus, whereas closing actions such as shutting cabinets, drawers, and microwaves appear far less frequently. This suggests uneven skill coverage, and the paper links such imbalance to downstream weaknesses on underrepresented behaviors (Li et al., 29 Jun 2026).

Several caveats recur across the two papers. Because annotations are generated by large VLMs, subtask decomposition, temporal boundaries, and completion reasoning can be noisy; ProcVLM therefore introduces a manually inspected high-quality refinement subset of about 15K trajectories, reported as 13,688 in the appendix table (Feng et al., 9 May 2026). Dense annotation is also computationally expensive, since each annotation involves a forward pass through a capable VLM, and the authors identify scalability as an open issue (Li et al., 29 Jun 2026). Coverage is broad but still centered on embodied manipulation and the retained screened datasets rather than all robotic domains (Feng et al., 9 May 2026). Finally, although ProcVLM provides a project page, the provided paper text does not explicitly state a direct ProcCorpus-60M download URL, release package, or license terms, so public availability of the corpus itself remains unclear from that text alone (Feng et al., 9 May 2026).

Taken together, ProcCorpus-60M is best understood as a procedure-enriched robotic trajectory corpus whose primary contribution is dense semantic structure at frame level. Its annotations bridge observations, subtasks, progress, grounding, and control-relevant reasoning; this makes it usable both as a source of procedure-grounded dense rewards and as a train-time semantic scaffold for cross-embodiment VLA pretraining (Feng et al., 9 May 2026, Li et al., 29 Jun 2026).