- The paper introduces ProcVLM, a method that uses vision-language guided procedural supervision to provide dense progress rewards in robotic manipulation.
- It details a high-throughput annotation pipeline that generates the ProcCorpus-60M dataset and employs a two-stage training process for granular subtask progression.
- Experimental results demonstrate ProcVLM's superior progress ranking, future planning, and reward-guided policy optimization compared to existing baselines.
Procedure-Grounded Progress Reward Modeling for Robotic Manipulation: An Analysis of ProcVLM
Motivation and Context
Long-horizon robotic manipulation tasks in unstructured settings require dense, semantically grounded feedback to ensure reliable skill acquisition and adaptation. Conventional reward models for such tasks exhibit systemic deficiencies, primarily due to reliance on sparse trajectory-level success signals, time-based interpolation, or outcome-driven pseudo-rewards. These approaches conflate execution time with procedural advancement and systematically fail to localize failures, stagnations, or partial completions within multi-step procedures. ProcVLM provides a significant advancement by establishing a procedure-grounded, progress-aware dense reward model that leverages vision-language paradigms for effective reward-guided policy optimization, thereby facilitating generalization and robust adaptation across tasks and modalities (2605.08774).
Procedural Supervision Synthesis and Dataset Construction
A core contribution is the design and deployment of a high-throughput, scalable pipeline for procedural supervision synthesis. This pipeline utilizes large vision-LLMs (VLMs) as hierarchical annotators, transforming raw, multimodal robot trajectories into frame-aligned subtask structures, explicit assignment of remaining actions, and per-frame reasoning meta-data. The annotator pipeline achieves effective decoupling between CPU/GPU workloads and produces dense, temporally consistent annotations via a combination of video-level and keyframe-level VLM inference.
The resultant dataset, ProcCorpus-60M, encompasses over 60M annotated frames from 400K trajectories, integrating real-robot and simulation data across 30 diverse manipulation datasets. This scope is unprecedented in procedure-aware annotation density and diversity, constituting a foundation for broad generalization. The authors additionally curate a high-quality subset (~15K trajectories) for refinement, ensuring that subtask alignments are robust and semantically meaningful.
Procedure-Defined Progress Labels and Model Architecture
ProcVLM operationalizes progress estimation as a function of subtask structure and intra-subtask visual dynamics, rejecting linear time as a proxy for procedural advancement. Progress is defined by integrating local visual changes, weighted by procedural budgets computed from subtask durations (with clipping to prevent dominance by outlier stages), and normalized across the full trajectory. The perceptual difference between adjacent frames supplies a low-cost proxy for continuous visual change, allowing granular attribution of progress within and across subtasks.
The model itself is realized atop a compact, instruction-tuned VLM backbone (Qwen3-VL-2B-Instruct), extended by a continuous-valued regression head gated via the presence of progress supervision. Training follows a two-stage pipeline: (1) large-scale pretraining using a multi-task VQA-style corpus (ProcVQA, ~20B tokens) constructed from ProcCorpus-60M, and (2) refinement on the curated high-quality subset to sharpen subtask/progress fidelity.
Multi-task supervision is central: in addition to continuous progress regression, the model is jointly optimized for action segmentation and future action planning. The response format explicitly enforces a reasoning-then-regression protocol, with the model first predicting remaining atomic actions before emitting a continuous progress value. Leakage from ground-truth tokens is mitigated through feature-level dropout on the answer span during training.
Experimental Results
Embodied Procedural Reasoning
On the in-distribution (ID) and out-of-distribution (OOD) splits of ProcVQA, ProcVLM consistently outperforms established VLM baselines (Qwen3VL, Qwen3.5, Gemini 3.1 Pro, GPT-5.4) by substantial margins in boundary F1 score (BF1@5), future plan success, VOC, and effective progress resolution (EPR@50). These results reinforce the value of explicit procedure-grounded supervision and architectural coupling of reasoning and regression for long-horizon embodied tasks.
Reward Model Benchmarking and Adaptation
Zero-shot evaluation on progress-reward tasks shows that ProcVLM yields higher VOC than both Robometer and RoboDopamine, demonstrating finer intra-trajectory progress ranking in unseen environments. One-shot adaptation experiments on RoboFAC exhibit large relative gains in progress ordering (VOC), binary success detection (MCC), and fault localization (MAE) after only a single successful (or failure-specific) demonstration. This one-shot efficiency highlights the transferability of procedural representations learned during large-scale pretraining, in contrast to baselines whose nominal reward structures (preference or step-wise reward fusion) exhibit limited adaptability in the low-data regime.
Ablation Studies
Ablative analysis confirms the necessity of both procedure-aware pretraining and explicit subtask-formatted reasoning: omission of pretraining substantially degrades all tested metrics, especially fault localization and binary success detection. Removing reasoning-formatted supervision similarly reduces fidelity across progress alignment and failure diagnosis, emphasizing the compounded role of structured procedural cues for generalizable reward inference.
Reward-Guided Policy Optimization
ProcVLM-based dense reward signals are demonstrably superior to standard supervised fine-tuning (SFT) for offline RL. On LIBERO-10 (simulation), reward-guided fine-tuning (RFT) provides consistent early-stage success rate improvements. On real-robot stack-bowls tasks, the gains are substantial (e.g., +25% at 5k steps), attributable to robust suppression of overfitting to suboptimal demonstrations and enhanced discrimination of useful exploration segments. These results indicate that procedure-grounded rewards are more sample-efficient and resilient to noise compared to traditional regression-to-demo methods.
Theoretical and Practical Implications
ProcVLM's paradigm—grounding reward regression in explicit procedural structure and structured reasoning—addresses a central causality gap in reward design for robotics. This progress-aware supervision provides dense, interpretable feedback tightly coupled to task semantics, overcoming the limitations of time-based proxies and sparse terminal labels.
Practically, this framework enables more robust and generalizable policy improvement, supporting effective transfer to unseen tasks, viewpoints, and failure modalities with minimal adaptation data. The pipeline for automated procedural annotation further opens the possibility of rapidly scaling dense reward modeling to new domains, providing a foundation for truly generalist robotic agents.
Theoretically, ProcVLM points toward architectural synergies between multimodal pretraining, explicit subgoal representation, and continuous progress modeling in RL. It also suggests that compositional task structure—explicitly represented and learned—may be a crucial prior for scalable robot learning.
Limitations and Future Directions
The fidelity of subtask decomposition and temporal boundary localization remains a bottleneck; annotation quality directly impacts model performance. Downstream experiments to date are limited in coverage (e.g., exclusive focus on reward-guided offline RL), so further studies are needed to evaluate integration with online RL, distributional RL, or more general preference-based learning pipelines. The regression head is lightweight; future work is warranted on robust calibration, distributional objectives, and comparison-based reward alignment.
Conclusion
ProcVLM presents a robust, scalable pipeline for dense, procedure-grounded reward modeling in vision-language-action robotic manipulation. By explicitly coupling structured procedural reasoning with continuous progress estimation, it yields transferable, discriminative, and adaptive reward signals for long-horizon policy optimization. This work lays foundational groundwork for a new class of dense reward models within the broader VLM paradigm, with far-reaching implications for scalable, generalist robotics and embodied intelligence (2605.08774).