Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation (2512.23703v1)

Abstract: The primary obstacle for applying reinforcement learning (RL) to real-world robotics is the design of effective reward functions. While recently learning-based Process Reward Models (PRMs) are a promising direction, they are often hindered by two fundamental limitations: their reward models lack step-aware understanding and rely on single-view perception, leading to unreliable assessments of fine-grained manipulation progress; and their reward shaping procedures are theoretically unsound, often inducing a semantic trap that misguides policy optimization. To address these, we introduce Dopamine-Reward, a novel reward modeling method for learning a general-purpose, step-aware process reward model from multi-view inputs. At its core is our General Reward Model (GRM), trained on a vast 3,400+ hour dataset, which leverages Step-wise Reward Discretization for structural understanding and Multi-Perspective Reward Fusion to overcome perceptual limitations. Building upon Dopamine-Reward, we propose Dopamine-RL, a robust policy learning framework that employs a theoretically-sound Policy-Invariant Reward Shaping method, which enables the agent to leverage dense rewards for efficient self-improvement without altering the optimal policy, thereby fundamentally avoiding the semantic trap. Extensive experiments across diverse simulated and real-world tasks validate our approach. GRM achieves state-of-the-art accuracy in reward assessment, and Dopamine-RL built on GRM significantly improves policy learning efficiency. For instance, after GRM is adapted to a new task in a one-shot manner from a single expert trajectory, the resulting reward model enables Dopamine-RL to improve the policy from near-zero to 95% success with only 150 online rollouts (approximately 1 hour of real robot interaction), while retaining strong generalization across tasks. Project website: https://robo-dopamine.github.io

• The paper introduces a unified, multi-view, step-aware reward modeling framework (GRM) that accurately quantifies task progress using vision-language inputs.
• The paper demonstrates that the Dopamine-RL framework achieves sample-efficient learning, reaching up to 95% success with rapid one-shot adaptation in real-world tasks.
• The paper validates the approach through rigorous ablations, showing that multi-perspective progress fusion and policy-invariant reward shaping are critical for robust performance.

General Process Reward Modeling for High-Precision Robotic Manipulation

Introduction and Motivation

The application of reinforcement learning (RL) for real-world robotic manipulation is fundamentally bottlenecked by reward function design. Traditional hand-crafted and outcome-based policy rewards lack both scalability and the capacity to handle the fine temporal distinctions necessary for high-precision, long-horizon, and contact-rich tasks. Prior learning-based Process Reward Models (PRMs) have shown promise in increasing reward density, yet generally suffer from inadequate step-awareness, single-view perceptual limitations, and theoretically unsound shaping that fosters suboptimal policies. "Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation" (2512.23703) addresses these key limitations with a unified, multi-view, step-aware reward modeling and RL framework.

Dopamine-Reward: Step-Aware Multi-View Reward Modeling

The core contribution is the Dopamine-Reward modeling pipeline, centered around a General Reward Model (GRM) capable of accurately quantifying task progress via vision-language inputs and multi-view images. GRM is trained on a large-scale, multi-domain corpus (35M samples; 3,400+ hours) segmented and labeled using a principled hop-based stepwise progress discretization. Each training sample encodes initial, goal, "before," and "after" states, along with a normalized relative progress ("hop") label; this guarantees accurate, bounded progress estimation and robust generalization across task types and embodiments.

Figure 1: The architecture integrates the Dopamine-Reward Model and the Dopamine-RL framework, featuring a multi-view, hop-based reward modeling module and policy-invariant reward shaping for reinforcement learning.

To further mitigate perceptual drift and occlusion errors, the reward estimation employs Multi-Perspective Progress Fusion—averaging incremental (local), forward-anchored (global start), and backward-anchored (global goal) predictions. This approach produces a progress signal that is both drift-resistant and sensitive to local and global task completion cues.

Figure 2: The GRM reward profile closely matches human annotations and correctly penalizes errors, outperforming VLAC baseline.

Dataset Construction and Model Architecture

The GRM is built atop the Qwen2.5-VL architecture, receiving a sequence of task instructions and synchronized multi-view images. The underlying training data is highly heterogeneous, drawn from real-world robots, high-fidelity simulations, and large-scale, egocentric human action datasets. The dataset is structured to ensure strong task, scene, and embodiment coverage, with explicit balancing via stratified hop binning and zero-hop negative anchors.

Figure 3: Diverse real-world evaluation tasks (insertion, assembly, folding, etc.) and the multi-view hardware platform with both wrist-mounted and third-person cameras.

Figure 4: Dataset overview highlights multi-domain, multi-view coverage and the natural long-tailed task distribution.

Figure 5: GRM model structure: a unified multimodal transformer conditioned on task prompts, initial/goal images, and step-pair observations.

Dopamine-RL: Policy-Invariant Reward Shaping for RL

A theoretically-grounded Policy-Invariant Reward Shaping (PBRS) is introduced to guarantee that the use of GRM's dense reward will preserve the optimal policy of the original (sparse) task, avoiding the "semantic trap" (i.e., optimizing for intermediate proxy rewards rather than true task completion). The shaped reward takes the form $$r_{\mathrm{GRM}}(s_t, a_t, s_{t+1}) = r_{\mathrm{gold}} + \gamma\Phi^{\star}(s_{t+1}) - \Phi^{\star}(s_t)$$, where the $\gamma\Phi^{\star}(s_{t+1}) - \Phi^{\star}(s_t)$ term provably telescopes to a state-dependent constant and has no effect on action selection at optimality.

The Dopamine-RL framework is universally compatible with contemporary RL algorithms and model classes—demonstrated with PPO, ReinFlow, OpenVLA-OFT, and Cal-QL—requiring only a single demonstration for rapid one-shot task adaptation via supervised fine-tuning.

Empirical Evaluation

Reward Assessment Accuracy

GRM achieves state-of-the-art performance in both fine-grained progress perception and high-level outcome classification. On temporal order benchmarks (Value-Order Consistency), the multi-view GRM reaches VOC >0.96 across challenging real, simulated, and human domains, substantially exceeding prior VLM and PRM baselines. In classification of task completion, the GRM attains 92.8% accuracy, outperforming both specialized reward models and generalist VLMs, particularly under occlusion and spatial ambiguity scenarios.

Policy Training Efficiency and Generalization

Dopamine-RL delivers striking sample efficiency: after one-shot adaptation, policies achieve up to 95% real-world success within only 150 online rollouts (≈1 hour robot time), a substantial reduction compared to RL + sparse reward or behavioral cloning (BC). Notably, the learned policies generalize robustly to out-of-distribution settings (e.g., novel object instances, layouts, backgrounds), with performance drops of only 8–20% compared to 50–60% for BC.

Figure 6: Learning curve on LIBERO-Goal: GRM-based reward shaping enables rapid, stable convergence and high success rate.

Progress Estimation Robustness

The GRM exhibits strong invariance to temporal sampling granularity and resilience to out-of-distribution states. Progress estimation remains consistent across frame strides (10–100 frames), and the architecture includes a bi-directional consistency check to suppress reward hallucination under OOD exploration.

Figure 7: Visualization of hop and progress prediction across various unseen tasks demonstrates semantic consistency and episodic progress monotonicity.

Figure 8: Progress curve overlap across variable frame intervals confirms temporal invariance and robustness.

Online RL Robustness

Dense, accurate reward feedback from GRM enables agents to recover from mid-episode perturbations and regressions, as demonstrated in challenging manipulation rollouts with adversarial disturbances. The model immediately detects loss of progress and supplies corrective signals for rapid policy recovery.

Figure 9: Real-world policy execution with online visual feedback; disturbance triggers rapid progress drop, driving consistent policy correction and successful task completion.

Ablation and Design Analysis

Component ablations highlight the criticality of multi-perspective progress fusion (single-prediction variants incur 15–22% performance drops) and reveal that omitting policy-invariant shaping leads to a catastrophic 43.7% reduction, validating the theoretical necessity of the shaping approach. One-shot adaptation is also essential for robust out-of-distribution performance.

Theoretical and Practical Implications

This work provides a rigorous recipe for constructing generalizable, multi-view process reward models and integrating them with policy-invariant shaping for high-performance robotic RL. GRM's architecture and fusion strategy address step-awareness and occlusion robustness, while the shaping term ensures sample-efficient policy learning that remains aligned to the original task.

Practically, this framework substantially lowers data and engineering requirements for real-world RL by reducing demonstration count, ensuring rapid adaptation to new tasks, and boosting generalization. Theoretically, the design offers a blueprint for reward modeling that is provably safe under policy optimization.

Future research avenues include efficient GRM inference (e.g., model quantization, KV-cache optimization), extension to continuous video reasoning with temporally explicit models, expansion to more dynamic and mobile tasks, and incorporation of additional sensory modalities such as tactile or audio for richer reward encoding.

Conclusion

The "Robo-Dopamine" framework operationalizes large-scale, multi-view, step-aware reward modeling and theoretically-motivated shaping as a scalable solution for sample-efficient, robust RL in high-precision robotic manipulation. By addressing both the perceptual and algorithmic bottlenecks of reward learning, this work sets a new performance baseline in dense reward assessment, rapid policy refinement, and generalization, marking a significant step toward universal embodied intelligence in robotics.