Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

Abstract: General-purpose robot reward models are typically trained to predict absolute task progress from expert demonstrations, providing only local, frame-level supervision. While effective for expert demonstrations, this paradigm scales poorly to large-scale robotics datasets where failed and suboptimal trajectories are abundant and assigning dense progress labels is ambiguous. We introduce Robometer, a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. Robometer is trained with a dual objective: a frame-level progress loss that anchors reward magnitude on expert data, and a trajectory-comparison preference loss that imposes global ordering constraints across trajectories of the same task, enabling effective learning from both real and augmented failed trajectories. To support this formulation at scale, we curate RBM-1M, a reward-learning dataset comprising over one million trajectories spanning diverse robot embodiments and tasks, including substantial suboptimal and failure data. Across benchmarks and real-world evaluations, Robometer learns more generalizable reward functions than prior methods and improves robot learning performance across a diverse set of downstream applications. Code, model weights, and videos at https://robometer.github.io/.

• The paper introduces a dual-objective reward model named Robometer that uses frame-level progress prediction and inter-trajectory preference loss to learn from both expert and failed trajectories.
• It employs a causally masked visual language model trained on over 1M trajectories to achieve higher value order correlation and improved trajectory ranking compared to baselines.
• Experimental results demonstrate robust failure detection and enhanced policy learning in online/offline RL and data filtering, validating its scalable and practical design.

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

Motivation and Contributions

Robometer addresses a critical bottleneck in robotic reward modeling by moving beyond absolute progress supervision towards hybrid supervision via trajectory-level preferences. Existing general-purpose reward models are limited by reliance on expert demonstrations for dense progress labeling, which precludes leveraging the large quantities of failed and suboptimal trajectories encountered during real-world robotic data collection. Labeling these is ambiguous and resource-intensive, leading to discarded data and reduced generalization capacity. Robometer introduces a dual objective: frame-level progress prediction (anchored on expert data) and inter-trajectory preference loss (ranking execution quality), enabling scalable reward learning from large, heterogeneous datasets including unlabeled failures.

Figure 1: Robometer is a VLM-based reward model, predicting per-frame progress, success, and trajectory-level preferences through multiple augmentation strategies for dataset curation.

To support this, Robometer is trained on RBM-1M, a dataset of over one million trajectories spanning 21 robot platforms, including substantial failure and suboptimal data. This enables the model to internalize globally consistent preference relations and leverage comparative supervision for tasks where absolute progress is ambiguous.

Model Architecture and Training Regime

Robometer utilizes a causally masked VLM (Qwen3-VL-4B-Instruct) as backbone, ingesting either one (for reward inference) or two video trajectories (for preference supervision) per prompt.

Figure 2: Robometer model architecture; language and two video trajectories are processed, generating progress, success, and preference outputs from shared embeddings.

Progress supervision is achieved by inserting learned tokens after frames in the first trajectory, whereas preference prediction is facilitated by a dedicated token at the sequence end, attending over both trajectories. The model discretizes progress into bins for categorical cross-entropy loss—following the C51 RL formulation—recovering continuous progress during inference. For preference prediction, a binary cross-entropy classifier on the hidden state of the preference token ranks which trajectory better achieves the language-specified task. Additionally, a success prediction head outputs binary completion signals.

Three strategies curate preference pairs for training:

• Progress-based comparisons: expert vs. failed/suboptimal executions for the same instruction.
• Instruction negatives: contrasting trajectories from different tasks.
• Video rewind augmentations: synthetic failures generated by reversing sub-sequences in demonstrations.

This methodology allows Robometer to leverage both reward-labeled and unlabeled, failed trajectories at scale, providing dense, globally grounded reward signals.

Empirical Evaluation

Robometer demonstrates persistent superiority across multiple evaluation axes. On RBM-EVAL-OOD (out-of-distribution benchmarking), the model achieves strong task-trajectory alignment, as evidenced by confusion matrices revealing high reward selectivity along the diagonal for correct language-trajectory pairs.

Figure 3: Confusion matrix comparison of demo videos and instructions; Robometer yields highly diagonal matrices, signifying strong reward-task alignment.

Quantitatively, Robometer surpasses baselines in Value Order Correlation (VOC) and trajectory ranking (Kendall $\tau_a$); for RBM-EVAL-OOD, it attains Kendall $\tau_a$ of $0.66$, outperforming RoboReward-4B ($0.50$) and RoboReward-8B ($0.47$).

Qualitative analysis further reveals the model's robustness: Robometer adjusts progress estimates upon task regression (e.g., marker drop), avoiding the over-assignment of progress during failures, which is a common error in prior models.

Figure 4: Progress estimates for failed, suboptimal, and successful trajectories; Robometer uniquely detects regressions and explicit task completion.

On RoboRewardBench (external benchmark), Robometer delivers MAE of $0.72$, better than RoboReward-4B ($0.85$), and is competitive with larger closed-source models.

Importantly, fine-tuning on task-specific failure datasets (RoboFAC) validates transfer capacity. Robometer exhibits superior reward evaluation after adaptation, with full-finetuned model achieving $VOC\,r=0.884$, $Kendall\,\tau=0.802$, and significant success-fail separation.

Ablation Studies: Supervision and Data Regimes

Ablations demonstrate mutual reinforcement: preference supervision improves reward alignment and policy ranking, even absent explicit failed data. Scaling with failed trajectories further enhances model calibration and success-fail reward separation (up to $4\times$). The architecture's reliance on pretrained VLMs is essential; attempts to use larger ReWiND architectures without VLM pretraining yield degraded performance.

Training on LIBERO simulation tasks, policies derived via Robometer reward models exhibit $2-4\times$ improved sample efficiency relative to sparse reward RL, confirming metric-reward translation to policy learning.

Figure 5: RL with ablation models on LIBERO tasks; preference and failed data supervision yield drastically improved sample efficiency and success rates.

Downstream Applications & Failure Detection

Robometer's dense, instruction-aligned rewards unlock diverse downstream applications:

Automatic Online RL

Robometer, when used in DSRL (Diffusion Steering RL) setups, automates online RL, improving base policy success rates from $20\%$ to $85\%$ (single-stage) and $70\%$ (multi-stage). It outperforms RoboReward by $2.5\times$, demonstrating robust avoidance of reward misalignment and automated stage progression/reward-based success detection.

Figure 6: DSRL online RL setup; Robometer enables automated success detection with large gains in policy improvement.

Offline RL

Combining expert and noisy trajectories, Robometer rewards enable IQL-trained policies to extract higher-quality behaviors with $2.4\times$ improvement over baselines per-task, particularly at lower $\gamma$ values due to consistent intermediate reward feedback.

Figure 7: Offline RL with IQL on mixed-quality data; Robometer rewards yield major improvements in policy extraction.

Data Filtering {content} Retrieval

For imitation learning, Robometer retrieves subtrajectories with high task relevance, exhibiting $4.5\times$ improvement in downstream policy success rates versus baselines. Both preference and progress-based retrieval objectives prove effective.

Figure 8: Retrieval quality and imitation success; Robometer consistently recovers more relevant segments and facilitates robust policy outcomes.

Failure Detection

In zero-shot failure detection on OOD manipulation tasks, Robometer achieves highest average F1 scores, accurately flagging irreversible (e.g., object drops) and non-terminal failures (e.g., oscillatory stalling, instruction mismatch).

Figure 9: Dropped object during transport; Robometer detects sharp progress regressions and flags failures promptly.

Figure 10: Dropped object during placement; irreversible failures prompt immediate progress regression and detection.

Figure 11: Wrong object grasped despite correct motion pattern; semantic failures revealed as persistently low progress.

Dataset Scaling and Diversity

RBM-1M is a deliberate aggregation of expert, human, simulated, and failed trajectories across an unprecedented diversity of embodiments and task categories, facilitating reward modeling that generalizes across seen and unseen robots, viewpoints, and task structures.

Figure 12: RBM-1M dataset composition, revealing broad coverage for robust embodiment and task generalization.

Limitations and Prospects

Robometer's frame-based model (subsampling $8$ frames/trajectory) restricts fine-grained temporal and long-horizon reasoning. Certain rare or subtle failure modes may not be captured given visual-only supervision; the model cannot directly exploit latent physical state (e.g., contact, force). Future extensions could leverage denser temporal representations, VQA-style supervision, or systematically curated failure data to expand coverage of real-world failure diversity.

Figure 13: Model-based RL with Robometer; integration into DreamZero improves trajectory ranking and policy execution.

Conclusion

Robometer introduces scalable reward learning via joint progress and preference supervision, enabling dense, instruction-grounded reward models for general robotic applications. Empirical results show robust generalization, improved policy learning, and practical failure detection across diverse robots and tasks. The RBM-1M dataset and novel architectural/training recipes position Robometer as a strong initialization for domain-specific reward adaptation. The implications are substantial for RL, IL, and data curation—huge robotic datasets are no longer bottlenecked by ambiguous progress labels, and preference-driven comparisons unlock dense performance signals from failed behaviors. Anticipated advances involve richer temporal modeling and broader sensor integration to further expand generalization and fine-grained reward precision.