Robustness of general-purpose VLMs as robot reward models

Determine whether state-of-the-art vision-language models pretrained on large, diverse internet-scale datasets can robustly provide accurate and reliable rewards for real-world robotic reinforcement learning at the level of precision required for effective policy training.

Background

Reinforcement learning for real-world robotics requires precise and reliable reward signals, but obtaining such rewards is labor-intensive. Vision-LLMs offer a potential automated alternative due to their broad perceptual and linguistic capabilities.

However, despite extensive pretraining on diverse data, it remains uncertain whether these general capabilities suffice to deliver rewards with the fidelity needed for reinforcement learning. This uncertainty motivates the creation of RoboReward and RoboRewardBench to rigorously evaluate reward accuracy across many robots and tasks.

References

While VLMs are pretrained on large datasets drawn from a diverse set of sources-endowing them with general vision-language abilities-it is not clear that these general abilities enable them, at present, to robustly provide rewards at the level of precision and reliability required by RL training.

— RoboReward: General-Purpose Vision-Language Reward Models for Robotics  (2601.00675 - Lee et al., 2 Jan 2026) in Introduction

Whether such signals improve quality-aware pretraining remains an open question.

— What Matters in Quality-Aware VLA Pretraining: A Benchmark and Empirical Study  (2610.18869 - Yan et al., 13 Oct 2026) in Discussion and Conclusion

Camera calibration is required, and sensitivity to real calibration errors is not yet established.

— AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attention  (2609.20106 - Tu et al., 17 Sep 2026) in Section Conclusion and Limitations

Finally, on the real robot we evaluate the adapted model only offline, as a reward model scoring recorded demonstrations; we have not yet trained a policy with its progress estimates as rewards, so whether more accurate progress estimates translate into faster or more reliable reinforcement learning remains an open question.

— AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attention  (2609.20106 - Tu et al., 17 Sep 2026) in Section Conclusion and Limitations

Whether a single multi-modal reward model, trained jointly on vision and touch across a large and diverse task suite, can transfer to new tasks without retraining remains open.

— TaRL: Learning General and Physical Rewards from Tactile Demonstrations  (2609.36785 - Wu et al., 29 Sep 2026) in Limitations section, immediately preceding Section Acknowledgement