CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

Abstract: Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-LLMs (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds. Videos can be found on our project website: https://core-2026.github.io/

• The paper presents a novel reward decomposition approach combining formal and residual modules for preference-aligned reinforcement learning.
• It leverages vision-language model feedback to integrate human-like intent and refine reward signals for complex tasks.
• Empirical results show significant gains in sample efficiency and real-world performance compared to state-of-the-art RL methods.

Combined Rewards with VLM Feedback for Preference-Aligned RL: An Expert Analysis of CoRe

Introduction

Reward design remains a primary obstacle in deploying reinforcement learning (RL) for complex embodiments such as robot manipulation. Traditional approaches utilize hand-crafted dense or sparse rewards, which frequently lead to reward misspecification, suboptimal policy convergence, or adverse side effects such as reward hacking. Conversely, learned reward models from human preferences or demonstrations—most notably in preference-based RL (PbRL)—suffer from inefficiency, poor scalability due to prohibitive human labeling requirements, instability, and noisy alignment with nuanced, task-specific human intent.

The "CoRe: Combined Rewards with Vision-LLM Feedback for Preference-Aligned Reinforcement Learning" (2607.01721) introduces a novel hybrid reward framework aimed at addressing these core deficiencies by dual decomposition and vision-LLM (VLM) feedback integration.

Methodological Framework

CoRe decomposes the reward function into two complementary modules:

• Formal Reward Module (FRM): Constructs interpretable, code-based reward functions from explicit task knowledge, iteratively leveraging LLMs for reward synthesis and optimization. Distinctly, reward selection and refinement are performed by matching candidate rewards not simply to task-completion metrics but directly to VLM-assisted preference-alignment labels, thus explicitly incorporating human-like intent into the optimization trajectory.
• Residual Reward Module (RRM): Supplements the FRM by learning residual signals from VLM-generated video-level preference labels, capturing nuanced, implicit aspects that are challenging or impossible to encode formally. State- or frame-wise importance weights derived from VLMs inform credit assignment through KL-regularization, enhancing the policy’s sensitivity to trajectories deemed task-relevant by the model.

The final reward signal is the sum of outputs from both modules. Policy optimization is carried out off-policy (SAC), while reward learning and updates for FRM and RRM proceed in parallel, with preference sampling and labeling handled end-to-end by VLMs (e.g., Gemini 2.0 Flash, with ablations including open-source Qwen3-VL and Video-LLaVA).

Formal Reward Module: Preference Alignment

The FRM employs LLM-synthesized reward candidates generated from task knowledge (state/action schema, task description) and policy feedback. Rather than relying on simple episodic task success for reward selection (which risks reward exploitation and partial solutions), the module applies preference-guided reward selection based upon VLM-labeled trajectory segment comparisons. Maximizing agreement with these preference labels ensures that code-based reward search stays anchored to aspirational behavior that would receive human approval, rather than maximizing proxy metrics that are easy to exploit.

Residual Reward Module: VLM-Guided Credit Assignment

The RRM receives as supervision pairs of trajectory segments (as video clips) and VLM-provided preference labels that specify which trajectory is preferable, along with importance weights per frame. The reward model optimizes both cross-entropy loss for label prediction and a KL term aligning the predicted reward distribution to the VLM’s importance weights within each segment. This multi-level alignment is essential for transferring high-level preference—which may involve temporally extended behaviors or emergent strategies—into robust, dense reward signals.

Experimental Results

CoRe demonstrates significant improvements over state-of-the-art LLM- and VLM-based PB-RL baselines—including CLIP Score [Rocamonde et al., 2024], Eureka [Ma et al., 2024], Text2Reward [Xie et al., 2024], RL-VLM-F [Wang et al., 2024], PrefVLM [Ghosh et al., 2025], and ERL-VLM [Luu et al., 2025]—across ten simulated tasks (MetaWorld, SoftGym) and five real-world robot manipulation tasks.

Key Numerical Findings

• MetaWorld Tasks: CoRe achieves an average success rate of 99.0%, surpassing the best (non-dense) baseline by more than 20% and closely matching human performance (99.3%).
• SoftGym (Deformable Manipulation): CoRe attains best episode returns in all evaluated tasks (e.g., 20.6 in Straighten Rope, -0.10 in Fold Cloth, -30.0 in Pass Water).
• Sample Efficiency: CoRe provides a 3–40x reduction in required feedback labels compared to prior VLM-based preference methods (e.g., CoRe uses 0.5K vs. 21K labels in MetaWorld).
• Alignment Metrics: Learned reward correlations with expert task progress are highest for CoRe (mean Spearman $\rho = 0.88$), while others range from 0.09 to 0.77.
• Real-World Deployment: CoRe policies transferred zero-shot to UR5 arm execution exhibit near-maximum success rates and best performance compared to baselines, including in noisy visual settings.

Critical Analysis and Ablations

Decomposition Efficacy

Ablation studies confirm the complementary strengths of FRM and RRM:

• FRM alone achieves strong performance in rigid-object tasks but underperforms in deformable manipulation or tasks involving complex implicit preferences.
• RRM alone leverages VLM feedback to excel in tasks where visual/temporal credit assignment dominates but faces instability and noise in long-horizon settings.
• The combination produces reliably stable, interpretable, and dense rewards for robust policy learning across all regimes.

Reward-Preference Alignment

Iterative reward-preference alignment in FRM outperforms task-completion-only selection and prevents reward exploitation (e.g., hammer handle strikes instead of intended hammerhead use), ensuring that selected formal rewards incentivize the correct behavior as judged by preference.

VLM Model Agnosticism and Scalability

Experiments with open-source VLMs (Qwen3-VL) demonstrate that CoRe's architecture does not rely on proprietary models for fundamental gains, and superior VLMs will further improve reward fidelity. The method also remains robust under environmental visual disturbances.

Computational Efficiency

Full training with CoRe averages $2.00$M tokens, $0.37 API cost, and$2.15$ hours wall-clock time per task—substantially outperforming other VLM-based RL methods in the accuracy-efficiency trade-off.

Implications and Future Directions

CoRe's decomposition of reward learning is conceptually aligned with cognitive science perspectives on human learning, emphasizing the value of explicit structure (prior/task knowledge) and flexible adaptation (implicit, observation-based preference).

Theoretical Implications:

• Offers a robust template for scalable, preference-aligned RL applicable to high-dimensional, real-world tasks without direct human labeling or hand-crafted dense reward engineering.
• Lays groundwork for integrating advanced VLM/LLM feedback for even more nuanced or high-level preferences (e.g., incorporating multi-agent or social objectives).

Practical Implications:

• Strong numerical and deployment results validate CoRe as a default paradigm for robotic policy learning where preference alignment and generalization are critical.
• Label/sample efficiency opens the path for deploying RL agents in domains with limited annotation resources.

Future Developments:

• Extending uncertainty-aware reward modeling and adaptive VLM querying to further minimize computational cost and maximize feedback informativeness.
• Evaluating generalizability and robustness in highly unstructured, real-world multi-agent scenarios and non-stationary environments.

Conclusion

CoRe introduces a model-agnostic framework synthesizing explicit code-level rewards with residual, preference-driven signals under VLM guidance, establishing a new standard for preference-aligned reward learning in RL. It achieves superior performance, exceptional sample efficiency, and robust real-world deployment capabilities by integrating principled reward decomposition with advanced multimodal model feedback.

This work delineates a promising path for scalable, human-aligned autonomous systems and marks a substantive advance in bridging RL policy learning with the structure and flexibility of human-like preference inference (2607.01721).