Automating Potential-based Reward Shaping with Vision Language Model Guidance

Abstract: Sparse rewards are inherently challenging for reinforcement learning agents as they lack intermediate feedback to guide exploration and to correctly attribute the sparse success rewards to relevant parts of the trajectory. Naive reward shaping can induce reward hacking, yielding policies that exploit auxiliary signals instead of solving the intended task. Potential-based reward shaping (PBRS) guarantees preservation of the optimal policy set, but requires the definition of a heuristic potential function over the state space. In this work, we introduce the VLM-guided PBRS framework VLM-PBRS that learns the potential function directly from vision LLM (VLM) feedback. We query a lightweight VLM to obtain preferences over image pairs and train a model of the potential function using these preferences. As this approach is based on potential-based reward shaping, it preserves the original optimal policies, and removes the need for expert-designed reward shaping terms. Because large VLMs are prohibitively expensive to invoke repeatedly during policy learning, we employ smaller, more computationally efficient VLMs. Although the resulting preference labels are less accurate, empirical evidence shows that the preference labels can still be used to accelerate learning. We validate our method empirically in the Meta-World and Franka Kitchen environments and highlight the connection between VLM preference label accuracy and sample efficiency improvements. Our contributions are threefold: (1) the first application of VLM preference-based learning to synthesize a potential function for PBRS, (2) a principled, low-cost solution that leverages small VLMs, and (3) extensive empirical demonstration of improved sample efficiency and robustness to reward hacking.

• The paper introduces VLM-PBRS to automate potential-based reward shaping using vision-language models, reducing the need for manual potential function design.
• It integrates preference-based reward learning with PBRS to boost sample efficiency and maintain policy robustness in sparse-reward environments.
• Empirical results show that even with modest VLM accuracy, the approach outperforms sparse and human-engineered dense rewards in complex robotic tasks.

Automating Potential-Based Reward Shaping with Vision LLM Guidance

Introduction and Motivation

The paper "Automating Potential-based Reward Shaping with Vision LLM Guidance" (2606.27180) introduces VLM-PBRS, an automated framework for constructing potential-based reward shaping (PBRS) functions using Vision LLMs (VLMs). The motivation arises from the challenge in reinforcement learning (RL) of efficiently learning policies in sparse reward environments, where the lack of intermediate signals impedes exploration and credit assignment. Traditional reward shaping, while helpful, often risks reward hacking and policy misalignment. PBRS, grounded in theoretical guarantees, preserves optimal policy sets but demands labor-intensive potential function engineering.

VLM-PBRS leverages VLMs to generate preference labels over environment image pairs given a textual goal description, thus learning the potential function required for PBRS without expert intervention. The method targets computational efficiency by utilizing smaller VLMs, tolerating reduced label accuracy due to PBRS's policy invariance, and empirically demonstrates sample efficiency gains and robustness.

Figure 1: VLM-PBRS framework, illustrating the iterative loop between VLM feedback, potential function learning, and RL policy improvement.

Methodological Framework

The core of VLM-PBRS is the integration of preference-based reward learning and potential-based shaping in RL. The algorithm proceeds by:

• Collecting image observations during initial exploration
• Periodically sampling image pairs and querying VLMs for preferences relative to a task description
• Training a parameterized preference model using these labels
• Transforming the preference model into a bounded potential function (with sigmoid activation and scaling)
• Computing potential-based shaping rewards and updating RL policy via standard off-policy methods (SAC)

Preference queries are operationalized via prompt pipelines that are task-agnostic, requiring only a succinct textual description from the user. The VLM generates preference labels over single-step image pairs, extracted directly from VLM output for computational expedience. Training ablates between cross-entropy and MAE losses, showing negligible differences, and dynamically updates the potential function during RL, leveraging PBRS's theoretical resilience to shaping non-stationarity.

Figure 2: Prompt pipeline for image pair labeling, combining goal description and images to form structured VLM input and extracting labels from output.

Empirical Results

Experiments span Meta-World (robotic manipulation tasks) and Franka Kitchen (high-dimensional, visually complex tasks). Key baselines include sparse rewards, human-engineered dense rewards, and RL-VLM-F (reward learning directly from VLM preferences). VLM-PBRS employs Ovis2-16B and Qwen3-VL-8B for Meta-World and Franka Kitchen, respectively.

Results reveal:

• Sample Efficiency: VLM-PBRS consistently outperforms sparse baselines and, in some tasks (e.g., button-press), even dense human rewards, despite using small VLMs and modest label accuracy.
• Policy Robustness: VLM-PBRS maintains the optimal policy set regardless of shaping function quality, whereas RL-VLM-F can suffer from suboptimality due to label errors.
• Label Quality: Empirical accuracy for preference labeling varies ($48-75\%$), with VLM-PBRS showing resilience to inaccuracies, provided label quality exceeds chance.
• Ablation Studies: Improvements in sample efficiency positively correlate with VLM label accuracy and number of preference queries per batch; however, performance saturates beyond moderate batch sizes.

Figure 3: Learning curves for the button-press task, showing VLM-PBRS exceeding dense reward baseline in convergence speed and final return.

Figure 4: Drawer-open task performance, demonstrating sample efficiency gains over sparse rewards despite at-chance VLM label accuracy.

Figure 5: Ablation of VLM label accuracy in door-open; sample efficiency gains manifest only above random label quality.

Theoretical and Practical Implications

VLM-PBRS exploits PBRS's invariance to the potential function, relaxing requirements on label accuracy and model size, with immediate implications:

• Scalability: Automated shaping generalizes across domains and scales with multi-modal foundation model advances. Only a textual goal description is required, minimizing human involvement.
• Computational Efficiency: Smaller VLMs suffice, reducing inference costs and enabling real-time or embedded scenarios.
• Safety and Policy Alignment: The method is robust against reward hacking and erroneous shaping, mitigating risks associated with noisy foundation model outputs.
• Future Directions: Increased expressiveness via richer modalities (video, temporal prompts), improved active selection of labeling examples, and architecturally tailored RL agents for non-stationary shaping functions.

Figure 6: Light-switch task in Franka Kitchen, illustrating VLM-PBRS's adaptability to visually noisy, realistic environments.

Conclusion

VLM-PBRS provides a principled, automated mechanism for reward shaping in RL, leveraging VLMs to guide potential function synthesis. Empirical evaluations demonstrate improved sample efficiency and policy robustness in sparse-reward robotic tasks, with resilience to imperfect labeling and computational constraints. The framework bridges foundational RL theory with contemporary multi-modal models, and its generality positions it as a scalable approach for reward specification in increasingly complex autonomous decision-making contexts. As VLMs evolve, VLM-PBRS is poised to further minimize reward engineering while maximizing exploration and learning fidelity.