Learning More from Less: Reinforcement Learning from Hindsight

Abstract: Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior. Such a failure, however, is a success at a different task. We present Learning from Hindsight (LfH), which brings hindsight relabeling to RL post-training of VLAs by scoring failed rollouts against the tasks they actually achieved. A single vision-LLM relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories. On out-of-distribution LIBERO-PRO tasks, where standard RL improves only slowly, LfH achieves $5\times$ improvement in sample efficiency, and outperforms a dense progress-reward baseline. The gains hold across VLA backbones and on a physical Franka robot.

• The paper introduces the Learning from Hindsight (LfH) framework, which repurposes failed rollouts by using a vision-language model to generate alternative supervisory signals.
• By relabeling trajectories with hindsight instructions, LfH improves sample efficiency by 5× over standard GRPO and outperforms dense reward shaping methods.
• Empirical evaluations on simulated and physical robotic tasks demonstrate that LfH enhances data utilization and promotes diverse behavioral exploration.

Reinforcement Learning from Hindsight for Vision-Language-Action Models

Motivation and Problem Setting

Autonomous robotic systems powered by vision-language-action (VLA) models have achieved significant advances in open-ended manipulation. However, RL fine-tuning of these systems is severely constrained by the cost of acquiring environment rollouts, which are slow and expensive to collect on physical robots. When reward feedback is sparse (i.e., the agent is only rewarded for full completion of a language instruction), the overwhelming majority of trajectories early in training are failures, providing no learning signal under standard RL objectives. Importantly, many of these "failures" correspond to meaningful behaviors that succeed at alternative tasks.

The central hypothesis of "Learning More from Less: Reinforcement Learning from Hindsight" (2607.09042) is that failed rollouts in sparse-reward VLA RL can and should be mined for alternative supervisory signal via automated hindsight relabeling. This approach leverages the compositional generalization capacity of VLA policies: if a robot, commanded to "close the microwave," instead picks up a cup, this trajectory is a perfect demonstration for "pick up the cup." This stands in contrast to reward shaping methods, which densify feedback for fixed tasks but do not change the assignment of supervision itself.

Method: Learning from Hindsight (LfH)

The proposed Learning from Hindsight (LfH) framework augments standard RL fine-tuning for VLAs by deriving additional training targets from failed rollouts via a vision-LLM (VLM)-driven relabeling mechanism. LfH operates on the following core insight: each collected trajectory can serve as positive supervision, not for the commanded instruction, but for whichever alternative instruction it actually fulfills.

Operational Pipeline:

• Standard RL (implemented via Group Relative Policy Optimization; GRPO) collects batches of rollouts for each instruction and computes the batch-wise binary success rewards.
• Groups where all rollouts fail (low signal under the commanded instruction) are the primary target of LfH.
• For such failed groups, a pretrained VLM inspects an anchor trajectory and automatically relabels it with a "hindsight instruction"—a language description of what behavior was actually achieved.
• The VLM then evaluates all group members under this new instruction, assigning relabeled rewards (success, ambiguity, or failure).
• Both the original and hindsight-relabeled groups are used as RL optimization targets: the former provides direct alignment with the commanded task distribution, while the latter unlocks learning signal from otherwise discarded interactions.

This process is illustrated in the conceptual overview below:

Figure 1: A trajectory that fails its original instruction is relabeled in language by a VLM, augmenting supervision for VLA fine-tuning, and turning failures into positive examples for alternative tasks.

By harnessing language-relabeled hindsight, LfH scales the amount of actionable feedback per environment interaction, increases data efficiency, and drives progress even from highly suboptimal policies.

Empirical Evaluation and Results

Sample Efficiency Gains

Experimental benchmarks focus on out-of-distribution tasks from the LIBERO-PRO suite, where initial policies exhibit near-zero success rates under sparse reward. Key findings include:

• Retention of Learning Signal: Standard GRPO discards 80%-100% of experience batches due to the absence of successful trajectories. LfH recovers and re-labels ~80% of these groups, providing alternative supervision in place of null gradients.
• Acceleration of Learning: LfH achieves the same final success rate on OOD tasks in approximately $1/5$ the number of training steps required by baseline GRPO; i.e., a 5× improvement in sample efficiency.
• Superiority to Dense Reward Shaping: LfH outperforms GRPO augmented with RoboMETER dense progress reward models. This demonstrates that simply providing denser rewards for the original task is less effective than altering the target of supervision via relabeling for VLA RL in the sparse/low-success regime.

Figure 2: Side-by-side training curves for LfH vs. GRPO, showing dramatically improved sample efficiency and higher final success rate for LfH across OOD tasks.

Robustness and Generalizability

LfH's benefits are verified across different VLA policy architectures, including GR00T and OpenVLA-OFT, with consistent sample-efficiency gains. The framework seamlessly transfers to physical-robot experiments (Franka FR3 platform), demonstrating substantial improvements in real-world online RL from binary human feedback.

Mechanism Analysis

Ablation studies confirm that the gains from LfH cannot be attributed to:

• Instruction augmentation alone (simply rephrasing prompts).
• Reward relabeling alone (relabeling rewards for the same instruction).
• Artificially injecting variance (random reward perturbations).

The unique advantage of LfH derives from coupling semantically meaningful instruction relabeling with corresponding reward evaluation on diverse behaviors, as formalized in the hindsight policy objective.

Diversity analysis indicates that LfH not only leverages but also encourages more varied behavioral exploration during RL. By mining alternative tasks, it mitigates the collapse to repetitive non-successful strategies, a common failure mode of sparse-reward RL.

Figure 3: As action sampling noise—and thus trajectory diversity—increases, LfH uniquely benefits from this diversity, converting more failures into informative learning signals, while standard RL does not.

Practical and Theoretical Implications

Data Utilization and RL for VLA Policies

LfH concretely increases the "bits per trial" extracted from each robot-environment interaction, addressing the single greatest cost in physical RL loops: data inefficiency. By associating each rollout with one or several instructions post hoc, LfH aligns with the inductive priors of pretrained VLA models, which are highly compositional and instruction-conditional, and exploits their natural extrapolation in language space.

Practically, LfH could reduce the amount of real-world data necessary for generalization, a critical factor when scaling VLA deployment on embodied platforms.

Limitations and Future Directions

The principal limitations of LfH are:

• Dependence on the diversity and interpretability of failed policy behaviors. Repetitive or uninformative failures cannot be relabeled meaningfully.
• Reliance on the accuracy and grounding ability of the VLM for instruction generation and reward evaluation. Noisy relabels or hallucinations can introduce suboptimal gradients.

Improvements in VLM reliability, enhanced filtering, human-in-the-loop verification, and adaptive group selection policies are proposed as further research. Moreover, LfH's efficacy in more temporally extended, multi-stage tasks and its integration with broader offline data regimes require exploration.

Figure 4: Qualitative examples where LfH relabels meaningful failures as alternative tasks or filters uninformative rollouts, demonstrating the necessity of robust VLM-driven relabeling for generalization.

Conclusion

LfH represents a principled augmentation to RL for VLA models, exploiting the dual structure of language and sensory experience. By transforming failed interactions into additional supervision via VLM-based relabeling, LfH drives dramatic gains in sample efficiency and task generalization for robotic manipulation, both in simulation and on physical platforms. The approach provides a scalable mechanism for unlocking more information per collected rollout—a key advance for practical RL in data-constrained, embodied environments.

The methodological paradigm of hindsight relabeling via VLMs, as advanced in this work, is likely to shape future developments in sample-efficient RL for grounded, instruction-following agents.