Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

Abstract: LLM agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing a source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved. HSL then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post-training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.

• The paper presents Hindsight Supervised Learning (HSL) to transform agent rollouts into diverse, successful demonstrations via trajectory relabeling.
• It introduces action masking and demonstration reweighting to mitigate bias, achieving a 97.76% success rate on ALFWorld.
• Empirical evaluations across ALFWorld, PlanCraft, and WebShop confirm that HSL significantly enhances sample efficiency and generalization.

Hindsight Supervised Learning for LLM Agents: Relabeling Trajectories for Efficient Demonstration Mining

Motivation and Problem Statement

The deployment of LLM agents in complex, partially observable, and long-horizon environments faces a fundamental bottleneck in acquiring sufficiently rich and diverse supervision. Traditional post-training methods—primarily SFT and RL-based approaches like PPO and DPO—either require costly expert demonstrations or struggle to efficiently utilize agent-generated rollouts with sparse rewards, especially in environments where exploration is constrained. This paper presents Hindsight Supervised Learning (HSL), a framework inspired by HER in robotic RL, which post-processes agent rollouts by mining unintended but successfully achieved goals. By relabeling trajectories with the full set of goals actually achieved, HSL transforms a broader spectrum of agent experiences into successful demonstrations, offering pathway to improve agent policies via fine-tuning.

Figure 1: Left: Overview of the HSL augmentation to conventional training pipelines. Right: Example of the relabeling process, where an auxiliary LLM assigns newly achieved goals to trajectory segments.

Methodological Contribution

Relabeling for Demonstration Extraction

HSL enhances post-training pipelines by introducing three stages: (1) trajectory collection, (2) goal relabeling, and (3) learning on relabeled demonstrations. For each agent-generated trajectory, an auxiliary LLM identifies all natural language instructions corresponding to achieved goals. Unlike prior goal relabeling efforts, which focus only on final goals or failed instructions, HSL mines all intermediate goals throughout a trajectory, leading to a richer demonstration buffer.

The relabeling process comprises:

• Goal identification: Zero-shot inference by the relabeler over complete agent trajectories and observations to derive a list of achieved goals.
• Action relevance labeling: For each relabeled segment, the relabeler assigns a binary relevance tag to each action, distinguishing necessary steps for the relabeled goal from irrelevant ones.

Loss Masking and Reweighting

To mitigate demonstration bias and suboptimality in agent-generated rollouts, HSL introduces:

• Irrelevant-action masking: The SFT loss is masked for actions marked irrelevant, preventing imitation of unnecessary behaviors and tightening expert-agent discrepancy.
• Demonstration reweighting: Relabeled demonstrations are sampled according to a weight that prioritizes optimal execution and task difficulty.

The combined objective leverages both ground-truth and relabeled demonstrations, with a tunable mixture parameter, enabling stable learning while maximizing coverage from mined successes.

Theoretical Analysis

The authors provide a finite-horizon discrepancy bound between the agent and expert policies, showing that SFT on both ground-truth and relabeled demonstrations strictly reduces the gap relative to SFT alone. The bound depends critically on occupancy coverage and optimality of the hindsight expert induced by the relabeling process. Continuous, on-policy relabeling enhances this coverage, especially as the base agent improves, demonstrating the value of dynamic demonstration mining.

Experimental Evaluation

HSL is evaluated across ALFWorld, PlanCraft, and WebShop, spanning tasks with varying horizon lengths and goal diversity. HSL consistently improves both SFT and DPO baselines, with the largest gains observed on ALFWorld's long-horizon, multi-goal environment. Notably, SFT+HSL achieves a success rate of 97.76% on ALFWorld (unseen split) and outperforms SFT and DPO using only a quarter of the ground-truth demonstrations, revealing pronounced sample efficiency.

Figure 2: Sample efficiency curves illustrating HSL’s substantial performance gains on ALFWorld and WebShop with limited ground-truth demonstrations.

Comparative analyses demonstrate that the auxiliary relabeler’s use for post-hoc demonstration extraction yields markedly larger improvements over the same relabeler used for direct reasoning and action prediction. Other offline data synthesis methods, even with strong external LLMs, underperform compared to HSL’s dynamic, on-policy relabeling strategy.

Ablation and Analysis

Ablation studies show that both demonstration reweighting and action masking are integral; removing masking, in particular, leads to substantial declines with limited training data. Mining only final achieved goals yields lower performance, reflecting the value of extracting all intermediate successes.

Figure 3: Ablation studies on ALFWorld quantifying the impact of HSL components; action masking and reweighting are critical for unseen generalization.

Analysis of relabeled demonstration distribution and success rates across goal types reveals a strong correlation between mined demonstration frequency and task performance. Underrepresented goals in relabeled demos correspond to lower success rates, highlighting both the exploratory bias and potential avenues for increasing coverage. Manual validation indicates 90% accuracy in relabeling, supporting the claim that goal inference in hindsight is more reliable than direct action prediction.

Figure 4: Top: Goal-type distributions in relabeled versus ground-truth demos. Bottom: Success rate by goal type for SFT+HSL.

Practical and Theoretical Implications

Practically, HSL enables high performance and generalization on long-horizon and open-ended tasks with significantly reduced reliance on expensive expert annotations. This has direct implications for embodied, web-based, and planning agents where interaction and data collection constraints are prevalent. By leveraging LLMs' reasoning abilities for trajectory relabeling rather than direct control, HSL transforms exploration data into high-quality demonstrations, optimizing compute utilization in training.

Theoretically, the demonstration mining framework points toward a paradigm in goal-conditioned agent learning that is less dependent on explicit reward signals or environment dynamics modeling, and instead exploits retrospective reasoning. Occupancy coverage and hindsight optimality emerge as central quantities, suggesting future directions in joint adaptation of relabeling models and agents, and in diversity expansion for mined demonstrations.

Future Directions

The reliance on external, zero-shot relabelers raises questions about adaptation for harder tasks, adversarial or malicious goal mining, and open-world environments lacking predefined goal spaces. Integrating fine-grained relevance scoring, adaptive relabeler training, and unsupervised exploration strategies will be required to further scale HSL's effectiveness.

Conclusion

HSL offers a principled, sample-efficient framework for LLM agent learning by converting unintended successes in rollouts into valuable demonstrations, bolstered by relevance-driven action masking and demonstration reweighting. Empirical evidence substantiates substantial improvements in success rates and generalization, particularly on challenging, long-horizon benchmarks. The approach paves the way for broader adoption of hindsight-based supervision in agentic AI systems and suggests promising future research directions in demonstration mining and reasoning-centric agent training (2607.04235).