Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair

Abstract: Learning robot control policies from demonstrations is a powerful paradigm, yet real-world data is often suboptimal, noisy, or otherwise imperfect, posing significant challenges for imitation and reinforcement learning. In this work, we present a formal framework that leverages Temporal Behavior Trees (TBT), an extension of Signal Temporal Logic (STL) with Behavior Tree semantics, to repair suboptimal trajectories prior to their use in downstream policy learning. Given demonstrations that violate a TBT specification, a model-based repair algorithm corrects trajectory segments to satisfy the formal constraints, yielding a dataset that is both logically consistent and interpretable. The repaired trajectories are then used to extract potential functions that shape the reward signal for reinforcement learning, guiding the agent toward task-consistent regions of the state space without requiring knowledge of the agent's kinematic model. We demonstrate the effectiveness of this framework on discrete grid-world navigation and continuous single and multi-agent reach-avoid tasks, highlighting its potential for data-efficient robot learning in settings where high-quality demonstrations cannot be assumed.

• The paper introduces a formal framework that repairs flawed trajectories using Temporal Behavior Trees for RL policy synthesis.
• The paper employs an MILP-based optimization to minimally adjust demonstration state sequences while satisfying logical task constraints.
• The paper demonstrates faster convergence and improved safety in RL agents by integrating potential-based reward shaping from repaired trajectories.

Learning from Imperfect Demonstrations with TBT-Guided Trajectory Repair

Introduction and Motivation

The paper "Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair" (2604.04225) proposes a principled, formal-methods-based framework for policy synthesis in RL settings where demonstration data is noisy, suboptimal, and may violate task constraints. Unlike prior works relying on demonstration optimality, rewarding only Boolean satisfaction, or requiring task-specific reward engineering, this approach leverages the formalism of Temporal Behavior Trees (TBTs) to systematically repair demonstration trajectories and extract instructional potential functions for RL reward shaping. The framework thus unifies logic-based supervision with data-driven RL without requiring access to agent actions or dynamics in the demonstrations.

Figure 1: Overview of the framework: TBT specifications guide both repair of suboptimal demonstrations and construction of TBT-based monitoring and reward potentials for RL.

Framework and Methodology

Temporal Behavior Trees as Task Specifications

TBTs extend Signal Temporal Logic (STL) by structuring specifications as hierarchical trees with compositional constructs (Sequence, Parallel, Fallback) and STL-leaf nodes for predicates. This enables succinct expression and segmentation of complex temporal and multi-agent tasks, with both Boolean and quantitative (robustness) semantics facilitating downstream repair and reward shaping.

Demonstration Repair Module

Given an imperfect dataset $\mathcal{D}$ and a TBT $\mathcal{T}$, the system performs model-based repair by solving an optimization problem for each demonstration that violates the specification. The repaired trajectory $\tau_R$ minimally deviates from the original, subject to both kinematic model constraints and logical satisfaction of $\mathcal{T}$. The core repair formulation is an MILP; practical computation leverages landmark-based strategies to reduce to LPs, supported by robustness-based heuristic landmark selection.

This repair mechanism operates purely on state sequences, does not require action or reward information, and the repaired dataset $\mathcal{D}_R$ is guaranteed to satisfy the logical task constraints.

Potential-Based Reward Shaping

From the set of repaired state sequences, the framework extracts two key pose-dependent potentials:

• Proximity Potential: Encourages spatial closeness to repaired demonstration states.
• Safety Potential: Penalizes deviations exceeding a threshold from repaired states to define implicit safety boundaries.

The total shaping signal is a weighted sum of these potentials, efficiently queried at runtime via KD-tree indexing of repaired poses. This potential-based shaping integrates seamlessly with standard RL algorithms, ensuring invariance of the optimal policy set while incorporating dense, specification-respecting feedback. The approach is kinematics-agnostic since it acts solely in pose space, enabling direct transfer of shaped rewards to agents with diverse, even unknown, dynamics.

Experimental Results

Grid-World Navigation

In a $10 \times 10$ deterministic grid-world with randomly placed obstacles, task specifications require sequential goal-reaching and persistent obstacle avoidance. Demonstrations, including deliberately suboptimal ones, are repaired via the TBT repair module. Extracted potential function heatmaps reveal explicit spatial structuring, aligning with task corridors and obstacles.

RL agents trained with the proposed TBT-based potential shaping converge faster and to higher asymptotic performance than PPO trained with sparse rewards, especially as task scaling worsens reward sparsity.

Figure 2: (a) Start/goal positions and obstacles; (b) Demo trajectories violating safety; (c) Heatmaps of extracted potential functions; (d) Comparison of episodic rewards and success rates.

Safety-Gymnasium Mobile Navigation

Single-Agent

An Ackermann-kinematic agent must reach a goal while avoiding hard and soft obstacles in randomized map configurations. Repaired demonstrations (using Dubins model) are used to shape the reward. Both TBT-based and expert-designed reward pipelines achieve equivalent $100\%$ success, but the proposed method demonstrates marginally faster convergence.

Multi-Agent

Two unicycle-kinematic agents must each reach distinct goals while avoiding obstacles—a combinatorial parallel specification compactly encoded by TBTs. The TBT-guided repair and potential extraction generalize seamlessly to this multi-agent, higher-dimensional observation space. Notably, the RL policy with TBT-shaped rewards achieves $100\%$ task success with a 47% lower obstacle cost rate compared to the expert-designed baseline.

Figure 3: Single-agent task layout in Safety-Gymnasium showing obstacles and goal configuration.

Figure 4: Performance comparisons between the proposed TBT-guided framework and RL baselines in Safety-Gymnasium reach-avoid tasks.

Implications, Limitations, and Future Directions

The paper robustly demonstrates that TBT-guided trajectory repair provides a formally certified, interpretable bridge from imperfect human demonstrations to RL-compatible reward functions, with significant practical advantages:

• Data-efficient RL: The dense, logically consistent shaping signals accelerate convergence and improve safety, especially in sparse or high-dimensional scenarios.
• Kinematic model independence: The method generalizes across a spectrum of robot dynamics without tailored reward function engineering.
• Multi-agent scalability: TBT compositionality avoids specification blowup typical in automata-based approaches.

Several limitations are identified:

• Offline repair dependency: The MILP formulation and need for a model restrict real-time deployment and pure model-free settings.
• Potential imprecision: In highly randomized environments, pose-based potentials from a limited repaired dataset may transfer suboptimally, motivating future methods conditioning on current observations or dynamically updating the potential.
• Specification knowledge: The current approach assumes provision of a correct TBT; next-step research could perform inference of task specifications from noisy demonstrations.

The framework sets a foundation for integrating formal methods and RL, with promising applications in domains requiring both flexible learning and rigorous safety or task satisfaction guarantees.

Conclusion

The paper presents a comprehensive methodology for leveraging TBT-guided repair of imperfect demonstrations to synthesize RL policies that are both data-efficient and specification-satisfying. Empirical results on both discrete and continuous robot navigation tasks validate the effectiveness of the approach, particularly its ability to outperform hand-engineered reward baselines in safety-critical settings, while supporting principled transfer across agent models. This strongly suggests that logical repair and potential-based shaping offer a robust paradigm for RL from real-world, imperfect data and merit further investigation in adaptive and online extensions.