Robust Maximum Entropy Behavior Cloning

• The paper introduces an adaptive weighting mechanism within a min–max entropy framework to robustly isolate informative demonstrations from adversarial ones.
• It employs a saddle-point optimization with feature-matching constraints to derive a Gibbs form policy, ensuring high sample efficiency.
• Empirical evaluations on Gridworld and OpenAI Gym tasks demonstrate that RM-ENT maintains high accuracy and rapid convergence despite noisy data.

Robust Maximum Entropy Behavior Cloning (RM-ENT) is a framework for imitation learning (IL) that enables robust policy learning from a set of demonstrations, some of which may be adversarial or noisy. By exploiting a min–max entropy-based objective and adaptive demonstration weights, RM-ENT automatically detects and suppresses uninformative or misleading demonstrations without requiring a simulator or additional environment interactions. The approach introduces a general saddle-point optimization scheme with feature-matching constraints to induce maximum-entropy distributions over actions, achieving strong sample efficiency and robustness to corrupted data (Hussein et al., 2021).

1. Formal Optimization Framework

RM-ENT builds upon the principle of maximum entropy imitation learning, augmenting it with an adversarial-aware weighting mechanism on demonstrations. Given a set of $D$ demonstrations $\mathcal{D} = \{d=1,\dots,D\}$, each induces an empirical state-action distribution $\tilde p(s,a|d)$. The learner aims to find a stochastic policy $\pi(a|s)$ and a set of weights $w = (w_1, \dots, w_D)$, with $w_d \in [0,1]$, $\sum_d w_d = M$ (where $M$ is the estimated number of trustworthy demonstrations).

The primary objective is a min–max (saddle-point) formulation: $$\min_{w\in\mathbb{R}^{D}}\,\max_\pi\Bigl\{-\sum_{s,a} \tilde p_w(s)\,\pi(a|s)\,\log\pi(a|s)\Bigr\}$$ subject to:

• Weighted feature matching: $$\sum_{d=1}^{D} w_d \sum_{s,a} \tilde p(s,a|d) [\pi(a|s) - \tilde\pi(a|s,d)] f_i(s,a) = 0 \qquad \forall\, i=1,\dots,n$$
• Policy normalization: $\mathcal{D} = \{d=1,\dots,D\}$0 for all $\mathcal{D} = \{d=1,\dots,D\}$1
• Demo weight constraints: $\mathcal{D} = \{d=1,\dots,D\}$2, $\mathcal{D} = \{d=1,\dots,D\}$3

Expressed in dual form over Lagrange multipliers $\mathcal{D} = \{d=1,\dots,D\}$4, the policy takes the Gibbs form: $\mathcal{D} = \{d=1,\dots,D\}$5 where $\mathcal{D} = \{d=1,\dots,D\}$6. The optimization is over the joint dual-primal objective: $\mathcal{D} = \{d=1,\dots,D\}$7 with linear constraints on $\mathcal{D} = \{d=1,\dots,D\}$8. The resulting problem is non-convex in $\mathcal{D} = \{d=1,\dots,D\}$9 and is solved using sequential quadratic programming (SQP) (Hussein et al., 2021).

2. Entropy-Based Demo Weight Mechanism and Adversarial Detection

A core feature of RM-ENT is the assignment of adaptive weights $\tilde p(s,a|d)$0 to demonstrations, governing their influence on policy induction. Each demonstration contributes a scalar

$\tilde p(s,a|d)$1

where: $\tilde p(s,a|d)$2 The min-over-$\tilde p(s,a|d)$3 step in the optimization suppresses weights on demonstrations that would force excessive entropy, as adversarial or stochastic data increases the entropy cost. A demonstration $\tilde p(s,a|d)$4 with low or negative $\tilde p(s,a|d)$5 is downweighted, with $\tilde p(s,a|d)$6 for strongly adversarial cases.

Optimization over $\tilde p(s,a|d)$7 at each step corresponds to solving a linear or quadratic program: $\tilde p(s,a|d)$8 This automated filtering suppresses adversarial or random demonstrations while retaining the influence of correct ones, enabling robust imitation learning.

3. Algorithmic Structure and Computational Outline

RM-ENT alternates between updating Lagrange multipliers $\tilde p(s,a|d)$9 for the dual (policy) variables and re-optimizing demonstration weights $\pi(a|s)$0 given the current policy. A high-level outline:

$w_d \in [0,1]$7

No environment simulator or rollouts are required after receiving the input demonstrations. The per-iteration computational cost scales favorably with the number of demonstrations $\pi(a|s)$1 (practical for $\pi(a|s)$2) (Hussein et al., 2021).

4. Theoretical Properties and Convergence

For any fixed set of weights $\pi(a|s)$3, the maximization in $\pi(a|s)$4 is strictly convex, yielding a unique maximum-entropy “Gibbs” policy $\pi(a|s)$5. The minimization over $\pi(a|s)$6 for any fixed $\pi(a|s)$7 is linear. The non-convexity of the joint problem in $\pi(a|s)$8 precludes global optimality guarantees; the SQP-based optimization converges only to local Karush–Kuhn–Tucker (KKT) points under standard smoothness and constraint qualifications. As a result, only local convergence is certified, mirroring limitations of established alternating-optimization or EM-style procedures in related imitation learning frameworks (Hussein et al., 2021).

5. Empirical Performance and Experimental Protocol

Empirical validation encompasses both tabular and control-suite tasks:

• Gridworld (5x5): Synthetic experiments with mixtures of correct, adversarial, and random demonstrations. Examples:
  • Two correct demos: $\pi(a|s)$9, 100% path-accuracy.
  • Two correct + 1 adversarial: $w = (w_1, \dots, w_D)$0, 83% accuracy.
  • Two correct + three random: $w = (w_1, \dots, w_D)$1, 92% accuracy.
• OpenAI Gym (MountainCar, Acrobot): Discrete-action, continuous-state domains. Baselines include supervised behavioral cloning (BC), maximum-entropy IRL with TRPO (“FEM”), and Game-Theoretic Apprenticeship Learning (GTAL).
  • When no adversarial data, RM-ENT performance matches BC (100% performance).
  • With one or more adversarial demonstrations, RM-ENT preserves high returns by suppressing their weights, whereas BC and IRL baselines degrade linearly with contamination.
  • Performance degrades only when adversarial demos outnumber experts, at which point RM-ENT collapses to random performance.
  • Computation time: RM-ENT converges in $w = (w_1, \dots, w_D)$2 10 seconds; IRL baselines take several minutes due to repeated simulator queries (Hussein et al., 2021).

Setting	RM-ENT Weighting	Accuracy/Return	BC/IRL Baseline
2 correct demos	$w = (w_1, \dots, w_D)$3	100%	100%
2 correct, 1 adv	$w = (w_1, \dots, w_D)$4	83%	$w = (w_1, \dots, w_D)$583%
2 correct, 3 rand	$w = (w_1, \dots, w_D)$6	92%	$w = (w_1, \dots, w_D)$792%
Gym: adv $w = (w_1, \dots, w_D)$8 expert	collapse	random-like	random-like

6. Practical Considerations and Limitations

• Sample complexity: RM-ENT requires zero environment interactions beyond provided demonstrations. In contrast, IRL baselines (e.g., TRPO-based) require hundreds of thousands of simulator steps.
• Computational efficiency: Each iteration involves:
  1. State-action pass for $w = (w_1, \dots, w_D)$9 (partition function)
  2. Per-demo $w_d \in [0,1]$0 computation
  3. QP solve in $w_d \in [0,1]$1 variables
  4. SQP-style $w_d \in [0,1]$2 update. For $w_d \in [0,1]$3 demos, overall runtime is in seconds.

• Characteristic constraints and limitations:

1. Only local optima are guaranteed due to joint non-convexity. 2. Discriminative feature design is required: features $w_d \in [0,1]$4 must distinguish correct from adversarial demonstrations. 3. Algorithm supports discrete actions only; continuous-action extensions remain an open direction. 4. The number of “trusted” demonstrations $w_d \in [0,1]$5 must be estimated, typically via cross-validation.

A plausible implication is that, while robust to adversarial data and highly sample efficient, the framework's performance hinges on feature selection and the accuracy of $w_d \in [0,1]$6. In summary, RM-ENT extends maximum-entropy behavior cloning with adaptive demo weighting, enabling policy learning that is robust to corrupted demonstrations and efficient in sample and computation, without reliance on simulators (Hussein et al., 2021).