Ensure Retrainability of Primal IRL Rewards

Establish a practical method to ensure that the final reward function returned by primal inverse reinforcement learning algorithms that alternate small updates to the discriminator and the policy (such as GAIL-style approaches solving min_π max_f J(π_E, f) − J(π, f)) permits effective retraining from scratch; specifically, when the recovered reward is optimised via reinforcement learning in the original environment to completion, the resulting policy should match the expert’s performance.

Background

The paper contrasts classical dual IRL methods, which perform full policy optimisation at each iteration and thereby enable retraining guarantees, with modern primal IRL methods that take small inner-loop policy updates and are more interaction-efficient but lack retrainability guarantees. The authors observe empirically that rewards recovered by modern deep IL algorithms often fail to yield strong policies when optimised from scratch, even in the source environment.

This motivates a concrete question about ensuring retrainability of rewards learned by primal IRL without sacrificing the interaction-efficiency benefits that make these methods practical at scale.