Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching (2411.07007v2)

Abstract: In inverse reinforcement learning (IRL), an agent seeks to replicate expert demonstrations through interactions with the environment. Traditionally, IRL is treated as an adversarial game, where an adversary searches over reward models, and a learner optimizes the reward through repeated RL procedures. This game-solving approach is both computationally expensive and difficult to stabilize. In this work, we propose a novel approach to IRL by direct policy optimization: exploiting a linear factorization of the return as the inner product of successor features and a reward vector, we design an IRL algorithm by policy gradient descent on the gap between the learner and expert features. Our non-adversarial method does not require learning a reward function and can be solved seamlessly with existing actor-critic RL algorithms. Remarkably, our approach works in state-only settings without expert action labels, a setting which behavior cloning (BC) cannot solve. Empirical results demonstrate that our method learns from as few as a single expert demonstration and achieves improved performance on various control tasks.

• The paper presents a non-adversarial IRL approach called Successor Feature Matching that bypasses explicit reward modeling for direct policy optimization.
• It achieves high sample efficiency by learning effectively from minimal expert demonstrations, improving performance by an average of 16% on control tasks.
• The method supports state-only learning with adaptive feature extraction, broadening its applicability to scenarios without expert action labels.

Overview of Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

The paper under discussion presents an innovative approach to Inverse Reinforcement Learning (IRL), shifting from traditional adversarial methods to a non-adversarial framework. The authors propose Successor Feature Matching (SFM), a novel algorithm that utilizes successor features for direct policy optimization without the need to explicitly model reward functions. This approach diverges from the conventionally used min-max optimization problems characteristic of adversarial methods, which are typically computationally expensive and suffer from stabilization challenges.

The proposed SFM methodology hinges on the concept of successor features (SF), allowing for a linear factorization of the return as an inner product of these features with a reward vector. This decouples the IRL process from the need to separately learn reward functions, offering a seamless integration with existing actor-critic reinforcement learning algorithms. A notable advantage of this approach is its applicability in state-only settings, where expert action labels are unavailable, a scenario where behavior cloning (BC) often fails.

Key Contributions and Methodological Insights

1. Feature Matching through Direct Policy Optimization: SFM reduces the problem of occupancy matching in IRL to a reinforcement learning problem by leveraging the policy gradient method to minimize the difference in successor features between the learned policy and the expert. This is achieved through a straightforward Mean Squared Error (MSE) objective defined over the gap in successor features, which is tractable and does not require an adversarial setup.
2. Learning from Limited Expert Demonstrations: The algorithm demonstrates robustness even with a minimal number of expert demonstrations. Empirical results in the paper show the capability of SFM to learn effectively from as few as a single demonstration, significantly outperforming traditional methods in similar conditions.
3. Avoiding Expert Action Labels: By focusing on state-based features, SFM allows for imitation learning from data formats such as video or motion-capture, where action labels are absent. This broadens the scope of applicable scenarios for SFM compared to methods reliant on action-annotated demonstrations, like Behavior Cloning and IQ-Learn.
4. Adaptive Feature Learning and Representation: The algorithm adaptively learns the class of features necessary for successful IRL, mitigating the reliance on pre-specified reward function classes. This is implemented using unsupervised reinforcement learning techniques, adapting the learned feature representation as learning progresses.

Experimental Evaluation and Results

The paper highlights that SFM consistently outperforms baseline methods such as behavior cloning (BC), IQ-Learn, and adversarial state-only methods like Moment Matching (MM) and Generative Adversarial Imitation from Observation (GAIfO) across a suite of control tasks. In numerical comparisons, SFM is shown to achieve an average improvement of 16% on mean normalized returns across tasks from the DMControl suite.

SFM's effectiveness is not only reflected in its mean performance but also in its higher sample efficiency and learning stability, as evidenced by faster convergence rates and reduced variance in performance across different task domains. The authors also systematically evaluate the impact of different underlying policy optimizers and feature functions, demonstrating that SFM maintains performance robustness across variations.

Implications and Future Directions

The introduction of SFM marks a significant shift in IRL, transitioning from reliance on adversarial dynamics to leveraging powerful, non-adversarial RL techniques for policy optimization. This simplification may facilitate broader adoption and application, particularly in computationally constrained environments or where expert action annotations are sparse.

Future research could explore extending SFM to stochastic policy settings or integrating it with exploration mechanisms that address the intrinsic challenges of IRL in complex, dynamic environments. The adaptability of learned feature representations in SFM could also be further studied in conjunction with pretrained models to enhance efficiency in learning from demonstrations.

Overall, this paper presents a compelling advancement in IRL, showcasing the potential to streamline imitation learning processes by circumventing the adversarial paradigms and focusing on more direct, stable, and accessible methodologies.