Offline Inverse Reinforcement Learning

• Offline IRL is a subfield that infers reward functions solely from pre-collected expert data, bypassing the need for online environment interaction.
• It employs diverse algorithmic paradigms—such as model-based min–max optimization, policy-free bi-level optimization, and distributional methods—to ensure sample efficiency and robust imitation.
• Recent advances offer theoretical guarantees on sample complexity and reward recovery, demonstrating practical success in real-world, safety-critical applications.

Offline Inverse Reinforcement Learning (Offline IRL) is a subfield of IRL focused on inferring a reward function and imitating expert policies using only fixed, pre-collected datasets—without any new online interaction with the environment. This contrasts with classical IRL, which typically assumes the ability to iteratively query or interact with the environment. Offline IRL addresses the practical necessity in domains where environment sampling is costly, unethical, or infeasible, such as healthcare, scientific experimentation, and real-world robotics. Recent theoretical advances have established the statistical and computational feasibility of offline IRL, and several algorithmic paradigms now achieve sample-efficient, robust imitation with strong empirical and occasionally statistical guarantees.

1. Problem Formulation and Foundational Models

Offline IRL is defined over a Markov Decision Process (MDP) $M = (\mathcal S, \mathcal A, P, r_*, \mu, \gamma)$, where both the transition dynamics $P$ and the (unknown) expert reward $r_*$ are not accessible to the learner. The agent is provided with one or more offline datasets: (a) a main dataset $\mathcal D_E = \{(s_i,a_i,s'_i)\}$ consisting of expert demonstrator transitions, and often (b) auxiliary datasets with non-expert or exploratory transitions. The core objective is to infer a reward function $r_\phi(s, a)$ (or, more generally, a feasible reward set) such that the induced optimal or soft-optimal policy explains the observed expert behavior and, as a downstream task, enables learning a policy that attains expert-level or superior performance in the true environment, all without further interaction (Ahn et al., 17 Oct 2025, Park, 27 Nov 2025, Lazzati et al., 2024).

At the mathematical core of many offline IRL methods is the maximum-entropy IRL formalism, which seeks to solve

$$\min_{r}\;\Bigl[\,\max_{\pi}\{\alpha H(\pi) + \mathbb E_{(s,a)\sim\rho^\pi}[r(s,a)]\} - \mathbb E_{(s,a)\sim\rho^E}[r(s,a)] + \psi(r)\,\Bigr],$$

with $H(\pi)$ the discounted causal entropy of the policy and $\psi(r)$ a regularizer. Some approaches eschew explicit policy learning via direct maximum-likelihood (e.g., BiCQL-ML) (Park, 27 Nov 2025), or by feasible-reward set estimation (Lazzati et al., 2024), or even entirely eliminate RL subroutines via “reset-based” occupancy-matching (Swamy et al., 2023).

2. Algorithmic Paradigms and Solution Methods

Offline IRL algorithms can be organized into several main paradigms:

• Model-based min–max optimization: OffSim (Ahn et al., 17 Oct 2025) jointly learns a high-entropy transition model and an IRL reward from expert data and uses the learned model as a simulator for fully offline policy learning. The transition model is trained for high entropy and reward-maximization, while the reward function discriminates between real and simulated transitions. In OffSim$^{+}$, multiple datasets can be incorporated by enforcing a margin between the expected expert and suboptimal rewards.
• Policy-free bi-level optimization: BiCQL-ML (Park, 27 Nov 2025) alternates between conservative Q-learning and maximum-likelihood reward fitting, avoiding explicit policy learning. The reward is updated to maximize expert Q-value and penalize out-of-distribution actions, while the Q-function is made conservative via CQL regularization.
• Distributional IRL: Distributional approaches (Wu et al., 3 Oct 2025) infer not only the mean reward but the entire conditional reward distribution, using first-order stochastic dominance (FSD) criteria and spectral risk measures (distortion risk measures, DRM) for risk-sensitive imitation.
• Preference-based IRL: TROFI (Sestini et al., 27 Jun 2025) applies trajectory-ranking (via T-REX) to enable reward learning from partial human trajectory preferences, followed by standard offline RL on the re-labeled dataset.
• Pessimistic Offline RL as an inner loop: Several approaches formalize and exploit the link between pessimistic offline RL and IRL (Wu et al., 2024, Zhao et al., 2023), plugging any conservative offline RL algorithm into the IRL inner loop to ensure sample-efficient, robust reward recovery.
• Set-valued estimation: IRLO and PIRLO (Lazzati et al., 2024) estimate feasible reward sets rather than single reward functions, providing inclusion-monotonic bounds and negative results on identifiability in limited coverage scenarios.

Algorithmic techniques include min–max optimization (OffSim), bi-level stochastic approximation (BiCQL-ML), quantile regression (Distributional IRL), and adversarial data augmentation via GANs (CAMERON (Jarboui et al., 2021)). Model selection and parameter tuning, regularizer choice, and explicit penalties for uncertainty in model-based methods (e.g., ensemble spread, margin constraints) play crucial roles.

3. Theoretical Guarantees and Sample Complexity

Offline IRL is subject to fundamental statistical constraints set by dataset coverage (concentrability), reward realizability, and epistemic uncertainty about both the reward and transition models. Recent work has proved near-optimal sample-complexity bounds (Zhao et al., 2023, Lazzati et al., 2024):

• For tabular MDPs (finite $S, A, H$), the number of required offline trajectories to ensure final reward error $P$0 scales as $P$1, with $P$2 the concentrability constant quantifying data coverage.
• Reward recovery and downstream policy optimality for pessimistic reward learning are guaranteed when the learned mapping is inclusion-monotonic and the pseudometric (Hausdorff) distance to the true feasible set is bounded.
• Plugging pessimistic offline RL into IRL inner loops yields immediate end-to-end sample-complexity reductions: if the offline RL algorithm arrives within $P$3 of optimal value with high probability, offline IRL inherits this guarantee up to an additive statistical term (Wu et al., 2024).

Distributional and Bayesian methods (distributional IRL, Bayesian robust IRL) also provide robustness and generalization in the presence of stochasticity or model/dynamics gap, with explicit risk-sensitive bounds (Wu et al., 3 Oct 2025, Wei et al., 2023).

4. Practical Implementations and Empirical Performance

Offline IRL methods have demonstrated superior performance across standard continuous-control benchmarks and in application-focused domains.

• OffSim and OffSim$P$4 exceed prior offline IRL methods (CLARE, ML-IRL, IQ-Learn, behavioral cloning) on D4RL MuJoCo tasks, often surpassing even the expert policy return, with stable performance across various datasets and margin values (Ahn et al., 17 Oct 2025).
• BiCQL-ML achieves not only higher final returns but faster convergence and lower variance, even in the low-data regime, compared to Behavioral Cloning and ValueDICE (Park, 27 Nov 2025).
• Distributional IRL recovers both mean and higher-order reward/return statistics on synthetic, biological, and risk-sensitive benchmarks, enabling risk-aware policy synthesis and supporting neuroscience analysis tasks (Wu et al., 3 Oct 2025).
• TROFI achieves or surpasses ground-truth reward baselines with preference learning over only a small subset of ranked trajectories (Sestini et al., 27 Jun 2025).
• Model-based methods with GAN augmentation (CAMERON) combine offline occupancy estimation and conservative policy optimization to outperform reinforced imitation and reward shaping baselines (Jarboui et al., 2021).
• Healthcare applications utilize causal Transformer-based cost learning (Constraint Transformer) with generative world models for safe, history-aware policy inference and constraint recovery, showing strong empirical correlation with mortality and elimination of unsafe actions (Fang et al., 2024).
• Diffusion-model-based policies (KANDI) provide generative, stable action refinement in clinical data settings and match or outperform classic offline RL on D4RL benchmarks (Liu et al., 22 Sep 2025).

Empirical ablations highlight the necessity of high-entropy/preventing overfitting in transition models, the critical importance of reward identifiability and model quality, and reward generalization across seen/unseen states.

5. Open Challenges, Limitations, and Future Directions

Several intrinsic challenges and limitations are recognized:

• Identifiability: Without sufficient coverage or control over the behavior policy, many reward functions can fit limited-data expert demonstrations. Without behavioral policy coverage over non-expert actions, offline IRL collapses to behavioral cloning (Lazzati et al., 2024). The minimal sample coverage required for feasible set estimation has been characterized; in practice, ensuring adequate exploratory data remains difficult.
• Generalization and dynamics uncertainty: Learned models may fail to generalize to out-of-distribution states; uncertainty penalties, robust Bayesian priors, and min–max optimization address but do not entirely resolve these issues (Wei et al., 2023, Zeng et al., 2023).
• Computational costs: Many state-of-the-art offline IRL methods (OffSim, ML-IRL, Distributional IRL) incur significant computational overhead due to the need for model ensemble training, inner-loop optimization, or adversarial data augmentation (GANs).
• Function approximation: The extension of theoretically sound offline IRL algorithms to high-dimensional state-action spaces with neural function approximation is ongoing, with new concentration and uniform convergence analyses required (Wu et al., 2024, Lazzati et al., 2024).
• Transfer and robustness: Statistically guaranteed transfer of learned rewards to new target environments has been demonstrated under transferability/concentrability assumptions, but analysis of the limits of transfer, and lower bounds, is an active research area (Zhao et al., 2023).
• Reward ambiguity and risk: Distributional IRL and set-valued estimation (Wu et al., 3 Oct 2025, Lazzati et al., 2024) address the ill-posedness of reward inference by targeting feasible reward sets or return distributions.

Emerging directions include adaptive hybrid offline–online update schemes, history-aware constraints (as in healthcare), integration of human preference feedback, and principled Bayesian uncertainty quantification.

6. Representative Algorithmic Strategies: Comparative Table

Algorithm/Class	Key Feature(s)	Empirical Setting
OffSim	High-entropy model-based min–max	D4RL continuous control
BiCQL-ML	Policy-free bi-level CQL objective	MuJoCo offline, D4RL
Distributional IRL	FSD constraints, risk-sensitive	Control, dopamine data
TROFI	Trajectory-ranking, preference IRL	Video game, MuJoCo
CAMERON	GAN-based occupancy augmentation	OpenAI Gym
Constraint Transformer	History-attentive constraints, offline RL	Healthcare
PIRLO/IRLO	Feasible-reward-set estimation, inclusion monotonicity	Tabular

This table documents only strategies and settings explicitly described in the primary arXiv papers (Ahn et al., 17 Oct 2025, Park, 27 Nov 2025, Wu et al., 3 Oct 2025, Sestini et al., 27 Jun 2025, Jarboui et al., 2021, Fang et al., 2024, Lazzati et al., 2024).

7. Conclusion and Perspectives

Offline Inverse Reinforcement Learning is now a mature research branch with well-posed formulations, provably efficient algorithms, and robust empirical validation in challenging continuous-control and real-world environments. Advances in pessimistic offline RL, model-based min–max learning, reward set estimation, and distribution-aware methodologies fundamentally expand the applicability of IRL to safety-critical, batch-only application domains. Open problems remain in scalability, identifiability under limited coverage, generalization to rich observation spaces, and rigorous transfer across environment classes. Ongoing work continues to strengthen both the foundational and practical contributions of Offline IRL, propelling its adoption across AI, robotics, and decision-critical applications.