Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models (2512.24407v1)

Abstract: Inverse reinforcement learning (IRL) and dynamic discrete choice (DDC) models explain sequential decision-making by recovering reward functions that rationalize observed behavior. Flexible IRL methods typically rely on machine learning but provide no guarantees for valid inference, while classical DDC approaches impose restrictive parametric specifications and often require repeated dynamic programming. We develop a semiparametric framework for debiased inverse reinforcement learning that yields statistically efficient inference for a broad class of reward-dependent functionals in maximum entropy IRL and Gumbel-shock DDC models. We show that the log-behavior policy acts as a pseudo-reward that point-identifies policy value differences and, under a simple normalization, the reward itself. We then formalize these targets, including policy values under known and counterfactual softmax policies and functionals of the normalized reward, as smooth functionals of the behavior policy and transition kernel, establish pathwise differentiability, and derive their efficient influence functions. Building on this characterization, we construct automatic debiased machine-learning estimators that allow flexible nonparametric estimation of nuisance components while achieving $\sqrt{n}$-consistency, asymptotic normality, and semiparametric efficiency. Our framework extends classical inference for DDC models to nonparametric rewards and modern machine-learning tools, providing a unified and computationally tractable approach to statistical inference in IRL.

• The paper demonstrates that reward-dependent functionals in softmax IRL and DDC models can be identified and efficiently estimated using semiparametric methods.
• It derives explicit efficient influence functions and constructs debiased machine learning estimators that achieve nonparametric efficiency for policy evaluation.
• The proposed approach bypasses nested dynamic programming, offering scalable and robust estimation for counterfactual policy evaluation and structural analysis.

Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models

Motivation and Context

Inverse reinforcement learning (IRL) and dynamic discrete choice (DDC) models are foundational for understanding sequential decision-making in domains where only behavioral data is observed. The central statistical challenge is to infer the reward mechanisms underlying observed choices, which enables counterfactual policy evaluation and behavioral analysis. Existing methods are bifurcated: classical econometric approaches (DDC) offer parametric inference with restrictive assumptions and substantial computational burden, while recent IRL methods provide flexibility but lack formal inferential guarantees. Notably, both paradigms converge on the same mathematical structure—agents implement (possibly normalized) softmax (entropy-regularized) policies—but reward identifiability is generally partial.

This work introduces a semiparametric inferential framework for IRL and DDC models under the softmax (Gumbel-shock) structure. The authors formalize statistical identification of reward-dependent functionals (policy values, value differences, functionals under normalized rewards), characterize their efficient influence functions, and propose debiased machine learning (DML) estimators with nonparametric efficiency guarantees.

Key Contributions

1. Identification in Maximum-Entropy IRL and Gumbel DDC

The authors demonstrate that, in softmax IRL and Gumbel DDC, a broad class of reward-dependent functionals can be written as smooth functionals of the log-behavior policy and the transition kernel. While the reward is only identified up to potential-based shaping, policy value differences and functionals of normalized rewards are point-identified. Explicitly, the log of the behavior policy ("pseudo-reward") encodes all information available from the observed process.

They formalize these identification results under weak modeling assumptions, extending to counterfactual evaluations (including alternative entropy regularizations and environment changes).

2. Pathwise Differentiability and Semiparametric Efficiency Bounds

For these identified functionals, the authors establish pathwise differentiability and derive the efficient influence functions (EIFs) in the nonparametric (and semiparametric) model. The differentiability analysis leverages the structure of the soft Bellman equation and inverse operators on the space of reward functions and transition kernels. This yields explicit EIFs that, in many cases, generalize canonical influence functions in off-policy evaluation and doubly robust RL.

3. Debiased Automatic Machine Learning Estimation

Building on the semiparametric theory, the paper constructs automatic debiased estimators for general functionals. These estimators admit flexible, nonparametric estimation of high-dimensional nuisance components (e.g., behavior policy, transition kernel, $Q$-functions, occupancy ratios) via modern ML, while maintaining $\sqrt{n}$-consistency, asymptotic normality, and attain the semiparametric lower variance bound. This dramatically generalizes classical GMM or value/density-matching approaches and avoids numerical dynamic programming or simulation-based inference.

4. Efficient Inference for Core IRL/DDC Quantities

The framework is instantiated for:

• Fixed policy values: EIFs and efficient estimators for average value of arbitrary target policies.
• Soft-optimal (entropy-regularized) policy values: treatment of counterfactual entropy parameters, identifying both values and advantage-weighted occupancy measures.
• Normalized rewards policy values: inference on structural and policy parameters under reward normalizations, integrating standard econometric normalizations as special cases.

5. Practical Algorithmic Implications

Estimators require only standard supervised learning for behavior policy and transition models, plus fitted $Q$-iteration or regression-based $Q$- and value-function estimation. Occupancy ratios can be estimated via minimax or regression-based dual formulations from recent RL literature. The estimation procedure avoids nested dynamic programming, increases robustness to misspecification, and is straightforward to implement with modern ML toolkits.

Numerical Results and Claims

The paper provides rigorous asymptotic guarantees for the proposed estimators, with precise characterization of the error decomposition and sufficient conditions under which DML estimators achieve nonparametric efficiency. The theory covers scenarios with potentially slow convergence of high-dimensional nuisance estimators, showing that as long as their product rates satisfy $o(n^{-1/2})$, efficiency is retained.

Concrete efficiency bounds are provided for all considered targets, and the conditions for identification, regularity, and estimator validity are detailed.

Notable Numerical Claims:

• AutoDML estimators achieve nonparametric efficiency bounds for value functionals under essentially arbitrary nonparametric estimation of nuisances.
• Efficient influence functions are computed explicitly for policy evaluation objectives under both behavior and counterfactual settings, as well as under various normalization schemes.
• Temporal difference-based EIFs subsume classical doubly robust estimators in off-policy evaluation but extend to the IRL/DDC regime where the reward is latent.

Theoretical Implications

The analysis connects the econometric identification structure of DDC models (potential-based shaping, normalization constraints) with the functional-analytic identification structure of modern IRL. It formally justifies the use of flexible machine learning methods for reward and value inference, provided debiasing corrections are applied.

The general methodology provides a blueprint for developing semiparametric inference in RL, IRL, and economic choice models, with potential applications to settings beyond Gumbel-shock/softmax models (e.g., alternative noise models or other optimality rationalizations).

Practical Implications

• Computational efficiency: The proposed approach sidesteps repeated or nested dynamic programming or costly simulation, enabling routine application at scale with deep learning.
• Policy evaluation and selection: Practitioners can reliably quantify uncertainty and construct confidence intervals for counterfactual or proposed policies, even when using highly flexible behavioral models.
• Structural estimation: The framework supports robust inference for structural parameters in economic DDC, including cases with nonparametric utility specifications and heteroskedasticity.

Limitations and Extensions

• The identification and efficiency results hinge on the softmax (Gumbel-shock) form, which may not fully capture agent stochasticity in some domains. The extension to general shock distributions, alternate regularizers, or multi-agent games is highlighted as a promising direction.
• The normalization approach is linear-policy based; extensions to more complex constraints (affine or nonlinear normalizations) are anticipated to be feasible within the developed machinery.
• The analysis is primarily for infinite-horizon, homogeneous MDPs; adaptation to nonstationary, finite-horizon settings or dependent transition data remains for future work.

Outlook for Future Research

• Generalized IRL models: Applying semiparametric inference methods to broader classes of behavioral models and heteroskedastic shock distributions.
• Adaptive normalization: Automated selection or learning of normalization constraints to improve interpretability and identification strength.
• Causal inference for RL: Integration with approaches to off-policy evaluation and causal identification in the presence of unmeasured confounding.
• Scalable computation: Empirical evaluation and scaling of DML estimators to high-dimensional, real-world sequential decision problems.

Conclusion

This work bridges a significant methodological gap between classical econometric inference and modern, ML-based IRL approaches. By formalizing and solving the semiparametric inference problem for softmax IRL and DDC models, the authors deliver both broad practical tools and foundational theoretical advances. The resulting estimators and influence function theory provide a template for efficient, robust, and interpretable behavioral modeling in a wide array of sequential decision domains.

Reference:

Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models (2512.24407)