Model-Free Inference of Investor Preferences: A Relative Entropy IRL Approach

Abstract: We present a framework using Relative Entropy Inverse Reinforcement Learning (RE-IRL) to recover investor reward functions from observed investment actions and market conditions. Unlike traditional IRL algorithms, RE-IRL is employed to account for environments where transition probabilities are unknown or inaccessible. To address the challenge of data sparsity, we utilize a $K$-nearest neighbor approach to estimate the observed behavior policy. Furthermore, we propose a statistical testing framework to evaluate the validity and robustness of the estimated results.

• The paper introduces a model-free IRL approach that infers investor reward functions without relying on restrictive parametric utility models.
• It employs a surrogate objective and K-nearest neighbor estimation to robustly recover investor preferences from discretized state-action trajectories.
• Statistical validation shows that higher estimated rewards correlate with increased institutional investment flows, offering new insights for asset pricing.

Model-Free Recovery of Investor Preferences via Relative Entropy IRL

Problem Motivation and Context

The paper "Model-Free Inference of Investor Preferences: A Relative Entropy IRL Approach" (2604.24280) addresses the challenge of inferring latent reward functions that govern institutional investor behavior in financial markets. Traditional asset pricing relies on parametric utility assumptions—e.g., CRRA models—which introduce rigidity and often fail to explain observed phenomena such as the equity premium puzzle. Instead, the paper proposes an Inverse Reinforcement Learning (IRL) methodology tailored to real-world investment dynamics, bypassing the need for explicit transition modeling and parametric utility specification.

MDP Formulation and Data Representation

Investor decision-making is cast as a Markov Decision Process (MDP), where the state vector $\bm{s}_t \in \mathbb{R}^K$ encodes market returns, firm-specific features, and other relevant data. Actions $a_t$ represent discrete adjustments in positions, reflecting institutional demand or supply pressure. The Markov property is justified by the aggregate and impactful nature of institutional trading (market impact).

Observed investor trajectories are delineated as sequences $(\bm{s}_t, a_t)$, with each trajectory corresponding to a stock over multiple time periods. Discretization of actions and states ensures the feasibility of the trajectory space for subsequent optimization. Data sparsity and unknown transition dynamics prohibit conventional IRL techniques reliant on generative models.

Relative Entropy IRL: Theoretical Foundation

The core inference technique is Relative Entropy IRL (RE-IRL) [boularias2011relative], where the reward function is expressed as a linear combination of state features: $$R_{\bm{\theta}^*}(\bm{s}) = (\bm{\theta}^*)' \bm{s}$$. The objective is to estimate the vector $\bm{\theta}^*$ that characterizes investor preferences. The RE-IRL framework optimizes the following constrained entropy minimization:

$$\min_{P} \sum_\tau P(\tau)\ln\frac{P(\tau)}{Q(\tau)}$$

subject to matching empirical feature counts within confidence tolerances, as derived via Hoeffding’s inequality.

The solution yields a trajectory distribution proportional to the exponential of cumulative reward, weighted by observed features:

$$P^*(\tau) = \frac{Q(\tau)e^{\sum_{k}\theta^*_k s_k^\tau}}{Z(\bm{\theta}^*)}$$

where $Q(\tau)$ is a reference uniform distribution, and $Z$ is a partition function ensuring normalization.

Surrogate Objective and Estimation Algorithm

The estimation of $\bm{\theta}^*$ is reframed as maximizing the surrogate objective:

$a_t$0

where $a_t$1 are empirical averages of feature counts and $a_t$2 are confidence tolerances. The algorithm uses stochastic gradient ascent with unbiased sample estimates incorporating importance weighting between observed and reference trajectories. The key computational steps involve:

• Sample trajectory average calculations,
• K-nearest neighbor (KNN) estimation of policy likelihoods $a_t$3 for observed state-action pairs,
• Mahalanobis distance-based similarity metrics for state matching,
• Laplace-smoothed probabilities for numerical stability.

Rolling estimation procedures ensure that policy inference respects temporal data structures and avoids look-ahead bias.

Statistical Testing Framework

Recovered $a_t$4 vectors are estimated for grouped trajectory lengths $a_t$5. Statistical significance is assessed via $a_t$6-tests for feature weights, quantifying which features exert positive or negative influence on investor rewards. Furthermore, regression analyses are used to validate that stocks with higher inferred reward function values experience greater institutional investment flows.

Implications, Limitations, and Forward Outlook

The RE-IRL framework delivers a model-free, data-driven methodology for extracting investor reward structures in complex, noisy market settings. The principal practical implication is that asset pricing and investment analytics can proceed without restrictive utility assumptions or explicit modeling of market transitions. This is particularly relevant for institutional settings where the economic environment is high-dimensional and evolving.

The theoretical implication is the formal justification of linear reward structures consistent with maximum entropy and observed feature constraints. The approach also suggests potential for generalized IRL applications in economics, where agent-environment dynamics are not fully specified or accessible.

Potential limitations include the necessity of discretizing the state and action spaces, which may affect fidelity in high-resolution datasets. The reliance on KNN for policy estimation introduces local bias and may struggle with outlier or regime-shift phenomena. Further, the model assumes that observed trajectories are sufficiently representative of aggregate investor strategy.

Future developments may involve scalable nonparametric policy estimation, extension to nonlinear reward functions, or integration with generative market simulation for deeper interpretability and policy evaluation. Validation across diverse asset classes and regimes would strengthen robustness claims.

Conclusion

This paper formalizes a scalable, model-free framework for recovering institutional investor reward functions via Relative Entropy IRL, leveraging local nonparametric policy estimation and robust statistical testing. The approach circumvents the limitations of classical utility-based asset pricing and provides a rigorous route to feature-driven inference of investor preferences in empirical financial datasets (2604.24280).