Learning the Preferences of a Learning Agent

Abstract: For AI systems to be useful to humans, they must understand and act in accordance with our values and preferences. Since specifying preferences is a hard task, inverse reinforcement learning (IRL) aims to develop methods that allow for inferring preferences from observed behavior. However, IRL assumes the human to be approximately optimal. This is a big limitation in cases where the human themselves may be learning to act optimally in an environment. In this paper, we formalize the problem of learning the preferences of a learning agent: a predictor observes a learner acting online and tries to infer the underlying reward function being (initially suboptimally) optimized by the learner. We model the learner as either being no-regret, or as converging to an optimal Boltzmann policy over time. In each of these settings, we establish theoretical guarantees for various preference learning algorithms, or otherwise show that such guarantees are impossible.

• The paper demonstrates that recovering full reward structures is infeasible for no-regret learners, proving linear cumulative error in norm-based recovery.
• The paper establishes sublinear error bounds for Boltzmann learners using an averaging strategy and KL-divergence metrics to formalize reward inference.
• The paper highlights practical implications for interactive systems, emphasizing the need for explicit exploration policies to model evolving user preferences accurately.

Learning the Preferences of a Learning Agent

Problem Formulation and Motivation

The paper "Learning the Preferences of a Learning Agent" (2605.09217) presents a formal and comprehensive treatment of preference inference in scenarios where the agent whose preferences are being learned is itself non-stationary—specifically, an agent undergoing learning. Conventional approaches in Inverse Reinforcement Learning (IRL) assume expert trajectories are approximately optimal, an assumption that is violated when the underlying agent (e.g., a human user or an RL agent) is still learning. This work systematically defines the statistical and computational boundaries of preference inference in this more general and realistic setting.

By modeling the learner as either (i) a no-regret agent or (ii) a Boltzmann-rational agent whose policies converge toward optimality over time, the authors address the minimal assumptions necessary for extracting reward functions from observations of changing, initially suboptimal behavior. The results have direct implications for interactive systems (e.g., recommender systems, human-in-the-loop RL, AI assistants) where user preferences or reward functions must be inferred online despite user learning and exploration.

Formal Models and Theoretical Results

Learner Models

The analysis distinguishes two key learner models:

• No-Regret Learner: The learner’s policy ensures vanishing average regret as $T \to \infty$, but the policy may rapidly converge on a single action (exploiting), offering minimal actionable information about the reward landscape.
• Boltzmann-Rational Learner: The learner's action distribution at each time is Boltzmann with respect to their current estimated reward (stateless) or Q-function (stateful), and these estimates converge toward the ground truth. This model preserves structured (though suboptimal) exploration, allowing inference over the reward or Q-function.

Evaluation Metrics

Preference inference is evaluated using three metrics of increasing stringency:

1. Best-Response Distance: Measures policy optimality alignment rather than reward structure recovery.
2. KL Divergence (Boltzmann Policy Distance): Quantifies the gap between the Boltzmann policies induced by predicted and true rewards.
3. Norm-Based Distances ($\ell_2$, $\ell_\infty$): Strongest form, requiring reconstruction of the full reward structure.

Negative and Positive Results

• For no-regret learners, the paper proves that norm-based recovery is information-theoretically impossible—linear cumulative error is unavoidable in both $\ell_2$ and $\ell_\infty$ norms. KL divergence minimization is analogously unattainable. The only achievable guarantee is on the optimal action (policy class equivalence), not the reward structure.
• For Boltzmann-learners, the authors establish tight upper bounds: using an averaging strategy for policy estimation, the cumulative norm-based error is bounded as $$O\left(\sqrt{T \log(|A|T/\epsilon)} + \sum_t f(t-1)/t\right)$$ (stateless), where $f$ encodes the learner's rate of convergence. In the stateful setting, a visit-count-weighted analysis yields analogous bounds for cumulative prediction error over state-action pairs.
• The transfer from KL-policy distance bounds to best-response distance is formalized, showing that minimizing the former yields guarantees on optimal policy matching.
• Open questions are identified in efficiently minimizing KL-divergence and in achieving tightness for the derived bounds.

Numerical Guarantees and Claims

• Impossibility for norm-based learning for no-regret learners is explicitly proven: any algorithm can be forced into $\Omega(T)$ error.
• For Boltzmann learners, the averaging method achieves $O(\sqrt{T \log(|A|T/\epsilon)})$ error after all actions are sufficiently explored, indicating sublinear identification error on the reward/Q structure tied directly to effective exploration and learning rate.

Implications

Practical Implications

• The results formalize critical limitations in preference learning for interactive systems. When users do not systematically explore or when their regret is vanishing, only their favorite option can be inferred with confidence; recovering full preference models becomes intractable.
• For systems with users who exhibit probabilistic, structured exploration (modeled as Boltzmann rational with learning), more nuanced preference models can be recovered, with explicit bounds dependent on both exploration rates and actual visit distributions.
• The work underscores the necessity of explicit user (or agent) exploration policies for robust preference inference, raising foundational questions for the design of learning interfaces and AI alignment.

Theoretical Implications

• The results delineate the boundary between structural identifiability and its impossibility in preference inference, subject to the learning policy class of the observed agent.
• The averaging strategy in the Boltzmann setting is supported with high-probability deviation bounds, and the reduction between per-timestep and final reward estimation is formalized.
• The notions of policy optimality matching and reward structure matching are unambiguously separated, strengthening theoretical understanding for IRL and preference learning in non-stationary settings.

Future Directions

• The extension to more realistic models of bounded rationality for humans—beyond classic Boltzmann rationality—remains open.
• Richer exploration of algorithms minimizing KL-divergence between predicted and true policies, including algorithmic and computational efficiency under practical constraints.
• Application to domains with stochastic rewards and in settings where the observing system can also act (not purely passive), enabling richer preference elicitation protocols.

Conclusion

This work provides a foundational analysis of preference learning when the demonstrator is itself a learning agent. By characterizing the limitations and achievable bounds under different learner assumptions and evaluation criteria, the paper both demarcates the feasible regime for preference recovery and motivates principled strategies and further algorithmic development for real-world preference-aligned AI systems. These results inform both the design of interactive AI and future theoretical work in robust reward inference, especially for human-AI interaction scenarios where user optimality cannot be assumed.