Inverse Q-Learning: Theory and Applications

• Inverse Q-Learning is a family of methods that infers expert behavior by directly optimizing a soft Q-function rather than explicitly estimating rewards.
• It replaces nested reinforcement learning loops with a single optimization over Q-values, using a soft Bellman operator for policy recovery.
• The framework extends to offline imitation, multi-agent scenarios, and safety constraints, demonstrating robust empirical performance in diverse applications.

Inverse Q-learning denotes a family of inverse reinforcement learning and imitation learning methods that perform inference in the space of action-value functions rather than separately parameterizing a reward and then solving a forward control problem. In the canonical soft setting, the central object is a soft $Q$-function that implicitly determines both a reward through an inverse Bellman operator and a policy through a Boltzmann distribution over actions. This reformulation was introduced to avoid nested reinforcement-learning loops and adversarial reward–policy optimization, and it has since been extended to offline imitation, avoidance from undesirable demonstrations, optimal stopping, safety-constrained learning, Bayesian posterior inference, multi-agent learning, financial hedging, and inverse optimal control (Garg et al., 2021, Kalweit et al., 2020, Bajgar et al., 2024).

1. Foundational problem formulation

Standard Q-learning assumes a known reward function $r(s,a)$ and learns an action-value function satisfying a Bellman relation. Inverse reinforcement learning instead assumes expert trajectories are available but the reward is unknown, and seeks a reward–policy pair explaining the demonstrations. In the maximum-entropy formulation used by several inverse Q-learning papers, this appears as

$$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$

where $\rho_E$ and $\rho_\pi$ denote discounted state–action occupancies, $H(\pi)$ is causal entropy, and $\psi$ is a convex regularizer (Garg et al., 2021, Hoang et al., 2024).

The defining move of inverse Q-learning is to replace explicit optimization over rewards by direct optimization over a $Q$-function. In the soft setting, for a fixed policy $\pi$, one defines the inverse soft-Bellman operator

$$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$

with

$r(s,a)$0

This yields a one-to-one correspondence between $r(s,a)$1 and $r(s,a)$2 for fixed $r(s,a)$3, so that $r(s,a)$4 (Garg et al., 2021).

This Q-space viewpoint is not confined to one objective. Some papers use it to rationalize expert behavior under a Boltzmann policy; some use it to define Bayesian posteriors directly over $r(s,a)$5 values; some reinterpret imitation as a saddle-point problem over policies and critics; and some derive domain-specific inverse control procedures from Bellman-type identities (Bajgar et al., 2024, Moulin et al., 26 May 2025, Du et al., 3 Jul 2026).

2. Canonical inverse soft-Q learning

The best-known formulation is inverse soft-Q learning as introduced in IQ-Learn. Starting from the soft Bellman backup and inverse operator, the IRL saddle-point can be rewritten entirely in $r(s,a)$6 as

$r(s,a)$7

For typical convex regularizers $r(s,a)$8, the inner minimization over $r(s,a)$9 has the closed-form solution

$$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$0

so the original max–min problem becomes a single optimization in $$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$1 (Garg et al., 2021).

Under the $$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$2-divergence special case, where $$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$3, the objective becomes a squared Bellman-error form on expert data. In practice, the implicit reward is

$$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$4

and training minimizes the mean-squared Bellman error on expert transitions. Once $$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$5 is learned, the recovered reward is

$$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$6

and the corresponding maximum-entropy policy is

$$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$7

IQ-Learn emphasizes that the update is dynamics-aware because it uses actual transitions $$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$8 from expert data, avoids adversarial saddle-point training, and in continuous control can use a standard Soft Actor-Critic step for policy improvement (Garg et al., 2021).

Least-Squares Inverse Q-Learning sharpens this picture by showing that a mixture-distribution $$\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),$$9 regularizer on the implicit reward turns the objective into squared Bellman-error minimization under the mixture

$\rho_E$0

It also shows that this regularization corresponds to minimizing a bounded Pearson $\rho_E$1-divergence and modifies the inverse Bellman operator so that absorbing states are assigned the closed-form value $\rho_E$2 instead of being forced to zero, thereby addressing termination bias (Al-Hafez et al., 2023).

3. Alternative Q-space inference paradigms

A distinct line of work uses a Boltzmann expert model to derive reward recovery directly from action probabilities. Inverse Action-Value Iteration and its sampling-based descendants exploit the relation

$\rho_E$3

to derive local linear systems for the unknown reward values at each state. This yields a model-based closed-form procedure, a model-free stochastic approximation variant, and deep continuous-state extensions with neural approximators for $\rho_E$4, $\rho_E$5, shifted $\rho_E$6, and empirical action frequencies. The same framework also admits constrained variants in which the Bellman backup is restricted to safe action sets (Kalweit et al., 2020).

Bayesian inverse reinforcement learning has also been reformulated in Q-space. ValueWalk places a prior over rewards, transfers it to a prior over $\rho_E$7 values via the change of variables $\rho_E$8, and samples from the posterior

$\rho_E$9

using Hamiltonian Monte Carlo. Because reward extraction from $\rho_\pi$0 is inexpensive relative to repeated forward planning from rewards, this makes gradient-based posterior sampling practical and supports direct reward recovery from accepted $\rho_\pi$1 samples (Bajgar et al., 2024).

An online variant appears in Bellman Gradient Iteration. There the demonstrator is assumed to act according to a Boltzmann distribution over the optimal $\rho_\pi$2, and the reward parameter $\rho_\pi$3 is updated from each new observation using gradients of $\rho_\pi$4 with respect to $\rho_\pi$5. Since the $\rho_\pi$6 operator obstructs differentiation, the method replaces it with either a $\rho_\pi$7-norm approximation or a generalized soft-max approximation, derives Bellman gradient equations, and performs stochastic gradient ascent on the instantaneous log-likelihood (Li et al., 2017).

A more theoretical offline imitation formulation is SPOIL, which assumes $\rho_\pi$8-realizable MDPs and optimizes the saddle-point loss

$\rho_\pi$9

In the linear case it achieves additive-error $H(\pi)$0 imitation with $H(\pi)$1 samples; in the general nonlinear case the sample complexity becomes $H(\pi)$2. The same analysis yields a critic loss of the form $H(\pi)$3 for deep imitation learning (Moulin et al., 26 May 2025).

4. Structured, constrained, and objective-specific variants

Several later methods preserve the Q-space perspective while changing the learning objective or the structure of the decision problem.

Variant	Setting	Distinctive mechanism
CIQL	Imperfect demonstrations	Per-transition confidence weighting; filtering or penalization
UNIQ	Undesirable demonstrations	Maximize divergence from undesirable occupancy; occupancy correction
DO-IQS	Optimal stopping	Explicit dynamics model, cumulative continuation gain, oversampling
SafeQIL	Unknown safety constraints	Support-aware Q bound and safety reward from discriminator
BiCQL-ML	Offline IRL	Bi-level conservative Q-learning and reward regression

Confidence-based Inverse soft-Q Learning augments IQ-Learn with transition-level confidence scores $H(\pi)$4. CIQL-Expert filters low-confidence samples by weighting the expert term with $H(\pi)$5, whereas CIQL-Agent adds a negative term $H(\pi)$6 so that low-confidence transitions are explicitly penalized. Its confidence scores are computed from transition geometry through an approach angle and a sigmoid map, rather than from ground-truth rewards or human supervision (Bu et al., 2023).

UNIQ inverts the usual imitation objective. Instead of matching an expert occupancy, it maximizes a statistical distance between the learner occupancy and an undesirable occupancy $H(\pi)$7. The resulting Q-space problem is cooperative rather than adversarial, uses occupancy-ratio correction when only mixed unlabeled data are available, and extracts the policy by Weighted Behavior Cloning with weights $H(\pi)$8 (Hoang et al., 2024).

DO-IQS specializes inverse Q-learning to inverse optimal stopping. It embeds stopping into a stopped Markov decision process with binary actions $H(\pi)$9, a cemetery state $\psi$0, and reward

$\psi$1

The method learns a Q-function, a neural-network dynamics model, and a continuation-gain model, augments states with a cumulative continuation-gain variable to handle non-Markovian continuation rewards, and uses confidence-based oversampling to mitigate sparsity near the stopping boundary. Its evaluation targets recovery of the unique stopping region $\psi$2 rather than the non-unique gain functions themselves (Kuchko, 5 Mar 2025).

SafeQIL addresses settings with observable task rewards but unknown safety constraints and hidden costs. It defines a safety reward $\psi$3 from a discriminator estimating support membership, forms a mixed reward from task and safety components, and constrains out-of-support Q-values to remain below the smallest expert Q-value. The resulting critic loss combines in-support Bellman regression, out-of-support pessimistic regression, and a local upper-bound penalty anchored to nearby demonstration states (Papadopoulos et al., 27 Feb 2026).

BiCQL-ML is a policy-free offline IRL method. Its lower level learns a conservative Q-function with a Bellman-error term plus a Conservative Q-Learning regularizer, while its upper level regresses the reward to the soft Bellman residual

$\psi$4

The paper proves convergence to a fixed point under a contraction condition and states that at the fixed point the learned reward makes the expert policy soft-optimal (Park, 27 Nov 2025).

5. Multi-agent, hierarchical, and latent-intention formulations

In cooperative multi-agent imitation learning, Inverse Factorized Q-Learning extends single-agent inverse soft-Q learning by assigning each agent a local soft-Q function $\psi$5 and then aggregating local values and inverse rewards through centralized mixing networks. The networks are required to be coordinate-wise nondecreasing and convex, which preserves convexity of the objective in the full vector of local Q-values. A special case with nonnegative linear mixtures reduces exactly to independent IQ-Learn (Bui et al., 2023).

A separate multi-agent approach, MAMQL, focuses on general-sum Markov games. For each agent $\psi$6, it defines a marginalized critic

$\psi$7

treats the other agents as part of the environment, and optimizes a single-agent soft-Q loss for each marginal dataset $\psi$8. Policies are then extracted by Boltzmann normalization of the marginalized critics (Haynam et al., 6 Mar 2025).

Hierarchical Inverse Q-Learning addresses non-stationary or multi-intention demonstrations by segmenting trajectories into latent intention segments and solving inverse Q-learning separately for each segment. The latent intention process may be independent across segments or Markovian, in which case Forward–Backward inference is used in the E-step of an EM procedure. The resulting reward or Q-function is piecewise constant over time,

$\psi$9

which the paper argues better models intention transitions as a step function than as a smoothly varying function (Zhu et al., 2023).

6. Empirical performance and application domains

IQ-Learn reports state-of-the-art results in offline and online imitation learning. With just 1 expert trajectory on CartPole and Acrobot, it achieves near-optimal returns and far surpasses ValueDICE, EDM, and behavioral cloning. In online imitation on MuJoCo tasks with 1 demo, it reaches expert level in fewer than 200K steps, and on Atari with 20 demos it converges $Q$0–$Q$1 faster than DQN baselines while outperforming GAIL and ValueDICE. The recovered rewards show Pearson correlation $Q$2 with the true rewards in MuJoCo, and the recovered reward maps in Gridworld and Reacher faithfully match ground truth (Garg et al., 2021).

CIQL reports that transition-based confidence scoring can increase the success rate of the baseline algorithm by $Q$3 on average. In the detailed results, CIQL-A gives $Q$4, CIQL-E gives $Q$5, and filtering only gives $Q$6; on a Better–Worse mixture, CIQL-A reaches approximately $Q$7 versus $Q$8 for filtering and $Q$9 for vanilla IQ-Learn (Bu et al., 2023).

For optimal stopping, DO-IQS improves the balanced-accuracy and the m-EMR versus m-TTE frontier on sparse 2D Brownian Motion data, and on Azure Predictive Maintenance and NASA Turbofan it matches or slightly exceeds survival-analysis hazard-rate baselines in the same trade-off. The reported compute profile is training in $\pi$0 min on 2 vCPU/4 GiB and inference cost $\pi$1 s/obs (Kuchko, 5 Mar 2025).

ValueWalk reports the same posterior over rewards as reward-space PolicyWalk while being $\pi$2–$\pi$3 faster per effective sample in gridworlds. Its HMC+NUTS chains reach $\pi$4 on each dimension within 1 K samples, yield ESS approximately $\pi$5 higher than reward-space chains, and take $\pi$6 s per sample versus $\pi$7 s for PolicyWalk-HMC on a $\pi$8 grid (Bajgar et al., 2024).

On Objectworld, the closed-form IAVI method reports Expected Value Difference $\pi$9 with runtime about $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$0 min, the sampling-based IQL reports EVD $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$1 with runtime about $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$2 min, MaxEnt IRL reports EVD $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$3 with runtime about $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$4 h, and single-step MaxEnt reports EVD $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$5 with runtime about $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$6 h. In SUMO lane-changing, Deep Constrained Inverse Q-learning achieves average speed around $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$7 m/s with zero keep-right violations and reaches near-expert speed even with $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$8 demonstrations (Kalweit et al., 2020).

Domain-specific inverse Q-learning has also been demonstrated in quantitative finance and stochastic control. In the QLBS hedging model, IRL plus fitted Q-iteration recovers the correct risk-aversion parameter and reproduces option prices such as $$(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],$$9 for the ATM put in the reported setup; off-policy fitted Q-iteration remains stable up to moderate action noise $r(s,a)$00 (Halperin, 2018). For linear quadratic systems with Poisson jumps, a model-free off-policy inverse Q-learning algorithm converges in approximately $r(s,a)$01 iterations and recovers an equivalent cost weight producing a gain within about $r(s,a)$02 of the expert gain (Du et al., 3 Jul 2026).

In multi-agent domains, MAMQL reports improvements in average reward, sample efficiency, and reward recovery by often more than $r(s,a)$03–$r(s,a)$04, and its benchmark returns on Gems, Overcooked, and Highway-Env are markedly above behavior cloning, independent IQ-Learn, and MA-AIRL in the reported tables (Haynam et al., 6 Mar 2025).

7. Identifiability, assumptions, and unresolved issues

A recurring theme is that inverse Q-learning often identifies equivalence classes rather than a unique ground-truth reward. DO-IQS states explicitly that $r(s,a)$05 are not unique even though the induced stopping region is unique. The Poisson-jump inverse optimal control paper recovers an equivalent state weight $r(s,a)$06 that need not equal the expert’s true $r(s,a)$07 but still yields the same optimal gain. By contrast, in the QLBS model the inverse problem is effectively one-dimensional in the trader’s risk-aversion parameter $r(s,a)$08, which makes reward recovery practically identifiable (Kuchko, 5 Mar 2025, Du et al., 3 Jul 2026, Halperin, 2018).

Most formulations rely on a soft-optimality or Boltzmann-rationality assumption. This assumption is central in IQ-Learn, UNIQ, ValueWalk, Deep Inverse Q-Learning with Constraints, MAMQL, and the online Bellman-gradient method. Several papers also make clear that approximation quality depends on this choice: the online method converges to a local optimum of an approximate likelihood, Deep Inverse Q-Learning with Constraints notes possible misspecification in non-Boltzmann or adversarial settings, and MAMQL notes that Boltzmann rationality may not capture highly suboptimal or exploratory experts (Li et al., 2017, Kalweit et al., 2020, Haynam et al., 6 Mar 2025).

The literature also emphasizes support mismatch and terminal-state handling. LS-IQ addresses absorbing states by regressing to the correct discounted absorbing return rather than zero, SafeQIL imposes local pessimism outside the demonstration support, and BiCQL-ML uses Conservative Q-Learning to suppress over-generalization to out-of-distribution actions (Al-Hafez et al., 2023, Papadopoulos et al., 27 Feb 2026, Park, 27 Nov 2025).

Several open directions are stated explicitly. IQ-Learn lists alternative statistical divergences, state-only reward learning, meta-IRL for multi-task or goal generalization, and learned dynamics models for recovering $r(s,a)$09 without environment access. DO-IQS suggests Bayesian or ensemble dynamics models, more principled oversampling, improved continuation-gain fitting, and broader comparisons to adversarial and variational IRL in inverse optimal stopping. Deep Inverse Q-Learning with Constraints identifies continuous action spaces and partial observability as future work (Garg et al., 2021, Kuchko, 5 Mar 2025, Kalweit et al., 2020).

Taken together, these results show that inverse Q-learning is best understood not as a single algorithm but as a Q-space design principle: expert behavior is explained by learning, regularizing, sampling, or constraining an action-value representation from which rewards and policies can be recovered. The specific object of identification may be a soft reward, a stopping boundary, a safety-aware critic, a latent intention segment, a marginalized multi-agent value, or an equivalent cost functional, depending on the structure imposed by the problem setting.