---
title: 'Inverse Q-Learning: Theory and Applications'
url: https://www.emergentmind.com/topics/inverse-q-learning
type: topic
---

# Inverse Q-Learning: Theory and Applications

Inverse Q-learning denotes a family of inverse reinforcement learning and imitation learning methods that perform inference in the space of action-value functions rather than separately parameterizing a reward and then solving a forward control problem. In the canonical soft setting, the central object is a soft $Q$-function that implicitly determines both a reward through an inverse Bellman operator and a policy through a Boltzmann distribution over actions. This reformulation was introduced to avoid nested reinforcement-learning loops and adversarial reward–policy optimization, and it has since been extended to offline imitation, avoidance from undesirable demonstrations, optimal stopping, safety-constrained learning, Bayesian posterior inference, multi-agent learning, financial hedging, and inverse optimal control [2106.12142], [2008.01712], [2407.10971].

## 1. Foundational problem formulation

Standard Q-learning assumes a known reward function $r(s,a)$ and learns an action-value function satisfying a Bellman relation. Inverse reinforcement learning instead assumes expert trajectories are available but the reward is unknown, and seeks a reward–policy pair explaining the demonstrations. In the maximum-entropy formulation used by several inverse Q-learning papers, this appears as
\[
\max_r \min_\pi \; \mathbb E_{\rho_E}[r]-\mathbb E_{\rho_\pi}[r]-H(\pi)-\psi(r),
\]
where $\rho_E$ and $\rho_\pi$ denote discounted state–action occupancies, $H(\pi)$ is causal entropy, and $\psi$ is a convex regularizer [2106.12142], [2410.08307].

The defining move of inverse Q-learning is to replace explicit optimization over rewards by direct optimization over a $Q$-function. In the soft setting, for a fixed policy $\pi$, one defines the inverse soft-Bellman operator
\[
(\mathcal T^\pi Q)(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V^\pi(s')],
\]
with
\[
V^\pi(s)=\mathbb E_{a\sim \pi}[Q(s,a)-\log \pi(a\mid s)].
\]
This yields a one-to-one correspondence between $Q$ and $r$ for fixed $\pi$, so that $r(s,a)=\mathcal T^\pi Q(s,a)$ [2106.12142].

This Q-space viewpoint is not confined to one objective. Some papers use it to rationalize expert behavior under a Boltzmann policy; some use it to define Bayesian posteriors directly over $Q$ values; some reinterpret imitation as a saddle-point problem over policies and critics; and some derive domain-specific inverse control procedures from Bellman-type identities [2407.10971], [2505.19946], [2607.03169].

## 2. Canonical inverse soft-Q learning

The best-known formulation is inverse soft-Q learning as introduced in IQ-Learn. Starting from the soft Bellman backup and inverse operator, the IRL saddle-point can be rewritten entirely in $(\pi,Q)$ as
\[
\mathcal J(\pi,Q)=\mathbb E_{\rho_E}[(\mathcal T^\pi Q)(s,a)]-\mathbb E_{\rho_\pi}[(\mathcal T^\pi Q)(s,a)]-H(\pi)-\psi(\mathcal T^\pi Q).
\]
For typical convex regularizers $\psi$, the inner minimization over $\pi$ has the closed-form solution
\[
\pi_Q(a\mid s)=\mathrm{softmax}_a\,Q(s,a),
\]
so the original max–min problem becomes a single optimization in $Q$ [2106.12142].

Under the $\chi^2$-divergence special case, where $\psi(r)=\tfrac14\alpha\|r\|^2_{\rho_E}$, the objective becomes a squared Bellman-error form on expert data. In practice, the implicit reward is
\[
r(s,a)=Q(s,a)-\gamma \,\mathbb E_{s'}[V(s')],
\]
and training minimizes the mean-squared Bellman error on expert transitions. Once $Q^\star$ is learned, the recovered reward is
\[
\hat r(s,a)=Q^\star(s,a)-\gamma \,\mathbb E_{s'}[V^\star(s')],
\]
and the corresponding maximum-entropy policy is
\[
\pi(a\mid s)=\exp\!\bigl(Q^\star(s,a)-V^\star(s)\bigr), \qquad
V^\star(s)=\log\sum_a \exp Q^\star(s,a).
\]
IQ-Learn emphasizes that the update is dynamics-aware because it uses actual transitions $(s,a,s')$ from expert data, avoids adversarial saddle-point training, and in continuous control can use a standard Soft Actor-Critic step for policy improvement [2106.12142].

Least-Squares Inverse Q-Learning sharpens this picture by showing that a mixture-distribution $L_2$ regularizer on the implicit reward turns the objective into squared Bellman-error minimization under the mixture
\[
d_{\mathrm{mix}}=\alpha d_{\pi_E}+(1-\alpha)d_\pi.
\]
It also shows that this regularization corresponds to minimizing a bounded Pearson $\chi^2$-divergence and modifies the inverse Bellman operator so that absorbing states are assigned the closed-form value $V(s_A)=r_A/(1-\gamma)$ instead of being forced to zero, thereby addressing termination bias [2303.00599].

## 3. Alternative Q-space inference paradigms

A distinct line of work uses a Boltzmann expert model to derive reward recovery directly from action probabilities. Inverse Action-Value Iteration and its sampling-based descendants exploit the relation
\[
\pi^E(a\mid s)=\frac{\exp(Q^\star(s,a))}{\sum_b \exp(Q^\star(s,b))}
\]
to derive local linear systems for the unknown reward values at each state. This yields a model-based closed-form procedure, a model-free stochastic approximation variant, and deep continuous-state extensions with neural approximators for $r$, $Q$, shifted $Q$, and empirical action frequencies. The same framework also admits constrained variants in which the Bellman backup is restricted to safe action sets [2008.01712].

Bayesian inverse reinforcement learning has also been reformulated in Q-space. ValueWalk places a prior over rewards, transfers it to a prior over $Q$ values via the change of variables $r=(I-\gamma \bar P)Q$, and samples from the posterior
\[
p(Q\mid D)\propto p_Q(Q)\,p(D\mid Q)
\]
using Hamiltonian Monte Carlo. Because reward extraction from $Q$ is inexpensive relative to repeated forward planning from rewards, this makes gradient-based posterior sampling practical and supports direct reward recovery from accepted $Q$ samples [2407.10971].

An online variant appears in Bellman Gradient Iteration. There the demonstrator is assumed to act according to a Boltzmann distribution over the optimal $Q^\star(s,a;\theta)$, and the reward parameter $\theta$ is updated from each new observation using gradients of $Q^\star$ with respect to $\theta$. Since the $\max$ operator obstructs differentiation, the method replaces it with either a $p$-norm approximation or a generalized soft-max approximation, derives Bellman gradient equations, and performs stochastic gradient ascent on the instantaneous log-likelihood [1707.09393].

A more theoretical offline imitation formulation is SPOIL, which assumes $Q^\pi$-realizable MDPs and optimizes the saddle-point loss
\[
L(\pi,Q)=\mathbb E_{\mu^E}[Q(x,a)-Q(x,\pi)].
\]
In the linear case it achieves additive-error $\varepsilon$ imitation with $\tilde O(\varepsilon^{-2})$ samples; in the general nonlinear case the sample complexity becomes $\tilde O(\varepsilon^{-4})$. The same analysis yields a critic loss of the form $\mathbb E_D[Q(x,a)]-\mathbb E_D[\mathbb E_{a\sim \pi}Q(x,a)]$ for deep imitation learning [2505.19946].

## 4. Structured, constrained, and objective-specific variants

Several later methods preserve the Q-space perspective while changing the learning objective or the structure of the decision problem.

| Variant | Setting | Distinctive mechanism |
|---|---|---|
| CIQL | Imperfect demonstrations | Per-transition confidence weighting; filtering or penalization |
| UNIQ | Undesirable demonstrations | Maximize divergence from undesirable occupancy; occupancy correction |
| DO-IQS | Optimal stopping | Explicit dynamics model, cumulative continuation gain, oversampling |
| SafeQIL | Unknown safety constraints | Support-aware Q bound and safety reward from discriminator |
| BiCQL-ML | Offline IRL | Bi-level conservative Q-learning and reward regression |

Confidence-based Inverse soft-Q Learning augments IQ-Learn with transition-level confidence scores $w(s,a)\in[0,1]$. CIQL-Expert filters low-confidence samples by weighting the expert term with $w(s,a)/\alpha$, whereas CIQL-Agent adds a negative term $-(1-w)\mathcal T^\pi Q$ so that low-confidence transitions are explicitly penalized. Its confidence scores are computed from transition geometry through an approach angle and a sigmoid map, rather than from ground-truth rewards or human supervision [2312.11194].

UNIQ inverts the usual imitation objective. Instead of matching an expert occupancy, it maximizes a statistical distance between the learner occupancy and an undesirable occupancy $\rho^{UN}$. The resulting Q-space problem is cooperative rather than adversarial, uses occupancy-ratio correction when only mixed unlabeled data are available, and extracts the policy by Weighted Behavior Cloning with weights $\exp(Q-V^Q)$ [2410.08307].

DO-IQS specializes inverse Q-learning to inverse optimal stopping. It embeds stopping into a stopped Markov decision process with binary actions $\{0,1\}$, a cemetery state $\Delta$, and reward
\[
r(s,a)=g(s)a+G(s)(1-a).
\]
The method learns a Q-function, a neural-network dynamics model, and a continuation-gain model, augments states with a cumulative continuation-gain variable to handle non-Markovian continuation rewards, and uses confidence-based oversampling to mitigate sparsity near the stopping boundary. Its evaluation targets recovery of the unique stopping region $D^\star$ rather than the non-unique gain functions themselves [2503.03515].

SafeQIL addresses settings with observable task rewards but unknown safety constraints and hidden costs. It defines a safety reward $r_s(s)=\log \phi_\omega(s)\le 0$ from a discriminator estimating support membership, forms a mixed reward from task and safety components, and constrains out-of-support Q-values to remain below the smallest expert Q-value. The resulting critic loss combines in-support Bellman regression, out-of-support pessimistic regression, and a local upper-bound penalty anchored to nearby demonstration states [2602.23816].

BiCQL-ML is a policy-free offline IRL method. Its lower level learns a conservative Q-function with a Bellman-error term plus a Conservative Q-Learning regularizer, while its upper level regresses the reward to the soft Bellman residual
\[
Q_\phi(s,a)-\gamma \log\sum_{a'} e^{Q_\phi(s',a')}.
\]
The paper proves convergence to a fixed point under a contraction condition and states that at the fixed point the learned reward makes the expert policy soft-optimal [2511.22210].

## 5. Multi-agent, hierarchical, and latent-intention formulations

In cooperative multi-agent imitation learning, Inverse Factorized Q-Learning extends single-agent inverse soft-Q learning by assigning each agent a local soft-Q function $Q_i(o_i,a_i)$ and then aggregating local values and inverse rewards through centralized mixing networks. The networks are required to be coordinate-wise nondecreasing and convex, which preserves convexity of the objective in the full vector of local Q-values. A special case with nonnegative linear mixtures reduces exactly to independent IQ-Learn [2310.06801].

A separate multi-agent approach, MAMQL, focuses on general-sum Markov games. For each agent $i$, it defines a marginalized critic
\[
Q_m^i(s,a^i)=\mathbb E_{a^{-i}\sim \pi_{-i}(\cdot\mid s)}[Q^i(s,a^i,a^{-i})],
\]
treats the other agents as part of the environment, and optimizes a single-agent soft-Q loss for each marginal dataset $\{(s,a^i)\}$. Policies are then extracted by Boltzmann normalization of the marginalized critics [2503.04679].

Hierarchical Inverse Q-Learning addresses non-stationary or multi-intention demonstrations by segmenting trajectories into latent intention segments and solving inverse Q-learning separately for each segment. The latent intention process may be independent across segments or Markovian, in which case Forward–Backward inference is used in the E-step of an EM procedure. The resulting reward or Q-function is piecewise constant over time,
\[
Q_t^\star(s,a)=\sum_{k=1}^K \mathbf 1\{z_t=k\}Q_k^\star(s,a),
\]
which the paper argues better models intention transitions as a step function than as a smoothly varying function [2311.13870].

## 6. Empirical performance and application domains

IQ-Learn reports state-of-the-art results in offline and online imitation learning. With just 1 expert trajectory on CartPole and Acrobot, it achieves near-optimal returns and far surpasses ValueDICE, EDM, and behavioral cloning. In online imitation on MuJoCo tasks with 1 demo, it reaches expert level in fewer than 200K steps, and on Atari with 20 demos it converges $3$–$7\times$ faster than DQN baselines while outperforming GAIL and ValueDICE. The recovered rewards show Pearson correlation $r\ge 0.99$ with the true rewards in MuJoCo, and the recovered reward maps in Gridworld and Reacher faithfully match ground truth [2106.12142].

CIQL reports that transition-based confidence scoring can increase the success rate of the baseline algorithm by $40.3\%$ on average. In the detailed results, CIQL-A gives $+40.3\%$, CIQL-E gives $+30.1\%$, and filtering only gives $+26.8\%$; on a Better–Worse mixture, CIQL-A reaches approximately $91.9\%$ versus $51.4\%$ for filtering and $27.7\%$ for vanilla IQ-Learn [2312.11194].

For optimal stopping, DO-IQS improves the balanced-accuracy and the m-EMR versus m-TTE frontier on sparse 2D Brownian Motion data, and on Azure Predictive Maintenance and NASA Turbofan it matches or slightly exceeds survival-analysis hazard-rate baselines in the same trade-off. The reported compute profile is training in $\le 12$ min on 2 vCPU/4 GiB and inference cost $0.003$ s/obs [2503.03515].

ValueWalk reports the same posterior over rewards as reward-space PolicyWalk while being $5$–$300\times$ faster per effective sample in gridworlds. Its HMC+NUTS chains reach $\hat R\le 1.01$ on each dimension within 1 K samples, yield ESS approximately $3\times$ higher than reward-space chains, and take $0.2$ s per sample versus $0.8$ s for PolicyWalk-HMC on a $3\times 3$ grid [2407.10971].

On Objectworld, the closed-form IAVI method reports Expected Value Difference $0.09$ with runtime about $1.8$ min, the sampling-based IQL reports EVD $1.47$ with runtime about $21$ min, MaxEnt IRL reports EVD $11.58$ with runtime about $8$ h, and single-step MaxEnt reports EVD $4.33$ with runtime about $12$ h. In SUMO lane-changing, Deep Constrained Inverse Q-learning achieves average speed around $19.7$ m/s with zero keep-right violations and reaches near-expert speed even with $10^3$ demonstrations [2008.01712].

Domain-specific inverse Q-learning has also been demonstrated in quantitative finance and stochastic control. In the QLBS hedging model, IRL plus fitted Q-iteration recovers the correct risk-aversion parameter and reproduces option prices such as $4.90\pm 0.12$ for the ATM put in the reported setup; off-policy fitted Q-iteration remains stable up to moderate action noise $\eta\approx 0.5$ [1801.06077]. For linear quadratic systems with Poisson jumps, a model-free off-policy inverse Q-learning algorithm converges in approximately $20$ iterations and recovers an equivalent cost weight producing a gain within about $1.5\times 10^{-2}$ of the expert gain [2607.03169].

In multi-agent domains, MAMQL reports improvements in average reward, sample efficiency, and reward recovery by often more than $2$–$5\times$, and its benchmark returns on Gems, Overcooked, and Highway-Env are markedly above behavior cloning, independent IQ-Learn, and MA-AIRL in the reported tables [2503.04679].

## 7. Identifiability, assumptions, and unresolved issues

A recurring theme is that inverse Q-learning often identifies equivalence classes rather than a unique ground-truth reward. DO-IQS states explicitly that $(g,G)$ are not unique even though the induced stopping region is unique. The Poisson-jump inverse optimal control paper recovers an equivalent state weight $N^\star$ that need not equal the expert’s true $N_T$ but still yields the same optimal gain. By contrast, in the QLBS model the inverse problem is effectively one-dimensional in the trader’s risk-aversion parameter $\lambda$, which makes reward recovery practically identifiable [2503.03515], [2607.03169], [1801.06077].

Most formulations rely on a soft-optimality or Boltzmann-rationality assumption. This assumption is central in IQ-Learn, UNIQ, ValueWalk, Deep Inverse Q-Learning with Constraints, MAMQL, and the online Bellman-gradient method. Several papers also make clear that approximation quality depends on this choice: the online method converges to a local optimum of an approximate likelihood, Deep Inverse Q-Learning with Constraints notes possible misspecification in non-Boltzmann or adversarial settings, and MAMQL notes that Boltzmann rationality may not capture highly suboptimal or exploratory experts [1707.09393], [2008.01712], [2503.04679].

The literature also emphasizes support mismatch and terminal-state handling. LS-IQ addresses absorbing states by regressing to the correct discounted absorbing return rather than zero, SafeQIL imposes local pessimism outside the demonstration support, and BiCQL-ML uses Conservative Q-Learning to suppress over-generalization to out-of-distribution actions [2303.00599], [2602.23816], [2511.22210].

Several open directions are stated explicitly. IQ-Learn lists alternative statistical divergences, state-only reward learning, meta-IRL for multi-task or goal generalization, and learned dynamics models for recovering $r(s,a,s')$ without environment access. DO-IQS suggests Bayesian or ensemble dynamics models, more principled oversampling, improved continuation-gain fitting, and broader comparisons to adversarial and variational IRL in inverse optimal stopping. Deep Inverse Q-Learning with Constraints identifies continuous action spaces and partial observability as future work [2106.12142], [2503.03515], [2008.01712].

Taken together, these results show that inverse Q-learning is best understood not as a single algorithm but as a Q-space design principle: expert behavior is explained by learning, regularizing, sampling, or constraining an action-value representation from which rewards and policies can be recovered. The specific object of identification may be a soft reward, a stopping boundary, a safety-aware critic, a latent intention segment, a marginalized multi-agent value, or an equivalent cost functional, depending on the structure imposed by the problem setting.

Source: https://www.emergentmind.com/topics/inverse-q-learning