---
title: KL-Regularised Q-Learning
url: https://www.emergentmind.com/topics/kl-regularised-q-learning-klq
type: topic
---

# KL-Regularised Q-Learning

Searching arXiv for recent and foundational papers on KL-regularised Q-learning and closely related regularised Q-learning formulations.
KL-regularised Q-Learning (KLQ) is a family of reinforcement-learning methods in which action selection is regularised by a Kullback–Leibler divergence to a reference policy, so that greedy improvement is replaced by a prior-weighted soft improvement rule. In its canonical form, the statewise control problem is
\[
\max_{\pi(\cdot\mid s)} \left\{ \sum_a \pi(a\mid s) Q(s,a) - \tau \,\mathrm{KL}(\pi(\cdot\mid s)\|\pi_{\mathrm{ref}}(\cdot\mid s)) \right\},
\]
whose optimizer is the Boltzmann tilt of the reference policy and whose optimal value is a log-partition or log-sum-exp backup [2604.28036]. In contemporary usage, the term spans several closely related objects: regularised Bellman operators for tabular or approximate control, mirror-descent or trust-region-style policy-improvement schemes, value-based algorithms such as Q-REPS and Munchausen-style methods, and, more recently, token-level action-value methods for online RLHF [2010.11151] [2508.17000]. It does not include every paper whose title contains “regularized Q-learning”: some works regularise parameter dynamics for stability under function approximation rather than policy optimisation, and therefore are not KLQ in the standard sense [2202.05404].

## 1. Conceptual scope and defining distinction

KLQ is defined by where the regularisation enters. In KL-regularised reinforcement learning, the control objective is modified by a statewise penalty of the form
\[
-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s))
\quad\text{or equivalently}\quad
\tau \mathcal H(\pi(\cdot|s))
\]
when the reference is uniform. This changes both policy improvement and Bellman optimality, replacing a hard maximisation over actions by a soft, prior-weighted optimisation [2604.28036] [2401.15196]. By contrast, some papers use the word “regularized” to describe shrinkage or damping in the Q-update or parameter dynamics. “Regularized Q-Learning” by Lim and Lee is of that latter kind: it studies Q-learning with linear function approximation and adds a stabilising regulariser to the update dynamics; it has no KL divergence, no reference policy \(\mu\), and no soft Bellman operator, so it is not a KLQ method [2202.05404].

This distinction is central because KLQ regularises policy selection, not merely numerical optimisation. In KLQ, the reference policy may be a previous iterate, a fixed prior, a behavior policy, an SFT model in RLHF, or a safe-support policy in safety-constrained settings. The regulariser therefore has a control-theoretic role: it constrains deviation from a reference mechanism while preserving a closed-form greedy step [2508.17000] [2604.25379].

A second important distinction is between explicit and implicit KL regularisation. Some methods write the KL penalty directly into the objective or Bellman operator, as in Q-REPS or safe-support KLQ [2010.11151] [2604.25379]. Others induce an equivalent or approximate KL effect through reward augmentation or advantage learning, as in Munchausen-style methods [2205.07467] [2205.07885]. This suggests that KLQ is best viewed as a broader design pattern organised around KL-shaped improvement geometry rather than a single algorithm.

## 2. Variational foundation and soft Bellman structure

The cleanest mathematical foundation in the supplied material is the exponential-family identity developed in “Exponential families from a single KL identity” [2604.28036]. For an exponential family \(p_\lambda\), the paper proves
\[
\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),
\]
and derives the Gibbs variational principle
\[
A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.
\]
Specialising to scalar reward \(r\) with \(\lambda=1/\beta\), it yields
\[
\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),
\]
with unique maximizer
\[
q^*(y)\propto a(y)e^{r(y)/\beta}.
\]
Mapped statewise to RL, \(a(\cdot)\) becomes \(\pi_{\mathrm{ref}}(\cdot\mid s)\), \(r(a)\) becomes \(Q(s,a)\), and \(\beta\) becomes \(\tau\), giving the canonical KLQ update
\[
\pi_{\mathrm{new}}(a\mid s)
=
\frac{\pi_{\mathrm{ref}}(a\mid s)\exp(Q(s,a)/\tau)}
{\sum_{a'}\pi_{\mathrm{ref}}(a'\mid s)\exp(Q(s,a')/\tau)},
\]
together with the soft value
\[
V(s)=\tau\log \sum_a \pi_{\mathrm{ref}}(a\mid s)\exp(Q(s,a)/\tau)
\]
[2604.28036].

Within the broader regularised-MDP literature, the same structure appears through convex conjugacy. “Regularized Q-Learning with Linear Function Approximation” defines a strongly convex policy regulariser \(G\), its conjugate \(G_\tau^*\), and the regularised Bellman operator
\[
\mathcal{B}_{\tau}X(s,a)
=
R(s,a)+\gamma \mathbb{E}_{s'}\left[G^*_\tau(X(s',\cdot))\right].
\]
For negative Shannon entropy, this reduces to
\[
G_\tau^*(q)=\tau\log\sum_a e^{q_a/\tau},
\]
recovering the standard soft-Q backup. The induced policy is
\[
\pi^*_{\tau}(\cdot \mid s)=\nabla G^*_{\tau}\left(Q^*_{\tau}(s,\cdot)\right),
\]
which becomes softmax in the Shannon case and prior-weighted Boltzmann under a KL-to-prior specialisation of \(G\) [2401.15196]. This suggests that KLQ is naturally embedded in a more general convex-dual framework, with KL and entropy as particular choices of regulariser.

The same statewise variational identity also underlies the bandit setting. “Near-Optimal Regret for KL-Regularized Multi-Armed Bandits” defines
\[
J(\pi)=\mathbb E_{a\sim\pi}[r(a)]-\eta^{-1}\mathrm{KL}(\pi\|\rho),
\]
whose optimizer is
\[
\pi^*(a)\propto \rho(a)\exp(\eta r(a)).
\]
This is the one-state analogue of KLQ’s soft greedy step, and the paper further proves
\[
\mathrm{SubOpt}(\pi,\pi^*)=\eta^{-1}\mathrm{KL}(\pi\|\pi^*),
\]
showing that the regularised suboptimality gap is itself a KL divergence to the soft optimum [2603.02155].

## 3. Core algorithmic forms

Several distinct algorithmic realisations of KLQ appear in the supplied sources.

The most direct value-based instance is Q-REPS, introduced in “Logistic Q-Learning” [2010.11151]. Starting from a regularised occupancy-measure LP, the paper introduces a dual Q-function \(Q_\theta=\Phi\theta\) and defines
\[
V_\theta(x)=\frac1\alpha \log\!\left(\sum_a \pi_0(a|x)e^{\alpha Q_\theta(x,a)}\right),
\]
\[
\Delta_\theta=r+\gamma PV_\theta-Q_\theta.
\]
The regularised primal optimum satisfies
\[
\mu^*(x,a)\propto \mu_0(x,a)e^{\eta \Delta_{\theta^*}(x,a)},
\]
and the policy update is
\[
\pi_{d^*}(a|x)=
\frac{\pi_0(a|x)e^{\alpha Q_{\theta^*}(x,a)}}
{\sum_{a'}\pi_0(a'|x)e^{\alpha Q_{\theta^*}(x,a')}}.
\]
This is a KL-regularised policy-improvement step around a reference policy \(\pi_0\), while critic learning is performed via the convex logistic Bellman error
\[
\mathcal G_k(\theta)=
\frac1\eta \log\!\left(\sum_{x,a} d_k(x,a)e^{\eta \Delta_\theta(x,a)}\right)
+(1-\gamma)\langle \nu_0,V_\theta\rangle
\]
[2010.11151]. Relative to canonical soft Q-learning, Q-REPS is distinctive because KL regularisation is derived from regularised occupancy optimisation and critic fitting uses a log-sum-exp Bellman objective rather than squared Bellman error.

A second line is KL-regularised approximate value iteration and mirror-descent value iteration. “Leverage the Average” studies the regularised greedy operator
\[
G_\mu^{\lambda,\tau}(q)
=
\arg\max_{\pi}
\Big(
\langle \pi,q\rangle
-\lambda \mathrm{KL}(\pi\|\mu)
+\tau H(\pi)
\Big),
\]
with Bellman evaluation
\[
T_{\pi|\mu}^{\lambda,\tau} q
=
r+\gamma P\Big(\langle \pi,q\rangle -\lambda \mathrm{KL}(\pi\|\mu)+\tau H(\pi)\Big).
\]
Its closed-form greedy solution is
\[
\pi^*(a|s)\propto
\mu(a|s)^{\frac{\lambda}{\lambda+\tau}}
\exp\!\left(\frac{q(s,a)}{\lambda+\tau}\right),
\]
specialising to
\[
\pi_{k+1}(a|s)\propto \pi_k(a|s)e^{q_k(s,a)/\lambda}
\]
in the pure-KL case [2003.14089]. “Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning” gives a closely related improvement rule
\[
\pi'_\epsilon
=
\frac{1}{Z_A}\pi\exp(\epsilon \tilde A_\alpha)
=
\frac{1}{Z_A}\pi^{1-\epsilon\alpha}\exp(\epsilon A_\alpha),
\]
interpolating continuously between the current policy and the softmax-greedy policy, thereby connecting policy gradient and soft Q-learning through a KL-proximal step [2005.08844].

A third line is Munchausen-style KLQ. “\(q\)-Munchausen Reinforcement Learning” shows that standard Munchausen DQN,
\[
Q_{k+1}
=
r+\alpha\tau \ln \pi_{k+1}
+\gamma P \Big\langle \pi_{k+1},\, Q_k - \tau \ln \pi_{k+1}\Big\rangle,
\]
can be rewritten in shifted coordinates as
\[
Q'_{k+1}
=
r+\gamma P\Big(
\langle \pi_{k+1},Q'_k\rangle
-\alpha\tau \mathrm{KL}(\pi_{k+1}\|\pi_k)
+(1-\alpha)\tau H(\pi_{k+1})
\Big),
\]
so Munchausen reward augmentation is an implicit KL regulariser to the previous policy in transformed Q-space [2205.07467]. “Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning” further argues that this equivalence is Shannon-specific because \(\tau\ln\pi=Q-V\) holds for Boltzmann policies but not for general Tsallis policies. It therefore proposes Tsallis Advantage Learning,
\[
Q_{k+1}
=
(T^{\pi_{k+1}})^m Q_k + \beta\bigl(Q_k-\pi_{k+1}Q_k\bigr),
\]
as an implicit KL-like surrogate via explicit action-gap enlargement rather than an explicit KL penalty [2205.07885].

## 4. Function approximation, dynamic regularisation, and stability

A major research question in KLQ concerns what survives under function approximation. “Regularized Q-Learning with Linear Function Approximation” addresses this directly for regularised MDPs. With \(\hat Q_\theta=\Phi\theta\), it studies the projected fixed-point equation
\[
\hat Q_{\theta^*}
=
\Pi_{D_\mu}\mathcal{B}_{\tau}\hat Q_{\theta^*},
\]
and emphasises that although \(\mathcal{B}_\tau\) is a contraction in \(\ell_\infty\), the composition \(\Pi_{D_\mu}\mathcal{B}_\tau\) is not generally a contraction in any norm [2401.15196]. To handle this, the paper introduces a bilevel formulation: the lower problem projects the regularised Bellman backup into the feature span, while the upper problem minimises the regularised mean squared projected Bellman error. The resulting single-loop two-timescale algorithm updates a fast variable \(\omega\) tracking the projected backup and a slow variable \(\theta\) moving toward stationarity of the projected objective. Under the paper’s assumptions, including Markovian noise, bounded features, and nonsingularity conditions, the method converges to a stationary point with rate
\[
\frac1T\sum_t \mathbb E\|\nabla J(\theta^t)\|=\mathcal O(T^{-1/4}),
\]
and the extracted regularised policies satisfy an explicit performance bound separating optimisation error, approximation error, and truncation bias [2401.15196].

A complementary perspective is dynamic KL regularisation. “Geometric Value Iteration” studies mirror-descent value iteration with a time-varying KL coefficient,
\[
\pi_{k+1}=G_{\pi_k}^{\lambda_k}(q_k),
\qquad
q_{k+1}=T^{\lambda_k}_{\pi_{k+1}\mid \pi_k}q_k+\epsilon_{k+1},
\]
and proves an asymptotic error bound in which the accumulated error appears as a weighted average
\[
\frac{1}{Z_k}\left\|\sum_{j=1}^k \eta_j \epsilon_j\right\|_\infty,
\qquad
\eta_j=\frac{1}{\lambda_j}.
\]
This motivates the adaptive rule
\[
\lambda_k=\max(\alpha_1\|\epsilon_k\|_\infty,\alpha_2\lambda_{k-1}),
\]
so large Bellman or TD errors trigger stronger KL regularisation [2107.07659]. In the deep version, the target takes a Munchausen-like form with a dynamic bootstrap factor, and the paper reports stable learning even without a target network where constant-KL variants oscillate or fail [2107.07659]. This suggests that in KLQ, \(\lambda\) or \(\tau\) is not merely a bias–exploration hyperparameter; it can be used as an error-control mechanism that modulates aggressiveness of policy improvement.

The averaging view of KL regularisation is sharpened in “Leverage the Average”. In the pure-KL case,
\[
\pi_{k+1}\propto \pi_k \exp(q_k/\lambda)
\]
implies
\[
\pi_{k+1}\propto \exp\!\left(\frac{1}{\lambda}\sum_{j=0}^k q_j\right),
\]
so the policy acts on an average of past Q-estimates rather than only the most recent one. The paper derives a bound
\[
\|q_* - q_{\pi_k}\|_{\infty}
\leq
\frac{2}{1-\gamma}
\left\|
\frac{1}{k}\sum_{j=1}^k \epsilon_j
\right\|_\infty
+
\frac{4}{1-\gamma}\frac{v^\lambda_{\max}}{k},
\]
which exhibits linear dependence on the horizon factor and dependence on the average of errors rather than an accumulated discounted norm [2003.14089]. This is one of the main theoretical arguments for KLQ’s robustness under approximation.

## 5. Variants, generalisations, and domain-specific instantiations

Several supplied papers extend KLQ beyond the standard Shannon/Boltzmann case or specialise it to particular domains.

A prominent generalisation replaces ordinary KL by Tsallis KL. “Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence” defines
\[
D^q_{\mathrm{KL}}(\pi\|\mu)
=
\left\langle \pi,\ -\ln_q\frac{\mu}{\pi}\right\rangle,
\]
and studies the policy iteration recursion
\[
\pi_{k+1}=\argmax_{\pi}\ \langle \pi,Q_k\rangle - D^q_{\mathrm{KL}}(\pi\|\pi_k),
\qquad
Q_{k+1}= r + \gamma P \langle \pi_{k+1}, Q_k\rangle.
\]
For \(q>1\), the induced policy is \(q\)-exponential and can be sparse, unlike the full-support Boltzmann policy of standard KLQ [2301.11476]. The paper’s practical algorithm, MVI\((q)\), is a generalised Munchausen value iteration, and for \(q=2\) it proves convergence to a unique regularised optimal policy [2301.11476]. The closely related “\(q\)-Munchausen Reinforcement Learning” derives an implicit Tsallis-KL-regularised Bellman recursion
\[
Q''_{k+1}
=
r +\gamma P\langle \pi_{k+1},Q''_k\rangle
-\alpha\tau D_q(\pi_{k+1}\|\pi_k)
+(1-\alpha)\tau S_q(\pi_{k+1}),
\]
showing that the Munchausen construction extends to Tsallis geometry when the ordinary logarithm is replaced by the dual \(q^*\)-logarithm [2205.07467].

A different extension concerns RLHF. “KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF” treats autoregressive generation as a token-level MDP, with reference policy \(\pi_b\) given by the SFT model and sparse terminal reward from a reward model [2508.17000]. The KL-regularised return is
\[
\bar{R}_\phi(x,y)=R_\phi(x,y)-\tau \log\!\left(\frac{\pi_\theta(y\mid x)}{\pi_b(y\mid x)}\right),
\]
and the crucial parameterisation is
\[
Q_\theta(s,a)=\tau\log\!\left(\frac{\pi_\theta(a\mid s)}{\pi_b(a\mid s)}\right)+V_\theta(s).
\]
Under this decomposition, the KL-regularised TD error simplifies to
\[
\delta_t = r_{t+1} + \gamma V_\theta(s_{t+1}) - Q_\theta(s_t,a_t),
\]
and the algorithm fits conservative \(\lambda\)-return targets with a squared regression loss rather than an actor loss [2508.17000]. The paper further proves an equivalence, in a specific sense, to a PPO-penalty-like policy/value update, with proximal coefficient
\[
\beta = \tau\left(\frac{1-\alpha}{\alpha}\right),
\]
thereby framing PPO-style RLHF as a KLQ procedure in Q-space [2508.17000].

Safety-constrained KLQ appears in “Safe-Support Q-Learning”. There the Bellman target is regularised toward a safe behavior policy \(\tilde\pi_b\):
\[
y_k^{\rm safe}
=
r_{k+1}
+
{\bf 1}(s_{k+1})\gamma
\max_{\pi}
\left\{
\sum_a \pi(a|s_{k+1})Q(s_{k+1},a)
-\lambda D_{\rm KL}(\pi(\cdot|s_{k+1})\|\tilde\pi_b(\cdot|s_{k+1}))
\right\},
\]
with exact closed form
\[
y_k^{\rm safe}
=
r_{k+1}
+
{\bf 1}(s_{k+1})\gamma\lambda
\ln
\left(
\sum_a \tilde\pi_b(a|s_{k+1})
\exp(Q(s_{k+1},a)/\lambda)
\right).
\]
The induced policy is
\[
\pi_\lambda^*(a|s)
=
\frac{\tilde\pi_b(a|s)e^{Q_\lambda^*(s,a)/\lambda}}
{\sum_{a'}\tilde\pi_b(a'|s)e^{Q_\lambda^*(s,a')/\lambda}},
\]
so improvement is explicitly anchored to a safe-support prior [2604.25379]. This is a canonical example of KLQ with a nonuniform prior used not for optimisation convenience but for structural safety.

Finally, “Residual Policy Gradient” shows that a KL penalty to a prior policy can often be reinterpreted as reward augmentation by \(\log \pi(a|s)\), yielding a soft-Q backup on an augmented MDP [2503.11019]. In the value-based residual setting,
\[
Q^{aug}(s,a)
\leftarrow
r_R(s,a)+\omega'\log \pi(a|s)
+\gamma \mathbb E_{s'} \Big[ \hat\alpha\log \int \exp(Q^{aug}(s',a')/\hat\alpha)\,da' \Big],
\]
which strongly suggests a residual-view interpretation of KLQ as soft Q-learning on a prior-shaped reward [2503.11019].

## 6. Theory, empirical findings, and interpretive issues

Across the supplied material, the main theoretical themes are curvature, averaging, and reference-policy anchoring. In bandits, strong KL regularisation changes the regret regime from \(\tilde \Theta(\sqrt{KT})\) to a fast-rate regime with upper bound
\[
\tilde O(\eta K\log^2 T)
\]
and lower bound
\[
\Omega\!\left(\eta K\log\!\frac{T}{\eta^2K}\right),
\]
with the key mechanism being a second-order regret decomposition under the regularised objective [2603.02155]. In offline zero-sum Markov games, KL regularisation alone suffices to stabilise fitted-Q learning and yields a fast \(\widetilde{\mathcal O}(1/n)\) duality-gap rate under unilateral concentrability, without explicit pessimism [2605.13025]. In approximate value iteration, KL regularisation averages Q-errors across iterations and improves error propagation [2003.14089]. These results suggest a common structural effect: KL regularisation does not only smooth policies; it changes the local geometry of the control problem in a way that can transform first-order error sensitivity into quadratic or averaged error terms.

Empirically, the picture is mixed but coherent. Q-REPS is reported as competitive or better than DQN, PPO, V-MPO, and a parametric-policy REPS baseline on Double Chain, Single Chain, RiverSwim, WideTree, Windy GridWorld, CartPole, and toy MDPs, with ablations showing that increasing \(\eta\) can increase bias through a risk-seeking effect and increasing \(\alpha\) can reduce the action gap [2010.11151]. Tsallis-Munchausen variants show that replacing KL by Tsallis KL can yield significant improvements over the standard Munchausen case across 35 Atari games, while also revealing a trade-off between sparsity and exploration in harder games such as Seaquest and PrivateEye [2301.11476]. \(q\)-Munchausen and TAL both report that a mismatch between policy geometry and reward augmentation can flatten learning curves, while geometry-matched generalisations restore performance [2205.07467] [2205.07885]. In safe-support KLQ, the reported outcome is stable learning, well-calibrated Q-values, and safer behavior with comparable or better performance than baselines, provided the behavior policy has adequate support [2604.25379]. In RLHF, KLQ matches PPO on reward/KL tradeoffs, has essentially identical wall-clock cost, and achieves consistently higher LLM-as-a-judge win rates on summarisation and dialogue tasks [2508.17000].

Several interpretive cautions recur. First, not every regularised Q-learning algorithm is KLQ; the Lim–Lee stabilisation paper is explicitly not [2202.05404]. Second, many practical deep-RL instantiations rely on approximations that are not covered by the strongest theory. “Leverage the Average” notes that exact greedy steps underpin its formal guarantees, and those assumptions do not generally hold with neural networks [2003.14089]. “Regularized Q-Learning with Linear Function Approximation” proves stationarity of a projected objective, not global optimality of the regularised MDP [2401.15196]. “KL-Regularised Q-Learning” for RLHF states that its equivalence to PPO is exact only in a specific sense at the level of idealised update rules [2508.17000]. Third, the choice of divergence is substantive: Tsallis KL changes the policy class from Boltzmann to \(q\)-exponential and can produce sparse support [2301.11476], while transport-based KL analogues such as WKL and KWKL were proposed precisely because ordinary KL can become pathological under support mismatch or low-noise limits [2602.02250]. This suggests that “KLQ” denotes a precise but not exhaustive regularisation geometry.

Taken together, the supplied literature supports a relatively sharp characterisation. KLQ is a value-based or fitted-Q approach to control in which the Bellman backup and induced policy are regularised by KL divergence to a reference policy. Its canonical update is Boltzmann with a prior. Its principal mathematical tools are convex duality, log-partition operators, mirror descent, and soft Bellman recursions. Its main theoretical virtues are robustness through averaging and curvature, and, in some regularised objectives, fast statistical rates. Its main practical virtues are stable improvement around a reference policy, interpretable policy extraction, and adaptability to offline RL, safe RL, and RLHF. Its main limitations are dependence on the quality and role of the reference policy, sensitivity of guarantees to approximation assumptions, and nontrivial interactions between divergence choice, support, and exploration.

Source: https://www.emergentmind.com/topics/kl-regularised-q-learning-klq