KL-Regularised Q-Learning

• KL-regularised Q-Learning is defined by incorporating a statewise KL penalty into the Bellman backup, resulting in a soft, prior-weighted policy improvement rule.
• It unifies algorithmic variants such as Q-REPS and Munchausen methods using convex duality and log-partition operators to enhance stability and error averaging.
• The approach adapts to diverse settings including RLHF and safe RL by anchoring policy updates to a reference, effectively balancing exploration and safety.

Searching arXiv for recent and foundational papers on KL-regularised Q-learning and closely related regularised Q-learning formulations. KL-regularised Q-Learning (KLQ) is a family of reinforcement-learning methods in which action selection is regularised by a Kullback–Leibler divergence to a reference policy, so that greedy improvement is replaced by a prior-weighted soft improvement rule. In its canonical form, the statewise control problem is

$$\max_{\pi(\cdot\mid s)} \left\{ \sum_a \pi(a\mid s) Q(s,a) - \tau \,\mathrm{KL}(\pi(\cdot\mid s)\|\pi_{\mathrm{ref}}(\cdot\mid s)) \right\},$$

whose optimizer is the Boltzmann tilt of the reference policy and whose optimal value is a log-partition or log-sum-exp backup (Dymetman, 30 Apr 2026). In contemporary usage, the term spans several closely related objects: regularised Bellman operators for tabular or approximate control, mirror-descent or trust-region-style policy-improvement schemes, value-based algorithms such as Q-REPS and Munchausen-style methods, and, more recently, token-level action-value methods for online RLHF (Bas-Serrano et al., 2020, Brown et al., 23 Aug 2025). It does not include every paper whose title contains “regularized Q-learning”: some works regularise parameter dynamics for stability under function approximation rather than policy optimisation, and therefore are not KLQ in the standard sense (Lim et al., 2022).

1. Conceptual scope and defining distinction

KLQ is defined by where the regularisation enters. In KL-regularised reinforcement learning, the control objective is modified by a statewise penalty of the form

$$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$

when the reference is uniform. This changes both policy improvement and Bellman optimality, replacing a hard maximisation over actions by a soft, prior-weighted optimisation (Dymetman, 30 Apr 2026, Xi et al., 2024). By contrast, some papers use the word “regularized” to describe shrinkage or damping in the Q-update or parameter dynamics. “Regularized Q-Learning” by Lim and Lee is of that latter kind: it studies Q-learning with linear function approximation and adds a stabilising regulariser to the update dynamics; it has no KL divergence, no reference policy $\mu$, and no soft Bellman operator, so it is not a KLQ method (Lim et al., 2022).

This distinction is central because KLQ regularises policy selection, not merely numerical optimisation. In KLQ, the reference policy may be a previous iterate, a fixed prior, a behavior policy, an SFT model in RLHF, or a safe-support policy in safety-constrained settings. The regulariser therefore has a control-theoretic role: it constrains deviation from a reference mechanism while preserving a closed-form greedy step (Brown et al., 23 Aug 2025, Lim et al., 28 Apr 2026).

A second important distinction is between explicit and implicit KL regularisation. Some methods write the KL penalty directly into the objective or Bellman operator, as in Q-REPS or safe-support KLQ (Bas-Serrano et al., 2020, Lim et al., 28 Apr 2026). Others induce an equivalent or approximate KL effect through reward augmentation or advantage learning, as in Munchausen-style methods (Zhu et al., 2022, Zhu et al., 2022). This suggests that KLQ is best viewed as a broader design pattern organised around KL-shaped improvement geometry rather than a single algorithm.

2. Variational foundation and soft Bellman structure

The cleanest mathematical foundation in the supplied material is the exponential-family identity developed in “Exponential families from a single KL identity” (Dymetman, 30 Apr 2026). For an exponential family $p_\lambda$, the paper proves

$$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$

and derives the Gibbs variational principle

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$

Specialising to scalar reward $r$ with $\lambda=1/\beta$, it yields

$$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$

with unique maximizer

$q^*(y)\propto a(y)e^{r(y)/\beta}.$

Mapped statewise to RL, $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$0 becomes $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$1, $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$2 becomes $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$3, and $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$4 becomes $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$5, giving the canonical KLQ update

$$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$6

together with the soft value

$$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$7

(Dymetman, 30 Apr 2026).

Within the broader regularised-MDP literature, the same structure appears through convex conjugacy. “Regularized Q-Learning with Linear Function Approximation” defines a strongly convex policy regulariser $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$8, its conjugate $$-\tau D_{\mathrm{KL}}(\pi(\cdot|s)\|\mu(\cdot|s)) \quad\text{or equivalently}\quad \tau \mathcal H(\pi(\cdot|s))$$9, and the regularised Bellman operator

$\mu$0

For negative Shannon entropy, this reduces to

$\mu$1

recovering the standard soft-Q backup. The induced policy is

$\mu$2

which becomes softmax in the Shannon case and prior-weighted Boltzmann under a KL-to-prior specialisation of $\mu$3 (Xi et al., 2024). This suggests that KLQ is naturally embedded in a more general convex-dual framework, with KL and entropy as particular choices of regulariser.

The same statewise variational identity also underlies the bandit setting. “Near-Optimal Regret for KL-Regularized Multi-Armed Bandits” defines

$\mu$4

whose optimizer is

$\mu$5

This is the one-state analogue of KLQ’s soft greedy step, and the paper further proves

$\mu$6

showing that the regularised suboptimality gap is itself a KL divergence to the soft optimum (Ji et al., 2 Mar 2026).

3. Core algorithmic forms

Several distinct algorithmic realisations of KLQ appear in the supplied sources.

The most direct value-based instance is Q-REPS, introduced in “Logistic Q-Learning” (Bas-Serrano et al., 2020). Starting from a regularised occupancy-measure LP, the paper introduces a dual Q-function $\mu$7 and defines

$\mu$8

$\mu$9

The regularised primal optimum satisfies

$p_\lambda$0

and the policy update is

$p_\lambda$1

This is a KL-regularised policy-improvement step around a reference policy $p_\lambda$2, while critic learning is performed via the convex logistic Bellman error

$p_\lambda$3

(Bas-Serrano et al., 2020). Relative to canonical soft Q-learning, Q-REPS is distinctive because KL regularisation is derived from regularised occupancy optimisation and critic fitting uses a log-sum-exp Bellman objective rather than squared Bellman error.

A second line is KL-regularised approximate value iteration and mirror-descent value iteration. “Leverage the Average” studies the regularised greedy operator

$p_\lambda$4

with Bellman evaluation

$p_\lambda$5

Its closed-form greedy solution is

$p_\lambda$6

specialising to

$p_\lambda$7

in the pure-KL case (Vieillard et al., 2020). “Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning” gives a closely related improvement rule

$p_\lambda$8

interpolating continuously between the current policy and the softmax-greedy policy, thereby connecting policy gradient and soft Q-learning through a KL-proximal step (Lee, 2020).

A third line is Munchausen-style KLQ. “$p_\lambda$9-Munchausen Reinforcement Learning” shows that standard Munchausen DQN,

$$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$0

can be rewritten in shifted coordinates as

$$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$1

so Munchausen reward augmentation is an implicit KL regulariser to the previous policy in transformed Q-space (Zhu et al., 2022). “Enforcing KL Regularization in General Tsallis Entropy Reinforcement Learning via Advantage Learning” further argues that this equivalence is Shannon-specific because $$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$2 holds for Boltzmann policies but not for general Tsallis policies. It therefore proposes Tsallis Advantage Learning,

$$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$3

as an implicit KL-like surrogate via explicit action-gap enlargement rather than an explicit KL penalty (Zhu et al., 2022).

4. Function approximation, dynamic regularisation, and stability

A major research question in KLQ concerns what survives under function approximation. “Regularized Q-Learning with Linear Function Approximation” addresses this directly for regularised MDPs. With $$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$4, it studies the projected fixed-point equation

$$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$5

and emphasises that although $$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$6 is a contraction in $$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$7, the composition $$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$8 is not generally a contraction in any norm (Xi et al., 2024). To handle this, the paper introduces a bilevel formulation: the lower problem projects the regularised Bellman backup into the feature span, while the upper problem minimises the regularised mean squared projected Bellman error. The resulting single-loop two-timescale algorithm updates a fast variable $$\mathrm{KL}(q \| p_{\lambda_2}) - \mathrm{KL}(q \| p_{\lambda_1}) = A(\lambda_2) - A(\lambda_1) + \mu_q \cdot (\lambda_1 - \lambda_2),$$9 tracking the projected backup and a slow variable $$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$0 moving toward stationarity of the projected objective. Under the paper’s assumptions, including Markovian noise, bounded features, and nonsingularity conditions, the method converges to a stationary point with rate

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$1

and the extracted regularised policies satisfy an explicit performance bound separating optimisation error, approximation error, and truncation bias (Xi et al., 2024).

A complementary perspective is dynamic KL regularisation. “Geometric Value Iteration” studies mirror-descent value iteration with a time-varying KL coefficient,

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$2

and proves an asymptotic error bound in which the accumulated error appears as a weighted average

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$3

This motivates the adaptive rule

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$4

so large Bellman or TD errors trigger stronger KL regularisation (Kitamura et al., 2021). In the deep version, the target takes a Munchausen-like form with a dynamic bootstrap factor, and the paper reports stable learning even without a target network where constant-KL variants oscillate or fail (Kitamura et al., 2021). This suggests that in KLQ, $$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$5 or $$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$6 is not merely a bias–exploration hyperparameter; it can be used as an error-control mechanism that modulates aggressiveness of policy improvement.

The averaging view of KL regularisation is sharpened in “Leverage the Average”. In the pure-KL case,

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$7

implies

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$8

so the policy acts on an average of past Q-estimates rather than only the most recent one. The paper derives a bound

$$A(\lambda)=\sup_q \left\{ \lambda\cdot\mu_q - \mathrm{KL}(q\|a)\right\}.$$9

which exhibits linear dependence on the horizon factor and dependence on the average of errors rather than an accumulated discounted norm (Vieillard et al., 2020). This is one of the main theoretical arguments for KLQ’s robustness under approximation.

5. Variants, generalisations, and domain-specific instantiations

Several supplied papers extend KLQ beyond the standard Shannon/Boltzmann case or specialise it to particular domains.

A prominent generalisation replaces ordinary KL by Tsallis KL. “Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence” defines

$r$0

and studies the policy iteration recursion

$r$1

For $r$2, the induced policy is $r$3-exponential and can be sparse, unlike the full-support Boltzmann policy of standard KLQ (Zhu et al., 2023). The paper’s practical algorithm, MVI$r$4, is a generalised Munchausen value iteration, and for $r$5 it proves convergence to a unique regularised optimal policy (Zhu et al., 2023). The closely related “$r$6-Munchausen Reinforcement Learning” derives an implicit Tsallis-KL-regularised Bellman recursion

$r$7

showing that the Munchausen construction extends to Tsallis geometry when the ordinary logarithm is replaced by the dual $r$8-logarithm (Zhu et al., 2022).

A different extension concerns RLHF. “KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF” treats autoregressive generation as a token-level MDP, with reference policy $r$9 given by the SFT model and sparse terminal reward from a reward model (Brown et al., 23 Aug 2025). The KL-regularised return is

$\lambda=1/\beta$0

and the crucial parameterisation is

$\lambda=1/\beta$1

Under this decomposition, the KL-regularised TD error simplifies to

$\lambda=1/\beta$2

and the algorithm fits conservative $\lambda=1/\beta$3-return targets with a squared regression loss rather than an actor loss (Brown et al., 23 Aug 2025). The paper further proves an equivalence, in a specific sense, to a PPO-penalty-like policy/value update, with proximal coefficient

$\lambda=1/\beta$4

thereby framing PPO-style RLHF as a KLQ procedure in Q-space (Brown et al., 23 Aug 2025).

Safety-constrained KLQ appears in “Safe-Support Q-Learning”. There the Bellman target is regularised toward a safe behavior policy $\lambda=1/\beta$5: $\lambda=1/\beta$6 with exact closed form

$\lambda=1/\beta$7

The induced policy is

$\lambda=1/\beta$8

so improvement is explicitly anchored to a safe-support prior (Lim et al., 28 Apr 2026). This is a canonical example of KLQ with a nonuniform prior used not for optimisation convenience but for structural safety.

Finally, “Residual Policy Gradient” shows that a KL penalty to a prior policy can often be reinterpreted as reward augmentation by $\lambda=1/\beta$9, yielding a soft-Q backup on an augmented MDP (Wang et al., 14 Mar 2025). In the value-based residual setting,

$$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$0

which strongly suggests a residual-view interpretation of KLQ as soft Q-learning on a prior-shaped reward (Wang et al., 14 Mar 2025).

6. Theory, empirical findings, and interpretive issues

Across the supplied material, the main theoretical themes are curvature, averaging, and reference-policy anchoring. In bandits, strong KL regularisation changes the regret regime from $$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$1 to a fast-rate regime with upper bound

$$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$2

and lower bound

$$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$3

with the key mechanism being a second-order regret decomposition under the regularised objective (Ji et al., 2 Mar 2026). In offline zero-sum Markov games, KL regularisation alone suffices to stabilise fitted-Q learning and yields a fast $$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$4 duality-gap rate under unilateral concentrability, without explicit pessimism (Chen et al., 13 May 2026). In approximate value iteration, KL regularisation averages Q-errors across iterations and improves error propagation (Vieillard et al., 2020). These results suggest a common structural effect: KL regularisation does not only smooth policies; it changes the local geometry of the control problem in a way that can transform first-order error sensitivity into quadratic or averaged error terms.

Empirically, the picture is mixed but coherent. Q-REPS is reported as competitive or better than DQN, PPO, V-MPO, and a parametric-policy REPS baseline on Double Chain, Single Chain, RiverSwim, WideTree, Windy GridWorld, CartPole, and toy MDPs, with ablations showing that increasing $$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$5 can increase bias through a risk-seeking effect and increasing $$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$6 can reduce the action gap (Bas-Serrano et al., 2020). Tsallis-Munchausen variants show that replacing KL by Tsallis KL can yield significant improvements over the standard Munchausen case across 35 Atari games, while also revealing a trade-off between sparsity and exploration in harder games such as Seaquest and PrivateEye (Zhu et al., 2023). $$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$7-Munchausen and TAL both report that a mismatch between policy geometry and reward augmentation can flatten learning curves, while geometry-matched generalisations restore performance (Zhu et al., 2022, Zhu et al., 2022). In safe-support KLQ, the reported outcome is stable learning, well-calibrated Q-values, and safer behavior with comparable or better performance than baselines, provided the behavior policy has adequate support (Lim et al., 28 Apr 2026). In RLHF, KLQ matches PPO on reward/KL tradeoffs, has essentially identical wall-clock cost, and achieves consistently higher LLM-as-a-judge win rates on summarisation and dialogue tasks (Brown et al., 23 Aug 2025).

Several interpretive cautions recur. First, not every regularised Q-learning algorithm is KLQ; the Lim–Lee stabilisation paper is explicitly not (Lim et al., 2022). Second, many practical deep-RL instantiations rely on approximations that are not covered by the strongest theory. “Leverage the Average” notes that exact greedy steps underpin its formal guarantees, and those assumptions do not generally hold with neural networks (Vieillard et al., 2020). “Regularized Q-Learning with Linear Function Approximation” proves stationarity of a projected objective, not global optimality of the regularised MDP (Xi et al., 2024). “KL-Regularised Q-Learning” for RLHF states that its equivalence to PPO is exact only in a specific sense at the level of idealised update rules (Brown et al., 23 Aug 2025). Third, the choice of divergence is substantive: Tsallis KL changes the policy class from Boltzmann to $$\max_q \left\{ E_q[r] - \beta\, \mathrm{KL}(q\|a) \right\} = \beta A(1/\beta),$$8-exponential and can produce sparse support (Zhu et al., 2023), while transport-based KL analogues such as WKL and KWKL were proposed precisely because ordinary KL can become pathological under support mismatch or low-noise limits (Stein et al., 2 Feb 2026). This suggests that “KLQ” denotes a precise but not exhaustive regularisation geometry.

Taken together, the supplied literature supports a relatively sharp characterisation. KLQ is a value-based or fitted-Q approach to control in which the Bellman backup and induced policy are regularised by KL divergence to a reference policy. Its canonical update is Boltzmann with a prior. Its principal mathematical tools are convex duality, log-partition operators, mirror descent, and soft Bellman recursions. Its main theoretical virtues are robustness through averaging and curvature, and, in some regularised objectives, fast statistical rates. Its main practical virtues are stable improvement around a reference policy, interpretable policy extraction, and adaptability to offline RL, safe RL, and RLHF. Its main limitations are dependence on the quality and role of the reference policy, sensitivity of guarantees to approximation assumptions, and nontrivial interactions between divergence choice, support, and exploration.