Boltzmann Q-Learning Methods

• Boltzmann Q-learning is a family of value-based reinforcement learning methods that incorporate Boltzmann distributions into policies, Bellman backups, or value approximators.
• Its approaches include softmax action selection, entropy-regularized Bellman operators, and free-energy-based Q-value estimates using Boltzmann or quantum Boltzmann machines.
• These methods present trade-offs in convergence, gap sensitivity, and implementation in continuous and multi-agent settings, driving ongoing research into stability and scalability.

Boltzmann Q-learning denotes a family of value-based reinforcement-learning methods in which a Boltzmann object enters the learning procedure through the policy, the Bellman backup, or the value approximator itself. In the literature, the term spans at least three distinct constructions: classical Q-learning with Boltzmann or softmax action selection, soft or entropy-regularized Bellman updates that replace the hard maximum by a Boltzmann-type operator, and free-energy-based methods in which $Q(s,a)$ is represented by the negative free energy of a Boltzmann machine or quantum Boltzmann machine (Kianercy et al., 2011, Pan et al., 2019, Crawford et al., 2016, Schenk et al., 2022).

1. Conceptual scope and formal core

At the level of control objective, these methods inherit the standard action-value formalism. One representative statement is

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$

with temporal-difference updating of the form

$$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$

Boltzmann variants modify either the policy used to choose actions, the operator used in the target, or the parameterization of $Q$ itself (Schenk et al., 2022).

Sense of “Boltzmann”	Mechanism	Representative papers
Softmax policy	$\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$	(Kianercy et al., 2011, D'Amico-Wong et al., 2024, Singh et al., 18 Feb 2026)
Soft Bellman backup	replace $\max_a Q(s,a)$ by a Boltzmann or log-sum-exp operator	(Pan et al., 2019, Jeong et al., 2024, Mahajan et al., 15 Apr 2026)
Free-energy value approximation	$Q(s,a)\approx -F(s,a)$ for a clamped BM or QBM	(Crawford et al., 2016, Schenk et al., 2022, Gerlach et al., 6 Nov 2025)

This taxonomy matters because the same label can denote substantively different algorithms. In the quantum-Boltzmann literature, for example, “Boltzmann” often refers to the statistical-mechanics model used to represent $Q$, while exploration is explicitly $\epsilon$-greedy rather than softmax over actions (Müller et al., 2021).

2. Softmax action selection and classical Boltzmann dynamics

In the classical usage, Boltzmann Q-learning means that actions are sampled according to a softmax over current action values. In continuous-time analysis for an agent with action values $Q_i(t)$, the policy takes the Gibbs form

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$0

where $$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$1 is the temperature (Kianercy et al., 2011). For two-player two-action games, this induces the reduced dynamics

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$2

For any non-zero exploration rate, the dynamics is dissipative, and trajectories converge to interior rest points that are generally different from the game’s Nash equilibria (Kianercy et al., 2011). In coordination and anti-coordination games, multiple rest points can exist at low exploration and collapse through bifurcation at critical exploration rates; with asymmetric temperatures, even some single-Nash-equilibrium games can exhibit additional non-Nash rest points over a finite temperature range (Kianercy et al., 2011).

In online discounted MDPs, Boltzmann exploration with decaying temperature has now been analyzed through high-probability regret bounds. The studied policy is

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$3

combined with standard asynchronous Q-learning updates (Singh et al., 18 Feb 2026). The resulting regret bound is

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$4

where the MDP gap is

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$5

The analysis shows that Boltzmann Q-learning can achieve sublinear regret for sufficiently large gaps, but for small gaps the regret deteriorates and can approach linear growth (Singh et al., 18 Feb 2026). The critical mechanism is that the suboptimal-action mass decays like $$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$6, so low-gap problems remain difficult even when the temperature is annealed.

3. Soft Bellman operators and entropy-regularized variants

A second line of work uses a Boltzmann operator inside the Bellman target rather than solely in the behavior policy. A central example is the Boltzmann softmax operator

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$7

which interpolates between averaging and maximization (Pan et al., 2019). Its direct, fixed-temperature use is problematic: it does not satisfy the non-expansion property, and its naive substitution into value iteration may fail to converge (Pan et al., 2019). The proposed remedy is the dynamic Boltzmann softmax update, in which $$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$8. The key estimate is

$$Q(s,a\vert\theta) = \mathbb{E}_\pi \left[\sum_{k=1}^{N} \gamma^{k-1} r_{t+k}\vert s_t=s, a_t=a\right],$$9

which yields convergence to $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$0 for value iteration and to $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$1 for tabular Q-learning when $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$2 (Pan et al., 2019). In deep RL, the corresponding DBS-DQN replaces the DQN max target by a dynamic Boltzmann target and was reported to outperform DQN in 40 out of 49 Atari games, with median human-normalized score $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$3 versus $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$4 for DQN (Pan et al., 2019).

Soft Q-learning in entropy-regularized control adopts a related but distinct construction. With a default policy $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$5 and temperature $$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$6, the induced Boltzmann policy is

$$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$7

and the soft state value is

$$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$8

The one-step TD error becomes

$$Q(s_t, a_t)\leftarrow Q(s_t, a_t) + \alpha \left[r_{t+1} + \gamma\, \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t)\right].$$9

and recent work extends this to formal $Q$0-step returns, a Soft Tree Backup operator, and an off-policy Soft $Q$1 with eligibility traces under arbitrary behavior policies (Mahajan et al., 15 Apr 2026). The off-policy trace decay is modulated by the target-policy probabilities $Q$2, giving a soft analogue of Tree Backup rather than importance-sampled SARSA (Mahajan et al., 15 Apr 2026).

A separate finite-time analysis of soft Q-learning studies both log-sum-exp and Boltzmann operators under constant stepsize and i.i.d. state-action sampling. For the Boltzmann variant, the update target uses

$Q$3

with $Q$4, and the finite-time error bound relative to $Q$5 has the structure

$Q$6

making explicit the separate contributions of switching transients, constant-stepsize noise, and Boltzmann approximation bias (Jeong et al., 2024).

4. Multi-agent and game-theoretic extensions

In multi-agent settings, Boltzmann Q-learning has been treated both as a baseline algorithm and as an object of dynamical analysis. In extensive-form and imperfect-information games, one formulation freezes all players’ policies, samples one trajectory for each agent, performs TD updates

$Q$7

and then updates the policy by softmaxing the Q-values:

$Q$8

This BQL is trajectory-based and bootstrapping, and the temperature schedule used in the experiments was

$Q$9

The same work shows that BQL converges in MDPs and perfect-information games under suitable schedules, but in imperfect-information multi-agent games it has no Nash-convergence guarantees; the ABCs algorithm uses child stationarity to interpolate between BQL and CFR, converging to the optimal policy in MDPs with at most an $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$0 slowdown compared to BQL and to a Nash equilibrium in two-player zero-sum games under a perfect stationarity oracle (D'Amico-Wong et al., 2024).

A more fine-grained deterministic model of incremental multi-agent Boltzmann Q-learning shows that earlier deterministic approximations actually model frequency-adjusted or batch variants rather than the original incremental algorithm. For the true incremental process, only the Q-value of the played action is updated, so the expected deterministic map contains a factor $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$1 multiplying the TD term. This update-frequency asymmetry fundamentally changes the dynamics: the process must be analyzed in Q-space rather than policy space, the discount factor $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$2 affects stability even in a single-state repeated game, and apparent cooperation in the Prisoner’s Dilemma can be a metastable transient rather than a true equilibrium (Goll et al., 2024). For $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$3 and $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$4, increasing $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$5 produces a supercritical Neimark-Sacker bifurcation at approximately $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$6, beyond which a stable limit cycle surrounds an unstable focus (Goll et al., 2024).

Boltzmann policies also appear in multi-agent inverse reinforcement learning through marginalized critics. In MAMQL, each agent learns

$\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$7

which yields the explicit Boltzmann policy

$\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$8

The paper reports that MAMQL significantly outperforms previous multi-agent methods in average reward, sample efficiency, and reward recovery by often more than $\pi(a\mid s)\propto \exp(Q(s,a)/\tau)$9-$\max_a Q(s,a)$0, with convergence roughly $\max_a Q(s,a)$1-$\max_a Q(s,a)$2 faster than IQ-Learn MA on Gems, Overcooked, and Highway (Haynam et al., 6 Mar 2025).

5. Free-energy-based Boltzmann machines and quantum variants

A distinct tradition identifies $\max_a Q(s,a)$3 with the negative free energy of a clamped Boltzmann machine. In this formulation, the visible variables encode a state-action pair, the hidden variables define a conditional Boltzmann or Gibbs distribution, and the value estimate is

$\max_a Q(s,a)$4

For clamped QBMs, the free energy is

$\max_a Q(s,a)$5

and weight updates are proportional to the TD residual multiplied by hidden expectations or correlations (Crawford et al., 2016). On small maze benchmarks, DBM-RL outperforms RBM-RL, and QBM-RL slightly outperforms DBM-RL, with the best-performing method using a nonzero final transverse field $\max_a Q(s,a)$6 (Crawford et al., 2016).

The same idea has been extended to multi-agent grid domains. One such architecture uses a QBM approximator, simulated quantum annealing, experience replay, and a target network. The stabilized method solves a $\max_a Q(s,a)$7 two-agent domain with median reward around $\max_a Q(s,a)$8, near optimum, but still fails on a $\max_a Q(s,a)$9 two-agent domain, with median reward around $Q(s,a)\approx -F(s,a)$0 (Müller et al., 2021). The paper explicitly states that its exploration policy is $Q(s,a)\approx -F(s,a)$1-greedy, so the Boltzmann component lies in the function approximator rather than the action-selection rule (Müller et al., 2021).

The CERN beam-line work develops free-energy-based reinforcement learning with clamped quantum Boltzmann machines for control tasks. Its key approximation is

$Q(s,a)\approx -F(s,a)$2

with the QBM Hamiltonian defined by a transverse-field Ising model and sampled through simulated or hardware quantum annealing (Schenk et al., 2022). For continuous states with discrete actions, the TT24-T4 study reports that FERL reaches $Q(s,a)\approx -F(s,a)$3 policy optimality in about 10 training steps, versus about 380 for the optimized DQN; on D-Wave hardware, the greedy policy is still $Q(s,a)\approx -F(s,a)$4 optimal after visiting only 15 states during training (Schenk et al., 2022). For continuous actions, the paper adopts a hybrid actor-critic based on DDPG in which the actor is classical and the critic remains a QBM free-energy estimator. On the 10-dimensional AWAKE beam line, the hybrid method reaches near-perfect behavior after about 50 agent-environment interactions, compared to about 70 for classical DDPG, while the actor gradient is computed by numerical finite differences because the entropy term in the free energy is not easy to differentiate analytically (Schenk et al., 2022).

A subsequent theoretical extension introduces the Continuous Semi-Quantum Boltzmann Machine, in which visible variables are continuous and classical, hidden variables remain quantum, and the free energy decomposes as

$Q(s,a)\approx -F(s,a)$5

with analytic gradients

$Q(s,a)\approx -F(s,a)$6

The proposed continuous Q-learning framework replaces explicit global maximization by sampling from the CSQBM-induced conditional action distribution, thereby avoiding the finite-difference difficulty of earlier continuous-action QBM schemes (Gerlach et al., 6 Nov 2025). The contribution is theoretical rather than empirical: the paper reports no RL experiments and leaves evaluation on real-world continuous-control problems to future work (Gerlach et al., 6 Nov 2025).

6. Recurring misconceptions, limitations, and open directions

A persistent source of confusion is that “Boltzmann Q-learning” does not name a single algorithm. In one body of work it means softmax exploration over tabular or parametric Q-values; in another it means a Boltzmann or log-sum-exp Bellman operator; in a third it means free-energy-based value approximation using Boltzmann machines or quantum Boltzmann machines. This suggests that the label identifies a shared statistical-mechanics motif rather than a unique update rule.

A second misconception is to equate all Boltzmann methods with Boltzmann exploration. The QBM papers are explicit that exploration is $Q(s,a)\approx -F(s,a)$7-greedy and that “Boltzmann” refers to the energy-based critic or value approximator, not the policy (Müller et al., 2021, Schenk et al., 2022). Conversely, the classical softmax literature uses Boltzmann action selection directly and shows that positive exploration can move learning dynamics toward interior rest points that generally do not coincide with Nash equilibria (Kianercy et al., 2011).

Theoretical guarantees are likewise heterogeneous. Fixed-temperature Boltzmann softmax backups do not satisfy the non-expansion property and may fail to converge even in value iteration, whereas dynamic schedules with $Q(s,a)\approx -F(s,a)$8 restore convergence (Pan et al., 2019). Online Boltzmann exploration in discounted MDPs is gap-sensitive: the regret can be sublinear when the MDP gap is sufficiently large, but can approach linear growth when the gap is small (Singh et al., 18 Feb 2026). In multi-agent games, plain BQL has no Nash-convergence guarantee, which is precisely why hybridizations such as ABCs or marginalized-critic formulations were proposed (D'Amico-Wong et al., 2024, Haynam et al., 6 Mar 2025).

The quantum and energy-based branch has its own limits. Continuous-action control required either a hybrid actor-critic with finite-difference actor gradients (Schenk et al., 2022) or a new semi-quantum continuous visible model with analytic gradients that, so far, remains theoretical (Gerlach et al., 6 Nov 2025). Hardware studies are constrained by QPU size and access time, and the CERN beam-line paper is explicit that it does not claim a provable quantum speedup; the benefits are empirical and task-dependent (Schenk et al., 2022). The multi-agent QBM study likewise identifies QPU size as a current bottleneck and reports that larger domains remain unsolved (Müller et al., 2021).

Taken together, the literature portrays Boltzmann Q-learning as a technically diverse class of methods organized around a common replacement of hard greedy structures by Boltzmann distributions, log-partition values, or free energies. The resulting algorithms range from softmax-exploratory Q-learning in tabular MDPs, to entropy-regularized multi-step off-policy control, to quantum Boltzmann critics for continuous control. The principal unresolved questions concern stability under nonstationarity, finite-time guarantees beyond specialized settings, scalable continuous-action formulations, and whether annealing-based free-energy critics can deliver advantages beyond the small and medium-scale regimes demonstrated so far.