Prior-Scaled Boltzmann Exploration

• Prior-scaled Boltzmann exploration is a refined softmax method that incorporates external and per-arm uncertainty scaling to balance exploration and exploitation.
• It employs strategies like non-monotone schedules, count-based adjustments, and pseudo-count initialization to mitigate the suboptimality of classical softmax.
• The approach achieves near-optimal or asymptotically optimal regret bounds in stochastic bandits and extends its utility to reinforcement learning and Bayesian optimization.

Prior-scaled Boltzmann exploration refers to Boltzmann or softmax action-selection schemes in which the exponential weighting is modified by an explicit scale: a global temperature or learning-rate schedule, problem-parameter-dependent tuning, per-arm uncertainty scaling, pull counts, or initialization through pseudo-counts. In the stochastic multi-armed bandit setting, the topic is motivated by a central negative result: classical Boltzmann exploration with any monotone learning-rate sequence induces suboptimal behavior, whereas more structured scaling can recover near-optimal or optimal regret (Cesa-Bianchi et al., 2017, Bian et al., 2021). Related uses appear in parameterized reinforcement learning, where the inverse temperature is meta-learned online, and in batch Bayesian optimization, where sampling is performed from a Boltzmann distribution over an acquisition function with temperature-controlled diversity (Khamassi et al., 2016, Bloor et al., 29 Jun 2026).

1. Classical softmax and the motivation for scaling

Standard Boltzmann Exploration (BE), also called softmax, selects arms with probability proportional to $\exp(\hat{\mu}_{t,i}/\sigma^2)$, where $\hat{\mu}_{t,i}$ is the empirical mean reward for arm $i$ and $\sigma^2$ is a temperature parameter (Bian et al., 2021). An equivalent parameterization writes the policy as $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$, with $\eta_t$ as a learning rate (Cesa-Bianchi et al., 2017). The attraction of this rule is its simplicity: larger estimated values receive exponentially larger mass.

The principal difficulty is that this classical form ignores uncertainty heterogeneity across arms. In particular, it uses the same learning rate for all arms, not accounting for uncertainty that decreases with more samples (Cesa-Bianchi et al., 2017). The theoretical consequences are severe. If $\eta_t$ increases too slowly, specifically $o(\log t / \Delta)$, the algorithm over-explores and regret is $\omega(\log T/\Delta)$; if $\eta_t > 2\log t$, the algorithm may commit to a suboptimal arm prematurely and suffer linear regret, $\hat{\mu}_{t,i}$0 (Cesa-Bianchi et al., 2017). The same work states that any monotonic $\hat{\mu}_{t,i}$1 schedule leads to problematic asymptotic behavior.

These results explain why prior scaling became necessary within the Boltzmann family. The problem is not merely that temperature must be “tuned well,” but that the naive global schedule is structurally misaligned with the information geometry of bandit learning. This suggests that successful Boltzmann methods require scales that reflect either prior knowledge of the instance or arm-specific evidence accumulated online.

2. What “prior scaling” denotes across the literature

One form of prior scaling uses external knowledge of key problem parameters. A non-monotone schedule proposed as a remedy sets

$\hat{\mu}_{t,i}$2

and achieves

$\hat{\mu}_{t,i}$3

Its limitation is explicit: it requires foreknowledge of the gap $\hat{\mu}_{t,i}$4 and horizon $\hat{\mu}_{t,i}$5 (Cesa-Bianchi et al., 2017). The same source describes this as an approach dependent on prior knowledge, or “prior-scales” in the sense of tuning to problem parameters.

A second form replaces global scaling by arm-specific scaling. Boltzmann–Gumbel Exploration (BGE) uses different learning rates for different arms, with

$\hat{\mu}_{t,i}$6

where $\hat{\mu}_{t,i}$7 is the number of times arm $\hat{\mu}_{t,i}$8 has been pulled and $\hat{\mu}_{t,i}$9 is a constant (Cesa-Bianchi et al., 2017). The scale of the perturbation is therefore inversely proportional to the square root of the number of samples. The paper states that this scaling plays the role of a prior-scale for each arm, controlled only by actual data.

A third form appears in count-based Gibbs distributions. In Maillard Sampling, the exponent is scaled by $i$0 rather than by the empirical mean alone, and priors can be incorporated by initializing $i$1 and the reward counts accordingly, which shifts the initial bias and impacts early exploration (Bian et al., 2021). Here prior scaling is tied to pseudo-counts and initial sufficient statistics.

A fourth form uses temperature over utilities rather than value estimates. In B3O, prior-scaling in Boltzmann or Gibbs-style exploration typically refers to rescales within the utility or via the temperature, and the inverse temperature $i$2 is the key parameter for prior scaling (Bloor et al., 29 Jun 2026). The paper also states that there is no explicit incorporation of a “prior scaling” factor beyond this temperature.

Taken together, these uses show that “prior scaling” is not limited to one mathematical device. This suggests that the term is best understood as a family resemblance: external schedules based on known instance parameters, per-arm scales derived from uncertainty, pseudo-count initialization, and utility-level annealing all instantiate the same effort to correct the failure modes of classical softmax.

3. Count-scaled Boltzmann exploration: Maillard Sampling and MS$i$3

Maillard Sampling (MS) revisits an algorithm from the PhD thesis of Maillard (2013) for the $i$4-armed bandit problem and computes the probability of choosing each arm in a closed form (Bian et al., 2021). The sampling probability is

$i$5

where $i$6 is the number of times arm $i$7 has been pulled so far, $i$8 is the empirical mean reward, and $i$9 is a tunable scaling parameter with $\sigma^2$0 for theoretical guarantees (Bian et al., 2021). The method is described as reminiscent of a Boltzmann or gibbsified distribution, but crucially the score is scaled by $\sigma^2$1, not just the mean.

The analysis reported for MS establishes both asymptotical optimality and a $\sigma^2$2 minimax regret bound, matching the known bounds for asymptotically optimal UCB (Bian et al., 2021). The same source emphasizes a practical merit of the closed-form probabilities: bandit-logged data from running MS can be readily used for counterfactual evaluation, unlike Thompson sampling.

MS$\sigma^2$3 augments the exponent with two correction terms:

$\sigma^2$4

Here $\sigma^2$5 and $\sigma^2$6 are additional tunable parameters; $\sigma^2$7 encourages, in non-asymptotic regimes, pulling the empirical best arm more, and $\sigma^2$8 helps in the theoretical regret guarantee, mostly nudging instance-independent minimax bounds (Bian et al., 2021).

The reported improvement is twofold. First, MS$\sigma^2$9 improves the minimax bound to $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$0 (Bian et al., 2021). Second, it can be tuned to be aggressive, meaning less exploration, without losing asymptotic optimality; the paper identifies this as a unique feature unavailable from existing bandit algorithms (Bian et al., 2021). The details further state that lowering $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$1 while keeping $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$2 and using positive $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$3 values makes MS$p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$4 more greedy, rapidly focusing on high-mean arms, yet aggressive tuning does not void the theoretical guarantees.

4. Per-arm scaling and prior-free Boltzmann exploration

“Boltzmann Exploration Done Right” develops a distinct remedy to classical softmax under the name Boltzmann–Gumbel Exploration (BGE) (Cesa-Bianchi et al., 2017). Rather than exponentiating empirical means directly, BGE samples independent standard Gumbel perturbations $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$5 and selects

$p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$6

with the arm-specific scale $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$7 (Cesa-Bianchi et al., 2017). The construction is described as analogous to randomized optimism, where the scale of the perturbation is inversely proportional to the square root of the number of samples and mirrors UCB’s bonus, but random.

The theoretical guarantees are near-optimal rather than asymptotically optimal in the exact sense claimed for MS. For $p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$8-subgaussian rewards, the distribution-dependent regret is

$p_{t,i} \propto e^{\eta_t \widehat{\mu}_{t,i}}$9

and the distribution-independent regret satisfies

$\eta_t$0

The same work states that the method achieves these bounds without requiring prior knowledge of $\eta_t$1 or $\eta_t$2 (Cesa-Bianchi et al., 2017). It also gives a heavy-tailed extension based on a robust mean estimator using Catoni’s influence function, with the same $\eta_t$3 distribution-independent rate and a variance-based distribution-dependent bound (Cesa-Bianchi et al., 2017).

Relative to prior-scaled schedules based on externally supplied instance parameters, BGE is explicitly prior-free. The contrast is important: with prior knowledge, the non-monotone schedule achieves near-optimal regret but is described as artificial and rarely applicable; without prior knowledge, BGE operates in a fully online and adaptive manner (Cesa-Bianchi et al., 2017). This suggests a bifurcation within prior-scaled Boltzmann research between methods that encode exogenous information and methods that derive the relevant scale from empirical uncertainty.

5. Beyond bandits: adaptive temperature in reinforcement learning and Boltzmann sampling in Bayesian optimization

In parameterized reinforcement learning, active exploration in structured continuous action space uses a Boltzmann softmax over discrete actions together with Gaussian exploration over continuous parameters (Khamassi et al., 2016). The discrete component is selected according to

$\eta_t$4

where $\eta_t$5 is an inverse temperature parameter controlling the exploitation versus exploration trade-off (Khamassi et al., 2016). The paper contrasts fixed or pre-defined annealing schedules with a meta-learning rule that adjusts $\eta_t$6 and the Gaussian width $\eta_t$7 from the difference between short-term and long-term reward running averages:

$\eta_t$8

When performance drops, $\eta_t$9 is reduced to encourage more exploration and $\eta_t$0 is increased (Khamassi et al., 2016).

This formulation is explicitly presented as active and meta-learned, as opposed to prior-scaled. The contrast clarifies a recurrent distinction in the literature: prior-scaled Boltzmann exploration fixes or anneals temperature through human specification or instance-level assumptions, whereas meta-learned Boltzmann exploration makes the temperature responsive to ongoing performance dynamics (Khamassi et al., 2016).

In batch Bayesian optimization, B3O defines a Boltzmann distribution over the acquisition function,

$\eta_t$1

and draws $\eta_t$2 i.i.d. samples to form the batch (Bloor et al., 29 Jun 2026). Low $\eta_t$3 yields sampling that is nearly uniform over the space and therefore exploratory; high $\eta_t$4 concentrates samples near maxima of the acquisition function and is therefore exploitative (Bloor et al., 29 Jun 2026). The paper states that the inverse temperature $\eta_t$5 is the key parameter for prior scaling, that it can be kept fixed or grown over time, and that theoretical regret analysis supports $\eta_t$6 (Bloor et al., 29 Jun 2026). It also states that there is no explicit mention of incorporating prior or posterior rescaling into the exponent as in certain kernelized bandit models.

These two extensions show that Boltzmann scaling serves different roles outside stochastic bandits. In parameterized RL it governs adaptation to non-stationarity through online meta-control, while in B3O it controls the diversity–concentration trade-off in direct sampling from acquisition landscapes. The common mechanism is still temperature, but the operational meaning of the scale is task-dependent.

6. Comparative properties, misconceptions, and conceptual distinctions

A recurring misconception is that Boltzmann exploration is theoretically benign provided that the temperature is annealed in some reasonable way. The bandit results directly contradict this: any monotone learning-rate sequence induces suboptimal behavior, and overly aggressive schedules can produce linear regret (Cesa-Bianchi et al., 2017). Another misconception is that “prior scaling” names a single canonical method. The surveyed papers instead use the term for several non-equivalent constructions: schedules depending on $\eta_t$7 and $\eta_t$8, per-arm uncertainty scales, pseudo-count initialization, and acquisition-temperature annealing (Cesa-Bianchi et al., 2017, Bian et al., 2021, Bloor et al., 29 Jun 2026).

A further distinction concerns what is preserved under aggressive tuning. In classical BE, temperature mis-tuning can cause linear regret if suboptimal arms are never explored; in contrast, MS$\eta_t$9 is reported to allow more aggressive tuning, meaning less exploration, without losing asymptotic optimality (Bian et al., 2021). The significance is not merely empirical convenience. It separates methods in which scaling is a fragile hyperparameter from methods in which scaling is part of the proof architecture.

Method	Scaling mechanism	Stated property
Classical BE	Global $o(\log t / \Delta)$0 or $o(\log t / \Delta)$1	Suboptimal; may be linear if mis-tuned
Non-monotone Boltzmann	Schedule using $o(\log t / \Delta)$2 and $o(\log t / \Delta)$3	Near-optimal, but needs prior knowledge
BGE	Per-arm $o(\log t / \Delta)$4	Prior-free; $o(\log t / \Delta)$5 distribution-independent bound
MS	Exponent scaled by $o(\log t / \Delta)$6	Asymptotically optimal; $o(\log t / \Delta)$7 minimax bound
MS$o(\log t / \Delta)$8	Adds $o(\log t / \Delta)$9 and $\omega(\log T/\Delta)$0	$\omega(\log T/\Delta)$1 minimax bound; aggressive tuning
B3O	Acquisition temperature $\omega(\log T/\Delta)$2	Batch diversity and negligible additional regret

Across these methods, the central design question is how the Boltzmann exponent should encode information absent from the classical softmax rule. The literature surveyed here answers that question in four main ways: with exogenous problem parameters, with empirical uncertainty, with count-weighted sufficient statistics and pseudo-counts, or with adaptive temperature schedules. This suggests that prior-scaled Boltzmann exploration is best understood not as a single algorithmic object, but as a technical program for repairing classical softmax by embedding a more informative scale into the Gibbs distribution.