Delightful Exploration

Abstract: Most exploration algorithms search broadly until uncertainty is resolved. When the action space is too large to resolve within budget, practitioners default to $\varepsilon$-greedy, which bounds disruption but spends its override blindly. We introduce \textit{Delight-gated exploration} (DE), a host--override rule that spends exploratory actions only when their prospective delight (expected improvement times surprisal) exceeds a gate price. This practical heuristic recovers a classical result: Pandora's reservation-value rule for costly search, with surprisal setting the effective inspection cost. Resolved arms exit the gate, fresh arms shut off above a prior-determined threshold, and selected linear-bandit overrides consume finite information budget. Across Bernoulli bandits, linear bandits, and tabular MDPs, the same hyperparameters transfer without retuning, and DE shows much weaker regret growth than Thompson Sampling and $\varepsilon$-greedy in the tested unresolved regimes. Delight improves acting for the same reason it improves learning: it prices scarce resources by the product of upside and surprisal.

• The paper introduces Delight-gated Exploration (DE), which selectively overrides greedy policies by targeting actions with high prospective delight.
• DE combines expected improvement with capped surprisal to efficiently manage exploration, resulting in nearly constant regret in large action spaces.
• Empirical results across bandit and RL environments show that DE generalizes without retuning, outperforming traditional methods like ε-greedy and Thompson Sampling.

Delight-gated Exploration: A Formal Summary

Motivation and Problem Setting

Most canonical exploration algorithms search exhaustively until uncertainty is resolved, operating under the assumption that agent-environment interaction budgets are sufficient to amortize broad search costs. However, practical systems typically face unresolved regimes—action spaces or horizons too large to resolve within available interactions—where $\varepsilon$-greedy is widely adopted for its simplicity and bounded disruption. The principal shortcoming of $\varepsilon$-greedy is its blind allocation of exploration, frequently expending effort on actions with negligible posterior upside. The paper introduces Delight-gated Exploration (DE), a host–override framework that selectively explores actions with prospective delight—expected improvement multiplied by surprisal—that exceed a preset gate price.

Algorithmic Formulation

DE augments a greedy host (typically a Boltzmann or deterministic policy) with a sparse override mechanism. With probability $1-\varepsilon_t$, the agent follows the host; with probability $\varepsilon_t$, it samples from a gated override distribution. Candidate actions are scored by their prospective delight, $\tilde{d}_t(a) = EI_t(a) \cdot s_t(a)$, where $EI_t(a)$ is expected improvement relative to the current host baseline, and $s_t(a)$ is capped surprisal under the host policy. Only actions with $\tilde{d}_t(a)\geq\lambda$ are eligible for exploratory overrides; the final acting policy is a mixture, $$\pi_t^{\mathrm{act}} = (1-\varepsilon_t)\pi_t^{\mathrm{host}} + \varepsilon_t q_t^\lambda$$, with $\varepsilon_t$ annealed according to a half-life parameter $\varepsilon$0.

A central conceptual insight is the equivalence of the DE gate with Pandora’s reservation-value eligibility in costly search: actions are explored only while the prospective delight surpasses the cost threshold, aligning DE with optimal revealed-value search policies.

Empirical Results

DE's empirical performance is evaluated in three paradigms with substantial unresolved regimes: Bernoulli bandits (independent arms, exact posteriors), linear bandits (feature-based generalization), and tabular MDPs (DeepSea, sequential structure). Notably, the same hyperparameters ($\varepsilon$1, $\varepsilon$2, $\varepsilon$3) transfer across all domains without retuning, provided rewards/values are unit-scaled.

In Bernoulli bandits, DE exhibits dramatically slower regret growth with increasing number of arms $\varepsilon$4. As shown below, cumulative regret at $\varepsilon$5 remains nearly flat for DE as $\varepsilon$6 grows, while Thompson Sampling and $\varepsilon$7-greedy degrade significantly.

Figure 1: Cumulative regret after $\varepsilon$8 rounds as the number of arms $\varepsilon$9 increases; DE remains nearly flat while TS and $1-\varepsilon_t$0-greedy deteriorate.

Learning curves further reinforce the dichotomy: for small $1-\varepsilon_t$1, DE matches TS; for large $1-\varepsilon_t$2, DE outperforms by shutting off unnecessary exploration.

Figure 2: Learning curves for $1-\varepsilon_t$3; DE matches TS in small environments, but strongly outperforms as the unresolved regime appears.

In linear bandits, DE continues to outperform across a sweep of problem dimensionality and noise settings, again without retuning.

Figure 3: Regret vs.\ dimension $1-\varepsilon_t$4 (with $1-\varepsilon_t$5); DE transfers the Bernoulli configuration and outperforms baselines in the structured case.

Tabular MDPs (DeepSea) highlight DE’s applicability to sequential settings, where the exploration problem scales combinatorially with horizon $1-\varepsilon_t$6. DE matches or outperforms PSRL and dominates $1-\varepsilon_t$7-greedy.

Figure 4: Regret at $1-\varepsilon_t$8 vs problem size $1-\varepsilon_t$9 for DeepSea; DE maintains lower regret than PSRL and $\varepsilon_t$0-greedy.

Learning curves for DeepSea environments with $\varepsilon_t$1 show robust DE performance.

Figure 5: Learning curves for DeepSea with $\varepsilon_t$2; DE tracks and outperforms PSRL, $\varepsilon_t$3-greedy incurs high regret.

Sensitivity analyses demonstrate that DE's gains are robust to hyperparameter choices; regret remains low and the competitive advantage over TS persists across broad ranges of annealing half-life $\varepsilon_t$4.

Figure 6: Sensitivity to annealing half-life $\varepsilon_t$5; DE outperforms TS across a wide range of $\varepsilon_t$6.

Structural and Theoretical Analysis

The paper provides structural analysis explaining DE’s scaling:

• Resolved arms exit the gate: Once an arm is sufficiently resolved and identified as suboptimal, its expected improvement decays and it exits the override-eligible set; the number of exploratory pulls is finite and independent of $\varepsilon_t$7.
• Fresh arms drop out above a reservation threshold: When the host baseline exceeds a computed threshold, untried arms are excluded from the gated set, preventing wasted exploration.
• Override consumption bounded by information budget: In linear bandits, exploratory overrides consume a finite uncertainty budget, scaling with model complexity rather than action-space size.

DE's gating mechanism is formalized as Pandora eligibility and admits finite selected-pull budgets for both tried and fresh arms. Regret in infinite-arm regimes achieves the optimal prior-tail rate when price and override frequency are calibrated according to search horizon and tail exponent.

Practical Implications and Contradictory Claims

Empirically, DE demonstrates strong numerical results: in unresolved regimes, DE’s regret is nearly constant as action-space size increases, in direct contradiction to the rapid growth in Thompson Sampling and $\varepsilon_t$8-greedy. The same configuration generalizes across environments, contrary to the necessity for instance-specific tuning in other exploration schemes.

Theoretically, DE contradicts optimistic algorithms by showing that uncertainty alone is not sufficient to warrant exploration in unresolved regimes; instead, upside must be priced. DE dominates $\varepsilon_t$9-greedy by eliminating wasteful late exploration and targeting override only to actions with sufficient expected improvement and surprise.

Implications, Limitations, and Future Directions

Practically, DE delivers scalable exploration for large-scale bandit and RL systems, facilitating transfer of configuration across disparate domains and obviating blind allocation of exploratory budget. Theoretically, DE establishes a principled equivalence to optimal costly search via Pandora’s rule, extending the delight principle from learning to acting.

However, a full regret bound for the greedy host–override mixture remains open, particularly in noisy bandit settings where the gate can reopen after temporary host failures. The fallback to expected improvement as a proxy for value-of-perfect-information is exact only in the revealed-value model and admits subtle deviation in correlated or structured posteriors.

Future developments may include adaptive pricing strategies based on horizon, prior tails, or information gain; extension to continuous or combinatorial action spaces using candidate aggregation or posterior retrieval; and formalizing regret bounds for stochastic hosts.

Conclusion

Delight-gated Exploration replaces blind $\tilde{d}_t(a) = EI_t(a) \cdot s_t(a)$0-greedy override with a priced gate on prospective delight, implementing satisficing search and achieving empirically superior regret scaling across bandit and RL domains. The algorithm’s principled connection to optimal costly search, its robustness to hyperparameter tuning, and its transferability across tasks represent substantial progress in practical, theoretically-grounded exploration. Price calibration and override frequency are identified as primary levers, with uncertainty only warranting exploration when paired with sufficient upside. Formal guarantees for stochastic host–override mixtures remain an avenue for future work.