---
title: Feel-Good Thompson Sampling (FG-TS)
url: https://www.emergentmind.com/topics/feel-good-thompson-sampling-fg-ts
type: topic
---

# Feel-Good Thompson Sampling (FG-TS)

Searching arXiv for recent and foundational papers on Feel-Good Thompson Sampling.
Feel-Good Thompson Sampling (FG-TS) is a family of posterior-sampling algorithms in which the posterior or pseudo-posterior is deliberately tilted toward models that predict high rewards, thereby injecting optimism into Thompson Sampling without adding a separate UCB-style action bonus. In the foundational formulation for contextual bandits and reinforcement learning, the modification is introduced to address a frequentist weakness of standard Thompson Sampling: in pessimistic or weakly identifying environments, the unmodified posterior can under-explore and incur suboptimal regret [2110.00871]. Subsequent work has extended the same idea to approximate posterior sampling, smoothed objectives, contextual dueling bandits, nonparametric BART-based bandits, variance-aware contextual bandits, and reinforcement learning with approximate sampling, while also clarifying where the method is effective and where it is sensitive to posterior approximation and tuning [2507.15290], [2404.06013], [2602.07767], [2511.02123], [2406.12241].

## 1. Origins and motivation

FG-TS arose from a frequentist analysis of contextual bandits in which the standard Thompson Sampling update is based on a prior \(p_0(\theta)\), a model value function \(f(\theta,x,a)\), and a loss \(L(\theta,x,a,r)\), producing the posterior
\[
p(\theta \mid S_{t-1}) \propto \exp\Big(-\sum_{s=1}^{t-1} L(\theta,x_s,a_s,r_s)\Big) p_0(\theta).
\]
Under the canonical squared-loss choice
\[
L(\theta,x,a,r)=\eta\big(f(\theta,x,a)-r\big)^2,
\]
standard Thompson Sampling samples \(\theta_t\) from this posterior and plays the greedy action \(a_t\in\arg\max_a f(\theta_t,x_t,a)\) [2110.00871].

The foundational analysis isolates a failure mode in which standard Thompson Sampling is “not aggressive enough” in exploration. In a two-action, finite-model example with a uniform prior over \(N\) models, the paper proves
\[
\sum_{t=1}^T \mathbb{E}[\Delta_t] \ge 0.5\,T(1-1/N)^T,
\]
so for \(T=N\) the regret is \(\Omega(T)\), even though the minimax rate in the finite-class setting is \(O(\sqrt{T\log N})\) [2110.00871]. The later empirical study frames the same issue more broadly: recent theory shows that standard Thompson Sampling “does not explore aggressively enough in high-dimensional problems,” and FG-TS is designed to correct that behavior by biasing the posterior toward high-reward models [2507.15290].

This motivation persists across later variants. In the BART-based nonparametric setting, the authors state that “standard Thompson sampling can under-explore and is difficult to analyze in nonparametric classes,” which is why a feel-good variant is introduced as a technical companion to Bayesian Forest Thompson Sampling (BFTS) [2602.07767]. In reinforcement learning, the same idea appears as a way to encourage deep exploration by favoring \(Q\)-functions that assign high value to the current initial state [2406.12241].

## 2. Posterior tilting and the core “feel-good” mechanism

The defining modification of FG-TS is the insertion of an optimism term directly into the posterior weight. In the contextual-bandit formulation, the per-sample loss becomes
\[
L(\theta,x,a,r)
=
\eta \big(f(\theta,x,a)-r\big)^2
-
\lambda\,\min\big(b,f(\theta,x)\big),
\]
where \(f(\theta,x)=\max_a f(\theta,x,a)\), \(\eta>0\) is the squared-loss weight, \(\lambda\ge 0\) controls the strength of the feel-good bias, and \(b\ge 1\) truncates extreme predictions [2110.00871]. The resulting posterior favors models whose greedy values on observed contexts are large.

A closely related contextual-bandit presentation writes the FG-TS loss as
\[
L^\text{FG}(\theta,x,r)
=
\eta\bigl(f_\theta(x)-r\bigr)^2
-
\lambda \min\bigl(b,f_\theta(x)\bigr),
\]
with posterior
\[
p(\theta\mid\mathcal D_{t-1})
\propto
\exp\bigl(-\mathcal L_t(\theta)\bigr)\,p_0(\theta),
\]
where \(\mathcal L_t\) is the cumulative FG loss. The decision rule is unchanged from standard Thompson Sampling:
\[
x_t=\arg\max_{x\in\mathcal X_t} f_{\theta_t}(x),
\]
so there is “no extra UCB-style bonus at decision time”; the optimism is injected into the posterior itself [2507.15290].

Two structural variants are prominent. First, “Smoothed Feel-Good Thompson Sampling” (SFG-TS) replaces the nonsmooth \(\min(b,\cdot)\) term by a differentiable surrogate,
\[
L^\text{SFG}(\theta,x,r)
=
\eta\bigl(f_\theta(x)-r\bigr)^2
-
\lambda\bigl(b-\Phi_s(b-f_\theta^\star)\bigr),
\]
where \(f_\theta^\star=\max_{x\in\mathcal X} f_\theta(x)\) and
\[
\Phi_s(u)=\frac{1}{s}\log(1+e^{su}).
\]
This smoothing is introduced specifically to make Langevin- and HMC-type samplers more stable [2507.15290].

Second, the BART-based FG-TS variant used inside BFTS keeps the same arm-wise BART posterior machinery as standard BFTS but adds a feel-good loss
\[
L_{\mathrm{FGTS}}(\theta,x,a,r)
=
\eta\big(f_\theta(x,a)-r\big)^2
-
\lambda\,\min\bigl(b,f_\theta^*(x)\bigr),
\qquad
f_\theta^*(x)=\max_{a'\in\mathcal A} f_\theta(x,a'),
\]
and in practice implements the optimism through a reweighting of posterior draws by \(\exp(\lambda S_j)\), where
\[
S_j=\sum_{s=1}^t \min\{b,f_{\theta^{(j)}}^*(X_s)\}.
\]
Setting \(\lambda=0\) recovers standard BFTS, and the theoretically optimal \(\lambda^\star\) is of order \(\varepsilon_T\to 0\), so the authors describe the feel-good modification as a “vanishing perturbation” of BFTS [2602.07767].

## 3. Foundational theory: regret decomposition and decoupling

The original theory rewrites instantaneous regret as the difference between a model-fit term and an optimism term. With \(a_t=a(\theta_t,x_t)\), the decomposition is
\[
\Delta_t
=
f_*(x_t)-f_*(x_t,a_t)
=
BE_t-FG_t,
\]
where \(BE_t\) is a Bellman-error-like prediction term and \(FG_t\) is the feel-good exploration term [2110.00871]. This identity is the basis for the later literature: the optimism term does not merely alter action probabilities, but appears explicitly with a negative sign in the regret bound.

To control the dependence between sampled model and chosen action, the theory introduces a decoupling coefficient \(dc\). In the finite-action contextual-bandit case, \(dc(x,b,\Omega,B)\le K\) when \(|\mathcal A(x)|\le K\). In the linearly embeddable contextual-bandit model
\[
f(\theta,x,a)=w(\theta,x)^\top \phi(x,a),\qquad
f_*(x,a)=w_*(x)^\top \phi(x,a),
\]
the same coefficient is bounded by the embedding dimension \(K\) [2110.00871]. This converts bandit regret into an online least-squares estimation problem plus a complexity term.

The resulting frequentist guarantees are minimax-rate up to logarithmic factors. For finite-action contextual bandits with a finite function class of size \(N\), FG-TS attains
\[
O\big(\sqrt{KT\ln N}\big)
\]
in the realizable case. For linearly embeddable bandits with parametric complexity \(d\) and embedding dimension \(K\), the bound is
\[
O\big(\sqrt{K d T\ln T}\big),
\]
and for standard linear bandits, where \(K=d\), this becomes
\[
O\big(d\sqrt{T\ln T}\big),
\]
which the paper identifies as minimax-optimal up to logarithmic factors [2110.00871].

The same paper also separates this frequentist story from the Bayesian one. Standard Thompson Sampling retains a Bayesian-regret guarantee, but the feel-good modification is what restores minimax-style frequentist control in the worst case [2110.00871]. A common misconception is therefore that FG-TS is merely a heuristic exploration bias; in the foundational treatment, the feel-good term is the object that makes the frequentist analysis go through.

## 4. Major variants and domain-specific formulations

Later work preserved the posterior-tilting idea but changed the form of the feel-good term to match the geometry of the problem class.

| Setting | Feel-good construction | Reported guarantee or role |
|---|---|---|
| Contextual bandits | Posterior tilt by \(\lambda\max_a f(x_t,a)\) or \(\lambda\min(b,f(\theta,x))\) | Minimax-rate frequentist regret up to logs in finite-action and linear cases |
| Contextual dueling bandits | Two independent pseudo-posteriors with a dueling-specific max-difference bonus | Nearly minimax-optimal \(\tilde{\mathcal O}(d\sqrt T)\) regret |
| Reinforcement learning | Feel-good prior at the initial state via \(-\lambda\max_a Q(w_1;x_1^k,a)\) | \(\tilde O(dH^{3/2}\sqrt T)\) in linear MDPs with approximate sampling |
| Variance-aware contextual bandits | Variance-weighted losses and \(\lambda_t=c\sqrt{\Lambda_t}/\bar\sigma_t^2\) | \(\tilde O(\sqrt{\mathrm{dc}\log|\mathcal F|\sum_t \sigma_t^2}+\mathrm{dc})\) |
| BART-based nonparametric bandits | Feel-good reweighting of arm-wise BART posterior draws | Minimax-optimal nonparametric regret up to logs for the feel-good variant |

In contextual dueling bandits, FGTS.CDB uses two independent posteriors \(p^1(\cdot\mid S_{t-1})\) and \(p^2(\cdot\mid S_{t-1})\), one for each selected arm. For player \(j\in\{1,2\}\), the loss is
\[
L^j(\theta,x,a^1,a^2,y)
=
\eta\,\sigma\!\big(y\langle \theta,\phi(x,a^1)-\phi(x,a^2)\rangle\big)
-
\mu \max_{a'\in\mathcal A}
\langle \theta,\phi(x,a')-\phi(x,a^{3-j})\rangle.
\]
The independent sampling of \(\theta_t^1\) and \(\theta_t^2\) is central to the analysis because it avoids cross terms in the regret decomposition. The main result is \(\tilde{\mathcal O}(d\sqrt T)\) regret, matching the lower bound up to logarithmic factors [2404.06013].

In reinforcement learning, the feel-good term is moved to a stage-0 loss at the current initial state,
\[
L_0^k(w_1)=-\lambda \max_{a\in\mathcal A} Q(w_1;x_1^k,a),
\]
while the remaining losses are temporal-difference squared losses,
\[
L_h^k(w_h)
=
\eta\sum_{\tau=1}^{k-1}
\Big[
r_h(x_h^\tau,a_h^\tau)
+
\max_{a\in\mathcal A} Q_{h+1}^k(x_{h+1}^\tau,a)
-
Q(w_h;\phi(x_h^\tau,a_h^\tau))
\Big]^2.
\]
The resulting FG-TS posterior over \(Q\)-function parameters is sampled approximately with LMC or ULMC inside least-squares value iteration with approximate sampling exploration. In linear MDPs, the regret bound is
\[
\tilde O(dH^{3/2}\sqrt T),
\]
with an additive term that depends on total-variation sampling error [2406.12241].

The variance-aware extension FGTS-VA modifies both the data-fit weights and the feel-good parameter. With \(\bar\sigma_t=\max\{\sigma_t,\alpha\}\) and \(\Lambda_t=\sum_{s=1}^t \bar\sigma_s^2\), it sets
\[
\eta_t=\bar\sigma_t^{-2},
\qquad
\lambda_t=c\,\frac{\sqrt{\Lambda_t}}{\bar\sigma_t^2},
\]
and uses the posterior
\[
p_t(f\mid S_{t-1})
\propto
p_0(f)\exp\Big(
-\sum_{s=1}^{t-1}\eta_s(r_s-f(x_s,a_s))^2
+
\lambda_t\max_{a\in\mathcal A_t} f(x_t,a)
\Big).
\]
The corresponding regret bound is
\[
\tilde O\Big(\sqrt{\mathrm{dc}\,\log|\mathcal F|\sum_{t=1}^T \sigma_t^2}+\mathrm{dc}\Big),
\]
and for linear contextual bandits it specializes to \(\tilde O(d\sqrt{\Lambda}+d)\), matching the best variance-aware UCB-type rates cited in that work [2511.02123].

In nonparametric contextual bandits with BART priors, the feel-good BFTS variant is analytically distinct from standard BFTS. The main BFTS algorithm receives a Bayesian regret bound of order \(\tilde{\mathcal O}(\sqrt T)\), while the feel-good variant obtains a frequentist nonparametric regret bound
\[
\widetilde{\mathcal O}\big(T\varepsilon_T\big),
\]
which yields
\[
\widetilde{\mathcal O}\big(K\,T^{(\alpha+d)/(2\alpha+d)}\operatorname{polylog}(T,p)\big)
\]
under covariate sparsity and Hölder smoothness. The paper states that this is minimax-optimal up to logarithmic factors and presents it as evidence for the “structural suitability of BART priors for non-parametric bandits” [2602.07767].

## 5. Approximate sampling, smoothing, and empirical behavior

A substantial later literature treats FG-TS as a sampling problem rather than only a regret-analysis device. The systematic MCMC study benchmarks FG-TS and SFG-TS across eleven real-world and synthetic problems under exact and approximate posterior sampling. The main empirical pattern is conditional: when posterior samples are accurate, such as in linear and logistic bandits with well-tuned MALA or LMC, small feel-good bonuses improve exploration and regret; when sampling noise dominates, especially in neural bandits, the same optimism can amplify approximation error and hurt performance [2507.15290].

This study also gives a concrete algorithmic view of approximate FG-TS. A generic MCMC-Thompson-sampling template runs an inner chain each round and then acts greedily with the terminal sample. The only change between TS, FG-TS, and SFG-TS is the loss used inside the sampler. The paper evaluates LMC, MALA, HMC, underdamped LMC, preconditioned variants, and SVRG-based variants, and reports that FG-TS “generally outperforms vanilla TS in linear and logistic bandits, but tends to be weaker in neural bandits.” It further recommends FG-TS and its variants as baselines in modern contextual-bandit benchmarks because they are competitive and easy to use [2507.15290].

The same paper emphasizes hyperparameter sensitivity. Ablations over \(\lambda\in\{0,0.01,0.1,0.5,1.0\}\) show that small \(\lambda\) can help, while larger values often cause regret spikes. In the linear setting, SFGMALATS with \(\lambda=0.01\) achieves cumulative regret comparable to or better than LinUCB and LinTS, but values such as \(0.5\) and \(1.0\) often harm performance. In neural contextual bandits, FG-NeuralTS and SFG-NeuralTS often collapse, while vanilla LMC-TS, Neural-\(\epsilon\)-Greedy, and NeuralUCB are more robust [2507.15290].

The reinforcement-learning approximation literature shows a different empirical regime. In N-Chain environments and on several Atari 57 games, approximate-sampling FG-TS variants such as FG-LMCDQN and FG-ULMCDQN perform significantly better than strong baselines in hard-exploration problems, and on several Atari games they are reported as better than or on par with other strong baselines from deep RL [2406.12241]. This suggests that the effect of posterior optimism depends strongly on whether exploration difficulty is primarily long-horizon and state-value based, or primarily a posterior-calibration problem in high-dimensional supervised reward models.

The BFTS paper is more cautious. Its feel-good variant is implemented only in a sensitivity study, not as the main empirical method. The reported findings are that the parameter `feel_good_eta` has large impact, aggressive values can cause normalized regret multipliers of \(4\text{–}8\times\), and FG-TS “does not consistently outperform BFTS.” The authors conclude that they “do not recommend the feel-good variant as a default in practice” [2602.07767].

## 6. Interpretation, misconceptions, and current status

The central conceptual point is that FG-TS is not UCB with randomized scoring. In the contextual-bandit formulation, “there is no extra UCB-style bonus at decision time”; the action rule remains greedy with respect to a sampled model, and the optimism is expressed by reshaping the posterior distribution over models [2507.15290]. The same distinction appears in the foundational paper, where the feel-good term enters the loss, not the action score [2110.00871].

A second point is that minimax-style frequentist guarantees are attached to specific FG-TS constructions, not to Thompson Sampling in general. The original paper proves Bayesian-regret guarantees for standard Thompson Sampling and frequentist guarantees for FG-TS [2110.00871]. The BART paper is explicit that the feel-good result “does not constitute a frequentist regret bound for BFTS itself”; instead, it shows that the same BART prior supports minimax-rate learning once paired with a vanishing feel-good perturbation [2602.07767]. The reinforcement-learning approximation framework likewise separates ideal FG-TS regret from the additional regret induced by sampling error through an additive term involving total-variation distance [2406.12241].

A third point concerns smoothness and implementability. The original \(\min(b,\cdot)\) bonus is nonsmooth, which is why SFG-TS replaces it by a log-sum-exp surrogate for gradient-based MCMC [2507.15290]. In practice, many later FG-TS algorithms are pseudo-posterior methods rather than exact Bayesian procedures: the posterior is a computational and analytical vehicle that blends squared-loss fitting with deliberate optimism. This is explicit in the dueling-bandit construction, which uses tailored pseudo-posteriors, and in the approximate-sampling RL work, which relies on stage-wise MCMC rather than exact joint posterior sampling [2404.06013], [2406.12241].

Across the literature, the present status of FG-TS is therefore mixed but technically clear. In linear contextual bandits and several moderate-dimensional settings, FG-TS and SFG-TS are supported by minimax-style theory and competitive empirical results. In variance-aware and nonparametric settings, the framework has been generalized in a principled way through weighted losses, generalized decoupling coefficients, and BART-based pseudo-posteriors. In neural contextual bandits and some approximate-posterior regimes, however, the optimism term is often reported as fragile. This suggests that the practical utility of FG-TS depends on the fidelity of posterior sampling, the smoothness of the objective, and the extent to which optimism amplifies rather than corrects model miscalibration [2511.02123], [2507.15290].

Source: https://www.emergentmind.com/topics/feel-good-thompson-sampling-fg-ts