Semantic-aware Thompson Sampling

• Semantic-aware Thompson sampling is a Bayesian framework that extends standard methods by incorporating structured, meaning-bearing latent hypotheses.
• It applies sequential Bayesian updating where expert weights and variable inclusion techniques inform reward-based action selection.
• The framework bridges bandit methods with causal and model-based inference by treating actions as interventions and embedding semantic structure into the posterior.

Searching arXiv for the cited papers and for the exact topic phrase to ground the article in current arXiv records. Semantic-aware Thompson sampling is not introduced explicitly in the cited arXiv literature. The closest technical usage is conceptual: Thompson sampling is driven by a probabilistic model over hypotheses or policies, and the same posterior-sampling logic could, in principle, be applied when those hypotheses are defined over semantic or meaning-bearing latent structures. Within that interpretation, the relevant literature consists of generalized Thompson sampling as a Bayesian sequential decision-making procedure, a contextual expert-weighting variant in which experts may encode structured hypotheses, and Thompson Variable Selection, where rewards are tied to posterior evidence from a black-box learner rather than to anonymous arm payoffs (Ortega et al., 2013, Zhou, 2015, Liu et al., 2020).

1. Scope of the term

The exact phrase semantic-aware Thompson sampling does not appear as the name of a formal algorithm in the cited works. One source states that it does not discuss “semantic-aware Thompson sampling” explicitly and that it does not introduce semantic embeddings, LLMs, or meaning-aware action selection. Another states that it does not explicitly discuss semantics, meaning, embeddings, side information, or structured textual context. In this literature, the term is therefore best understood as an interpretive umbrella rather than a standardized method name (Ortega et al., 2013, Zhou, 2015).

The strongest explicit connection to a “semantic-aware” reading appears where Thompson-style exploration is coupled to structured hypotheses rather than generic arm identities. In one case, the “structured” or “semantic-aware” interpretation is strongest in the way Thompson Variable Selection defines its reward and oracle: the selection process is structured and meaningful because the reward is tied to posterior evidence from a black-box learner such as BART, and the chosen subset is the one that maximizes a Bayesian-style objective over variables. In another case, generalized Thompson sampling over experts is described as especially relevant because experts can encode semantic models, contextual predictors, neural predictors, or other structured hypotheses (Liu et al., 2020, Zhou, 2015).

A plausible implication is that, in this body of work, “semantic-aware” does not denote a separate sampling rule. It denotes where the latent hypotheses, expert classes, or variable-selection rewards are already endowed with task-specific structure.

2. Bayesian posterior sampling over policies

In generalized Thompson sampling, the core object is policy uncertainty. The agent does not know the true environment, maintains a posterior over latent environment parameters $\theta \in \Theta$, and associates to each $\theta$ a corresponding optimal policy $B(a_t \mid \theta, a_{<t}, o_{<t})$. At each time step, the agent samples a $\theta$ from the posterior and then samples or chooses the action prescribed by that $\theta$-specific policy. Thompson sampling is therefore treated as sampling actions from the predictive distribution over optimal actions (Ortega et al., 2013).

The interaction model is formulated as an I/O system with action set $A$ and observation set $O$. For a length-$T$ interaction, the joint probability under agent $P$ and environment $Q$ factorizes as

$\theta$0

This makes explicit that the agent generates actions and the environment generates observations conditioned on the full action history. When the environment is unknown, the agent assumes a latent parameter $\theta$1 indexing possible environments $\theta$2, with prior $\theta$3, and forms the Bayesian mixture

$\theta$4

The adaptive-agent definition is then

$\theta$5

The action rule itself is a Bayesian superposition of candidate optimal policies: $\theta$6 The predictive distribution over observations is defined analogously by

$\theta$7

The paper emphasizes that, in a Bayesian view, uncertainty about the policy should be represented by a distribution rather than a point estimate. This is the principal sense in which generalized Thompson sampling is presented as a principled Bayesian method rather than merely a bandit heuristic (Ortega et al., 2013).

3. Causal calculus and the semantics of intervention

A central technical feature of generalized Thompson sampling is that actions and observations enter posterior updating differently. Observations provide evidence about $\theta$8 and update beliefs by standard Bayesian conditioning. Past actions do not provide evidence about the environment in the same way, because they are chosen by the agent. For that reason, the posterior is written as

$\theta$9

not $B(a_t \mid \theta, a_{<t}, o_{<t})$0, where Pearl-style notation $B(a_t \mid \theta, a_{<t}, o_{<t})$1 marks an intervention (Ortega et al., 2013).

The intervened posterior is

$B(a_t \mid \theta, a_{<t}, o_{<t})$2

The crucial point is that the action terms disappear from the posterior update, because intervened actions do not provide evidence about $B(a_t \mid \theta, a_{<t}, o_{<t})$3. The paper also notes that

$B(a_t \mid \theta, a_{<t}, o_{<t})$4

so once $B(a_t \mid \theta, a_{<t}, o_{<t})$5 is fixed, the action model is unaffected by whether past actions are treated as interventions or ordinary variables (Ortega et al., 2013).

This same framework is used for causal induction. Different hypotheses $B(a_t \mid \theta, a_{<t}, o_{<t})$6 may correspond to different causal structures, and sequential interaction creates asymmetries that observational data alone may not reveal. The paper’s illustrative example considers two correlated light bulbs $B(a_t \mid \theta, a_{<t}, o_{<t})$7 and $B(a_t \mid \theta, a_{<t}, o_{<t})$8 with competing hypotheses: $B(a_t \mid \theta, a_{<t}, o_{<t})$9 is “green causes red,” and $\theta$0 is “red causes green.” Observing $\theta$1 and $\theta$2 jointly may not distinguish these hypotheses, but intervening on one variable does. The sequential procedure is: sample a causal hypothesis, intervene according to the policy associated with that hypothesis, observe the outcome, update the posterior, and repeat. This is the clearest sense in which generalized Thompson sampling acquires a semantics of action: actions are interventions, not merely observations (Ortega et al., 2013).

4. From adaptive opponents to expert-weighted contextual models

Generalized Thompson sampling also extends to repeated interaction between multiple adaptive agents. Each agent treats the other as part of its environment and maintains Bayesian beliefs over the other’s behavior. In a one-shot simultaneous-move game, best responses are written as

$\theta$3

$\theta$4

A Nash equilibrium $\theta$5 is a fixed point of these best-response maps. When both players use generalized Thompson sampling, the paper argues that, under suitable assumptions, the adaptive dynamics can converge to a Nash equilibrium or “lock in” to a stable pair of predictive models. The matching pennies game is given as a concrete example in which repeated sampling and adaptation can converge toward the mixed-strategy Nash equilibrium (Ortega et al., 2013).

A distinct but related generalization appears in the contextual bandit and expert-learning setting. Generalized Thompson Sampling (GTS) is described as a contextual bandit or expert-learning variant that “includes Thompson Sampling as a special case.” At round $\theta$6, after observing context $\theta$7, the learner assigns arm probabilities by

$\theta$8

where $\theta$9 is a set of experts and expert $\theta$0 recommends

$\theta$1

The weights update according to

$\theta$2

With logarithmic loss,

$\theta$3

the weight dynamics behave like Bayesian posteriors over experts (Zhou, 2015).

This is the most direct bridge from standard posterior sampling to a semantic or model-based reading. The note states that experts can encode semantic models, contextual predictors, neural predictors, or other structured hypotheses. A plausible implication is that “semantic awareness” enters not through a new sampling equation, but through the representational content of the experts whose weights evolve as posterior-like quantities (Zhou, 2015).

5. Thompson Variable Selection as structure-aware posterior sampling

Thompson Variable Selection (TVS) reinterprets variable selection as a combinatorial bandit problem and solves it with a Thompson-sampling-style policy. The action at time $\theta$4 is a subset $\theta$5 rather than a single arm, and the reward is produced by a variable-screening rule based on a learning algorithm applied to the selected subset. The paper explicitly frames this as a bridge between spike-and-slab Bayesian variable selection and combinatorial bandits (Liu et al., 2020).

The Bayesian formulation gives each variable its own Bernoulli inclusion probability: $\theta$6 At each round,

$\theta$7

and the posterior parameters are updated by observed binary rewards: if $\theta$8, then $\theta$9; otherwise $A$0. The running proxy for posterior inclusion probability is

$A$1

The final selected model is obtained by thresholding these $A$2 values (Liu et al., 2020).

The key optimization criterion uses the global reward

$A$3

with expectation

$A$4

The oracle subset is

$A$5

For the special choice

$A$6

the threshold becomes $A$7, and the oracle is exactly the median probability model: $A$8 The paper identifies this as a key semantic link: subset selection becomes “posterior probability exceeds threshold,” aligning Thompson sampling with Bayesian inclusion logic rather than purely deterministic optimization (Liu et al., 2020).

The offline and online regimes make the structured character of the reward explicit. In the offline setting, the reward is produced by running BART on only the variables in $A$9, with local reward

$O$0

In the online or streaming setting, data are divided into minibatches $O$1, and the reward becomes

$O$2

where $O$3 is an importance measure such as the average number of times variable $O$4 is used in the fitted forest. The posterior update is

$O$5

The framework is not limited to BART; LASSO, random forests, or any method producing variable importance can be wrapped into the reward mechanism (Liu et al., 2020).

6. Regret, consistency, and interpretive boundaries

The cited papers provide several theoretical guarantees, but they concern different objects. In the standard Thompson sampling exposition, the probability of choosing arm $O$6 is

$O$7

and the Beta-Bernoulli implementation samples

$O$8

The same note contrasts Thompson sampling with Information-Directed Sampling, which instead minimizes

$O$9

This distinction matters because semantic or structured priors can be attached to posterior sampling without changing Thompson sampling into IDS (Zhou, 2015).

For GTS, the note states a regret bound under assumptions on immediate regret and self-bounded loss: $T$0 It further states that with square loss the regret is

$T$1

and with logarithmic loss,

$T$2

These are guarantees for an expert-mixture contextual procedure, not for a semantics-specific variant as such (Zhou, 2015).

For TVS, regret is defined by

$T$3

When the optimal model size $T$4 is known and the arms are independent, the paper gives

$T$5

When $T$6 is unknown and rewards are independent, it gives

$T$7

For correlated or related arms, under the stated strong identifiability condition,

$T$8

The same paper also proves the almost-sure consistency statement

$T$9

for fixed $P$0 and $P$1 (Liu et al., 2020).

The main interpretive boundary is explicit in the sources themselves. The papers do not present semantic embeddings, LLMs, or meaning-aware action selection as part of Thompson sampling. The connection is conceptual rather than explicit: posterior sampling over latent structure, posterior-like weighting over experts, and structured rewards tied to model-specific evidence provide the available routes by which Thompson sampling becomes “semantic-aware” in a limited and technical sense (Ortega et al., 2013, Zhou, 2015, Liu et al., 2020).