Multi-Play Combinatorial Semi-Bandit

• MP-CSB is a class of online learning problems where a learner selects multiple arms under combinatorial constraints and receives individual (semi-bandit) feedback.
• The framework supports both binary and integer-action settings, enabling applications such as optimal transport, network routing, and energy management.
• Algorithmic paradigms including UCB variants, Thompson Sampling, and adversarial methods provide rigorous performance guarantees like logarithmic and polylogarithmic regret bounds.

Multi-Play Combinatorial Semi-Bandit (MP-CSB) denotes a class of online learning problems in which a learner selects multiple base arms subject to combinatorial constraints and receives semi-bandit feedback, meaning that the individual outcomes of the selected components are observed. In the standard stochastic formulation, a round consists of choosing a super arm $S_t \subseteq [m]$ or a decision vector $\mathbf{x}_t \in \{0,1\}^K$, observing the rewards of the chosen or triggered arms, and optimizing either the sum of rewards or a more general reward function. A more recent formulation extends the action space to $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$, so that a single arm can be sampled multiple times in one round and yields multiple feedbacks; this extension was proposed to overcome the restriction of classical combinatorial semi-bandits to binary decision spaces, which excludes important cases involving non-negative integer flows or allocations such as optimal transport and knapsack (Wang et al., 2018, Nourani-Koliji et al., 2023, Nakamura et al., 12 Sep 2025).

1. Formal scope and terminology

In the classical combinatorial semi-bandit model, there is a ground set of base arms, and at each round the learner selects a feasible subset subject to combinatorial constraints such as cardinality, matching, matroid, path, or spanning-tree structure. The chosen object is often called a super arm. Semi-bandit feedback reveals the individual stochastic rewards of the selected base arms, and the per-round reward is frequently the sum of those rewards [(Wen et al., 2014); (Combes et al., 2015)].

A common binary-action formulation writes the action as $\mathbf{x}_t \in \{0,1\}^K$ with $\|\mathbf{x}_t\|_0 \le m$, so the learner selects at most $m$ arms per round. This formulation appears in work on piecewise-stationary combinatorial semi-bandits with causally related rewards, where the feasible action set is

$$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$

and in adversarial multiple-play bandits, where the learner selects a subset of $m$ arms out of $K$ total under semi-bandit feedback (Nourani-Koliji et al., 2023, Vural et al., 2019).

A distinct, newer usage reserves MP-CSB for the integer-action setting in which the action is $$\boldsymbol{a}(t) \in \mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$$. In that model, coordinate $\mathbf{x}_t \in \{0,1\}^K$0 specifies how many times arm $\mathbf{x}_t \in \{0,1\}^K$1 is played in round $\mathbf{x}_t \in \{0,1\}^K$2, and the learner receives $\mathbf{x}_t \in \{0,1\}^K$3 independent feedbacks from that arm. This formulation subsumes the binary case as a special case when $\mathbf{x}_t \in \{0,1\}^K$4 (Nakamura et al., 12 Sep 2025).

A recurrent source of ambiguity is therefore terminological rather than mathematical: some papers use “multi-play combinatorial semi-bandit” for the standard subset-selection model, whereas later work introduces MP-CSB precisely to distinguish integer allocations from binary selections. This suggests that the term is best interpreted through the action space and feedback model adopted by a given paper, rather than through nomenclature alone (Vural et al., 2019, Nakamura et al., 12 Sep 2025).

2. Objectives and performance criteria

The canonical objective is cumulative expected reward maximization. In the linear-reward setting of large-scale combinatorial semi-bandits with linear generalization, the expected reward of item $\mathbf{x}_t \in \{0,1\}^K$5 is

$\mathbf{x}_t \in \{0,1\}^K$6

the reward of an action $\mathbf{x}_t \in \{0,1\}^K$7 is $\mathbf{x}_t \in \{0,1\}^K$8, and cumulative regret is

$\mathbf{x}_t \in \{0,1\}^K$9

Related stochastic formulations define regret against the best feasible subset under the true mean vector $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$0 or $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$1 [(Wen et al., 2014); (Combes et al., 2015)].

The objective need not be expectation-based. In risk-aware combinatorial semi-bandits, the goal is to maximize the Conditional Value-at-Risk (CVaR) of a super arm at level $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$2,

$\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$3

with regret measured against the super arm of maximum CVaR: $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$4 The paper introducing this objective studies both Gaussian and bounded arm rewards (Ayyagari et al., 2021).

Other work changes the temporal or inferential target. In pure exploration, the goal is not cumulative reward but fixed-confidence identification of the best combinatorial answer, with sample complexity characterized through a sequential zero-sum game and a lower bound of the form

$\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$5

In non-stationary environments, regret is benchmarked against the time-varying optimal action, either under switching budgets or under variation budgets. In adversarial multiple-play bandits, the comparator may be the best fixed $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$6-arm strategy or the best switching $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$7-arm strategy with $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$8 segments (Jourdan et al., 2021, Chen et al., 2020, Vural et al., 2019).

These variants preserve the same core feedback pattern—multiple observations at the arm level—but change what is being optimized: expected value, tail risk, identification accuracy, or robustness to distributional drift and adversarial sequences.

3. Algorithmic paradigms

A dominant design principle is oracle-based optimism. CombLinUCB computes an upper-confidence value for each item from a posterior covariance and then calls an offline combinatorial optimization oracle; ESCB computes combinatorial confidence indexes that generalize KL-UCB and Hoeffding-style confidence bounds to super arms; CUCB-style methods and their descendants repeatedly solve an offline maximization problem using confidence-adjusted arm statistics [(Wen et al., 2014); (Combes et al., 2015)].

Risk-aware algorithms preserve the same structure but replace expected reward by a confidence bound on CVaR. For Gaussian rewards, $\mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$9-CUCB-G maintains per-arm estimates of means and variances, computes a UCB for the mean and an LCB for the variance, and selects the super arm with the best confidence bound on CVaR. For bounded rewards, $\mathbf{x}_t \in \{0,1\}^K$0-SDCB maintains empirical CDFs and lower confidence CDFs, forms a stochastically dominant super-arm CDF by convolution, and then computes its CVaR. A discretized version, D-$\mathbf{x}_t \in \{0,1\}^K$1-SDCB, is introduced to control the computational burden of convolution (Ayyagari et al., 2021).

A second major family is posterior sampling. CombLinTS samples a parameter vector $\mathbf{x}_t \in \{0,1\}^K$2, converts it into item weights, and optimizes over the combinatorial family. Combinatorial Thompson Sampling (CTS) for general CMAB samples one posterior mean per arm, uses an exact oracle on the sampled parameter vector, and updates posteriors with semi-bandit feedback. In the integer-action MP-CSB model, GenCTS updates each arm with all observed feedbacks in a round and remains computationally feasible when the underlying combinatorial optimization oracle is efficient [(Wen et al., 2014); (Wang et al., 2018); (Nakamura et al., 12 Sep 2025)].

Adversarial and best-of-both-worlds methods rely on expert-mixing or optimistic FTRL machinery. Exp4.MP is an expert-mixing algorithm for adversarial multiple-play bandits, while Exp3.MSP efficiently tracks the best switching $\mathbf{x}_t \in \{0,1\}^K$3-arm strategy. In the integer-action model, GenLBINFV is an optimistic Follow-the-Regularized-Leader method with hybrid regularization that yields logarithmic stochastic regret and worst-case $\mathbf{x}_t \in \{0,1\}^K$4 adversarial regret (Vural et al., 2019, Nakamura et al., 12 Sep 2025).

Specialized settings introduce additional mechanisms. PS-SEM-UCB-Gr combines UCB, GLR-based change-point detection, group restart, and online causal graph learning in piecewise-stationary environments with causally related rewards. Oracle-efficient rare-query frameworks, AROQ and SROQ, reduce oracle calls from linear in $\mathbf{x}_t \in \{0,1\}^K$5 to doubly logarithmic in $\mathbf{x}_t \in \{0,1\}^K$6 while retaining $\mathbf{x}_t \in \{0,1\}^K$7 regret guarantees (Nourani-Koliji et al., 2023, Kim et al., 24 Oct 2025).

4. Regret theory and lower bounds

Theoretical analysis in MP-CSB is organized around problem-dependent lower bounds, instance-dependent logarithmic regret, and gap-free worst-case guarantees. In stochastic combinatorial semi-bandits with linear rewards, a Lai-Robbins-type lower bound takes the form

$\mathbf{x}_t \in \{0,1\}^K$8

where $\mathbf{x}_t \in \{0,1\}^K$9 is defined by a semi-infinite linear program over suboptimal combinatorial actions. ESCB attains

$\|\mathbf{x}_t\|_0 \le m$0

improving earlier generic bounds by a factor of $\|\mathbf{x}_t\|_0 \le m$1, while the lower bound scales as $\|\mathbf{x}_t\|_0 \le m$2 in many cases (Combes et al., 2015).

Linear generalization changes the scaling dimension. CombLinTS and CombLinUCB obtain regret bounds that are independent of the size of the ground set $\|\mathbf{x}_t\|_0 \le m$3 up to logarithmic terms and sublinear in time. In particular, the Bayesian regret of CombLinTS is

$\|\mathbf{x}_t\|_0 \le m$4

and CombLinUCB admits a similarly $\|\mathbf{x}_t\|_0 \le m$5-independent bound under bounded rewards and an exact offline oracle (Wen et al., 2014).

For Thompson sampling in general CMAB with independent arms, the leading term of the regret bound is

$\|\mathbf{x}_t\|_0 \le m$6

which improves a prior $\|\mathbf{x}_t\|_0 \le m$7 dependence. In the matroid bandit special case, the independence assumption can be removed and the regret bound matches the lower bound. The same line of work also proves a negative result: one cannot directly replace the exact offline oracle in Thompson sampling with an approximation oracle, even for classical MAB, without risking linear regret (Wang et al., 2018).

Risk-aware MP-CSB retains logarithmic or polylogarithmic time dependence. For $\|\mathbf{x}_t\|_0 \le m$8-CUCB-G, the Gaussian-case regret is polylogarithmic in $\|\mathbf{x}_t\|_0 \le m$9 and polynomial in problem parameters such as $m$0, $m$1, $m$2, and $m$3. For $m$4-SDCB, the bounded-case regret is logarithmic in $m$5 and polynomial in $m$6 and $m$7 (Ayyagari et al., 2021).

Adversarial multiple-play bandits admit minimax-optimal high-probability bounds. For the best fixed $m$8-arm benchmark,

$m$9

with probability at least $$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$0, and for the best switching $$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$1-arm benchmark with $$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$2 segments,

$$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$3

These results close an $$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$4 high-probability gap relative to earlier efficient algorithms (Vural et al., 2019).

The newer integer-action MP-CSB formulation yields logarithmic stochastic regret without resorting to the “duplicating trick.” GenCTS satisfies

$$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$5

while GenLBINFV satisfies

$$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$6

in the stochastic regime and

$$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$7

in the adversarial regime, where $$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$8 adapts to quantities such as $$\mathcal{X} = \left\{ \mathbf{x} \in \{0,1\}^K : \|\mathbf{x}\|_0 \leq m \right\},$$9, $m$0, and $m$1 (Nakamura et al., 12 Sep 2025).

5. Structural extensions and specialized regimes

Several strands of work modify the structural assumptions rather than the basic feedback pattern. Linear generalization assumes that mean rewards lie in the span of known features, allowing learning complexity to scale with the feature dimension $m$2 rather than the number of items $m$3. Contextual formulations make the feature vector time dependent, and network bottleneck identification adapts LinUCB, BayesUCB, Thompson Sampling, and NeuralUCB to super-arm selection with path costs defined by the maximum loss along the chosen path [(Wen et al., 2014); (Hoseini et al., 2022)].

Dependency across arms can be exploited when covariance information is available. In combinatorial semi-bandits with known covariance, the noise is subgaussian with covariance matrix $m$4, and the learner knows a positive semidefinite matrix $m$5 satisfying $m$6. OLS-UCB adapts its regret to the dependency structure, interpolating between independent and fully correlated regimes through a correlation parameter $m$7, and matches a new lower bound up to polylogarithmic factors in the number of pulled arms (Degenne et al., 2016).

Triggered-feedback and non-stationary models broaden the observation process. With probabilistically triggered arms, regret scales as

$m$8

where $m$9 is the minimum non-zero triggering probability of arm $K$0. In piecewise-stationary CSB, sliding-window CUCB achieves $K$1 distribution-dependent regret in the switching case and $K$2 in the dynamic case, while parameter-free methods trade optimality for adaptivity (Hüyük et al., 2018, Chen et al., 2020).

Privacy, incentives, and pure exploration introduce further constraints. Under $K$3-bounded smoothness, $K$4-LDP and $K$5-DP CSB admit regret rates

$K$6

with no additional dependence on feedback size $K$7. For incentivized exploration, Thompson Sampling becomes Bayesian incentive-compatible after a sufficient warm start in which each atom has enough initial samples. For best-arm identification with semi-bandit feedback, the CombGame meta-algorithm yields asymptotically optimal sample complexity with an oracle-efficient instantiation based on projection-free online learning (Chen et al., 2020, Hu et al., 2022, Jourdan et al., 2021).

Budgeted multiple-play bandits provide another special case. Here each arm has a known cost, there is a per-round budget, and the oracle problem becomes a fractional knapsack. KL-UCB and Thompson Sampling variants are asymptotically optimal in both rate and leading constants, including the thick margin setting where multiple arms fall on the decision boundary (Luedtke et al., 2016).

6. Applications and interpretive themes

MP-CSB is used as a modeling layer for large combinatorial decision problems in which per-component feedback is observable and exploitable. In long-distance navigation for battery electric vehicles, the action is a feasible path in a pre-processed road network, the observed edge-level feedback includes queue time and charging time, and combinatorial Thompson Sampling, BayesUCB, and $K$8-greedy are coupled with a shortest-path oracle. Simulation studies on Norway, Sweden, and Finland show that Thompson Sampling achieves the lowest cumulative regret in all but one scenario (Åkerblom et al., 2023).

In network bottleneck identification, each base arm is an edge, each super arm is a feasible path, the path cost is the maximum loss among chosen arms, and contextual information is available for every arm at every round. The framework adapts $K$9-greedy, LinUCB, BayesUCB, Thompson Sampling, and NeuralUCB, and reports that NeuralUCB achieves the fastest convergence and lowest cumulative regret on a real-world road network in Luxembourg (Hoseini et al., 2022).

In epidemiological monitoring, PS-SEM-UCB-Gr is applied to daily new infection records for Italian regions. The learner groups regions geographically, tracks change-points in reward distributions and in the causal graph, and uses directed total variation regularization for graph learning. The paper reports that the method tracks changes in both infection distributions and the interaction network, and that group restart enables timely adaptation to regime shifts (Nourani-Koliji et al., 2023).

Energy management motivates a related combinatorial multi-bandit problem in which $$\boldsymbol{a}(t) \in \mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$$0 actors, each with $$\boldsymbol{a}(t) \in \mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$$1 actions, are coordinated over $$\boldsymbol{a}(t) \in \mathcal{A} \subseteq \mathbb{Z}_{\ge 0}^d$$2 episodes. The paper emphasizes that individual observability enables parallelized exploration and that mathematical programming can exploit a known combinatorial objective without exponential exploration over the full joint action space (Jacobs et al., 2020).

Across these applications, a persistent interpretive theme is that semi-bandit feedback enables knowledge transfer across overlapping super arms. This is the chief statistical advantage of MP-CSB over approaches that treat every super arm as an atomic arm. The main practical limitation is computational: many algorithms assume an exact offline oracle, some bounded-reward risk-aware methods require expensive convolutions, and older reductions from integer allocations to binary CSB suffer from a severe blow-up through duplicated arms [(Ayyagari et al., 2021); (Wen et al., 2014); (Nakamura et al., 12 Sep 2025); (Kim et al., 24 Oct 2025)].