Hybrid CUCB for CMAB-T

• The paper introduces Hybrid CUCB, which integrates offline batch data with online UCB exploration to minimize cumulative regret in combinatorial multi-armed bandit settings.
• It employs explicit bias correction by combining offline estimates with adaptive online updates, ensuring robust performance even under distribution shifts.
• Empirical results on synthetic and real-world benchmarks demonstrate that Hybrid CUCB achieves accelerated learning and improved data efficiency compared to traditional CUCB methods.

A hybrid CUCB algorithm is a principled approach for leveraging both offline (“batch”) and online (“interactive”) data within the framework of combinatorial multi-armed bandits with probabilistically triggered arms (CMAB-T). It integrates observations from an offline dataset, bias quantification, and adaptive online exploration to accelerate learning and reduce cumulative regret, outperforming both purely online and purely offline algorithms when the offline data are informative, while remaining robust to their bias or misalignment (Zhou et al., 26 Dec 2025).

1. Hybrid CMAB-T Problem Structure

The hybrid setting is formalized as H-CMAB-T. There are $m$ base arms $i=1,\dots,m$; at each round $t$, the agent selects a combinatorial super arm $S_t\in\mathcal{S}$. Upon selection, an environment sample $X(t)\sim D^{on}$ is drawn, and a triggered set $\tau_t\subset[m]$ is realized stochastically via $D^{trig}(S_t,X(t))$, with the identity property $\mathbb{E}[X_i|i\in\tau_t] = \mu^{on}_i$. The reward is $R(S_t,X(t),\tau_t)\geq 0$, and the agent’s objective is to maximize cumulative expected reward.

Before online interaction, the learner has access to an offline dataset $\mathcal{B}$: for each arm $i$, $N_i$ samples $Y_{i,1},...,Y_{i,N_i}$ drawn from a (possibly biased) offline environment with mean $\mu^{off}_i$. The discrepancy between offline and online means is bounded: $|\mu^{off}_i-\mu^{on}_i|\leq V_i$, with $\omega_i=V_i + (\mu^{off}_i-\mu^{on}_i)\in[0,2V_i]$.

An $(\alpha,\beta)$-approximation oracle $\mathcal{O}(\hat{\mu})$ outputs a super arm $S$ such that with probability at least $\beta$, $$r_S(\hat{\mu})\geq\alpha\cdot\mathrm{opt}_{\hat{\mu}}$$, where $\mathrm{opt}_{\hat{\mu}}$ is the optimal expected reward given means $\hat{\mu}$.

The performance metric is $(\alpha,\beta)$-approximate regret: $$\mathrm{Reg}(T) = \alpha\beta\,T\,\mathrm{opt}_{\mu^{on}} - \mathbb{E}\left[\sum_{t=1}^T R(S_t,\dots)\right]$$ (Zhou et al., 26 Dec 2025).

2. Algorithmic Procedure and Update Rules

The Hybrid CUCB algorithm combines the exploitation of high-confidence offline arm means (with explicit bias correction) and adaptive online UCB-driven exploration. The following table concisely summarizes the update equations and indices:

Quantity	Formula	Description
$\hat{y}_i^{off}$	$\frac{1}{N_i}\sum_{s=1}^{N_i}Y_{i,s}$	Offline mean estimate for arm $i$
$\hat{y}_i^{on}$	$\frac{1}{T_i}\sum_{s=1}^{T_i}X_{i,s}$ (updated only for $i\in\tau_t$)	Online mean estimate for arm $i$
$rad_t(i)$	$\sqrt{2\ln(4mt^3) / \max\{T_i,1\}}$	Pure-online UCB radius
$rad_t^S(i)$	$$\sqrt{2\ln(4mt^3)/\max\{N_i+T_i,1\}} + \frac{N_i}{N_i+T_i}V_i$$	Hybrid (offline+online) UCB radius (bias-aware)
$UCB_t(i)$	$\hat{y}_i^{on} + rad_t(i)$	Pure-online UCB
$UCB_t^S(i)$	$$\frac{N_i\hat{y}_i^{off} + T_i\hat{y}_i^{on}}{N_i+T_i} + rad_t^S(i)$$	Hybrid UCB (offline+online, bias-corrected)
$\bar{\mu}_i$	$\min\{UCB_t(i),\, UCB_t^S(i),\, 1\}$	Combined index for arm selection

The agent chooses $S_t = \mathcal{O}(\bar{\mu}_1, ..., \bar{\mu}_m)$, i.e., inputs the coordinate-wise minimum of the pure-online and hybrid UCB bounds into the $(\alpha,\beta)$-approximation oracle.

The selection mechanism ensures that if the offline arm estimates are reliable (large $N_i$, small $V_i$), the hybrid bound dominates and accelerates learning. In cases of small or high-bias offline data, the algorithm defaults to the conservative, safe pure-online bound (Zhou et al., 26 Dec 2025).

3. Regret Analysis and Theoretical Guarantees

Under typical CMAB-T assumptions—reward monotonicity in each arm mean, 1-norm TPM bounded smoothness $$|r_S(\mu)-r_S(\mu')|\leq B\sum_i p_i^{D,S}|\mu_i-\mu_i'|$$, rewards in $[0,1]$, and known bias bounds $V_i$—gap-dependent and gap-independent regret rates are established.

• Gap-dependent bound:

For arm gap $\Delta^i_{min}>0$, discrepancy $\omega_i$, and $K=\max_{S}| \{i: p_i^{D,S}>0\}|$, define

$$N'_i = N_i\max \left\{1-\frac{2B K \omega_i}{\Delta^i_{min}},\, 0\right\}^2$$

Then:

$$\mathrm{Reg}(T)\leq \sum_{i=1}^m\max\left\{\frac{64\sqrt{2}B^2K\ln(4mT^3)}{\Delta^i_{min}} - 8B\sqrt{2 N'_i\ln(4mT^3)},\,0\right\} + 4Bm + \frac{\pi^2}{6}\Delta_{\max}$$

A salient implication is that the hybrid algorithm interpolates smoothly between the purely online $O(\sum_i\frac{B^2K}{\Delta^i_{min}}\ln T)$ regime (when $N'_i=0$) and accelerated learning when many arms have large, accurate offline samples and small bias.

• Gap-independent bound:

The regret satisfies

$$\mathrm{Reg}(T)\leq \min\{\psi,\,\gamma\} + 4Bm + \frac{\pi^2}{6}\Delta_{\max}$$

with $\psi$ and $\gamma$ quantifying contributions from offline sample size, bias, and an auxiliary LP covering term. Explicit formulas are given in (Zhou et al., 26 Dec 2025).

The proof strategy links the per-round suboptimality to the hybrid radii, bounds failure events, and efficiently aggregates the savings attributable to the offline data by integrating over the learning trajectory.

4. Empirical Findings and Evaluation

Extensive experiments substantiate the theory:

• Synthetic cascade ranking (Bernoulli arms, $m=10,k=5$):

When the offline sample has no bias ($V=0$) and $N\geq 200$, Hybrid CUCB achieves near-constant regret, overtaking the purely online CUCB baseline. Under moderate bias ($V$ up to $0.4$), Hybrid CUCB matches or improves over CUCB, always outperforming the offline-only CLCB algorithm by correcting for online distribution shift.

• Real-world MovieLens benchmarks:

The algorithm is robust to distributional shift between the offline and online phases, maintaining substantially lower regret than both CUCB and CLCB.

Key metrics include cumulative regret trajectories over $T$ rounds (averaged over 20 seeds), revealing that the hybrid approach yields both improved data efficiency and final performance, as quantified in (Zhou et al., 26 Dec 2025).

5. Practical Considerations and Implementation Guidance

• Bias Bound Setting:

The possibility of distribution shift from offline to online data is accounted for through bias bounds $V_i$. These should be tuned to reflect prior confidence in the logs: if alignment is likely, small $V_i$ can be chosen, but conservative settings default to $V_i=1$.

• Log Factors and Approximation Oracle:

The theoretical guarantees follow from using $\ln(4mt^3)$ in the confidence radii, though in practice $\ln(mt)$ or $\ln T$ may suffice. The $(\alpha,\beta)$ oracle can be instantiated with any fast approximate solver—the algorithm is agnostic to combinatorial structure.

• Algorithmic Robustness:

Rapid improvement is observed when $N_i$ are large and bias is low; the hybrid UCB dominates exploration. If not, the fallback to pure-online ensures no performance loss compared to standard CUCB.

6. Relationship to Other CUCB-type Algorithms

Hybrid CUCB differs from previous variants such as the CUCB-Avg algorithm (Li et al., 2020), which utilizes upper confidence bounds together with sample averages in fixed or time-varying target combinatorial bandits, and GLR-CUCB (1908.10402), which augments CUCB with sequential change-point detection in piecewise-stationary environments. Unlike those, Hybrid CUCB explicitly integrates bias-aware offline estimates with online updating and uses the minimum of pure and hybrid UCBs for each arm, achieving provable improvements in regret whenever informative offline data are available (Zhou et al., 26 Dec 2025).

7. Significance and Research Outlook

The Hybrid CUCB algorithm constitutes a unifying advance for CMAB-T, bridging the gap between purely online and offline paradigms. It leverages prior data—provided that coverage quality and bias are quantitatively controlled—to accelerate learning while retaining robustness to distributional mismatch. Empirical evidence indicates that Hybrid CUCB consistently demonstrates superior regret minimization in both synthetic and real-world domains, and its theoretical characterization sets a benchmark for future advances in hybrid bandit frameworks (Zhou et al., 26 Dec 2025).