Best-of-Both-Worlds Multi-Dueling Bandits: Unified Algorithms for Stochastic and Adversarial Preferences under Condorcet and Borda Objectives

Abstract: Multi-dueling bandits, where a learner selects $m \geq 2$ arms per round and observes only the winner, arise naturally in many applications including ranking and recommendation systems, yet a fundamental question has remained open: can a single algorithm perform optimally in both stochastic and adversarial environments, without knowing which regime it faces? We answer this affirmatively, providing the first best-of-both-worlds algorithms for multi-dueling bandits under both Condorcet and Borda objectives. For the Condorcet setting, we propose \texttt{MetaDueling}, a black-box reduction that converts any dueling bandit algorithm into a multi-dueling bandit algorithm by transforming multi-way winner feedback into an unbiased pairwise signal. Instantiating our reduction with \texttt{Versatile-DB} yields the first best-of-both-worlds algorithm for multi-dueling bandits: it achieves $O(\sqrt{KT})$ pseudo-regret against adversarial preferences and the instance-optimal $$O!\left(\sum_{i \neq a^\star} \frac{\log T}{Δ<em>i}\right)$$ pseudo-regret under stochastic preferences, both simultaneously and without prior knowledge of the regime. For the Borda setting, we propose \AlgBorda, a stochastic-and-adversarial algorithm that achieves $O\left(K² \log KT + K \log² T + \sum</em>{i: Δ_i^{\mathrm{B}} &gt; 0} \frac{K\log KT}{(Δ_i<sup>{\mathrm{B}})<sup>2}\right)$ regret in stochastic environments and $$O\left(K \sqrt{T \log KT} + K^{1/3} T^{2/3} (\log K)<sup>{1/3}\right)$$ regret against adversaries, again without prior knowledge of the regime. We complement our upper bounds with matching lower bounds for the Condorcet setting. For the Borda setting, our upper bounds are near-optimal with respect to the lower bounds (within a factor of $K$) and match the best-known results in the literature.

• The paper introduces MetaDueling, a black-box reduction that transfers dueling-bandit guarantees to multi-dueling settings with at most a constant-factor loss, regardless of the number of selected arms.
• The paper achieves simultaneously optimal Condorcet regret—O(√KT) adversarial and instance-dependent O(Σ log T/Δᵢ) stochastic—without knowing the environment’s regime.
• The paper proposes SA-MiDEX for Borda objectives, obtaining gap-dependent stochastic and near-optimal adversarial guarantees while highlighting open challenges such as quadratic exploration costs and a remaining factor-K gap.

This paper resolves an open question in preference-based online learning: whether a single algorithm can achieve optimal regret in both stochastic and adversarial multi-dueling bandit environments without prior knowledge of which regime it faces. The authors answer affirmatively for both Condorcet and Borda objectives, providing the first best-of-both-worlds guarantees for multi-dueling bandits, where a learner selects $m \geq 2$ arms per round and observes only the winner (2603.18972).

Setting and problem statement

The learner operates over $K$ arms for $T$ rounds. At each round, the environment (obliviously) selects a reciprocal preference matrix $P_t$, the learner plays a multiset of size $m$, and observes only the winning arm under the pairwise-subset choice model of Saha and Gopalan, where each arm's win probability is the average of its pairwise preferences within the multiset. In the stochastic regime, $P_t = P$ is fixed; in the adversarial regime, $\{P_t\}$ is arbitrary.

Two optimality criteria are considered. The Condorcet setting assumes an arm $a^\star$ beating all others, with pseudo-regret measured by average sub-optimality gaps of played arms. The Borda setting measures regret against the arm maximizing average pairwise win probability; this criterion always admits a winner even when preferences are cyclic — the paper notes that no Condorcet winner exists in the MSLR-WEB10k dataset — and is more robust to estimation errors when the two notions diverge. Prior work addressed these regimes separately: gap-dependent stochastic bounds under Condorcet winners [Saha and Gopalan 2018] and $O((K\log K)^{1/3}T^{2/3})$ adversarial bounds under Borda scores [Gajane 2024], but no algorithm adapted to both regimes simultaneously.

MetaDueling: a black-box reduction for Condorcet objectives

The central technical device is MetaDueling, a reduction converting any dueling bandit algorithm into a multi-dueling algorithm. Each round, the base learner proposes two arms $(x_t, y_t)$; the reduction constructs a multiset containing only these two arms, with counts $K$0 summing to $K$1. For odd $K$2, a random symmetrization of the split ensures unbiasedness.

The key observation is that duplicate copies introduce tie bias: some probability mass goes to identical-arm comparisons won with probability $K$3. The paper shows this bias corresponds exactly to an affine rescaling of the preference matrix,

$K$4

with constants depending only on $K$5. Three structural facts make the reduction work: (i) the observed binary outcome is unbiased for $K$6 rather than $K$7; (ii) $K$8 is itself a valid preference matrix that preserves reciprocity, self-comparison, boundedness, and the Condorcet winner; and (iii) gaps rescale as $K$9 with $T$0 for all $T$1. Consequently, multi-dueling Condorcet regret equals base-algorithm dueling regret under rescaled preferences up to the factor $T$2 — a universal constant independent of $T$3.

Instantiating the reduction with Versatile-DB yields the main result: simultaneously, $T$4 adversarial pseudo-regret and instance-optimal $T$5 stochastic pseudo-regret, both without knowledge of the regime. Notably, the stochastic bound depends on individual gaps rather than only the minimum gap, improving on the $T$6 bound of prior stochastic work while additionally covering the adversarial case. The guarantee extends immediately to time-varying subset sizes $T$7, since the rescaling holds pointwise per round.

Matching lower bounds for the Condorcet setting

The paper proves matching lower bounds via reductions from dueling bandits: any multi-dueling algorithm can be simulated by a dueling algorithm that samples two indices uniformly from the chosen multiset, and this simulation preserves expected regret exactly. Combined with known minimax and KL-based lower bounds, this yields $T$8 adversarial and $T$9 stochastic lower bounds, establishing that the upper bounds are optimal in both regimes. The paper also correctly notes that best-of-both-worlds guarantees must be stated in expectation: following Auer and Chiang, no algorithm can achieve optimal high-probability bounds in both regimes simultaneously, making pseudo-regret the natural target.

SA-MiDEX for Borda objectives

The Borda setting requires different machinery because estimating an arm's score requires comparisons against all opponents, not just one. The proposed SA-MiDEX follows the stochastic-and-adversarial paradigm of Auer and Chiang rather than entropy regularization. It uses MetaDueling internally, applying the inverse affine transformation $P_t$0 to recover unbiased estimates of true pairwise preferences.

In stochastic mode, the algorithm proceeds through three stages: a baseline exploration phase of $P_t$1 round-robin rounds establishing high-confidence pairwise estimates; successive elimination over an active set with uniform opponent sampling to keep Borda estimates unbiased; and background monitoring that forces uniform exploration with decaying probability $P_t$2 to feed deviation statistics. Deviation detection tracks cumulative departures from the locked baseline per pair using Freedman-type martingale concentration. Upon detection, the algorithm irreversibly switches to adversarial mode, running EXP3 with explicit uniform exploration ($P_t$3) to bound importance-weighting variance, using an importance-weighted shifted-Borda estimator whose unbiasedness is proven directly.

The resulting guarantees are:

Regime	Regret bound
Stochastic	$P_t$4
Adversarial	$P_t$5

Both hold simultaneously without regime knowledge. Against the known lower bounds — $P_t$6 stochastic and $P_t$7 adversarial — these are near-optimal within a factor of $P_t$8 and match the best-known results in the literature. The adversarial analysis carefully splits into switch/no-switch cases: undetected pre-switch manipulation is bounded by $P_t$9 via Cauchy–Schwarz over pairs, while post-switch EXP3 contributes the $m$0 term; neither dominates across all horizons, so both are retained.

Limitations and open questions

The paper concedes several limitations at the points where they bear on the results. First, SA-MiDEX's $m$1 exploration overhead — required to learn all pairwise preferences for unbiased Borda estimation — dominates for small $m$2 or large $m$3; structured preference models such as low-rank or parametric assumptions might reduce it, but this is left unexplored. Second, both algorithms construct multisets containing only two distinct arms, discarding the potential information in genuine $m$4 comparisons; exploiting all $m$5 pairwise signals would likely sacrifice the black-box property of the reduction. Third, the Condorcet results require the existence of a Condorcet winner, and the self-bounding condition of Versatile-DB is invoked for the stochastic bound; extending best-of-both-worlds guarantees to cyclic preferences would require a new regret notion that remains sublinear adversarially. Fourth, the Borda upper bounds carry a factor-$m$6 gap relative to the lower bounds, and closing it is open. Finally, the framework targets winner-only feedback; extension to richer feedback (full rankings, pairwise margins) and to dynamic benchmarks with time-varying winners remain open questions explicitly posed by the authors.

Conclusion

This paper establishes the first best-of-both-worlds guarantees for multi-dueling bandits. For Condorcet objectives, the MetaDueling reduction shows that the entire complexity of the $m$7-way interaction reduces to a constant factor at most 2, transferring optimal dueling-bandit guarantees verbatim and achieving matching lower bounds in both regimes. For Borda objectives, SA-MiDEX demonstrates that gap-dependent stochastic rates and near-optimal adversarial rates are simultaneously attainable despite the harder estimation problem, at the cost of a quadratic-in-$m$8 exploration overhead. The modularity of the reduction means future improvements to dueling bandit algorithms transfer automatically to the multi-dueling setting.