Dueling-Bandit Selection: Sparse and Efficient Methods

• Dueling-bandit selection is the process of identifying optimal items through noisy pairwise duels, emphasizing relative feedback over direct rewards.
• Algorithms employ winner criteria such as Borda, Copeland, and Condorcet, with methods like SECS exploiting sparsity to efficiently eliminate suboptimal choices.
• Theoretical guarantees and empirical evaluations demonstrate reduced sample complexity and robust performance in both synthetic simulations and real-world ranking tasks.

Dueling-bandit selection is the problem of sequentially identifying optimal or near-optimal items from a finite set based solely on noisy pairwise comparisons, rather than absolute reward signals. This setting is fundamental in interactive learning, online ranking, and preference elicitation, where only relative (comparative) feedback is available. The field covers sample-optimal pure-exploration and regret-minimization algorithms under a variety of winner criteria, including the Borda, Copeland, and Condorcet notions, with specialized methodologies leveraging structure such as sparsity or transitivity in the underlying preference matrix.

1. Problem Formulation and Winner Criteria

Consider $K$ arms, with each action a pairwise "duel" $(i, j)$ between distinct arms, yielding a random outcome $Z^{(t)}_{i,j} \in \{0,1\}$, where $P[Z^{(t)}_{i,j}=1]=p_{i,j}$, $p_{j,i}=1-p_{i,j}$. The collection $P=(p_{i,j})$ is called the preference matrix, generally unknown and fixed but possibly subject to regularity constraints such as $p_{i,j} \in [3/8,5/8]$ for identification purposes (Jamieson et al., 2015).

Winner Criteria:

• Borda Winner: The unique arm $i^*$ maximizing the Borda score

$s_i = \frac{1}{K-1}\sum_{j \neq i} p_{i,j}.$

$i^*$ is the Borda winner if $(i, j)$0 for all $(i, j)$1.

• Copeland Winner: For each arm $(i, j)$2, define the Copeland score as the fraction of arms $(i, j)$3 that $(i, j)$4 beats (i.e., $(i, j)$5), normalized by $(i, j)$6. The Copeland winner(s) maximize this score. Unlike the Condorcet winner, the Copeland winner always exists.
• Condorcet Winner: An arm $(i, j)$7 such that $(i, j)$8 for all $(i, j)$9. There may not always be a Condorcet winner, but a Borda or Copeland winner exists under mild assumptions (Zoghi et al., 2015).

The goal can be pure exploration (best-arm identification with high confidence and minimal samples) or regret minimization (minimize cumulative gap to optimality over time).

2. Sample Complexity and Lower Bounds

Borda Selection: Minimax Rates and Optimality

In the absence of special structure, achieving $Z^{(t)}_{i,j} \in \{0,1\}$0-PAC selection of the Borda winner requires

$Z^{(t)}_{i,j} \in \{0,1\}$1

duels, where $Z^{(t)}_{i,j} \in \{0,1\}$2 (Jamieson et al., 2015). The lower bound matches this up to constant and lower-order logarithmic terms:

$Z^{(t)}_{i,j} \in \{0,1\}$3

Thus, in the generic (non-structured) case, the minimax sample complexity is $Z^{(t)}_{i,j} \in \{0,1\}$4.

Impact of Sparsity

A notable advance is the exploitation of "sparsity" in the comparison matrix. Let $Z^{(t)}_{i,j} \in \{0,1\}$5 for each suboptimal arm $Z^{(t)}_{i,j} \in \{0,1\}$6 (assuming arm 1 is the unique Borda winner). If, for each $Z^{(t)}_{i,j} \in \{0,1\}$7, $Z^{(t)}_{i,j} \in \{0,1\}$8 is $Z^{(t)}_{i,j} \in \{0,1\}$9-approximately sparse—i.e., the sum of discriminative gaps outside the $P[Z^{(t)}_{i,j}=1]=p_{i,j}$0 largest entries is at most $P[Z^{(t)}_{i,j}=1]=p_{i,j}$1 times the main partial gap for the $P[Z^{(t)}_{i,j}=1]=p_{i,j}$2 indices—then the leading term of $P[Z^{(t)}_{i,j}=1]=p_{i,j}$3 in sample complexity can be reduced by a factor of $P[Z^{(t)}_{i,j}=1]=p_{i,j}$4 (Jamieson et al., 2015). That is, only $P[Z^{(t)}_{i,j}=1]=p_{i,j}$5 duels are needed per arm.

3. Algorithmic Approaches: SECS and Successive Elimination

The SECS Algorithm

Successive Elimination with Comparison Sparsity (SECS) augments classical elimination with a sparse test. At each round, the algorithm either:

1. Applies standard Borda elimination: if the empirical Borda gap for $P[Z^{(t)}_{i,j}=1]=p_{i,j}$6 is large enough, eliminate $P[Z^{(t)}_{i,j}=1]=p_{i,j}$7.
2. Applies a sparse partial-gap test: evaluate empirical pairwise gaps on the $P[Z^{(t)}_{i,j}=1]=p_{i,j}$8 most discriminative comparison arms for $P[Z^{(t)}_{i,j}=1]=p_{i,j}$9. If the partial gap exceeds a threshold, eliminate $p_{j,i}=1-p_{i,j}$0.

Pseudocode Outline [algorithmic core from (Jamieson et al., 2015)]:

• Input: sparsity $p_{j,i}=1-p_{i,j}$1, confidence $p_{j,i}=1-p_{i,j}$2, (optionally) a time-gate $p_{j,i}=1-p_{i,j}$3
• Maintain an active set $p_{j,i}=1-p_{i,j}$4; at each time step:
  • For each $p_{j,i}=1-p_{i,j}$5, sample duels $p_{j,i}=1-p_{i,j}$6 against a random arm $p_{j,i}=1-p_{i,j}$7.
  • For $p_{j,i}=1-p_{i,j}$8, eliminate if there exists $p_{j,i}=1-p_{i,j}$9 such that:
  • (a) $P=(p_{i,j})$0 and the partial empirical gap over the $P=(p_{i,j})$1 maximal discrepancy arms exceeds $P=(p_{i,j})$2.
  • (b) The full empirical gap in Borda estimates is large enough: $P=(p_{i,j})$3.

Theoretical Guarantees

Under the $P=(p_{i,j})$4 sparsity condition (for, e.g., $P=(p_{i,j})$5), SECS identifies the Borda winner with high probability using at most

$P=(p_{i,j})$6

samples (Jamieson et al., 2015). For $P=(p_{i,j})$7 and moderate gaps, this provides a $P=(p_{i,j})$8 improvement in sample complexity relative to the non-sparse baseline.

If $P=(p_{i,j})$9, SECS reduces to standard successive elimination (full Borda reduction), matching known minimax rates (Jamieson et al., 2015).

4. Empirical Evaluation and Practical Impact

Synthetic and Real-World Data

• Synthetic case (P1 matrix): For $p_{i,j} \in [3/8,5/8]$0, with one top-2 distinguished group differing only on one other arm ($p_{i,j} \in [3/8,5/8]$1 sparsity), SECS achieves $p_{i,j} \in [3/8,5/8]$2 sample complexity, versus $p_{i,j} \in [3/8,5/8]$3 for standard Borda reduction. Log-log slopes are $p_{i,j} \in [3/8,5/8]$4 (standard) vs.\ $p_{i,j} \in [3/8,5/8]$5 (SECS).
• Real data (MSLR-WEB10k, MQ2008): In web ranking settings (arms as rankers/features), SECS with small $p_{i,j} \in [3/8,5/8]$6 (e.g., $p_{i,j} \in [3/8,5/8]$7) reduced the number of necessary duels by approximately $p_{i,j} \in [3/8,5/8]$8 compared to standard algorithms, with performance smoothly degrading to the full Borda cost as $p_{i,j} \in [3/8,5/8]$9. In all cases with unique Borda winners, SECS reliably identified the correct arm.

Practical Guidance

• When to use sparsity-based methods: If top candidates are only separated on a small number of discriminative arms/attributes, SECS yields substantial gains.
• Parameter tuning: Small $i^*$0 (e.g., $i^*$1) suffices for many real datasets; setting $i^*$2 is often acceptable; the confidence $i^*$3 as in any PAC algorithm.
• Otherwise: Use the standard Borda reduction—successive elimination based on Borda scores—if no structure is apparent.

5. Broader Connections and Related Developments

Extensions to Other Winner Criteria and Problem Structures

• Copeland and Condorcet settings: Algorithms such as Copeland Confidence Bound (CCB) and Scalable Copeland Bandits (SCB) generalize to broader winner criteria and retain $i^*$4 regret without Condorcet assumptions (Zoghi et al., 2015).
• Multi-dueling and Battling Bandits: Recent frameworks allow comparing larger subsets at each timestep and exploit richer feedback; however, fundamental sample complexity for winner-of-subset feedback is not improved unless richer ranking information is available (Saha et al., 2018). Under pure winner-information feedback, the lower bound matches the ordinary dueling-bandit setting.
• Combinatorial Structures: In combinatorial pure exploration settings (e.g., best matching in a bipartite graph), Borda-type objectives admit reductions to CPE-MAB settings, and Condorcet objectives require min-max combinatorial strategies (Chen et al., 2020).

Algorithmic Summary Table

Method	Structure Leveraged	Sample Complexity / Regret
Borda Reduction	None	$i^*$5
SECS (Sparse)	$i^*$6-sparsity	$i^*$7
CCB/SCB (Copeland)	None/minimal	$i^*$8 for CCB/SCB under mild assumptions
PAC-Battling Bandits	PL/Subset structure	Winner info: $i^*$9, Top-m: $s_i = \frac{1}{K-1}\sum_{j \neq i} p_{i,j}.$0

6. Theoretical and Practical Implications

Sparsity-aware algorithms such as SECS highlight that structural properties of the preference matrix can be exploited for substantial reductions in sample requirements, resulting in near-optimal practical and theoretical performance in large-scale systems with moderate discriminative structure. In the absence of such structure, classical reduction to multi-armed bandit techniques yield minimax bounds. The study of dueling-bandit selection serves as both a methodological foundation and a unifying paradigm for relative-feedback online decision-making in a wide range of applications.

• "Sparse Dueling Bandits" (Jamieson et al., 2015)
• "Copeland Dueling Bandits" (Zoghi et al., 2015)
• "PAC Battling Bandits in the Plackett-Luce Model" (Saha et al., 2018)
• "Combinatorial Pure Exploration of Dueling Bandit" (Chen et al., 2020)

These works collectively establish both tight theoretical rates and practical guidelines for sample-efficient selection in preference-based online learning frameworks.