Dueling Kernel: Bias-Robust Bayesian Optimization

• Dueling Kernel is a kernelized dueling-bandit method that uses pairwise reward differences to overcome additive, action-independent biases.
• It reformulates confounded Bayesian optimization into a quantitative dueling feedback model with one-point and two-point reductions, enabling unbiased difference estimation.
• The method employs RKHS regression with confidence quantification and an IDS-based decision rule to achieve sublinear cumulative regret.

Searching arXiv for the cited paper and closely related work on kernelized/dueling bandits and GP-based Bayesian optimization. Searching arXiv for: (Kirschner et al., 2021), semiparametric linear bandits, GP-UCB, and dueling bandits. “Dueling Kernel” denotes the kernelized dueling-bandit construction introduced in “Bias-Robust Bayesian Optimization via Dueling Bandits” (Kirschner et al., 2021). It addresses Bayesian optimization when the observed reward is corrupted by an additive, adversarially chosen bias that does not depend on the chosen action, and it does so by shifting attention from absolute rewards to reward differences. The resulting framework combines a reduction from confounded reward observations to quantitative dueling feedback with an information-directed sampling (IDS) algorithm defined in an RKHS. In the formulation studied in the paper, the unknown objective $f:\mathcal X\to\mathbb R$ lies in a known RKHS $\mathcal H$ with kernel $k$, bounded norm $\|f\|_{\mathcal H}\le B$, and $k(x,x)\le 1$, while the learner observes

$y_t = f(x_t) + b_t + \epsilon_t,$

with zero-mean $\sigma^2$-sub-Gaussian noise $\epsilon_t$ and an unobserved confounding term $b_t$. The central point is that the argmax of $f$ is invariant to additive shifts, so the optimizer can be recovered from pairwise differences even when absolute levels are not identifiable (Kirschner et al., 2021).

1. Problem formulation and identifiability

The setting is a confounded Bayesian optimization problem over a compact action space $\mathcal H$0. The cumulative regret objective is

$\mathcal H$1

where

$\mathcal H$2

The confounding process is modeled in two ways: bounded bias, $\mathcal H$3, and bounded drift between consecutive biases, $\mathcal H$4, with $\mathcal H$5 fixed before the current action or action pair is chosen (Kirschner et al., 2021).

The identifiability issue is structural. Because the bias is additive and action-independent, $\mathcal H$6 is not identifiable up to an additive constant. However, $\mathcal H$7 remains identifiable. This leads to a reformulation in which pairwise comparisons, rather than direct reward levels, become the sufficient statistical object for optimization. The paper treats this as the decisive conceptual bridge between bias-robust Bayesian optimization and dueling bandits.

A common misconception is to interpret the method as a generic solution to arbitrary corruption. The model is narrower: the bias cannot depend on the current action. The paper’s guarantees derive precisely from that restriction, because it preserves reward differences under confounding.

2. Reduction to quantitative dueling feedback

The dueling feedback model used in the construction is

$\mathcal H$8

where $\mathcal H$9 is $k$0-sub-Gaussian. Although binary preference feedback is included as a special case, the framework studied here is quantitative: the feedback is a noisy estimate of a reward difference (Kirschner et al., 2021).

Two reductions are given.

Reduction	Constructed feedback	Assumption used
Two-point reduction	$k$1	$k$2
One-point reduction	$k$3	$k$4

In the two-point reduction, the learner chooses $k$5, evaluates both with randomized order,

$k$6

with $k$7, and sets $k$8. Randomization makes the induced difference unbiased: $k$9 Under bounded bias difference, the paper states the variance proxy

$\|f\|_{\mathcal H}\le B$0

In the one-point reduction, the learner randomly selects one of two candidate points,

$\|f\|_{\mathcal H}\le B$1

and defines

$\|f\|_{\mathcal H}\le B$2

Again,

$\|f\|_{\mathcal H}\le B$3

and if $\|f\|_{\mathcal H}\le B$4, the induced dueling noise is sub-Gaussian with variance proxy

$\|f\|_{\mathcal H}\le B$5

This reduction matters because it replaces direct bias correction with randomized comparison design. A plausible implication is that the hard part of the original problem is not estimating the bias process itself, but constructing unbiased difference observations despite the bias.

3. RKHS least-squares estimator on reward differences

Once the problem is rewritten as dueling feedback, the learning problem becomes kernel regression on pairwise differences. Given previous duels

$\|f\|_{\mathcal H}\le B$6

the estimator is

$\|f\|_{\mathcal H}\le B$7

The associated kernel matrix is defined entrywise by

$\|f\|_{\mathcal H}\le B$8

and for any $\|f\|_{\mathcal H}\le B$9,

$k(x,x)\le 1$0

The estimator admits the closed form

$k(x,x)\le 1$1

with $k(x,x)\le 1$2 (Kirschner et al., 2021).

Uncertainty is also expressed in difference form. The posterior-style kernel correction is

$k(x,x)\le 1$3

and the pairwise uncertainty quantity is

$k(x,x)\le 1$4

With probability at least $k(x,x)\le 1$5,

$k(x,x)\le 1$6

where

$k(x,x)\le 1$7

The paper then turns this pairwise confidence set into a bound on action suboptimality. Let

$k(x,x)\le 1$8

$k(x,x)\le 1$9

and

$y_t = f(x_t) + b_t + \epsilon_t,$0

The key lemma gives

$y_t = f(x_t) + b_t + \epsilon_t,$1

with probability $y_t = f(x_t) + b_t + \epsilon_t,$2, for all $y_t = f(x_t) + b_t + \epsilon_t,$3. This makes the regret of any action controllable through difference-based RKHS confidence estimates rather than absolute reward recovery.

4. Information-directed sampling and approximate optimization

The decision rule is an IDS procedure specialized to dueling bandits. Information gain for a pair is defined as

$y_t = f(x_t) + b_t + \epsilon_t,$4

and the cumulative information gain is

$y_t = f(x_t) + b_t + \epsilon_t,$5

The paper lists the examples

$y_t = f(x_t) + b_t + \epsilon_t,$6

and

$y_t = f(x_t) + b_t + \epsilon_t,$7

as cited from Srinivas et al. (Kirschner et al., 2021).

The exact IDS problem over distributions on action pairs is described as computationally expensive, so the paper proposes an efficient approximation. At round $y_t = f(x_t) + b_t + \epsilon_t,$8, the procedure computes the current optimistic maximizer $y_t = f(x_t) + b_t + \epsilon_t,$9, the gap estimates $\sigma^2$0, and the information gains $\sigma^2$1, then solves

$\sigma^2$2

With probability $\sigma^2$3, it plays the duel $\sigma^2$4; otherwise it plays the degenerate pair $\sigma^2$5, which yields no new information.

The optimization criterion can be read as squared regret per unit information. In the paper’s interpretation, $\sigma^2$6 measures uncertainty about whether $\sigma^2$7 is optimal, $\sigma^2$8 measures how bad a comparison with $\sigma^2$9 might be, and $\epsilon_t$0 measures how much the duel teaches. The approximation reduces the per-round cost on finite action sets from $\epsilon_t$1 for direct IDS to $\epsilon_t$2 after kernel quantities are available. The paper further states overall computational costs of $\epsilon_t$3 in the kernelized setting and $\epsilon_t$4 in the linear setting (Kirschner et al., 2021).

5. Regret guarantees and transfer back to confounded Bayesian optimization

For dueling feedback, regret is defined as

$\epsilon_t$5

The main theorem states that for $\epsilon_t$6-sub-Gaussian dueling feedback, the algorithm satisfies with probability at least $\epsilon_t$7,

$\epsilon_t$8

The more informative rate exposed by the proof is the standard IDS-style bound

$\epsilon_t$9

up to log factors and constants, via an information-ratio argument using

$b_t$0

The paper also gives a gap-dependent result: if the optimum is unique and

$b_t$1

then

$b_t$2

For kernels, the text highlights that the linear-kernel scaling matches LinUCB, while the RBF-kernel bound depends on $b_t$3-type factors (Kirschner et al., 2021).

Composing the reduction with the dueling algorithm yields guarantees in the original confounded observation model. Under the one-point reduction with $b_t$4,

$b_t$5

Under the two-point reduction with $b_t$6,

$b_t$7

These statements are notable because the dependence is on the magnitude of the bias or on the difference between consecutive biases. The paper explicitly notes that the method can tolerate even some forms of unbounded drift under the two-point scheme.

A second common misconception is to view these bounds as standard GP-UCB-style guarantees under ordinary noisy observations. The point of the construction is precisely that standard Bayesian optimization procedures can fail under adversarial additive bias or unbounded drift, whereas the difference-based formulation preserves usable signal.

6. Relation to semiparametric linear bandits and prior work

The paper explicitly connects the kernelized dueling construction to the semiparametric linear bandit model of Krishnamurthy et al. In the linear case,

$b_t$8

That earlier line of work uses a doubly-robust estimator

$b_t$9

where $f$0. The key observation is that if $f$1 is uniform over a pair $f$2, then

$f$3

so the doubly-robust estimator coincides with the least-squares estimator induced by the dueling feedback reduction (Kirschner et al., 2021).

This identifies the “Dueling Kernel” perspective as more than a kernelization of an existing linear estimator. It generalizes the semiparametric linear model to non-linear RKHS reward functions and provides an alternative interpretation of doubly-robust estimation through pairwise comparisons. This suggests that the role of randomization is simultaneously statistical and structural: it both removes action-independent confounding and induces the geometry needed for kernel least squares on differences.

Relative to prior work, the paper distinguishes itself in four ways. Compared to semiparametric linear bandits such as BOSE and SemiTS, it avoids explicitly solving for low-variance exploration distributions in the confounded observation model. Compared to robust Bayesian optimization with action-dependent corruption, its bias model is narrower because the bias cannot depend on the current action, but it allows sublinear regret even with persistent constant bias or certain drift processes. Compared to standard dueling bandits, it handles quantitative dueling feedback and provides the first efficient kernelized algorithm with cumulative regret guarantees. Compared to GP-UCB, it is designed for robustness to confounding rather than ordinary stochastic noise (Kirschner et al., 2021).

Within that landscape, “Dueling Kernel” is best understood as an overview of three ingredients: a reduction from confounded Bayesian optimization to dueling bandits, RKHS regression and confidence quantification on reward differences, and an efficient IDS rule that balances estimated regret against information gain.