Kernel Stein Discrepancies (KSDs): Theory & Applications

• Kernel Stein Discrepancies (KSDs) are integral probability metrics that combine Stein's method with RKHS theory to measure discrepancies between sample and target distributions.
• They provide closed-form estimators based solely on the target’s score function and kernel evaluations, enabling efficient goodness-of-fit tests and sampler comparisons.
• Using slowly decaying kernels like the IMQ kernel, KSDs ensure convergence-determining properties, robust diagnostics, and scalability for high-dimensional inference.

Kernel Stein Discrepancies (KSDs) are a class of integral probability metrics that utilize Stein’s method and reproducing kernel Hilbert space (RKHS) theory to quantitatively assess the discrepancy between a sample distribution and a target probability measure. Central to their practical appeal is the availability of closed-form estimators that require only the score function of the target density and pairwise kernel evaluations. KSDs provide a computationally tractable tool for goodness-of-fit testing, MCMC diagnostics, model criticism, and sample quality improvement in both exact and approximate inference, and have been rigorously analyzed for their convergence-determining properties, separation power, and practical deployment.

1. Principle and Formulation

The construction of a KSD starts from Stein's method, where for a target distribution $P$ with density $p$ and score function $b(x) = \nabla \log p(x)$, the Langevin Stein operator $T$ satisfies $E_P[Tg] = 0$ for all $g$ in a suitable class of test functions. By choosing $g$ from the RKHS associated with a positive-definite kernel $k$, the test function set becomes the "kernel Stein set" $\mathcal{G}_k$: $$\mathcal{G}_k = \left\{g = (g_1, \ldots, g_d): \|[g_1, \ldots, g_d]\|_{\mathrm{RKHS}^d} \leq 1\right\}$$ The kernel Stein discrepancy is defined as

$$\mathrm{KSD}(Q, P) = \sup_{g \in \mathcal{G}_k} |E_Q[Tg] - E_P[Tg]|$$

where $Q$ is the sample distribution.

A crucial property is the closed-form representation. For $b_j$ the $j$th coordinate of the score, the Stein kernel in the $j$th direction is

$$k_0^j(x, y) = b_j(x) b_j(y) k(x, y) + b_j(x) \partial_{y_j} k(x, y) + b_j(y) \partial_{x_j} k(x, y) + \partial_{x_j} \partial_{y_j} k(x, y)$$

and the squared KSD is

$$\mathrm{KSD}^2(Q, P) = \sum_{j=1}^d E_{X, \tilde{X} \sim Q}[k_0^j(X, \tilde{X})]$$

This structure makes KSDs amenable to efficient pairwise computation and massive parallelization (Gorham et al., 2017).

2. Convergence and Separation Properties

KSD convergence theory distinguishes between two core properties: whether the KSD is convergence-determining (controls weak convergence to $P$) and whether it is separating ($\mathrm{KSD}(Q,P) = 0 \implies Q = P$). Not all kernel choices yield convergence-determining KSDs. Kernels such as the Gaussian, Matérn, or other rapidly decaying functions may be insensitive to departure from the target in the tails: the KSD with these kernels can converge to zero even when the empirical measure does not weakly converge to $P$ (i.e., mass escapes to infinity) (Gorham et al., 2017).

To counter this, the use of slowly decaying kernels such as the inverse multiquadric (IMQ) kernel,

$$k(x,y) = (c^2 + \|x - y\|^2)^\beta,\quad \beta \in (-1, 0)$$

is proposed. An IMQ-based KSD penalizes tail deviations and is proven to control both tightness and weak convergence: if $\mathrm{KSD}(Q_n, P) \rightarrow 0$, then $Q_n$ is uniformly tight and thus converges weakly to $P$ (Gorham et al., 2017). This distinguishes IMQ KSDs as convergence-determining discrepancies, rectifying the failure modes of lighter-tailed kernels.

3. Applications and Diagnostics

KSDs, especially when equipped with slowly decaying kernels, have demonstrated a wide diversity of applications:

• Sample Quality Assessment: KSD reliably distinguishes high-quality approximations of the target from poor ones, including cases where classical diagnostics—such as effective sample size (ESS) or moment matching—fail due to undetected asymptotic bias.
• Comparing Samplers: Sequence comparison via KSD values allows principled selection among biased and unbiased samplers, as well as deterministic and random sampling schemes.
• Hyperparameter Selection: KSD can be minimized as a function of algorithm hyperparameters, allowing selection of, for instance, MCMC step size for an optimal trade-off between bias and variance that is often invisible to traditional criteria like ESS.
• Hypothesis Testing: One-sample tests based on KSD enjoy closed-form test statistics and, if appropriately regularized with IMQ kernels, maintain high power in high-dimensional settings where Gaussian-kernel-based tests suffer severe power loss.
• Sample Improvement: Given a set of samples, reweighting or thinning to minimize the KSD has been shown to reduce worst-case quadrature error over smooth function classes (Gorham et al., 2017).

Empirical results robustly demonstrate that IMQ KSD maintains sensitivity even as dimension increases or under alternatives that evade moment-based diagnostics.

4. Kernel Choice and Limitations

The sensitivity of KSDs is fundamentally modulated by the chosen kernel’s tail decay. Theoretical warnings show that Gaussian and similar kernels are “blind” to sequences $Q_n$ that lose tightness, since the kernel values and their derivatives decay faster than the score function grows in the tails (Gorham et al., 2017). Conversely, the IMQ kernel’s slow decay ensures that the discrepancy remains sensitive to large excursions from the target's bulk.

A representative counterexample constructed in (Gorham et al., 2017) demonstrates that for $d \geq 3$, there exist non-tight empirical measures $Q_n$ supported on isolated points at infinity such that $\mathrm{KSD}(Q_n, P_\text{Gaussian}) \to 0$ under the Gaussian or Matérn kernels, even though $Q_n$ does not converge to $P$.

Thus, for high-dimensional and heavy-tailed targets, the kernel’s tail behavior is decisive for convergence-determining power.

5. Computational and Scalability Aspects

KSD test statistics for a sample of size $n$ are expressed as pairwise sums over kernel evaluations and their derivatives, leading to a computational complexity of $O(n^2 d)$ per KSD evaluation (for $d$ vector components). This avoids linear or quadratic programming subroutines required by some earlier Stein discrepancies and can be massive parallelized. KSD computations are efficient and practical for large-scale problems and have been demonstrated to scale to high-dimensional inference scenarios (Gorham et al., 2017).

Additionally, the closed form of the KSD functional enables differentiable and parallel computations that are essential for integration into automated hyperparameter tuning or particle optimization schemes.

6. Broader Impact and Practical Implications

The advances of KSD methodology in (Gorham et al., 2017) clarify both the theoretical limits and practical opportunities for sample-based convergence diagnostics:

• Diagnostic Trustworthiness: IMQ KSD’s convergence-determining property ensures that small KSD values guarantee not only moment-matching but also that mass does not escape into the tails—a critical assurance not provided by classical statistics.
• Scalability: KSD’s pairwise, kernel-based structure makes it suitable for high-volume and high-dimensional data, supporting modern statistical computing pipelines.
• Reliable Automation: By providing a theoretically validated, extensible, and efficiently computable metric, KSDs can serve as robust backbone diagnostics for automated algorithms in MCMC, variational inference, sample postprocessing, and goodness-of-fit testing.

In summary, the combination of RKHS structure, Stein-type operators, and careful kernel choice yields a practically indispensable tool for distributional approximation. The insights and methods elucidated in (Gorham et al., 2017) delineate rigorous boundaries for the power and limitations of KSDs and provide concrete guidance for their use in statistical machine learning and Bayesian computation.