Finite-Sample Convergence Guarantees

• Finite-sample convergence guarantees are nonasymptotic bounds that quantify estimator accuracy and convergence rates based on sample size and the underlying geometric properties.
• They reveal a multi-scale behavior where effective dimensions vary with resolution, demonstrating rapid convergence at coarse scales and slower refinement at finer scales.
• These guarantees have practical applications in Monte Carlo integration, clustering, and nonparametric inference, informing sample complexity and algorithm optimization in structured data.

Finite-sample convergence guarantees refer to explicit, nonasymptotic bounds that characterize the behavior of estimators, optimization methods, or learning algorithms for any finite number of samples or data points. Unlike asymptotic results, which describe limits as the sample size $n \to \infty$, finite-sample guarantees quantify rates and accuracy in terms of actual sample size, the underlying geometric or statistical structure of the problem, and—in modern work—can reveal a “multi-scale” picture where convergence rates depend on how the signal or distribution behaves at various resolutions or scales.

1. Finite-sample Convergence in Wasserstein Distance

A paradigmatic example of finite-sample convergence theory arises in studying the convergence of the empirical measure $\hat\mu_n$ built from $n$ i.i.d. samples from a probability measure $\mu$, to $\mu$ itself, with respect to the Wasserstein distance $W_p$. The rate at which $\mathbb{E}[W_p(\mu, \hat\mu_n)]$ decays as $n$ increases plays a central role in quantifying the reliability of sampling-based approximations in statistics, probability, and machine learning (Weed et al., 2017).

Sharp finite-sample rates are expressed in terms of geometric properties of $\mu$, specifically its covering numbers at scale $\varepsilon$, yielding scale-dependent “effective dimensions.” Let $d_n$ denote such dimension (see §3 below). For suitable measures and $d_n > 2p$, the bound

$$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1\, n^{-p/d_n}$$

holds, so that

$$\mathbb{E}[W_p(\mu, \hat\mu_n)] \lesssim n^{-1/d_n},$$

where $C_1$ is an explicit constant and the rates are non-asymptotic, applying for all $n$ above a threshold determined by the regularity of $\mu$.

When the geometric complexity (quantified via covering number–related quantities $m_n$) dominates, one also obtains bounds like

$$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1 \sqrt{m_n/n}.$$

These bounds are non-asymptotic and track the true measure-empirical discrepancy for any finite $n$.

2. Multi-scale Nature of Convergence Rates

A distinctive phenomenon revealed by finite-sample analysis is "multi-scale" behavior: measures often have different "effective dimensions" at various observational scales. For example, at coarse scales, $\mu$ may appear clustered or nearly discrete (low-dimensional), whereas at finer resolutions it exhibits complex, high-dimensional structure. This is formalized by examining how $d_n$ changes with $n$.

Mathematically, for any $\varepsilon' > 0$, if there exists $s > 2p$ such that

$$d_{\varepsilon}(\mu, \varepsilon^p) \leq s \quad \forall \varepsilon \le \varepsilon',$$

then, for all sufficiently large $n$,

$$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1 n^{-p/s} + C_2 n^{-1/2}.$$

This rate holds until $n$ is large enough that finer structure dominates, at which point the rate transitions (often slows) in accordance with intrinsic dimension at the newly resolved scale.

This multi-scale behavior accounts for cases where empirical measures converge much faster than the worst-case global asymptotic rate, as is typical when $\mu$ is a finite mixture of well-separated clusters or a convolution of point masses with a small Gaussian.

3. Geometric Quantification: Covering Numbers and Scale-adaptive Dimension

The mathematical machinery underpinning these results leverages metric geometry and covering numbers. Let $N(\mu, \varepsilon)$ be the minimal number of metric balls of radius $\varepsilon$ covering the support of $\mu$. The scale-adaptive dimension $d_n$ is defined as

$$d_n = \inf_{\varepsilon > 0} \max\left\{ d_{\ge \varepsilon}(\mu, \varepsilon^p),\, \frac{\log n}{-\log\varepsilon} \right\},$$

where $d_{\ge \varepsilon}(\mu,\varepsilon^p)$ captures the local covering complexity above scale $\varepsilon$.

Practical finite-sample bounds in this framework take the form: $$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1 n^{-p/d_n};$$ thus, the explicit convergence rate is dictated by the interplay of $n$ and the scale at which the geometry of $\mu$ “saturates” relative to sampling error.

An illustrative bound (Proposition 4.1 in (Weed et al., 2017)) is: $$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1 n^{-p/s} + C_2 n^{-1/2},\qquad s > 2p,$$ with explicit constants: $$C_1 = 27^p\left(2+\frac{1}{3^{\frac{s}{2}-p}-1}\right),\quad C_2 = (27/\varepsilon')^{s/2}.$$

4. Applications: Numerical Integration, Learning, and Clustering

Finite-sample convergence rates in Wasserstein distance have critical implications across multiple domains:

• Numerical integration: For Monte Carlo quadrature, with approximation error for Lipschitz functionals controlled by $\mathbb{E}[W_1(\mu, \hat\mu_n)]$, the results justify the surprisingly efficient empirical behavior of sample mean approximations—especially when the underlying distribution is “effectively low-dimensional” at sample-accessible scales.
• Unsupervised learning/clustering: Many clustering or quantization algorithms (e.g., $k$-means, discrete approximations to continuous distributions) require bounds on the quality of empirical representations. The rapid convergence for measures exhibiting coarse-scale discretization justifies nearly optimality of empirical $k$-means centroids relative to the population objective.
• Statistical estimation and nonparametric inference: When constructing estimators of probability measures from samples, e.g., in density estimation or GAN training, these bounds directly inform the error between the empirical and population distributions in powerful, geometry-adaptive manners.

5. Comparison with Asymptotic Theory: Transition and Complementarity

Classical asymptotic results, such as Dudley’s, assert that for measures with full $d$-dimensional support,

$W_1(\mu, \hat\mu_n) \sim n^{-1/d}.$

However, this is only the limiting rate as $n \to \infty$. Finite-sample theory uncovers the sharper fact that empirical convergence may initially follow much faster rates $n^{-1/d'}$ for an effective dimension $d' < d$ at accessible scales—slowing only as $n$ grows large enough to resolve high-complexity microstructure.

Thus, finite-sample and asymptotic results together describe a transition: fast convergence at low-resolution, possibly clustered scales, then gradual approach to the limiting, possibly slow, worst-case rate. This complementarity is essential for understanding error in data-driven algorithms (especially in high dimension, nonuniform, or clustered regimes).

6. Practical Implications and Theoretical Insights

These results fundamentally change the interpretation of empirical approximation error in statistical learning and computational mathematics. They demonstrate that sample-based methods benefit quantitatively from favorable geometric structure (i.e., concentration or low-dimensional support) of the data-generating measure, and can far outperform predictions based solely on ambient dimension.

Key numerical observations:

• For $n$ not extremely large, effective sample complexity is dramatically improved if $\mu$ is clustered or nearly discrete at the relevant observational scale.
• For measures where the covering number grows polynomially with $1/\varepsilon$ (dimension $d$), the classical $n^{-1/d}$ asymptotic rate is recovered, but for measures that are mixtures of Diracs or have “intrinsically” low-dimensional support at moderate scales, the observed rate is much faster.

In practical terms, practitioners can leverage these results to:

• Justify faster-than-expected empirical convergence in high-dimensional but structured data,
• Guide the necessary sample size for a desired accuracy in function integration or distributional approximation,
• Inform the design of learning algorithms sensitive to underlying geometric structure (such as adaptive quantization or cluster-based modeling).

7. Summary Table of Core Results

Setting	Convergence Bound	Dimension Parameter
General measure	$$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1 n^{-p/d_n}$$	$d_n$ (scale-adaptive)
Scale $s > 2p$	$$\mathbb{E}[W_p^p(\mu, \hat\mu_n)] \leq C_1 n^{-p/s} + C_2 n^{-1/2}$$	$s$
Effective at scale $\varepsilon$	$d_{\varepsilon}(\mu, \varepsilon^{p})$ controls local rate	$d_{\varepsilon}$

These finite-sample convergence guarantees (Weed et al., 2017) offer a precise quantitative link between the geometry of a measure—via covering numbers, clustering, and “local dimension”—and the rate at which empirical measures approximate the underlying truth, both in theory and in the implementation of modern data-driven algorithms.