Non-Asymptotic Convergence Rates

• Non-Asymptotic Convergence Rates are explicit error bounds that quantify finite-sample or iteration performance in statistical estimation, optimization, and sampling.
• They establish concrete metrics such as risk estimation, algorithmic convergence, and sampling bias to guide practical algorithm design.
• These rates inform parameter selection and initialization strategies to achieve robust and efficient outcomes in real-world, non-idealized settings.

Non-asymptotic convergence rates quantify the finite-sample or finite-iteration error of estimators, optimization algorithms, or sampling schemes, providing explicit error bounds that hold uniformly over all steps or sample sizes, rather than just asymptotically as the sample size or number of iterations tends to infinity. These rates are central to modern mathematical statistics, machine learning theory, stochastic optimization, sampling algorithms, and computational mathematics, as they formalize the speed and robustness with which practical algorithms approach their target solutions in realistic, non-idealized settings.

1. Foundations and Notions in Non-Asymptotic Convergence

Non-asymptotic convergence analyses provide uniform bounds on algorithmic or statistical errors as explicit functions of finite $n$ (sample size), $k$ (iteration), or discretization step size $h$. Canonically, these take the form:

• Risk estimation: $|R_n - R| \leq \mathcal{O}(n^{-\alpha})$ for some $\alpha>0$
• Algorithmic convergence: $\|x_k - x^*\| \leq \mathcal{O}(k^{-\beta})$ for some $\beta > 0$
• Sampling bias: $W_2(\mu_n, \pi) \leq \mathcal{O}(n^{-\gamma})$ in Wasserstein distance

These rates explicitly track constants determined by problem parameters, function class complexity, and measures of regularity or curvature—unlike asymptotic statements, which only describe limiting behavior. The framework allows for sharp comparisons between algorithms and for the rigorous design of step sizes, model classes, or architectures to optimize performance for realistic sample sizes.

2. Statistical Estimation and Learning: Plug-in and Optimization Rates

In statistical estimation, non-asymptotic rates depend on properties such as function class entropy (covering number exponent $\gamma$), tail behavior (e.g., sub-Weibull or polynomial decay), and moment or regularity assumptions. For plug-in estimators of law-invariant risk measures (such as AVaR, OCE, or shortfall risk), sharp non-asymptotic expectation and deviation bounds of the form

$$\mathbb{E}[|T^\infty(F) - T_N^\infty(F)|] \leq \frac{C}{\sqrt{N}},\quad \mathbb{P}(|T^\infty(F) - T_N^\infty(F)| \geq \epsilon) \leq C\,e^{-c N \epsilon^2}$$

can be obtained independently of the dimension of $F$, and hold even when hedging is included as an optimization over a potentially large set of portfolios (Bartl et al., 2020).

For optimal transport map estimation, non-asymptotic rates bound the $L^2(P)$ error between the empirical and true OT map as a function of sample size and function class complexity: $$\|\nabla \hat\varphi_{n,N} - \nabla \varphi_0\|_{L^2(P)}^2 \lesssim S(\bar{\varphi}) - S(\varphi_0) + \tilde{n}^{-1/\gamma} + \tilde{N}^{-1/\gamma}$$ where $\gamma$ is the covering number exponent of the function class and the bound improves to essentially $n^{-2/(\gamma+2)}$ for Donsker classes (Ding et al., 11 Dec 2024). These bounds are often minimax optimal under the basic moment assumptions permitted by the underlying theory (e.g., Brenier’s Theorem for OT).

In nonparametric density estimation with GANs, optimal convergence rates (up to logarithmic factors) can be achieved: $$\mathrm{JS}(p_{\hat{w}}, p^*) \lesssim \left(\frac{\log n}{n}\right)^{2\beta/(2\beta+d)} + \frac{\log(1/\delta)}{n}$$ where $\beta$ is the smoothness of the class and $d$ is the ambient dimension (Puchkin et al., 2021).

For contrastive divergence estimators in exponential families, under regularity, the non-asymptotic parametric rate is attainable: $$\mathbb{E}[\|\bar{\psi}_n - \psi^*\|^2 ]^{1/2} \leq 2 \sqrt{ \frac{ \mathrm{tr}\,( \mathcal{I}(\psi^*)^{-1} ) } { n } } + o(n^{-1/2})$$ with constants controlled by the Fisher information (Glaser et al., 15 Oct 2025).

3. Optimization Algorithms: Finite-Time and Superlinear Guarantees

Non-asymptotic convergence rates have become a foundational aspect of the theory of stochastic and deterministic optimization:

• Stochastic Proximal Point (SPP): Under weak linear regularity (a relaxed quadratic growth condition),

$\mathbb{E}[\|x^k - x^*\|^2] \leq \mathcal{O}(1/k)$

and the rate improves to linear (geometric) convergence under an interpolation condition (Patrascu, 2019).

• SGD and Variants: In the streaming/minibatch setting,

$$\mathbb{E}[\|\theta_t - \theta^*\|^2] \leq \mathcal{O}(N_t^{-\alpha})$$

for appropriate choices of learning rate and mini-batch scaling, and the averaged iterate can achieve

$$(\mathbb{E}[ \|\bar{\theta}_t - \theta^*\|^2 ])^{1/2} \leq \Lambda^{1/2} / N_t^{1/2}$$

matching Cramér-Rao asymptotic efficiency (Godichon-Baggioni et al., 2021).

• Adaptive Methods (AdaGrad, etc.): Explicitly parameter-dependent non-asymptotic bounds for deterministic and stochastic convex/quasiconvex problems, e.g.,

$$\frac{1}{T}\sum_{t=1}^T (F(x_t) - F^*) \leq \frac{C\,b_T}{T}$$

and, in the stochastic non-convex setting, nearly optimal rates in expectation:

$$\frac{1}{T}\sum_{n=1}^T \mathbb{E}[\|\nabla g(\theta_n)\|^2] \leq O\left( \frac{\ln T}{\sqrt{T}} \right )$$

utilizing stopping time-based Lyapunov arguments (Liu et al., 2022, Jin et al., 8 Sep 2024).

• Quasi-Newton Methods and Superlinear Global Rates: Global, non-asymptotic superlinear rates have been rigorously established for regularized (e.g., cubic or gradient regularized) SR1 and BFGS variants—remarkably under minimal (often no strong convexity) assumptions:

$$\|F'(x_N)\| \leq \left( \frac{C}{N^{1/2}} \right)^{N/2} \|F'(x_0)\|,\quad \text{or} \quad \left( \frac{C}{(k - k_0)^{1/2}} \right)^{(k-k_0)/2}$$

These rates are achieved using Lyapunov-type arguments and are valid beyond the convex setting under the Kurdyka–Łojasiewicz property (Wang et al., 15 Oct 2024, Wang et al., 31 May 2025).

4. Non-Asymptotic Rates for Sampling and Stochastic Simulation

For sampling from high-dimensional or non-logconcave targets, precise non-asymptotic error rates are established in Wasserstein distance employing drift conditions that are much weaker than global convexity. For instance, in the absence of log-concavity but under a contractivity-at-infinity assumption: $$W_2(\mathcal{L}(X_k),\pi) \leq (A (1 - c h)^k \mathbb{E}[ f(\lvert X_0 - Y_0 \rvert)])^{1/2} + \tilde{\mu}_2 h^{1/4}$$ where the first term decays exponentially and the second arises from Euler discretization; similar bounds exist in $W_1$ with improved dependence on step size (Majka et al., 2018).

For high-order Langevin Monte Carlo algorithms targeting super-linear potentials, the use of taming and higher-order discretization yields state-of-the-art rates: $$W_1( \mathcal{L}( \theta_n), \pi_\beta ) \leq C_1 e^{-C_0 \lambda n} ( \mathbb{E}[ |\theta_0|^{16(\rho+1)} ] + 1 ) + C_2 \lambda^{1+q/2 }$$ with convergence rates that depend on the Hölder exponent $q$ of the third derivative of the potential (Neufeld et al., 9 May 2024).

In diffusion generative modeling, both deterministic and stochastic samplers exhibit non-asymptotic total variation error bounds as functions of the step count $T$, dimension, and score estimation error, e.g.,

$$\mathrm{TV}(q_1, p_1) \leq C [ (d^2 \log^4 T)/T + \sqrt{d \log^3 T} \, \varepsilon_{\mathrm{score}} + d(\log T) \varepsilon_{\mathrm{Jacobi}} ]$$

and can be accelerated to reach $1/T^2$ or $1/T$ rates with additional correction terms (Li et al., 2023).

5. Mean-Field Games and Non-Asymptotic Large-System Limits

In high-dimensional game-theoretic problems, non-asymptotic convergence rates describe the quantitative error between the value functions and Nash equilibria of finite-$N$ mean-field games and their limiting (McKean–Vlasov) formulations: $$|V^{i,N} - V^{\hat\xi}|^2 \leq C \left( \frac{1}{N} + N R^2_N + \gamma_N \right)$$ with explicit error contributions from empirical measure convergence, control interaction errors, and Wasserstein metric deviations; these bounds hold under mild regularity and dissipativity at infinity (Possamaï et al., 2021).

6. Central Limit Theorem and Inference with Non-Asymptotic Guarantees

Non-asymptotic variants of the classical central limit theorem have been established for both SGD and Markov chain functionals:

• For Polyak–Ruppert averages of SGD, using martingale CLT and Stein’s method,

$$| \mathbb{E}[ h( \sqrt{t} \, \bar{\Delta}_t ) ] - \mathbb{E}[ h( A^{-1} V^{1/2} Z ) ] | \leq C \frac{d^2}{\sqrt{t}}$$

where $Z$ is standard normal and $h$ is a test function (Anastasiou et al., 2019).

• For Markov chain functionals, via Stein’s method and Poisson’s equation,

$$d_{\mathcal{W}}(U_n, \Sigma^{1/2}_\infty Z) = O( (\ln n)/\sqrt{n} )$$

This underpins valid non-asymptotic inference—especially for reinforcement learning algorithms such as TD-learning (Srikant, 28 Jan 2024).

7. Algorithmic Design and Practical Implications

The explicit, uniform nature of non-asymptotic convergence rates underpins:

• The selection of sample size or iteration count to guarantee prescribed error.
• The structure–complexity trade-off (e.g., in function class for plug-in estimators, generator/discriminator capacity in GANs, or step-size strategies in SGD).
• The design and use of averaging (Polyak–Ruppert) for variance reduction and optimality in stochastic approximation.
• Choice of initialization and regularization (e.g., in quasi-Newton schemes) to balance linear and superlinear phases of convergence (Jin et al., 1 Apr 2024).
• Robustness and fail-safeness in adaptive optimization even under heavy-tailed or non-convex scenarios, via methods such as adaptive stepsizes and stopping-time partitions (Jin et al., 8 Sep 2024).

Non-asymptotic analyses enable practitioners to explicitly quantify how fast finite-sample or finite-iteration outputs approximate their targets, aligning theoretical guarantees with the realities of contemporary data, computation, and problem structure. They bridge the gap between theory and practice, guiding the scalable and reliable deployment of modern statistical and optimization methods across scientific and engineering domains.