Non-Ergodic Convergence Rate Analysis

• Non-Ergodic Convergence Rate is defined as the rate at which the current iterate (rather than the averaged sequence) converges, preserving properties like sparsity and feasibility.
• Recent advances in convex, nonconvex, and stochastic optimization, including methods like ADMM and inertial schemes, achieve O(1/k) or faster rates under practical conditions.
• Analytical techniques such as discrete Lyapunov functions and quadratic recurrences underpin non-ergodic rate proofs, impacting both algorithm design and theoretical ergodic analysis.

A non-ergodic convergence rate characterizes the iteration complexity for the last (or current) iterate of an algorithm, as opposed to the rate for ergodic (averaged) sequences. In contrast to the classical ergodic rate, which typically refers to bounds for running averages of iterates, non-ergodic rates quantify how quickly the actual sequence of iterates produced by optimization or dynamical algorithms approaches a solution (optimality, feasibility, stationarity). Non-ergodic rates have become central in modern convex, non-convex, stochastic, and PDE optimization, as well as analysis of operator-theoretic ergodic theorems without spectral gap.

1. Definition and Distinction: Non-Ergodic versus Ergodic Rates

Given a sequence $\{x_k\}$ generated by an optimization or dynamical process, the ergodic rate bounds the error of the averaged sequence $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$, whereas the non-ergodic rate bounds the error for $x_K$ itself. The non-ergodic rate is typically more relevant in practice, especially in large-scale or online settings where only the final iterate is of interest.

In optimization, ergodic rates trace back to classical convex methods such as ADMM or the augmented Lagrangian method, where $O(1/K)$ sublinear rates are obtained for function value and feasibility in terms of $\bar{x}_K$. Non-ergodic rates had long proved difficult for non-monotone methods (e.g., heavy-ball or inertial schemes) until recent advances established $O(1/k)$ and faster rates directly for $x_k$, preserving structural properties such as sparsity and feasibility that may be lost under averaging (Liu et al., 2016, Sun et al., 2018, Sun et al., 2018).

2. Fundamental Results: Archetypes and Main Theorems

A broad template for non-ergodic convergence appears in both deterministic and stochastic settings. For deterministic convex composite problems

$$\min_x \; F(x) := f(x) + g(x) \quad \text{subject to } A x = b,$$

with $f$ smooth convex, $g$ potentially nonsmooth, and $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$0 full row rank, non-ergodic rates are exemplified by the inexact augmented Lagrangian (IAL) or multi-block ADMM frameworks. For IAL, if the inexactness $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$1 in the primal subproblem is summable and decays as $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$2, then the dual gap, constraint violation, and function value for $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$3 themselves all decay as $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$4 or $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$5 under stronger structure (Liu et al., 2016, Zhang et al., 2023). Specifically, with $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$6, $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$7,

$\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$8

and if $\bar{x}_K = \frac{1}{K} \sum_{k=1}^K x_k$9, $x_K$0 non-ergodic rates follow.

There are analogous results for inertial-type and heavy-ball methods: for PIGD and heavy-ball recurrence

$x_K$1

with constant $x_K$2 and nonincreasing $x_K$3, if $x_K$4 is convex and coercive, the function value gap for $x_K$5 itself admits $x_K$6 decay (Sun et al., 2018, Sun et al., 2018).

3. Algorithmic Frameworks with Non-Ergodic Rates

Several algorithmic families now admit sharp non-ergodic rate guarantees:

Algorithmic Family	Non-Ergodic Rate	Structural Requirements
IAL / ADMM (det. convex)	$x_K$7, $x_K$8	Convexity, possibly strong convexity
PIGD / Heavy-ball	$x_K$9, linear (optimal strong cvx)	Convexity ($O(1/K)$0), Strong cvx (lin)
Inertial primal-dual	$O(1/K)$1	Convex-strong-convex saddle structure
Stochastic ADMM (SVRG / momentum)	$O(1/K)$2 (w/ VR, Nesterov extrap.)	Convexity, finite-sum structure
Delayed/stochastic gradient descent	Linear ($O(1/K)$3), or $O(1/K)$4 (PL)	(PL) or strongly convex smooth
Adam (non-convex, PL)	$O(1/K)$5	PL in function, relaxed hyperparameters
Distributed consensus	$O(1/K)$6	Convexity, block structure

For deterministic convex composite programs with constraints, prediction-correction Lagrangian-based schemes incorporating Nesterov-style acceleration in the prediction and mild correction in the multipliers achieve $O(1/K)$7 or $O(1/K)$8 non-ergodic rates for both primal optimality and feasibility, depending on function structure (Zhang et al., 2023).

For linearly constrained stochastic optimization, stochastic ADMM with variance reduction and Nesterov-type extrapolation achieves the optimal non-ergodic $O(1/K)$9 rate for the last iterate, not just for averaging, matching known lower bounds (Fang et al., 2017).

In time-delay and communication-prone settings, delayed gradient descent with fixed step-size and $\bar{x}_K$0-strongly convex, $\bar{x}_K$1-smooth $\bar{x}_K$2 converges at a non-ergodic linear rate for the sequence $\bar{x}_K$3 for appropriately chosen $\bar{x}_K$4, and under the Polyak-Łojasiewicz condition, $\bar{x}_K$5 decay for $\bar{x}_K$6 (Choi et al., 2023).

4. Analysis Techniques and Proof Structures

Non-ergodic rate proofs utilize discrete Lyapunov (energy) functions that encode both the objective gap and (when present) momentum, feasibility, or primal-dual residuals.

• In IAL-type methods, the main recurrence is

$\bar{x}_K$7

where $\bar{x}_K$8 is the dual gap. ODE-type analysis of this quadratic recursion enables extraction of $\bar{x}_K$9 rates if $O(1/k)$0 (Liu et al., 2016).

• For inertial/accelerated algorithms (heavy-ball, PIGD, Nesterov), a Lyapunov function of the form

$O(1/k)$1

satisfies a sufficient descent and discrete error bound. Discrete analogues of $O(1/k)$2 with quadratic-to-linear coupling yield $O(1/k)$3 or $O(1/k)$4 for $O(1/k)$5 non-ergodically (Sun et al., 2018, Sun et al., 2018, He et al., 2023).

• In stochastic methods, variance reduction (e.g., SVRG in ADMM) and momentum/extrapolation are combined to offset the variance contribution and ensure telescoping of the “energy,” resulting in non-ergodic $O(1/k)$6 bounds (Fang et al., 2017).
• In distributed and coupling-constrained problems, blockwise Lyapunov functions and their monotonic decrease control both optimality and feasibility residuals of the last iterate, giving $O(1/k)$7 rates under only convexity (Qiu et al., 24 Nov 2025).

5. Non-Ergodic Rates in Ergodic Theorems and PDEs

Outside optimization, non-ergodic convergence rates play a role in the analysis of ergodic theorems for operator semigroups and related dynamics.

• In von Neumann’s ergodic theorem, the decay of ergodic averages $O(1/k)$8 toward the projection onto invariant functions is governed non-ergodically by the spectral density of the generator at zero (continuous case) or of the unitary operator at $O(1/k)$9 (discrete); specifically, if the spectral measure scales as $x_k$0 at $x_k$1, then

$x_k$2

for the last average, not just for Cesàro means (Ben-Artzi et al., 2019, Aloisio et al., 2022). Subsequence-dependent (lim inf/lim sup) exponents are possible in the absence of a spectral gap.

• For non-gradient Fokker-Planck equations arising in computational neuroscience, the solution $x_k$3 converges exponentially to the steady state $x_k$4 in the last iterate (not only in time averages), with decay rate dictated by the best constant in a weighted Poincaré inequality: $x_k$5 for all $x_k$6, under suitable structural and boundary assumptions (Carrillo et al., 2016).

6. Broader Implications and Practical Significance

Non-ergodic rates are generally stronger in both theory and practice because:

• They guarantee convergence for the actual sequence as it would be used in computations, ensuring properties such as feasibility, constraints, and structural features are not lost to averaging.
• In many large-scale or non-smooth problems, only the last iterate is tractably accessible.
• Algorithms such as heavy-ball, PIGD, stochastic ADMM, or Adam (in the nonconvex/PL setting) have been recently proven to admit $x_k$7 or even faster rates non-ergodically, often under milder parameter or smoothness assumptions than previously required for ergodic bounds (Sun et al., 2018, Sun et al., 2018, He et al., 2023, Fang et al., 2017).

Empirical investigations on multi-block ADMM, accelerated primal-dual, and distributed consensus methods confirm that the non-ergodic rates predict the practical decay of the last-iterate suboptimality and constraint violation (He et al., 2023, Qiu et al., 24 Nov 2025).

7. Current Directions and Open Questions

Recent lines of work focus on:

• Tightening the gap between ergodic and non-ergodic rates under minimal conditions (e.g., lack of smoothness, presence of inertia, stochastic gradients with delay).
• Extending non-ergodic rate analysis to broader classes (e.g., non-convex objectives under the PL condition, as for Adam).
• Developing sharper recursive inequalities (“quadratic recursion” or Lyapunov-based) that handle inexactness, stochasticity, and block structure.
• Unification of discrete-time non-ergodic rates in optimization with spectral and measure-theoretic rates in ergodic theory and dynamical systems (Aloisio et al., 2022).

Ongoing research continues to generalize these frameworks and relax technical constraints (e.g., on parameter choices, inexactness levels, boundedness), further broadening the applicability of non-ergodic rate theory across mathematical optimization and applied dynamical systems.