Average Sample Complexity

• Average sample complexity is a measure that averages the sample requirements to achieve an ε-accurate outcome across diverse settings such as reinforcement learning and dynamical systems.
• In reinforcement learning, it captures the sample budget needed for accurate policy or value estimates under models characterized by mixing times, bias spans, or transient behaviors.
• In stochastic optimization and dynamics, it serves as an invariant or empirical estimator quantifying the averaged informational or error bounds over partial observations.

Average sample complexity is not a single universally standardized object; in the literature represented here, it denotes several technically distinct notions that share an averaging principle over observations, samples, or sampled time sets. In reinforcement learning and stochastic optimization, it refers to the number of samples needed to produce an $\varepsilon$-accurate policy, value estimate, or optimizer, typically under a generative model or Sample Average Approximation (SAA) framework. In dynamical systems, it is a cover- or partition-level invariant that averages complexity over subsets of a finite orbit window rather than over the full window alone (Petersen et al., 2015, Huang et al., 25 Sep 2025). A narrower algorithmic usage also appears in generalized Leapfrogging Samplesort, where a sorted sample is used recursively to partition an unsorted block, yielding an average-case recurrence

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$

with $n=s+(2k-1)(s+1)$, and hence $A(n)=O(n\log n)$ for constant $k$ (Albacea, 2018). This suggests that the term is best understood through its local context: statistical learning, stochastic optimization, dynamical complexity, or sample-based partitioning.

1. Terminological scope and mathematical forms

In average-reward Markov decision processes (MDPs), average sample complexity is tied to the task of learning an $\varepsilon$-optimal policy from samples drawn by a generative model. A representative objective is to output $\hat\pi$ such that

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$

with probability at least $1-\delta$, while minimizing the total number of queried transitions (Wang et al., 2023). In constrained and robust variants, the target is modified to include feasibility conditions or worst-case transitions, but the operative quantity remains the sample budget required for a prescribed error tolerance (Wei et al., 20 Sep 2025, Chen et al., 15 May 2025).

In stochastic programming, the analogous role is played by SAA. One replaces an expectation $F(x)=\mathbb E[f(x,\xi)]$ by an empirical objective $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$0, or by a nested empirical estimator in conditional stochastic optimization, and then asks how large $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$1 must be so that an empirical minimizer is $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$2-optimal in expectation or with high probability (Liu et al., 2024, Hu et al., 2019). In this setting, “average” refers to empirical averaging over random samples rather than long-run average reward.

In dynamical systems, by contrast, average sample complexity is itself the invariant. For a topological system $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$3 and open cover $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$4, it is defined by

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$5

while for a measure-preserving system and finite partition $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$6,

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$7

Its companion quantity, intricacy, satisfies

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$8

in both topological and measure-theoretic forms (Petersen et al., 2015).

2. Average-reward MDPs and the emergence of optimal rates

A central modern use of average sample complexity is in average-reward reinforcement learning under a generative model. Early work established a reduction from average-reward MDPs to discounted MDPs under a mixing-time assumption: for every deterministic stationary policy, the induced chain has mixing time at most $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$9. Under this regime, an $n=s+(2k-1)(s+1)$0-optimal deterministic stationary policy can be found using

$n=s+(2k-1)(s+1)$1

oblivious samples per state-action pair, while any algorithm using deterministic oblivious samples requires at least

$n=s+(2k-1)(s+1)$2

in the worst case (Jin et al., 2021).

Subsequent work closed the principal gap for uniformly ergodic average-reward MDPs. Using the minorization parameter

$n=s+(2k-1)(s+1)$3

with $n=s+(2k-1)(s+1)$4 up to universal constants, the optimal policy-learning complexity became

$n=s+(2k-1)(s+1)$5

or equivalently

$n=s+(2k-1)(s+1)$6

thereby matching the literature’s lower bound up to logarithmic factors (Wang et al., 2023).

A different line of work replaced uniform mixing by the span of the optimal bias function,

$n=s+(2k-1)(s+1)$7

and proved, for weakly communicating MDPs,

$n=s+(2k-1)(s+1)$8

samples under a generative model. This is minimax optimal up to logarithmic factors in $n=s+(2k-1)(s+1)$9, $A(n)=O(n\log n)$0, $A(n)=O(n\log n)$1, and $A(n)=O(n\log n)$2, and it is strictly more general than guarantees based on a uniform mixing-time bound because $A(n)=O(n\log n)$3 can remain finite when uniform mixing is not available (Zurek et al., 2023). The multichain extension showed that span alone is insufficient in general average-reward MDPs; a bounded transient-time parameter

$A(n)=O(n\log n)$4

is necessary, leading to the optimal bound

$A(n)=O(n\log n)$5

with a matching minimax lower bound up to logarithmic factors (Zurek et al., 2024).

The plug-in approach, long regarded as the simplest model-based baseline, was subsequently given its first finite-sample analysis. With empirical transitions, optional anchoring, and exact average-reward planning in the estimated model, it achieves the optimal diameter-based and mixing-based rates

$A(n)=O(n\log n)$6

without prior knowledge of $A(n)=O(n\log n)$7 or $A(n)=O(n\log n)$8, and also admits span-based bounds with unavoidable empirical-span dependence (Zurek et al., 2024).

The main average-reward sample-complexity laws established in these works can be organized as follows.

Setting	Complexity law	Structural parameter
Uniformly ergodic AMDP	$A(n)=O(n\log n)$9	$k$0 / minorize
Weakly communicating AMDP	$k$1	$k$2
General multichain AMDP	$k$3	$k$4
Plug-in, communicating or uniformly mixing	$k$5, $k$6	$k$7, $k$8

These results collectively show that the “right” average sample complexity parameter depends on the structural class: uniform ergodicity favors $k$9, weak communication favors $\varepsilon$0, and multichain structure requires $\varepsilon$1 in addition to $\varepsilon$2 (Wang et al., 2023, Zurek et al., 2023, Zurek et al., 2024, Zurek et al., 2024).

3. Constrained, robust, offline, and model-free extensions

Constrained average-reward MDPs introduce long-run feasibility conditions. For a stationary policy $\varepsilon$3,

$\varepsilon$4

and the goal is

$\varepsilon$5

Under a generative model, model-based primal-dual learning yields a sharp separation between relaxed and strict feasibility: $\varepsilon$6 samples for relaxed feasibility and

$\varepsilon$7

for strict feasibility, where

$\varepsilon$8

is the Slater constant. A matching lower bound

$\varepsilon$9

shows that the $\hat\pi$0 penalty is unavoidable when exact feasibility is required (Wei et al., 20 Sep 2025).

Robust and distributionally robust variants alter the transition model rather than adding explicit constraints. For robust average-reward policy evaluation under contamination, total variation, and Wasserstein uncertainty sets, the robust Bellman operator is a contraction under the span seminorm,

$\hat\pi$1

and truncated Multi-Level Monte Carlo (MLMC) yields the first finite-sample guarantee: $\hat\pi$2 for both robust value estimation and robust average reward estimation, with explicit dependence on $\hat\pi$3, $\hat\pi$4, $\hat\pi$5, $\hat\pi$6, and the uncertainty radius $\hat\pi$7 (Xu et al., 24 Feb 2025). For distributionally robust average-reward reinforcement learning under KL and $\hat\pi$8-divergence uncertainty sets, two algorithms—one based on reduction to a robust discounted MDP and one based on anchoring—both achieve

$\hat\pi$9

under uniform ergodicity and sufficiently small uncertainty radius, providing the first finite-sample convergence guarantee for this setting (Chen et al., 15 May 2025).

Offline average-reward RL introduces nonuniform coverage effects. In weakly communicating MDPs, a single-policy guarantee is possible in terms of the target policy’s bias span and policy hitting radius. If $0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$0 is deterministic and unichain and the tabular offline data satisfy

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$1

then with high probability the returned policy obeys

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$2

The additive transient-coverage term is essential; stationary-distribution coverage alone is insufficient (Zurek et al., 26 Jun 2025).

Model-free average-reward Q-learning remains statistically less efficient than the best model-based rates, but recent work gives explicit guarantees. Under weakly communicating MDPs, single-agent Q-learning with stage-wise discount scheduling achieves

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$3

while a federated version with $0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$4 agents reduces the per-agent sample complexity to

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$5

and requires only

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$6

communication rounds (Jiao et al., 20 Jan 2026).

4. Sample Average Approximation and stochastic optimization

In stochastic programming, average sample complexity is usually expressed through the sample size required by SAA. For conditional stochastic optimization,

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$7

the nested empirical objective

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$8

is generally biased. Under Lipschitz assumptions, the total sample complexity is

$0\le \bar\alpha-\alpha^{\hat\pi}\le \epsilon$9

it improves to

$1-\delta$0

when the outer function is smooth, and under a Hölderian error bound it further improves to

$1-\delta$1

in the smooth quadratic-growth case. In the special case where $1-\delta$2 and $1-\delta$3 are independent and one uses a modified SAA with shared inner samples, the total complexity becomes

$1-\delta$4

(Hu et al., 2019).

For convex and strongly convex stochastic programming, metric entropy-free analyses have materially changed the status of SAA. In the strongly convex case, if

$1-\delta$5

then any SAA solution satisfies

$1-\delta$6

and the high-probability counterpart introduces only a $1-\delta$7 factor. These rates contain no covering-number or metric-entropy term and therefore match the canonical stochastic mirror descent rate up to minor logarithmic factors (Liu et al., 2024).

When sampling is biased because the random input is itself approximated, the complexity depends on the weak-error order. In biased SAA with

$1-\delta$8

standard Monte Carlo yields total cost

$1-\delta$9

By introducing MLMC into the SAA objective, one obtains

$F(x)=\mathbb E[f(x,\xi)]$0

so MLMC recovers the unbiased Monte Carlo rate when $F(x)=\mathbb E[f(x,\xi)]$1 (Sinha et al., 2024).

Distributionally robust SAA over a $F(x)=\mathbb E[f(x,\xi)]$2-divergence ball exhibits a sharp dichotomy. If

$F(x)=\mathbb E[f(x,\xi)]$3

then a $F(x)=\mathbb E[f(x,\xi)]$4-independent sample complexity exists and can be bounded through a growth-controlled factor $F(x)=\mathbb E[f(x,\xi)]$5; for example, if

$F(x)=\mathbb E[f(x,\xi)]$6

then the truncated SAA estimator attains $F(x)=\mathbb E[f(x,\xi)]$7-accuracy with probability at least $F(x)=\mathbb E[f(x,\xi)]$8. If superlinear growth fails, then every estimator has a $F(x)=\mathbb E[f(x,\xi)]$9-dependent lower bound that can be made arbitrarily large by changing the nominal law $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$00 (Li, 12 Apr 2026).

5. Dynamical-systems average sample complexity

In topological and measure-preserving dynamics, average sample complexity is not a sample-size bound but a dynamical invariant. For a coefficient system $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$01 satisfying

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$02

the topological quantity is

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$03

and the measure-theoretic counterpart is

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$04

The limits exist by subadditivity, and in both settings the associated intricacy satisfies

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$05

Moreover,

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$06

so the supremum over all covers or partitions recovers ordinary entropy (Petersen et al., 2015).

These quantities nonetheless discriminate between systems at fixed cover or partition. For a full $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$07-shift,

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$08

while for certain subshifts of finite type with adjacency matrix $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$09 satisfying $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$10,

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$11

The measure-theoretic theory also yields explicit formulas for Markov shifts, such as

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$12

for a $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$13-step Markov shift with the time-$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$14 partition (Petersen et al., 2015).

The amenable-group extension introduces local and relative versions. For a factor map $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$15, finite cover $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$16, Følner sequence $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$17, and uniform coefficients $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$18,

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$19

and similarly for the measure-theoretic relative quantity with $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$20. The paper proves the equivalence

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$21

for general amenable groups, and establishes a local variational principle: $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$22 In uniquely ergodic $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$23-systems, the topological and the two measure-theoretic local versions coincide for open covers (Huang et al., 25 Sep 2025).

6. Structural parameters, separations, and persistent misconceptions

A recurring theme is that average sample complexity is controlled by structure, not just by ambient dimension or state-action cardinality. In average-reward MDPs, the operative parameter may be the uniform mixing time $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$24, the minorization constant, the optimal bias span $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$25, the diameter $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$26, or the transient time $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$27, depending on the class of MDP being studied (Wang et al., 2023, Zurek et al., 2023, Zurek et al., 2024, Zurek et al., 2024). A common misconception is that span alone always characterizes average-reward difficulty. The multichain lower-bound constructions show otherwise: there exist instances with

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$28

that still require many samples because of long transient behavior, which is precisely why $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$29 appears in the general complexity law (Zurek et al., 2024).

A second misconception is that exact feasibility or robustness merely changes constants. In constrained average-reward MDPs, strict feasibility incurs the unavoidable factor $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$30 absent from relaxed feasibility (Wei et al., 20 Sep 2025). In offline average-reward RL, stationary coverage alone is not enough; the transient-coverage term is necessary (Zurek et al., 26 Jun 2025). In plug-in average-reward planning, a purely $A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$31-based guarantee is impossible for the plug-in template because a lower bound forces additional empirical-span dependence (Zurek et al., 2024). In distributionally robust SAA, distribution-free sample complexity is possible only when the divergence function grows superlinearly; otherwise the lower bound is inherently nominal-distribution-dependent (Li, 12 Apr 2026).

The dynamical-systems literature resolves a different misunderstanding. Since

$A(n)=A(s)+(2k-1)(s+1)\log(s+1)+(s+1)A(2k-1),$32

at the system level, one might conclude that average sample complexity is merely entropy in disguise. The explicit formulas for symbolic systems and the relative cover-level theory show that this is not the case: at fixed cover, partition, or factor, average sample complexity detects structure that ordinary entropy does not isolate (Petersen et al., 2015, Huang et al., 25 Sep 2025).

Taken together, these results indicate that “average sample complexity” is best treated as a family of parameter-sensitive asymptotic laws or invariants. In learning and optimization, it quantifies the sample budget needed to achieve a target accuracy under average-reward, nested-expectation, or robust criteria. In dynamics, it measures the average informational content of sampled orbit subsets. The shared mathematical intuition is averaging over partial observations; the specific object being averaged, and the structural parameter governing it, are determined by the problem class.