Sample Complexity Upper Bounds

• The paper outlines key sample complexity upper bounds, demonstrating how minimal sample counts are derived via complexity measures like VC dimension and margin-adapted dimensions.
• Sample complexity upper bounds are defined as functions that guarantee error ε and confidence 1-δ, with formulations tailored to tasks in classification, regression, and reinforcement learning.
• The study emphasizes practical implications in high-dimensional and distribution-specific settings, highlighting phase transitions and efficiency gains in learning algorithms.

Sample complexity upper bounds quantify the minimal number of samples required, as a function of problem parameters, to ensure with high probability that a statistical learning or estimation procedure achieves a desired level of accuracy. These upper bounds are fundamental to modern learning theory, statistical estimation, and information theory, as they set the operational guarantees for algorithms under well-specified data and noise models. Over the last decades, increasingly tight and distribution-specific sample complexity upper bounds have been established for a wide array of tasks—ranging from classification, regression, reinforcement learning, density estimation, and distributional property estimation, to learning in noisy or high-dimensional regimes. This article synthesizes leading paradigms, mathematical frameworks, and representative results for sample complexity upper bounds as supported by contemporary research.

1. Key Principles and Core Definitions

A sample complexity upper bound specifies, for a given learning or estimation problem, an explicit function $m(\epsilon, \delta, \Theta)$ such that with $m \geq m(\cdot)$ samples, a prescribed procedure returns a solution within error $\epsilon$ and confidence $1-\delta$, where $\Theta$ captures relevant problem-specific parameters (dimension, margin, smoothness, etc).

Upper bounds are typically expressed in terms of:

• Information measures (e.g., VC dimension, covering/packing numbers, fat-shattering dimension, margin-adapted dimension).
• Distributional structure: sub-Gaussianity, covariance structure, or noise characteristics.
• Geometric parameters: ambient dimension, signal-to-noise ratio (SNR), or simplex regularity.

It is now standard to compare sample complexity upper bounds with matching lower bounds to assess the tightness and optimality of proposed methods. For many learning problems, the upper bounds are non-asymptotic and are distribution- or instance-dependent.

2. Distribution-Specific Upper Bounds: The Margin-Adapted Paradigm

The "tight" sample complexity characterization for large-margin classification with $\ell_2$ regularization is governed by the "margin-adapted dimension" $k_\gamma$ rather than just ambient dimension $d$ or average squared norm. This concept captures how many principal directions of the data's covariance matrix have variance above the scale set by the required margin $\gamma$.

Key Formulations

Let $D_x$ denote the distribution over $\mathbb{R}^d$ with covariance matrix whose eigenvalues are $\lambda_1 \geq \ldots \geq \lambda_d$. Then, for margin parameter $\gamma > 0$,

$$k_\gamma = \min \left\{ k : \sum_{i=k+1}^d \lambda_i \leq \gamma^2 k \right\}.$$

The minimax sample complexity for large-margin learning then satisfies, up to logarithmic factors,

$$\Omega(k_\gamma) \leq m(\epsilon, \gamma, D) \leq \tilde{O}(k_\gamma / \epsilon^2)$$

where $\tilde{O}(\cdot)$ absorbs polylogarithmic factors (Sabato et al., 2010, Sabato et al., 2012).

For sub-Gaussian distributions with independent coordinates, $k_\gamma$ precisely controls both sample complexity upper and lower bounds. This framework generalizes classical bounds—where using just the dimension $d$ or the average squared norm $E[\|x\|^2]/\gamma^2$ is sub-optimal—for anisotropic, high-dimensional, or low-rank data.

Implications:

• If the spectrum decays rapidly ($\lambda_1 \gg \lambda_2 \gg \ldots$), $k_\gamma \ll d$, leading to sample efficiency.
• For full-rank, isotropic settings, $k_\gamma \sim d$.

This distribution-specific approach is now prevalent in margin-based learning and is extensible to contexts such as active learning, settings with irrelevant features, and comparative studies between $\ell_2$- and $\ell_1$-regularized learners (Sabato et al., 2012).

3. Task-Specific Upper Bounds Across Domains

Sample complexity upper bounds adapt to the statistical and computational constraints of different learning tasks. Select paradigms and results include:

3.1 PAC Learning: Realizable Case

Optimal upper bounds for PAC learning in the realizable setting are

$$m(\epsilon, \delta) = O\left( \frac{1}{\epsilon}(d + \ln(1/\delta)) \right)$$

where $d$ is the VC dimension of the hypothesis class. This matches lower bounds exactly up to constants, resolving historical logarithmic gaps in VC theory (Hanneke, 2015).

3.2 Episodic Reinforcement Learning (RL)

For learning an $\epsilon$-optimal policy in finite episodic MDPs with $|\mathcal{S}|$ states, $|\mathcal{A}|$ actions, horizon $H$, and confidence $1-\delta$: $$\tilde{O}\left( \frac{|\mathcal{S}|^2 |\mathcal{A}| H^2}{\epsilon^2}\ln\frac{1}{\delta} \right)$$ episodes suffice for PAC guarantee. The bound is tight (matching a lower bound up to a factor of $|\mathcal{S}|$) and is achieved via improved variance-based concentration; it reduces the horizon dependence from $H^3$ (prior results) to $H^2$ (Dann et al., 2015).

3.3 Population Recovery under Lossy/Noisy Channels

For recovering a population vector from $n$ incomplete or corrupted binary samples:

• Lossy model: For erasure probability $\epsilon$, the minimax sample complexity is

$$\tilde{\Theta}\left( \delta^{-2 \max\left( \frac{\epsilon}{1-\epsilon}, 1 \right)} \right)$$

exhibiting a phase transition at $\epsilon=1/2$: parametric rate ($1/n$) below $1/2$, nonparametric above (Polyanskiy et al., 2017).

• Noisy model: Sample complexity depends exponentially on dimension:

$\exp( \Theta(d^{1/3} (\log(1/\delta))^{2/3}) )$

The minimax-optimal estimators are derived via linear programming and are statistically optimal up to polylogarithmic factors.

3.4 Distribution Learning: Gaussian Mixtures and Log-Concave Densities

Learning mixtures of $k$ $d$-dimensional Gaussians to total variation error $\epsilon$ requires

$$\tilde{\Theta}\left( \frac{k d^2}{\epsilon^2} \right)$$

samples for general mixtures, and $\tilde{O}(k d/\epsilon^2)$ for axis-aligned mixtures (Ashtiani et al., 2017). The upper bounds are realized using robust sample compression schemes, providing nearly tight rates.

For log-concave densities in $\mathbb{R}^d$, the maximum likelihood estimator satisfies

$\tilde{O}_d\left( (1/\epsilon)^{(d+3)/2} \right)$

samples for squared Hellinger error $\epsilon$, which matches information-theoretic lower bounds up to an $\tilde{O}(1/\epsilon)$ factor (Carpenter et al., 2018).

3.5 Recurrent Neural Networks (RNNs)

For real-valued RNNs with $a$ units, input length $b$, and error $\epsilon$: $\tilde{O}\left( \frac{a^4 b}{\epsilon^2} \right)$ samples are sufficient for uniform convergence (Akpinar et al., 2019). For size-adaptive RNNs on $n$-node graphs, this yields $\tilde{O}(n^6/\epsilon^2)$, which is polynomial despite the problem's exponential instance set.

4. Analytical and Methodological Techniques

Sample complexity upper bounds are derived via several central mechanisms:

• Complexity Measures: VC dimension, fat-shattering dimension, covering/packing numbers, margin-adapted dimension, etc., are used to relate empirical and true risks, and to exploit problem-specific structure (Musayeva, 2020).
• Concentration Inequalities: Bernstein’s, Hoeffding’s, and more advanced martingale inequalities (e.g., block martingale small-ball conditions) are applied to control deviations, especially in RL and system identification settings (Dann et al., 2015, Chatzikiriakos et al., 17 Sep 2024).
• Information-Theoretic Arguments: Covering, packing, and KL-divergence-based data-processing inequalities underpin many lower and upper bounds, as well as design of distribution-specific minimax strategies (Guo et al., 2019, Saberi et al., 11 Jun 2025).
• Algorithmic Innovations: Sample compression schemes (for robust density estimation) and specialized estimators, such as weak Schur sampling in quantum trace estimation, yield dimension-independent bounds (Ashtiani et al., 2017, Chen et al., 14 May 2025).
• Adaptive and Data-Driven Methods: Instance-dependent bounds and procedures, such as data-adaptive influence maximization (Sadeh et al., 2019) and Iterative-Insertion-Ranking for exact ranking (Ren et al., 2019), allow tighter bounds based on local instance properties.

5. Impact, Applications, and Theoretical Significance

Sample complexity upper bounds delineate achievable rates for fundamental learning and estimation tasks, clarify tradeoffs between statistical efficiency, computation, and model structure, and inform design of efficient algorithms for modern data-analytic settings:

• Discriminative vs. Generative Learning: Comparison of sample complexities for large-margin (discriminative) and generative approaches makes explicit quantitative gaps under distributional assumptions (Sabato et al., 2012).
• Complexity of Neural Function Classes: Reveals that RNNs and deep architectures can be learned with polynomial sample size despite enormous combinatorial input spaces (Akpinar et al., 2019).
• Fundamental Barriers in High Dimensions: Precise dependence of sample complexity for learning simplices, log-concave densities, or quantum state functionals demonstrates both where “curse of dimensionality” can be avoided and where it remains inevitable (Saberi et al., 11 Jun 2025, Carpenter et al., 2018, Chen et al., 14 May 2025).
• Constrained and Structured Estimation: The gap between unconstrained and strictly-constrained reinforcement learning is sharply captured through explicit dependence on feasibility and slack parameters (Vaswani et al., 2022).
• Minimax Optimality and Statistical-Computational Tradeoffs: Distribution-specific upper bounds provide a unifying language for stating and proving minimax rates, and for exposing residual room for algorithmic improvement.

6. Recent Developments and Open Questions

Recent advances include:

• Non-asymptotic, dimension-free bounds for quantum property estimation via non-plug-in estimators (Chen et al., 14 May 2025).
• Nearly tight upper/lower bounds for system identification without stability assumptions (Chatzikiriakos et al., 17 Sep 2024).
• Sharp characterization of phase transitions (e.g., in population recovery as erasure probability crosses $1/2$) (Polyanskiy et al., 2017).
• Exploiting local versus global metric covers for sample efficiency under differential privacy (Aden-Ali et al., 2020).

Notable directions for future work include:

• Refinement of logarithmic or constant factors in sample complexity expressions.
• Extension of compression and covering arguments to new distribution classes and regimes.
• Understanding computational complexity lower bounds matching sharp statistical upper bounds—especially in high-dimensional, noisy, or quantum data models.
• Achieving distribution-dependent or instance-optimal adaptivity in sample usage, particularly in non-i.i.d. or adversarial settings.

7. Summary Table: Paradigm Results for Sample Complexity Upper Bounds

Domain	Upper Bound (principal term)	Key Parameter(s)
Large-margin ($\ell_2$ reg.)	$\tilde{O}(k_\gamma/\epsilon^2)$	Margin-adapted dim $k_\gamma$
PAC learning (realizable)	$O((d+\ln(1/\delta))/\epsilon)$	VC dim $d$, confidence $\delta$
RL (episodic, PAC)	$$\tilde{O}(|\mathcal{S}|^2|\mathcal{A}|H^2/\epsilon^2)$$	States, actions, horizon $H$
Gaussian mixtures	$\tilde{\Theta}(kd^2/\epsilon^2)$	#components $k$, dimension $d$
Score-matching (deep ReLU)	$\tilde{O}((\sigma^2 d \log n)/(\epsilon^2 n))$	Noise $\sigma^2$, dim $d$, net size
Quantum trace estimation	$\tilde{\Theta}(1/\epsilon^2)$ ($q>2$)	Additive error $\epsilon$, power $q$
High-dim. simplex learning	$\tilde{O}((K^2/\epsilon^2)e^{O(K/\mathrm{SNR}^2)})$	Dim $K$, error $\epsilon$, SNR

This landscape continues to be refined as methodologies advance and new problem regimes are explored. The algebraic, geometric, and information-theoretic characterizations of sample complexity upper bounds remain foundational in both theoretical research and the design of statistically efficient machine learning and inference systems.