--- title: Sample Complexity Bounds url: https://www.emergentmind.com/topics/sample-complexity-bounds-4df81adb-15ba-47f8-b38d-3ed255d57bae type: topic --- # Sample Complexity Bounds Sample complexity bounds quantify the minimal number of data samples required by an algorithm or estimator to achieve a specified level of accuracy and confidence in learning, system identification, estimation, or decision-making under uncertainty. These bounds are foundational in learning theory, statistics, signal processing, control, statistics, and quantum information, providing finite-sample guarantees and characterizing achievable performance. Cutting across classical PAC learning, margin-based classification, convex optimization, dynamical systems, reinforcement learning, quantum measurement learning, and modern high-dimensional models, sample complexity analysis reveals both minimax and instance-dependent rates, clarifies the trade-offs induced by structure (such as invariances, privacy, or model families), and leads to algorithmic designs optimized for statistical efficiency. ## 1. Foundational Definitions and Frameworks Sample complexity bounds have diverse meanings across technical domains but are always anchored to three ingredients: a class of target objects (concepts, distributions, policies, measurements), an allowable error or loss metric (such as misclassification, total variation, prediction error, or regret), and a probability or confidence parameter. - **PAC Learning (Realizable Case):** Given concept class $C$ of VC-dimension $d$, permissible error $\epsilon$, and confidence $1-\delta$, the sample complexity $m(\epsilon,\delta)$ is the smallest $m$ such that a learner $A$ produces $h$ with $\Pr[\mathrm{err}(h)\leq\epsilon] \geq 1-\delta$ for any $c \in C$. The optimal rate is $m(\epsilon,\delta) = \Theta\left( \frac{d+\log(1/\delta)}{\epsilon} \right)$ [1507.00473]. - **Margin-based Classification:** For large-margin (e.g., SVM-type) classification with margin parameter $\gamma$ and distribution $P$, the sample complexity is governed by the *margin-adapted dimension* $d_m(\gamma,P)$, with $m = \widetilde{O}\left((d_m(\gamma,P) + \log(1/\delta)) / \epsilon^2\right)$ [1204.1276]. - **Differential Privacy:** In pure differentially private PAC learning, sample complexity $SCDP(C)$ for class $C$ is tightly characterized by randomized one-way public-coin communication complexity, and lower bounded by the Littlestone dimension: $SCDP(C) = \Omega(LDim(C))$, with possible gaps $SCDP(C) \gg VC(C)$ [1402.6278]. - **Convex Stochastic Programs:** For SAA in convex stochastic programming, new bounds without metric entropy terms reach $N = O(\sigma^2/\epsilon^2)$ under mild regularity and bypass linear dependence on dimension $d$ [2401.00664]. - **Quantum Learning:** Quantum PAC sample complexity for measurement classes depends on the shadow-norm parameter $V_{\mathcal{C}^*}$ and the size of extreme points; $n = O(V_{\mathcal{C}^*} \log |\mathcal{C}^*| / \epsilon^2)$, realizing the first non-linear scaling in the quantum setting [2408.12683]. ## 2. Classical Minimax and Instance-dependent Rates Across learning, optimization, and decision-making, both *worst-case* (minimax) and *instance-dependent* rates arise. - **Best-$k$-Arm Identification:** In multi-armed bandit selection, the minimal expected samples required to identify the top-$k$ arms is governed by *instance-wise* gap-dependent terms ($H^+$ and $H^-$), which can be strictly smaller than the classical sum-of-inverse-gap-squared measure. Nearly instance-optimal algorithms achieve the lower bounds up to doubly-logarithmic factors [1702.03605]. - **Zero-Sum Matrix Games:** Approximate Nash equilibrium identification in $n\times2$ matrix games admits sample complexity lower bounds scaling as $O(\max\{1/\Delta_{\min}^2,\,1/(\epsilon|D|)\}\log(1/\delta))$, where $\Delta_{\min}$ and $D$ are gap parameters of the payoff matrix. For favorable instances, exponentially fewer samples are needed [2303.10565]. - **Reinforcement Learning—Actor-Critic:** In infinite-horizon discounted MDPs, actor-critic methods for $\epsilon$-stationarity achieve sample complexity $O((1-\gamma)^{-2}\epsilon^{-2}\log(1/\epsilon))$ (AC) and $O((1-\gamma)^{-4}\epsilon^{-3}\log(1/\epsilon))$ (NAC), strictly better than policy gradient (PG) and natural policy gradient (NPG), where critic-based variance reduction is essential [2004.12956]. - **Diffusion Model Training:** For high-dimensional deep generative models (e.g., diffusion models), end-to-end sample complexity scales only polylogarithmically in the inverse desired Wasserstein error, achieving $m = {\rm poly}(d) \cdot \operatorname{polylog}(1/\gamma)$, closing a gap versus prior polynomial-in-$1/\gamma$ rates [2311.13745]. - **Constrained MDPs:** For constrained average-reward MDPs, the minimax-optimal sample complexity is $\widetilde{O}(SA(B+H)/(\epsilon^2 \zeta^2))$ under strict feasibility; here $S,A$ are sizes, $B$ bounds transients, $H$ is the span, and $\zeta$ the Slater constant. This reveals a precise statistical penalty for feasibility constraints [2509.16586]. ## 3. Structure-adapted and Distribution-dependent Bounds Many sample complexity bounds leverage distributional or algebraic structure to obtain sharper results. - **Distributional Dependence in Margin-based Learning:** The *margin-adapted dimension* $d_m(\gamma,P)$ exactly interpolates between worst-case (VC-dimension, $\operatorname{Tr}(\Sigma)/\gamma^2$) and favorable anisotropic regimes (effective dimension $\ll d$), yielding tight sample complexity $O(d_m(\gamma,P)/\epsilon^2)$ for large-margin classification under sub-Gaussian product laws [1204.1276]. - **Group Invariances:** For distributions invariant under a group $G$ acting on a manifold $M$, sample complexity for estimating Wasserstein, Sobolev, MMD, and $L^2$/ $L^\infty$ density divergences is reduced by a factor of $|G|$ (for finite $G$) or by the normalized measure of the quotient $M/G$ (for continuous $G$), and convergence rates in $n$ improve to match the "effective dimension" $d = \dim(M/G)$ [2311.02868]. - **Denoising and High-dimensional Estimation:** In learning a simplex from noisy (additive Gaussian) samples in $\mathbb{R}^K$, the sample complexity is $n = \widetilde{O}(K^2/\epsilon^2) \exp(\Omega(K/{\rm SNR}^2))$, but with SNR $\gtrsim \sqrt{K}$, the exponential phase vanishes, matching the noiseless case [2209.05953]. ## 4. Algorithmic and Information-theoretic Techniques Sample complexity bounds are derived using a range of analytical and constructive tools, often leveraging information theory, combinatorial constructions, and algorithmic design. - **Compression Schemes:** For density estimation, the existence of data-dependent robust compression schemes directly yields tight upper bounds for mixtures and products (e.g., learning $k$-mixtures of $d$-dimensional Gaussians in TV requires $\widetilde{\Theta}(k d^2 / \epsilon^2)$ samples in the agnostic setting) [1710.05209]. - **Information-theoretic Lower Bounds:** Change-of-measure (likelihood ratio) arguments, Fano-type constructions, and KL-divergence sensitivity are used to prove lower bounds that often match upper bounds up to constants or logarithms (as in system identification [1903.10343, 2409.11141], margin learning [1204.1276], bandit feedback [1702.03605]). - **Finite-sample Monte Carlo:** For Sequential Monte Carlo (SMC) schemes, the finite-sample complexity is determined by mixing properties, importance weight regularity, and the number of tempering stages, producing bounds within logarithmic factors of Markov chain Monte Carlo for log-concave and high-dimensional targets [1803.09365]. - **First-order and Gradient-based Methods in Bilevel and RL:** Recent advances in bilevel RL circumvent Hessian dependence using the Polyak-Łojasiewicz property and penalty surrogates, attaining $O(\epsilon^{-4})$ rates in continuous state-action domains, improving dramatically over past $O(\epsilon^{-6})$ and higher orders [2503.17644]. ## 5. Separation Results and Limits Situations where privacy, quantum structure, or other constraints fundamentally increase the sample complexity are precisely characterized. - **Private vs. Non-private Learning:** In PAC learning, pure ($\beta = 0$) differential privacy can require exponentially more samples than both non-private or approximate ($\beta>0$) DP; exact separations are constructed using representation dimension and communication complexity, with explicit classes achieving arbitrarily large SCDP/VC gaps [1402.6278]. - **Quantum Learning vs. Classical:** Quantum PAC sample complexity for measurement classes can scale linearly in $|\mathcal{C}|$ in the worst case, versus log-linear in the shadow-norm-exposed extreme points: $n = O(V_{\mathcal{C}^*} \log |\mathcal{C}^*|)$, restoring a near-classical logarithmic dependence under bounded-norm circumstances [2408.12683]. - **Strict-constrained MDPs:** Enforcing zero constraint violation in average-reward CMDPs requires $1/\zeta^2$ more samples compared to allowing $\epsilon$-approximate feasibility, reflecting a statistical hardness not present in unconstrained or discounted/finite-horizon analogues [2509.16586]. ## 6. Parameter Dependence, Tightness, and Open Directions Sample complexity bounds are sensitive to task-specific parameters, variance, model size, smoothness, problem structure, and required accuracy. In many cases, upper and lower bounds match up to logarithmic or constant factors; in others, subtle gaps remain. | Setting | Core Sample Complexity | Key Parameters | Tightness/Separations | |:-------------------------------|:------------------------------|:---------------------------|:-------------------------------------| | PAC-realizable [1507.00473] | $\Theta((d+\log(1/\delta))/\epsilon)$ | VC-dim $d$, error/conf | Tight, improved constant, no $\log(1/\epsilon)$ | | Margin-based [1204.1276] | $\widetilde{O}(d_m(\gamma,P)/\epsilon^2)$ | Margin, covariance | Tight both sides for sub-Gaussians | | Private PAC [1402.6278] | $\Omega(LDim(C))$ (pure DP) | $LDim(C)$ vs $VC(C)$ | Unbounded gap possible, separation pure/approximate DP | | RNN regression [1901.10289] | $\widetilde{O}(a^4 b/\epsilon^2)$ | width, input length | Deterministic upper bound | | Influence maximization [1907.13301] | $O(s \tau \epsilon^{-2} \log(n/\delta))$ | seed size, diffusion steps | Removal of $n$-factor, near-optimal | | CMDP strict feasibility [2509.16586] | $\widetilde{O}(SA(B+H)/(\epsilon^2 \zeta^2))$ | problem size, slack | Minimax tight for CAMDPs | Many open problems persist: refining constants in optimal PAC bounds [1507.00473], closing remaining logarithmic gaps in convex stochastic programming and SAA [2401.00664], extending metric-entropy-free rates to nonconvex or non-Lipschitz settings, generalization to more complex reinforcement learning and stochastic control models, and fully characterizing quantum-classical separations in high-dimensional limit regimes. Source: https://www.emergentmind.com/topics/sample-complexity-bounds-4df81adb-15ba-47f8-b38d-3ed255d57bae