Power-law Sample Complexity Insights

• Power-law sample complexity is defined by polynomial scaling laws relating sample size to estimation accuracy in heavy-tailed data distributions.
• It underpins methodologies across maximum likelihood estimation, network sampling, sparse PCA, and kernel ridge regression, highlighting trade-offs in bias and variance.
• Leveraging tail-biased sampling and curriculum learning can significantly reduce sample requirements in network analysis, high-dimensional statistics, and compositional tasks.

Power-law sample complexity refers to the scaling laws and statistical efficiency associated with learning, estimation, and discovery tasks when the underlying data or model exhibits a power-law distribution. Such laws commonly arise in networks, high-dimensional statistics, kernelized regression, combinatorial learning, and discrete stochastic processes, as well as in the estimation of distributional parameters themselves. The essential feature is that either the data, the underlying process, or the statistical task is governed by rarity of large deviations—leading to non-exponential, often polynomial, relationships between the number of samples, estimation accuracy, and intrinsic parameters such as tail exponents or effective dimensions.

1. Parametric Power-Law Distributions and Exponent Estimation

A fundamental problem is estimating the scaling exponent $\alpha$ of a power-law distribution, typically modeled for continuous variables as

$$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$

with $C = \alpha-1$ for normalization when $\alpha > 1$. Maximum likelihood estimation (MLE) yields an estimator

$$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$

with asymptotic standard error $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$ for large $n$ (D'Huys et al., 2016). The Cramér–Rao lower bound scales accordingly, leading to a required sample size

$n \gtrsim \frac{(\alpha-1)^2}{\epsilon^2}$

for target standard error $\epsilon$. In comparison, graphical estimation methods (e.g., log-log histogram regression) require $10^2$–$$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$0 more samples for comparable precision, and their bias persists at sample sizes where MLE is already accurate. MLE’s reliability is maintained for $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$1 in the moderate exponent regime ($$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$2), with clear practical guidelines emphasizing automatic threshold selection (e.g., via Kolmogorov–Smirnov statistic minimization), reporting of uncertainty, and care in the low-sample regime (D'Huys et al., 2016).

2. Sample Complexity in Network Power Laws: Uniform vs. Tail-Biased Sampling

In network science, estimating degree exponents for power-law degree distributions presents additional challenges due to the scarcity of high-degree nodes in typical samples. Under the continuous power-law model $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$3 for degrees $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$4 and $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$5, ordinary uniform node sampling yields log-likelihood

$$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$6

with Fisher information $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$7, leading to variance lower bound $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$8 for any unbiased estimator. An alternative, exploiting the friendship paradox (sampling edge endpoints instead of nodes), yields an effective exponent shift $$p(x; \alpha) = C x^{-\alpha}, \quad x \geq x_{\min}$$9 in the sampled distribution, producing improved Fisher information $C = \alpha-1$0 and decreasing the sample complexity to

$C = \alpha-1$1

(Nettasinghe et al., 2019). As $C = \alpha-1$2, the sample-size reduction becomes unbounded, and even for moderate $C = \alpha-1$3, a strictly smaller constant factor is achieved. However, this method requires true uniform-edge sampling or long random walks for approximation, and finite-sample bias may become non-negligible when $C = \alpha-1$4 is near 2 or the network is small. Extensions to other parametric families (exponential laws) and to time-varying or directed networks are also possible through analogous sampling strategies.

3. Power Law Effects in High-Dimensional Statistical Learning

In models where signal structure or data eigen-spectrum follows a power law, sample complexity can be dramatically reduced compared to isotropic or “flat” cases.

Sparse Principal Component Analysis: When a spike vector $C = \alpha-1$5 has entries decaying as $C = \alpha-1$6, sample complexity for support/energy recovery—using the Spectral Energy Pursuit (SEP) algorithm—is governed by the structure function $C = \alpha-1$7 and

$C = \alpha-1$8

(Xu et al., 17 Dec 2025). For flat spikes $C = \alpha-1$9, $\alpha > 1$0, while as $\alpha > 1$1, $\alpha > 1$2 is achieved, matching information-theoretic limits. The algorithm is adaptive—no knowledge of $\alpha > 1$3 is required—and one iteration of “truncated power” refinement ensures purely statistical error rates.

Kernel Ridge Regression (KRR): For data with covariance spectrum $\alpha > 1$4, $\alpha > 1$5, and polynomial kernel $\alpha > 1$6, the effective dimension scales as $\alpha > 1$7 for small regularization $\alpha > 1$8. The excess risk decomposes into bias $\alpha > 1$9 and variance $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$0; balancing terms, the optimal rate is $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$1, and to reach accuracy $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$2, the required sample size is

$$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$3

(Wortsman et al., 6 Oct 2025). This power-law scaling replaces the exponential barrier found in isotropic or finite-rank scenarios and highlights the advantage of “soft” spectrum decay.

4. Range-Renewal, Discovery, and Power-Law Complexity

When observing samples from a heavy-tailed discrete distribution $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$4 (with $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$5 slowly varying and $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$6), the number of observed distinct elements $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$7 after $$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$8 draws satisfies a law of large numbers

$$\hat\alpha = 1 + n \left[ \sum_{i=1}^{n} \ln \frac{x_i}{x_{\min}} \right]^{-1}$$9

a.s., where $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$0 encapsulates slow variation (Chen et al., 2013). To observe $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$1 distinct elements, the necessary number of draws scales as $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$2—superlinear in $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$3 for $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$4, reflecting the overwhelming contribution of the rarest symbols. This phenomenon governs discovery tasks, species estimation, and coverage in heavy-tailed settings. The same formalism extends to induced random graphs: degree distributions, small-world properties, and higher-order frequency counts all inherit power-law scaling based on $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$5.

5. Power Law Sampling and Complexity in Neural and Compositional Tasks

In multi-stage or compositional reasoning tasks—such as neural learning of skill compositions, state tracking, or multi-step arithmetic—the empirical data distribution’s tail behavior fundamentally alters sample complexity.

For the $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$6-multiplicative composition problem, uniformly sampling from $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$7 skills yields a provable lower bound: no polynomial-time (or polynomial-sample) gradient-based learner can succeed unless $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$8, due to the flattening of all correlations (Wang et al., 24 Apr 2026). In contrast, if task instances are sampled according to a Zipf (power-law) distribution $\sigma(\hat\alpha) \approx (\hat\alpha-1)/\sqrt{n}$9 ($n$0), then basic stochastic gradient descent recovers the full underlying skill vector in $n$1 samples, a polynomial bound. This “asymmetric” sampling accelerates the learning of high-frequency (head) components, which in turn serve as stepping stones to learn rare (tail) components efficiently—a dynamic observed both in theoretical analysis (PL inequalities, weighted population gradients) and in empirical evaluations (statistical learning on arithmetic, state-tracking, and multi-hop QA).

A key conclusion is that when the true empirical distribution is power law, “curriculum learning” or uniformization may harm rather than help: tail-biased sampling provably unlocks tractable sample complexity for long-tailed compositional tasks.

6. Synthesis: Regimes and Interpretations of Power-Law Sample Complexity

Power-law sample complexity arises from several distinct mechanisms, summarized as follows:

Setting	Scaling Law	Comments
Parametric tail estimation	$n$2	MLE optimal, graphical much worse
FP network sampling	$n$3	Strictly smaller, diverges for $n$4
Sparse PCA (profile-adapted)	$n$5 to $n$6	Interpolates by spike decay
KRR under power-law spectrum	$n$7	Benefit grows with $n$8
Range-renewal processes	$n$9	Sublinear $n \gtrsim \frac{(\alpha-1)^2}{\epsilon^2}$0, superlinear coverage
Compositional learning	Polynomial in $n \gtrsim \frac{(\alpha-1)^2}{\epsilon^2}$1 for power-law, exp. for uniform	Asymmetry critical

The common thread is that heavy tails or power-law behavior—whether in underlying distributions, energy profiles, or data sampling—modulate sample efficiency by concentrating statistical power on the frequent and enabling rare-event estimation via targeted or tail-biased schemes. These effects must be leveraged or accounted for in the design of efficient inference, estimation, or learning pipelines across domains.