Sparsity-Dependent Capacity Function

• Sparsity-dependent capacity functions define a mapping from a sparsity parameter to operational limits such as communication rate, storage, or recovery probability across various domains.
• The formulations span sensor networks, polar-based coding, neural encoding, and deep networks, illustrating how threshold conditions and finite-blocklength effects modulate performance.
• These capacity laws reveal that structural conditions, phase transitions, and sparsity constraints jointly influence practical metrics like error exponents, effective rank, and representational capacity.

The expression sparsity-dependent capacity function is used in several technical literatures for a mapping from a sparsity parameter to an operational notion of capacity, such as achievable communication rate, sensing capacity, storage capacity, recoverability, or effective representational capacity. Across these uses, sparsity may refer to support fraction, generator-matrix column weight, number of active latent variables, fraction of nonzero synapses, or active feature density; capacity may refer to Shannon rate, dimensions per measurement, number of storable associations, block-error exponent, or effective rank (0704.3434, Pang et al., 2023, Baldi et al., 2021).

1. Scope and canonical forms

A recurring structure is that sparsity enters through an explicit scalar parameter and capacity is then expressed either as a direct function of that parameter or through a phase-transition condition. In some settings the function is monotone decreasing, as in fixed-SNR sensing of very sparse phenomena; in others it is monotone increasing, as in storage of sparse target codes; and in still others capacity is asymptotically flat in sparsity while finite-blocklength reliability remains sparsity-sensitive. This suggests that the term denotes a family of domain-specific capacity maps rather than a single universal formula.

Setting	Sparsity parameter	Capacity object
Sensor networks	$s=\alpha$	$C(d_0)$, dimensions per observation
Polar-based coding	$s,\lambda$	Capacity-achieving under GM column-weight bounds
Sparse neural encoding	$q$	Storage threshold in $K$
Continual learning	$\xi$	Effective-rank capacity per task

In the sensor-network formulation, sensing capacity is the maximum number of signal dimensions reliably identified per sensor observation under an average distortion constraint $d_0$, and upper bounds depend explicitly on the sparsity fraction $s=\alpha$ (0704.3434). In sparse neural encoding, the sharp threshold is given by

$2Kq\log(K/n)\approx n,$

so the critical number of storable associations grows as target activity $q$ decreases (Baldi et al., 2021). In polar-based coding, sparsity is imposed through sublinear generator-matrix column weights such as $C(d_0)$0 or $C(d_0)$1, and the capacity question becomes whether such sparsity constraints are compatible with capacity achievement (Pang et al., 2023). In continual learning, a capacity function is formulated through task-level effective rank, with sparser inputs empirically using more effective directions (Wasilewski et al., 18 Jun 2026).

2. Communication, sensing, and physical-channel formulations

In fixed-SNR sensor networks, sensing capacity is defined as

$C(d_0)$2

and sparsity is parameterized by $C(d_0)$3. For binary sparse signals with Hamming distortion,

$C(d_0)$4

while for a continuous sparse mixture,

$C(d_0)$5

The main asymptotic conclusion is that under the fixed-SNR linear observation model, $C(d_0)$6, with upper bounds decaying like $C(d_0)$7; the paper interprets this as meaning that disproportionately more sensors are required to monitor very sparse events (0704.3434).

Sparse wideband multipath channels use a different capacity map. Here sparsity is expressed through sublinear delay and Doppler degrees of freedom,

$C(d_0)$8

which induce coherence

$C(d_0)$9

When $s,\lambda$0, the training-based achievable rate satisfies

$s,\lambda$1

First-order optimality holds if and only if $s,\lambda$2, and second-order optimality holds if and only if $s,\lambda$3. In this formulation, sparsity increases coherence and thereby relaxes noncoherent learning requirements; sparse channels are asymptotically coherent, and the requirement of peaky signals can be eliminated or relaxed in sparse environments (0705.2847, 0705.2848).

With partial feedback, sparse wideband channels admit a related capacity law. One-bit feedback per coherence subspace with threshold $s,\lambda$4, $s,\lambda$5, achieves the coherent limited-feedback benchmark under average power, and under instantaneous power the same first-order gain is achieved when $s,\lambda$6. In the training-based noncoherent scheme, multipath sparsity enables the benchmark gain under both average and instantaneous power constraints as long as the channel coherence scales at a sufficiently fast rate with signal space dimensions (0801.3521).

Sparse reciprocal wireless channels lead to still another capacity function, now for secret-key generation. In the wideband regime with white sounding and equal SNRs, the ergodic secret-key capacity obeys the approximation

$s,\lambda$7

Here a sparser channel can achieve a higher ergodic secret key rate than a richer channel can, whereas richer channels achieve larger outage exponents because $s,\lambda$8 decays exponentially in $s,\lambda$9 (Chou et al., 2012).

For sparse superposition codes, the provided material supports a narrower sparsity-rate relation. With section size $q$0 and one nonzero per section, relative sparsity is $q$1, and the SPARC rate can be written as

$q$2

Using the standard AWGN capacity expression, the gap

$q$3

gives a direct sparsity-dependent capacity gap. The same material explicitly notes that the requested spatial-coupling and AMP details are not present in the provided content, so only this schematic rate-capacity relation is supported there (Rush et al., 2020).

3. Coding-theoretic formulations: sparse generators and sparse columns

In coding theory, sparsity-dependent capacity functions often describe whether capacity can still be achieved when code structure is forced to be sparse. For polar-based codes over a BMS channel $q$4, one formulation constrains every generator-matrix column weight to be sublinear: $q$5 For any fixed $q$6, there exist capacity-achieving polar codes under successive-cancellation decoding whose GM column weights are bounded by $q$7, with polarization rate $q$8. To improve the sparsity versus error-rate trade-off, the paper develops DRS for the BEC and ADRS for general BMS channels. The resulting threshold constants are

$q$9

For any $K$0, the BEC construction yields capacity-achieving polar-based codes with all GM column weights bounded from above by $K$1 for any $K$2, error probability $K$3, and decoding complexity $K$4; the analogous BMS construction holds for $K$5 with the same decoding complexity (Pang et al., 2023).

The same literature makes clear that the dependence on sparsity is not always a dependence of asymptotic Shannon capacity itself. For random binary sparse generating matrices over the BSC and BEC, the asymptotic capacity is flat in sparsity under the stated ensembles and decoder conditions: for fixed $K$6, sparse Bernoulli or row-regular generating matrices are capacity-achieving on the BSC, while on the BEC even $K$7 suffices. The sparsity-dependent effect appears instead in finite-blocklength performance and the error exponent. Smaller $K$8 gives worse finite-$K$9 reliability, but as $\xi$0 the performance becomes independent of sparsity under the proved conditions (Kakhaki et al., 2011).

This contrast is central. In the polar-based constructions, sparsity constraints are traded directly against polarization speed, rate loss, and overhead thresholds. In the random sparse-generator results, asymptotic capacity does not depend on sparsity level, while the finite-blocklength regime remains strongly sparsity-sensitive. The phrase sparsity-dependent capacity function therefore covers both threshold-preserving and threshold-distorting behaviors, depending on which asymptotic and operational quantity is being held fixed (Pang et al., 2023, Kakhaki et al., 2011).

4. Storage, recovery, and estimation capacity

In sparse neural encoding, the capacity variable is the number $\xi$1 of input-target associations that can be realized by a threshold map $\xi$2. For Gaussian inputs and Bernoulli-$\xi$3 targets, the main phase transition is

$\xi$4

and

$\xi$5

Thus, up to logarithms, $\xi$6 grows inversely with $\xi$7; the paper states that sparsity in the target layers increases the storage capacity of the map. The same work extends the picture to sub-gaussian inputs, Bernoulli inputs, and sparse Hebbian constructions, with comparable conditions of the form $\xi$8 (Baldi et al., 2021).

A related but distinct capacity law appears in an exactly solvable binary perceptron with sparsity-inducing constraints on synapses. There the relevant quantity is the memory load $\xi$9, and connectivity sparsity is $d_0$0, where $d_0$1. In the dense limit without sparsity-inducing constraints, $d_0$2. Under annealed $d_0$3, $d_0$4 is the optimal sparsity-dependent capacity curve; in the homogeneous case, $d_0$5 gives $d_0$6. Under a gap constraint plus $d_0$7, $d_0$8 gives $d_0$9, only about $s=\alpha$0 below the $s=\alpha$1 benchmark. The paper frames this as nearly the same efficiency being obtainable by eliminating weak connections (Baig et al., 2024).

For sparse recovery via Basis Pursuit, the capacity object is not Shannon rate but the probability that a support of size $s=\alpha$2 is $s=\alpha$3-reconstructible. Let

$s=\alpha$4

where $s=\alpha$5 are the entries of the capacity vector derived from $s=\alpha$6. Then, for any $s=\alpha$7,

$s=\alpha$8

A pairwise variant uses the capacity matrix $s=\alpha$9 and gives, for even $2Kq\log(K/n)\approx n,$0,

$2Kq\log(K/n)\approx n,$1

Here the sparsity-dependent capacity function is explicitly the recovery-probability curve $2Kq\log(K/n)\approx n,$2 (Shtok et al., 2010).

In dependent-data $2Kq\log(K/n)\approx n,$3-penalized estimation, the term is used for effective estimation complexity. With sparsity $2Kq\log(K/n)\approx n,$4, restricted-eigenvalue constant $2Kq\log(K/n)\approx n,$5, dependence-adjusted effective sample size $2Kq\log(K/n)\approx n,$6, and tail function $2Kq\log(K/n)\approx n,$7, the synthesis gives

$2Kq\log(K/n)\approx n,$8

which reduces in exponential short-range regimes to

$2Kq\log(K/n)\approx n,$9

This is another instance in which sparsity-dependent capacity means a complexity law for reliable estimation rather than a communication rate (Alquier et al., 2011).

5. Deep networks, latent models, and effective representational capacity

In static sparsity for deep networks, capacity is operationalized as the number of nonzero parameters after masking: $q$0 and under uniform sparsity,

$q$1

Capacity-preserving width scaling in convolutional networks obeys

$q$2

while depth scaling gives

$q$3

The empirical robustness function at fixed capacity is unimodal: robustness improves as sparsity increases from $q$4 to a moderate level, then declines at extreme sparsity where clean accuracy also drops. The paper’s central claim is that the rapid robustness drop caused by network compression is due to a reduced network capacity rather than sparsity (Timpl et al., 2022).

A more activation-centered capacity model is built from receptive-field activation vectors. For a layer $q$5, with Softmax probabilities $q$6 over filter responses, RFAV entropy is

$q$7

mean entropy is

$q$8

and layer sparsity is

$q$9

The paper introduces targeted sparsity regularization

$C(d_0)$00

and a decorrelation term

$C(d_0)$01

The synthesis then proposes an operational layer-capacity proxy

$C(d_0)$02

and a global capacity

$C(d_0)$03

where $C(d_0)$04 is a correlation-based redundancy factor. In this usage, higher entropy and lower filter redundancy correspond to fuller use of network capacity (Huesmann et al., 2020).

Variational autoencoders supply a latent-capacity interpretation. The latent dimension $C(d_0)$05 is the basic capacity variable, inactive latents have near-zero per-dimension KL divergence, and the aggregate information capacity is

$C(d_0)$06

The synthesis states that as $C(d_0)$07 increases beyond an architecture-dependent active count $C(d_0)$08, additional latent dimensions are pruned by the KL term, and a simple piecewise function supported by the reported findings is

$C(d_0)$09

This formulation treats sparsity as a self-regularizing indicator that capacity has exceeded the intrinsic active latent dimension (Asperti, 2018).

In continual learning, effective rank becomes the capacity variable. With sparsity level $C(d_0)$10, where $C(d_0)$11 is expected active feature fraction, the paper measures task-level capacity by effective rank and reports that it grows with sparsity. A flexible saturating form consistent with the reported trends is

$C(d_0)$12

with $C(d_0)$13. The same study emphasizes that higher feature sparsity induces more superposition yet does not inevitably cause forgetting; when representations remain strong, forgetting can be reduced despite overlap (Wasilewski et al., 18 Jun 2026).

6. Recurrent trade-offs, thresholds, and misconceptions

Taken together, these works suggest two broad regimes. In measurement-limited physical systems, deeper sparsity can reduce per-observation information: fixed-SNR sensing capacity goes to zero as $C(d_0)$14 (0704.3434). In sparse coding, neural storage, and sparse-selection problems, deeper sparsity can raise usable capacity by reducing combinatorial or representational interference: the threshold $C(d_0)$15 implies larger $C(d_0)$16 for smaller $C(d_0)$17, and basis pursuit in high-dimensional asset pricing benefits from very large candidate spaces precisely because it can extract a sparse priced-risk structure (Baldi et al., 2021, Afsharhajari et al., 18 Apr 2026).

This directly addresses a recurring misconception that sparsity and capacity are necessarily competing principles. In high-dimensional asset pricing, the paper distinguishes capacity sparsity from factor sparsity and argues that the two are complements: expanding capacity enables the discovery of factor sparsity. Empirically, with $C(d_0)$18, a critical complexity threshold $C(d_0)$19 is reported, corresponding to $C(d_0)$20 features; beyond this point, nonlinear feature expansions combined with basis pursuit dominate ridgeless benchmarks out of sample, while the selected support remains only about $C(d_0)$21–$C(d_0)$22 active factors (Afsharhajari et al., 18 Apr 2026).

A second misconception is that sparsity alone explains degradation in learning systems. In static sparse deep networks, the reported robustness drop in compressed models is attributed to reduced capacity rather than sparsity per se (Timpl et al., 2022). In continual learning, the reported results nuance the common intuition that more superposition leads to more forgetting: overlap interacts with representation strength and capacity allocation, and higher sparsity can increase effective-rank capacity even while increasing superposition (Wasilewski et al., 18 Jun 2026).

The mathematically important feature of these formulations is that sparsity rarely acts in isolation. Capacity laws are modulated by threshold constants and structural conditions: $C(d_0)$23 and $C(d_0)$24 in sparse wideband coherence scaling (0705.2847); $C(d_0)$25 and $C(d_0)$26 for sparse polar-based codes (Pang et al., 2023); $C(d_0)$27 and support size at most $C(d_0)$28 in sparse basis pursuit for SDF estimation (Afsharhajari et al., 18 Apr 2026); and restricted-eigenvalue or null-space conditions in sparse estimation and recovery (Alquier et al., 2011, Shtok et al., 2010). A sparsity-dependent capacity function is therefore best understood as a structural law that links a sparsity control variable to an operational limit under a specified estimator, channel model, or architecture, rather than as a single invariant quantity shared across fields.