Bounded Capacity Theorem

• Bounded Capacity Theorem is a framework that establishes rigorous upper bounds on the capacity of systems via submultiplicative properties and LP relaxations, using tools like the Lovász theta function and Haemers’ rank bounds.
• It demonstrates how feedback in queueing theory and fading conditions in MIMO and wireless channels can sharply alter capacity limits, leading to practical design implications.
• In neural networks and semantic parsing, bounded parameters and memory constraints define finite effective degrees of freedom, influencing approximation capabilities and model expressiveness.

The Bounded Capacity Theorem refers to a collection of rigorous upper bounds on the capacity—broadly construed as the maximal rate of reliable information transfer, approximation, or expressive power—of a range of mathematical and engineering systems under explicit resource constraints. The term encompasses advances in Shannon-theoretic graph capacity, feedback in queueing theory, MIMO and wireless network communication, neural networks with parameter bounds, and syntactic or semantic mechanisms with bounded state or memory. Fundamentally, such results characterize the exact limits imposed by finiteness, boundedness, or submultiplicativity in the system, and often reveal thresholds or sharp separations between finite and unbounded capacity regimes.

1. The Bounded Capacity Theorem in Shannon Graph Theory

The Bounded Capacity Theorem in the context of graph theory strengthens the classic approach to upper bounding Shannon capacity via the independence number, fractional independence number, Lovász theta function θ(G), and Haemers’ minimum rank bound μ_F(G).

Let $G=(V,E)$ be a finite, undirected simple graph. The Shannon capacity $\Theta(G)$ is defined as

$$\Theta(G) = \lim_{n\to\infty} \alpha(G^{\otimes n})^{1/n} = \sup_n \alpha(G^{\otimes n})^{1/n},$$

where $G^{\otimes n}$ is the strong product of $n$ copies of $G$, and $\alpha(\cdot)$ denotes the independence number.

The Bounded Capacity Theorem (Hu et al., 2018) states: Given any function $f$ on graphs satisfying (i) $\alpha(G)\leq f(G)$ for all $G$, and (ii) $f(G\otimes H)\leq f(G)f(H)$ (submultiplicativity), define $f^*(G)$ as the value of the following linear program:

• Primal form:

maximize $\sum_{x\in V(G)} w(x)$ subject to $$\forall S\subseteq V(G): \sum_{x\in S} w(x)\leq f(G_S)$$, $w(x)\geq0$;

• Dual form:

minimize $\sum_{S\subseteq V(G)} q(S)f(G_S)$ subject to $\forall x\in V(G): \sum_{S\ni x} q(S)\geq1$, $q(S)\geq0$.

Then,

$\Theta(G) \leq f^*(G).$

If $f$ is the Lovász theta function, $f^*(G)=\theta(G)$. For Haemers’ minimum rank $\mu_F(G)$, the LP yields a novel μ_F^*(G) that can strictly improve on both θ(G) and μ_F(G), as in concrete examples for various graph classes. The LP relaxation unifies and sharpens previous approaches. Additionally, the technique extends to new bounds on the index-coding broadcast rate (Hu et al., 2018).

2. Queueing Theory: Feedback and Capacity Under Bounded Support

In discrete-time or continuous-time single-server FIFO queues with i.i.d. service times $S$ supported within $[a,b]$ $(0<a<b<\infty)$, the Bounded Capacity Theorem (Sahasranand et al., 2023) establishes a strict separation between the capacities achievable with and without feedback.

Let $C_{noFb}$, $C_{wFb}$, and $C_{fFb}$ denote the capacities without feedback, with weak feedback, and with full feedback, respectively, defined via appropriate single-letter mutual information rates. In the bounded-support regime:

• For any output-rate $X<p=1/\mathbb E[S]$,

$C_{fFb}(X) > C_{noFb}(X).$

That is, full feedback strictly increases capacity whenever service times are supported on a finite interval. This arises from the non-invertibility of the integral operator induced by the compact support, which implies that constrained waiting-time strategies under weak feedback fail to exhaust the entropy maximization available under full feedback. The capacity separation can be numerically demonstrated for truncated exponential and uniform service-time distributions, and persists for both discrete and continuous time (Sahasranand et al., 2023).

3. Multi-Antenna and Wireless Channels: Sublinear and Bounded Capacity

a. Multi-Antenna MIMO Channels

The Bounded Capacity Theorem (Bentosela et al., 2012) addresses MIMO channels constructed from physical scattering models. When the fading matrix $F$ (obtained by Fourier-transforming $HH^\ast$) obeys:

• (A1) Almost-diagonal structure: for all $i$, the off-diagonal entries decay as a bounded multiple of $f_{i,i}$;
• (A2) Power-law decay: $f_i\leq f_1 i^{-r}$ for $r>1$;

then, for any $y>1$,

$C_M \leq A\,M^{1/y} + B\ln M,$

for suitable $A,B$, implying capacity scaling is strictly sub-linear in the number of antennas $M$ (Bentosela et al., 2012).

b. Noncoherent and Interference Networks

For discrete-time, noncoherent multipath fading channels with path variances $\{\alpha_\ell\}$ not decaying faster than geometrically, i.e.,

$$\liminf_{\ell\to\infty} \frac{\alpha_{\ell+1}}{\alpha_\ell} > 0,$$

the capacity $C(SNR)$ is bounded uniformly over all SNR (0711.3152). Similarly, in infinite wireless interference networks with exponentially or slower decaying $\alpha_\ell$ (per-interferer fading variances), the SNR-independent upper bound holds:

$$C(\mathrm{SNR}) \leq \mathcal{O}(1/\rho),\quad \forall \mathrm{SNR}>0,$$

where $\rho$ denotes the geometric decay parameter (Villacrés et al., 2015).

c. Vector Gaussian Channels with Peak and Average Power

In deterministic vector Gaussian (AWGN) channels with identity channel matrix and constraints $\|X\|^2 \leq u_p$, $\mathbb E\|X\|^2\leq u_a$, the Bounded Capacity Theorem (Rassouli et al., 2014) shows that the capacity-achieving input law has a finite set of amplitude values (a discrete support in magnitude), and the capacity is correspondingly bounded. In the high-dimensional/relaxed average-power regime, constant-amplitude signaling is optimal, and the capacity converges to that achieved by Gaussian signaling as the number of antennas grows.

4. Bounded Capacity in Machine Learning and Neural Approximation

For deep feed-forward neural networks with real-analytic activation functions and all nonlinear parameters (weights and biases) bounded within a compact set $S$, the Bounded Capacity Theorem (Liu et al., 2024) establishes:

• For any fixed numerical tolerance $\epsilon>0$, there exists $M=M(\epsilon,g,S,L,n)<\infty$ such that any $L$-layer network of arbitrary width, with activations in $S$, can realize at most $M$ effective degrees of freedom (Numerical Span Dimension) at approximation scale $\epsilon$.
• Thus, neural networks with bounded weights are not universal in the practical, numeric sense—bounded numerical capacity replaces the unbounded approximation property of the classical Universal Approximation Theorem.
• Empirically, as the width increases, the singular-value spectrum of the hidden-layer output matrix saturates rapidly, indicating severe redundancy once the width exceeds $M$.
• Additional theoretical results demonstrate equivalence (at infinite width) between back-propagation networks and random-parameter networks, and clarify why regularization, depth, and parameter condensation phenomena emerge.

5. Capacity Bounds under Adversarial and Stochastic Channel Models

In physical communication channels with continuous-time and delay/noise errors, Khanna & Sudan (Khanna et al., 2011) provide a dichotomy captured by the Bounded Capacity Theorem:

• The capacity is finitely bounded if and only if at least one of the two error mechanisms (delay or noise) is adversarial (with sufficient power, e.g., adversarial delay or noise that knows the delay realization).
• If both delay and noise are purely stochastic (independent), or if noise is adversarial but unaware of delay, the channel admits unbounded capacity through arbitrarily fine time discretization.
• These results extend the classical amplitude-constrained capacity limits by showing that adversarial timing joins noise as an essential mechanism for enforcing finite per-unit-time bit rates.

6. Compositional Semantics, Memory Constraints, and Expressive Capacity

In compositional semantic parsing models, the Bounded Capacity Theorem (Venant et al., 2019) formally relates memory limits and the class of generable semantic structures:

• For any $n$-bounded, projective (adjacent-span combining) semantic mechanism, there exists a sentence of length $2(n+1)$ whose intended cross-serial semantic dependencies cannot be constructed by the mechanism.
• This places a strict upper bound on the expressive capacity of projective, finite-state or bounded-memory models: they fail on semantic structures with unbounded cross-serial dependencies.
• Nonprojective combinators or non-finite memory mechanisms are necessary to represent wider classes of semantic relations. This result clarifies capacity limitations in both grammar-based and neural parsing architectures.

7. Perspectives, Open Problems, and Broader Implications

The bounded capacity paradigm unifies a variety of settings wherein inherent physical or combinatorial constraints induce sharp, quantifiable upper bounds on system capacity:

• In channel coding, such bounds guide practical system design by illuminating which physical effects—multipath, delay, feedback, or adversarial interference—cause capacity saturation or require fundamentally new architectures to overcome.
• For neural function approximation, bounded capacity metrics such as the $\epsilon$-outer measure or NSdim explain empirical bottlenecks and supply a rigorous foundation for regularization and architectural decisions.
• In semantics and formal language theory, bounded-state limits yield rigorous separation results regarding the necessity of unbounded memory or non-projective transformations.

Several technical challenges remain open, for example:

• Developing efficient algorithms for the exponentially-constrained linear programs underlying graph capacity bounds (Hu et al., 2018).
• Extending vector-minrank and LP bounds to directed graphs or capacity-like invariants (Hu et al., 2018).
• Characterizing the gap between finite-parameter and infinite-parameter expressive power in deep architectures quantitatively (Liu et al., 2024).

A plausible implication is that, across communication, computation, and modeling, the search for tight bounded capacity results will remain central wherever scalable reliability, performance, or expressive power are required under real-world constraints.