---
title: Memorization Capacity in Deep ReLU Networks
url: https://www.emergentmind.com/topics/memorization-result-for-deep-relu-networks
type: topic
---

# Memorization Capacity in Deep ReLU Networks

Memorization in deep ReLU networks refers to their capacity to interpolate arbitrary labelings of finite datasets—realizing any mapping from a set of given inputs to specified outputs. This property is a canonical measure of expressivity, distinct from generalization, and has played a central role in recent theoretical analyses of deep learning's empirical success. Recent work has established sharp quantitative bounds on memorization capacity, parameter-count trade-offs, and associated activation complexity, exposing deep links between architectural design, depth–width scaling, bit complexity, and the structure of learned decision boundaries.

## 1. Theoretical Limits of Memorization Capacity

Classical constructions show that arbitrarily labeled data of size $N$ in $\mathbb{R}^{d}$ can be memorized by a fully connected ReLU network with $\mathcal{O}(N)$ parameters; this corresponds to using a single hidden layer with $N$ neurons, each isolating one datapoint [1810.07770]. However, by exploiting increased depth, recent work has established much sharper upper and lower bounds:

- **Tight Bounds (General Position/Separation):** The minimal number of required parameters is $\tilde{\Theta}(\sqrt{N})$, where $\tilde{\Theta}(\cdot)$ suppresses logarithmic factors [2110.03187]. This is achievable under a mild pairwise separation assumption: $\|x_i - x_j\| \geq \delta > 0$ for all $i \neq j$.
- **Width–Depth Trade-off:** For an $L$-layer network, the parameter requirement reduces to $\tilde{O}(N/L)$, optimal up to logarithmic factors. Depth can thus be traded for width.
- **Bit Complexity and Precision:** Achieving these sub-linear parameter counts requires weights of large bit-complexity, packing $\sim\sqrt{N}$ bits into single parameters; with limited precision $B$, the parameter count scales as $\tilde{O}(N/B)$ [2110.03187].
- **Necessary Condition:** These results are tight due to VC-dimension lower bounds, specifically $VCdim=O(W^2)$ for networks with $W$ parameters, meaning memorizing $N$ patterns requires $W = \Omega(\sqrt{N})$.

The constructive method uses projections, interval bucketing, and bit-extraction networks—implementable with layered, narrow ReLU architectures [2010.13363, 2110.03187].

## 2. Depth, Width, and Overparameterization: Fine-Grained Scaling Laws

Explicit constructions and lower bounds clarify how depth and width interplay in determining memorization:

| Model Class                          | Parameter Complexity   | Key Architectural Conditions                  |
|:-------------------------------------|:---------------------:|:---------------------------------------------|
| 2-layer ReLU FNN                     | $\Theta(N)$           | Width $N$                                     |
| 3-layer ReLU FNN                     | $\Theta(\sqrt{N})$    | Width $\sim\sqrt{N}$ per hidden layer         |
| General $L$-layer ReLU               | $\tilde{O}(N/L)$      | Total hidden-layer parameters $W=\Omega(N)$   |
| Narrow, deep (width-2) ReLU [2409.06555] | Depth $2N + 4M - 1$   | Width $2$, $M$ classes, all parameters explicit |

Depth enables parameter efficiency; for any $\Delta$-separated dataset, width-3 ReLU nets suffice with $O(N^{2/3})$ parameters and $O(N^{1/3})$ depth [2010.13363]. The extreme depth–width trade-off is realized in [2409.06555], which provides a constructive scheme achieving memorization with width 2 and depth $2N + 4M - 1$ for $N$ samples and $M$ classes. Conversely, with fixed depth, width must increase: three-layer networks require at least $\Theta(\sqrt{N})$ width to achieve full memorization, and this is both necessary and sufficient [1810.07770, 2110.03187].

## 3. Geometric, Combinatorial, and Architectural Mechanisms

Memorization constructions leverage geometric and combinatorial mechanisms:

- **Random Orthogonalization:** Early layers map sufficiently separated input points to nearly orthogonal hidden representations using random Gaussian weights and sparsifying biases [2001.06938].
- **Bit-Packing and Extraction:** For optimal parameter efficiency, label information and input identity are packed into large-weight parameters, and depth is used for bit-extraction and range folding [2110.03187, 2010.13363].
- **Sparsity via Large Biases:** Sparse activation patterns, achieved by high biases, greatly aid the decorrelation and separability of activations, reducing the sample complexity per hidden connection [2001.06938].
- **Convex Carving via ReLU Dynamics:** Explicit, width-2 constructions use sequential “carving”—using the geometry of ReLU-induced halfspaces to sequentially isolate, collapse, and assign target outputs to each sample [2409.06555].
- **ResNet and General Position:** For data in general position, residual connections allow memorization with $O(N/d_x)$ hidden nodes rather than $O(N)$, via hyperplane carving and class-region separation [1810.07770].
- **Ensemble Controllability View:** Memorizing networks can be interpreted as discrete control systems that synchronously steer each sample’s trajectory through the network to its assigned label [2409.06555].

For random labelings or maximally challenging memorization tasks, these constructions remain near-optimal under known bounds.

## 4. Activation Region Complexity and Expressivity Gap

A key intrinsic limitation of deep ReLU networks is the scaling of their activation-region (or pattern) complexity during memorization:

- **Depth-Independence of Region Count:** For a ReLU FNN with $N_\mathrm{neurons}$ total hidden neurons and input dimension $n_0$, the expected number of activation regions intersecting a fixed input cube satisfies $E[\# \mathrm{regions}] \leq T \cdot (N_\mathrm{neurons})^{n_0}/n_0!$ for $N_\mathrm{neurons} \geq n_0$, with $T\approx1$ for standard random initializations [1906.00904]. Critically, this bound is polynomial in the total number of neurons, not exponential in depth.
- **Empirical Upper Bounds:** Even under full data memorization or random-label training, region counts grow at most by a small constant factor above initialization and never approach the exponential combinatorial bound [1906.00904]. For example, with $N_\mathrm{neurons}=96$, $n_0=2$, observed region counts cluster around theoretical predictions ($\sim 4000$–$5500$ out of a theoretical max of $2^{96}$).
- **Optimization Bias:** Gradient-based training, even for pure memorization, exploits only a polynomially sized subspace of the network's full combinatorial region capacity.

This highlights a sharp “expressivity gap”: deep ReLU nets can, in principle, realize exponentially many activation patterns, but memorization tasks—regardless of label or input structure—remain confined to a much smaller pattern subset in practice.

## 5. Robust Memorization and Parameter Complexity under Perturbations

Memorization under adversarial or robust requirements fundamentally increases parameter complexity:

- **Robustness Ratio and Complexity Scaling:** For a prescribed robustness radius $\mu$ (where $\mu = \rho \epsilon_D$, $\epsilon_D$ being the minimum across-class separation), the required parameter count transitions through three regimes [2510.24643]:
  - For $\rho\leq 1/(5N\sqrt{d})$, $P=\tilde{O}(\sqrt{N})$—matching non-robust memorization bounds.
  - For $1/(5N\sqrt{d})<\rho\leq 1/(5\sqrt{d})$, $P=\tilde{O}(N\,d^{1/4}\,\rho^{1/2})$.
  - For $\rho>1/(5\sqrt{d})$, $P=\tilde{O}(N\, d^2\,\rho^4)$, reaching the dense network regime as $\rho\to 1$.

Robust memorization thus interpolates smoothly from the standard regime to the worst-case $Nd^2$ scaling with increasing robustness requirements.

## 6. Activation Nonlinearity, Internal Phases, and Detection of Memorization

The explicit detection and characterization of memorization in deep ReLU nets can be approached via internal measures of activation nonlinearity:

- **Non-Negative Rank and Memorization:** The non-negative rank of an activation matrix induced by a batch of similar samples measures proximity to linearity; high non-negative rank is strongly indicative of memorization [1810.03372]. Networks generalizing well show increased linearity (low rank) across deep layers, while networks forced to memorize (e.g., random-label training) retain high nonlinearity.
- **Three Distinct Phases:**
  - *Feature Extraction*: Early layers are robust to low-rank compression, capturing generic features.
  - *Memorization*: Intermediate layers exhibit label-noise/proportion-dependent nonlinearity increases.
  - *Clustering*: Late layers focus on bringing like-labeled activations together for final classification.
- **Practical Application**: Tracking the area under the accuracy-vs-rank curve (NMF AuC) over training enables principled early-stopping criteria aligned with the onset of memorization and overfitting.

Such internal diagnostic tools refine the understanding of how, when, and why a network transitions from feature learning to memorization during training.

## 7. Implications for Optimization, Training Dynamics, and Practical Regimes

Analysis of memorization has several concrete implications for training and model selection:

- **Gradient Descent and NTK:** A positive lower bound on the smallest eigenvalue of the neural tangent kernel (NTK) ensures that gradient flow achieves exact memorization for networks with a single wide layer ($n_k=\widetilde{\Omega}(N)$) [2012.11654].
- **SGD Around Global Minima:** SGD, initialized near an exact memorizing solution, rapidly and exponentially reduces empirical risk in the tangent space of memorized gradients [1810.07770].
- **Width–vs–Depth Trade-offs in Practice:** Networks employed in contemporary practice are typically wide and overparameterized, far exceeding the theoretical minimum required for memorization, but optimization dynamics (implicit bias, path smoothness) and data structure favor such architectures for generalization.
- **Design Guidelines:** Minimal overparameterization for memorization is $\#\text{connections} \gtrsim N\, \log^5 N$; architectural sparsity induced via large biases and careful scaling allows orthogonalization and efficient pattern isolation [2001.06938].

These findings collectively reveal that memorization efficiency, activation-space geometry, and training dynamics are tightly interwoven in deep ReLU networks. Depth provides parameter efficiency given sufficient width and precision, but the achievable complexity of learned functions in practice is substantially lower than the maximum combinatorial capacity, fundamentally constrained by optimization and input structure.

---

**References:**  
- "Deep ReLU Networks Have Surprisingly Few Activation Patterns" [1906.00904]  
- "Provable Memorization via Deep Neural Networks using Sub-linear Parameters" [2010.13363]  
- "On the Optimal Memorization Power of ReLU Neural Networks" [2110.03187]  
- "Detecting Memorization in ReLU Networks" [1810.03372]  
- "The Cost of Robustness: Tighter Bounds on Parameter Complexity for Robust Memorization in ReLU Nets" [2510.24643]  
- "Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks" [2012.11654]  
- "Memory capacity of neural networks with threshold and ReLU activations" [2001.06938]  
- "Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity" [1810.07770]  
- "Constructive Universal Approximation and Finite Sample Memorization by Narrow Deep ReLU Networks" [2409.06555]

Source: https://www.emergentmind.com/topics/memorization-result-for-deep-relu-networks