Shared Compression Multilayer Perceptron

• The paper demonstrates that a shared compression MLP significantly reduces parameters via randomized pruning and quantization, maintaining performance within a controlled approximation error.
• The methodology employs a layer-wise greedy masking approach with Lindeberg interpolation to cancel first-order error contributions and ensure stable compression.
• The framework extends to structured pruning and convolutional models, offering theoretical guarantees that inspire further improvements in deep network compression.

A Shared Compression Multilayer Perceptron (MLP) refers to a class of neural network models in which substantial parameter reduction is achieved via pruning and quantization without retraining, enabled by a rigorous post-training compression algorithm. This approach is characterized by the application of randomized greedy strategies across the layers of a wide MLP, ensuring that a subnetwork preserves the performance of the original up to a controlled approximation error. The resulting framework not only justifies empirical successes in deep network compression but also establishes theoretical guarantees for both unstructured and structured sparsification, extending further to convolutional architectures (Cheairi et al., 6 Dec 2025).

1. Compression Framework and Algorithmic Principles

Compression in MLPs is formalized as the search for a subnetwork $\hat\Phi$—formed by setting a fraction of weights to zero (pruning) or to quantized values—such that the excess mean squared loss is bounded:

$$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$

where $\Phi$ is the original network. The algorithm operates layer-by-layer in a top–down manner, targeting a large compression ratio (e.g., $\alpha \approx 0.99$ weights per layer to be compressed). At each iteration, an uncompressed weight or block is selected such that its randomized replacement most minimally increases the expected layer-wise loss. For pruning, replacement is by a zero-mean scaled Bernoulli random variable; for quantization, via two-point randomized rounding to the nearest quantization grid points.

A crucial aspect is the projection step: after compression of a layer, layer activations are projected onto an $\ell_2$-ball determined by the product of operator norms, which controls error amplification. The algorithm is a randomized, iterative analogue of the classic Optimal Brain Damage (OBD) method, distinguished by its zero-mean perturbation that cancels first-order Taylor error contributions, obviating the need for weight retraining or zero gradient preconditions.

2. Theoretical Compression Bounds and Conditions

Formally, the compression guarantees are encapsulated through two main bounds:

• Unstructured Pruning: Subject to each activation being 1-Lipschitz and layers satisfying suitable spectral and entrywise norm constraints ($\|W_\ell\| \leq c_1$, $$\|W_\ell\|_\infty \leq c_2/\sqrt{n_\ell \vee n_{\ell+1}}$$), one can select a $p$-sparse subnetwork such that

$$\mathbb{E}\|\Phi(x) - \hat{\Phi}(x)\|^2 \leq c_1^{2m}(1+\xi)^m\,\xi,$$

with excess risk controlled correspondingly. For each wide or bottleneck layer, the number of retained parameters scales as $(0.01+1.01p)\,n_\ell n_{\ell+1}$.

• Unstructured Quantization: By applying $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$0-point rounding, at most $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$1 parameters fall outside of the $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$2-point alphabet, and the same error bound applies under analogous norm constraints.

Network width plays a pivotal role: wide layers ($$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$3) or bottleneck structures (where input and output dimensions differ markedly) enable linear parameter sparsities while explicitly controlling second-order contributions to loss (as quantified by Hessian-saliency proxies depending on $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$4). This provides, for the first time, rigorous proof that overparameterized (wide) networks are fundamentally easier to compress post-training, with no dependence on the data distribution.

3. Proof Techniques—Lindeberg Interpolation and Second-Order Analysis

The error bounds follow from a series of analytical steps combining classical probability ideas and layer-wise error analysis:

• Lindeberg-Style Interpolation: The impact on loss of randomly replacing a weight is bounded by interpolating between the current and perturbed network and considering the second-order Taylor expansion. The expectation eliminates the first-order (gradient) term, leaving only Hessian-induced curvature contributions.
• Hessian Bound Strategies: Quantitative bounds on average Hessian curvature across weights ensure that most directions are "flat," enabling safe aggressive compression.
• Greedy Masking and Telescoping: Repeated compression steps are analyzed via a telescoping sum, with the cumulative excess error scaling linearly in the number compressed, under control of the per-step variance.
• Layer-Wise Error Recursion: Layerwise error is propagated forward using recurrence inequalities. For a wide layer, the $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$5 error in activations after compression at layer $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$6 is upper bounded recursively, and the bounds unroll across $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$7 layers to yield the global result.

These techniques enable an expectation-to-existence argument, extracting a deterministic sparse subnetwork from the random ensemble with identical risk properties.

4. Extension to Structured Pruning and Convolutional Models

The framework extends to structured pruning variants, wherein entire input or output neurons (i.e., entire columns or rows of $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$8) are zeroed. The extension holds with minimal change to the algorithm, and corresponding theoretical bounds are provided for the case when up to a fraction $$\mathbb{E}_x \|\Phi(x) - \hat\Phi(x)\|^2 \leq \epsilon,$$9 of neurons per wide or bottleneck layer are pruned. Analogous arguments apply to convolutional architectures: each circularly padded convolutional layer is cast as a block-circulant matrix, with filter or channel pruning carried out on block-rows or block-columns. The metric and proof structure remain unchanged except for multiplicative kernel-size factors.

5. Practical Implications and Limitations

No assumption on data distribution is made aside from boundedness of covariates, or a mild noisy-labels lower bound if clean relative risk forms are desired. The required spectral and entrywise norm bounds match those satisfied by standard initialization schemes such as Xavier or Kaiming methods after conventional training.

Key practical considerations include:

• Very wide or bottleneck layers are information-theoretically compressible at linear sparsity levels, rigorously explaining the practical success of so-called parameter redundancy and empirically observed ease of pruning as model width increases.
• The randomized construction provides a theoretical rationale for alternative importance-sampling masking strategies, suggesting further algorithmic exploration.
• Limiting factors are the exponential-in-depth dependency of error bounds (via $\Phi$0), the seeming necessity of activation projection steps for theoretical analysis (which can often be omitted in practice), and pessimistic hidden constants in some uniform bounds.
• Worst-case Hessian curvature may be overly conservative compared to the typically “flatter” structure encountered in real data regimes.

A plausible implication is that, while the theoretical framework motivates new algorithms for compression, actual practical tolerance to compression may often exceed the specified pessimistic rates, especially in very deep or extremely wide networks.

6. Unified Theoretical Foundation for Post-Training Compression

The framework constitutes the first unified theoretical justification for post-training compression across unstructured, structured, and convolutional neural network models (Cheairi et al., 6 Dec 2025). By leveraging randomized second-order pruning and quantization, Lindeberg-style interpolation, and error-propagation analysis, the method rigorously grounds the empirical fact that millions of parameters in modern deep networks can be compressed away with little or no accuracy degradation—particularly as the network width increases. The approach is extensible across architectures and compression schemes, closing a longstanding gap between empirical practice and mathematical theory.