Big2Small: A Unifying Neural Network Framework for Model Compression

Abstract: With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed \textit{Big2Small}, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. \textit{Big2Small} trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that \textit{Big2Small} achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.

• The paper presents a unified measure-theoretic framework that reinterprets diverse compression techniques as parameter space projections with guaranteed approximation bounds.
• It introduces Big2Small, a data-free, post-training compression method that uses dual-MLP based implicit neural representations to reconstruct weight tensors with high fidelity.
• Empirical evaluations on benchmarks like ImageNet show that Big2Small achieves competitive accuracy and compression ratios while effectively handling high-frequency weight features.

Big2Small: A Unified Mathematical Perspective and Data-Free Framework for Model Compression

Introduction

The exponential growth in deep neural network complexity necessitates increasingly efficient compression methodologies, especially for deployment in resource-constrained environments. Traditionally, the field has approached compression via a set of heuristically driven but largely disparate techniques—low-rank decomposition, quantization, pruning, ergodic dynamic systems, and knowledge distillation. This paper introduces a rigorously unified measure-theoretic framework for model compression and proposes Big2Small, a novel post-training, data-free compression paradigm predicated on Implicit Neural Representations (INRs) (2603.29768). The unification not only relates existing techniques as specific manifestations of a more general principle but also reveals a new avenue for neural network model compression rooted in signal approximation.

Theoretical Unification of Model Compression

A fundamental contribution of the paper is the formalization of model compression as a projection in parameter space, deduced via measure theory and set mappings. Specifically, model compression is interpreted as a mapping $g: \Sigma \rightarrow \Sigma^{\dagger}$ that projects the original parameter set $\Sigma$ onto a structurally reduced set $\Sigma^{\dagger}$, with a quantitatively bounded approximation error and reduced “set size” as characterized by a size functional incorporating Lebesgue measure and point encoding cost.

The Universal Compressibility Theorem (Theorem 1) establishes that the five major compression paradigms are particular realizations of this projection, each characterized by a specialized mapping and associated approximation bound. The formal treatment proves both a strict reduction in set size and a guaranteed capability to approximate the original weights within any prescribed error tolerance for each paradigm. This theoretical abstraction transcends heuristic fragmentation and positions model compression as a principled subdiscipline grounded in mathematical structure.

Figure 1: Theoretical framework of Theorem 1 uniting different compression paradigms via measure-theoretic projections.

Building on this, the Structural Equivalence Theorem (Theorem 2) rigorously demonstrates that every classical compression method is isomorphic to a neural network architecture with specific regularization constraints. In particular, low-rank decomposition corresponds to linear bottlenecks, quantization to networks with discretizing staircase activations, pruning to masked networks, and ergodic systems to RNNs generating parameter sequences. This result reinterprets all compression pipelines as constrained neural function approximation, providing a common language to analyze, compare, and extend them.

Figure 2: Visualization of structural equivalence, mapping classical compression approaches to neural function approximators.

Big2Small: Implicit Neural Networks for Data-Free Compression

Leveraging the equivalence between model compression and neural approximation, the Big2Small framework is instantiated as a system that learns a lightweight INR for each parameter tensor of a trained model. Unlike prior art, Big2Small is data-free, operates post-training, and reconstructs weight tensors at inference via the learned implicit networks. The approach uses a “compression–decompression” pipeline: parameter tensors are encoded by compact INRs (much smaller than the original weights), whose outputs are used to reconstruct the original weights dynamically.

Figure 3: Overview of Big2Small’s compression–decompression architecture using per-layer INRs as compact signal encoders.

Architectural and Methodological Innovations

The Big2Small system advances the state-of-the-art in three main areas:

1. Outlier-Aware Preprocessing: Extreme values—particularly the lowest and highest 1%—are identified and stored explicitly outside the INR representation for each tensor. This stratagem bypasses approximation bottlenecks associated with weight outliers, analogous to selectively lossless encoding.
2. Frequency-Aware Loss: Standard MSE, by itself, underestimates the significance of high-frequency artifacts in weights. The framework introduces a composite objective that adds FFT-based frequency loss and gradient difference penalties, fostering high-fidelity reconstruction at all spatial/scalar frequencies.
3. INR Design: A dual-MLP structure with both positional encoding (Fourier features) and a learnable frequency modulation path is adopted, yielding strong signal representation for both low- and high-frequency components.

   Figure 4: Structure of the INR, showing dual-MLP branches for synthesis and modulation with explicit frequency control.

Empirical Evaluation

Image Classification and Segmentation

Benchmarks on standard datasets such as ImageNet and Carvana image segmentation establish the practical robustness of Big2Small. The framework exhibits competitive or superior accuracy and compression ratios compared to other post-training, data-free compression baselines (e.g., DSG, Squant, UDFC, RieM). Notably, on ResNet-50, Big2Small-8bit achieves a Top-1 ImageNet accuracy of 73.98% at a 7.7× compression ratio.

Figure 5: Visualizations of segmentation results for UNet and compressed models; antennas are preserved under Big2Small, demonstrating high-frequency reconstruction fidelity.

Weight Reconstruction Quality

Big2Small outperforms RieM and other peer encoders (e.g., SIREN, VAE, GAN) under quantitative and qualitative measures of weight tensor reconstruction—including MSE, cosine similarity, and percentile Q-Q plots. Figures directly compare empirical distributions and error behaviors, indicating the superiority of INR-based approximation, especially after frequency-aware training.

Figure 6: Weight reconstruction quality comparison for the convolutional layer of ResNet50; Big2Small closely matches the original distribution.

Figure 7: Reconstruction error as a function of compression ratio. Big2Small achieves lower MAE across all tested regimes.

Ablation and Augmentation Capabilities

Ablation studies substantiate the efficacy of outlier handling and frequency-aware loss components. The framework is shown to be orthogonally combinable with traditional compression techniques, such as additional layer-wise quantization or low-rank approximations. For example, cascading Big2Small with 6-bit quantization achieves a 5.9× compression ratio with only a minor drop in accuracy.

Figure 8: Q–Q plot analysis of different loss functions, highlighting the enhanced statistical fit from frequency-aware objectives.

Inference Efficiency

Inference latency is moderately affected due to online weight reconstruction, with approximately a 30% increase compared to static models. The trade-off is considered acceptable in settings where storage, not real-time inference, is the primary bottleneck.

Figure 9: Throughput comparison between raw and Big2Small models on ImageNet validation.

Encoder Comparisons

Big2Small demonstrates superior compression–fidelity tradeoff compared to SIREN, VAE, and GAN-based alternatives when encoding model weights, as evidenced by MAE and other error metrics.

Figure 10: Encoder comparison on ResNet50 convolutional weights; Big2Small yields the lowest relative MAE.

Implications and Future Directions

The measure-theoretic and structural equivalence perspective presented opens several theoretically and practically valuable avenues:

• Principled Compression Design: The unification permits principled design and systematic analysis of new compression architectures, moving beyond ad hoc heuristics.
• Cross-Paradigm Hybrids: Since all major primitives reduce to constrained neural networks, hybrid or “meta-compression” architectures become more transparent to analyze and optimize.
• Data-Free Model Release: Big2Small and similar paradigms can address data privacy and reproducibility concerns in domains where raw training data cannot be shared.
• Network Architecture Search: Automated tuning of INR structure can enable adaptive compression that self-balances fidelity and storage.
• Extensions to Polynomial/Non-MLP Approximators: The presented theory naturally generalizes to more expressive approximator networks (e.g., polynomial nets), promising even higher compression ratios and fidelity.

Conclusion

This work reframes model compression as a well-posed neural function approximation and projects all classic methods into this schema. The Big2Small framework operationalizes this abstraction into a data-free, post-training compression paradigm that leverages implicit representation learning to encode large-scale models with high accuracy and strong compression. The unified theoretical foundation, architectural innovations, and competitive empirical results lay the groundwork for principled, extensible compression strategies for the next generation of deep models, as well as new research directions in the theory and practice of efficient learning systems.