Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

Abstract: Deep neural networks often contain substantial hidden-state redundancy, but most compression methods operate directly on weights, neurons, or quantised representations without explicitly characterising the dynamical role of internal states. This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks. By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. The resulting A/B/C tests estimate layer-wise reachable, observable, and jointly reachable--observable ranks. These ranks are then used not only as diagnostic measures of hidden-state redundancy, but also as actual compressed layer widths for realised reduced networks. Experiments on MNIST and CIFAR-10 compare the proposed balanced realisation against projection-based reduction, unstructured pruning, structured pruning, low-rank SVD, dynamic INT8 quantisation, and linear baselines. On MNIST, a four-layer SiLU DNN is reduced from state order 1024 to 277, giving 72.95% state compression and 73.48% parameter compression, while maintaining 95.45% accuracy compared with 96.60% for the full model. On CIFAR-10, a larger SiLU DNN is reduced from state order 4608 to 1339, giving 70.94% state compression and 83.09% parameter compression, while preserving accuracy from 54.45% to 54.44% and reducing CUDA inference latency by approximately 3X. The results show that balanced reachable-observable ranks provide a principled empirical minimal-realisation criterion for designing compact neural architectures with little or no loss in accuracy.

• The paper demonstrates a control-theoretic framework using empirical Gramians to identify minimal hidden-state subspaces in DNNs.
• It presents a method that leverages reachability and observability tests to compute layer-wise minimal ranks, resulting in significant compression with minimal accuracy loss.
• Empirical evaluations on MNIST and CIFAR-10 validate aggressive compression ratios and improved latency, demonstrating practical efficiency for DNN deployment.

Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability–Observability Tests

Introduction and Motivation

The paper "Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests" (2607.05457) addresses intrinsic hidden-state redundancy in DNNs by formulating a state-space, control-theoretic compression framework. Conventional approaches—including pruning, quantization, and low-rank factorization—typically operate on weights or features without explicitly confronting the empirical state-space dimensionality that is actually utilized by the data distribution and necessary for downstream tasks. The authors adopt a dynamical-systems perspective, viewing trained feedforward DNNs as depth-indexed nonlinear state-space systems, to estimate the minimal subset of hidden directions that are both data-excitable (reachable) and output-relevant (observable).

Methodological Framework

Nonlinear State-Space View

The key formalism treats a DNN as a discretized, depth-indexed dynamical system:

• Hidden activations $h_\ell$ are the system state at each layer $\ell$.
• State transitions $h_{\ell+1} = f_\ell(h_\ell)$ and final output $z = g(h_L)$ construct a layered, input-driven system analogous to nonlinear state-space models.

Empirical Gramians and A/B/C Tests

For each layer, the method empirically estimates three Gramians:

• Reachability Gramian ($G_{c,\ell}$): Hidden-state covariance over input data, identifying directions strongly excited by data (Test A).
• Observability Gramian ($G_{o,\ell}$): Output-logit Jacobian covariance w.r.t. hidden activations, capturing which directions propagate information to the output (Test B).
• Balanced Gramian ($B_\ell$): Combines reachability and observability (Test C), estimating the minimal jointly reachable–observable subspace using principal component energy thresholds.

Layer-wise "useful" rank is determined by retaining enough principal components to explain $1-\epsilon$ of energy (with $\epsilon=10^{-4}$ in the experiments).

Realised C-Balanced Compression

Moving beyond projection-based analysis, the C-balanced test yields actual layer widths for constructing a new compact architecture. The compressed DNN is trained from scratch or via distillation. This "realised" model directly reduces parameter count and state order, offering both interpretability and practical deployability.

Empirical Evaluation

Datasets and Architectures

Fully-connected SiLU DNNs were studied on MNIST and CIFAR-10:

• MNIST: [256, 256, 256, 256] hidden widths; 1024 total state order; 400,906 parameters.
• CIFAR-10: [2048, 1024, 1024, 512] widths; 4608 total state order; 9,971,210 parameters.

Summary of Numerical Results

MNIST:

Model	State Order	Param.	Acc. (%)	Compression (% state/param)	Latency (ms/sample)
Full	1024	400,906	96.60	0 / 0	0.000427
Realised C-balanced	277	106,323	95.45	72.95 / 73.48	0.000409

CIFAR-10:

Model	State Order	Param.	Acc. (%)	Compression (% state/param)	Latency (ms/sample)
Full	4608	9,971,210	54.45	0 / 0	0.001582
Realised C-balanced	1339	1,685,917	54.44	70.94 / 83.09	0.000525

These show that compression ratios exceeding 70% (state) and 73% (params) are possible on MNIST (with only a 1.2 point accuracy drop), and over 70% (state) and 83% (params) on CIFAR-10, with no loss in accuracy and ~3× lower CUDA latency.

Baseline Comparisons & Observations

• Standard pruning preserves accuracy but does not guarantee hardware efficiency unless widths are reduced; unstructured pruning did not improve latency here.
• Low-rank SVD delivers less effective parameter compression and can degrade accuracy due to lacking a guarantee of dynamical relevance for retained directions.
• Quantization and distillation are complementary but orthogonal to hidden-state order reduction.
• Projection-based C-balanced models diagnose hidden redundancy but incur storage cost; only realised C-balanced architectures yield practical compression.

Theoretical and Practical Implications

The key innovation is formulating and empirically realizing a state-based, rather than parameter-based, compression regime, rooted in classic controllability and observability but adapted to nonlinear, data-driven DNNs. The approach yields new insight:

• Not all high-weight or high-activation directions are needed; only those both reached by data and observable at the output survive balanced tests.
• The empirical minimal-realisation estimation is directly translatable into lean architectures that minimize resource usage with negligible or no accuracy penalty.
• Layer-wise ranks derived from A/B/C tests offer a diagnostic for redundancy—a tool for model design and a target for width-aware pruning.

This paradigm can potentially generalize to CNNs and architectures with residual/skip pathways (defining “state” as pooled feature channels), though further work is needed for token-based and attention modules in transformers.

Limitations and Future Directions

Several limitations are acknowledged:

• Empirical nature: Ranks depend on the sampled data and are not worst-case; distribution shifts may require re-estimation.
• Observability complexity: Output Jacobian computation is tractable for small output spaces but less so for large vocabularies or high-dimensional outputs.
• Retraining requirement: Realised models must be retrained from scratch or via distillation, unlike direct post-training methods.
• Architecture focus: Main results are on fully-connected nets; convolutional/sequence models and hardware-aware rank selection require further research.

Future work should address generalization bounds under nonlinear model reduction, hardware-constrained optimization, and straightforward extensions to convolutional, attention-based, and recurrent networks.

Conclusion

This work demonstrates that a control-theoretic, data-driven approach to model compression—using empirical reachability, observability, and balanced tests—enables principled estimation and realization of minimal hidden-state DNN architectures. The C-balanced ranks translate into practical, width-reduced models that achieve aggressive compression with minimal loss in performance and substantial speedup. This empirical minimal-realisation framework provides a new theoretical and practical tool for efficient DNN deployment and serves as a foundation for future extensions to more complex neural systems.