ReduNet: White-Box Neural Network Design

• ReduNet is a network built using the maximal coding rate reduction principle to maximize feature coding rate differences, ensuring compact intra-class and diverse inter-class representations.
• It constructs each layer analytically through convex optimization, resulting in explicit white-box layers with residual maps, skip-connections, and efficient convergence.
• Extensions like AR-ReduNet and ESS-ReduNet improve performance with adaptive regularization and dynamic expansion, achieving higher classification accuracy and faster training.

ReduNet is a class of white-box deep neural networks explicitly constructed to maximize discriminative representation through the principle of Maximal Coding Rate Reduction (MCR²). Unlike standard networks trained by cross-entropy that often collapse class features and obscure internal mechanisms, ReduNet builds its architecture and parameters layer-by-layer through convex optimization, with all transformations governed by explicit, interpretable matrix-analytic objectives. The resulting network architecture features residual maps, skip-connections, and (when formulated for shift-invariant data) convolutional structure, all arising natively from information-theoretic coding rate definitions. ReduNet and its extensions—most notably Adaptive Regularized ReduNet (AR-ReduNet) and ESS-ReduNet—serve as prototypes of analytically tractable, interpretable networks for supervised and unsupervised representation learning.

1. The Maximal Coding Rate Reduction Principle

ReduNet is rooted in the principle of Maximal Coding Rate Reduction (MCR²), originally developed by Yu et al. (Chan et al., 2021). For a dataset $X\in\mathbb{R}^{n\times m}$ with $m$ normalized vectors and $k$ class partitions, the MCR² approach seeks a feature map $Z$ that maximizes the difference between the coding rate of the entire data and the sum of rates of the individual classes. Specifically, the coding rate of $Z$ at distortion $\epsilon$ is

$$R(Z, \epsilon) = \frac{1}{2} \log \det \Big( I_n + \frac{n}{m\epsilon^2} Z Z^T \Big),$$

and the per-class coding rate for class $j$ is

$$R^c(Z, \epsilon \mid \Pi_j) = \frac{\mathrm{tr}\,\Pi_j}{2m} \log \det \Big( I_n + \frac{n}{(\mathrm{tr}\,\Pi_j)\epsilon^2} Z\Pi_j Z^T \Big),$$

where $\Pi_j$ is the diagonal indicator of class $j$. The MCR² objective is

$$\Delta R(Z) = R(Z, \epsilon) - \sum_{j=1}^k R^c(Z, \epsilon \mid \Pi_j).$$

Maximizing $\Delta R$ enforces intra-class compactness and inter-class diversity: class features are compressed while different class subspaces are driven apart, resulting in maximally discriminative, high-entropy representations (Chan et al., 2021).

2. ReduNet: White-Box Neural Network Construction

ReduNet constructs each network layer by ascending the gradient of $\Delta R$ with respect to the feature matrix $Z$, using operators that have explicit algebraic form. The update at layer $\ell$ is

$$Z^{(\ell+1)} = \text{Proj}_{\|\cdot\|=1} \Big\{ Z^{(\ell)} + \eta \Big( E^{(\ell)}Z^{(\ell)} - \sum_{j=1}^k C_j^{(\ell)}Z^{(\ell)}\Pi_j \Big) \Big\},$$

where the matrices

$$E^{(\ell)} = \frac{n}{m\epsilon^2} \left( I_n + \frac{n}{m\epsilon^2} \Sigma^{(\ell)} \right)^{-1}, \quad C_j^{(\ell)} = \frac{n}{(\mathrm{tr}\,\Pi_j)\epsilon^2} \left( I_n + \frac{n}{(\mathrm{tr}\,\Pi_j)\epsilon^2} \Sigma_j^{(\ell)} \right)^{-1},$$

with $\Sigma^{(\ell)} = \frac{1}{m}Z^{(\ell)}Z^{(\ell)T}$ and $$\Sigma_j^{(\ell)} = \frac{1}{\mathrm{tr}\,\Pi_j}Z^{(\ell)}\Pi_j Z^{(\ell)T}$$. This layered construction yields both “expansion” (amplifying overall feature diversity) and “compression” (projecting features toward lower-dimensional class-specific subspaces). Unlike backpropagation-based networks, all layer transforms in ReduNet are computed in closed form from the data at each step, allowing for full interpretability and efficient “forward” construction (Chan et al., 2021).

Invariance to group actions (e.g., translation or rotation) can be built in by constructing the residual operators as circulant (or more general equivariant) matrices, yielding deep convolutional architectures whose parameters remain tractable and analyzable (Chan et al., 2021).

3. Rate-Distortion Theory and Analytical Approximations

The theoretical core of ReduNet involves the multivariate Gaussian rate-distortion (RD) function. For a zero-mean Gaussian $X\sim\mathcal{N}(0,\Sigma)$, the exact RD function at mean-squared error distortion $D$ is

$$R(D) = \frac{1}{2} \sum_{i=1}^n \max\{0, \log(\lambda_i/d_i)\}, \quad \text{with } D = \sum_i d_i,$$

where $\lambda_i$ are eigenvalues of $\Sigma$. However, the coupling in $\{d_i\}$ renders $R(D)$ analytically inconvenient.

ReduNet circumvents this by introducing tractable surrogates, most commonly

$$R_1(D) = \frac{1}{2}\log\det\left(I_n+\frac{n}{D}\Sigma\right).$$

A recent advance provides a one-parameter family of approximations

$$R_\alpha(D) = \frac{1}{2}\log\det\left(\alpha I_n + \frac{n}{D}\Sigma\right),\qquad \alpha \in [0,1],$$

with per-dimension error sharply bounded in terms of the condition number $\kappa$ of $\Sigma$. The optimal $\alpha^*$ is chosen to match the endpoint $R_{\alpha^*}(\mathrm{tr}\,\Sigma)=0$. As $\kappa\to1$ (well-conditioned), $R_{\alpha^*}(D)$ becomes exact (Huang et al., 23 Jun 2025). This approximation supports efficient, numerically stable layer updates and forms the core of AR-ReduNet.

4. Extensions: Adaptive Regularized and ESS-ReduNet

Adaptive Regularized ReduNet (AR-ReduNet): AR-ReduNet enhances the original formulation by replacing every $I+\frac{n}{D}\Sigma$ by $\alpha I + \frac{n}{D}\Sigma$, with $\alpha$ adaptively solved via bisection at each layer to enforce $R_\alpha(\mathrm{tr}\,\Sigma)=0$. This adaptation decreases approximation error by up to two orders of magnitude (e.g., 98.9% for $n=23$), yields higher downstream classification accuracy (e.g., $92.1\%$ vs $89.8\%$ on MNIST after PCA), and accelerates convergence (200 vs 350 layers for 90% test accuracy) (Huang et al., 23 Jun 2025).

ESS-ReduNet: Recent work identifies an issue in ReduNet where the absence of ground-truth labels in intermediate layers allows poor class separation and slow convergence. ESS-ReduNet addresses this by dynamically amplifying the expansion operator to ensure inter-class separability, incorporates Bayesian inference from labels to correct class assignment confusion, and halts training via monitoring the stabilization of the condition number of each class covariance. This results in 10–20$\times$ faster convergence, and up to 47% improved SVM accuracy on certain datasets relative to ReduNet (Yu et al., 2024).

Model	Key Enhancement	Speedup	Accuracy Gain
ReduNet	Baseline MCR², closed-form layers	–	–
AR-ReduNet	Layerwise $\alpha$ regularization	Up to $1.8\times$	+2–3% on MNIST (PCA)
ESS-ReduNet	Dynamic expansion + Bayes cor.	$10\times$–$20\times$	+37% to +47% (SVM, ESR dataset)

5. Architectural and Computational Features

ReduNet layers can be interpreted as multi-branch residual blocks, each comprising: (1) expansion transforms (amplifying inter-class variance), (2) per-class compression transforms, (3) nonlinearity in the form of class-membership weighting, and (4) normalization projecting features onto the unit sphere. In the presence of shift invariance, all matrix multiplications become cycles of circulant convolutions, and spectral decomposition via FFT enables block-diagonalization for rapid matrix inverses. For an image or signal of length $n$ and $C$ classes, inversion complexity drops from $O(n^3C^3)$ to $O(nC^3)$ (Chan et al., 2021).

Both classification and clustering can be accomplished either by nearest-subspace classifiers on the learned features or by further fine-tuning downstream heads, with the intermediate representations exhibiting orthogonal, well-separated class subspaces and balanced intra-class variance.

6. Empirical Performance and Applications

ReduNet and its variants have been empirically evaluated on both synthetic controlled mixtures (where class separation and cluster geometry can be visualized), and standard benchmark datasets such as MNIST, CIFAR-10, ESR, HAR, Covertype, and Gas.

• With nearest-subspace classifiers, ReduNet attains $\sim$92% test accuracy on CIFAR-10 features, demonstrating robust class separation. On MNIST, AR-ReduNet outperforms the baseline in both accuracy and speed of convergence. Under severe condition number scenarios ($\kappa \gg 1$), the improvements in approximation error and downstream accuracy from AR-ReduNet are more pronounced (Huang et al., 23 Jun 2025).
• ESS-ReduNet delivers up to a $47\%$ increase in SVM accuracy and can reduce the number of layers to reach numerical stability by an order of magnitude (Yu et al., 2024).
• ReduNet with group-invariant architectures (e.g., polar or toroidal circulant lifts) achieves features perfectly invariant to transformations, with subspaces that are linearly separable and orthogonalized regardless of original data symmetry (Chan et al., 2021).
• In self-supervised learning and clustering settings, ReduNet enables state-of-the-art clustering accuracy when augmentations are treated as pseudo-classes and the MCR²-CTRL objective is used (Chan et al., 2021).

7. Significance and Extensions

ReduNet represents a new paradigm in neural network design, where deep architectures are derived from an explicit information-theoretic objective and constructed analytically, rather than heuristically searched via gradient-based parameterization. The transparency (“white-box” nature) of its layers enables precise statistical and geometric interpretation, ease of diagnostic analysis, and straightforward incorporation of prior knowledge or invariance.

A plausible implication is that the techniques developed for ReduNet—specifically, analytical layer construction based on coding rate or information bottleneck ideas, adaptive regularization tied to condition number, and integration of Bayesian correction for unreliable class assignments—may generalize broadly to other structured, interpretable models. Extensions are anticipated for distributed semantic communication, control-based representation learning (e.g., CTRL), white-box transformer architectures, and diverse applications involving multivariate Gaussian rate-distortion blocks (Huang et al., 23 Jun 2025).

In summary, ReduNet, AR-ReduNet, and ESS-ReduNet offer analytically grounded, interpretable, white-box networks for supervised, unsupervised, and invariant representation learning, fusing information theory and deep learning with strong empirical performance and conceptual transparency (Chan et al., 2021, Huang et al., 23 Jun 2025, Yu et al., 2024).