On the Entropy Formula for Real, Complex, and Quaternionic Deep Linear Networks

Abstract: We extend the entropy formula of Menon and Yu for the real Deep Linear Network (DLN) to its complex and quaternionic analogues, obtaining a unified formula for DLNs over $\mathbb{R}$, $\mathbb{C}$, and $\mathbb{H}$.

• The paper derives an explicit Boltzmann entropy formula for DLNs by analyzing balanced factorizations and leveraging random matrix theory.
• It develops a unified framework across ℝ, ℂ, and ℍ, revealing structural parallels to RMT β-ensembles while lacking classical singular-value repulsion.
• The work establishes practical guidelines for entropic regularization in infinite-depth networks and informs future deep learning optimization.

Unified Entropy Formula for Deep Linear Networks over Real, Complex, and Quaternionic Fields

Introduction

This paper develops a comprehensive entropy formula for functions represented by Deep Linear Networks (DLNs) over the three classical division algebras: $\mathbb{R}$, $\mathbb{C}$, and $\mathbb{H}$, extending the work of Menon and Yu for real-valued networks to the complex and quaternionic cases. The analysis leverages geometric and thermodynamic perspectives, connecting DLN structure with random matrix theory (RMT), and rigorously computes the Boltzmann entropy for the class of functions implementable by DLNs of fixed layer depth and width.

Formalism and Entropy Calculation

For a DLN of depth $N \geq 2$ and width $d$, with parameters over a field $F \in \{\mathbb{R}, \mathbb{C}, \mathbb{H}\}$, the model is described by the map

$$\varphi: (M_d(F))^N \to M_d(F),\qquad \varphi(W_N, \ldots, W_1) = W_N \cdots W_1,$$

where $M_d(F)$ is the space of $d \times d$ matrices over $F$ and $\mathbb{C}$0 its invertible subgroup. The paper analyzes the set of "balanced factorizations" of a matrix $\mathbb{C}$1, the locus in parameter space where each layer is balanced (the product of conjugate transposes equates adjacent layers), and computes the entropy as the logarithm of the induced volume.

The principal result is an explicit formula for the Boltzmann entropy:

$\mathbb{C}$2

where $\mathbb{C}$3 is the diagonal matrix of singular values of $\mathbb{C}$4, $\mathbb{C}$5 ($\mathbb{C}$6 for $\mathbb{C}$7, $\mathbb{C}$8 for $\mathbb{C}$9, $\mathbb{H}$0 for $\mathbb{H}$1), $\mathbb{H}$2 is the Haar volume of the corresponding maximal compact group ($\mathbb{H}$3, $\mathbb{H}$4, $\mathbb{H}$5), and $\mathbb{H}$6 denotes the Vandermonde determinant.

Structural and Mathematical Insights

The result exhibits structural parallels to RMT $\mathbb{H}$7-ensembles, with the parameter $\mathbb{H}$8 playing a role analogous to the Dyson index. However, the entropy term in the DLN context uniquely lacks singular-value repulsion, diverging from classical logarithmic repulsion seen in Dyson Brownian motion. The analytic continuation through coinciding singular values is facilitated by polynomial quotient formulas, allowing the entropy to remain regular in the limiting cases.

For complex ($\mathbb{H}$9) and quaternionic ($N \geq 2$0) networks, the generalization introduces entropy contributions from the additional imaginary directions in the underlying Lie algebras and compact groups. The metric computations exploit the real trace pairing and block-diagonalization of the pushforward metric induced on group-orbit directions in balanced factorizations.

Technical Achievements

• Orbit Parametrization: The orbit of balanced factorizations for a fixed $N \geq 2$1 with distinct singular values is shown to be diffeomorphic to $N \geq 2$2, the product of maximal compact groups, up to stabilizers of the singular values. This yields compact, smooth manifold structure for the space of balanced factorizations.
• Induced Metric and Volume: By constructing an explicit real orthonormal basis in the Lie algebras of $N \geq 2$3, $N \geq 2$4, $N \geq 2$5 and using the Riemannian metric induced from ambient Euclidean structure, the paper computes block-tridiagonal determinants whose product (after appropriate normalization) computes the volume of the orbit.
• Infinite Depth Limit: The paper also furnishes an entropy formula in the infinite-depth DLN limit, connecting with recent work on entropic regularization and convergence properties in deep matrix completion (Chen et al., 5 Dec 2025).

Numerical and Analytical Claims

The explicit formulas for entropy and volume demonstrate:

• For networks over $N \geq 2$6, the entropy formula simplifies due to the vanishing of determinant contributions from imaginary diagonal directions.
• In the infinite-depth limit, the renormalized metric yields entropy expressions involving log-differences of singular values, maintaining regularity under collisions.

The paper provides detailed determinant identities for block tridiagonal matrices and characterizes the normalization constants, allowing for direct computation of entropy for any given singular value spectrum.

Implications and Future Directions

The theory here gives a precise geometric and metric foundation for DLNs across real, complex, and quaternionic domains, enabling rigorous analysis of overparameterized learning dynamics and entropic properties of function classes representable by DLNs. The lack of singular-value repulsion at arbitrary $N \geq 2$7 introduces new phenomena compared to classical RMT, suggesting avenues for further investigation into associated matrix models and symmetry reductions, particularly in the quaternionic setting.

From a practical perspective, this unified entropy formula offers insights into the measure concentration and information-theoretic landscape of DLN function spaces. The structural identification of the balanced orbit as a compact manifold parametrized by $N \geq 2$8 may inform future work in optimization, deep learning analysis, and non-Euclidean geometry for parameter spaces.

Theoretical extensions could explore stochastic flows on these manifolds, matrix models at general $N \geq 2$9, and applications to entropic regularization in nonlinear deep networks. The lack of a tridiagonal matrix model at arbitrary $d$0 stands as an open problem, ripe for further research.

Conclusion

This work rigorously extends the entropy formula for DLNs to encompass real, complex, and quaternionic architectures, providing a unified and explicit characterization of the entropy as a function of singular values and network depth/width. The approach connects geometric, algebraic, and probabilistic structures, offering both practical tools for deep learning analysis and foundational directions for future mathematical investigation in network geometry and random matrix theory.