Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

Abstract: The massive computational costs of scaling modern deep learning architectures have driven the widespread use of parameter-efficient low-rank structures, such as LoRA and low-rank factorization. However, theoretical guarantees for their expressive power are less explored, often relying on restrictive priors like a pretrained base matrix, ReLU activations or non-verifiable singularity conditions. We first investigate the limits of neural networks constrained strictly to low-rank manifolds without pretrained dense priors. We demonstrate a theoretical paradox: while purely rank-1 layers can exactly interpolate arbitrary scalar datasets, they collapse for function approximations. To overcome this bottleneck without surrendering parameter efficiency, we introduce a unified \textit{Structural Correspondence} framework. We prove that augmenting low-rank layers with only a minimal sparse diagonal component, say a Diagonal plus Low-Rank (DLoR) structure, is sufficient to reach Universal Approximation. We show that any full-rank transformation can be exactly reconstructed using these DLoR components by trading off network width (additive decomposition) or depth (multiplicative decomposition). By tracking asymptotic Taylor remainders, we prove that DLoR neural networks fully restore the Universal Approximation Theorem for general activation functions. Finally, we establish that multiplicative depth provides superior parameter-to-expressivity scaling compared to additive width. Our results show that dense matrices and specific activation functions are not topological prerequisites for universal expressivity.

• The paper demonstrates that adding a diagonal residual to low-rank architectures (DLoR) restores universal approximation even under strict parameter constraints.
• It rigorously characterizes the limitations of pure rank-1 networks and introduces both additive and multiplicative decompositions to overcome affine drift.
• Experimental results validate that deep DLoR networks achieve lower error and better optimization efficiency compared to wide variants, matching dense network performance.

Structural Correspondence and Universal Approximation in Diagonal plus Low-Rank Neural Networks

Motivation and Problem Statement

Scaling deep neural architectures imposes severe computational and memory bottlenecks, leading to widespread adoption of parameter-efficient structures such as LoRA and sparse-plus-low-rank matrix decompositions. Despite empirical successes, foundational theoretical guarantees for universal approximation with strictly low-rank constraints remain scant and often rely on dense priors, specific activation functions, or unverifiable singularity conditions. This paper rigorously formalizes the expressive boundaries of neural networks restricted to low-rank manifolds and introduces the Diagonal plus Low-Rank (DLoR) structure to restore universal expressivity.

Limits of Pure Low-Rank Architectures

The first major theoretical contribution is a precise characterization of the expressivity paradox in pure rank-1 networks:

• Exact Interpolation for Scalar Outputs: It is proven that pure rank-1 networks can interpolate arbitrary scalar datasets under general activation functions by exploiting non-mean-periodic activations (Theorem 1). This replicates results for full-rank networks but without dense matrix requirements.
• Orthogonal Blindness and Collapse in Function Approximation: When target outputs are multi-dimensional, rank-1 networks collapse geometrically, becoming incapable of capturing variations orthogonal to the projection vector. This orthogonal blindness precludes function approximation in spaces of dimension greater than one (Theorem 2).

Figure 1: Rank-1 neural networks interpolate scalar data but fail to capture multidimensional function geometry due to orthogonal blindness.

Figure 2: Illustration of approximation failure for rank-one neural networks; only one direction of variation is captured, resulting in inability to approximate independent dimensions.

Structural Correspondence: DLoR and Approximation Recovery

To overcome expressive bottlenecks, the paper introduces a minimal sparse augmentation: the DLoR structure ($M = \alpha I + UV^T$). This provides a residual pathway, allowing the network to bypass the low-rank bottleneck.

• Additive Decomposition (Width Expansion): Any full-rank matrix can be expressed as a sum of DLoR components. When projected through nonlinearity, this reconstructs the target affine transformation by expanding network width and utilizing parallel ensembles. This formulation eliminates affine drift and does not require bias recalibration due to a zero-sum constraint on decomposition coefficients (Theorem 3).
• Multiplicative Decomposition (Depth Extension): Any invertible matrix can be decomposed into a product of DLoR components. This achieves expressivity by extending depth but maintaining original width, leveraging the locally linear regime of activation functions (Theorem 4).

Figure 1: Width and depth expansion with DLoR structures: additive decomposition increases parallelism, multiplicative exploits sequential composition.

Universal Approximation Theorems for DLoR Networks

Embedding DLoR decompositions into wide or deep architectures restores universal approximation power:

• Universal Approximation via Depth (Deep DLoR): Deep DLoR networks with bounded width ($N = n+m+2$) and arbitrary depth are shown to be dense in $C(\mathcal{K}, \mathbb{R}^m)$ for general continuous, non-affine activations (Theorem 5). This holds for any rank constraint $r \ge 1$.
• Universal Approximation via Width (Wide DLoR): Parallel DLoR expansion with arbitrarily large width is also universally approximative.
• Parameter Efficiency: DLoR networks preserve the optimal parameter complexity for function approximation in Sobolev spaces, matching dense architectures ($\mathcal{O}(\epsilon^{-d/s} \log(1/\epsilon))$). Depth is formally proven to yield exponentially better expressivity scaling than width (see Appendix).

Experimental Validation

Numerical experiments corroborate theoretical results:

• Direct Constructions: Both deep and wide DLoR networks approximate dense MLPs and target functions with error converging to zero as scaling parameter $h \rightarrow 0$. This validates constructive aspects of the theorems.

Figure 3: Comparison of test error for deep/wide DLoR networks and parameter-matched dense MLPs; deep DLoR achieves lower error and better optimization reliability.

• Training Efficiency: Deep DLoR structures consistently outperform wide variants in fixed-budget approximation and early stopping experiments, achieving higher reliability and efficient convergence to prescribed accuracy thresholds.
• Spectral Analysis: Deep DLoR layers retain dominant singular directions (low-rank) and broad spectrum (identity), while wide networks distribute representation across branches without spectral separation.

Practical and Theoretical Implications

The formalism has direct consequences for AI and deep learning:

• Parameter-Efficient Training: DLoR structures justify parameter-efficient pretraining and fine-tuning in PEFT regimes without reliance on pretrained dense priors.
• Architectural Design: The elimination of affine drift via width expansion enables decoupled and robust bias management in distributed settings.
• Hardware Optimization: Structural correspondence allows for tailoring models to parallelism or sequential computation, maximizing utilization and scalability.
• Foundation for MoE Variants: The additive framework aligns with Mixture of Experts-type ensembles, establishing exact static reconstruction without dynamic routing.
• Theoretical Universality: Dense matrices and restrictive activations are not required for universal expressivity; the structural duality holds for general nonlinearities.

Conclusion

The paper rigorously establishes the limits and recovery mechanisms for function approximation in low-rank neural networks. While pure rank-1 networks suffer catastrophic collapse for multidimensional targets, minimal augmentation via DLoR restores universal approximation, supporting robust architectural strategies for parameter-efficient training. Multiplicative depth delivers superior parameter-to-expressivity scaling, and the findings provide both an axiomatic basis and actionable blueprint for future AI model design and compression, with broad implications across neural architecture optimization.