Wide Two-Layer Configuration

• Wide two-layer configurations are system architectures with two layers of scalable width, enabling universal function approximation and robust computational guarantees.
• They support diverse applications including neural networks, distributed energy markets, and programmable photonic devices, ensuring high expressivity and efficient resource aggregation.
• Careful hyperparameter tuning and mean-field dynamics underpin their convergence and performance, addressing challenges in optimization, generalization, and computational scalability.

A wide two-layer configuration is a system architecture consisting of two principal layers—each with potentially large width—designed to enable universal function approximation, distributed optimization, or complete controllability in both physical and computational domains. This concept arises prominently in neural networks (particularly in the mean-field and infinite-width regime), energy-sharing markets for prosumer systems, and programmable photonic devices. A wide two-layer configuration is characterized by architectural simplicity (fixed depth, width scalable) but robust expressive, approximation, or coverage guarantees when the layer width is sufficiently large.

1. Mathematical Structure and Instantiations

A wide two-layer configuration typically features an input layer, a first transformation ("hidden" or active computing) layer with large width, a second transformation or aggregation layer, and an output or global aggregator.

Neural Networks: In two-layer neural models, the network computes

$h_m(w;x) = \frac{1}{m} \sum_{j=1}^m \phi(w_j, x)$

where $m$ is the width (number of units/neurons), $\phi$ is a homogeneous or non-homogeneous nonlinearity (e.g., ReLU), and $w_j$ are typically high-dimensional parameters. In the infinite-width regime ($m\to\infty$), the network output is represented as an integral over a measure $\mu$ on parameter space: $h(\mu;x) = \int_{\mathbb{R}^p} \phi(w,x)\,d\mu(w)$ (Chizat et al., 2020, Jin et al., 2020, Hajjar et al., 2022).

Distributed Optimization/Energy Markets: In wide-area energy-sharing systems, the two layers appear as:

• Lower Layer: Multiple local-area markets (LAMs), each sharing resources among a wide set of agents (prosumers).
• Upper Layer: A wide-area market (WAM) aggregates boundary/surplus variables from LAMs and solves for system-wide constraints and equilibria (Su et al., 2024).

Programmable Photonics: A minimal-depth, wide two-layer architecture can universally realize arbitrary $N\times N$ matrix operations in photonic circuits by interleaving two active (programmable diagonal) layers with fixed passive mixing layers. Universality is guaranteed once the width (number of ports) exceeds a problem-dependent threshold: $K_c = \lceil N^2 \rceil$ for phase-only devices with $M=2$ active layers (Markowitz et al., 5 Mar 2025).

2. Universal Approximation and Expressivity

The ability of two-layer configurations to serve as universal approximators is established via both constructive and mean-field methods.

Neural Networks: Classical universal approximation theorems show that sufficiently wide two-layer (and even two-hidden-layer) feedforward networks can approximate any continuous function on a compact domain. Constructive approaches use triangulations and simplicial maps, precisely relating the required width of hidden layers to the granularity of mesh subdivisions and the geometry of target function (Gonzalez-Diaz et al., 2019). For ReLU networks, in the infinite-width limit, the space of functions representable corresponds to a variation-norm ball or a space of splines/polyharmonic splines, depending on the initialization (Jin et al., 2020).

Programmable Photonics: Two active layers interleaved with passive mixing planes suffice to implement any (embedded) $N\times N$ complex matrix transformation, provided the width $K$ meets the universal scaling law cited above (Markowitz et al., 5 Mar 2025).

Energy Sharing: The two-layer market design enables near-social-optimal allocation in large-scale systems; as the sizes of the lower-layer clusters grow, scalable welfare-optimality is achieved (Su et al., 2024).

3. Optimization Dynamics and Mean-Field Behavior

Gradient flow and stochastic optimization in wide two-layer systems are governed by emergent deterministic dynamics at large width, i.e., mean-field theory (MFT).

Population Dynamics: The empirical measure of parameters converges to the solution of a deterministic McKean–Vlasov equation, which can be described as a Wasserstein gradient flow for neural networks (Hajjar et al., 2022, Descours et al., 2022). These flows reflect the symmetries and invariances of the data and loss.

Convergence Properties: For both deterministic and stochastic training (e.g., SGD), convergence to global minima occurs at rates controlled by the minimum eigenvalues of associated Gram matrices. For physics-informed neural networks (PINNs), linear convergence of (S)GD can be rigorously established with explicit width-over-parameterization thresholds (Jin et al., 29 Aug 2025). For SGD with noise and mini-batching, law of large numbers and central limit theorems detail both the limiting dynamics and fluctuation regimes (Descours et al., 2022).

Implicit Bias: Gradient descent in wide two-layer networks induces a non-trivial implicit bias: the limiting solution minimizes a convex "variation-norm" (e.g., an $F_1$ norm in classification, or a curvature norm in regression). This explains generalization properties and kernel regime behavior (Chizat et al., 2020, Jin et al., 2020).

4. Algorithmic and Computational Implications

Decentralization and Scalability: In distributed systems (e.g., energy sharing), the wide two-layer decomposition enables drastically reduced computational complexity. Every agent (prosumer) solves a small convex subproblem; each community aggregates a single scalar to communicate with the upper layer. The upper-layer operator only needs to collect these aggregates to ensure systemwise balance and constraints. This hierarchical split is key for handling massive systems (e.g., $>10^4$ users) (Su et al., 2024).

Photonic Processors: The two-active-layer photonic design achieves universality with dramatically reduced component count relative to traditional mesh or SVD-based designs, offering a practical hardware route to scalable programmable linear algebra (Markowitz et al., 5 Mar 2025).

Algorithmic Barriers: For storage capacity and constraint satisfaction in infinite-width two-layer neural systems, algorithmic hardness results show that gradient-based methods frequently stall far below the optimal capacity—well before physical (SAT/UNSAT) limits—especially in glassy (full-RSB/OGP) phases (Annesi et al., 2024).

5. Generalization, Structure Adaptation, and Symmetry

Generalization Bounds: Wide two-layer networks trained with logistic or MSE loss achieve dimension-independent generalization if the data concentrate near a low-dimensional manifold, with gapless margins controlled by hidden structure rather than the ambient dimension (Chizat et al., 2020).

Symmetry Preservation: Mean-field gradient flows in wide two-layer neural systems preserve any orthogonal symmetry present in the target or data distribution. For odd targets, the flow collapses to a linear subspace with exponential convergence. When the target exhibits low-dimensional structure, the mean-field PDE reduces to fewer degrees of freedom, yielding effective dimension reduction and parameter alignment (Hajjar et al., 2022).

Adversarial Learning: For wide two-layer ReLU networks, adversarial perturbations—though they appear as noise—retain sufficient class-specific information. Training solely on adversarial samples can recover the same decision boundary as training on clean data, provided the network width is large enough to remain in a lazy NTK-like regime (Kumano et al., 2024).

6. Design Principles and Practical Guidelines

Optimal utilization of wide two-layer configurations requires attention to hyperparameters, scaling, and initialization:

• Scale the width polynomially in problem size, data dimension, and relevant Gram matrix conditioning parameters to obtain robust deterministic mean-field dynamics and ensure positive-definite kernels throughout training (Jin et al., 29 Aug 2025).
• For market or distributed resource systems, use elastic price rules in the lower layer, scale price elasticity inversely with cluster size, employ dual price updates in the upper layer, and rely on convexity for fast convergence (Su et al., 2024).
• In photonic implementations, increasing width while fixing depth at two ensures universal expressivity, dramatically reduces footprint, and maintains reconfigurability (Markowitz et al., 5 Mar 2025).
• For constructive network synthesis, mesh granularity and subdivision depth determine required layer widths and thus approximation accuracy (Gonzalez-Diaz et al., 2019).

7. Limitations and Open Problems

Despite their expressive power, wide two-layer configurations have structural and algorithmic limitations:

• In constraint satisfaction regimes with full-RSB (replica symmetry breaking), typical solutions are clustered and disconnected (overlap-gap). Gradient-based algorithms cannot access much of the feasible phase, indicating a fundamental hardness barrier (Annesi et al., 2024).
• For stochastic training, insufficiently wide networks or overly strong training noise ($\beta \leq 3/4$) yield divergent variances and instability in prediction dynamics (Descours et al., 2022).
• Full universality in some physical (e.g., photonic) contexts may require accepting small additional losses or embedding into higher-dimensional architectures (Markowitz et al., 5 Mar 2025).

Overall, wide two-layer configurations represent a foundational design pattern in both learning theory and physical computation, underpinning modern advances in scalable neural architecture, distributed optimization, and programmable analog computation.