EYS Initialization in Autoencoders and SMoE

• EYS Initialization is a deterministic strategy that uses spectral and edge-based techniques to initialize deep symmetric autoencoders and steered mixture-of-experts models.
• It employs iterated SVD in autoencoders to optimally set encoder-decoder weights and uses Canny edge detection with clustering to guide expert placement in SMoE models.
• Empirical results show that this approach accelerates convergence, reduces reconstruction error, and improves performance compared to random or heuristic initialization methods.

Edge-aligned initialization, known as EYS initialization, denotes two distinct but conceptually analogous strategies for initializing machine learning models: in deep symmetric autoencoders, where it leverages the Eckart–Young–Schmidt (EYS) theorem to derive optimal low-rank linear projections via iterated SVD; and in steered mixture-of-experts (SMoE) image models, where it deterministically places and parameterizes experts along prominent image edges using classical edge detection and clustering methods. Both approaches emphasize data-informed, deterministic initialization over stochastic schemes, resulting in accelerated convergence, superior early reconstruction, and often improved final performance.

1. Mathematical Foundations: Eckart–Young–Schmidt for Autoencoders

EYS initialization for symmetric autoencoders is rooted in the Eckart–Young–Schmidt theorem. For a matrix $A \in \mathbb{R}^{m\times n}$ with SVD $A = U \Sigma V^T$, the best rank-$r$ approximation in the Frobenius norm is the truncated SVD: $A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$ which minimizes $\|A - A_r\|_F^2$ and guarantees the minimum sum of squared discarded singular values. In the context of linear or orthogonality-constrained autoencoders, this result identifies the optimal encoder/decoder subspaces.

Deep symmetric autoencoders extend this framework by composing several such projections interleaved with nonlinear, typically invertible activations, maintaining weight symmetry (decoder mirrors encoder). EYS initialization generalizes the optimal linear solution to the deep, nonlinear setting by computing SVDs of the (possibly nonlinearly transformed) data representation at each successive layer, thus seeding all weight matrices and biases to closely approximate the optimal low-rank affine encoder/decoder pair before any gradient-based learning (Brivio et al., 13 Jun 2025).

2. EYS Initialization Algorithms: Derivation and Pipeline

2.1. Symmetric Autoencoders

Given training samples $X \in \mathbb{R}^{n_0 \times S}$, and target layer widths $n_0 > n_1 > \dots > n_\ell$:

1. At each layer $j$:
   • Center the current representations: $b_j = \text{mean}(Z^{(j-1)})$, $\widehat{Z} = Z^{(j-1)} - b_j1^T$.
   • Compute the rank-$A = U \Sigma V^T$0 SVD of $A = U \Sigma V^T$1.
   • Set the encoder weight $A = U \Sigma V^T$2, decoder $A = U \Sigma V^T$3.
   • Set biases $A = U \Sigma V^T$4.
   • Project data for the next layer: $A = U \Sigma V^T$5.
2. Repeat for all $A = U \Sigma V^T$6 layers.

The process is fully deterministic, and all layers are initialized such that encoding–decoding maps are exact inverses in the linear subspace associated with each SVD (Brivio et al., 13 Jun 2025).

2.2. Steered Mixture-of-Experts (SMoE)

EYS initialization for SMoE involves edge-informed kernel placement and expert initialization:

1. Canny edge detection: Denoise the image, compute gradients and orientations, perform non-maximum suppression, and apply double-threshold hysteresis to form a binary edge map $A = U \Sigma V^T$7.
2. Segment extraction: Scan for line segments in canonical directions (0°, 90°, ±45°), recording chain centers and their angles.
3. Importance-weighted reduction: For each candidate, compute an importance score based on intra- and inter-orientation neighborhood distances; sort, cluster (via DBSCAN), and retain a compact set of key edge midpoints and angles.
4. Kernel placement: For each selected center–angle pair, place two Gaussian kernels offset perpendicularly to the segment, both assigned the segment orientation.
5. Expert coefficients: Assign initial coefficients from direct pixel values or closed-form least-squares over local patches.
6. Optional refinement: Iteratively adjust coefficients to match local intensities.

This procedure is deterministic and ensures kernel population is concentrated along salient image structure, as opposed to random or grid-based initialization (Determann et al., 2 Feb 2026).

3. Computational Complexity and Implementation

Symmetric Autoencoders

Each truncated SVD on an $A = U \Sigma V^T$8 matrix at layer $A = U \Sigma V^T$9 to rank $r$0 costs $r$1 (Lanczos) or $r$2 (randomized SVD). Total cost sums over all layers. For large sample count ($r$3), covariance methods may reduce cost to $r$4.

SMoE

EYS initialization in SMoE is $r$5, with $r$6 the number of image pixels and $r$7 the number of resulting kernels (typically $r$8). The majority of cost is in edge extraction and clustering, with memory storing the edge mask, contours, and final kernel parameters (Determann et al., 2 Feb 2026).

A high-level algorithmic comparison is summarized as follows:

Method	Computational Complexity	Memory Usage
Random/Grid init	$r$9 per kernel	$A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$0
SMoE Gradient-based	$A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$1	$A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$2
SMoE EYS (edge-aligned)	$A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$3	$A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$4

4. Hyperparameter Selection and Influence

Symmetric Autoencoders

Critical choices include:

• Layer widths $A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$5: Typically set by retaining a target fraction (e.g., 95–99%) of variance in the top singular values at each layer.
• SVD Algorithm: Lanczos, randomized SVD, or direct eigen-decomposition based on computational regime.
• Network depth $A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$6 and activation $A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$7: Deeper networks allow greater model capacity, but due to error amplification (as per Theorem 3.3 in (Brivio et al., 13 Jun 2025)), excessive depth or overly sharp Lipschitz constants can degrade performance.

SMoE

Key hyperparameters include Canny filter width ($A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$8), threshold settings, kernel offset ($A_r = U_{:,1:r} \Sigma_{1:r,1:r} V_{:,1:r}^T,$9), maximal number of kernels, clustering radius (DBSCAN $\|A - A_r\|_F^2$0), importance weighting ($\|A - A_r\|_F^2$1), and expert adjustment rate ($\|A - A_r\|_F^2$2). Practical guidelines are given on typical value ranges, and cross-validated selection is recommended for optimal results (Determann et al., 2 Feb 2026).

5. Empirical Results and Comparative Evaluation

Symmetric Autoencoders

Experiments demonstrate that EYS initialization yields orders-of-magnitude lower initial reconstruction error, dramatically faster loss descent (<50 epochs to near-optimal MSE versus hundreds for random/Xavier/He initializations), and improved final accuracy (10–50% lower test MSE) as well as reduced variance across random seeds. These benefits persist across unconstrained, biorthogonal, and strictly orthogonal symmetric architectures on diverse synthetic datasets: parameterized Gaussian waves (PGA), elasticity collision (ELS), and rod diffusion (ROD) (Brivio et al., 13 Jun 2025).

SMoE

On standard test images (e.g., 512×512 "Barbara," "Peppers," "Flowers," "Parrots"), EYS initialization with roughly 200 kernels achieves post-training PSNR ≈ 27.5 dB, SSIM ≈ 0.92, compared to PSNR ≈ 25.0 dB, SSIM ≈ 0.86 for grid-initialized SMoE. Convergence time to final PSNR is reduced from ≈3.8 s (grid init) and ≈2.1 s (segmentation-based) to ≈0.9 s with EYS. The method yields consistently better initial structures, fewer required gradient steps, and improved final metric values per kernel (Determann et al., 2 Feb 2026).

6. Practical Implications and Applicability

EYS initialization, both in autoencoder and SMoE contexts, offers a principled, data-aligned alternative to stochastic or naïve heuristics, yielding deterministic, reproducible initializations that closely match the data manifold or structure. This deterministic seeding facilitates stable nonconvex optimization, reduces sensitivity to hyperparameters, and provides compressive representations that are interpretable—an essential property for scientific modeling, compression, or reduced order modeling in high-dimensional systems (Brivio et al., 13 Jun 2025, Determann et al., 2 Feb 2026).

A plausible implication is that EYS-based protocols can serve as general templates for deterministically initializing deep, layered models wherever linear low-rank approximations (via SVD) or edge-aware priors are meaningful.

7. Comparison and Taxonomy

EYS initialization is unified by two central principles: using spectral (SVD-based) or structural (edge-based) decompositions of the target dataset to inform the initial parameterization. In symmetric autoencoders, the emphasis is on spectral optimality in reconstruction, while in SMoE, the focus is on placement of experts at data-adaptive edge structure.

Domain	EYS Principle	Primary Mechanism	Empirical Gains
Symmetric Autoencoder	Spectral alignment	Iterated SVD/projector	Lower initial/final MSE, rapid convergence
SMoE image models	Edge alignment	Canny+clustering	Higher PSNR/SSIM, reduced training time

Their adoption in modern architectures is supported by rigorous mathematical derivation and empirical evidence across diverse test beds. Both variants remove the dependence on random seeds and stochastic search for parameter selection, establishing a deterministic baseline for further model refinement.