Structured First-Layer Initialization (SFLI)

• SFLI is a suite of initialization strategies that replaces random first-layer weights with structured, analytically or data-derived configurations to boost feature diversity at startup.
• It incorporates design families such as analytic orthogonal bases, supervised data-dependent methods, and mimetic statistical constructions to optimize scaling, conditioning, and orthogonality.
• Empirical studies show that SFLI improves convergence speed, accuracy, and variance control across diverse domains including scientific computing, medical deep learning, and vision transformer models.

Searching arXiv for the cited papers and closely related work on structured first-layer initialization. Searching for "Structured First-Layer Initialization" and the provided arXiv IDs. I’ll look up the relevant arXiv entries to ground the article in the cited literature. Structured First-Layer Initialization (SFLI) denotes a family of initialization strategies in which the first trainable transformation is assigned weights with explicit structure rather than being sampled as i.i.d. zero-mean noise. In the literature, this structure is derived from several sources: analytically constructed orthogonal bases, class-separating discriminants, tree-induced sparse feature interactions, convolutional impulse patterns for attention, covariance patterns observed in trained networks, or activation-aware constructions designed to raise the initial $\varepsilon$-rank of the hidden representation. Although the explicit label “SFLI” is used most directly in scientific-computing work, closely related first-layer-focused methods have been reported for feed-forward networks, medical deep learning, tabular MLPs, Vision Transformers, and MLP channel-mixing layers (Tang et al., 16 Jul 2025, Masden et al., 2020, Shkolnikov, 30 Mar 2026, Lutz et al., 2022, Zheng et al., 2024, Trockman et al., 6 Feb 2026).

1. Definition, scope, and first-layer specificity

SFLI is motivated by the observation that the first layer is not merely another affine map. It is the interface between raw inputs and learned internal features, so its scaling, geometry, and inductive bias affect the entire subsequent optimization trajectory. In the $\varepsilon$-rank formulation, the objective is to make the first hidden layer “already feature-rich at initialization,” so that optimization does not need to discover diverse basis functions from scratch (Tang et al., 16 Jul 2025). In the scaling-calculus formulation for ReLU networks, the first layer is treated specially because input scaling, weight scaling, and bias scaling interact differently there; the analysis derives a first-layer condition

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$

and recommends including an initial forward scaling factor of $1/(n_0k^2)^{1/4}$ to balance learning dynamics (Defazio et al., 2019).

The term SFLI does not denote a single algorithm. Some papers use it for explicit first-layer-only procedures, such as $\varepsilon$-rank pre-training in scientific computing (Tang et al., 16 Jul 2025). Others are best understood as first-layer-focused structured initializers even when the implementation extends beyond the first layer. Sinusoidal initialization, for example, is theoretically strongest in its first-layer analysis through exact row-sum cancellation, but in experiments it is manually applied to all convolutional and linear layers (Fernández-Hernández et al., 19 May 2025). Likewise, depth-aware initialization organizes variance across all layers, yet the first layer is the anchor of the schedule (Pandey, 5 Sep 2025). By contrast, FLIP is a broader structured initializer for parametrized quantum circuits and is not first-layer-only; it learns a full per-parameter initialization rule across arbitrary circuit sizes (Sauvage et al., 2021).

A common misconception is that SFLI is synonymous with deterministic initialization. The literature is broader. Some methods are deterministic and analytic, such as DCT-based orthogonal filters (Shkolnikov, 30 Mar 2026). Some are deterministic but data-dependent, such as linear discriminant sorting (Masden et al., 2020). Some are structured yet still partly random, such as the row-broadcasted Gaussian mean shift for MLPs,

$W_1' = W_1 + \mathbf{1}_p b_n^T,$

with $b_n \sim \mathcal{N}(0,\sigma I_n)$ (Trockman et al., 6 Feb 2026).

2. Principal design families

The published literature organizes naturally into several families distinguished by where the structure comes from and what property it is intended to enforce.

Family	First-layer structure	Representative papers
Analytic basis constructions	Orthogonal or sinusoidal filters, often deterministic	(Shkolnikov, 30 Mar 2026, Fernández-Hernández et al., 19 May 2025)
Supervised data-dependent constructions	Linear discriminants, tree partitions, BLA state-space seeds	(Masden et al., 2020, Lutz et al., 2022, Schoukens et al., 2020)
Mimetic/statistical constructions	Weight statistics inspired by trained models	(Trockman et al., 6 Feb 2026)
Inductive-bias transfer to attention	Impulse-like attention maps from positional structure	(Zheng et al., 2024, Zheng et al., 26 May 2025)
Rank-oriented scientific-computing constructions	$\varepsilon$-linearly independent neurons in the input layer	(Tang et al., 16 Jul 2025)

Analytic basis constructions replace random first-layer sampling with deterministic filters drawn from structured orthogonal bases. In medical deep learning, the first convolutional layer is initialized from a DCT-II basis, with support for Hadamard, Hartley, and sinusoidal bases as alternatives; the first-layer filters are described as orthogonal or near-orthogonal after scaling to a target variance (Shkolnikov, 30 Mar 2026). Sinusoidal initialization similarly defines each row of a layer weight matrix as a sinusoidal waveform with frequency $k_i=2\pi i$, phase $\phi_i=2\pi i/m$, and input grid $\varepsilon$0 (Fernández-Hernández et al., 19 May 2025).

Supervised data-dependent constructions use the training data to place the first-layer hyperplanes or sparse connections in a class-aware or model-aware manner. Linear discriminant sorting computes class-vs-rest discriminants and uses them as first-layer weights (Masden et al., 2020). Sparse tree-based initialization first trains a Random Forest, GBDT/XGBoost, or Deep Forest, translates that ensemble into an equivalent neural representation, and uses the translated sparse prefix to initialize the first layers of an MLP (Lutz et al., 2022). In nonlinear LFR identification, the Best Linear Approximation provides the initial linear dynamics, and the nonlinear block is initialized so that the overall model initially matches the BLA and preserves stability if the BLA is stable (Schoukens et al., 2020).

Mimetic/statistical constructions use trained networks as case studies of good initialization. For channel-mixing MLPs, the first layer is given a nonzero mean inspired by covariance patterns observed in trained weights, producing a lightweight first-layer bias in the weight matrix rather than in the explicit bias parameter (Trockman et al., 6 Feb 2026). For Vision Transformers, the first attention layer can be initialized so that its attention maps resemble convolutional impulse filters, thereby injecting a CNN-like inductive bias through initialization alone (Zheng et al., 2024, Zheng et al., 26 May 2025).

3. Canonical constructions and mathematical forms

In the explicit SFLI formulation for scientific computing, first-layer neurons are written as

$\varepsilon$1

with $\varepsilon$2, $\varepsilon$3, and a practical recommendation

$\varepsilon$4

The construction is activation-specific: cosine SFLI samples $\varepsilon$5, tanh and hat SFLI sample $\varepsilon$6, and Gaussian SFLI uses radial neurons

$\varepsilon$7

with $\varepsilon$8 uniformly distributed in the domain (Tang et al., 16 Jul 2025).

Linear discriminant initialization uses the top Fisher/LDA direction as a first-layer weight vector. The standard discriminant direction is

$\varepsilon$9

followed by normalization to the unit component vector, scalar projections

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$0

and selection of a bias $E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$1 that maximizes the number of points correctly placed relative to the separating hyperplane $E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$2 (Masden et al., 2020).

Mimetic initialization for MLPs modifies the first linear layer of

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$3

by either a row-broadcasted random offset,

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$4

or a constant scalar mean shift,

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$5

The second form is the practical “small nonzero mean” initialization emphasized in the paper (Trockman et al., 6 Feb 2026).

Structured orthogonal initialization for convolutional first layers constructs basis filters analytically. For $E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$6, one DCT-II construction is

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$7

and the orthonormal DCT-II matrix satisfies

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$8

These rows are used as first-layer filters after zero-meaning and scaling (Shkolnikov, 30 Mar 2026).

For ViT attention, the structured objective is to initialize a head so that

$E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}$9

resembles an impulse convolution matrix. One formulation uses positional encoding $1/(n_0k^2)^{1/4}$0 as pseudo-input and seeks

$1/(n_0k^2)^{1/4}$1

with a pseudo-inverse/SVD factorization in one account (Zheng et al., 26 May 2025), whereas another solves

$1/(n_0k^2)^{1/4}$2

by direct optimization with MSE loss against the impulse target (Zheng et al., 2024).

4. Mechanistic rationales

The theoretical arguments for SFLI are heterogeneous, but several recurrent mechanisms appear.

One rationale is feature diversity. The $1/(n_0k^2)^{1/4}$3-rank framework defines the Gram matrix

$1/(n_0k^2)^{1/4}$4

and the effective rank

$1/(n_0k^2)^{1/4}$5

The associated empirical claim is a staircase phenomenon: long loss plateaus are followed by sharp drops, and those drops correlate with increases in $1/(n_0k^2)^{1/4}$6-rank. SFLI is therefore designed to make first-layer neurons approximately $1/(n_0k^2)^{1/4}$7-linearly independent at initialization (Tang et al., 16 Jul 2025).

A second rationale is scaling and conditioning. The scaling calculus for ReLU networks introduces a layer scaling factor $1/(n_0k^2)^{1/4}$8 and argues that networks with constant $1/(n_0k^2)^{1/4}$9 are “preconditioned.” The same framework connects $\varepsilon$0 to the weight-to-gradient ratio

$\varepsilon$1

and to the average squared singular value of a diagonal Hessian block, yielding a structured justification for geometric-mean variance initialization and special treatment of the first layer (Defazio et al., 2019).

A third rationale is orthogonality. Structured orthogonal initialization emphasizes exact or near-exact norm preservation. For the orthonormal DCT-II matrix, the condition number is

$\varepsilon$2

and Parseval’s theorem gives

$\varepsilon$3

The paper contrasts this with Kaiming/Gaussian matrices, whose condition number worsens with dimensionality, and further argues that DCT approximately whitens correlated signals,

$\varepsilon$4

which is presented as a useful inductive bias for ECG (Shkolnikov, 30 Mar 2026).

A fourth rationale is structured balance in the first layer. Sinusoidal initialization studies the weight-sum statistic

$\varepsilon$5

and links large $\varepsilon$6 to skewed neurons. Its central structural theorem is row cancellation:

$\varepsilon$7

This guarantees exact first-layer row-sum centering in the proposed sinusoidal matrix (Fernández-Hernández et al., 19 May 2025).

Mimetic MLP initialization offers a different mechanistic argument. The empirical covariance of

$\varepsilon$8

exhibits “striped” patterns in trained models, suggesting row or column correlations and possible nonzero group means. The first-layer mean shift is meant to mimic this structure and place the network in a more favorable optimization region (Trockman et al., 6 Feb 2026).

5. Empirical record across domains

In feed-forward classification, linear discriminant initialization is reported to reduce the number of training epochs needed to reach threshold accuracy and to lower minimum validation error. On MNIST and Fashion-MNIST, the LDA procedure produced fixed first-layer sizes of 21 weights and 28 weights, respectively. On Fashion-MNIST, the reported savings in epochs to threshold accuracy were about 4.5 to 29 epochs depending on batch size and learning rate, and minimum validation error improved by roughly 0.03% to 0.78%. In an AlexNet/CIFAR-10 fine-tuning experiment, sorting-game initialization reached threshold accuracy on average 7.2 epochs sooner than Xavier-normal and achieved about 2.13 percentage points lower minimum validation error, with a 95% confidence interval of 1.86 to 2.41 percentage points (Masden et al., 2020).

In scientific computing, the explicit SFLI pre-training method is reported to raise initial $\varepsilon$9-rank, reduce early plateaus, mitigate spectral bias, and improve final errors in both function approximation and PDE benchmarks. Representative results include high-dimensional smooth-function relative test errors of $W_1' = W_1 + \mathbf{1}_p b_n^T,$0 versus $W_1' = W_1 + \mathbf{1}_p b_n^T,$1 at $W_1' = W_1 + \mathbf{1}_p b_n^T,$2 and $W_1' = W_1 + \mathbf{1}_p b_n^T,$3 versus $W_1' = W_1 + \mathbf{1}_p b_n^T,$4 at $W_1' = W_1 + \mathbf{1}_p b_n^T,$5 for baseline versus SFLI-Gaussian. On the lid-driven cavity flow benchmark at $W_1' = W_1 + \mathbf{1}_p b_n^T,$6, the reported final relative $W_1' = W_1 + \mathbf{1}_p b_n^T,$7 errors were 88.3% without SFLI and 3.75% with SFLI. On a time-marching Navier–Stokes problem in a torus, the reported final vorticity errors were $W_1' = W_1 + \mathbf{1}_p b_n^T,$8 without SFLI and $W_1' = W_1 + \mathbf{1}_p b_n^T,$9 with SFLI (Tang et al., 16 Jul 2025).

In medical deep learning, structured orthogonal first-layer initialization is presented as part of a bit-identical training pipeline. The pipeline removes randomness from weight initialization, batch ordering, and GPU kernels, yielding MD5-verified identical trained weights across independent runs. On PTB-XL ECG rhythm classification, structured initialization significantly exceeded Kaiming across two architectures, with Conformer results of $b_n \sim \mathcal{N}(0,\sigma I_n)$0 versus $b_n \sim \mathcal{N}(0,\sigma I_n)$1 and baseline CNN results of $b_n \sim \mathcal{N}(0,\sigma I_n)$2 versus $b_n \sim \mathcal{N}(0,\sigma I_n)$3. Aggregate variance was reduced by 2–3x, and rare-class variability was sharply reduced; the TRIGU range was 4.1 percentage points under structured initialization versus 30.9 percentage points under Kaiming. A four-basis comparison at $b_n \sim \mathcal{N}(0,\sigma I_n)$4 found no significant difference across DCT, Hadamard, Hartley, and sinusoidal bases, with Friedman $b_n \sim \mathcal{N}(0,\sigma I_n)$5, supporting the claim that the effect comes from deterministic structured initialization itself rather than from one privileged basis (Shkolnikov, 30 Mar 2026).

In tabular learning, sparse tree-based initialization uses a trained tree ensemble to shape the first layers of an MLP into a sparse feature extractor. Across 10 tabular datasets, RF or GBDT initialization strictly outperformed random MLP initialization on all datasets except Covertype, where performance was roughly similar. The initialized MLPs were often competitive with SAINT and with strong tree baselines, while preserving the sparse first-layer structure through training (Lutz et al., 2022).

In Vision Transformers, structured initialization of the first attention layer consistently improved data-efficient training. One study reported on ViT-T that SFLI impulse initialization achieved 91.62 on CIFAR-10, 70.46 on CIFAR-100, 97.23 on SVHN, and 74.40 on ImageNet-1K, compared with 88.63, 66.50, 93.20, and 73.42 for truncated normal initialization, and competitive results relative to mimetic initialization (Zheng et al., 2024). A subsequent formulation reported ViT-T gains from 92.29 to 94.67 on CIFAR-10, from 71.67 to 77.02 on CIFAR-100, from 64.60 to 73.18 on Flowers, and maintained competitive ImageNet-1K performance; it also reported gains for Swin Transformer and MLP-Mixer, including 83.14 to 83.55 for Swin-B on ImageNet-1K and 87.00 to 88.78 for MLP-Mixer on CIFAR-10 (Zheng et al., 26 May 2025).

For MLP channel mixing, the first-layer nonzero-mean mimetic scheme improved early training on CIFAR-10 and ImageNet-1K and was complementary to spatial-mixing mimetic initializations. On DeiT-Small for ImageNet-1K, the reported final accuracies over 100 epochs were 68.8% with the proposed MLP initialization and 68.4% for the baseline, described as a small but consistent advantage (Trockman et al., 6 Feb 2026).

6. Limitations, misconceptions, and open problems

The literature is explicit that SFLI is not a universal substitute for conventional initialization. The evidence is often strongest for the first layer only. Linear discriminant initialization did not yield positive results when naively extended to a second layer by applying LDA to the image of data under the first layer (Masden et al., 2020). The MLP mimetic mean-shift paper describes its own effect as much smaller than earlier spatial-mixing mimetic initializations, strongest on small-scale or short-duration training, and diminishing as training proceeds (Trockman et al., 6 Feb 2026). The scientific-computing SFLI paper states that extending the $b_n \sim \mathcal{N}(0,\sigma I_n)$6-rank theory and the associated initialization schemes to convolutional, attention-based, or graph architectures is nontrivial and future work (Tang et al., 16 Jul 2025).

A second misconception is that any apparent gain can be reduced to ordinary bias initialization. This is contradicted directly in the mimetic MLP study: changing the linear-layer bias initialization, adding a learnable scalar bias per $b_n \sim \mathcal{N}(0,\sigma I_n)$7, or making the bias parameter small and constant did not reproduce the effect of shifting the first-layer weight matrix mean (Trockman et al., 6 Feb 2026). Conversely, the medical orthogonal-initialization paper shows that the main contribution is not tied to any single analytic basis, since DCT, Hadamard, Hartley, and sinusoidal bases were statistically equivalent in the reported four-basis comparison (Shkolnikov, 30 Mar 2026).

Implementation-specific limitations also recur. Tree-to-MLP translation can suffer from catastrophic cancellation in the exact output-layer formula, and the relaxed $b_n \sim \mathcal{N}(0,\sigma I_n)$8-based translation introduces extra hyperparameters (Lutz et al., 2022). ViT structured-attention initialization depends on a pseudo-input, usually positional encoding, and is constrained by the number of heads and the fact that only $b_n \sim \mathcal{N}(0,\sigma I_n)$9 and $\varepsilon$0 are initialized structurally, not the value projection or patch embedding (Zheng et al., 2024). Sinusoidal and depth-aware methods are often cited in SFLI discussions because their theory is strongly first-layer oriented, yet their actual implementations are layerwise rather than strictly first-layer-only (Fernández-Hernández et al., 19 May 2025, Pandey, 5 Sep 2025).

Taken together, these results suggest that SFLI is best viewed not as a single initializer but as a design principle: the first trainable interaction with the input can be engineered to encode discriminative geometry, spectral structure, sparsity, convolutional locality, conditioning, or feature diversity before SGD begins. The practical success of that principle is domain-dependent, but the accumulated evidence indicates that first-layer structure can measurably alter convergence, variance, and generalization without requiring architectural changes (Tang et al., 16 Jul 2025, Shkolnikov, 30 Mar 2026, Zheng et al., 2024).