---
title: Structured First-Layer Initialization (SFLI)
url: https://www.emergentmind.com/topics/structured-first-layer-initialization-sfli
type: topic
---

# Structured First-Layer Initialization (SFLI)

Searching arXiv for the cited papers and closely related work on structured first-layer initialization.
Searching for "Structured First-Layer Initialization" and the provided arXiv IDs.
I’ll look up the relevant arXiv entries to ground the article in the cited literature.
Structured First-Layer Initialization (SFLI) denotes a family of initialization strategies in which the first trainable transformation is assigned weights with explicit structure rather than being sampled as i.i.d. zero-mean noise. In the literature, this structure is derived from several sources: analytically constructed orthogonal bases, class-separating discriminants, tree-induced sparse feature interactions, convolutional impulse patterns for attention, covariance patterns observed in trained networks, or activation-aware constructions designed to raise the initial $\varepsilon$-rank of the hidden representation. Although the explicit label “SFLI” is used most directly in scientific-computing work, closely related first-layer-focused methods have been reported for feed-forward networks, medical deep learning, tabular MLPs, Vision Transformers, and MLP channel-mixing layers [2507.11962, 2007.12782, 2603.28040, 2209.15283, 2404.01139, 2602.07156].

## 1. Definition, scope, and first-layer specificity

SFLI is motivated by the observation that the first layer is not merely another affine map. It is the interface between raw inputs and learned internal features, so its scaling, geometry, and inductive bias affect the entire subsequent optimization trajectory. In the $\varepsilon$-rank formulation, the objective is to make the first hidden layer “already feature-rich at initialization,” so that optimization does not need to discover diverse basis functions from scratch [2507.11962]. In the scaling-calculus formulation for ReLU networks, the first layer is treated specially because input scaling, weight scaling, and bias scaling interact differently there; the analysis derives a first-layer condition
$$
E\left[x_0^2\right]=\frac{1}{\sqrt{n_0k_0^2}}
$$
and recommends including an initial forward scaling factor of $1/(n_0k^2)^{1/4}$ to balance learning dynamics [1906.04267].

The term SFLI does not denote a single algorithm. Some papers use it for explicit first-layer-only procedures, such as $\varepsilon$-rank pre-training in scientific computing [2507.11962]. Others are best understood as first-layer-focused structured initializers even when the implementation extends beyond the first layer. Sinusoidal initialization, for example, is theoretically strongest in its first-layer analysis through exact row-sum cancellation, but in experiments it is manually applied to all convolutional and linear layers [2505.12909]. Likewise, depth-aware initialization organizes variance across all layers, yet the first layer is the anchor of the schedule [2509.05018]. By contrast, FLIP is a broader structured initializer for parametrized quantum circuits and is not first-layer-only; it learns a full per-parameter initialization rule across arbitrary circuit sizes [2103.08572].

A common misconception is that SFLI is synonymous with deterministic initialization. The literature is broader. Some methods are deterministic and analytic, such as DCT-based orthogonal filters [2603.28040]. Some are deterministic but data-dependent, such as linear discriminant sorting [2007.12782]. Some are structured yet still partly random, such as the row-broadcasted Gaussian mean shift for MLPs,
$$
W_1' = W_1 + \mathbf{1}_p b_n^T,
$$
with $b_n \sim \mathcal{N}(0,\sigma I_n)$ [2602.07156].

## 2. Principal design families

The published literature organizes naturally into several families distinguished by where the structure comes from and what property it is intended to enforce.

| Family | First-layer structure | Representative papers |
|---|---|---|
| Analytic basis constructions | Orthogonal or sinusoidal filters, often deterministic | [2603.28040], [2505.12909] |
| Supervised data-dependent constructions | Linear discriminants, tree partitions, BLA state-space seeds | [2007.12782], [2209.15283], [2004.05040] |
| Mimetic/statistical constructions | Weight statistics inspired by trained models | [2602.07156] |
| Inductive-bias transfer to attention | Impulse-like attention maps from positional structure | [2404.01139], [2505.19985] |
| Rank-oriented scientific-computing constructions | $\varepsilon$-linearly independent neurons in the input layer | [2507.11962] |

Analytic basis constructions replace random first-layer sampling with deterministic filters drawn from structured orthogonal bases. In medical deep learning, the first convolutional layer is initialized from a DCT-II basis, with support for Hadamard, Hartley, and sinusoidal bases as alternatives; the first-layer filters are described as orthogonal or near-orthogonal after scaling to a target variance [2603.28040]. Sinusoidal initialization similarly defines each row of a layer weight matrix as a sinusoidal waveform with frequency $k_i=2\pi i$, phase $\phi_i=2\pi i/m$, and input grid $x_j=j/n$ [2505.12909].

Supervised data-dependent constructions use the training data to place the first-layer hyperplanes or sparse connections in a class-aware or model-aware manner. Linear discriminant sorting computes class-vs-rest discriminants and uses them as first-layer weights [2007.12782]. Sparse tree-based initialization first trains a Random Forest, GBDT/XGBoost, or Deep Forest, translates that ensemble into an equivalent neural representation, and uses the translated sparse prefix to initialize the first layers of an MLP [2209.15283]. In nonlinear LFR identification, the Best Linear Approximation provides the initial linear dynamics, and the nonlinear block is initialized so that the overall model initially matches the BLA and preserves stability if the BLA is stable [2004.05040].

Mimetic/statistical constructions use trained networks as case studies of good initialization. For channel-mixing MLPs, the first layer is given a nonzero mean inspired by covariance patterns observed in trained weights, producing a lightweight first-layer bias in the weight matrix rather than in the explicit bias parameter [2602.07156]. For Vision Transformers, the first attention layer can be initialized so that its attention maps resemble convolutional impulse filters, thereby injecting a CNN-like inductive bias through initialization alone [2404.01139, 2505.19985].

## 3. Canonical constructions and mathematical forms

In the explicit SFLI formulation for scientific computing, first-layer neurons are written as
$$
F(x)=\sigma(Wx+b), \qquad
F_i(x;w_i,b_i)=\sigma(w_i\cdot x+b_i)=\sigma\!\big(\gamma(\alpha_i\cdot x+c_i)\big),
$$
with $w_i=\gamma\alpha_i$, $b_i=\gamma c_i$, and a practical recommendation
$$
\gamma = C\cdot \frac{n^{1/d}-1}{|\Omega|^{1/d}}, \qquad C\in[0.5,2].
$$
The construction is activation-specific: cosine SFLI samples $\alpha_i\sim\mathcal N(\mathbf 0,\mathbf I_d)$, tanh and hat SFLI sample $\alpha_i\in\mathbb S^d$, and Gaussian SFLI uses radial neurons
$$
F_i(x;\gamma,c_i)=\exp\!\left(-\gamma^2\|x-c_i\|^2\right)
$$
with $c_i$ uniformly distributed in the domain [2507.11962].

Linear discriminant initialization uses the top Fisher/LDA direction as a first-layer weight vector. The standard discriminant direction is
$$
\mathbf{w} \propto S_W^{-1}(\boldsymbol{\mu}_1-\boldsymbol{\mu}_2),
$$
followed by normalization to the unit component vector, scalar projections
$$
z_k=\vec w\cdot x_k,
$$
and selection of a bias $b$ that maximizes the number of points correctly placed relative to the separating hyperplane $\vec w\cdot x=b$ [2007.12782].

Mimetic initialization for MLPs modifies the first linear layer of
$$
y=W_2\sigma(W_1x)
$$
by either a row-broadcasted random offset,
$$
W_1' = W_1 + \mathbf{1}_p b_n^T,
$$
or a constant scalar mean shift,
$$
W_1' = W_1 + b\,\mathbf{1}_p\mathbf{1}_n^T.
$$
The second form is the practical “small nonzero mean” initialization emphasized in the paper [2602.07156].

Structured orthogonal initialization for convolutional first layers constructs basis filters analytically. For $\mathrm{Conv1d}(C_\text{in},C_\text{out},K)$, one DCT-II construction is
$$
d_i[j]=\cos\!\left(\frac{\pi\, i\, (2j+1)}{2\, C_\text{in}\, K}\right),
\qquad i\in[0,C_\text{out}),\ j\in[0,C_\text{in}\cdot K),
$$
and the orthonormal DCT-II matrix satisfies
$$
C_{k,n}=\alpha_k\cos\!\left(\frac{\pi(n+\tfrac12)k}{N}\right), \qquad
\mathbf C^\top \mathbf C=\mathbf C\mathbf C^\top=\mathbf I_N.
$$
These rows are used as first-layer filters after zero-meaning and scaling [2603.28040].

For ViT attention, the structured objective is to initialize a head so that
$$
M_i=\mathrm{softmax}\!\left(X\mathbf Q_i\mathbf K_i^TX^T\right)
$$
resembles an impulse convolution matrix. One formulation uses positional encoding $P$ as pseudo-input and seeks
$$
M_{\text{init}}=\mathrm{softmax}\!\left(P\mathbf Q_{\text{init}}\mathbf K_{\text{init}}^TP^T\right)\approx H_{\text{impulse}},
$$
with a pseudo-inverse/SVD factorization in one account [2505.19985], whereas another solves
$$
\hat H_{\text{impulse}}=\mathrm{softmax}\!\left(\sigma \tilde X\mathbf Q_{\text{init}}\mathbf K_{\text{init}}^T\tilde X^T\right)
$$
by direct optimization with MSE loss against the impulse target [2404.01139].

## 4. Mechanistic rationales

The theoretical arguments for SFLI are heterogeneous, but several recurrent mechanisms appear.

One rationale is feature diversity. The $\varepsilon$-rank framework defines the Gram matrix
$$
(M_u)_{ij}=\int_\Omega \phi_i(x;\theta)\,\phi_j(x;\theta)\,dx
$$
and the effective rank
$$
r_\varepsilon(M_u)=\#\{\lambda\in\lambda(M_u)\mid \lambda>\varepsilon\}.
$$
The associated empirical claim is a staircase phenomenon: long loss plateaus are followed by sharp drops, and those drops correlate with increases in $\varepsilon$-rank. SFLI is therefore designed to make first-layer neurons approximately $\varepsilon$-linearly independent at initialization [2507.11962].

A second rationale is scaling and conditioning. The scaling calculus for ReLU networks introduces a layer scaling factor $\gamma_l$ and argues that networks with constant $\gamma_l$ are “preconditioned.” The same framework connects $\gamma_l$ to the weight-to-gradient ratio
$$
\nu_l \doteq \frac{E[\Delta W_l^2]}{E[W_l^2]}
$$
and to the average squared singular value of a diagonal Hessian block, yielding a structured justification for geometric-mean variance initialization and special treatment of the first layer [1906.04267].

A third rationale is orthogonality. Structured orthogonal initialization emphasizes exact or near-exact norm preservation. For the orthonormal DCT-II matrix, the condition number is
$$
\kappa(\mathbf C)=1,
$$
and Parseval’s theorem gives
$$
\|\mathbf C\mathbf x\|^2=\|\mathbf x\|^2.
$$
The paper contrasts this with Kaiming/Gaussian matrices, whose condition number worsens with dimensionality, and further argues that DCT approximately whitens correlated signals,
$$
\mathbf C\boldsymbol\Sigma_x\mathbf C^\top \approx \mathrm{diag}(\lambda_0,\ldots,\lambda_{N-1}),
$$
which is presented as a useful inductive bias for ECG [2603.28040].

A fourth rationale is structured balance in the first layer. Sinusoidal initialization studies the weight-sum statistic
$$
S=W_1+\cdots+W_n
$$
and links large $|S|$ to skewed neurons. Its central structural theorem is row cancellation:
$$
\sum_{j=1}^n W[i,j]=0 \qquad \text{for every } i.
$$
This guarantees exact first-layer row-sum centering in the proposed sinusoidal matrix [2505.12909].

Mimetic MLP initialization offers a different mechanistic argument. The empirical covariance of
$$
[\mathrm{vec}(W_1)\;\; \mathrm{vec}(W_2)]
$$
exhibits “striped” patterns in trained models, suggesting row or column correlations and possible nonzero group means. The first-layer mean shift is meant to mimic this structure and place the network in a more favorable optimization region [2602.07156].

## 5. Empirical record across domains

In feed-forward classification, linear discriminant initialization is reported to reduce the number of training epochs needed to reach threshold accuracy and to lower minimum validation error. On MNIST and Fashion-MNIST, the LDA procedure produced fixed first-layer sizes of 21 weights and 28 weights, respectively. On Fashion-MNIST, the reported savings in epochs to threshold accuracy were about 4.5 to 29 epochs depending on batch size and learning rate, and minimum validation error improved by roughly 0.03% to 0.78%. In an AlexNet/CIFAR-10 fine-tuning experiment, sorting-game initialization reached threshold accuracy on average 7.2 epochs sooner than Xavier-normal and achieved about 2.13 percentage points lower minimum validation error, with a 95% confidence interval of 1.86 to 2.41 percentage points [2007.12782].

In scientific computing, the explicit SFLI pre-training method is reported to raise initial $\varepsilon$-rank, reduce early plateaus, mitigate spectral bias, and improve final errors in both function approximation and PDE benchmarks. Representative results include high-dimensional smooth-function relative test errors of $1.79\times 10^{-2}$ versus $2.96\times 10^{-3}$ at $d=5$ and $9.96\times 10^{-1}$ versus $3.11\times 10^{-2}$ at $d=50$ for baseline versus SFLI-Gaussian. On the lid-driven cavity flow benchmark at $\mathrm{Re}=3200$, the reported final relative $L^2$ errors were 88.3% without SFLI and 3.75% with SFLI. On a time-marching Navier–Stokes problem in a torus, the reported final vorticity errors were $2.96\times 10^{-1}$ without SFLI and $8.99\times 10^{-2}$ with SFLI [2507.11962].

In medical deep learning, structured orthogonal first-layer initialization is presented as part of a bit-identical training pipeline. The pipeline removes randomness from weight initialization, batch ordering, and GPU kernels, yielding MD5-verified identical trained weights across independent runs. On PTB-XL ECG rhythm classification, structured initialization significantly exceeded Kaiming across two architectures, with Conformer results of $0.959\pm 0.005$ versus $0.953\pm 0.010$ and baseline CNN results of $0.956\pm 0.004$ versus $0.947\pm 0.007$. Aggregate variance was reduced by 2–3x, and rare-class variability was sharply reduced; the TRIGU range was 4.1 percentage points under structured initialization versus 30.9 percentage points under Kaiming. A four-basis comparison at $n=20$ found no significant difference across DCT, Hadamard, Hartley, and sinusoidal bases, with Friedman $p=0.48$, supporting the claim that the effect comes from deterministic structured initialization itself rather than from one privileged basis [2603.28040].

In tabular learning, sparse tree-based initialization uses a trained tree ensemble to shape the first layers of an MLP into a sparse feature extractor. Across 10 tabular datasets, RF or GBDT initialization strictly outperformed random MLP initialization on all datasets except Covertype, where performance was roughly similar. The initialized MLPs were often competitive with SAINT and with strong tree baselines, while preserving the sparse first-layer structure through training [2209.15283].

In Vision Transformers, structured initialization of the first attention layer consistently improved data-efficient training. One study reported on ViT-T that SFLI impulse initialization achieved 91.62 on CIFAR-10, 70.46 on CIFAR-100, 97.23 on SVHN, and 74.40 on ImageNet-1K, compared with 88.63, 66.50, 93.20, and 73.42 for truncated normal initialization, and competitive results relative to mimetic initialization [2404.01139]. A subsequent formulation reported ViT-T gains from 92.29 to 94.67 on CIFAR-10, from 71.67 to 77.02 on CIFAR-100, from 64.60 to 73.18 on Flowers, and maintained competitive ImageNet-1K performance; it also reported gains for Swin Transformer and MLP-Mixer, including 83.14 to 83.55 for Swin-B on ImageNet-1K and 87.00 to 88.78 for MLP-Mixer on CIFAR-10 [2505.19985].

For MLP channel mixing, the first-layer nonzero-mean mimetic scheme improved early training on CIFAR-10 and ImageNet-1K and was complementary to spatial-mixing mimetic initializations. On DeiT-Small for ImageNet-1K, the reported final accuracies over 100 epochs were 68.8% with the proposed MLP initialization and 68.4% for the baseline, described as a small but consistent advantage [2602.07156].

## 6. Limitations, misconceptions, and open problems

The literature is explicit that SFLI is not a universal substitute for conventional initialization. The evidence is often strongest for the first layer only. Linear discriminant initialization did not yield positive results when naively extended to a second layer by applying LDA to the image of data under the first layer [2007.12782]. The MLP mimetic mean-shift paper describes its own effect as much smaller than earlier spatial-mixing mimetic initializations, strongest on small-scale or short-duration training, and diminishing as training proceeds [2602.07156]. The scientific-computing SFLI paper states that extending the $\varepsilon$-rank theory and the associated initialization schemes to convolutional, attention-based, or graph architectures is nontrivial and future work [2507.11962].

A second misconception is that any apparent gain can be reduced to ordinary bias initialization. This is contradicted directly in the mimetic MLP study: changing the linear-layer bias initialization, adding a learnable scalar bias per $W_1$, or making the bias parameter small and constant did not reproduce the effect of shifting the first-layer weight matrix mean [2602.07156]. Conversely, the medical orthogonal-initialization paper shows that the main contribution is not tied to any single analytic basis, since DCT, Hadamard, Hartley, and sinusoidal bases were statistically equivalent in the reported four-basis comparison [2603.28040].

Implementation-specific limitations also recur. Tree-to-MLP translation can suffer from catastrophic cancellation in the exact output-layer formula, and the relaxed $\tanh$-based translation introduces extra hyperparameters [2209.15283]. ViT structured-attention initialization depends on a pseudo-input, usually positional encoding, and is constrained by the number of heads and the fact that only $Q$ and $K$ are initialized structurally, not the value projection or patch embedding [2404.01139]. Sinusoidal and depth-aware methods are often cited in SFLI discussions because their theory is strongly first-layer oriented, yet their actual implementations are layerwise rather than strictly first-layer-only [2505.12909, 2509.05018].

Taken together, these results suggest that SFLI is best viewed not as a single initializer but as a design principle: the first trainable interaction with the input can be engineered to encode discriminative geometry, spectral structure, sparsity, convolutional locality, conditioning, or feature diversity before SGD begins. The practical success of that principle is domain-dependent, but the accumulated evidence indicates that first-layer structure can measurably alter convergence, variance, and generalization without requiring architectural changes [2507.11962, 2603.28040, 2404.01139].

Source: https://www.emergentmind.com/topics/structured-first-layer-initialization-sfli