Attention-Inspired Initialization

• Attention-Inspired Initialization is a strategy that sets neural network weights with deliberate, attention-based structures to facilitate robust convergence and enhanced performance.
• Methods like Structured Convolutional Impulse Initialization and Mimetic Initialization embed convolutional and pretrained biases, improving signal propagation and learning dynamics.
• Empirical studies show these techniques yield significant gains in accuracy and stability, particularly in transformers, state space models, and 3D reconstruction tasks on small or noisy datasets.

Attention-Inspired Initialization encompasses a class of initialization techniques for neural architectures—especially those with self-attention or related mixing mechanisms—where parameters are deliberately set to realize functional properties observed in attention-based models. These mechanisms differ from generic random or purely variance-scaled initializations by injecting structure (e.g., local convolutional patterns, identity-like weight products, or mimetic copying of pretrained statistics) into the initial parameter states. This facilitates robust convergence, mitigates early-stage optimization pathologies, and frequently yields superior performance when training from scratch on small or noisy datasets. The following sections catalog methodologies, theoretical rationales, and experimental validations across Transformers, state space models, object-centric architectures, and structured 3D reconstruction.

1. Structured Convolutional Impulse Initialization in Vision Transformers

Structured Convolutional Impulse Initialization (SCII) sets the query and key matrices in ViT attention modules such that, prior to any data exposure, the corresponding attention map approximates a spatial convolution with block-circulant impulse filters. This is operationalized by:

• Constructing a pseudo-input $P \in \mathbb{R}^{N \times D}$ using sinusoidal positional encodings.
• Defining a convolution matrix $H_{\text{impulse}} \in \{0,1\}^{N\times N}$ that encodes a local $f \times f$ receptive pattern.
• Optimizing $Q_{\text{init}}, K_{\text{init}}$ to minimize the discrepancy

$$\frac{1}{N^2}\| H_{\text{impulse}} - \mathrm{softmax}(\sigma P Q K^T P^T) \|_F^2$$

where $\sigma = 1/\sqrt{D/h}$ is attention scaling.

SCII introduces convolutional inductive bias without modifying ViT architectures or training pipelines. Experimental results demonstrate 2–4% absolute improvements in accuracy on CIFAR-10, CIFAR-100, and SVHN (e.g., $91.62\%$ vs.\ $88.63\%$ on CIFAR-10 for SCII vs.\ standard TruncNormal), while retaining scalability for ImageNet (Zheng et al., 2024). Ablations reveal stronger gains with increased attention head count and confirm that the pseudo-input choice—specifically pure sinusoidal encodings—is critical for optimal accuracy. Attention map visualizations post-SCII maintain prescribed local structures in early layers, in contrast to randomly initialized models.

2. Mimetic Initialization of Self-Attention and State Space Models

Mimetic Initialization entails analytically crafting weight matrices such that the inner products $(W_q W_k^\top)$ and $(W_v W_{\text{proj}}^\top)$ emulate empirically observed patterns in pretrained models: the former approximates the identity while the latter targets the negative identity. Explicitly:

• For each self-attention head, derive $M_{qk} = \alpha_1 Z_1 + \beta_1 I_k$ and $M_{vp} = \alpha_2 Z_2 - \beta_2 I_d$, with $Z_1, Z_2$ Gaussian noise.
• Use SVD to factor $M_{qk}$ and $M_{vp}$ into $W_q, W_k, W_v, W_{\text{proj}}$ respectively.

Default hyperparameters ($\alpha_1 = \beta_1 = 0.7$, $\alpha_2 = \beta_2 = 0.4$) yield near-diagonal attention and robust signal propagation. In state space models (SSMs) such as Mamba, mimetic initialization parametrizes transition and input gains ($A, W_\Delta, b_\Delta$) such that the recurrence is nearly $h_{t+1} = h_t + W_B x_t$ and $W_C^\top W_B \approx I$, aligning the implicit mixing matrix with linear attention (Trockman et al., 2023, Trockman et al., 2024).

These techniques dramatically accelerate convergence and increase final accuracy, particularly on small datasets. For instance, ViT-Tiny with mimetic initialization achieves $90.78\%$ on CIFAR-10 (cf.\ $86.07\%$ baseline), $71.92\%$ on ImageNet (cf.\ $67.80\%$). SSMs initialized mimetically exhibit improved copy/recall generalization, matching hybrid architectures with added attention blocks.

3. Effective Theory and Criticality-Driven Scaling

Effective-theory analysis prescribes initialization variance scalings and learning-rate groupings to maintain order-one signal propagation and stable gradients in deep, wide Transformers. Attention weights ($Q, K, V, U$) are initialized with variances proportional to inverse model dimension and head count; e.g., $\text{Var}(Q) = 1/d$, $\text{Var}(U) = H/d$, with LayerNorm parameters (scale=1, bias=0) and positional embeddings (variance=1). For AdamW, per-parameter learning rates are adjusted to maintain an $O(1)$ Neural Tangent Kernel (Dinan et al., 2023).

Empirical validation on large-scale ImageNet, BART, and R2C2 models confirms that criticality-driven initializations yield smoother loss landscapes and modest robustness gains without modifying model architecture.

4. Object-Centric Aggregation via Attention-Guided Re-Initialization

Slot Attention models initialize $K$ object-level "slots" $S^0$ typically from learned Gaussian or random draws, then iteratively update these slots via competitive cross-attention over $T$ rounds. Redundant slots—detected by clustering assignment masks—are removed after warmup iterations. Remaining slots are re-initialized for an additional attention pass, allowing survivors to "re-compete" for feature ownership without interference. The attention mask for discarded slots is zeroed via logit masking.

Further, self-distillation aligns the initial attention map ($A_a^1$) to the improved final attention ($A_a'$) using cross-entropy loss after index matching. This strategy yields cleaner object segmentations and improves object discovery metrics (ARI, mIoU, classification accuracy) relative to naive slot reuse (Zhao et al., 31 Jul 2025).

5. Geometric and Appearance Attention for Initialization-Free 3D Reconstruction

AttentionGS eliminates reliance on high-quality SfM point clouds for 3D Gaussian Splatting by initializing scene Gaussians with large variances and random positions, then steering their evolution using a pair of attention-inspired modules:

• Geometric attention computes per-pixel weights from edge maps, focusing gradient updates toward global structure and boundaries via weighted photometric loss.
• Appearance attention weighs color residuals per pixel-channel, refining fine detail once coarse geometry emerges.

A sigmoid scheduler modulates the predominance of geometry vs.\ appearance loss over training. Opacity-weighted gradients further guide densification, ensuring that foreground Gaussians are split preferentially. AttentionGS achieves substantial performance gains over baseline initialization (e.g., +3.84 dB PSNR on Mip-NeRF 360, +12.5 dB on LLFF), with particular robustness under sparse or texture-deficient conditions (Liu et al., 30 Jun 2025).

6. Theoretical Characterization of Attention-Inspired Initialization in In-Context Learning

Recent analysis reveals that in linear regression-based in-context learning (ICL), standard multi-head linear self-attention (LSA) approximates one-step gradient descent only under highly restrictive initializations (e.g., zero mean priors). By augmenting the query embedding with a trainable initial guess $y_q$, the $y_q$-LSA variant closes the performance gap even when priors are non-zero mean:

• Embedding extended as $E_w = [X^\top \; x_q; y^\top \; w^\top x_q]$.
• Prediction $f_{y_q\text{-LSA}}(X, y, x_q) = f_{LSA}(E_w)$.

It is proven that $d+1$ heads suffice for full expressivity in LSA, yet when the prior mean is nonzero, multi-head LSA loss is strictly greater than that of gradient descent; $y_q$-LSA restores equivalence (Xie et al., 3 Dec 2025). Empirically, $y_q$-LSA matches true GD in risk and converges identically on synthetic linear tasks. Extension to LLMs using semantic similarity prompts also demonstrates consistent mean squared error improvements when initial guess tokens are incorporated.

7. Limitations and Scope

Attention-inspired initialization confers pronounced benefits in matching inductive biases of successful pretrained or architecturally constrained models when training from scratch, especially on small-scale, data-scarce, or noisy environments. Gains diminish in regimes with extended training or heavy augmentation, and are sensitive to the choice of positional encodings and initialization patterns. Architectures and tasks where attention map structure (e.g., diagonal, local, negative/positive identity in weight products) is less informative may show reduced impact. The technique is generally learning-free and architecturally compatible, but application to non-Transformer models (e.g., state space, object-centric, 3D scene synthesis) requires domain-specific adaptations.

A plausible implication is that, as the theoretical foundations of initialization align more closely with architectural and data priors, optimization bottlenecks in novel sequence architectures, compositional inference, and memory-rich problems may be dynamically mitigated, enabling the development of increasingly expressive, data-efficient models without recourse to extensive pretraining.