Moment-Aware Initialization

• Moment-aware initialization is a set of strategies that use statistical moments to control signal propagation and prevent vanishing or exploding outputs in deep networks.
• These methods encompass fractional moment-preserving techniques, angle-calibrated updates in optimization, and architectures for temporal and multi-modal alignment.
• Empirical findings show that moment-aware approaches enhance convergence, generalization, and performance in tasks like video-text retrieval and segmentation.

Moment-aware initialization refers to a collection of principled initialization strategies and training frameworks that adaptively utilize statistical properties—moments of order $s$ of network outputs, gradients, or feature representations—at either the architectural or algorithmic level. These approaches explicitly target stability, expressiveness, and inductive bias in deep learning systems, with applications in network parameter initialization, adaptive optimization, and multi-modal temporal alignment.

1. Fractional Moment-Preserving Initialization in Deep Neural Networks

The fractional moment-preserving initialization framework generalizes classical weight initialization methods by conditioning the propagation of the $s$-th moment ($0 < s \leq 2$) of network outputs across layers. Given a fully connected feed-forward network with $d$ inputs, the $s$-th moment of the output at layer $k$ is defined as:

$M_s^{(k)} = \mathbb{E}\left[ \|x^{(k)}\|^s \right]$

The initialization scale, usually the standard deviation $\sigma_d$ of weights, is chosen to ensure the “fractional moment ratio”

$$r(s) = \mathbb{E}\left[\frac{\|x^{(1)}\|^s}{\|x^{(0)}\|^s}\right] = \sigma_d^s \sum_{n=1}^d p(n) m(n,s/2)$$

remains close to unity for a prescribed $s$. Here, $p(n)$ is the probability that exactly $n$ neurons contribute (binomial for ReLU networks), and $m(n,s/2)$ is the $s/2$-th moment of the chi-square distribution:

$$m(n, s/2) = 2^{s/2} \frac{\Gamma\left(\frac{n}{2} + \frac{s}{2}\right)}{\Gamma\left(\frac{n}{2}\right)}$$

Solution of $\sigma_d$ for a desired $s$:

$$\sigma_d = \frac{1}{\left(\sum_{n=1}^d p(n) m(n,s/2)\right)^{1/s}}$$

This setting recovers classical variance-based schemes (e.g., $\sigma_d^2 = 2/d$ for ReLU) as $s\rightarrow 2$.

These initializations guarantee that neither vanishing nor explosive output propagation arises for networks with heavy-tailed weights or pre-activations, and can produce stationary output distributions admitting finite lower-order moments even if higher ones (e.g., variance) diverge under heavy-tailed conditions. The propagation of signals is controlled by the top Lyapunov exponent $\rho$, and the limiting output norm’s logarithm converges in law to a Gaussian with explicit mean and variance determined by $s$, $\sigma_d$, and the activation function.

2. Alternative Moment-Aware Strategies in Optimization Algorithms

Moment-aware principles extend to optimization via schemes such as angle-calibrated moment methods. Adaptive optimizers (Adam, RMSProp) often rely on second-moment ($v_t$) statistics, leading to coordinate-wise learning rate adaptation. ACMo (“Angle-Calibrated Moment method”) reframes moment-awareness as penalizing unwarranted projection of descent directions onto past gradients, modulating the first-moment update direction to preserve acute angles between the current and aggregated historical gradients:

$$\theta_{t+1} = \theta_t - \alpha_t \left[ g_t + \beta_t \frac{\|g_t\|}{\|m_{t-1}\| + \delta_t} m_{t-1} \right]$$

with $g_t$ as the current gradient, $m_{t-1}$ as aggregate past direction, $\beta_t$ momentum hyperparameter. This achieves competitive convergence and generalization by encoding moment information in the directional update itself, harmonizing benefits of SGD and Adam without explicit second-moment tracking (Huang et al., 2020).

Further, proper initialization of second-moment estimates in adaptive optimizers is critical for stable early updates. Initializing $v_0$ at zero (as in standard Adam) causes the first update to collapse into sign-descent, resulting in unstable and potentially excessive parameter changes:

$\Delta\theta_1 = -\alpha \cdot \mathrm{sign}(g_1)$

Nonzero moment-aware initializations (via data-driven statistics or random draws) stabilize the early learning rate denominator, yielding controlled, magnitude-sensitive updates and reducing drift in moment statistics, substantially improving convergence and generalization (Abuduweili et al., 3 Dec 2024).

3. Moment-Aware Initialization in Temporal and Multi-Modal Deep Learning

Recent advances in video-language understanding, segmentation, and retrieval apply moment-awareness at the architectural and supervision levels.

Event-based and Moment-Guided Initialization: In EventFormer for video corpus moment retrieval, model representations are pre-initialized in a two-level hierarchy—frames aggregated into events—allowing encoding of temporally coherent “moments”:

• Contextualized frame features are aggregated into event units using self-similarity convolution, clustering, or window segmentation.
• Anchor multi-head self-attention focuses on local neighborhoods to promote moment delineation, enforcing sensitivity to temporal context during initialization and training.

Contrastive two-branch learning forces encoder outputs for event and frame branches to be discriminative in relation to moment boundaries, effectively producing “moment-aware” representations primed for retrieval and localization tasks (Hou et al., 21 Feb 2024).

Moment-aware video-text alignment and segmentation: The SAMDWICH framework in referring video object segmentation applies moment-aware annotation and supervision:

• Temporal moments ($M_i$) for each object are explicitly annotated and only frames with active object-expression alignment are used for training (selective supervision).
• Moment-guided dual-path propagation (MDP) separates features into text-relevant and irrelevant streams, propagating memory bank updates only from relevant moments.
• OSS strategy filters out masks for non-aligned objects, reducing semantic noise in learning (Lee et al., 16 Aug 2025).

This results in segmentation models that are robust to irrelevant frames, show improved object grounding, and achieve superior temporal alignment and performance, as shown by joint metric scores on the MeViS-M dataset.

4. Theoretical Analysis: Heavy-Tailed Behavior and Moment Control

Fractional moment-aware initialization provides a theoretical lens for understanding signal propagation and heavy-tailed effects in deep networks. By controlling fractional moments, stable stationary distributions for network outputs can be established even where variance diverges. The explicit calculation of the limiting distribution, the Lyapunov exponent, and the stationary measure parameters yields precise control over when and how heavy tails originate:

• For $\rho < 0$, stationary distribution exists and the output norm converges almost surely.
• If the initialization scale $\sigma_d$ is tuned above critical threshold $(1/d + c/d^2)$ with $c>1$, iterates are shown to diverge.
• Explicit means, variances, and distributional forms (Gaussian, heavy-tailed) can be calculated for different activation functions and moment orders.

This suggests that fractional moment-awareness offers flexible inductive bias selection at initialization, enabling practitioners to tailor networks for robust signal propagation according to data characteristics or architectural depth (Gurbuzbalaban et al., 2020).

5. Empirical Observations and Applications

Moment-aware schemes demonstrate quantifiable improvements in experimental settings:

• Fractional moment-preserving initializations avoid exponential blow-up or decay (vanishing outputs) in deep feed-forward or convolutional networks and yield higher test performance compared with standard initializations.
• In optimization, ACMo and nonzero moment-initialized Adam achieve convergence rates and generalization on par with state-of-the-art methods, but often with reduced memory overhead and greater stability.
• In temporal video understanding, event/moment-aware architectures and training strategies produce improved retrieval, alignment, and segmentation metrics, evidenced by state-of-the-art results on TVR, ANetCaps, DiDeMo, and MeViS benchmarks.

6. Comparison to Classical Approaches and Further Generalizations

By generalizing initialization to arbitrary moment order $s$, fractional schemes recover variance-preserving (Kaiming/He/Xavier) methods in the $s = 2$ limit but allow for more expressive initial distributions. This flexibility lends itself to exploration of heavy-tailed regimes, adaptivity in signal scaling, and robustness to layer depth. Additionally, the moment-awareness concept is extensible:

• To other distributions (e.g., symmetric $\alpha$-stable, Laplace, Weibull) by appropriate scale adjustment for moment control.
• To architectural design and supervision, where temporal, semantic, or segment-level structures can be encoded in initialization or early representation layers, as shown in multi-modal alignment and segmentation tasks.

A plausible implication is that broader incorporation of moment-aware principles—as both statistical initialization and strategic architecture/training design—may yield further progress in stabilizing and enriching deep learning model expressiveness and efficiency across domains.