Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

Abstract: Standard transformer attention computes pairwise token similarity but treats all tokens as equally salient and all positions as equally local, regardless of the informational structure of the input. We identify two complementary inductive biases that standard attention lacks: energy salience (which tokens concentrate informational energy, learned end-to-end without explicit frequency decomposition) and scale-selective locality (how far positional influence extends at each frequency, implemented via Morlet wavelet encoding). We address both with two simple components. Energy-Gated Attention (EGA) gates value aggregation by a learned energy estimate of key token embeddings, computed via a single linear projection; it selects what to attend to. Morlet Positional Encoding (MoPE) replaces fixed sinusoidal encodings with learned Gaussian-windowed wavelets that adapt the joint position-frequency localization to the corpus; it specifies where attention operates at each scale. On TinyShakespeare, EGA alone achieves +0.092 validation loss improvement over standard attention (+0.103 over Phase 1-3 baseline); MoPE alone is -0.032 (below baseline as a standalone encoding); but their combination achieves +0.119 -- more than the sum of parts. This superadditivity, observed across two independent training runs, is the central empirical finding: salience and locality are complementary inductive biases, each addressing a gap the other cannot fill alone. Ablations confirm that structured spectral priors (Morlet wavelet gates, scale-initialized heads, fixed sinusoidal PE) consistently underperform their unconstrained learned counterparts, while complementary learned components interact superadditively. All experiments are at small scale (<=6M parameters, character-level benchmarks, single seed); larger-scale multi-seed validation is the most important direction for future work.

• The paper presents a novel integration of Energy-Gated Attention and adaptive Morlet Positional Encoding that captures both spectral salience and scale-selective locality.
• It employs learned gating mechanisms and adaptive wavelet features to efficiently highlight salient tokens and adjust positional influence dynamically.
• Empirical results on a character-level benchmark show that the combined approach outperforms individual components with minimal parameter overhead (<0.3%), validating its superadditive benefits.

Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention

Motivation and Theoretical Background

Transformer attention mechanisms, as formalized by Vaswani et al., compute similarity between query and key embeddings but do not distinguish inherently informative tokens or adapt how far positional influence extends across linguistic scales. This paper proposes two complementary inductive biases: spectral salience and scale-selective locality. Spectral salience models which tokens concentrate informational energy, implemented via learned gating (EGA), while scale-selective locality determines the positional extent of influence at each frequency, addressed via adaptive Morlet wavelet positional encoding (MoPE).

The cross-correlation interpretation adopted frames standard dot-product attention as a zero-lag estimate, discarding lag structure and spectral salience. Theoretical analysis reveals that learned energy gates and adaptive locality interact multiplicatively, not additively, leading to superadditive improvements in performance.

Energy-Gated Attention (EGA)

EGA introduces a learned gate over value aggregation, projecting token embeddings onto a learned direction to estimate local energy. This process entails linear projection, normalization, sigmoid gating, and renormalization of attention weights. The mechanism directly selects salient tokens—those carrying disproportionate informational density—such as morphological boundaries, syntactic heads, or discourse markers. EGA is causally implemented, adds negligible parameter overhead ($<0.3\%$), and theoretically aligns with Parseval's identity, supporting framing as spectral energy estimation.

Morlet Positional Encoding (MoPE)

MoPE replaces fixed sinusoidal encoding (sin/cos PE) with adaptive Gaussian-windowed wavelet representations. Each dimension learns its center frequency and locality bandwidth, providing joint localization in position and frequency:

$$MoPE(b, 2i) = \cos(\omega_i b) e^{-b^2 / 2 \sigma_i^2}, \quad MoPE(b, 2i+1) = \sin(\omega_i b) e^{-b^2 / 2 \sigma_i^2}$$

MoPE strictly generalizes sin/cos PE ($\sigma_i \to \infty$) and connects theoretically to RoPE and ALiBi as specific limiting cases. The cross-correlation kernel from MoPE exhibits optimal time-frequency localization per the Heisenberg bound, supporting persistence and periodicity properties needed for hierarchical linguistic relationships.

Empirical Evaluation and Main Results

Application of EGA and MoPE, individually and in combination, was benchmarked on character-level TinyShakespeare (6M parameters). The combination (EGA-MORLET) yielded $+0.119$ validation loss improvement over standard dot-product attention, exceeding the sum of its independent components ($+0.060$), thus confirming the complementarity and superadditive interaction hypothesized.

MoPE alone performed below baseline ($-0.032$), whereas EGA alone improved substantially ($+0.092$). The only models substantially above baseline were EGA and EGA-MORLET.

Figure 1: Validation loss curves and final benchmark results, demonstrating fastest convergence and lowest loss for EGA-MORLET and EGA-1.

Ablations reveal that nonzero lags in attention (convolutional variants) marginally improve performance, and that unconstrained learned projections outperform structured spectral priors (Morlet energy gates, Daubechies DWT, scale-initialized heads, sin/cos PE). Specifically, phase coherence gating led to negative or neutral outcomes, and spectral flux gating, operating as a boundary detector, delivered minor improvements.

Learned MoPE parameters clustered around distinct temporal scales—character, word, clause, sentence—highlighting emergent scale specialization from corpus data without explicit prior hierarchy.

Implications and Future Directions

The results indicate that introducing spectral salience and adaptive locality into attention mechanisms yields empirically and theoretically robust improvements, particularly via superadditivity. The findings advocate for unconstrained, corpus-specific learning strategies at small scale, with structured priors only aiding when providing orthogonal, non-redundant inductive bias.

Practical implications include the prospect of efficient, data-dependent sparse attention schemes and scale-specialized context windows, pertinent for long-sequence models and hierarchical tasks. The adaptability of MoPE locality parameters suggests high potential for models operating at extended contexts ($T \geq 4096$), where context-range specialization can be exploited.

The spectral cascade observed—progressive coarsening of spectral content across transformer layers—aligns with multiscale filter bank interpretations and opens avenues for mechanistic interpretability grounded in operational spectral analysis.

Limitations and Prospects

Experimental limitations include reliance on character-level benchmarks, restricted parameter count, and single-seed runs. Effect sizes for secondary ablations are preliminary and warrant multi-seed confirmation. Besides scaling EGA-MORLET to word-level datasets and larger models (50M–100M parameters), developing RoPE-compatible MoPE variants for mainstream deployment, and deeper interpretability studies into spectral salience correlation with DFT-computed energy, are foundational next steps.

Conclusion

The study delineates standard transformer attention's deficits in modeling spectral salience and scale-selective locality. By integrating EGA and MoPE, transformer attention is augmented to select both what matters (salience) and where it matters (locality), yielding superadditive gains and empirically supporting complementarity. Empirical ablations underscore the inefficacy of structured spectral priors at small scale, except where they address genuine gaps in unconstrained learning. Theoretical and practical implications point to scalable, interpretable, and efficient modifications of attention, with recommendations for future validation and expansion.