Morlet Positional Encoding (MoPE)

• MoPE is a wavelet-based positional encoding that replaces fixed sin/cos features with learnable frequency and Gaussian locality parameters.
• It generalizes sinusoidal and RoPE encodings by adding a Gaussian envelope, enabling scale-selective locality for varied sequence patterns.
• MoPE integrates efficiently into transformer architectures, offering complementary benefits when combined with Energy-Gated Attention.

Searching arXiv for the cited MoPE papers and closely related references. Morlet Positional Encoding (MoPE) is a wavelet-based positional encoding for transformer architectures in which each embedding-dimension pair learns both a characteristic frequency $\omega_i$ and a locality bandwidth $\sigma_i$, producing Gaussian-windowed sinusoidal features that are localized jointly in position and frequency. In the formulation introduced by Zeris, MoPE replaces fixed, nonlocal sin/cos features with learned Morlet-like atoms, thereby making the spatial extent of positional influence a learned property rather than a fixed design choice. The encoding strictly generalizes sinusoidal positional encoding through the $\sigma_i \to \infty$ limit, and its phase recovers the RoPE rotation angle, while its Gaussian amplitude adds a locality prior absent from standard sin/cos PE and RoPE (Zeris, 25 May 2026, Zeris, 31 May 2026).

1. Conceptual role in transformer position modeling

MoPE was proposed to address what the underlying papers describe as a missing inductive bias in standard attention: locality, and specifically scale-selective locality. Standard sin/cos positional encoding assigns fixed frequencies to embedding dimensions without a spatial envelope, so every position contributes globally at every scale. RoPE encodes relative position via rotations, but it likewise uses fixed, nonlocal frequencies. In this framing, MoPE answers the question of where positional information should matter, and at what scale (Zeris, 25 May 2026).

The motivating claim is that real sequences contain phenomena operating over distinct positional ranges. Character-level patterns are highly local, whereas clause- and sentence-level dependencies may span tens to hundreds of tokens. MoPE is intended to let each embedding dimension learn both a frequency and the range over which that frequency remains informative. This gives each dimension its own notion of “how far” positional influence extends.

The wavelet basis is central to that motivation. The Morlet, or Gabor, atom is a Gaussian-windowed sinusoid, and the papers characterize it as achieving minimum-uncertainty localization in position and frequency within the Gaussian window class. This makes it a natural mechanism for injecting locality without discarding oscillatory phase structure. In the broader architecture proposed in the companion work, MoPE supplies the locality inductive bias, while Energy-Gated Attention (EGA) supplies salience, or “what to attend to” (Zeris, 31 May 2026).

2. Formal definition and induced kernel

In real form, for token position $b$ and embedding dimension pair $(2i,2i+1)$, MoPE is defined as

$$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$

$$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$

Here $\omega_i$ and $\sigma_i$ are learned per embedding-dimension pair. The implementation uses real-valued features; no complex arithmetic is required. For analysis, the same object is also written in complex form as

$$\text{MoPE}(b, i) = e^{i\omega_i b}\cdot e^{-b^2/2\sigma_i^2},$$

which makes the phase–amplitude decomposition explicit: the phase is $\sigma_i$0, and the amplitude is the Gaussian envelope $\sigma_i$1 (Zeris, 31 May 2026).

A key limiting case is

$\sigma_i$2

with the analogous result for the sine channel. This establishes sinusoidal PE as a degenerate, nonlocal case of MoPE. The same limit also removes the Gaussian amplitude from the correlation structure, yielding the RoPE phase factor (Zeris, 25 May 2026).

The lag-space structure induced by MoPE is central to its interpretation. In the same-scale autocorrelation view, ignoring boundary effects, the kernel has the Morlet-like form

$\sigma_i$3

The first factor encodes locality through Gaussian decay in lag $\sigma_i$4; the second encodes periodicity at frequency $\sigma_i$5. Fine scales, with small $\sigma_i$6, decay quickly and emphasize local structure. Coarse scales, with large $\sigma_i$7, decay more slowly and can represent longer dependencies (Zeris, 25 May 2026).

The more exact cross-correlation written in the second paper is

$\sigma_i$8

Under the paper’s heuristic position-averaging assumptions, this approximately factorizes into the RoPE phase term multiplied by a Gaussian locality kernel in $\sigma_i$9. This suggests that MoPE can be interpreted as RoPE-like phase behavior with learned distance-sensitive attenuation (Zeris, 31 May 2026).

3. Admissibility, parameterization, and optimization constraints

MoPE is grounded in the complex Morlet atom

$\sigma_i \to \infty$0

which is described as approximately admissible when $\sigma_i \to \infty$1. The admissibility discussion is operationalized through the product constraint

$\sigma_i \to \infty$2

One paper describes this as enforcing an “admissibility minimum” during training; the other expresses it through a forward-pass projection

$\sigma_i \to \infty$3

which defines the wavelet admissibility boundary $\sigma_i \to \infty$4 in the $\sigma_i \to \infty$5 plane (Zeris, 25 May 2026, Zeris, 31 May 2026).

Both $\sigma_i \to \infty$6 and $\sigma_i \to \infty$7 are stored in log space to ensure positivity. The implementation uses one frequency and one bandwidth per sin/cos pair, giving $\sigma_i \to \infty$8 learned frequencies and $\sigma_i \to \infty$9 learned bandwidths for a $b$0-dimensional positional vector. The granularity is per embedding dimension rather than per head or per layer. Because the projection matrices for $b$1 and $b$2 split embedding dimensions across heads, dimension-local MoPE affects each head that draws on those dimensions (Zeris, 25 May 2026).

Initialization is dyadic-like: frequencies are spaced up to approximately $b$3, and bandwidths are initialized as $b$4 so that $b$5 at the start of training. The papers treat this boundary as both an admissibility device and a structural prior. They also note that projection-based enforcement may bias optimization, and explicitly caution that unconstrained or penalty-based validation is needed to determine whether boundary saturation is intrinsic or an artifact of the clamp (Zeris, 31 May 2026).

The Heisenberg interpretation follows directly from the Gaussian window:

$b$6

Within the Gaussian class, this is the minimum uncertainty product. The significance attributed to this fact is not that MoPE guarantees better downstream performance, but that it supplies a principled basis for trading positional localization against frequency localization (Zeris, 31 May 2026).

4. Integration into attention and relation to existing positional schemes

MoPE is used as an absolute positional encoding added directly to token embeddings:

$b$7

Queries, keys, and values are then computed in the usual way,

$b$8

Its effect on attention is therefore indirect: it modifies $b$9 and $(2i,2i+1)$0 through the added positional vector, but it does not directly alter value aggregation and does not apply rotary operations. The implementation is explicitly additive (Zeris, 25 May 2026).

The computational cost is modest. Computing MoPE features costs $(2i,2i+1)$1 per forward pass, comparable to fixed sin/cos PE, with the incremental overhead arising from learned parameters and the Gaussian envelope. The papers state that this overhead is negligible relative to attention’s $(2i,2i+1)$2 cost. Parameter growth is also small: MoPE adds $(2i,2i+1)$3 learnable parameters (Zeris, 25 May 2026).

The papers place MoPE in direct relation to three standard positional mechanisms. First, sinusoidal PE is the $(2i,2i+1)$4 limit of MoPE. Second, RoPE is recovered at the phase level in the same limit: the MoPE phase difference between positions $(2i,2i+1)$5 and $(2i,2i+1)$6 is $(2i,2i+1)$7, exactly matching the RoPE rotation angle when $(2i,2i+1)$8 is identified with RoPE’s $(2i,2i+1)$9. However, the additive MoPE mechanism is not equivalent to RoPE’s rotational composition in $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$0 space. Third, ALiBi is described as corresponding qualitatively to the zero-frequency limit $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$1, i.e., a pure locality envelope without oscillation (Zeris, 25 May 2026, Zeris, 31 May 2026).

A common misconception is therefore that MoPE is merely RoPE with a different initialization or merely sinusoidal PE with learnable frequencies. The papers argue for a stronger claim: MoPE preserves sinusoidal phase structure while adding a learned Gaussian locality kernel. Its novelty lies in the amplitude term, not in replacing oscillatory phase with a different phase law (Zeris, 31 May 2026).

5. Complementarity with Energy-Gated Attention and empirical results

The empirical setting reported for MoPE is small-scale: a GPT-style decoder with $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$2 layers, $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$3 heads, $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$4, context length $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$5, trained for 5,000 steps with AdamW and cosine learning-rate decay starting at $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$6 on character-level TinyShakespeare, with identical mini-batches across models and total parameter count at or below 6M (Zeris, 25 May 2026).

In that setting, MoPE by itself does not outperform the main baseline, whereas its combination with Energy-Gated Attention does. The core interpretation is that MoPE supplies adaptive locality and EGA supplies salience; the papers treat these as complementary inductive biases that affect different parts of the computation. MoPE alters the position-aware content of $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$7, while EGA reweights which tokens contribute during aggregation (Zeris, 25 May 2026).

The reported validation losses are as follows:

Model	Validation loss	$$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$8 vs BASE-DOT
BASE-DOT	1.4742	reference
PE-SINCOS	1.5863	-0.112
PE-ROPE	1.4637	+0.011
PE-MORLET	1.5060	-0.032
EGA-1	1.3821	+0.092
EGA-MORLET	1.3550	+0.119

Two empirical points are emphasized. First, MoPE alone underperforms BASE-DOT by $$MoPE(b, 2i) = \cos(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}$$9, although it outperforms fixed sin/cos by $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$0. Second, the combination EGA+MoPE improves over BASE-DOT by $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$1, exceeding the sum of the isolated effects of EGA and MoPE, $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$2, by an excess of $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$3. The first paper identifies this superadditivity as the central empirical finding and states that it was observed across two independent training runs; the second paper presents the same interaction as evidence that salience and locality are complementary (Zeris, 25 May 2026, Zeris, 31 May 2026).

The EGA component is described in the first paper through a learned energy projection, z-normalization, logistic gating, and renormalization of attention weights. The companion MoPE paper cites the EGA mechanism as originating in Zeris (2026a) and does not reproduce the gating equations, but consistently uses the “EGA-1 energy gate” configuration (Zeris, 21 May 2026, Zeris, 31 May 2026).

6. Learned parameter behavior, limitations, and proposed extensions

A striking empirical observation from the second paper is that all 128 learned frequency–bandwidth pairs satisfy $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$4 exactly after training, meaning that every dimension saturates the admissibility boundary. The learned ranges are reported as $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$5 tokens and $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$6, implying that the learned positional capacity concentrates at character-to-short-word scale in the TinyShakespeare setting. Both sine and cosine channels receive non-zero downstream weights, which the paper interprets as evidence that the model uses the full quadrature representation (Zeris, 31 May 2026).

The same papers, however, explicitly warn against overinterpreting these findings. Most results are single-seed, all experiments are at small scale, and effects of at most $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$7 are described as preliminary. MoPE alone slightly hurts at this scale, and RoPE remains the strongest structured positional encoding when used alone. The first paper also reports that hand-crafted spectral structures, including Morlet energy gates, Daubechies DWT gates, scale-initialized heads, and fixed sin/cos PE, consistently underperform unconstrained learned counterparts (Zeris, 25 May 2026).

Several limitations are structural rather than merely empirical. Because the Gaussian envelope is centered at the sequence origin,

$$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$8

MoPE introduces an absolute origin prior, so tokens near $$MoPE(b, 2i+1) = \sin(\omega_i b) \cdot e^{-b^2/2\sigma_i^2}.$$9 have higher positional amplitude. The second paper recommends a natural generalization with learned centers,

$\omega_i$0

to mitigate origin bias, but notes that this extension was not evaluated (Zeris, 31 May 2026). The first paper similarly notes possible boundary effects and suggests that anchoring or sliding windows may be worth considering in longer contexts or different segmenting schemes.

Future work proposed in the two papers includes larger multi-seed validation, word-level benchmarks such as WikiText-103 and OpenWebText, and a RoPE-compatible “Morlet-RoPE” variant that would preserve relative-position rotation while adding learned Gaussian locality (Zeris, 25 May 2026, Zeris, 31 May 2026). A plausible implication is that MoPE’s benefits may become more pronounced when long contexts and stronger positional non-stationarity make fixed nonlocal encodings less well matched to the data. The papers do not claim that this has been established; they present it as the next empirical question.