Weierstrass Elliptic Positional Encoding (WEF-PE)

• WEF-PE is a mathematically principled positional encoding that maps 2D patch coordinates to the complex plane using the Weierstrass elliptic function.
• It leverages doubly periodic and nonlinear geometric properties to inject explicit 2D locality priors and maintain resolution invariance.
• Empirical evaluations on CIFAR-100 and ImageNet show significant accuracy improvements over traditional absolute and Fourier-based positional encodings.

Weierstrass Elliptic Function Positional Encoding (WEF-PE) is a mathematically principled positional encoding framework for vision transformers (ViTs) that directly encodes two-dimensional spatial coordinates using the Weierstrass elliptic function $\wp(z;\omega_1, \omega_2)$ and its analytic structure in the complex domain. Unlike traditional one-dimensional or non-geometric position encodings, WEF-PE leverages the doubly periodic and nonlinear geometric properties of elliptic functions to inject explicit 2D locality priors, ensuring monotonic correspondence between Euclidean and representation distances, and enabling seamless adaptation to differing image resolutions without interpolation artifacts (Xin et al., 26 Aug 2025).

1. Mathematical Foundations

1.1. Weierstrass Elliptic Function and Lattice Construction

Let $\omega_1, \omega_2 \in \mathbb{C}$ be two $\mathbb{R}$-linearly independent half-periods, generating the period lattice

$$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$

The Weierstrass elliptic function is defined as

$$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$

rendering it uniquely meromorphic and doubly periodic with fundamental periods $2\omega_1,\, 2\omega_2$ (Definition 2.4, Eq. 2.6). The function and its derivative $\wp'(z)$ satisfy the classical elliptic curve differential equation:

$$\left(\wp'(z)\right)^2 = 4\,\wp(z)^3 - g_2\wp(z) - g_3$$

(Theorem 2.2, Eq. 2.7).

1.2. Algebraic Addition Formula and Relative Position Algebra

For any $z_1, z_2$ not differing by a lattice vector, the addition formula holds:

$$\wp(z_1 + z_2) = -\wp(z_1) - \wp(z_2) + \frac{1}{4} \left( \frac{\wp'(z_1) - \wp'(z_2)}{\wp(z_1) - \wp(z_2)} \right)^2$$

(Theorem 2.3, Eq. 2.8). In WEF-PE, this enables algebraic recovery of relative encoding $\omega_1, \omega_2 \in \mathbb{C}$0 from the absolute encodings at $\omega_1, \omega_2 \in \mathbb{C}$1 and $\omega_1, \omega_2 \in \mathbb{C}$2 without a secondary lookup table.

1.3. Geometric and Invariance Properties

The functional form $\omega_1, \omega_2 \in \mathbb{C}$3 guarantees double periodicity:

$\omega_1, \omega_2 \in \mathbb{C}$4

enforcing translation invariance corresponding to shifting grid indices $\omega_1, \omega_2 \in \mathbb{C}$5. Continuity and the meromorphic nature of $\omega_1, \omega_2 \in \mathbb{C}$6 support evaluation at any complex location, implying strict resolution invariance for positional encodings.

2. Construction of WEF-PE Positional Features

2.1. Mapping Patch Coordinates to Complex Plane

Given a $\omega_1, \omega_2 \in \mathbb{C}$7 patch grid, patch coordinates $\omega_1, \omega_2 \in \mathbb{C}$8 are normalized:

$\omega_1, \omega_2 \in \mathbb{C}$9

A linear isomorphism $\mathbb{R}$0 maps $\mathbb{R}$1 to

$\mathbb{R}$2

(Eq. 3.2), where $\mathbb{R}$3 are learnable scale factors and $\mathbb{R}$4 is a half-period parameter.

2.2. Feature Extraction and Compression

The vector of raw features is

$\mathbb{R}$5

Optionally, each channel $\mathbb{R}$6 undergoes adaptive compression via:

$\mathbb{R}$7

(Eq. 3.8), followed by concatenation to yield a 4D feature vector.

2.3. Implementation Variants

• From-scratch: Direct lattice summation truncated to $\mathbb{R}$8, with parameter learning on $\mathbb{R}$9 and $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$0.
• Fine-tuning: Replaces series by a Fourier-like expansion:

$$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$1

(Eq. 3.12), with learnable $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$2.

2.4. Projection and Model Integration

A linear map (or shallow MLP) $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$3 and LayerNorm project $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$4 to match the ViT's embedding dimension:

$$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$5

(Eq. 3.14), plus the special $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$6 token embedding.

3. Key Theoretical Properties

3.1. Monotonic Distance-Decay

For two patches at grid distance $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$7, let $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$8 be their WEF-PE vectors in $$\Lambda = \{ 2m\omega_1 + 2n\omega_2 : m, n \in \mathbb{Z} \}.$$9. Then the expected dot product

$$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$0

is strictly monotonically decreasing in $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$1 (Sec. 4.3.3). This is a direct consequence of the linear mapping $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$2 and the Lipschitz-continuity of $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$3 away from poles, implying that encoding similarity decays naturally and smoothly with Euclidean patch distance.

3.2. Algebraic Relative Position Recovery

The addition formula enables direct computation of relative position signals $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$4 algebraically from $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$5 and $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$6, enabling relative positional bias or enrichment of attention $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$7 vectors without need for explicit pairwise lookup tables.

3.3. Resolution Invariance

Because encodings are parameterized by a continuous function and the $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$8 normalization adjusts smoothly with $$\wp(z; \omega_1, \omega_2) = \frac{1}{z^2} + \sum_{\omega \in \Lambda \setminus \{0\}} \left( \frac{1}{(z - \omega)^2} - \frac{1}{\omega^2} \right),$$9, WEF-PE adapts seamlessly to arbitrary resolution increases, requiring no interpolation or retraining of position encodings.

4. Integration with Vision Transformer Architectures

4.1. Absolute and Relative Position Embedding

WEF-PE supports absolute position embeddings by constructing $2\omega_1,\, 2\omega_2$0 and forming ViT's input as

$2\omega_1,\, 2\omega_2$1

(Eq. 3.15).

For relative encoding in self-attention:

• Option A: Add algebraically-computed bias $2\omega_1,\, 2\omega_2$2 to attention scores $2\omega_1,\, 2\omega_2$3.
• Option B: Concatenate or rotate self-attention $2\omega_1,\, 2\omega_2$4 by the relative feature vector.

4.2. Hybrid Fine-Tuning

A hybrid fine-tuning strategy maintains pretrained learned embeddings $2\omega_1,\, 2\omega_2$5 and newly computed WEF-PE $2\omega_1,\, 2\omega_2$6, then linearly mixes:

$2\omega_1,\, 2\omega_2$7

(Section 3.2.1).

4.3. Parameters and Training

The method introduces a minimal set of jointly-optimized parameters: $2\omega_1,\, 2\omega_2$8 (lattice), $2\omega_1,\, 2\omega_2$9 (tanh), $\wp'(z)$0 (global PE strength), and, where applicable, Fourier series terms $\wp'(z)$1. No changes are required to core ViT block structure apart from prepending the WEF-PE computation.

5. Empirical Evaluation

5.1. From-Scratch Results

On CIFAR-100, ViT-Tiny (192-dim, 12 layers, 3 heads, $\wp'(z)$2 grid), WEF-PE achieves 63.78%, outperforming:

• Learnable absolute PE (APE): 56.46%
• Rotary (RoPE): 57.29%
• Fourier (FoPE): 57.70% (Table 5.1)

The hybrid DHVT-Ti model also shows marked improvement: WEF-PE reaches 76.53% vs 74.78% (Table 5.2).

5.2. Fine-Tuning and Transfer

Replacement of position encodings with WEF-PE in ViT-Base/16 trained on ImageNet-21k, fine-tuned on CIFAR-100, yields 93.28% (vs baseline 91–92%). On VTAB-1k, WEF-PE consistently improves accuracy across tasks such as SVHN (84.58% vs 80.90%), Resisc45 (86.10% vs 85.20%), and DMLab (54.24% vs 41.90%) (Table 5.3).

5.3. Ablation and Structural Analysis

Experiment	Accuracy (%)	Difference
4D (full WEF-PE)	63.78	–
Drop $\wp'(z)$3 channels (2D only)	63.08	–0.70
Fixed vs. learnable $\wp'(z)$4	62.88	–0.90
Lemniscatic vs. non-lemniscatic invariants	63.20	–0.58
Fixed PE strength vs. adaptive $\wp'(z)$5	62.60	–1.18

Ablations confirm that full 4D feature extraction and adaptive parameter tuning are critical for peak performance (Table 5.4).

5.4. Attention and Topological Visualizations

Untrained WEF-PE produces grid-like, isotropic distance decay in attention maps, whereas learned APE presents random or noisy patterns (Figure 1.1). PCA on untrained encodings shows that WEF-PE forms a 2D spiral manifold preserving image grid topology, while APE collapses to a Gaussian blob. In trained models, WEF-PE attention is more coherent with full object coverage (Figure 1.4).

5.5. Empirical Validation of Distance-Decay

Cosine similarity across all patch pairs in a $\wp'(z)$6 grid decreases monotonically with Euclidean distance, exhibiting Pearson $\wp'(z)$7 and a 16.4% drop in mean similarity, directly empirically substantiating the distance-decay theorem (Figure 1.5, Table 5.5).

6. Implementation, Efficiency, and Hyperparameters

The core $\wp'(z)$8 and $\wp'(z)$9 computation employs modulus-sorted lattice truncation and pole-clipping, plus adaptive tanh compression (Listing A.1). For from-scratch use, the method is GPU-optimized in PyTorch. From-scratch pretraining uses 4 RTX 3090 GPUs, batch size 512, strong augmentations, AdamW, and cosine-annealed learning rate; fine-tuning is typically performed on 2–4 GPUs. Whereas baseline PEs require bilinear interpolation across resolutions, WEF-PE remains interpolation-free.

7. Significance and Implications

WEF-PE introduces a mathematically exact 2D positional prior aligned with the spatial and translational structure of visual data via the doubly periodic, meromorphic Weierstrass elliptic function. The monotonic and smoothly decaying encoding similarity with distance directly addresses the geometric mismatch inherent in 1D or flat learned positional encodings, facilitating improved inductive bias for both local and global patterns in vision transformers. Continuous function evaluation ensures inherent resolution invariance, simplifying upsampling and cross-resolution adaptation. The algebraic addition formula facilitates efficient relative positioning operations and explicit modeling of pairwise spatial relationships. Empirical results demonstrate strong improvements in both data-limited and transfer settings, with minimal architectural adjustments required (Xin et al., 26 Aug 2025).