WIRE: Wavelet-Induced Rotary Encodings

• WIRE is a generalization of rotary position encodings that integrates spectral graph theory and wavelet transforms to encode both positional and topological context.
• It applies blockwise rotations on query and key vectors using angles computed from Laplacian eigenvectors, ensuring permutation equivariance and linear attention compatibility.
• Empirical results show that WIRE improves performance on graph regression, shortest-path prediction, and point-cloud classification tasks, achieving up to 15% reduction in normalized RMSE.

WIRE (Wavelet-Induced Rotary Encodings) generalizes the concept of Rotary Position Encodings (RoPE) beyond regular grids to arbitrary structured domains, namely undirected graphs and multivariate sequences. By leveraging spectral graph theory and wavelet transforms, WIRE integrates positional and topological context into token representations for attention mechanisms, enabling both theoretical guarantees (such as permutation equivariance) and practical improvements on a range of graph and sequence modeling tasks.

1. Foundational Principles of WIRE

WIRE is motivated by the limitations of classic positional encoding strategies in domains lacking regular sequential or Cartesian structure. In standard RoPE, each token $i$ receives a positional coordinate $r_i \in \mathbb{R}^d$, and the token's query/key vectors are transformed by a series of blockwise rotations parameterized by pairwise coordinate-dependent angles. On grids, these coordinates reflect sequence indices or pixel locations; for graphs, such canonical coordinates are unavailable.

WIRE resolves this by replacing regular coordinates with spectral graph coordinates ("wavelets"). These coordinates are derived from the leading $m$ nontrivial eigenvectors of the graph Laplacian $L = D - A$, where $A$ is the adjacency matrix and $D$ the degree matrix. For a node $i$, the spectral coordinate is

$$r_i := (u_1[i],\,u_2[i],\,\ldots,\,u_m[i])^\top \in \mathbb{R}^m,$$

where $u_k$ is the $k$th Laplacian eigenvector. Low-frequency components capture large-scale graph structure, while higher components encode finer topology. In time series applications, WIRE can also be implemented within the wavelet-transformed domain, supplying multi-scale, band-limited coordinates per variable or node.

2. Mathematical Construction and Encoding Mechanism

WIRE applies blockwise two-dimensional rotations to each (even-dimensional) query/key vector $z_i \in \mathbb{R}^d$:

• $z_i$ is partitioned into $d/2$ blocks of size 2,
• For each block $\ell$, a cumulative rotation angle is computed:

$\Theta_\ell = \sum_{k<m} \omega_k \, r_i[k]$

where $\{\omega_k\}$ are learnable frequency scalars (per attention head).

• The rotated block is:

$$\mathrm{ROPE}(r_i) \cdot z_i^{(\ell)} = \begin{pmatrix} \cos \Theta_\ell & -\sin \Theta_\ell\ \sin \Theta_\ell & \cos \Theta_\ell \end{pmatrix} z_i^{(\ell)}$$

In vectorized form, using a permutation matrix $P$ that swaps pairs, and with $\Omega \in \mathbb{R}^{(d/2)\times m}$ broadcasting $\omega_k$ across blocks:

$$\mathrm{ROPE}(r_i) z_i = \cos(\Omega r_i) \odot z_i + P [\sin(\Omega r_i)] \odot z_i,$$

where $\odot$ denotes elementwise product.

At each transformer layer, the query and key projections are replaced by their WIRE-rotated forms, injecting spectral/topological context into the attention mechanism.

3. Theoretical Properties

WIRE possesses several theoretically desirable properties:

• Permutation Equivariance: Any permutation $\pi$ on node indices maps the spectral coordinates as $r_{\pi(i)} = r_i$, and the entire WIRE transformation strictly commutes with $\pi$. Consequently, attention is invariant under arbitrary node relabellings.
• Compatibility with Linear Attention: Since WIRE transforms the $q_i$ and $k_j$ independently (as opposed to biasing $q_i^\top k_j$ directly), kernel-based factorizations for linear attention are preserved.
• Asymptotic Dependence on Graph Resistance Distance: When the $\omega_k$ are sampled with mean zero, the expected dot product after rotary encoding is

$$\mathbb{E}_\omega[(\mathrm{ROPE}(r_i)q_i)^\top (\mathrm{ROPE}(r_j)k_j)] \approx q_i^\top k_j \cdot (1 - \frac{1}{2}\sigma^2 R(i,j)) + O(\sigma^4)$$

where $R(i,j)$ is the effective resistance metric derived from the Laplacian pseudoinverse and gives a lower bound on shortest-path distance. Thus, WIRE implicitly endows attention mechanisms with a topological inductive bias, downweighting interactions between distant nodes.

4. Experimental Evidence and Empirical Results

WIRE's efficacy has been validated across synthetic, geometric, and standard graph-learning tasks:

• Synthetic Graphs:
  • On monochromatic subgraph regression (5×5 grids with edge deletions), 4-layer Transformer with WIRE ($m \in \{3,5,10\}$) reduced normalized RMSE by up to 15% versus no positional encoding.
  • On shortest-path distance prediction (Watts–Strogatz graphs, $k=2$, $p=0.6$), WIRE ($m=5$) reduced RMSE from ~0.065 to ~0.038.
• Point-Cloud Classification & Segmentation:
  • On k-NN graphs (k=20) for ModelNet40 and ShapeNet, 4-layer Transformer/Performer architectures yield the following test accuracies (in %):

  Method	ModelNet40 (cls)	ShapeNet (seg)
  NoPE	91.8	93.1
  CartRoPE	91.8	93.2
  WIRE (m=10)	93.4	93.2

• Standard Graph Benchmarks:

  • Integrating WIRE into GraphGPS (with ReLU-Performer attention) across MNIST-graphs, CIFAR10-graphs, PATTERN, CLUSTER and molecular datasets produces consistent improvements of 0.5–3 points over no positional encoding, often closing the gap to quadratic-complexity softmax baselines.
  • Similar gains are realized in SGFormer and BigBird integration.

Empirical results establish WIRE as most advantageous in settings where global or local graph topology is critical for model success.

5. Comparison to Previous Approaches

WIRE recovers classic RoPE exactly for grid graphs—formalized as Cartesian products of paths—demonstrating that RoPE is a special case of the more general WIRE construction. Unlike absolute positional encodings or Laplacian eigenvalue-based scalars, WIRE is permutation-equivalent by construction and leverages vector-valued, low-frequency wavelet coordinates rather than arbitrary or hand-crafted features.

Compared to node-attribute-only models or approaches lacking structural encodings, WIRE provides a principled mechanism to inject inductive bias towards topological relations and graph distance, all while maintaining scaling efficiency for large graphs through compatibility with linear attention and kernelized mechanisms.

6. Algorithmic Complexity and Implementation Considerations

Wire's computational overhead arises from two sources:

• Spectral Precomputation: Exact eigendecomposition of the Laplacian is $O(N^3)$; however, fast approximate methods (Lanczos, random-projection-based algorithms) make the computation tractable for large, sparse graphs. Only $m \ll d$ leading eigenvectors are required.
• Parameter Overhead: Each attention head requires only $m$ learnable frequencies; this increment is small relative to standard attention architectures.
• Integration with Transformer Models: WIRE can be immediately incorporated into any transformer-based graph neural network or attention-based point-cloud model without altering the linear or softmax attention schema, as positional encoding is injected into the projections themselves.

The choice of $m$ (spectral coordinate dimensionality) is task-dependent, with higher $m$ potentially capturing finer structural nuance, while unnecessarily large $m$ may add parameter redundancy.

7. Limitations and Potential Applications

Limitations of WIRE include:

• The dependence on Laplacian eigenvectors implies $O(N^3)$ cost for small graphs (though often mitigated for sparse or structured graphs by approximate solvers).
• Effectiveness depends on tasks being sufficiently topology-sensitive; WIRE's improvements may diminish on tasks where attributes dominate over structural relationships.

WIRE is broadly applicable in domains where transformer-based models are utilized in combination with graph-structured or point-cloud data, including but not limited to:

• General-purpose graph transformers,
• GNN–attention hybrids,
• Point-cloud transformers for 3D data,
• Long-range graph tasks where inductive topological bias is desired.

In contexts requiring global structure awareness, efficient scaling, and explicit permutation invariance, WIRE provides a mathematically grounded, empirically validated positional encoding strategy that subsumes RoPE and extends transformer architectures to a new class of structured problems (Reid et al., 26 Sep 2025).