High-Dimensional Positional Encoding

• High-dimensional positional encoding (HPE) is a method that maps low-dimensional coordinates into a higher-dimensional space using techniques like Fourier features, spline bases, and hyperbolic embeddings.
• HPE methods enhance geometric fidelity and identifiability by employing basis expansion, geometry preservation, and architectural decoupling to improve neural model performance.
• Empirical studies across domains such as vision, language, and point clouds demonstrate that HPE can boost accuracy, convergence, and reconstruction quality compared to standard positional encodings.

High-dimensional Positional Encoding (HPE) denotes a class of positional representation schemes that map indices, coordinates, or relative offsets from low-dimensional domains such as sequences, images, volumes, graphs, and point clouds into higher-dimensional feature spaces before they are injected into a neural architecture. In contemporary literature, the term covers several related but non-identical constructions: Fourier-feature encodings for arbitrary $d$-dimensional positions, faithful discrete Fourier transform encodings for sequences, grid-cell-inspired encodings with translation-invariant kernels, spline and orthogonal-polynomial bases, anisotropic and hyperbolic positional embeddings, and high-frequency rotary variants designed for long-context language modeling (Li et al., 2024, Idé et al., 2024, Li et al., 2021, Chen et al., 2024, Bose et al., 2023). Their shared objective is to improve the identifiability, geometric fidelity, and task utility of positional information beyond what is typically obtained from low-dimensional coordinates or standard sinusoidal encodings.

1. Conceptual scope and mathematical basis

A common formal pattern in HPE is a lifting map $x \mapsto e(x)$, where $x$ is a sequence position, coordinate, or relative displacement and $e(x)\in\mathbb{R}^D$ or $\mathbb{C}^D$ is the encoding used by the downstream model. In learnable Fourier-feature formulations for multi-dimensional spatial encoding, a position $x\in\mathbb{R}^d$ is mapped to

$$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$

with $W_r\in\mathbb{R}^{D/2\times d}$ trainable, after which a small MLP may modulate the representation before addition or concatenation with the content embedding (Li et al., 2021). In GridPE, the basis functions are complex exponentials $\phi_i(x)=e^{j k_i^T x}$, assembled into a vector whose inner products depend only on displacement, not absolute location (Li et al., 2024). In faithful DFT-based positional encoding, a discrete position $s$ is represented by evaluating an orthonormal real DFT basis at $x \mapsto e(x)$0, yielding an encoding that is injective over the supported index set (Idé et al., 2024).

These constructions differ in basis choice and downstream parameterization, but they repeatedly exploit three mathematical devices. The first is basis expansion: sinusoidal, Fourier, spline, Legendre-polynomial, or hyperbolic transforms replace raw coordinates by structured coordinates in a richer feature space (Wang et al., 2021, Aggarwal, 2024, Bose et al., 2023). The second is geometry preservation: several methods explicitly target distance- or displacement-sensitive kernels, including translation-invariant inner products in Fourier encodings and anisotropic weighted norms in medical imaging (Li et al., 2024, Jabareen et al., 2 Sep 2025). The third is architectural decoupling: positional structure is encoded by a dedicated map rather than being inferred entirely through content features or recurrence (Morita, 2024, Ito et al., 2024).

The acronym itself is used heterogeneously. Some papers use HPE to mean high-dimensional positional encoding in the generic sense of lifting positions into a larger feature space; HyPE-GT uses hyperbolic positional encodings for graph transformers; HoPE denotes a high-frequency rotary position encoding that keeps only part of the RoPE spectrum and replaces the rest with position-independent components (Bose et al., 2023, Chen et al., 2024). This terminological variation reflects a broader methodological family rather than a single canonical algorithm.

2. Principal construction families

Recent work partitions naturally into several design families.

Family	Core mechanism	Representative papers
Fourier and DFT encodings	Fixed or learnable sinusoidal/Fourier bases; often kernel-oriented	(Idé et al., 2024, Li et al., 2021, Li et al., 2024)
Alternative bases	Spline or Legendre polynomial expansions	(Wang et al., 2021, Aggarwal, 2024)
Geometry-aware encodings	Anisotropic Euclidean or hyperbolic embeddings	(Jabareen et al., 2 Sep 2025, Bose et al., 2023)
Architecture-specific variants	Rotary high-frequency selection, RNN concatenation, point-cloud relative-offset lifting, INR projected sampling	(Chen et al., 2024, Morita, 2024, Zou et al., 4 Mar 2026, Perez et al., 27 Apr 2026)

Fourier-feature methods dominate the current landscape. Learnable Fourier features for multi-dimensional spatial positional encoding treat $x \mapsto e(x)$1-dimensional positions as continuous variables, use a trainable frequency matrix, and bias the initial dot-product kernel toward a Gaussian RBF. GridPE extends this logic by modeling grid-cell firing as a weighted sum of planar oscillations and by deriving an optimal scale ratio $x \mapsto e(x)$2, with $x \mapsto e(x)$3 for $x \mapsto e(x)$4-dimensional encoding (Li et al., 2021, Li et al., 2024). Faithful DFT encoding is more restrictive but sharper in its guarantee: with an orthonormal real DFT basis and $x \mapsto e(x)$5, positional order is preserved exactly through invertible reconstruction of the one-hot reference function (Idé et al., 2024).

Alternative bases are motivated by deficiencies in sinusoidal encodings. Spline Positional Encoding replaces fixed sinusoidal functions by trainable B-spline basis functions defined over projected 1D coordinates of a 3D point, with learnable weights and projection directions. PoPE replaces sinusoidal terms by Legendre polynomials, emphasizing orthogonality, non-periodicity, and a different correlation structure in high dimensions (Wang et al., 2021, Aggarwal, 2024).

Geometry-aware HPE variants adapt the embedding space to the data domain. AFPE generalizes isotropic Fourier features by sampling each spatial axis with its own scale $x \mapsto e(x)$6, thereby encoding anisotropy in images, volumes, or videos. HyPE-GT moves positional encodings to a manifold of constant negative curvature, using either the Poincaré ball or hyperboloid model and a hyperbolic neural module to transform initial graph-topological positional descriptors such as LapPE or RWPE (Jabareen et al., 2 Sep 2025, Bose et al., 2023).

Finally, several methods are tightly coupled to particular architectures or modalities. HoPE is a rotary formulation that retains only the top-$x \mapsto e(x)$7 highest-frequency RoPE blocks while replacing the remaining blocks with the identity. HPENet projects relative 3D point offsets into high-dimensional embeddings inside the ABS–REF view for point-cloud processing. PEPS interprets each sinusoidal positional feature as a point of interest and samples a learned grid at all such projected points (Chen et al., 2024, Zou et al., 4 Mar 2026, Perez et al., 27 Apr 2026).

3. Theoretical properties and recurrent critiques of standard encodings

A central theoretical theme is translation- or displacement-sensitivity. For GridPE,

$x \mapsto e(x)$8

so the inner product is a function only of the displacement $x \mapsto e(x)$9, not of the absolute positions. The same shift-invariant structure appears in learnable Fourier features, where $x$0 becomes a sum of cosines of $x$1, approximating a stationary kernel under Gaussian initialization (Li et al., 2024, Li et al., 2021). This is particularly relevant in vision, point clouds, and other domains where relative geometry is often more important than absolute index.

A second theme is faithfulness or injectivity. The DFT-based encoding of Idée, Labaien, and Chen is explicitly designed so that the mapping $x$2 is one-to-one, and the one-hot position function can be reconstructed exactly through the orthonormal basis. The paper contrasts this with standard sinusoidal PE, whose Fourier coverage is strongly low-pass and whose reconstruction of $x$3 is blurred over $x$4 positions or more for typical $x$5 (Idé et al., 2024). PoPE raises a related critique from a different angle: in a $x$6-dimensional sinusoidal encoding, empirical correlations in high-dimensional subspaces can exceed $x$7 for distant positions, indicating that large parts of the embedding become nearly indistinguishable (Aggarwal, 2024).

A third theme is the role of basis orthogonality and non-periodicity. PoPE attributes some of the failure modes of sinusoidal absolute and rotary encodings to the periodicity of sine and cosine bases and proposes Legendre polynomials to obtain orthogonality on $x$8, non-periodicity, and a three-term recurrence that can express algebraic relations between neighboring basis orders (Aggarwal, 2024). Spline Positional Encoding reaches a similar objective through local support rather than orthogonality: compactly supported B-splines permit local learning of high-frequency detail and progressive refinement by knot subdivision (Wang et al., 2021).

Long-context language modeling introduces a distinct theoretical dispute. HoPE rejects the long-term decay assumption embedded in many relative positional encodings, arguing from empirical analyses that trained models exhibit steep local decay together with a global U-shaped emphasis on early tokens. Its derivation identifies a band of RoPE frequencies that becomes “activated,” forms shortcut U-shaped attention patterns during training, and undergoes half-cycle out-of-distribution flips when extrapolated beyond the training length. HoPE therefore keeps only the highest-frequency rotary blocks and replaces the remainder by position-independent components (Chen et al., 2024). This does not merely alter frequency allocation; it challenges the premise that useful positional bias should decay globally with distance.

4. Architectural integration across model classes

The most direct integration strategy is additive injection. Faithful DFT PE follows the original Transformer pattern exactly: at position $x$9, the token or segment embedding is updated by $e(x)\in\mathbb{R}^D$0, with no modification to the self-attention equations. Learnable Fourier features also produce a positional vector that is added or concatenated before attention (Idé et al., 2024, Li et al., 2021).

GridPE supports multiple integration modes. In a Pyramid Vision Transformer, the original 2-D positional bias in each self-attention layer is replaced by GridPE applied to flattened spatial coordinates. For each token at grid location $e(x)\in\mathbb{R}^D$1, the encoding $e(x)\in\mathbb{R}^D$2 can be used in an add mode, a rotate mode with block-diagonal rotations, or a project mode using a small MLP $e(x)\in\mathbb{R}^D$3. Standard LayerNorm and residual connections remain unchanged, and the only architectural constraint is that $e(x)\in\mathbb{R}^D$4 match the hidden size, with an optional complex-to-real split (Li et al., 2024).

Rotary and rotary-derived methods modify the attention mechanism more structurally. HoPE inherits RoPE’s block-diagonal rotation formulation in query and key space but alters which frequency blocks are position-dependent. The attention score remains $e(x)\in\mathbb{R}^D$5, preserving a purely relative formulation while changing the spectral content of the positional operator (Chen et al., 2024).

Graph and non-Euclidean models require additional machinery. HyPE-GT first generates hyperbolic positional embeddings from initial graph-topological encodings using an HNN or HGCN on either the hyperboloid or Poincaré-ball manifold. Integration can then proceed by mapping Euclidean node features into hyperbolic space, Möbius-adding the hyperbolic PE, and mapping back to Euclidean space, or by logging the hyperbolic PE to Euclidean space and adding it there before standard attention (Bose et al., 2023).

Outside Transformers, HPE is used as a general input-lifting mechanism. In Spline Positional Encoding, the encoded 3D coordinate replaces the raw coordinate as input to a 4-layer Softplus MLP for signed distance field learning. In HPENet, relative offsets $e(x)\in\mathbb{R}^D$6 are encoded and concatenated with neighbor features in both the ABS and REF stages. In Morita’s RNN study, sinusoidal positional vectors are concatenated with token embeddings at every time step, so the recurrent input becomes $e(x)\in\mathbb{R}^D$7 for Elman RNNs, GRUs, and LSTMs (Wang et al., 2021, Zou et al., 4 Mar 2026, Morita, 2024).

5. Empirical results across domains

The empirical record is distributed across many application areas rather than a single benchmark tradition.

Setting	Baseline result	HPE-family result
Time-series anomaly classification	Elevator F1 $e(x)\in\mathbb{R}^D$8, SMD F1 $e(x)\in\mathbb{R}^D$9, MSL F1 $\mathbb{C}^D$0 with original PE	Elevator $\mathbb{C}^D$1, SMD $\mathbb{C}^D$2, MSL $\mathbb{C}^D$3 with DFT PE (Idé et al., 2024)
PVT on ImageNet-1000, top-5 accuracy on 10-class subset	PVT-o $\mathbb{C}^D$4, PVT-abs $\mathbb{C}^D$5, CPVT $\mathbb{C}^D$6	GridPVT-rotate $\mathbb{C}^D$7, GridPVT-merge $\mathbb{C}^D$8, GridPVT-complex $\mathbb{C}^D$9, GridPVT-deep $x\in\mathbb{R}^d$0 (Li et al., 2024)
ViT-B/16 on ImageNet	Embed-1D $x\in\mathbb{R}^d$1 top-1	Fourier+MLP $x\in\mathbb{R}^d$2 top-1 (Li et al., 2021)
EchoNet Dynamic EF regression	SPE $x\in\mathbb{R}^d$3, IFPE $x\in\mathbb{R}^d$4	AFPE $x\in\mathbb{R}^d$5 (Jabareen et al., 2 Sep 2025)
Multi30K EN$x\in\mathbb{R}^d$6DE	Transformer-base + sinusoidal APE $x\in\mathbb{R}^d$7 BLEU	Transformer-base + PoPE $x\in\mathbb{R}^d$8 BLEU (Aggarwal, 2024)
Reverse-ordering with LSTM at $x\in\mathbb{R}^d$9	vanilla $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$0	HPE $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$1 (Morita, 2024)

These results span sequence modeling, vision, medical imaging, translation, and recurrent architectures. In vision, learnable Fourier features improved convergence and final bits-per-dim on ImageNet-64 generation, improved DETR object detection from $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$2 AP with Sine-2D to $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$3 AP with Fourier+MLP, and improved few-shot linear-probe ImageNet accuracy from $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$4 to $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$5 (Li et al., 2021). In point clouds, HPENet ablations on S3DIS Area-5 showed $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$6 mIoU when HPE was applied in both ABS and REF, compared with $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$7 for a standard learnable PE in the same ABS+REF placement and $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$8 without HPE or BFM (Zou et al., 4 Mar 2026).

For implicit neural representations, Spline Positional Encoding reported Chamfer $$r_x=\frac{1}{\sqrt{D}}\,[\cos(W_r x)\,\Vert\,\sin(W_r x)],$$9 and MAE $W_r\in\mathbb{R}^{D/2\times d}$0 on single-shape SDF reconstruction, while Fourier Feature Encoding failed under gradient constraints with Chamfer $W_r\in\mathbb{R}^{D/2\times d}$1. On DFaust shape-space learning, Spe achieved Chamfer $W_r\in\mathbb{R}^{D/2\times d}$2, compared with $W_r\in\mathbb{R}^{D/2\times d}$3 for Igr, $W_r\in\mathbb{R}^{D/2\times d}$4 for Siren, and $W_r\in\mathbb{R}^{D/2\times d}$5 for Fpe (Wang et al., 2021). PEPS likewise reported stronger image and SDF reconstruction; for Kodak image representation, Grid-PEPS achieved PSNR $W_r\in\mathbb{R}^{D/2\times d}$6 against $W_r\in\mathbb{R}^{D/2\times d}$7 for a plain grid, and in SDF compression Grid-PEPS improved IoU from $W_r\in\mathbb{R}^{D/2\times d}$8 to $W_r\in\mathbb{R}^{D/2\times d}$9 over a trilinear grid baseline (Perez et al., 27 Apr 2026).

Language-model and graph results point to broader positional-design effects. HoPE improved small-model zero-shot perplexity from RoPE $\phi_i(x)=e^{j k_i^T x}$0 to HPE $\phi_i(x)=e^{j k_i^T x}$1 at test length $\phi_i(x)=e^{j k_i^T x}$2, reduced the gap at $\phi_i(x)=e^{j k_i^T x}$3 from RoPE $\phi_i(x)=e^{j k_i^T x}$4 to HPE $\phi_i(x)=e^{j k_i^T x}$5, and raised copying accuracy from RoPE $\phi_i(x)=e^{j k_i^T x}$6 to HPE $\phi_i(x)=e^{j k_i^T x}$7. In a 3B Llama setting, average performance across eight benchmarks increased from $\phi_i(x)=e^{j k_i^T x}$8 with RoPE to $\phi_i(x)=e^{j k_i^T x}$9 with HPE (Chen et al., 2024). HyPE-GT reported top-three performance on PATTERN, CLUSTER, MNIST, and CIFAR10 among Transformer-based methods, improved AUROC on ogbg-molhiv over GraphGPS and SAN by $s$0, and mitigated over-smoothing in deep GCNs up to depth $s$1 (Bose et al., 2023).

6. Design trade-offs, misconceptions, and open problems

One recurrent misconception is that positional encoding is only a Transformer-specific timekeeping device. Morita’s RNN experiments directly contradict that view: concatenating sinusoidal positional encodings to recurrent inputs improved reverse-ordering and sorting accuracy at large vocabulary sizes, and the proposed explanation was gradient stabilization for rare tokens rather than extension of raw memory span. The same study also showed no improvement on the copying-memory task, confirming that HPE did not combat vanishing memory itself (Morita, 2024). This suggests that positional lifting can play optimization and conditioning roles distinct from sequence-order disambiguation.

A second misconception is that any increase in positional dimensionality is automatically beneficial. Faithful DFT PE requires $s$2 to guarantee full invertibility over $s$3 positions, so the encoding dimension is coupled to the model hidden size. Learnable Fourier features add approximately $s$4 per-step overhead relative to fixed embeddings or sinusoids. PEPS notes that very high-dimensional PEPS vectors can be underutilized when the MLP is tiny. HPENet ablations found a non-monotonic dependence on embedding width, with $s$5 outperforming $s$6, $s$7, $s$8, and $s$9 on S3DIS Area-5 (Idé et al., 2024, Li et al., 2021, Perez et al., 27 Apr 2026, Zou et al., 4 Mar 2026).

A third issue concerns geometry mismatch. AFPE argues that isotropic Fourier features preserve Euclidean distances equally in all directions but cannot represent anisotropy, while sinusoidal encodings extended axis-wise fail to preserve Euclidean distances for diagonal or higher-dimensional offsets. HyPE-GT argues that Euclidean positional encodings distort hierarchical graph structure and that negative curvature provides a better metric substrate for graph neighborhoods. These positions are not contradictory; they imply that positional geometry should be chosen to match domain structure rather than inherited from generic sequence models (Jabareen et al., 2 Sep 2025, Bose et al., 2023).

Initialization and spectral allocation remain unsettled design variables. Ito et al. found that learnable positional encodings initialized with a small-norm Gaussian, especially $x \mapsto e(x)$00, produced more interpretable multi-dimensional positional structure and higher validation accuracy than standard or large-norm initializations in a 2D relational reasoning task. HoPE, by contrast, argues that selective removal of low-frequency and “activated” RoPE components improves extrapolation, while PoPE argues that sinusoidal periodicity itself is part of the problem. This suggests that current HPE research is divided between three remedies for positional failure modes: improved initialization of learnable tables, replacement of the basis family, and restriction or reshaping of the frequency spectrum (Ito et al., 2024, Chen et al., 2024, Aggarwal, 2024).

Open directions stated in the literature include combinations of faithful or Fourier encodings with learned or relative schemes, adaptive or dynamic band selection beyond a fixed training length, interactions with sparse long-context attention kernels, broader evaluation on very long sequences, and extension to other anisotropic or non-Euclidean domains (Idé et al., 2024, Chen et al., 2024, Jabareen et al., 2 Sep 2025). A plausible implication is that “high-dimensional positional encoding” will remain less a single method than a design space in which basis, geometry, dimensionality, and integration strategy are tuned jointly to the inductive demands of the target architecture and data modality.