Input-Dependent Positional Encoding

• Input-dependent positional encoding is a mechanism that creates adaptive, context-aware positional representations by integrating input data features rather than fixed indices.
• Representative methodologies such as dynamic wavelet encodings, local grid-Fourier schemes, and neural ODE flows tailor positional signals to various tasks including time series analysis, image reconstruction, and language modeling.
• Empirical studies show these encodings yield measurable improvements, like a 9.1% accuracy gain in time series tasks and robust extrapolation in long-context language modeling, enhancing overall model generalization.

Input-dependent positional encoding refers to any mechanism that generates positional representations as explicit functions of the input sequence, rather than solely from a static position index. Unlike classical positional encodings, which are agnostic to sequence content, input-dependent schemes adapt positional signals based on the input’s local or global characteristics. This paradigm has gained momentum across deep learning domains, including transformers for sequence modeling, MLP-based function representation, and scientific or robotics applications, as a means to endow neural networks with richer inductive biases and improved generalization for complex or non-stationary data.

1. Motivation and Theoretical Foundations

Traditional positional encoding schemes in transformers—including sinusoidal embeddings and various learned absolute or relative encodings—produce embeddings solely from integer indices $i \in \{0, ..., L-1\}$, irrespective of the underlying sequence content. While this approach is effective for relatively homogeneous data such as text, it introduces two fundamental limitations when applied to more complex or non-stationary signals:

• No awareness of local dynamics: Identical positional encodings are assigned to indices regardless of varying signal regimes (e.g., a calm segment vs. a bursty period), forcing the model to jointly infer position and content from scratch (Irani et al., 18 Sep 2025).
• Lack of multi-scale adaptation: Real-world signals often exhibit hierarchical or transient dynamics at multiple temporal or spatial scales. Static encodings cannot locally adapt, degrading inductive bias and hampering performance on variable or long-range sequences (Irani et al., 18 Sep 2025, Fujieda et al., 2023).

By making positional embeddings explicit functions of the input, input-dependent schemes directly address these deficiencies, enabling representations that reflect both the geometry of the data and its temporal or spatial context. In formal terms, these methods replace fixed mappings $PE(i)$ with input-conditioned mappings $PE(x, i)$ where $x$ is (local or global) information from the input sequence.

2. Representative Methodologies

A range of input-dependent positional encoding frameworks have emerged, targeting distinct domains and encoder architectures. Core approaches include:

Dynamic Wavelet Positional Encoding (DyWPE)

DyWPE constructs positional embeddings for time series transformers by leveraging the discrete wavelet transform (DWT) of the input signal. For multivariate input $X \in \mathbb{R}^{B \times L \times d_x}$, a channel projection collapses inputs to a mono-channel, followed by a multi-level wavelet decomposition. Learnable embeddings for each wavelet scale are dynamically modulated by the corresponding coefficient maps through learned gating functions and outer products. A content-driven multi-scale embedding $$P_{DyWPE} \in \mathbb{R}^{B \times L \times d_{model}}$$ is reconstructed by inverse DWT and injected directly into the transformer’s input at every layer (Irani et al., 18 Sep 2025).

Local Positional Encoding (LPE) for MLPs

LPE combines grid and Fourier features in coordinate-based MLPs. For position $x_g \in [0,1]^d$, a low-resolution grid decomposition determines the cell and a local offset, enabling per-cell interpolation of learned coefficients. These modulate the amplitude of locally varying sinusoidal features, yielding an embedding that captures both spatial locality and local frequency content in a memory- and parameter-efficient manner (Fujieda et al., 2023).

Context-aware and Content-Driven Rotary Embeddings

CARoPE extends rotary positional encoding (RoPE) by replacing static, global frequencies with head- and token-dependent frequencies computed by a learned function of the token embeddings. This produces phase accumulations that are sequence- and context-sensitive, while preserving RoPE’s efficiency and streaming compatibility (Veisi et al., 30 Jul 2025). Token-Aware Phase Attention (TAPA) further factors the phase directly as a function of token content, enabling complete elimination of fixed distance-dependent biases and robust extrapolation to long contexts (Yu et al., 16 Sep 2025).

Lie-Group–Based Encodings for Cognitive Maps

In MapFormer, input tokens produce “integration time” increments via learnable projections, generating input-dependent rotation matrices over block-diagonal Lie groups. These rotations, accumulated over the sequence, define both absolute and relative positional embeddings capable of representing path-integration or group-theoretic structure as required by spatial and cognitive navigation tasks (Rambaud et al., 24 Nov 2025).

Neural ODE–Driven Continuous Flows

Input-dependent encodings can be realized via a continuous ordinary differential equation (ODE) that models the evolution of positional embeddings as a function of continuous time or position. This “dynamical system” approach not only enables content-driven adaptation but allows extrapolation to arbitrary sequence lengths, as the flow can be queried at unseen indices (Liu et al., 2020).

3. Mathematical Formulations and Algorithmic Structure

Across domains, input-dependent positional encodings are formulated by composing parameterized functions—often neural networks—with input features, local context, or signal transforms. Representative examples include:

• Wavelet-based: $$P_{DyWPE}[b, \cdot] = \mathrm{IDWT}(Yl_{mod}[b, \cdot], Yh_{mod}[b, \cdot])$$, where $Yl_{mod}, Yh_{mod}$ are themselves outputs of dynamic modulation between learned scale prototypes $e$ and DWT coefficient maps via a gating function over the input (Irani et al., 18 Sep 2025).
• Local grid-Fourier: For grid cell $z$ and local offset $x_l$, $LPE(x_g)$ combines trilinearly interpolated trainable coefficients with locally computed sinusoids, yielding region-specific amplitude and phase (Fujieda et al., 2023).
• Token-aware phase: In TAPA, the phase $\phi(q, k) = q^T \mathcal{N} k$ introduces a content-dependent oscillatory attenuation into the attention mechanism:

$$\mathrm{Attn}_{\phi,\mathcal M,\alpha}(q, k) = q^\top\mathcal M k\,\times\,\cos(2\pi|m-n|^{\alpha} \phi(q, k)).$$

Quadratic $\phi$ ensures stationary properties and stability (Yu et al., 16 Sep 2025).

• Lie-group action: In MapFormer, action-dependent increments $\Delta_t$ are mapped into per-block rotation angles, and the product of these rotations over time commutes due to block-diagonal structure. Path integration is expressed as $R_{0 \rightarrow t} = \prod_{s \leq t} R_s$ (Rambaud et al., 24 Nov 2025).
• Neural ODE flow: Continuous positional signal $h(t)$ is the solution to an ODE:

$\frac{d h(t)}{dt} = f(h(t), t ; \theta),$

where $f$ is a small neural network and $h(0)$ learnable initial state (Liu et al., 2020).

4. Empirical Performance and Impact

Empirical studies consistently demonstrate that input-dependent positional encodings yield measurable improvements over static baselines, particularly in domains where local or multi-scale dynamics are critical.

• DyWPE outperforms eight existing state-of-the-art positional encodings—including sinusoidal, learned, relative, and hybrid schemes—on ten time series datasets, achieving an average relative accuracy improvement of $9.1\%$ (e.g., $88.2\%$ vs $84.1\%$ on Sleep EEG, $79.1\%$ vs $67.3\%$ on ElectricDevices) while maintaining comparable computational efficiency ($1.48\times$ the no-PE baseline, similar to other advanced encodings) (Irani et al., 18 Sep 2025).
• LPE in MLPs enables higher-quality image and shape reconstructions from coordinate queries, substantially improving PSNR (up to $6$ dB gain over global Fourier PE) and SSIM, and matches or surpasses memory-comparable hierarchical grid approaches (Fujieda et al., 2023).
• CARoPE and TAPA outperform RoPE and sinusoidal/learned baselines in long-context language modeling, achieving lower perplexity and greater robustness to context length extension (e.g., CARoPE yields perplexity $21.39$ vs RoPE $56.61$ on GPT-2 Small at $1024$ tokens; TAPA maintains stable perplexity at $64$k sequence length where RoPE collapses) (Veisi et al., 30 Jul 2025, Yu et al., 16 Sep 2025).
• MapFormer exhibits near-perfect OOD generalization for navigation and relational reasoning tasks, significantly outperforming fixed and relative PE schemes (e.g., $0.99$–$1.0$ accuracy vs RoPE $0.29$–$0.39$ in 2D grid navigation under OOD splits) (Rambaud et al., 24 Nov 2025).
• Neural ODE–based encoding (“FLOATER”) provides consistent gains over both sinusoidal and learned lookup-table PEs in machine translation and language understanding tasks, with improved extrapolation to unseen sequence lengths (Liu et al., 2020).

A common finding is that the expressive capacity brought by input dependence must be carefully regularized. For example, excessively high-frequency input-dependent features can harm generalization (as observed in collision checking with large $L$ in Fourier bases), necessitating tuning of spectral complexity (Kulecki et al., 9 Sep 2025).

5. Applications Across Domains

Input-dependent positional encodings have been deployed in a wide range of machine learning subfields:

• Time Series Analysis: DyWPE enables signal-aware transformer models to exploit the inherent nonstationarity and multi-scale properties of real-world time series, critical for biomedical signals, geospatial data, or financial streams (Irani et al., 18 Sep 2025).
• Coordinate-based Neural Representations: LPE and similar techniques improve high-fidelity encoding of images, signed distance functions, and scientific simulations, producing sharper reconstructions and reducing memory requirements for compact MLP architectures (Fujieda et al., 2023).
• Robotic Collision Checking: Frequency-based input positional encodings in MLPs enhance classification of high-frequency configuration space boundaries, yielding faster and more accurate collision checking than geometric algorithms (Kulecki et al., 9 Sep 2025).
• Language Modeling and Reasoning: Content-aware rotary embeddings and token-aware phase mechanisms deliver stable and precise modeling of long-range dependencies beyond the training context, enhancing LLMs’ extrapolation ability (Veisi et al., 30 Jul 2025, Yu et al., 16 Sep 2025).
• Cognitive Map and Navigation Modeling: Input-dependent Lie-group encodings in MapFormer are essential for learning disentangled structure-content representations underlying path-integration and relational reasoning, enabling robust adaptation to new or longer navigation tasks (Rambaud et al., 24 Nov 2025).

6. Limitations, Open Challenges, and Future Directions

While input-dependent positional encoding represents a substantial advance in conveying local and global structure to neural architectures, current implementations face several challenges:

• Complexity and Overhead: Methods leveraging wavelet decompositions (Irani et al., 18 Sep 2025), neural ODEs (Liu et al., 2020), or advanced gating networks introduce increased training and inference time (e.g., $20$–$30$\% slowdown in some ODE flows).
• Stability and Generalization: Not all designs yield robust generalization, particularly as the frequency or complexity of input-dependent terms increases. Overly rich encoding bases may lead to overfitting or poor extrapolation outside data regimes.
• Integration with Relative Positioning: Input-dependent encodings must often be reconciled with relative-position modeling to capture arbitrary permutations or group actions as required in navigation and multi-agent environments.
• Architectural and Task Adaptivity: Dynamic encodings can introduce new hyperparameters (e.g., spectral range, gating function complexity), and their adaptation across distinct transformer structures and data types remains an area of active exploration (Gu et al., 19 May 2025).
• Interpretability and Structural Bias: Some methods, such as the MapFormer approach, suggest a close correspondence between encoding structure and the algebraic properties of the underlying task (e.g., group theory for navigation), but general frameworks for learning such biases remain to be fully developed (Rambaud et al., 24 Nov 2025).

A plausible implication is that future research will further unify the design of input-dependent positional encoding with task-specific inductive priors, possibly leveraging theory from Lie groups, spectral analysis, or dynamical systems.

7. Summary Table: Key Methods and Domains

Method	Input Dependence	Domains / Models
DyWPE (Irani et al., 18 Sep 2025)	DWT of input + gated scale embeddings	Time Series / Transformers
LPE (Fujieda et al., 2023)	Grid cell & local offset modulate basis	MLPs (images, SDF, regression)
CARoPE (Veisi et al., 30 Jul 2025)	Frequency function of token embeddings	Transformers / Language modeling
TAPA (Yu et al., 16 Sep 2025)	Attention phase from token pairs	Transformers / Language modeling
MapFormer (Rambaud et al., 24 Nov 2025)	Action-driven rotation matrices	Navigation / Cognitive maps
Neural ODE PE (Liu et al., 2020)	ODE flow over position (dynamic)	Transformers (NLP, MT)
Fourier PE for MLP (Kulecki et al., 9 Sep 2025)	$\sin$, $\cos$ on inputs (config.)	Robotics, geometric learning

Input-dependent positional encoding represents a paradigm shift from position-indexed static features to architecture- and sequence-aware representations, equipping modern neural networks with the capacity to explicitly encode and exploit the structure present in their input data. Ongoing advances in this field continue to improve model fidelity, extrapolation, and understanding of the geometries underlying complex tasks.