LaMPE: Adaptive Multi-grained Positional Encoding

• The paper introduces LaMPE, a training-free method that dynamically maps input lengths with a scaled sigmoid function to optimize positional encoding in RoPE-based LLMs.
• It employs a multi-grained attention mechanism that preserves fine-grained details in the head and tail regions while compressing middle regions to maintain long-range dependencies.
• Experimental results show that LaMPE outperforms fixed remapping techniques, achieving superior performance on benchmarks like LongBench and L-Eval with consistent improvements.

Length-aware Multi-grained Positional Encoding (LaMPE) is a training-free method for adaptive long-context scaling in RoPE-based LLMs. It is designed for the setting in which LLMs experience significant performance degradation when the input exceeds the pretraining context window, primarily due to the out-of-distribution behavior of Rotary Position Embedding (RoPE). LaMPE addresses this by establishing a dynamic relationship between mapping length and input length through a parametric scaled sigmoid function, and by introducing a multi-grained attention mechanism that allocates positional resolution differently across sequence regions. In the reported experiments, it is applied to three representative LLMs across five mainstream long-context benchmarks and achieves significant performance improvements compared to existing length extrapolation methods (Zhang et al., 4 Aug 2025).

1. Problem formulation and conceptual basis

LaMPE is situated within the broader problem of length extrapolation in Transformers. Transformer-based models including LLMs suffer from a preset length limit and can hardly generalize from short training sequences to longer inference ones, and positional encoding has been considered the primary factor on length extrapolation (Zhao et al., 2023). In RoPE-based LLMs specifically, degradation arises when input positions move outside the range effectively covered during pretraining, so recent long-context methods often remap out-of-distribution positions back into an in-distribution range.

The specific premise of LaMPE is that fixed mapping strategies are insufficient because they ignore the dynamic relationship between input length and the model’s effective context window. Its abstract states that recent studies mitigate this problem by remapping OOD positions into the in-distribution range with fixed mapping strategies, ignoring the dynamic relationship between input length and the model's effective context window. LaMPE is therefore “length-aware” in a precise operational sense: the mapping rule is not constant across inputs, but varies as a function of the actual sequence length (Zhang et al., 4 Aug 2025).

The “multi-grained” component addresses a second limitation of uniform remapping. Uniform grouping or remapping can dilute crucial fine-grained relationships where locality and sequence boundaries matter most. LaMPE instead divides the sequence into head, middle, and tail regions and uses region-specific positional granularities, with the stated goal of capturing both fine-grained locality and long-range dependencies (Zhang et al., 4 Aug 2025).

2. Dynamic mapping between input length and positional capacity

The first methodological component of LaMPE is Length-aware Dynamic Mapping. Instead of using a fixed rule to map OOD positions back into the trained window, LaMPE dynamically determines the mapping length for each input based on the actual input length using a parametric scaled sigmoid function. The stated motivation is to optimally utilize well-trained positions without wasting positional capacity on undertrained locations (Zhang et al., 4 Aug 2025).

The reported empirical analysis found that the optimal mapping length rises S-shaped with the input length. For short inputs, small mapping lengths often suffice, described as a V-shaped pattern of perplexity vs. mapping length. For long inputs, the best mapping length monotonically increases, approaching the model’s effective context window. The mapping length is given as

$m = \frac{L}{1 + e^{-(a l + b)}}$

where $m$ is the optimal mapping length for the current sequence, $l$ is input length, $L$ is the upper limit, and $a, b$ are sigmoid shape parameters fitted on a small number of points (Zhang et al., 4 Aug 2025).

Given input length $l$ and mapping length $m$, LaMPE linearly remaps each relative position:

$$PE[i][j] = \left\lfloor \frac{m}{l} (i - j) \right\rfloor$$

This compresses the position space to be compatible with the mapping length and provides a dynamic transformation for any input size. The paper frames this as a way to fully utilize the model’s effective context window for adaptive long-context scaling in LLMs (Zhang et al., 4 Aug 2025).

A common misunderstanding in the length-extrapolation literature is that a single remapping rule should suffice once the target context window is fixed. LaMPE explicitly rejects that assumption. Its formulation makes the mapping length itself depend on the current input, rather than treating extrapolation as a one-time global rescaling.

3. Multi-grained attention over head, middle, and tail regions

The second methodological component is the Multi-grained Attention Mechanism. LaMPE splits each sequence of length $l$ into head, middle, and tail regions, and assigns different encoding granularities to them. The head region, consisting of the first $s_1$ tokens, uses fine-grained identity mapping $m$0, preserving local relationships. The middle region is linearly compressed using the adaptive scaling described above. The tail region, consisting of the last $m$1 tokens, also preserves fine granularity to maintain dependencies with starting tokens and for retrieval of past context (Zhang et al., 4 Aug 2025).

The positional encoding matrix is defined as

$m$2

Region-specific indices for RoPE rotations are then assigned as follows. In the head region, $m$3 and $m$4. In the middle region,

$m$5

In the tail region, $m$6 and $m$7 (Zhang et al., 4 Aug 2025).

Two properties are explicitly stated for this construction: monotonicity and continuity across boundaries, with mathematical proof in Appendix B, and plug-in compatibility with efficient attention such as FlashAttention2. Methodologically, this scheme allocates fine-grained positional detail where locality and sequence boundaries matter, while using coarser granularity elsewhere. This suggests that LaMPE treats positional capacity as a limited resource to be distributed non-uniformly across the sequence.

4. Experimental evaluation and reported performance

LaMPE is evaluated on Llama2-7B-Chat with a 4K window, Llama3-8B-Instruct with an 8K window, and Llama3.1-8B-Instruct with a 128K window. The benchmarks are LongBench, L-Eval, InfinityBench, RULER, and PG-19. The baselines include RoPE, NTK-RoPE, YaRN, SelfExtend, DCA, ReRoPE, and STRING (Zhang et al., 4 Aug 2025).

The reported summary results are:

Model/Window	LaMPE (Avg)	Best Baseline (Avg)
Llama2-7B-Chat 4K	35.07	34.62 (SelfExtend)
Llama3-8B-Ins 8K	46.99	45.90 (YaRN)
Llama3.1-8B-Ins 128K	62.85	60.72 (STRING)

Across LongBench and L-Eval, LaMPE delivers the best average performance for each LLM, outperforming the best baseline by 0.45–1.09 points on LongBench and 0.99–2.54 points on L-Eval. On InfinityBench at 32K and 64K, LaMPE beats baselines and maintains higher stability as context increases, while other methods degrade. On RULER, LaMPE achieves top scores from window up to 128K, both within pretraining window and in extrapolation, including a reported boost of RoPE from 88.76 to 90.57. On PG-19, LaMPE either achieves lowest perplexity or is very close to the best, and suffers much less performance decay as context grows (Zhang et al., 4 Aug 2025).

These results are notable because the gains are not limited to extrapolation beyond the original training window. The RULER result explicitly shows improvement within the pretraining window. That observation counters the common assumption that training-free long-context extensions help only in out-of-distribution length regimes.

5. Ablations, robustness, and mechanism validation

The ablation studies reported for LaMPE isolate both major components of the method. Dynamic Mapping is described as critical: removing the sigmoid-based dynamic adaptation sharply reduces long-context performance as input grows. The tail region is also described as important for long-range dependencies, and removing it degrades tasks relying on start-end distance. Analysis of head and tail region hyperparameters shows performance is robust when head and tail are small compared to the mapping window, but recovers sharply when a small but reasonable tail is present (Zhang et al., 4 Aug 2025).

These observations align with the methodological intent of the design. The dynamic mapping component governs how much of the model’s effective context window is used for a given input length, while the head/middle/tail partition governs where that positional capacity is spent. A plausible implication is that the two components solve different failure modes of uniform remapping: one addresses input-length mismatch, and the other addresses non-uniform utility of positional resolution across the sequence.

The broader literature contains conceptually related but structurally different approaches. DPE, for example, is also training-free and adapts extrapolation non-uniformly, but it operates by detecting effective length for every RoPE dimension and identifying key dimensions using 2-norm Attention Contribution, then manipulating only those key dimensions’ position indices (Lu et al., 26 Apr 2025). BiPE instead disentangles intra-segment absolute encoding from inter-segment relative encoding, explicitly separating local within-segment order from global across-segment relationships (He et al., 2024). Ms-PoE assigns different scaling ratios to different attention heads to mitigate the lost-in-the-middle problem without fine-tuning or additional overhead (Zhang et al., 2024). Relative to those methods, LaMPE is defined by adaptive mapping length as a function of input length and by sequence-region-specific granularity.

6. Position within the positional-encoding literature

LaMPE belongs to a broader shift from static positional rules toward adaptive and structure-sensitive schemes. A survey of length extrapolation from the perspective of positional encoding highlights flexible PE that supports variable granularity and explicit length-control as an open and practically important research direction (Zhao et al., 2023). LaMPE directly addresses that agenda through adaptive long-context scaling without training and through explicit multi-grained allocation of positional resolution (Zhang et al., 4 Aug 2025).

The method also fits alongside work arguing that fixed positional encodings remain limited after training because they do not adapt to the given input data. DAPE, for instance, proposes a Data-Adaptive Positional Encoding that dynamically and semantically adjusts based on input context and learned fixed priors, and reports statistically significant improvements in both trained length and length generalization (Zheng et al., 2024). LaMPE does not use DAPE’s context-conditioned bias formulation; instead, it remains a training-free remapping method. Even so, both methods are responses to the same general limitation of static PE.

From an encyclopedic perspective, LaMPE is best understood not as an isolated RoPE tweak, but as a concrete instantiation of two broader design principles. The first is length-awareness: positional manipulation should depend on the actual input length rather than a fixed extrapolation rule. The second is multi-grained encoding: positional resolution should be distributed unevenly across the sequence so that fine-grained locality and long-range dependency capture are both retained. The paper’s reported results across LongBench, L-Eval, InfinityBench, RULER, and PG-19 place LaMPE among the training-free methods that operationalize those principles directly in RoPE-based LLMs (Zhang et al., 4 Aug 2025).