Layer-Specific Scaling of Positional Encodings

• The paper proposes a novel layer-specific scaling of positional encodings that independently modulates positional contributions across transformer layers, addressing decay and bias in uniform injection.
• The method employs independent layer normalization and per-layer scaling factors—applied in Vision Transformers and long-context language models—to overcome issues like the 'lost-in-the-middle' phenomenon.
• Empirical results demonstrate significant improvements in model performance, with gains across visual and language tasks validating the benefits of depth-aware positional control.

Layer-specific scaling of positional encodings denotes a family of mechanisms in which positional information is not injected with a single depth-independent rule, but is instead modulated as a function of layer. In the literature summarized here, this idea appears in several technically distinct forms: independent per-layer normalization and affine control over position embeddings in Vision Transformers, per-layer scaling of Rotary Position Embeddings in long-context LLMs, and, as a closely related adjacent formulation, scale-consistent coordinate encodings designed to remain invariant under generator-layer transformations in arbitrary-scale image synthesis (Yu et al., 2022, Wang et al., 6 Mar 2025, Lv et al., 26 Jun 2026, Ntavelis et al., 2022). The unifying objective is to preserve or rebalance positional information across depth when a uniform positional mechanism is either too rigid or produces undesirable decay, coupling, or distortion.

1. Conceptual and mathematical basis

The need for layer-specific scaling arises from different failure modes in different architectures. In a standard Vision Transformer, the input is formed as

$x_0=\alpha+\omega,$

where $\alpha\in\mathbb{R}^{N\times D}$ are patch embeddings and $\omega\in\mathbb{R}^{N\times D}$ is the position embedding. Encoder layers then apply the usual pre-normalized self-attention and feed-forward updates. The key observation in "Position Embedding Needs an Independent Layer Normalization" is that the default joining method applies the same LayerNorm affine transform to token embedding and PE, so the learned $(\gamma_\ell,\beta_\ell)$ must trade off between content and position information. Under the paper’s reparameterization,

$$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$

which means the positional component can only be globally scaled and shifted in the same way as the token component. This is presented as an expressiveness limitation of the default scheme (Yu et al., 2022).

In RoPE-based Transformers, the issue is different. RoPE rotates each query-key coordinate pair according to token index, so that attention depends on relative offset. With $q,k\in\mathbb{R}^d$ and a block-diagonal rotation matrix $R(n)$, one obtains attention terms of the form

$a_{ij}=q^\top R(i-j)k.$

The long-context papers attribute the "lost-in-the-middle" problem to rapid decay in the RoPE-induced long-range interaction as $|i-j|$ grows. Layer-specific scaling therefore acts on the positional transform itself, introducing a distinct scale factor per layer in order to slow the decay or rebalance attention toward middle-context tokens (Wang et al., 6 Mar 2025, Lv et al., 26 Jun 2026).

A related but not identical formulation appears in arbitrary-scale image synthesis. There, image generation is framed as sampling a continuous two-dimensional function $I_z$ at coordinates determined by center $\alpha\in\mathbb{R}^{N\times D}$0, scale $\alpha\in\mathbb{R}^{N\times D}$1, and resolution $\alpha\in\mathbb{R}^{N\times D}$2, and positional encoding is defined directly as a uniformly sampled grid of pixel-center coordinates. The emphasis is not per-layer rescaling parameters, but invariance of the positional grid to the generator’s layer transformations, so that translation and scale equivariances hold exactly through the network (Ntavelis et al., 2022). This suggests that the broader research theme is depth-aware control of positional geometry, even when the concrete mechanism differs.

2. Layer-adaptive position embedding in Vision Transformers

LaPE introduces two independent LayerNorms at every Transformer layer: one for the token stream and one for the position embedding. Let $\alpha\in\mathbb{R}^{N\times D}$3 be the output of layer $\alpha\in\mathbb{R}^{N\times D}$4, and let $\alpha\in\mathbb{R}^{N\times D}$5 be the shared fixed or learnable PE. The method defines

$\alpha\in\mathbb{R}^{N\times D}$6

followed by

$\alpha\in\mathbb{R}^{N\times D}$7

Because $\alpha\in\mathbb{R}^{N\times D}$8 and $\alpha\in\mathbb{R}^{N\times D}$9 are distinct at each depth, the model can amplify or attenuate the positional contribution differently in early and late layers. The paper names this mechanism Layer-adaptive Position Embedding, abbreviated as LaPE (Yu et al., 2022).

Implementation changes are minimal. In a standard block, $a_{ij}=q^\top R(i-j)k.$2 the first part becomes $a_{ij}=q^\top R(i-j)k.$3 Each $\omega\in\mathbb{R}^{N\times D}$0 adds $\omega\in\mathbb{R}^{N\times D}$1 scalars. The paper gives the concrete example that, for a 12-layer model with $\omega\in\mathbb{R}^{N\times D}$2, this is $\omega\in\mathbb{R}^{N\times D}$3 extra parameters, with FLOPs increased only by an extra LN on $\omega\in\mathbb{R}^{N\times D}$4, which can be precomputed at inference (Yu et al., 2022).

The paper also distinguishes LaPE from simply assigning different positional embeddings to different layers. In its ablations, LaPE outperforms adding layer-distinct $\omega\in\mathbb{R}^{N\times D}$5 embeddings because it keeps $\omega\in\mathbb{R}^{N\times D}$6 shared while allowing per-layer rescaling through normalization and affine parameters. This directly addresses the misconception that depth-aware positional control necessarily requires duplicating the positional representation itself (Yu et al., 2022).

3. Layer-specific RoPE scaling in long-context LLMs

The long-context LLM literature formulates layer-specific positional scaling as a remedy for positional attention bias. In "Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling," each Transformer layer $\omega\in\mathbb{R}^{N\times D}$7 receives its own scale $\omega\in\mathbb{R}^{N\times D}$8, so that relative distance is effectively reduced from $\omega\in\mathbb{R}^{N\times D}$9 to $(\gamma_\ell,\beta_\ell)$0. One implementation is

$(\gamma_\ell,\beta_\ell)$1

leading to

$(\gamma_\ell,\beta_\ell)$2

The paper also gives an alternative formulation, $(\gamma_\ell,\beta_\ell)$3, and states that empirically both formulations slow the fall-off of attention with distance (Wang et al., 6 Mar 2025).

"Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling" presents LPES, which likewise assigns each layer $(\gamma_\ell,\beta_\ell)$4 its own scale $(\gamma_\ell,\beta_\ell)$5. The paper defines a modified rotation

$(\gamma_\ell,\beta_\ell)$6

and attention

$(\gamma_\ell,\beta_\ell)$7

In its implementation guidance, LPES states that one can replace the angle computation $(\gamma_\ell,\beta_\ell)$8 with $(\gamma_\ell,\beta_\ell)$9 for layer $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$0, or precompute $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$1 per layer for all positions (Lv et al., 26 Jun 2026).

Both papers tie the method to the "lost-in-the-middle" phenomenon, but with slightly different emphasis. The 2025 paper attributes the issue to rapid long-term decay in RoPE and focuses on improving middle-context utilization; the 2026 paper adds the language of positional attention bias and "attention waves," stating that layer-specific scaling can damp oscillatory patterns as well as compensate long-term decay (Wang et al., 6 Mar 2025, Lv et al., 26 Jun 2026). A plausible implication is that the same per-layer scaling mechanism is being used to target both monotone attenuation and structured distortions in the attention landscape.

4. Search and parameterization of per-layer scales

A central practical problem is that independently choosing one scale per layer produces a high-dimensional search space. The two long-context papers therefore constrain the scale profile to a Bézier curve. In the 2025 formulation, layers are indexed $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$2, $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$3, and the curve is

$$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$4

with control points $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$5, $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$6, $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$7, and strictly increasing $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$8. The scale is then taken as the $$LN_\ell(x_\ell+\omega)=\lambda_1\cdot LN_\ell(x_\ell)+\lambda_2\cdot LN_\ell(\omega)+\lambda_3\cdot\beta_\ell,$$9-coordinate at each $q,k\in\mathbb{R}^d$0. The paper states that discretizing $q,k\in\mathbb{R}^d$1 and $q,k\in\mathbb{R}^d$2 reduces brute-force complexity from $q,k\in\mathbb{R}^d$3 to $q,k\in\mathbb{R}^d$4; for $q,k\in\mathbb{R}^d$5 and $q,k\in\mathbb{R}^d$6, this is a $q,k\in\mathbb{R}^d$7 reduction (Wang et al., 6 Mar 2025).

LPES fixes degree $q,k\in\mathbb{R}^d$8, so that four control points generate all layer scales. It assigns each layer

$q,k\in\mathbb{R}^d$9

The genetic algorithm maintains a population of Bézier-control-point sets with monotonic $R(n)$0 and $R(n)$1, evaluates fitness on beginning, middle, and end subsets, and produces offspring through crossover and mutation while rejecting invalid candidates (Lv et al., 26 Jun 2026).

The optimization objective also reflects the targeted positional pathology. In the 2025 paper, fitness is

$R(n)$2

where $R(n)$3 are accuracies when the answer document is in the first, middle, and last third of the context. In the 2026 LPES paper, fitness is

$R(n)$4

with weights summing to $R(n)$5 over beginning, middle, and end sub-datasets (Wang et al., 6 Mar 2025, Lv et al., 26 Jun 2026). In both cases, the search objective is explicitly position-structured rather than a generic validation loss.

5. Reported empirical behavior

The reported gains differ by modality, architecture family, and task, but they are consistently framed as benefits from permitting depth-dependent positional control.

Method	Setting	Reported result
LaPE	ViT-Lite on CIFAR-10	+0.94% top-1
LaPE	CCT on CIFAR-100	+0.98%
LaPE	DeiT-Ti on ImageNet-1K	+1.72%
Layer-specific RoPE scaling	Vicuna-7B-v1.5 on Key-Value Retrieval	74.8% to 94.8%
LPES	Vicuna on MDQA	59.9 to 63.6
LPES	Vicuna on Key-Value Retrieval	80.9 to 88.8

For LaPE, the experiments span CIFAR-10/100 and ImageNet-1K, with models including ViT-Lite, CVT, CCT, DeiT-Ti/S, T2T-ViT-7, DeiT-distill, Swin-Ti/S, and CeiT-Ti/S. The paper reports that gains also extend to sinusoidal or relative PE, including +0.19% on T2T-ViT-7 and +0.30% on Swin-Ti. It further reports faster convergence, and for DeiT-Ti, the gap between 1-D sinusoidal and learnable PE shrinks from 3.84% to 1.04% once LaPE is used (Yu et al., 2022).

For long-context modeling, the 2025 paper reports on MDQA, Key-Value Retrieval, ZeroSCROLLS, and PG19. On Key-Value Retrieval with Vicuna-7B-v1.5, the baseline is 74.8%, PI is 90.8%, Ms-PoE is 79.8%, and the layer-specific method reaches 94.8%, which the paper describes as +20.0 points over baseline and +4.0 over PI. On MDQA with the same model, it reports 61.9% for baseline, 63.3% for PI, 64.8% for Ms-PoE, and 64.6% for the proposed method. On ZeroSCROLLS, average scores are reported as 16.91 to 19.35 for Vicuna, 19.35 to 21.01 for LLaMA-2-7B-chat, and 20.34 to 22.12 for StableBeluga-7B (Wang et al., 6 Mar 2025).

LPES reports similar but not identical benchmarks: MDQA, Key-Value Retrieval, ZeroSCROLLS, L-Eval, and MMLU/C-Eval. It states that on Vicuna, average MDQA accuracy improves from 59.9 to 63.6 and Key-Value Retrieval from 80.9 to 88.8, while general capabilities on MMLU/C-Eval change negligibly, by less than 0.5%. For efficiency, LPES reports zero latency over baseline and gives the concrete Vicuna timing comparison: 0.71 s/sample for LPES, 1.03 s/sample for Ms-PoE, and 1.72 s/sample for MoICE (Lv et al., 26 Jun 2026).

The extrapolation ablations in the 2025 paper are particularly important because they show that layer-specific interpolation is not merely an in-distribution calibration device. On PG19 at 5K–8K tokens, both "Ours-PI" and "Ours-Dy-NTK" achieve lower perplexity than uniform PI or Dynamic-NTK for LLaMA-2-7B-hf and Vicuna-7B-v1.5, and the paper states that this confirms that layer-wise scales better correct OOD attention shifts (Wang et al., 6 Mar 2025).

6. Related formulations, limits, and adjacent developments

A common misconception is that positional scaling is a single technique with a single implementation. The literature here shows at least three distinct mechanisms. LaPE rescales the positional contribution indirectly through independent normalization and affine parameters while keeping the positional embedding itself shared across layers. The RoPE papers alter the positional transform by inserting per-layer scale factors into the rotary mechanism. The arbitrary-scale image synthesis work does not assign a distinct learned scale per layer; instead, it constructs a positional encoding whose spatial period remains exact under pad-free convolutions and bilinear upsampling, thereby making scale consistency a layer-invariant property rather than a layer-specific parameter (Yu et al., 2022, Wang et al., 6 Mar 2025, Ntavelis et al., 2022).

In "Arbitrary-Scale Image Synthesis," image generation is written as sampling

$R(n)$6

with $R(n)$7, and positional encoding is

$R(n)$8

Doubling grid resolution halves spacing exactly, which the paper identifies as the core of scale consistency. The generator uses zero-padding-free $R(n)$9 convolutions and bilinear upsampling without corner alignment, together with feature unfolding. With $a_{ij}=q^\top R(i-j)k.$0, the layer sizes satisfy

$a_{ij}=q^\top R(i-j)k.$1

The paper argues that this preserves exact translation and scale equivariance at each layer because padding and resizing never distort the positional grid itself (Ntavelis et al., 2022).

That image-synthesis work also illustrates a broader lesson about positional control. Inter-scale augmentations, partial generation training, and scale-aligned noise are used so that the positional encoding functions as the sole scale anchor. The paper reports that grid-sampling one large noise map per layer raises Self-SSIM by 0.02–0.03 compared to random or constant reuse, and on FFHQ a single generator trained with scales 256 and 384 can generate at any integer scale between 128 and 1024, with Self-SSIM(5 k) scores of 0.85–0.88 at unseen scales such as 320, 384, 448, and 512 (Ntavelis et al., 2022). This suggests that depth-aware handling of positional structure is not confined to Transformer attention; it also appears in generative architectures where scale and translation equivariance must remain coherent across layers.

Across these works, the main point of agreement is not a single formula but a common diagnosis: positional information that is injected uniformly across depth can become either under-expressive, over-coupled to content, or biased toward particular regions of the input. Layer-specific scaling—or, in the adjacent image-synthesis setting, layer-consistent positional geometry—is the mechanism used to restore control over how positional structure evolves through depth (Yu et al., 2022, Wang et al., 6 Mar 2025, Lv et al., 26 Jun 2026, Ntavelis et al., 2022).