Shifted Position Embedding (ShiftPE)

• Shifted Position Embedding (ShiftPE) is a positional encoding framework that reassigns indices—either stochastically or deterministically—to mitigate reliance on fixed absolute positions.
• It encompasses variants such as SHAPE for sequence Transformers, DiT ShiftPE for image generation, and local-frame formulations for language models, each addressing unique limitations.
• Empirical results show that ShiftPE enhances extrapolation performance, prevents positional collisions, and introduces minimal overhead during training.

Shifted Position Embedding (ShiftPE) denotes a family of positional encoding schemes in which positional indices are deliberately reassigned so that a model is less dependent on fixed absolute locations or, in multimodal settings, does not suffer positional collisions. In the recent literature, the term spans at least three technically distinct constructions: SHAPE, or Shifted Absolute Position Embedding, for sequence Transformers (Kiyono et al., 2021); the non-overlapping Rotary Position Embedding shift introduced for attention sharing in Diffusion Transformers (DiT) (Zhang et al., 21 Sep 2025); and shift-invariant relative-offset formulations proposed as a remedy for the brittleness of learned absolute position embeddings in LLMs (Sinha et al., 2022). The common theme is positional reindexing, but the operational setting, mathematical form, and failure mode addressed differ substantially.

1. Scope of the term and major formulations

The phrase “Shifted Position Embedding” is not monosemous. In sequence modeling, SHAPE replaces the positional vector for token position $i$ by the vector for $i+k$, where $k$ is randomly sampled during training, so that the model cannot rely on a fixed absolute slot (Kiyono et al., 2021). In DiT-based image generation, ShiftPE assigns each image in a shared-attention batch a disjoint block of positional indices, preventing two images from ever sharing the same Rotary Position Embedding (RoPE) (Zhang et al., 21 Sep 2025). In language-model analysis, shifted-position evaluation is used to expose the fragility of learned absolute position embeddings, and a related “ShiftPE” proposal replaces $P(i)$ by a function of the local offset $i-s$ from a sentence start index $s$ (Sinha et al., 2022).

Variant	Core definition	Primary setting
SHAPE	$\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$ with $k\sim\mathcal{U}\{0,\dots,K\}$	Sequence Transformers
DiT ShiftPE	$P_{\text{shifted}}(i,k)=P(i+k\cdot N)$	Diffusion Transformers
Local-frame ShiftPE	$P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$	Language-model robustness

This terminological plurality matters because the three schemes do not intervene at the same level. SHAPE is a training-time stochastic perturbation of otherwise standard absolute embeddings; DiT ShiftPE is a deterministic reallocation of RoPE indices across images; the local-frame proposal is a relative-position parameterization that explicitly centers each sentence or prompt in its own coordinate frame.

2. SHAPE as shifted absolute position embedding in Transformers

SHAPE was introduced to address two limitations of common positional schemes in Transformers: the poor extrapolation of Absolute Position Embeddings (APE) and the architectural and computational overhead of Relative Position Embeddings (RPE) (Kiyono et al., 2021). The paper identifies shift invariance as the key desideratum. A function $i+k$0 of a sequence of length $i+k$1 is shift-invariant if, for any offset $i+k$2, shifting every position by $i+k$3 leaves the outputs unchanged, up to global re-indexing:

$i+k$4

when all indices are in range.

For standard sinusoidal APE with model dimension $i+k$5, position $i+k$6 is encoded as $i+k$7 with

$i+k$8

SHAPE leaves this embedding form unchanged and instead shifts the index:

$i+k$9

$k$0

During training, one offset $k$1 is drawn for the source sequence and an analogous independent offset $k$2 for the target sequence. The encoder input becomes

$k$3

At inference time, $k$4, so the model defaults to vanilla APE.

Architecturally, SHAPE requires no modification of the self-attention equations and no new parameter tensors. The implementation is a shift of the integer index passed to an existing APE lookup or sinusoid formula. The paper recommends a maximum shift $k$5 of 500 and reports robustness to $k$6, with one $k$7 drawn per sequence per forward pass. Computationally, SHAPE has the same $k$8 complexity as APE, plus the negligible cost of adding $k$9 to $P(i)$0, whereas RPE modifies self-attention and adds relative-distance tables of size $P(i)$1 for every head (Kiyono et al., 2021).

The theoretical intuition is that random offsetting prevents the network from associating a task-relevant signal with any absolute index. Because every training example is seen at many shifted locations, the model must internalize information that is stable under uniform shifts, which in practice means relying on relative distances rather than absolute slots.

3. Empirical behavior of SHAPE in machine translation

The empirical evaluation of SHAPE was conducted on the WMT 2016 English$P(i)$2German machine-translation benchmark under three settings: Vanilla, Extrapolate, and Interpolate (Kiyono et al., 2021). Vanilla uses all sentence pairs and evaluates on newstest2010–2016. Extrapolate trains on pairs with length $P(i)$3 subwords and evaluates on the full-length newstest sets. Interpolate constructs long sequences by concatenating blocks of 10 sentences separated by a special token. All models are standard Transformer-base models, tokenized and BPE-segmented as in Ott et al. (2018), with evaluation by detokenized sacreBLEU.

On the aggregated newstest2010–2016 averages, APE (sinus.) obtained 30.46 BLEU on Vanilla, 29.22 on Extrapolate, and 38.23 on Interpolate, with relative speed 1.00$P(i)$4. RPE (shaw) obtained 30.54 on Vanilla and 29.86 on Extrapolate, with relative speed 0.91$P(i)$5; Interpolate is marked as infeasible because of slowdown. SHAPE with $P(i)$6 obtained 30.49 on Vanilla, 29.80 on Extrapolate, and 39.09 on Interpolate, with relative speed 0.99$P(i)$7 (Kiyono et al., 2021). Within the paper’s interpretation, all three methods tie within 0.1 BLEU on Vanilla; on Extrapolate, APE drops by about 1.2 BLEU relative to RPE while SHAPE matches RPE; on Interpolate, SHAPE improves on APE by about 0.9 BLEU.

Length-conditioned analysis showed that both RPE and SHAPE outperform APE on source lengths beyond those seen in training under Extrapolate and on the long sequences in Interpolate. Two invariance checks are central. First, swapping the order of concatenated sentences, described as a global cyclic “shift,” drops APE’s BLEU by 8 points but only about 1.5 points for SHAPE. Second, cosine similarity of hidden states under different random $P(i)$8 values is nearly 1.0 for SHAPE, whereas APE’s hidden states vary widely across $P(i)$9 (Kiyono et al., 2021).

The runtime profile is correspondingly favorable. SHAPE adds no parameters and less than 2% runtime overhead relative to APE, whereas RPE adds extra parameter tensors and slows training by about 9–10% on WMT En$i-s$0De. On the long-sequence Interpolate setting, SHAPE is reported as about $i-s$1 faster to train than RPE, which failed to complete (Kiyono et al., 2021). In this setting, ShiftPE is best understood as a minimal intervention that approximates the shift-invariance benefits of relative methods without their attention-level complexity.

4. ShiftPE in Diffusion Transformers and attention sharing

A distinct use of ShiftPE was introduced for DiT-based image generation, where the immediate problem is not length extrapolation but the failure of naive attention sharing across images (Zhang et al., 21 Sep 2025). In U-Net–based diffusion models, self-attention keys and values from a reference image can be shared into target-image generation to encourage style consistency. A direct transfer of this strategy to DiT concatenates reference and target keys and values inside a Multi-Modal Attention block. However, DiT mixes image and text tokens and applies RoPE to encode absolute 2D image location.

The failure mode is positional collision. If a token at coordinate $i-s$2 in the reference and a token at $i-s$3 in the target receive the identical RoPE, then attention is driven to the same spatial coordinate irrespective of content or text prompt, producing severe content leakage and collapse (Zhang et al., 21 Sep 2025). ShiftPE resolves this by allocating non-overlapping position-index blocks to each image. If $i-s$4 is the number of image tokens and $i-s$5 is the original rotary embedding for flattened index $i-s$6, then for image $i-s$7:

$i-s$8

In 2D terms, the reference image occupies $i-s$9, the first target occupies $s$0, the second target occupies $s$1, and so on.

The operational consequence is that a target query $s$2 and a reference key $s$3 no longer share identical rotary angles. Attention therefore cannot “cheat” by maximizing the dot product at the same spatial coordinate. Instead, it is free to distribute over semantically relevant keys. The paper describes this as a transition from attention maps that remain peaked at the exact coordinate under naive RoPE sharing to maps that fan out to semantically related regions under ShiftPE (Zhang et al., 21 Sep 2025).

ShiftPE is inserted at the point where RoPE is applied. It adds zero learned parameters and only trivial integer arithmetic, because the only change is the replacement of $s$4 by $s$5. This shift underpins Advanced Attention Sharing (AAS): selectively sharing only image-token keys and values, scaling reference keys by a factor $s$6, and applying sharing only in style-sensitive blocks $s$7. The paper states that without ShiftPE these AAS techniques would still collapse due to positional collision (Zhang et al., 21 Sep 2025).

Empirically, the unmodified DiT (“Original,” no sharing, no ShiftPE) scores $s$8, $s$9, and $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$0. Naive sharing without ShiftPE literally collapses. With ShiftPE and sharing only in Group 2 layers, style consistency rises to $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$1 and $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$2. The full AAS suite, with ShiftPE as a prerequisite, yields $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$3, $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$4, and $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$5, compared with StyleAligned at $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$6 and $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$7 (Zhang et al., 21 Sep 2025).

5. Shifted-position evaluation and local-frame remedies in LLMs

“The Curious Case of Absolute Position Embeddings” studies the fragility of learned APEs under positional shifts and uses “ShiftPE” in both diagnostic and remedial senses (Sinha et al., 2022). In the evaluated models, token-level embeddings are formed as

$\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$8

where $\mathrm{PE}(i)\rightarrow \mathrm{PE}(i+k)$9 is the token embedding matrix and $k\sim\mathcal{U}\{0,\dots,K\}$0 is the learned absolute position embedding matrix.

The paper introduces a shifted-position evaluation protocol, also called a “phase-shift” or ShiftPE test. In BLiMP grammatical acceptability, a sentence that would normally occupy positions $k\sim\mathcal{U}\{0,\dots,K\}$1 is shifted to $k\sim\mathcal{U}\{0,\dots,K\}$2, while special tokens are handled so as to avoid artifacts. In in-context learning, the entire prompt and any exemplars are offset by $k\sim\mathcal{U}\{0,\dots,K\}$3. In fine-tuning, models are trained at one shift $k\sim\mathcal{U}\{0,\dots,K\}$4 and evaluated across all four shifts (Sinha et al., 2022).

The reported degradation is substantial. On BLiMP, most models are near 90% at $k\sim\mathcal{U}\{0,\dots,K\}$5, with RoBERTa_BASE near 95%; by $k\sim\mathcal{U}\{0,\dots,K\}$6, RoBERTa_BASE drops by about 5 points, GPT2_MEDIUM by about 20 points, and OPT$k\sim\mathcal{U}\{0,\dots,K\}$7 by about 15 points. On six in-context learning benchmarks, moving to $k\sim\mathcal{U}\{0,\dots,K\}$8 drops zero-shot accuracy by an absolute 5–20% in many tasks, and at $k\sim\mathcal{U}\{0,\dots,K\}$9 the degradation can reach 30–40%; five-shot performance also degrades by 2 to 25 points. In fine-tuning, models trained at $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$0 and tested at $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$1 lose 10–25% relative accuracy or Matthews correlation, and out-of-distribution shifts can cause dramatic falloffs, especially on CoLA, where the decline exceeds 30 points (Sinha et al., 2022).

To mitigate this brittleness, the paper recommends replacing or augmenting APEs with a shift-invariant relative offset mechanism. A minimal design introduces a learned relative table $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$2 and computes

$P_{\text{shifted}}(i,k)=P(i+k\cdot N)$3

where $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$4 is the start index of the current sentence or prompt. The more general form is

$P_{\text{shifted}}(i,k)=P(i+k\cdot N)$5

with $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$6 implemented, for example, as a small MLP or rotary embedding. The paper also proposes a hybrid form,

$P_{\text{shifted}}(i,k)=P(i+k\cdot N)$7

where $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$8 is a learned scalar (Sinha et al., 2022).

The implementation rationale is explicit. A relative table $P_{\text{shifted}}(i,k)=P(i+k\cdot N)$9 can be more parameter-efficient than a full absolute table when $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$0; computing $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$1 is $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$2; the scheme aligns every sentence to a local coordinate frame; and task-specific updates to the smaller relative table can generalize across relative positions rather than a single absolute slot. The recommendations include detecting or predicting the sentence start $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$3, optionally combining relative and absolute components, randomizing $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$4 during pretraining, and always computing $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$5 during fine-tuning and inference (Sinha et al., 2022).

6. Comparative interpretation, misconceptions, and limitations

A frequent misconception is that ShiftPE refers to a single standardized positional encoding. The literature instead uses the term for several non-equivalent mechanisms. In SHAPE, the shift is a stochastic scalar sampled once per sequence per forward pass; in DiT ShiftPE, the shift is a deterministic image-specific block offset; in local-frame ShiftPE, the shift is a recentering of indices around a sentence or prompt start (Kiyono et al., 2021, Zhang et al., 21 Sep 2025, Sinha et al., 2022). This suggests a unifying principle—index reassignment to reduce harmful dependence on absolute positions—but not a single canonical algorithm.

The limitations are likewise setting-specific. SHAPE introduces a maximum-shift hyperparameter $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$6; very small $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$7 may under-cover the test-time absolute index range, and dedicated learnable RPE may still outperform SHAPE on some tasks (Kiyono et al., 2021). In the DiT setting, very large $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$8 can harm text alignment, and sharing in the earliest layers can introduce artifacts; hence the emphasis on layer-selective application in style-sensitive blocks, such as mid-to-late layers (Zhang et al., 21 Sep 2025). In the local-frame language-model formulation, one must detect or predict the sentence start $P_{\text{ShiftPE}}(i,s)=f(i-s;\theta_{\text{rel}})$9, and the paper therefore also considers hybrid schemes that retain some absolute positional bias through a scaled absolute term (Sinha et al., 2022).

Open directions also diverge. For SHAPE, proposed future work includes combining the method with learned or hybrid relative–absolute schemes, extending it beyond machine translation to language modeling and summarization, and integrating it into lightweight attention variants (Kiyono et al., 2021). For DiT ShiftPE, the same block-allocation idea already generalizes to more than two images by assigning each image its own $i+k$00 offset (Zhang et al., 21 Sep 2025). For the local-frame approach, the principal recommendation is to enforce shift invariance systematically during pretraining by randomizing sentence starts, thereby guaranteeing that any sentence at any context index maps to the same relative embeddings (Sinha et al., 2022).

Taken together, these strands of research establish ShiftPE not as a single model component but as a recurrent design pattern for overcoming absolute-position brittleness. In sequence Transformers it approximates the extrapolation behavior of relative methods while retaining the simplicity of APE; in DiT it prevents RoPE-induced content leakage during attention sharing; and in language-model analysis it serves both as a stress test for absolute embeddings and as a blueprint for more robust local-coordinate positional schemes.