The Position Curse: LLMs Struggle to Locate the Last Few Items in a List

Abstract: Modern LLMs can find a needle in a haystack (locating a single relevant fact buried among hundreds of thousands of irrelevant tokens) with near-saturated accuracy, yet fail to retrieve the last few items in a short list. We call this failure the Position Curse. For instance, even in a two-line code snippet, Claude Opus 4.6 misidentifies the second-to-last line most of the time. To characterize this failure, we evaluated two complementary queries: given a position in a sequence (of letters or words), retrieve the corresponding item; and given an item, return its position. Each position is specified as a forward or backward offset from an anchor, either an endpoint of the list (its start or end) or another item in the list. Across both open-source and frontier closed-source models, backward retrieval substantially lags forward retrieval. To test whether this capability can be rescued by post-training, we constructed PosBench, a position-focused training dataset. LoRA fine-tuning improves both forward and backward retrieval and generalizes to a held-out code-understanding benchmark (PyIndex), yet absolute performance remains far from saturated. As LLM coding agents increasingly operate over large codebases where precise indexing becomes essential for code understanding and editing, position-based retrieval emerges as a key capability for future pretraining objectives and model design.

• The paper identifies that LLMs consistently fail in retrieving list items from backward positions, even in simple sequences.
• It reveals that while forward retrieval scales with model size, backward and relative queries remain substantially inaccurate despite basic counting success.
• Post-training interventions like LoRA boost performance moderately but do not eliminate the Position Curse, posing challenges for coding and indexing tasks.

The Position Curse: Analysis of LLMs’ Deficits in Position-Based Retrieval

Overview

This paper rigorously investigates a fundamental failure mode in current LLMs: their difficulty in position-based retrieval, especially for retrieving the last few items in a short list. While state-of-the-art LLMs attain near-ceiling accuracy on content-based retrieval tasks over vast contexts, they consistently misidentify items that require indexing from the sequence end or relative positions, a failure termed the “Position Curse.” The authors systematically dissect this phenomenon across model families, task variants, and post-training interventions, demonstrating both the persistence of the error and the limited effectiveness of available remedies. This work carries broad implications for LLM architecture, pretraining, and downstream applications, especially in code intelligence.

Experimental Formulation and Scope

The authors formalize position-based retrieval as a deterministic mapping: given an ordered sequence $S = [s_1, ..., s_L]$, the model must either retrieve the item at a specified position or return the index of a specified item. Queries are structured as:

• Position→Item: Given a position (with offset and anchor), return the corresponding item.
• Item→Position: Given an item, return its position relative to an anchor.

Tests span anchors at endpoints (start/end of the list) or relative to other items, and directions both forward and backward from anchors, across different sequence types (letters, words, code lines) and sequence lengths ($L = 5, 10, 20$). Critically, queries include both forward-referenced (e.g., the 2nd item from the start) and backward-referenced (e.g., the 2nd-to-last item) variants.

Empirical Findings

1. Position Curse Manifestation Across Models

In initial evaluations, both open- and closed-source LLMs—including Claude Opus 4.6, GPT, Qwen, Gemma, and Llama—demonstrate an acute and systematic defect: retrieval accuracy for backward queries is dramatically lower than for forward queries, even on trivial sequence lengths. For instance, Claude Opus misidentifies the second-to-last letter in a four-letter list over 27% of the time.

Figure 1: Example position-based queries and failures in models; even basic backward position retrieval fails for leading LLMs.

2. Counting Is Intact, but Relative Indexing Fails

The paper verifies that these models’ difficulties do not stem from basic counting deficits. On length-counting control tasks (e.g., "How many items are in the list?"), accuracy is saturated for nearly all tested models and sequence lengths.

Figure 2: Left—examples of task prompts; Center—Qwen confusion matrices illustrating diagonal dominance in forward queries and mass collapse for backward queries; Right—accuracy breakdown by model family.

3. Nature of Forward-Backward Asymmetry

Forward endpoint retrieval ($\mathsf{End}^{+}$, e.g., "the 3rd from start") is learned with moderate to high sample efficiency as model size grows; e.g., Qwen3.5 rises from 23% to 84% at $L=20$ as model scale increases. Backward endpoint retrieval ($\mathsf{End}^{-}$, e.g., "2nd from end") remains extremely low (5–16%) even in the largest models. Similar gaps appear for relative anchors. This strongly implicates a structural and data-driven origin, as opposed to mere capacity constraints.

Figure 2: Model confusion heatmaps show forward queries yield near-diagonal accuracy, while backward queries show dysfunction characterized by collapsed or streaked predictions.

4. Reasoning Traces Yield Limited Benefit

Allowing models to "think step by step" (chain-of-thought; up to 256 intermediate reasoning tokens) offers only partial and model-size-dependent relief. For smaller models, reasoning does not help and may slightly degrade accuracy. For the largest Qwen model, reasoning improves accuracy in backward cases (to as high as 94%), but does not consistently close the gap.

Figure 3: Comparison of mean retrieval accuracy under no-reasoning (gray) and chain-of-thought-enabled (orange) conditions; only large models exhibit gains from explicit reasoning.

5. Post-training: Fine-tuning and Generalization

The authors introduce PosBench, a position-focused training dataset, and experiment with both LoRA and full-parameter SFT adaptation. LoRA consistently improves position-based retrieval accuracy across multiple LLM families, but fine-tuning does not saturate accuracy, especially for backward or relative queries, and the capacity to generalize is limited.

Figure 4: Retrieval accuracy improvements with LoRA and SFT adaptation; major gains over pre-trained baselines but with persistent shortfall, especially on backward and relative queries.

Transfer to Code Understanding: PyIndex Benchmark

To assess real-world impact, they introduce PyIndex—a synthetic benchmark reflecting Python-style indexing semantics, with direct, backward, nested, expression, and chained indexing queries. Fine-tuned models more than double their base accuracy on this set (e.g., Qwen3.5-4B rises from 32% to 71% with LoRA), but performance remains far below ceiling, especially for complex cases like nested or backward indexing.

Figure 5: PyIndex accuracy across fine-tuning methods and model families, highlighting pronounced improvements yet large remaining gap to perfect retrieval.

Mechanistic and Theoretical Interpretation

The authors propose a mechanistic basis for the Position Curse grounded in current transformer architectures and residual stream interpretability:

• Counting is Directly Encoded: As tokens are processed, ordinal indices are effectively written into the residual stream; the model can read off the position of the last (or any forward) item directly.
• Backward Retrieval Requires Inefficient Computation: Backward operations necessitate position subtraction or offset arithmetic not natively represented in the model’s hidden state. Position information under rotary embeddings (RoPE) is introduced via rotation, not readily amenable to subtraction in the residual stream—limiting LLMs’ native ability to implement flexible backward addressivity.
• Fine-tuning Partially Mitigates but Does Not Eliminate This Limitation: Post-training yields some localized circuits for the data-distribution seen, with poor extrapolation to novel or hierarchical queries.

Implications for LLM Design and Use

Practical Implications:

• LLMs deployed as coding agents (refactoring, log inspection, code navigation) are currently unreliable for precise backward or relative retrieval tasks, even in contexts where the immediate context is minimal.
• This is highly relevant for applications such as automated code patching, program synthesis, and conversational coding assistants.

Theoretical Implications:

• The Position Curse identifies a core inductive bias mismatch between transformer-based LLMs and classical sequence-addressing machinery (as in Neural Turing Machines and explicit memory architectures).
• Integrating architectural innovations (e.g., enhanced position-addressability, new positional encoding schemes, or explicit index computation modules) or richer pretraining signals is essential to rectify this limitation.

Future Directions

Future work is required to engineer robust, compositional position-based retrieval mechanisms—either at the architectural or algorithmic level—in LLMs. Broader evaluation across diverse retrieval settings is warranted, and optimized training objectives or data curation aiming at position generalization may be necessary. Theoretical analysis of attention-based position encoding limitations should inform the next generation of LLM backbone models.

Conclusion

The Position Curse demonstrates that LLMs achieve near-optimal content-based retrieval in the long-context regime but remain fundamentally limited in position-based access, particularly in backward and relative addressing even for simple, short sequences. Current architectures and training pipelines do not endow LLMs with the analog of random-access indexability underpinning procedural reasoning and symbolic manipulation. Overcoming this limitation is critical for the reliable deployment of LLMs in real-world agent and coding scenarios and will require explicit advances in model architecture, training data, and evaluation methodology.