---
title: 'PosBench: Addressing the Position Curse'
url: https://www.emergentmind.com/topics/posbench
type: topic
---

# PosBench: Addressing the Position Curse

Searching arXiv for the PosBench source paper and closely related context.
arxiv_search(query="2605.07127 OR \"The Position Curse: LLMs Struggle to Locate the Last Few Items in a List\"", max_results=5)
PosBench is a position-focused supervised fine-tuning dataset introduced to test whether the central failure mode identified in "The Position Curse: LLMs Struggle to Locate the Last Few Items in a List" can be repaired by post-training [2605.07127]. It is designed around **position-based retrieval** over ordered sequences, rather than content retrieval or counting. In the underlying study, modern LLMs are reported to be strong at content-based retrieval in long contexts and near-saturated on simple counting, yet to perform poorly on short-sequence positional access, especially when retrieval is specified **backward** from the end of a list or **relative** to another item. PosBench serves both as a training corpus and as an in-distribution evaluation suite for this capability gap, with particular motivation from code understanding and editing, where exact identification of lines, branches, list elements, and slices by position is often required.

## 1. Position-based retrieval and the “Position Curse”

The motivating phenomenon for PosBench is the **Position Curse**: large language models can often locate a single relevant fact in very long contexts, yet fail to retrieve the last few items in a short list. The paper treats this as a distinct capability deficit rather than a simple counting failure. Counting on the same short lists is described as near-saturated, whereas positional retrieval remains poor, especially in **backward retrieval** settings.

The core asymmetry is between **forward retrieval** and **backward retrieval**. Forward retrieval means indexing “from the beginning” or “after” a reference item. Backward retrieval means indexing “from the end” or “before” a reference item. The paper reports that, across both open-source and frontier closed-source models, backward retrieval substantially lags forward retrieval. This asymmetry is central to PosBench’s design: the dataset intentionally concentrates supervision on the settings that most strongly expose the failure.

The coding motivation is explicit. Code agents routinely need to identify exact positions in sequences of lines or program elements, not merely retrieve semantically relevant content. A plausible implication is that positional access is a low-level capability whose failure can propagate into broader code-understanding and code-editing errors, even when higher-level semantic reasoning is strong.

## 2. Formal task family underlying PosBench

The paper formalizes position-based retrieval over an ordered sequence

$$
S = [s_1,\ldots,s_L].
$$

A query specifies an offset $n \ge 1$, an anchor, and a direction. Anchors are either an endpoint of the sequence or another item in the sequence. Endpoint-anchored queries are denoted by $\mathsf{End}$ and relative-item-anchored queries by $\mathsf{Rel}$; forward and backward directions are denoted by superscripts $+$ and $-$.

The four index operators are

$$
\mathsf{End}^{+}: S[n], \qquad
\mathsf{End}^{-}: S[-n], \qquad
\mathsf{Rel}^{+}: S[r+n], \qquad
\mathsf{Rel}^{-}: S[r-n],
$$

with

$$
S[-n] \equiv s_{L-n+1},
$$

and offsets sampled so that the resulting index remains in $[1,L]$.

These operators support two exact retrieval task definitions. The first is **position$\rightarrow$item**, in which a position is specified and the model must return the corresponding item. The second is **item$\rightarrow$position**, in which the target item is given and the model must return the offset $n$ relative to the specified anchor and direction. The same operator family is used for both training and evaluation.

The framework covers endpoint and relative anchors, forward and backward directions, and multiple item types including letters and words. Examples in the paper include queries such as “the 2nd from the end” and “one position before $X$.” PosBench is therefore not a single prompt format, but a structured task family defined by operator choice, anchor type, direction, sequence type, and offset.

## 3. Dataset composition and construction

In the appendix, PosBench is described as a mixed supervised fine-tuning corpus combining three sources of ordered structure: synthetic retrieval examples, code-retrieval examples, and adapted real-data examples. The dataset contains **20K synthetic retrieval examples**, **4K code-retrieval examples**, and **46K adapted real-data examples**.

| Component | Size | Source and role |
|---|---:|---|
| Synthetic component $\mathcal{D}_{\mathrm{syn}}$ | 20K | Controlled coverage of the operator family |
| Code component $\mathcal{D}_{\mathrm{code}}$ | 4K | Retrieval over structured code-line sequences |
| Adapted real-data component $\mathcal{D}_{\mathrm{adapt}}$ | 46K | Linguistically diverse ordered structures from existing instruction and code data |

The synthetic component covers both position$\rightarrow$item and item$\rightarrow$position tasks, for both endpoint anchors $\mathsf{End}^{\pm}$ and relative anchors $\mathsf{Rel}^{\pm}$. Sequence lengths are sampled from $[10,50]$. Items come from multiple categorical pools, including letters, digits, animals, fruits, cities, elements, languages, and instruments. Prompt phrasing and list formatting are randomized, and the corpus includes a small number of multi-turn list conversations so that positional queries also appear in more natural dialog contexts.

The code component uses **BigCode self-oss-instruct-sc2-exec-filter-50k**. Each usable code snippet is split into non-empty lines; snippets with fewer than five lines are discarded; longer snippets are windowed into contiguous spans of **5–30 lines**; and positional queries are instantiated over those code-line sequences. The basic item type in this component is therefore the **code line**, directly matching the paper’s code-understanding motivation.

The adapted real-data component extracts naturally ordered structures from **Open-Orca / SlimOrca**, **OpenHermes-2.5**, and **tiny-codes**. The converted structures include numbered lists, bullet lists, markdown tables, and code blocks, provided they contain at least **five usable items/rows/lines**. Extracted items are deduplicated, kept short, and reformatted before query generation. For tiny-codes, the original dialogue context is preserved and the positional query is appended as a follow-up turn.

The sampling policy explicitly overrepresents difficult settings. The construction sets

$$
P(\mathrm{forward}) = 0.3, \qquad P(\mathrm{endpoint}) = 0.3,
$$

so the complementary cases—especially backward and relative addressing—appear more often. The paper states that those settings “most strongly expose the Position Curse.” No separate curriculum schedule is described.

## 4. Prompting, supervision, and fine-tuning protocol

PosBench examples instantiate the same operator family used in evaluation. The task space includes position$\rightarrow$item and item$\rightarrow$position, forward and backward indexing, endpoint and relative anchors, and sequence types ranging from synthetic categorical lists to natural ordered text structures and code-line sequences. The paper does not provide a single canonical template for all PosBench samples; instead, prompt wording and formatting are randomized.

At evaluation time, prompts are **three-shot**. Each test query is preceded by **three in-context demonstrations** of the same operator using independently sampled sequences. The stated purpose is to remove ambiguity about conventions such as 1-based indexing and answer format, so that failures reflect positional retrieval rather than prompt misunderstanding.

The fine-tuning objective uses **answer-span supervision** rather than supervising the entire assistant response. Let $m_t \in \{0,1\}$ indicate whether token $t$ belongs to the target answer span. The retrieval loss is

$$
\mathcal{L}_{\mathrm{retrieval}} = -\sum_t m_t \log p_\theta(y_t \mid y_{<t}, x).
$$

All non-answer tokens are masked out, so only answer tokens contribute to the loss. The paper explains that this concentrates learning on deterministic position lookup rather than on prompt scaffolding or stylistic response framing.

Evaluation uses exact-match accuracy after task-specific parsing. Accuracy over a trial subset $\mathcal{T}$ is defined as

$$
\operatorname{Acc}(\mathcal{T}) =
\frac{1}{|\mathcal{T}|}\sum_{t \in \mathcal{T}}
\mathbf{1}[\hat{y}^{(t)} = y^{(t)}].
$$

Per-offset analyses use subsets

$$
\mathcal{T}_n = \{ t : n^{(t)} = n \}.
$$

Responses are normalized into the appropriate answer space: for item retrieval, outputs are matched against candidate items in the sequence; for position or count tasks, the first integer answer is taken; and for code-style short answers, formatting artifacts are stripped.

For matched fine-tuning comparisons, the default backbone is **Qwen3.5-4B**. The LoRA configuration uses rank $r=32$, $\alpha=32$, dropout $=0$, and is applied to attention projections $(q,k,v,o)$ and MLP projections (gate, up, down). The training hyperparameters are **1 epoch**, learning rate $10^{-4}$, weight decay $0.05$, per-device batch size 16, gradient accumulation 2, bf16 precision, maximum sequence length 512, cosine learning-rate decay, 3% warmup, and seed 42.

The full-parameter supervised fine-tuning baseline uses **0.2 epochs**, learning rate $2 \times 10^{-5}$, weight decay $0.1$, per-device batch size 4, gradient accumulation 8, gradient clipping 1, and NEFTune noise $\alpha=5$. The paper’s reported comparison is that LoRA is more reliably beneficial than full-parameter supervised fine-tuning on PosBench.

## 5. Evaluation design and empirical findings

PosBench evaluation uses held-out sequences sampled independently from the training corpus [2605.07127]. Each evaluation condition fixes the query type, anchor type, indexing direction, item type, prompt variant, and sequence length $L \in \{5,10,20\}$. For each condition, the paper samples **50 independent sequences**, and for each sequence instantiates all valid offsets. The paper does not elaborate a distinct validation split in detail; the described split is between the training corpus and independently sampled held-out evaluation conditions.

Within the paper, PosBench is used in two roles. First, it is a **training corpus** for targeted adaptation. Second, it is an **in-distribution evaluation suite** for the same underlying operator family. The headline in-distribution evaluation shown in Figure 4 uses **letter sequences**, **endpoint anchoring**, $L=20$, and averages over four tasks: forward position$\rightarrow$item, backward position$\rightarrow$item, forward item$\rightarrow$position, and backward item$\rightarrow$position.

The main empirical conclusion is that **LoRA fine-tuning on PosBench reliably improves position-based retrieval across model families**. Figure 4 is reported to show consistent improvement over the base model for **Qwen3.5-4B**, **Llama 3.2-3B**, **Gemma4 E2B**, and **Ministral 3-3B**. By contrast, full-parameter fine-tuning does not produce similarly consistent gains.

The appendix’s per-query-position analyses indicate that LoRA improves accuracy broadly across queried offsets, but backward conditions remain much weaker and more uneven than forward ones. The paper is explicit that PosBench does **not** solve the Position Curse. It “closes part of the gap,” but absolute performance remains “far from saturated.” Forward retrieval can become strong; backward retrieval also improves, but remains harder; relative and compositional indexing remain difficult; and the reported results do not show that post-training fully instills a general positional algorithm.

The paper also reports residual error structure after fine-tuning. Confusion patterns improve under LoRA, yet residual collapse and off-by-one-style mistakes remain visible rather than perfectly diagonal retrieval behavior. This suggests that the learned behavior is partially corrective but not algorithmically robust across all operator variants.

## 6. Transfer, significance, and limitations

A central external generalization test is transfer from PosBench to **PyIndex**, a separate held-out benchmark of Python indexing semantics. PyIndex defines a Python list $X=(x_0,\ldots,x_{L-1})$ and asks the model to return the value of an indexing expression. Its five subcategories are **Forward** ($X[i]$), **Backward** ($X[-i]$), **Nested** (for example $X[X[i]]$), **Expression** (for example $X[a+b]$, $X[L-k]$, $X[a \bmod L]$, $X.\mathrm{index}(v)$), and **Chained** (for example $X[a:b][i]$, $\operatorname{reversed}(X)[i]$, $\operatorname{sorted}(X)[i]$). Each model is evaluated on **20 examples per category**, for **100 total**, and the paper reports the unweighted mean across categories.

The reported transfer gains are substantial. For **Qwen3.5-4B**, the mean score improves from **32.2%** to **70.8%** with LoRA, while full-parameter fine-tuning reaches only **36.2%**. For **Qwen3.5-2B**, the score improves from **19.2%** to **31.4%** with LoRA. For **Qwen3.6-35B-A3B**, it improves from **43.8%** to **52.6%**. For Qwen3.5-4B, the paper also reports subcategory gains: **Forward** $93\%\rightarrow 99\%$, **Backward** $17\%\rightarrow 71\%$, **Chained** $23\%\rightarrow 72\%$, **Expression** $22\%\rightarrow 68\%$, and **Nested** $6\%\rightarrow 44\%$.

These transfer results are presented as evidence that PosBench is not merely teaching one fixed prompt format. The strongest gains occur in backward, expression, and chained indexing, which aligns with the benchmark’s emphasis on hard positional transformations. At the same time, **nested indexing remains hardest** even after large gains. This suggests incomplete generalization when position-based retrieval must compose with hierarchical structure.

The broader scaling observations in the paper are also relevant to PosBench. Scale helps more for forward endpoint retrieval than for backward or relative retrieval. PosBench fine-tuning partly compensates for this weakness, but does not erase it. Larger models are therefore not automatically cured by size alone. The paper also contrasts positional adaptation with explicit reasoning: larger Qwen3.6 models benefit from enabling a reasoning channel, but reasoning is described as still insufficient and inefficient for something this basic. PosBench is presented as a stronger intervention than merely allowing more test-time thinking.

The practical implication is that current LLMs lack a robust generic mechanism for position-based access, especially for “from-the-end” and relative queries. This matters directly for code workflows involving the second-to-last branch, a line before or after a known statement, negative indexing and slices, or manipulation of arrays and code blocks by exact offset. The paper argues that, as coding agents operate over larger codebases, precise indexing becomes a fundamental capability rather than a niche skill.

The limitations stated by the authors are narrow but important. Evaluation focuses on the targeted positional benchmarks **PosBench** and **PyIndex**, so broader real-world validation remains future work. The paper also identifies the design of post-training procedures that yield truly **generalizable** position-based retrieval, rather than primarily in-distribution gains, as an important next step. PosBench is therefore best understood as a deliberately focused dataset and evaluation framework that both exposes and partially remedies a major blind spot in current LLMs, while simultaneously showing that the underlying capability gap remains unresolved.

Source: https://www.emergentmind.com/topics/posbench