PosBench: Addressing the Position Curse

• PosBench is a position-based supervised fine-tuning dataset that tests how LLMs retrieve items from ordered sequences using forward, backward, and relative indexing.
• It combines synthetic, code, and adapted real-data examples to provide comprehensive coverage of positional retrieval tasks for practical applications like code editing.
• Empirical results show that LoRA fine-tuning significantly improves positional retrieval accuracy, particularly in backward indexing scenarios, though some challenges remain.

Searching arXiv for the PosBench source paper and closely related context. arxiv_search(query="^^^^^^^^1(Zhang et al., 8 May 2026) OR \1^^^^^^^^"The Position Curse: LLMs Struggle to Locate the Last Few Items in a List\"", max_results=5) PosBench is a position-focused supervised fine-tuning dataset introduced to test whether the central failure mode identified in "The Position Curse: LLMs Struggle to Locate the Last Few Items in a List" can be repaired by post-training (&&&^^^^1(Zhang et al., 8 May 2026) OR \1^^^^&&&). It is designed around position-based retrieval over ordered sequences, rather than content retrieval or counting. In the underlying study, modern LLMs are reported to be strong at content-based retrieval in long contexts and near-saturated on simple counting, yet to perform poorly on short-sequence positional access, especially when retrieval is specified backward from the end of a list or relative to another item. PosBench serves both as a training corpus and as an in-distribution evaluation suite for this capability gap, with particular motivation from code understanding and editing, where exact identification of lines, branches, list elements, and slices by position is often required.

1. Position-based retrieval and the “Position Curse”

The motivating phenomenon for PosBench is the Position Curse: LLMs can often locate a single relevant fact in very long contexts, yet fail to retrieve the last few items in a short list. The paper treats this as a distinct capability deficit rather than a simple counting failure. Counting on the same short lists is described as near-saturated, whereas positional retrieval remains poor, especially in backward retrieval settings.

The core asymmetry is between forward retrieval and backward retrieval. Forward retrieval means indexing “from the beginning” or “after” a reference item. Backward retrieval means indexing “from the end” or “before” a reference item. The paper reports that, across both open-source and frontier closed-source models, backward retrieval substantially lags forward retrieval. This asymmetry is central to PosBench’s design: the dataset intentionally concentrates supervision on the settings that most strongly expose the failure.

The coding motivation is explicit. Code agents routinely need to identify exact positions in sequences of lines or program elements, not merely retrieve semantically relevant content. A plausible implication is that positional access is a low-level capability whose failure can propagate into broader code-understanding and code-editing errors, even when higher-level semantic reasoning is strong.

2. Formal task family underlying PosBench

The paper formalizes position-based retrieval over an ordered sequence

PRESERVED_PLACEHOLDER_^^^^1(Zhang et al., 8 May 2026) OR \1^^^^

A query specifies an offset $n \ge 1$, an anchor, and a direction. Anchors are either an endpoint of the sequence or another item in the sequence. Endpoint-anchored queries are denoted by $\mathsf{End}$ and relative-item-anchored queries by $\mathsf{Rel}$; forward and backward directions are denoted by superscripts $+$ and $-$.

The four index operators are

$$\mathsf{End}^{+}: S[n], \qquad \mathsf{End}^{-}: S[-n], \qquad \mathsf{Rel}^{+}: S[r+n], \qquad \mathsf{Rel}^{-}: S[r-n],$$

with

$S[-n] \equiv s_{L-n+1},$

and offsets sampled so that the resulting index remains in $[1,L]$.

These operators support two exact retrieval task definitions. The first is position$\rightarrow$item, in which a position is specified and the model must return the corresponding item. The second is item$n \ge 1$^^^^1(Zhang et al., 8 May 2026) OR \1^^^^position, in which the target item is given and the model must return the offset $n \ge 1$1 relative to the specified anchor and direction. The same operator family is used for both training and evaluation.

The framework covers endpoint and relative anchors, forward and backward directions, and multiple item types including letters and words. Examples in the paper include queries such as “the 2nd from the end” and “one position before $n \ge 1$2.” PosBench is therefore not a single prompt format, but a structured task family defined by operator choice, anchor type, direction, sequence type, and offset.

3. Dataset composition and construction

In the appendix, PosBench is described as a mixed supervised fine-tuning corpus combining three sources of ordered structure: synthetic retrieval examples, code-retrieval examples, and adapted real-data examples. The dataset contains 2^^^^1(Zhang et al., 8 May 2026) OR \1^^^^K synthetic retrieval examples, 4K code-retrieval examples, and 46K adapted real-data examples.

Component	Size	Source and role
Synthetic component $n \ge 1$3	2^^^^1(Zhang et al., 8 May 2026) OR \1^^^^K	Controlled coverage of the operator family
Code component $n \ge 1$4	4K	Retrieval over structured code-line sequences
Adapted real-data component $n \ge 1$5	46K	Linguistically diverse ordered structures from existing instruction and code data

The synthetic component covers both position$n \ge 1$6item and item$n \ge 1$7position tasks, for both endpoint anchors $n \ge 1$8 and relative anchors $n \ge 1$9. Sequence lengths are sampled from $\mathsf{End}$^^^^1(Zhang et al., 8 May 2026) OR \1^^^^. Items come from multiple categorical pools, including letters, digits, animals, fruits, cities, elements, languages, and instruments. Prompt phrasing and list formatting are randomized, and the corpus includes a small number of multi-turn list conversations so that positional queries also appear in more natural dialog contexts.

The code component uses BigCode self-oss-instruct-sc2-exec-filter-5^^^^1(Zhang et al., 8 May 2026) OR \1^^^^k. Each usable code snippet is split into non-empty lines; snippets with fewer than five lines are discarded; longer snippets are windowed into contiguous spans of 5–3^^^^1(Zhang et al., 8 May 2026) OR \1^^^^ lines; and positional queries are instantiated over those code-line sequences. The basic item type in this component is therefore the code line, directly matching the paper’s code-understanding motivation.

The adapted real-data component extracts naturally ordered structures from Open-Orca / SlimOrca, OpenHermes-2.5, and tiny-codes. The converted structures include numbered lists, bullet lists, markdown tables, and code blocks, provided they contain at least five usable items/rows/lines. Extracted items are deduplicated, kept short, and reformatted before query generation. For tiny-codes, the original dialogue context is preserved and the positional query is appended as a follow-up turn.

The sampling policy explicitly overrepresents difficult settings. The construction sets

$\mathsf{End}$1

so the complementary cases—especially backward and relative addressing—appear more often. The paper states that those settings “most strongly expose the Position Curse.” No separate curriculum schedule is described.

4. Prompting, supervision, and fine-tuning protocol

PosBench examples instantiate the same operator family used in evaluation. The task space includes position$\mathsf{End}$2item and item$\mathsf{End}$3position, forward and backward indexing, endpoint and relative anchors, and sequence types ranging from synthetic categorical lists to natural ordered text structures and code-line sequences. The paper does not provide a single canonical template for all PosBench samples; instead, prompt wording and formatting are randomized.

At evaluation time, prompts are three-shot. Each test query is preceded by three in-context demonstrations of the same operator using independently sampled sequences. The stated purpose is to remove ambiguity about conventions such as 1-based indexing and answer format, so that failures reflect positional retrieval rather than prompt misunderstanding.

The fine-tuning objective uses answer-span supervision rather than supervising the entire assistant response. Let $\mathsf{End}$4 indicate whether token $\mathsf{End}$5 belongs to the target answer span. The retrieval loss is

$\mathsf{End}$6

All non-answer tokens are masked out, so only answer tokens contribute to the loss. The paper explains that this concentrates learning on deterministic position lookup rather than on prompt scaffolding or stylistic response framing.

Evaluation uses exact-match accuracy after task-specific parsing. Accuracy over a trial subset $\mathsf{End}$7 is defined as

$\mathsf{End}$8

Per-offset analyses use subsets

$\mathsf{End}$9

Responses are normalized into the appropriate answer space: for item retrieval, outputs are matched against candidate items in the sequence; for position or count tasks, the first integer answer is taken; and for code-style short answers, formatting artifacts are stripped.

For matched fine-tuning comparisons, the default backbone is Qwen3.5-4B. The LoRA configuration uses rank $\mathsf{Rel}$^^^^1(Zhang et al., 8 May 2026) OR \1^^^^, $\mathsf{Rel}$1, dropout $\mathsf{Rel}$2, and is applied to attention projections $\mathsf{Rel}$3 and MLP projections (gate, up, down). The training hyperparameters are 1 epoch, learning rate $\mathsf{Rel}$4, weight decay $\mathsf{Rel}$5, per-device batch size 16, gradient accumulation 2, bf16 precision, maximum sequence length 512, cosine learning-rate decay, 3% warmup, and seed 42.

The full-parameter supervised fine-tuning baseline uses ^^^^1(Zhang et al., 8 May 2026) OR \1^^^^.2 epochs, learning rate $\mathsf{Rel}$6, weight decay $\mathsf{Rel}$7, per-device batch size 4, gradient accumulation 8, gradient clipping 1, and NEFTune noise $\mathsf{Rel}$8. The paper’s reported comparison is that LoRA is more reliably beneficial than full-parameter supervised fine-tuning on PosBench.

5. Evaluation design and empirical findings

PosBench evaluation uses held-out sequences sampled independently from the training corpus (&&&^^^^1(Zhang et al., 8 May 2026) OR \1^^^^&&&). Each evaluation condition fixes the query type, anchor type, indexing direction, item type, prompt variant, and sequence length $\mathsf{Rel}$9. For each condition, the paper samples 5^^^^1(Zhang et al., 8 May 2026) OR \1^^^^ independent sequences, and for each sequence instantiates all valid offsets. The paper does not elaborate a distinct validation split in detail; the described split is between the training corpus and independently sampled held-out evaluation conditions.

Within the paper, PosBench is used in two roles. First, it is a training corpus for targeted adaptation. Second, it is an in-distribution evaluation suite for the same underlying operator family. The headline in-distribution evaluation shown in Figure 1 uses letter sequences, endpoint anchoring, $+$^^^^1(Zhang et al., 8 May 2026) OR \1^^^^, and averages over four tasks: forward position$+$1item, backward position$+$2item, forward item$+$3position, and backward item$+$4position.

The main empirical conclusion is that LoRA fine-tuning on PosBench reliably improves position-based retrieval across model families. Figure 1 is reported to show consistent improvement over the base model for Qwen3.5-4B, Llama 3.2-3B, Gemma4 E2B, and Ministral 3-3B. By contrast, full-parameter fine-tuning does not produce similarly consistent gains.

The appendix’s per-query-position analyses indicate that LoRA improves accuracy broadly across queried offsets, but backward conditions remain much weaker and more uneven than forward ones. The paper is explicit that PosBench does not solve the Position Curse. It “closes part of the gap,” but absolute performance remains “far from saturated.” Forward retrieval can become strong; backward retrieval also improves, but remains harder; relative and compositional indexing remain difficult; and the reported results do not show that post-training fully instills a general positional algorithm.

The paper also reports residual error structure after fine-tuning. Confusion patterns improve under LoRA, yet residual collapse and off-by-one-style mistakes remain visible rather than perfectly diagonal retrieval behavior. This suggests that the learned behavior is partially corrective but not algorithmically robust across all operator variants.

6. Transfer, significance, and limitations

A central external generalization test is transfer from PosBench to PyIndex, a separate held-out benchmark of Python indexing semantics. PyIndex defines a Python list $+$5 and asks the model to return the value of an indexing expression. Its five subcategories are Forward ($+$6), Backward ($+$7), Nested (for example $+$8), Expression (for example $+$9, $-$^^^^1(Zhang et al., 8 May 2026) OR \1^^^^, $-$1, $-$2), and Chained (for example $-$3, $-$4, $-$5). Each model is evaluated on 2^^^^1(Zhang et al., 8 May 2026) OR \1^^^^ examples per category, for 1^^^^1(Zhang et al., 8 May 2026) OR \1(Zhang et al., 8 May 2026) OR \1^^^^ total, and the paper reports the unweighted mean across categories.

The reported transfer gains are substantial. For Qwen3.5-4B, the mean score improves from 32.2% to 7^^^^1(Zhang et al., 8 May 2026) OR \1^^^^.8% with LoRA, while full-parameter fine-tuning reaches only 36.2%. For Qwen3.5-2B, the score improves from 19.2% to 31.4% with LoRA. For Qwen3.6-35B-A3B, it improves from 43.8% to 52.6%. For Qwen3.5-4B, the paper also reports subcategory gains: Forward $-$6, Backward $-$7, Chained $-$8, Expression $-$9, and Nested $$\mathsf{End}^{+}: S[n], \qquad \mathsf{End}^{-}: S[-n], \qquad \mathsf{Rel}^{+}: S[r+n], \qquad \mathsf{Rel}^{-}: S[r-n],$$^^^^1(Zhang et al., 8 May 2026) OR \1^^^^.

These transfer results are presented as evidence that PosBench is not merely teaching one fixed prompt format. The strongest gains occur in backward, expression, and chained indexing, which aligns with the benchmark’s emphasis on hard positional transformations. At the same time, nested indexing remains hardest even after large gains. This suggests incomplete generalization when position-based retrieval must compose with hierarchical structure.

The broader scaling observations in the paper are also relevant to PosBench. Scale helps more for forward endpoint retrieval than for backward or relative retrieval. PosBench fine-tuning partly compensates for this weakness, but does not erase it. Larger models are therefore not automatically cured by size alone. The paper also contrasts positional adaptation with explicit reasoning: larger Qwen3.6 models benefit from enabling a reasoning channel, but reasoning is described as still insufficient and inefficient for something this basic. PosBench is presented as a stronger intervention than merely allowing more test-time thinking.

The practical implication is that current LLMs lack a robust generic mechanism for position-based access, especially for “from-the-end” and relative queries. This matters directly for code workflows involving the second-to-last branch, a line before or after a known statement, negative indexing and slices, or manipulation of arrays and code blocks by exact offset. The paper argues that, as coding agents operate over larger codebases, precise indexing becomes a fundamental capability rather than a niche skill.

The limitations stated by the authors are narrow but important. Evaluation focuses on the targeted positional benchmarks PosBench and PyIndex, so broader real-world validation remains future work. The paper also identifies the design of post-training procedures that yield truly generalizable position-based retrieval, rather than primarily in-distribution gains, as an important next step. PosBench is therefore best understood as a deliberately focused dataset and evaluation framework that both exposes and partially remedies a major blind spot in current LLMs, while simultaneously showing that the underlying capability gap remains unresolved.