---
title: 'SinkTrack: LLM Context Anchoring'
url: https://www.emergentmind.com/topics/sinktrack
type: topic
---

# SinkTrack: LLM Context Anchoring

Searching arXiv for the specified paper to ground the article in the current record.
SinkTrack is a context anchoring method for large language models that exploits an intrinsic property of decoder-only transformers: the attention sink on the first token, \( \mathrm{BOS} \). It is presented as a training-free, plug-and-play mechanism that repurposes the normally semantically sparse \( \mathrm{BOS} \) token into an information anchor by injecting contextual features derived from the initial instruction, textual context, and/or image features into its internal representation. The stated objective is to counteract attention drift, a recency-biased shift of attention toward newly generated tokens and away from the initial input context, which the paper identifies as a primary mechanistic driver of hallucination and context forgetting in both text-only LLMs and multimodal LLMs [2604.10027].

## 1. Problem setting and conceptual basis

The motivating failures are defined in two parts. Hallucination is the generation of content that is factually inconsistent with the provided input, such as producing “planes on the road” when the image contains buses. Context forgetting occurs in long-form tasks and multi-step dialogue when the model loses track of the initial instructions or salient input context, leading to answers that violate earlier constraints or instructions [2604.10027].

The paper attributes both failures primarily to attention drift during autoregressive decoding. As generation proceeds, attention progressively concentrates on newly generated tokens, and decays for earlier tokens that often carry key information, including instructions, retrieved text, or image features. This recency bias is described as widely observed in LLMs and as a source of under-attention to the initial context.

Against this tendency, SinkTrack leverages attention sink, an emergent phenomenon in which the very first token of the sequence, \( \mathrm{BOS} \), consistently receives high attention throughout generation despite being semantically sparse. The paper’s diagnostics indicate that many heads and layers allocate persistent mass to \( \mathrm{BOS} \), making it the most stable point of focus in the attention topology. SinkTrack reframes that stable focus as an active information carrier rather than a passive artifact. This suggests a change in emphasis from modifying global decoding behavior to modifying one persistent conduit inside the pre-trained attention structure.

## 2. Method: from passive sink to active anchor

The core design is to inject key contextual features into the internal representation of \( \mathrm{BOS} \), so that the model remains anchored to the initial context over the entire generation. The method is explicitly described as inference-time only: there is no fine-tuning, no added memory tokens, and no repeated per-token interventions during decoding. The intervention is one-off during prefill, after which the resulting KV states are cached and subsequent autoregressive steps proceed unchanged [2604.10027].

The paper distinguishes three stages in the development of the method. The initial exploration, termed hard injection, directly replaced the Value vector of \( \mathrm{BOS} \) in the KV cache with a pooled context vector \( f_{\mathrm{info}} \). For text tasks, \( f_{\mathrm{info}} \) was mean-pooled prompt embeddings; for multimodal tasks, mean-pooled features from the model’s vision encoder. This direct replacement was reported to cause collapse because it violated the model’s learned computational flow and attention hierarchies.

A second variant, soft injection, blended external information into the \( \mathrm{BOS} \) hidden state through a weighted sum,
\[
H^{(l)}_{\mathrm{BOS}} \leftarrow \alpha \cdot H^{(l)}_{\mathrm{BOS}} + (1-\alpha)\cdot f_{\mathrm{info}},
\]
with \( \alpha \) as a hand-tuned strength and injections applied at selected layers. This preserved flow and improved performance, but required manual tuning and introduced noise on long contexts because the fusion remained indiscriminate and mean-pooled.

The final SinkTrack formulation replaces pooling-based fusion with a dual-track attention mechanism. At designated injection layers, Track 1 uses the \( \mathrm{BOS} \) hidden state as the query and the external contextual features as keys and values:
\[
H^{(l)}_{\mathrm{BOS}} \leftarrow \mathrm{MHA}\!\left(Q=H^{(l)}_{\mathrm{BOS}},\,K=f_{\mathrm{info}},\,V=f_{\mathrm{info}}\right).
\]
Track 2 leaves all other tokens on standard causal self-attention over the original sequence. The updated \( \mathrm{BOS} \) output from Track 1 is then concatenated with the regular token outputs from Track 2 within the same attention block, passed through the output projection, and continued through the FFN. The paper presents this as preserving the pre-trained attention hierarchy while enriching \( \mathrm{BOS} \) with salient context [2604.10027].

A critical design change is the elimination of mean pooling. In the final method, \( f_{\mathrm{info}} \) is the full sequence of contextual features, whether text tokens or visual tokens, projected into K/V spaces. The paper argues that this exploits attention’s natural variable-length handling and avoids information bottlenecks from pooling.

## 3. Mathematical formulation and inference-time procedure

SinkTrack is defined within the standard decoder self-attention framework. For a single attention head in layer \( l \), with hidden size \( d \),
\[
\mathrm{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d}}\right)V,
\qquad
\alpha_{i,j} = \frac{\exp\!\left(q_i^\top k_j/\sqrt{d}\right)}{\sum_t \exp\!\left(q_i^\top k_t/\sqrt{d}\right)}.
\]
The decoder block is given in pre-norm form as
\[
M^{(l)} = \mathrm{LayerNorm}\!\left(H^{(l-1)} + \mathrm{MHA}(Q=H^{(l-1)},K=H^{(l-1)},V=H^{(l-1)})\right),
\]
\[
H^{(l)} = \mathrm{LayerNorm}\!\left(M^{(l)} + \mathrm{FFN}(M^{(l)})\right).
\]

The inference-time procedure is a one-off prefill intervention. First, contextual features \( h_{\mathrm{info}} \) are extracted from the source. For text-only settings, the model’s own token embeddings or hidden states for the prompt/context are used; for multimodal settings, the model’s native vision encoder produces image features such as patch tokens. In the final method, these remain as sequences rather than pooled vectors [2604.10027].

At designated injection layers \( l \), both the original hidden states \( h_{\mathrm{ori}} \) and the contextual features \( h_{\mathrm{info}} \) are projected with the layer’s learned projections:
\[
Q_{\mathrm{ori}},K_{\mathrm{ori}},V_{\mathrm{ori}} = W_Q h_{\mathrm{ori}},\, W_K h_{\mathrm{ori}},\, W_V h_{\mathrm{ori}},
\]
\[
K_{\mathrm{info}},V_{\mathrm{info}} = W_K h_{\mathrm{info}},\, W_V h_{\mathrm{info}}.
\]
The query is split into \( q_{\mathrm{BOS}} \) and \( Q_{\mathrm{rest}} \). Cross-attention is then applied only for \( \mathrm{BOS} \),
\[
o_{\mathrm{BOS}} = \mathrm{Attn}(q_{\mathrm{BOS}},K_{\mathrm{info}},V_{\mathrm{info}}),
\]
while the remaining tokens use standard causal self-attention,
\[
o_{\mathrm{rest}} = \mathrm{Attn}(Q_{\mathrm{rest}},K_{\mathrm{ori}},V_{\mathrm{ori}};\,\text{causal mask}).
\]
The outputs are concatenated, projected, and cached in the standard KV cache; generation then proceeds normally.

The default scheduling choices are also part of the method definition. Intermittent injection every 5 layers performs best, whereas dense injection every layer degrades performance by disrupting flow. The paper reports that a decaying schedule for soft injection, with stronger early layers, was superior and motivated SinkTrack’s emphasis on early anchoring. SinkTrack itself has no scalar strength parameter such as \( \lambda \) or \( \alpha \); retrieval from \( h_{\mathrm{info}} \) is adaptive through cross-attention [2604.10027].

## 4. Empirical results across text and multimodal benchmarks

The paper evaluates SinkTrack on both textual and multimodal benchmarks, reporting mean \( \pm \) variance over 3 seeds. The principal baselines are Direct prompting and CoT. CoT is reported to sometimes degrade performance in long-context tasks, with the interpretation that extended chains exacerbate drift and error accumulation [2604.10027].

| Setting | Baseline | SinkTrack |
|---|---|---|
| Qwen2.5-VL-7B-Instruct on M3CoT | CoT 44.11±0.56 | 66.94±0.21 |
| Qwen2.5-VL-7B-Instruct average Acc/Macro-F1 | CoT 60.37/65.72 | 70.42/75.37 |
| Llama3.1-8B-Instruct on QuAC (full, 7,354) | CoT 49.13±0.27 | 56.20±0.09 |
| Llama3.1-8B on SQuAD2.0 | Direct 78.69±0.02 | 79.83±0.07 |

On multimodal benchmarks with Qwen2.5-VL-7B-Instruct, the paper reports RealWorldQA at 65.49±0.37 for SinkTrack versus 57.86±1.29 for CoT; MMStar at 63.78±0.19 versus 55.84±0.74; M3CoT at 66.94±0.21 versus 44.11±0.56, identified as \(+22.8\%\) over CoT; and POPE at 85.47±0.02 with Macro-F1 91.67±0.01, compared with CoT 83.65±0.08. The average Acc/Macro-F1 is 70.42/75.37 for SinkTrack versus 60.37/65.72 for CoT. For Gemma3-12B-Instruct, the average Acc/Macro-F1 is 66.37/72.86 versus 65.78/72.22, and for Qwen2.5-VL-3B-Instruct it is 55.37/58.99 versus 39.05/46.82 [2604.10027].

On text benchmarks, Llama3.1-8B-Instruct on QuAC yields 59.40±0.10 accuracy on QuAC-1 versus Direct 54.10±0.02, 58.05±0.05 on QuAC-2 versus 54.25±0.08, and 56.20±0.09 on QuAC full versus CoT 49.13±0.27, corresponding to an accuracy gain of 7.07 over CoT. On SQuAD2.0, Llama3.1-8B reaches 79.83±0.07 versus Direct 78.69±0.02, while Qwen2.5-7B reaches 81.04±0.09 versus 80.97±0.13. The paper also highlights a \(+21.6\%\) improvement on SQuAD2.0 with Llama3.1-8B-Instruct relative to CoT [2604.10027].

The ablations are central to the empirical argument. Intermittent injection every 5 layers outperforms dense injection every layer by 9.0% accuracy on M3CoT. Removing mean pooling and using full sequences for \( K_{\mathrm{info}},V_{\mathrm{info}} \) eliminates diminishing gains on long-dialogue QuAC. The paper also reports consistent gains across model families and scales from 3B to 12B, and across both text and vision-language settings, with low variance across seeds.

## 5. Mechanistic interpretation and information delivery

SinkTrack’s analysis is framed as information delivery rather than explicit attention reweighting. The paper states that increasing the informativeness of \( \mathrm{BOS} \) boosts \( q_i^\top k_{\mathrm{BOS}} \) dot products across generation steps, keeping \( \alpha_{i,\mathrm{BOS}} \) salient without altering the attention weights explicitly. Since attention outputs aggregate value vectors,
\[
O_t = \sum_j \alpha_{t,j} V_j,
\]
a richer \( V_{\mathrm{BOS}} \) increases the capacity of the \( \mathrm{BOS} \) channel to inject context into every step [2604.10027].

The vertical part of the analysis focuses on values. Layer-wise diagnostics show large increases in the L1 norm of \( V_{\mathrm{BOS}} \) after injection, and the paper interprets this as information gain. The horizontal part focuses on downstream attention. Tracking \( \alpha_{t,\mathrm{BOS}} \) over generated tokens shows natural decay but sustained magnitude; the reported before/after curves overlap, which is used to argue that anchoring is non-destructive rather than a forced reallocation of the attention map.

The strongest quantitative diagnostic is the preservation of the pre-trained hierarchy. Spearman correlation of layerwise \( \mathrm{BOS} \) attention patterns before and after SinkTrack is reported as \( \rho=0.9985 \), with \( p<2.22\times 10^{-17} \). In a “Drift Test” on Llama3.1-8B with \(>8\)k input tokens and 1024 generated tokens, attention to \( \mathrm{BOS} \) remains dominant, with 0.582 at step 1024, approximately 14 times larger than the maximum attention to any other token. This supports the claim that SinkTrack enriches content while preserving the model’s original attention topology [2604.10027].

A common misconception is that the method operates by calibrating attention weights directly. The paper does not describe SinkTrack in those terms. Instead, it modifies only the \( \mathrm{BOS} \) representation and relies on persistent downstream attention to that position for delivery. Another misconception is that it is a retrieval or external-memory method. The paper contrasts SinkTrack with RAG and tool augmentation by noting that SinkTrack does not rely on external modules and targets internal attention dynamics.

## 6. Implementation, limitations, and prospective extensions

The paper gives implementation guidance for Hugging Face Transformers and vLLM or Triton-backed engines. In the Hugging Face setting, the recommended approach is to create a wrapper around the model’s attention module that implements the hybrid attention path and to register forward hooks at chosen layers to obtain \( h_{\mathrm{ori}} \) and cached \( h_{\mathrm{info}} \), compute the layer projections, split the query into \( q_{\mathrm{BOS}} \) and \( Q_{\mathrm{rest}} \), run the one-query cross-attention for \( \mathrm{BOS} \), concatenate outputs, and continue through the block. Batched inputs must handle one \( \mathrm{BOS} \) per sequence, and \( h_{\mathrm{info}} \) should remain on the same device and dtype as the model [2604.10027].

For vLLM or Triton-backed engines, the paper suggests a custom attention kernel or a light shim that runs a one-query cross-attention for the \( \mathrm{BOS} \) position at injection layers, or alternatively an auxiliary attention call before output projection with splicing of the \( \mathrm{BOS} \) row. Because outputs and keys/values produced at prefill are cached as usual, the decoding loop, batching, and speculative decoding remain compatible.

The reported overhead is small. On an RTX 4090, prefill latency for Llama3.1-8B increases from 35.90 ms to 36.66 ms, a \(+0.76\) ms change, and for Qwen2.5-VL-7B it increases from 107.13 ms to 110.26 ms, a \(+3.13\) ms change. Autoregressive token generation is unchanged because the intervention is cached [2604.10027].

The method is validated on Llama3.1-8B, Qwen2.5-7B, MiniCPM3-4B, Gemma3-4B/12B, and Qwen2.5-VL-3B/7B, with no architecture-specific weights or retraining. The paper nevertheless identifies several limitations and risks. Over-anchoring may bias tasks that genuinely require flexible attention away from the initial context, such as creative generation or late-stage context shifts. A single \( \mathrm{BOS} \) vector has finite capacity, so extremely long or dense contexts may expose a single-anchor bottleneck; the reported diminishing gains on very long dialogues are presented as evidence of this. Anchoring poor-quality or noisy inputs can entrench errors, and an overly strong early anchor may conflict with necessary recency in some tasks. The paper also notes that better coherence and confidence may amplify persuasive but biased content if the source context is biased, so calibration effects should be monitored.

Future work is described in terms of multi-anchor designs, dynamic gating and adaptive schedules, learned anchors, integration with RAG and memory modules, and attention-aware normalization. These directions follow directly from the paper’s diagnosis: if the main advantage comes from stable information delivery through persistent anchor positions, then distributing or adapting that anchoring mechanism is a plausible next step.

Source: https://www.emergentmind.com/topics/sinktrack