History Tokens in Sequential Systems

• History tokens are representations that carry historical context across computational models, ranging from visual cues in GUIs to memory tokens in language systems.
• They are managed through techniques like pruning, recency weighting, and sparse bottlenecking to reduce redundancy while preserving essential spatial and temporal information.
• Advanced methods leverage learnable prefix states and token provenance mechanisms to enhance predictive performance and maintain trusted, auditable histories.

History tokens are representations of prior context carried forward into later computation. Across recent work, the term denotes visual tokens from past GUI screenshots, token-level units extracted from dialogue histories, special learnable tokens inserted into causal Transformers to summarize sequence prefixes, fixed-size bottleneck or memory tokens in recurrent vision and robotics models, prior dialogue tokens in interactive chat systems, and provenance-tracked token instances in revision corpora (Li et al., 27 Mar 2026, Zhong et al., 2022, Karpukhin et al., 2 Aug 2025, Weinzaepfel et al., 19 Jun 2026, Wei et al., 2024, Flöck et al., 2017). Taken together, these usages suggest that history tokens are best understood not as a single architectural primitive, but as a family of mechanisms for retaining, compressing, querying, or auditing past information under computational, statistical, and trust constraints.

1. Domain-specific meanings

Recent work uses the same term for several technically distinct objects. In all cases, the common role is to make past state available to a present decision, but the representational substrate varies markedly.

Setting	What “history tokens” denote	Representative sources
GUI and computer-use agents	Visual tokens produced from past screenshots	(Li et al., 27 Mar 2026, Abaskohi et al., 11 May 2026)
Personalized dialogue	Fine-grained words or subwords extracted from past responses	(Zhong et al., 2022)
Event-sequence modeling	Learnable tokens inserted into a causal Transformer as compact prefix summaries	(Karpukhin et al., 2 Aug 2025)
Long-horizon robotics	Bottleneck tokens, memory slots, or moment tokens summarizing observation history	(Weinzaepfel et al., 19 Jun 2026, Koo et al., 1 Oct 2025)
Interactive chat and revision corpora	Prior dialogue tokens in a serialized prompt, or token instances tracked across revisions	(Wei et al., 2024, Flöck et al., 2017)

In GUI agents, the history is typically a truncated window of past screenshots $I_t = \{o_{t-\tau}, \dots, o_{t-1}\}$, and the corresponding history tokens are the visual tokens originating from those historical images (Li et al., 27 Mar 2026). In personalized dialogue, history tokens are the token-level units selected from the current user’s and similar users’ past responses as persona profiles (Zhong et al., 2022). In HT-Transformer, by contrast, a history token is a special learnable token inserted into a causal Transformer so that next-token prediction can optimize a single state-like embedding of the prefix (Karpukhin et al., 2 Aug 2025). In recurrent robotic memory, the teacher’s bottleneck tokens $\mathbf{B}_t \in \mathbb{R}^{B \times E}$, the student memory bank $M_t \in \mathbb{R}^{B \times E}$, and HAMLET’s per-timestep moment tokens all serve as fixed-size carriers of past observations (Weinzaepfel et al., 19 Jun 2026, Koo et al., 1 Oct 2025).

2. Selection, pruning, and redundancy reduction

A large fraction of the recent literature treats history tokens as a compression problem. In GUI visual agents, the central issue is that high-resolution screenshots generate many visual tokens per frame, so historical context quickly dominates compute. One study reports that on AndroidControl, total FLOPs jump from 14.93 T to 54.28 T when historical images are included (Li et al., 27 Mar 2026). That work identifies three non-obvious constraints on pruning. First, GUI screenshots have a foreground-background semantic composition in which background patches are not semantically disposable: across four datasets, background patches constitute about 51.9% of all patches, and background-only history remains close to full-history performance, including 59.00% versus 59.00% on AndroidControl. Second, random pruning is a strong baseline because it preserves spatial uniformity; under a 50% history budget, random pruning remains essentially at the upper bound on Mind2Web and matches or exceeds designed methods on AndroidControl. Third, a recency-weighted allocation

$$N_{\text{retain}}^{(k)} = N_{\text{total}} \cdot \lambda^k$$

with $\lambda = 0.5$ reduces prefill FLOPs from 0.93 T to 0.74 T, about 20.5%, while keeping performance retention at about 99–100% on AITW (Li et al., 27 Mar 2026).

ReVision addresses the same bottleneck at patch level by removing temporal redundancy across consecutive GUI screenshots while preserving positional structure. Across datasets, the average screenshot yields 2,315 patches, of which 1,083 are redundant on average, or 45.4% (Abaskohi et al., 11 May 2026). With 5 history screenshots on Qwen2.5-VL-7B, ReVision reduces token usage by approximately 46% on average while improving success rate by 3% over the no-drop baseline. It also changes the interpretation of history saturation: the no-drop baseline peaks around 7 images, whereas ReVision continues to improve up to around 11 images before plateauing, with both regimes saturating near a combined context length of roughly 23k tokens (Abaskohi et al., 11 May 2026).

In dialogue systems, the pressure comes from long, noisy textual histories rather than dense images. MSP formalizes history refinement through a user refiner, topic refiner, and token refiner, and then applies a TopK bottleneck over query-aware attention scores to retain only the most valuable history tokens (Zhong et al., 2022). The underlying histories are long—72.4 responses on Reddit and 30.8 on Weibo on average—and the selected token budgets are sparse, with $k_p = 30$ on Reddit and $k_p = 200$ on Weibo. This sparse refinement is reported to reduce training time by 41.6% versus using full history while improving personalization metrics, including Weibo P-Cover from 0.144 for DHAP to 0.297 for MSP (Zhong et al., 2022).

3. Learned prefix states and memory tokens

A second line of work does not prune history tokens after they are formed; it makes them the primary latent state of the model. HT-Transformer begins from the claim that causal Transformers trained with next-token prediction do not naturally provide a good single vector for the whole prefix, unlike RNNs (Karpukhin et al., 2 Aug 2025). It therefore inserts learnable history tokens into the sequence under a special attention mask, with the number of such tokens given by $\max(1, fL)$, where $f$ is a frequency hyperparameter. During inference, a single history token is appended at the end of the sequence and its representation is used as the sequence embedding. In pretrain-only evaluation, this improves over a plain NTP Transformer on all five datasets, including Churn 83.34 versus 80.92 and AgePred 60.10 versus 56.16. The same study also finds a clear trade-off: history tokens trained by next-token prediction are best for local or recent predictive context, whereas contrastive CoLES remains stronger on some global tasks in the pure embedding setting (Karpukhin et al., 2 Aug 2025).

“Compressing Observation History into Agent Memory: Distilling Transformers into Recurrent Transformers” reframes history tokens as a teacher bottleneck and student memory alignment problem (Weinzaepfel et al., 19 Jun 2026). The teacher, a Latent Bottleneck History Transformer, produces $\mathbf{B}_t \in \mathbb{R}^{20 \times 3072}$, while the recurrent student maintains $\mathbf{B}_t \in \mathbb{R}^{B \times E}$0 and is directly supervised with a segment-wise $\mathbf{B}_t \in \mathbb{R}^{B \times E}$1 loss at selected sequence endpoints. This substantially narrows the gap between a recurrent model and a full-history transformer: at sequence length 200 and the $\mathbf{B}_t \in \mathbb{R}^{B \times E}$2 metric, Chimera reaches 70% versus 63% for Kinaema and 72% for the teacher; at sequence length 800, Chimera reaches 45% versus 37% for Kinaema and 44% for the teacher (Weinzaepfel et al., 19 Jun 2026).

HAMLET adapts vision-language-action models into history-aware policies by introducing per-timestep moment tokens, initializing them with time-contrastive learning, and feeding them through a lightweight Transformer memory module before action prediction (Koo et al., 1 Oct 2025). The moment tokens are appended to the VLM input and updated into $\mathbf{B}_t \in \mathbb{R}^{B \times E}$3, which are then stacked across the last $\mathbf{B}_t \in \mathbb{R}^{B \times E}$4 decision steps and transformed into history-augmented features $\mathbf{B}_t \in \mathbb{R}^{B \times E}$5. On real-world history-dependent tasks, HAMLET raises average success from 29.2% for GR00T N1.5 and 45.8% for a 4-frame baseline to 76.4%. Its inference overhead is comparatively small: at history length 4, latency is 82.4 ms versus 80.5 ms for GR00T N1.5 and 108.5 ms for the multi-frame baseline, while peak GPU memory is 566 MB versus 289 MB and 1051 MB respectively (Koo et al., 1 Oct 2025).

4. Externalized and budgeted histories

Some systems move history tokens outside the original modality altogether. AgentOCR replaces a long textual history

$\mathbf{B}_t \in \mathbb{R}^{B \times E}$6

with a rendered image $\mathbf{B}_t \in \mathbb{R}^{B \times E}$7, and then lets a vision-LLM condition on that image instead of on raw text tokens (Feng et al., 8 Jan 2026). On ALFWorld with 7B models, average tokens per step fall from about 0.95k to 0.43k, a 54.7% reduction, while success changes from 81.8% to 81.2%. On search-based QA, average tokens per step fall from about 0.73k to 0.36k, a 50.7% reduction, and the maximum drops from 13.84k to 2.65k, an 80.9% reduction, while average EM changes from 41.9% to 40.1% (Feng et al., 8 Jan 2026). The same framework adds segment optical caching, which yields a 20.8x rendering speedup versus no cache, and agentic self-compression, where a learned compression factor can reduce vision tokens from 458.1 to 381.7 with success 78.2% versus 78.4% when intermittent reward is used (Feng et al., 8 Jan 2026).

Budgeted Dynamic Trace Structures treat history tokens as a data-structural resource under an explicit byte or token budget rather than as internal model activations (Alpay et al., 20 May 2026). BDTS combines rooted trace graphs, append-only histories, bounded cost caches, and summary-plus-suffix compaction. Its compaction operator replaces a long prefix by a summary item and retains the longest recent suffix whose measured cost fits within a budget. On synthetic traces with 10,000–40,000 vertices, the prototype builds graphs in 0.58–2.72 ms, enumerates all descendants in 0.24–1.42 ms, and compacts histories of 350k–2.71M approximate tokens to 1,048–4,120 approximate tokens. Under real tokenizers and forward measurements, it reduces 3,359–3,360 trace tokens to 432–433 tokens (Alpay et al., 20 May 2026). This external view differs from model-internal history tokens, but it addresses the same core question: how to preserve a bounded, recent, queryable view of long sequential traces.

5. Provenance, tampering, and token history as an auditable object

In interactive chat systems, history tokens are also a security boundary. “Hidden in Plain Sight” argues that chat models operate on a single flat token sequence containing system messages, chat history, and the current user message, and therefore cannot separate user input from server-provided context (Wei et al., 2024). This makes fabricated history tokens operationally effective: a user can embed fake multi-turn dialogue inside a current message using role-like templates, leading the model to interpret those tokens as genuine history. In the paper’s evaluation, the combined acceptance-elicitation and anonymization attack reaches $\mathbf{B}_t \in \mathbb{R}^{B \times E}$8 97.5% on GPT-3.5, 61.22% on GPT-4, and 89.41% on average across ten models (Wei et al., 2024). The issue is not merely prompt phrasing; it is the inability of the model to attach provenance or trust level to different segments of the history sequence.

TokTrack studies a different aspect of provenance: not whether history can be forged, but how token histories can be reconstructed exactly from revision data (Flöck et al., 2017). The dataset covers 5,275,388 English Wikipedia articles, 451,350,901 revisions, and 13,545,349,787 token instances up to October 2016. Each token instance records its origin_rev_id, together with ordered out and in lists capturing every deletion and reinsertion event. This representation supports token-level provenance, survival analysis, conflict metrics, and fine-grained revert analysis. At revision scale, 14.84% of all revisions fully undo another revision and 50.65% partially revert some other revision (Flöck et al., 2017). Here the history token is not a learned embedding but a persistent textual unit with an explicit event log.

6. Recurring principles and unresolved issues

Several recurring principles emerge across these otherwise disjoint formulations. History tokens are rarely useful in raw, undifferentiated form: GUI agents benefit from recency-weighted budgets and from retaining background and layout cues rather than pruning solely by saliency; computer-use agents improve when temporal redundancy is removed rather than when history is shortened; personalized dialogue improves when history tokens are selected by user, topic, and query relevance; and history-aware robotics improves when past observations are first compressed into temporally distinctive moment tokens rather than concatenated as raw frames (Li et al., 27 Mar 2026, Abaskohi et al., 11 May 2026, Zhong et al., 2022, Koo et al., 1 Oct 2025). A related misconception is directly challenged by the GUI literature: background is not “low semantic” in historical screenshots, and history saturation is not evidence that past observations are intrinsically unhelpful (Li et al., 27 Mar 2026, Abaskohi et al., 11 May 2026).

The open problems are likewise consistent. Positional reasoning under pruning remains unresolved in GUI agents, where preserving positional indices is insufficient and pruning-aware positional compensation is explicitly left open (Li et al., 27 Mar 2026). HT-Transformer reports sensitivity to history-token placement and notes that its current dense-mask implementation does not exploit the sparsity of the intended attention pattern (Karpukhin et al., 2 Aug 2025). Distilled recurrent memory depends on a strong full-history teacher and is computationally expensive to train (Weinzaepfel et al., 19 Jun 2026). AgentOCR does not address histories that are already multimodal rather than primarily textual (Feng et al., 8 Jan 2026). Chat history tampering indicates that token-efficient history management and provenance-aware history management are different problems, and that interactive systems need a distinction between trusted and untrusted history channels (Wei et al., 2024). BDTS, finally, makes the budget explicit but leaves summary generation application-specific (Alpay et al., 20 May 2026). Taken together, these works suggest that future history-token systems will be judged not only by compression ratio or downstream accuracy, but also by how well they preserve spatial and temporal structure, expose provenance, and enforce trust boundaries.