Compressing Observation History into Agent Memory: Distilling Transformers into Recurrent Transformers

Abstract: Transformers are AI's workhorse with strong performance in modeling sequential data, but their computational cost becomes prohibitive when processing long sequences. We target long-horizon streaming vision and robotics applications like map-free pose estimation, where it is particularly impractical to store and maintain a history of observations. Recurrent Transformers address this limitation by maintaining fixed-size memory but their performance lags behind that of transformers operating over the full observation history. We argue that this gap does not stem from architectural limitations, but from differences in how these models learn to compress past information. Without access to an observation history, recurrent models must explicitly decide what to retain in memory at each step, a significantly harder learning problem. In this work, we propose a distillation approach that transfers the compression strategy of a classical full-history transformer to a recurrent variant. We enable this by designing a teacher model that explicitly compresses its observation history into a fixed-size bottleneck representation. By directly supervising the student's memory with this bottleneck representation, we align the two compression mechanisms. We show that this approach allows to train a recurrent latent robotic memory with linear-time complexity while substantially narrowing the performance gap to full-history transformers.

• The paper demonstrates a distillation method that transfers a history transformer's compression into a recurrent transformer's memory, narrowing the performance gap.
• It introduces the Chimera model which uses segment-wise L1 supervision to align latent bottleneck representations, ensuring constant computational costs and stability in long sequences.
• Empirical results on map-free pose estimation show the distilled recurrent model nearly matches or exceeds teacher performance over both moderate and ultra-long test sequences.

Distilling Long-Horizon Transformer Compression into Recurrent Agent Memory

Introduction: Motivation and Problem Setting

This work addresses the challenge of compressing the history of sequential observations into fixed-size agent memory for efficient, scalable long-horizon inference, with a focus on robotics and streaming vision tasks such as map-free pose estimation. Standard transformers (History Transformers, HTs) can attend over the entire sequence history but suffer from prohibitive computational and storage costs, scaling quadratically with sequence length. Recurrent Transformers (RTs), in contrast, operate with constant per-step complexity by maintaining a fixed-size memory bank updated incrementally, but empirically exhibit a significant performance gap relative to their HT counterparts.

The paper posits that this gap is not inherent to the architectural expressivity of RTs, but rather is a consequence of their unsupervised compression mechanism: at each time-step, the RT must decide what to retain, with no recourse to historical re-attention or correction. In contrast, HTs compress by attention with direct gradient supervision over all historical representations. The authors advocate for a distillation-based solution, explicitly transferring the sequence compression inductive bias of HTs (with a learnable bottleneck) into the memory state of RTs, effectively realigning the two memory formation processes.

Architecture and Distillation Methodology

The core of the proposed method, Chimera, is a distillation pipeline aligning a Latent Bottleneck History Transformer (LBHT) teacher with a Kinaema-style RT student. The teacher encodes the sequence using full attention and compresses the observation history via a learned bottleneck of fixed-size embedding tokens, mimicking Perceiver Resampler semantics but tailored to the streaming, goal-driven pose estimation domain.

The RT student receives only the current observation and its own latent memory at each time step. Its update mechanism involves concatenation, linear correction, intra-memory self-attention, and a gated update using a shared GRU. The most important step in the pipeline is the dimensional alignment between the teacher's bottleneck and the student's memory bank, allowing for direct $L_1$ supervision at selected segment boundaries during sequence unrolling.

The distillation loss is applied segment-wise—rather than at every frame—to optimize computational efficiency and memory usage. This approach simulates curriculum learning, with memory banks first matched on sub-segments of shorter length and eventually on full-length sequences. This also mitigates vanishing gradients in long-horizon training. The teacher itself is trained for goal-agnostic scene encoding and pose prediction using combined pose and masked image modeling losses.

Figure 1: The Chimera distillation procedure: recurrent transformer student undergoes segment-wise memory supervision from a bottleneck-equipped history transformer teacher.

Evaluation on Long-Horizon Map-Free Pose Estimation

The primary empirical domain is Mem-RPE: streaming, agent-centric relative pose estimation from sequential RGB and odometry data in large-scale 3D environments (HM3D and Gibson). Importantly, the evaluation covers strict train/test shifts, including sequence lengths much longer than in training and different step scales.

Chimera, the distilled recurrent transformer, demonstrates a marked gain over vanilla RTs and approaches or even exceeds the performance of the teacher model for long sequences. For example, for evaluation episodes of length 200, Chimera achieves 70% accuracy at a $2\mathrm{m}, 90^\circ$ threshold (teacher: 72%, Kinaema: 63%). For ultra-long test sequences (800 steps), Chimera maintains better stability and generalization than the history transformer, which suffers a significant performance drop beyond its training context. Notably, the RT architecture of Chimera is unchanged from Kinaema, establishing that the improved performance stems solely from the distilled compression policy.

Figure 2: Mem-RPE accuracy as a function of query recency and encoded sequence length, demonstrating Chimera’s consistent advantage over non-distilled recurrent baselines and comparable performance to the teacher.

The Chimera model achieves this while maintaining $\mathcal{O}(1)$ inference cost and constant memory footprint, as demonstrated by direct measurements. In contrast, the LBHT teacher—and by extension standard HTs—exhibits quadratic (for runtime) and linear (for memory) complexity.

Figure 3: Inference timing and memory usage for Chimera (RT) and LBHT Teacher (HT), highlighting constant-complexity recurrent inference.

Memory Analysis and Internal Representation Structure

Beyond end-task performance, the paper investigates the properties of the learned agent memory. The distilled model's memory updates exhibit greater long-term stability, with normalized update norms that remain consistent across sequence length, reminiscent of the teacher’s bottleneck representations. In contrast, baseline RTs display a regime switch: rapid updates within the (short) sequence lengths seen at training, followed by stagnation.

There is also a pronounced specialization among memory tokens. In Chimera, a small subset of tokens is highly dynamic, concentrating most of the updates, while the majority are more static, suggestive of emergent persistent and transient memory separation. This structural property is less marked in non-distilled RTs.

Figure 4: Memory stability as a function of sequence length for teacher, Chimera, and Kinaema, revealing Chimera’s alignment with history-based compression dynamics.

Visualization of Attention over Memory Tokens

The paper also provides a qualitative analysis of how queries attend over the learned memory tokens as memory and query content evolve along a real navigation trajectory. The resulting attention distributions are coherent over time and appear semantically grounded, consistent with the hypothesis that Chimera’s memory encodes robust episodic state.

Figure 5: Attention distribution over memory tokens in Chimera along an agent’s trajectory, for both fixed query and evolving memory, and vice versa.

Implications and Outlook

The key result is that the challenge in recurrent long-horizon agent models lies not in insufficient memory bank expressivity, but in the inherent difficulty of learning optimal lossy compression under purely delayed, task-based supervision. By substituting supervised bottleneck matching—leveraging a strong, offline history-attentive teacher—this work closes much of the performance gap while preserving the favorable computational complexity of RTs.

Practically, this enables deployment of high-capacity, high-fidelity sequential world models on resource-constrained robotic platforms, with direct applicability to localization, SLAM variants, and policy learning in partially observable MDPs. Theoretically, the results support the hypothesis that learned memory organization and lifelong context compression in agents can be modularized and distilled, suggesting avenues for task-agnostic, multi-teacher bootstrapping.

The explicit distillation of sequence compression is orthogonal to other advances in long-context models (e.g., SSMs, chunk-based transformers) and can be directly combined with architecture and hardware-aware optimizations.

Conclusion

This paper demonstrates that explicit distillation of latent bottleneck representations from history-transformer teachers into recurrent transformer memory substantially improves long-horizon task performance and internal agent memory fidelity, without modifying recurrent architecture or computational cost. The empirical results support the view that the main limitation in recurrent, fixed-memory models for sequential decision-making is not lack of expressivity, but weakness in autonomous history compression learning. This approach opens viable new routes for constructing fast, efficient, and robust world models and agent policies with tractable resource requirements in realistic, lifelong domains.

(2606.21562)