Absorber LLM: Harnessing Causal Synchronization for Test-Time Training

Abstract: Transformers suffer from a high computational cost that grows with sequence length for self-attention, making inference in long streams prohibited by memory consumption. Constant-memory alternatives such as RNNs and SSMs compress history into states with fixed size and thus lose long-tail dependencies, while methods that memorize contexts into parameters, such as Test-Time Training (TTT), are prone to overfitting token-level projection and fail to preserve the causal effect of context in pretrained LLMs. We propose Absorber LLM, which formulates long-context retention as a self-supervised causal synchronization: after absorbing historical contexts into parameters, a contextless model should match the original model with full context on future generations. We optimize this objective by synchronizing internal behaviors of the updated model with the original one, ensuring context absorption and generalization. Experiments on long-context and streaming benchmarks show that Absorber LLM reduces inference memory and improves accuracy over prior parameter-as-memory baselines.

• The paper introduces causal synchronization to compress long historical contexts into model parameters, achieving O(1) per-token inference.
• The methodology employs hidden state alignment via LoRA adapters to fine-tune and internalize context without direct token-level projection.
• Empirical results show enhanced in-context reasoning and scalability, outperforming standard transformers on long-context tasks.

Absorber LLM: Functional Causal Synchronization for Long-Context Parameterization

Introduction

This work introduces Absorber LLM, a novel framework for compressing long historical contexts into model parameters via causal synchronization, circumventing the quadratic complexity and context window limitations imposed by conventional transformers. Unlike prior Test-Time Training (TTT) paradigms that rely on token-level context projection, Absorber LLM enforces functional alignment between a contextless model and its full-context counterpart, preserving the causal mechanisms fundamental to long-context reasoning and in-context generalization.

Figure 1: Absorber LLM absorbs historical contexts through synchronization, ensuring the causal influence of absorbed context on all future generations via functional alignment.

Motivation: Beyond Context Windows and State Compression

The scaling limitations of transformer-based LLMs originate from their self-attention mechanism—specifically, the requirement to attend to all prior states, yielding $\mathcal{O}(N^2)$ computational and $\mathcal{O}(N)$ KV cache complexity. Linear attention variants, SSMs, and RNN architectures reduce complexity to $\mathcal{O}(N)$ or even $\mathcal{O}(1)$, but fail to maintain robust long-tail dependencies due to fixed-size hidden state bottlenecks and intrinsic expressiveness loss [gu2024mamba, merrill2024illusion].

Alternatively, TTT approaches encode context into the high-dimensional parameter spaces, leveraging greater representational capacity. However, classic TTT objectives emphasize target projection or token-level reconstruction, often resulting in overfitting, loss of causal semantics, and failure to capture high-level, task-relevant dependencies inherent in autoregressive inference [sun2024learning, cao2025analyzing].

Absorber LLM posits that proper context compression must internalize not just the surface form, but the causal contribution of historic tokens on all possible futures. This functional equivalence is achievable by synchronizing the inference behaviors—down to the hidden state propagation—between a model deprived of historical context and a full-context oracle.

Figure 2: Reducing inference complexity by absorbing history into parameters allows deduction with $\mathcal{O}(1)$ main-memory cost per token, independent of context length.

Methodology: Causal Synchronization and Hidden State Alignment

Absorber LLM operationalizes context absorption as a self-supervised causal synchronization problem. Let $X$ denote the historical context to be absorbed, and $Y$ the subsequent text to be generated or inferred. The objective is to update the pretrained parameters $W$ to $W^*$ so that the model $f_{W^*}$, operating on $\mathcal{O}(N)$0 alone, functionally replicates $\mathcal{O}(N)$1 with direct access to $\mathcal{O}(N)$2 across the entire inference trajectory:

$\mathcal{O}(N)$3

This is not a shallow token-level projection: the alignment occurs via the minimization of divergence between all layer-wise hidden states of $\mathcal{O}(N)$4 and $\mathcal{O}(N)$5, enforcing equivalence across the entire computational graph.

The absorption process is realized by fine-tuning (efficiently implemented via LoRA adapters) on the behavioral loss over sliding context windows, continually synchronizing $\mathcal{O}(N)$6 to $\mathcal{O}(N)$7. This continuous parameter update yields a model whose deductive capacity for future generations reflects the semantic and causal influence of history, obviating the need for direct context retention or extended KV cache.

Empirical Evaluation

Absorber LLM is benchmarked on LLaMA2-7B across computational efficiency, in-context learning, long-context chain-of-thought reasoning, and long text summarization, compared against standard transformers, Mamba, and TTT.

• Inference Complexity: Absorber LLM achieves near-constant ($\mathcal{O}(N)$8) per-token latency across context lengths up to $\mathcal{O}(N)$9 tokens, whereas standard transformers scale quadratically and hit OOM errors beyond $\mathcal{O}(N)$0 tokens. Although initial overhead is marginally higher for short contexts, Absorber LLM’s decoupled context length allows robust long-sequence deployment.
• In-Context Reasoning and Summarization: In many-shot Agnews classification, Absorber LLM outperforms Mamba and TTT by 7–10% in accuracy for $\mathcal{O}(N)$1–$\mathcal{O}(N)$2 token ranges. On Musique multi-step reasoning and Samsum summarization, Absorber LLM remains performant as context length expands, remaining operational when standard transformers are infeasible due to memory constraints.
• Ablations: Hidden-state alignment vastly outperforms token-level alignment schemes, confirming that deep representational synchronization is key for effective context absorption. L1 regularization is shown to be optimal for parameter-space memory, encouraging sparse—but precise—internalization of causal dependencies.

Theoretical and Practical Implications

Absorber LLM reframes parametric memory not as static token memorization but as the construction of a causal surrogate: the absorbed parameters replace the historical context in all deductive mechanisms, filtering task-irrelevant noise and preserving only the predictive information relevant for future tasks. This stands in contrast to prior SSM and TTT paradigms, which either collapse long-tail dependencies due to fixed vector bottlenecks or accrue storage overhead and drift due to shallow reconstruction loss.

Practically, Absorber LLM enables:

• Scalable Streaming Inference: By eliminating context-length-dependent compute and memory, production LLM deployments can sustain interactive sessions, lifelong user histories, and continual instruction absorption without retraining.
• Adaptive Long-Context Generalization: Dynamic fine-tuning on session histories allows on-the-fly adaptation and personalized recall within a fixed parameter budget.
• Seamless Memory Editability: The causal and parametric nature of absorption enables future work in selective forgetting, privacy, and dynamic update of internalized memories.

Future Directions

This methodology opens avenues for integrating continual learning paradigms, supporting selective and causal memory editing, and expansion to multimodal sequence domains. Challenges remain in calibrating the absorption window for maximal retention versus overfitting, defining stable continual update protocols, and generalizing alignment approaches beyond text-only autoregressive models.

Conclusion

Absorber LLM delivers a scalable and causally-aligned approach to parameter-based context absorption, subsuming both the computational and generalization limitations of existing long-context models. Experimental findings substantiate its superior efficiency and inference stability, with theoretical underpinnings rooted in causal abstraction and information-theoretic predictability. This approach establishes a promising foundation for future research on scalable, adaptive, and controllable long-context LLMs.

---

References

See (2604.20915) for in-depth empirical results, ablations, and citations to foundational works discussed herein.