Ultra-long Context Processing

• Ultra-long context processing is the method enabling language models to manage sequences of hundreds of thousands to millions of tokens using scalable techniques.
• It employs dynamic sparse attention, chunked processing, and hybrid retrieval-augmented memory to overcome the quadratic self-attention bottleneck.
• These approaches improve document understanding, codebase modeling, and recommendation systems while addressing challenges like memory scaling and boundary fragmentation.

Ultra-long context processing refers to the algorithmic and systems-level methods that allow LLMs and sequence models to efficiently and effectively handle input sequences ranging from hundreds of thousands to millions of tokens—a regime far beyond the typical context length of standard Transformer architectures. This area spans architectural changes, compression and memory techniques, dynamic sparse attention, multi-agent pipelines, and comprehensive recipes for scalable pretraining. The central challenge is to solve the quadratic scaling bottleneck of self-attention (both compute and memory), while maintaining or improving accuracy, recall fidelity, random access, and generalization across in-distribution and out-of-distribution input lengths. Ultra-long context processing has immediate implications for document understanding, codebase modeling, retrieval, recommendation, and any domain requiring reasoning over extended sequential dependencies.

1. Architectural Paradigms for Ultra-Long Contexts

1.1. Non-Attention and Hybrid Models

One direction eliminates full self-attention in favor of more scalable operators. The model in “Breaking Quadratic Barriers: A Non-Attention LLM for Ultra-Long Context Horizons” avoids token-to-token attention by composing four components: (a) State Space (S4-inspired) blocks for intra-chunk global mixing; (b) Multi-resolution convolutions for efficient local context capture; (c) an external retrieval-augmented memory at the chunk level; (d) a lightweight RNN supervisor for threading global state (Kiruluta et al., 9 May 2025). The input is divided into chunks $X_m$; each chunk is independently embedded, processed via SSM and multi-scale Conv1d, summarized to a chunk-level vector, write/read to external memory for history, and fused via a gating MLP. A recurrent cell additionally propagates global state. The architecture achieves $O(n\log c)$ or $O(n)$ time and $O(B c d + B h)$ memory per chunk, compared to $O(n^2)$ for attention, and matches or exceeds the perplexity and bit-per-char (bpc) of best-in-class Transformer variants.

Another paradigm is hierarchical memory: models such as HMT (Hierarchical Memory Transformer) impose a multi-level persistent memory inspired by human memory hierarchy, passing memory tokens forward across chunks, recording “sensory”, “short-term”, and “long-term” information, and recalling with lightweight cross-segment attention (He et al., 2024).

1.2. Chunked and Streaming Processing

Across architectures, chunking is the dominant motif: the input sequence is split into segments for local or hierarchical processing (often with overlap), enabling streaming, constant-memory pipelines, and separate random-access or recurrent pathways for global dependencies. Chunk summaries are combined, retrieved, or updated per chunk (or per head/layer) to ensure coverage over long horizons.

1.3. Hybrid Retrieval and Compression

Retrieval-augmented memory—external buffers of learned embeddings or summary vectors—is standard in non-attention models and many hybrid systems. These allow both local and global information to be fused without ever allocating quadratic-sized attention maps.

For compatibility with conventional Transformers, compression schemes such as ParallelComp implement dynamic chunking, self-information-based chunk eviction, and selective token-level key/value (KV) cache pruning to support extrapolation to 128K tokens and beyond with minimal loss (Xiong et al., 20 Feb 2025). REFORM applies a two-phase inference strategy: incremental chunk processing with early exit and “heavy-hitter” token retention, followed by similarity-based gathering and recomputation for full-fidelity KV cache (Song et al., 1 Jun 2025).

2. Sparse, Hierarchical, and Dynamic Attention

2.1. Hierarchical Sparse Attention (HSA)

HSA, introduced in “Every Token Counts: Generalizing 16M Ultra-Long Context in LLMs,” is a learnable method that satisfies three criteria: sparsity, random-access, and strong extrapolation to longer-than-training contexts (Hu et al., 28 Nov 2025). Sequence is chunked, each chunk produces a “landmark” key for retrieval scoring, and per-step routing selects the top-$k$ relevant chunks for intra-chunk attention. Only those chunks’ token memories are available to each decoder step, fusing local fine-grained attention with global, sparse random selection. HSA is slotted alongside sliding-window attention (SWA), and only sparse/landmark keys and “mid-layer” chunk summaries are maintained across layers. HSA-trained models support up to 16M-token contexts, achieving >90% accuracy on retrieval tasks at that length.

2.2. System-Level Sparse and Block Sparse Kernels

Sparse attention accelerates both training and serving, but dynamic sparse pattern scheduling introduces significant communication and load imbalance in distributed training. MTraining resolves this via three innovations: dynamic vertical + slash line sparse patterns (learned per batch), block-striped ring attention for step/worker load balance, and a hierarchical ring for efficient multi-node scaling (Li et al., 21 Oct 2025). This results in linear scaling on 32xA100s, with up to 6x throughput improvement at 512K tokens.

FlashPrefill, by contrast, targets the inference prefill phase: instantaneous (“one-pass”) block-score pattern discovery (vertical, slash, block-sparse) is performed, followed by max-based thresholding that prunes low-score blocks without costly sorting. The output is a highly sparse index for block-wise attention, yielding a measured 27.78x speedup on 256K contexts over dense attention baselines (Fan et al., 6 Mar 2026).

3. Training Recipes and Scaling Strategies

3.1. Efficient Pretraining and SFT

Multiple works show that context extension does not necessitate retraining from scratch. UltraLong-8B extends Llama-3.1-8B-Instruct from 128K to 1M–4M tokens by continued pretraining with a tailored long-document mix, context-aware RoPE rescaling (YaRN), explicit document separations, and efficient global full-attention with tensor and context parallelism (Xu et al., 8 Apr 2025). No synthetic or long-instruction SFT data is required to preserve reasoning and instruction-following. Short-context SFT is sufficient, and one-step context scaling outperforms staged curriculum.

LongSkywork demonstrates that a few hundred LC-SFT steps with synthetic data (“chunk interleaved pretraining” and “synthetic lengthy tables”) suffices to convert a standard SFT model into a 200K-context model (Zhao et al., 2024). LC-SFT need only comprise 200 steps and benefit from programmatic QA generation, outperforming standard book QA.

Fine-tuning for extended RoPE base, as in Llama-3-8B-80K QLoRA, operates via low-rank adapters and positional encoding extension, with a small number of synthetic (GPT-4) super-long samples (Zhang et al., 2024). This produces perfect retrieval at 80K+ tokens without meaningful short-context degradation.

3.2. Distributed Long-Context Training Kernels

FPDT achieves a 16x increase in trainable context by combining pipeline model parallelism, sequence chunking within each layer, host offload of inactive keys/values, and double-buffering to overlap CPU-GPU transfer with compute (Yao et al., 2024). Such approaches allow, for example, training 8B models at sequence length 2M on only 4 GPUs.

4. Advanced Inference and Question-Aware Selection

4.1. Dynamic Chunking and Selection

For QA and reading comprehension, semantic-aware chunking combined with question-sensitive chunk selection has been shown to outperform fixed-length chunking. DCS (Dynamic Chunking and Selection) uses Sentence-BERT to form variable-length, semantically coherent chunks, and a question-aware MLP to select chunks likely to answer a given query—yielding stable QA accuracy on up to 256K tokens at a fraction of the attention cost (Sheng et al., 1 Jun 2025).

XpandA exemplifies multi-agent orchestration for ultra-long context—breaking input into adaptively sized, overlapping chunks, assigning explorer agents to process each, and updating a shared memory table of question–information pairs (Xiao et al., 27 May 2025). The protocol involves iterative replay of parts of the context (as needed) using a decider agent with state-tracking to resolve cross-chunk dependencies, minimizing information loss and cumulative query latency, and improving F1 by up to 20% over standard agent and RAG baselines on up to 1M tokens.

4.2. Compression, Latent Compilation, and Test-Time Adaptation

Latent Context Compilation (LCC) reframes adaptation as one-shot “compilation” of long input into a small portable buffer token cache using a disposable LoRA compiler (Li et al., 31 Jan 2026). The buffer tokens capture context via a reconstruction loss and are regularized by random instruction queries to reside on the model manifold. These tokens—stateless, of size up to a 16× compression ratio—can be prepended to any frozen base model, with negligible loss up to 16×. The approach matches full-context quality on CoQA, BookSum, and XSum benchmarks while saving 94% of memory.

InfiniteICL formalizes parameter-updating as the canonical mechanism for infinite context integration: raw context is distilled into parameter deltas via elicitation, selection, and consolidation, equivalent to long-to-short term memory transformation, reducing total inference memory and achieving 103% of full-context prompting while reducing context by 90% (Cao et al., 2 Apr 2025).

5. Application Domains

5.1. Recommendation and Sequential User Modeling

Ultra-long sequence attention is required to model user behavior at production scale. VQL (Vector Quantization Attention) compresses only attention keys using a learnable codebook, yielding inference that is $L$-independent (key-only quantization) and supports efficient group-based multi-scale quantization, temporal/contextual kernels, and cacheability (Li et al., 23 Aug 2025). End-to-end training preserves ranking accuracy and recall on up to 10K-length user behavior with lower latency than all baselines.

For candidate retrieval, LongRetriever leverages in-context training and multi-context retrieval: user–candidate pairs are modeled with category-filtered sequences, and at serving, a small set of contexts are selected to query category-specific ANN indices. A/B tests show significant boosts in retrieval and conversion on a billion-user platform (Qin et al., 21 Aug 2025).

5.2. Database, QA, and Summarization

Long-context NL2SQL pipelines for multiturn QA and program synthesis are practical on gemini-1.5-pro with careful prompt organization, inclusion of schema, in-context examples, user hints, disambiguations, and schema-aware data—pushing execution accuracy to 67%+ with hundreds of thousands of context tokens, at linear latency scaling (Chung et al., 21 Jan 2025).

Ultra-long context summarization, book-level QA, and codebase modeling have been demonstrated at scales exceeding 1M tokens, relying on efficient chunked, sparse, hybrid, or fully memory-based architectures (Liao et al., 2024, Zhang et al., 2024, Xu et al., 8 Apr 2025, Hu et al., 28 Nov 2025).

6. Complexity, Scaling Laws, and Empirical Frontiers

The dominant complexity reduction across all approaches arises from chunk-wise and sparse mechanisms: standard self-attention is $O(n^2)$, but chunked models (S4-based, convolutional, retrieval-augmented, RNN-supplemented) achieve $O(n\log c)$ or $O(n)$ time and peak memory; sparse attention (HSA, block-sparse, ring-striped) reaches practical linear time by dynamically routing attention; compressed and buffer-token models amortize inference/serving over buffer size rather than the raw context length. Empirically, all these methods have established accurate or perfect long-range recall, robust QA, and minimal short-context regression for context lengths $10^5$–$10^7$ tokens (Kiruluta et al., 9 May 2025, Hu et al., 28 Nov 2025, Zhang et al., 2024, Zhao et al., 2024, Xu et al., 8 Apr 2025).

Scaling strategies—one-step pretraining, explicit document separation, advanced RoPE scaling (YaRN), and flexible tensor/context parallelism—are critical for training models at these lengths with constant or sub-quadratic hardware cost.

7. Limitations, Open Challenges, and Outlook

While ultra-long context processing has advanced substantially, key open problems remain:

• For hybrid/sparse attention, the balance between local detail and global retrieval, kernel-level inefficiencies at shorter context, and head ratio bottlenecks constrain ultimate flexibility (Hu et al., 28 Nov 2025).
• Non-attention architectures can underperform on ultra-fine-grained order or in domains requiring precise position recall within chunks (Kiruluta et al., 9 May 2025).
• Chunk-boundary fragmentation and information loss across boundaries are limiting factors for all chunked and selection-based methods; variable or dynamic chunking, and hierarchical or learned chunk-splitting, may alleviate this (Sheng et al., 1 Jun 2025, He et al., 2024).
• Compilation and buffer-token methods shift the bottleneck to instance-specific GPU compute at compilation time, may blur details beyond certain compression ratios, and are unsuited for low-latency requests unless amortized (Li et al., 31 Jan 2026).
• Distributed training on ultra-long contexts still incurs significant communication costs, especially at inter-node scale, and may be further improved with hardware co-design (Li et al., 21 Oct 2025, Yao et al., 2024).
• Memory-augmented/recurrent models may accumulate “stale” memory over million-token horizons; dynamic memory allocation and multi-level tag-based recall are proposed as mitigations (He et al., 2024).

The field continues to converge on systems and algorithms that move beyond quadratic scaling, with architectural flexibility suited to both training and inference. The combinatorial space of memory, retrieval, selection, compression, and distributed optimization will likely shape the next generation of ultra-long context LLMs.