LongFilter: A Framework for Long-Context Data Curation

• LongFilter is a data curation framework that identifies and prioritizes training samples with genuine long-range dependencies.
• It employs an information-theoretic approach, comparing short-context and long-context predictions to filter out redundant or less informative text.
• Empirical results demonstrate improved performance on long-context tasks while reducing data requirements by focusing on sequences that truly benefit from extended context.

LongFilter is a data curation framework for long-context continued pretraining that measures whether a longer context provides real predictive benefit over a short context, and then uses that signal to select training data in which long-range dependencies are genuinely informative rather than merely present by length alone. It was introduced to address a specific inefficiency in long-context LLM training: much of the readily available long text on the web and in common corpora can be predicted from local context, or consists of repetitions, independent segments, or non-prose formatting, so naïvely training on long sequences does not necessarily teach a model to use extended context. In experiments extending LLaMA-3-8B from 8K to 64K context, LongFilter improved performance on HELMET, LongBench, and RULER while reducing the amount of data required to reach a given long-context performance level (Deng et al., 29 Oct 2025).

1. Motivation and problem formulation

Long-context LLMs are intended to exploit dependencies across extended spans of text for reasoning, code generation, and document summarization. LongFilter begins from the observation that “long” text is not equivalent to long-context data: many tokens remain predictable from nearby context alone, and some long sequences are largely composed of independent segments, repetitions, tables, markup, or other structures that contribute little long-range signal. Because causal LLM training loss is averaged over all tokens, a large fraction of updates on such data may fail to strengthen the model’s ability to use distant context (Deng et al., 29 Oct 2025).

The framework therefore targets the marginal predictive contribution of extended context. Its goal is to identify sequences in which access to a longer prefix actually reduces uncertainty about the next token, and to concentrate continued-pretraining compute on those sequences. A common misconception in long-context pretraining is that simply increasing the proportion of long documents is sufficient; LongFilter is explicitly designed to reject that premise by asking whether the additional span changes prediction quality, rather than whether the sequence merely exceeds a threshold length.

2. Information-theoretic basis

LongFilter formalizes long-range usefulness at the token level. For a sequence $X=(x_1,\dots,x_N)$ and position $i$, it defines a short context window $K$,

$C_i^{\text{short}} = x_{i-K:i-1},$

a long context window $L>K$,

$C_i^{\text{long}} = x_{i-L:i-1},$

and the extended context

$E_i = x_{i-L:i-K-1},$

so that $C_i^{\text{long}} = (E_i, C_i^{\text{short}})$ (Deng et al., 29 Oct 2025).

The ideal quantity is the conditional mutual information between the next token $T \equiv x_i$ and the extended context $E_i$ given the short context $i$0:

$i$1

Equivalently,

$i$2

For a single context instance $i$3, this becomes

$i$4

Because computing the full KL at every position requires summing over the vocabulary, LongFilter uses a practical surrogate centered on the ground-truth token $i$5:

$i$6

At the sequence level, the sample score is

$i$7

The stated intuition is that this prefers sequences where long context reduces loss substantially on tokens the model deems plausible under the long context. This suggests that LongFilter is not simply measuring surprise or ease, but a specific form of grounded improvement attributable to distant evidence.

3. Scoring procedure and implementation

LongFilter uses a separate evaluator model rather than the model being continued-pretrained. In the reported implementation, Llama-3.1-8B, with context up to 128K, computes the short- and long-context next-token probabilities, while the target model is LLaMA-3-8B, whose context is extended from 8K to 64K in the training pipeline. The short window is set to $i$8 tokens and the long window to $i$9 tokens (Deng et al., 29 Oct 2025).

The scoring pipeline contains two prediction modes. For long-context modeling, the full sequence, up to 64K tokens, is processed once to obtain next-token distributions under the long context. For short-context modeling, the sequence is segmented into $K$0-sized chunks with overlap so that boundary tokens are predicted with sufficient local context; the overlap is introduced to avoid the cold-start issue at chunk heads, although no specific stride fraction is prescribed. Token-level surrogate scores are then computed at every position and averaged over the sequence length to produce $K$1. No special clipping, ratio normalization, or domain weights are applied; length normalization is achieved solely by averaging over $K$2.

The compute pattern consists of one long-context pass and multiple short-context passes over overlapping chunks. Complexity scales linearly with the number of tokens, with a small overhead from chunk overlap. In practice, each SlimPajama sub-corpus is scored in approximately one day using 32 NVIDIA H100 GPUs, with batching for throughput and standard transformer prefix-caching used to amortize repeated prefixes in the short-context windows (Deng et al., 29 Oct 2025).

4. Data selection and continued-pretraining pipeline

The base source corpus is SlimPajama-627B. The reported study uses three sub-corpora—ArXiv, Books, and CommonCrawl—and separates “long” from “short” examples by domain-specific thresholds: ArXiv $K$3 tokens, Book $K$4, and CommonCrawl $K$5, with the long portion totaling approximately 19B tokens per domain. The continued-pretraining dataset mixes 80% long and 20% short sequences, and LongFilter is applied only to the long portion. After scoring, instances are ranked by $K$6 and the top 20% are selected for training; the method uses hard filtering rather than reweighting, and there is no per-domain weighting beyond the chosen mixing ratios (Deng et al., 29 Oct 2025).

The training configuration keeps optimizer, learning rate schedule, and data packing settings identical to ProLong. LLaMA-3-8B is continued-pretrained with packed 64K-token sequences, a batch size of 4M tokens, and 1,000 steps for a total of 4B tokens processed. RoPE base frequency is increased from $K$7 to $K$8, and no other architectural changes are made. For instruction-based benchmarks, specifically RULER and LongBench, the models are additionally SFT’d on 1B tokens of UltraChat following ProLong’s settings to enable instruction comprehension (Deng et al., 29 Oct 2025).

Within the paper’s taxonomy, length-only mixing such as ProLong is described as a “0→1” step because it exposes the model to long sequences, whereas LongFilter is a “1→2” step because it prioritizes sequences whose extended context actually changes predictions. This distinction is central to the framework’s intended contribution.

5. Empirical results

Evaluation is conducted on HELMET, LongBench, and RULER. HELMET covers Recall, RAG, Re-rank, In-Context Learning, and QA; LongBench reports category-wise scores for SingleQA, MultiQA, Summarization, ICL, Synthetic, and Code; RULER stress-tests long-range retrieval and compositionality, including several NIAH sub-tasks and other synthetic tasks. Across all three domain settings—ArXiv, Book, and CommonCrawl—LongFilter shows the fastest and most stable improvements with training scale on HELMET Recall/NIAH, surpasses ProLong and LongWanjuan at small scales of 0.5–1.0B tokens, and reaches above 90 SubEM by 4B tokens. The paper also reports up to a 10% accuracy gain on recall tasks at 1B tokens compared to unfiltered training (Deng et al., 29 Oct 2025).

Domain	LongBench overall	RULER overall
ArXiv	39.52	70.13
Book	39.81	78.95
CommonCrawl	40.66	75.37

These overall scores were the highest among the compared methods for RULER in all three domains. On LongBench, LongFilter achieved 39.52 on ArXiv, 39.81 on Book, and 40.66 on CommonCrawl, compared with ProLong’s 38.58, 37.47, and 38.37, and LongWanjuan’s 39.36, 37.69, and 40.02, respectively. The reported gains are especially large on Book, where LongFilter improves by +2.34 over ProLong and +2.12 over LongWanjuan, and on CommonCrawl, where it improves by +2.29 over ProLong and +0.64 over LongWanjuan. Gains are particularly notable in Synthetic tasks and competitive in Code. On RULER, the largest reported margin is also on Book: +5.60 over ProLong and +5.21 over LongWanjuan, with especially strong improvements in structured multi-key and multi-value NIAH sub-tasks (Deng et al., 29 Oct 2025).

The efficiency result is equally prominent: comparable HELMET performance is reached with approximately 1.5B filtered tokens versus 3–4B unfiltered tokens, indicating roughly halved data requirements. Qualitative case studies align with the formal objective. High scores appear on coherent academic prose, such as PhD thesis text, where earlier sections constrain later terminology and argumentation, while low scores appear on repetitive, non-prose segments such as LaTeX TikZ drawing commands. Abrupt score changes can occur when preprocessed submissions concatenate multiple .tex files, which the paper identifies as a document-boundary artifact rather than a modeling principle.

6. Robustness, limitations, and relation to adjacent filtering strategies

The reported configuration fixes $K$9 and $C_i^{\text{short}} = x_{i-K:i-1},$0 and uses the surrogate weighting $C_i^{\text{short}} = x_{i-K:i-1},$1. The paper does not present a formal ablation over $C_i^{\text{short}} = x_{i-K:i-1},$2, alternative scoring functions such as unweighted log-likelihood differences or ratios, or systematic variation of the selection threshold $C_i^{\text{short}} = x_{i-K:i-1},$3, although top-20% selection yields consistent improvements across ArXiv, Books, and CommonCrawl. Short-context task effects are not the focus; the emphasis is long-context generalization, while post-SFT performance indicates transfer to instruction-following (Deng et al., 29 Oct 2025).

The main limitations are tied to the scoring signal. Highly repetitive or non-semantic content, including boilerplate, tables, markup, or code blocks with local patterns, tends to receive low scores, which is intended behavior, but in domains such as codebases or logs, long-range dependencies may be subtle and not well captured by next-token probabilities alone. Formatting artifacts, especially concatenated heterogeneous files, can induce abrupt document-level score shifts. The evaluator model’s own inductive biases also matter: a model poorly attuned to a domain such as specialized code or mathematics may misestimate the gain from long context. Finally, LongFilter explicitly prioritizes long-range dependencies, so aggressive use may under-select samples that are valuable for local reasoning (Deng et al., 29 Oct 2025).

The recommended mitigations are pragmatic rather than architectural: use domain-aware evaluators when possible, combine LongFilter with conventional quality filters such as deduplication, toxicity filtering, and language identification, inspect token-level score distributions on sampled documents, and adjust the selection fraction by domain. In conceptual terms, LongFilter positions itself “beyond length.” It differs from perplexity-only filtering, which selects generally easy or hard sequences without explicitly contrasting short and long contexts, and from retrieval-based filtering, which targets task-specific structure. Its defining claim is that long-context quality should be measured by the marginal predictive gain of extended context, with an explicit information-theoretic rationale. Code and scripts for preprocessing, scoring, training, evaluation, and visualization are released at the project repository linked from the paper (Deng et al., 29 Oct 2025).