EntropyLong: Verified Long-Context Construction

• EntropyLong is a long-context data construction method that uses model-in-the-loop entropy verification to ensure genuine long-range dependencies rather than simple token concatenation.
• It employs a Shannon entropy-based uncertainty reduction criterion to verify candidate context chunks retrieved from large corpora, ensuring measurable information gain.
• Experiments show EntropyLong outperforms heuristic methods on benchmarks like RULER and LongBench-v2, especially at extended context lengths.

EntropyLong is a long-context data construction method for training LLMs on documents that contain verified long-range dependencies rather than merely long token sequences. It addresses the observation that long-context LLMs—including architectures associated with Longformer, BigBird, and rotary-embeddings—may support very large context windows while remaining bottlenecked by the scarcity of genuine long-range dependencies in pre-training data (Jia et al., 26 Sep 2025). Its central mechanism is model-in-the-loop verification: candidate distant context is accepted only if it reduces a model’s token-level predictive uncertainty, measured by Shannon entropy, by more than a specified threshold. In the reported implementation, this procedure is used with FineWebEdu and Cosmopedia to construct 128K-length sequences containing verified dependencies, and models trained on the resulting data improve on RULER and, after instruction fine-tuning, on LongBench-v2 (Jia et al., 26 Sep 2025).

1. Problem setting and motivation

EntropyLong is motivated by a specific failure mode in long-context training data construction. Standard concatenation of unrelated documents produces superficially long sequences but does not guarantee real information flow across distant spans. Heuristic synthesis methods such as Quest and NExtLong assume topical coherence or interleave distractors, but they do not verify whether the added context actually helps the model predict tokens in the root document (Jia et al., 26 Sep 2025).

The method therefore reframes long-context synthesis as an information-theoretic verification problem. Its key insight is that a model’s predictive uncertainty at the token level can identify positions where the current local context is insufficient. If a distant chunk reduces that uncertainty, the chunk is treated as evidence of a genuine long-range dependency rather than a spurious correlation (Jia et al., 26 Sep 2025). This suggests that EntropyLong is not merely a retrieval-and-concatenation pipeline; it is a dependency screening procedure grounded in measurable information gain.

A later paper, PolicyLong, describes EntropyLong as a “one-shot, off-policy data synthesis procedure” and uses it as the baseline formulation of entropy-verified long-context construction (Jia et al., 9 Apr 2026). That later characterization does not alter EntropyLong’s original objective, but it is relevant for situating the method within subsequent work on context extension.

2. Information-theoretic formalism

EntropyLong defines uncertainty with the predictive distribution of a LLM. Let $M$ be a LLM over vocabulary $V$. At position $t$, with predictive distribution

$p_t(w)=M(w\mid x_1\ldots x_{t-1}),$

the token-level predictive entropy is

$H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$

If a candidate context $C_{\text{new}}$ is prepended before $x_1\ldots x_{t-1}$, the entropy reduction is

$\Delta H = H(p_t)-H(p_t\mid C_{\text{new}}).$

EntropyLong accepts $C_{\text{new}}$ as a verified dependency only if $\Delta H$ exceeds a threshold $V$0 (Jia et al., 26 Sep 2025).

The PolicyLong exposition presents the same construction in normalized form. For a root document $V$1 and candidate chunk $V$2, it defines

$V$3

and notes that a substantial entropy drop implies a non-trivial mutual information signal:

$V$4

when the conditioned entropy is smaller than the unconditioned entropy (Jia et al., 9 Apr 2026). In EntropyLong’s own formulation, the emphasis is on measurable uncertainty reduction rather than on a separate contrastive objective (Jia et al., 26 Sep 2025).

Within this framework, high-entropy positions are treated as “uncertainty hotspots” where additional context may resolve an information deficit (Jia et al., 9 Apr 2026). The underlying methodological claim is narrow but precise: semantic similarity alone cannot certify dependency quality; entropy reduction supplies the verification criterion (Jia et al., 26 Sep 2025).

3. Data construction pipeline

EntropyLong’s pipeline operates in four stages (Jia et al., 26 Sep 2025).

First, for each document $V$5, the method computes per-token entropies $V$6 for $V$7. It then computes the document-level mean $V$8 and standard deviation $V$9 of $t$0 and sets

$t$1

with $t$2 in practice. Positions

$t$3

are marked as high-uncertainty (Jia et al., 26 Sep 2025).

Second, for each $t$4, the method extracts a short query window

$t$5

with $t$6 tokens, and uses a dense retriever, specified as jina-embeddings-v3 + Faiss, to retrieve the top-$t$7 semantically similar chunks from a large external corpus (Jia et al., 26 Sep 2025).

Third, for each candidate chunk $t$8, the method prepends the chunk before the root document and recomputes the entropy at the target position. It computes $t$9 and accepts $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$0 as a verified dependency if $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$1, with $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$2 (Jia et al., 26 Sep 2025). The pseudocode sketch further indicates that, among unused candidates, the procedure tracks the best verified chunk for a position before adding it to the verified set (Jia et al., 26 Sep 2025).

Fourth, the method collects all verified chunks for document $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$3, randomly shuffles them, and concatenates them with $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$4:

$p_t(w)=M(w\mid x_1\ldots x_{t-1}),$5

with truncation or padding to the target length $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$6 tokens (Jia et al., 26 Sep 2025).

A concise summary of the operational settings reported for the pipeline is given below.

Component	Reported setting
High-entropy threshold	$p_t(w)=M(w\mid x_1\ldots x_{t-1}),$7, $p_t(w)=M(w\mid x_1\ldots x_{t-1}),$8
Query window	$p_t(w)=M(w\mid x_1\ldots x_{t-1}),$9 tokens
Retriever	jina-embeddings-v3 + Faiss
Verification threshold	$H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$0
Target sequence length	$H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$1 tokens

The construction is explicitly model-in-the-loop because the same base model both identifies uncertainty hotspots and validates whether a retrieved context produces measurable information gain (Jia et al., 26 Sep 2025). A plausible implication is that the resulting training sequences are biased toward contexts the model can demonstrably exploit, rather than contexts that only appear topically compatible.

4. Dataset construction and training configuration

The reported source corpora are FineWebEdu and Cosmopedia, together comprising more than 1 billion documents. From these corpora, 100K documents are randomly sampled as root texts, while the full set is used for retrieval (Jia et al., 26 Sep 2025).

The resulting dataset contains 4 billion tokens of 128K-length sequences. Each sequence contains on average 46 verified long-range dependencies, and the mean Contextual Information Gain is reported as $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$2 per dependency (Jia et al., 26 Sep 2025). These quantities are central to the paper’s claim that the constructed long-context samples are not generic concatenations but sequences with verified dependency structure.

The base model in the main experiments is Meta-Llama-3-8B with rotary-emb-base=$H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$3 and context window $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$4. Training is reported as 1 000 iterations, global batch size $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$5 million tokens, learning rate $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$6 with cosine decay, bfloat16, and grad-clip $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$7 (Jia et al., 26 Sep 2025). The baselines are Quest and NExtLong (Jia et al., 26 Sep 2025).

The paper evaluates on RULER before instruction tuning and on LongBench-v2 after instruction fine-tuning with UltraChat (Jia et al., 26 Sep 2025). A later comparison in PolicyLong uses Qwen2.5-3B and reports EntropyLong’s absolute scores on RULER, HELMET, and LongBench-v2 in that separate setup (Jia et al., 9 Apr 2026). Those results belong to the later comparative study rather than to EntropyLong’s original Meta-Llama-3-8B experiments.

5. Empirical performance

On RULER, EntropyLong outperforms both Quest and NExtLong across all tested context lengths from 8K to 128K in the reported Meta-Llama-3-8B setup (Jia et al., 26 Sep 2025). The full table is as follows.

Method	8K	16K	32K	64K	128K	avg
Quest	91.39	89.72	84.37	77.07	60.11	80.53
NExtLong	89.99	88.58	86.04	83.52	77.99	85.22
EntropyLong	91.50	90.11	88.95	85.04	81.26	87.37

The paper highlights that the largest gain occurs at 128K, where EntropyLong is ahead of NExtLong by $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$8 points (Jia et al., 26 Sep 2025). This suggests that the advantage of verified dependencies becomes more pronounced as retrieval distance increases and long-context tasks become less tolerant of spurious context.

After instruction fine-tuning, EntropyLong also exceeds both baselines on LongBench-v2 (Jia et al., 26 Sep 2025).

Method	Easy	Hard	Short	Medium	Long	Overall
Quest	17.70	25.10	25.60	20.00	21.30	22.30
NExtLong	21.40	25.70	27.20	21.90	23.10	24.10
EntropyLong	25.50	28.90	30.00	23.70	31.50	27.60

The reported “Long” category gain is $H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).$9 points versus the best baseline (Jia et al., 26 Sep 2025). Because LongBench-v2 is described as containing real-world long-context tasks, the result is presented as evidence that the verified-dependency pretraining signal transfers beyond synthetic retrieval settings (Jia et al., 26 Sep 2025).

PolicyLong later reports that, on Qwen2.5-3B, EntropyLong achieves 69.86 at 64K and 63.45 at 128K on RULER, 43.88 at 64K and 40.08 at 128K on HELMET, and 24.2 on Medium and 28.7 on Long in LongBench-v2 after SFT (Jia et al., 9 Apr 2026). In that paper these numbers are used to show that EntropyLong improves over random concatenation or embedding-based heuristics, while also motivating on-policy refinement (Jia et al., 9 Apr 2026).

6. Ablations, analysis, and limitations

The ablation studies emphasize that entropy-based verification is not incidental to performance. In the comparison between full EntropyLong and EntropyLong-NoVerify, where the latter always accepts the top-1 retrieval without entropy checking, the full method improves average RULER from 85.82 to 87.37, with gains up to $C_{\text{new}}$0 at 32K (Jia et al., 26 Sep 2025). This directly supports the claim that heuristic retrieval without verification is insufficient.

Threshold selection also matters. Varying the entropy threshold parameter $C_{\text{new}}$1 changes the number of high-entropy positions and average RULER; among the reported settings, $C_{\text{new}}$2 yields the best average score of 87.37 with 292 high-entropy tokens, whereas $C_{\text{new}}$3 yields 82.49 with 913 tokens and $C_{\text{new}}$4 yields 85.52 with 83 tokens (Jia et al., 26 Sep 2025). Similarly, for the verification threshold $C_{\text{new}}$5, the reported optimum is $C_{\text{new}}$6, which produces 46 verified dependencies and average RULER 87.37; both lower and higher thresholds reduce performance (Jia et al., 26 Sep 2025).

The query window size $C_{\text{new}}$7 is also ablated, with $C_{\text{new}}$8 giving the best average RULER of 87.37 compared with 86.61 for $C_{\text{new}}$9, 86.04 for $x_1\ldots x_{t-1}$0, and 86.85 for $x_1\ldots x_{t-1}$1 (Jia et al., 26 Sep 2025). For concatenation strategy, random shuffling slightly exceeds sequential ordering, 87.37 versus 87.06 (Jia et al., 26 Sep 2025).

The short-text evaluation shows minimal degradation: Llama3-8B scores 61.77 on six short-text benchmarks, while +EntropyLong scores 61.83 (Jia et al., 26 Sep 2025). The paper also reports attention-pattern analysis indicating that EntropyLong maintains higher attention to correct answers as context length grows, mitigating “lost-in-middle,” with relative gains up to $x_1\ldots x_{t-1}$2 in middle positions (Jia et al., 26 Sep 2025).

The limitations stated in the paper are specific. Retrieval quality depends on external corpus coverage, so low recall may miss dependencies. The thresholds $x_1\ldots x_{t-1}$3 and $x_1\ldots x_{t-1}$4 require tuning per corpus and model. Future directions proposed in the paper include adaptive thresholds, alternative uncertainty signals such as mutual information, and integration with retrieval-augmented generation (Jia et al., 26 Sep 2025).

A later critique in PolicyLong isolates two structural drawbacks of EntropyLong’s offline design: static screening and lack of hard negatives (Jia et al., 9 Apr 2026). The off-policy argument is that data are constructed once using the initial base model, so the selected high-entropy positions may cease to be informative as the model improves. PolicyLong reports that when those original positions are reevaluated on a later checkpoint, their median entropy drops from 6.63 to 5.34, and it describes this as an off-policy gap (Jia et al., 9 Apr 2026). This is a later interpretation rather than a claim in the original EntropyLong paper, but it has become part of the method’s subsequent reception.

7. Position within long-context data curation

EntropyLong is best understood as a shift from heuristic long-context synthesis toward uncertainty-verified synthesis. Its stated contribution is to replace generic concatenation or heuristic variants with a model-in-the-loop, information-theoretic approach that filters out spurious dependencies and guarantees measurable information gain (Jia et al., 26 Sep 2025). In that sense, the method treats long-context training data curation as an empirical verification problem rather than a topical similarity problem.

The method’s broader significance lies in connecting data construction to the predictive state of a LLM. High-entropy positions operationalize where the model lacks information; verified retrieved chunks operationalize which distant contexts actually resolve that deficit (Jia et al., 26 Sep 2025). This suggests a general pattern for long-context supervision in which the unit of curation is not the document pair but the uncertainty reduction event.

Subsequent work has extended this line of thought. PolicyLong retains entropy computation, retrieval, and verification, but re-executes screening using the current model so that the training distribution tracks evolving capabilities, introducing what it calls an emergent self-curriculum and adding hard negative contexts derived from the same entropy landscape (Jia et al., 9 Apr 2026). That development indicates that EntropyLong established a reusable framework for principled data curation even as later work identified the limitations of its one-shot offline form.