---
title: 'EntropyLong: Verified Long-Context Construction'
url: https://www.emergentmind.com/topics/entropylong
type: topic
---

# EntropyLong: Verified Long-Context Construction

EntropyLong is a long-context data construction method for training language models on documents that contain verified long-range dependencies rather than merely long token sequences. It addresses the observation that long-context LLMs—including architectures associated with Longformer, BigBird, and rotary-embeddings—may support very large context windows while remaining bottlenecked by the scarcity of *genuine* long-range dependencies in pre-training data [2510.02330]. Its central mechanism is model-in-the-loop verification: candidate distant context is accepted only if it reduces a model’s token-level predictive uncertainty, measured by Shannon entropy, by more than a specified threshold. In the reported implementation, this procedure is used with FineWebEdu and Cosmopedia to construct 128K-length sequences containing verified dependencies, and models trained on the resulting data improve on RULER and, after instruction fine-tuning, on LongBench-v2 [2510.02330].

## 1. Problem setting and motivation

EntropyLong is motivated by a specific failure mode in long-context training data construction. Standard concatenation of unrelated documents produces superficially long sequences but does not guarantee real information flow across distant spans. Heuristic synthesis methods such as Quest and NExtLong assume topical coherence or interleave distractors, but they do not verify whether the added context actually helps the model predict tokens in the root document [2510.02330].

The method therefore reframes long-context synthesis as an information-theoretic verification problem. Its key insight is that a model’s predictive uncertainty at the token level can identify positions where the current local context is insufficient. If a distant chunk reduces that uncertainty, the chunk is treated as evidence of a genuine long-range dependency rather than a spurious correlation [2510.02330]. This suggests that EntropyLong is not merely a retrieval-and-concatenation pipeline; it is a dependency screening procedure grounded in measurable information gain.

A later paper, PolicyLong, describes EntropyLong as a “one-shot, off-policy data synthesis procedure” and uses it as the baseline formulation of entropy-verified long-context construction [2604.07809]. That later characterization does not alter EntropyLong’s original objective, but it is relevant for situating the method within subsequent work on context extension.

## 2. Information-theoretic formalism

EntropyLong defines uncertainty with the predictive distribution of a language model. Let $M$ be a language model over vocabulary $V$. At position $t$, with predictive distribution
$$
p_t(w)=M(w\mid x_1\ldots x_{t-1}),
$$
the token-level predictive entropy is
$$
H(p_t)=-\sum_{w\in V} p_t(w)\cdot \log p_t(w).
$$
If a candidate context $C_{\text{new}}$ is prepended before $x_1\ldots x_{t-1}$, the entropy reduction is
$$
\Delta H = H(p_t)-H(p_t\mid C_{\text{new}}).
$$
EntropyLong accepts $C_{\text{new}}$ as a verified dependency only if $\Delta H$ exceeds a threshold $\epsilon$ [2510.02330].

The PolicyLong exposition presents the same construction in normalized form. For a root document $A$ and candidate chunk $B_{i,j}$, it defines
$$
\Delta H(B_{i,j},x_i)
=
\frac{H_M(x_i\mid x_{<i})-H_M(x_i\mid B_{i,j},x_{<i})}
{H_M(x_i\mid x_{<i})},
$$
and notes that a substantial entropy drop implies a non-trivial mutual information signal:
$$
I_M(x_i;B\mid x_{<i})
=
H_M(x_i\mid x_{<i})-H_M(x_i\mid B,x_{<i})>0
$$
when the conditioned entropy is smaller than the unconditioned entropy [2604.07809]. In EntropyLong’s own formulation, the emphasis is on measurable uncertainty reduction rather than on a separate contrastive objective [2510.02330].

Within this framework, high-entropy positions are treated as “uncertainty hotspots” where additional context may resolve an information deficit [2604.07809]. The underlying methodological claim is narrow but precise: semantic similarity alone cannot certify dependency quality; entropy reduction supplies the verification criterion [2510.02330].

## 3. Data construction pipeline

EntropyLong’s pipeline operates in four stages [2510.02330].

First, for each document $D=x_1\ldots x_n$, the method computes per-token entropies $H_t$ for $t=1\ldots n$. It then computes the document-level mean $\mu_H$ and standard deviation $\sigma_H$ of $\{H_t\}$ and sets
$$
TH=\mu_H+\alpha\cdot\sigma_H,
$$
with $\alpha=2.0$ in practice. Positions
$$
U=\{t\mid H_t>TH\}
$$
are marked as high-uncertainty [2510.02330].

Second, for each $t\in U$, the method extracts a short query window
$$
q_t=x_{t-w:t+w},
$$
with $w=16$ tokens, and uses a dense retriever, specified as jina-embeddings-v3 + Faiss, to retrieve the top-$K$ semantically similar chunks from a large external corpus [2510.02330].

Third, for each candidate chunk $C_j$, the method prepends the chunk before the root document and recomputes the entropy at the target position. It computes $\Delta H_j=H_t-H'_t$ and accepts $C_j$ as a verified dependency if $\Delta H_j>\epsilon$, with $\epsilon=0.4$ [2510.02330]. The pseudocode sketch further indicates that, among unused candidates, the procedure tracks the best verified chunk for a position before adding it to the verified set [2510.02330].

Fourth, the method collects all verified chunks for document $D$, randomly shuffles them, and concatenates them with $D$:
$$
S=[C_1^*;C_2^*;\ldots;C_m^*;D],
$$
with truncation or padding to the target length $L_{\text{target}}=131{,}072$ tokens [2510.02330].

A concise summary of the operational settings reported for the pipeline is given below.

| Component | Reported setting |
|---|---|
| High-entropy threshold | $TH=\mu_H+\alpha\cdot\sigma_H$, $\alpha=2.0$ |
| Query window | $w=16$ tokens |
| Retriever | jina-embeddings-v3 + Faiss |
| Verification threshold | $\epsilon=0.4$ |
| Target sequence length | $131{,}072$ tokens |

The construction is explicitly model-in-the-loop because the same base model both identifies uncertainty hotspots and validates whether a retrieved context produces measurable information gain [2510.02330]. A plausible implication is that the resulting training sequences are biased toward contexts the model can demonstrably exploit, rather than contexts that only appear topically compatible.

## 4. Dataset construction and training configuration

The reported source corpora are FineWebEdu and Cosmopedia, together comprising more than 1 billion documents. From these corpora, 100K documents are randomly sampled as root texts, while the full set is used for retrieval [2510.02330].

The resulting dataset contains 4 billion tokens of 128K-length sequences. Each sequence contains on average 46 verified long-range dependencies, and the mean Contextual Information Gain is reported as $AI=0.68$ per dependency [2510.02330]. These quantities are central to the paper’s claim that the constructed long-context samples are not generic concatenations but sequences with verified dependency structure.

The base model in the main experiments is Meta-Llama-3-8B with rotary-emb-base=$2e8$ and context window $128K$. Training is reported as 1 000 iterations, global batch size $4$ million tokens, learning rate $4\times 10^{-5}$ with cosine decay, bfloat16, and grad-clip $=1.0$ [2510.02330]. The baselines are Quest and NExtLong [2510.02330].

The paper evaluates on RULER before instruction tuning and on LongBench-v2 after instruction fine-tuning with UltraChat [2510.02330]. A later comparison in PolicyLong uses Qwen2.5-3B and reports EntropyLong’s absolute scores on RULER, HELMET, and LongBench-v2 in that separate setup [2604.07809]. Those results belong to the later comparative study rather than to EntropyLong’s original Meta-Llama-3-8B experiments.

## 5. Empirical performance

On RULER, EntropyLong outperforms both Quest and NExtLong across all tested context lengths from 8K to 128K in the reported Meta-Llama-3-8B setup [2510.02330]. The full table is as follows.

| Method | 8K | 16K | 32K | 64K | 128K | avg |
|---|---:|---:|---:|---:|---:|---:|
| Quest | 91.39 | 89.72 | 84.37 | 77.07 | 60.11 | 80.53 |
| NExtLong | 89.99 | 88.58 | 86.04 | 83.52 | 77.99 | 85.22 |
| EntropyLong | 91.50 | 90.11 | 88.95 | 85.04 | 81.26 | 87.37 |

The paper highlights that the largest gain occurs at 128K, where EntropyLong is ahead of NExtLong by $+3.27$ points [2510.02330]. This suggests that the advantage of verified dependencies becomes more pronounced as retrieval distance increases and long-context tasks become less tolerant of spurious context.

After instruction fine-tuning, EntropyLong also exceeds both baselines on LongBench-v2 [2510.02330].

| Method | Easy | Hard | Short | Medium | Long | Overall |
|---|---:|---:|---:|---:|---:|---:|
| Quest | 17.70 | 25.10 | 25.60 | 20.00 | 21.30 | 22.30 |
| NExtLong | 21.40 | 25.70 | 27.20 | 21.90 | 23.10 | 24.10 |
| EntropyLong | 25.50 | 28.90 | 30.00 | 23.70 | 31.50 | 27.60 |

The reported “Long” category gain is $+8.4$ points versus the best baseline [2510.02330]. Because LongBench-v2 is described as containing real-world long-context tasks, the result is presented as evidence that the verified-dependency pretraining signal transfers beyond synthetic retrieval settings [2510.02330].

PolicyLong later reports that, on Qwen2.5-3B, EntropyLong achieves 69.86 at 64K and 63.45 at 128K on RULER, 43.88 at 64K and 40.08 at 128K on HELMET, and 24.2 on Medium and 28.7 on Long in LongBench-v2 after SFT [2604.07809]. In that paper these numbers are used to show that EntropyLong improves over random concatenation or embedding-based heuristics, while also motivating on-policy refinement [2604.07809].

## 6. Ablations, analysis, and limitations

The ablation studies emphasize that entropy-based verification is not incidental to performance. In the comparison between full EntropyLong and EntropyLong-NoVerify, where the latter always accepts the top-1 retrieval without entropy checking, the full method improves average RULER from 85.82 to 87.37, with gains up to $+2.66$ at 32K [2510.02330]. This directly supports the claim that heuristic retrieval without verification is insufficient.

Threshold selection also matters. Varying the entropy threshold parameter $\alpha$ changes the number of high-entropy positions and average RULER; among the reported settings, $\alpha=2.0$ yields the best average score of 87.37 with 292 high-entropy tokens, whereas $\alpha=1.5$ yields 82.49 with 913 tokens and $\alpha=2.5$ yields 85.52 with 83 tokens [2510.02330]. Similarly, for the verification threshold $\epsilon$, the reported optimum is $\epsilon=0.4$, which produces 46 verified dependencies and average RULER 87.37; both lower and higher thresholds reduce performance [2510.02330].

The query window size $w$ is also ablated, with $w=16$ giving the best average RULER of 87.37 compared with 86.61 for $w=2$, 86.04 for $w=4$, and 86.85 for $w=8$ [2510.02330]. For concatenation strategy, random shuffling slightly exceeds sequential ordering, 87.37 versus 87.06 [2510.02330].

The short-text evaluation shows minimal degradation: Llama3-8B scores 61.77 on six short-text benchmarks, while +EntropyLong scores 61.83 [2510.02330]. The paper also reports attention-pattern analysis indicating that EntropyLong maintains higher attention to correct answers as context length grows, mitigating “lost-in-middle,” with relative gains up to $+44.4\%$ in middle positions [2510.02330].

The limitations stated in the paper are specific. Retrieval quality depends on external corpus coverage, so low recall may miss dependencies. The thresholds $\alpha$ and $\epsilon$ require tuning per corpus and model. Future directions proposed in the paper include adaptive thresholds, alternative uncertainty signals such as mutual information, and integration with retrieval-augmented generation [2510.02330].

A later critique in PolicyLong isolates two structural drawbacks of EntropyLong’s offline design: static screening and lack of hard negatives [2604.07809]. The off-policy argument is that data are constructed once using the initial base model, so the selected high-entropy positions may cease to be informative as the model improves. PolicyLong reports that when those original positions are reevaluated on a later checkpoint, their median entropy drops from 6.63 to 5.34, and it describes this as an off-policy gap [2604.07809]. This is a later interpretation rather than a claim in the original EntropyLong paper, but it has become part of the method’s subsequent reception.

## 7. Position within long-context data curation

EntropyLong is best understood as a shift from heuristic long-context synthesis toward uncertainty-verified synthesis. Its stated contribution is to replace generic concatenation or heuristic variants with a model-in-the-loop, information-theoretic approach that filters out spurious dependencies and guarantees measurable information gain [2510.02330]. In that sense, the method treats long-context training data curation as an empirical verification problem rather than a topical similarity problem.

The method’s broader significance lies in connecting data construction to the predictive state of a language model. High-entropy positions operationalize where the model lacks information; verified retrieved chunks operationalize which distant contexts actually resolve that deficit [2510.02330]. This suggests a general pattern for long-context supervision in which the unit of curation is not the document pair but the uncertainty reduction event.

Subsequent work has extended this line of thought. PolicyLong retains entropy computation, retrieval, and verification, but re-executes screening using the current model so that the training distribution tracks evolving capabilities, introducing what it calls an emergent self-curriculum and adding hard negative contexts derived from the same entropy landscape [2604.07809]. That development indicates that EntropyLong established a reusable framework for principled data curation even as later work identified the limitations of its one-shot offline form.

Source: https://www.emergentmind.com/topics/entropylong