Long-Context Continual Pre-training (LCCP)

• LCCP is a specialized adaptation that extends LLM context windows via token-prediction training on long documents while preserving original performance.
• It employs multi-stage pre-training pipelines with advanced positional scaling and synthetic data augmentation to robustly model long-term dependencies.
• It also integrates efficient systems techniques like chunking and optimized attention mechanisms to overcome compute challenges and maintain short-context abilities.

Long-Context Continual Pre-training (LCCP) is a specialized adaptation phase in which a pre-trained LLM’s context window is extended by continuing token-prediction training on long-sequence documents, for example from 32K to 64K tokens, with the goal of robust long-dependency modeling such as retrieval, comprehension, and copy-paste without catastrophic forgetting of the model’s original capabilities (Liang et al., 3 Apr 2026). In recent work, LCCP has been used to extend models from short contexts such as 4K or 8K to 128K, 200K, 512K, 1M, 2M, and 4M tokens through combinations of continued pretraining, positional-scaling adjustments, data engineering, and post-extension instruction tuning (Gao et al., 2024, Zhao et al., 2024, Xu et al., 8 Apr 2025).

1. Definition, scope, and scale regimes

The central problem addressed by LCCP is that Transformer-based LMs pretrained on short contexts degrade sharply once asked to attend to much longer inputs. The literature describes three recurrent constraints: very long documents are rare in standard web-scale corpora, naively increasing sequence length incurs high compute and memory costs, and off-the-shelf positional encodings do not extrapolate well. Within this framing, LCCP is not a replacement for base pretraining, but a distinct adaptation stage aimed at changing the model’s usable context window while preserving its prior competence (Gao et al., 2024).

A major organizing distinction in the literature is between small-scale and industrial-grade LCCP. Small-scale setups are described as typically using 1–10 B-parameter models and $O(1\text{–}20\ \mathrm{B})$ LCCP tokens, with reported saturations often around tens of billions of tokens. Industrial-grade setups are described as using models with tens of billions of parameters or Sparse MoE variants and $O(100\text{–}200\ \mathrm{B})$ LCCP tokens; one study tracks Hunyuan-A13B, with 80 B total and 13 B active parameters, across a 200 B-token LCCP trajectory (Liang et al., 3 Apr 2026). This scale distinction matters because the same source argues that naively transplanting small-scale recipes to production models risks “deceptive convergence” and insufficient adaptation.

The range of target windows in published recipes is correspondingly broad. LongSkywork describes a 4 K-context LLM extended to 200,000 tokens, ProLong-8B reports effective processing up to 512K tokens, and UltraLong-8B describes one-step extension from 128K to 1M, 2M, and 4M tokens (Zhao et al., 2024, Gao et al., 2024, Xu et al., 8 Apr 2025). This suggests that “long context” in LCCP is not a single regime, but a family of regimes whose data budgets, systems constraints, and convergence behavior differ materially with model scale and target length.

2. Canonical training pipelines and positional scaling

Most LCCP recipes retain the standard causal language-modeling objective during the continual stage. One formulation writes

$$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$

with long sequences built by concatenating documents and continuing pretraining at the target window (Xu et al., 8 Apr 2025). What varies across papers is the staging strategy. ProLong uses a two-stage continual pretraining pipeline on top of Llama-3-8B-Instruct: 20 B tokens at 64 K, then 20 B tokens at 512 K, with RoPE base frequency $\alpha=8\times 10^6$ for 64 K and $\alpha=1.28\times 10^8$ for 512 K, followed by 1 B tokens of UltraChat instruction tuning (Gao et al., 2024). LongSkywork instead presents a four-stage logic of base pretraining, standard SFT at 4 K, long-context continual pre-training at 64 K, and long-context SFT at 100 K, with the claim that a long-context SFT stage following the standard SFT stage is the critical element in enhancing long-context processing capability (Zhao et al., 2024). UltraLong describes a two-stage pipeline in which Stage 1 directly extends a 128K instruct model to 1M, 2M, or 4M with continued pretraining on about 1 billion tokens and Stage 2 applies short-context instruction tuning on 100 K examples (Xu et al., 8 Apr 2025).

Positional adaptation is a second invariant. ProLong retains RoPE but rescales its base frequency so that

$\theta_k = \alpha^{-2k/d},$

with aggressive $\alpha$ values selected for the new maximum context; smaller $\alpha$ values, including the default $5\times 10^5$, are reported to fail catastrophically or to be suboptimal (Gao et al., 2024). LongSkywork changes the RoPE base from $10{,}000$ to $O(100\text{–}200\ \mathrm{B})$0 and attributes this to NTK-aware scaling, while UltraLong uses YaRN scaling with multiplicative factors $O(100\text{–}200\ \mathrm{B})$1 for 1M, 2M, and 4M contexts respectively (Zhao et al., 2024, Xu et al., 8 Apr 2025).

The post-extension SFT stage is one of the clearest points of divergence. ProLong reports that “using only short instruction datasets yields strong performance on long-context tasks,” and UltraLong states that “no long-context instruction data is needed to preserve reasoning.” LongSkywork, by contrast, identifies long-context SFT as critical and uses a $O(100\text{–}200\ \mathrm{B})$2 normal SFT and $O(100\text{–}200\ \mathrm{B})$3 long-context SFT mixture at a 100 K window (Gao et al., 2024, Xu et al., 8 Apr 2025, Zhao et al., 2024). This suggests that the role of long-context SFT is recipe-dependent rather than universally settled.

3. Data mixture, selection, and synthetic augmentation

Because natural long-context data are scarce, LCCP research devotes substantial attention to corpus construction. ProLong reports that code repositories and books are excellent sources of long data, but that they must be combined with high-quality short-context data to preserve short-context abilities. Its final mixture is $O(100\text{–}200\ \mathrm{B})$4 long documents—$O(100\text{–}200\ \mathrm{B})$5 code repos, $O(100\text{–}200\ \mathrm{B})$6 books, $O(100\text{–}200\ \mathrm{B})$7 high-quality textbooks—and $O(100\text{–}200\ \mathrm{B})$8 “ShortMix,” with FineWeb / FineWeb-Edu comprising $O(100\text{–}200\ \mathrm{B})$9 of ShortMix and Wikipedia, Tulu-v2, StackExchange, ArXiv, and OpenWebMath each comprising $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$0. A $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$1 long / $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$2 short ratio is reported to maximize HELMET downstream scores after SFT, while pure long data hurts RAG and re-ranking once SFT is added (Gao et al., 2024).

Another line of work treats long-context quality as a data-selection problem. LADM introduces Attention-based Dependency Measurement, which assigns each 32 K sample a Contextual Dependency Score by partitioning it into 128-token spans, computing pairwise Focus Scores from attention, aggregating them into an Aggregated Focus Score, and finally ranking samples to select the top 1 B tokens while keeping original domain proportions. In the reported experiments, LADM reaches up to $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$3 Needle-in-the-Haystack retrieval and improves the LongBench average to $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$4, compared with $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$5 for ProLong-style delta-perplexity filtering and $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$6 for random sampling (Chen et al., 4 Mar 2025).

Synthetic construction is equally prominent. LongSkywork’s chunk-interleaved pretraining forms long synthetic sequences by splitting multiple short documents into chunks and interleaving them; its synthetic long-context SFT data (“SynL”) are generated as large tables and converted into information retrieval, chain-of-thought reasoning, and global comprehension tasks (Zhao et al., 2024). Untie the Knots (UtK) is a pure data-augmentation layer: documents are chunked, shuffled across a minibatch, marked with chunk IDs and knot markers, and finished with a backtrace prompt requiring the model to reproduce the original chunk-ID sequence. UtK reports $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$7 average accuracy on RULER at 128 K for Qwen2-7B and $$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$8 for Qwen2-72B after 20 B tokens, and states that ABF’s 20 B-token performance can be reached after only about 1 B UtK tokens (Tian et al., 2024). Taken together, these results indicate that LCCP data engineering has moved well beyond simple concatenation of naturally long documents.

4. Evaluation, learning dynamics, and saturation

Evaluation is a central controversy in LCCP. ProLong argues that perplexity or simple Needle-in-a-Haystack tests do not correlate reliably with real-world uses and therefore evaluates after SFT on HELMET, a suite of realistic long-context tasks including JSON KV Recall, RAG, re-ranking, many-shot ICL, book-length QA, and summarization (Gao et al., 2024). By contrast, the Hunyuan-A13B study proposes a hierarchical framework that explicitly combines behavioral, probabilistic, and mechanistic views of convergence. At the behavioral level, lightweight SFT probes are run on 0.25 B tokens spanning general, reasoning, and long-context tasks, then evaluated on RULER, MRCR, and LongBio at uniformly sampled sequence lengths between 32K and 64K. At the probabilistic level, the study uses perplexity,

$$\mathcal{L}_{LM}(\theta) = -\sum_{t=1}^N \log p_\theta(x_t \mid x_{<t}),$$9

and converts Needle-in-a-Haystack into a continuous PPL-based measure by computing perplexity only over answer tokens in “context + question + answer” sequences. At the mechanistic level, it tracks retrieval heads using the retrieval score

$\alpha=8\times 10^6$0

with threshold $\alpha=8\times 10^6$1 for identifying a retrieval head (Liang et al., 3 Apr 2026).

The resulting learning dynamics differ sharply from traditional binary retrieval evaluation. In Hunyuan-A13B, traditional NIAH score saturates early, around 20 B tokens and reaches $\alpha=8\times 10^6$2 at 50 B, whereas continuous NIAH PPL declines steadily and only plateaus near 150 B. The same study reports that RULER Pass@3 climbs from $\alpha=8\times 10^6$3 to $\alpha=8\times 10^6$4, saturating only after about 100 B tokens, while probabilistic and mechanistic indicators continue improving until about 150 B. Pearson correlations with downstream SFT are also stronger for NIAH PPL than for NIAH Score: the average correlation is $\alpha=8\times 10^6$5 for NIAH PPL versus $\alpha=8\times 10^6$6 for NIAH Score (Liang et al., 3 Apr 2026). The paper names this discrepancy “deceptive saturation” versus “intrinsic saturation.”

Mechanistic monitoring extends this argument. The number of retrieval heads increases from about 10 to about 40 over training, the average retrieval score of top heads rises monotonically, and the top-30 retrieval heads overlap by more than $\alpha=8\times 10^6$7 across checkpoints after the first 20 B tokens, with Spearman $\alpha=8\times 10^6$8. The reported average correlation with downstream SFT is $\alpha=8\times 10^6$9 for the number of retrieval heads and $\alpha=1.28\times 10^8$0 for the average retrieval score (Liang et al., 3 Apr 2026). These results imply that no single benchmark exhausts LCCP evaluation: downstream task suites after SFT expose task transfer, while continuous PPL and retrieval-head trajectories expose intrinsic convergence and training stability.

5. Systems methods for efficient LCCP

LCCP is constrained not only by data and evaluation, but also by the quadratic time and memory of dense attention and by the variable-length distribution of long corpora. ChunkFlow addresses the distributed-training side of this problem by reorganizing input sequences into uniformly sized chunks, splitting long sequences and packing short ones, then using state-aware chunk scheduling so that peak memory is bounded by

$\alpha=1.28\times 10^8$1

rather than the naïve $\alpha=1.28\times 10^8$2. In reported long-context fine-tuning experiments, ChunkFlow is up to $\alpha=1.28\times 10^8$3 faster than Megatron-LM, raises average GPU utilization from $\alpha=1.28\times 10^8$4 to $\alpha=1.28\times 10^8$5, and in a small continual-pretraining pilot of 2 B tokens with an 8 K$\alpha=1.28\times 10^8$632 K$\alpha=1.28\times 10^8$7128 K curriculum yields a $\alpha=1.28\times 10^8$8 end-to-end speedup over naïve retraining from scratch (Yuan et al., 4 Mar 2025).

Seamless Packing attacks a different failure mode of continual pre-training: context discontinuity and truncation from concatenate-and-truncate pipelines. Its first stage uses a sliding window with controlled overlap $\alpha=1.28\times 10^8$9, and its second stage uses First-Fit-Decreasing bin packing with slight over-capacity and dropping. The paper reports $\theta_k = \alpha^{-2k/d},$0 as optimal across multiple distributions, perplexity improvements such as PubMed $\theta_k = \alpha^{-2k/d},$1 for CT, BFD, and SP respectively, average downstream accuracy gain of $\theta_k = \alpha^{-2k/d},$2 versus CT, and padding-token reductions of up to $\theta_k = \alpha^{-2k/d},$3 on BBC News (Yin et al., 28 May 2025). Although these experiments are framed as continual pre-training more generally, the motivation is directly relevant to LCCP because boundary fragmentation becomes more damaging as the target window grows.

At the attention-kernel level, Lighthouse Attention proposes a training-only hierarchical selection-based wrapper around ordinary SDPA. It compresses $\theta_k = \alpha^{-2k/d},$4, $\theta_k = \alpha^{-2k/d},$5, and $\theta_k = \alpha^{-2k/d},$6 symmetrically into a pyramid, applies parameter-free scoring and Top-$\theta_k = \alpha^{-2k/d},$7 selection, runs dense attention on the gathered sub-sequence, and then scatters outputs back to full length while preserving causality. The method is used for most of the token budget and then removed during a short dense-attention recovery phase. In preliminary 530 M-parameter pretraining at sequence length 98,304, the default 10k + 6k Lighthouse-to-SDPA schedule reports stage-1 throughput of $\theta_k = \alpha^{-2k/d},$8 K tok/s/GPU versus $\theta_k = \alpha^{-2k/d},$9 K tok/s/GPU for dense SDPA, end-to-end wall-clock reduction from $\alpha$0 h to $\alpha$1 h, and a final loss of $\alpha$2 versus $\alpha$3 for the dense baseline (Peng et al., 7 May 2026). This places LCCP systems research on two fronts at once: packing and scheduling for variable-length corpora, and subquadratic training-time substitutes for dense attention.

6. Retention of original capabilities, operational guidance, and unresolved questions

A recurrent concern in LCCP is short-text degradation. LongReD attributes this degradation to distribution drift in hidden states and attention scores and to catastrophic forgetting during continual pre-training. Its remedy combines long-text continual pre-training, short-text restoration distillation on selected layers, and short-to-long distillation using skipped positional indices. In the reported results, LongReD-C on Llama-3-8B extended to 32K with ABF reaches a short-text average of $\alpha$4 versus $\alpha$5 for long-text continual pre-training alone and retains $\alpha$6 of original performance versus $\alpha$7 for the CPT baseline; on Mistral-7B 128K, the short-text average is $\alpha$8 versus $\alpha$9 for CPT (Dong et al., 11 Feb 2025). These findings establish that long-context adaptation and short-context preservation are not automatically aligned.

The practical recommendations in the industrial-scale Hunyuan-A13B study are correspondingly conservative. It recommends LCCP budgets of at least 150 B tokens for LLMs with tens of billions of parameters, warns against sole reliance on binary retrieval benchmarks because of deceptive saturation, recommends continuous PPL measures and standard PPL on held-out long-context corpora, and proposes tracking the number of retrieval heads and their average scores every few billion tokens. The same source describes a dashboard logic in which a deceptive plateau is indicated by flat NIAH score and falling PPL, while true convergence is indicated by flat PPL and stable retrieval metrics; it further recommends extending context windows to 128K or 256K only when retrieval heads and PPL continue to improve with additional tokens (Liang et al., 3 Apr 2026).

The field nevertheless remains heterogeneous. Smaller-scale or specialized recipes report effective long-context adaptation with 1 B selected tokens in LADM, 20 B tokens in UtK, 40 B continual tokens plus 1 B SFT in ProLong, or 200–500 long-context pretraining steps in LongSkywork (Chen et al., 4 Mar 2025, Tian et al., 2024, Gao et al., 2024, Zhao et al., 2024). Industrial-scale evidence, however, argues that dozens of billions of tokens are insufficient for industrial-grade models and that true plateau may not emerge until beyond 150 B tokens (Liang et al., 3 Apr 2026). A plausible implication is that LCCP has no scale-invariant recipe: the relevant token budget, evaluation protocol, and retention strategy depend jointly on model size, target window, positional-scaling method, and the degree to which downstream performance rather than intrinsic convergence is treated as the stopping criterion.