---
title: Training Re-evaluation Curve (TREC)
url: https://www.emergentmind.com/topics/training-re-evaluation-curve-trec
type: topic
---

# Training Re-evaluation Curve (TREC)

Searching arXiv for the cited TREC-related papers to ground the article in current literature.
Search arXiv: "Training Re-evaluation Curve TREC 2509.25380"
Training Re-evaluation Curve (TREC) is a diagnostic for large language model training defined by retrospectively evaluating each original training batch with the final model parameters. In the formulation introduced in "Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs" [2509.25380], the curve is $L(t) := \mathcal{L}(\mathcal{B}_t; \theta_T)$ for training steps $t=1,\dots,T$, where $\mathcal{B}_t$ is the batch seen at step $t$ and $\theta_T$ is the fully trained parameter vector. The resulting time-indexed loss profile characterizes how well the final model retains training data as a function of when that data was encountered, and its low points are proposed as preferred insertion regions for scarce high-quality data. The acronym is distinct from the classic information retrieval usage of TREC as Text REtrieval Conference and TREC-style evaluation methodology [2205.04105].

## 1. Definition and interpretive framework

TREC is constructed after training finishes. Instead of recording the loss of $\mathcal{B}_t$ at the moment it was seen, the final model is frozen and re-evaluated on the training batches in their original order. The horizontal axis is training time, represented either by the discrete batch index $t$ or the normalized training fraction $t/T \in [0,1]$. The vertical axis is the retrospective loss of the final model on the batch seen at that time, typically cross-entropy loss, plotted either in absolute form or normalized/min-max scaled form when the emphasis is on shape rather than magnitude [2509.25380].

The central empirical observation is that the curve is typically not flat. It often exhibits a valley: a region of training whose batches receive lower final loss than batches from much earlier or later in training. Low TREC loss at time $t$ indicates that the final model is especially well aligned with batches from that period; high TREC loss indicates that those batches are less well retained, more forgotten, or less aligned with the final-state optimization geometry. A TREC minimum is therefore treated as the region where injected high-quality data should have the greatest impact.

The proposal is explicitly curricular rather than merely descriptive. The paper defines high-quality data operationally: given training distribution $D$, a distribution $D_h$ is high-quality for a task if replacing $D$ samples with $D_h$ samples improves task performance. Under that definition, code-heavy or math-heavy mixtures are high-quality only relative to the corresponding target validation tasks. The framework also assumes homogeneous i.i.d. training data for defining and analyzing TREC shape, so that the curve reflects training dynamics rather than content heterogeneity [2509.25380].

A key conceptual distinction is between training loss and re-evaluation loss. The standard heuristic that high-quality data should be placed at the very end relies on low instantaneous training loss late in training, but the paper reports that TREC can rise near the end under decay-to-zero or step-drop schedules. This means late placement can be suboptimal even when training loss is low. This suggests that TREC is intended to measure retained usefulness under the final model rather than instantaneous ease of fitting.

## 2. Mathematical formulation and optimizer-level explanation

The optimizer-level account begins with AdamW:
$$
\theta_t = (1 - \eta\lambda)\theta_{t-1} - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon},
$$
where $\eta$ is learning rate, $\lambda$ is weight decay, $\hat m_t$ and $\hat v_t$ are bias-corrected first- and second-moment estimates, and $\epsilon$ is the numerical stabilizer. Following the paper’s formulation, this can be interpreted as an exponential moving average of updates,
$$
y_t = (1-\alpha)y_{t-1} + \alpha x_t,
$$
with $y_t=\theta_t$, $\alpha=\eta\lambda$, and
$$
x_t = -\frac{1}{\lambda}\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}.
$$
This yields an optimizer memory timescale of roughly $\frac{1}{\eta\lambda}$ and, for single-epoch LLM pretraining, the normalized AdamW timescale
$$
\tau = \frac{1}{\eta\lambda T} = \frac{B}{\eta\lambda D},
$$
where $B$ is batch size in tokens, $D$ is total number of training tokens, and $T=D/B$ is the number of optimization steps [2509.25380].

For time-varying learning rates, the smoothing parameter becomes $\alpha_t=\eta_t\lambda$, and the unrolled EMA for the final parameters induces stepwise contribution coefficients
$$
c_i = \eta_i \lambda \prod_{j=i+1}^{T} (1 - \eta_j \lambda).
$$
The continuous function $c(t)$ over normalized training fraction is interpreted as the retrospective weighting of updates from each training time into the final model. These coefficients are not themselves the TREC, but they are the first major ingredient in its prediction.

The predictive approximation for normalized TREC shape is
$$
\hat{L}(t) = 1 - c(t)^p \cdot t^m,
$$
where $c(t)$ is the normalized AdamW EMA coefficient for the final weights, $p$ controls the strength of the EMA contribution, and $m$ is a training-fraction exponent capturing drift or decay in the usefulness of earlier gradients. The experiments fix $p=0.5$ and focus on fitting or predicting $m$ [2509.25380].

The justification for the extra $t^m$ term comes from a simplified quadratic analysis. If batch-specific local minimizers drift as training progresses, then EMA weighting alone overstates the influence of early updates. In the appendix, the paper writes the retrospective loss using the final parameters and shows that low TREC loss requires both high optimizer weight $c_t$ and good alignment between the update at time $t$ and the final batch-specific minimizer. This is why the curve is governed by both optimizer persistence and minimizer drift rather than by learning-rate schedule alone.

The paper also derives a small-weight-decay limit in which $c_i$ becomes proportional to $\eta_i\lambda$, so the TREC shape approaches a learning-rate-shaped limit as $\lambda \to 0$. This provides the analytical basis for the reported transition toward learning-rate-dominated behavior at very small weight decay [2509.25380].

## 3. Empirical structure across model size, schedule, and scaling regime

The empirical study spans models from **111M to 3.9B** parameters, **578 TREC computations and 41 placement-trained models**, and datasets from **20 to 1280 TPP**, using GPT2-style LLMs with AdamW, context lengths **2048 or 8192**, and optimizer settings $\beta_1=0.9$, $\beta_2=0.95$ [2509.25380]. Across this range, the paper reports that TREC shape is largely controlled by the AdamW timescale $\tau$ for a fixed learning-rate schedule.

For **610M** models at **80 TPP**, sweeping learning rate $\eta$, weight decay $\lambda$, and batch size $B$ while matching
$$
\tau = \frac{B}{\eta\lambda D}
$$
produces very similar normalized TREC shapes. As $\tau$ increases, the TREC valley shifts earlier in training. When models are trained at fixed TPP and similar $\tau$, TRECs align from **111M to 3.3B**, despite about **1000× difference in training FLOPs**. At larger TPP, the valley shifts slightly later and the absolute depth of the TREC drop shrinks. The authors interpret this as reduced memorization or reduced plasticity in overtrained regimes.

The paper further reports that normalized TREC shape is stable across batch sizes up to around critical batch size but diverges at very large batches far beyond it. Sweeping $\beta_1,\beta_2$ shows that normalized TREC shape is mostly insensitive to momentum settings, with the weight-decay timescale dominating [2509.25380].

Prediction accuracy is evaluated at the level of shape rather than absolute scale. The exponent $m$ is fit using **111M** models and transferred across scales. The reported fit $R^2$ for predicted $m^\star$ is about **99% at 111M**, declining to **77% at 3.3B**, while predicted TREC shape remains highly accurate, with Pearson correlation around **98%** across scales. Schedule transfer is also reported: using a decay-to-zero fit on $10\times$ decay runs gives $\rho = 94.2\%$, and schedule-specific fitting improves this to $\rho = 97.8\%$. This suggests that exact estimation of $m$ is less critical than preservation of the overall curve geometry.

## 4. Curriculum placement and continual pre-training

The core curricular claim is direct: place high-quality data at low points of the TREC [2509.25380]. In the paper’s mid-training placement study, **610M** models are trained on **50B tokens (82 TPP)** consisting of **45B “general blend” tokens** and **5B “code blend” high-quality tokens**, with code validation as the target. The code blend upweights GitHub and StackExchange relative to the general blend. For each learning-rate schedule, **10 models** are trained, each placing the **5B** high-quality code tokens into a different **10%** segment of training.

Across tested schedules—**step-drop schedules**, **linear decay-to-zero**, and **linear decay-to-\(10^{-x}\) variants**—the best validation performance occurs when high-quality data is placed at the **lowest point of the TREC**. The paper also reports that optimal placement beats uniform mixing of the same high-quality data throughout training. A complementary ablation evaluates general-blend validation after replacing some general-blend data with code-blend data; the largest degradation occurs when code displaces general-blend data at the TREC minimum, implying that those positions are also where the baseline data was most valuable.

The framework is also used to reinterpret published training recipes. The paper states that **Llama-3 405B** likely saw no gain from late annealing on GSM8k/MATH because the final **40M tokens** over batch size **16M** amount to only about **3 optimizer steps** at vanishing learning rate. It further reports that the best onset for the high-quality phase in **Feng et al.** aligns with the predicted TREC bottom, that **OLMo-2 13B** places its high-quality blend during the final **5.7%** in a way that aligns with a narrow TREC dip, and that **Pangu-Ultra 135B** aligns reasoning and annealing phases well with the predicted TREC valley, though perhaps slightly too late [2509.25380].

The largest demonstration concerns continual pre-training. A **3.9B** model is pretrained on **900B tokens** (about **234 TPP**) and then continued for an extra **18B tokens** (about **5 TPP**) during continual pre-training. A **1.3B-token high-quality math blend** that upweights OpenWebMath and UltraTextBooks is inserted either halfway through continual pre-training, where TREC is low, or at the very end, where TREC returns to baseline. Placing the high-quality data at the TREC minimum outperforms end placement across all tested continual-pre-training learning rates. At the same time, the paper reports an important caveat: across different continual-pre-training learning rates, deeper TREC valleys do not always imply better validation, so TREC is reliable within a schedule but not always for choosing among schedules [2509.25380].

## 5. Practical use and methodological limits

The paper presents two operational modes [2509.25380]. In retrospective analysis mode, one saves or reconstructs the sequence of training batches or coarse training segments, evaluates the final model $\theta_T$ on those batches in original order, plots $L(t)=\mathcal L(\mathcal B_t;\theta_T)$ against training fraction, identifies valleys or minima, and uses those regions for data placement in a retraining run. In predictive curriculum-design mode, one specifies the planned learning-rate schedule $\eta_t$, weight decay $\lambda$, total training tokens $D$, batch size $B$, total steps $T=D/B$, tokens-per-parameter, and a schedule-specific fit or estimate for $m$; computes
$$
c_i = \eta_i \lambda \prod_{j=i+1}^{T} (1 - \eta_j \lambda);
$$
forms
$$
\hat L(t)=1-c(t)^{0.5}t^m;
$$
and places scarce high-quality data into windows around the minima of the predicted curve.

Several methodological constraints are explicit. Observed TREC is optimizer-agnostic, but the predictor is **AdamW-specific**. The predictive model targets **normalized shape**, not absolute retrospective cross-entropy values. The main evidence is for **20 TPP and above**, i.e. mostly compute-optimal and overtrained regimes rather than very data-scarce undertrained settings. The formal derivation assumes **i.i.d. homogeneous batches**, although the paper reports that TREC-informed placement still transfers to heterogeneous inserted datasets. It also notes that observed TRECs require explicit re-evaluation on training data and can therefore be costly, whereas predicted TRECs require only schedule arithmetic plus a pre-fit exponent model.

A further limitation concerns schedule comparison. The paper explicitly distinguishes **intra-schedule placement**, where TREC is reliable, from **cross-schedule selection**, where absolute TREC values may fail to predict validation performance. This suggests that TREC is best viewed as a placement diagnostic conditioned on a training recipe rather than as a universal objective for choosing the recipe itself.

## 6. Acronym ambiguity and related usages

The term TREC is polysemous in arXiv literature. In "Re-thinking Knowledge Graph Completion Evaluation from an Information Retrieval Perspective" [2205.04105], TREC refers to the classic information retrieval paradigm of **Text REtrieval Conference**, **TREC-style pooling**, and **judgment-set construction** rather than to a training diagnostic. That paper imports TREC methodology into knowledge graph completion by treating triple questions as IR queries and recommends **TREC-style pooling** and **IR-like macro metrics** to mitigate incomplete judgments. Its use of the acronym is therefore unrelated to Training Re-evaluation Curve.

A separate usage appears in "Generative Relevance Feedback and Convergence of Adaptive Re-Ranking: University of Glasgow Terrier Team at TREC DL 2023" [2405.01122], where TREC denotes the **TREC 2023 Deep Learning Track**. That work studies adaptive re-ranking as a budgeted iterative process and reports effectiveness and ranking-correlation curves as re-ranking budget increases, but it does not define a training re-evaluation curve. Its convergence analysis is relevant only in the looser sense of measuring system behavior as additional evaluation effort is spent.

Within this landscape, Training Re-evaluation Curve designates a specific optimizer-aware diagnostic for LLM training. Its novelty lies in combining a retrospective loss curve over original training batches, the claim that TREC minima indicate optimal insertion windows for scarce high-quality data, and a predictive approximation derived from AdamW EMA coefficients and a drift term [2509.25380]. This suggests that careful disambiguation is necessary whenever the acronym TREC appears in machine learning and information retrieval research.

Source: https://www.emergentmind.com/topics/training-re-evaluation-curve-trec