- The paper introduces twelve 1.3B-parameter DatedGPT models trained from scratch on roughly 100B temporally filtered tokens per year from 2013–2024, plus instruction-tuned and finance-specific variants.
- Perplexity probing shows each model’s exposure to quarterly company-news headlines rises sharply after its designated cutoff, supporting effective prevention of post-cutoff parametric leakage.
- DatedGPT-Instruct models achieve stable general benchmark performance across cutoff years and substantially outperform similar baselines on IFEval, although finance forecasting gains remain untested and crawl-date and teacher-model leakage remain limitations.
DatedGPT addresses a methodological problem that is specific to the use of LLMs in temporal prediction tasks: lookahead bias. When an LLM pretrained on internet-scale corpora is asked to forecast, for example, stock returns from news headlines, it may have already encountered the realized outcome during pretraining. Any apparent predictive success then conflates memorization with genuine in-context reasoning, invalidating claims of forecasting ability. The paper introduces DatedGPT, a family of twelve 1.3B-parameter LLMs trained from scratch on temporally partitioned data with strict annual cutoffs spanning 2013 to 2024, together with instruction-tuned variants and finance-specific fine-tuning data designed under the same temporal constraints (2603.11838).
Motivation and positioning
The motivating example is concrete: an LLM that "predicts" the 8.8% S&P 500 crash of September 29, 2008 following Congress's rejection of the bailout plan may simply be recalling the event rather than reasoning over the input. The authors argue that uncontaminated evaluation requires both that the input information set be limited to what was observable at prediction time and that the model itself never have been exposed to future outcomes — precisely the condition that standard pretrained LLMs are most likely to breach.
Prior work offers only partial remedies. Prompting strategies such as masking firm identifiers or anonymizing temporal cues leave leakage at the parametric level unresolved (2603.11838), and econometric tests of lookahead bias suffer from limited statistical power when the out-of-sample window is short for widely used models. The closest related efforts are StoriesLM, a series of 110M-parameter BERT models with cutoffs from 1900 to 1963, and the concurrent ChronoGPT, which uses continual pretraining with incremental yearly updates on fewer than 10B tokens per year. DatedGPT distinguishes itself by training each model independently from scratch on roughly 100B tokens per year at 1.3B parameters, and by providing instruction-tuned variants with curated finance-domain data.
Time-aware data curation
The pretraining corpus is derived from FineWeb-Edu, filtered by Common Crawl timestamp rounded to the year. The authors are explicit about a limitation here: crawl timestamps do not reflect true document creation dates, so a page crawled in 2015 may have been authored decades earlier. Filtering by crawl date therefore cannot guarantee that a model sees only content produced before its cutoff; it guarantees only that no data crawled after the cutoff enters training. This is a defensible but imperfect proxy, and the paper concedes it plainly.
For instruction tuning, the pipeline handles two categories of data. General-domain instruction datasets (OpenHermes-2.5, Tulu-3 persona-based instructions, Coconot) are filtered by prompting Llama-3.3-70B-Instruct to classify each sample as time-sensitive or not; removal rates range from about 19% for OpenHermes-2.5 to 39.15% for Coconot. The high Coconot removal rate is attributed largely to copyright-related queries that presuppose knowledge of media released at specific dates — an instructive observation that even safety-oriented instruction data can constitute implicit temporal leakage.
The finance-specific instruction data targets two tasks: predicting stock return direction from news headlines and forecasting capital expenditure from earnings call transcripts. Both sources carry precise publication timestamps, which the authors exploit to anchor each training instance to its actual information availability date. Approximately 6,000 examples per year per task are generated using Llama-3.3-70B-Instruct as teacher. A caveat worth noting: because the target responses are LLM-generated rather than ground-truth outcomes, these datasets teach reasoning patterns distilled from a teacher model whose own outputs may embed post-hoc knowledge; the paper does not address whether the teacher's responses themselves leak future information beyond the anchor date.
Training setup
Each of the twelve base models uses a Llama-style 1.3B architecture (24 layers, RoPE, SwiGLU, RMSNorm, sequence length 2048) trained for 25,000 iterations (~100B tokens) following the LLM.c GPT-2 reproduction recipe. Pretraining costs approximately 2,000 A100 GPU-hours per model. Training curves are reported as smooth, without the loss spikes seen in larger runs. Instruction tuning mixes general-domain and year-specific data for three epochs, totaling roughly 1% of the pretraining token budget.
Evaluation results
Zero-shot results for representative instruction-tuned models against scale-matched baselines:
| Model |
ARC-C |
HellaSwag |
PIQA |
IFEval |
MMLU |
TruthfulQA |
Avg. |
| SmolLM-1.7B-Instruct |
34.9 |
56.1 |
72.4 |
15.5 |
27.1 |
27.7 |
41.1 |
| GPT2-XL |
28.4 |
50.8 |
70.5 |
15.0 |
25.3 |
22.4 |
37.6 |
| TinyLlama-1.1B |
30.5 |
55.0 |
72.5 |
14.8 |
24.8 |
26.0 |
38.2 |
| OPT-1.3B |
27.8 |
53.7 |
70.9 |
17.6 |
25.2 |
23.8 |
38.6 |
| Pythia-1B |
27.1 |
47.1 |
69.3 |
16.8 |
23.1 |
23.6 |
36.6 |
| DatedGPT-Instruct-2013 |
33.6 |
47.6 |
66.4 |
34.2 |
25.3 |
25.1 |
40.1 |
| DatedGPT-Instruct-2024 |
34.7 |
53.2 |
70.5 |
35.3 |
24.3 |
28.6 |
42.7 |
Two findings stand out. First, the IFEval gains are substantial: DatedGPT-Instruct scores 30.7–35.9 across all years versus 14.8–17.6 for every baseline, indicating that the instruction-tuning stage is highly effective relative to compute cost. Second, performance is remarkably stable across cutoff years (average 40.1–42.7), supporting the claim that models without access to recent data can still develop strong general language understanding. The full series confirms this consistency, though MMLU remains modest (~24–26), typical of models at this scale and token budget.
For memorization verification, the authors adopt perplexity-based probing on quarterly public-company news headlines. For each model, perplexity rises sharply after the model's cutoff year — e.g., DatedGPT-base-2020 shows a clear reversal around Q4 2020 — and this pattern holds consistently across all twelve cutoffs. This is the paper's central empirical validation: parametric knowledge is effectively bounded by the designated cutoff, which is exactly the property needed for uncontaminated backtesting. An interactive web demo allows side-by-side comparison of responses across cutoff years.
Limitations and open questions
Several limitations bear directly on interpretation. The crawl-date proxy means contamination by older-than-crawl-date content is possible in principle, though this affects all years symmetrically and does not threaten the post-cutoff exclusion property. The finance instruction data derives its supervision from a teacher LLM, introducing a potential secondary leakage channel the paper does not quantify. Benchmark evaluations use standard suites whose test items may themselves postdate some cutoffs, complicating strict comparability across the series. Finally, while perplexity probing demonstrates bounded exposure to news text patterns, it does not directly demonstrate improved out-of-sample forecasting validity in financial tasks; whether DatedGPT yields economically significant predictive gains free of bias is left as an open empirical question.
Conclusion
DatedGPT provides the largest yearly-cutoff model series to date — twelve 1.3B-parameter models trained independently from scratch on ~100B tokens each, spanning 2013–2024 — with instruction-tuned variants and finance-specific data respecting the same temporal boundaries. Perplexity probing confirms effective knowledge bounding at each cutoff, and benchmark performance is competitive with similarly sized models despite the constrained data. The release of checkpoints, data, and tooling makes it a practical resource for lookahead-bias-free evaluation of LLM forecasting in finance and other time-sensitive domains.