Temporally Ordered Pre-training

• Temporally ordered pre-training is a method that uses inherent sequence and timestamps as primary supervision to capture temporal dynamics.
• It replaces static or shuffled training with objectives that preserve event order and chronological context, enhancing performance in tasks like semantic change detection and localization.
• Applications span language, video, time-series, and autonomous driving, with tailored methods like time-aware attention and contrastive learning addressing specific temporal challenges.

Searching arXiv for recent and foundational papers on temporally ordered pre-training across language, video, time series, and embodied learning. Temporally ordered pre-training denotes a family of pre-training strategies in which temporal structure is treated as a primary supervision signal rather than an incidental property of the corpus. Across language modeling, video-language learning, time-series modeling, autonomous driving, and embodied learning, these strategies replace temporally indiscriminate training with objectives that preserve timestamps, sequence order, temporal overlap, foreground–background structure, or chronological data presentation. The common premise is that standard shuffled or static pre-training can obscure diachronic regularities, event progression, and time-sensitive factual knowledge, whereas temporally structured training can yield representations that are more temporally grounded, more precise about when facts or events hold, and better aligned with downstream tasks such as semantic change detection, retrieval, localization, entity tracking, forecasting, and imitation learning (Rosin et al., 2022, Hippolyte et al., 21 May 2026, Drinkall et al., 2024).

1. Temporal structure as a supervision signal

A central motivation for temporally ordered pre-training is that conventional pre-training pipelines are often temporally blind or temporally indiscriminate. In language modeling, training corpora may contain publication times or revision histories, yet ordinary pretrained LLMs are trained on the textual data alone, treating language as if usage were static. In LLM pre-training, global shuffling of web corpora destroys temporal structure and can bias models toward older, repeated facts. In video-language and video representation learning, image-language style objectives often align a whole video with text while underusing the defining property of video, namely temporal structure. In procedural text, recipes and other instructions are not a bag of sentences but ordered steps whose dependencies are directly relevant to entity tracking. In time series, random masking and series-wise contrastive similarity can distort or neglect inherent temporal correlations (Rosin et al., 2022, Hippolyte et al., 21 May 2026, Ye et al., 2022, Nandy et al., 2024, Dong et al., 2024).

The literature uses several distinct notions of temporality. One line uses explicit timestamps or point-in-time corpora, so that a model can condition on when a sentence was written or restrict itself to what could have been known by a cutoff date. A second line uses temporal order within sequences, as in recipe steps, event chains, or frame sequences, and converts that order into a pretext task such as permutation prediction, ranking, or denoising reconstruction. A third line uses temporal overlap and temporal neighborhood structure to define better positives and harder negatives for contrastive learning. A fourth line uses temporal-region supervision, moment–event hierarchy, or continuity constraints so that features become sensitive to boundaries, progression, and local coherence rather than only to global class identity (Drinkall et al., 2024, Lin et al., 2020, Xu et al., 2021, Alwassel et al., 2020, Zhang et al., 3 Feb 2025).

A common misconception is that temporal pre-training is equivalent to appending a date token or preserving file order. The surveyed work distinguishes more tightly integrated mechanisms. Some methods place time inside the attention computation itself rather than treating it as just another token. Others train yearly models from scratch rather than continuing pre-training from a future-aware base model. Still others define explicit supervision on permutations, future-step ranking, temporal overlap, or semantic differences between temporally separated frames. This suggests that temporally ordered pre-training is best understood as a design principle for data construction, objective design, and architectural conditioning, not as a single algorithm (Rosin et al., 2022, Drinkall et al., 2024, Nandy et al., 2024, Zhang et al., 3 Feb 2025).

2. Time-aware LLM pre-training

In transformer LLMs, a direct formulation of temporal conditioning is to modify self-attention so that attention scores depend on time. “Temporal Attention for LLMs” extends self-attention with a time matrix $T$ derived from the time point attached to the input text and defines

$$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$

For a sequence written at time $t$, the model produces time-specific contextualized word representations $x_1^t,\dots,x_n^t$. The key effect is that time is not just appended as another token; instead, it modulates the attention scores themselves. Applied to BERT and evaluated on SemEval-2020 Task 1 semantic change detection in English, Latin, and German, the method achieves state-of-the-art on all three datasets, with English $r=0.620$, $\rho=0.520$; Latin $r=0.661$, $\rho=0.565$; and German $r=0.767$, $\rho=0.763$, and the paper notes that all correlations are statistically significant ($$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$0) and that improvements are large, around 7%–36% over baselines (Rosin et al., 2022).

The same work clarifies a more general relation between temporally ordered pre-training and time-annotated corpora. Instead of training one static LLM over all text, it trains a transformer whose internal attention depends on the writing time of each input. It also reports that prepending time tokens alone is inferior to temporal attention, while a hybrid model gives mixed results: the combination is best on English, temporal attention alone is better on Latin and German, and smaller BERT variants can outperform larger ones for this task. Specifically, Temp. [Att](https://www.emergentmind.com/topics/attentional-aggregation-att). base yields $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$1, $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$2; Temp. Att. small yields $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$3, $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$4; and Temp. Att. tiny yields $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$5, $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$6 (Rosin et al., 2022).

A second language-model line focuses not on conditioning a single model on time labels, but on constructing chronologically valid model families. “Time Machine GPT” builds a series of yearly GPT-2-small-style models, denoted across years such as TiMaGPT/11 through TiMaGPT/22, where each yearly model only knows what could have been known up to the end of that year. The training data are assembled from Wikipedia revision history and WMT News with strict temporal inclusion criteria. Wikipedia is reconstructed “as it existed” on December 31 of each year from 2004 to 2023, and WMT News is sampled from a moving 5-year window ending at the target year with recency bias defined by

$$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$7

The domain mixture is fixed across annual datasets at 0.6 WMT News and 0.4 Wikipedia, and the models are trained from scratch rather than by continued pre-training, specifically to avoid future leakage (Drinkall et al., 2024).

This construction addresses a distinct problem from semantic change detection: point-in-time integrity. TiMaGPT is described as nonprognosticative, meaning that the model should not contain foresight. The paper shows historically plausible behavior on terms such as “COVID-19” and “coronavirus” and on names of country leaders around inauguration dates: TiMaGPT has high perplexity before those terms became prevalent and low perplexity afterward, whereas conventional temporally adapted models show implausibly early familiarity. The paper also notes that yearly models perform broadly comparably to other small models like GPT-2 small, OPT-125m, and GPT-Neo 125m on static benchmarks such as HellaSwag, TruthfulQA, PIQA, WSC, and Winogrande, which supports the claim that the temporal differences are not simply a byproduct of degraded base capability (Drinkall et al., 2024).

A broader pre-training study isolates data order itself as the critical variable. “Understanding Data Temporality Impact on LLMs Pre-training” compares 6B-parameter models pretrained on temporally ordered Common Crawl snapshots against standard shuffled pre-training. Both use the same 6B Transformer decoder architecture and train for 2.5T tokens, but the sequential curriculum processes yearly snapshots and produces eight checkpoints corresponding to data cutoffs from 2018 to 2025. On OLMES, sequential pre-training matches the shuffled baseline at the end of training. On KairosQA, a benchmark of 7,167 subject–relation pairs built from Wikidata, sequential models show a recency peak, while shuffled models exhibit temporal alignment inertia, often showing local maxima around 2015 and 2020 and dropping toward random performance for 2024 facts. The same paper also reports that a targeted cooldown experiment using 2024 data on the 2021 sequential checkpoint improved OLMES by +1.5%, from 71.7% to 73.2%, while also noting a trade-off: sequential training improves freshness but increases relative forgetting of older facts (Hippolyte et al., 21 May 2026).

3. Order-based objectives for textual and event-sequence understanding

For procedural text, temporally ordered pre-training uses step order directly as supervision. “Order-Based Pre-training Strategies for Procedural Text Understanding” compares three sequence-level objectives on recipe corpora: Permutation Classification, Embedding Regression, and Skip-Clip. In Permutation Classification, a recipe with $$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$8 steps is permuted using a small set of maximally diverse permutations selected by maximum Hamming distance, and the model predicts the permutation index with

$$TemporalAttention(Q, K, V, T) = \mathrm{softmax}\left(\frac{Q \frac{T^\top T}{\lVert T \rVert} K^\top}{\sqrt{d_k}}\right)V.$$9

In Embedding Regression, the target is a continuous embedding of the permutation, using either Hamming or Lehmer encoding, with mean squared error

$t$0

In Skip-Clip, the first $t$1 steps provide context and later steps are ranked by temporal proximity to that context using a hinge ranking loss

$t$2

These objectives operate at the level of ordered step sequences rather than token-level masking (Nandy et al., 2024).

The downstream task in that work is entity tracking on NPN-Cooking and ProPara, including entity state tracking and entity location tracking. On NPN-Cooking, Skip-Clip is best on test accuracy with 65.33, compared with 64.28 for RoBERTa-BASE, an improvement of about 1.6%. On ProPara, Permutation Classification is strongest in the open-domain setting, reporting Location Acc: 57.71, Status Acc: 70.57, Cat1: 73.72, Cat2: 43.16, Cat3: 32.72, and Avg Cat Acc: 49.87, improving over RoBERTa-BASE by about 7.4% on Status Accuracy and about 9% on Average Category Accuracy. The paper further notes that Emb Lehmer outperforms Emb Hamming on 5 of 6 metrics and that sequential combinations of objectives perform worse than individual methods because the supervision signals are mismatched (Nandy et al., 2024).

Related event-sequence work uses temporally ordered denoising rather than step classification. “Conditional Generation of Temporally-ordered Event Sequences” trains a BART-based conditional generation model on event sequences extracted from narrative text, assuming that discourse order in narrative text is a good approximation to temporal order. A clean sequence $t$3 is corrupted by random shuffling and event deletion with probability $t$4, and the model learns to reconstruct the original ordered sequence by maximizing $t$5. This training objective teaches both temporal ordering and event infilling. On CaTeRS, the temporal ordering task yields 77.1 pairwise accuracy for TemporalBART and 79.7 for TemporalBART-indexed, compared with 65.7 for Pairwise+SSVM and 54.1 for the BERT-based pointer network. On event infilling, TemporalBART-indexed reports 58.4 EM / 87.4 Top2, compared with 38.8 / 73.5 for Infilling GPT-2, and an ablation removing event deletion drops performance to 42.4 EM / 73.0 Top2, confirming that deletion corruption is essential for infilling (Lin et al., 2020).

These textual methods differ from timestamp-conditioned LLMs. They do not primarily aim to prevent future knowledge leakage or to model date-conditioned lexical semantics. Instead, they turn latent sequential structure into an explicit supervision target: global order reconstruction, structured ranking, or denoising recovery of temporally ordered event chains. A plausible implication is that temporally ordered pre-training in text splits into at least two regimes: one centered on external calendar time and one centered on intrinsic process order (Nandy et al., 2024, Lin et al., 2020, Rosin et al., 2022).

4. Temporal supervision in video-language pre-training

In video-language contrastive learning, temporal ordering can enter through pair construction rather than through timestamps. “VideoCLIP” trains a unified video-text model by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest-neighbor retrieval. A positive pair is constructed by sampling a text clip first, sampling a timestamp inside the text clip as the center of the video clip, and then growing a video clip with random duration up to 32 seconds around that center. This addresses what the paper calls low relevance temporal alignment, namely the fact that ASR and visual actions in instructional videos are often not perfectly synchronized. The training objective is a symmetric InfoNCE-style loss over average-pooled video and text representations:

$t$6

At the start of each epoch, VideoCLIP computes a global embedding for each video, builds a dense nearest-neighbor index, retrieves its $t$7 nearest neighbors, and samples $t$8 videos from these $t$9 to form a cluster for hard negatives (Xu et al., 2021).

Its ablations show that the temporal construction itself matters. On Youcook2 retrieval, baseline zero-shot VideoCLIP yields R@1 = 22.7; removing retrieval augmentation gives 18.5; removing both retrieval and overlap gives 12.4; and replacing the setup with MIL-NCE clips and loss gives 16.1. The paper explicitly concludes that overlap + retrieval together produce about a 50% relative improvement in recall over the weakest ablated setup. The model is then evaluated on text-to-video retrieval, VideoQA, action segmentation, and action step localization, with strong zero-shot performance and state-of-the-art fine-tuned performance on several tasks (Xu et al., 2021).

A more explicitly hierarchical design appears in “HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training.” HiTeA constructs a Long-view video $x_1^t,\dots,x_n^t$0, representing the whole video, and a Short-view video $x_1^t,\dots,x_n^t$1, representing a randomly truncated segment. It introduces Cross-Modal Moment Exploration (CME), which aligns the short-view clip with the most relevant words in the caption selected by cosine similarity, and Multi-Modal Temporal Relation Exploration (MTRE), which aligns short-view and long-view multi-modal representations with a SimSiam-style objective. The total pre-training objective is

$x_1^t,\dots,x_n^t$2

The paper also proposes a shuffling test, measuring Original, Shuffled, and Gap = Original − Shuffled, to quantify how much a dataset or model depends on temporal order (Ye et al., 2022).

The shuffling test is methodologically important because it shows that many benchmark datasets are only weakly temporal. On retrieval datasets such as MSRVTT, DiDeMo, LSMDC, and ActivityNet Caption, the average-recall gap is around 0.8–1.9. By contrast, temporally oriented datasets are far more sensitive, especially SSv2 Template, with a gap of 22.8, and SSv2 Label, with a gap of 3.2. Correspondingly, HiTeA’s largest gains appear on those temporal benchmarks: on SSv2, HiTeA reports 55.2 / 81.4 / 89.1 on SSv2-Label and 85.6 / 100.0 / 100.0 on SSv2-Template, improving R@1 by +11.1% and +8.6% over Singularity, respectively. It also improves VideoQA and captioning and shows strong zero-shot transfer (Ye et al., 2022).

Not all temporal video training is pre-training in the same sense. “TEMPLE:Temporal Preference Learning of Video LLMs via Difficulty Scheduling and Pre-SFT Alignment” is described as a temporally structured alignment method rather than a classic pretraining-on-ordered-video method. It generates preference pairs by selecting temporally rich videos, applying temporal perturbations such as clip dropping, clip shuffling, and clip reversal, and then treating the model’s response on the clean video as chosen and the response on the perturbed video as rejected. The method trains with DPO before SFT, a regime termed Pre-SFT Alignment, and uses a curriculum over perturbation difficulty with $x_1^t,\dots,x_n^t$3, then $x_1^t,\dots,x_n^t$4, then $x_1^t,\dots,x_n^t$5, then $x_1^t,\dots,x_n^t$6. On Qwen2-VL-7B, it improves VideoMME TP by +3.6, VideoMME TR by +2.8, MLVU overall by +1.3, and Vinoground average by +2.6. This suggests that temporally ordered pre-training and temporally structured post-pretraining alignment are adjacent but non-identical ideas (Li et al., 21 Mar 2025).

5. Temporally sensitive pre-training beyond text–video alignment

A distinct video line treats temporal sensitivity as a property of clip features for localization. “TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks” argues that trimmed action classification learns what action class is present but not necessarily whether a clip lies inside or outside an action instance. TSP therefore trains on untrimmed videos with temporal annotations, where each sampled clip has an action class label $x_1^t,\dots,x_n^t$7 if it is foreground and a temporal-region label $x_1^t,\dots,x_n^t$8 indicating foreground or background. The encoder output $x_1^t,\dots,x_n^t$9 is paired with a global video feature

$r=0.620$0

and the loss combines action classification and region classification:

$r=0.620$1

In the experiments, $r=0.620$2, foreground and background are balanced, and the method is evaluated across ResNet3D-18, R(2+1)D-18, and R(2+1)D-34 on ActivityNet v1.3 and THUMOS14 (Alwassel et al., 2020).

TSP improves temporal action localization, action proposal generation, and dense video captioning. On ActivityNet with R(2+1)D-34, it improves TAL average mAP to 35.81, proposal AUC to 69.04, and dense-captioning METEOR to 8.75. The paper emphasizes that gains are strongest at high tIoU thresholds and for extra-short and short actions, which supports the claim that TSP sharpens boundary sensitivity rather than only improving coarse recognition. Ablations show that TSP w/o GVF still beats trimmed-action baselines, but full TSP is better overall, and that most of the gain comes from the temporal region head (Alwassel et al., 2020).

Temporal ordering can also supervise dense perception in autonomous driving. “Self-Supervised Representation Learning from Temporal Ordering of Automated Driving Sequences” proposes TempO, a temporal ordering pretext task for pre-training region-level features. Each frame is represented as an unordered set of proposal features $r=0.620$3, and given an anchor frame and subsequent frames in arbitrary order, the model predicts next-image probabilities

$r=0.620$4

with a ranking loss

$r=0.620$5

The multi-frame sorting head scales less than quadratic with sequence length, and the pre-training is done for 6 epochs on 4 NVIDIA V100 GPUs (Lang et al., 2023).

TempO is evaluated on object detection, multi-object tracking, and frame retrieval. Using Sparse R-CNN, it achieves 31.4 mAP on BDD100K validation, improving over supervised COCO initialization by about +0.7 mAP and over the single-frame RPD baseline by as much as +2.0 mAP. On nuImages, Sparse R-CNN + TempO reaches 34.6 AP. For tracking on BDD100K validation, the best reported setup reaches 39.2 mHOTA, and on MOT17 test, Sparse R-CNN + TempO gives 72.8 MOTA and 54.3 HOTA, which the paper states is +1.4 HOTA and +2.3 MOTA better than compared supervised and unsupervised initialization strategies (Lang et al., 2023).

Time-series pre-training offers another variant of temporal ordering. “TimeSiam” samples an earlier subseries $r=0.620$6 and a later subseries $r=0.620$7, augments the current series with masking, and uses shared-weight Siamese encoders with learnable lineage embeddings that encode temporal distance. A decoder with cross-attention and self-attention reconstructs the original current series, optimizing

$r=0.620$8

The task is explicitly past-to-current reconstruction rather than self-reconstruction. Across 13 benchmarks, the paper reports average MSE reduction of 5.7% on PatchTST and 3.2% on iTransformer relative to random initialization, as well as strong classification performance such as 89.93% on AD, 90.67% on TDBrain, and 91.32% on PTB in the in-domain setting (Dong et al., 2024).

6. Continuity, forgetting, and the scope of temporal generalization

Temporally ordered pre-training does not only concern ordering; some work argues that useful representations must also be continuous. “Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents” proposes Action Temporal Coherence Learning (AcTOL), combining a Vision-Language Ordering (VLO) objective with a local Brownian bridge continuity regularizer:

$r=0.620$9

VLO defines a semantic alignment score

$\rho=0.520$0

and negatives are frames at least as far away in time as the comparison frame. The paper states that optimizing VLO encourages representations satisfying a Vision-Language Ordering property, and the Brownian bridge regularizer enforces smooth local transitions and robustness to language perturbations (Zhang et al., 3 Feb 2025).

AcTOL is evaluated in language-conditioned behavior cloning on Franka Kitchen and Metaworld with frozen pretrained encoders. Relative to baselines such as CLIP, R3M, LIV, and DecisionNCE, the paper reports improvements over the state of the art of up to 49.0% at 5 demos in Franka Kitchen, 46.4% at 15 demos, and 26.3% at 25 demos. It also finds that removing the Brownian bridge term consistently reduces performance and that the learned representation yields reward curves that rise during the relevant action, peak near completion, and fall when the action ends or a different action begins (Zhang et al., 3 Feb 2025).

Across the surveyed literature, three recurring limitations appear. First, temporally ordered pre-training usually requires temporally annotated or temporally recoverable data: publication dates, revision histories, frame sequences, step order, temporal boundaries, or persistent identity across frames. Second, stronger temporal freshness can trade off against retention of older knowledge: the sequential 6B LLMs improve recent factual knowledge but exhibit relative forgetting of older facts, and simple model soup or replay do not eliminate that trade-off (Hippolyte et al., 21 May 2026). Third, benchmark choice is consequential: HiTeA’s shuffling test shows that many standard video-language datasets have very small gaps between ordered and shuffled input, so claims about temporal reasoning can be benchmark-dependent (Ye et al., 2022).

A further misconception is that temporal training necessarily improves every aspect of modeling. The evidence is more specific. Sequential pre-training can match shuffled baselines on general language understanding while improving freshness, but it does not automatically preserve old knowledge. Temporal preference learning can improve temporal reasoning in Video LLMs, but it is best viewed as a complement to pre-training rather than a replacement. Order-based recipe objectives can outperform both traditional baselines and several large LLMs on procedural reasoning, but combinations of different order objectives can perform worse than individual methods. Temporal supervision therefore appears most effective when its inductive bias matches the downstream requirement: diachronic semantics for semantic change, point-in-time integrity for forecasting, overlap and moment alignment for video-language grounding, foreground/background discrimination for localization, and continuity-aware ordering for embodied control (Li et al., 21 Mar 2025, Nandy et al., 2024, Hippolyte et al., 21 May 2026).

Taken together, these results define temporally ordered pre-training as a broad methodological shift from static corpus modeling toward training procedures that preserve or exploit temporal origin, temporal relation, and temporal progression. In language, this includes time-aware attention and point-in-time training; in text understanding, order-supervised reconstruction and ranking; in video, temporally overlapping positives, moment–event hierarchy, and temporally sensitive localization features; in time series and driving, past-to-current reconstruction and transition prediction; and in embodied learning, ordered and continuous vision-language trajectories. The shared claim is not simply that time exists in the data, but that models improve when pre-training objectives are explicitly designed around what happens when (Rosin et al., 2022, Drinkall et al., 2024, Xu et al., 2021, Alwassel et al., 2020, Dong et al., 2024).