Memory-based Temporal Relation Distillation

• Memory-based Temporal Relation Distillation is a design pattern that records and abstracts temporally structured evidence to capture dependencies across individual episodes and longer sequences.
• It employs diverse memory representations—including explicit stores, recurrent hidden states, and checkpoint sequences—to maintain inter-step and cross-episode temporal relationships in tasks like RL, diffusion, and BEV detection.
• The distillation mechanisms convert temporal signals into trainable supervision, improving performance metrics while addressing challenges such as temporal drift, fragmentation, and misalignment.

Memory-based Temporal Relation Distillation denotes a class of mechanisms that preserve temporal structure by storing temporally organized evidence in an explicit memory, a recurrent state, or a temporally indexed representation, and then distilling that structure into a policy, a student model, or a retrieval-ready memory layer. In the recent literature, the phrase is used for online procedural memory in reinforcement learning with verifiable rewards, semantic-time memory in personalized agents, time-aware pixel queues in low-bit diffusion, checkpoint- and Conv-LSTM-based review in supervised knowledge distillation, hidden-state discrepancy transfer in recurrent time-series classification, and temporally aggregated teacher targets in BEV detection (Liu et al., 1 Jul 2026, Su et al., 12 Jan 2026, Feng et al., 6 Jul 2025, Wang et al., 2023, Udayangani et al., 7 Jan 2026, Zheng et al., 2024).

1. Core problem formulation

Across these systems, the motivating claim is that temporally structured signals are usually richer than point-wise or episode-local supervision. In Procedural Memory Distillation (PMD), RLVR and SDPO evaluate each rollout with a verifier, but supervision remains episode-local and therefore misses cross-episode signals such as strategies that consistently pass verification, persistent failure modes, and reusable reasoning sequences. PMD explicitly targets both intra-episode temporal dependencies and cross-episode signals, treating them as procedural knowledge to be stored and distilled into parameters (Liu et al., 1 Jul 2026).

Temporal Semantic Memory (TSM) frames the problem differently but with the same temporal emphasis. It identifies temporal inaccuracy, where memories are indexed by dialogue time rather than occurrence time, and temporal fragmentation, where coherent durative states are split into isolated snippets. The resulting failure is not merely retrieval error; it is a mismatch between stored memory and real-world temporal validity (Su et al., 12 Jan 2026).

In low-bit diffusion, MPQ-DMv2 identifies a different failure mode: time-wise activation quantization can make the student locally correct at timestep $t$ while globally inconsistent over the denoising trajectory, producing temporal drift. Its memory-based temporal relation distillation is therefore designed to preserve inter-step relational structure rather than only per-step feature alignment (Feng et al., 6 Jul 2025).

Comparable formulations appear in other domains. TKD for active perception uses selective teacher invocation because temporal dependencies among frames make full teacher execution unnecessary on every frame (Farhadi et al., 2019). TempDistiller for camera-based BEV detection compresses long-term temporal knowledge from a teacher with more frames into a student with fewer frames (Zheng et al., 2024). MemKD argues that conventional KD neglects the memory retention behavior of recurrent time-series models and therefore fails to preserve hidden-state dynamics over subsequences (Udayangani et al., 7 Jan 2026). Document-level temporal dependency graph construction uses discourse profiling as a proxy for global temporal frames, and ETRE work such as MulCo treats event-pair distance bands as distinct temporal regimes requiring cross-scale transfer (Choubey et al., 2022, Yao et al., 2022).

2. Memory representations and temporal units

A central distinction in this literature is the form taken by “memory.” In some systems the memory is explicit and external; in others it is a recurrent hidden state or a bank of checkpoints. The temporal relation being distilled changes accordingly: action order, validity intervals, denoising-step similarity, inter-frame affinity, or hidden-state evolution.

Framework	Memory form	Temporal relation encoded
PMD	Level-0 trajectories, Level-1 insights, Level-2 behaviors	Attempt $\rightarrow$ feedback, strategy scripts, recurring step order
TSM	TKG plus durative topics/personas	Occurrence time, valid intervals, slice-level continuity
MPQ-DMv2	Per-timestep FIFO queues $\mathcal{Q}_t$	Global inter-timestep relational distributions
TSKD	Saved student checkpoints plus Conv-LSTM state	Evolution of attention increments across training time
TempDistiller	Teacher temporal self-attention target	Long-term BEV/PV memory and inter-frame affinities
MemKD	Teacher and student hidden states $h^{(t)}$	Subsequence-level memory response
TKD	LSTM selector over student features	Keyframe-triggered adaptation over frame sequences

PMD is the most explicit hierarchical design. It stores Level-0 raw trajectories per problem, Level-1 self-reflected strategies and lessons, and Level-2 higher-level recurring behavioral patterns. Level-0 keeps up to five successful responses and up to three failure+feedback pairs per problem, with cosine-similarity diversity gating and redundancy-based eviction. Level-1 extracts strategies from successes and lessons from failures, calibrating each item by $c_i = n_{\text{success}}/n_{\text{total}}$. Level-2 clusters semantically related problems using Qwen3-Embedding-0.6B and abstracts insights into named skills, error archetypes, and stepwise patterns (Liu et al., 1 Jul 2026).

TSM uses a different ontology. Episodic memory is a temporal knowledge graph

$\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$

with valid_time and invalid_time maintained by DUPLICATE, ADD, INVALIDATE, and UPDATE operations. Durative memory is then constructed over monthly slices

$\mathcal{G}=\bigcup_k \mathcal{G}^{(k)},$

where entities are clustered with a Gaussian Mixture Model over name embeddings and summarized into time-anchored topics and personas (Su et al., 12 Jan 2026).

In MPQ-DMv2, the memory is neither symbolic nor episodic but a per-timestep queue,

$$\mathcal{Q}=\{\mathcal{Q}_1,\mathcal{Q}_2,\dots,\mathcal{Q}_T\}, \quad \mathcal{Q}_t\in\mathbb{R}^{L\times d},$$

containing sampled full-precision features from timestep $t$. The global reference matrix $\mathbf{F}_{\text{ref}}$ is then formed by sampling $\rightarrow$0 features from each queue, yielding a time-aware reference bank spanning the whole denoising process (Feng et al., 6 Jul 2025).

TSKD’s memory is training-time rather than task-time. It periodically saves student checkpoints as memory nodes, computes attention-map increments between nodes, and feeds the resulting temporal sequence into a Conv-LSTM. TempDistiller uses teacher temporal self-attention to produce a long-term target $\rightarrow$1 for each student frame. MemKD uses no external bank at all; its operative memory is the recurrent hidden state, and temporal relation is defined directly by hidden-state change over a subsequence (Wang et al., 2023, Zheng et al., 2024, Udayangani et al., 7 Jan 2026).

3. Distillation mechanisms

The distillation step is where stored temporal structure is converted into trainable supervision. In PMD, the teacher is memory-conditioned:

$\rightarrow$2

and the student matches this teacher with SDPO-style reverse-KL on the same prefixes:

$\rightarrow$3

This converts cross-episode temporal relations into teacher-side guidance at token level (Liu et al., 1 Jul 2026).

MPQ-DMv2 uses relation distributions over a global temporal bank:

$\rightarrow$4

with loss

$\rightarrow$5

The overall objective is

$\rightarrow$6

The distinctive feature is that similarity to features from all timesteps becomes the distillation target, not just alignment at the current timestep (Feng et al., 6 Jul 2025).

MemKD defines temporal relation by normalized hidden-state difference:

$\rightarrow$7

and constructs teacher and student memory-retention metrics

$\rightarrow$8

The distillation term is a Smooth L1 loss,

$\rightarrow$9

combined with cross-entropy as

$\mathcal{Q}_t$0

Short subsequences use $\mathcal{Q}_t$1; long subsequences use $\mathcal{Q}_t$2 (Udayangani et al., 7 Jan 2026).

TSKD turns training history into a dynamic target. The Conv-LSTM predicts a next-step increment

$\mathcal{Q}_t$3

while the teacher supplies the current absolute increment

$\mathcal{Q}_t$4

Temporal supervised KD uses

$\mathcal{Q}_t$5

or a multi-layer sum over aligned layer pairs (Wang et al., 2023).

TempDistiller distinguishes reconstruction and relation distillation. With fewer student frames, it reconstructs masked student features against a temporally aggregated teacher target. With full frames, it instead aligns inter-frame relations using

$\mathcal{Q}_t$6

and a KL loss over row-wise softmaxed similarity matrices with $\mathcal{Q}_t$7 (Zheng et al., 2024).

4. Language-model and agent settings

PMD places memory-based temporal relation distillation inside RLVR training. Its central principle is co-evolution: the policy generates rollouts that update memory, and the updated memory conditions the teacher that supervises the next policy update. Level-0 and Level-1 are updated on every batch, Level-2 is consolidated periodically, and the teacher context is composed from problem-specific experience, insight, and retrieved global behaviors. On Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8–5.5% on SCIKNOWEVAL and 7.9–13.6% on LIVECODEBENCH. The reported numbers are Qwen3-8B 74.4 $\mathcal{Q}_t$8 77.2 and OLMo3-7B 69.5 $\mathcal{Q}_t$9 73.3 on SciKnowEval avg@16, and Qwen3-8B 47.9 $h^{(t)}$0 51.7 and OLMo3-7B 45.0 $h^{(t)}$1 51.1 on LiveCodeBench score@4. The ablations are especially diagnostic: Evolving Memory + Frozen Policy gives 54.0 and 35.9, Frozen Memory + Evolving Policy gives 65.0 and 47.5, PMD-Transient gives 75.7 and 48.1, and full PMD gives 77.2 and 51.7; freezing either memory or policy degrades performance by more than 10 percentage points on SciKnowEval (Liu et al., 1 Jul 2026).

TSM uses “distillation” in a more structural sense. It distills dispersed episodic mentions and their temporal relations into a semantic timeline, a TKG with explicit occurrence times, and durative topics/personas anchored to temporal slices. Query handling is likewise temporal: a query issued at current time $h^{(t)}$2 is mapped to a semantic time constraint

$h^{(t)}$3

implemented with spaCy for explicit and relative expressions. Retrieval then combines dense semantic search with temporal filtering and lexicographic re-ranking so that only time-valid topics and personas are used. On LongMemEval_S, TSM achieves 74.80% overall ACC with GPT-4o-mini, surpassing A-MEM at 62.60%, and shows the largest category gains on Temporal at 69.92% and Multi-Session at 69.17%. On LoCoMo, among memory-based methods it reaches 76.69% with GPT-4o-mini and 71.23% with Qwen3. Ablations show w.o.temporal reduces LongMemEval_S overall from 74.8 to 72.8 and Temporal by 6.0 points; w.o.summary reduces overall by 1.2 and Single-Session Preference by 10.0 (Su et al., 12 Jan 2026).

These two systems illustrate two distinct meanings of temporal distillation in LLM settings. In PMD, temporally structured experience is distilled into weights during optimization and yields a memory-free model at inference. In TSM, temporal distillation compresses dialogues into durable, time-valid memory representations for later retrieval.

5. Vision, perception, and generative modeling

TKD for active perception uses teacher invocation itself as a temporal decision problem. A lightweight LSTM-based keyframe selector reads the student’s pre-decoder feature tensor and decides whether to distill at the current frame, with a Bernoulli safeguard and a $h^{(t)}$4 training-prevention window. Distillation uses a teacher-bounded tensor MSE loss that directly matches high-confidence teacher regions while bounding teacher influence in low-expectation regions with $h^{(t)}$5. Empirically, TKD selects about 24–26% of frames, achieves mAP $h^{(t)}$6 and F-1 $h^{(t)}$7 at IoU $h^{(t)}$8 on YouTube-Objects, reaches F-1 $h^{(t)}$9 on Hollywood and $c_i = n_{\text{success}}/n_{\text{total}}$0 on The Pursuit of Happyness, and runs at about 91 FPS sequentially or up to $c_i = n_{\text{success}}/n_{\text{total}}$1 FPS with parallel execution (Farhadi et al., 2019).

MPQ-DMv2 extends temporal distillation to extremely low-bit diffusion. Its per-timestep queues store full-precision activations, with ImageNet defaults $c_i = n_{\text{success}}/n_{\text{total}}$2 and $c_i = n_{\text{success}}/n_{\text{total}}$3, and LSUN/COCO defaults $c_i = n_{\text{success}}/n_{\text{total}}$4 and $c_i = n_{\text{success}}/n_{\text{total}}$5. On LDM-4 ImageNet $c_i = n_{\text{success}}/n_{\text{total}}$6, W2A4 ablations show MPQ-DM baseline FID 36.59, +FZRMQ 33.58, +MTRD (MSE) 33.26, +MTRD (KL) 33.02, and +OOLRI 32.55 with IS 49.79, sFID 12.12, and Precision 54.40%. The KL formulation is better than MSE for the relational target, and $c_i = n_{\text{success}}/n_{\text{total}}$7 is reported as the best distillation weight (Feng et al., 6 Jul 2025).

TempDistiller addresses long-term temporal compression in camera-based BEV 3D detection. The teacher uses $c_i = n_{\text{success}}/n_{\text{total}}$8 frames in the main setting, while the student typically uses $c_i = n_{\text{success}}/n_{\text{total}}$9. Temporal self-attention produces reconstruction targets for masked BEV and PV features, and when $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$0 the method switches to temporal relation distillation over inter-frame BEV affinities. On nuScenes val, SparseBEV ResNet-50 with $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$1 improves from NDS 52.9 and mAP 41.6 to NDS 54.0 and mAP 43.2, with mAVE improving to 0.253 and inference speed unchanged at 26.1 FPS. With full frames $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$2, mAP improves from 44.7 to 45.1 and mAVE from 0.251 to 0.238. Compressing temporal knowledge from 8 to 4 frames yields about +6 FPS (Zheng et al., 2024).

TSKD treats the student’s own feature evolution as a temporal process worth distilling. Its memorize–review schedule stores checkpoints, forms sequences of attention increments, and trains a Conv-LSTM reviewer. On CIFAR-100, reported gains include ResNet56 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$3 ResNet20 from 69.06 to 71.76, WRN40-2 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$4 WRN16-2 from 73.26 to 75.77, and VGG13 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$5 VGG8 from 70.36 to 74.75. On ImageNet, ResNet-50 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$6 MobileNetV2 improves from 68.87/88.76 to 72.22/90.99 in Top-1/Top-5, and ResNet-34 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$7 ResNet-18 improves from 69.75/89.07 to 71.90/90.76. On MS-COCO detection, TSKD+ReviewKD improves R101 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$8 R50 from 40.36 to 40.65 mAP and R50 $\mathcal{G}=\{(e_s,r,e_o,t)\mid t\in\mathbb{T}\},$9 MobileNetV2 from 33.71 to 34.03 (Wang et al., 2023).

6. Structured temporal reasoning and sequential classification

Memory-based temporal relation distillation is not confined to generative or perception models. In document-level temporal dependency graph construction, discourse profiling is used as an auxiliary temporal teacher. The model ranks candidate parents for events and timexes, while a discourse-profiling head distills sentence-level roles such as Main event, Previous event, Historical event, and Expectation. RoBERTa DP-Distillation improves overall accuracy from 77.26 to 78.93 on test and cross-sentence F1 from 72.83 to 75.70, with the gain concentrated in difficult long-range links. The work explicitly states that there is no explicit memory module; instead, the role-aware encoder representation functions as an implicit document-level memory (Choubey et al., 2022).

MemKD makes the memory metaphor literal for recurrent time-series models. The teacher is a three-layer LSTM with hidden size 100 and 812,008 parameters; the student is a single-layer LSTM with hidden size 8 and 1,480 parameters, yielding approximately 500× compression and a reported memory reduction from 0.7744 MB to 0.0014 MB. Across 56 UCR datasets, Base, Base-KD, and MemKD achieve average AUC-PRC 63.89, 68.65, and 74.05; average AUC-ROC 77.00, 80.36, and 84.65; and average accuracy 62.94, 67.23, and 71.41, with win/loss counts 0/56, 7/49, and 49/7 respectively. Ablations show that short-only memory discrepancy yields average AUC-PRC 72.24, long-only 74.04, and combined MemKD 74.78 in the 12-dataset setting (Udayangani et al., 7 Jan 2026).

MulCo for event temporal relation extraction is an important boundary case. It performs multi-scale co-distillation between a BERT encoder and a GNN encoder, using SAT-compressed structural representations and a symmetric stop-gradient InfoNCE objective. It does not use an external memory bank, but its design is directly relevant because it shares knowledge across short and long event-pair proximity bands. The reported F1 scores are 55.1 on TDDMan, 77.1 on TDDAuto, 90.4 on MATRES, and 85.6 on TB-Dense (Yao et al., 2022).

A second boundary case is LLM-guided distillation for temporal knowledge graph reasoning. The framework uses a high-capacity temporal teacher and an auxiliary LLM instructor, but explicitly states that it does not introduce memory networks, replay buffers, or memory banks of interactions. Temporal dynamics are modeled through time-aware embeddings and, for TADistMult, an LSTM-based relation-time encoder. With teacher dimension 400 and student dimension 25, the student retains a compact inference path and obtains small but consistent improvements over BKD on YAGO11k and WIKIdata12k (Xing et al., 16 Feb 2026).

7. Empirical patterns, limitations, and conceptual distinctions

Several regularities recur across the literature. First, explicit temporal memory tends to help most when local supervision is known to be insufficient: cross-episode RLVR in PMD, semantic-time misalignment in TSM, timestep-wise quantization drift in diffusion, sparse long-term motion cues in BEV detection, or subsequence-level retention in RNNs. Second, temporal relation distillation is often most effective when the final model is memory-free at inference: PMD distills procedural knowledge into weights, TempDistiller discards TSA and generators at test time, and the TKG distillation framework uses only the compact student at deployment (Liu et al., 1 Jul 2026, Zheng et al., 2024, Xing et al., 16 Feb 2026).

The associated costs are domain-specific. PMD increases training-time step latency by $\mathcal{G}=\bigcup_k \mathcal{G}^{(k)},$0 relative to SDPO, although peak GPU memory is essentially unchanged because memory stores and reflection modules are off-GPU or asynchronous. MPQ-DMv2 reports queue overhead of approximately 293 MB with an additional $\mathcal{G}=\bigcup_k \mathcal{G}^{(k)},$1 MB for reference sampling on ImageNet defaults, yet calibration time and GPU memory remain close to MPQ-DM at 3.13 h and 19.8 GB versus 3.12 h and 19.7 GB. TempDistiller adds training-time reconstruction and relation losses but zero inference overhead beyond the student detector. TSM amortizes consolidation by a “sleep-time” monthly summarization stage rather than online per-query recomputation (Liu et al., 1 Jul 2026, Feng et al., 6 Jul 2025, Zheng et al., 2024, Su et al., 12 Jan 2026).

The failure modes are likewise recurrent. PMD identifies memory contamination, shortcut insights, overfitting to spurious patterns, and drift when memory or policy evolves alone. TSM notes fixed granularity, ambiguous or implicit times, and non-contiguous intervals. MPQ-DMv2 reports sensitivity to too-small queue sizes. TSKD depends on appropriate choices of memory interval $\mathcal{G}=\bigcup_k \mathcal{G}^{(k)},$2 and number of memory nodes $\mathcal{G}=\bigcup_k \mathcal{G}^{(k)},$3. TKD includes an explicit Bernoulli safeguard because the selector can miss useful keyframes (Liu et al., 1 Jul 2026, Su et al., 12 Jan 2026, Feng et al., 6 Jul 2025, Wang et al., 2023, Farhadi et al., 2019).

A common misconception is that any temporally informed distillation is “memory-based” in the same sense. The literature is more nuanced. PMD, TSM, MPQ-DMv2, TempDistiller, and TKD use explicit temporally indexed stores or targets. MemKD uses recurrent hidden state as the operative memory. Document TDG distillation uses an implicit memory in shaped encoder states, while MulCo and the LLM-guided TKG framework are better viewed as temporally structured distillation without an external memory module (Choubey et al., 2022, Yao et al., 2022, Xing et al., 16 Feb 2026).

This suggests a broad but technically coherent definition: memory-based temporal relation distillation is a design pattern in which temporally structured evidence is recorded, abstracted, or accumulated in a form that persists beyond a single local decision, and is then distilled so that a student, policy, or retriever internalizes temporal dependencies rather than merely matching isolated outputs. The specific temporal primitive varies—before/after edges, valid intervals, keyframes, denoising timesteps, frame affinities, or hidden-state responses—but the recurring objective is the same: preserve temporal structure under compression, adaptation, or efficiency constraints.