Quantify the Effectiveness of Self-Forcing Distillation for Exposure-Bias Reduction

Determine how much of the image-quality gap between training with clean ground-truth context and inference with model-generated context is closed by applying Self Forcing distillation to both the retrieved memory bank and the recent context window in Memorizon.

Background

Memorizon trains world models with clean ground-truth memory-bank entries and recent chunks, whereas inference conditions generation on the model’s own previously generated outputs. The paper identifies this mismatch as exposure bias and attributes much of the observed image-quality loss—particularly for longer training spans—to conditioning on generated frames through the memory bank.

The authors apply Self Forcing distillation to the retrieved bank as well as the recent context window, but do not establish quantitatively how much this intervention closes the training–inference quality gap. This remains unresolved in the paper.

References

This is the exposure bias that Self Forcing \citep{selfforcing} removes by training on the model's own rollouts, and our distillation (\hyperref[sec:distill]{Sec.~\ref*{sec:distill}) applies it to the bank as well as to the recent window; how much of the gap it closes remains to be measured.

— Memorizon: Training World Models Beyond Their Context Window  (2610.00544 - Liao et al., 30 Sep 2026) in Section 6, Discussion, paragraph “Limitations”