Long-timescale consolidation during streaming pretraining

Incorporate long-timescale consolidation directly into self-supervised streaming pretraining for continuous video, rather than applying checkpoint averaging only at evaluation time.

Background

The paper studies StreamMAE for self-supervised representation learning from temporally ordered video streams without global reshuffling or long-term replay buffers. For long streaming runs, the authors evaluate checkpoint averaging as a post hoc procedure and find that averaging checkpoints from different stages can improve dense prediction while leaving classification performance nearly unchanged. However, the training process itself does not perform consolidation across widely separated stages of the stream. The authors explicitly identify integrating such long-timescale consolidation into streaming pretraining as an unresolved direction.