LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
This presentation examines LeVJEPA, a video self-supervised learning method that achieves significant computational efficiency by replacing conventional anti-collapse mechanisms with a distributional regularizer and aggressively dropping 95% of video tokens. The work demonstrates that sparse, causal video encoders can match or exceed the performance of more complex baselines while reducing pretraining cost by up to 20 times, particularly excelling at appearance-centric transfer tasks while revealing interesting trade-offs in motion-centric evaluation.Script
Training video models typically means processing thousands of tokens per clip and layering on architectural tricks to keep the system from collapsing. LeVJEPA throws out 95% of those tokens, strips away the extra machinery, and still outperforms the competition.
The method is surprisingly simple. Take one video clip, create a global view and a few cropped local views, and train a shared encoder to produce similar embeddings for all of them. A distributional regularizer called SIGReg prevents collapse by nudging embeddings toward an isotropic Gaussian, and that is the entire training objective.
Here is the counterintuitive result: accuracy climbs monotonically as you drop more tokens. At 95% sparsity, ImageNet probing hits 47.6%, compared with just 33.9% when you keep everything. Token dropping is not an approximation, it is a regularizer that forces the model to learn from sparse, randomly sampled observations.
When you match total pretraining compute, LeVJEPA runs longer because each sample is cheaper. A ViT-B trained this way reaches 61.0% on ImageNet, beating VideoMAEv2 by 7.6 points and V-JEPA 2 by 9.4 points. The efficiency advantage ranges from 5 times at large scales to over 20 times at small scales.
Despite training only the clip-level class token, LeVJEPA's patch tokens organize themselves semantically. PCA reveals clean separation between foreground objects and background, comparable to methods that explicitly supervise dense features. However, the paper does not evaluate segmentation or tracking, so whether this emergent structure is quantitatively sufficient for dense prediction remains an open question.
The simplicity pays off in accessibility. A tiny model trained for 12 hours on one consumer GPU with just 620,000 frames improves ImageNet accuracy from 8.9% to 25.2%. LeVJEPA makes video pretraining feasible on hardware that would struggle to fit a conventional video encoder. To dive deeper into this work or create your own video summaries of the latest research, visit EmergentMind.com and see what you can learn next.
