LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

Abstract: Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.

• The paper demonstrates a more efficient video self-supervised learning approach by replacing complex architectures with SIGReg, a distributional regularizer.
• Uniform random dropping of 95% of video patch tokens improves ImageNet transfer accuracy to 61.0%, making it 7.6 points higher than the existing baselines while reducing computational cost substantially.
• Block-causal attention allows tether to new sequence-independently , preserving past temporal dependencies and proving advantageous in certain scenarios .

LeVJEPA addresses a specific inefficiency in video self-supervised learning: the computational cost of processing long spatiotemporal token sequences is compounded by architectural mechanisms introduced solely to prevent representation collapse. The paper proposes replacing these mechanisms with a distributional regularizer, SIGReg, and uses the resulting simplification to examine sparse token processing, causal attention, and video pretraining at controlled compute budgets. The central claim is that video representations can be learned with an encoder and projector alone, while aggressive token dropping simultaneously reduces computation and improves several downstream metrics.

Methodological premise

Prior joint-embedding approaches such as BYOL and V-JEPA prevent collapse through branch asymmetry. They use an online encoder, an exponential-moving-average target encoder, stop-gradient operations, and—in V-JEPA—the prediction of masked target representations. Masked autoencoders avoid collapse through pixel-space reconstruction, but require a decoder and masking patterns designed to make imputation nontrivial. LeVJEPA removes both classes of machinery.

The method constructs one global and several local spatial views of the same 16-frame clip. All views share the same temporal window, while local views differ through cropping and photometric augmentation. Each view is processed by a shared video ViT. Only the [cls] token is projected into a 256-dimensional embedding space and supervised. The loss contains two terms: an invariance loss that minimizes the mean-squared distance between global and local embeddings, and SIGReg, which regularizes the embedding distribution toward an isotropic Gaussian.

The invariance term alone would admit constant representations. SIGReg excludes this solution by testing whether random one-dimensional projections of the batch embeddings match a standard Gaussian. Its implementation uses 1,024 random directions and 17 quadrature knots over the interval $[0,3]$. The regularization coefficient is fixed at $\lambda = 0.02$ for every experiment, rather than tuned per architecture, dataset, or benchmark. The trainable system therefore consists only of the encoder, the projector, and the single combined objective. A Polyak-averaged encoder is retained as an evaluation checkpoint, but unlike a BYOL- or V-JEPA-style target encoder, it does not participate in training.

The distributional formulation is consequential beyond architectural economy. It supplies an explicit anti-collapse criterion rather than relying on an empirically effective training asymmetry whose dynamics are difficult to characterize analytically. The paper’s theoretical motivation follows LeJEPA, which argues that an isotropic Gaussian embedding distribution minimizes worst-case downstream probing risk (Balestriero et al., 11 Nov 2025). LeVJEPA’s video contribution is to show that this objective remains effective when the input is a temporally structured, heavily subsampled clip.

Sparse token processing as both efficiency mechanism and augmentation

LeVJEPA tokenizes each frame independently with a $16 \times 16$ spatial patch embedding. For a $224 \times 224$ global view of 16 frames, this produces 3,136 patch tokens; a $96 \times 96$ local view produces 576. The method then drops 95% of patch tokens uniformly at random before the transformer blocks. The [cls] token is retained.

This design is not presented as an approximation to full-sequence processing. It changes the learning problem: the clip-level representation must remain inferable from a sparse, randomly located observation of the video. The resulting ablation is unusually strong. ImageNet-1K probing accuracy rises monotonically from 33.9% with no token dropping to 47.6% when 95% of tokens are discarded. The comparison between 90% and 95% dropping is especially important: accuracy is effectively unchanged, at 47.4% versus 47.6%, despite halving the number of processed tokens.

Figure 1: ImageNet probing accuracy as a function of the token-dropping ratio, showing monotonic improvement under increasingly sparse random observations.

The implication is that token dropping is simultaneously a computational reduction and a stochastic regularizer. Since transformer feed-forward cost scales approximately linearly with sequence length, the 95% dropping configuration reduces this component of the cost by roughly twentyfold. The result also challenges the interpretation of sparse video masking inherited from masked prediction. In reconstruction-based objectives, structured tube masks prevent the model from copying information from nearby spatial or temporal tokens. LeVJEPA reconstructs nothing, so that rationale no longer applies.

Indeed, tube dropping performs substantially worse than uniform random dropping. On ImageNet-1K, accuracy falls from 50.7% with uniform dropping to 39.6% with tube dropping; on Something-Something-v2 under temporal patch aggregation, it falls from 28.8% to 26.4%. Uniform sampling distributes observations across both space and time, whereas a tube pattern permanently removes the same spatial regions throughout the clip. The contrast supports the paper’s objective-dependent interpretation of masking: a mask suitable for imputation is not necessarily suitable when the retained tokens are the complete input to an invariant encoder.

The benefit is task-dependent. On Something-Something-v2, aggressive dropping beyond 30% reduces motion-centric accuracy at short training schedules. The paper reports that longer schedules largely recover this loss while preserving lower total compute. This qualification matters: sparse tokenization improves static appearance transfer more reliably than short-horizon motion recognition. The proposed explanation is that random sparsification makes cross-frame correspondences less frequently available within a single view, increasing the number of optimization steps required to learn motion-sensitive features.

Simplifying temporal tokenization and attention

The paper also removes temporal patch aggregation. Conventional video ViTs often combine adjacent frames at the input, for example with temporal patch extent $\tau = 2$, to reduce the initial sequence length. LeVJEPA instead uses per-frame tokens, $\tau = 1$, and compensates with a lower retention rate so that the two configurations process the same number of retained tokens during pretraining.

Under this matched token budget, per-frame tokenization is better on both evaluated tasks: 50.7% versus 47.4% on ImageNet-1K and 30.4% versus 28.8% on Something-Something-v2. The latter result directly bears on the usual motivation for temporal aggregation. Although aggregation is often justified as a mechanism for capturing short-range motion, its removal improves rather than harms motion-centric accuracy in this experiment. The evidence does not establish that temporal aggregation is universally detrimental; it establishes that LeVJEPA’s objective does not require it and that per-frame representations are preferable under the reported evaluation protocol.

LeVJEPA further exploits the absence of branch asymmetry to impose block-causal attention. Patch tokens attend bidirectionally within each frame and causally to preceding frames. The [cls] token attends to the entire sequence but does not serve as an input to the patch-token representations. Consequently, the representation of frame $t$ depends only on frames up to and including $t$.

Contrary to the expected trade-off between causality and representational quality, block-causal attention slightly outperforms bidirectional attention on ImageNet-1K: 51.2% versus 50.7%. The difference is within the range where variability should be considered, but there is no measured accuracy penalty. The practical implication is stronger than the numerical difference: the encoder itself produces temporally ordered frame representations, and new frames can be appended without re-encoding the past. This property is relevant to streaming inference and autoregressive world-model interfaces, where a bidirectional encoder requires either recomputation or a separate temporal model.

Figure 2: Cosine similarity between a query patch and all patch tokens, illustrating spatially localized semantic correspondence in LeVJEPA representations.

Controlled comparison with video baselines

The comparisons are designed to reduce common confounds in self-supervised video evaluation. Baselines are retrained on the same 20% class-balanced subsample of K710, for 240 epochs, with effective batch size 3,072, and evaluated using frozen attentive probing. The compared encoders include V-JEPA 2 and VideoMAEv2 across ViT-S, ViT-B, and ViT-L scales.

At matched epochs, LeVJEPA reaches comparable or higher ImageNet accuracy at substantially lower pretraining compute. The reported efficiency advantage ranges from 5.6 times at ViT-L to 20.8 times at ViT-S. The ViT-B comparison is particularly illustrative: LeVJEPA requires 4.8 ExaFLOPs versus 36.4 ExaFLOPs for V-JEPA 2, while remaining within one accuracy point. At ViT-L, LeVJEPA surpasses V-JEPA 2 by 1.9 percentage points while using 5.6 times less compute. The paper also emphasizes that the LeVJEPA ViT-L uses less compute than the V-JEPA 2 ViT-S in this comparison.

Figure 3: ImageNet-1K attentive-probing accuracy versus total pretraining compute at matched epochs across ViT-S, ViT-B, and ViT-L models.

The efficiency difference follows from multiple reductions rather than one isolated optimization. LeVJEPA does not process full token sequences during pretraining, performs no target-encoder forward pass, and has no predictor over masked queries. Thus, nearly all major operations contribute directly to the single training objective.

At equal total pretraining FLOPs, LeVJEPA receives a longer schedule because each training sample is cheaper. The ViT-B model is trained for 1,085 epochs with 10 local views. It achieves 61.0% on ImageNet-1K, compared with 53.4% for VideoMAEv2 and 51.6% for V-JEPA 2. The 7.6-point advantage over the strongest baseline is the paper’s clearest compute-controlled result. On Kinetics-400, LeVJEPA also leads, obtaining 44.6% versus 37.4% for VideoMAEv2 and 40.7% for V-JEPA 2. On Something-Something-v2, however, it reaches 40.4%, below VideoMAEv2’s 43.6% and V-JEPA 2’s 42.5%.

Method	ImageNet-1K	Something-Something-v2	Kinetics-400
VideoMAEv2	53.4	43.6	37.4
V-JEPA 2	51.6	42.5	40.7
LeVJEPA	61.0	40.4	44.6

The benchmark pattern is therefore asymmetric. LeVJEPA is particularly strong on appearance-centric transfer and Kinetics-400, while motion-centric performance remains less dominant. This is consistent with the token-dropping ablation: the method’s most aggressive sparsity is beneficial overall but can make temporal correspondence learning more difficult at a fixed training duration.

Video versus image pretraining

The paper compares LeVJEPA against DINOv2 trained on individual frames sampled from the same source videos, with total pretraining FLOPs matched. This isolates the contribution of temporal structure from the contribution of data source and compute.

DINOv2 retains a 3.1-point advantage on ImageNet-1K, reaching 53.8% versus 50.7% for LeVJEPA. On Something-Something-v2, however, LeVJEPA obtains 30.4%, nearly twice DINOv2’s 16.9%. The result weakens the conventional assumption that video pretraining is useful primarily for motion while necessarily sacrificing general visual transfer. Under this setup, the appearance gap is modest, whereas the motion advantage is substantial.

The comparison should nevertheless be interpreted within the stated evaluation design. DINOv2 and LeVJEPA use different pretraining input structures and objectives, and the experiment does not establish parity at larger data or model scales. It does show that, at matched compute and identical source data, temporal video pretraining can approach a strong image-pretraining baseline on static recognition while retaining a large advantage on temporal recognition.

Scaling data and reducing hardware requirements

When the pretraining corpus is expanded beyond the controlled K710 subset to include K710, Something-Something-v2, Walking Tours, and the PE Video Dataset, a ViT-L trained for 100 epochs reaches 69.5% on ImageNet-1K and 55.0% on Something-Something-v2 under frozen attentive probing. This improves ImageNet accuracy by 9.5 points relative to the ViT-L trained on the restricted corpus, despite the shorter schedule. The objective and its fixed regularization coefficient are unchanged.

The paper also reports a small-scale hardware experiment. A ViT-Tiny trained for 12 hours on one RTX 5080 using approximately 620,000 frames from eight Walking Tours videos improves ImageNet accuracy from 8.9% at initialization to 25.2%. On the same 16 GB GPU, LeVJEPA supports batch size 128 in under 8 GB, whereas an equivalently sized V-JEPA configuration reaches only batch size 28. These results demonstrate that sparse sequences reduce memory pressure as well as arithmetic cost. They do not, however, imply that the resulting representation is competitive with large-scale foundation models; the experiment establishes accessibility and nontrivial learning from limited, uncurated data.

Emergent dense structure from clip-level supervision

A notable qualitative result concerns patch-token organization. The loss is applied exclusively to the clip-level [cls] token, with no direct patch-level objective. Nevertheless, PCA visualizations of patch tokens show semantic grouping and separation between foreground objects and background regions. Cosine similarity from a query patch is also spatially concentrated on the corresponding object.

Figure 4: Patch-token PCA for a common input image, showing semantically organized regions in LeVJEPA representations despite the absence of patch-level supervision.

LeVJEPA’s patch-token organization is comparable in the presented visualization to V-JEPA 2.1, which introduces an auxiliary patch-level loss specifically to improve dense features (Mur-Labadia et al., 15 Mar 2026). V-JEPA 2, without that auxiliary objective, does not exhibit comparable visible organization in the cited comparison. The result is important but limited in scope: qualitative PCA and cosine-similarity visualizations establish emergent structure, not performance on segmentation, tracking, optical flow, or other dense prediction tasks. Whether the emergent features are quantitatively sufficient for those tasks remains unresolved.

Limitations and open questions

The strongest limitation is the residual weakness on motion-centric evaluation under aggressive sparsity. LeVJEPA is below both principal video baselines on Something-Something-v2 in the FLOP-matched comparison, and high token-dropping ratios initially reduce performance on that benchmark. Longer training recovers much of the deficit, but the reported remedy increases optimization duration rather than preserving temporal information more effectively. The paper leaves open whether a sparsity pattern can retain the computational benefits of random dropping while explicitly preserving cross-frame correspondences.

The controlled baseline experiments use a restricted 20% K710 corpus and scale only through ViT-L. The larger-corpus result is encouraging, but it is not a controlled comparison against V-JEPA 2 or other methods at the same expanded data scale. The behavior of SIGReg under substantially larger models, batches, datasets, and token dimensions is therefore not established by the experiments presented.

The evaluation is also limited to frozen probing. Attentive probing is more expressive than linear probing and is appropriate for measuring information content, but it introduces a learned adaptation layer whose capacity can affect comparisons. End-to-end fine-tuning, dense prediction, tracking, long-context temporal reasoning, and action-conditioned modeling are not evaluated. In particular, causal attention guarantees a past-only receptive field in the encoder, but the paper does not demonstrate that this yields superior performance in streaming perception or world-model planning.

Finally, the theoretical anti-collapse guarantee concerns the embedding distribution under the assumptions of SIGReg and does not by itself guarantee useful temporal representations, optimal invariances, or robustness to distribution shift. The empirical success of the method depends on the global-local view construction, sparse sampling, optimization schedule, and probe protocol even though the loss has only one explicitly tuned coefficient.

Conclusion

LeVJEPA presents a compact video-pretraining formulation based on invariance plus SIGReg. Its principal empirical contribution is the demonstration that uniform random dropping of 95% of video patch tokens can reduce pretraining cost while improving ImageNet transfer, enabling a ViT-B to achieve 61.0% ImageNet accuracy at matched compute compared with 53.4% for VideoMAEv2 and 51.6% for V-JEPA 2. Its principal architectural contribution is showing that block-causal attention and per-frame tokenization can be used without a measured loss in frozen-probe accuracy.

The method does not dominate every benchmark: motion-centric performance remains its principal weakness, particularly under short schedules and extreme sparsity. Within those qualifications, the results support the paper’s central thesis that collapse-free distributional regularization can replace substantial video-pretraining machinery and that sparse, causal video encoders can provide an efficient alternative to image-only pretraining.