Next-Embedding Prediction Makes Strong Vision Learners (2512.16922v1)

Abstract: Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.

• The paper introduces NEPA, a self-supervised pretraining method that predicts next patch embeddings via a causal Transformer.
• It employs a minimalist Vision Transformer enhanced with techniques like RoPE, LayerScale, SwiGLU, and QK-Norm for stable training.
• Empirical results demonstrate NEPA's competitive performance on ImageNet accuracy and ADE20K segmentation benchmarks.

Next-Embedding Prediction for Vision: A Comprehensive Analysis of NEPA

Introduction

This paper introduces Next-Embedding Predictive Autoregression (NEPA), a novel self-supervised vision pretraining framework designed to shift the focus from representation learning to direct predictive modeling. NEPA builds on the autoregressive paradigm that underlies modern language modeling: instead of extracting features for downstream use, it formulates visual pretraining as the task of predicting future patch embeddings from past ones using a causal Transformer. Compared with contrastive, masked reconstruction, and JEPA-style approaches, NEPA eliminates the need for task-specific heads, pixel-level decoders, or auxiliary losses. The method is realized with a minimalist yet scalable Transformer architecture and demonstrates strong downstream performance on classification and segmentation benchmarks.

Figure 1: The NEPA framework decomposes an image into patches and sequentially predicts each patch embedding, mirroring next-token prediction in LLMs.

Methodology

Next-Embedding Prediction Objective

NEPA operates by dividing an input image into $T$ non-overlapping patches, each linearly embedded. The predictive model is a causal Transformer trained to estimate the next embedding $\widehat{z}_{t+1}$ given all prior embeddings $\{z_1,\ldots,z_t\}$. The objective function is a negative cosine similarity between the predicted embedding and the true, stop-gradient embedding from the encoder:

$$\mathcal{L} = -\frac{1}{T-1}\sum_{t=1}^{T-1} \left(\frac{z_{t+1}}{\|z_{t+1}\|_2} \cdot \frac{\widehat{z}_{t+1}}{\|\widehat{z}_{t+1}\|_2}\right)$$

This loss, inspired by SimSiam, ensures the model learns semantically meaningful predictions without explicit reconstruction of pixel values. The stop-gradient mechanism on targets is critical for avoiding degenerate collapsing solutions.

Architectural Design and Stabilization

The backbone is a Vision Transformer (ViT) with causal attention masking. Notably, the model receives patches embedded via a Conv2d layer and uses only a single forward pass for both context encoding and prediction.

Figure 2: The image pipeline with Conv2d patch embedding, pre-norm Transformer backbone, and stabilization components LayerScale, RoPE, SwiGLU, and QK-Norm.

Key stabilization and optimization techniques in NEPA include:

• RoPE (Rotary Position Embedding): Encodes relative position efficiently for arbitrary sequence lengths.
• LayerScale: Learnable per-layer scaling of residuals for increased stability in deep Transformers.
• SwiGLU: Nonlinear gated MLP variant, aligning with recent advances in vision/language architectures.
• QK-Norm: Query-key normalization in attention to suppress gradient explosions.

Comparison to Related Paradigms

NEPA is architecturally simpler than JEPA, avoiding dual encoders and dense prediction heads. Its predictive coding formulation is also distinct from contrastive and masked autoencoder approaches by obviating negative samples, additional decoders, or asymmetrical branches.

Figure 3: Architectural contrast between JEPA (left: dual encoders and heavy head) and NEPA (right: unified causal Transformer).

Empirical Findings

Ablation and Stability

Extensive ablation identifies the necessity of key components:

Figure 4: Ablation results for NEPA's main algorithmic and architectural choices on accuracy, loss, and gradient metrics.

• The autoregressive shift is essential: training without it diverges.
• Stop-gradient removal causes immediate representation collapse.
• LayerScale and QK-Norm yield smoother optimization and stable gradients.
• High random input masking is detrimental, in contrast to the masked autoencoder paradigm.

Downstream Performance

NEPA-pretrained models demonstrate strong results on standard vision tasks. ViT-B achieves 83.8% and ViT-L reaches 85.3% top-1 accuracy on ImageNet-1K after fine-tuning with causal attention—on par with or exceeding models using more complex pretraining signals. For ADE20K segmentation, NEPA achieves 48.3% (ViT-B) and 54.0% (ViT-L) mean Intersection-over-Union, outperforming or matching models pretrained with labeled or reconstruction-based objectives.

Figure 5: Top-1 ImageNet-1K accuracy versus training epochs for NEPA-based ViT-Base and ViT-Large models.

These results are robust to changes in attention masking and methods of downstream head instantiation. Unlike many competitive vision self-supervised models, NEPA maintains high performance without requiring a decoder or task-specific reconstruction objectives.

Analysis of Learned Representations

Qualitative analyses of NEPA's attention patterns and embedding structure indicate robust semantic grouping and object-centric reasoning even without explicit supervision.

Figure 6: Validation images from ImageNet-1K highlight NEPA's selective, long-range, object-centric attention allocation and semantically meaningful embedding organization.

The attention maps produced by NEPA preferentially track spatially and semantically related regions, often corresponding to contiguous object parts, regardless of spatial distance. The predicted next-embedding shows high cosine similarity with patches from the same object, further evidencing coherent semantic grouping.

These representational properties extend to out-of-distribution datasets and remain strong even without explicit region or object labels.

Training Dynamics and Failure Modes

Training curves and progressive visualization of attention maps evidence stable optimization and continual improvement with scale.

Figure 7: NEPA pretraining dynamics showing monotonically improving loss and evolving attention maps; global attention structures emerge progressively.

However, NEPA demonstrates characteristic failure cases under challenging visual conditions: strong reflections, backlighting, small or overlapping objects, and ambiguous textures can lead to embedding confusion or incorrect spatial grouping.

Figure 8: Failure cases—NEPA confusing highlight reflections, misinterpreting shadows, and struggling with occluded or ambiguous object regions.

These limitations predominantly reflect dataset constraints (e.g., ImageNet biases) rather than architectural inadequacy.

Theoretical and Practical Implications

The core contribution of NEPA is exposing the sufficiency of next-embedding prediction for scalable, transferable vision pretraining. Notably, the architectural and training minimalism achieved by NEPA challenges the necessity of decoders, negative sampling, or heavily engineered pipelines for robust representation learning.

NEPA pushes the vision community towards modal-agnostic pretraining: direct embedding prediction is a unifying abstraction applicable across both vision and language. The observed semantic coherence of embedding space under autoregressive prediction validates theoretical connections to predictive coding frameworks in neuroscience and underpins recent multi-modal model designs with tied token/embedding matrices.

The potential extension to generative modeling is also clear: with autoregressive embedding predictors and access to suitable decoders, NEPA-style pretraining can be adapted for image synthesis or editing tasks, promising a seamless bridge between discriminative and generative capabilities within a single, unified model architecture.

Conclusion

NEPA demonstrates that autoregressive next-embedding prediction—operating solely in the latent space and eschewing common architectural/artifactual complexity—yields vision models with strong performance in classification and segmentation transfer. The evidence supports a fundamental paradigm shift: causal, generative objectives at the embedding level are not only competitive with but in certain respects superior to established contrastive or reconstruction-based self-supervised methods for vision, both in simplicity and transferability. These results invite a reevaluation of self-supervised pretraining strategies, with implications for scalable, modality-unified, and generative vision systems (2512.16922).