In Pursuit of Pixel Supervision for Visual Pre-training (2512.15715v1)

Abstract: At the most basic level, pixels are the source of the visual information through which we perceive the world. Pixels contain information at all levels, ranging from low-level attributes to high-level concepts. Autoencoders represent a classical and long-standing paradigm for learning representations from pixels or other raw inputs. In this work, we demonstrate that autoencoder-based self-supervised learning remains competitive today and can produce strong representations for downstream tasks, while remaining simple, stable, and efficient. Our model, codenamed "Pixio", is an enhanced masked autoencoder (MAE) with more challenging pre-training tasks and more capable architectures. The model is trained on 2B web-crawled images with a self-curation strategy with minimal human curation. Pixio performs competitively across a wide range of downstream tasks in the wild, including monocular depth estimation (e.g., Depth Anything), feed-forward 3D reconstruction (i.e., MapAnything), semantic segmentation, and robot learning, outperforming or matching DINOv3 trained at similar scales. Our results suggest that pixel-space self-supervised learning can serve as a promising alternative and a complement to latent-space approaches.

• The paper introduces Pixio, enhancing masked autoencoders by using pixel-level supervision to yield more transferable features.
• The approach incorporates a deeper decoder, larger masking blocks, and multiple class tokens to improve geometric and semantic understanding.
• The study confirms that web-scale, minimally curated data supports efficient representation learning and robust transfer to dense tasks.

Pixel-Space Supervision for Scalable Visual Pre-training: A Technical Analysis of Pixio

Motivation and Context

This paper presents Pixio, an enhanced masked autoencoder (MAE) designed to capitalize on pixel-level supervision for visual representation learning at web scale. The authors advocate for pixel-based objectives as an unbiased, information-rich, and scalable alternative to conventional supervision paradigms that depend on human-defined labels or language annotations. They systematically evaluate the performance and characteristics of Pixio against state-of-the-art latent space models such as DINOv2 and DINOv3, with particular focus on dense vision tasks requiring geometric and semantic understanding.

Algorithmic Innovations in Pixio

Pixio extends the MAE paradigm through four critical modifications driven by empirical and theoretical analysis:

1. Deeper Decoder: The authors identify that the original MAE's shallow decoder constrains pixel regression capability, resulting in suboptimal features in late encoder blocks. By increasing decoder depth and width, Pixio shifts the burden of pixel-level reconstruction from the encoder to the decoder, thereby enabling higher-quality, transferable representations in the encoder.
2. Larger Masking Blocks: Instead of single-patch masking, Pixio adopts larger blocks (e.g., 4×4 patches), mitigating local reconstruction shortcuts and encouraging genuine visual reasoning over broader context.
3. Multiple Class Tokens: The introduction of multiple [CLS] tokens allows Pixio to capture diverse global properties, reflecting scene semantics, pose, style, and other image-level attributes that a single class token cannot adequately represent.
4. Scale and Data Curation: Pixio is trained on 2B web-crawled images via a self-curation strategy that maximizes label diversity while minimizing human intervention. This diverse, minimally curated dataset is used to reduce benchmark-specific bias and improve broad generalization.

   Figure 1: Pixio architecture introduces a deeper decoder, larger mask blocks, multiple class tokens, and web-scale training data to optimize for pixel-space regression.

Representation Analysis and Ablations

A central observation is that, with shallow decoders, optimal features emerge far from the last encoder block, compromising semantic abstraction in favor of reconstructive detail. Increasing decoder depth realigns this, and Pixio's final encoder block achieves near-optimal feature quality.

Figure 2: Probing reveals the original MAE's optimal features are located away from the final encoder block, especially for dense prediction tasks.

Comprehensive ablation reveals:

• Decoder Capacity: Increasing decoder depth and width improves performance, but over-parameterization leads to memorization rather than transferable representations.
• Masking Granularity: Larger mask blocks provide more challenging pretext tasks, improving downstream accuracy for both regression and classification. Excessively large blocks, however, diminish predictability.
• Class Token Number: Additional class tokens consistently improve linear classification and dense prediction results, supporting richer global embedding.
• Pre-training Data Curation: Models pre-trained on scale-curated web datasets outperform those trained on conventional benchmarks, illustrating the importance of up-to-date and diverse supervision.

  Figure 3: Ablation study demonstrates that increasing decoder depth and width in the MAE leads to significant performance gains in depth estimation and segmentation.

  

  Figure 4: Larger masking granularity in MAE results in better downstream performance, confirming the utility of challenging reconstruction contexts.

  

  Figure 5: Increasing the number of class tokens leads to systematic gains in classification accuracy and dense prediction quality.

Downstream Task Evaluation

Pixio's performance is evaluated across monocular depth estimation, feed-forward 3D reconstruction, semantic segmentation, and robot learning:

• Monocular Depth Estimation: Pixio surpasses DINOv2/v3 and original MAE in RMSE and $\delta_1$ metrics. With a DPT head, Pixio achieves RMSE 0.268 and $\delta_1$ 95.5 on NYUv2.
• Zero-Shot Transfer (Depth Anything V2): Pixio maintains or exceeds DINOv3 performance on most distributions except KITTI, which is attributed to DINO's specialized data injection.
• 3D Reconstruction (MapAnything): Pixio achieves higher accuracy in scale, point correspondence, pose estimation, and depth metrics across ScanNet++, ETH3D, and TartanAirV2-WB benchmarks.
• Semantic Segmentation: Pixio matches or outperforms DINOv3, including on satellite imagery, despite using simpler objectives and fewer parameters.
• Robot Learning (CortexBench): Pixio demonstrates superior average performance compared to specialized and generic visual encoders, indicating strong policy transferability.

  Figure 6: Fine-tuning results on ImageNet reveal the narrowing gap between Pixio and DINOv3, especially when training the final encoder block.

Implications and Theoretical Impact

The empirical results underscore several key implications:

• Pixel-Space Supervision Viability: Directly regressing pixels remains highly competitive for representation learning in foundation models, challenging the dominance of latent-space contrastive objectives.
• Simplicity and Efficiency: The MAE-based approach offers algorithmic and implementation simplicity, with stable training and reduced need for elaborate stabilization techniques.
• Bias Reduction: Web-scale pixel supervision minimizes benchmark-centric and language-induced bias, advancing model robustness to novel data distributions.
• Scalability and Transferability: The framework supports scaling to billions of images, facilitating transfer to diverse downstream tasks, including geometry-aware and agent-based vision.

The authors argue for a future direction prioritizing temporal video data, where prediction tasks can leverage causally grounded objectives rather than artificial masking. This aspiration aligns with the limitations of static image pretraining and points toward less biased and richer pretraining paradigms.

Conclusion

Pixio advances MAE-based self-supervised learning with targeted, minimal refinements optimizing for pixel-space reconstruction objectives at scale. Through algorithmic analysis and extensive downstream evaluation, Pixio establishes that pixel-based supervision can achieve and often surpass contemporary latent-space approaches in both dense and semantic tasks. The findings motivate further exploration of scalable pixel and video supervision for next-generation visual foundation models.