OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation

Abstract: This paper presents a family of advanced vision encoder, named OpenVision 3, that learns a single, unified visual representation that can serve both image understanding and image generation. Our core architecture is simple: we feed VAE-compressed image latents to a ViT encoder and train its output to support two complementary roles. First, the encoder output is passed to the ViT-VAE decoder to reconstruct the original image, encouraging the representation to capture generative structure. Second, the same representation is optimized with contrastive learning and image-captioning objectives, strengthening semantic features. By jointly optimizing reconstruction- and semantics-driven signals in a shared latent space, the encoder learns representations that synergize and generalize well across both regimes. We validate this unified design through extensive downstream evaluations with the encoder frozen. For multimodal understanding, we plug the encoder into the LLaVA-1.5 framework: it performs comparably with a standard CLIP vision encoder (e.g., 62.4 vs 62.2 on SeedBench, and 83.7 vs 82.9 on POPE). For generation, we test it under the RAE framework: ours substantially surpasses the standard CLIP-based encoder (e.g., gFID: 1.89 vs 2.54 on ImageNet). We hope this work can spur future research on unified modeling.

• The paper introduces a unified VAE–ViT approach that enhances both image reconstruction and semantic understanding, achieving a PSNR of 30.33 dB and an LPIPS of 0.061 on ImageNet.
• The methodology leverages a continuous latent space with a frozen FLUX.1 VAE and a trainable ViT, eliminating the need for discrete tokenization and minimizing gradient flow issues.
• The empirical results demonstrate that joint optimization of reconstruction and semantic objectives not only improves visual generation but also achieves competitive results on benchmarks like SeedBench and POPE.

OpenVision 3: Unified Visual Encoder for Understanding and Generation

Overview

The OpenVision 3 framework, as introduced in "OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation" (2601.15369), addresses the dual challenge of visual semantic understanding and high-fidelity image generation within a single architectural paradigm. Leveraging a streamlined VAE–ViT hybrid tokenizer, OpenVision 3 presents an empirically validated approach that eliminates the architectural bifurcation seen in prior unified multimodal models, providing robust performance gains with respect to both generative and semantic downstream evaluations.

Architectural Innovations

OpenVision 3's core contribution is the integration of a VAE encoder for spatial compression with a Vision Transformer (ViT) acting as a feature tokenizer. The image is encoded by the pre-trained (and frozen) FLUX.1 VAE, producing dense latent representations that are then tokenized by a randomly initialized ViT. The resulting unified representation is dual-purposed by being routed to two disjoint branches:

• Generation branch: Features are decoded via a symmetric ViT decoder, noise-perturbed for regularization, and subsequently upsampled by the VAE decoder. The objective enforces pixel-level and latent-level fidelity via reconstruction and perceptual losses (LPIPS).
• Understanding branch: The same token sequence is optimized for semantic alignment via joint contrastive learning (against text encodings) and image-captioning objectives, enabling high-level semantic structure acquisition.

Unlike discrete tokenization approaches (e.g., VQ-GAN, UniTok), OpenVision 3 preserves a continuous latent space, reducing discretization artifacts and supporting better gradient flow across all tasks.

Training Paradigm

The training regimen utilizes progressive input resolution, initially pretraining with images at 128×128 before finetuning at higher resolutions (224×224 or 256×256), with the vast majority of computation allocated to lower-resolution stages for efficiency. The FLUX.1 VAE is kept frozen, ensuring stability and facilitating the learning of the ViT-based unified tokenizer from scratch. Losses from both branches are combined with a weighting scheme that prioritizes semantics (understanding) over pure reconstruction (2:1 ratio). Models are trained at scale on the DataComp dataset with synthetic captions provided by LLaVA-Llama-3 recaptioning.

Empirical Results

Image Reconstruction and Generation

OpenVision 3 substantially surpasses prior unified tokenizers in reconstruction quality, achieving a PSNR of 30.33 dB on ImageNet and an LPIPS of 0.061, both far superior to strong baselines such as UniTok (25.34 dB, 0.132 LPIPS) and OmniTokenizer. In generative evaluation (class-conditional image generation under the RAE framework), OpenVision 3 achieves a gFID of 1.89 on ImageNet 256×256, outperforming CLIP (gFID 2.54) and continuous rivals such as SD-VAE (gFID 2.06). This indicates notable gains in low-level fidelity and structural preservation while benefiting from semantically-informed tokenization.

Multimodal Understanding

Integrated into the LLaVA-1.5 framework, OpenVision 3 provides understanding performance on par or superior to CLIP-based encoders. For example, on SeedBench, OpenVision 3 achieves 62.4 (B/2) and 66.0 (L/2) compared to 62.2 and 65.4 for CLIP (B/16, L/14, respectively). On POPE, performance is similarly strong, with OpenVision 3 reaching 83.7–85.3 versus CLIP's 82.9–84.7. These results support the claim that continuous, unified tokenization via their proposed VAE–ViT pipeline does not incur meaningful degradation in semantic alignment benchmarks relative to specialist models.

Objective Synergy

A key experimental insight is the mutual benefit derived from joint optimization of reconstruction and semantic objectives. Empirical analysis reveals that even in the absence of explicit reconstruction loss, semantic optimization reduces reconstruction errors, and vice versa—optimizing for reconstruction can lead to improved semantic metrics. This cross-regularization effect substantiates that a well-designed continuous representation space admits robust multi-task generalization, addressing the long-standing optimization tension in unified multimodal encoders.

Implications and Future Directions

Practical Impact

OpenVision 3's performance and architectural simplicity lower the implementation and training barrier for unified visual representation learning. Its ability to match and often surpass existing, more complex rival approaches suggests a preferred design choice for future multimodal foundation models targeting joint understanding and generative capabilities. The continuous unified representation also interfaces naturally with both autoregressive and diffusion-based generative models, and blends with popular multimodal LLM frameworks.

Theoretical Perspectives

The observed synergistic effect between reconstruction and semantic signals provides empirical support for the Platonic Representation Hypothesis, reinforcing the conceptual move toward shared latent spaces for vision-LLMs. It also suggests new research avenues in objective composition and multi-task representation learning, especially for scenarios where interpretability and controllability of generated outputs are critical.

Prospective Extensions

OpenVision 3 opens several pathways for future exploration:

• Extending the approach to higher resolution and video data, leveraging the scalability of ViT and modern VAEs.
• Broadening benchmarks to include more complex visual reasoning and compositional generation tasks.
• Investigating cross-modal transfer, fine-grained controllability, and adversarial robustness in continuous unified spaces.
• Optimizing training efficiency and alignment with larger LLMs, further unlocking performance and generalization in truly universal multimodal AI architectures.

Conclusion

OpenVision 3 advances the state-of-the-art in unified visual encoders, offering a robust, efficient, and versatile solution to the dual challenge of semantic understanding and high-fidelity generation. Its combination of a frozen VAE with a jointly trained ViT in a continuous latent space is empirically validated to outperform or match leading specialized tokenizers and encoders, both in pixel-level and semantic benchmarks. Open-sourcing of code and models promises rapid community adoption and further innovation in unified multimodal learning (2601.15369).