OpenVision 3: Unified Visual Encoder

• OpenVision 3 is a unified visual encoder that integrates a frozen VAE with a trainable ViT encoder to create a continuous latent space for both image generation and understanding.
• It simultaneously optimizes pixel-level reconstruction and semantic alignment through joint objectives, yielding improved performance on benchmarks like ImageNet and COCO.
• The dual training regime leverages contrastive learning and captioning losses alongside generative supervision to outperform traditional CLIP-based encoders in multimodal tasks.

OpenVision 3 is a family of unified visual encoders that learns a single continuous visual representation for both image understanding and image generation. Its core design combines a frozen VAE with a trainable ViT encoder and trains the resulting representation to support image reconstruction, contrastive image–text alignment, and image captioning in a shared latent space. In the reported downstream setting, the encoder is frozen and used both as a CLIP-like vision tower in LLaVA‑1.5 and as a tokenizer in the RAE framework, where it is comparable to a standard CLIP vision encoder on multimodal understanding and substantially stronger than a standard CLIP-based encoder for generation, including a reported ImageNet gFID of $1.89$ versus $2.54$ for CLIP under RAE (Zhang et al., 21 Jan 2026).

1. Historical position within the OpenVision line

OpenVision 3 follows two earlier stages in the OpenVision program. OpenVision introduced a fully-open family of vision encoders intended as drop-in visual backbones for multimodal LLMs, with complete transparency of data, code, training recipes, and checkpoints, and it emphasized multimodal validation rather than relying only on classic CLIP benchmarks (Li et al., 7 May 2025). OpenVision 2 then simplified that recipe by removing the text encoder and contrastive loss, retaining only captioning loss as a purely generative training signal; it reported lower FLOPs, lower memory, and scalability beyond $1$ billion parameters while remaining competitive on multimodal benchmarks (Liu et al., 1 Sep 2025).

Against that background, OpenVision 3 changes the problem definition. Rather than optimizing only for multimodal understanding, it is presented as a unified visual encoder for both understanding and generation. The stated motivation is that prior systems often separate “semantic” tokenizers from “low-level reconstructable” tokenizers, or rely on unified discrete tokenizers whose quantization introduces quantization error, blur, and artifacts (Zhang et al., 21 Jan 2026).

This places OpenVision 3 at the intersection of CLIP-style vision-language pretraining, VAE-based latent modeling, and unified multimodal tokenization. A plausible implication is that the central research question is no longer whether a vision encoder is merely a good backbone for a VLM, but whether one continuous latent space can support both semantic alignment and high-fidelity reconstruction without forcing a dual-tokenizer design.

2. Core architecture

OpenVision 3 uses a frozen FLUX.1-dev VAE as the first stage. For an input image $x \in \mathbb{R}^{H \times W \times C}$, the VAE encoder produces latents

$$z_{\text{vae}} = E_{\text{vae}}(x) \in \mathbb{R}^{H/8 \times W/8 \times D_{\text{vae}}}.$$

The VAE applies an 8× downsampling factor in height and width, and all subsequent training of the ViT encoder and decoders is carried out in this latent space, with the VAE weights frozen (Zhang et al., 21 Jan 2026).

These VAE latents are then processed by a ViT encoder $E_{\text{vit}}$. The latent-space patch size is 2×2, which, together with the VAE’s 8× downsampling, yields an overall image compression of 16×. For a 256×256 input image, the VAE output is 32×32, and patching with 2×2 produces 16×16 tokens: $$z_u = E_{\text{vit}}(z_{\text{vae}}) \in \mathbb{R}^{16 \times 16 \times D_u}.$$ The paper focuses primarily on OpenVision 3-B, described as VAE + ViT-B/2, and OpenVision 3-L, described as VAE + ViT-L/2 (Zhang et al., 21 Jan 2026).

The reconstruction branch takes these unified tokens and adds Gaussian noise: $$\tilde{z}_u = z_u + T \sigma \epsilon,\quad \epsilon \sim \mathcal{N}(0, I),\ \sigma \sim \text{Uniform}[0, 7].$$ A ViT decoder with 1×1 patch size maps noisy unified tokens back to VAE latent space, after which the frozen VAE decoder reconstructs the image. In parallel, the same unified tokens are used for contrastive image–text alignment and caption generation (Zhang et al., 21 Jan 2026).

This architectural choice directly distinguishes OpenVision 3 from the earlier OpenVision models. OpenVision used a ViT-style two-tower CLIP-like model during pretraining, with a vision encoder, a text encoder, and an auxiliary text decoder (Li et al., 7 May 2025). OpenVision 2 removed the text encoder entirely and used a decoder-only generative setup with visual tokens as a prefix (Liu et al., 1 Sep 2025). OpenVision 3 instead adds a generative reconstruction branch centered on VAE latents while retaining both semantic objectives.

3. Joint objectives and training procedure

The reconstruction branch is trained with pixel-space, latent-space, and perceptual supervision: $$\mathcal{L}_{\text{rec}} = \|x - \hat{x}\|_1 + \beta \,\|z_{\text{vae}} - \hat{z}_{\text{vae}}\|_1 + \alpha\, \text{LPIPS}(x, \hat{x}),$$ with $\alpha = 0.5$ and $2.54$0 (Zhang et al., 21 Jan 2026).

The understanding branch combines captioning loss with CLIP-style contrastive learning. The captioning term is

$2.54$1

while the semantic branch as a whole is defined as

$2.54$2

with $2.54$3. The full objective is

$2.54$4

where $2.54$5 and $2.54$6, so semantic loss is weighted 2× reconstruction loss (Zhang et al., 21 Jan 2026).

Training uses a progressive two-stage schedule. Stage 1 pre-trains at 128×128; Stage 2 fine-tunes at 224×224 or 256×256; and the epoch ratio is approximately 10:1, so most compute is spent at low resolution. The reported optimization settings are a global batch size of 8K with learning rate $2.54$7 for pretraining, and a global batch size of 4K with learning rate $2.54$8 for fine-tuning, with cosine decay (Zhang et al., 21 Jan 2026).

The training data is DataComp, described as a curated web-scale dataset of image–text pairs, and all images are recaptioned by LLaVA-Llama-3 to provide higher-quality semantic supervision than raw alt-text (Zhang et al., 21 Jan 2026). This continues a broader OpenVision pattern: the original OpenVision emphasized recaptioned public corpora, specifically Recap-DataComp-1B, and attributed multimodal gains to synthetic captions and auxiliary generative supervision (Li et al., 7 May 2025). OpenVision 2 likewise reported that synthetic captions were crucial and that ReCap-DataComp-1B v2 was particularly strong on OCR and MME reasoning (Liu et al., 1 Sep 2025).

4. Unified representation and ablation evidence

A central claim of OpenVision 3 is that reconstruction-driven and semantics-driven supervision are mutually beneficial in one shared latent space. The paper frames this in terms of a shared underlying reality behind visual and textual descriptions and presents ablations intended to show synergy rather than interference (Zhang et al., 21 Jan 2026).

When reconstruction loss is removed and the model is trained only with semantic loss, the paper reports that pixel recon loss and latent recon loss still decrease significantly. The interpretation given is that semantics-based training guides the encoder to structure latent space in a way that makes it easier to reconstruct via the decoder. Conversely, adding reconstruction loss does not hurt caption or contrastive losses, which are described as essentially unaffected (Zhang et al., 21 Jan 2026).

When understanding loss is removed and the model is trained only with reconstruction, caption and contrastive losses still decrease somewhat, with caption loss decreasing more than contrastive. The paper also reports that adding semantic loss further improves reconstruction performance, specifically that pixel reconstruction loss is lower when semantic losses are present (Zhang et al., 21 Jan 2026).

This is notable in the context of the earlier OpenVision trajectory. OpenVision had already argued that auxiliary decoder training and synthetic captions improved the semantic richness of visual representations (Li et al., 7 May 2025), and OpenVision 2 argued that purely generative supervision could preserve multimodal benchmark performance while simplifying optimization (Liu et al., 1 Sep 2025). OpenVision 3 extends that line by asserting not only that generative supervision is useful for understanding, but that low-level reconstruction and high-level semantic alignment can be jointly optimized in a single continuous tokenizer.

A common misconception is that semantic alignment and reconstructability are intrinsically opposed. The ablations are presented precisely to dispute that view: training with only semantic loss still improves reconstruction, while training with only reconstruction still improves semantic metrics to some extent (Zhang et al., 21 Jan 2026). This suggests that, at least under the reported setup, the shared latent space is not merely a compromise but a cooperative training substrate.

5. Downstream understanding performance

For multimodal understanding, OpenVision 3 is evaluated by replacing CLIP’s vision encoder inside LLaVA‑1.5. The tokenizer, consisting of the frozen VAE plus the frozen ViT encoder, produces unified visual tokens that are mapped to the LLM embedding space through a projection adapter, following standard LLaVA-style multimodal alignment (Zhang et al., 21 Jan 2026).

The comparison is explicitly made under identical token counts. For the B-scale setup, OpenVision 3 uses VAE + B/2 and is compared with OpenAI‑CLIP B/16. For SeedBench, the reported values are 62.4 for OpenVision 3 and 62.2 for CLIP; for POPE, 83.7 versus 82.9. For the L-scale setup, VAE + L/2 is compared with OpenAI‑CLIP L/14. The reported values are 66.0 versus 65.4 on MME‑P, 72.8 versus 73.9 on MME‑C, 61.1 versus 60.6 on SeedBench, and 85.3 versus 84.7 on POPE (Zhang et al., 21 Jan 2026).

Scale	CLIP baseline	OpenVision 3	Selected reported comparison
B	B/16	VAE + B/2	SeedBench: 62.2 vs 62.4; POPE: 82.9 vs 83.7
L	L/14	VAE + L/2	MME‑P: 65.4 vs 66.0; MME‑C: 73.9 vs 72.8; SeedBench: 60.6 vs 61.1; POPE: 84.7 vs 85.3

The reported conclusion is that, as a frozen encoder, OpenVision 3 is on par with or slightly better than CLIP in general multimodal understanding (Zhang et al., 21 Jan 2026). This is technically significant because OpenVision 3 is not trained as an understanding-only encoder. Earlier OpenVision work optimized specifically for multimodal backbones in LLaVA-style systems (Li et al., 7 May 2025), whereas OpenVision 3 attempts to preserve that level of semantic utility while also functioning as a generative tokenizer.

6. Reconstruction, generation, limitations, and research significance

OpenVision 3 is evaluated on reconstruction on ImageNet and COCO validation sets, with images resized and center-cropped to 256×256. On ImageNet, the reported reconstruction metrics are PSNR 30.33, SSIM 0.885, LPIPS 0.061, and rFID 0.216. On COCO, the reported values are PSNR 30.20, SSIM 0.893, LPIPS 0.058, and rFID 1.798 (Zhang et al., 21 Jan 2026).

These numbers are contrasted with both generation-oriented tokenizers and prior unified tokenizers. Relative to unified tokenizers such as UniTok, OmniTokenizer, and Vila-U, OpenVision 3 is reported to be markedly stronger on reconstruction. Relative to pure VAEs such as FLUX-VAE, it remains somewhat behind on the best reconstruction metric, which the paper presents as an expected gap given that FLUX-VAE is optimized purely for reconstruction (Zhang et al., 21 Jan 2026).

For generation, OpenVision 3 is used as the tokenizer in the RAE framework on ImageNet 256×256. The reported metrics include gFID, Inception Score, Precision, and Recall. The key comparison is between CLIP + RAE and OpenVision 3 + RAE. The paper reports CLIP + RAE at gFID 2.54, IS 256.4, Precision 0.80, and Recall 0.54, while OpenVision 3 in its best improved RAE setup reports gFID 1.89, IS 289.2, Precision 0.84, and Recall 0.59 (Zhang et al., 21 Jan 2026).

Tokenizer	Generator	gFID ↓	IS ↑
CLIP	RAE	2.54	256.4
OpenVision 3	RAE	2.44	262.2
OpenVision 3	RAE (improved)	1.89	289.2

This result underwrites the main claim of OpenVision 3: a unified tokenizer can retain CLIP-level semantics while becoming substantially more suitable for generative modeling than a standard CLIP-based encoder. It also reframes a recurring debate in multimodal representation learning. OpenVision 2 had argued that CLIP-style contrastive objectives are not indispensable for strong multimodal LLM performance (Liu et al., 1 Sep 2025). OpenVision 3 does not abandon contrastive learning; instead, it places it alongside captioning and reconstruction in a single latent space. A plausible implication is that the controversy is shifting from “contrastive versus generative” toward “how to compose semantic and reconstructive supervision without sacrificing either regime.”

The paper also implies several limitations. Reconstruction remains slightly behind the best pure VAEs, evaluation is centered on ImageNet 256×256 for generation and a standard but limited set of VQA-style benchmarks for understanding, and the design is tied to a specific FLUX.1 VAE and ViT-based stack (Zhang et al., 21 Jan 2026). These constraints leave open whether the same unified principle will scale to richer generative settings, broader vision tasks, or alternative latent backbones. Even so, OpenVision 3 establishes a fully specified recipe for a single continuous visual tokenizer that is simultaneously reconstructable, semantically aligned, and directly usable in both LLaVA-style understanding systems and RAE-style generative models (Zhang et al., 21 Jan 2026).