---
title: 'OpenVision 3: Unified Visual Encoder'
url: https://www.emergentmind.com/topics/openvision-3
type: topic
---

# OpenVision 3: Unified Visual Encoder

OpenVision 3 is a family of unified visual encoders that learns a single continuous visual representation for both image understanding and image generation. Its core design combines a frozen VAE with a trainable ViT encoder and trains the resulting representation to support image reconstruction, contrastive image–text alignment, and image captioning in a shared latent space. In the reported downstream setting, the encoder is frozen and used both as a CLIP-like vision tower in LLaVA‑1.5 and as a tokenizer in the RAE framework, where it is comparable to a standard CLIP vision encoder on multimodal understanding and substantially stronger than a standard CLIP-based encoder for generation, including a reported ImageNet gFID of \(1.89\) versus \(2.54\) for CLIP under RAE [2601.15369].

## 1. Historical position within the OpenVision line

OpenVision 3 follows two earlier stages in the OpenVision program. OpenVision introduced a fully-open family of vision encoders intended as drop-in visual backbones for multimodal LLMs, with complete transparency of data, code, training recipes, and checkpoints, and it emphasized multimodal validation rather than relying only on classic CLIP benchmarks [2505.04601]. OpenVision 2 then simplified that recipe by removing the text encoder and contrastive loss, retaining only captioning loss as a purely generative training signal; it reported lower FLOPs, lower memory, and scalability beyond \(1\) billion parameters while remaining competitive on multimodal benchmarks [2509.01644].

Against that background, OpenVision 3 changes the problem definition. Rather than optimizing only for multimodal understanding, it is presented as a unified visual encoder for both understanding and generation. The stated motivation is that prior systems often separate “semantic” tokenizers from “low-level reconstructable” tokenizers, or rely on unified discrete tokenizers whose quantization introduces quantization error, blur, and artifacts [2601.15369].

This places OpenVision 3 at the intersection of CLIP-style vision-language pretraining, VAE-based latent modeling, and unified multimodal tokenization. A plausible implication is that the central research question is no longer whether a vision encoder is merely a good backbone for a VLM, but whether one continuous latent space can support both semantic alignment and high-fidelity reconstruction without forcing a dual-tokenizer design.

## 2. Core architecture

OpenVision 3 uses a frozen **FLUX.1-dev VAE** as the first stage. For an input image \(x \in \mathbb{R}^{H \times W \times C}\), the VAE encoder produces latents
\[
z_{\text{vae}} = E_{\text{vae}}(x) \in \mathbb{R}^{H/8 \times W/8 \times D_{\text{vae}}}.
\]
The VAE applies an **8×** downsampling factor in height and width, and all subsequent training of the ViT encoder and decoders is carried out in this latent space, with the VAE weights frozen [2601.15369].

These VAE latents are then processed by a ViT encoder \(E_{\text{vit}}\). The latent-space patch size is **2×2**, which, together with the VAE’s 8× downsampling, yields an overall image compression of **16×**. For a **256×256** input image, the VAE output is **32×32**, and patching with **2×2** produces **16×16** tokens:
\[
z_u = E_{\text{vit}}(z_{\text{vae}}) \in \mathbb{R}^{16 \times 16 \times D_u}.
\]
The paper focuses primarily on **OpenVision 3-B**, described as **VAE + ViT-B/2**, and **OpenVision 3-L**, described as **VAE + ViT-L/2** [2601.15369].

The reconstruction branch takes these unified tokens and adds Gaussian noise:
\[
\tilde{z}_u = z_u + T \sigma \epsilon,\quad \epsilon \sim \mathcal{N}(0, I),\ \sigma \sim \text{Uniform}[0, 7].
\]
A ViT decoder with **1×1** patch size maps noisy unified tokens back to VAE latent space, after which the frozen VAE decoder reconstructs the image. In parallel, the same unified tokens are used for contrastive image–text alignment and caption generation [2601.15369].

This architectural choice directly distinguishes OpenVision 3 from the earlier OpenVision models. OpenVision used a ViT-style two-tower CLIP-like model during pretraining, with a vision encoder, a text encoder, and an auxiliary text decoder [2505.04601]. OpenVision 2 removed the text encoder entirely and used a decoder-only generative setup with visual tokens as a prefix [2509.01644]. OpenVision 3 instead adds a generative reconstruction branch centered on VAE latents while retaining both semantic objectives.

## 3. Joint objectives and training procedure

The reconstruction branch is trained with pixel-space, latent-space, and perceptual supervision:
\[
\mathcal{L}_{\text{rec}} =
\|x - \hat{x}\|_1
+ \beta \,\|z_{\text{vae}} - \hat{z}_{\text{vae}}\|_1
+ \alpha\, \text{LPIPS}(x, \hat{x}),
\]
with \(\alpha = 0.5\) and \(\beta = 0.4\) [2601.15369].

The understanding branch combines captioning loss with CLIP-style contrastive learning. The captioning term is
\[
\mathcal{L}_{\text{caption}} = - \mathbb{E}_{(x, y)} \left[ \sum_{t} \log p(y_t \mid y_{<t}, z_u) \right],
\]
while the semantic branch as a whole is defined as
\[
\mathcal{L}_{\text{und}} = \mathcal{L}_{\text{caption}} + \lambda_{\text{contrast}}\,\mathcal{L}_{\text{contrast}}(z_u, z_{\text{txt}})
\]
with \(\lambda_{\text{contrast}} = 1.0\). The full objective is
\[
\mathcal{L}_{\text{overall}} = \omega_{\text{rec}}\,\mathcal{L}_{\text{rec}} + \omega_{\text{und}}\,\mathcal{L}_{\text{und}},
\]
where \(\omega_{\text{rec}} = 0.5\) and \(\omega_{\text{und}} = 1.0\), so semantic loss is weighted **2×** reconstruction loss [2601.15369].

Training uses a progressive two-stage schedule. **Stage 1** pre-trains at **128×128**; **Stage 2** fine-tunes at **224×224** or **256×256**; and the epoch ratio is approximately **10:1**, so most compute is spent at low resolution. The reported optimization settings are a global batch size of **8K** with learning rate \(8\times 10^{-6}\) for pretraining, and a global batch size of **4K** with learning rate \(4\times 10^{-7}\) for fine-tuning, with cosine decay [2601.15369].

The training data is **DataComp**, described as a curated web-scale dataset of image–text pairs, and all images are recaptioned by **LLaVA-Llama-3** to provide higher-quality semantic supervision than raw alt-text [2601.15369]. This continues a broader OpenVision pattern: the original OpenVision emphasized recaptioned public corpora, specifically Recap-DataComp-1B, and attributed multimodal gains to synthetic captions and auxiliary generative supervision [2505.04601]. OpenVision 2 likewise reported that synthetic captions were crucial and that ReCap-DataComp-1B v2 was particularly strong on OCR and MME reasoning [2509.01644].

## 4. Unified representation and ablation evidence

A central claim of OpenVision 3 is that reconstruction-driven and semantics-driven supervision are mutually beneficial in one shared latent space. The paper frames this in terms of a shared underlying reality behind visual and textual descriptions and presents ablations intended to show synergy rather than interference [2601.15369].

When reconstruction loss is removed and the model is trained only with semantic loss, the paper reports that **pixel recon loss** and **latent recon loss** still decrease significantly. The interpretation given is that semantics-based training guides the encoder to structure latent space in a way that makes it easier to reconstruct via the decoder. Conversely, adding reconstruction loss does not hurt caption or contrastive losses, which are described as essentially unaffected [2601.15369].

When understanding loss is removed and the model is trained only with reconstruction, **caption and contrastive losses** still decrease somewhat, with caption loss decreasing more than contrastive. The paper also reports that adding semantic loss further improves reconstruction performance, specifically that pixel reconstruction loss is lower when semantic losses are present [2601.15369].

This is notable in the context of the earlier OpenVision trajectory. OpenVision had already argued that auxiliary decoder training and synthetic captions improved the semantic richness of visual representations [2505.04601], and OpenVision 2 argued that purely generative supervision could preserve multimodal benchmark performance while simplifying optimization [2509.01644]. OpenVision 3 extends that line by asserting not only that generative supervision is useful for understanding, but that low-level reconstruction and high-level semantic alignment can be jointly optimized in a single continuous tokenizer.

A common misconception is that semantic alignment and reconstructability are intrinsically opposed. The ablations are presented precisely to dispute that view: training with only semantic loss still improves reconstruction, while training with only reconstruction still improves semantic metrics to some extent [2601.15369]. This suggests that, at least under the reported setup, the shared latent space is not merely a compromise but a cooperative training substrate.

## 5. Downstream understanding performance

For multimodal understanding, OpenVision 3 is evaluated by replacing CLIP’s vision encoder inside **LLaVA‑1.5**. The tokenizer, consisting of the frozen VAE plus the frozen ViT encoder, produces unified visual tokens that are mapped to the language model embedding space through a projection adapter, following standard LLaVA-style multimodal alignment [2601.15369].

The comparison is explicitly made under **identical token counts**. For the **B-scale** setup, OpenVision 3 uses **VAE + B/2** and is compared with **OpenAI‑CLIP B/16**. For **SeedBench**, the reported values are **62.4** for OpenVision 3 and **62.2** for CLIP; for **POPE**, **83.7** versus **82.9**. For the **L-scale** setup, **VAE + L/2** is compared with **OpenAI‑CLIP L/14**. The reported values are **66.0** versus **65.4** on **MME‑P**, **72.8** versus **73.9** on **MME‑C**, **61.1** versus **60.6** on **SeedBench**, and **85.3** versus **84.7** on **POPE** [2601.15369].

| Scale | CLIP baseline | OpenVision 3 | Selected reported comparison |
|---|---|---|---|
| B | B/16 | VAE + B/2 | SeedBench: 62.2 vs 62.4; POPE: 82.9 vs 83.7 |
| L | L/14 | VAE + L/2 | MME‑P: 65.4 vs 66.0; MME‑C: 73.9 vs 72.8; SeedBench: 60.6 vs 61.1; POPE: 84.7 vs 85.3 |

The reported conclusion is that, as a frozen encoder, OpenVision 3 is on par with or slightly better than CLIP in general multimodal understanding [2601.15369]. This is technically significant because OpenVision 3 is not trained as an understanding-only encoder. Earlier OpenVision work optimized specifically for multimodal backbones in LLaVA-style systems [2505.04601], whereas OpenVision 3 attempts to preserve that level of semantic utility while also functioning as a generative tokenizer.

## 6. Reconstruction, generation, limitations, and research significance

OpenVision 3 is evaluated on reconstruction on **ImageNet** and **COCO** validation sets, with images resized and center-cropped to **256×256**. On **ImageNet**, the reported reconstruction metrics are **PSNR 30.33**, **SSIM 0.885**, **LPIPS 0.061**, and **rFID 0.216**. On **COCO**, the reported values are **PSNR 30.20**, **SSIM 0.893**, **LPIPS 0.058**, and **rFID 1.798** [2601.15369].

These numbers are contrasted with both generation-oriented tokenizers and prior unified tokenizers. Relative to unified tokenizers such as **UniTok**, **OmniTokenizer**, and **Vila-U**, OpenVision 3 is reported to be markedly stronger on reconstruction. Relative to pure VAEs such as **FLUX-VAE**, it remains somewhat behind on the best reconstruction metric, which the paper presents as an expected gap given that FLUX-VAE is optimized purely for reconstruction [2601.15369].

For generation, OpenVision 3 is used as the tokenizer in the **RAE** framework on **ImageNet 256×256**. The reported metrics include **gFID**, **Inception Score**, **Precision**, and **Recall**. The key comparison is between **CLIP + RAE** and **OpenVision 3 + RAE**. The paper reports **CLIP + RAE** at **gFID 2.54**, **IS 256.4**, **Precision 0.80**, and **Recall 0.54**, while **OpenVision 3** in its best improved RAE setup reports **gFID 1.89**, **IS 289.2**, **Precision 0.84**, and **Recall 0.59** [2601.15369].

| Tokenizer | Generator | gFID ↓ | IS ↑ |
|---|---|---:|---:|
| CLIP | RAE | 2.54 | 256.4 |
| OpenVision 3 | RAE | 2.44 | 262.2 |
| OpenVision 3 | RAE (improved) | 1.89 | 289.2 |

This result underwrites the main claim of OpenVision 3: a unified tokenizer can retain CLIP-level semantics while becoming substantially more suitable for generative modeling than a standard CLIP-based encoder. It also reframes a recurring debate in multimodal representation learning. OpenVision 2 had argued that CLIP-style contrastive objectives are not indispensable for strong multimodal LLM performance [2509.01644]. OpenVision 3 does not abandon contrastive learning; instead, it places it alongside captioning and reconstruction in a single latent space. A plausible implication is that the controversy is shifting from “contrastive versus generative” toward “how to compose semantic and reconstructive supervision without sacrificing either regime.”

The paper also implies several limitations. Reconstruction remains slightly behind the best pure VAEs, evaluation is centered on **ImageNet 256×256** for generation and a standard but limited set of VQA-style benchmarks for understanding, and the design is tied to a specific **FLUX.1** VAE and ViT-based stack [2601.15369]. These constraints leave open whether the same unified principle will scale to richer generative settings, broader vision tasks, or alternative latent backbones. Even so, OpenVision 3 establishes a fully specified recipe for a single continuous visual tokenizer that is simultaneously reconstructable, semantically aligned, and directly usable in both LLaVA-style understanding systems and RAE-style generative models [2601.15369].

Source: https://www.emergentmind.com/topics/openvision-3