UniLIP: Unified CLIP Multimodal Framework

• UniLIP is a unified multimodal framework that adapts CLIP into a continuous image tokenizer to support reconstruction, generation, and editing, while preserving strong image comprehension.
• It employs a novel two-stage reconstruction training with self-distillation and a dual-condition architecture that bridges an MLLM with a diffusion transformer for enhanced multimodal synthesis.
• Empirical evaluations demonstrate competitive performance with metrics like rFID 0.32, PSNR 24.84, and strong understanding scores, validating its integrated approach over traditional methods.

Searching arXiv for “UniLIP” and closely related entries to ground the article in current papers. UniLIP is a unified multimodal framework that adapts CLIP from an understanding-oriented visual encoder into a reconstructable continuous image tokenizer that supports multimodal understanding, reconstruction, generation, and editing (Tang et al., 31 Jul 2025). In the terminology of the paper, its central aim is to remove the trade-off faced by earlier CLIP-based unified methods: quantization-based systems degrade comprehension, while frozen-CLIP-plus-diffusion-decoder systems often reconstruct images inconsistently with the source (Tang et al., 31 Jul 2025). UniLIP addresses this with a two-stage reconstruction adaptation procedure and a dual-condition architecture linking an MLLM to a diffusion transformer, while preserving the original comprehension performance of the CLIP backbone (Tang et al., 31 Jul 2025).

1. Nomenclature and scope

The name “UniLIP” most directly corresponds to “UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing” (Tang et al., 31 Jul 2025). That paper consistently presents UniLIP as a method for extending CLIP to reconstruction, generation, and editing, thereby building a unified tokenizer on top of CLIP’s comprehension capabilities (Tang et al., 31 Jul 2025).

The term can be confused with two neighboring lines of work. First, “ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding” is a distinct method for 3D point-cloud representation learning rather than a unified image tokenizer (Xue et al., 2022). ULIP learns a unified representation of images, texts, and 3D point clouds by aligning a trainable 3D encoder to a frozen pretrained image-text space, using automatically synthesized triplets from ShapeNet55 (Xue et al., 2022). Second, “Uni-Mlip: Unified Self-supervision for Medical Vision Language Pre-training” is a medical vision-language pretraining framework with unified self-supervision objectives, but it is not presented as “UniLIP” and does not claim to be the same model (Bawazir et al., 2024).

A plausible implication is that “UniLIP” belongs to a broader family of “unified” multimodal methods built around CLIP-like alignment, but in the specific literature provided here, the canonical referent is the 2025 UniLiP paper on unified multimodal understanding, generation, and editing (Tang et al., 31 Jul 2025).

2. Conceptual motivation and problem formulation

UniLIP is motivated by a division in multimodal systems between image understanding models and image generation models. In the formulation of the paper, image understanding systems are typically based on CLIP-like encoders aligned with LLMs, while image generation systems typically rely on VAEs, VQ-based tokenizers, and diffusion models (Tang et al., 31 Jul 2025). This separation makes it difficult to build a single model that both interprets and produces images effectively.

The paper frames prior CLIP-based unified methods as falling into two problematic categories. Quantization-based approaches such as VILA-U, TokenFlow, TokLIP, and ILLUME discretize CLIP features into tokens; according to the paper, this introduces information loss and pulls the representation away from the original continuous CLIP space, yielding weaker understanding than the original CLIP backbone (Tang et al., 31 Jul 2025). The second category, represented by Emu2 and inherited by methods like BLIP3-o for reconstruction, keeps CLIP frozen and uses CLIP features only as conditioning for a diffusion decoder; this preserves CLIP semantics but can yield reconstructions with altered fine details, including wrong hole counts, wrong positions, and missing text (Tang et al., 31 Jul 2025).

UniLIP’s core claim is that CLIP itself should be adapted to encode more reconstructable detail while remaining anchored to its original semantic distribution (Tang et al., 31 Jul 2025). This yields a model that is simultaneously a strengthened CLIP-style image tokenizer and a unified multimodal architecture for understanding, text-to-image generation, and image editing (Tang et al., 31 Jul 2025). The paper implements UniLIP on top of InternViT from InternVL3-1B, with the resulting visual representation intended to remain strong for understanding while becoming directly usable for reconstruction and as the visual bridge for generation and editing (Tang et al., 31 Jul 2025).

3. Architecture and representation design

UniLIP contains two intertwined subsystems: a reconstruction subsystem and a generation/editing subsystem (Tang et al., 31 Jul 2025).

For reconstruction, the paper pairs the CLIP encoder with a lightweight pixel decoder $D_{\text{pix}}$ and a projection module $h_{\phi}$, giving the reconstruction equation

$$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$

The input image is $I \in \mathbb{R}^{H \times W \times 3}$, the CLIP feature map is denoted

$$F_{\text{clip}} \in \mathbb{R}^{\frac{H}{p} \times \frac{W}{p} \times d},$$

and $h_{\phi}$ maps CLIP features into the channel space required by the pixel decoder (Tang et al., 31 Jul 2025). In practice, $h_{\phi}$ is implemented with several MLPs (Tang et al., 31 Jul 2025). This subsystem is the mechanism by which CLIP becomes a reconstructable tokenizer.

For generation and editing, UniLIP couples an MLLM with a diffusion transformer and modifies the bridge between them (Tang et al., 31 Jul 2025). The major components named in the paper are InternVL3-1B as the MLLM, the adapted UniLIP encoder initialized from InternViT as the visual encoder/tokenizer, a 6-layer transformer connector with the same structure as the InternVL3-1B LLM, $N=256$ learnable queries, SANA-0.6B as the diffusion transformer, and a DC-AE decoder as the pixel decoder (Tang et al., 31 Jul 2025).

The architectural novelty is the dual-condition mechanism. Instead of conditioning the diffusion transformer only through compressed query embeddings, UniLIP uses both learnable queries and the last-layer multimodal hidden states of the MLLM as joint conditions (Tang et al., 31 Jul 2025). During generation, the last-layer multimodal embeddings and query embeddings are concatenated, passed through the connector, and used as the condition for the diffusion transformer (Tang et al., 31 Jul 2025). During editing, the same mechanism is used, but the MLLM input includes a reference image together with the editing instruction, allowing reference-image detail to survive in the last-layer hidden states rather than being compressed entirely into a fixed query set (Tang et al., 31 Jul 2025).

This suggests that UniLIP treats compressed reasoning signals and richer multimodal context as complementary conditioning channels. The paper explicitly associates query embeddings with high-level reasoning and the last-layer hidden states with richer information preservation, especially for editing consistency (Tang et al., 31 Jul 2025).

4. Training procedure and objectives

The most important technical contribution in the paper is a two-stage reconstruction training scheme combined with self-distillation (Tang et al., 31 Jul 2025).

In stage 1, CLIP is frozen and only the projection $h_{\phi}$ and pixel decoder $D_{\text{pix}}$ are trained. The objective is

$h_{\phi}$0

Here, $h_{\phi}$1 is a pixel-wise mean squared error reconstruction loss and $h_{\phi}$2 is the LPIPS perceptual loss (Tang et al., 31 Jul 2025). The purpose of this stage is to extract whatever weak reconstruction signal is already present in frozen CLIP features without modifying their distribution (Tang et al., 31 Jul 2025).

In stage 2, the CLIP encoder is unfrozen and adapted for reconstruction, but a self-distillation penalty is added: $h_{\phi}$3 In this expression, $h_{\phi}$4 denotes features from the original frozen CLIP teacher, $h_{\phi}$5 denotes features from the fine-tuned CLIP student, and $h_{\phi}$6 is the weight on the self-distillation term (Tang et al., 31 Jul 2025). The paper identifies this self-distillation term as the mechanism that anchors the student to the original semantic embedding distribution while allowing it to encode more fine-grained detail for reconstruction (Tang et al., 31 Jul 2025).

The training schedule is specified in concrete terms. Reconstruction training uses the BLIP3-o pretraining dataset, batch size 64, and a fixed learning rate of $h_{\phi}$7 on 4 A100 GPUs (Tang et al., 31 Jul 2025). Stage 1 runs for 300k steps at $h_{\phi}$8 resolution (Tang et al., 31 Jul 2025). Stage 2 runs for 400k steps total, with the first 300k steps at $h_{\phi}$9 and the final 100k steps at $$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$0; during stage 2, the CLIP learning rate is reduced to $$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$1 (Tang et al., 31 Jul 2025).

After reconstructable UniLIP features are obtained, generation and editing are trained with a three-stage schedule. Stage A freezes the MLLM and diffusion transformer and trains only the connector for 50k steps on generation data (Tang et al., 31 Jul 2025). Stage B trains the connector and diffusion transformer together for 200k steps on both generation and editing data (Tang et al., 31 Jul 2025). Stage C performs instruction tuning on both generation and editing tasks for 20k steps (Tang et al., 31 Jul 2025). All three stages use batch size 512 and cosine learning-rate decay from $$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$2 to $$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$3 (Tang et al., 31 Jul 2025).

The data mixture is also specified. For image generation, the paper uses BLIP3-o’s training data, including CC12M, SAM-1B, JourneyDB, 27M samples with detailed captions generated by Qwen2.5-VL-7B, and 5M samples from short CC12M captions (Tang et al., 31 Jul 2025). For generation instruction tuning, it uses 60K high-quality image-text pairs generated using GPT-4o prompting and synthesized images (Tang et al., 31 Jul 2025). For editing, it uses about 1M editing samples mainly from ImgEdit and SEED-X, collected from UniWorld-V1, plus 46K instruction-tuning editing samples from ShareGPT-4o-Image (Tang et al., 31 Jul 2025).

5. Empirical performance

The paper evaluates UniLIP across understanding, reconstruction, text-to-image generation, and image editing (Tang et al., 31 Jul 2025). The reported results are central to its claim that a continuous CLIP-based unified tokenizer can preserve comprehension while becoming competitive for creation tasks.

On understanding benchmarks, UniLIP-1B achieves MME-P 1499, MMBench 72.6, MMMU 43.3, MM-Vet 59.4, SEED 71.0, AI2D 70.7, and MMVP 68.7 (Tang et al., 31 Jul 2025). Compared with InternVL3-1B, the paper reports that understanding is preserved essentially intact and that some detailed-perception tasks improve, citing InternVL3-1B scores of MME-P 1492, MMBench 72.6, MMVP 67.3, and AI2D 69.4 (Tang et al., 31 Jul 2025).

On reconstruction, evaluated on the ImageNet 50k validation set, UniLIP reports at $$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$4 an rFID of 0.79, PSNR of 22.99, and SSIM of 0.747 (Tang et al., 31 Jul 2025). At $$\hat{I} = D_{\text{pix}}(h_{\phi}(\text{CLIP}(I))).$$5, compared with Emu2, UniLIP reports rFID 0.32, PSNR 24.84, and SSIM 0.792, while Emu2 reports rFID 3.27, PSNR 13.49, and SSIM 0.423 (Tang et al., 31 Jul 2025). The paper interprets these as evidence that direct adaptation of CLIP for reconstruction yields more faithful reconstructions than frozen-CLIP-conditioned diffusion decoders (Tang et al., 31 Jul 2025).

On GenEval, UniLIP-1B reports Single Obj. 1.00, Two Obj. 0.93, Counting 0.83, Colors 0.90, Position 0.80, Color Attr. 0.76, and Overall 0.87 (Tang et al., 31 Jul 2025). On WISE, it reports Cultural 0.51, Time 0.54, Space 0.68, Biology 0.47, Physics 0.57, Chemistry 0.42, and Overall 0.53 (Tang et al., 31 Jul 2025). On ImgEdit-Bench, it reports Add 3.84, Adj. 3.77, Ext. 1.90, Repl. 4.33, Rmv. 3.52, Bkg. 3.67, Style 4.75, Hyb. 2.51, Act. 4.31, and Overall 3.62 (Tang et al., 31 Jul 2025).

The following table condenses the principal benchmark figures explicitly given in the paper.

Capability	Benchmark	Reported UniLIP result
Understanding	MMMU	43.3
Reconstruction	ImageNet 50k, 448×448	rFID 0.32, PSNR 24.84, SSIM 0.792
Generation	GenEval	0.87 overall
Knowledge-grounded generation	WISE	0.53 overall
Editing	ImgEdit-Bench	3.62 overall

A plausible implication is that UniLIP’s reported strength is not in any single metric alone, but in maintaining a high-level CLIP/MLLM understanding profile while also becoming directly reconstructable and competitive on generation and editing. That combination is the core empirical claim of the paper (Tang et al., 31 Jul 2025).

6. Ablations, interpretation, and relation to adjacent methods

The ablations emphasize the role of self-distillation and the dual-condition bridge. In the reconstruction ablation, direct fine-tuning of CLIP for reconstruction yields strong reconstruction PSNR but severely degrades understanding, with MMMU dropping from about 43.4 to 15.2 (Tang et al., 31 Jul 2025). The full method, combining staged decoder-first training, self-distillation, and lower CLIP learning rate, yields MMMU 43.3 and PSNR 24.84 (Tang et al., 31 Jul 2025). The paper further states that removing self-distillation drops MMMU by 6.8 points, identifying it as especially important for preserving CLIP semantics (Tang et al., 31 Jul 2025).

The dual-condition ablation compares three conditioning settings: last-layer hidden states only, query embeddings only, and both together (Tang et al., 31 Jul 2025). The reported results are last-layer only with WISE 0.44 and ImgEdit 3.37, query only with WISE 0.50 and ImgEdit 3.12, and dual condition with WISE 0.53 and ImgEdit 3.62 (Tang et al., 31 Jul 2025). This supports the paper’s interpretation that query embeddings help reasoning-heavy generation, while last-layer embeddings help information preservation in editing (Tang et al., 31 Jul 2025).

UniLIP is explicitly situated against several prior families. Relative to VQ/VAE-based unified multimodal models such as Chameleon, Emu3, Show-O, and Transfusion, the paper argues that their latent spaces are not as semantically rich as CLIP, so comprehension suffers (Tang et al., 31 Jul 2025). Relative to quantized-CLIP systems such as VILA-U, TokenFlow, TokLIP, and ILLUME, UniLIP preserves continuous CLIP features rather than discretizing them (Tang et al., 31 Jul 2025). Relative to Emu2 and BLIP3-o, UniLIP avoids relying on a separate diffusion reconstruction process from frozen CLIP and instead teaches CLIP itself to encode more detail, decoding directly with a lightweight pixel decoder (Tang et al., 31 Jul 2025). Relative to MetaQuery-like bridges, UniLIP adopts learnable query conditioning but argues that fixed-length queries alone are insufficient for editing, motivating the dual-condition extension (Tang et al., 31 Jul 2025). Relative to UniWorld-V1, the paper positions UniLIP as using one UniLIP feature space for both semantic understanding and reconstructable visual conditioning (Tang et al., 31 Jul 2025).

In relation to the two neighboring “Uni-” papers in the record, UniLIP differs both in modality and objective. ULIP targets 3D understanding by aligning a 3D encoder with a pretrained image-text space, using point clouds, rendered images, and prompted text descriptions (Xue et al., 2022). Uni-Mlip targets medical vision-language pretraining by combining cross-modality, uni-modality, and fused-modality self-supervision, with a CLIP-style medical dual encoder, multimodal transformer fusion, and a medical-domain-specific treatment of batch normalization for SimCLR-style image-only SSL (Bawazir et al., 2024). These works share a family resemblance in unified pretraining design, but they address different data regimes, modalities, and downstream tasks.

7. Significance and limitations

UniLIP’s significance lies in its claim that continuous CLIP features can serve not only as strong representations for understanding but also as a practical latent space for reconstruction, generation, and editing, provided that CLIP is adapted carefully rather than quantized or left entirely frozen (Tang et al., 31 Jul 2025). The paper’s bottom-line formulation is that UniLIP expands the application scope of continuous CLIP features from “best for understanding” to “competitive for understanding, reconstruction, generation, and editing alike” (Tang et al., 31 Jul 2025).

Several limitations are visible in the paper’s own framing. The method still depends on a fairly complex multi-stage pipeline, and the diffusion training component is not analyzed as deeply as the reconstruction adaptation component (Tang et al., 31 Jul 2025). The demonstrations are mainly at the 1B-scale MLLM setting, and some benchmarks still trail proprietary frontier systems such as GPT-4o (Tang et al., 31 Jul 2025). The paper also makes heavy use of large mixed public datasets and staged freezing/unfreezing schedules, which suggests substantial training complexity even if the unified tokenizer principle is conceptually cleaner than earlier CLIP-based alternatives (Tang et al., 31 Jul 2025).

Taken together, the available literature suggests a useful disambiguation. If “UniLIP” denotes a specific named model, it is the CLIP-based unified multimodal framework of 2025 (Tang et al., 31 Jul 2025). If used more loosely as a label for unified language-image pretraining, the term sits alongside but should not be conflated with ULIP’s 3D alignment framework (Xue et al., 2022) or Uni-Mlip’s medical vision-language self-supervision framework (Bawazir et al., 2024).