Let ViT Speak: Generative Language-Image Pre-training

Abstract: In this paper, we present \textbf{Gen}erative \textbf{L}anguage-\textbf{I}mage \textbf{P}re-training (GenLIP), a minimalist generative pretraining framework for Vision Transformers (ViTs) designed for multimodal LLMs (MLLMs). To better align vision encoders with the autoregressive nature of LLMs, GenLIP trains a ViT to predict language tokens directly from visual tokens using a standard language modeling objective, without contrastive batch construction or an additional text decoder. This design offers three key advantages: (1) \textbf{Simplicity}: a single transformer jointly models visual and textual tokens; (2) \textbf{Scalability}: it scales effectively with both data and model size; and (3) \textbf{Performance}: it achieves competitive or superior results across diverse multimodal benchmarks. Trained on 8B samples from Recap-DataComp-1B, GenLIP matches or surpasses strong baselines despite using substantially less pretraining data. After continued pretraining on multi-resolution images at native aspect ratios, GenLIP further improves on detail-sensitive tasks such as OCR and chart understanding, making it a strong foundation for vision encoders in MLLMs.

• The paper introduces GenLIP, a minimalist generative framework that directly aligns vision tokens with language output using a unified Transformer.
• It leverages gated attention and multimodal rotary encoding to enhance performance in tasks like caption generation, OCR, and semantic segmentation.
• Empirical results demonstrate superior data efficiency and scaling, outperforming traditional dual-encoder and encoder-decoder baselines.

Minimalist Generative Vision-Language Pretraining with GenLIP

Motivation and Key Architectural Principles

The paper "Let ViT Speak: Generative Language-Image Pre-training" (2605.00809) introduces GenLIP, a minimalist generative vision-language pretraining framework designed to optimize Vision Transformers (ViTs) for multimodal LLMs (MLLMs). GenLIP departs from the prevalent dual-encoder contrastive paradigms and encoder-decoder schemes by leveraging a single unified Transformer architecture that processes visual and textual inputs in a concatenated sequence. The vision encoder is directly trained to predict language tokens that describe visual content using a standard autoregressive language modeling objective, aligning pretraining objectives with downstream generative tasks and eschewing contrastive batch construction or auxiliary text modules.

Figure 1: GenLIP eliminates dual-tower complexity by adopting a single unified Transformer for vision-language pretraining.

The design philosophy emphasizes simplicity, scalability, and alignment with multimodal generative objectives. GenLIP's architecture is characterized by:

• Early fusion of image patch embeddings and text token embeddings in a single sequence.
• Prefix-LM attention, enabling bidirectional attention for image tokens and causal attention for text tokens.
• Multimodal Rotary Position Encoding (MRoPE) for robust position representation without absolute position embeddings.
• Introduction of a gated attention mechanism to suppress attention sink phenomena and promote distributed information flow.

  Figure 2: GenLIP framework overview with unified Transformer architecture, gated attention, Prefix-LM attention mechanism, and MRoPE encoding.

Training Paradigm and Objective Alignment

GenLIP is trained on large-scale image-text corpora (up to 8B pairs from Recap-DataComp-1B), utilizing two pretraining stages: fixed low-resolution foundational training followed by multi-resolution adaptation at native aspect ratios using high-quality caption datasets. The model processes concatenated visual and textual inputs and predicts text tokens conditioned on preceding visual and textual tokens, forming a generative modeling objective directly matched to downstream MLLM generation paradigms.

Unlike prior generative approaches that utilize vision encoders coupled to text decoders, GenLIP optimizes vision representations directly without indirect supervision or architectural redundancy. Gated attention further mitigates attention sink issues, which are pronounced under Prefix-LM attention where the first tokens absorb excessive attention mass, resulting in degraded spatial diversity and less informative visual features.

Figure 3: Attention sink illustrated—without gating, the first token absorbs most attention mass, undermining spatial feature diversity.

Empirical Evaluation and Analysis

Caption Generation and Patch Semantics Alignment

Direct generative evaluation demonstrates GenLIP's ability to produce fluent, semantically accurate descriptions from visual inputs and to align image patches with meaningful language concepts via patch semantics readout. Model scaling and multi-stage pretraining result in improved granularity and semantic correctness.

Figure 4: GenLIP prompt generations demonstrate fluent and grounded image descriptions with increasing accuracy across model scales and stages.

Figure 5: Patch semantics readout reveals spontaneous alignment of local regions to language concepts without explicit supervision.

Data and Model Scalability

Scaling analysis highlights GenLIP's strong data efficiency and favorable model scaling characteristics. When increasing pretraining samples from 1B to 8B, performance improves steeply up to 4B, then plateaus, supporting efficient learning at moderate data volumes. Larger model variants (L/16, So/16, g/16) show consistent gains, with GenLIP-g/16 achieving the highest scores.

Figure 6: Data scaling curves confirm robust performance gains as pretraining sample count increases, especially for OCR, VQA, and Caption tasks.

Native Aspect Ratio Adaptation

Native aspect ratio adaptation further enhances visual representation quality for detail-sensitive tasks such as OCR and chart understanding. Performance is maximized when evaluating at higher resolutions matching image content, confirming the importance of resolution-aware adaptation.

Figure 7: Native aspect adaptation boosts performance on OCR, VQA, and Caption tasks across evaluation resolutions.

Discriminative Representation Quality

Frozen backbone evaluations on ImageNet-1K and ADE20K confirm that GenLIP's visual representations retain strong discriminative power, outperforming pure contrastive methods on semantic segmentation tasks. Gated attention is crucial for mitigating attention sink effects and preserving discriminative quality.

OCR and Patch Semantics Case Studies

Supplementary case studies illustrate GenLIP's emergent OCR ability and detailed patch-language mapping, especially in the large model variants. These qualitative analyses expose remaining failure modes in long number string recognition, precise spatial layout encoding, and tiny text extraction.

Figure 8: Additional OCR generations reflect genLIP's non-trivial, scalable capacity for fine-grained detail extraction.

Figure 9: Further patch semantics readouts reveal increased alignment fidelity in stage-2 models.

Comparative Ablation and Controlled Data Efficiency Evaluation

Controlled comparisons under equal data budgets demonstrate that GenLIP surpasses encoder-decoder and contrastive baselines in multimodal benchmarks, validating its superior data efficiency and representational alignment. Gated attention consistently improves performance across varying data regimes.

Implications and Theoretical Considerations

GenLIP delivers compelling evidence that minimalist generative vision-language pretraining is not only feasible but optimal for modern MLLMs. By directly aligning the pretraining objective to generative downstream tasks and employing a unified architecture, GenLIP achieves both strong numerical results and resource-efficient scalability. The introduction of gated attention addresses theoretical attention sink limitations inherent in Prefix-LM fusion, ensuring robust visual feature utility.

Practical implications include streamlined vision encoder integration in multimodal pipelines, reduced model complexity, and broader applicability across detail-sensitive domains (e.g., visual document understanding, chart analysis, OCR-centric reasoning). Theoretical implications suggest that future research should prioritize unified generative paradigms, further explore attention regulation methods, and quantify scaling limits with increasingly large and high-fidelity datasets.

Conclusion

GenLIP provides a direct, scalable, and data-efficient solution for generative vision-language pretraining, leveraging a unified Transformer architecture and a simple autoregressive language modeling objective. The empirical results and qualitative case analyses indicate that architectural minimalism and objective alignment yield superior multimodal representations, outperforming traditional contrastive and encoder-decoder baselines even with less pretraining data. These findings establish GenLIP as a foundational approach for vision encoder pretraining toward modular, highly performant MLLMs. Further scalability studies and adaptation to advanced MLLM architectures and larger datasets remain crucial future directions.