ViZer: Unified Vision-Language Alignment

• The paper demonstrates that a lightweight mapper with a cosine contrastive loss enhances visual grounding in caption generation without annotated data.
• ViZer uses a modular, self-aligning framework to bridge image and text embeddings, leading to more context-aware and accurate captions.
• Empirical results indicate reduced hallucinations, improved semantic clustering, and scalability to broader vision–language tasks.

Unified Vision-Language Alignment for Zero-Label Enhancement (ViZer) is a training framework that enables vision-LLMs (VLMs) to improve the quality and grounding of generated captions without the need for any annotated text labels. By aligning vision and language representation features in the embedding space, ViZer provides a practical mechanism for zero-label learning in image captioning and stands as a starting point for scalable and adaptable zero-label solutions across broader vision-language tasks (Byun et al., 14 Oct 2025).

1. Conceptual Foundation

ViZer directly addresses the scalability constraints of prevailing VLMs, which depend heavily on large-scale manually annotated image–caption datasets for pretraining. These constraints limit deployment in open-world or rapidly evolving domains where labeled data is scarce. The central tenet of ViZer is to actively and continuously align the latent feature spaces of vision and language modalities, leveraging only unlabeled images for enhancement training. Rather than requiring paired image–text supervision, ViZer introduces a lightweight, modular mapper network that sits between the visual feature extractor and the language modeling head or decoder. This mapper is optimized to bridge the gap between visual and language embeddings via a contrastive objective, providing a foundation for label-free improvement in generation quality.

2. Methodological Framework

The core methodology of ViZer consists of the following steps:

1. Feature Extraction: The vision encoder $\mathcal{V}_e$ computes visual features $F_I$ for input image $I$:

$F_I = \mathcal{V}_e(I)$

1. Language Side Representation: The text encoder (and tokenizer, $E(\cdot)$) produces token-level embeddings, but in zero-label mode, these are not conditioned on training-time ground-truth captions.
2. Mapper Network: A mapper function $M_\phi(\cdot)$, typically realized as a small MLP stacked over intermediate transformer layers, transforms the latent text representation to align with the visual features:

$M_\phi(\cdot) = h_z(f_y(\cdot))$

where $h_z(\cdot)$ is a multi-layer perceptron and $f_y$ are intermediate transformer layers prior to the language modeling head.

1. Training Objective: The principal loss is a cosine contrastive loss operating over the aligned latent spaces:

$\mathcal{L}_{\text{ViZer}} = 1 - \cos(F_I, F_T)$

Here, $F_T = M_\phi(\text{language tokens})$ represents the mapped (latent) text embedding.

1. Training Variants:
   • ViZerGT: When ground-truth captions are available, the mapper can be optionally initialized or refined via supervised alignment.
   • ViZerG: In full zero-label mode, captions are generated (e.g., from the model itself using a seed prompt $P$), and mappings are optimized solely by self-alignment between generated captions and the image.

This training process allows the VLM to iteratively co-adapt its representations such that image features and generated captions become semantically and structurally closer in the embedding space, with no external textual supervision required.

3. Advantages and Distinguishing Features

ViZer's zero-label enhancement methodology yields several empirically verified advantages:

• Zero-Annotated Data Requirement: No human-annotated or synthetic captions are needed during enhancement, enabling efficient use of the vast pool of unlabeled images.
• Grounded Caption Improvement: Captions generated after ViZer enhancement exhibit better grounding to image content, capturing scene-specific details, object interactions, textures, and relationships that may be underrepresented or omitted in conventional supervised baselines.
• Modular Integration: The mapper is implemented as a lightweight module. Existing VLMs, such as SmolVLM-Base and Qwen2-VL, can be equipped with ViZer without modifying or retraining large-scale encoder/decoder components.
• Reduced Annotation Bias and Hallucination: ViZer-generated captions demonstrate lower susceptibility to spurious hallucinations (introducing details not in the image) and are less biased toward stylistic conventions imposed by human-labeled reference corpora.
• Continuous Adaptation Potential: Because ViZer operates over unpaired, unlabeled images, it is well-suited to continuous learning in changing environments or when extending to new domains.

4. Evaluation Methodology and Metrics

ViZer-enhanced models are evaluated using a combination of standard automatic metrics and qualitative assessments:

• CIDEr: TF-IDF weighted n-gram consensus-based similarity to human references, appropriate for capturing informativeness and relevance in captions.
• BERTScore: Semantic similarity between contextual embeddings of generated captions and references, more robust to lexical variation than n-gram matching.
• CLIPScore: Cosine similarity in a multimodal embedding space computed via a pretrained vision-LLM, providing a reference-free proxy for image-text alignment.
• BLEU, ROUGE-L: Additional n-gram and longest common subsequence metrics.
• Qualitative analysis: Human evaluation highlights that, due to possible reference bias, automated metrics such as CIDEr and BERTScore sometimes undervalue ViZer’s qualitative improvements; the enhanced captions often identify more visually grounded or additional scene details absent from human-provided references.

5. Empirical Performance and Qualitative Gains

Applying ViZer to models like SmolVLM-Base and Qwen2-VL yields consistent qualitative improvements in the generated captions:

• Rich and Scene-Specific Descriptions: Captions shift from generic (e.g., “<PERSON> in 2008”) to richer, more context-aware outputs (e.g., “woman surfing in the ocean”).
• More Accurate Structural and Contextual Details: The model captures relationships and attributes (object count, spatial relationships, background description) better than baselines.
• Reduction in Hallucination and Errors: Artifact details or errors (e.g., spurious numerical quantities or mismatched objects) decrease post-enhancement.
• PCA Visualization of Embeddings: After ViZer training, the clustering between caption and image embeddings in the shared space is tighter, evidencing improved semantic alignment.

6. Broader Applicability and Theoretical Impact

Although ViZer is introduced and validated in the image captioning context, its latent alignment paradigm is immediately extensible:

• Visual Question Answering: The improved vision–language alignment can facilitate more accurate answering by ensuring the representations for both images and questions inhabit a semantically coherent space.
• Multimodal Reasoning and Dialogue: ViZer’s continuous latent updating can support systems that must perform grounded reasoning or hold interactive, content-aware conversations.
• Label-Free Adaptation in Other Tasks: By removing dependency on human annotation, ViZer’s active alignment is conceptually suitable for tasks like retrieval, visual search, or content moderation, especially in uncurated, label-scarce environments.
• Continuous Learning: As a modular, lightweight component, ViZer allows for periodic or ongoing enhancement using only new, unlabeled image data as it becomes available.

A plausible implication is that as the field seeks ever-larger and more diverse datasets, schemes like ViZer will become integral to ensuring VLMs can scale and adapt without the “label bottleneck” that stymies classical supervised pipelines.

7. Limitations and Considerations

• Evaluation Metric Bias: Automated captioning metrics may undervalue qualitative improvements that include scene-accurate but reference-absent content.
• Scope of Reference-Free Learning: While ViZer’s alignment does not require labels for enhancement, initial model performance is still contingent on the pretraining quality of the underlying VLM.

---

Unified Vision-Language Alignment for Zero-Label Enhancement (ViZer) represents a modular, contrastive, zero-label enhancement training paradigm in vision-language modeling. Its method of active latent alignment increases descriptive grounding and reduces label dependency in image captioning, with promising implications for broader vision–language adaptation tasks (Byun et al., 14 Oct 2025).