Conditional Vision Language Models

• ConditionalVLMs are multimodal architectures that use explicit condition tokens to control the output type (e.g., captions, dense descriptions, Q&A) from combined visual and linguistic inputs.
• They utilize dynamic prompt tuning and modular conditioning mechanisms, with reported accuracy improvements (e.g., from 63.2% to 71.7%) on new class generalization benchmarks.
• Advanced techniques like plug-and-play inference, context compression, and counterfactual fine-tuning enhance efficiency, reduce hallucinations, and bolster robust multimodal reasoning.

A Conditional Vision LLM (ConditionalVLM) is a class of multimodal machine learning architectures that generate language or perform multimodal reasoning under explicit control signals or conditioning variables. These models typically unify visual and linguistic understanding and generation, with conditioning mechanisms that modulate the type, style, or function of the output—such as varying between captions, dense scene descriptions, or question-answering—given an image and a specified condition. The research landscape for ConditionalVLMs has rapidly expanded to include self-training unified models, dynamic conditional prompts, plug-and-play conditional inference, modular control for robotics, context compression for efficiency, and sophisticated loss formulations for robust alignment and causal understanding.

1. Foundational Architectures and Conditioning Mechanisms

ConditionalVLMs integrate visual and linguistic encoders in ways that support multiple conditional outputs through architectural innovations and conditioning tokens. A typical example is the Unified Conditional Model (UCM), which utilizes a two-stream transformer architecture (bi-directional and uni-directional branches) with cross-attention layers (Yang et al., 2022). The distinguishing component is a “condition token” [CND], introduced early in the input sequence, which allows the model to switch output mode (e.g., caption, dense caption, question) by altering the condition flag at inference.

The training regime combines Conditional Masked LLMing (CMLM) for both branches:

$$L_{\text{CMLM-Bi}}(\theta) = -\mathbb{E}_{w,v}\left[\log P_\theta(w_m \mid w_{\backslash m}, v, c)\right]$$

$$L_{\text{CMLM-One}}(\theta) = -\mathbb{E}_{w,v}\left[\log P_\theta(w_m \mid w_{<m}, v, c)\right]$$

Auxiliary objectives—Image-Text Matching, Masked Object and Attributes Modeling (MOAM), and Masked Feature Regression (MFR)—ensure dual skills in understanding and generation, and fuse visual-linguistic reasoning.

Zero-shot conditional generation is directly enabled by training the generative (one-directional) branch under explicit conditions, allowing models to produce different language outputs per condition flag at test time, without further fine-tuning.

2. Conditional Prompt Learning and Domain Generalization

Static prompt learning, as deployed in methods such as CoOp, improves adaptation of large vision-LLMs but suffers from overfitting to base classes. Conditional prompts, notably Conditional Context Optimization (CoCoOp) (Zhou et al., 2022), address this by generating dynamic, instance-dependent prompts. A lightweight network ("Meta-Net") takes an image feature $x$ and adjusts the prompt tokens:

$$v_m(x) = v_m + \pi \quad\text{with}\quad \pi = h_\theta(x)$$

The overall prompt for class $i$ is:

$t_i(x) = \{v_1(x),\dots, v_M(x), c_i\}$

Prediction operates via cosine similarity between image and dynamically prompted text embeddings. Empirical results demonstrate marked improvements in new class accuracy (often increasing from 63.2% to 71.7% across benchmarks), narrowing the generalization gap between base and unseen classes, with only slight compromise in base accuracy.

Further advances in conditional prompt tuning critically analyze conditioning signals: research on Class-adaptive Prompt Tuning (CaPT) (Zhang et al., 30 Jun 2025) demonstrates that conditioning prompts on Textual Class Information (TCI) derived from class names (via pretrained text encoders) is more effective than visual features (VII) or even random noise. CaPT employs a meta-network to transform prompts:

$$v_k(e_{\text{base}}^i) = v_k + h_\alpha(e_{\text{base}}^i)$$

The margin-ITM loss is additionally used to temper the distinction between closely related classes:

$$p(c_i|x_j) = \frac{\exp(\langle g(c_i(e_{\text{base}}^i)), f_j\rangle/\tau)}{\sum_{i'} \exp((\langle g(c_{i'}(e_{\text{base}}^{i'})), f_j\rangle - s_{i'i})/\tau)}$$

This formulation enables simultaneous high base class performance and strong generalization to new classes.

3. Plug-and-Play Conditional Inference and Prompt Regularization

ConditionalVLMs can be composed at inference via "importance sampling" or masking strategies. VLIS (Chung et al., 2023) fuses a pre-trained LLM's output with a VLM by reweighting token likelihoods using pointwise mutual information (PMI):

$$\text{PMI}(x_t \mid c, x_{<t}) = \log\left[\frac{p_{vl}(x_t \mid c, x_{<t})}{p_{vl}(x_t \mid x_{<t})}\right]$$

$$f(x_t) = \bar{p}_{text}(x_t | c, x_{<t}) \cdot \exp(\text{PMI}(x_t, c \mid x_{<t}))$$

Tokens that are both linguistically fluent and visually grounded are promoted. This approach is shown to improve commonsense VQA, complex text generation, and explanation of visual oddities—attributes not reliably addressed in conventional VLMs which often overfit to either modality.

ProMIM (Bui et al., 7 Aug 2025) introduces masked image modeling (MIM) into prompt generation, masking a large percentage of image patches during prompt formation. The prompt meta-network takes only the visible sub-image features: this prevents "leakage" of full-image information into conditional prompts and mitigates overfitting, with minimal additional computation. Such regularization, when combined with alignment losses (as in KgCoOp), further increases generalization and robustness across dataset shifts.

4. Efficient and Scalable ConditionalVLMs

Several architectural extensions focus on improving data and compute efficiency. Prismatic VLMs (Karamcheti et al., 12 Feb 2024) compile a modular architecture: a pretrained visual backbone feeds a learned projector, concatenates projected features with prompt embeddings, and feeds the sequence into a LLM. Training strategies include both base and instruct-tuned LMs, with single-stage optimization preferred for computational and performance efficiency (saving up to 25% compute).

Vision Concept Modeling (VCM) (Luo et al., 28 Apr 2025) introduces dynamic concept-level selection, reducing the number of dense visual tokens by aligning only those that are semantically relevant. Using implicit contrastive learning and a forward–backward dynamic programming algorithm, VCM collapses visual inputs into a sparse set of conceptual tokens, achieving up to 85% reduction in FLOPs with negligible loss in accuracy on image understanding tasks.

ContextVLA (Jang et al., 5 Oct 2025) applies amortized multi-frame context compression for efficient vision-language-action models in robotic settings. Temporal context from multi-frame sequences is compressed via average pooling to a single context token, injected into later model layers, retaining the temporal dependencies needed for action generation with substantially reduced memory and compute.

5. Robust Multimodal Reasoning: Causality, Hallucination Mitigation, and Attribute Reasoning

ConditionalVLMs face key challenges in causal reasoning and hallucination reduction.

CF-VLM (Zhang et al., 10 Jun 2025) employs counterfactual fine-tuning, generating counterfactual image-text pairs with minimal semantic differences and utilizing a composite loss:

• Alignment loss $L_{\text{align}}$ for overall cross-modal matching.
• Counterfactual scenario discrimination loss $L_{\text{csd}}$:

$$L_{\text{csd}} = \frac{1}{K} \sum_k \max(0, S(I_{cf,k}, T_{cf,k}) - S(I_{anchor}, T_{anchor}) + m_1)$$

• Fine-grained causal discrimination loss $L_{\text{fcd}}$, focusing on minimal attribute or relation edits.

This training leads to improved compositional reasoning, reduced hallucinations, and higher robustness on high-stakes tasks.

A complementary perspective is taken by (Fang et al., 26 May 2025), introducing a Conditional PMI (C-PMI) calibrated decoding regime. Token generation is modulated by

$$C\text{-}PMI_t(v, y|x) = \log p_\theta(y_t|v,x,y_{<t}) - \log p_\theta(y_t|x,y_{<t})$$

This mutual information dynamic, solved as a bi-level optimization with a lightweight purifier network, ensures only visual tokens most relevant to each response are retained, further reducing hallucinated content during generation.

For fine-grained attribute recognition, generative retrieval formulations (Zhu et al., 7 Aug 2024) leverage autoregressive models to factorize the probability of an object-attribute sentence given an image, capturing conditional dependencies and outperforming embedding-level contrastive retrieval.

6. Applications: Robotics, Moderation, Task Planning, and Context Conditioning

ConditionalVLMs are now critical in a range of real-world tasks:

• Robotic Manipulation and Planning: VLMPC (Zhao et al., 13 Jul 2024) and ContextVLA (Jang et al., 5 Oct 2025) integrate conditional action sampling—where the VLM proposes candidate action sequences under task conditions—and amortized context token compression for temporally-aware control. Behavior tree generation (Wake et al., 7 Jan 2025) leverages a VLM to build hierarchical task programs with free-form visual conditions, using runtime self-prompting to verify branching on real images.
• Content Moderation: ConditionalVLMs grounded on pretrained unsafe image classifiers and counterfactual reasoning algorithms provide interpretability and minimal obfuscation of unsafe image regions (Bethany et al., 19 Jan 2024).
• Task Generalization and Instruction Tuning: MoCLE (Gou et al., 2023) clusters instruction embeddings and routes samples to mixture-of-experts (LoRA) branches, integrating a universal expert for robustness on out-of-distribution and composite tasks.

In all these settings, the modular use of condition tokens, class-conditional prompt networks, and bi-level optimization (for instance, in C-PMI or VCM) is crucial for ensuring models adapt fluidly to varying task demands while maintaining factual precision.

7. Future Directions and Synthesis

Research points toward several fronts:

• Meta-level Prompt and Condition Design: The trend is towards more sophisticated or semantically structured conditioning sources—class semantics (CaPT), external MLLMs for high-level features (MuGCP (Yang et al., 11 Jul 2025)), or multi-view environment encodings for RL agents (Cachet et al., 24 Sep 2024).
• Architecture and Optimization: Combining single-stage training with concept-level selection and mutual information–grounded decoding appears productive for both efficiency and alignment.
• Benchmarking and Evaluation: Adoption of unified evaluation frameworks, normalized Z-scores, and task-specialized loss functions enables calibration of advancements across the diverse modalities and conditioning strategies.

ConditionalVLMs embody a family of models in which explicit, context-dependent control over multimodal reasoning and generation is realized through architectural, training, and inference-time conditioning. Advances continue in designing modular, efficient, and factually robust systems that can perform and generalize across the spectrum of tasks demanded by contemporary AI applications.