GenVLM: Vision-Language Generative Systems

• GenVLM is a family of vision–language models that generate multimodal outputs such as text, symbolic programs, and images, integrating various generative strategies.
• It leverages pretrained vision and language backbones with task-specific connectors to transform visual inputs into structured outputs across planning, narration, and control tasks.
• Empirical results demonstrate that GenVLM systems achieve efficient, diverse, and context-aware generative performance in tasks ranging from captioning to decision-making.

GenVLM denotes a heterogeneous but increasingly coherent research direction in which vision–LLMs are used as generative systems over multimodal outputs. In current arXiv usage, the term may refer to visually conditioned language generation with frozen LLMs, reward-proportional generation of reasoning-and-action trajectories, PDDL domain induction, generation of beam indices as text, coupled video generation–narration pipelines, or unified text-to-image models that reuse a VLM’s own visual understanding space (Jian et al., 2023, Kang et al., 9 Mar 2025, Hao et al., 3 Oct 2025, Kou et al., 23 Feb 2026, Fu et al., 2024, Chen et al., 8 Oct 2025, Guo et al., 28 Nov 2025). Taken together, these works suggest that GenVLM is best understood not as a single architecture but as a family of VLM-centric generative formulations whose outputs can be free-form text, structured symbolic programs, decision trajectories, continuous visual latents, or pixels.

1. Terminology and conceptual scope

The literature uses “GenVLM” in several distinct but related ways. Some works use it to denote a visually conditioned language generator built from a frozen vision encoder and frozen LLM, whereas others use it for a symbolic planner, a decision-making policy, or a unified understanding-generation model.

Work	GenVLM role	Generated object
EVL$_{\text{Gen}}$ (Jian et al., 2023)	one-stage visually conditioned language generation	captions or video-conditioned text
GFlowVLM (Kang et al., 9 Mar 2025)	generative, generalist decision-making VLM	CoT-guided action trajectories
DriveGenVLM (Fu et al., 2024)	driving video + narration pipeline	future driving video and text narration
GenVLM in VLMFP (Hao et al., 3 Oct 2025)	symbolic planning module	PDDL domain and problem files
BeamVLM (Kou et al., 23 Feb 2026)	VQA-style structured predictor	beam indices as language tokens
VUGEN (Chen et al., 8 Oct 2025)	unified understanding-generation VLM	text-conditioned image latents and images
VGT (Guo et al., 28 Nov 2025)	tuning paradigm for generative VLM conversion	continuous visual latents and images
G1 (Chen et al., 19 May 2025)	RL-trained interactive VLM agent	perception, reasoning, and actions

This terminological spread is not merely lexical variation. It reflects a substantive shift in how generation is defined around VLMs. In EVL$_{\text{Gen}}$, generation remains visually conditioned language production; in GFlowVLM and G1, the generated object is a multimodal control trajectory; in VLMFP, it is a formal symbolic model; in BeamVLM, it is a comma-separated sequence of future beam indices; and in VUGEN and VGT, it is an image mediated by VLM-native semantic latents. A plausible implication is that GenVLM has become an organizing label for VLM systems whose core competence lies in generative multimodal inference rather than static classification.

2. Architectural patterns

A recurring architectural pattern is the reuse of pretrained vision and language backbones while relocating task specificity into a connector, latent mapper, decoder, or policy wrapper. EVL$_{\text{Gen}}$ follows the frozen-backbone paradigm explicitly: a pretrained ViT produces patch tokens, a TomeFormer initialized from BERT base compresses them, and a frozen OPT-2.7B or Vicuna-7B consumes the compressed tokens as a soft prompt. In the main configuration, 256 tokens are reduced to 28, and only the connector and projections are trained (Jian et al., 2023).

GFlowVLM wraps LLaVA-v1.6-Mistral-7B inside a GFlowNet defined over a non-Markovian decision process. Its state is $z_t=(o_t,p_t)$, where $p_t$ encodes goal description, state history, action history, admissible actions, and optionally $d(o_t)$. At each step the VLM first emits chain-of-thought $c_t$, then an action $a_t$, and the forward policy combines both terms with a weighting factor $\lambda$, with $\lambda=0.4$ reported as best empirically (Kang et al., 9 Mar 2025).

Unified image-generation variants adopt a different pattern: they reuse the native visual representation of a VLM instead of introducing a separate reconstruction-oriented tokenizer. VUGEN starts from PLM-1B, maps the understanding embedding $_{\text{Gen}}$0 through a learnable dimension reducer $_{\text{Gen}}$1 into a lower-dimensional $_{\text{Gen}}$2, trains a Mixture-of-Transformers image generation tower to sample $_{\text{Gen}}$3, and decodes with either an LDM or a VAE-free pixel diffusion decoder $_{\text{Gen}}$4 (Chen et al., 8 Oct 2025). VGT likewise treats a pretrained VLM vision encoder as a semantic encoder, introduces VGT-AE to compress to 32-dimensional continuous visual tokens at 28x compression, and reuses the decoder-only LLM as a QueryAR backbone over continuous latents (Guo et al., 28 Nov 2025).

Domain-specialized GenVLM systems preserve the same principle while changing the target space. BeamVLM is built on Qwen2.5-VL-3B and reframes beam prediction as VQA: eight consecutive frames plus a textual instruction are mapped to five future beam indices rendered as textual integers. DriveGenVLM couples a conditional DDPM video generator with the pretrained BLIP-2-based EILEV model, which interprets generated or real driving clips by natural-language narration. In VLMFP, GenVLM is instantiated as GPT-4o and functions as the symbolic writer that generates and iteratively repairs PDDL domain and problem files, while a separate SimVLM supplies visual simulation and feedback (Kou et al., 23 Feb 2026, Fu et al., 2024, Hao et al., 3 Oct 2025).

3. Learning objectives and generative mechanisms

The objective functions associated with GenVLM are notably diverse. EVL$_{\text{Gen}}$5 adopts the simplest formulation: one-stage, single-loss training with captioning cross-entropy,

$_{\text{Gen}}$6

thereby bypassing BLIP-2-style Stage 1 contrastive and matching losses (Jian et al., 2023).

GFlowVLM defines generation over complete reasoning trajectories rather than token strings alone. Its central proportionality relation is

$_{\text{Gen}}$7

and the forward policy combines CoT and action log-probabilities. Training uses existing GFlowNet objectives adapted to non-Markovian multimodal states: Variance Trajectory Balance, Subtrajectory Balance, and Detailed Balance. A practical prerequisite is explicit termination modeling via a new [DONE] token and an SFT stage that teaches $_{\text{Gen}}$8 before GFlowNet optimization (Kang et al., 9 Mar 2025).

DriveGenVLM uses a conditional DDPM for video prediction. Its forward process is

$_{\text{Gen}}$9

and the reverse model generates future frames conditioned on observed ones. Long-horizon video generation is handled through Flexible Diffusion Model sampling schemes such as autoreg, hierarchy-2, and adaptive hierarchy-2 (Fu et al., 2024).

VUGEN factorizes text-to-image generation as

$_{\text{Gen}}$0

and trains the VLM generation tower with rectified flow matching: $_{\text{Gen}}$1 Its decoder stage can be an LDM or a VAE-free pixel diffusion decoder trained with flow matching, LPIPS, and REPA alignment to DINOv2 features (Chen et al., 8 Oct 2025). VGT also uses flow matching, but inside a continuous latent autoregressive formulation. For latent target $_{\text{Gen}}$2, it defines

$_{\text{Gen}}$3

and optimizes

$_{\text{Gen}}$4

This is paired with a two-stage autoencoding procedure: semantic-preserving reconstruction plus latent normalization and noise regularization (Guo et al., 28 Nov 2025).

Decision-making and symbolic-generation variants replace or augment likelihood training with environment-grounded feedback. G1 uses GRPO on interactive visual games, with grouped rollouts and a scalar reward

$_{\text{Gen}}$5

where $_{\text{Gen}}$6 is game reward, $_{\text{Gen}}$7 is format reward, and $_{\text{Gen}}$8 is perception reward. Its cold start is standard SFT on teacher-generated > and <answer> paired with environment-ground-truth <perception> (Chen et al., 19 May 2025). VLMFP instead performs prescreening, simulation consistency checking via an Exploration Walk score, and iterative regeneration of PDDL with structured interfaces such as add_or_update_predicates and modify_action (Hao et al., 3 Oct 2025).

BeamVLM remains token-likelihood based, but its target is a structured engineering output. It learns

$_{\text{Gen}}$9

with teacher forcing, where the output tokens encode future beam indices directly as integers (Kou et al., 23 Feb 2026).

4. Domains and system roles

GenVLM research spans at least four major operational regimes. The first is sequential decision-making. GFlowVLM evaluates NumberLine, Blackjack, and ALFWorld, with non-Markovian prompts that expose state and action history. G1 introduces VLM-Gym with 2048, Shisen-Sho, Shisen-Sho-CIFAR10, and Swap, and forces the policy to emit <perception>, <think>, and <answer> segments. VLMFP applies GenVLM to formal visual planning over six grid-world domains, where the generated object is a reusable PDDL domain file rather than a direct action (Kang et al., 9 Mar 2025, Chen et al., 19 May 2025, Hao et al., 3 Oct 2025).

The second regime is visually conditioned language generation in the more classical sense. EVL$z_t=(o_t,p_t)$0 targets image- and video-conditioned language generation with a one-stage frozen-LLM pipeline. DriveGenVLM uses generated Waymo videos as inputs to EILEV, whose narrations describe camera viewpoint, traffic scene, and day-versus-night conditions. In this setting, generation is distributed across two stages: video synthesis at the pixel level and semantic narration at the language level (Jian et al., 2023, Fu et al., 2024).

The third regime is structured numeric or symbolic sequence prediction. BeamVLM treats low-altitude beam prediction as a VQA problem and outputs exactly five integers separated by commas. VLMFP’s GenVLM outputs formal action schemas, predicates, and problem initializations. This suggests that, within GenVLM usage, “generation” frequently denotes structured outputs that are executable by another system rather than only human-readable text (Kou et al., 23 Feb 2026, Hao et al., 3 Oct 2025).

The fourth regime is unified understanding-and-generation image modeling. VUGEN reuses PLM-1B’s perception encoder to generate images from the VLM’s own reduced understanding latent space. VGT frames image generation as an ability latent in pretrained multimodal understanding models such as InternVL3 and Qwen2.5-VL, then exposes that ability through VGT-AE plus continuous latent autoregression. Both reject the assumption that a separate VAE tokenizer is the only viable bridge between understanding and generation (Chen et al., 8 Oct 2025, Guo et al., 28 Nov 2025).

5. Empirical record

The empirical record is correspondingly heterogeneous, combining efficiency, diversity, OOD generalization, formal validity, and image fidelity.

System	Reported outcome	Benchmark
GFlowVLM (Kang et al., 9 Mar 2025)	NumberLine OOD 17.3% / 16.7% / 18.6% off-policy for Var-TB / SubTB / DB; ALFWorld avg 26.1%, OOD 12.3, Div@16 1.40	NumberLine, ALFWorld
EVL$z_t=(o_t,p_t)$1 (Jian et al., 2023)	about 5× faster end-to-end than BLIP-2 competitive configuration; VQA 48.4, GQA 30.9, OK-VQA 27.2, COCO 139.1	zero-shot image-text evaluation
DriveGenVLM (Fu et al., 2024)	adaptive hierarchy-2 achieves FVD 1174.6 / 812.4 / 1122.2	Waymo front / front-left / front-right
VLMFP GenVLM (Hao et al., 3 Oct 2025)	70.0% valid plans on seen appearance, 54.1% on unseen appearance	formal visual planning
BeamVLM (Kou et al., 23 Feb 2026)	UAV Top-1 83.3% at $z_t=(o_t,p_t)$2 and 71.4% at $z_t=(o_t,p_t)$3; V2I Top-1 72.1% at $z_t=(o_t,p_t)$4 and 52.9% at $z_t=(o_t,p_t)$5	DeepSense 6G Scenario 23 and Scenario 8
VUGEN (Chen et al., 8 Oct 2025)	DPG Bench 71.17 to 74.32; FID 11.86 to 9.06 on COCO	text-to-image generation
VGT (Guo et al., 28 Nov 2025)	20× speedup; 26.67 PSNR; 0.50 rFID; GenEval 0.77; DPG-Bench 78.73	reconstruction and autoregressive T2I
G1 (Chen et al., 19 May 2025)	2048 1070; Shisen-Sho 17.5; Shisen-Sho-CIFAR10 14.1; Swap 0.78	VLM-Gym

Beyond headline metrics, several studies emphasize secondary properties that are central to the GenVLM idea. GFlowVLM reports higher solution diversity and convergence to peak performance with about 10k fewer environment steps than RL4VLM. EVL$z_t=(o_t,p_t)$6 reports strong performance even with 11M image-text pairs, or about 1/10 of BLIP-2’s 104M pretraining data. VUGEN reports that the VAE-free pixel diffusion decoder is on par or better than commonly used latent diffusion decoders while fully preserving the base VLM’s understanding capabilities. VGT reports that semantic latent design, rather than purely pixel-oriented tokenization, is the main driver of its autoregressive generation gains. G1 shows that a Qwen2.5-VL-7B-based policy can surpass Claude-3.7-Sonnet-Thinking across all four games once perception-enhanced SFT is followed by RL (Kang et al., 9 Mar 2025, Jian et al., 2023, Chen et al., 8 Oct 2025, Guo et al., 28 Nov 2025, Chen et al., 19 May 2025).

6. Limitations, misconceptions, and research directions

A common misconception is to equate GenVLM exclusively with text-to-image synthesis. The literature instead treats generation as production of trajectories, symbolic rules, numeric control outputs, narrations, and images. Another misconception is that a single training recipe defines the area. In practice, GenVLM systems span cross-entropy training, GFlowNet objectives, GRPO-based RL, DDPM sampling, rectified flow matching, and iterative simulator-guided repair. This suggests that the unifying element is not objective class but the use of a VLM-centered representation as the generative substrate.

The main technical bottlenecks differ by subfamily. GFlowVLM depends on non-negative reward shaping and accurate estimation of $z_t=(o_t,p_t)$7, with SubTB and DB particularly sensitive to [DONE] modeling. BeamVLM is materially heavier than its discriminative baselines, with runtime $z_t=(o_t,p_t)$8 s per sample, and it depends on reliable camera input aligned with the beam sector. DriveGenVLM is trained on 138 Waymo clips at $z_t=(o_t,p_t)$9, struggles with buildings, pedestrians, and complex real-world logic, and does not yet close the loop into control (Kang et al., 9 Mar 2025, Kou et al., 23 Feb 2026, Fu et al., 2024).

Open directions are correspondingly varied. VLMFP points toward more structured domain-learning interfaces and stronger automated detection of logical inconsistencies. G1 points toward larger-scale multi-game RL, better handling of sparse reward, and transfer beyond simple games. VUGEN highlights scaling to larger backbones and resolutions, while VGT identifies a persistent reconstruction-versus-generation trade-off in latent design. GFlowVLM points toward larger VLMs, richer embodied benchmarks, multi-agent settings, and better reward modeling. A plausible synthesis is that future GenVLM systems will increasingly combine three properties now distributed across separate papers: native VLM semantic latents for generation, environment-grounded optimization for actionability, and explicit structural outputs that can be consumed by planners, simulators, or communication stacks (Hao et al., 3 Oct 2025, Chen et al., 19 May 2025, Chen et al., 8 Oct 2025, Guo et al., 28 Nov 2025).