OneVL: Unified Vision-Language-Action Modeling

• OneVL is a unified framework that consolidates image, video, and language modalities into a single model using shared latent representations.
• The approach employs auxiliary decoders to supervise both semantic and visual future-frame predictions, facilitating efficient one-step latent reasoning and planning.
• Empirical results show that OneVL achieves real-time performance with competitive accuracy, streamlining deployment in safety-critical applications.

OneVL encompasses multiple distinct but convergent lines of research in unified modeling for vision-language (VL) and vision-language-action (VLA) tasks. The term “OneVL” is associated primarily with two families: (1) foundation models that unify image-language and video-language pipelines via a single model and pretraining regime (notably OmniVL), and (2) unified vision-language-action world model architectures for one-step latent reasoning and planning that compress explicit reasoning into a compact latent space for high-efficiency inference (OneVL as in “One-step latent reasoning and planning with Vision-Language explanations”). Both strands represent a shift from pipeline-fragmented, multi-component reasoning toward joint, end-to-end frameworks with generalization and real-time efficiency at scale.

1. Unified Vision-Language(-Action): Concept and Motivation

“OneVL” denotes architectures and training objectives designed to collapse multiple modalities—image, video, and natural language—into a single, parameter-shared model, eliminating the need for per-modality encoders or fusion transformers found in traditional VLP systems. The impetus is twofold:

• Modality synergies: Visual and linguistic modalities, when processed jointly, encourage richer representations that benefit both domains; e.g., spatial structure from image-language tasks and temporal structure from video-language tasks (Wang et al., 2022).
• Efficiency and deployment: Autoregressive explicit Chain-of-Thought (CoT) reasoning, widely adopted for interpretable VLA reasoning, incurs prohibitive latency due to sequential token generation. In safety-critical applications such as autonomous driving, even a 1–2 second delay for CoT reasoning can be unacceptable (Lu et al., 20 Apr 2026).

A further driver is the empirical observation that latent CoT methods—which compress reasoning into continuous thought vectors—often underperform explicit CoT, as latent spaces induced by language alone fail to internalize the causal dynamics of the environment. Unification thus entails not only combining image/video/text pipelines but also integrating symbolic and causal abstraction within one latent space (Lu et al., 20 Apr 2026).

2. Model Architectures and Latent Structure

One-Step Latent CoT for Vision-Language-Action (OneVL, (Lu et al., 20 Apr 2026))

The OneVL world model extends a pretrained VLM backbone (Qwen3-VL-4B, ViT encoder and LLM) with a set of inserted latent tokens:

• Visual latents ($C_v=4$): <|latent-vis|> tokens designed to encode spatio-temporal and dynamical information.
• Language latents ($C_t=2$): <|latent|> tokens to encode semantic and symbolic reasoning.

The backbone produces matrices $H_v\in\mathbb R^{C_v\times d}$ and $H_l\in\mathbb R^{C_t\times d}$, with $d=2048$, representing the hidden states at latent positions.

Auxiliary supervision is provided through two decoders:

• Language auxiliary decoder ($\mathcal D_{\text{lang}}$) reconstructs the ground-truth text Chain-of-Thought from the latents.
• Visual auxiliary decoder ($\mathcal D_{\text{vis}}$) predicts a sequence of discrete tokens representing predicted future camera frames at +0.5 s and +1.0 s, via an IBQ tokenizer and a 131K-code vocabulary.

The main trajectory (action) head produces the predicted trajectory as a sequence of 8 two-dimensional waypoints, decoded from the LLM. A deployment-optimized variant uses an MLP on the last language latent for direct trajectory regression.

Unified Image-/Video-Language Pretraining (OmniVL, (Wang et al., 2022)) and SimVLT (Wang et al., 2022)

Foundation models termed “OneVL” in the image/video-language literature feature:

• Unified visual encoder: A single TimeSformer-style Transformer backbone with patch-tokenization (2D for images, 3D for video). Spatial and temporal position encodings are decoupled: temporal attention is bypassed for still images.
• Parameter sharing: All vision Transformer weights are shared, except for initial patch-tokens and any temporal attention heads.
• BERT-based text encoder and two “visual-grounded” decoders:
  • Alignment decoder (bidirectional) for matching
  • Generation decoder (causal) for text generation

A key innovation is merging supervised (image/video-label) and webly-crawled (image/text, video/text) data in pretraining with a unified contrastive loss.

3. Training Regimes and Objectives

OneVL (Latent CoT, (Lu et al., 20 Apr 2026))

The model is optimized via a multi-objective loss:

$$L = L_{\text{traj}} + \lambda_l\,L_{\text{lang}} + \lambda_v\,L_{\text{vis}}$$

where $L_{\text{traj}}$ is trajectory cross-entropy, $L_{\text{lang}}$ is auxiliary language reconstruction, and $C_t=2$0 is visual future-frame reconstruction. Weights are set to $C_t=2$1, $C_t=2$2.

Three-stage curriculum:

1. Pretraining ($C_t=2$3 only): Trains the visual decoder unconditionally (video prediction), establishing spatial-temporal priors.
2. Stage 0 (warmup): Trajectory head only; auxiliary decoders untrained. Forces latents to become action-useful before decoding.
3. Stage 1 (auxiliary warmup): Latent states frozen, train decoders for language and vision to align interpretability.
4. Stage 2 (joint fine-tuning): All modules jointly optimized.

This staged approach stabilizes optimization, prevents “gradient shock,” and ensures latents are both compressive and predictive.

OmniVL/SimVLT (Wang et al., 2022, Wang et al., 2022)

Pretraining unfolds in phases:

• Stage 1: Image-language only (spatial-only) to consolidate spatial representations.
• Stage 2: Mix of image-/video-language and -label data for full spatio-temporal coverage.

Unified Vision–Language Contrastive (UniVLC) loss averages bidirectional contrastive losses over normalized visual/text embeddings within group labels (supervised and webly data). Additional objectives: Vision-Language Matching (binary) and Language Modeling (autoregressive generation).

SimVLT relies on:

• Video-Text Matching (VTM)
• Masked Language Modeling (MLM)
• Video-Text Contrastive (VTC, for retrieval finetuning)

Temporal Token Rolling (TTR) is applied in the transformer blocks to integrate temporal dependencies efficiently.

4. Inference Efficiency and Deployment

The defining operational feature of the OneVL (Latent CoT) approach is that all latent tokens are prefilled into the prompt as special tokens. Transformers process these tokens in parallel, and only the final trajectory output is decoded autoregressively, yielding “answer-only” latency, matching the speed of models that directly predict actions without reasoning.

Empirical inference times:

Benchmark	OneVL (s)	AR Answer-only (s)	AR CoT+Answer (s)
NAVSIM	4.46	4.49	6.58
ROADWork	4.71	4.74	10.74
Impromptu	4.02	4.24	6.84
APR1	3.23	3.06	3.51

A feed-forward variant (MLP-on-latent) can further reduce runtime to 0.24 s (4.16 Hz), albeit with a modest metric drop.

Foundation models (OmniVL, SimVLT) eliminate the need for per-task adaptors. Their transformer backbones can be efficiently fine-tuned or used zero-shot across visual-only, cross-modal alignment, and multimodal generation tasks, with competitive throughput (SimVLT-Base: 89 v/s on $C_t=2$4 for $C_t=2$5 frames).

5. Empirical Performance and Ablation

OneVL (Lu et al., 20 Apr 2026) on Trajectory and Reasoning

Across four benchmarks, OneVL is the first latent CoT method to both outperform explicit (textual) CoT in accuracy and to match answer-only latency.

Benchmark	Metric	OneVL	AR CoT+Answer	Latent CoT (prev)
NAVSIM	PDM	88.84	88.29	~84
ROADWork	ADE/FDE	12.49/28.80	13.18/29.98	-
Impromptu	ADE/FDE	1.34/3.70	1.42/3.96	-
APR1	ADE	2.62	2.99	-

Ablation studies demonstrate the importance of staged training (drop of 21.71 PDM if omitted), visual decoder supervision ($C_t=2$6 PDM), and language decoder supervision ($C_t=2$7 PDM).

Foundation Models (Wang et al., 2022, Wang et al., 2022) on VL Benchmarks

Unified Transformer models (OmniVL, SimVLT) achieve state-of-the-art or competitive results across classification, retrieval, captioning, and question answering:

Task	Model	Main Metric	Score	Comparison
Image classif. (LP)	OmniVL	avg acc.	86.7%	BLIP 81.5%
Video action (UCF101)	OmniVL	accuracy	93.2%	TimeSformer 82.9%
COCO Retrieval I→T	OmniVL	R@1	82.1	BLIP 80.6
VQA	OmniVL	accuracy	78.33%	BLIP 77.54%
MSRVTT T→V	OmniVL	R@1 (zero-shot)	34.6	VIOLET 34.5
Video QA (MSVD)	OmniVL	accuracy	51.0%	-
Video QA (MSRVTT)	SimVLT-B	accuracy	44.3	VIOLET 43.1
Video retrieval	SimVLT-B	R@1 (MSR-VTT 9K)	37.9	OA-Trans 35.8

6. Limitations, Interpretability, and Future Directions

OneVL (Lu et al., 20 Apr 2026)

• Model capacity vs. compactness: Latent counts ($C_t=2$8, $C_t=2$9) are empirically chosen; increasing dimensionality could improve expressivity at the cost of efficiency.
• Training overhead: The three-model training pipeline consumes roughly threefold memory; partial mitigation is possible via optimizer partitioning (ZeRO-2).
• Interpretability: By supervising the latent space with both language and visual future frame decoding, the bottleneck internalizes factors that are causally predictive and semantically meaningful. This dual supervision yields generalizable and interpretable representations not achievable with purely linguistic or purely visual bottlenecks.
• Future work: Directions include parallel (non-AR) trajectory decoding, multi-camera extension for 360° context, debugging interfaces visualizing the “visual CoT” rollouts, and reinforcement learning finetuning for deployment robustness.

Unified VL Foundation Models

• Pretraining regime: Mixing supervised and noisy web data increases robustness and transferability; decoupled spatial-temporal stages are found to stabilize video-language learning.
• Modularity: The absence of per-task adaptors and fusion blocks enhances practicality for new downstream tasks.

A plausible implication is that unified latent and pretraining frameworks, particularly those combining symbolic and causally predictive bottlenecks, offer not only efficiency gains but also improved generalization in open-world reasoning tasks.

7. Relationship to Related Research and Distinctions

The “OneVL” paradigm distinguishes itself from dual-stream and late-fusion architectures by merging all modalities and objectives in both architectural and optimization regimes. Unlike earlier latent CoT approaches (COCONUT, CODI, SIM-CoT), OneVL’s two-auxiliary-decoder supervision compels the latent to encode not only compressed semantic abstraction but also the causal dynamics necessary for open-ended spatio-temporal reasoning (Lu et al., 20 Apr 2026).

In summation, OneVL and its counterparts (OmniVL, SimVLT) mark a transition to genuinely unified, efficient, and general-purpose vision-language (and action) modeling, demonstrating performance that meets or exceeds those of prior multi-component or explicit reasoning pipelines (Lu et al., 20 Apr 2026, Wang et al., 2022, Wang et al., 2022).