DREAMSTEER: Latent World Models Can Steer VLA Policies During Deployment Without Any Finetuning

Abstract: Pretrained vision-language-action (VLA) policies show promising zero-shot generalization, but often fail under deployment-time distribution shift, leading to decreased robustness and inconsistent instruction following. While prior work commonly tackles this by finetuning on in-distribution data, it assumes demonstrations collected on tasks in the target environment. In this work, we propose DREAMSTEER, a deployment-time steering framework for pretrained VLAs without any finetuning or parameter modifications. The key insight in DREAMSTEER is to leverage a latent world model and a value model to steer pretrained VLA policies. During deployment, DREAMSTEER samples candidate action chunks from a VLA policy and predefined motion primitives, imagines their outcomes using an action-conditioned latent world model, and ranks the imagined trajectories with a language-conditioned value model. Across four real-world manipulation benchmarks with unseen objects, DREAMSTEER improves task success rate from 23.75% to 66.25% and instruction-following accuracy from 38.75% to 56.25% over the base VLA policy.

• The paper introduces a test-time policy steering framework that improves robotic manipulation by combining frozen VLA policies with a latent world model.
• It leverages a spatio-temporal transformer in latent space and a language-conditioned value model to rank candidate action sequences, achieving significant gains on OOD tasks.
• Deployment-time steering via imagined rollouts enables robust decision-making without updating pretrained policies, reducing the need for expensive environment-specific finetuning.

DREAMSTEER: Deployment-Time Policy Steering for VLAs Using Latent World Models

Introduction and Problem Setting

Pretrained vision-language-action (VLA) policies, constructed from large-scale multimodal data, exhibit strong zero-shot generalization in robotic manipulation tasks. Nonetheless, their robustness and instruction-following consistency degrade sharply under deployment-time distribution shift, particularly when confronted with out-of-distribution (OOD) objects or environment changes. Traditional solutions rely on finetuning with new demonstrations in the deployment environment, which is often impractical due to data collection costs and the desire to preserve the integrity of the pretrained policy.

DREAMSTEER introduces a test-time policy steering framework that obviates the need for finetuning. It leverages a plug-and-play composition of a frozen VLA policy, a generalized, action-conditioned latent world model, and a language-conditioned value model. The architecture enables deployment-time evaluation and selection among candidate action chunks by imagining their consequences in a learned latent state space, ranking rollouts according to alignment with the natural language instruction.

Methodology

Latent World Model

The world model is central to DREAMSTEER, trained across multiple robot embodiments and human demonstrations for generality, employing the DINOv2 latent feature space for efficient inference. Its architecture features a spatio-temporal transformer with spatial and causal temporal factorization, reducing rollout complexity to linear in the horizon. The model predicts future visual latent states conditioned on current observation and a candidate action chunk, not at the pixel level, but embedded in a compact, semantically rich representation space.

Key technical points:

• Multi-embodiment Training: Trained with robot and human manipulation datasets, supporting broad generalization across devices and views.
• Latent-space Dynamics: Predicts in DINOv2 latent space, allowing fast rollouts and preserving features required for value-based ranking.
• Spatio-temporal Transformer: Enables efficient, scalable trajectory modeling, needed when evaluating many candidate action sequences per control cycle.

Deployment-Time Steering

At each decision point, DREAMSTEER samples a pool of candidate action chunks from:

• The pretrained stochastic VLA policy ("policy proposals"),
• A library of short-horizon Cartesian motion primitives (e.g., move up, left, close/open gripper).

For each candidate:

1. The world model imagines the rollout in latent space from the current observation.
2. The latent sequence is decoded to visual observations for interpretability and downstream evaluation.
3. The value model (a Vision-Language-Action-Critic, VLAC) scores trajectory progress against the language instruction.
4. The action chunk with the highest score is executed.

No component is modified or finetuned using deployment-environment data. All models remain frozen, and the approach is fully training-free in the deployment phase.

Empirical Evaluation

Experimental Setting

Evaluations are performed with a 7-DoF Franka Panda manipulator on two axes:

• OOD Manipulation: Pick-and-place tasks with objects not present during policy training (e.g., phone, mustard, tape).
• Instruction Following: Tasks emphasizing language-grounded selection amidst distractors.

Testing is performed in a new laboratory to enforce strong distribution shift (variations in robot hardware, controller, camera positions, and background).

Numerical Results

DREAMSTEER demonstrates significant deployment-time improvements:

Task Type	Base VLA Policy (To)	DREAMSTEER (To + primitives)
OOD Object Manipulation	23.75%	66.25%
Instruction Following	38.75%	56.25%

These metrics are aggregated across multiple objects and scenes, with success defined objectively per benchmark protocol.

Critical ablation findings include:

• Value-based selection among multiple VLA samples improves over single-sample execution, confirming the presence of useful behaviors in the policy distribution that are not reliably selected by naive sampling.
• Using primitives alone or random selection yields no improvement, emphasizing the necessity of value-guided candidate ranking.
• The value model’s trajectory ranking on latent rollouts correlates positively (Pearson $r=0.66$) with scores on ground-truth video clips, validating the informativeness of latent rollouts for evaluation.

Efficiency and Latency

Generating and evaluating all candidates per timestep requires approximately 13 seconds (with 13 candidates; policy inference + world model rollout + value scoring), dominated by world model and value model computation. The process is fully parallelizable but is not yet optimized for real-time deployment.

Discussion

Mechanism of Improvement

DREAMSTEER shifts the policy execution paradigm from single-sample, one-pass action generation to a trajectory-ranking regime over imagined futures. This enables the exploitation of diversity in the action proposals, mediated via efficient latent-space simulation and value-based reranking. Components' generalization characteristics complement each other: while policy may be brittle under shift, the world and value models, trained on broader or more heterogeneous data, may generalize better to the new environment, allowing successful steering when the policy alone would fail.

Failure Modes and Limitations

Two principal failure modes are identified:

• Limited Candidate Coverage: When neither the VLA proposals nor the motion primitives are sufficient, the system cannot recover. Proposed avenues for improvement include integrating learned latent-planning or iterative candidate refinement.
• Imperfect Value Ranking: Ambiguity in visual observations (e.g., due to single-view limitations) can cause misranking of generated trajectories. Multi-view aggregation could improve ranking reliability.

Efficiency Bottlenecks

Inference latency is a nontrivial challenge. Significant speedups are possible through parallel rollout/evaluation, memory optimization, and next-generation attention kernels.

Implications and Future Directions

Practically, DREAMSTEER substantially increases the out-of-the-box reliability of robotic policies in unanticipated deployment settings without requiring new data or retraining. It establishes a general-purpose framework for plug-and-play policy steering, analogous to test-time output reranking in LLMs, but adapted to the unique challenges of physically grounded action selection.

Theoretically, the work demonstrates that decision-time model-based evaluation in latent spaces—decoupled from photorealistic simulation—can suffice for downstream control and instruction adherence.

Further progress may be realized by:

• Enriching the diversity and coverage of candidate action proposal mechanisms, e.g., via learned latent planners.
• Improving world model rollout stability and spatial coverage, possibly leveraging multi-view vision or multi-modal (audio, tactile) observation prediction.
• Enhancing value evaluation via ensembles or hierarchical critics.

Conclusion

DREAMSTEER provides an efficient, training-free framework that steers frozen VLA policies at deployment via imagined rollout evaluation in latent space. The method yields substantial gains in OOD manipulation and instruction-following accuracy relative to the base policy, without target-domain adaptation or parameter modification. It exemplifies the potential of modular, model-based architectures for robust policy deployment in robotics, and motivates further work on interactive latent-planning, efficient rollout architectures, and generalizable evaluators.