Guided Action Flow: Q-Guided Inference for Flow-Matching Vision-Language-Action Policies

Abstract: Flow-matching vision-language-action policies generate robot action chunks through an iterative transport process, creating an opportunity for test-time guidance without retraining the base policy. We study this opportunity in Guided Action Flow, an inference-time framework that keeps a pretrained SmolVLA policy frozen and uses a learned action-chunk critic to guide its reverse-time flow sampler. The critic is trained from real success and failure rollouts, can condition on task-description features from the frozen SmolVLA language pathway, and is used only through action gradients during sampling. We evaluate the approach on LIBERO manipulation tasks. A single-task critic improves success from 68.0% to 82.0% on one seed window and from 82.0% to 86.0% on another. A multi-family task-description critic improves validation success from 46.0% to 56.0%, while the locked held-out test gain is positive but modest, from 65.0% to 67.5%. These results support the feasibility of Q-guided inference for frozen flow-matching VLA policies, while showing that critic generalization and uncertainty-aware guidance remain the central bottlenecks.

• The paper presents Guided Action Flow, enhancing closed-loop robotic tasks by using a Q-guided critic ensemble to adjust action velocities without retraining.
• It employs a reverse-time flow-matching sampler where critic gradients bias the action generation, yielding notable success rate improvements on LIBERO benchmarks.
• The study identifies limitations in critic generalization and parameter sensitivity, suggesting future work on robust, OOD-aware guidance for real-world applications.

Guided Action Flow: Q-Guided Inference for Flow-Matching Vision-Language-Action Policies

Introduction and Motivation

Vision-language-action (VLA) policies, leveraging large-scale data and strong language-vision backbones, have set a new standard for language-conditioned robotic manipulation across diverse benchmarks such as LIBERO, LIBERO-Plus, and LIBERO-PRO. Despite this progress, robust adaptation to task, object, and distributional shifts remains an unsolved problem, particularly when full-policy fine-tuning is impractical due to hardware constraints or scarce task-local data. The central thesis of "Guided Action Flow: Q-Guided Inference for Flow-Matching Vision-Language-Action Policies" (2607.02092) is that inference-time, value-guided perturbations to a frozen flow-matching VLA policy—without further base policy training—can realize non-trivial performance improvements in closed-loop robotic tasks.

The paper proposes Guided Action Flow, an inference-time wrapper that augments a frozen SmolVLA policy with a learned action-chunk critic ensemble. This critic is trained offline on real success/failure rollouts and provides value gradients with respect to action chunks during the iterative flow-matching sampling of the base policy. The intent is to bias the policy toward higher-value actions with minimal system retraining.

Figure 1: System overview of Guided Action Flow. Offline rollouts from the frozen policy are converted into success/failure episodes and action-chunk training examples for a separately trained critic ensemble. At test time, SmolVLA remains frozen while QGF uses the critic ensemble through critic gradients, uncertainty gating, and velocity adjustment to guide action-chunk generation during closed-loop LIBERO evaluation.

Methodology

Problem Setup

Given a pretrained and frozen VLA policy (SmolVLA), the problem is to improve closed-loop task performance through test-time guidance. The base policy produces action chunks via a reverse-time flow-matching sampler, integrating from random noise ($t=1$) toward action ($t=0$). Guided Action Flow intervenes in the velocity computation at each denoising step by injecting gradients derived from a learned critic.

Critic Training

The critic, $Q_\phi$, is a multilayer perceptron trained to predict a sparse success-to-go signal for candidate action chunks, conditioned on observation features and language/task descriptors. Training data is collected from offline rollouts of the frozen SmolVLA on target tasks, with episodes segmented into overlapping action chunks. The critic architecture accommodates task conditioning through mean-pooled hidden state features from the SmolVLA language pathway, which allows transferability beyond fixed task IDs. An ensemble of critics is trained to facilitate uncertainty estimation via their disagreement.

Inference-Time Guidance Protocol

At each SmolVLA flow-matching denoising step:

• The current sample and policy-predicted velocity are used to estimate the clean action chunk.
• The critic ensemble evaluates this estimate, and the mean value gradient with respect to the action is computed.
• The action gradient is clipped and then adaptively weighted using an exponential disagreement gate that suppresses guidance when ensemble variance is high.
• The base velocity is modified by subtracting the (gated, scaled) critic gradient, consistent with the reverse-time SmolVLA convention: this increases the probability of higher-value actions under the critic.
• The policy weights remain strictly frozen; all adaptation occurs through velocity perturbation.

Unlike reranking methods, the critic here operates in a continuous action space and delivers local, differentiable feedback at each sampling timestep, shaping the entire action-generation trajectory.

Empirical Results

The experiments primarily evaluate within the LIBERO benchmark, focusing on the following questions:

1. Can a critic ensemble improve single-task closed-loop performance?
2. Does a critic trained across related tasks generalize to unseen variations?
3. Do improvements on validation tasks translate to locked, held-out tests?

Single-Task and Multi-Task Results

• Single-task evaluation: On a representative LIBERO spatial task, QGF raises success from 68.0% to 82.0% (seed 3000 window), and from 82.0% to 86.0% (seed 4000 window), establishing the feasibility of value-gradient-based test-time guidance.
• Multi-task (spatial-only) transfer: Here, a critic trained only on related spatial tasks did not improve (or marginally decreased) performance on validation episodes, indicating the limits of generalization for narrowly trained critics.
• Multi-family, task-description conditioned: Training the critic on both spatial and object tasks, and conditioning on language pathway features, raised validation success from 46.0% to 56.0%. On locked, held-out tasks, the gain was more modest (65.0% to 67.5%).

These results establish that Q-guided inference can yield measurable improvements for frozen VLA policies, though held-out generalization is currently limited by critic robustness.

Technical Analysis and Implications

The Guided Action Flow framework demonstrates that flow-matching VLA policies expose an interface for modular test-time adaptation via external value gradients. This presents several technical implications:

• Critic quality and uncertainty: The method is bottlenecked by the generalization capacity of the action-chunk critic. Task-conditioned critics, leveraging language pathway features, outperform fixed-ID or shallow encoding variants. Ensemble disagreement gating provides an effective, lightweight mechanism for mitigating catastrophic value extrapolation, but more principled OOD detectors are needed for robust deployment.
• No base policy retraining: The framework decouples policy and adaptation costs. This is highly attractive in environments with limited data, hardware, or safety budgets for full fine-tuning. All empirical improvements are obtained with a strictly frozen SmolVLA checkpoint.
• Sign and protocol correctness: The reverse-time flow-matching convention for action estimation in the SmolVLA sampler is non-trivial; correct sign handling is essential for beneficial gradient flow.
• Evaluation discipline: The paper emphasizes episode-level train/validation splits to prevent trajectory context leakage and relies on locked held-out tests for honest performance characterization.

Practically, this method enables rapid, modular adaptation of generalist policies to new tasks by assembling lightweight, offline-learned critics, with no access to privileged simulator state at deployment.

Limitations and Future Directions

• Limited critic generalization: On held-out tasks, improvements remain moderate—a consequence of restricted critic coverage, sparse reward targets, and limited OOD handling.
• Parameter sensitivity: Empirical performance is sensitive to ensemble size, guidance strength ($\beta$), and gating parameters, requiring careful validation-only tuning.
• Evaluation scope: Experiments remain in simulation with relatively small held-out test budgets. LIBERO-Plus and LIBERO-PRO require further investigation due to near-zero base policy performance under current setups.
• No real-robot validation: The method has not yet been validated under real-world sensor perturbations or system noise.

Future work should prioritize richer observation features for the critic, advanced contrastive or uncertainty-aware loss functions, and integration of OOD-aware guidance rules. Extending evaluation to direct deployment on physical robotic platforms remains an open challenge for assessing practical utility.

Conclusion

Guided Action Flow provides an inference-time adaptation mechanism for pretrained flow-matching vision-language-action policies without further policy retraining. By integrating a learned, task-conditioned action-chunk critic as a gradient-based guide during the action flow sampling process, it achieves clear performance improvements in both single-task and multi-family settings on the LIBERO benchmark. The work establishes the technical feasibility of Q-guided inference in modular robotic pipelines, while systematically identifying critic quality and distributional robustness as central open problems for the generalization of this adaptation paradigm.

(2607.02092)