DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models

Abstract: Many modern vision-LLMs (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.

• The paper introduces DRIFT, which integrates a coarse predictor with a residual flow adapter to refine continuous outputs.
• It improves sample efficiency, training stability, and precision by localizing the generative task to a fine-grained residual correction.
• Empirical results demonstrate DRIFT’s superior performance on robotic control and temporal video grounding over traditional methods.

DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-LLMs

Motivation and Problem Formulation

This paper identifies a crucial limitation in the current architecture of vision-LLMs (VLMs): their reliance on autoregressive, discrete token-based decoding. While this approach has enabled scaling and generalization across tasks amenable to textual interfaces, it fundamentally restricts the ability to generate precise continuous outputs, which are essential for tasks such as temporal video grounding (TVG) and robotic action generation. Existing alternatives—tokenization of continuous targets, direct regression heads, or diffusion/flow-based decoders—are individually inadequate: the first two are deterministic and struggle with uncertainty and multimodality, while the latter suffer from high sample complexity and data inefficiency.

The central question addressed is: How can pretrained VLMs with discrete output heads be adapted to high-precision continuous-output tasks without sacrificing their pretraining-induced generalization and transfer capabilities? The solution must allow the preservation of reusable transformer-based backbones and leverage the generalization strengths of large multimodal pretraining.

Methodology: The PSRFM/DRIFT Framework

The paper introduces DRIFT (Decoding via Residual Flow Adapter; denoted as PSRFM in the figures), a general adaptation paradigm for VLMs. DRIFT is characterized by a cascaded architecture consisting of:

1. Base Predictor: Provides a coarse, deterministic estimate of the continuous target. This can be the output of a token decoder (for discretized targets) or a simple MLP regressor.
2. Generative Refinement Module (Residual Flow Adapter): Given the coarse estimate as a strong prior, this component models the local residual distribution using flow matching. Rather than learning the global conditional distribution from scratch, the model transports a localized Gaussian (centered at the base prediction) to the true conditional target in a learned, data-efficient fashion.

This residual flow formulation not only enables iterative refinement, but, crucially, it restricts the modeling task to a much smaller, local correction, thereby improving optimization speed, stability, and sample efficiency.

Figure 1: PSRFM learns a velocity field transporting a distribution centered at the initial coarse prediction toward the target, enabling lower-variance, more accurate, and stable generation for actions and event boundaries.

Figure 2: Overview of PSRFM's structure and its performance on synthetic 2D tasks; the residual refinement yields faster convergence and sharper recovery of multimodal target distributions.

The refinement module’s velocity field is parameterized to predict the target as a convex combination (gated by a learned parameter) of the base prediction and a refinement network, enabling flexible skip connections and joint optimization. The training objective follows flow matching, minimizing the squared deviation from the target velocity on the interpolant between the noisy prior and the ground truth, with theoretical analysis affirming that residual flow strictly improves error relative to either direct MLP regression or flow matching without residualization whenever the prior carries any meaningful signal.

Experimental Evaluation

Robotic Policy Decoding (VLA Tasks)

DRIFT is instantiated with both Qwen3-VL-2B and OpenVLA backbones. The method is evaluated on challenging manipulation benchmarks (Libero, Simpler WidowX) and compared to state-of-the-art robotic VLMs, diffusion/flow-based policies, and direct regression approaches.

• Libero: DRIFT + Qwen3-VL-2B achieves 97.9% average success, outperforming strong diffusion and flow-based competitors as well as regression-based adapters.
• Simpler: On robotic simulation benchmarks, DRIFT achieves 61.5%, surpassing contemporaries especially where base pretrained representations are strong.

These results hold both for regression-based prior heads (Qwen3-VL-2B, which lacks explicit continuous output heads) and discrete-token-based heads (OpenVLA), demonstrating DRIFT’s flexibility.

Figure 3: Qualitative improvement in action trajectory stability and event boundary sharpness achieved through DRIFT’s iterative residual flow refinement.

Temporal Video Grounding (TVG)

Using ET-Chat as a backbone, DRIFT is applied to TVG on Charades-STA and ActivityNet Captions. Here, the approach replaces a cosine-matching head with an MLP prior, refined by residual flow.

• On Charades-STA, DRIFT surpasses the best previous generalist models by 2.6% ([email protected]); on ActivityNet Captions, it leads by 2.1% ([email protected]), also improving or matching state-of-the-art on mIoU and high-IoU recall metrics.
• DRIFT capably handles multimodal, high-variance boundaries—producing lower-variance, more concentrated temporal segmentation distributions.

  Figure 4: DRIFT boundary predictions illustrate refinement from coarse initial estimates to sharp, high-precision localization, mitigating annotation ambiguities in TVG.

Generalization and Ablation

• Spatial Grounding: Integrating DRIFT into Qwen3-VL-2B on RefCOCO series datasets yields consistent accuracy improvements over regression-only baselines.
• World Action Models (WAMs): DRIFT applied to FastWAM (already utilizing flow matching) shows consistent additive benefit, confirming that residualization can enhance even sophisticated generative decoders.
• Ablation: DRIFT outperforms standalone MLP, Diffusion, and Flow Matching decoders. Detailed studies verify that the skip connection, direct signal prediction, and joint optimization all contribute to the observed gains.

Theoretical Implications

DRIFT's design is rigorously justified through analysis of its risk properties. The core insight is that residualization localizes the refinement task, such that the generative model only needs to handle the irreducible uncertainty (conditional variance) and nuanced nonlinearity not already captured by $g(z)$. This strictly reduces both signal and velocity field second moments, leading to lower-variance training targets, faster convergence, and improved generalization in finite-sample regimes. At the population optimum, the Bayes risk is unchanged, but practical optimization is fundamentally easier (see supplementary proofs).

Practical and Theoretical Impact

• Post-hoc Adaptation: DRIFT provides a generic recipe for retrofitting existing VLMs for high-precision, continuous-output applications without modifying backbone architectures or requiring large-scale end-to-end pretraining on continuous supervision.
• Low-Data Regimes: By confining the generative modeling to a local, data-dependent correction, DRIFT yields markedly higher sample efficiency, critical when downstream continuous annotations are scarce.
• Modular Design: As both MLP-based and tokenizer-based predictors are supported, DRIFT is broadly compatible with open-world, compositional VLM backbones.

Limitations and Future Directions

The present approach depends on the existence of a strong, informative base predictor (i.e., the backbone has learned representations correlated with the target task). In scenarios with weak or misaligned priors, DRIFT's incremental gains over direct global generative modeling may not materialize. Most experiments are in simulation, so real-world scaling and sim-to-real transfer for robotic policy adaptation merit investigation. Further, extending to more complex continuous-object outputs (structured predictions, trajectories with temporal dependencies) is a rich avenue for research.

Conclusion

This work provides a formal, empirically validated, and theoretically sound framework for adapting VLMs to precise continuous decoding tasks by integrating deterministic priors with localized generative refinement via residual flow matching. The DRIFT paradigm achieves strong improvements in perception (visual and temporal grounding) and action domains (robotic control), outcompeting existing state-of-the-art methods across multiple tasks and model families. The framework is widely applicable, modular in design, and underpinned by statistical learning guarantees, supporting future advances in hybrid discrete-continuous modeling with large-scale pretrained vision-language architectures.