FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space

Abstract: Pretrained generative robot policies based on flow matching and diffusion have achieved impressive results across a wide range of manipulation tasks. Yet real-world deployments routinely expose failure modes outside the pretraining distribution. Closing these gaps typically requires large-scale data collection or online reinforcement learning on physical hardware, which is impractical for rapid and safe adaptation. We present FlowDAgger, a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space. Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy, using reverse-time integration followed by local refinement. The resulting inverted noise provides supervision for a lightweight latent policy that steers the base model at deployment time, enabling rapid skill acquisition while preserving its behavioral priors. We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation, adapting both action-head VLAs and world-action models from a handful of interventions. FlowDAgger outperforms supervised fine-tuning and latent-space RL baselines and preserves pretrained skills on held-out tasks, offering a practical path for adapting robot foundation models in the real world. Website: https://microsoft.github.io/FlowDAgger

• The paper demonstrates that mapping human corrections into the latent space via action inversion efficiently adapts frozen generative policies.
• It leverages per-step fixed-point iteration for inversion, achieving over an order of magnitude improvement in reconstruction accuracy compared to alternatives.
• Experiments on simulated and real-world tasks show a mean success rate improvement of +0.25, preserving pretrained priors while ensuring compute-efficient adaptation.

FlowDAgger: Latent-Space Human-in-the-Loop Adaptation for Generative Robot Policies

Introduction and Motivation

FlowDAgger addresses a critical challenge in robotic manipulation: adapting large, pretrained generative policies to novel real-world scenarios where out-of-distribution errors often surface. Despite the broad generalization capabilities of foundation models constructed using flow-matching or diffusion-based methods, practical deployment frequently exposes failure modes due to unmodeled objects, dynamics, or idiosyncratic embodiment differences. Standard adaptation protocols—offline fine-tuning, online RL, or residual policy learning—either demand prohibitive data/compute or risk catastrophic forgetting and policy drift. FlowDAgger introduces a highly sample- and compute-efficient alternative for this problem, leveraging human-in-the-loop interventions mapped through policy inversion into the latent space of the underlying generative model.

Methodology: Action Inversion and Latent Policy Supervision

FlowDAgger comprises two principal components. First, it enables action inversion: mapping a corrective action—intervened by a human expert—back into the latent noise that would produce this action under the frozen base generative policy. For flow-matching or diffusion-head policies, the approach uses per-step fixed-point iteration to robustly invert the forward ODE integrator, ensuring high-fidelity reconstructions even for the few-step schedules common in robot action generation. For world-action models (WAMs) with entangled action and state generation (e.g., Cosmos-Policy), a modified inversion is introduced that targets the joint process.

Figure 1: A schematic of FlowDAgger. Human interventions are mapped into the latent (noise) space of a frozen generative policy via action inversion, enabling efficient adaptation through a lightweight latent-space policy.

Once inverted noise vectors are obtained for each intervention, FlowDAgger aggregates these into a dataset for a lightweight, state-conditioned latent policy. This noise policy is trained online to regress towards the inversion targets, directly modulating the noise input to the frozen base policy at each timestep. Importantly, the base, high-capacity model remains constant throughout, with adaptation capacity provided solely by the low-dimensional latent policy.

Experimental Validation

MetaWorld Adaptation and Baseline Comparison

FlowDAgger is evaluated on a diverse suite of simulated manipulation tasks drawn from MetaWorld, as well as on real-world bimanual and unimanual robotic tasks. The primary architectural baselines include:

• SFT: Full-parameter supervised fine-tuning via behavioral cloning
• LoRA-DAgger: DAgger with low-rank adaptation
• Residual-DAgger: Human-gated DAgger in action space with base policy frozen
• DSRL: Noise-space RL via sparse rewards
• Frozen Base: No adaptation

Across twelve MetaWorld tasks, FlowDAgger consistently outperforms both weight-space and action-space baselines in terms of improvement over the frozen base, with a mean success rate increment of +0.25 versus considerably smaller improvements from other methods. Its superiority is most pronounced on tasks where straightforward action residuals are ill-conditioned, highlighting the advantage of steering within the generative manifold rather than unconstrained action or weight space.

Figure 2: Success rate versus adaptation rollouts for different adaptation methods on $\pi_{0.5}$; FlowDAgger consistently achieves faster and higher success than baselines.

Compute Efficiency and Policy Prior Preservation

Training efficiency is a critical concern for robotics; FlowDAgger requires only the compute footprint needed for inference with the frozen base plus a small MLP, in contrast to full fine-tuning approaches. This enables policy adaptation to run on modest consumer hardware without sacrificing competence.

Central to FlowDAgger's design is preservation of pretrained priors. On tasks where the base policy already approaches saturation, FlowDAgger's adaptation incurs minimal negative transfer or catastrophic forgetting, whereas action- and weight-space DAgger methods erode success on held-out tasks through overfitting or instability.

Generalizability Across Model Families

The efficacy of FlowDAgger is demonstrated across both action-head Vision-Language-Action (VLA) models (e.g., $\pi_{0.5}$, Gr00t N1.7) and joint world-action models (e.g., Cosmos-Policy), with consistent improvements observed independent of the underlying generative architecture. The inversion machinery is shown to generalize to both ODE-based flow-matchers and EDM-based diffusion processes, supporting a broad spectrum of robot foundation models.

Real-World Robotic Evaluation

On several real-robot tasks spanning single- and bimanual manipulation, FlowDAgger achieves substantial performance gains with as few as five to twenty intervention episodes per task. For example, on Toolbox Packing, ten interventions raise the base policy's accuracy from 13% to 80%—well above full parameter fine-tuning—without fine-tuning the foundation model.

Additional Experimental Analyses

The paper provides further empirical substantiation regarding the inversion process itself, demonstrating via ablation that per-step fixed-point inversion yields more accurate action reconstructions (by over an order of magnitude in MSE) and better downstream adaptation performance than explicit Euler, trajectory-level, or optimization-based inverses.

Figure 3: FlowDAgger rapidly adapts Gr00t N1.7 on previously unseen tasks, reaching near-perfect success with minimal additional data, whereas DSRL plateaus at lower performance.

Figure 4: FlowDAgger extends to vanilla diffusion policies, reaching asymptotic task success rates substantially higher than those possible via reward-based RL in latent space.

Theoretical and Practical Implications

FlowDAgger demonstrates that action-space human corrections, when mapped into the noise space of frozen generative models, are a sufficient and highly effective supervision signal for continual robot policy adaptation. By ensuring that all adaptation occurs within the data manifold spanned by the foundation model—rather than through unconstrained weight or action modifications—FlowDAgger preserves broad prior competence and mitigates regression on previously mastered tasks.

Practically, this approach enables sample-efficient, real-time adaptation to long-tail deployment errors without any retraining of large vision-language or world-model backbones, providing a scalable pathway for integrating human feedback in both simulation and on hardware. The method is immediately relevant for advancing deployment robustness of robot foundation models, particularly in environments exhibiting significant task variability and covariate shift.

Theoretically, FlowDAgger suggests a decoupling of representation learning and task adaptation in robotics: large, frozen generative models serve as stable priors, while lightweight latent policies allow rapid correction and specialization under human supervision. This perspective may generalize to other inference-time controllable generative agents, opening directions in modular foundation model deployment across diverse domains.

Limitations include the fact that adaptation capacity is upper-bounded by the foundational policy's coverage: behaviors far outside the manifold are not accessible without collecting new data and fine-tuning. Policy improvement is also limited to regions where human supervision is provided, as with other DAgger-style approaches, and inversion accuracy can be affected by sharp multimodality or conditioning ill-posedness.

Conclusion

FlowDAgger provides a robust, scalable framework for adapting pretrained robotic foundation models via human-in-the-loop corrections, exploiting latent-space steering for efficient policy improvement. By marrying accurate action inversion with an interplay of autonomous and intervention-driven noise supervision, FlowDAgger achieves strong gains in sample efficiency, preserves policy priors, and extends across a spectrum of modern policy architectures. This methodology encourages further research on decoupled, interaction-driven adaptation for large foundation models in embodied AI, with promising implications for lifelong learning, deployment safety, and rapid correction of out-of-distribution errors.