Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience

Abstract: Robots trained on real world data tend to be imprecise, slow, and brittle to perturbations. Improving these policies with reinforcement learning (RL) is an appealing alternative, but this process often requires expensive training in the real world. Performing policy improvement in simulation instead provides a far cheaper alternative, but unconstrained RL in simulation can exploit contact and dynamics mismatches, resulting in unsafe behaviors that do not transfer to hardware. Common forms of regularization can furthermore limit improvement by overconstraining to an imperfect behavior prior. In this work, we propose Support-Constrained Off-Domain REinforcement (SCORE), a real-to-sim-to-real framework that constrains RL in simulation to the support of a generative policy pretrained on real data. We instantiate this constraint through flow steering, restricting SCORE to actions the base policy can already produce, which ensures transferable behaviors while maximizing policy improvement. Improving a policy with SCORE requires minimal effort: it learns from sparse rewards, avoids distillation, and leaves the base policy untouched. Across eight real-world dexterous multi-fingered robotic manipulation tasks, SCORE improves average success rate from 37.8% to 89.9%, compared to 59.5% for the best baseline, and reaches success in 36.8% fewer steps than the base policy. Ultimately, through extensive experiments and ablations, we show that simulation can substantially improve real-world manipulation policies when policy optimization is appropriately constrained, introducing a new paradigm for real-to-sim-to-real policy improvement. Videos and code are available at https://weirdlabuw.github.io/score/.

• The paper demonstrates that support-constrained RL via the SCORE framework safely improves dexterous manipulation by using only simulated interactions.
• It employs a frozen base policy with latent flow steering to strictly remain within real-world action support, boosting success rates from 37.8% to 89.9% and execution speed by 36.8%.
• Extensive evaluations show that enforcing support constraints avoids reward hacking and transfer failures, highlighting the importance of diverse demonstration data for effective policy refinement.

Support-Constrained RL for Real-World Policy Improvement Without Real-World Experience

Problem Formulation and Motivation

The paper introduces Support-Constrained Off-Domain REinforcement (SCORE), addressing the critical challenge of policy improvement for real-world dexterous manipulation without additional costly and risky real-world data collection. Imitation-learned policies, while effective for deploying robotic skills, are typically suboptimal: they inherit limitations from the demonstration data (e.g., slowness, imprecision, failure to recover). Efforts to improve such policies via on-hardware RL are generally infeasible due to high operational costs, safety concerns, and low sample efficiency. Sim-to-real approaches, where the policy is improved in simulation before real-world deployment, are appealing but suffer from the “reality gap”: unconstrained RL in simulation exploits subtle dynamics mismatches, leading to policies that may be unsafe, suboptimal, or fail on hardware.

Conventional methods mitigate these issues with distributional constraints (e.g., BC loss, KL-divergence regularization) to keep policies near the base distribution during simulation fine-tuning. However, these induce a fundamental tradeoff between policy improvement (which would benefit from relaxed constraints) and transferability (which requires strict constraints and thus limits achievable gains). The paper highlights that policy improvement should be constrained to the support (not just the distribution) of the pretrained real-world policy: only actions demonstrated in the real world should see probability mass during optimization.

The SCORE Framework

SCORE formalizes off-domain policy improvement: enhancing a base policy $\pi_{\mathrm{base}}$ trained in the real world using only simulated interaction so that $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$, without any real-world experience during optimization. The solution departs from both unconstrained and distributionally-constrained methods by enforcing a strict support constraint: the improved policy acts only within the action support of the base policy for all observations. This guarantees transferability, as all seen behaviors already occurred in the real world, while still allowing significant policy refinement by maximizing performance over the support set (e.g., emphasizing fast, robust modes).

Technically, SCORE instantiates support constraints via "flow steering." The pretrained base policy, a conditional flow matching model trained on real data, is frozen. RL is done in simulation by learning a lightweight steering policy over the latent inputs of the base model, altering the base’s internal sampling (e.g., by biasing the latent variable $z$ to prefer different action modes), but never leaving its support. Two implementations are provided: pure latent steering (hard support constraint; all actions lie strictly inside the model-induced support) and residual flow steering (soft constraint; small additive residuals are allowed).

Methodology and Experimental Pipeline

The method proceeds in three phases:

1. Pretraining: A generative flow-matching policy is trained via imitation learning on human demonstrations (teleoperation with diverse behaviors, including suboptimal and retry trajectories). Observations include point clouds and proprioception.
2. Simulation Policy Improvement: The environment is recreated and randomized in IsaacLab using scanned meshes. The base policy is frozen; RL (PPO with an asymmetric actor-critic) optimizes a steering policy using only privileged state in the critic, ensuring sample efficiency and stability under sparse reward regimes.
3. Deployment: The composed policy (base + steering) is deployed on the robot without additional finetuning or distillation.

Empirical Evaluation

SCORE is systematically evaluated on eight dexterous manipulation tasks (e.g., grasping, reorientation, pushing) using a Franka FR3 + LEAP hand system, with performance measured via real-world average success rates and execution speed. Baselines include unconstrained RL (FPO), distributionally-regularized RL (RialTo, Residual RL), and ablations.

Key results:

• SCORE improves average real-world success rate from 37.8% to 89.9% (best baseline achieves 59.5%). Every task sees improvement; three tasks transition from <11% to >80-100% success (Ball Pour, Credit Card Pick, Cube Pinch).
• Execution speed increases by 36.8% compared to base policies, surpassing Residual RL (19.0% improvement).
• In simulation, unconstrained RL agents exhibit reward hacking and catastrophic transfer failures (e.g., learning to strike the table in Soccer Push, exploiting mis-modeled contacts), while distributional penalties overly constrain improvement and can entrench suboptimal behavior.
• Empirical ablations establish that SCORE’s gains are directly linked to the diversity of the base policy’s support, not just its average performance: retry/play data that do not yield observable improvements in the unconditional policy can dramatically expand post-training capabilities when steering is applied.

Theoretical Implications and Analysis

A formal analysis in the paper demonstrates that unconstrained RL in simulation can yield unrecoverable policies due to overlooked real-world constraints. Distributional constraints can fail when the optimal improvement is outside any small divergence neighborhood, especially in continuous action spaces exhibiting "mode-missing." In contrast, support constraints are both necessary and sufficient for safe improvement as long as the base policy provides adequate coverage. This is substantiated by showing that, under reasonable assumptions (support realizability is preserved, simulation is a faithful evaluator for in-support policies), support-constrained RL can always recover the optimal in-support real-world policy.

Practical Design Insights and System Engineering

The method is highly sample-efficient, leveraging up to 4096 parallel simulation environments and an asymmetric actor-critic to inject privileged state without sacrificing deployability. The implementation uses action chunks in absolute joint space to maximize transfer and robustness to low-level parameter miscalibration. Artificially injecting failures during simulation RL preserves coverage of recovery behaviors, preventing catastrophic policy forgetting.

A critical discovery is that broader, more diverse pretraining data (including retries, suboptimal "play" actions, and multi-task demonstrations) do not necessarily yield a better unconditional policy but substantively enhance the "steerability" of the policy post-optimization. Conversely, when the base support is missing modes required for new environments (e.g., unseen objects, distractors), SCORE’s improvement is limited—a stark demonstration of the method’s safety and its reliance on dataset support.

Relation to Prior Art

While prior methods address sim-to-real transfer via distributional regularization [3], or leverage flow/diffusion policies for RL [9,11,12], SCORE is unique in guaranteeing hard support constraints tied to real-world data. Compared to methods like ExpertGen [24] and RFS [12], which use synthetic priors and distillation, SCORE works exclusively with priors derived from real hardware data, and can deploy the steered policy directly, bypassing the risk and complexity of additional finetuning/distillation steps.

Implications and Future Directions

SCORE points toward a promising paradigm in robotics where simulation is repurposed as a tool for improving existing physical-world policies, not merely generating new ones. The method’s efficacy hinges on the coverage and diversity of the original demonstration data. As such, large-scale, generalist datasets (e.g., task-agnostic play, broad multi-task collections) and pretraining regimes optimized for steerability rather than immediate performance may become the linchpin for future policy improvement frameworks. Scaling SCORE to high-diversity or open-world settings will require advances in capturing, curating, and exploiting broad behavior priors.

Conclusion

SCORE establishes that support-constrained RL in simulation is both effective and safe for real-world policy improvement. By learning only within the action support of a real-world generative prior, the method preserves transferability while substantially increasing performance and efficiency across a range of dexterous manipulation tasks. The framework avoids hand-tuned reward functions, does not rely on real-world finetuning, and underscores the critical role of data diversity and support in real-to-sim-to-real pipelines. Extensions to larger-scale, more diverse data regimes and improved support estimation offer a promising direction for scalable robot learning.