Support-Constrained Off-Domain RL (SCORE)

• The paper introduces a real-to-sim-to-real framework that enforces hard support constraints from a pretrained real-data generative policy to prevent simulator exploitation.
• SCORE leverages flow-matching and latent steering to reweight actions within the base policy’s support, enabling significant improvement in real-world policy performance.
• Empirical results demonstrate that SCORE markedly boosts real-world success rates compared to unconstrained simulation RL and behavior cloning regularization baselines.

Searching arXiv for SCORE and closely related support-constrained offline RL papers. Support-Constrained Off-Domain REinforcement (SCORE) is a real-to-sim-to-real framework for policy improvement that constrains reinforcement learning in simulation to the support of a generative policy pretrained on real-world data. The setting is off-domain policy improvement: starting from a base policy trained on real data, the goal is to learn an improved policy using additional interaction only in simulation and then deploy it back on hardware, despite a mismatch between simulator and real-world dynamics (Yu et al., 25 Jun 2026). SCORE’s central claim is that the relevant constraint is support, not distributional closeness: the improved policy should be allowed to reweight behaviors already producible by the base policy, while being prevented from inventing simulator-specific actions that exploit contact and dynamics mismatch (Yu et al., 25 Jun 2026).

1. Problem formulation and off-domain setting

SCORE studies real-to-sim-to-real policy improvement using two MDPs, a simulator MDP $\mathcal{M}_{\mathrm{sim}}$ and a real-world MDP $\mathcal{M}_{\mathrm{real}}$, which share state and action spaces and reward, but differ in transition dynamics $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$ and $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$ (Yu et al., 25 Jun 2026). Real-world performance is defined as

$$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$

The paper defines off-domain policy improvement as finding $\hat{\pi}$ such that

$$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$

using additional interaction only in simulation (Yu et al., 25 Jun 2026).

The motivation is that RL on hardware is expensive, unsafe, and slow to scale, especially in contact-rich dexterous manipulation. Unconstrained RL in simulation can exploit contact modeling, friction, object motion, low-level controller behavior, and collision-model mismatch, thereby discovering policies that score well in simulation but are bad or unsafe on hardware (Yu et al., 25 Jun 2026). The appendix formalizes a related notion of realizability: a state-action pair $(s,a)$ is realizable in the real world if $\rho^\pi(s,a)>0$ for some policy $\pi$ in $\mathcal{M}_{\mathrm{real}}$0 (Yu et al., 25 Jun 2026). This suggests that the real issue is not merely policy improvement under domain shift, but policy improvement under a realizability constraint induced by real-world data.

The paper contrasts this with standard distributionally constrained objectives of the form

$\mathcal{M}_{\mathrm{real}}$1

where $\mathcal{M}_{\mathrm{real}}$2 may be instantiated by behavior-cloning or KL-style regularization (Yu et al., 25 Jun 2026). The stated failure mode is a familiar one in offline RL: weak regularization allows simulator exploitation, while strong regularization preserves slow, imprecise, and brittle modes of the base policy. Related offline RL work makes the same critique in a different setting, arguing that density matching is often more restrictive than support preservation, because it preserves the full behavior distribution rather than only excluding unsupported actions (Gao et al., 7 Mar 2025, Wu et al., 2022, Wu et al., 2022).

2. Support constraint as the central design principle

SCORE’s central objective is

$\mathcal{M}_{\mathrm{real}}$3

The paper explicitly distinguishes this from distributional regularization (Yu et al., 25 Jun 2026). Distributional constraints say, in effect, stay near the original action distribution; support constraints say, only use actions the base policy can already generate, but reweight them arbitrarily. That distinction is the method’s defining conceptual move.

This support-only view aligns closely with a broader line of offline RL work arguing that the key safety requirement is support membership rather than density matching. OSC, for example, frames policy improvement as maximizing return while maximizing the learned policy’s total probability mass on the support of the behavior policy, without preferring high-probability over low-probability in-support actions (Gao et al., 7 Mar 2025). STR makes a similar distinction by defining a support-constrained policy class $\mathcal{M}_{\mathrm{real}}$4 and optimizing within it rather than imposing density closeness to $\mathcal{M}_{\mathrm{real}}$5 (Mao et al., 2023). In that sense, SCORE is part of the same conceptual family, though it is instantiated for real-to-sim-to-real policy improvement rather than standard offline RL.

The practical support set in SCORE is induced by a frozen generative base policy. Once the base policy is trained, the model-induced action set at observation $\mathcal{M}_{\mathrm{real}}$6 is

$\mathcal{M}_{\mathrm{real}}$7

The paper states that $\mathcal{M}_{\mathrm{real}}$8 is used as the support for steering (Yu et al., 25 Jun 2026). This means support is not estimated by a thresholded density or classifier; it is instantiated as the image of the latent-to-action map of a frozen conditional flow policy.

The claimed benefit is twofold. First, actions in the base support are grounded in real data, so they are more likely to transfer. Second, because only support is constrained, RL can amplify fast or precise modes and suppress brittle or slow ones. The paper repeatedly contrasts this with KL or BC penalties, which preserve the entire base distribution, including poor modes (Yu et al., 25 Jun 2026). A plausible implication is that SCORE operationalizes the same support-only philosophy articulated in OSC and SPOT, but in an off-domain control setting where the support anchor comes from a real-data generative prior rather than a behavior model fitted to an offline benchmark (Gao et al., 7 Mar 2025, Wu et al., 2022).

3. Flow steering and policy parameterization

SCORE instantiates the support constraint through a conditional flow-matching base policy $\mathcal{M}_{\mathrm{real}}$9, trained on real-world robot data (Yu et al., 25 Jun 2026). Flow matching learns a velocity field

$p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$0

that transports samples from a prior $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$1 to the behavior distribution. With $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$2, $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$3, $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$4, and interpolant

$p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$5

the training loss is

$p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$6

At inference, one samples latent noise $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$7 and integrates

$p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$8

from $p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)$9 to obtain $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$0 (Yu et al., 25 Jun 2026).

The base flow policy is frozen during RL. What is learned in simulation is a steering policy over the latent variables of the flow model, denoted $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$1, or over a concatenated steering variable $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$2 in the residual variant (Yu et al., 25 Jun 2026). This yields two instantiations.

In DSRL, the steering policy chooses only the latent: $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$3 The paper describes this as a hard model-induced support constraint, because every action lies in $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$4 (Yu et al., 25 Jun 2026).

In RFS, the action is

$p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$5

The paper interprets this as a soft support constraint: latent steering stays anchored to the base support while a small residual permits local refinement important for dexterous precision (Yu et al., 25 Jun 2026). The PPO actor is then a single Gaussian over $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$6.

The paper emphasizes that SCORE leaves the base policy untouched, avoids distillation, and deploys the steered policy directly on hardware (Yu et al., 25 Jun 2026). This parameterization choice is also what distinguishes it from many offline RL support methods. SPOT and OSC constrain actor learning through behavior-density penalties (Wu et al., 2022, Gao et al., 7 Mar 2025); CDSA performs test-time action correction using score models and inverse dynamics (Liu et al., 2024); STR constrains policy iteration through support projection and trust regions (Mao et al., 2023). SCORE instead hard-codes the support into the policy class by steering only through the latent action manifold of a real-data generative model.

4. Optimization objective, assumptions, and theoretical interpretation

With pure latent steering, SCORE optimizes

$p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$7

This is the paper’s practical simulation objective under a hard support constraint (Yu et al., 25 Jun 2026). In RFS, the same idea applies with $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$8.

The appendix states several assumptions. One assumes simulation preserves the ordering of realizable policies: $p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)$9 for realizable $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$0 (Yu et al., 25 Jun 2026). Another assumes a realizability neighborhood: for every state $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$1 and every

$$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$2

the pair $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$3 is realizable for all $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$4 satisfying $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$5 (Yu et al., 25 Jun 2026). A third assumes realizability is closed under support: if $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$6 is realizable and $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$7 for all $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$8, then $$J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].$$9 is also realizable (Yu et al., 25 Jun 2026).

Under these assumptions, the appendix defines

$\hat{\pi}$0

and states that any support-constrained RL procedure returning

$\hat{\pi}$1

yields a realizable policy with

$\hat{\pi}$2

This is the paper’s idealized justification for support-constrained RL in the off-domain setting (Yu et al., 25 Jun 2026).

The theoretical stance differs from offline RL approaches that penalize off-support actions in the critic rather than constraining the policy class. DARC, for example, modifies reward by a transition log-density ratio

$\hat{\pi}$3

which acts as a soft transition-support penalty under dynamics shift (Eysenbach et al., 2020). MOBODY deliberately expands target support using learned dynamics and synthetic rollouts, trading strict support preservation for model-based target exploration (Guo et al., 10 Jun 2025). CEDGE similarly uses a source trajectory prior and energy guidance to perform soft, trajectory-level support control rather than a hard feasible-set restriction (Yang et al., 24 May 2026). SCORE is stricter: it does not primarily regularize values or reweight trajectories; it constrains policy optimization to the action support of a real-data generative prior (Yu et al., 25 Jun 2026).

5. Training pipeline, rewards, and experimental setting

The training pipeline is sequential. First, real-world data are collected using an Apple Vision Pro teleoperation interface (Yu et al., 25 Jun 2026). The total dataset contains 890 demonstrations across 8 tasks (Yu et al., 25 Jun 2026). Second, a conditional flow-matching base policy is trained on the real data. Third, a simulation is built in IsaacLab with domain randomization over object scale, mass, friction, spawn locations, disturbances, and mid-episode perturbations (Yu et al., 25 Jun 2026). Fourth, the base policy is frozen and a steering policy is trained in simulation using PPO with sparse rewards and an asymmetric actor-critic: the actor sees deployable observations and the critic sees privileged simulator state (Yu et al., 25 Jun 2026). Finally, the steered policy is deployed directly on the robot without distillation or base-policy finetuning.

The appendix gives the PPO actor objective for asymmetric RFS as

$\hat{\pi}$4

with ratio

$\hat{\pi}$5

The paper states that the same hyperparameters are used across tasks and that simulation uses 4096 parallel environments (Yu et al., 25 Jun 2026).

The robot platform is a Franka Research 3 arm with a LEAP Hand and a single Orbbec Femto Bolt depth camera. Policies run at 10 Hz in closed loop (Yu et al., 25 Jun 2026). Observations include robot joint positions, a 512-point XYZ point cloud, and histories of prior robot states and actions. Actions are 23-dimensional absolute joint targets predicted in chunks of horizon 16 (Yu et al., 25 Jun 2026). The base flow architecture uses PointNet, a 1D U-Net, FiLM conditioning, and 5 integration steps at inference (Yu et al., 25 Jun 2026).

The tasks are Bottle Grasp, Cup Grasp, Cube Pinch, Lightbulb Screw, Dishrack Place, Credit Card Pick, Ball Pour, and Soccer Push (Yu et al., 25 Jun 2026). Rewards are sparse and match evaluation success conditions, with shared lightweight penalties on joint velocity and action rate. For example, Bottle Grasp and Cup Grasp use $\hat{\pi}$6, Lightbulb Screw uses $\hat{\pi}$7 when cumulative rotation exceeds $\hat{\pi}$8, and Dishrack Place uses $\hat{\pi}$9 (Yu et al., 25 Jun 2026).

6. Empirical results, ablations, and relation to adjacent methods

The headline result is that average real-world success improves from $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$0 for the base policy to $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$1 for SCORE, compared to $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$2 for the best baseline (Yu et al., 25 Jun 2026). The paper also reports that SCORE reaches success in $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$3 fewer steps than the base policy on average, whereas Residual RL improves speed by $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$4 (Yu et al., 25 Jun 2026). Per-task real-world success rates for SCORE are reported as 100.0% on Bottle Grasp, Cup Grasp, Cube Pinch, and Lightbulb Screw; 90.0% on Dishrack Place; 80.0% on Credit Card Pick; 88.9% on Ball Pour; and 60.0% on Soccer Push (Yu et al., 25 Jun 2026).

The baseline comparison is structurally important. The main baselines are the base policy, unconstrained flow-policy RL fine-tuning (FPO), RialTo as a BC-regularized real-to-sim-to-real method, Residual RL, SCORE(DSRL), and SCORE(RFS) (Yu et al., 25 Jun 2026). Unconstrained FPO achieves average simulation success of $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$5 but only $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$6 average real-world success, which the paper interprets as direct evidence that unconstrained sim RL discovers unsafe or non-transferable actions (Yu et al., 25 Jun 2026). SCORE(DSRL) reaches $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$7 average real-world success and SCORE(RFS) reaches $$J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})$$8, indicating that the residual extension improves precision while the hard-support variant already captures most of the transfer benefit (Yu et al., 25 Jun 2026).

Several qualitative failures of unconstrained RL are documented. In Soccer Push, FPO learns to strike the table and trap the ball using simulator-specific contact dynamics. In Credit Card Pick, it swipes the card off the shelf. In Lightbulb Screw, it uses excessive force and twists or pulls the fixture unnaturally (Yu et al., 25 Jun 2026). The paper characterizes SCORE failures differently: they are mainly due to precision demands, contact sensitivity, or task difficulty, rather than support drift or reward hacking (Yu et al., 25 Jun 2026).

The ablations reinforce the support-based interpretation. Comparing DSRL and RFS isolates hard versus soft support constraints: both outperform baselines, but RFS is better in final performance (Yu et al., 25 Jun 2026). Varying BC regularization in RialTo exposes the familiar tradeoff already emphasized in offline RL: small coefficients improve sim performance but hurt transfer; large coefficients are overconstrained and limit improvement; intermediate coefficients help but remain well below SCORE (Yu et al., 25 Jun 2026). This is closely related to the critique of density-constrained offline RL in OSC and ReDS, which argue that matching behavior probabilities inside support biases learning toward poor but common actions (Gao et al., 7 Mar 2025, Singh et al., 2022).

The data-coverage ablations are especially informative. On Cube Pinch, more demonstrations only modestly improve the base policy but dramatically improve SCORE built on top of it; with 100 demonstrations, SCORE reaches 100% success (Yu et al., 25 Jun 2026). Adding retry demonstrations changes SCORE from 40% to 100% in the reported comparison, even though the base policy changes little (Yu et al., 25 Jun 2026). Adding play data improves SCORE from 30% to 64% (Yu et al., 25 Jun 2026). The paper interprets these outcomes as evidence that broader support coverage matters more than immediate imitation quality. This suggests a general principle: support-rich priors may be more valuable for constrained improvement than narrowly optimal priors, because reweighting within support is only as effective as the diversity of realizable behavior already encoded in the generator.

Cross-object adaptation makes the same point sharply. A bottle-grasp prior, when steered in a carrot environment, improves carrot pinch success from 22% to 67%, whereas a cup-grasp prior fails because it lacks compatible pinch behavior (Yu et al., 25 Jun 2026). The paper presents this as evidence for the support interpretation itself: when the prior does not contain the relevant behavior mode, support-constrained improvement cannot invent it. This is the main caveat of SCORE and a point of contrast with methods such as MOBODY or CEDGE, which attempt to go beyond observed support through learned target rollouts or guided trajectory generation (Guo et al., 10 Jun 2025, Yang et al., 24 May 2026).

Overall, SCORE contributes a specific answer to a recurring question in offline and off-domain RL: when the optimizer is allowed to search in a mismatched domain, what should be constrained? Its answer is the support of a real-data generative policy, instantiated by flow steering and preserved throughout simulation RL (Yu et al., 25 Jun 2026). In the broader literature, that places it alongside support-only approaches that reject full-distribution imitation as unnecessarily conservative, while distinguishing it from critic-side pessimism, density-matching regularization, and explicit support expansion methods (Gao et al., 7 Mar 2025, Mao et al., 2023, Chen et al., 18 May 2026).