---
title: Support-Constrained Off-Domain RL (SCORE)
url: https://www.emergentmind.com/topics/support-constrained-off-domain-reinforcement-score
type: topic
---

# Support-Constrained Off-Domain RL (SCORE)

Searching arXiv for SCORE and closely related support-constrained offline RL papers.
Support-Constrained Off-Domain REinforcement (SCORE) is a real-to-sim-to-real framework for policy improvement that constrains reinforcement learning in simulation to the support of a generative policy pretrained on real-world data. The setting is off-domain policy improvement: starting from a base policy trained on real data, the goal is to learn an improved policy using additional interaction only in simulation and then deploy it back on hardware, despite a mismatch between simulator and real-world dynamics [2606.27475]. SCORE’s central claim is that the relevant constraint is support, not distributional closeness: the improved policy should be allowed to reweight behaviors already producible by the base policy, while being prevented from inventing simulator-specific actions that exploit contact and dynamics mismatch [2606.27475].

## 1. Problem formulation and off-domain setting

SCORE studies real-to-sim-to-real policy improvement using two MDPs, a simulator MDP \(\mathcal{M}_{\mathrm{sim}}\) and a real-world MDP \(\mathcal{M}_{\mathrm{real}}\), which share state and action spaces and reward, but differ in transition dynamics \(p_{\mathrm{sim}}(s_{t+1}\mid s_t,a_t)\) and \(p_{\mathrm{real}}(s_{t+1}\mid s_t,a_t)\) [2606.27475]. Real-world performance is defined as
\[
J_{\mathrm{real}}(\pi) = \mathbb{E}_{\pi, p_{\mathrm{real}}} \left[ \sum_{t=1}^{T} \gamma^{t-1} r(s_t,a_t) \right].
\]
The paper defines off-domain policy improvement as finding \(\hat{\pi}\) such that
\[
J_{\mathrm{real}}(\hat{\pi}) > J_{\mathrm{real}}(\pi_{\mathrm{base}})
\]
using additional interaction only in simulation [2606.27475].

The motivation is that RL on hardware is expensive, unsafe, and slow to scale, especially in contact-rich dexterous manipulation. Unconstrained RL in simulation can exploit contact modeling, friction, object motion, low-level controller behavior, and collision-model mismatch, thereby discovering policies that score well in simulation but are bad or unsafe on hardware [2606.27475]. The appendix formalizes a related notion of realizability: a state-action pair \((s,a)\) is realizable in the real world if \(\rho^\pi(s,a)>0\) for some policy \(\pi\) in \(\mathcal{M}_{\mathrm{real}}\) [2606.27475]. This suggests that the real issue is not merely policy improvement under domain shift, but policy improvement under a realizability constraint induced by real-world data.

The paper contrasts this with standard distributionally constrained objectives of the form
\[
\hat{\pi} = \arg\max_{\pi} J_{\mathrm{sim}}(\pi)
\quad \text{s.t.} \quad
D(\pi,\pi_{\mathrm{base}}) < \epsilon,
\]
where \(D\) may be instantiated by behavior-cloning or KL-style regularization [2606.27475]. The stated failure mode is a familiar one in offline RL: weak regularization allows simulator exploitation, while strong regularization preserves slow, imprecise, and brittle modes of the base policy. Related offline RL work makes the same critique in a different setting, arguing that density matching is often more restrictive than support preservation, because it preserves the full behavior distribution rather than only excluding unsupported actions [2503.05207], [2202.06239], [2202.06239].

## 2. Support constraint as the central design principle

SCORE’s central objective is
\[
\hat{\pi} = \arg\max_{\pi} J_{\mathrm{sim}}(\pi)
\quad \text{s.t.} \quad
\mathrm{supp}(\pi) \subseteq \mathrm{supp}(\pi_{\mathrm{base}}).
\]
The paper explicitly distinguishes this from distributional regularization [2606.27475]. Distributional constraints say, in effect, stay near the original action distribution; support constraints say, only use actions the base policy can already generate, but reweight them arbitrarily. That distinction is the method’s defining conceptual move.

This support-only view aligns closely with a broader line of offline RL work arguing that the key safety requirement is support membership rather than density matching. OSC, for example, frames policy improvement as maximizing return while maximizing the learned policy’s total probability mass on the support of the behavior policy, without preferring high-probability over low-probability in-support actions [2503.05207]. STR makes a similar distinction by defining a support-constrained policy class \(\Pi=\{\pi \mid \pi(a|s)=0 \text{ whenever } \beta(a|s)=0\}\) and optimizing within it rather than imposing density closeness to \(\beta\) [2311.08935]. In that sense, SCORE is part of the same conceptual family, though it is instantiated for real-to-sim-to-real policy improvement rather than standard offline RL.

The practical support set in SCORE is induced by a frozen generative base policy. Once the base policy is trained, the model-induced action set at observation \(o\) is
\[
\mathcal{A}_{\mathrm{base}}(o) = \{\pi_{\mathrm{base}}(o,z)\mid z\in\mathcal{Z}\}.
\]
The paper states that \(\mathcal{A}_{\mathrm{base}}(o)\) is used as the support for steering [2606.27475]. This means support is not estimated by a thresholded density or classifier; it is instantiated as the image of the latent-to-action map of a frozen conditional flow policy.

The claimed benefit is twofold. First, actions in the base support are grounded in real data, so they are more likely to transfer. Second, because only support is constrained, RL can amplify fast or precise modes and suppress brittle or slow ones. The paper repeatedly contrasts this with KL or BC penalties, which preserve the entire base distribution, including poor modes [2606.27475]. A plausible implication is that SCORE operationalizes the same support-only philosophy articulated in OSC and SPOT, but in an off-domain control setting where the support anchor comes from a real-data generative prior rather than a behavior model fitted to an offline benchmark [2503.05207], [2202.06239].

## 3. Flow steering and policy parameterization

SCORE instantiates the support constraint through a conditional flow-matching base policy \(\pi_{\mathrm{base}}(a\mid o)\), trained on real-world robot data [2606.27475]. Flow matching learns a velocity field
\[
v_\theta(x_\tau,\tau,o)
\]
that transports samples from a prior \(p_0\) to the behavior distribution. With \((o,a)\sim\mathcal{D}_{\mathrm{real}}\), \(x_0\sim p_0\), \(\tau\sim\mathcal{U}[0,1]\), and interpolant
\[
x_\tau=(1-\tau)x_0+\tau a,
\]
the training loss is
\[
\mathcal{L}_{\mathrm{FM}}(\theta)
=\mathbb{E}_{\tau,(o,a),x_0}\!\left[
\left\|v_\theta(x_\tau,\tau,o)-(a-x_0)\right\|^2
\right].
\]
At inference, one samples latent noise \(z\sim p_0\) and integrates
\[
\frac{dx_\tau}{d\tau}=v_\theta(x_\tau,\tau,o)
\]
from \(x_0=z\) to obtain \(a=x_1\) [2606.27475].

The base flow policy is frozen during RL. What is learned in simulation is a steering policy over the latent variables of the flow model, denoted \(\pi_{\mathrm{steer}}(z\mid o)\), or over a concatenated steering variable \(\xi=[a_r,z]\) in the residual variant [2606.27475]. This yields two instantiations.

In DSRL, the steering policy chooses only the latent:
\[
z_k \sim \pi_{\mathrm{steer}}(\cdot\mid o_k), \qquad
a_k = \pi_{\mathrm{base}}(o_k,z_k).
\]
The paper describes this as a hard model-induced support constraint, because every action lies in \(\mathcal{A}_{\mathrm{base}}(o_k)\) [2606.27475].

In RFS, the action is
\[
a_k=\pi_{\mathrm{base}}(o_k,z_k)+a_r.
\]
The paper interprets this as a soft support constraint: latent steering stays anchored to the base support while a small residual permits local refinement important for dexterous precision [2606.27475]. The PPO actor is then a single Gaussian over \(\xi=[a_r,z]\).

The paper emphasizes that SCORE leaves the base policy untouched, avoids distillation, and deploys the steered policy directly on hardware [2606.27475]. This parameterization choice is also what distinguishes it from many offline RL support methods. SPOT and OSC constrain actor learning through behavior-density penalties [2202.06239], [2503.05207]; CDSA performs test-time action correction using score models and inverse dynamics [2406.07541]; STR constrains policy iteration through support projection and trust regions [2311.08935]. SCORE instead hard-codes the support into the policy class by steering only through the latent action manifold of a real-data generative model.

## 4. Optimization objective, assumptions, and theoretical interpretation

With pure latent steering, SCORE optimizes
\[
\underset{\pi_{\mathrm{steer}}}{\max}\;
J_{\mathrm{sim}}(\pi_{\mathrm{base}} \circ \pi_{\mathrm{steer}})
=
\mathbb{E}_{z_k \sim \pi_{\mathrm{steer}}(\cdot \mid o_k), \; a_k = \pi_{\mathrm{base}}(o_k,z_k), \; p_{\mathrm{sim}}}
\left[ \sum_{k=1}^{T} \gamma^{k-1} r(s_k,a_k) \right].
\]
This is the paper’s practical simulation objective under a hard support constraint [2606.27475]. In RFS, the same idea applies with \(a_k=\pi_{\mathrm{base}}(o_k,z_k)+a_r\).

The appendix states several assumptions. One assumes simulation preserves the ordering of realizable policies:
\[
J_{\mathcal{M}_{\mathrm{sim}}}(\pi) \geq J_{\mathcal{M}_{\mathrm{sim}}}(\pi')
\implies
J_{\mathcal{M}_{\mathrm{real}}}(\pi) \geq J_{\mathcal{M}_{\mathrm{real}}}(\pi')
\]
for realizable \(\pi,\pi'\) [2606.27475]. Another assumes a realizability neighborhood: for every state \(s\) and every
\[
a \in \text{supp}(\pi_{\mathrm{base}}(\cdot|s)),
\]
the pair \((s,a')\) is realizable for all \(a'\) satisfying \(\|a'-a\|\le \delta\) [2606.27475]. A third assumes realizability is closed under support: if \(\pi\) is realizable and \(\pi'(\cdot|s)\ll \pi(\cdot|s)\) for all \(s\), then \(\pi'\) is also realizable [2606.27475].

Under these assumptions, the appendix defines
\[
\pi^*_{\mathrm{base}} = \arg\max_{\pi \ll \pi_{\mathrm{base}}} J_{\mathcal{M}_{\mathrm{real}}}(\pi)
\]
and states that any support-constrained RL procedure returning
\[
\pi_{\mathrm{supp}} = \arg\max_{\pi \ll \pi_{\mathrm{base}}} J_{\mathcal{M}_{\mathrm{sim}}}(\pi)
\]
yields a realizable policy with
\[
J_{\mathcal{M}_{\mathrm{real}}}(\pi_{\mathrm{supp}})
\geq
J_{\mathcal{M}_{\mathrm{real}}}(\pi^*_{\mathrm{base}}).
\]
This is the paper’s idealized justification for support-constrained RL in the off-domain setting [2606.27475].

The theoretical stance differs from offline RL approaches that penalize off-support actions in the critic rather than constraining the policy class. DARC, for example, modifies reward by a transition log-density ratio
\[
\Delta r(s,a,s') = \log p_{\text{target}}(s'|s,a)-\log p_{\text{source}}(s'|s,a),
\]
which acts as a soft transition-support penalty under dynamics shift [2006.13916]. MOBODY deliberately expands target support using learned dynamics and synthetic rollouts, trading strict support preservation for model-based target exploration [2506.08460]. CEDGE similarly uses a source trajectory prior and energy guidance to perform soft, trajectory-level support control rather than a hard feasible-set restriction [2605.24810]. SCORE is stricter: it does not primarily regularize values or reweight trajectories; it constrains policy optimization to the action support of a real-data generative prior [2606.27475].

## 5. Training pipeline, rewards, and experimental setting

The training pipeline is sequential. First, real-world data are collected using an Apple Vision Pro teleoperation interface [2606.27475]. The total dataset contains 890 demonstrations across 8 tasks [2606.27475]. Second, a conditional flow-matching base policy is trained on the real data. Third, a simulation is built in IsaacLab with domain randomization over object scale, mass, friction, spawn locations, disturbances, and mid-episode perturbations [2606.27475]. Fourth, the base policy is frozen and a steering policy is trained in simulation using PPO with sparse rewards and an asymmetric actor-critic: the actor sees deployable observations and the critic sees privileged simulator state [2606.27475]. Finally, the steered policy is deployed directly on the robot without distillation or base-policy finetuning.

The appendix gives the PPO actor objective for asymmetric RFS as
\[
J_{\mathrm{actor}}(\theta)
=
\mathbb{E}_{(o,s,\xi) \sim \mathcal{B}}
\left[
\min\!\left(
\rho_{\theta}\,\hat{A},
\mathrm{clip}(\rho_{\theta}, 1-\varepsilon, 1+\varepsilon)\,\hat{A}
\right)
\right],
\]
with ratio
\[
\rho_{\theta} =
\frac{\pi_{\mathrm{steer}}(\xi \mid o)}
{\pi_{\mathrm{steer}}^{\mathrm{old}}(\xi \mid o)}.
\]
The paper states that the same hyperparameters are used across tasks and that simulation uses 4096 parallel environments [2606.27475].

The robot platform is a Franka Research 3 arm with a LEAP Hand and a single Orbbec Femto Bolt depth camera. Policies run at 10 Hz in closed loop [2606.27475]. Observations include robot joint positions, a 512-point XYZ point cloud, and histories of prior robot states and actions. Actions are 23-dimensional absolute joint targets predicted in chunks of horizon 16 [2606.27475]. The base flow architecture uses PointNet, a 1D U-Net, FiLM conditioning, and 5 integration steps at inference [2606.27475].

The tasks are Bottle Grasp, Cup Grasp, Cube Pinch, Lightbulb Screw, Dishrack Place, Credit Card Pick, Ball Pour, and Soccer Push [2606.27475]. Rewards are sparse and match evaluation success conditions, with shared lightweight penalties on joint velocity and action rate. For example, Bottle Grasp and Cup Grasp use \(10.0\,r_{\mathrm{success}}\), Lightbulb Screw uses \(10.0\,r_{\mathrm{success}}\) when cumulative rotation exceeds \(2\pi\), and Dishrack Place uses \(50.0\,r_{\mathrm{success}} + 2.0\,r_{\mathrm{return\_home}}\) [2606.27475].

## 6. Empirical results, ablations, and relation to adjacent methods

The headline result is that average real-world success improves from \(37.8\%\) for the base policy to \(89.9\%\) for SCORE, compared to \(59.5\%\) for the best baseline [2606.27475]. The paper also reports that SCORE reaches success in \(36.8\%\) fewer steps than the base policy on average, whereas Residual RL improves speed by \(19.0\%\) [2606.27475]. Per-task real-world success rates for SCORE are reported as 100.0% on Bottle Grasp, Cup Grasp, Cube Pinch, and Lightbulb Screw; 90.0% on Dishrack Place; 80.0% on Credit Card Pick; 88.9% on Ball Pour; and 60.0% on Soccer Push [2606.27475].

The baseline comparison is structurally important. The main baselines are the base policy, unconstrained flow-policy RL fine-tuning (FPO), RialTo as a BC-regularized real-to-sim-to-real method, Residual RL, SCORE(DSRL), and SCORE(RFS) [2606.27475]. Unconstrained FPO achieves average simulation success of \(61.8\%\) but only \(17.8\%\) average real-world success, which the paper interprets as direct evidence that unconstrained sim RL discovers unsafe or non-transferable actions [2606.27475]. SCORE(DSRL) reaches \(79.2\%\) average real-world success and SCORE(RFS) reaches \(89.9\%\), indicating that the residual extension improves precision while the hard-support variant already captures most of the transfer benefit [2606.27475].

Several qualitative failures of unconstrained RL are documented. In Soccer Push, FPO learns to strike the table and trap the ball using simulator-specific contact dynamics. In Credit Card Pick, it swipes the card off the shelf. In Lightbulb Screw, it uses excessive force and twists or pulls the fixture unnaturally [2606.27475]. The paper characterizes SCORE failures differently: they are mainly due to precision demands, contact sensitivity, or task difficulty, rather than support drift or reward hacking [2606.27475].

The ablations reinforce the support-based interpretation. Comparing DSRL and RFS isolates hard versus soft support constraints: both outperform baselines, but RFS is better in final performance [2606.27475]. Varying BC regularization in RialTo exposes the familiar tradeoff already emphasized in offline RL: small coefficients improve sim performance but hurt transfer; large coefficients are overconstrained and limit improvement; intermediate coefficients help but remain well below SCORE [2606.27475]. This is closely related to the critique of density-constrained offline RL in OSC and ReDS, which argue that matching behavior probabilities inside support biases learning toward poor but common actions [2503.05207], [2211.01052].

The data-coverage ablations are especially informative. On Cube Pinch, more demonstrations only modestly improve the base policy but dramatically improve SCORE built on top of it; with 100 demonstrations, SCORE reaches 100% success [2606.27475]. Adding retry demonstrations changes SCORE from 40% to 100% in the reported comparison, even though the base policy changes little [2606.27475]. Adding play data improves SCORE from 30% to 64% [2606.27475]. The paper interprets these outcomes as evidence that broader support coverage matters more than immediate imitation quality. This suggests a general principle: support-rich priors may be more valuable for constrained improvement than narrowly optimal priors, because reweighting within support is only as effective as the diversity of realizable behavior already encoded in the generator.

Cross-object adaptation makes the same point sharply. A bottle-grasp prior, when steered in a carrot environment, improves carrot pinch success from 22% to 67%, whereas a cup-grasp prior fails because it lacks compatible pinch behavior [2606.27475]. The paper presents this as evidence for the support interpretation itself: when the prior does not contain the relevant behavior mode, support-constrained improvement cannot invent it. This is the main caveat of SCORE and a point of contrast with methods such as MOBODY or CEDGE, which attempt to go beyond observed support through learned target rollouts or guided trajectory generation [2506.08460], [2605.24810].

Overall, SCORE contributes a specific answer to a recurring question in offline and off-domain RL: when the optimizer is allowed to search in a mismatched domain, what should be constrained? Its answer is the support of a real-data generative policy, instantiated by flow steering and preserved throughout simulation RL [2606.27475]. In the broader literature, that places it alongside support-only approaches that reject full-distribution imitation as unnecessarily conservative, while distinguishing it from critic-side pessimism, density-matching regularization, and explicit support expansion methods [2503.05207], [2311.08935], [2605.18320].

Source: https://www.emergentmind.com/topics/support-constrained-off-domain-reinforcement-score