Zero-Shot Policy Matching

• Zero-shot policy matching is a framework that directly maps new rewards or task descriptions to policies, enabling immediate execution without retraining.
• It employs methods like direct reward-to-policy mapping, successor features, occupancy matching, and cross-embodiment alignment to handle diverse domains.
• Practical results in robotics and navigation show robust performance, with methods achieving high success rates in cross-morphology and visual alignment tasks.

Zero-shot policy matching denotes a family of formulations in which a controller, policy representation, or decision module is constructed so that, when a new task, reward, morphology, embodiment, partner, or domain is presented at inference time, the system can immediately select or execute a policy matched to that objective without retraining, fine-tuning, or task-specific adaptation. Across the cited literature, the term spans direct reward-to-policy mappings, compositional successor-based methods, occupancy matching in imitation, morphology-aligned transfer across robots, and structured decision modules that map the current episode state to a behavior choice (Ventura et al., 23 Oct 2025, Rupf et al., 2024, Spraggett, 13 Dec 2025, Zhang et al., 2024).

1. Definition and problem setting

In the unified zero-shot RL formulation, the environment family is written as

$$\mathcal{M}^\mathcal{R} \equiv \left\{ (\mathcal{S}, \mathcal{A}, p, p_0, r, \gamma) \,\middle|\, r \in \mathcal{R} \right\},$$

so the transfer problem is explicitly one of shared dynamics and varying reward. The central object is no longer a single-task policy, but a mapping from rewards or task descriptions to action distributions. In that formulation, zero-shot policy matching means learning a representation that can be queried with a new reward and yields the policy that best matches that reward immediately, without additional training, planning, or substantial computation at test time (Ventura et al., 23 Oct 2025).

Other papers instantiate the same logic with different matching variables. In multi-floor ObjectNav, the Multi-floor Navigation Policy (MFNP) is a policy matching module that maps the current episode state to a floor-switching decision using stair existence, timestep, explored object ratio, explored-area growth, and LLM judgment (Zhang et al., 2024). In cross-embodiment manipulation, Mirage aligns a target robot’s state and image observations to the source robot’s embodiment at execution time so that the source policy can be queried directly (Chen et al., 2024). In zero-shot coordination, the relevant object is the unseen partner policy: Conditional Policy Similarity between Training and Testing partners (CPSTT) measures how closely the test partner matches the training partner from the main agent’s perspective (Yu et al., 2023).

A recurrent distinction in the literature concerns what “zero-shot” excludes. In the humanoid get-up setting, “zero-shot” means deployment on a morphology the policy never saw during training, with no fine-tuning or adaptation (Spraggett, 13 Dec 2025). In zero-shot off-policy learning, it means adapting to new tasks from reward-free offline data without additional training, finetuning, or rollouts (Asadulaev et al., 2 Feb 2026). In zero-shot offline imitation learning, it means receiving a single unseen behavior sequence at inference time and imitating it without training on that demonstration beforehand (Rupf et al., 2024).

2. Formal families and learned objects

The unified framework separates methods along two axes: direct versus compositional representations, and reward-free versus pseudo-reward-free learning. Direct methods learn the optimal action-value function itself,

$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$

with policy extraction by greedy action selection,

$\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$

Compositional methods instead learn an intermediate quantity whose structure can be combined with a new reward at inference time. For successor-based approaches, the key identity is

$Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$

when the reward admits the linear form $r=\phi^\top w_r$ (Ventura et al., 23 Oct 2025).

Family	Learned object	Representative inference
Direct representation	$Q^*(s,a,r)$	$\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$
Successor-based compositional	$\psi^\pi(s,a)$ or $M^\pi$	combine with $w_r$ or $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$0
Occupancy matching	$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$1 or $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$2	minimize occupancy distance

For compositional zero-shot RL, the successor measure is the common underlying object: $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$3 The value function is then recovered by integrating reward over future occupancy. Forward-backward methods factorize the successor measure as

$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$4

embed a reward by

$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$5

and infer the policy through

$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$6

The unifying point is that policy matching is implemented by aligning a learned occupancy representation with the downstream reward (Ventura et al., 23 Oct 2025).

Zero-Shot Off-Policy Learning makes the same relation explicit through stationary density ratios. It defines

$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$7

and shows the exact identity

$$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$8

With a Forward-Backward factorization, the ratio becomes a low-rank estimator used for training-free test-time latent optimization over $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$9 (Asadulaev et al., 2 Feb 2026).

Zero-shot offline imitation learning frames the same issue as occupancy matching. ZILOT defines the state and goal occupancies

$\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$0

and optimizes

$\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$1

Here the matching object is the entire visitation distribution rather than the next subgoal (Rupf et al., 2024).

3. Cross-embodiment and morphology matching

A prominent line of work treats zero-shot policy matching as embodiment alignment. In humanoid fall recovery, a single DRL policy is trained across seven humanoid robots—Bez1, OP3-Rot, Bez2, Bez3, Sigmaban, Wolfgang, and NUGUS—using a common action space of desired joint angle changes for the shoulder, elbow, hip, knee, and ankle pitch,

$\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$2

The observation space is morphology-agnostic, and the policy is morphology-blind: it receives no explicit robot ID, morphology vector, joint graph, or structural descriptor. Training uses SAC augmented with CrossQ, 16 parallel MuJoCo environments, about 600k time steps, a 512-512-256 network, domain randomization, and randomized fall initializations. In leave-one-out evaluation with 10 random seeds and 100 episodes per morphology, the best zero-shot transfer reaches $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$3 success with $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$4 CI $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$5; 21 of 42 off-diagonal cases exceed $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$6; the shared policy sometimes exceeds specialist baselines; and on NUGUS the shared policy improves by about $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$7 over the specialist. The morph scaling analysis further shows that diversity helps only when it forms continuous overlapping coverage of morphology space (Spraggett, 13 Dec 2025).

Mirage addresses cross-embodiment transfer for robot arms by making the unseen target robot behave as if it were the source robot that the policy was trained on. For state-based transfer, the target pose is mapped into the source frame, the source policy is queried, the source robot’s action is stepped in simulation, and the target robot is commanded to the corresponding target-frame pose. For vision-based transfer, Mirage performs “cross-painting”: mask out the target robot pixels, inpaint the missing region, render the source robot at the same end-effector pose, and overlay it to produce $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$8. The method relies on a learned source forward dynamics model and a high-gain or blocking controller rather than direct replay of source delta actions. Across 8 tasks and multiple target robots, same-gripper state-based transfer is often near source performance, while vision-based Mirage substantially improves over naive zero-shot; for example, on Lift with UR5e, Oracle is $\pi^*_r(s) \in \arg\max_a Q^*(s,a,r).$9, Naive 0-shot is $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$0, and Mirage is $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$1 (Chen et al., 2024).

DexGrasp-Zero extends the same principle to dexterous hands by replacing hand-specific state and action spaces with a morphology-aligned graph representation and tri-axial orthogonal motion primitives. Each hand $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$2 is represented as $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$3 with anatomically grounded node types—wrist, metacarpal, proximal, middle, distal, fingertip—and node features

$Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$4

The action space uses FLEX, ABD, and ROT primitives, which are mapped to executable joint commands by a fixed hand-specific mapping $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$5. The Morphology-Aligned Graph Convolutional Network (MAGCN) injects physical properties—link lengths, joint ranges, velocity limits, damping, axis geometry—at every layer, and uses an activation mask with feasibility penalty

$Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$6

Joint training on Allegro, Shadow, Ability, and Schunk yields $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$7 average success on seen hands and $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$8 success on unseen LEAP and Inspire, reported as a $Q^\pi_r(s,a) = \psi^\pi(s,a)^\top w_r,$9 improvement over the prior state of the art; real-world experiments on LEAP, Inspire, and Revo2 achieve $r=\phi^\top w_r$0 average success on unseen objects (Wu et al., 17 Mar 2026).

4. Semantic, visual, and spatial policy alignment

In multi-floor ObjectNav, zero-shot policy matching is implemented without training a new navigation policy on floor-labeled data. MFNP aggregates stair existence $r=\phi^\top w_r$1, timestep $r=\phi^\top w_r$2, discovered object ratio $r=\phi^\top w_r$3, explored-area growth $r=\phi^\top w_r$4, and LLM judgment $r=\phi^\top w_r$5, with the time factor

$r=\phi^\top w_r$6

and decision score

$r=\phi^\top w_r$7

Stair ascent is not allowed in the first 150 timesteps or the last 200 timesteps, and if $r=\phi^\top w_r$8, the agent moves to another floor via stairs. The semantic map is $r=\phi^\top w_r$9, the action space is $Q^*(s,a,r)$0, success requires stopping within $Q^*(s,a,r)$1 m of the target object, and evaluation uses SR, SPL, and DTG. On HM3D, MFNP reports SR $Q^*(s,a,r)$2, SPL $Q^*(s,a,r)$3, DTG $Q^*(s,a,r)$4, with $Q^*(s,a,r)$5 SR and $Q^*(s,a,r)$6 SPL over the previous SOTA; on MP3D, SR is $Q^*(s,a,r)$7, SPL $Q^*(s,a,r)$8, DTG $Q^*(s,a,r)$9, with $\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$0 SR. Real-world experiments show a Unitree quadruped successfully navigating multiple floors in a completely unseen environment (Zhang et al., 2024).

Prompt-based Visual Alignment addresses zero-shot policy transfer under observation-domain shift. PVA uses a frozen CLIP visual encoder $\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$1, a text encoder $\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$2, prompt tuning for global, domain-specific, and instance-conditional semantics, and a visual aligner $\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$3 that maps an image $\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$4 to an aligned image $\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$5. Training combines the global match loss

$\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$6

the patch loss, the VGG-based feature loss

$\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$7

and the total loss

$\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$8

Only 100 images from 2 domains are used to learn prompts and the visual aligner. In CARLA, with seen domains ClearNoon and HardRainNoon and unseen domains ClearSunset, WetCloudySunset, and SoftRainSunset, PVA reports 2004.6, 1825.9, 2178.4, 1775.65, and 1789.2 respectively, outperforming DARLA, CURL, LUSR, UNIT4RL, CycleGAN, pix2pix, and AdaIN in the reported tables (Gao et al., 2024).

A narrower but operationally important variant appears in zero-shot sim-to-real control of a scale autonomous vehicle. The policy is a feed-forward neural network trained only in simulation, with the same ROS2 autonomy stack and the same onboard NVIDIA Jetson hardware in simulation and reality. The error-state input is

$\pi_r^*(s)\in \arg\max_a Q^*(s,a,r)$9

and the learned mapping is $\psi^\pi(s,a)$0, where $\psi^\pi(s,a)$1. Training data come from either human-in-the-loop simulation or an MPC teacher; deployment uses no domain randomization, no domain adaptation, and no real-world corrective training. The reported result is immediate real-world trajectory following with only a small sim2real gap (Zhang et al., 2023).

5. Task composition, prompting, and occupancy-based adaptation

Disentangled task composition in meta-RL treats a task $\psi^\pi(s,a)$2 as a combination of degrees of variation,

$\psi^\pi(s,a)$3

and replaces a single encoder with $\psi^\pi(s,a)$4 encoders that produce DoV-specific latents

$\psi^\pi(s,a)$5

A KL-based disentanglement loss aligns identical DoV labels to label-specific Gaussian targets updated by EMA with $\psi^\pi(s,a)$6. In Scenario S1, all DoV labels are seen in training, and the test embedding is formed directly from stored EMA prototypes with no extra interaction. In Scenario S2, one DoV label is unseen in training; one held-out test task is used to infer that sub-embedding, which is then reused compositionally for zero-shot transfer to other tasks. On a real FANUC Mate 200iD insertion setup, training in simulation and inferring the real-world physics embedding from one held-out real task yields $\psi^\pi(s,a)$7 success over unseen real-world tasks (Wu et al., 2022).

Prompt-based zero-shot policy learning with Decision Transformers argues that demonstrations are over-specified and that the minimally sufficient prompt is the task parameter itself. The task prompt is

$\psi^\pi(s,a)$8

aligned with returns-to-go, state, and action token types, and the policy is written as

$\psi^\pi(s,a)$9

where $M^\pi$0 is an additional learnable prompt. Training minimizes

$M^\pi$1

Across Cheetah-Velocity, Ant-Direction, ML1-Reach, ML1-Push, ML1-Pick-Place, ML10, and Maze2D, Task-DT and Task-Learned-DT are reported as on par with or better than Trajectory-DT; for example, on ML1-Pick-Place, Task-Learned-DT is $M^\pi$2, Task-DT is $M^\pi$3, and Trajectory-DT is $M^\pi$4 (Song et al., 2024).

Distributional successor-feature methods replace one-step dynamics modeling with a distribution over long-horizon outcomes. In the GOM formulation, a cumulant function $M^\pi$5 induces the outcome

$M^\pi$6

and the learned object is $M^\pi$7, trained with the fixed-point objective

$M^\pi$8

A readout policy $M^\pi$9 maps desired outcomes to actions. For a new reward $w_r$0, the test-time step is

$w_r$1

followed by action decoding $w_r$2. The practical implementation uses conditional diffusion models and guided diffusion or random shooting, and reports strong performance on D4RL Antmaze, Franka Kitchen, Preference Antmaze, and Roboverse (Zhu et al., 2024).

ZILOT treats zero-shot imitation as non-myopic occupancy matching from offline, potentially suboptimal data. A single expert goal sequence $w_r$3 is given only at inference time; a learned world model $w_r$4 predicts future occupancies; and a goal-conditioned value function defines the transport cost by

$w_r$5

Planning minimizes the entropically regularized optimal transport distance between predicted learner occupancy and expert occupancy, solved with Sinkhorn and executed in MPC fashion through iCEM or MPPI. The reported qualitative conclusion is that ZILOT achieves lower Wasserstein distance to the demonstration trajectory and better goal-order adherence than hierarchical goal-following baselines, especially in tasks such as fetch_slide_large_2D where myopic behavior is catastrophic (Rupf et al., 2024).

6. Evaluation, limitations, and recurrent misunderstandings

In multi-agent coordination, policy matching is made explicit by CPSTT,

$w_r$6

which becomes the similarity between training and testing partners in zero-shot coordination. On Hanabi, cross-play score increases almost linearly with CPSTT, with Pearson correlations $w_r$7 for IQL, $w_r$8 for SAD, and $w_r$9 for VDN. Similarity-Based Robust Training perturbs partner actions so that a prescribed similarity $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$00 is enforced during training; with $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$01, $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$02, and $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$03, SBRT improves both intra- and inter-algorithm cross-play over vanilla self-play baselines and Other-Play in the reported experiments (Yu et al., 2023).

A central clarification from the zero-shot RL framework is that “zero-shot” is not a binary label but a spectrum determined by how much inference-time computation is still needed. Some methods amortize policy selection almost completely, while others still require search over latent variables, constrained optimization, or planning. The same framework also identifies recurring assumptions: support of the test reward distribution should be contained in the support of the training reward distribution for pseudo-reward-free methods; successor-feature methods require reward linearizability $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$04; and failures can arise from linearization error, inference error, or approximation error in the learned successor representation (Ventura et al., 23 Oct 2025).

Across embodiments, a recurring misconception is that more diversity is always sufficient for transfer. The humanoid get-up study reports instead that the morphologies need to form continuous overlapping coverage of the morphology space: for Sigmaban, a deliberately diverse but disconnected set such as $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$05 performs very poorly, at $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$06 (Spraggett, 13 Dec 2025). Mirage shows a related but distinct limitation: direct execution of the same delta actions on the target robot hurts performance substantially, and larger camera pose differences reduce performance even though reprojection helps; background changes are not handled well, and the Jaco 3-jaw gripper is a notable failure mode (Chen et al., 2024). PVA similarly depends on explicit semantic constraints and careful prompt design: it requires a pretrained VLM, assumes a small multi-domain dataset for prompt tuning, and is sensitive to prompt length, with overly short or overly long prompts hurting performance (Gao et al., 2024).

Another recurrent misunderstanding concerns what is minimally necessary at test time. The Decision Transformer study argues that demonstration prompts are not necessary when the model is given the underlying task parameters directly (Song et al., 2024). By contrast, the disentangled meta-RL study distinguishes a strict zero-shot composition regime, S1, from S2, in which one held-out test task is used to infer an unseen DoV embedding before transfer to the remaining tasks (Wu et al., 2022). Zero-Shot Off-Policy Learning sits between these poles: all network parameters are fixed, but latent $$Q^*:\mathcal S\times\mathcal A\times\mathcal R\to\mathbb R,\qquad (s,a,r)\mapsto Q^*_r(s,a),$$07 is still optimized at test time using occupancy-ratio estimates rather than environment interaction (Asadulaev et al., 2 Feb 2026).

This suggests that the most stable forms of zero-shot policy matching are those in which the alignment object is explicit: a reward embedding, a successor representation, an occupancy measure, a morphology-aligned control space, or a semantically constrained perceptual representation. The cited literature converges on that point even though the matched object differs sharply across RL, imitation, navigation, control, and cross-embodiment robotics.