Cross-Domain Skill Diffusion

• Cross-domain skill diffusion is a framework that transfers semantically abstract, high-level skills between domains by factoring out task specifics.
• It employs diffusion models, hierarchical decompositions, and latent embeddings to overcome challenges like dynamics mismatch and embodiment differences.
• Applications span robotics, image composition, recommendation systems, and web agents, demonstrating enhanced transfer, robustness, and efficiency.

Cross-domain skill diffusion denotes a family of transfer mechanisms in which reusable skills, semantic subroutines, trajectories, or interaction patterns are learned in one domain and then adapted, regenerated, composed, or grounded in another. Across the recent literature, the source–target gap may involve environment dynamics, agent embodiment, task horizon, object categories, robots, websites, visual style, or recommendation spaces, while the transferred object may be a semantic skill, a prototype skill latent, an object-centric motion flow, a target-domain user embedding, or an externalized skill package (Huang et al., 16 Jan 2026, Yoo et al., 4 Sep 2025, Niu et al., 2024). The central technical problem is to preserve behaviorally meaningful invariants while discarding or reparameterizing domain-specific factors.

1. Semantic abstraction as the precondition for transfer

A recurring premise is that direct transfer of low-level actions is usually too brittle under embodiment or dynamics mismatch, whereas higher-level skill abstractions can remain stable. In semantic skill imitation, semantic skills are defined as high-level, discrete, context-independent actions such as “opening the microwave” or “turning on the stove,” and the transfer objective is to imitate the demonstrated semantic skill sequence rather than joint-level behavior (Pertsch et al., 2022). This formulation enables transfer across environments and embodiments, including real-world human videos to a simulated kitchen, and it matched demonstration-accelerated RL methods that require in-domain demonstrations while using less than 3 minutes of human video demonstrations (Pertsch et al., 2022).

Later systems refine this abstraction in more structured ways. SemTra uses a two-level hierarchy of task adaptation and skill adaptation: a pretrained LLM translates extracted source-domain skills into a target-domain semantic skill sequence, and a separate context-sensitive stage instantiates each skill for the target domain (Shin et al., 2024). SemGro makes the hierarchy explicit by distinguishing short-horizon, low-semantic skills that are broadly portable from long-horizon, rich-semantic skills that are specialized to particular domains; grounding then proceeds by iterative decomposition until each planned skill is executable in the target environment (Shin et al., 2024). XSkill replaces manually defined semantic labels with learned cross-embodiment skill prototypes discovered from unlabeled human and robot videos, then conditions a diffusion policy on those prototypes for robotic execution (Xu et al., 2023).

Taken together, these works establish a consistent interpretation of cross-domain skill diffusion: transfer depends less on raw behavioral cloning than on the existence of an intermediate representation that factorizes “what to do” from “how to do it here.” This interpretation is directly instantiated in semantic skills, prototype skills, skill prototypes, and hierarchical decompositions across multiple papers (Pertsch et al., 2022, Shin et al., 2024, Shin et al., 2024, Xu et al., 2023).

2. Transferable units and representational design

The literature does not use a single universal skill object. Instead, each system defines a transferable unit appropriate to its domain, and the design of that unit largely determines the achievable cross-domain invariance.

System	Transferable unit	Core representational mechanism
SADiff	Skill-aware latent embeddings; object-centric motion flow	Learnable skill tokens; skill-aware encoding; skill-retrieval transformation
ICPAD	Domain-agnostic prototype skills	Skill encoder; domain encoder; domain-grounded skill adapter; dynamic domain prompting
DuSkill	Domain-invariant and domain-variant embeddings	Hierarchical encoding; guided diffusion-based skill decoder
XSkill	Skill prototypes	Self-supervised prototype discovery; conditional diffusion policy
SkillMigrator	Transferable Interaction Patterns (TIPs)	Intent; operation template; slot schema; structural sketch
SkillX	Planning, functional, and atomic skills	Multi-Level Skills Design; Iterative Skills Refinement; Exploratory Skills Expansion

In robotic manipulation, SADiff explicitly introduces learnable skill tokens $\mathbf{s}_k$ and a skill-aware encoding module that produces skill-aware latent embeddings used to condition diffusion; the generated object-centric motion flow is later converted into executable 3D actions through a skill-retrieval transformation (Huang et al., 16 Jan 2026). ICPAD instead learns domain-agnostic prototype skills offline and couples them with a domain-grounded skill adapter, so that prototype skills act as a “lingua franca” across domains (Yoo et al., 4 Sep 2025). DuSkill makes the invariance structure explicit by disentangling skill embeddings into a domain-invariant component $z_\rho$ and a domain-variant component $z_\sigma$ (Kim et al., 2024).

In cross-embodiment imitation, XSkill discovers $K$ learnable skill prototypes from unlabeled human and robot videos and uses them as discrete shared anchors across embodiments, rather than relying on aligned demonstrations or explicit task labels (Xu et al., 2023). In web agents, the transferable unit is no longer a motion primitive but a Transferable Interaction Pattern: a skill paired with an intent, an operation template, a slot schema, and a structural sketch of the inducing page layout, with no site-specific identifiers stored (He et al., 16 Jun 2026). In agentic knowledge-base construction, SkillX organizes reusable knowledge into planning, functional, and atomic skills, thereby turning trajectories into a reusable hierarchy rather than a flat memory buffer (Wang et al., 6 Apr 2026).

At the workflow level, SkillFlow formalizes a task instance as $\mathcal{T}=(V,E,\lambda,\gamma)$ and its domain-agnostic execution flow as $\mathcal{F}=\phi(\mathcal{T})=(V_F,E_F,\lambda_F)$, so that transfer is defined over operation types and dependency structure while task-specific grounding is stripped away (Zhang et al., 19 Apr 2026). A plausible implication is that representational success depends on choosing the invariant at the correct granularity: too low-level and transfer collapses under domain shift; too high-level and executability becomes ill-posed.

3. Diffusion as translator, editor, and compositor

Diffusion is used in at least four distinct ways. First, it serves as a skill-conditioned action generator. SADiff conditions a diffusion model on skill-aware embeddings and generates object-centric motion flow, while ICPAD formulates the generation of action sequences as a DDPM conditioned on state, domain embedding, and prototype skill (Huang et al., 16 Jan 2026, Yoo et al., 4 Sep 2025).

Second, diffusion serves as a disentangled decoder. In DuSkill, the denoising field is explicitly decomposed into domain-invariant and domain-variant components,

$$\epsilon(x^k, k, s_t, z_\sigma, z_\rho) = (1-\delta)\,\epsilon_\rho(x^k, k, s_t, z_\rho) + \delta\,\epsilon_\sigma(x^k, k, s_t, z_\sigma),$$

so executable behavior can be generated by balancing invariant behavior patterns against domain-specific modulation (Kim et al., 2024).

Third, diffusion is used as a data-level editor rather than a policy model. xTED adds noise to source-domain trajectories and denoises them with a model trained on target-domain trajectories, thereby transforming source trajectories into target-aligned ones while preserving their semantic information (Niu et al., 2024). Its training objective is the standard denoising loss

$$\mathcal{L}_\theta = \mathbb{E}_{k,\mathbf{x}_0,\epsilon} \left[ \|\epsilon-\epsilon_\theta(\mathbf{x}_k,k)\|^2 \right].$$

BDGxRL replaces standard DDPM editing with a Diffusion Schrödinger Bridge and defines dynamics alignment as a stochastic optimal transport problem,

$$\mathbb{P}^* = \arg\min_{\mathbb{P}\in\mathcal{M}(\Pi_0,\Pi_1)} \left\{ \mathrm{KL}(\mathbb{P}\,\|\,\mathbb{Q}) \;\middle|\; \mathbb{P}_0=\Pi_0,\ \mathbb{P}_1=\Pi_1 \right\},$$

then combines aligned transitions with reward modulation so policy learning can remain entirely in the source domain (Zhang et al., 27 Feb 2026).

Fourth, diffusion is compositional. GSC learns a diffusion model for each primitive skill distribution $q_\pi(s,a,s')$ and composes these models at inference by summing and subtracting score terms so that multi-skill plans can be sampled jointly rather than greedily (Mishra et al., 2023). This shifts diffusion from imitation or editing to long-horizon planning over skill chains.

These formulations show that “skill diffusion” is not a single algorithmic motif. It may denote denoising in action space, latent space, transition space, or workflow space, provided the denoising process preserves transferable semantics while adapting domain-specific structure.

4. Robotics and reinforcement learning

Robotic manipulation is the domain in which the phrase is used most literally. SADiff is designed for robust generalization in manipulation by exploiting shared motion patterns among tasks within the same skill, and it introduces IsaacSkill, a high-fidelity dataset containing fundamental robotic skills for comprehensive evaluation and sim-to-real transfer (Huang et al., 16 Jan 2026). XSkill addresses a different transfer axis—human to robot—and reports only about a 5% drop in success when transferring from human to robot, with simulation performance up to approximately 90% and real-world unseen tasks around 60% versus less than 25% for baselines, while its Skill Alignment Transformer improves robustness to speed mismatches and perturbations (Xu et al., 2023).

ICPAD targets long-horizon multi-task RL under stringent constraints: no model updates in the target domain and only limited target-domain data. On Metaworld, it improves success rates by 14%–20% over the closest competitor, and at 8% task coverage its performance drops by only 6.8% versus 32% for DCMRL; on CARLA it achieves 11.6%–21.6% higher normalized returns under variations in vehicle embodiment and weather (Yoo et al., 4 Sep 2025). The framework’s dynamic domain prompting also extends to language-based prompting.

Offline and cross-domain RL works place greater emphasis on robustness under limited target interaction. DuSkill reports only a 7.6% reward drop as domains diverge from training, compared with 25.3% for the best baseline, and in new domains it outperformed SPiRL-c by 89.16%; removing either the hierarchical encoder or the guided diffusion decoders caused an 8–9% average performance drop (Kim et al., 2024). xTED demonstrates that data reuse without adaptation can be harmful: directly augmenting target data with unedited source trajectories often degrades performance, whereas xTED-edited data yielded +50% and +20% normalized return improvements in Walker2d and HalfCheetah, and in real-robot experiments it often increased success rates by over 100% or even 200%, with some target-plus-source settings dropping to zero success before editing (Niu et al., 2024). BDGxRL further reports a D4RL-normalized score of 89.5 on Walker2d with a friction gap and Medium-Expert demonstrations, compared with 85.4 for DARAIL and 77.2 for GAIL, while requiring no target-environment interaction or target rewards (Zhang et al., 27 Feb 2026).

For long-horizon planning, GSC evaluates on nine long-horizon manipulation tasks and uses classifier-based guidance to enforce constraints during joint diffusion sampling; on constrained packing tasks, test-time constraints can improve success up to 1.0, and the framework supports closed-loop replanning on a real Franka Panda robot (Mishra et al., 2023). A separate line of work, the robotic skill learning system built on diffusion policies and foundation models, shows how diffusion-based low-level skill acquisition can be embedded in an interactive system with LLM skill selection and VLM precondition checking: multi-view observations improved crate-pushing IoU from 39% to 83% in simulation, real-world tasks reached 60–90% success with 50–101 demonstrations, GPT-4 achieved 96.3% skill-match accuracy, and the best combined LLM·VLM pipeline reached about 74.6% end-to-end success (Ingelhag et al., 2024).

5. Generalization beyond control: images, recommendation, and web agents

Outside control, the same transfer logic reappears with different semantics. TF-ICON formulates cross-domain image-guided composition as seamless insertion of a reference object into a target visual context without additional training, fine-tuning, or optimization. Its “exceptional prompt” is an information-less prompt used only during inversion, and when coupled with Stable Diffusion and DPM-Solver++ it yields nearly-perfect inversion on CelebA-HQ with MAE=0.019, LPIPS=0.047, and SSIM=0.918, while composition results outperform prior baselines across photorealistic, painting, sketch, and cartoon domains (Lu et al., 2023). Here, “skill diffusion” denotes the transfer of structure, style, and semantic alignment through composite self- and cross-attention injection rather than motor behavior.

In recommendation, diffusion replaces deterministic mapping modules. DiffCDR interprets a diffusion probabilistic model as a generative mapping from source-domain to target-domain user embeddings, stabilized by an Alignment Module and specialized by a task-oriented loss, and it reports superior performance on Amazon-derived cold-start and warm-start tasks (Xuan, 2024). MuSiC extends this logic by using multimodal LLMs to extract item and user features, then training a cross-domain diffusion module in two stages: side users teach the target-domain feature distribution, and overlapping users teach the cross-domain conditional mapping. The resulting vectorized U-Net generates target-domain embeddings for cold-start users and achieves state-of-the-art performance on the Amazon dataset (Zhang et al., 5 Jul 2025).

Web and agentic systems move the notion of skill from latent control to reusable external procedures. SkillMigrator stores web skills as TIPs and retrieves them by a weighted combination of textual similarity and layout similarity based on tree edit distance over accessibility-tree sketches. On WebArena and Mind2Web, it reduces average LLM-action count on successful trajectories by 8–10% at matched or near-matched success rate, and on the hardest Mind2Web cross-domain split it attains 63.0% success with 35.4% skill reuse using a smaller library than PolySkill (He et al., 16 Jun 2026). SkillX builds a plug-and-play skill knowledge base through Multi-Level Skills Design, Iterative Skills Refinement, and Exploratory Skills Expansion; when transferred to weaker base agents, the library improves success by roughly 8–10 absolute points across AppWorld, BFCL-v3, and $z_\rho$0-Bench (Wang et al., 6 Apr 2026). SkillFlow then turns lifelong cross-task transfer into a benchmark: it contains 166 tasks across 20 families organized by Domain-Agnostic Execution Flow, and lifelong skill evolution improves Claude Opus 4.6 from 62.65% to 71.08% (+8.43 points), while Kimi K2.5 gains only +0.60 points despite 66.87% skill usage (Zhang et al., 19 Apr 2026).

6. Evaluation, misconceptions, and failure modes

Evaluation protocols vary with the transferred object. In embodied instruction-following, SemGro evaluates Success Rate, Completed Goal Conditions, planning accuracy, and executability across 300 cross-domain VirtualHome scenarios with object-location, physical-attribute, and room-structure shifts, and it reports gains over prior state of the art up to $z_\rho$1 in Success Rate (Shin et al., 2024). In web agents, average LLM-action count, success rate, skill reuse rate, and library size are central because transfer is useful only if it reduces policy-facing deliberation without lowering completion (He et al., 16 Jun 2026). In image composition, reconstruction and semantic metrics such as MAE, LPIPS, SSIM, and CLIP scores are appropriate because the transferred “skill” is visual consistency rather than action execution (Lu et al., 2023).

Several misconceptions are explicitly challenged. First, high skill usage does not imply high utility: SkillFlow shows that Kimi K2.5 can reach 66.87% skill usage with only +0.60 points, and Qwen-Coder-Next can still regress relative to the vanilla setting even while using skills (Zhang et al., 19 Apr 2026). Second, a fixed abstraction level is not sufficient for cross-domain grounding: SemGro reports that using either the highest or the lowest abstraction level performs poorly, whereas adaptive iterative decomposition is essential (Shin et al., 2024). Third, text-only similarity is insufficient for reusable web skills: SkillMigrator’s text-only retrieval causes a large success drop on both WebArena and Mind2Web, confirming that layout signal is crucial for cross-domain transfer (He et al., 16 Jun 2026).

The dominant failure modes are also increasingly well characterized. SkillFlow identifies incorrect skill entrenchment, fragmented skill libraries, a creation–repair gap, cognitive overload from proliferating skills, and verification mismatches as recurrent causes of negative transfer (Zhang et al., 19 Apr 2026). In control domains, ablations are consistent with the same pattern: removing SADiff’s skill token or retrieval transformation degrades performance, removing ICPAD’s cross-domain consistency losses or dynamic prompting lowers adaptation effectiveness, and removing DuSkill’s hierarchical encoder or guided decoder reduces robustness (Huang et al., 16 Jan 2026, Yoo et al., 4 Sep 2025, Kim et al., 2024). This suggests that cross-domain skill diffusion is governed not only by generative expressivity, but by three coupled requirements: a compact invariant representation, a domain-grounding operator, and a repair mechanism when transfer assumptions fail.