---
title: Reference-Grounded Skill Discovery (RGSD)
url: https://www.emergentmind.com/topics/reference-grounded-skill-discovery-rgsd
type: topic
---

# Reference-Grounded Skill Discovery (RGSD)

Reference-Grounded Skill Discovery (RGSD) is a two-stage method for high-dimensional, reward-free control that uses a small set of reference motions to organize the latent skill space before reinforcement learning begins. In RGSD, reference trajectories are embedded on a unit hypersphere so that each motion occupies a distinct direction; reinforcement learning then mixes imitation of those reference directions with discovery from randomly sampled directions, allowing the policy to reproduce known behaviors and discover semantically related variants. The method was introduced for a simulated SMPL humanoid with 359-D observations and 69-D actions, where it learns structured skills such as walking, running, punching, and side stepping, and also discovers related novel behaviors; in downstream control tasks, it outperforms imitation-based skill acquisition baselines [2510.06203].

## 1. Definition and scope

RGSD is formulated in a reward-free Markov decision process
$$
M := \{\mathcal{S}, \mathcal{A}, \mathcal{P}, \rho_0, \gamma\},
$$
with a latent-conditioned policy $\pi_\theta(\cdot \mid s,z)$ and a latent skill $z$ sampled once per episode. Its target regime is high-DoF control, where the combinatorics of possible motions expand rapidly while the subset of useful motions remains sparse and structured. The paper’s central premise is that semantic meaningfulness becomes essential in this regime, because unconstrained diversity objectives can spend capacity on arbitrary joint-space variation rather than recognizable behaviors [2510.06203].

Within the broader skill-discovery literature, RGSD occupies a middle position between imitation learning and unguided unsupervised reinforcement learning. Purely unsupervised methods such as DIAYN-style mutual-information maximization, Lipschitz-constrained displacement learning, controllability-aware distance maximization, and behavior-level contrastive skill discovery seek distinguishable or far-reaching behaviors without external references [2202.00914], [2302.05103], [2305.04477]. By contrast, RGSD uses reference trajectories as an explicit semantic scaffold. This makes it neither a pure behavior-cloning pipeline nor a pure diversity-maximization method.

A useful characterization is that RGSD grounds a continuous latent skill manifold with reference data, then performs reward-free exploration in that grounded space. This positioning suggests that its main novelty is not merely adding demonstrations, but constraining latent geometry so that subsequent discovery remains near a semantically meaningful motion manifold.

## 2. Motivation in the context of guided skill discovery

The immediate motivation for RGSD is the failure of standard unsupervised skill discovery to scale cleanly to high-dimensional humanoid control. The DIAYN lower bound used as a baseline is
$$
\mathcal{F}(\theta) = \mathcal{I}(s;z) + \mathcal{H}(a \mid s,z) \geq \mathbb{E}_{z \sim p(z), \, s \sim \pi(z)} \Big[- \log p(z) + \log q_\phi(z \mid s) + \mathcal{H}\big[\pi_\theta(\cdot \mid s,z)\big]\Big] \triangleq \mathcal{G}(\theta, \phi),
$$
with policy optimization under
$$
r_z = -\log p(z) + \log q_\phi(z \mid s).
$$
The critique is that distinguishability alone does not ensure semantically meaningful skill structure in high-DoF systems [2510.06203].

RGSD belongs to a broader family of externally guided skill-discovery methods, but its grounding mechanism is distinct. “Learning Task Agnostic Skills with Data-driven Guidance” learns a projection from reference occupancy data and measures DIAYN-style discriminability in that projected space rather than in raw state space [2108.01869]. “Language Guided Skill Discovery” uses prompt-conditioned LLM descriptions and a language distance to define semantic diversity [2406.06615]. “Semantically Relevant Skill Discovery” uses human semantic labels and a learned relevance reward to bias skills toward human-relevant classes [2604.24127]. “Guiding Skill Discovery with Foundation Models” re-weights METRA rewards by a foundation-model-derived state desirability score [2510.23167]. RGSD differs from all of these in that its reference signal is a small set of motion trajectories that directly define latent directions before online reinforcement learning.

That difference matters because RGSD does not merely score states or constrain occupancy; it restructures the latent space itself. A plausible implication is that reference motions in RGSD play a stronger geometric role than prompts, labels, or scalar desirability scores in adjacent methods.

## 3. Contrastive grounding on the unit hypersphere

The first stage of RGSD is contrastive pretraining over a reference dataset
$$
\mathcal{M}=\{m_i\}_{i=1}^n,
$$
where each $m_i$ is a reference trajectory of states. The encoder $q_\phi$ is an MLP with layers $[1024, 1024, 1024, 512]$ and ReLU activations, and it takes as input 5 stacked observation frames,
$$
s_t = (o_{t-4}, o_{t-3}, o_{t-2}, o_{t-1}, o_t).
$$
Pretraining uses latent dimension $k=16$, batch size 256, 3000 epochs, Adam, and learning rate $10^{-4}$ [2510.06203].

The latent space is the unit sphere
$$
\mathcal{Z}=\{v\in \mathbb{R}^k:\|v\|_2=1\}.
$$
The encoder defines a von Mises-Fisher distribution
$$
q_\phi(z \mid s) = C_d(\kappa)\,\exp\!\big(\kappa\,\mu_\phi(s)^\top z\big), \quad \|z\|_2=\|\mu_\phi(s)\|_2=1,
$$
so $\mu_\phi(s)$ is a normalized directional embedding and $\kappa$ is a fixed concentration parameter. Because the vectors are unit norm, dot product coincides with cosine similarity.

The contrastive supervision is trajectory-identity based. For a sampled motion $m$, an anchor $s^a$ and positive $s^+$ are drawn from the same motion, while negatives $s_j^-$ are drawn from different motions. Writing
$$
z^a = \mu_\phi(s^a), \quad z^+ = \mu_\phi(s^+), \quad z_j^- = \mu_\phi(s_j^-),
$$
the InfoNCE objective is
$$
\mathcal{L}_{\mathrm{InfoNCE}} = -\log \frac{\exp\!\big(\mathrm{sim}(z^a, z^+)/T\big)}
{\exp\!\big(\mathrm{sim}(z^a, z^+)/T\big) + \sum_j \exp\!\big(\mathrm{sim}(z^a, z_j^-)/T\big)},
$$
with
$$
\mathrm{sim}(z_i, z_j)=z_i^\top z_j, \qquad T = 1/\kappa.
$$
The appendix derives the equivalent NCE form directly from the vMF likelihood. It also proves that, at the optimum, states from the same motion align to the same direction, so each reference motion $m$ induces a canonical latent direction [2510.06203].

That direction is computed by averaging the frame embeddings:
$$
z_m = \frac{1}{l}\sum_{s \in m} \mu_\phi(s),
$$
where $l$ is motion length. Under perfect within-trajectory alignment,
$$
\mu_\phi(s_1)=\mu_\phi(s_2)=\cdots=\mu_\phi(s_{l_m})
$$
for all states in the same motion. The result is a latent sphere on which each reference trajectory occupies a distinct semantic direction.

## 4. Reinforcement learning with parallel imitation and discovery

After pretraining, the latent space is frozen and used as a semantic scaffold for reinforcement learning. RGSD then mixes two rollout modes:
$$
z \sim
\begin{cases}
\mu_\phi^-(m), & \text{with probability } p, \\
k / \|k\| , \quad k \sim \mathcal{N}(0, I), & \text{with probability } 1-p .
\end{cases}
$$
In experiments, $p=0.7$ [2510.06203]. With probability $p$, the policy receives a reference latent and performs imitation of a known motion. With probability $1-p$, it receives a random unit vector and performs discovery. The policy and value function are shared across both modes.

This construction defines the central balance in RGSD. Exact reference directions recover reference behaviors, while random directions populate neighboring parts of the same latent geometry. The paper motivates this as a mechanism for learning both imitation-compatible skills and semantically related diverse behaviors. A plausible interpretation is that the sphere acts as a reference-conditioned chart: exact directions recover anchors, and nearby directions support controlled variation.

The paper explicitly situates this against imitation-only acquisition. In RGSD, the reference motions are not the terminal product of training; they are the latent basis from which exploration proceeds. This is why the method is presented as discovering novel skills rather than merely reproducing a fixed motion library.

## 5. Empirical behavior and downstream use

The empirical domain is a simulated SMPL humanoid with 359-D observations and 69-D actions. In that setting, RGSD learns structured skills including walking, running, punching, and side stepping, and also discovers related novel behaviors [2510.06203]. The contrast in the paper is with DIAYN and METRA, which are reported to yield unstructured movements on the same humanoid.

The downstream result reported in the abstract is that RGSD outperforms imitation-based skill acquisition baselines on control tasks [2510.06203]. The article materials do not provide a metric table in the supplied text, so the empirical claim should be read at that level of specificity. The stated qualitative finding is that lightweight reference-guided grounding is sufficient to make high-DoF skill discovery produce semantically rich and structured behaviors.

The significance of these findings is methodological. RGSD does not require dense task rewards during pretraining, and it does not replace exploration with full imitation. Instead, it uses a small reference set to organize the latent manifold before reward-free control learning. This suggests a practical path for high-DoF systems in which reference data are available but too sparse or too narrow to justify pure imitation learning.

## 6. Relation to adjacent methods, limitations, and interpretation

RGSD is one instantiation of a larger shift from unguided skill diversity to semantically constrained discovery. Compared with prompt-grounded methods such as LGSD, its grounding object is a reference trajectory rather than an LLM-generated language metric [2406.06615]. Compared with occupancy-guided DIAYN variants, its reference is a motion manifold rather than an expert-versus-random projection [2108.01869]. Compared with SRSD, it does not rely on interactive human semantic labels or a discrete relevance taxonomy [2604.24127]. Compared with FoG, it changes the latent geometry through reference motions rather than re-weighting an intrinsic reward by a desirability score [2510.23167].

The method also differs from reference-free hierarchical decomposition systems. “Unsupervised Hierarchical Skill Discovery” segments trajectories and induces grammars without any external grounding signal [2601.23156]. RGSD instead uses references from the outset, so its latent structure is semantically biased before reinforcement learning begins. This suggests that RGSD is best understood as a reference-conditioned alternative to purely unsupervised skill segmentation or diversity maximization, rather than as a variant of hierarchical decomposition.

Several boundaries are explicit in the supplied material. RGSD depends on reference trajectories; it is therefore not a fully unguided method. Its semantic prior is only as rich as the reference set used in contrastive pretraining. The provided text also describes a two-stage pipeline with a frozen latent scaffold, which suggests that reference geometry is imposed before policy learning rather than jointly adapted with online discovery. A plausible implication is that RGSD trades some flexibility for semantic structure: it narrows the search space in exchange for better alignment with meaningful motion families.

Within the skill-discovery literature, the distinguishing idea of RGSD is therefore precise: semantically meaningful reference motions are used to define a continuous latent sphere, and reward-free skill discovery is subsequently carried out inside that grounded representation.

Source: https://www.emergentmind.com/topics/reference-grounded-skill-discovery-rgsd