State-Conditioned Skill Prior

• State-conditioned skill prior is a conditional probability model over latent skill embeddings that guides high-level decisions in RL by leveraging state-dependent expert demonstrations.
• The approach employs VAE-based encoding and adaptive architectures like Gaussian, flow-based, or mixture priors to map environmental states to effective skill parameters.
• Empirical studies in robotics, autonomous driving, and transfer RL show substantial improvements in exploration speed and sample efficiency when using state-conditioned skill priors.

A state-conditioned skill prior is a conditional probability distribution over latent, temporally extended action parameters ("skills") given the current environment state. It systematically leverages state-dependent statistics of skill usage, which are learned—typically from expert demonstrations or prior experience—to guide high-level decision making in hierarchical or skill-based reinforcement learning (RL). The approach directly biases exploration and policy learning toward skills that are relevant and effective in a given state, substantially accelerating downstream RL and improving data efficiency across diverse domains including robotics, manipulation, autonomous driving, and transfer RL.

1. Formal Definition and Mathematical Framework

Let $s \in \mathcal{S}$ denote the environment state and $z \in \mathbb{R}^d$ a latent skill embedding or parameter vector specifying a temporally extended sequence of actions (skill). A state-conditioned skill prior is a density $p(z|s)$ capturing, for any state $s$, the distribution over $z$-values that are likely to yield meaningful (typically expert or successful) behaviors. The general training paradigm comprises learning both a skill embedding and the conditional prior $p(z|s)$.

A synthesis of common frameworks follows, exemplified by the ReSkill, SPiRL, and ASAP-RL pipelines (Rana et al., 2022, Pertsch et al., 2020, Wang et al., 2023):

1. Skill Embedding (VAE):
   • Encode demonstration snippets $(s_{t:t+H-1}, a_{t:t+H-1})$ into $z$ using an encoder $q_\phi(z|s,a)$.
   • Decode $z$ to actions using $z \in \mathbb{R}^d$0 or $z \in \mathbb{R}^d$1.
   • Embed via a $z \in \mathbb{R}^d$2-VAE objective:

   $z \in \mathbb{R}^d$3

2. Learning the State-Conditioned Prior:

   • Gaussian Parameterization: $z \in \mathbb{R}^d$4 (Pertsch et al., 2020).
   • Conditional Normalizing Flow: $z \in \mathbb{R}^d$5 maps $z \in \mathbb{R}^d$6 to a base density $z \in \mathbb{R}^d$7, giving $z \in \mathbb{R}^d$8 via $z \in \mathbb{R}^d$9 (Rana et al., 2022).
   • Mixture Priors: Mixture of multiple priors with adaptive, state-dependent weights $p(z|s)$0, $p(z|s)$1 (Xu et al., 2022).
   • Objective (reverse-KL):

   $p(z|s)$2

• In flow-based settings, negative log-likelihood in the transformed space is minimized:

  $p(z|s)$3

1. Total Skill Learning Loss:

$p(z|s)$4

2. Model Architectures and Implementation

The implementation of $p(z|s)$5 and associated encoders/decoders is domain-dependent but shares common structural elements:

Component	Description	Used in
Encoder ($p(z|s)$6)	LSTM (128 units) or MLP for trajectory-to-latent mapping	(Rana et al., 2022, Pertsch et al., 2020)
Decoder ($p(z|s)$7)	3-layer MLP or LSTM; maps $p(z|s)$8 or $p(z|s)$9 to actions	(Rana et al., 2022, Pertsch et al., 2020)
State-conditioned Prior ($s$0)	Real-NVP flow (4 coupling layers) or Gaussian/softmax mixture	(Rana et al., 2022, Xu et al., 2022)
Adaptive Weight Module (AWM)	6-layer MLP with softmax output for $s$1	(Xu et al., 2022)

For mixture or compositional settings, a set of task- or primitive-specific priors $s$2 are pre-trained, and an adaptive weighting module combines them per state (Xu et al., 2022). Information asymmetry and soft masking over state features is handled by attention modules or learned masks, as in APES (Salter et al., 2022).

3. Accelerated Exploration and Sample Efficiency

State-conditioned priors directly bias high-level skill sampling to relevant regions of the skill space, mitigating unstructured exploration and avoiding "dead" or unsafe zones. This leads to documented gains in exploration efficiency and sample efficiency across manipulation, navigation, and autonomous driving:

• ReSkill (Rana et al., 2022):

  • Object interaction rate in first 20k steps:
  • Gaussian atomic-action: 0.56%
  • Unconditioned skill sampling: 9.39%
  • Single-step prior: 4.72%
  • State-conditioned prior: 45.4%
• SPiRL (Pertsch et al., 2020): Baseline methods fail to reach goal or sufficiently explore in sparse reward environments; state-conditioned priors enable 10–50× faster learning and task completion.
• ASAP-RL (Autonomous Driving) (Wang et al., 2023): Tenfold gain in sample efficiency; e.g., policy converges in ~50k skill steps (vs. ~200k for vanilla SAC), with superior rates of success and safety.

4. Integration into RL Algorithms

State-conditioned skill priors are tightly coupled with hierarchical or KL-regularized RL. The high-level policy operates over $s$3 and is regularized or initialized by $s$4:

• KL-regularization:

RL objective incorporates a penalty:

$s$5

(Pertsch et al., 2020, Salter et al., 2022)

• Mixture/compositional priors:

Adaptive mixture $s$6 guides $s$7 via weighted KL-divergences (Xu et al., 2022).

• Residual policies:

Low-level corrections $s$8 are added to the decoded skill action $s$9 to preserve policy expressivity and adaptability (Rana et al., 2022).

The roll-out loop typically alternates high-level selection of $z$0 according to $z$1, low-level skill execution (possibly with residual correction), and environmental advancement.

5. Skill Priors for Transfer, Compositionality, and Adaptation

Skill priors facilitate not only faster RL but also more robust transfer and compositionality:

• ASPiRe (Xu et al., 2022):
  • Leverages multiple specialized state-conditioned priors $z$2.
  • A learned AWM assigns adaptive weights $z$3 enabling task-dependent composition.
  • Demonstrated ability to (1) neglect irrelevant primitives, (2) select single modes, or (3) construct concurrent mixtures when both are required by the environment.
• APES (Salter et al., 2022):
  • Learns information-asymmetric state-conditioned priors (masked or attention-weighted state input).
  • Optimizes the tradeoff between expressivity (richness of input conditioning) and transferability (robustness to covariate shift) via explicit regularization and information-theoretic theorems.
  • Empirically, soft-masked state-conditioned priors outperform both unconditioned and fully conditioned priors across transfer and extrapolation tasks.

6. Empirical Results and Benchmarks

Key experiments substantiate the impact of state-conditioned skill priors:

Domain	Method	Performance Improvement	Reference
Robotic manipulation	ReSkill with prior	5× increase in effective exploration, fastest learning, highest final reward	(Rana et al., 2022)
Maze/blocks/kitchen	SPiRL	10–50× faster learning; solves tasks unreachable by flat policies	(Pertsch et al., 2020)
Dense-traffic driving	ASAP-RL	Converges 4× faster, 10% higher asymptotic success, 30% fewer collisions	(Wang et al., 2023)
Transfer learning (APES)	Learned mask prior	Outperforms all fixed/prior-free baselines by wide margin	(Salter et al., 2022)
Multi-prior composition	ASPiRe	Near-perfect success in harder long-horizon and compositional tasks	(Xu et al., 2022)

Ablations consistently show that removing state conditioning, using unconditioned priors, or handicapping residual adaptability noticeably retards learning or caps final performance.

7. Theoretical Trade-offs: Expressivity, Transferability, and Information Asymmetry

The choice of how much state information is fed to the skill prior (degree of conditioning, so-called information asymmetry) is nontrivial and domain-dependent (Salter et al., 2022):

• Expressivity: Conditioning on more state variables allows matching policy and prior more closely in situ, reducing KL-divergence and enabling expressive skill assignment.
• Transferability: Greater conditioning increases sensitivity to covariate shift between source and target tasks, reducing robustness in transfer and extrapolation settings.
• APES addresses this via learned soft masks, seeking an optimal conditioning subset; proven theorems ground this expressivity–transferability tension in KL-divergence properties.

In sum, state-conditioned skill priors represent a central advance in skill-based RL, encoding state-dependent knowledge to focus exploration, improve sample efficiency, boost transferability, and enable dynamic composition across primitive or high-level behaviors. These mechanisms now underlie most scalable approaches to skill-based RL in continuous and complex domains (Rana et al., 2022, Pertsch et al., 2020, Xu et al., 2022, Wang et al., 2023, Salter et al., 2022).