Hierarchical Kickstarting in Reinforcement Learning

• Hierarchical Kickstarting is a reinforcement learning strategy that fuses pretrained expert skills into a shallow hierarchy to guide agent learning.
• It uses a learned, state-dependent mixture of teacher policies to form distillation targets via KL divergence and entropy regularization.
• Empirical evaluations on SkillHack show HKS outperforms traditional methods, converging faster and achieving higher success rates in sparse-reward settings.

Hierarchical Kickstarting (HKS) is a policy transfer methodology in reinforcement learning (RL) that leverages a predefined library of skill policies (teacher policies), integrating them into a shallow hierarchical structure to accelerate and guide learning on challenging tasks with large state-action spaces and sparse rewards. HKS is designed to combine the sample efficiency and inductive bias of hierarchical RL with the flexible guidance of policy distillation, enabling agents to efficiently transfer structured procedural knowledge and outperform flat RL and classical transfer techniques on tasks requiring compositional skill execution (Matthews et al., 2022).

1. Formal Structure and Core Principles

HKS assumes access to $K$ pretrained expert policies $\{\pi_1, \pi_2, \dots, \pi_K\}$, referred to as "skills" or "teachers", each optimized to near-optimal performance on a specific, narrowly defined environment (e.g., "PickUp", "NavigateWater"). The target task is modeled as a Markov Decision Process (MDP) $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$, where the agent must maximize the expected discounted return. The central innovation of HKS is to introduce a per-state, learned mixture over the teacher policies via a policy-over-teachers controller: $\pi_H(k \mid s),\ \ k = 1, \ldots, K$ which assigns a distribution over teachers for each state $s$. At each decision point, HKS forms a convex combination of teachers' action distributions, constructing a "distillation target": $$\bar{\pi}_T(a \mid s) = \sum_{k=1}^K \pi_H(k \mid s)\, \pi_k(a \mid s)$$ The student policy $\pi_\theta(a \mid s)$ is regularized towards matching this target via KL divergence or cross-entropy in addition to optimizing its own return, while $\pi_H$ is jointly adapted to select mixtures that best explain and guide the student's evolving policy. This design creates a self-consistent, soft options-like hierarchy without long commitment periods.

2. Mathematical Objective

The HKS objective augments the standard RL loss with state-conditional distillation and entropy regularization: $$\mathcal{L}_{\text{total}}(\theta, \phi) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ - \sum_{t=0}^T \gamma^t r_t\ +\ \lambda \sum_{t=0}^T \mathrm{KL}\left(\bar{\pi}_T(\cdot \mid s_t) \,\|\, \pi_\theta(\cdot \mid s_t)\right) -\ \kappa \sum_{t=0}^T \mathcal{H}\left(\pi_H(\cdot \mid s_t)\right) \right]$$ where:

• $L_{\text{RL}}$ is the standard discounted return loss (e.g., V-Trace for IMPALA);
• $\{\pi_1, \pi_2, \dots, \pi_K\}$0 controls the distillation strength;
• $\{\pi_1, \pi_2, \dots, \pi_K\}$1 encourages skill mixture diversity, weighted by $\{\pi_1, \pi_2, \dots, \pi_K\}$2.

Many experiments use the cross-entropy version of the distillation loss, which is equivalent up to an additive constant dependent only on the teacher policies.

3. Training Algorithm and Workflow

HKS operates in a multi-worker actor-critic regime (e.g., IMPALA), with one iteration consisting of the following steps:

1. Roll out student policy $\{\pi_1, \pi_2, \dots, \pi_K\}$3 in parallel across multiple actors; collect trajectories $\{\pi_1, \pi_2, \dots, \pi_K\}$4.
2. For each step $\{\pi_1, \pi_2, \dots, \pi_K\}$5 in trajectory $\{\pi_1, \pi_2, \dots, \pi_K\}$6:
   • Evaluate each teacher $\{\pi_1, \pi_2, \dots, \pi_K\}$7.
   • Compute the mixture $\{\pi_1, \pi_2, \dots, \pi_K\}$8.
3. Compute actor-critic loss $\{\pi_1, \pi_2, \dots, \pi_K\}$9 for the student.
4. Compute distillation loss $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$0.
5. Compute entropy regularizer $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$1.
6. Update $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$2 to minimize $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$3, and $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$4 to minimize $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$5.
7. Synchronize updated parameters with actors.

Teacher policies are frozen; the student and policy-over-teachers are updated without propagating gradients to the skill networks.

4. Network Architectures and Skill Modules

The HKS network shares a unified, multi-modal observation encoder across all policies:

• Five-layer CNN processing NetHack's glyph arrays;
• Embeddings for textual messages and inventory slots;
• Scalar statistics;
• Outputs concatenated and passed through an LSTM (hidden size 256).

The student network branches into:

• $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$6: action head (linear + softmax, $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$732 actions);
• $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$8: value head;
• $$M_T = \langle \mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma \rangle$$9: policy-over-teachers head (linear + softmax over $\pi_H(k \mid s),\ \ k = 1, \ldots, K$0).

Teachers use the same base encoder but are feed-forward (LSTM removed for stability). At each timestep, $\pi_H(k \mid s),\ \ k = 1, \ldots, K$1 produces a distribution across skills, which is used to compute the distillation target $\pi_H(k \mid s),\ \ k = 1, \ldots, K$2.

5. Empirical Evaluation on SkillHack

The SkillHack benchmark comprises eight procedurally generated MiniHack tasks built from compositions of 16 basic skill environments (e.g., "TakeOff", "PickUp", "ZapWandOfDeath"). Skills are pretrained (≥90% success) using IMPALA in isolation over $\pi_H(k \mid s),\ \ k = 1, \ldots, K$3 steps. In each target task, agents must sequence and compose these skills to succeed.

The evaluation contrasts HKS with vanilla RL (IMPALA), Random Network Distillation (exploration), an Options framework (policy over one-step options), and classical Kickstarting with fixed $\pi_H(k \mid s),\ \ k = 1, \ldots, K$4. Table: summary of average success rates across all tasks:

Method	Avg. Success Rate (%)
IMPALA	~0
RND	~0
Options	~32
Kickstarting	~41
HKS	~56

HKS converges at around $\pi_H(k \mid s),\ \ k = 1, \ldots, K$5 timesteps and consistently outperforms all baselines. Individual task breakdowns confirm HKS' dominance or parity against all alternatives, apart from minor exceptions in rare scenarios (e.g., "Target Practice").

Ablation experiments indicate HKS is robust to missing skills, typically substituting the closest relevant teacher. However, performance degrades if the skill set is diluted with many irrelevant teachers. Analysis of the $\pi_H(k \mid s),\ \ k = 1, \ldots, K$6 output during learning confirms that HKS learns sensible, temporally coherent skill weightings aligned with task demands.

Key hyperparameters:

• $\pi_H(k \mid s),\ \ k = 1, \ldots, K$7: linearly decayed 10$\pi_H(k \mid s),\ \ k = 1, \ldots, K$81 over $\pi_H(k \mid s),\ \ k = 1, \ldots, K$9 steps (not to zero).
• $s$0: decayed 20$s$10 over $s$2 steps.
• IMPALA-specific: 256 actors, batch size 32, rollout length 80, $s$3.

6. Contributions and Limitations

HKS advances the state of policy transfer in RL by:

• Introducing a generalization of Kickstarting via learned, per-state mixtures over teacher policies ("Options-style soft hierarchy") for state-dependent, compositional imitation.
• Demonstrating significant empirical gains on SkillHack, a new benchmark for procedural skill transfer in sparse-reward settings.
• Outperforming a comprehensive set of competitive baselines, highlighting the practicality of combining pretrained skill modules with online adaptation.

Principal limitations include:

• Necessity for predefined, high-quality teacher skills, which can require substantial domain knowledge and computational investment.
• Sensitivity to domain shift between the skill-pretraining and target environments.
• Need for careful tuning of the distillation schedule ($s$4) to prevent collapse or premature domination by rewards vs. teachers.

Failure cases arise when skill environments diverge in dynamics or interface from the target, or if the agent fails to collect reward early, resulting in cycles over non-useful teacher mixtures.

7. Broader Context and Future Directions

HKS is situated at the intersection of policy distillation, soft options, and transfer learning. In contrast to rigidly hierarchical or multi-timescale RL approaches, HKS employs a shallow, learnable mixture, enabling state-specific, compositional skill transfer without strict temporal abstraction.

Recent work on flattening hierarchies in goal-conditioned RL (e.g., policy bootstrapping in (Zhou et al., 20 May 2025)) highlights alternative approaches for capturing hierarchical credit assignment or compositional behavior without explicit modularity. In these frameworks, bootstrapping and advantage-weighted KL-divergence can unify options, distillation, and flat policy extraction, suggesting that hierarchical skill distillation, as used in HKS, represents a design point on a broader continuum of policy transfer and credit assignment mechanisms.

Key directions for future research include:

• Automated discovery (rather than predesign) of teacher skills.
• Pruning and curriculum-based adaptation of skill libraries.
• Improved off-policy correction and handling domain mismatch.
• Integration with unsupervised skill acquisition frameworks.

In sum, Hierarchical Kickstarting operationalizes a practical strategy for the injection of structured prior knowledge via compositional skill distillation, significantly advancing agent capabilities on sparse-reward, compositional RL tasks (Matthews et al., 2022).