---
title: Bottom-Up Skill Discovery
url: https://www.emergentmind.com/topics/bottom-up-skill-discovery
type: topic
---

# Bottom-Up Skill Discovery

Bottom-up skill discovery is a paradigm within unsupervised reinforcement learning (RL) and imitation learning in which an agent autonomously acquires a repertoire of diverse, reusable behaviors (“skills”) by acting in its environment with minimal or no task-specific guidance. Unlike top-down approaches, which require humans to specify task decomposition or skill libraries, bottom-up methodologies enable skills to emerge incrementally or hierarchically through exploration, guided objective optimization, or structure-aware discovery mechanisms. This article presents the core methodologies, mathematical formalisms, representative algorithms, empirical findings, and current research trajectories associated with bottom-up unsupervised skill discovery.

## 1. Mathematical Foundations and Objectives

The standard mathematical formalism for bottom-up skill discovery is built on maximizing the mutual information (MI) between a skill variable $Z$ and some measurement of the agent's behavior—states $S$, state transitions, or successor representations. The canonical objective is
$$
I(Z;S) = H(Z) - H(Z|S) = H(S) - H(S|Z)
$$
which rewards acquiring a set of $k$ policies $\{\pi(\cdot|s,z):\,z=1,\dots,k\}$ whose induced state distributions $\rho_\pi(s|z)$ are easily distinguishable and collectively cover a broad region of the environment [2002.03647, 2107.08398, 2202.00914].

However, MI-based objectives often yield “static” or trivial skills due to invariance properties, leading to the development of alternative formulations:

- **Distance- or Direction-maximizing objectives**: Encourage dynamic, far-reaching skills by maximizing expected movement in a learnable, often Lipschitz-constrained representation—e.g., $J_\mathrm{LSD}(\pi,f) = \mathbb{E}_{z,\tau}[(f(s_T)-f(s_0))^\top z]$ with a Lipschitz constraint on $f$ [2202.00914, 2602.01619].
- **State-density deviation**: Encourage each skill $z$ to explore states not visited by other skills, e.g., maximizing the deviation of $d^\pi_{z}(s)$ from $\sum_{z' \neq z} d^\pi_{z'}(s)$; implemented via variational autoencoders or contrastive estimators [2506.14420].
- **Empowerment and state-marginal covering**: Direct computation or ELBO-based optimization of state-covering policies via latent-variable generative models pre-fit to coverage distributions $p(s)$, decoupling skill assignment from policy reachability [2002.03647].
- **Regret-based curriculum**: Adaptive, adversarial games between policy and skill generator, driving discovery along directions in skill space where the agent's value function is still improving; this ensures continual expansion and avoids redundancy—$\min_{\theta_1}\max_{\theta_2}\mathbb{E}_{z\sim P^k_z}[V_{\pi_{\theta_1}^k}(s_0|z) - V_{\pi_{\theta_1}^{k-1}}(s_0|z)]$ [2506.21044].

## 2. Structured State Factorization and Compositionality

In environments with natural factorization (multibody robotics, multi-object scenes), representing the state as $S = S_1 \times \dots \times S_N$ enables more fine-grained and efficient skill discovery [2602.01619, 2410.18416]. Each latent skill vector $z = (z^1, \dots, z^N)$ can be aligned to specific factors ("editor’s term"), and exploration can be adaptively weighted per factor by observing the novelty or coverage of each component.

- **Structured Unsupervised Skill Discovery (SUSD)** implements factor-specific embedding functions $\phi_i:S_i \rightarrow \mathbb{R}^D$, defines factorized intrinsic rewards, and adaptively focuses on under-covered factors via a learned density model $q_e(s'|s)$ [2602.01619]. This ensures disentangled skills capable of controlling specific entities and facilitates downstream hierarchical RL.
- **Skill Discovery from Local Dependencies (SkiLD)** targets not just independent factor coverage but also the interaction graph (e.g., which entities affect each other during transitions). Skills are indexed by dependency matrices $g \in \{0,1\}^{N \times (N+1)}$, capturing minimal causally-inducible subgraphs. Coverage of these graphs is directly linked to downstream manipulation success [2410.18416].

Compared to MI-only approaches, structured factorization yields a combinatorial reduction in the effective search space and unlocks skills that are semantically richer and more directly applicable in complex, compositional environments. Empirical results demonstrate SUSD’s $2-3\times$ improvement in worst-factor coverage and dramatically faster sample complexity on compositional HRL benchmarks [2602.01619]; SkiLD outperforms state-coverage methods on long-horizon multi-object tasks, being the only method with nonzero success on the hardest benchmarks [2410.18416].

## 3. Algorithmic Pipelines and Hierarchical Organization

A majority of bottom-up skill discovery frameworks adopt either one of:

- **Simultaneous skill learning:** All skills are discovered in parallel (e.g., fixed-size latents), usually with MI or state-density differentiation objectives [2202.00914, 2107.08398, 2506.14420].
- **Incremental/Sequential:** Skills are acquired one after another, freezing previously mastered policies to prevent catastrophic forgetting—each new skill is driven to cover previously unmodeled or altered environment regions [2203.11176].
- **Hierarchical or tree-structured policies:** Skill libraries are grown by curriculums such as tree policies (ELSIM), hierarchical selection over a topological cluster graph (DisTop), or multi-layered controllers that select skill/subgoal/atomic action [2006.12903, 2110.10809, 2505.17673].

A representative, algorithm-agnostic pipeline for skill discovery and hierarchical reuse is:

| Stage                  | Key Operations / Methods                                              |
|------------------------|----------------------------------------------------------------------|
| Skill Initialization   | Random, VQ-VAE, iterative growth, or LLM-based task synthesis        |
| Skill Optimization     | MI/distance/density-deviation objectives, dual updates over embedding/skill |
| Coverage Monitoring    | State-factor per-skill, topological clusters, or interaction graphs   |
| Downstream Composition | High-level policy (SAC, RL, LLM selection, clustering) over skill library |
| Continual/Evolving     | Skill library is grown (sequential/incremental) or refined adaptively|

Hierarchical execution often involves a high-level controller selecting a skill or subgoal every $L$ steps, passing control to the low-level policy $\pi_\theta(a|s,z)$, which is typically frozen after discovery [2602.01619, 2006.12903, 2110.10809].

## 4. Bottom-Up Skill Discovery Beyond RL: Language and Imitation

Recent work extends bottom-up paradigms to LLM-driven and demonstration-based settings:

- **LLM-Driven Autonomous Skill Bootstrapping**: An LLM proposes atomic tasks, codes success/reward functions, and orchestrates RL policy training; successful policies are verified by VLMs and aggregated without any prior skill library [2405.15019]. This yields a growing, reliable skill set starting from zero primitives, supporting further skill proposals and compositional, top-down planning as needed.
- **Unsegmented Demonstration Parsing**: Hierarchical task structure is recovered from unsegmented demonstrations via clustering, skills are identified from recurring patterns, and meta-controllers compose these skills for long-horizon tasks. Skills extracted from multi-task demos boost average task success by 8% relative to per-task extraction [2109.13841 (*abstract only*)].

This line of work supports emergent libraries with semantically meaningful, verifiable skills applicable in open-ended, human-in-the-loop or sim2real scenarios.

## 5. Empirical Evaluation and Comparative Performance

Skill discovery methods are evaluated on criteria including:

- **State coverage**: Fraction or count of state bins visited by at least one skill; coverage of each factor or interaction-graph component [2602.01619, 2410.18416].
- **Skill diversity**: Pairwise endpoint diversity/consistency (Hausdorff, variance metrics) [2203.11176, 2202.00914].
- **Downstream task acceleration**: Speed and success rate of hierarchical RL or finetuning on tasks requiring composition of discovered skills [2002.03647, 2110.10809, 2410.04855, 2602.01619].
- **Zero-shot goal-reaching**: Ability to reach arbitrary, possibly unseen goals using latent-conditioned policies and learned embedding [2202.00914].
- **Continual adaptation and retention**: Performance in non-stationary/evolving environments, measured by skill coverage, non-forgetting, and transfer to new contexts [2203.11176, 2506.21044].

Empirical findings indicate that bottom-up, structure-aware and curriculum-driven methods achieve state-of-the-art performance in both coverage and sample efficiency compared to mutual-information or count-based baselines. Regret-aware skill generators, for example, improve zero-shot task success by up to 15% in high-dimensional settings [2506.21044], while language-driven bootstrapping achieves robust, semantically meaningful behavior with full vision-language verification protocols [2405.15019].

## 6. Challenges, Limitations, and Open Directions

While bottom-up skill discovery has advanced substantially, key open research problems include:

- **Representation constraints**: Methods with state-metric or feature requirements (e.g., Lipschitz conditioning, state factorization) may suffer in high-dimensional observation spaces without structured priors [2202.00914, 2602.01619].
- **Skill redundancy and scalability**: Uniform sampling or excessive overlap among skill policies, especially in non-factorized or large latent spaces. Curriculum or regret-aware selection mitigates but does not eliminate this [2506.21044].
- **Skill functionalization and parameterization**: Most skill libraries consist of flat, fixed policies; automatic task/parameter slot inference for reusable, generalized skill functions remains a major unsolved challenge [2505.17673, 2405.15019].
- **Continual adaptation**: Parameter footprint grows linearly with the number of independent skill modules. Selective parameter sharing, modularization, or memory-efficient extensions are needed for lifelong deployment [2203.11176, 2006.12903].
- **Evaluation and benchmarking**: Open-ended domains (e.g., games, real robotics) lack reproducible, standardized reset protocols, complicating benchmarking and skill success evaluation [2505.17673, 2405.15019].
- **Sim2real transfer**: While spectral decomposition methods [2404.05051] and robust verification address some gaps, systematic sim2real skill transfer and adaptation continue to be active directions.

## 7. Directions for Future Research

Advances in bottom-up skill discovery are converging with large-scale unsupervised representation learning, language-driven robotics, open-ended game agents, and lifelong RL. Promising directions include:

- **Tightly coupled factorization and compositional architectures**, enabling efficient manipulation and interaction-centric behaviors in rich domains [2602.01619, 2410.18416].
- **Regret- or curriculum-guided self-pacing**, with learnable skill generators and adaptive skill mixture policies [2506.21044].
- **Language-grounded, autonomous skill construction** leveraging LLM generation of tasks, success/reward functionals, and compositional skill-chaining for open vocabulary primitives with RL and imitation [2405.15019].
- **Enhanced downstream composition, reuse, and abstraction**, possibly through multiplicative or modular compositional policies [2410.04855].
- **Robust sim2real pipelines**, including spectral skill basis expansions and orthogonalization to handle dynamic mismatches [2404.05051].
- **Hybrid implicit–explicit objectives**, combining MI/density maximizing with directed, graph- or task-conditioned constraints.
- **Open-ended agent ecosystems** with asynchronous skill sharing, consensus, and decentralized conflict resolution [2505.17673].

The rapid evolution of bottom-up skill discovery continues to redefine the practical and theoretical limits of autonomous lifelong learning, with deep implications for embodied AI, robotics, and open-world agent design.

Source: https://www.emergentmind.com/topics/bottom-up-skill-discovery