---
title: Unsupervised Action Discovery
url: https://www.emergentmind.com/topics/unsupervised-acton-discovery
type: topic
---

# Unsupervised Action Discovery

Unsupervised Action Discovery

Unsupervised action discovery refers to the suite of computational frameworks for automatically detecting, segmenting, and characterizing the structure of actions—typically as atomic or compositional units—without access to any labeled supervision. This problem spans hierarchical reinforcement learning (HRL), video understanding, robotics, and symbolic planning. Across domains, the central goal is to autonomously acquire, from unstructured experiences or perceptual streams, a discrete or structured set of "actions," "skills," "options," or "interaction modes" that facilitate downstream problem solving, efficient exploration, or semantic understanding. The field synthesizes information-theoretic objectives (e.g., mutual information), intrinsic motivation, unsupervised segmentation, clustering, and sequence modeling.

## 1. Problem Formulations Across Domains

Unsupervised action discovery spans a spectrum of problem settings:

- **Hierarchical Reinforcement Learning (HRL):** Discovering temporally extended options or subgoals from exploratory experience, with the aim of enabling abstraction, compositionality, and improved exploration in sparse-reward environments. This is typically framed as mining subgoals, skills, or options from raw experience without extrinsic task reward [1911.10164, 2602.01619, 2305.04477, 2410.20258, 2410.07877, 2302.05103].
- **Video Understanding and Segmentation:** Parsing raw video collections into temporally localized atomic actions or semantic steps by unsupervised means—partitioning sequences based on visual or multimodal cues, and clustering segments into a vocabulary [2106.14733, 2508.05529, 1605.03324, 2106.03738].
- **Symbolic Model Discovery:** Inducing STRIPS/PDDL action schemas solely from unordered state transition traces, potentially with unknown action signatures, under constraints of generalization and parsimony [2001.11457].
- **Robotic Manipulation:** Inferring discrete "interaction modes" (e.g., open, close, pull, push) underlying continuous control trajectories without labeled primitives, typically via unsupervised representation learning and clustering over collected task observations [2410.20258].

Although terminology varies (options, skills, subgoals, actions, primitives, interaction modes), the shared challenge is to recover a compositional structure of behavior or task decomposition that facilitates generalization and efficient learning without reliance on ground truth partitioning.

## 2. Information-Theoretic and Intrinsic Objectives for Skill Discovery

A central methodological axis is the use of information-theoretic objectives—primarily mutual information (MI)—and intrinsic motivation to drive unsupervised acquisition of skills:

- **Mutual Information Maximization:** Many methods maximize $I(Z;S)$, the MI between skill variable $Z$ (latent code) and state visitation $S$, often with a variational posterior $q_\phi(z|s)$ providing a discriminative reward [2602.01619, 2305.04477, 2202.00161, 2310.20178, 2410.07877, 2302.05103]. This induces diverse, discriminable skills, each targeting a distinct subset of state space, but can collapse to static solutions if not regularized.
- **Distance- or Difficulty-Aware Maximization:** Distance-maximizing skill discovery (DSD) encourages policies to move the environment along learned or task-driven "hard" directions, often subject to distance constraints or using learned state factorization [2602.01619, 2410.07877, 2302.05103].
- **Curiosity and Empowerment:** Intrinsic rewards based on empowerment (maximizing agent's influence over future states) or surprise (e.g., negative log-likelihood under a learned density model, -log q(s'|s)) promote discovery of controllable and underexplored behaviors [1905.09334, 2602.01619]. Controllability-aware methods adaptively shift the discovery pressure toward harder-to-control state transitions as previously easy transitions are mastered [2302.05103].
- **Contrastive Skill Discovery:** Recent approaches use contrastive learning to maximize skill-behavior discriminability via InfoNCE objectives, as in CIC (contrastive between transitions and skills) or BeCL (contrastive between pairs of behaviors under the same skill), and use embedding entropy as an intrinsic reward (see Table 1 below) [2202.00161, 2305.04477]. This addresses the collapse-to-static/degenerate-skills problem in vanilla MI maximization.

| Method/family                  | Core discovery objective                       | Salient regularizer/bonus     |
|-------------------------------|-----------------------------------------------|------------------------------|
| MI-based (DIAYN, SMM, DADS)   | $\max I(Z; S)$                                | Discriminator: $q(z|s)$      |
| Distance-maximizing (DSD, LSD, METRA) | $\max (\phi(s')-\phi(s))^\top z$    | Constraint: $|\phi(s)-\phi(s')| \le d(s, s')$ |
| Controllability-aware (CSD)   | DSD with $d(s,s') = -\log q(s'|s)$            | Hardness-driven reward       |
| Contrastive (CIC, BeCL)       | InfoNCE over $(s, s')$ or behaviors           | Embedding entropy reward     |
| Empowerment                   | $I($option$; $trajectory$)$                   | Option decoding accuracy     |

Empirical results, e.g. SUSD [2602.01619], show that factorization and curiosity-driven exploration (guiding the agent to target underexplored subspaces of the state factors) improve factor coverage, balance the discovery between environment components (entities, objects), and enable efficient downstream hierarchical RL.

## 3. Subgoal and Option Discovery in Hierarchical RL

In HRL contexts, unsupervised action discovery involves two elements: identifying promising subgoals and constructing temporally extended options or composite policies. Rafati & Noelle [1911.10164] describe a two-level model-free HRL scheme in which:

- Exploration is driven by intrinsic motivation for subgoal attainment, with experience memories mined periodically via anomaly detection (for rare, high-reward states) and K-means clustering (for typical but representative states).
- Discovered subgoals (both anomalies and centroids) are turned into options with initiation sets, internal controller policies, and termination conditions.
- The representation network $\phi(\cdot)$ is shared between state clustering and value functions, enabling unified learning without auxiliary models.
- Agent coverage (e.g., in Four-Rooms and Montezuma's Revenge) is dramatically improved compared to standard random or non-hierarchical approaches, with subgoal-attainment rates exceeding 90%.

SUSD extends such ideas by factorizing the state and latent skill spaces, ensuring that all controllable factors are represented and that HRL controllers may later select and coordinate among multiple disentangled skills for complex tasks [2602.01619].

## 4. Unsupervised Action Segmentation in Video and Instructional Data

Video-based action discovery methods perform unsupervised temporal segmentation and clustering into action primitives. Canonical approaches include:

- **Sequential Stochastic Autoregressive Models:** [2106.14733, 2106.03738] use RNNs/grammars with Gumbel-Softmax sampling to probabilistically segment sequences. Self-labeling proceeds via stochastic rule sampling, followed by cost-based selection favoring action occurrence, length-consistency, and visual appearance agreement.
- **Generative Bayesian Nonparametrics:** [1605.03324] models per-frame multimodal features (visual atoms, language atoms) with Beta-Process HMM priors, inferring both the number and semantics of steps. Segments are aligned across a collection by feature sharing; textual descriptors emerge via language modeling.
- **Mixed-Supervision Discovery:** [2508.05529] considers "action discovery" with only partial annotations (some actions labeled, others unknown), using annotated granularity to guide temporal segmentation (GGSM) and performing cluster assignment (UASA) on unlabelled segments using GMM and K-means over learned embeddings. This yields significant gains in F1 metrics for unknown actions compared to unsupervised or oracle baselines.

State-of-the-art models match or exceed weakly supervised methods on challenging datasets (NIV, 50Salads, Breakfast), demonstrating the effectiveness of constraints combined with self-labeling, cross-embedding losses, and hierarchical clustering.

## 5. Unsupervised Symbolic Action Model Induction

In classical planning, unsupervised action discovery targets the synthesis of STRIPS/PDDL action models from goal-oriented transition traces without intermediate state or action annotations [2001.11457]. The approach is to:

- Compile the schema discovery problem into a single parallel planning task, where edit-actions successively hypotheses headers, preconditions, and effects for $k$ candidate schemas with arity $r$.
- Use anomaly detection and symbolic constraints to systematically prune and generalize, including both statically determined predicates and variable-effect arities.
- Leverage SAT-based planners to validate and select minimal-cost action models for all traces in parallel.
- Empirical results on blocksworld, hanoi, and grid domains show that fully unsupervised discovered models generalize to all validation problems, with best solutions reached in modest compute times.

A key challenge is balancing parsimony with over-generalization, especially when only goal states (and not negative examples) are available.

## 6. Unsupervised Discovery of Robotic Action Modes

Recent robotic learning work has focused on discovering discrete interaction modes (primitives) underlying continuous control data without privileged labels [2410.20258]. In ActAIM2:

- A conditional GMM-VAE selector clusters task embeddings (derived as visual differences between initial and final object states), forming discrete interaction modes.
- A behavior-cloning transformer predicts low-level action trajectories conditioned on both sensory input and the discovered discrete mode.
- The pipeline operates entirely self-supervised, relying only on visually defined thresholds to label successful interaction attempts.
- ActAIM2 achieves higher sample success rates in both seen and unseen objects/categories relative to unsupervised and weakly supervised baselines, with robust grounding in real-robot demonstrates.
- Empirical ablations demonstrate mode separability is critically dependent on informative clustering (e.g., GMM but not VQVAE can yield disentangled modes).

A general implication is that discrete, unsupervised primitives discovered in this way provide a practical basis for high-level control and composable planning in robotics, without reliance on expert-annotated data.

## 7. Future Directions and Challenges

Several open avenues and limitations persist:

- **Coverage and Expressivity:** Scalability to high-dimension environments, richer or hierarchical compositionality (e.g., curriculum learning, stacking discovered options/skills), and balancing coverage of rare and frequent factors remain central [2602.01619, 2310.20178].
- **Difficulty-Aware and Goal-Conditioned Discovery:** The integration of controllability measures, curriculum mechanisms, and goal-conditioned planning with unsupervised skill discovery is under active exploration [2302.05103, 2310.20178].
- **Robustness and Transfer:** Robust unsupervised discovery in the presence of noise, partial observability, or complex sensorimotor contingencies, and the transferability of skills to novel tasks or domains, especially in real-world hardware deployments, continue to challenge current methods [2410.20258, 2410.07877].
- **Symbolic-Subsymbolic Bridging:** Extending symbolic schema discovery to handle raw perceptual input (e.g., via representation learning linking visual features to predicates) is a major open question [2001.11457].
- **Open-World and Continual Discovery:** Future approaches are investigating continual, open-world action discovery with streaming data and the potential for continual integration of new actions as agents encounter novel domains [2508.05529].

Unsupervised action discovery remains a foundational problem at the intersection of RL, perception, video understanding, and planning, with ongoing advances in factorization, information-theoretic objectives, and cross-modal representation learning accelerating empirical progress.

Source: https://www.emergentmind.com/topics/unsupervised-acton-discovery