Action Discovery in AI & Robotics

• Action discovery is the automatic inference of action representations from data that replaces fixed vocabularies across domains like robotics, reinforcement learning, and video analysis.
• It employs diverse methodologies such as latent discretization, chunking, and clustering to discover primitives, macro-actions, and causal subsets that support downstream tasks.
• Active exploration, uncertainty estimation, and partial supervision drive the discovery process to enhance prediction, planning, and system transfer across varied applications.

Action discovery is the problem of inferring useful action representations, action sets, or action structure from data rather than assuming a fixed action vocabulary. In contemporary research, the discovered entity varies by domain: it may be a symbolic manipulation primitive in robotics, a high-level action abstraction or macro-action in reinforcement learning, a subset of causal action dimensions in continuous control, an unknown action class in partially annotated video, an action concept grounded in image–sentence corpora, a stochastic action-order graph in process mining, or a STRIPS action schema in classical planning (Kilic et al., 23 May 2025, Boussif et al., 2024, Sun et al., 2022, Spurio et al., 7 Aug 2025, Gao et al., 2016, Alkhammash et al., 2023, Suárez-Hernández et al., 2020). Across these settings, the common objective is to replace hand-specified action vocabularies with discovered structures that improve prediction, exploration, planning, interpretability, or transfer.

1. Scope and problem formulations

Action discovery is not a single task but a family of problems defined by what counts as an “action” and what information is available during discovery. In robotic manipulation, the target is often an executable primitive grounded in continuous control parameters. One recent formulation learns an encoder–decoder effect predictor over object features $o \in \mathbb{R}^4$, continuous actions $a \in \mathbb{R}^{12}$, and effects $e \in \mathbb{R}^3$, then discretizes the action latent space into symbolic action primitives for planning (Kilic et al., 23 May 2025). In amortized sampling and long-horizon RL, the target can instead be a high-level action obtained by chunking repeated subsequences in sampled trajectories and adding them back to the action space (Boussif et al., 2024). In Meta-RL, a macro-action may be a latent variable $z$ that summarizes a multi-step transition from $s_t$ to $s_{t+M}$ (Cho et al., 2024).

In other settings, the action is not a temporally extended control primitive but a structural variable. State-Wise Action Refined learning treats action discovery as a state-dependent selection problem over action coordinates, using a binary mask $G(a \mid s)\in\{0,1\}^L$ to identify task-relevant dimensions and suppress nuisance dimensions (Sun et al., 2022). In agentic LLMs with very large tool sets, action discovery is formulated as recovery of a small support set $S^\star\subseteq\mathcal{A}$ of relevant actions under a block-sparse contextual linear reward model (Majumdar, 13 Jan 2026).

Observation-driven formulations are equally diverse. Unsupervised semantic action discovery from instructional videos seeks recurring latent activity steps shared across videos, with a Beta Process Hidden Markov Model deciding which steps are present, where they occur, and how they transition (Sener et al., 2016). Video novel category discovery addresses partially labeled action spaces in which known actions are labeled and unknown actions must be clustered as disjoint novel classes (Hasan et al., 2023). Temporal Action Segmentation has also adopted an “Action Discovery” setting in which unknown actions are labeled as background during training and must later be segmented and assigned semantic labels (Spurio et al., 7 Aug 2025).

Planning and process modeling introduce still different formulations. Process discovery from event logs infers observable actions and directly-follows relations, represented as nodes and arcs in a Stochastic Directed Action Graph with stochastic-language semantics (Alkhammash et al., 2023). STRIPS action discovery infers action signatures, preconditions, and add/delete effects from traces containing only initial and goal states, with no observed actions or intermediate states (Suárez-Hernández et al., 2020).

Setting	What is discovered	Representative approach
Robotics and manipulation	Symbolic primitives, affordances	Latent binarization, uncertainty-guided exploration
RL and Meta-RL	Action abstractions, macro-actions, causal action dimensions	Chunking, VAE-style latent actions, state-wise masks
Vision and video	Semantic steps, unknown action classes, action concepts	BP-HMM, clustering, partial-label discovery
Planning and process modeling	Action schemas, action-order graphs	Compilation to planning, grammatical inference
Large-action agents	Relevant action subset	Block-sparse recovery

This diversity suggests that action discovery is best understood as a representational problem coupled to an execution or inference problem. The discovered object must not only partition data; it must support downstream use.

2. Representation learning and abstraction mechanisms

A central design choice is the level at which actions are abstracted. In manipulation, one approach learns separate object and action encoders, concatenates their latent embeddings, and predicts a Gaussian effect model

$$p(e \mid a,o)=\mathcal{N}(e;\mu(a,o),\sigma^2(a,o)).$$

Action symbols emerge by binarizing the action encoder output,

$z_a=b(\phi_a(a))\in\{0,1\}^k,$

so continuous motor parameters are converted into discrete bit vectors. With $a \in \mathbb{R}^{12}$0, the learned action space admits up to $a \in \mathbb{R}^{12}$1 symbols (Kilic et al., 23 May 2025). This mechanism treats action discovery as latent discretization constrained by effect prediction.

A different abstraction mechanism appears in ActionPiece, where actions are discovered online from trajectories by Byte Pair Encoding over action sequences. Frequent adjacent primitive pairs are merged into composite tokens, and the policy’s action space is augmented with these chunks. The policy handles the changing action set through action embeddings rather than a fixed output layer, with logits computed by similarity between a query vector and learned action embeddings; the action encoder itself is an LSTM over the primitive-action sequence comprising a chunk (Boussif et al., 2024). Here, discovery is explicitly compositional: abstractions are induced from repeated subsequences rather than latent binarization.

HiMeta uses a modified VAE-style latent-variable model to discover a macro-action $a \in \mathbb{R}^{12}$2 from the current state $a \in \mathbb{R}^{12}$3, task representation $a \in \mathbb{R}^{12}$4, and a future goal state. The encoder samples

$a \in \mathbb{R}^{12}$5

and the decoder predicts a future ego-state $a \in \mathbb{R}^{12}$6. Because the decoder is trained only on $a \in \mathbb{R}^{12}$7, not the full state $a \in \mathbb{R}^{12}$8, the discovered macro-actions are intended to be task-agnostic and reusable across tasks (Cho et al., 2024). This suggests a distinct criterion for action abstraction: invariance to task-specific state components.

In continuous control, abstraction can take the form of dimension selection rather than temporal extension. SWAR defines a refined action

$a \in \mathbb{R}^{12}$9

and learns the selector $e \in \mathbb{R}^3$0 to preserve TD or dynamics prediction while minimizing the number of selected coordinates. The discovered “action” is therefore a state-wise causal subset of the full action vector, not a new primitive in the conventional sense (Sun et al., 2022).

Observation-driven work also relies on latent structures. Instructional video discovery represents each frame as a binary multimodal vector over language atoms and visual atoms, then uses a Beta Process Hidden Markov Model with latent step indicators $e \in \mathbb{R}^3$1, state assignments $e \in \mathbb{R}^3$2, and Bernoulli emissions to discover shared activity steps without fixed cardinality (Sener et al., 2016). In still images, ACD extracts verb–object pairs, filters them by visualness with a linear SVM on VGG-16 features, and clusters the survivors using a multimodal embedding that concatenates visual and linguistic representations,

$e \in \mathbb{R}^3$3

with cosine similarity for cluster formation (Gao et al., 2016).

These methods differ in supervision, modality, and inductive bias, but all impose a structural bottleneck between raw observations and the final action representation. That bottleneck may be binary, sequential, variational, or clustering-based; in each case it is the locus of discovery.

3. Discovery through interaction, uncertainty, and exploration

A substantial branch of the literature treats action discovery as an active process in which the system must choose interactions that make useful actions observable. In symbolic manipulation, curiosity is defined directly from predictive uncertainty. For each sampled candidate action, the system evaluates the entropy of the predicted Gaussian effect distribution,

$e \in \mathbb{R}^3$4

and executes the action with maximal mean entropy. On a UR10 robot in PyBullet simulation, exploration runs for 10,000 steps; 2,000 candidate actions are sampled at each step; retraining occurs every 512 steps for 10 epochs; and the learned primitives support 82% single-object planning success and 59% double-object planning success, compared with 56% and 38% for active learning and 13% and 9% for random exploration (Kilic et al., 23 May 2025). The reported result is not merely better prediction error; it is discovery of a richer set of executable primitives.

Information-Driven Affordance Discovery uses a closely related but distinct quantity: expected information gain over model parameters. With an ensemble of affordance predictors, the information gain of a context–action pair is

$e \in \mathbb{R}^3$5

which is the Jensen–Shannon divergence of the ensemble predictive distributions. Actions are chosen by a UCB-style rule

$e \in \mathbb{R}^3$6

The method discovers visual affordances for grasping, stacking, and opening, improves data efficiency in ManiSkill2 simulation, and reaches up to 90% grasp success on a UFACTORY XArm 6 real-world setup (Mazzaglia et al., 2024).

Interactive perception introduces another interpretation of discovery: selecting actions that reveal latent structure. Structure from Action predicts informative 3D hold–push interactions from the current RGB point cloud and a history volume $e \in \mathbb{R}^3$7, then uses the resulting motion to discover object parts and joints. The interaction policy prefers holds on no-flow regions and pushes on points with large predicted flow, so action choice is explicitly tied to part isolation and articulation inference (Nie et al., 2022). The discovered object structure depends on action selection; action discovery and structure discovery are therefore coupled.

Sparse action discovery in agentic LLMs replaces uncertainty-guided exploration with statistical recovery guarantees. Under a contextual linear reward model $e \in \mathbb{R}^3$8 and row sparsity $e \in \mathbb{R}^3$9 for all $z$0, Contextual Block-OMP iteratively selects the action block most correlated with the residual and refits least squares. Exact support recovery holds with high probability when $z$1, under coverage, block incoherence, and minimum signal-strength conditions (Majumdar, 13 Jan 2026). This formulation is theoretical rather than interactive in the robotic sense, but it shares the same premise: only a small subset of actions is informative.

A common misconception is that action discovery is equivalent to passive clustering of behavioral data. The literature includes a large class of methods in which discovery is inseparable from targeted intervention, uncertainty estimation, or support recovery.

4. Discovery from observation, partial supervision, and traces

Observation-driven action discovery addresses settings in which the system cannot intervene directly, or where the relevant actions are only partially labeled. In instructional videos, the objective is to recover recurring semantic steps from large video collections without step labels, object labels, or temporal boundaries. The multimodal BP-HMM of “Unsupervised Semantic Action Discovery from Video Collections” discovers a latent inventory of activity steps and their temporal alignments, and its full model reports $z$2, $z$3, and $z$4, outperforming HMM and kernel temporal segmentation baselines (Sener et al., 2016). The discovered storyline is a sequence of objective steps rather than low-level motion primitives.

Partially labeled video recognition introduces a different challenge: the unlabeled set contains unknown-but-disjoint action classes. NEV-NCD addresses this with supervised cross-entropy on labeled classes, negative learning on unlabeled clips,

$z$5

entropy minimization, and variance regularization to enforce equipartition among novel categories. On UCF101, NEV-NCD reports 92.4% labeled top-1 accuracy and 82.7% clustering accuracy on unlabeled data (Hasan et al., 2023). The method does not discover temporal segments; it discovers novel action categories in an open-world video recognition setting.

Temporal Action Segmentation extends partial labeling to frame-wise discovery. In the Action Discovery setting, only known actions are annotated during training; unknown actions are labeled as UNK or background. The proposed pipeline first trains an MS-TCN backbone on known classes plus UNK, then applies the Granularity-Guided Segmentation Module to refine unknown intervals by matching the granularity of known annotations, and finally applies Unknown Action Segment Assignment to cluster segment means and estimate the number of unknown classes by BIC. On DesktopAssembly, the combined GGSM+UASA method reports unknown-action MoF 51.3, Edit 52.8, and F1@10/25/50 of 62.5/58.0/45.4, versus 39.9, 17.4, and 25.6/25.6/14.2 for the baseline (Spurio et al., 7 Aug 2025).

Weakly supervised action discovery from image–sentence corpora operates at the concept level rather than the temporal level. ACD extracts candidate verb–object pairs, removes terms with frequency below $z$6, verifies visualness by cross-validated SVM AP, retains 171 action concepts from 327 candidates, and clusters them into about 84 multimodal concept clusters. On PASCAL VOC 2012 action classification, ACD-A reaches 54.3 mAP, compared with 48.3 for ACD and 44.2 for a keyword-based weak baseline (Gao et al., 2016).

Process mining offers a trace-centric analogue. Starting from a finite multiset of positive traces, stochastic directly-follows process discovery infers a Stochastic Directed Action Graph

$z$7

where nodes are labeled by observable actions, arcs encode execution order, and outgoing probabilities are normalized. The model has stochastic-language semantics and can represent trace likelihoods as sums of execution probabilities (Alkhammash et al., 2023). Here, action discovery means recovering action-order structure and frequency-aware semantics from logs rather than labeling perceptual data.

These observation-driven formulations show that action discovery can target classes, segments, concepts, or trace languages. The shared element is latent structure induced from incomplete or weak supervision.

5. Planning, execution, and downstream use

Discovered actions are valuable only to the extent that they support downstream reasoning or execution. In symbolic manipulation, the action symbols learned from binarized latent codes are used in breadth-first search over sequences of discrete primitives. Because execution still requires continuous controls, the paper introduces parameter distillation: the action encoder is frozen and continuous action parameters are optimized by SGD so that the encoder output matches a target binary symbol (Kilic et al., 23 May 2025). This closes the loop from continuous actions to symbols and back to executable motor commands.

ActionPiece uses discovered chunks as actual actions during training. When a chunk is selected, its constituent primitive actions are executed in order, unless masked by invalidity at the current state. The framework reports improved mode discovery and density estimation for GFlowNets, including an increase from about 43.24 to 48.67 discovered modes on L14_RNA1 for GFlowNet with ActionPiece-Increment + BPE, and a graph-task ELBO gap improvement from 0.72 to 0.25 for Atomic versus ActionPiece-Increment (Boussif et al., 2024). The action discovery mechanism is therefore embedded directly in the sampler’s optimization loop.

HiMeta conditions the primitive policy on both task representation $z$8 and macro-action $z$9,

$s_t$0

and adds an intrinsic reward that encourages sign agreement between macro-action and primitive action. In MetaWorld ML10, the reported average success is $s_t$1 on train and $s_t$2 on test for HiMeta-CD, versus $s_t$3 and $s_t$4 for SD (Cho et al., 2024). The discovered macro-actions thus function as guidance variables for low-level control and cross-task adaptation.

In planning, STRIPS action discovery compiles model synthesis into a classical planning problem. The learner searches over the number of action schemas $s_t$5, maximum arity $s_t$6, and trace set size $s_t$7, edits partially defined actions by removing preconditions and adding effects, then validates the resulting model across all traces. The compilation is sound and complete, and the learned models achieve 30/30 coverage on held-out validation instances in hanoi, blocks, and visitall (Suárez-Hernández et al., 2020). The discovered action theory is useful precisely because it generalizes to unseen planning problems.

Process mining makes a similar downstream claim for frequency-aware action models. Because SDAGs are obtained from SDFAs with preserved stochastic-language semantics, they support reasoning about likely future actions, simulation, and trace frequencies, not just visualization of directly-follows relations (Alkhammash et al., 2023). Sparse action discovery for agentic LLMs adds a decision-theoretic version of the same idea: once the support $s_t$8 is recovered, the plug-in policy optimizes only over discovered relevant actions and is near-optimal on new latent states, with decision loss bounded by parameter estimation error (Majumdar, 13 Jan 2026).

A plausible implication is that action discovery is increasingly evaluated by its operational consequences rather than by intrinsic cluster quality alone. Executability, compositionality, support recovery, and planning performance have become central criteria.

6. Assumptions, limitations, and open directions

The literature is explicit that discovered actions are shaped by modeling assumptions. Symbolic manipulation work assumes known object features, a fixed 12D action parameterization, a single-target object effect signal during learning, and simulation-based evaluation; the breadth-first planner is workable on small benchmarks but may become slow as the number of objects or task complexity grows (Kilic et al., 23 May 2025). Information-driven affordance discovery assumes hand-designed primitive families, binary success labels, invalid-action masking, and short-horizon interaction (Mazzaglia et al., 2024). Structure from Action assumes that one joint is activated per interaction step and does not estimate other physical parameters such as friction (Nie et al., 2022).

In RL, action abstractions can improve exploration but introduce tradeoffs. ActionPiece improves GFlowNet mode discovery across all tested tasks, but for RL methods chunking can increase reward while decreasing diversity, suggesting over-specialized chunks and a negative feedback loop toward a small region of the state space (Boussif et al., 2024). SWAR identifies task-related action dimensions, but TD-SWAR is less stable in higher-dimensional tasks because TD targets move during training, while Dyn-SWAR models only the dynamics subset rather than the full union of reward- and dynamics-relevant actions (Sun et al., 2022). HiMeta mitigates tri-level instability with independently tailored training schemes, but its effectiveness depends on the ego-state decomposition and the chosen goal-state construction (Cho et al., 2024).

Observation-driven discovery inherits limitations from supervision and representation quality. Unsupervised instructional-video discovery assumes a collection of videos from the same broad category with recurring latent steps, and some steps that always co-occur may be merged (Sener et al., 2016). NEV-NCD assumes a known number of novel classes and benefits from large batch sizes for the variance term (Hasan et al., 2023). Temporal Action Discovery depends on the quality of learned embeddings and on hyperparameters such as $s_t$9, $s_{t+M}$0, and $s_{t+M}$1; the authors also note a tradeoff in which GGSM can reduce known-action performance while improving unknown discovery (Spurio et al., 7 Aug 2025). ACD depends on descriptive captions, verb–object extraction, and heuristic thresholds such as the AP $s_{t+M}$2 visualness criterion and the frequency cutoff $s_{t+M}$3 (Gao et al., 2016).

Symbolic and trace-based discovery methods face their own constraints. STRIPS Action Discovery assumes a noise-free, offline setting with known predicates and objects, but no action observations or intermediate states (Suárez-Hernández et al., 2020). Stochastic directly-follows discovery relies on positive examples only and uses a multi-objective genetic algorithm to tune inference parameters toward small, frequency-faithful models (Alkhammash et al., 2023). Sparse action discovery for agentic LLMs is established under incoherence, coverage, and minimum signal-strength assumptions; the theory also shows that without sparsity or sufficient coverage, sample complexity must grow at least linearly in the number of actions in simplified settings (Majumdar, 13 Jan 2026).

A common misconception is that action discovery always aims to recover a single, human-readable action vocabulary. The surveyed work instead supports multiple end states: executable primitives, reusable chunks, latent macro-actions, causal action subsets, unknown semantic classes, stochastic action graphs, and planning schemas. This suggests that future work will continue to treat “action” as a domain-relative construct, with discovery methods judged by how well the resulting construct supports the surrounding system’s objectives.