Macro-action Construction

• Macro-action construction is defined as a process that builds temporally extended action sequences from primitive actions to streamline decision-making.
• It employs methods like pattern mining, genetic algorithms, VAE-based discovery, and end-to-end RL to extract and integrate macro-actions effectively.
• Utilizing macro-actions reduces the decision space, improves sample efficiency, and enhances transferability and robustness in complex, long-horizon tasks.

A macro-action is a temporally extended sequence of primitive actions that is treated as a single decision unit in planning or reinforcement learning. Macro-action construction concerns the algorithms, principles, and representations for identifying, organizing, and leveraging such sequence-level abstractions to accelerate policy learning, planning, or acting in high-dimensional or long-horizon domains.

1. Formal Notions and Structural Definitions

Macro-actions, also called "temporal abstractions" or "options" in the literature, are defined in terms of sequences of primitive actions $\mathcal{A}$, with their semantics, domain of applicability, and termination rules varying by context.

• Open-loop macro-action: An ordered sequence $m = (a_1, a_2, ..., a_\ell)$, $a_i \in \mathcal{A}$, executed without mid-sequence feedback (Chang et al., 2019).
• Option (Semi-Markov abstraction): A triple $(I, \pi, \beta)$, with initiation set $I$, policy $\pi$, and termination condition $\beta$ (Zhang et al., 2022).
• Macro as local policy: In abstract MDPs, a macro for region $R$ is a (possibly stochastic) local policy $\mu: R \to \mathcal{A}$, terminating when certain boundary (exit) states are reached (Hauskrecht et al., 2013).
• Continuous embedding: In meta-RL, a macro-action $z$ can be a continuous latent vector functioning as a summary of an action sequence connecting start and goal states within a horizon $M$ (Cho et al., 2024).

The representation may encode fixed-length n-grams (e.g., token blocks in RLHF), variable-length sequences (e.g., dynamic macro length in STRAW (Alexander et al., 2016)), or parametric plans (e.g., Bézier-parameterized trajectories in MAGIC (Lee et al., 2020)).

2. Construction Algorithms and Automated Learning

Macro-action construction admits a range of algorithmic paradigms:

Data Mining and Pattern Mining: Macro-actions are extracted as frequent or closed sequential patterns from a corpus of solved plans using algorithms like BIDE+; their usability is ranked by frequency (support), without additional relational generalization (Castellanos-Paez et al., 2016). This approach is typically offline and domain-agnostic.

Genetic Algorithms: Candidate macros are evolved using mutation (append, alteration) and selection, with episodic return as the fitness function. Only macros which yield significant empirical performance gain when added to the primitive action set are kept (Chang et al., 2019).

VAE-based Latent Macro Discovery: Factorized and disentangled sequential VAEs are trained on expert demonstration trajectories to construct a low-dimensional latent code space, each code generating a macro through decoding. This allows flexible length and compositionality, with policy outputs in the latent rather than primitive space (Kim et al., 2019, Cho et al., 2024).

End-to-end RL Joint Discovery: Some architectures (e.g., STRAW) propose a joint model that learns both to generate plans (macro proposals) and to decide when to commit or replan, thus jointly learning macro boundaries and their content via reinforcement signals only (Alexander et al., 2016). These architectures scale to high-dimensional policy domains.

Focused Macro Search: For classical planning, best-first search is conducted in the space of action sequences, explicitly favoring macros with small effect size (number of state variables changed), yielding "focused" macros that synergize with heuristic search (Allen et al., 2020).

Expert Policy Induction: Given a set of (possibly sub-optimal) expert policies, EASpace constructs macro sets by executing each policy for a range of fixed durations, producing macro-actions parameterized by the source policy and execution time (Zhang et al., 2022).

Online Plateau Escape: In planners such as Marvin, macro-actions are constructed online as the shortest plateau-escape sequence found during enforced hill-climbing; the sequence is abstracted by parameterization and used to accelerate future search within similar plateaux (Coles et al., 2011).

3. Macro-action Integration and Usage in Decision Processes

Macro-actions are utilized in various types of decision processes, including:

• Classical and Temporal Planning: Macros augment the operator set in A* or temporal planners, with sequential macro-actions constructed by right-associative composition and effect-safe transformation to preserve validity under concurrency and durative semantics (Castellanos-Paez et al., 2016, Bortoli et al., 2023).
• (Semi-)Markov Decision Processes: Macro-actions are treated as options in SMDPs, with extended Bellman equations and value backup steps using macro-transitions and macro-rewards, thus shortening the effective decision horizon (Hauskrecht et al., 2013, Zhang et al., 2022, Chang et al., 2019).
• Hierarchical RL and Meta-RL: Macro-actions enable multi-level control hierarchies, where upper levels select macro-actions (or latent embeddings thereof), to be refined into primitive actions at lower levels (Cho et al., 2024, Lee et al., 2020).
• LLM RLHF: Macro-actions are sequences of tokens (n-grams, parsed constituents, or low perplexity chunks), and credit assignment is performed at the macro level to reduce variance and improve convergence (Chai et al., 2024).

Typical integration strategies either extend the set of available actions to include macros, or, in hierarchical or abstract MDPs, restrict policies to only macro-choices at "boundary" or high-level states.

4. Theoretical and Computational Implications

Macro-action construction and deployment yields both computational and theoretical benefits:

• Search Space Reduction: By decreasing the number of decision points, macro-actions flatten the depth of search trees and reduce the size of reachable state-action space, e.g., O(|A|^H) is replaced by O(K^{H/L}) with macro-length L and K macros (Lee et al., 2020, Hauskrecht et al., 2013).
• Sample Efficiency & Credit Assignment: In RL, longer temporal abstraction via macro-actions brings rewards closer to their causes, tightening credit assignment and reducing the variance of policy gradient estimators (Chai et al., 2024, Alexander et al., 2016).
• Value Function Embedding: Embedding macro-actions changes both the evaluation and embedding effect in value back-ups—altering intermediate value propagation (Chang et al., 2019).
• Solution Quality Guarantees: Under certain conditions (local macro optimality, accurate peripheral seed functions), hierarchical decomposition produces near-optimal policies with explicit $\epsilon$-optimality bounds (Hauskrecht et al., 2013).

Empirical results confirm dramatic speed-ups (e.g., 5×–20× in MAGIC, orders of magnitude in planning domains (Lee et al., 2020, Allen et al., 2020, Bortoli et al., 2023)), without substantial loss in solution quality.

5. Macro-action Properties: Reusability, Transferability, and Robustness

Macros with minimal dependence on specific task details or trajectories yield favorable generalization properties:

• Reusability: A macro discovered with one RL algorithm or in one task is effective for policy learning with different algorithms in the same domain (Chang et al., 2019, Cho et al., 2024).
• Transferability: Macros constructed for dense-reward or base environments accelerate learning in related tasks with altered dynamics or reward sparsity (Chang et al., 2019).
• Task-invariance: Regularization or masking (e.g., egocentric state restriction in HiMeta) drives macros to encode information reusable across tasks, boosting meta-RL performance (Cho et al., 2024).
• Robustness to exploration/exploitation settings: Empirically, longer macro-actions can either accelerate exploration or introduce suboptimality, requiring careful intrinsic reward shaping (Zhang et al., 2022).
• Online versus offline construction: Patterns mined offline (frequent subsequences, VAE-based chunks) can be beneficially combined with online, plateau-triggered macro discovery for adaptable coverage (Castellanos-Paez et al., 2016, Coles et al., 2011).

Table: Macro-action Construction Paradigms

Paradigm	Representative Work	Key Feature
Pattern mining	(Castellanos-Paez et al., 2016)	Frequent subsequence extraction
Genetic algorithm	(Chang et al., 2019)	Mutation/selection, fitness via return
VAE-based latent	(Kim et al., 2019, Cho et al., 2024)	Disentangled latent factor learning
End-to-end RL	(Alexander et al., 2016, Lee et al., 2020)	Integrated planning and termination
Focused effect BFS	(Allen et al., 2020)	Best-first search for small net effect size
Expert policy-based	(Zhang et al., 2022)	Option families parameterized by duration
Online plateau escape	(Coles et al., 2011)	Plateau escape in search as macro candidates

6. Practical Guidelines and Domain-specific Strategies

Construction and deployment must be tuned to the domain's dynamics and intended use:

• Time-block selection: E.g., optimal block size $L$ in multi-time-scale DP should balance computation and model accuracy (Rahimpour et al., 2019).
• Macro candidate filtering: Employ support thresholds (pattern mining), mutation rates (GA), effect-size limits (BFS), or structure regularization (VAE) to prevent macro proliferation and the "utility problem" (Castellanos-Paez et al., 2016, Chang et al., 2019, Allen et al., 2020).
• Action applicability and parameterization: For lifted macros, ensure abstract parameter schemas enable reusability across state instances (Coles et al., 2011).
• Integration with safety/mutex mechanisms: In temporal planning, inject effect-safe mutex locks to preserve concurrency where possible (Bortoli et al., 2023).
• Intrinsic reward balancing: Proper reward shaping balances exploitation of long macro-actions with their risk of credit diffusion (Zhang et al., 2022).

Macro-action construction thus encompasses a broad spectrum of methodologies, providing a critical lever for scaling learning and planning systems to tasks with deep, high-dimensional, or structured temporal dependencies. As experimental and theoretical evidence across classical planning, RL, meta-learning, and sequence modeling attests, judicious macro-action construction remains a central, active theme in decision-making research.