Skill-It Framework Overview

• Skill-It Framework is a hierarchical, skill-centric system that integrates high-level skill acquisition with automatic symbolic state abstraction.
• It alternates between acquiring temporally extended actions (options) and inducing abstract representations, recursively constructing Markov decision processes.
• The framework facilitates efficient planning and compact reasoning, making it applicable to hierarchical reinforcement learning, robotic control, and automated agent design.

The Skill-It Framework encapsulates a class of hierarchical, skill-centric learning and planning methodologies that alternate between the acquisition of high-level skills (temporally extended actions, options, or reusable behaviors) and the induction of symbolic abstract representations tailor-made to these skills. By iteratively building abstraction hierarchies in Markov decision processes, Skill-It enables fast planning and compact reasoning over complex sequential tasks. The framework tightly couples the set of available skills with the structure of abstract state representations, facilitating efficient computational planning in domains such as hierarchical reinforcement learning, robotic control, and automated agent design.

1. Alternating Skill Acquisition and Representation Induction

The foundational mechanism is the skill-symbol loop, in which two iterative phases are alternated:

• Skill Acquisition: The agent discovers or is provided with high-level skills, formalized as options—tuples ($I_o, \pi_o, \beta_o$) encompassing initiation sets, option policies, and termination conditions. These can be subgoal options targeting a particular effect set or abstract subgoal options manipulating only relevant state factors.
• Representation Acquisition: Based on the structure of acquired skills, the agent induces an abstract symbolic state space in which propositional symbols ($\sigma_Z$, with set membership tests $\tau_Z(s) = 1 \iff s \in Z$) partition the environment according to skill preconditions and effects.

This loop yields a recursive hierarchy of Markov decision processes: at each abstraction level $j$, the MDP $M_j = (S_j, A_j, R_j, P_j)$ is constructed, where $A_j$ is the set of options over $M_{j-1}$ and $S_j$ comprises abstract symbols relevant to those options.

2. Formal Structure: Skill-Driven Hierarchical Markov Decision Processes

The Skill-It framework generalizes classic MDPs. At each level $j$:

$M_j = (S_j, A_j, R_j, P_j)$

• $S_j$ is the abstract state set, derived from symbols induced by the current collection of skills.
• $A_j$ consists of options (temporally abstract actions) operating over $M_{j-1}$.
• Option definition: Each option $o$ is a tuple ($I_o, \pi_o, \beta_o$), where:
  • $I_o \subset S$ (initiation set)
  • $\pi_o$: option policy mapping states to primitive actions
  • $\beta_o$: termination condition
  • Effect set (image): the set of states where the option will terminate

Abstract representations are formed by logical combinations of propositional symbols. The grounding operator ($\mathcal{G}$) enables reasoning about intersection and containment of initiation/effect sets, supporting symbolic plan graph construction.

3. Symbolic Representation and Plan Graph Construction

Upon acquiring a set of options, the system induces a symbolic plan graph:

• Nodes: Abstract symbols representing sets of states defined by option initiation/effect sets.
• Edges: Chaining relationships—an edge exists if the image of one option is a subset of another's initiation set.
• Operations: Logical combinations (intersection, containment) via the grounding operator.

This abstraction enables high-level planning by traversing the plan graph: the agent can devise a plan at the abstract level using symbolic reasoning and then refine it into low-level actions.

4. Case Study: Taxi Domain Hierarchy Construction

The Taxi domain exemplifies hierarchical construction:

• Base MDP ($M_0$): 5×5 grid, 650 states (taxi and passenger positions).
• Level $M_1$: Introduction of navigation options (drive-to-depot, pick-up, put-down), yielding an abstracted state space factoring depot and passenger locations.
• Level $M_2$: Higher-level options (passenger-to-depot), further abstracting to just passenger location per depot.

Planning queries illustrate the effectiveness: some problems are solved immediately at the highest-level abstraction; others require refinement to lower levels, with backtracking only when the abstraction does not capture the necessary detail (e.g., query conditions ignored by coarse abstraction).

5. Computational Efficiency, Refinement, and Trade-Offs

Advantages:

• Fast Planning: Reduces dimensionality and decision complexity; macro-actions over abstract state space mean fewer steps to plan.
• Automatic State Space Abstraction: Skills acquired trigger corresponding abstraction; only relevant state features are considered.
• Downward Refinement: Abstract plans can be incrementally refined to produce base-level action sequences, typically without backtracking.

Limitations:

• Hierarchy Construction Overhead: Coarse abstraction may omit critical state details; queries mismatched to the abstraction require fallback, increasing computational cost.
• False Positive Matches: High-level symbolic plans may not correspond to feasible base-level sequences, necessitating deeper search.
• Skill Discovery Dependency: Performance relies on efficacy of skill discovery; options not optimally constructed can hinder abstraction structuring and planning utility.

6. Relation to Other Hierarchical RL Frameworks

Skill-It diverges from approaches such as MAX-Q and Hierarchy of Abstract Machines:

• In traditional HRL, state representation is fixed and skill learning can occur independently; Skill-It couples skill acquisition with automatic abstraction of state space, tightly integrating symbolic planning with option learning.
• The use of formal symbolic representations (akin to STRIPS/PDDL planning formalisms) and logical operators allows leveraging modern symbolic planners atop RL-acquired skill sets.
• This dual focus—skills as the foundation for both option-space expansion and representation abstraction—provides systematic reduction in planning complexity and better links between high-level reasoning and low-level perception/action.

7. Research Directions and Challenges

The Skill-It mechanism highlights several ongoing challenges:

• Optimizing Skill Discovery Algorithms: Automated discovery that yields temporally abstract and planning-relevant options remains an open problem.
• Balancing Representation Fidelity: Retaining critical environment details during successive abstractions is vital for applicability to a broader class of queries.
• Scalable Symbolic Reasoning: Managing the combinatorial expansion of symbolic representations and plan graphs, especially in domains with complex dynamics or high-dimensional states, requires efficient reasoning techniques.

This suggests the Skill-It paradigm forms the basis for extensible, skill-driven abstraction hierarchies, offering robust mechanisms for symbolic reasoning and efficient planning—provided the agent’s skill discovery remains aligned with representational precision and task requirements.