Task-Insertion Refinement Methods

• Task-insertion-based refinement is a paradigm that integrates auxiliary sub-tasks into existing structures to repair incompleteness and enhance flexibility.
• It utilizes techniques like completion profiles in HTN planning, Bayesian optimization for robotic primitives, and hierarchical RL for adaptive insertion strategies.
• Empirical results indicate improved task solvability, efficient skill acquisition, and the emergence of creative behaviors across symbolic and continuous domains.

Task-insertion-based refinement refers to a family of methodologies in hierarchical planning, robot skill learning, and reinforcement learning wherein tasks or subroutines (“inserted tasks”) are added or composed within an existing task structure or skill policy in order to repair incompleteness, enable rapid adaptation, or facilitate the discovery of complex behaviors under sparse or ambiguous supervision. This paradigm is central to recent advances in both symbolic and continuous domains, including Hierarchical Task Network (HTN) planning and robotic manipulation, and is supported by algorithmic innovations spanning from the use of prioritized preferences and completion profiles to demonstration-driven dense reward modeling and hierarchical policy optimization.

1. Formal Foundations and Problem Statement

In symbolic planning, task-insertion-based refinement is operationalized in the context of Hierarchical Task Networks (HTNs) as a process of augmenting an initially incomplete set of methods $M$ for decomposing compound tasks $C$ into primitive tasks $O$ over a logical domain $L$ (Xiao et al., 2019). The refinement problem is defined as follows:

Given an HTN planning domain $D = (L, O, C, M)$, a prioritized preference partition $P = \langle P_1, \ldots, P_n \rangle$ over methods in $M$ (with $P_i$ as methods at priority $i$), and a set of planning instances $I = \{(s_0^j, t_0^j)\}$ (each an initial state and a root compound task), the objective is to find a minimal set of refined methods $M'$ such that, for the extended domain $D^+ = (L, O, C, M \cup M')$, every instance in $I$ is solvable, and $M'$ is minimal with respect to the lexicographic preference $\leq_P$.

In motor-skill and robot learning, the analogous objective is to extend a library of parameterized primitives (e.g., "move until contact", "search", "insert") so as to efficiently acquire and adapt insertion skills across a variety of objects and environments. The refinement process is cast as black-box optimization, typically over a vector of primitive parameters $\theta \in \mathbb{R}^d$, subject to dense, demonstration-driven reward modeling (Wu et al., 2022); or as hierarchical composition and insertion of auxiliary tasks—each with a reward function—into a global control policy or scheduler (Vezzani et al., 2020).

2. Core Algorithms and Theoretical Constructs

Mechanisms for task-insertion-based refinement vary across domains but share several key constructs.

2.1 Inserted Tasks and Completion Profiles (HTN/Planning)

A TIHTN (Task-Insertion HTN) planner is allowed to insert primitive tasks not initially specified in $M$, resulting in a plan $\pi$ and decomposition tree $T$. Inserted tasks are mapped to inner nodes of $T$ through a completion profile $\mathcal{P}: I_\pi \to N_T$, subject to causality-preserving constraints. Preferred completion profiles are constructed by greedily associating insertions with the highest-priority incomplete compound nodes, as specified by $P$ (Xiao et al., 2019).

2.2 Method Substitution and Minimalization

Refined methods with the same head are homologous and considered substitutable if one covers all usages of the other under the decomposition tree $T$. The overall refinement algorithm comprises collecting TIHTN-driven refinements from each instance, constructing their completion profiles, and then greedily selecting a minimal set of refinements per priority strata to ensure solvability of all training instances. The process is polynomial in the size of plans and domains, with the set-minimalization step exponential in the worst case but typically small-scale in practice.

2.3 Parameterized Primitive Optimization (Robotics)

In frameworks such as Prim-LAfD, primitives are described by parameter vectors $\theta$, where each entry corresponds to dynamic or geometric properties of a low-level motion routine. The learning objective

$$\theta^* = \arg\max_\theta \mathbb{E}[R(\theta)] - \lambda C(\theta)$$

combines a demonstration-driven, dense reward $R(\theta)$ and regularization $C(\theta)$. Gaussian-process Bayesian Optimization is employed iteratively to evaluate candidate $\theta_k$, update posterior reward estimates, and select acquisition-maximizing points for the next trial. The approach enables task-insertion-based refinement by allowing rapid optimization of motion primitives for both new and previously encountered insertion tasks (Wu et al., 2022).

2.4 Hierarchical and Multi-task RL with Task Insertion

In RL, task insertion manifests as compositional learning over a set of auxiliary intentions $T_1, \ldots, T_K$, each defined by distinct but related reward functions (e.g., reaching, grasping, pushing, aligning) that are scheduled and executed conditionally by a higher-level policy (Vezzani et al., 2020). Scheduled Auxiliary Control (SAC-X) jointly optimizes low-level controllers and a discrete scheduler. Regularized Hierarchical Policy Optimization (RHPO) applies KL-regularization to ensure stable, sample-efficient updates of the multi-purpose policy $\pi_\theta(a|s,T)$ across tasks and the main objective.

3. Demonstration-Driven Reward Modeling and Data Efficiency

Dense, demonstration-derived rewards play a central role in data-efficient task-insertion-based refinement. In Prim-LAfD, expert kinesthetic demonstrations $\mathcal{E}_d = \{\xi_i\}$ are encoded via Gaussian-mixture models over pairs $(x_i, x_{i-1})$, allowing the per-rollout reward $J(\xi; \theta) = \log p(\xi; \theta) + B$ (where $B$ denotes sparse success bonuses). This formulation ensures that even failure trials yield graded feedback, guiding optimization toward primitive parameters inducing expert-like behavior. The demonstration-driven paradigm circumvents the limitations of sparse rewards and enables effective learning with limited physical trials (e.g., skill acquisition in under one hour, adaptation to unseen tasks in ≈15 minutes) (Wu et al., 2022).

In multi-task RL, auxiliary intentions supply denser reward structures than the sparse final-task success signal, unlocking exploration and enabling the discovery of complex manipulation strategies via task-insertion scheduling and hierarchical policy updates (Vezzani et al., 2020).

4. Empirical Results and Application Domains

Task-insertion-based refinement methods have been validated in both symbolic and motor-skills domains.

Domain	Method/Framework	Key Metrics	Outcomes
HTN Planning	MethodRefine with TIHTN	Solving Rate, Methods	100% solving with 5–10 train instances; orders-of-magnitude compactness vs. HTN-MAKER (Xiao et al., 2019)
Robotic Insertion	Prim-LAfD	BO Iterations, Success	<1h acquisition, >90% success, 60% faster adaptation to novel tasks via task parameter transfer (Wu et al., 2022)
RL-based Insertion	SAC-X + RHPO	RL Episodes, % Success	Near-perfect simulation, 85–90% real-world success, emergent creative skills (Vezzani et al., 2020)

In HTN planning, MethodRefine demonstrates both high solving rates and minimal method set sizes compared to goal-annotated alternatives, especially under high-incompleteness regimes. In physical robot insertion scenarios, Prim-LAfD achieves high sample-efficiency and robust adaptation. RL-based approaches leveraging SAC-X and RHPO enable agents to solve under-actuated insertion tasks from scratch, discover new skills via inserted auxiliary tasks, and attain state-of-the-art success rates.

5. Algorithmic Limitations and Refined Directions

Task-insertion-based refinement frameworks exhibit several limitations:

• Current methods in symbolic planning rely on exhaustive search or greedy minimalization, incurring exponential complexity in the worst case within each priority stratum (Xiao et al., 2019). Their applicability depends on practical instance and method set sizes.
• Robotic motion primitive approaches (e.g., Prim-LAfD) depend on hand-designed primitives and state machines. Richer, end-to-end differentiable primitive representations are necessary for broader generalization. Reward models may also be limited by Markovian assumptions and lack of high-dimensional sensory integration (Wu et al., 2022).
• Shape-based similarity measures for transfer ignore frictional and compliance properties; incorporating force or learned embeddings could enhance adaptation robustness.
• RL frameworks may require thousands of episodes to achieve success in real-world hardware, though off-policy sharing and hierarchical regularization significantly mitigate sample inefficiency (Vezzani et al., 2020).

A plausible implication is that future work will further integrate differentiable, data-driven primitive representations and non-parametric meta-learning over broader sensory modalities to improve adaptability and generalization.

6. Generality, Emergent Behavior, and Theoretical Insights

Across domains, task-insertion enables handling of model incompleteness, facilitates creative composition of subroutines, and promotes generalization to novel task configurations. In symbolic planning, the reuse of inserted tasks for method refinement allows the planner’s own output to systematically repair incomplete domains, guided by prioritized preferences (Xiao et al., 2019). In motor skills, insertion and scheduling of auxiliary tasks not only hasten learning but also result in emergent behaviors—such as drop-and-flip or poke-and-flip strategies in under-actuated peg insertion—unanticipated at design time but necessary for successful task completion (Vezzani et al., 2020). This evidence underscores the essential role of task-insertion-based refinement in both explicitly encoded and autonomously discovered hierarchical control architectures.

7. Experimental Methodology and Comparative Analysis

Empirical evaluations adhere to rigorous standards:

• In HTN refinement, experiments on Logistics, Satellite, and Blocks-World use IPC problem generators, simulate varying degrees of method incompleteness, and measure solving rate across held-out test instances, explicitly comparing against HTN-MAKER and evaluating the effect of preference stratification (Xiao et al., 2019).
• In Prim-LAfD, eight insertion tasks (covering diverse hole geometries and commercial sockets) are used. Acquisition and adaptation speeds, as well as transfer learning efficacy, are quantified via iterations to success and comparative success rates under time-minimizing vs. dense reward BO objectives (Wu et al., 2022).
• RL-based task-insertion experiments precisely specify MDP structure, episode segmentation, reward shaping, and hyperparameters (network, optimization, replay schemes), and directly report first-insertion times, final success rates, and the qualitative nature of emergent policies (Vezzani et al., 2020).

The comparative analyses demonstrate the superiority of task-insertion-based refinement in enabling concise, data-efficient, and generalizable task-solving frameworks. These results support the ongoing integration of task-insertion into both symbolic planning and robotic policy learning pipelines.