Impedance Primitive-Augmented HRL

• The paper shows IMP-HRL integrates adaptive stiffness control with hierarchical primitive selection to enhance robustness in sequential contact tasks.
• The methodology employs a high-level policy for primitive selection and a low-level policy that outputs both position and stiffness parameters, with online tuning for compliance.
• Experimental evaluations on simulated and real Franka Panda robots demonstrate reduced interaction forces, improved success rates, and enhanced safety in robotic manipulation.

Impedance Primitive-augmented Hierarchical Reinforcement Learning (IMP-HRL) is a hierarchical reinforcement learning framework for sequential contact tasks in robotic manipulation that augments low-level behavior primitives with variable stiffness control capabilities (Tahmaz et al., 27 Aug 2025). In this formulation, a high-level policy selects behavior primitives, while a low-level policy outputs both position and initial stiffness parameters for the selected primitive, and an adaptive stiffness controller further tunes stiffness online during primitive execution (Tahmaz et al., 27 Aug 2025). The framework is presented for block lifting, door opening, object pushing, and surface cleaning, with simulated and real-world evaluations on a Franka Emika Panda in Robosuite and on the physical robot (Tahmaz et al., 27 Aug 2025). More broadly, IMP-HRL lies at the intersection of hierarchical reinforcement learning, impedance control, primitive-based policy decomposition, and contact-rich robot learning, and it can be situated alongside earlier work on decentralized primitive selection (Goyal et al., 2019), hierarchical primitive sequencing (Li et al., 2019), primitive-enabled preference learning (Singh et al., 2024), and reinforcement learning for adaptive impedance control in prosthetics, rehabilitation, and contact manipulation (Wu et al., 2021, Zhang et al., 2021, Chang et al., 2022).

1. Conceptual definition and problem setting

IMP-HRL is introduced as an approach for efficient robotic manipulation in sequential contact tasks by sequentially executing behavior primitives with variable stiffness control capabilities (Tahmaz et al., 27 Aug 2025). Its stated key components are an action space enabling variable stiffness control, an adaptive stiffness controller for dynamic stiffness adjustments during primitive execution, and affordance coupling for efficient exploration while encouraging compliance (Tahmaz et al., 27 Aug 2025). The central design decision is that stiffness is not treated as a fixed controller parameter outside the learning loop; instead, it is embedded into primitive execution and adjusted online (Tahmaz et al., 27 Aug 2025).

The framework is built on Parameterized Action MDPs (PAMDP), with a high-level policy $\pi_H$ that chooses behavior primitives such as reach, grasp, and push, and a low-level policy $\pi_{param}$ that outputs position and initial stiffness parameters for the selected primitive (Tahmaz et al., 27 Aug 2025). The policy composition is written as

$\pi(s) = \pi_{param}^{\pi_H(s)}(s).$

This makes primitive selection and primitive parameterization distinct but coupled stages within the hierarchy (Tahmaz et al., 27 Aug 2025).

The emphasis on contact tasks places IMP-HRL within a class of methods where compliance is operationally important. Earlier impedance-learning work in manipulation framed stiffness as the mechanism that balances between following a position or force reference trajectory, with high stiffness tracking position and low stiffness tracking force (Chang et al., 2022). In prosthetics and gait rehabilitation, impedance parameters likewise govern torque production or assistive force fields and are adapted phase-dependently or cycle-by-cycle (Wu et al., 2021, Zhang et al., 2021). This suggests that IMP-HRL extends a well-established control idea—task-dependent impedance modulation—into a primitive-based hierarchical RL setting for sequential manipulation.

2. Hierarchical architecture and parameterized primitive execution

In IMP-HRL, the high-level policy selects behavior primitives, and the low-level policy outputs the primitive parameters (Tahmaz et al., 27 Aug 2025). Unlike standard HRL such as MAPLE, where primitive parameters are purely positional, the IMP-HRL action space is augmented with impedance parameters (Tahmaz et al., 27 Aug 2025). For each primitive, the parameter vector is

$\theta = (x, y, z, K_x, K_y, K_z, K_{\psi}),$

where $(x,y,z)$ is the Cartesian target position, $K_x, K_y, K_z$ are stiffness values along each axis, and $K_{\psi}$ is rotational stiffness for orientation (Tahmaz et al., 27 Aug 2025). Damping is not learned but set for critical damping: $D(t) = 2\sqrt{K(t)}.$

These parameterized primitives are executed as closed-loop controllers that minimize error to both position and force targets (Tahmaz et al., 27 Aug 2025). That design directly links IMP-HRL to prior formulations in which low-level controllers track reference trajectories under impedance or admittance dynamics. In the contact DMP setting, for example, the admittance law

$$\mathbf{M}\ddot{\mathbf{x}_e} + \mathbf{D}\dot{\mathbf{x}_e} + \mathbf{K}\mathbf{x}_e = \mathbf{F}_{ext} - \mathbf{F}_d$$

uses stiffness $\mathbf{K}$ as the principal quantity adapted by the RL policy (Chang et al., 2022). In the robotic knee prosthesis setting, a finite state machine impedance controller divides gait into four phases—Stance Flexion, Stance Extension, Swing Flexion, and Swing Extension—and impedance parameters $\pi_{param}$0, $\pi_{param}$1, and $\pi_{param}$2 determine the torque profile (Wu et al., 2021). Although these systems address different embodiments and tasks, they share the structural premise that high-level adaptation acts on low-level impedance parameters rather than directly on torques alone.

Earlier HRL work without impedance augmentation often decomposed action generation differently. "Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning" (Li et al., 2019) divided locomotion into learning primitive skills and using model-based planning to sequence them, with primitives parameterized as cyclic movements and planned by MPC over coarse dynamics models. "Model Primitive Hierarchical Lifelong Reinforcement Learning" (Wu et al., 2019) learned specialized subpolicies coordinated by a gating controller, while "Hierarchical Primitive Composition" (Lee et al., 2021) allowed simultaneous activation of skills with inconsistent action dimensions through multiplicative Gaussian distributions. IMP-HRL differs in that its primitive parameterization explicitly includes stiffness and its low-level execution is formulated around contact-aware variable impedance (Tahmaz et al., 27 Aug 2025).

3. Variable stiffness control and adaptive stiffness regulation

A defining feature of IMP-HRL is that initial stiffness selected by the low-level policy is further tuned online by an adaptive controller inspired by human motor behavior (Tahmaz et al., 27 Aug 2025). The controller updates stiffness according to

$\pi_{param}$3

where $\pi_{param}$4 is the current control error, $\pi_{param}$5 is energy consumed by robot joints as a proxy for effort, and $\pi_{param}$6 are scaling coefficients determined via kinesthetic demonstration or parameter tuning (Tahmaz et al., 27 Aug 2025). After each update,

$\pi_{param}$7

The stated effect is to reduce stiffness for compliance unless high error necessitates stiffening (Tahmaz et al., 27 Aug 2025).

This mechanism closely parallels the broader literature on RL-based impedance adaptation. In the contact DMP framework, a high-level RL policy adapts linear and angular stiffness values for each axis during execution, while damping is set for critical damping given stiffness (Chang et al., 2022). That work reports that RL-adapted impedance improved stability and rate of successful insertions in peg-in-hole and yielded highest success rates and robustness in adhesive strip application, whereas fixed stiffness performed worse under perturbation (Chang et al., 2022). In robotic gait rehabilitation, PI2-AAN learns a phase-dependent impedance landscape $\pi_{param}$8 from subject responses, and the controller switches between intervention and compliance modes depending on aggregate performance bounds (Zhang et al., 2021). In the robotic knee prosthesis work, RL performs online policy iteration over updates

$\pi_{param}$9

to tune stiffness, damping, and equilibrium position in a co-adapting environment (Wu et al., 2021).

Within IMP-HRL, the specific novelty is the placement of such adaptive stiffness regulation inside primitive execution rather than as a stand-alone controller adaptation problem (Tahmaz et al., 27 Aug 2025). A plausible implication is that the primitive abstraction becomes richer than a purely kinematic option: each primitive carries not only a target pose but also a compliance profile that can evolve during execution.

4. Affordance coupling, exploration, and compliance

IMP-HRL introduces affordance coupling to guide exploration and encourage compliance (Tahmaz et al., 27 Aug 2025). The framework couples position and stiffness affordances. The position affordance is defined as

$\pi(s) = \pi_{param}^{\pi_H(s)}(s).$0

where $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$1 is the set of object keypoints and $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$2 is a threshold (Tahmaz et al., 27 Aug 2025). The stiffness affordance rewards low stiffness except when the task requires it, thereby promoting compliance and safety during learning (Tahmaz et al., 27 Aug 2025). Both affordances are jointly considered in the reward, biasing early exploration toward safer, compliant behaviors without expensive manual reward tuning (Tahmaz et al., 27 Aug 2025).

This design choice addresses a standard difficulty in contact-rich RL: efficient exploration without excessive interaction forces. IMP-HRL reports that the framework reduces maximum interaction forces significantly, as shown in simulated force profiles and real robot runs (Tahmaz et al., 27 Aug 2025). The same theme appears in earlier impedance-adaptation work. In contact DMPs, low stiffness enables better force tracking and compliance when accurate force following is critical, especially after contact is made (Chang et al., 2022). In assist-as-needed gait training, impedance control is used to create a compliant force field around a target motion while allowing moderate kinematic errors, and the proposed controller autonomously reshapes this force field in a phase-dependent manner according to each individual's motor abilities and task requirements (Zhang et al., 2021).

From a hierarchical RL perspective, affordance coupling also serves as a structural prior on primitive parameterization. Primitive-based HRL often requires an additional mechanism to keep exploration aligned with feasible subgoals or relevant state regions. DIPPER uses primitive-informed regularization based on the lower-level value function so that the higher-level policy is regularized toward subgoals that the lower-level can actually achieve (Singh et al., 2024). The information-constrained primitive ensemble of (Goyal et al., 2019) instead uses an information bottleneck and competition among primitives to induce specialization without a meta-policy. IMP-HRL’s affordance coupling is different in form, but it plays a comparable role in restricting the effective search space toward safer and more task-relevant primitive instantiations (Tahmaz et al., 27 Aug 2025).

5. Experimental domains, performance, and sim2real evaluation

IMP-HRL is evaluated in simulation with a Franka Emika Panda in the Robosuite simulator on four tasks: Lift, Door, Cleanup, and Wipe (Tahmaz et al., 27 Aug 2025). Randomization is applied to table friction, object pose, and robot start position (Tahmaz et al., 27 Aug 2025). Real-world tasks include Lift, Cleanup, and Wipe on the actual Franka Panda arm with RGB-D perception and Deep Object Pose for object localization (Tahmaz et al., 27 Aug 2025).

The reported results distinguish among learning efficiency, compositionality, contact behavior, and success rates (Tahmaz et al., 27 Aug 2025). IMP-HRL outperforms or matches MAPLE in convergence time, especially for Wipe, while MAPLE may learn slightly faster on Lift due to fewer constraints from affordance coupling (Tahmaz et al., 27 Aug 2025). IMP-HRL yields more consistent primitive selection patterns for a given task, and compositionality matches or exceeds MAPLE (Tahmaz et al., 27 Aug 2025). The framework also reduces maximum interaction forces significantly in simulation and on the real robot (Tahmaz et al., 27 Aug 2025).

The success-rate table reported in the paper is as follows (Tahmaz et al., 27 Aug 2025):

Method	Task	Success
MAPLE (sim)	Lift / Door / Wipe / Cleanup	100% / 100% / 42% / 91%
Ours (sim)	Lift / Door / Wipe / Cleanup	100% / 100% / 86% / 87%
Ours (real)	Lift / Wipe / Cleanup	90% / 70% / 80%

The Wipe task is the clearest differentiator: IMP-HRL nearly doubles MAPLE's success rate in simulation, and the stated explanation is that stiffness adaptation preserves compliant contact whereas position-only control failed due to loss of compliant contact (Tahmaz et al., 27 Aug 2025). Lift and Door are reported as comparable, while Cleanup is slightly lower in simulation under the compliance-oriented design (Tahmaz et al., 27 Aug 2025). Real-world evaluations are described as confirming sim2real capability, with minor or no fine-tuning (Tahmaz et al., 27 Aug 2025).

Earlier studies provide related empirical context. In the contact DMP paper, RL impedance adaptation maintained over 90% success under a deliberate $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$3 positional offset in adhesive strip application, while fixed-stiffness methods deteriorated sharply (Chang et al., 2022). In the prosthesis paper, OpenSim simulations across 30 random trials achieved 100% convergence to tracking error tolerance and average convergence in approximately 19 updates, and human subject testing found significant reductions in tracking errors with $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$4 (Wu et al., 2021). Those results concern different embodiments, but they support the more general proposition that online impedance adaptation is particularly valuable when the environment or human counterpart is co-adapting or when contact geometry varies.

6. Relation to broader primitive-based hierarchical reinforcement learning

IMP-HRL belongs to a wider family of methods that structure behavior through primitives, subpolicies, or options rather than learning a single flat policy. "Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives" (Goyal et al., 2019) removes the explicit meta-policy and lets primitives compete through information-theoretic selection, with the primitive that requests the most information about the current state acting in the world. Each primitive is regularized by an information bottleneck

$\pi(s) = \pi_{param}^{\pi_H(s)}(s).$5

and primitives are selected using softmax scores $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$6 derived from their information costs (Goyal et al., 2019). The reported result is improved generalization relative to flat and hierarchical baselines in MiniGrid, Four-Rooms, AntMaze, and motion imitation (Goyal et al., 2019). This suggests one possible alternative to explicit high-level primitive selection in contact control: decentralized competition rather than a central meta-policy.

Other primitive-oriented HRL methods emphasize different aspects. "Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning" (Li et al., 2019) learns cyclic locomotion primitives and sequences them with MPC to reach arbitrary goals up to 12 m away after about two hours of hardware training from scratch. "Model Primitive Hierarchical Lifelong Reinforcement Learning" (Wu et al., 2019) uses diverse suboptimal world models to induce bottom-up task decomposition and transfer subpolicies across tasks. "Hierarchical Primitive Composition" (Lee et al., 2021) allows simultaneous activation of skills with inconsistent action dimensions in multiple recursive hierarchies through multiplicative Gaussian composition. "DIPPER" (Singh et al., 2024) learns the higher-level policy from human trajectory preferences using direct preference optimization while regularizing the higher-level policy with lower-level value estimates to mitigate non-stationarity and infeasible subgoal generation.

IMP-HRL is distinguished from these approaches by the explicit integration of variable stiffness into the primitive parameter vector, the adaptive stiffness controller for online tuning, and affordance coupling that jointly optimizes for position and compliance (Tahmaz et al., 27 Aug 2025). A plausible implication is that IMP-HRL can be read as a contact-centric specialization of primitive-based HRL: primitives are not only reusable temporal abstractions, but also carriers of compliance structure.

7. Limitations, tradeoffs, and research directions

The limitations reported for IMP-HRL are specific. Affordance coupling favors compliance, which can slow learning for tasks that require high precision, such as delicate grasping (Tahmaz et al., 27 Aug 2025). The adaptive controller parameters $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$7 and $\pi(s) = \pi_{param}^{\pi_H(s)}(s).$8 require either kinesthetic demonstrations or manual tuning, which is described as potentially time-consuming (Tahmaz et al., 27 Aug 2025). The stated future directions are task-adaptive affordance coupling, automating or learning the adaptive controller’s parameters, and extending to more complex, long-horizon, or multi-stage tasks and larger primitive libraries (Tahmaz et al., 27 Aug 2025).

These limitations align with tensions already visible in adjacent work. In rehabilitation, hand-tuned or simplistically adapted impedance parameters were one of the motivations for replacing conventional assist-as-needed control with real-time RL-based reshaping of an impedance landscape (Zhang et al., 2021). In prosthetics, the co-adapting environment makes fixed mathematical modeling impractical, motivating online policy iteration over impedance settings (Wu et al., 2021). In primitive-based HRL more generally, success depends on decomposition quality: model primitive diversity is critical in MPHRL (Wu et al., 2019), inconsistent state-space assumptions can limit reuse in HPC (Lee et al., 2021), and hand-designed primitives do not scale well in DIPPER’s comparisons (Singh et al., 2024).

A common misconception is that hierarchical reinforcement learning automatically guarantees better contact behavior once primitives are introduced. The literature summarized here does not support that claim. IMP-HRL attributes its gains specifically to variable stiffness control, adaptive stiffness updates, and affordance coupling (Tahmaz et al., 27 Aug 2025). Earlier work on contact manipulation similarly shows that position trajectories alone may not be safe or robust over variation in contact geometry and that fixed impedance leads to suboptimal performance and sensitivity to disturbances (Chang et al., 2022). The evidence therefore points not to hierarchy alone, but to the joint design of hierarchy, low-level compliance, and exploration bias.

In that sense, IMP-HRL marks a particular synthesis: hierarchical selection of behavior primitives, parameterized primitive execution in position-and-stiffness space, and online stiffness adaptation under a compliance-oriented exploration scheme (Tahmaz et al., 27 Aug 2025). This combination defines its contribution within the broader development of primitive-based and impedance-adaptive reinforcement learning.