Learning Variable Impedance Control for Contact Sensitive Tasks (1907.07500v2)

Abstract: Reinforcement learning algorithms have shown great success in solving different problems ranging from playing video games to robotics. However, they struggle to solve delicate robotic problems, especially those involving contact interactions. Though in principle a policy directly outputting joint torques should be able to learn to perform these tasks, in practice we see that it has difficulty to robustly solve the problem without any given structure in the action space. In this paper, we investigate how the choice of action space can give robust performance in presence of contact uncertainties. We propose learning a policy giving as output impedance and desired position in joint space and compare the performance of that approach to torque and position control under different contact uncertainties. Furthermore, we propose an additional reward term designed to regularize these variable impedance control policies, giving them interpretability and facilitating their transfer to real systems. We present extensive experiments in simulation of both floating and fixed-base systems in tasks involving contact uncertainties, as well as results for running the learned policies on a real system.

Learning Variable Impedance Control for Contact Sensitive Tasks

The paper "Learning Variable Impedance Control for Contact Sensitive Tasks" addresses the challenges faced by reinforcement learning (RL) algorithms when applied to robotic tasks involving complex contact interactions. The authors introduce a novel approach that adapts variable impedance control to enhance robustness and performance in the presence of contact uncertainties. This paper compares the efficacy of different action space representations in RL, specifically examining torque, fixed-gain, and variable-gain position control methodologies.

Problem Context and Motivation

Robotic systems often engage in tasks that necessitate intricate physical interactions. These operations, such as object manipulation or locomotion, invariably involve establishing and severing contact with external entities, which can complicate dynamic modeling due to abrupt changes in system behavior. Traditional RL techniques have exhibited notable success in observation-heavy tasks yet struggle with purely dynamic interaction aspects. The paper hypothesizes that optimizing the action space configuration can lead to substantial improvements in learning efficiency and task execution under contact conditions.

Approach and Technical Contribution

The paper investigates variable impedance control strategies in joint space, distinct from the commonly employed operational space approaches. By doing so, it aims to capitalize on the potential flexibility offered by modulating joint impedance parameters, which adaptively respond to different operational demands.

The authors propose three different control policy parametrizations:

1. Direct Torque Control: Directly outputs joint torques with no structural constraints, potentially offering precise interaction force control at the cost of increased learning complexity.
2. Fixed Gain PD Control: Utilizes pre-defined feedback gains with the RL policy controlling desired joint positions, simplifying exploration but offering limited adaptability in dynamic environments.
3. Variable Gain PD Control: Allows RL policies to dynamically adjust both joint positions and impedance, facilitating robust and adaptive interaction handling.

Results indicate that the variable gain approach prominently outperformed competing strategies in simulated environments and was effectively transferred to real robotic systems. Notably, the robustness of these policies remained high across scenarios involving varied contact friction, location, and stiffness parameters.

Key Findings and Empirical Results

The empirical analysis encompassed two distinct robotic setups—a hopping task on a single-leg robot and a fixed-base manipulator performing a force-sensitive wiping task. The authors demonstrated that variable gain policies significantly streamlined the learning process and improved robustness to environmental uncertainties, as compared to fixed-gain and direct torque policies.

• In the hopping task, the variable gain controller achieved superior performance, demonstrating smoother motion transitions upon contact.
• The manipulator task highlighted how dynamic impedance modulation facilitated stable interaction force control even amidst uncertain surface characteristics.
• A trajectory tracking regularization term was introduced, simplifying policy outputs without sacrificing performance. This term ensured policy interpretability and enabled direct deployment onto real hardware without notable loss in efficacy.

Implications for Future Research and Applications

This paper entails meaningful implications for the design of RL frameworks in robotics, emphasizing the importance of adaptive control strategies over traditional fixed action spaces. The approach can be extended to more complex robotic systems, where robustness to environmental unpredictability is crucial—such as autonomous vehicular navigation or humanoid robot manipulation tasks. Future work might focus on integrating these strategies with multi-agent learning paradigms or exploring their efficacy in environments with dynamically evolving constraints.

Conclusion

The paper presents a compelling case for the use of variable impedance control within joint space as a viable reinforcement learning strategy for contact-sensitive robotic tasks. By methodically evaluating several controller configurations, the authors provide profound insights into enhancing RL-based robotic interaction with complex environments, thus paving the way for more versatile and resilient autonomous robotic applications.