PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics

Abstract: Simulated virtual environments serve as one of the main driving forces behind developing and evaluating skill learning algorithms. However, existing environments typically only simulate rigid body physics. Additionally, the simulation process usually does not provide gradients that might be useful for planning and control optimizations. We introduce a new differentiable physics benchmark called PasticineLab, which includes a diverse collection of soft body manipulation tasks. In each task, the agent uses manipulators to deform the plasticine into the desired configuration. The underlying physics engine supports differentiable elastic and plastic deformation using the DiffTaichi system, posing many under-explored challenges to robotic agents. We evaluate several existing reinforcement learning (RL) methods and gradient-based methods on this benchmark. Experimental results suggest that 1) RL-based approaches struggle to solve most of the tasks efficiently; 2) gradient-based approaches, by optimizing open-loop control sequences with the built-in differentiable physics engine, can rapidly find a solution within tens of iterations, but still fall short on multi-stage tasks that require long-term planning. We expect that PlasticineLab will encourage the development of novel algorithms that combine differentiable physics and RL for more complex physics-based skill learning tasks.

• The paper introduces PlasticineLab—a benchmark that integrates differentiable physics for soft-body manipulation, offering ten distinct tasks for robust RL evaluation.
• Its novel simulation employs the MLS-MPM method to accurately model elastic and plastic deformations, enabling gradient-based optimization for multi-stage planning.
• Experimental results reveal that traditional RL methods struggle in these environments, while gradient-based approaches show rapid convergence, highlighting the potential for hybrid techniques.

PlasticineLab: A Novel Benchmark for Soft-Body Manipulation

Introduction

The paper "PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics" (2104.03311) presents a new benchmark designed to address the limited exploration of soft-body dynamics within reinforcement learning (RL) frameworks. Traditional virtual environments, such as ALE, MuJoCo, and OpenAI Gym, cater primarily to rigid-body dynamics, thus overlooking the complexities inherent in soft-body manipulation tasks. PlasticineLab introduces ten distinct manipulation tasks revolving around plasticine, augmented by the DiffTaichi system to offer gradients for enhanced planning and optimization.

Figure 1: Left: A child deforming a piece of plasticine into a thin pie using a rolling pin. Right: The challenging RollingPin scene in PlasticineLab. The agent needs to flatten the material by rolling the pin back and forth, so that the plasticine deforms into the target shape.

Technical Approach

PlasticineLab is constructed around a differentiable physics engine, a significant departure from conventional simulation models. This engine offers differentiable elastic and plastic deformation capabilities by leveraging the Moving Least Squares Material Point Method (MLS-MPM). Such innovations enable the computation of gradients essential for optimizing open-loop control sequences, thus posing intricate challenges for RL algorithms.

The tasks in PlasticineLab range from pinching and rolling to more complex operations like multi-stage planning. Each task can be modeled as an MDP with defined state spaces, actions, goals, and rewards tailored to soft-body dynamics. The simulation's realism is bolstered by soft-rigid material interaction, encapsulated within tasks such as Chopsticks and RollingPin, showcasing the nuanced interplay between manipulation techniques and deformable materials.

Experimental Evaluation

The benchmark serves as an evaluation ground for a variety of RL and gradient-based algorithms. Despite the innovative framework, results indicate that most RL methods struggle with task completion, especially those involving multi-stage planning and high degrees of freedom typical in soft-body environments. Traditional RL methods like SAC, TD3, and PPO demonstrate suboptimal performance due to their failure to capitalize on the rich gradient information available.

Conversely, gradient-based approaches that utilize the differentiable physics engine reveal an ability to rapidly converge on solutions, albeit with limitations in handling complex, multi-stage tasks that demand long-term planning foresight. These findings underscore a critical need for algorithms that effectively integrate the strengths of differentiable physics and RL techniques.

Figure 2: The final normalized incremental IoU score achieved by RL methods within $10^4$ epochs. Scores lower than 0 are clamped. The dashed orange line indicates the theoretical upper limit.

Implications and Future Directions

PlasticineLab opens avenues for the development of hybrid algorithms that marry differentiable physics with RL strategies. Such combinations hold promise in addressing current limitations observed in handling long-term, multi-stage tasks within soft-body dynamics. Moreover, the benchmark's design and differentiable nature encourage exploration beyond traditional discrete-action settings, potentially impacting fields like virtual surgery modeling, soft robotics, and biomimetic actuators.

The paper suggests prospective research trajectories, including the enhancement of RL policies through better reward shaping, the design of neural networks capturing intricate shape variations, and the examination of generalization capabilities across variably configured environments. Additionally, transferring trained policies from simulation to real-world applications presents an unexplored frontier, with the potential for the simulator to aid in trajectory planning and sim-to-real transfer methods.

Conclusion

PlasticineLab presents a comprehensive benchmark for soft-body manipulation, pioneering the integration of differentiable physics within RL frameworks. While revealing the current limitations of state-of-the-art algorithms, it sets the stage for emerging research that could define the future landscape of intelligent soft-body manipulation. By bridging the gap between theoretical underpinnings and applied techniques, PlasticineLab offers a platform for significant advancements in both academic research and practical applications.