Compact Multi-Object Placement Using Adjacency-Aware Reinforcement Learning
Abstract: Close and precise placement of irregularly shaped objects requires a skilled robotic system. The manipulation of objects that have sensitive top surfaces and a fixed set of neighbors is particularly challenging. To avoid damaging the surface, the robot has to grasp them from the side, and during placement, it has to maintain the spatial relations with adjacent objects, while considering the physical gripper extent. In this work, we propose a framework to learn an agent based on reinforcement learning that generates end-effector motions for placing objects as closely as possible to one another. During the placement, our agent considers the spatial constraints with neighbors defined in a given layout of the objects while avoiding collisions. Our approach learns to place compact object assemblies without the need for predefined spacing between objects, as required by traditional methods. We thoroughly evaluated our approach using a two-finger gripper mounted on a robotic arm with six degrees of freedom. The results demonstrate that our agent significantly outperforms two baseline approaches in object assembly compactness, thereby reducing the space required to position the objects while adhering to specified spatial constraints.
- A. Lobbezoo, Y. Qian, and H.-J. Kwon, “Reinforcement Learning for Pick and Place Operations in Robotics: A Survey,” Robotics, vol. 10, no. 3, p. 105, 2021.
- K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber, “A Survey on Learning-Based Robotic Grasping,” Current Robotics Reports, vol. 1, no. 4, pp. 239–249, 2020.
- A. X. Lee, C. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, C. Fantacci, J. E. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, S. Saliceti, F. Casarini, M. Riedmiller, R. Hadsell, and F. Nori, “Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes,” 2021.
- J. Zhang, W. Zhang, R. Song, L. Ma, and Y. Li, “Grasp for Stacking via Deep Reinforcement Learning,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 2543–2549.
- L. Berscheid, P. Meißner, and T. Kröger, “Self-Supervised Learning for Precise Pick-and-Place Without Object Model,” IEEE Trans. on Robotics (T-RO), vol. 5, no. 3, pp. 4828–4835, 2020.
- F. Wang and K. Hauser, “Dense Robotic Packing of Irregular and Novel 3D Objects,” IEEE Transactions on Robotics, vol. 38, no. 2, pp. 1160–1173, 2022.
- H. Ren and R. Zhong, “An Autonomous Ore Packing system through Deep Reinforcement Learning,” Advances in Space Research, p. S0273117724001273, 2024.
- S. Liu, T. Tang, W. Zhang, J. Cui, and X. Xu, “A Brief Review of Recent Hierarchical Reinforcement Learning for Robotic Manipulation,” in 2022 10th International Conference on Information Systems and Computing Technology (ISCTech), 2022, pp. 721–728.
- I. J. Park, H. Han, J. Cho, and J. H. Park, “Knowledge-Based Reinforcement Learning for Industrial Robotic Assembly,” in 2023 14th International Conference on Information and Communication Technology Convergence (ICTC), 2023, pp. 1684–1689.
- A. Orsula, S. Bøgh, M. Olivares-Mendez, and C. Martinez, “Learning to Grasp on the Moon from 3D Octree Observations with Deep Reinforcement Learning,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 4112–4119.
- F. Ceola, E. Maiettini, L. Rosasco, and L. Natale, “A Grasp Pose is All You Need: Learning Multi-fingered Grasping with Deep Reinforcement Learning from Vision and Touch,” 2023.
- S. Kim and A. Rodriguez, “Active Extrinsic Contact Sensing: Application to General Peg-in-Hole Insertion,” 2022.
- Z. He, N. Chavan-Dafle, J. Huh, S. Song, and V. Isler, “Pick2Place: Task-Aware 6DoF Grasp Estimation Via Object-Centric Perspective Affordance,” in Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 2023.
- A. Kuznetsov, P. Shvechikov, A. Grishin, and D. Vetrov, “Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics,” in Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 5556–5566.
- P. Harel and O. Ben-Shahar, “Crossing cuts polygonal puzzles: Models and Solvers,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3083–3092.
- E. Coumans and Y. Bai, “PyBullet, a Python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016.
- S. Q. Xie, G. G. Wang, and Y. Liu, “Nesting of two-dimensional irregular parts: An integrated approach,” International Journal of Computer Integrated Manufacturing, vol. 20, no. 8, pp. 741–756, 2007.
- A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-Baselines3: Reliable Reinforcement Learning Implementations.”
- G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “OpenAI Gym,” 2016.
Paper Prompts
Sign up for free to create and run prompts on this paper.