Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Neural Implicit Representation for Building Digital Twins of Unknown Articulated Objects (2404.01440v2)

Published 1 Apr 2024 in cs.CV, cs.AI, cs.GR, and cs.RO

Abstract: We address the problem of building digital twins of unknown articulated objects from two RGBD scans of the object at different articulation states. We decompose the problem into two stages, each addressing distinct aspects. Our method first reconstructs object-level shape at each state, then recovers the underlying articulation model including part segmentation and joint articulations that associate the two states. By explicitly modeling point-level correspondences and exploiting cues from images, 3D reconstructions, and kinematics, our method yields more accurate and stable results compared to prior work. It also handles more than one movable part and does not rely on any object shape or structure priors. Project page: https://github.com/NVlabs/DigitalTwinArt

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Learning to infer kinematic hierarchies for novel object instances. In 2022 International Conference on Robotics and Automation (ICRA), pages 8461–8467. IEEE, 2022.
  2. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2022.
  3. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15752–15761, 2021.
  4. Category-independent articulated object tracking with factor graphs. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3800–3807. IEEE, 2022.
  5. Carto: Category and joint agnostic reconstruction of articulated objects. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2023.
  6. Ditto in the house: Building articulation models of indoor scenes through interactive perception. ICRA, 2023.
  7. Learning to predict part mobility from a single static snapshot. ACM Transactions on Graphics (TOG), 36(6):1–13, 2017.
  8. Multibodysync: Multi-body segmentation and motion estimation via 3d scan synchronization. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2021.
  9. Occlusion-aware multi-view reconstruction of articulated objects for manipulation. Robotics and Autonomous Systems, 62(4):497–505, 2014.
  10. Opd: Single-view 3d openable part detection. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX, pages 410–426. Springer, 2022a.
  11. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2022b.
  12. Interactive segmentation, tracking, and kinematic modeling of unknown 3d articulated objects. In 2013 IEEE International Conference on Robotics and Automation, pages 5003–5010. IEEE, 2013.
  13. Skinning with dual quaternions. In Proceedings of the 2007 symposium on Interactive 3D graphics and games, pages 39–46, 2007.
  14. Unsupervised pose-aware part decomposition for man-made articulated objects. In European Conference on Computer Vision, pages 558–575. Springer, 2022.
  15. Category-level articulated object pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3706–3715, 2020.
  16. Paris: Part-level reconstruction and motion analysis for articulated objects. In Proceedings of the International Conference on Computer Vision (ICCV), 2023a.
  17. Building rearticulable models for arbitrary 3d objects from 4d point clouds. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2023b.
  18. Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pages 347–353. 1998.
  19. Sim2real2: Actively building explicit physics model for precise articulated object manipulation. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2023.
  20. Multiscan: Scalable rgbd scanning for 3d environments with articulated objects. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
  21. An integrated approach to visual perception of articulated objects. In 2016 IEEE international conference on robotics and automation (ICRA), pages 5091–5097. IEEE, 2016.
  22. Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6813–6823, 2021.
  23. A-sdf: Learning disentangled signed distance functions for articulated shape representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13001–13011, 2021.
  24. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  25. Watch it move: Unsupervised discovery of 3d joints for re-posing of articulated objects. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2022.
  26. Colored point cloud registration revisited. In Proceedings of the IEEE international conference on computer vision, pages 143–152, 2017.
  27. Learning articulated motions from visual demonstration. RSS, 2014.
  28. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
  29. Self-supervised learning of part mobility from point cloud sequence. In Computer Graphics Forum, pages 104–116. Wiley Online Library, 2021.
  30. A probabilistic framework for learning kinematic models of articulated objects. Journal of Artificial Intelligence Research, 41:477–526, 2011.
  31. Loftr: Detector-free local feature matching with transformers. In Proceedings of the Computer Vision and Pattern Recognition (CVPR), 2021.
  32. Cla-nerf: Category-level articulated neural radiance field. In 2022 International Conference on Robotics and Automation (ICRA), pages 8454–8460. IEEE, 2022.
  33. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876–8884, 2019.
  34. Self-supervised neural articulated shape and appearance models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15816–15826, 2022.
  35. Bundlesdf: Neural 6-dof tracking and 3d reconstruction of unknown objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 606–617, 2023.
  36. Captra: Category-level pose tracking for rigid and articulated objects from point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13209–13218, 2021.
  37. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097–11107, 2020.
  38. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3173–3182, 2021.
  39. Deep part induction from articulated object pairs. ACM Transactions on Graphics, 37(6):1–15, 2018.
  40. Visual identification of articulated object parts. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2443–2450. IEEE, 2021.
  41. Fast global registration. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 766–782. Springer, 2016.
  42. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2019.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets