Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models (2401.12978v3)

Published 23 Jan 2024 in cs.CV

Abstract: Understanding the inherent human knowledge in interacting with a given environment (e.g., affordance) is essential for improving AI to better assist humans. While existing approaches primarily focus on human-object contacts during interactions, such affordance representation cannot fully address other important aspects of human-object interactions (HOIs), i.e., patterns of relative positions and orientations. In this paper, we introduce a novel affordance representation, named Comprehensive Affordance (ComA). Given a 3D object mesh, ComA models the distribution of relative orientation and proximity of vertices in interacting human meshes, capturing plausible patterns of contact, relative orientations, and spatial relationships. To construct the distribution, we present a novel pipeline that synthesizes diverse and realistic 3D HOI samples given any 3D object mesh. The pipeline leverages a pre-trained 2D inpainting diffusion model to generate HOI images from object renderings and lifts them into 3D. To avoid the generation of false affordances, we propose a new inpainting framework, Adaptive Mask Inpainting. Since ComA is built on synthetic samples, it can extend to any object in an unbounded manner. Through extensive experiments, we demonstrate that ComA outperforms competitors that rely on human annotations in modeling contact-based affordance. Importantly, we also showcase the potential of ComA to reconstruct human-object interactions in 3D through an optimization framework, highlighting its advantage in incorporating both contact and non-contact properties.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (107)
  1. Affordances from human videos as a versatile representation for robotics. In CVPR, 2023.
  2. Behave: Dataset and method for tracking human object interactions. In CVPR, 2022.
  3. Contactdb: Analyzing and predicting grasp contact via thermal imaging. In CVPR, 2019a.
  4. Contactgrasp: Functional multi-finger grasp synthesis from contact. In IROS, 2019b.
  5. Contactpose: A dataset of grasps with object contact and hand pose. In ECCV, 2020.
  6. Understanding hand-object manipulation with grasp types and object attributes. In RSS, 2016.
  7. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE TPAMI, 2019.
  8. Long-term human motion prediction with scene context. In ECCV, 2020.
  9. Ensembling with deep generative views. In CVPR, 2021.
  10. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], 2015.
  11. Dexycb: A benchmark for capturing hand grasping of objects. CVPR, 2021.
  12. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. 2023.
  13. Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. In ICCV, 2019.
  14. 3d affordancenet: A benchmark for visual object affordance understanding. In CVPR, 2021.
  15. A. R. et al. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  16. Articulated objects in free-form hand interaction. CVPR, 2023.
  17. Three-dimensional reconstruction of human interactions. In CVPR, 2020.
  18. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 1981.
  19. W. Gao and R. Tedrake. kpam-sc: Generalizable manipulation planning using keypoint affordance and shape completion. In ICRA, 2021.
  20. Advanced fibonacci sequence with golden ratio. IJSER, 2014.
  21. J. J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979.
  22. Generative adversarial nets. In NeurIPS, 2014.
  23. ContactOpt: Optimizing contact to improve grasps. In CVPR, 2021.
  24. Honnotate: A method for 3d annotation of hand and object poses. CVPR, 2020.
  25. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3d pose estimation. CVPR, 2022.
  26. S. Han and H. Joo. Learning canonicalized 3d human-object spatial relations from unbounded synthesized images. In ICCV, 2023.
  27. Gancraft: Unsupervised 3d neural rendering of minecraft worlds. In ICCV, 2021.
  28. Resolving 3d human pose ambiguities with 3d scene constraints. In ICCV, 2019.
  29. Populating 3d scenes by learning human-scene interaction. In CVPR, 2021.
  30. Learning joint reconstruction of hands and manipulated objects. In CVPR, 2019.
  31. Is synthetic data from generative models ready for image recognition? arXiv preprint arXiv:2210.07574, 2022.
  32. Affordance prediction via learned object attributes. In Proc. Intl. Conf. on Robotics and Automation, 2011.
  33. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
  34. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.11239, 2020.
  35. Capturing and inferring dense full-body human-scene contact. In CVPR, 2022a.
  36. Capturing and inferring dense full-body human-scene contact. In CVPR, 2022b.
  37. InterCap: Joint markerless 3D tracking of humans and objects in interaction. In GCPR, 2022c.
  38. Generative models as a data source for multiview representation learning. arXiv preprint arXiv:2106.05258, 2021.
  39. Affordpose: A large-scale dataset of hand-object interactions with affordance-driven hand pose. In ICCV, 2023.
  40. Coherent reconstruction of multiple humans from a single image. In CVPR, 2020.
  41. Neuralhofusion: Neural volumetric rendering under human-object interactions. In CVPR, 2022.
  42. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
  43. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  44. Pointrend: Image segmentation as rendering. In CVPR, 2020.
  45. Segment anything. In ICCV, 2023.
  46. Putting people in their place: Affordance-aware human insertion into scenes. In CVPR, 2023.
  47. Y. J. Lee and K. Grauman. Predicting important objects for egocentric video summarization. IJCV, 2015. URL https://api.semanticscholar.org/CorpusID:5617021.
  48. S. Levine and D. Shah. Learning robotic navigation from experience: principles, methods and recent results. Philosophical Transactions of the Royal Society B: Biological Sciences, 2022.
  49. Locate: Localize and transfer object parts for weakly supervised affordance grounding. In CVPR, 2023.
  50. Estimating 3d motion and forces of person-object interactions from monocular video. In CVPR, 2019.
  51. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
  52. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
  53. SMPL: A skinned multi-person linear model. Proc. ACM SIGGRAPH Asia, 2015.
  54. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
  55. Cad-estate: Large-scale cad model annotation in rgb videos. In ICCV, 2023.
  56. Generative interventions for causal learning. In CVPR, 2021.
  57. M. McCool and E. Fiume. Hierarchical poisson disk sampling distributions. In Graphics interface, 1992.
  58. Finding an unsupervised image segmenter in each of your deep generative models. arXiv preprint arXiv:2105.08127, 2021.
  59. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
  60. COAP: Compositional articulated occupancy of people. In CVPR, 2022.
  61. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794, 2022.
  62. imapper: interaction-guided scene mapping from monocular videos. ACM TOG, 2019.
  63. Accurate 3d hand pose estimation for whole-body 3d human mesh estimation. In CVPRW, 2022.
  64. T. Nagarajan and K. Grauman. Learning affordance landscapes for interaction exploration in 3d environments. In NeurIPS, 2020.
  65. Signed distance fields: A natural representation for both mapping and planning. In RSS 2016 workshop, 2016.
  66. OpenAI. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/chatgpt/, 2023a.
  67. OpenAI. Gpt-4v(ision) system card. https://openai.com/research/gpt-4v-system-card, 2023b.
  68. Do 2d gans know 3d shape? unsupervised 3d shape reconstruction from 2d image gans. arXiv preprint arXiv:2011.00844, 2020.
  69. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
  70. Gan-supervised dense visual alignment. In CVPR, 2022.
  71. Object pop-up: Can we infer 3d objects and their poses from human interactions alone? In CVPR, 2023.
  72. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
  73. Contact and human dynamics from monocular video. In ECCV, 2020.
  74. Texture: Text-guided texturing of 3d shapes. In Proc. ACM SIGGRAPH, 2023.
  75. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  76. Embodied hands: Modeling and capturing hands and bodies together. Proc. ACM SIGGRAPH Asia, 2017.
  77. Pigraphs: learning interaction snapshots from observations. ACM TOG, 2016.
  78. SG_161222. Realistic vision v5.1. https://civitai.com/models/4201?modelVersionId=130090, 2023.
  79. C. E. Shannon. A mathematical theory of communication. ACM SIGMOBILE, 2001.
  80. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NeurIPS, 2021.
  81. Sketchfab. Sketchfab. https://sketchfab.com/, 2023.
  82. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
  83. Denoising diffusion implicit models. arXiv:2010.02502, 2020. URL https://arxiv.org/abs/2010.02502.
  84. Neural free-viewpoint performance rendering under complex human-object interactions. In ACM MM, 2021.
  85. Color indexing. IJCV, 1991.
  86. Grab: A dataset of whole-body human grasping of objects. In ECCV, 2020.
  87. F. H. K. d. S. Tanaka and C. Aranha. Data augmentation using gans. arXiv preprint arXiv:1904.09135, 2019.
  88. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944, 2023.
  89. DECO: Dense estimation of 3D human-scene contact in the wild. In ICCV, 2023.
  90. Repurposing gans for one-shot semantic part segmentation. In CVPR, 2021.
  91. Datasetdm: Synthesizing data with perception annotations using diffusion models. arXiv preprint arXiv:2308.06160, 2023.
  92. Chore: Contact, human and object reconstruction from a single rgb image. In ECCV, 2022.
  93. Automated labeling for robotic autonomous navigation through multi-sensory semi-supervised learning on big data. IEEE TBD, 2021.
  94. InterDiff: Generating 3d human-object interactions with physics-informed diffusion. In ICCV, 2023.
  95. CPF: Learning a contact potential field to model the hand-object interaction. In ICCV, 2021.
  96. Grounding 3d object affordance from 2d interactions in images. In ICCV, 2023.
  97. Affordance diffusion: Synthesizing hand-object interactions. In CVPR, 2023.
  98. Human-aware object placement for visual environment reconstruction. In CVPR, 2022.
  99. Perceiving 3d human-object spatial arrangements from a single image in the wild. In ECCV, 2020a.
  100. Place: Proximity learning of articulation and contact in 3d environments. In 3DV, 2020b.
  101. Couch: Towards controllable human-chair interactions. In ECCV, 2022.
  102. Image gans meet differentiable rendering for inverse graphics and interpretable 3d neural rendering. arXiv preprint arXiv:2010.09125, 2020c.
  103. Generating 3d people in scenes without people. In CVPR, 2020d.
  104. Datasetgan: Efficient labeled data factory with minimal human effort. In CVPR, 2021.
  105. Toch: Spatio-temporal object correspondence to hand for motion refinement. In ECCV, 2022.
  106. Stereo magnification: Learning view synthesis using multiplane images. In Proc. ACM SIGGRAPH, 2018.
  107. Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. ICCV, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Hyeonwoo Kim (13 papers)
  2. Sookwan Han (4 papers)
  3. Patrick Kwon (7 papers)
  4. Hanbyul Joo (37 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now