Real2Code: Reconstruct Articulated Objects via Code Generation (2406.08474v2)

Abstract: We present Real2Code, a novel approach to reconstructing articulated objects via code generation. Given visual observations of an object, we first reconstruct its part geometry using an image segmentation model and a shape completion model. We then represent the object parts with oriented bounding boxes, which are input to a fine-tuned LLM to predict joint articulation as code. By leveraging pre-trained vision and LLMs, our approach scales elegantly with the number of articulated parts, and generalizes from synthetic training data to real world objects in unstructured environments. Experimental results demonstrate that Real2Code significantly outperforms previous state-of-the-art in reconstruction accuracy, and is the first approach to extrapolate beyond objects' structural complexity in the training set, and reconstructs objects with up to 10 articulated parts. When incorporated with a stereo reconstruction model, Real2Code also generalizes to real world objects from a handful of multi-view RGB images, without the need for depth or camera information.

• The paper introduces a novel two-tier method combining part geometry reconstruction with joint articulation prediction via code generation.
• It leverages vision models like SAM and a fine-tuned CodeLlama to enhance accuracy over traditional 3D reconstruction methods.
• Empirical tests on PartNet-Mobility reveal significant improvements in handling complex articulated structures for VR/AR applications.

Real2Code: Reconstructing Articulated Objects via Code Generation

The paper "Real2Code: Reconstruct Articulated Objects via Code Generation" introduces a novel method to reconstruct articulated objects through the innovative use of code generation. The approach leverages advancements in pre-trained vision and LLMs to enhance the reconstruction process of objects with complex articulation structures. This paper provides a two-tiered solution: part geometry reconstruction followed by joint articulation prediction using a LLM fine-tuned for code generation.

Methodology and Technical Innovations

The methodology of Real2Code is divided into two main components: the reconstruction of object parts' geometry and the prediction of joint articulation through code generation. Below is a detailed overview of the two main steps:

1. Part Geometry Reconstruction:
   • A combination of a fine-tuned Segmentation and a 3D Shape Completion Model is employed.
   • The paper uses a pre-trained Segment Anything Model (SAM) for 2D segmentation, refined to focus on kinematic structures. This adaptation leverages large-scale datasets for better generalization to real-world objects, overcoming the common limitations seen in synthetic training datasets.
   • For objects in unstructured environments with solely RGB images available, the Dense Unified Stereo Reconstruction (DUSt3R) model is incorporated to achieve multi-view 3D segmentation from pose-free RGB images.
2. Joint Articulation Prediction via Code Generation:
   • This step translates the problem of joint prediction into a code generation task using a fine-tuned LLM, specifically CodeLlama.
   • A significant innovation lies in the use of Oriented Bounding Boxes (OBBs) to abstract raw sensory data. OBBs simplify the numerical complexity, facilitating more accurate joint prediction as a classification task via the LLM.
   • The fine-tuned model generates code outputs that can directly be executed in simulation environments, thus bypassing the need for manual cleanups often required in traditional methods.

Experimental Results and Comparative Analysis

Empirical evaluations conducted on the well-established PartNet-Mobility dataset demonstrate substantial performance improvements. Key findings include:

• Reconstruction Quality:
  • Real2Code consistently outperforms state-of-the-art methods including PARIS and Ditto in terms of reconstruction accuracy, evidencing adeptness in handling up to ten articulated parts in objects.
  • Direct comparisons showcase a notable enhancement in both parts' reconstruction accuracy and complete object reconstruction, validating the effectiveness of the two-tiered approach in Real2Code.
• Generalization Capability:
  • Unlike traditional methods, Real2Code's reliance on pre-trained models and abstracted input (OBBs) allows it to generalize beyond the structural complexities present in training data, successfully extending its applicability to unseen, complex objects.

Practical Implications and Future Directions

Real2Code significantly advances the domain of articulated object reconstruction with several practical and theoretical benefits:

• Practical Implications:
  • The method facilitates the digital replication of real-world articulated objects for VR/AR applications, enabling more accurate simulations for robotics and embodied agents.
  • The approach can automate asset creation for interactive simulations, reducing the manual effort in animating and segmenting complex objects.
• Theoretical Contributions:
  • By reframing joint prediction as a code generation task, the methodology exemplifies a novel intersection of computer vision and natural language processing, opening new avenues for cross-disciplinary innovations.
  • The effective use of large-scale pretrained models underscores the importance of leveraging extensive, generalized datasets to overcome domain-specific limitations.

Conclusion

Real2Code advances the field of articulated object reconstruction through its innovative use of code generation models and robust part geometry reconstruction techniques. This paper effectively bridges gaps between visual observations and simulation-ready object models, setting a new benchmark in handling objects with intricate kinematic structures. Future work may explore extending the method to multi-object scenes or integrating more complex joint parameter estimations, potentially involving iterative interactions, to broaden the scope and utility of this approach in real-world applications.