Don't Mesh with Me: Generating Constructive Solid Geometry Instead of Meshes by Fine-Tuning a Code-Generation LLM (2411.15279v2)

Abstract: While recent advancements in machine learning, such as LLMs, are revolutionizing software development and creative industries, they have had minimal impact on engineers designing mechanical parts, which remains largely a manual process. Existing approaches to generating 3D geometry most commonly use meshes as a 3D representation. While meshes are suitable for assets in video games or animations, they lack sufficient precision and adaptability for mechanical engineering purposes. This paper introduces a novel approach for the generation of 3D geometry that generates surface-based Constructive Solid Geometry (CSG) by leveraging a code-generation LLM. First, we create a dataset of 3D mechanical parts represented as code scripts by converting Boundary Representation geometry (BREP) into CSG-based Python scripts. Second, we create annotations in natural language using GPT-4. The resulting dataset is used to fine-tune a code-generation LLM. The fine-tuned LLM can complete geometries based on positional input and natural language in a plausible way, demonstrating geometric understanding.

• The paper introduces a novel method to generate Constructive Solid Geometry (CSG) for 3D models using a fine-tuned code-generation large language model, addressing limitations of mesh-based approaches.
• A key contribution is the development of a dataset creation pipeline that converts BREP geometries into CSG-based Python scripts, enhanced with GPT-4 generated natural language descriptions for training.
• The fine-tuned LLM demonstrates the ability to complete 3D geometries based on partial inputs and text descriptions, showing promise for AI-driven CAD design with intuitive text interfaces.

The paper "Don't Mesh with Me: Generating Constructive Solid Geometry Instead of Meshes by Fine-Tuning a Code-Generation LLM" proposes an innovative methodology for the generation of 3D geometries using a novel approach centered around Constructive Solid Geometry (CSG), rather than traditional mesh-based methods. This initiative is driven by the need for higher precision and modifiability in the design of mechanical parts, areas where mesh representations often fall short due to their approximative nature.

Key Contributions:

1. Dataset Creation Pipeline: The work introduces a sophisticated pipeline for the conversion of Boundary Representation (BREP) geometries into CSG-based Python scripts. This conversion facilitates the training of a code-generation LLM on data that is inherently compatible with any modern CAD software, broadening the applicability and flexibility of the approach.
2. Natural Language Annotation: The authors employ GPT-4 to generate natural language descriptions of the 3D geometries, thus enriching the dataset. These descriptions serve as textual annotations that the LLM uses to enhance its understanding and control over the geometry generation process.
3. Fine-Tuning a Code-Generation LLM: A code-generation LLM is fine-tuned using the curated dataset. This LLM demonstrates the capability to complete 3D geometries based on partial input, incorporating both the geometric and text-based instructions, which suggests an understanding of geometric relationships and descriptors.

Methodology and Evaluation:

The methodology involves several steps, starting with the decomposition of 3D models into sequences of cells using the GEOUNED tool, which facilitates the creation of CSG from BREP representations. By transforming these sequences into structured Python code, the model gains the ability to handle complex geometric tasks through reusable and parameterized operations.

The fine-tuned LLM's performance was evaluated on a modified version of the ABC dataset, tailored to include parts that can be constructed using axis-parallel cylinders and planes. The model's geometry completion and its alignment with textual instructions were assessed quantitatively and qualitatively. Results showed that the model generates syntactically correct and connected geometries in the majority of cases. The inclusion of text descriptions improved the likelihood of the model generating geometries that matched human design intents for simple structures.

Limitations and Future Work:

One significant challenge noted is the complexity of providing accurate text descriptions for intricate geometric shapes, which can impact the model's ability to learn and reproduce complex designs accurately. The model's current limitations in generating geometries with orientations beyond axis-alignment are also acknowledged, suggesting that future work could involve expanding the LLM’s capabilities to include more geometrically diverse datasets and exploring architectures with larger model capacities.

In conclusion, the paper presents a robust framework for generating precise geometrical structures in CAD applications, setting the stage for further enhancing AI-driven design tools that can accommodate natural language inputs. This approach holds promise for reducing the manual labor involved in designing mechanical parts and could potentially democratize access to design tools by allowing users with varying expertise to create detailed components using intuitive, text-based interfaces.