AutoPartGen: Compositional 3D Reconstruction

• AutoPartGen is a generative model for compositional 3D reconstruction that represents an object as a sequence of 3D parts generated autoregressively.
• The method employs an autoregressive diffusion process, conditioning on previously generated parts and optional 2D or 3D evidence to ensure geometric coherence.
• Empirical evaluations highlight strong compositional behavior, enabling seamless part assembly without additional optimization and competitive performance on key 3D metrics.

AutoPartGen is a generative model for compositional $3$D reconstruction that represents an object as a sequence of $3$D parts and generates those parts autoregressively. It accepts an image of an object, $2$D masks of the object’s parts, or an existing $3$D object, and produces a corresponding compositional $3$D reconstruction. The method builds on 3DShape2VecSet, uses a latent space reported to have strong compositional properties, predicts one part at a time while conditioning on previously generated parts and optional $2$D or $3$D evidence, and stops when it decides that all parts have been generated. The resulting parts can be assembled into coherent objects or scenes without additional optimization, and the method is presented as not requiring $3$D part annotations for training or post-hoc fitting during inference (Chen et al., 17 Jul 2025).

1. Conceptual scope

AutoPartGen addresses part-based $3$D generation rather than monolithic whole-shape synthesis. In the formulation given for the model, an object is represented as a union of parts,

$x = \bigcup_{k=1}^{K} x^{(k)},$

with variable $3$0, potentially multiple valid decompositions, and a requirement that the parts remain geometrically consistent when assembled (Chen et al., 17 Jul 2025).

This framing targets several tasks simultaneously. In object-to-parts mode, the input is an existing $3$1D mesh and the output is a decomposition or completion into coherent $3$2D parts. In image-to-parts mode, a single RGB image conditions the generation of a compositional $3$3D object. In masks-to-parts mode, $3$4D part masks provide explicit decomposition control, allowing the same underlying object to be reconstructed under different part granularities (Chen et al., 17 Jul 2025).

A central term in the paper is “part discovery.” In this context, it refers both to the automatic decomposition of an object or image into $3$5D parts when no masks are supplied and to the stochastic variation in decomposition induced by diffusion-based sampling. The model therefore treats the number of parts, their order, and their geometry as generated quantities rather than fixed external annotations (Chen et al., 17 Jul 2025).

2. Latent representation and compositionality

The underlying representation is 3DShape2VecSet. An object surface $3$6 is sampled to produce a point set $3$7, then subsampled to $3$8, and encoded by a transformer-based set encoder $3$9 into a sequence of latent tokens,

$2$0

Decoding uses a Perceiver-style transformer $2$1 that predicts signed distance values,

$2$2

so the zero level set defines the surface and can be meshed with Marching Cubes (Chen et al., 17 Jul 2025).

The AutoPartGen paper emphasizes a specific empirical property of this latent space: compositionality under concatenation. If two shapes $2$3 and $2$4 have latent token sequences $2$5 and $2$6, then concatenating the tokens,

$2$7

produces a decoded surface that is reported to approximate the union $2$8 (Chen et al., 17 Jul 2025).

This compositional behavior is structurally important. It allows part latents to remain in a shared metric coordinate system and makes “assembly” a latent-space operation rather than a downstream optimization problem. It also supplies the mechanism by which previously generated parts are summarized: their decoded surfaces are unioned and re-encoded into a fused latent representation used as context for the next part (Chen et al., 17 Jul 2025).

3. Autoregressive diffusion over parts

AutoPartGen models a distribution over sequences of part latents. For the $2$9-th part latent $3$0, the conditional factorization is

$3$1

where $3$2 is a latent code for the overall object, $3$3 is a fused latent for the union of previously generated parts, and $3$4 denotes optional conditioning evidence such as an image or masked image (Chen et al., 17 Jul 2025).

The fused previous-parts latent is constructed by decoding earlier parts, sampling their union, and re-encoding that union: $3$5 This makes the autoregressive state geometric rather than symbolic (Chen et al., 17 Jul 2025).

Each conditional distribution is implemented as a latent diffusion model. For a clean part latent $3$6, the noisy latent at time $3$7 is

$3$8

and the model predicts the corresponding flow velocity with a DiT-style transformer. The training objective is the mean-squared error between predicted and target velocity fields (Chen et al., 17 Jul 2025).

The diffusion backbone is reported as a DiT with width $3$9 and $3$0 layers. To modulate different conditioning sources, the method uses classifier-free guidance with separate image and geometry guidance weights; the default values reported are $3$1 and $3$2 (Chen et al., 17 Jul 2025).

Generation proceeds until an explicit end-of-trajectory condition is met. In masks-to-parts mode this occurs when all masks have been consumed; in the unconstrained modes the model stops when it predicts an empty-shape or end-of-trajectory token (Chen et al., 17 Jul 2025).

4. Conditioning modes and part assembly

In object-to-parts mode, a complete or partial $3$3D mesh $3$4 is encoded into $3$5. No image tokens are used. The model then samples a sequence of part latents conditioned on $3$6 and the fused latent of previously generated parts (Chen et al., 17 Jul 2025).

In image-to-parts mode, DINOv2 features are extracted from an RGB image, and a separate 3DShape2VecSet-based latent diffusion model first predicts an object latent $3$7. The autoregressive part generator then conditions on both $3$8 and image features to produce a compositional $3$9D reconstruction from a single view (Chen et al., 17 Jul 2025).

In masks-to-parts mode, the conditioning is finer-grained. For each part step $2$0, a masked image is formed as

$2$1

where $2$2 is a binary mask and $2$3 is the RGB image. DINOv2 features are extracted independently from $2$4 and $2$5, then concatenated and projected before being passed to the diffusion transformer. This mechanism is used to align the generated $2$6D part with the specified $2$7D region and to support different user-specified decompositions of the same object (Chen et al., 17 Jul 2025).

Assembly is latent and direct. The paper states that the resulting parts can be “seamlessly assembled into coherent objects or scenes without requiring additional optimization.” Operationally, the full object can be represented by concatenating the part tokens,

$2$8

or by unioning the meshes decoded from individual parts. Because all parts are generated in a shared coordinate frame, no explicit alignment stage is introduced (Chen et al., 17 Jul 2025).

5. Training corpus and empirical evaluation

The training assets are reported as approximately $2$9K $3$0D assets and approximately $3$1M parts, drawn from a larger pool of $3$2M licensed assets in glTF/GLB format. The assets often contain hierarchical mesh structure, and components are merged up to a maximum of $3$3 meshes per asset for tractability (Chen et al., 17 Jul 2025).

For quantitative evaluation, the paper uses PartObjaverse-Tiny for part completion and also shows object-to-parts examples on Google Scanned Objects. The reported metrics are IoU on $3$4 voxel grids, Chamfer distance, and F-score with threshold $3$5, each computed both at the part level and for the union of all predicted parts (Chen et al., 17 Jul 2025).

On the reported $3$6D part completion benchmark, AutoPartGen achieves parts IoU $3$7, parts F-score $3$8, parts Chamfer distance $3$9, whole-object IoU $3$0, whole-object F-score $3$1, and whole-object Chamfer distance $3$2. The same table reports HoloPart at parts IoU $3$3, parts F-score $3$4, parts Chamfer distance $3$5, whole-object IoU $3$6, whole-object F-score $3$7, and whole-object Chamfer distance $3$8, and PartGen at parts IoU $3$9, parts F-score $3$0, parts Chamfer distance $3$1, whole-object IoU $3$2, whole-object F-score $3$3, and whole-object Chamfer distance $3$4 (Chen et al., 17 Jul 2025).

The paper also reports an ablation on autoregression: removing conditioning on previously generated parts in masks-to-parts mode leads to substantially worse geometric coherence, with more overlaps and self-intersections. A second ablation varies $3$5 and $3$6, showing a trade-off between adherence to image or mask evidence and adherence to the learned geometric prior (Chen et al., 17 Jul 2025).

6. Relation to adjacent part-aware research

AutoPartGen occupies one point in a broader technical landscape that includes part discovery, segmentation, assembly, and automotive perception. In unsupervised part discovery, “Unsupervised Part Discovery via Feature Alignment” detects parts without ground-truth part or keypoint annotations and reports $3$7 mAP on VehiclePart (Guo et al., 2020). In $3$8D assembly, “Generative 3D Part Assembly via Dynamic Graph Learning” predicts $3$9-DoF part poses $x = \bigcup_{k=1}^{K} x^{(k)},$0 and, on tables, reports Chamfer distance $x = \bigcup_{k=1}^{K} x^{(k)},$1, Part Accuracy $x = \bigcup_{k=1}^{K} x^{(k)},$2, and Connectivity Accuracy $x = \bigcup_{k=1}^{K} x^{(k)},$3 (Huang et al., 2020).

In vehicle-specific $x = \bigcup_{k=1}^{K} x^{(k)},$4D perception, “Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark” introduces SAV, a prompt-free multi-class vehicle part segmentation framework with a $x = \bigcup_{k=1}^{K} x^{(k)},$5-part ontology and the VehicleSeg10K benchmark containing $x = \bigcup_{k=1}^{K} x^{(k)},$6 images. On VehicleSeg10K, SAV reports mIoU $x = \bigcup_{k=1}^{K} x^{(k)},$7 and mAcc $x = \bigcup_{k=1}^{K} x^{(k)},$8 (Wang et al., 6 Aug 2025). “Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation” introduces a VUS dataset with $x = \bigcup_{k=1}^{K} x^{(k)},$9 real traffic images and $3$00 vehicles in uncommon states, and reports that its data augmentation approach advances baseline $3$01D detection and instance segmentation by over $3$02 (Lu et al., 2020). “A Large-Scale Car Parts (LSCP) Dataset for Lightweight Fine-Grained Detection” contributes $3$03 images across $3$04 car-part categories (Jie et al., 2023).

In automotive inspection and recognition, “Automotive Parts Assessment: Applying Real-time Instance-Segmentation Models to Identify Vehicle Parts” reports a Yolact-based method with mAP $3$05 on DSMLR and SipMask++ with object-detection mAP $3$06 on a workshop dataset (Yusuf et al., 2022). “Domain Adaptation with Joint Learning for Generic, Optical Car Part Recognition and Detection Systems (Go-CaRD)” reports an F1 score of $3$07 for recognition and mAP $3$08 for detection (Stappen et al., 2020). These systems are not generative $3$09D models, but they define the practical perception and dataset infrastructure within which part-based $3$10D generation can be deployed.

The term “AutoPartGen” also appears in a distinct, unrelated sense in “Knowledge-Based Automatic Generation of Partitioned Matrix Expressions,” where it denotes automatic generation of Partitioned Matrix Expressions for linear algebra operations (Fabregat-Traver et al., 2014). That usage concerns symbolic algorithm derivation rather than compositional $3$11D object generation.

7. Limitations and outlook

Several constraints are explicit or implicit in the AutoPartGen formulation. The model relies on artist-provided mesh components as training parts, even though the paper states that no $3$12D part annotations are required; semantically labeled parts are not needed, but mesh-level decomposition structure is still present in the training assets (Chen et al., 17 Jul 2025).

The method is object-centered rather than scene-centered, even though the paper shows scene and city examples by repeated composition. The diffusion model operates in a high-dimensional latent space with a large DiT, so inference is nontrivial even if it is substantially cheaper than full $3$13D diffusion over voxels or radiance fields. The paper also does not introduce explicit geometric regularizers for non-overlap or connectivity, relying instead on global conditioning and autoregressive context to learn coherence (Chen et al., 17 Jul 2025).

These properties suggest a research program rather than a closed endpoint. The work itself positions compositional latent generation as a basis for part-level editing, reverse engineering, and controllable $3$14D reconstruction. A plausible implication is that subsequent systems can combine AutoPartGen-style latent part generation with the structural priors, retrieval modules, and fine-grained perception systems already developed in adjacent research on vehicle parts, articulated states, and industrial part inventories (Chen et al., 17 Jul 2025).