DreamPartGen: Part-Aware Text-to-3D

• DreamPartGen is a part-aware text-to-3D framework that uses Duplex Part Latents and Relational Semantic Latents to model both geometry and appearance.
• It employs a synchronized co-denoising process to align intra-part features and inter-part dependencies, ensuring coherent assembly and text alignment.
• Evaluated on PartRel3D and standard benchmarks, it significantly improves geometric fidelity, text-shape alignment, and part disentanglement over previous methods.

Searching arXiv for DreamPartGen and closely related part-aware 3D generation papers. DreamPartGen is a framework for semantically grounded, part-aware text-to-3D generation that treats 3D objects as compositions of meaningful parts rather than monolithic wholes. It introduces Duplex Part Latents (DPLs) to jointly model each part’s geometry and appearance, and Relational Semantic Latents (RSLs) to encode inter-part dependencies derived from language. Its central mechanism is a synchronized co-denoising process in which geometry, appearance, and relational semantics are updated collaboratively so that generated parts remain coherent, interpretable, and text-aligned throughout synthesis (Yu et al., 19 Mar 2026). Within the recent literature on part-aware 3D generation, DreamPartGen is situated against geometry-centered decomposition pipelines such as PartGen (Chen et al., 2024), latent part-manipulation autoencoders such as LPMNet (Öngün et al., 2020), and explicit or hybrid full-resolution part generators such as the latent neural voxel field method of 2024 (Huang et al., 2024) and FullPart (Ding et al., 30 Oct 2025).

1. Conceptual position and problem setting

DreamPartGen is motivated by a limitation of standard text-to-3D generation: many methods treat an object as a single latent entity, thereby entangling geometry, appearance, and semantics, and providing no explicit representation of parts or their relationships (Yu et al., 19 Mar 2026). The paper places this limitation in the context of earlier text-to-3D systems such as DreamFusion, ProlificDreamer, and LucidDreamer, and argues that language is often used only as a one-shot global condition rather than as a persistent source of compositional structure (Yu et al., 19 Mar 2026).

The framework also positions itself relative to recent part-aware methods. According to the paper, approaches such as Part123, PartGen, and HoloPart add decomposition but remain largely geometry-focused: they emphasize segmentations or boxes, treat parts as geometrically isolated, and do not model how parts relate functionally or spatially, nor how text aligns with parts and their assembly (Yu et al., 19 Mar 2026). This contrasts with PartGen’s staged pipeline, where multi-view diffusion models first produce plausible and view-consistent part segmentations and then complete occluded part views before 3D reconstruction (Chen et al., 2024). PartGen therefore provides explicit part decomposition and reconstruction, but DreamPartGen’s stated objective is different: persistent language grounding at the level of parts and inter-part relations (Yu et al., 19 Mar 2026).

A further contrast arises with LPMNet, which performs latent modification and generation of 3D point cloud models with respect to semantic parts through a single end-to-end autoencoder, enabling part exchange, interpolation, composition, and global modification directly in latent space (Öngün et al., 2020). LPMNet is part-aware and supports semantic editing, but its mechanism is centered on latent manipulation of reconstructed point clouds rather than language-grounded collaborative diffusion. This suggests that DreamPartGen extends the part-aware agenda from latent editing and decomposition toward explicit semantic and relational conditioning.

2. Representational core: DPLs and RSLs

DreamPartGen is described as a language-grounded, collaborative diffusion framework whose core abstractions are DPLs and RSLs (Yu et al., 19 Mar 2026). For an object with $N$ parts $O = \{p_i\}_{i=1}^N$, each part receives a duplex representation containing both geometry and appearance streams.

The Duplex Part Latents consist of three elements. First, each part has 3D tokens from a 3D VAE on mesh point samples, $\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$, which encode geometry. Second, each part has 2D tokens from an image-VAE with renders, $\mathbf{L}^{2D}_i \in \mathbb{R}^{T_{2D} \times d}$, which encode appearance. Third, each part is assigned a part-identity embedding $e_i \in \mathbb{R}^d$, a learnable vector used to persistently track part slots across diffusion steps (Yu et al., 19 Mar 2026). The paper attributes several properties to this construction: permutation robustness, because slot IDs make latent order irrelevant; slot persistence, because part swapping through denoising is prevented; and modularity, because shared parts such as a “leg” or “wheel” can transfer across objects and categories (Yu et al., 19 Mar 2026).

The Relational Semantic Latents provide the semantic counterpart. DreamPartGen distinguishes two forms. Global relational tokens $\mathbf{S}^{glb}$ are derived from canonicalized triplets such as $(seat, legs, above)$ or $(handle, body, attached\text{-}to)$, and encode explicit part-part functional and spatial relations (Yu et al., 19 Mar 2026). Each such token $\mathbf{s}^{glb}_{ij,\rho} \in \mathbb{R}^d$ acts as a persistent planner or constraint throughout denoising. Local semantic tokens $\mathbf{S}^{loc}$ encode part-level attributes from fine-grained descriptions such as “metal blade” or “wooden handle,” and are diffused jointly with DPLs to refine per-part details (Yu et al., 19 Mar 2026).

A key distinction is persistence. The paper states that global tokens persist through denoising, providing structural planning cues for assembly, while local tokens co-evolve with DPLs, acting as controllers for detail, material, and appearance refinement (Yu et al., 19 Mar 2026). This differs from conditioning schemes in which text is supplied only at input time. A plausible implication is that DreamPartGen treats language not merely as an external prompt but as a recurrent latent structure that remains active throughout generation.

3. Synchronized co-denoising and collaborative diffusion

DreamPartGen’s defining algorithmic contribution is the synchronized co-denoising process, which enforces mutual consistency among geometry, appearance, and relational semantics (Yu et al., 19 Mar 2026). At each diffusion step $O = \{p_i\}_{i=1}^N$0, the model alternates between intra-part synchronization and inter-part synchronization.

During intra-part synchronization, geometry and appearance are aligned within each part by cross-attention between the 3D and 2D DPL streams:

$O = \{p_i\}_{i=1}^N$1

Noised local semantic tokens $O = \{p_i\}_{i=1}^N$2 are then injected to refine part details:

$O = \{p_i\}_{i=1}^N$3

During inter-part synchronization, all parts communicate so that assembly remains plausible in context:

$O = \{p_i\}_{i=1}^N$4

This cross-part interaction is further guided by the global relational tokens:

$O = \{p_i\}_{i=1}^N$5

The relational tokens are themselves updated bottom-up from pooled part latents:

$O = \{p_i\}_{i=1}^N$6

The paper characterizes this as an alternating, persistent interaction across geometry, appearance, and relational semantics, ensuring that each part is shaped and textured in context while honoring linguistic instructions and functional constraints (Yu et al., 19 Mar 2026). This mechanism differs structurally from PartGen’s staged reconstruction pipeline, in which segmentation, completion, and reconstruction are separate operations (Chen et al., 2024), and from LPMNet’s direct editing of latent part vectors within an autoencoder (Öngün et al., 2020). DreamPartGen instead makes semantic coordination endogenous to the denoising dynamics.

4. Architecture, objectives, and generation procedure

The backbone of DreamPartGen is a Transformer-based DiT architecture with cross-attention for modal and part reasoning (Yu et al., 19 Mar 2026). Geometry and rendered appearance of each part are encoded by 3D/2D VAEs, while a frozen text encoder—for example, Gemma—followed by a learned projection is used for RSL extraction (Yu et al., 19 Mar 2026). The overall system thus combines tokenized geometric latents, tokenized appearance latents, and text-derived semantic latents within a single synchronized diffusion framework.

The training objective is the joint denoising of 3D and 2D DPLs under semantic constraints from the RSLs. For diffusion step $O = \{p_i\}_{i=1}^N$7, the 3D and 2D denoising losses are

$O = \{p_i\}_{i=1}^N$8

and

$O = \{p_i\}_{i=1}^N$9

The total loss uses an SNR-based curriculum:

$\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$0

with

$\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$1

Training proceeds in two phases. Phase 1 pretrains DPLs with semantic synchronization using part-level and relational signals. Phase 2 performs joint fine-tuning with both intra- and inter-part synchronization, while the curriculum progressively emphasizes relational alignment over early-phase geometric denoising (Yu et al., 19 Mar 2026).

At inference time, the prompt is parsed into global triplets and local attributes. Initial latent samples $\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$2 and $\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$3 are drawn from noise, then jointly denoised while conditioning on persistent global semantics. Finally, the 3D VAE decoder reconstructs meshes, and assembly is guided by decoded DPL correspondence (Yu et al., 19 Mar 2026). This end-to-end text-to-part-to-object procedure differs from approaches that begin from rendered multi-view images, as in PartGen (Chen et al., 2024), or from part layouts followed by per-part voxel synthesis, as in FullPart (Ding et al., 30 Oct 2025).

5. Benchmarks, datasets, and reported performance

DreamPartGen is evaluated on a new dataset, PartRel3D, introduced in the same work. The dataset contains 11K objects × 175 categories × 90K parts × 300K canonicalized triplets (Yu et al., 19 Mar 2026). Each object has 8.2 parts and 27 inter-part relations (on average), and the triplets cover diverse, assembly-relevant functional predicates such as “support,” “attach,” and “hinge,” as well as spatial predicates such as “above,” “symmetric-with,” and “touching” (Yu et al., 19 Mar 2026). Additional benchmarks listed by the paper are Objaverse, ShapeNet, ABO, Partverse (Yu et al., 19 Mar 2026).

The evaluation protocol spans three categories of metrics. Geometric Fidelity is assessed with Chamfer Distance (CD), Earth Mover’s Distance (EMD), F-Score, Render-FID/KID, PointNet++ FID/KID. Text-Shape Alignment is measured using CLIP (N-T, I-T) and ULIP-T. Structure/Part Independence is assessed by the average pairwise IoU among parts, with lower values indicating better part disentanglement (Yu et al., 19 Mar 2026).

The paper reports state-of-the-art performance in geometric fidelity and text-shape alignment (Yu et al., 19 Mar 2026). Specifically, it claims consistently lowest Chamfer Distance with a reduction of approximately 53%, EMD reduced by approximately 33%, best F-score for per-part geometry, and best render-FID/perceptual metrics across all benchmarks (Yu et al., 19 Mar 2026). For text-shape alignment, object-level improvement is reported as at least 20%, and part-level improvement as at least 35%, with the highest CLIP and ULIP-T scores (Yu et al., 19 Mar 2026). The method is also reported to generalize more strongly to rare parts and held-out relation predicates, with smaller performance drops in out-of-distribution settings than baselines (Yu et al., 19 Mar 2026).

A concise comparison of the evaluation dimensions explicitly stated in the source materials is given below.

Framework	Primary evaluation focus	Example reported outcomes
DreamPartGen	CD, EMD, F-Score, Render-FID/KID, PointNet++ FID/KID, CLIP, ULIP-T, part IoU	lowest Chamfer Distance; EMD down $\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$433%; text-shape alignment improved by at least $\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$520% object-level and $\mathbf{L}^{3D}_i \in \mathbb{R}^{T_{3D} \times d}$635% part-level (Yu et al., 19 Mar 2026)
PartGen	segmentation mAP, CLIP similarity, LPIPS, PSNR, structured reconstruction quality	54.2–59.3 mAP; CLIP 0.974 for completion and 0.936 for 3D with context (Chen et al., 2024)
LPMNet	CD, EMD, MMD, Coverage, JSD, TMD	low reconstruction error across varying bottleneck sizes and object types; part-exchange achieves best similarity-to-real (Öngün et al., 2020)
FullPart	F-Score, Chamfer Dist., Part-CD, ULIP Score	F-Score 0.81; Chamfer Dist. 0.11; Part-CD 0.36; ULIP Score 0.24 (Ding et al., 30 Oct 2025)

The metrics are not directly interchangeable because the frameworks solve related but non-identical problems. PartGen, for example, begins from text, an image, or an unstructured 3D object and focuses on decomposition and reconstruction into meaningful parts (Chen et al., 2024), while DreamPartGen is evaluated specifically as a semantically grounded text-to-3D generator (Yu et al., 19 Mar 2026).

6. Ablations, capabilities, and relation to adjacent methods

The DreamPartGen paper emphasizes several ablation findings. Removing global RSLs causes a massive degradation in part assembly and semantic alignment. Removing local RSLs leads to severe breakdown in part fidelity and alignment. Removing part identity worsens part disentanglement and semantic scores (Yu et al., 19 Mar 2026). These results are used to support the claim that relational semantics, local attributes, and persistent part-slot identity are all integral to the framework rather than auxiliary components.

Qualitatively, DreamPartGen is reported to generate 3D objects with faithful part boundaries, correct topology, and spatially consistent assemblies, and to outperform baselines in cases where other methods produce detached parts, geometric drift, or unstable attachments (Yu et al., 19 Mar 2026). The paper further states that it enables text-driven part editing and mini-scene generation via relational cues encoded as language triplets (Yu et al., 19 Mar 2026). This places it in a broader line of research in which part representations are valued not only for decomposition quality but also for downstream controllability.

Several neighboring approaches illuminate this design space. PartGen uses a two-diffusion pipeline in which one multi-view diffusion model predicts plausible and view-consistent part segmentations and a second fills in occlusions before a 3D reconstruction network produces complete, watertight meshes for each part (Chen et al., 2024). Its strengths are stochastic part decompositions, contextual completion, and the ability to hallucinate invisible parts (Chen et al., 2024). LPMNet instead learns latent part representations within a single autoencoder through two-stage symmetric pooling over point cloud features, enabling part exchange, interpolation, composition, and global modification without part-based training or extra auxiliary losses beyond reconstruction loss (Öngün et al., 2020). Part-aware Shape Generation with Latent 3D Diffusion of Neural Voxel Fields adopts a latent 3D diffusion process over neural voxel fields plus a part-aware shape decoder with cross-attention and self-attention, emphasizing high-resolution part-aware generation and rendering (Huang et al., 2024). FullPart combines implicit layout diffusion with full-resolution explicit voxel grids for each part and introduces center-corner encoding to preserve global coherence across parts of different scales (Ding et al., 30 Oct 2025).

These comparisons clarify DreamPartGen’s distinct emphasis. Its novelty is not merely that objects are decomposed into parts, but that inter-part dependencies derived from language are represented explicitly and participate in the denoising dynamics (Yu et al., 19 Mar 2026). A plausible implication is that DreamPartGen shifts part-aware 3D generation from decomposition-centered or geometry-centered modeling toward relationally conditioned generation in which semantics remain active as persistent planners throughout synthesis.