PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding

Abstract: Recent advances in 3D multimodal LLMs (3D-MLLMs) have enabled unified solutions for 3D scene understanding tasks, including visual question answering, captioning, and referring segmentation. However, existing 3D-MLLMs remain largely object-centric, limiting their ability to model fine-grained part structures that are essential for embodied interaction with 3D environments. In this work, we present PAR3D, a unified part-aware 3D-MLLM framework that enables models to understand, reason about, and ground both objects and their parts in 3D scenes. To enable training and evaluation of part-aware 3D scene understanding, we introduce ScenePart, a synthetic 3D scene dataset with part-level annotations and language instructions. We further develop Part-Aware 3D Representation Learning to enrich 3D visual representations with fine-grained part-level semantics, and propose Hierarchical Segmentation Query Generation to ground part targets via hierarchical object-part queries. Extensive experiments show that our method substantially improves part-level question answering and referring segmentation, while also achieving strong performance across object-level vision-language tasks.

• The paper introduces a unified part-aware 3D-MLLM that integrates the synthetic ScenePart dataset to address fine-grained part-level supervision in 3D scene understanding.
• It employs part-aware contrastive loss and representation-preserving self-distillation to achieve robust intra-part feature compactness and inter-part dispersion.
• Experimental results show significant performance gains in object and part segmentation tasks, advancing 3D VQA, captioning, and referential segmentation benchmarks.

PAR3D: Unified Part-Aware 3D-MLLM for Scene Understanding

Motivation and Problem Statement

The development of 3D multimodal LLMs (3D-MLLMs) has provided a foundation for unified solutions in 3D scene understanding, supporting tasks such as visual question answering (VQA), captioning, and referring segmentation. However, the prevailing object-centric paradigm in current 3D-MLLMs limits their effectiveness in modeling fine-grained part structures necessary for embodied interaction, affordance reasoning, and controllable manipulation of 3D environments.

Most existing datasets and models focus exclusively on object-level annotations and representations. This architectural and data bias results in models that fail to localize, reason about, and interact with object parts—functional subcomponents such as the handle of a mug, seat of a chair, or drawer of a cabinet—which are essential for both embodied agents and scene-level manipulations.

ScenePart: A Synthetic Part-Aware 3D Scene Dataset

To mitigate the lack of part-level supervision in 3D vision-language data, the paper introduces ScenePart, a synthetic 3D scene dataset explicitly designed for part-aware tasks. ScenePart synthesizes realistic indoor scenes by composing part-annotated 3D objects, providing both object-level and part-level mask annotations, as well as language instructions contextualized for scene-level understanding. ScenePart supports comprehensive multi-task and multi-granularity supervision by offering dense annotations and language-task pairs, enabling the training and evaluation of 3D-MLLMs on both object and part understanding.

Figure 1: PAR3D architecture and illustration of the ScenePart dataset, which provides fine-grained object-part annotations and instructions for 3D scene understanding.

Figure 2: The ScenePart data construction pipeline, integrating part-annotated objects into synthetic indoor layouts with full scene and language supervision.

PAR3D Framework: Methodological Advances

PAR3D addresses the core limitations of previous work by proposing three principal components:

1. ScenePart Dataset Integration: To inject part-level supervision into the training dynamics.
2. Part-Aware 3D Representation Learning: Enhancing the representation power of the 3D backbone to discriminate and encode part-level geometric and semantic features.
3. Hierarchical Segmentation Query Generation: Introducing granularity-aware grounding tokens that disentangle object- and part-level queries, mitigating the granularity conflicts inherent in earlier formulations.

The unified framework is built upon a backbone similar to 3D-LLaVA but augments it with targeted learning objectives and architectural modifications for part-awareness.

Part-Aware Representation Learning

The 3D encoder utilizes a pretrained Point Transformer as the visual backbone, integrating two additional regularization objectives during training:

• Part-aware Contrastive Loss: Enforces intra-part feature compactness and inter-part dispersion at the superpoint level, leveraging ScenePart's dense part annotations.
• Representation-Preserving Self-Distillation: Maintains semantic fidelity between decoder and frozen encoder features, preventing excessive task drift during adaptation.

This dual-objective mechanism ensures robustness and fine-grained discriminative capacity in the learned 3D representations.

Hierarchical Segmentation Query Generation

Instead of a single segmentation token, the model employs hierarchical grounding tokens (i.e., $[\mathrm{OBJ}]$ for objects, $[\mathrm{PART}]$ for parts), enabling explicit structural coupling between object and part queries. This mechanism supports hierarchical mask prediction and unifies object- and part-level grounding without cross-granularity interference.

Figure 3: The two-stage training pipeline and hierarchical grounding token design of PAR3D.

Integrated Training Strategy

Training proceeds in two stages:

1. Part-Aware Backbone Pretraining: Adaptation with object- and part-level supervision; freezing the main encoder while training the decoder with contrastive and representation-preserving objectives.
2. Instruction Tuning: LoRA-based finetuning of the LLM and output heads on joint object- and part-level language-vision instructions, with grounding supervision across granularities.

Experimental Results

PAR3D is evaluated on both canonical object-level benchmarks (ScanRefer, Multi3DRefer, ScanQA, SQA3D, Scan2Cap) and new part-level tasks constructed in ScenePart (ScenePart-Seg, ScenePart-QA). Across these benchmarks, PAR3D demonstrates:

• Significant improvements in part-aware referring segmentation and VQA, surpassing state-of-the-art and data-augmented baselines in mIoU, [email protected], CIDEr, BLEU, METEOR, and ROUGE metrics.
• Consistently superior object-level performance, indicating that enriching part-aware capabilities does not degrade, and often enhances, traditional 3D scene understanding.

Notably, incorporation of hierarchical token-based grounding and contrastive representation learning yields cumulative performance benefits in both ablation and benchmark studies.

Figure 4: Representative qualitative examples of part-aware referring segmentation and VQA on real and synthetic scenes.

Figure 5: Comparative VQA results highlighting improved answer quality by PAR3D on diverse datasets.

Figure 6: Comparative segmentation results on various benchmarks; PAR3D produces more accurate object and part masks than 3D-LLaVA.

Implications and Future Directions

PAR3D demonstrates that integrating synthetic, structured part-level data with matched architectural modifications yields substantial gains in fine-grained 3D scene understanding. The hierarchical segmentation query paradigm and part-aware representation learning offer a scalable template for future unified 3D-MLLMs, especially for scenarios involving complex spatial manipulations, affordance reasoning, and sub-object interaction.

Practical implications include:

• Robustness for embodied agents in robotics and AR, enabling precise manipulation at the part level.
• Improved scene editing and content creation pipelines via part-level referential interfaces.
• A bridge toward open-vocabulary, fine-grained 3D visual reasoning in real-world and synthetic domains.

The domain gap between synthetic data and real-world scenes persists as a primary limitation. Expanding annotation schemes to encompass real-scene part-level groundings and extending open-set generalization to unrestricted part categories remain open challenges.

Conclusion

PAR3D establishes a new standard for unified 3D-MLLMs by explicitly modeling and grounding both objects and their constituent parts. Advancements in data, representation learning, and hierarchical grounding are empirically validated to yield robust improvements across part-aware and object-centric tasks in 3D scene perception and multimodal reasoning (2606.06485). This work represents a key step toward fine-grained embodied intelligence and opens avenues for broader generalization and deployment in complex 3D environments.