3D-PLOT-LLM: Part-Level Object Tokens for 3D Large Language Models

Abstract: 3D multimodal LLMs (3D MLLMs) describe a 3D object as a whole but cannot address, name, or reason about its parts. Prior part-aware attempts add segmentation decoders, heavier 3D encoders, or bounding-box grammars at substantial parameter cost. We take a fundamentally different path: we reorganize the input token stream so that parts become directly addressable through the LLM's own vocabulary. Our model, 3D-PLOT-LLM, partitions the frozen point encoder's patches into K locally coherent regions and inserts, before each region's patch tokens, a learnable per-region marker and a reserved vocabulary token <part_k>; a Marker-Space Refinement (MSR) module then conditions each marker on its region's spatial statistics and adjacency neighbors. The model thus cites parts in its output and follows prompts that refer to parts by token, a capability absent from prior object-level 3D MLLMs. To probe this interface, we construct PartVerse-QA, a vocabulary-level part-QA benchmark adapted from PartVerse mesh annotations (77K training pairs and 588 held-out queries on disjoint object splits), on which 3D-PLOT-LLM reaches caption-to-slots Jaccard 0.459 and Exact-match 13.78%, with a slot-to-caption GPT-4o judge of 44.68. On the 3DCoMPaT-GrIn part-aware grounded description benchmark, 3D-PLOT-LLM outperforms PointLLM, Kestrel, PARIS3D, and SegPoint on every text-output metric, and ShapeLLM on 3 of 4, with up to +3.03 GPT-4o judge over PointLLM. On Objaverse whole-object captioning, adding PartVerse-QA at Stage 2 yields +0.65 SBERT and +1.85 GPT-4o over PointLLM, and tops PointLLM-PiSA on 4 of 5 traditional metrics (SBERT, SimCSE, BLEU-1, METEOR) despite targeting a different (part-grounded) objective. All with under 1M new trainable parameters on a frozen point encoder, an order of magnitude below prior part-aware 3D MLLMs, and no segmentation decoder or bounding-box head.

• The paper introduces a novel method for embedding addressable part-level tokens in 3D LLMs by partitioning patch tokens into spatially coherent regions with reserved markers.
• It employs a lightweight Marker-Space Refinement module that refines tokens using region statistics and inter-region adjacency to boost captioning and QA performance.
• Empirical evaluations show significant gains over prior methods with efficient performance, using less than 1M additional trainable parameters.

3D-PLOT-LLM: Addressable Part-Level Object Tokens in 3D LLMs

Motivation and Architectural Innovation

3D multimodal LLMs (MLLMs) have achieved notable success in whole-object 3D understanding via tokenization of point cloud patches aligned to LLMs. However, these models expose a flat sequence of patch tokens and are fundamentally incapable of addressing, naming, or reasoning about object parts at the vocabulary level. Previous efforts toward part-awareness—such as segmentation decoders (e.g., Kestrel [ahmed2025kestrel]) and bounding-box grammars (Part-X-MLLM [wang2025part])—rely on external modules, significantly increase parameter cost, and do not make parts directly addressable within the LLM's token stream.

3D-PLOT-LLM introduces a paradigm shift by reorganizing the point-token input stream such that each geometric region becomes a first-class addressable token in the LLM's vocabulary. The model partitions a frozen point encoder's patch tokens into $K$ spatially coherent regions, then inserts a learnable per-region marker and a reserved vocabulary token <part{k} prior to each patch token sequence. A lightweight Marker-Space Refinement (MSR) module conditions the marker on region spatial statistics and inter-region adjacency. This enables both reading and writing of parts in response to text prompts, a functionality strictly unavailable in prior architectures.

Figure 1: 3D-PLOT-LLM treats each geometric region of a 3D object as a first-class addressable token in the LLM's vocabulary, with patch tokens partitioned into $K$ regions and each region indexed by a reserved token.

Pipeline and Marker-Space Refinement

The 3D-PLOT-LLM pipeline begins with a frozen point encoder generating 512 patch tokens from an 8192-point cloud. These tokens undergo three transformative stages: unsupervised geometric region partitioning (yielding $K$ spatial regions and adjacency statistics), MSR conditioning (refining per-region markers using region-level context and graph message passing), and token assembly (inserting [CLS], <part{k}, markers, and region patch tokens as a structured LLM input).

Figure 2: 3D-PLOT-LLM pipeline from point encoder through region partition, MSR refinement, and token assembly for LLM input.

The MSR module operates via two residual branches: one maps region statistics via an MLP, and the other executes message passing over adjacencies. Residual scaling ensures the module initially acts nearly as identity, enabling effective end-to-end learning of part-token bindings.

Training Protocol and Data Construction

Training follows a two-stage protocol: Stage 1 aligns the model on 660K Cap3D captions of Objaverse objects, freezing the encoder and LLM, and preparing the projector and per-region markers. Stage 2 instruction tunes the full model (including <part{k} tokens in the tokenizer) on PointLLM's 70K complex instructions and 77K PartVerse-QA pairs constructed from mesh-level part annotations, supporting both caption-to-slots (C2S) and slot-to-caption (S2C) part addressing.

Empirical Evaluation across Benchmarks

Objaverse Captioning

On whole-object captioning, 3D-PLOT-LLM outperforms PointLLM under matched-data conditions (up to $+1.48$ GPT-4o judge), confirming architectural superiority. Introducing PartVerse-QA supervision yields further gains ($+0.65$ SBERT, $+1.85$ GPT-4o, with METEOR significance), exceeding PointLLM-PiSA on the majority of standard metrics, despite PiSA's focus on caption augmentation rather than part grounding.

PartVerse-QA Vocabulary-Level Addressing

PartVerse-QA establishes vocabulary-level evaluation for bi-directional part addressing. 3D-PLOT-LLM achieves Jaccard $0.459$ and Exact-match $13.78\%$, representing a $+64\%$ improvement over the strongest non-MSR variant, and demonstrates strong slot-to-caption generation ($44.68$ GPT-4o judge).

Figure 3: Qualitative examples of part-aware tokenization, showing exact mapping between free-form captions and slot sets.

A permutation probe reveals substantial collapse in C2S Jaccard and Exact-match when <part{k} identifiers are randomly shuffled at inference, confirming the learned token-region binding is substantive and non-cosmetic.

3DCoMPaT-GrIn and Parameter Efficiency

On the 3DCoMPaT-GrIn grounded description benchmark, 3D-PLOT-LLM achieves superiority over PointLLM, Kestrel, PARIS3D, SegPoint, and ShapeLLM (3/4 metrics), with up to $K$0 GPT-4o judge over PointLLM. These results are attained with under $K$1M new trainable parameters—an order of magnitude lower than segmentation-decoder methods.

Figure 4: Semantic parts per object in aligned PartVerse pool, motivating $K$2 slot budget.

Ablations and Scaling

Component and data-scaling ablations confirm the value of each architectural element: token reorganization alone provides significant gains, while MSR is pivotal for effective marker conditioning. Supervision scaling exhibits near-monotonic improvements in bidirectional part addressing as PartVerse-QA data increases.

Figure 5: PartVerse-QA supervision scaling curves for Caption-to-slots Jaccard and Slot-to-caption GPT-4o judge.

Qualitative Examples and Failure Modes

Qualitative analysis demonstrates that, unlike flat-token models, 3D-PLOT-LLM remains anchored in correct object class and recovers fine-grained multi-part attributes. Failure modes include symmetric part ambiguity, benchmark-induced degradation, and encoder-level limitations in 2D-surface encoding.

Figure 6: Failure modes visualized across all benchmarks, highlighting causes from query ambiguity to encoder limitations.

Implications and Future Directions

This work establishes vocabulary-level part addressing as a viable and efficient mechanism for part-aware 3D MLLMs. By reframing part segmentation as a tokenization problem and decoupling it from dense-decoder architectures, the paper demonstrates that part-grounded supervision can complement traditional captioning objectives. Further directions include variable-$K$3 or hierarchical slot partitioning for objects with complex part topologies, integration of stronger point encoders, and extension to scene-level part reasoning.

Conclusion

3D-PLOT-LLM redefines object-level part-awareness in 3D LLMs by encoding part geometry as addressable tokens within the vocabulary itself. Empirical benchmarks across captioning, QA, and grounded descriptions demonstrate consistent performance gains and efficiency. The architectural paradigm provides new avenues for lightweight, decoder-free part-aware modeling, and suggests that part-grounded supervision is a potent axis for the scaling of 3D LLMs.