Instance-Structured 3D Tokenization

• Instance-Structured 3D tokenization is defined as the process of decomposing 3D scenes into discrete, semantically coherent tokens that correspond to object-level entities.
• It enables instance-aware manipulation by decoupling local geometry from global context using methods like two-level factorization and adaptive spatial partitioning.
• The approach improves reconstruction fidelity and segmentation efficiency through cross-modal token fusion and semantic ordering, outperforming traditional spatial representations.

Instance-structured 3D tokenization refers to the process of decomposing 3D scenes, objects, or molecular structures into discrete, tokenized representations that directly correspond to meaningful object-level or part-level entities (instances), as opposed to background-agnostic spatial partitions such as fixed grid cells, patches, or primitive clusters. This paradigm is motivated by the need for concise, semantically coherent, and manipulation-friendly codes that fundamentally expose object-level structure as a first-class interface for downstream reconstruction, reasoning, editing, and retrieval. Unlike traditional “compression-centric” spatial codes, instance-structured 3D tokenization aims to make entity-ownership, compositional relationships, and local geometry natively accessible in the latent space.

1. Motivation and Foundational Principles

Instance-structured 3D tokenization is driven by a recognition that object-centric structure is the natural basis for scene understanding and manipulation in 3D data. Conventional feed-forward 3D representations typically output dense, unstructured sets of points or Gaussians, for which object boundaries and instances must be segmented post-hoc. This leads to representations that are inefficient, entangled, and unsuitable for downstream applications requiring object-level control or reasoning (Yoo et al., 28 Jun 2026). In semantic and generative modeling, traditional spatial hierarchies (e.g., octree, kNN, or uniform grids) are token-inefficient and insensitive to the semantic salience or compositional boundaries inherent in the input data (Dutt et al., 18 Mar 2026, Deng et al., 3 Apr 2025, Chen et al., 2024).

Instance-structured tokenization instead factorizes the latent representation into units that explicitly correspond to objects, parts, or functional substructures, with each token or group encapsulating both the identity and associated local geometry or appearance. This approach creates a direct operational interface for instance-aware manipulation, editing, and reasoning across heterogeneous domains, including 3D scene reconstruction, molecular modeling, and CAD assemblies (Yoo et al., 28 Jun 2026, Pei et al., 2024, Chen et al., 11 May 2026).

2. Representative Methodologies and Architectural Patterns

Multiple architectures instantiate instance-structured 3D tokenization, often combining multi-level or multi-modal factorization, assignment mechanisms, and specialized encoders. The primary methodologies are outlined below.

(a) Two-Level Factorization with Grouped Tokens

A canonical example (Yoo et al., 28 Jun 2026) employs a two-stage token factorization:

• Anchor tokens are constructed to encode local 3D geometry and appearance, initialized at 3D patch centers and contextualized via cross-attention with fused multi-view image features.
• Instance/group tokens serve as global identity carriers, being learned as latent vectors that aggregate and condition the anchors through soft-assignment weights:

$$\pi_{k,\ell} = \frac{\exp(\langle A_k, G_\ell \rangle)}{\sum_{\ell'} \exp(\langle A_k, G_{\ell'} \rangle)}$$

Here, each anchor token $A_k$ is bound softly to exactly one instance/group token $G_\ell$.

Object identity (instance tokens) and local appearance/geometry (anchor tokens) are thus decoupled, enabling direct control of instance granularity.

(b) Semantic Token Ordering and Hierarchical Decoding

Level-of-Semantics Tokenization (LoST) encodes 3D shapes into register tokens ordered by semantic salience rather than spatial granularity (Dutt et al., 18 Mar 2026), using ViT-style masked attention to enforce that leading tokens capture most informative, object-level details. Subsequent tokens refine geometric and instance-specific details, supporting progressive and semantically meaningful decoding.

(c) Part- and Component-centric Factorizations

In molecular modeling, the 3D-MolT5 framework uses atom-centric substructure extraction (E3FP) to produce distinct tokens for each atom’s geometric environment (Pei et al., 2024). In CAD assemblies and open-world assets, the C2LT-3D approach (Chen et al., 11 May 2026) factorizes the generative state into (1) canonical local geometry tokens, (2) partition-conditioned context tokens reflecting part/ownership, and (3) explicit relational seam tokens encoding attachment validity between components.

(d) Cross-modal and Multi-view Token Fusion

Instance-level 3D tokens can be constructed by fusing 3D geometry descriptors with multi-view or multi-modal 2D semantic features, as in Inst3D-LMM (Yu et al., 1 Mar 2025) and SAM-Guided tokenization (Chen et al., 2024). These methods leverage per-instance feature pooling, cross-attention, and region-level assignment to ensure robust semantic alignment.

(e) Adaptive Spatial Partitioning

Some methods employ adaptive spatial partitioning with instance granularity determined by surface complexity or semantic boundaries. Examples include quadric-error-based octrees for shape-adaptive tokenization (Deng et al., 3 Apr 2025), and image-region-driven token assignment derived from foundation model masks (Chen et al., 2024).

A summary of key architectural ingredients appears below:

Method	Tokenization Granularity	Assignment Mechanism
(Yoo et al., 28 Jun 2026)	Anchor tokens + instance groups	Cross-attention, soft-assign
(Dutt et al., 18 Mar 2026)	Semantic tokens (“registers”)	Masked ViT, semantic order
(Pei et al., 2024)	Atom-centric substructures	E3FP hashing
(Chen et al., 11 May 2026)	Part, context, seam tokens	Local patch quantization, attachments
(Yu et al., 1 Mar 2025)	Instance tokens	Cross-modal fusion
(Chen et al., 2024)	SAM instance regions	2D→3D mask projection

3. Tokenization Algorithms and Mathematical Formulations

The formal algorithmic details are domain-dependent but follow common steps:

• Instance delineation: Explicit assignment of points, patches, or atoms to discrete object/part regions. In molecular tasks, E3FP hashes atom-centric shells; in scenes, SAM-based or segmentation-driven projections define region membership (Pei et al., 2024, Chen et al., 2024).
• Feature aggregation: Each instance region is encoded by a local network (e.g., MLP, PointNet, foundation encoder), producing an embedding or discrete code for the instance (Yoo et al., 28 Jun 2026, Chen et al., 2024).
• Group/context tokens: Higher-order tokens capture global or relational context (e.g., group/instance tokens in (Yoo et al., 28 Jun 2026), spatial relation tokens in (Yu et al., 1 Mar 2025), partition-context variables in (Chen et al., 11 May 2026)).
• Quantization/assignment: Continuous embeddings may be discretized via vector quantization, hashing, or codebook-based lookup; soft assignment mechanisms connect local (anchor) and global (group or part) tokens (Yoo et al., 28 Jun 2026, Dutt et al., 18 Mar 2026).
• Losses: Training objectives typically combine reconstruction (e.g., differentiable rendering, field estimation), semantic alignment (e.g., RIDA (Dutt et al., 18 Mar 2026)), and task-specific supervision (e.g., mask matching, seam compatibility (Chen et al., 11 May 2026)).

Pseudocode and full formulae are provided in the corresponding primary sources (Yoo et al., 28 Jun 2026, Yu et al., 1 Mar 2025, Pei et al., 2024, Chen et al., 11 May 2026).

4. Evaluation Metrics and Comparative Results

Instance-structured 3D tokenization is evaluated according to:

• Token efficiency: Number of tokens required for high-fidelity reconstruction. Instance-structured methods achieve an order-of-magnitude improvement versus spatial hierarchies (e.g., LoST with 128–512 tokens achieves lower FID than 10^4 LoD-based tokens (Dutt et al., 18 Mar 2026)).
• Instance segmentation and retrieval: Metrics such as AP, mIoU, and CLIP-score for open-vocabulary retrieval. On ScanNet, instance-structured tokenization yields AP₅₀=0.438 (8 views), outperforming Gaussian grouping and ObjectGS baselines (Yoo et al., 28 Jun 2026).
• Reconstruction fidelity: PSNR, SSIM, LPIPS for novel view synthesis, Chamfer Distance or F-score for geometry, and downstream property prediction for molecular tasks (Pei et al., 2024).
• Editing and manipulation: Native support for instance-level removal, insertion, or rigid transformation by direct token/group operation, without mask recomputation or test-time optimization (Yoo et al., 28 Jun 2026, Chen et al., 11 May 2026).

A representative table highlights token efficiency in LoST (Dutt et al., 18 Mar 2026):

#Tokens	CD (↓)	FID (↓)	DINO (↑)
1	2.27	31.65	0.73
16	0.72	26.56	0.81
512	0.23	13.59	0.92

Instance-structured approaches dominate over fixed-grid or LoD-token baselines in both geometric accuracy and semantic interpretability per token budget.

5. Domain-Specific Instantiations

(a) Vision: Scene Reconstruction and Segmentation

Feed-forward frameworks reconstruct scenes into groups of anchor and instance tokens from multi-view images, optimizing for joint instance segmentation and surface fidelity via differentiable rendering (Yoo et al., 28 Jun 2026). SAM-guided projections leverage 2D mask regions for region-aligned 3D grouping (Chen et al., 2024). Multi-view cross-modal fusion injects rich 2D semantics into 3D instance descriptors, followed by spatial-relation encoding (Yu et al., 1 Mar 2025).

(b) Molecular Modeling

Molecule–text foundation models apply atom-centric 3D tokenization by encoding layered atom neighborhood structure into fixed vocabulary indices, aligning with 1D sequence tokens for fine-grained multi-modal understanding (Pei et al., 2024).

(c) CAD and Assembly

Component-conditioned tokens represent local geometry, context, and attachment seams, supporting attachment validation and assembly repair without post-hoc graph inference (Chen et al., 11 May 2026).

(d) Adaptive Representation and Generation

Octree-based adaptive schemes allocate token capacity according to local geometric complexity, yielding instance-level variable-length codes suitable for efficient autoregressive generation (Deng et al., 3 Apr 2025).

6. Structural Reasoning, Manipulability, and Operational Advantages

A defining feature of instance-structured 3D tokenization is the operational interpretability of tokens: object instances, parts, or modules become actionable variables rather than opaque units. This facilitates:

• Direct instance editing: Removal, translation, or composition is performed by manipulating token groups, followed by straightforward decoding (no test-time optimization, no mask recomputation) (Yoo et al., 28 Jun 2026).
• Assembly-level structural intervention: Seam tokens allow attachment queries, parent re-attachment, and constrained decoding—eliminating the need for a separate structure recovery step (Chen et al., 11 May 2026).
• Open-vocabulary and cross-modal retrieval: Semantic embeddings associated with instance tokens make retrieval complexity scale sublinearly with scene or object size, improving efficiency and interpretability (Yoo et al., 28 Jun 2026, Dutt et al., 18 Mar 2026).
• Zero-shot transfer and generalization: Factorization into discrete, object-level units supports robust application to previously unseen assets, categories, or granularities, as evidenced in zero-shot CAD and scene benchmarks (Chen et al., 11 May 2026, Yoo et al., 28 Jun 2026).

7. Limitations and Open Challenges

While instance-structured tokenization affords numerous advantages, several challenges persist:

• Instance boundary ambiguity: SEM-based and segmentation-dependent schemes may be sensitive to mask quality or part definition, especially in cases of occlusion or contact.
• Hierarchical or relational reasoning: Flat instance token sets may inadequately capture parent–child or compositional hierarchies unless complemented by explicit relational tokens.
• Semantic drift in generative modeling: Ensuring that tokens retain operational relevance—e.g., valid attachment in assembly or chemical connectivity in molecules—during autoregressive sampling is nontrivial (Chen et al., 11 May 2026).
• Scalability to highly complex scenes: While object/token scaling is sublinear compared to primitives, extremely dense or cluttered environments may challenge memory and runtime efficiency.

A plausible implication is that future research will continue to augment instance-structured tokenization with richer hierarchical, relational, and semantic priors, as well as more robust multi-modal and open-world alignment.

---

References:

(Yoo et al., 28 Jun 2026, Dutt et al., 18 Mar 2026, Pei et al., 2024, Chen et al., 11 May 2026, Yu et al., 1 Mar 2025, Deng et al., 3 Apr 2025, Chen et al., 2024)