Instance-Structured 3D Token Groups

• The paper demonstrates that grouping tokens by object instances, parts, and local regions enhances 3D representations and downstream tasks.
• It details mechanisms like learned soft grouping, geometry-aware pooling, and instance-guided construction, each balancing precision and efficiency.
• Empirical evaluations in segmentation, pose estimation, and scene reconstruction underscore the practical benefits of structured token grouping.

Searching arXiv for papers on instance-structured and related 3D tokenization to ground the article. Instance-Structured 3D Token Groups are 3D representations in which the token is no longer merely a fixed patch, voxel, or point, but a grouped unit tied to an object, part, region, pose, or component-conditioned local support. Recent work spans explicit object-centric formulations, in which a scene is decomposed into instance tokens and associated local tokens, and weaker formulations, in which tokens are learned soft regional summaries, Gaussian surface groups, superpoint-constrained patches, or adaptive octree cells. The main technical question is therefore not only how many tokens to use, but what a token means: a strict object instance, an unsupervised macro-component, a geometry-aware local region, or a latent semantic group over a 3D support field (Yoo et al., 28 Jun 2026, Xia et al., 28 May 2026, Tshimanga et al., 23 Feb 2026).

1. Definitional scope and boundary cases

The recent literature does not use a single uniform notion of instance structure. In the strongest object-centric sense, the representation itself exposes object-level units as first-class latent variables. "Scenes as Objects, Not Primitives" decomposes a scene into token groups in which each group contains one instance token and multiple anchor tokens, and it treats object instances as the native interface of the 3D representation rather than a product recovered after primitive-level reconstruction (Yoo et al., 28 Jun 2026). "3DVLA" similarly introduces an explicit 3D instance pathway with high-level instance tokens, completion tokens, and geometry-aware aggregation into a pretrained VLA stack (Xia et al., 28 May 2026).

Other papers only partially match that definition. "Token-UNet" learns a fixed small set of soft, semantically organized regional tokens from a 3D encoded feature map, but the paper explicitly states that these tokens are not hard object instances, not fixed voxel patches, and not explicit object proposals (Tshimanga et al., 23 Feb 2026). "GST-VLA" pools anisotropic Gaussian primitives into structured spatial tokens with partial object-centric behavior, yet also states that the tokens are not explicit object slots and not supervised by instance masks (Sarowar et al., 10 Mar 2026). "3D Token Representation Layer" recovers a 3D position for each token and improves viewpoint-agnosticity, but it does not explicitly group tokens into object instances or parts (Shang et al., 2022).

This suggests a practical distinction between four regimes. The first is strict instance-structured tokenization, where one token or token group is explicitly tied to an object instance. The second is part-structured tokenization, where unsupervised or semantically grounded object-internal regions become addressable. The third is region-structured tokenization, where tokens are coherent soft or geometric regions without an explicit instance claim. The fourth is geometry-structured tokenization, where tokens are organized by pose, affine frame, or spatial hierarchy rather than by semantic ownership. A recurring misconception is that any 3D token with geometric content is therefore an instance token; the surveyed papers repeatedly reject that equivalence (Sarowar et al., 10 Mar 2026, Shang et al., 2022).

2. Object-centric and part-centric token hierarchies

The clearest object-centric formulation is the two-level factorization introduced in "Scenes as Objects, Not Primitives." The model uses $K=1024$ anchor tokens and a maximum of $L=100$ group tokens; each group token acts as an instance token, each anchor token carries fine local geometry and appearance, and each anchor is softly assigned to a group by

$$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$

Each anchor then predicts $N_g=32$ Gaussians, and every Gaussian inherits the assignment score of its parent anchor, so instance structure is preserved all the way into rendering and 2D mask supervision (Yoo et al., 28 Jun 2026).

"3DVLA" implements a related but proposal-like hierarchy. It constructs a unified 3D memory from multi-view visual tokens and then decodes high-level state probes

$$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$

with $N_q=300$. After matching and filtering, the model keeps up to 16 confident instance tokens per scene. These tokens are explicitly anchored in 3D, updated by gated multi-view evidence, and later fused with local completion tokens through uncertainty-guided geometric routing (Xia et al., 28 May 2026).

"Inst3D-LMM" is object-centric at the level of language-interface design rather than 3D rendering. It begins from class-agnostic 3D object proposals, associates each proposal with the top $K=5$ views, summarizes those views, and cross-attends the resulting multi-view token group into a single fused instance-level token per object. It then adds a relation-aware scene representation via 3D-ISR, so the final LLM input is a compact set of object tokens plus scene-level relational context rather than a dense point-token stream (Yu et al., 1 Mar 2025).

Two recent systems push object or part grouping directly into the autoregressive token stream. "Aligning Text, Images, and 3D Structure Token-by-Token" serializes a scene as a list of object records delimited by [OBJECT-START] and [OBJECT-END]; with explicit geometry, each object contributes a contiguous block of 512 quantized shape tokens plus position and pose fields, making each object a literal token group in sequence space (Sahoo et al., 9 Jun 2025). "3D-PLOT-LLM" partitions a frozen point encoder’s 512 patch tokens into $K=16$ locally coherent regions and prefixes each region with a learnable marker and a reserved vocabulary token $\texttt{<part\_k>}$, so parts become directly addressable by the LLM’s own vocabulary (Xue et al., 18 Jun 2026).

Approach	Grouping unit	Structural meaning
(Yoo et al., 28 Jun 2026)	instance token + anchor tokens	one object instance with Gaussian-generating local supports
(Xia et al., 28 May 2026)	3D probe + completion tokens	object-centric proposal token with nearby geometry context
(Yu et al., 1 Mar 2025)	fused object token	one object summarized from 3D and grouped multi-view evidence
(Sahoo et al., 9 Jun 2025)	object record	one scene object serialized as a contiguous token block
(Xue et al., 18 Jun 2026)	$\texttt{<part\_k>}$ block	one unsupervised object-internal region with stable slot identity

The common pattern is a move away from scene-level primitive soups toward token groups that preserve ownership. The differences lie in whether ownership is explicit and object-level, object-internal and part-like, or only implicit through grouped evidence.

3. Group formation mechanisms

One major mechanism is learned soft grouping over semantically enriched 3D features. In "Token-UNet," a residual 3D convolutional encoder first produces a bottleneck feature map $L=100$0 with $L=100$1. TokenLearner then predicts $L=100$2 spatial masks, normalized over $L=100$3, and pools tokens by weighted global aggregation:

$L=100$4

The paper explicitly interprets each token as a size-independent set of variously akin voxels, rather than a fixed patch or object slot (Tshimanga et al., 23 Feb 2026).

A second mechanism is instance-guided construction from structural priors. In "IVT," one instance-guided token $L=100$5 is created per candidate person center, but that token is initialized from a grouped set of $L=100$6 joint-conditioned feature samples obtained from 2D center-to-joint offsets. The model therefore does not keep a persistent set of joint tokens; instead, it compresses a structured body-joint group into one instance token for later spatial and temporal attention (Qiu et al., 2022).

A third mechanism is geometry-aware pooling of explicit 3D primitives. "GST-VLA" first turns each of 256 semantic patches into an anisotropic Gaussian primitive parameterized by residual mean $L=100$7, diagonal log-scale $L=100$8, and opacity $L=100$9, then pools those 256 raw primitives into $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$0 spatial tokens using learned attention queries. The resulting tokens are metrically grounded, geometry-aware, and nonuniformly allocated, but the paper is explicit that they are still implicit object-/surface-centric groups rather than supervised instance slots (Sarowar et al., 10 Mar 2026).

A fourth mechanism is region construction by external or geometric partitioning. "SAM-Guided Masked Token Prediction for 3D Scene Understanding" replaces KNN patches with 3D point-token groups induced by projected SAM mask regions, so each token corresponds to one SAM-consistent 3D region rather than a purely Euclidean neighborhood (Chen et al., 2024). "S4Token" uses superpoint extraction, weighted FPS, and superpoint-constrained grouping to produce superpoint-constrained local surface regions; the paper explicitly states that these groups are not full object instances, but they often align better with parts or semantic regions than naive $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$1NN patches (Mei et al., 24 May 2025). "Efficient Autoregressive Shape Generation via Octree-Based Adaptive Tokenization" builds an adaptive octree whose cells form variable-sized hierarchical token groups according to a quadric-error subdivision criterion, yielding shape-instance-specific but not semantic-part-specific token groups (Deng et al., 3 Apr 2025).

A fifth mechanism is component-conditioned grouping by ownership and interface. "Beyond Spatial Compression" samples local charts $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$2, assigns each chart a partition hint $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$3, contextualizes tokens with partition- and geometry-biased attention, and adds seam/interface variables between groups. Here the group is an unsupervised macro-component, not a semantic part class, but ownership is explicit and operational in the latent state (Chen et al., 11 May 2026).

4. Relations, decoding, and operational state

Grouped tokens become useful only if they can interact and be projected back into dense or actionable 3D structure. "Token-UNet" makes this explicit with TokenFuser. After a small Transformer operates on the compact token set $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$4, TokenFuser predicts fresh voxelwise token-affinity maps $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$5, mixes the transformed tokens with a learned matrix $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$6, broadcasts them back into 3D space via

$$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$7

and adds the result residually back to the encoded feature map before decoding. Spatial detail is therefore not reconstructed by inverting the original pooling masks; it is restored through a learned token-to-space projection conditioned on the retained dense feature scaffold (Tshimanga et al., 23 Feb 2026).

"IVT" uses the opposite direction: once instance-guided tokens are formed, spatial attention acts over all person-centered tokens within a frame, and temporal attention aligns same-index tokens across frames after optical-flow alignment. The representation remains token-centric during reasoning, and 3D joint coordinates are decoded only after those instance-level interactions (Qiu et al., 2022).

In object-centric reconstruction, "Scenes as Objects, Not Primitives" keeps group identity attached to the rendering primitives themselves. Each anchor token predicts Gaussians, every Gaussian inherits the anchor’s group assignment, and 2D instance masks are produced by rendering those assignment probabilities and matching them to ground-truth masks with Hungarian matching. This makes object identity viewpoint-consistent by construction rather than by post-hoc clustering (Yoo et al., 28 Jun 2026).

"3DVLA" uses a related aggregation pattern for robotics. Instance tokens are first formed by multi-view evidence gathering, while a masked self-supervised predictor produces completion tokens at masked and unobserved coordinates. Spatially-Conditioned Geometry Aggregation then retrieves the $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$8-nearest completion points around each instance and routes local completion features into the instance token under uncertainty-gated cross-attention, producing object tokens that are explicitly conditioned on nearby hidden geometry and end-effector-relative position (Xia et al., 28 May 2026).

Some papers generalize the idea of grouped tokens beyond object ownership into relational state. "Beyond Spatial Compression" adds seam compatibility $$\pi_{k,\ell} = \mathrm{softmax}\big(\{\langle A_k, G_{\ell'} \rangle\}_{\ell'=1}^{L}\big)_\ell.$$9, relative-pose refinement $N_g=32$0, and collision evidence $N_g=32$1 between candidate chart groups, then uses a seam-penalized decoding energy to constrain generation and repair. In that formulation, the token group is not just a compressed code; it is part of an operational state that can validate, reject, or repair an attachment (Chen et al., 11 May 2026).

At the most abstract end, "The Token Is a Group Element" treats the token itself as a matrix Lie group element $N_g=32$2, so pairwise structure is canonical:

$N_g=32$3

The output update is group-valued,

$N_g=32$4

which makes equivariance under global group action and the cocycle consistency of relative relations automatic. For $N_g=32$5, $N_g=32$6, and $N_g=32$7, this offers a directly pose-structured notion of 3D token, relevant whenever the “instance” is naturally a frame or transform rather than a surface region (Musialski, 18 Jun 2026).

5. Empirical behavior across tasks

In 3D medical segmentation, grouped-token bottlenecks can be both efficient and competitive. The heaviest Token-UNet reduces memory footprint, inference time, and parameter count to $N_g=32$8, $N_g=32$9, and $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$0 of SwinUNETR, respectively, while improving average Dice from $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$1 to $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$2. The paper also states that adding TokenLearner and TokenFuser is more impactful than adding the Transformer itself, which it interprets as evidence that the grouped-token bottleneck increases semantic throughput (Tshimanga et al., 23 Feb 2026).

In 3D human pose estimation, instance-guided tokens improve both local body reasoning and multi-person temporal reasoning. On CMU Panoptic, the ablation sequence improves from $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$3 mm MPJPE for ordinary tokens, to $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$4 mm with instance-guided spatial attention, to $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$5 mm with temporal attention, and to $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$6 mm with cross-scale attention. The full method also reports $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$7 mm MPJPE on Human3.6M Protocol 1, $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$8 mm PA-MPJPE on Protocol 2, $$\mathcal{Q} = \{(\mathbf{c}_j,\mathbf{p}_j)\}_{j=1}^{N_q},$$9 mm PA-MPJPE on 3DPW, and $N_q=300$0 mm MPJPE on CMU Panoptic (Qiu et al., 2022).

In VLA settings, structure in the token groups is repeatedly isolated by ablation. "GST-VLA" reaches $N_q=300$1 on LIBERO and $N_q=300$2 on SimplerEnv; replacing anisotropic Gaussians with dense depth scalars drops performance to $N_q=300$3, and replacing attention pooling with average pooling costs $N_q=300$4 points, indicating that geometry-aware grouped tokens matter beyond raw 3D coordinates (Sarowar et al., 10 Mar 2026). "3DVLA" raises LIBERO-Plus average success from $N_q=300$5 to $N_q=300$6 across successive additions of the instance module, self-supervised predictor, and uncertainty-guided routing, and on RoboTwin 2.0 with $N_q=300$7 it improves Easy/Hard from $N_q=300$8 to $N_q=300$9 (Xia et al., 28 May 2026).

In scene-level object-centric reconstruction, "Scenes as Objects, Not Primitives" reports class-agnostic target-view instance segmentation AP/AP$K=5$0/AP$K=5$1 of $K=5$2 on ScanNet, ahead of Gaussian Grouping $K=5$3, ObjectGS $K=5$4, and IGGT+LUDVIG $K=5$5. The same paper emphasizes that these gains are obtained in a feed-forward system rather than through per-scene optimization (Yoo et al., 28 Jun 2026).

In part-aware 3D MLLMs, direct vocabulary-level region handles produce measurable gains. "3D-PLOT-LLM" reaches caption-to-slots Jaccard $K=5$6, Exact-match $K=5$7, and slot-to-caption GPT-4o judge $K=5$8 on PartVerse-QA, while also improving whole-object captioning and part-aware grounded description over prior 3D MLLMs with under 1M new trainable parameters and no segmentation decoder (Xue et al., 18 Jun 2026).

6. Limitations, ambiguities, and open directions

A persistent limitation is that many structured 3D token groups are not strict instances. Token-UNet, GST-VLA, S4Token, SAM-guided tokenization, and adaptive octree tokenization all introduce stronger organization than fixed patch tokenization, but they stop short of one-token-per-object decomposition, and their own papers state that explicitly (Tshimanga et al., 23 Feb 2026, Sarowar et al., 10 Mar 2026, Mei et al., 24 May 2025, Chen et al., 2024, Deng et al., 3 Apr 2025). This matters because performance gains from grouped tokens do not, by themselves, prove that explicit instance tokens are always the correct abstraction.

A second limitation is supervision and preprocessing dependence. "3DVLA" avoids extra manual labels but still relies on pseudo ground truth from VL-SAM-v2 for 2D boxes, masks, and 3D centroids (Xia et al., 28 May 2026). SAM-guided masked token prediction depends on offline SAM masks and aligned RGB-D projection (Chen et al., 2024). C2LT-3D depends on chart construction, unsupervised macro-component hints, and seam targets derived from geometric preprocessing (Chen et al., 11 May 2026). These systems therefore expose structured groups, but often only after substantial external structure recovery.

A third issue is slot semantics and token-budget rigidity. Token-UNet fixes $K=5$9 tokens and explicitly notes that future work could study reducing or increasing how many tokens are pooled and at what scale (Tshimanga et al., 23 Feb 2026). 3D-PLOT-LLM fixes $K=16$0 regions and states that hierarchical or variable-$K=16$1 partitioning is future work; its $K=16$2 slots are object-local handles, not category-consistent semantic labels across objects (Xue et al., 18 Jun 2026). "Scenes as Objects, Not Primitives" fixes a maximum of $K=16$3 groups and lists scaling to larger scenes as a limitation (Yoo et al., 28 Jun 2026). "Aligning Text, Images, and 3D Structure Token-by-Token" provides explicit object grouping, but its structure is shallow: scene-to-object, without explicit part graphs or scene-graph edges (Sahoo et al., 9 Jun 2025).

A fourth issue is that some structurally rich token ontologies remain mathematically specialized. Lie-Algebra Attention requires matrix Lie groups and a logarithm chart containing the relative poses; the paper explicitly notes restrictions near branch cuts and principal-log domains, especially for affine groups (Musialski, 18 Jun 2026). This is powerful for pose-structured tokens, but it does not directly solve ownership or semantic grouping.

The broader literature also suggests that the field has not converged on a single best interface. "Pts3D-LLM" deliberately excludes object-centric tokenization from its core comparison, yet finds that object-based ordering of point tokens helps substantially, especially for 3D dense captioning. The paper therefore supports the weaker claim that 3D token structure should reflect meaningful organization, while leaving object-centric or adaptive tokenization as future work (Thomas et al., 6 Jun 2025). A plausible implication is that future systems will need to combine the controllability of explicit instance groups, the efficiency of learned soft grouping, and the geometric faithfulness of pose- or surface-structured tokens, rather than treating these as mutually exclusive alternatives.