Multi-Scale Normal Distributions Transform

• Multi-Scale Normal Distributions Transform is a method that divides 3D space into voxel cells, modeling local geometry with Gaussian distributions and capturing both spatial and color features.
• The approach employs a transformer decoder to fuse cross-scale features, significantly improving the accuracy of 3D vision-language tasks.
• It underpins NDTokenizer3D by enabling precise segmentation, visual question answering, and interactive prompting in complex 3D environments.

NDTokenizer3D is a generalist 3D vision–LLM that transforms raw, high-resolution point clouds into holistic scene tokens for downstream language modeling tasks. It employs a three-stage pipeline: constructing multi-scale Normal Distributions Transform (NDT) representations, extracting per-cell 3D geometric features, and fusing these cross-scale features using a Multi-Scale NDT Decoder (MSDec) to produce compact, information-rich embeddings. These holistic scene tokens natively support a variety of 3D understanding tasks—referring segmentation, visual question answering (VQA), dense captioning, situated QA—and human interaction primitives such as point-, box-, and mask-based prompting, thereby unifying spatial and language reasoning within a single architecture (Tang et al., 26 Nov 2025).

1. Multi-Scale NDT Representation

NDTokenizer3D initiates its pipeline by partitioning the input point cloud $\mathbf{X} = \{x_i \in \mathbb{R}^3\}_{i=1}^{N_p}$ into $R$ regular grid scales ($s = 1,\ldots,R$), each subdividing the space into $N_s$ voxel cells $\{C_s^j\}_{j=1}^{N_s}$. Within each cell $C_s^j$, the local geometry is modeled by a 3D Gaussian distribution:

$$p(x \mid C_s^j) = \mathcal{N}(x; \mu_s^j, \Sigma_s^j),$$

where

$$\mu_s^j = \frac{1}{n} \sum_{i=1}^n x_i,\qquad \Sigma_s^j = \frac{1}{n-1}\sum_{i=1}^n (x_i-\mu_s^j)(x_i-\mu_s^j)^{T}.$$

From each cell, geometric descriptors are extracted, including the mean $\mu_s^j$, the three eigenvalues $$\{\lambda_{s,1}^j, \lambda_{s,2}^j, \lambda_{s,3}^j\}$$ of the covariance $R$0, and optional normalized shape features. Color features $R$1 can be projected from RGB cameras. The resulting per-cell vector

$R$2

captures both spatial and appearance attributes. For each scale, all cell descriptors are stacked as $R$3.

2. Multi-Scale NDT Decoder (MSDec) and Cross-Scale Fusion

The MSDec module consists of a transformer decoder stack with $R$4 layers. At each layer $R$5, the scale-$R$6 encoder outputs $R$7 serve as Key–Value pairs, and a set of learnable queries $R$8 serve as Query vectors. The queries are updated by sequential cross-attention to each scale, followed by self-attention and a feed-forward network:

$R$9

with

$s = 1,\ldots,R$0

Cross-scale fusion in MSDec is mathematically formalized as

$s = 1,\ldots,R$1

with

$s = 1,\ldots,R$2

Learnable positional embeddings $s = 1,\ldots,R$3 encoding 3D center and scale are incorporated into encoder features for spatial awareness. Ablation results confirm multi-scale NDT markedly surpasses naïve downsampling and three scales are optimal.

3. Holistic Scene Tokenization and LLM Integration

Following $s = 1,\ldots,R$4 MSDec layers, the decoder output $s = 1,\ldots,R$5 is linearly projected into $s = 1,\ldots,R$6 scene tokens for the LLM: $s = 1,\ldots,R$7 These tokens $s = 1,\ldots,R$8 compose the visual context $s = 1,\ldots,R$9 for multimodal LLM endpoints, serving as high-level scene encoding for vision–language tasks.

4. Unified Interface for Human-Interactive Prompting and Segmentation Decoding

NDTokenizer3D’s architecture natively supports human-interactive input (points, bounding boxes, masks). A prompt is rasterized into a mask $N_s$0 and pooled from fine-scale features:

$N_s$1

This vector is projected to $N_s$2 and concatenated to initial queries, propagating prompt-guided information throughout the MSDec hierarchy. The final prompt token $N_s$3 is projected to form a guidance embedding $N_s$4 for the LLM.

For segmentation decoding, when the LLM emits [SEG], the hidden state $N_s$5 triggers a segmentation head $N_s$6 generating a query $N_s$7. MSDec processes this query to $N_s$8, producing a mask kernel $N_s$9; mask logits and probabilities are computed by

$\{C_s^j\}_{j=1}^{N_s}$0

with sigmoid activation. Multiple loss components are applied for pre-training and instruction tuning:

• Cross-entropy for semantic classification: $\{C_s^j\}_{j=1}^{N_s}$1
• Binary cross-entropy + Dice loss for masks: $\{C_s^j\}_{j=1}^{N_s}$2
• Cosine-similarity alignment to CLIP features: $\{C_s^j\}_{j=1}^{N_s}$3
• Token cross-entropy for language: $\{C_s^j\}_{j=1}^{N_s}$4

5. Performance Benchmarks and Comparative Evaluation

NDTokenizer3D demonstrates competitive and state-of-the-art metrics across key 3D vision–language benchmarks (relative to generalist competitor 3D-LLaVA):

Task/Metric	3D-LLaVA	NDTokenizer3D	Δ
3D Ref Seg (Multi3DRefer mIoU)	42.7	46.0	+3.3
3D VQA (ScanQA, CiDEr/B-4/M/R)	92.6/17.1/18.4/43.1	98.6/17.0/19.4/44.9	see table
Situated QA (SQA3D EM/EM-R)	54.5/56.6	54.4/57.1
Dense Captioning (Scan2Cap IoU 0.5)	78.8/36.9/27.1/57.7	79.0/36.7/27.1/57.7
Hallucination (3D-POPE neg. accuracy)	Random: 75.5%	84.1%
	Popular: 66.9%	75.5%
	Adversarial:63.1%	72.0%

Ablative studies indicate that multi-scale NDT is superior to conventional downsampling, three scales yield optimal results, and approximately 800 queries suffice.

6. Architectural Innovations and Significance

NDTokenizer3D introduces several innovations:

• Multi-scale NDT representation: Compactly encodes both global context and local geometry for raw 3D input.
• MSDec framework: Hierarchical transformer decoder designed for efficient cross-scale feature fusion; supports unified handling of vision, spatial reasoning, segmentation, and interaction.
• LLM integration: Scene tokens and prompt-based guidance seamlessly bridge the gap between 3D spatial input and natural language reasoning in a multimodal model.

This approach enables a unified, flexible interface for 3D understanding, denoting a methodological advance in generalist 3D VLMs.

7. Context and Implications for 3D Vision-Language Research

NDTokenizer3D’s pipeline unifies scene tokenization, multi-scale geometric representation, and human-interactive prompting in a single architecture, supporting both holistic scene understanding and precise segmentation. Intrinsic support for language-level reasoning over 3D spatial data advances the state-of-the-art in 3D referring tasks, VQA, dense captioning, and hallucination resistance. A plausible implication is wider adoption of multi-scale Gaussian cell representations and transformer-based cross-scale fusion modules in next-generation 3D VLMs aimed at generalist multimodal understanding (Tang et al., 26 Nov 2025).