UniScene3D: Unified 3D Scene Representation

• UniScene3D is a unified 3D scene understanding framework that fuses multi-view colored pointmaps to jointly capture image appearance and global geometry.
• It employs a transformer-based encoder with early fusion and multi-level contrastive alignment, including cross-view geometric and grounded view alignment, to support diverse tasks.
• State-of-the-art results on viewpoint grounding, scene retrieval, classification, and 3D VQA demonstrate its effectiveness in low-shot and fine-tuned scenarios.

Searching arXiv for UniScene3D and closely related unified 3D scene representation work. UniScene3D is a 3D vision-language pretraining framework for unified 3D scene understanding that learns scene representations from multi-view colored pointmaps, jointly modeling image appearance and geometry with a transformer-based encoder. Its central claim is that colored pointmaps provide an intermediate representation that preserves RGB cues while exposing globally aligned 3D structure, thereby enabling a single pretrained backbone to transfer across viewpoint grounding, scene retrieval, scene type classification, and 3D VQA. The method introduces cross-view geometric alignment and grounded view alignment in addition to view-level and scene-level language alignment, and reports state-of-the-art performance in low-shot and task-specific fine-tuning evaluations (Mao et al., 2 Apr 2026).

1. Problem formulation and conceptual scope

UniScene3D addresses a specific deficiency in existing 3D pretraining pipelines: many methods specialize in a single modality. Point clouds encode geometry well but are awkward for 2D transformer backbones; multi-view images preserve appearance but do not by themselves provide explicit shared 3D coordinates; depth maps supply geometry but remain fundamentally view-centric (Mao et al., 2 Apr 2026).

The framework is designed around three requirements for unified 3D scene understanding. First, scene representations should be geometrically consistent across views of the same scene. Second, they should be semantically grounded in language. Third, they should transfer across multiple downstream tasks rather than being optimized for a single benchmark. In this formulation, “unified” refers to a shared representation that supports grounding, retrieval, classification, and question answering within one pretraining regime.

A central premise is that single views provide only partial observations. The representation therefore must capture how views of the same scene relate both geometrically and semantically. UniScene3D operationalizes this premise through colored pointmaps and multi-level contrastive alignment, rather than through explicit reconstruction, occupancy prediction, or generative rendering.

2. Colored pointmaps and encoder architecture

The model takes $V$ aligned view pairs

$\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$

where $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$ is an RGB image and $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$ is a pointmap in the world coordinate frame. The pointmap is constructed by back-projecting depth using camera intrinsics and extrinsics, so each pixel stores a 3D point in a globally aligned scene coordinate system (Mao et al., 2 Apr 2026).

UniScene3D is initialized from a pretrained 2D image encoder from FG-CLIP and uses early fusion at the patch embedding stage. For each view,

$$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$

followed by

$\mathbf{z}_v = \mathbf{z}^I_v + \mathbf{z}^P_v.$

This design preserves compatibility with pretrained 2D weights while injecting geometry before transformer processing. A learnable class token is appended to each view token sequence, and the resulting sequence is passed through $N$ transformer blocks. The output class token yields the view representation $\mathbf{h}_v$, while the scene representation is obtained by mean pooling: $$\bar{\mathbf{h}}_s = \text{mean pool}\left(\{\mathbf{h}_v\}\right).$$

The architectural significance of colored pointmaps lies in their dual role. They are image-like enough to reuse ViT-style machinery, yet geometrically explicit enough to encode coherent world-frame structure across views. The paper’s ablations further indicate that this is not a superficial implementation choice: simple concatenation plus projection performs worse than the proposed early summation fusion, suggesting that preserving pretrained patch embeddings and positional encoding is materially beneficial.

3. Contrastive pretraining objectives

UniScene3D pretraining uses four alignment losses:

1. cross-view geometric alignment,
2. grounded view alignment,
3. view-level alignment,
4. scene-level alignment.

The total objective is

$$\mathcal{L}_{\text{total}} = \lambda \mathcal{L}_{\text{geo}} + \mathcal{L}_{\text{ground}} + \mathcal{L}_{\text{view}} + \mathcal{L}_{\text{scene}},$$

with $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$0 (Mao et al., 2 Apr 2026).

Cross-view geometric alignment makes embedding similarity reflect geometric overlap between views. For two views $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$1 and $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$2, the paper defines the symmetric Chamfer distance

$\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$3

where $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$4. Views in the same scene are ranked by increasing Chamfer distance, and the ranking is converted into a mixed hard-soft target distribution. The training signal is a soft-label cross-entropy over view similarities. This makes the learned embedding space encode relative geometric proximity rather than a purely binary positive-negative relation.

Grounded view alignment introduces object-level semantic grounding. Let $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$5 denote the embedding of text referring to object $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$6. A view is positive for an object if its pointmap intersects the object’s 3D mask from SceneVerse. The resulting bidirectional contrastive loss aligns views with object-referring text and object text with views. This directly enforces semantic consistency across views that observe the same referred object.

View-level alignment and scene-level alignment extend the supervision to captions. The former aligns individual view embeddings with view captions; the latter aligns mean-pooled scene embeddings with scene captions. Taken together, the four losses act at different granularity levels: geometric relations between views, object-level grounding, per-view semantics, and whole-scene semantics. A plausible implication is that the representation’s transferability derives from this layered supervision structure rather than from any single contrastive term in isolation.

4. Pretraining corpora and evaluation tasks

UniScene3D is pretrained on 6,562 indoor scenes from ScanNet, 3RScan, and ARKitScenes. For language supervision, SceneVerse provides LLM-generated referring expressions for grounded view alignment, and POMA-3D provides view-level and scene-level captions (Mao et al., 2 Apr 2026).

The evaluation suite spans four downstream tasks.

Task	Datasets	Metrics
Viewpoint grounding	ScanRefer, Nr3D, Sr3D	$\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$7, $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$8, $\{(\mathbf{I}_v, \mathbf{P}_v)\}_{v=1}^{V},$9
Scene retrieval	ScanRefer, Nr3D, Sr3D	$\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$0, $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$1
Scene type classification	21 ScanNet room categories	zero-shot accuracy; few-shot linear probing
3D VQA	ScanQA, SQA3D, Hypo3D	Exact Match at top-1 and top-10

For scene retrieval, captions are formed by concatenating multiple referring expressions, with evaluation reported for $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$2 and $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$3 expressions per caption. For scene type classification, few-shot evaluation uses linear probing with L-BFGS. For 3D VQA, the protocol is task-specific fine-tuning with a shallow QA head and a BERT language encoder, while the visual encoder remains frozen.

This task design is notable because it mixes zero-shot, few-shot, and fine-tuned regimes. The benchmark suite therefore probes not only representational quality after adaptation, but also whether the pretrained embeddings retain semantic and geometric structure under limited supervision.

5. Reported empirical performance and ablation findings

The paper reports state-of-the-art performance across all evaluated tasks, outperforming image-only, point cloud-only, and prior pointmap-based baselines, including POMA-3D (Mao et al., 2 Apr 2026).

On zero-shot viewpoint grounding, UniScene3D achieves $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$4 on ScanRefer, $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$5 on Nr3D, and $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$6 on Sr3D. The reported POMA-3D baselines are $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$7, $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$8, and $\mathbf{I}_v \in \mathbb{R}^{H \times W \times 3}$9, respectively. On scene retrieval with $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$0, UniScene3D reaches $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$1 on ScanRefer, $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$2 on Nr3D, and $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$3 on Sr3D.

For scene type classification, the zero-shot result is $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$4, compared with $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$5 for the best baseline POMA-3D. Few-shot linear probing yields $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$6 in the 1-shot setting, $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$7 in the 5-shot setting, and $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$8 in the 10-shot setting. On 3D VQA, UniScene3D reports $\mathbf{P}_v \in \mathbb{R}^{H \times W \times 3}$9 on ScanQA, $$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$0 on SQA3D, and $$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$1 on Hypo3D, where each pair denotes top-1 and top-10 Exact Match.

The ablations clarify why these gains occur. Removing image input degrades performance, but removing pointmap input hurts more on grounding and retrieval, indicating that appearance and geometry are complementary rather than substitutable. Removing $$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$2 reduces performance across tasks, showing that explicit cross-view geometric structuring matters. Removing $$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$3 causes a particularly large drop on viewpoint grounding, supporting the claim that object-level semantic alignment is critical. The appendix further reports that removing either view-level alignment or scene-level alignment broadly harms performance, with scene-level alignment being especially important for scene classification and view-level alignment especially important for grounding.

Two additional findings bear on generalization. First, the model is robust to reduced view count: even with 16 views, UniScene3D can outperform POMA-3D with 32 views in retrieval. Second, performance improves steadily as the number of pretraining scenes increases, suggesting that the method scales favorably with additional real-world 3D data.

6. Position within unified 3D research

UniScene3D belongs to a broader line of work seeking unified 3D representations, but its notion of unification is distinct. It does not unify reconstruction and reasoning in the manner of Uni3DR$$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$4, which learns a volumetric representation for both TSDF/occupancy reconstruction and LLM-oriented downstream reasoning from multi-view RGB video using frozen SAM and CLIP features (Chu et al., 2024). Nor does it unify occupancy prediction and object detection in the manner of UniVision, which couples voxel and BEV representations for vision-centric 3D perception through an explicit-implicit view transform and joint multitask optimization (Hong et al., 2024).

Instead, UniScene3D unifies 3D scene understanding through a common appearance-geometry-language embedding space. Its reported downstream tasks are viewpoint grounding, scene retrieval, scene type classification, and 3D VQA, all driven by the same pretrained colored-pointmap encoder. This suggests a narrower but cleaner target: a general-purpose scene representation for semantic transfer, rather than a full-stack system for reconstruction, rendering, or sensor-level perception.

A common misconception is to treat all “unified 3D” methods as variants of the same problem. The available evidence indicates otherwise. In UniScene3D, the central object is a contrastively pretrained scene encoder over multi-view colored pointmaps. In Uni3DR$$\mathbf{z}^I_v = \phi_I(\mathbf{I}_v) + \mathbf{E}_{pos}, \qquad \mathbf{z}^P_v = \phi_P(\mathbf{P}_v),$$5, it is a geometry-semantic volumetric representation coupled to reconstruction and an LLM interface. In UniVision, it is a multitask 3D perception backbone spanning occupancy and detection. The family resemblance lies in the use of shared 3D representations; the operational goals, supervision, and downstream interfaces are materially different.

Within that taxonomy, UniScene3D’s specific contribution is the demonstration that colored pointmaps, when paired with cross-view geometric alignment and grounded view alignment, form a strong substrate for generalizable 3D scene understanding. Its empirical profile—especially zero-shot grounding, few-shot classification, and frozen-backbone 3D VQA—supports the view that the representation itself, rather than only downstream adaptation, is the main locus of improvement.