NOVA3R: Global Visual Transformer for 3D

• NOVA3R is a non-pixel-aligned visual Transformer architecture that enables amodal 3D reconstruction from unposed multi-view images.
• It aggregates global scene tokens with multi-view self-attention to overcome visibility bias and ray duplication inherent in traditional methods.
• The approach employs a diffusion-based decoder to generate complete, physically consistent point clouds validated across both scene-level and object-level benchmarks.

NOVA3R is a non-pixel-aligned visual Transformer architecture designed for amodal 3D reconstruction from unposed multi-view images. Unlike traditional 3D reconstruction pipelines that rely on per-pixel (pixel-aligned) raycasting or volumetric mapping, NOVA3R employs a global, view-agnostic latent scene representation. This mechanism enables the model to infer complete point clouds that encompass both visible and occluded surfaces with minimal duplication and superior physical plausibility, advancing the state of the art for both scene-level and object-level 3D understanding (Chen et al., 4 Mar 2026).

1. Motivation and Core Problem

Most established approaches to 3D reconstruction from images rely on pixel-alignment, wherein predicted geometry is generated along rays cast from image pixels (e.g., depth maps, per-pixel point clouds). Such methods exhibit two fundamental limitations:

• Visibility bias: Geometry is reconstructed only for surfaces explicitly visible in the input images. Occluded (amodal) regions remain unmodeled.
• Ray duplication artifacts: When multiple images capture the same 3D region, pixel-aligned methods redundantly reconstruct multiple points on overlapping scene geometry, resulting in physically implausible density and multi-layer artifacts.

NOVA3R addresses these issues by learning a global scene representation that aggregates information across all input views. Reconstruction thus becomes independent of per-pixel ray directions, enabling the recovery of both visible and invisible surfaces and yielding point clouds free of duplicated structures (Chen et al., 4 Mar 2026).

2. Scene-Token Aggregation and Global Representation

The cornerstone of NOVA3R is its learnable scene-token mechanism. Specifically, a set of $M$ scene tokens $\mathbf{t}_s\in\mathbb{R}^{M\times C}$ is introduced as a latent memory encoding the global structure and semantics of the 3D scene.

• Image Feature Extraction: Each of the $K$ unposed images is encoded into a sequence of $N_I$ patch-level tokens $\mathbf{t}_I^i\in\mathbb{R}^{N_I\times C}$ (using standard Vision Transformer embedding schemes).
• Token Sequence Formation: All image-patch tokens from all $K$ images, together with the $M$ scene tokens, are concatenated into a single sequence of length $K N_I+M$.
• Transformer Processing: This sequence is processed by $L$ Transformer blocks with multi-head self-attention.
  • Scene tokens are updated through cross-attention over all image-patch and scene tokens.
  • Via repeated attention and mixing, the scene tokens $\mathbf{Z}\doteq\mathbf{t}_s^{(L)}$ aggregate multi-view information into a compact, view-agnostic latent encoding.

This approach enables every image token to contribute information to every scene token, supporting robust amodal representation and eliminating view-dependent redundancies (Chen et al., 4 Mar 2026).

3. Diffusion-Based 3D Decoding

NOVA3R reconstructs complete 3D point clouds using a diffusion-based flow-matching decoder $\mathbf{t}_s\in\mathbb{R}^{M\times C}$0. The generation pipeline is as follows:

• Noising Process: Given a ground-truth point cloud $\mathbf{t}_s\in\mathbb{R}^{M\times C}$1, a noisy version $\mathbf{t}_s\in\mathbb{R}^{M\times C}$2 is produced for $\mathbf{t}_s\in\mathbb{R}^{M\times C}$3 as $\mathbf{t}_s\in\mathbb{R}^{M\times C}$4 with $\mathbf{t}_s\in\mathbb{R}^{M\times C}$5.
• Decoder Training: $\mathbf{t}_s\in\mathbb{R}^{M\times C}$6 is trained to predict the original noise $\mathbf{t}_s\in\mathbb{R}^{M\times C}$7 from $\mathbf{t}_s\in\mathbb{R}^{M\times C}$8, scene tokens $\mathbf{t}_s\in\mathbb{R}^{M\times C}$9, and continuous time $K$0 by minimizing the flow-matching objective:

$K$1

• Generation via ODE: At inference, given the scene tokens, a denoising ODE is solved (by numerical integration from $K$2 to $K$3) to recover a consistent, permutation-invariant point cloud representing the amodal 3D scene.

This decoder is both flexible and globally consistent, enabling the network to generate complete and physically coherent geometries (Chen et al., 4 Mar 2026).

4. Training Procedure and Objectives

NOVA3R employs a two-stage training framework:

1. Stage 1: 3D Latent Autoencoding
   • The decoder $K$4 is trained on ground-truth point clouds to reconstruct clean shapes from their noisy counterparts using flow-matching loss.
2. Stage 2: Image-to-Latent Alignment
   • The image encoder and scene tokens are trained to generate latent codes that, when decoded by the (frozen) $K$5, yield faithful reconstructions.
   • The objective remains the flow-matching loss, evaluated with respect to the 3D decoder's predictions.

No explicit occupancy or Chamfer loss is required; permutation invariance and coverage of invisible geometry are handled intrinsically by the flow-matching loss on unordered point sets (Chen et al., 4 Mar 2026).

5. Empirical Evaluation and Results

NOVA3R was validated on diverse scene-level and object-level benchmarks:

• Scene-Level Datasets: 3D-FRONT, ScanNet++V2, ARKitScenes (training); SCRREAM (zero-shot completion); 7-Scenes, NRGBD (multi-view, visible reconstruction).
• Object-Level Datasets: Objaverse (training), Google Scanned Objects (GSO; evaluation).
• Outdoor Scenes: Virtual KITTI 2 (finetuned).

Key quantitative results include:

Method	Dataset	View Count	CD↓	[email protected]↑	Acc (m)↓	Comp (m)↓	NC↑
DUST3R	SCRREAM	1	0.086	0.757	–	–	–
VGGT	SCRREAM	1	0.070	0.810	–	–	–
NOVA3R	SCRREAM	1	0.063	0.836	–	–	–
NOVA3R	SCRREAM	2	0.059	0.858	–	–	–
NOVA3R	7-Scenes	2	–	–	0.041	0.033	0.794
LaRI	GSO	1	0.025	0.966	–	–	–
NOVA3R	GSO	1	0.020	0.985	–	–	–
NOVA3R	GSO	2	0.023	0.978	–	–	–

• Hole area ratio: NOVA3R achieves 0.088–0.121 (vs. >0.23 for pixel-based methods).
• Density variance: NOVA3R yields more uniform point clouds and reduces artifact generation as the number of input views increases.

NOVA3R achieves or surpasses previously established results on completeness, accuracy, and normal consistency. Outdoor reconstruction on Virtual KITTI 2 confirms generalization to larger-scale and unconstrained scenarios (Chen et al., 4 Mar 2026).

6. Significance and Implications

NOVA3R marks a departure from pixel-centric geometry inference toward unified amodal 3D modeling. By disentangling geometry from camera rays and leveraging a Transformer-based, globally aggregated latent architecture, NOVA3R:

• Mitigates view redundancy and duplication,
• Enables robust inference of occluded structure,
• Delivers physically consistent point distributions,
• Generalizes across scenes, objects, and varying numbers and poses of input views,
• Demonstrates scalable applicability to both indoor and outdoor scenes.

The strong benchmarking performance and versatile representation capacity suggest NOVA3R establishes a new direction for amodal, permutation-invariant 3D scene understanding in vision (Chen et al., 4 Mar 2026).