NAS3R: Self-Supervised 3D Reconstruction

• NAS3R is a self-supervised 3D reconstruction framework that learns explicit 3D Gaussian representations, camera poses, and depth maps from multi-view images without relying on ground-truth 3D labels.
• It employs a Vision Transformer backbone with masked attention and a depth-based Gaussian formulation to jointly predict geometry and camera parameters in unconstrained scenes.
• The approach enables robust novel view synthesis, accurate pose estimation, and competitive depth prediction, making it suitable for in-the-wild data and scalable self-supervised training.

Searching arXiv for the NAS3R paper and directly related reconstruction methods to ground the article in current literature. NAS3R, introduced in "From None to All: Self-Supervised 3D Reconstruction via Novel View Synthesis" (Huang et al., 29 Mar 2026), is a self-supervised, feed-forward 3D reconstruction framework that learns explicit 3D Gaussian scene representations, camera extrinsics and intrinsics, and per-view depth maps from uncalibrated, unposed multi-view images, with no ground-truth 3D labels, no pose labels, and no pretrained priors required during training. Its central premise is that, if a model can reconstruct scene geometry and predict camera parameters well enough to render unseen target views, then agreement between rendered target images and true target images can serve as the supervision signal. Within that formulation, NAS3R is presented as a geometry-aware approach to 3D reconstruction from unconstrained data, and as a self-supervised training framework compatible with strong supervised 3D reconstruction architectures (Huang et al., 29 Mar 2026).

1. Problem formulation and motivation

NAS3R addresses the setting in which a model receives multi-view images partitioned into context views $\mathcal{I_C}$ and target views $\mathcal{I_T}$, with $V = V_C + V_T$, but has access to neither calibrated cameras nor ground-truth geometry. Unlike many prior methods, it does not require known intrinsics or poses during training; the first view is normalized as the canonical reference frame (Huang et al., 29 Mar 2026).

The motivation arises from limitations of large supervised 3D datasets and SfM-derived annotations. The stated obstacles are that 3D labels are expensive to obtain, camera poses from SfM can fail in low-texture, sparse-view, or low-overlap scenes, and many self-supervised methods still need known intrinsics or pretrained geometric priors to avoid degenerate optimization. NAS3R is positioned as a response to that gap by showing that a transformer-based reconstruction model can be trained entirely from 2D photometric supervision while jointly learning geometry and camera parameters from scratch.

This suggests that NAS3R is not merely a new reconstruction module but a training regime aimed at reducing dependence on manual annotation and SfM preprocessing. A plausible implication is that the framework is especially relevant for internet-scale or in-the-wild multi-view data, where pose metadata and clean geometric supervision are often unavailable or unreliable.

2. Architectural organization and information flow

NAS3R is built on a standard Vision Transformer (ViT) backbone with a shared-weight encoder, a masked decoder, and parallel heads for geometry and camera prediction (Huang et al., 29 Mar 2026). Each input image is patchified and encoded into tokens $\mathbf{F}^v \in \mathbb{R}^{L \times C}$ for view $v$, where $L$ is the token count and $C$ the channel dimension. A learnable camera token $\mathbf{g}^v$ is concatenated so that $\mathbf{F}^v := [\mathbf{g}^v,\mathbf{F}^v]$.

A defining architectural claim is that all 3D quantities are predicted from the same visual representation: depth, Gaussian parameters, camera intrinsics, and camera extrinsics. The paper describes this as encouraging consistent joint feature learning rather than allowing separate modules to drift apart. In that sense, the shared backbone is not only an efficiency choice but part of the optimization strategy.

The decoder uses masked cross-attention to regulate information flow. Context tokens can attend only to context tokens, while target tokens can attend to both context and target tokens (Huang et al., 29 Mar 2026). The stated purpose is to prevent information leakage: if context reconstruction were allowed to directly attend to target content, the model could “cheat” by leaking the answer into the reconstruction pathway. Under this masking scheme, context geometry is reconstructed without target-view information, while target views remain usable for camera estimation and global scene reasoning.

This masking policy is significant because self-supervised novel view synthesis is underconstrained. The architecture therefore encodes a specific causal structure: geometry is reconstructed from context, and target views act as supervision and as part of the global inference process, but not as direct reconstruction evidence for the context geometry.

3. Camera prediction and depth-based Gaussian parameterization

For each view $v$, NAS3R predicts intrinsics $\mathcal{I_T}$0 and relative extrinsics $\mathcal{I_T}$1, defined relative to the first view as canonical reference (Huang et al., 29 Mar 2026). Camera predictions are normalized so that the first input view is the reference. Intrinsics are parameterized by field of view; the model assumes identical horizontal and vertical FOV, and places the principal point at the image center. For the MASt3R-based variant, the camera head is a lightweight 3-layer MLP that predicts rotation in 6D representation, translation in 4D homogeneous coordinates, and intrinsics through FOV. The camera head is initialized to predict identity poses and focal length equal to image size, which the paper presents as a stabilization trick that avoids early collapse.

A core design choice is the depth-based Gaussian formulation. Earlier pose-free methods are described as often using a canonical-space formulation that directly regresses 3D Gaussian centers in a shared space, which the paper argues is hard to optimize from scratch because those canonical points are only implicitly constrained by poses, intrinsics, and depth. NAS3R instead uses a local-to-global depth-based formulation: it predicts per-pixel depth, lifts depth into 3D using predicted camera parameters, and uses the resulting 3D points as Gaussian centers.

Depth is predicted by a DPT-style depth head from refined context tokens $\mathcal{I_T}$2. Predicted depths pass through a sigmoid and are then linearly mapped between near and far planes, yielding a bounded depth range (Huang et al., 29 Mar 2026). The resulting depth map $\mathcal{I_T}$3 is lifted into 3D using estimated intrinsics and extrinsics; pixel coordinates plus depth become 3D points in the reference frame, and those points become Gaussian centers. The paper emphasizes that this keeps centers on valid camera rays, strongly constrains geometry, avoids exploding or unbounded depth values, and gives a well-conditioned initialization for self-supervised training.

In addition to centers, each Gaussian predicts a rotation quaternion $\mathcal{I_T}$4, scale $\mathcal{I_T}$5, opacity $\mathcal{I_T}$6, and spherical harmonics coefficients $\mathcal{I_T}$7. This follows the standard explicit 3D Gaussian Splatting representation. The Gaussian head is also DPT-style and takes refined context tokens together with the original context images, with the latter used to preserve fine spatial details.

4. Self-supervised novel view synthesis objective

The training pipeline follows a feed-forward novel view synthesis setup. Context views are used to reconstruct the scene; target views are held out and used for photometric supervision (Huang et al., 29 Mar 2026). The model predicts camera parameters $\mathcal{I_T}$8 for all views, depth maps for context views, and Gaussian parameters for the scene. Context-view geometry is converted into explicit 3D Gaussian primitives, and those Gaussians are rendered into target images using the model’s own predicted camera parameters for the target views.

Rendering is performed by a differentiable Gaussian splatting renderer: $\mathcal{I_T}$9 where $V = V_C + V_T$0 is the differentiable Gaussian rendering function. The paper describes this process as a kind of photometric bundle adjustment: the model adjusts geometry and camera parameters so that rendered pixels match the true target image, and gradients flow through the renderer to depth, poses, intrinsics, and Gaussian parameters.

The training objective combines pixel fidelity and perceptual similarity through an MSE/L2 term and LPIPS, with $V = V_C + V_T$1 balancing the two; the reported implementation uses $V = V_C + V_T$2 (Huang et al., 29 Mar 2026). This creates the self-supervised loop: geometry and camera estimates are improved by rendering losses from unseen views rather than by direct 3D or pose supervision.

The paper identifies five stability factors: the shared transformer backbone, masked attention, depth-based Gaussian centers, good camera initialization, and a progressive interval curriculum. The curriculum begins with smaller frame intervals and gradually increases them, making correspondence learning easier early on. Even without the curriculum, the method is reported to converge, though with reduced performance.

5. Compatibility with existing backbones and use of priors

NAS3R is presented not only as a standalone model but as a self-supervised training framework compatible with strong existing architectures, including VGGT, MASt3R-based multi-view models, and related 3D transformer backbones (Huang et al., 29 Mar 2026). For VGGT-based variants, the original encoder/decoder structure is kept and a DPT-based Gaussian head is added. For MASt3R-based variants, the pairwise decoder is extended to multi-view and depth, Gaussian, and camera heads are added.

This compatibility is important because it frames NAS3R as a training recipe for supervised reconstruction systems rather than as an isolated architecture. A plausible implication is that the method can be used to convert architectures originally trained under full supervision into models that exploit unlabeled multi-view data.

Although NAS3R is designed to work without priors, the paper states that it can also benefit from them. Variants initialized from MASt3R and VGGT improve performance further, especially for novel view synthesis and pose estimation. When ground-truth intrinsics are available, the model can incorporate them explicitly; the variant called NAS3R-I uses GT intrinsics, which improves pose estimation and reduces scale ambiguity (Huang et al., 29 Mar 2026). The paper also notes that intrinsics are often embedded into the decoder via a linear layer and concatenation with tokens.

The framework is also described as useful for self-supervised pretraining: NAS3R pretrained self-supervisedly and then fine-tuned with ground-truth supervision outperforms training from scratch. In that sense, its role extends beyond direct inference to representation and geometry initialization for downstream supervised tasks.

6. Experimental configuration and reported findings

NAS3R is trained on RealEstate10K (RE10K) and DL3DV, and evaluated on RE10K, ACID, DTU, ScanNet++, and BlendedMVS (Huang et al., 29 Mar 2026). The reported implementation is in PyTorch, trained on a single NVIDIA A100, with learning rate $V = V_C + V_T$3, LPIPS weight 0.05, and default input of two views. The batch size is 10 for VGGT-based models and 16 for MASt3R-based models. Input resolution is $V = V_C + V_T$4 for VGGT-based variants and $V = V_C + V_T$5 for MASt3R-based variants.

The paper reports that NAS3R outperforms other self-supervised methods in novel view synthesis and is competitive with supervised methods. On RE10K, ACID, DL3DV, and DTU, it consistently improves PSNR, SSIM, and LPIPS over SelfSplat, SPFSplat, NoPoSplat variants, and PF3plat, with especially strong performance under wide viewpoint changes (Huang et al., 29 Mar 2026).

For relative pose estimation, NAS3R achieves the best or near-best pose AUC scores among self-supervised methods and remains strong even compared to some supervised baselines. The paper explicitly notes that pose prediction is not simply a side product; it is part of the learned self-supervised geometry. It further states that NAS3R can approach or surpass SuperPoint + SuperGlue in some settings, despite using only photometric self-supervision.

For depth estimation, the paper reports that NAS3R achieves the best depth results on BlendedMVS among compared methods, including some supervised baselines, with better relative error and better inlier ratio. Visualizations are said to show cleaner Gaussian distributions, more coherent surfaces, and sharper depth renderings than the baselines, with geometry that is less noisy and less smeared. The method also scales with more data: training on RE10K + DL3DV improves NVS, pose, and depth, supporting the paper’s claim that NAS3R is a scalable way to exploit unlabeled multi-view data.

7. Relation to prior self-supervised methods, limitations, and significance

The paper contrasts NAS3R with several families of self-supervised methods (Huang et al., 29 Mar 2026). RayZer and other latent-space methods are described as avoiding explicit 3D optimization and as being able to synthesize images well, but not providing transferable camera poses or explicit geometry. NAS3R is therefore characterized as more geometry-aware and more suitable for downstream 3D tasks.

It also contrasts NAS3R with prior-free or pose-free explicit 3D methods such as SelfSplat, SPFSplat, PF3plat, NoPoSplat, and AnySplat. These methods are described as often still depending on pretrained priors, GT intrinsics, pseudo-labels, or context-only supervision. NAS3R’s stated distinctions are that it removes the need for pretrained priors during training, predicts intrinsics and poses jointly, supervises on held-out target views, and uses bounded depth to stabilize training.

The comparison with AnySplat is especially explicit. The paper states that AnySplat uses exponential depth activation and applies rendering loss only on input context views, whereas NAS3R uses bounded sigmoid plus near/far depth and renders and supervises novel target views (Huang et al., 29 Mar 2026). The reported interpretation is that this leads NAS3R to generalize better and train more stably without priors.

The stated limitations are twofold. First, scalability is described as promising, but future larger-scale training may improve generalization further. Second, because 3D Gaussian Splatting is not ideal for exact surface recovery, fine-tuning with ground-truth depth still helps for the highest-quality depth reconstruction. NAS3R is therefore not presented as a complete replacement for supervised geometry when precise surface accuracy is required.

Within those limits, NAS3R’s significance lies in demonstrating that high-quality 3D reconstruction can be learned from unconstrained unlabeled multi-view data without relying on SfM or pretrained 3D priors. This suggests a transition toward a regime in which explicit geometry and usable camera estimates can be learned from broadly available image collections, while preserving compatibility with established supervised reconstruction backbones.