---
title: 'WorldMirror 2.0: Unified 3D Reconstruction'
url: https://www.emergentmind.com/topics/worldmirror-2-0
type: topic
---

# WorldMirror 2.0: Unified 3D Reconstruction

WorldMirror 2.0 is the reconstruction backbone of HY-World 2.0: a unified, feed-forward multi-view 3D prediction model that takes a set of images together with optional geometric priors and predicts a consistent geometric description of the scene, including point maps, depth, normals, camera parameters, and per-pixel 3D Gaussian Splatting attributes. Within HY-World 2.0 it serves both as a standalone world reconstruction model for multi-view images or videos and as the geometry extractor in the final “World Composition” stage, where generated keyframes are converted into geometry that can be aligned, fused, and optimized into a navigable 3DGS world [2604.14268].

## 1. System role and problem scope

WorldMirror 2.0 occupies the reconstructive component of HY-World 2.0. It is not the module that initializes panoramas, plans exploration trajectories, or synthesizes new views. Those roles are assigned to HY-Pano 2.0, WorldNav, and WorldStereo 2.0, respectively. WorldMirror 2.0 instead recovers 3D structure from existing observations, either in standalone reconstruction from multi-view images or videos, or inside the generative pipeline during world composition [2604.14268].

In the world-composition stage, the paper summarizes its role by
$$
\{\mathbf{D}_i^m,\mathbf{N}_i^m\}_{i=1}^{T'_{ex}}
=
\Phi\!\left(
\{\mathbf{V}_j,\mathbf{C}_j\}_{j=1}^{T_{pan}},
\{\mathbf{V}_i,\mathbf{C}_i\}_{i=1}^{T'_{ex}}
\right),
$$
where $\Phi(\cdot)$ is WorldMirror 2.0, $\{\mathbf{V}_j,\mathbf{C}_j\}_{j=1}^{T_{pan}}$ are perspective views subdivided from the input panorama and their cameras, and $\{\mathbf{V}_i,\mathbf{C}_i\}_{i=1}^{T'_{ex}}$ are the selected generated keyframes and their cameras. The outputs are per-frame depth $\mathbf{D}_i^m$ and normals $\mathbf{N}_i^m$ [2604.14268].

A central design claim is that WorldMirror 2.0 is reconstructive rather than generative. It does not hallucinate new views, and the paper does not state that it accepts text. Text conditioning exists upstream in HY-Pano 2.0, while WorldMirror 2.0 operates on multi-view observations and optional geometric priors. This distinction is important because it places the model at the interface between 2D image generation and explicit 3D scene composition rather than among multimodal image synthesizers [2604.14268].

## 2. Inputs, outputs, and scene representation

A defining feature inherited from the earlier WorldMirror formulation is “Any-Modal Tokenization,” under which all available inputs are tokenized into a common sequence. The paper states that all input modalities—images, camera poses, intrinsics, and depth maps—are tokenized into a unified sequence, and that each prior modality is dropped independently with probability $0.5$ during training so that the model can operate under arbitrary missing-modality conditions [2604.14268].

The supported inputs are multi-view RGB images, video frames treated as multiple views, camera poses, camera intrinsics, and depth maps as optional priors. The paper’s figure caption explicitly describes WorldMirror 2.0 as taking “multi-view images with optional geometric priors (camera poses, intrinsics, depth maps) as input.” The paper does not state that WorldMirror 2.0 accepts text; that functionality belongs to other HY-World 2.0 modules [2604.14268].

The model is multi-task. It predicts dense point maps or point clouds, multi-view depth maps, surface normals, camera parameters, and pixel-wise 3D Gaussian Splatting attributes. In WorldMirror 2.0 an additional depth mask head is added for invalid-pixel modeling. The paper distinguishes between primary geometric outputs—point maps, depth, normals, and camera estimation—and 3DGS-related outputs. In the broader HY-World 2.0 pipeline, the final world is not produced by directly taking the feed-forward Gaussian prediction as the final scene; instead, WorldMirror 2.0 provides geometry that initializes and supervises a separate 3DGS optimization stage [2604.14268].

The treatment of video is also explicit: frames are handled as multi-view observations. The paper does not describe a dedicated temporal module, temporal loss, or recurrent memory. Temporal consistency is therefore implicit in multi-view processing rather than enforced by a video-specific mechanism. Likewise, the paper does not describe an explicit occlusion module inside WorldMirror 2.0; occlusion handling is partly absorbed by improved invalid-pixel prediction and by downstream alignment filters in world composition [2604.14268].

## 3. Architecture and the transition from WorldMirror 1.0

WorldMirror 2.0 uses a Transformer backbone with global-local attention mechanisms, inherited from WorldMirror 1.0, followed by task-specific DPT decoder heads. A major architectural change is the replacement of standard absolute-index 2D RoPE with normalized RoPE. For a patch grid $H_p\times W_p$, with $H_p=H/p$ and $W_p=W/p$, normalized patch coordinates are defined as
$$
\hat{x}_i=\frac{2i+1}{H_p}-1,
\qquad
\hat{y}_j=\frac{2j+1}{W_p}-1,
$$
with $\hat{x}_i,\hat{y}_j\in[-1,1]$. The stated purpose is to map all resolutions into the same coordinate range so that test-time extrapolation becomes interpolation [2604.14268].

The paper motivates the 2.0 upgrade by three shortcomings of WorldMirror 1.0: poor generalization to inference resolutions different from training resolution, limited geometric consistency between depth and normals, and memory or latency bottlenecks when scaling to many views. WorldMirror 2.0 addresses these through positional encoding changes, explicit geometric coupling, improved invalid-pixel handling, revised data, a new curriculum, and distributed inference strategies [2604.14268].

The main differences are concise enough to tabulate.

| Aspect | WorldMirror 1.0 | WorldMirror 2.0 |
|---|---|---|
| Position encoding | absolute RoPE | normalized RoPE |
| Depth supervision | GT depth only | GT depth + GT/pseudo normal through explicit depth-to-normal coupling |
| Invalid-pixel modeling | confidence only | confidence + dedicated depth-mask head |
| Acceleration | none | token/frame sequence parallelism, BF16, FSDP |
| Data | open-source datasets | plus internal Unreal Engine synthetic renderings |
| Pseudo-label enhancement | not normal-focused | pseudo normal labels added |
| Training strategy | independent sampling of view count and resolution; 2-stage curriculum | token-budget dynamic batching; 3-stage curriculum; broader resolution range |

The model preserves the shared-backbone, multi-head philosophy of the earlier “WorldMirror: Universal 3D World Reconstruction with Any-Prior Prompting,” which is described as an all-in-one, feed-forward model for versatile 3D geometric prediction tasks that flexibly integrates camera poses, intrinsics, and depth maps while jointly predicting dense point clouds, multi-view depth maps, camera parameters, surface normals, and 3D Gaussians [2510.10726]. WorldMirror 2.0 specializes that prior-aware formulation for the HY-World 2.0 stack and adds explicit machinery for resolution robustness, depth-normal consistency, and scalable multi-view inference [2604.14268].

A practical misconception addressed by the paper is that WorldMirror 2.0 might behave like WorldStereo 2.0. It does not. The paper explicitly states that it has no memory bank, retrieval module, or temporal recurrence; its consistency derives from jointly processing all views in one Transformer pass, aided by optional geometric priors [2604.14268].

## 4. Learning strategy, geometric coupling, and invalid-pixel modeling

WorldMirror 2.0 uses a three-stage curriculum. In Stage 1, all geometry heads are trained using native annotations, without pseudo-label enhancement and without depth-to-normal loss. In Stage 2, the depth-to-normal loss is introduced and the synthetic-data proportion is increased. In Stage 3, the backbone and geometry heads are frozen, and only the 3DGS head initialized from the depth head weights is trained. This extends the two-stage strategy of WorldMirror 1.0 into a geometry-first, geometry-coupled, then 3DGS-specialized schedule [2604.14268].

The paper also introduces token-budget dynamic batching. For a GPU token budget $T_{\max}$, after sampling image resolution and patch size, the per-image token count is
$$
t=\frac{H}{p}\times\frac{W}{p}.
$$
The maximum number of views is then
$$
N_{\max}=\min\!\left(N_{\text{cap}},\left\lfloor\frac{T_{\max}}{t}\right\rfloor\right),
$$
and the total per-GPU token budget is constrained by
$$
T_{\text{total}}=N\times\frac{H}{p}\times\frac{W}{p}\le T_{\max}.
$$
This permits many low-resolution views or fewer high-resolution views under a fixed token budget [2604.14268].

One of the central additions is explicit depth-to-normal coupling. Given predicted depth $\hat{\mathbf{D}}_i$ and intrinsics $\mathbf{K}$, the model back-projects into 3D as
$$
\mathbf{P}_i=\mathbf{K}^{-1}\hat{\mathbf{D}}_i\cdot [u,v,1]^\top,
$$
and derives a normal by
$$
\tilde{\mathbf{N}}_i(x)=\text{normalize}\!\left(
\frac{\partial \mathbf{P}_i}{\partial u}\times
\frac{\partial \mathbf{P}_i}{\partial v}
\right).
$$
The angular supervision loss is
$$
\mathcal{L}_{\text{d2n}}
=
\frac{1}{|\mathcal{V}|}
\sum_{x\in\mathcal{V}}
\arccos\!\left(
\frac{\tilde{\mathbf{N}}_i(x)\cdot \hat{\mathbf{N}}_i(x)}
{\|\tilde{\mathbf{N}}_i(x)\|\,\|\hat{\mathbf{N}}_i(x)\|}
\right),
$$
with targets derived from GT depth in synthetic data and from pseudo normals produced by a monocular normal teacher on real data. The paper argues against pseudo-depth labels for multi-view training because per-view pseudo-depth is often globally inconsistent, whereas normals encode local orientation more robustly [2604.14268].

Invalid depth pixels are handled by a dedicated depth mask head. It predicts a per-pixel validity logit $\hat{m}(x)$ with BCE supervision:
$$
\mathcal{L}_{\text{mask}}
=
-\frac{1}{|\mathcal{M}|}
\sum_{x\in\mathcal{M}}
\left[
m^*(x)\log \sigma(\hat{m}(x))
+
(1-m^*(x))\log(1-\sigma(\hat{m}(x)))
\right].
$$
This makes invalid-pixel prediction explicit rather than relying only on confidence weighting, which was the WorldMirror 1.0 design [2604.14268].

The training data combine public multi-view reconstruction datasets, high-quality synthetic Unreal Engine scenes, and pseudo normals on real data. The paper does not specify an epipolar-attention mechanism, explicit reprojection loss, or bundle-adjustment-style optimization inside WorldMirror 2.0. Its consistency is learned from multi-view supervision and shared Transformer processing rather than from iterative geometric optimization at inference time [2604.14268].

## 5. World composition, depth alignment, and integration with 3DGS

Inside HY-World 2.0, world composition begins from a panorama $\mathbf{I}^{pan}$, a panoramic point cloud $\mathbf{P}^{pan}$, generated keyframes $\{\mathbf{V}_i\}_{i=1}^{T_{ex}}$, and corresponding cameras $\{\mathbf{C}_i\}_{i=1}^{T_{ex}}$. A subset of generated frames is sampled, and WorldMirror 2.0 predicts depth and normals for them using both panorama-derived perspective views and selected generated views. The next step is alignment, because WorldMirror depth has scale ambiguity and may not match the panoramic point-cloud frame [2604.14268].

The paper renders the panoramic point cloud into each target camera to obtain guidance depth $\mathbf{D}_i^g$, then aligns predicted depth $\mathbf{D}_i^m$ by
$$
\mathbf{D}_i^a=\varphi_{align}(\mathbf{D}_i^m,\mathbf{D}_i^g,\mathbf{M}_i),
$$
where the valid mask is
$$
\mathbf{M}_i
=
\mathbf{M}_i^m
\cap
\mathbf{M}_i^g
\cap
\mathbf{M}_i^n
\cap
\mathbf{M}_i^p
\cap
\overline{\mathbf{M}_i^{sky}}.
$$
Here $\mathbf{M}_i^m$ denotes valid WorldMirror regions, $\mathbf{M}_i^g$ valid panorama-guidance regions, $\mathbf{M}_i^n$ normal-consistent regions, $\mathbf{M}_i^p$ a percentile-based depth-discrepancy filter, and $\overline{\mathbf{M}_i^{sky}}$ the non-sky mask [2604.14268].

The aligned depth is modeled as a per-frame linear transform,
$$
\mathbf{D}_i^a=\gamma_i\mathbf{D}_i^m+\beta_i,
$$
implemented in disparity space in practice. To reject poor alignments, the method uses anchor values $A_q$ with $Q=9$, computes transformed anchors $\mathcal{V}_{i,q}=\gamma_i A_q+\beta_i$, aggregates medians across frames, and replaces outlier coefficients when the maximum relative deviation exceeds the 90th percentile. This outlier-rejection procedure is a notable component of the system’s synthetic-to-world alignment logic [2604.14268].

After alignment, the depths are back-projected to obtain an extension point cloud $\mathbf{P}^{ex}$, and the union with panorama points is voxel-downsampled:
$$
\tilde{\mathbf{P}}
=
\text{VoxelDownsample}\!\left(\mathbf{P}^{pan}\cup \mathbf{P}^{ex}\right).
$$
This initializes the final 3DGS scene. Each Gaussian has opacity $\sigma_k\in[0,1]$, center $\boldsymbol{\mu}_k\in\mathbb{R}^{3\times1}$, covariance $\mathbf{\Sigma}_k\in\mathbb{R}^{3\times3}$, and color $\mathbf{c}_k\in\mathbb{R}^3$, with
$$
\mathbf{\Sigma}_k=\mathbf{R}_k\mathbf{S}_k\mathbf{S}_k^T\mathbf{R}_k^T.
$$
The final renderer is optimized with photometric and geometric losses, and the geometric term explicitly uses aligned WorldMirror depth and normals:
$$
\mathcal{L}_{\text{geo}}
=
\lambda_d\,\mathcal{L}_1(\hat{\mathbf{D}}_i,\mathbf{D}_i^a)
+
\lambda_n\,\bigl(1-\cos(\hat{\mathbf{N}}_i,\mathbf{N}_i)\bigr).
$$
In this sense, WorldMirror 2.0 is not merely a predictor of geometric intermediates; it also supplies initialization geometry, aligned supervision, and stabilization signals for the downstream 3DGS optimization [2604.14268].

## 6. Empirical profile, scaling behavior, and limitations

The paper attributes strong benchmark gains to WorldMirror 2.0. On point-map reconstruction across 7-Scenes, NRGBD, and DTU, it improves over WorldMirror 1.0 at all resolutions and especially resolves the high-resolution collapse of the earlier model. Reported examples include a 7-Scenes medium accuracy mean improvement from **0.043 → 0.033**, a 7-Scenes high accuracy mean improvement from **0.079 → 0.037**, a 7-Scenes high + all priors result of **0.012**, and a DTU high + all priors result of **0.554 acc mean / 0.771 comp mean** [2604.14268].

On RealEstate10K, the reported camera AUC@30 improves from **66.29 to 86.89** at high resolution; depth AbsRel improves from **0.195 to 0.162** at high resolution; and $\delta<1.25$ reaches **0.815** at high resolution. On novel-view synthesis, WorldMirror 1.0 is reported to collapse at high resolution, with PSNR **21.34 (M) → 17.78 (H)**, whereas WorldMirror 2.0 remains stable at **20.14 / 20.07 / 19.98** across low, medium, and high resolution, with high-resolution SSIM reaching **0.726**. On surface normals, the paper reports ScanNet mean error **12.3 (M), 12.5 (H)** for WorldMirror 2.0, while WorldMirror 1.0 degrades from **13.8 (M) to 17.6 (H)** [2604.14268].

The scaling strategy is equally prominent. The model is evaluated up to **256 views** and uses token-level sequence parallelism in the Transformer, frame-level sequence parallelism in DPT decoders, BF16 mixed precision for most parameters, and FSDP sharding. At **\(518\times378\)** on NVIDIA H20, the paper reports that a baseline FP32 single GPU is OOM at 256 views, whereas full SP + BF16 + FSDP on 4 GPUs runs **128 views in 5.60 s, 42.71 GB/GPU** and **256 views in 17.52 s, 78.78 GB/GPU**. In the full HY-World 2.0 pipeline, the “Recon and Align” stage, which includes WorldMirror 2.0 plus depth alignment, takes **102 s**. The paper also compares its linear alignment pipeline against video2world and claims similar quality with **less than 2 minutes** runtime versus **about 5 hours per scene** for feature-matched ICP in video2world [2604.14268].

The limitations are also explicit. WorldMirror 2.0 is not explicitly tailored for panoramic reconstruction. Its predicted depth has scale ambiguity and must be aligned to panorama coordinates. The paper states that it still struggles in highly challenging outdoor scenes, that bad alignments can occur when guidance masks are sparse or trajectories are difficult, and that it has no explicit temporal memory or recurrent mechanism. It also does not provide exact loss weights, optimizer settings, or internal 3DGS-head parameterization in the reported text. These limitations are consistent with its design emphasis: a feed-forward, prior-aware reconstruction engine that improves substantially over the prior WorldMirror formulation, but that still relies on downstream alignment and optimization to produce the final world representation [2604.14268].

Source: https://www.emergentmind.com/topics/worldmirror-2-0