WorldRecon Framework

• WorldRecon reconstructs large-scale indoor scenes by combining approximate camera and point-cloud estimation with block-wise generative reconstruction and neighbour-informed latent inpainting.
• It uses VGGT-Ω as a global geometric scaffold and a modified TRELLIS.2 generator to create spatially coherent scene blocks, yielding unified meshes rather than isolated depth maps or point clouds.
• WorldRecon produces high-quality, physically based rendered (PBR) scene meshes by progressively generating and blending overlapping blocks with a neighbour-aware latent inpainting, Overlapping regions readjust seamlessly using multi-view image and camera conditioning and propagation.

WorldRecon is a generative multi-view reconstruction framework for recovering complete, large-scale indoor-scene meshes from unposed, unordered, sparse, and heterogeneous RGB images. It combines approximate camera and point-cloud estimation with block-wise generative reconstruction, neighbour-aware latent inpainting, and causal propagation of structure, geometry, and physically based materials. The framework uses VGGT-$\Omega$ as a global geometric scaffold and a modified TRELLIS.2 generator to produce spatially coherent scene blocks, yielding unified meshes rather than isolated depth maps, point clouds, Gaussian primitives, or independently generated objects (Zhang et al., 13 Oct 2026).

1. Problem formulation and design objectives

WorldRecon addresses the reconstruction of large indoor environments from an arbitrary number of RGB images whose camera poses and, potentially, intrinsics are unknown. The input is represented as

$\mathcal I=\{I^n\}_{n=1}^{N},$

where the images may be unordered, sparsely distributed, unevenly sampled, and captured by heterogeneous sources, including synthetic images, real scans, and generated-video frames.

The desired output is a scene-level mesh

$\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$

where $\mathcal V$ and $\mathcal F$ denote vertices and faces, while $\mathcal A$ contains appearance and physically based rendering attributes such as materials and textures. The framework also evaluates derived depth and surface-normal renderings.

The reconstruction problem contains several coupled difficulties. Unknown camera parameters create pose ambiguity; incomplete visibility leaves surfaces unobserved; large scenes exceed the practical capacity of a single high-resolution latent representation; and generative completion must infer plausible geometry and materials without contradicting visible evidence. Independently generated spatial regions can additionally exhibit inconsistent wall heights, broken floors, duplicated objects, or mismatched textures.

WorldRecon therefore separates global grounding from local generative reconstruction. A reconstruction model estimates an approximate global layout, camera configuration, and point cloud. This scaffold is not treated as a final geometric solution. Instead, it provides spatial constraints for a generative model that denoises, completes, and refines local scene regions. The framework’s central principle is that imperfect geometric estimates can remain useful when incorporated as conditioning rather than imposed as immutable ground truth.

2. Global reconstruction and spatial decomposition

Global scaffold generation

Given the input images, VGGT-$\Omega$ predicts for each image:

• a depth map $\widehat D^n$;
• a confidence map $\widehat C^n$;
• camera intrinsics $\widehat K^n$;
• camera extrinsics $\mathcal I=\{I^n\}_{n=1}^{N},$0.

Depth pixels are unprojected into world-space point maps. Points with insufficient confidence are discarded, statistical outliers are removed, and connected-component filtering retains the largest component. The resulting global point cloud is

$\mathcal I=\{I^n\}_{n=1}^{N},$1

The point cloud supplies scene extent, approximate spatial arrangement, and coarse geometric grounding. It is deliberately regarded as noisy and incomplete, allowing the generative stages to reconstruct surfaces that are sparse, missing, or geometrically imperfect in the initial scaffold.

The scene is transformed into a canonical coordinate system in which gravity is aligned with $\mathcal I=\{I^n\}_{n=1}^{N},$2, dominant wall directions are aligned with $\mathcal I=\{I^n\}_{n=1}^{N},$3 and $\mathcal I=\{I^n\}_{n=1}^{N},$4, and the scene is rescaled to a fixed vertical extent of $\mathcal I=\{I^n\}_{n=1}^{N},$5 m at inference. Global similarity ambiguity is handled during evaluation with Horn $\mathcal I=\{I^n\}_{n=1}^{N},$6 alignment. Gravity is estimated using PCA axes and horizontal surface normals, followed by dominant wall-normal voting for yaw alignment.

Block partitioning

The aligned $\mathcal I=\{I^n\}_{n=1}^{N},$7 bounding box of $\mathcal I=\{I^n\}_{n=1}^{N},$8 is divided into cubic spatial blocks of side length

$\mathcal I=\{I^n\}_{n=1}^{N},$9

Adjacent blocks have a stride of $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$0 m and therefore overlap by $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$1 m, or $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$2. Blocks share a vertical centre and cover the full vertical extent. The described implementation is consequently a one-layer, single-story representation.

For block $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$3, $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$4 denotes its world-space cube and $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$5 the corresponding point-cloud crop. Blocks with fewer than 100 reconstructed points are discarded. The remaining crop is subsampled to 8192 points using uniform subsampling followed by farthest-point sampling. A point $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$6 is mapped into block-canonical coordinates by

$\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$7

where $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$8 is the block centre. Each block consequently occupies $\mathcal M=(\mathcal V,\mathcal F,\mathcal A),$9.

Selection of relevant image observations

VGGT-$\mathcal V$0 preserves pixel-to-3D correspondences through its predicted depth maps. An image is considered a candidate observation for block $\mathcal V$1 when at least 15% of its pixels project inside that block.

Candidate views are voxelized at resolution $\mathcal V$2. Let $\mathcal V$3 denote the occupied voxel set of the block crop and $\mathcal V$4 the occupied voxels visible in candidate view $\mathcal V$5. Up to four views are selected to maximize joint voxel coverage:

$\mathcal V$6

For every selected view, only pixels whose reconstructed 3D points lie inside $\mathcal V$7 are retained. These pixels form block-local RGBA images, with the alpha channel encoding an in-block visibility mask. View selection is therefore based on complementary spatial coverage rather than simply camera proximity or image count.

Causal block ordering

Blocks are generated progressively. A seed block is selected using voxel occupancy, after which a connected frontier is expanded greedily. Each newly generated block has at most two previously generated four-connected neighbours. In the two-neighbour case, the neighbours form an L-shaped configuration, with one neighbour along the $\mathcal V$8 direction and one along $\mathcal V$9.

This ordering defines the context available to each block and avoids unconstrained simultaneous generation. It also establishes the one-neighbour and two-neighbour configurations used for training the latent-inpainting models.

3. TRELLIS.2-based generative reconstruction

WorldRecon modifies TRELLIS.2, a generative 3D model based on the O-Voxel structured sparse representation

$\mathcal F$0

where $\mathcal F$1 is the number of active voxels, $\mathcal F$2 is the position of active voxel $\mathcal F$3, and $\mathcal F$4 and $\mathcal F$5 are geometric and material features.

Generation proceeds through three cascaded flow-matching DiT stages:

1. Sparse Structure (SS): predicts active voxel positions and structural occupancy.
2. Geometry: predicts geometry latents aligned with the active structure.
3. Material: predicts material latents aligned with the geometry latents.

The corresponding latent variables are

$\mathcal F$6

The decoders transform these latent variables into sparse structure, geometric features, and material features, which are ultimately decoded into a mesh with PBR materials. WorldRecon starts from a $\mathcal F$7-resolution TRELLIS.2 model and fine-tunes it from object-level generation to block-level scene generation.

Multi-view image and camera conditioning

The original single-image conditioning is replaced by a joint representation of multiple images and their camera parameters. Each block image is encoded with a frozen DINOv3 ViT-L/16 patch encoder at $\mathcal F$8 resolution. For view $\mathcal F$9, its image patch tokens are denoted by $\mathcal A$0.

Camera geometry is represented by five Plücker rays sampled at the image centre and four corners. For pixel coordinate $\mathcal A$1, intrinsics $\mathcal A$2, camera-to-world rotation $\mathcal A$3, and camera centre $\mathcal A$4, the normalized ray is

$\mathcal A$5

and the corresponding Plücker coordinate is

$\mathcal A$6

A two-layer MLP maps these camera descriptors to camera tokens $\mathcal A$7, which are prepended to the image tokens:

$\mathcal A$8

Learnable query tokens fuse the variable number of views into a fixed-length condition:

$\mathcal A$9

with the final multi-view representation defined as

$\Omega$0

This construction supplies both visual evidence and camera awareness without directly projecting image features into a pose-dependent 3D feature volume. It is intended to be more tolerant of imperfect camera estimates than explicit geometric feature projection.

Point-cloud conditioning

The sparse-structure stage additionally receives the local point cloud

$\Omega$1

Normalized point coordinates are encoded using Fourier positional features $\Omega$2, passed through a learnable projection $\Omega$3, and concatenated with ten learned modality tokens:

$\Omega$4

The point-cloud and multi-view streams share a 1024-dimensional feature space. The complete point-cloud stream is dropped with probability $\Omega$5, enabling classifier-free guidance and point-free inference behaviour.

The sparse-structure condition is

$\Omega$6

whereas geometry and material stages use the multi-view condition together with their native preceding-stage conditions. All conditioning is injected into the DiT through cross-attention layers.

4. Neighbour-aware latent inpainting and context propagation

Latent inpainting

Independent block generation would permit discontinuous walls and floors, incompatible surface heights, duplicated objects, and inconsistent materials. WorldRecon therefore trains inpainting versions of all three TRELLIS.2 denoisers.

Let $\Omega$7 be a noisy latent at flow time $\Omega$8 for stage

$\Omega$9

Let $\widehat D^n$0 be a binary mask, with $\widehat D^n$1 denoting known latent values supplied by previously generated neighbours and $\widehat D^n$2 denoting the region to be generated. The known clean latent is

$\widehat D^n$3

The inpainting input is formed by concatenating the native latent, the mask, and the masked known latent:

$\widehat D^n$4

For the material stage, geometry is also provided as a condition:

$\widehat D^n$5

The additional inpainting channels are zero-initialized, while the original input projection is inherited from the base generator. The initial behaviour consequently remains close to the pretrained generator, with neighbour conditioning learned during fine-tuning.

Training masks reproduce inference conditions by representing one or two generated neighbours. Two-neighbour masks form an L-shape. Known edge margins are sampled between $\widehat D^n$6 and $\widehat D^n$7. At inference, the raw overlap is $\widehat D^n$8, but only $\widehat D^n$9 is treated as fixed context; the remaining $\widehat C^n$0 forms an editable transition band.

Sparse-structure propagation

WorldRecon maintains shared world-space sparse-structure latents and occupancy tensors. The seed block is sampled normally. Subsequent blocks are inpainted using overlap latents from previously generated neighbours.

At flow time $\widehat C^n$1, the known overlap follows its forward-flow trajectory:

$\widehat C^n$2

where $\widehat C^n$3 is Gaussian noise. If $\widehat C^n$4 is the inpainter prediction, the state is clamped by

$\widehat C^n$5

The resulting clean latent is decoded into an occupancy grid and written only to previously unoccupied world locations. Later blocks therefore cannot overwrite established sparse structure.

Geometry and material blending

Once the world sparse structure is complete, geometry and material stages are processed in the same causal order. Geometry is generated or inpainted first, followed by material generation.

Rather than copying overlap values directly, WorldRecon blends clean block latents into shared world latents using distance-based feathering. Let $\widehat C^n$6 be a world latent-grid coordinate, $\widehat C^n$7 its block-local coordinate, $\widehat C^n$8 the latent resolution per axis, and $\widehat C^n$9 the set of overlapping axes. For each overlapping axis, the feather weight is

$\widehat K^n$0

For stage $\widehat K^n$1, let $\widehat K^n$2 and $\widehat K^n$3 be the accumulated world latent and weight before integrating block $\widehat K^n$4, and let $\widehat K^n$5 be the current block latent. The update is

$\widehat K^n$6

$\widehat K^n$7

This process propagates sparse structure through occupancy and clamping, geometry through overlap inpainting and latent blending, and materials through material-latent inpainting and the same blending mechanism. Local geometry and material latents are finally decoded into block meshes, transformed into world coordinates, and merged sequentially.

5. Reconstruction outputs and scalability

WorldRecon produces unified scene meshes with detailed geometry and PBR materials. Its output differs from representations that remain exclusively as depth maps, point clouds, Gaussian primitives, or independently generated object meshes. The explicit mesh representation supports derived depth and normal rendering and permits evaluation after block merging.

The framework scales to large scenes by restricting generative computation to local $\widehat K^n$8-m blocks rather than applying a single high-resolution generator to the entire environment. Overlapping blocks provide local context, while causal ordering and latent blending maintain scene-level consistency. The block process also allows sparse geometric scaffolding to guide generation only where sufficient evidence exists; blocks with fewer than 100 reconstructed points are discarded.

The current implementation has important structural constraints. Blocks share a vertical centre and cover the full vertical extent, making the reported representation effectively single-story and one-layer. The available description does not establish support for multi-floor scenes, arbitrary vertical partitioning, dynamic objects, temporal updates, or online reconstruction. It also does not report a full computational complexity analysis, memory scaling law, or runtime breakdown for global reconstruction, view selection, block generation, latent blending, mesh extraction, and merging.

Because the model performs generative completion, unobserved surfaces are inferred rather than directly measured. The global point cloud provides spatial grounding but does not guarantee metric correctness. A plausible completion can therefore remain geometrically uncertain where observations are sparse or absent. This distinction is fundamental: WorldRecon’s output is a generatively completed scene mesh, not a complete measurement of every physical surface.

6. Evaluation, reported performance, and limitations

The abstract reports that WorldRecon achieves state-of-the-art reconstruction performance. Relative to the strongest baselines, it reduces Chamfer distance by $\widehat K^n$9 and $\mathcal I=\{I^n\}_{n=1}^{N},$00, while improving F-score by $\mathcal I=\{I^n\}_{n=1}^{N},$01 and $\mathcal I=\{I^n\}_{n=1}^{N},$02 on synthetic and real-world indoor scenes, respectively (Zhang et al., 13 Oct 2026).

The framework’s reported evaluation emphasizes mesh reconstruction quality and generalization between synthetic and real indoor scenes. The supplied description does not provide the complete benchmark names, dataset sizes, baseline table, per-scene metrics, training schedule, inference latency, or the exact evaluation protocol. Those details are therefore not specified here.

The principal technical limitations follow from the described method:

• Dependence on global reconstruction quality: VGGT-$\mathcal I=\{I^n\}_{n=1}^{N},$03 must provide sufficiently coherent depth, confidence, intrinsics, extrinsics, and point correspondences for block selection and spatial grounding.
• Generative ambiguity: sparsely observed or unobserved geometry and materials are completed by the generative prior and may not be uniquely determined by the input images.
• Cross-block consistency: neighbour-aware inpainting and latent blending mitigate seams, but they do not establish exact physical consistency in every overlap.
• Single-story structure: the described block layout uses a shared vertical centre and full vertical coverage, limiting the directly specified representation to a one-layer scene decomposition.
• Static-scene assumption: no dynamic-object or temporal-consistency mechanism is specified.
• Metric uncertainty: canonical alignment, fixed vertical scaling, and evaluation-time $\mathcal I=\{I^n\}_{n=1}^{N},$04 alignment accommodate global ambiguity but do not by themselves guarantee absolute metric accuracy.
• Limited explicit uncertainty: confidence is used in global point-cloud filtering, but the generated mesh is not described as carrying per-surface posterior uncertainty or observation-versus-completion labels.
• Unspecified implementation details: the supplied description does not state the complete flow-matching losses, optimizer settings, training data composition, computational cost, or mesh-merging implementation.

WorldRecon’s significance lies in its combination of approximate geometric grounding and generative local reconstruction. Earlier reconstruction pipelines can fail when camera estimates are imperfect, images are sparse, or large portions of a scene are unseen; purely generative scene synthesis can produce locally plausible but globally inconsistent regions. WorldRecon addresses these tensions by using a noisy global point cloud to establish spatial support, selecting complementary block views, generating blocks causally, and propagating overlap information through latent inpainting and world-space blending.

The resulting design can be summarized as

$\mathcal I=\{I^n\}_{n=1}^{N},$05

This architecture positions WorldRecon as a large-scale generative reconstruction system rather than a conventional photogrammetric pipeline. Its central contribution is the integration of coarse global geometry with block-wise generative completion while preserving scene-level coherence across spatially overlapping regions.