PixWorld: Pixel-Space Diffusion for 3D Scenes

• PixWorld is a unified pixel-space diffusion framework for 3D scene modeling that directly supervises rendered images of predicted 3D representations.
• It overcomes latent-space bottlenecks by applying MSE, LPIPS, and a geometry perception loss to guide both reconstruction and generation.
• Empirical results on datasets like RealEstate10K and DL3DV-10K show state-of-the-art performance in reconstruction and competitive results in conditional generation.

Searching arXiv for the specified topic and related disambiguations. PixWorld is a unified pixel-space diffusion framework for 3D scene modeling that “models 3D scenes directly from posed multi-view images, solving both reconstruction and generation with a single model” (Gao et al., 6 Jul 2026). It reformulates two tasks that are commonly treated by separate paradigms—pixel-based regression for reconstruction and latent diffusion for generation—under a single pixel-space objective supervised on rendered images of a predicted 3D representation rather than on latent features (Gao et al., 6 Jul 2026). In the literature surrounding similarly named systems, PixWorld should be distinguished from PicWorld, a benchmark for implicit world knowledge reasoning in text-to-image models that is “sometimes misread as ‘PixWorld’” (Han et al., 23 Nov 2025), and from PixelWorld, a benchmark for the “Perceive Everything as Pixels (PEAP)” paradigm in unified vision-language perception (Lyu et al., 31 Jan 2025).

1. Definition and problem setting

PixWorld is presented as an alternative to latent-space unification for 3D reconstruction and generation. Prior latent-space approaches define the generative objective on the latent features of a pretrained VAE or RAE and then decode those latents into 3D, which introduces an information bottleneck and makes the optimization target indirect with respect to the underlying 3D representation (Gao et al., 6 Jul 2026). PixWorld instead “supervises diffusion directly in pixel space on differentiable renderings of a predicted 3D scene representation,” thereby removing dependence on pretrained VAEs or RAEs and aligning optimization with “the actual fidelity of the underlying 3D scene” (Gao et al., 6 Jul 2026).

The motivating claim is not merely architectural simplification. The paper argues that when diffusion is defined on intermediate latents, gradients shape latent distributions rather than directly correcting geometry and appearance in 3D (Gao et al., 6 Jul 2026). PixWorld addresses this by allowing losses on rendered multi-view images to back-propagate through a differentiable renderer into the scene representation itself (Gao et al., 6 Jul 2026). A central consequence is that a single model can perform reconstruction when all inputs are clean and generation when some views are noisy, without changing the overall mechanism (Gao et al., 6 Jul 2026).

This framing places PixWorld at the intersection of feed-forward multi-view reconstruction, diffusion-based generation, and 3D-aware rendering. The paper’s summary states that PixWorld “consistently outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods,” which the authors interpret as evidence for the superiority of the unified pixel-space approach (Gao et al., 6 Jul 2026).

2. Model formulation and scene representation

The core scene representation in PixWorld is a pixel-aligned 3D Gaussian Splatting representation (Gao et al., 6 Jul 2026). For each input view $n$ and pixel $p$, the model predicts a depth value and then unprojects that pixel into a 3D Gaussian center in world coordinates using the camera parameters. The paper gives the unprojection formula as

$\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$

where $\Pi^{-1}$ is the inverse camera projection using the intrinsics and extrinsics in $\mathcal{T}^n$ (Gao et al., 6 Jul 2026). Aggregating the per-pixel Gaussians across views yields the scene-level 3DGS $\hat{\mathcal{G}}$, which is then rendered differentiably under arbitrary cameras (Gao et al., 6 Jul 2026).

The denoiser $f_\theta$ is a two-stream DiT. Clean inputs and noisy inputs are embedded by separate patchify projections and passed through shared transformer blocks, with the noisy stream conditioned on the sampled diffusion timestep $t$ and the clean stream receiving the fully denoised time embedding $t=1$ (Gao et al., 6 Jul 2026). Cameras are injected through PRoPE, described as “Cameras as Relative Positional Encoding,” and optional text conditioning is fused via cross-attention (Gao et al., 6 Jul 2026). The model mapping is written as

$$f_\theta : (\tilde{\mathcal{I}_t,\mathcal{T},y})\mapsto (\hat{\mathcal{D}},\hat{\mathcal{G}}),$$

where $p$0 denotes predicted depth maps and $p$1 the pixel-aligned 3DGS (Gao et al., 6 Jul 2026).

The architecture details given in the paper are specific. PixWorld uses a 24-layer DiT with hidden width $p$2, 16 attention heads, head dimension 64, SwiGLU feed-forward layers, RMSNorm on $p$3, and adaLN-Zero timestep modulation (Gao et al., 6 Jul 2026). Inputs use $p$4 patchify with learnable positional embeddings, and outputs are per-pixel depth and 3D-Gaussian attributes via stream-specific heads (Gao et al., 6 Jul 2026). The total trainable parameter count is approximately $p$5B (Gao et al., 6 Jul 2026).

A plausible implication is that PixWorld is not only a unification of tasks but also a unification of conditioning regimes: geometry enters through camera-aware multi-view inputs, while appearance and optional semantic control can be mediated through rendered supervision and cross-attended text (Gao et al., 6 Jul 2026).

3. Pixel-space diffusion and training objectives

PixWorld adopts pixel-space flow matching “in the spirit of ‘Back to basics: Let denoising generative models denoise’” (Gao et al., 6 Jul 2026). For a clean image $p$6 and timestep $p$7, a noisy input is created by linear interpolation,

$p$8

with $p$9 (Gao et al., 6 Jul 2026). The denoiser is parameterized directly as an image predictor,

$\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$0

and the predicted image induces a velocity field $\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$1, yielding the flow-matching objective

$\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$2

with $\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$3 (Gao et al., 6 Jul 2026).

For multi-view scenes, the view indices are partitioned into clean and noisy subsets,

$\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$4

and the per-view mixed inputs are defined by

$\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$5

This formulation is what allows the same model to span pure reconstruction and conditional generation (Gao et al., 6 Jul 2026).

The total training objective combines rendering supervision, depth supervision, and a geometry perception loss. The rendering loss is

$\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$6

where clean views contribute an MSE reconstruction term and noisy views contribute the FM loss (Gao et al., 6 Jul 2026). LPIPS is gated by $\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$7, with $\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$8 in the reported implementation (Gao et al., 6 Jul 2026). The model also samples 4–8 additional novel views per iteration and supervises them by MSE and LPIPS (Gao et al., 6 Jul 2026).

Depth supervision uses pseudo-depth labels from DA3: $\mu_p^n = \Pi^{-1}(p, \hat d_p^n, \mathcal{T}^n),$9 where $\Pi^{-1}$0 is a per-pixel Huber loss in log-depth space (Gao et al., 6 Jul 2026). The full objective is

$\Pi^{-1}$1

with reported weights $\Pi^{-1}$2, $\Pi^{-1}$3, and $\Pi^{-1}$4 (Gao et al., 6 Jul 2026).

4. Geometry perception loss and unified operation

A distinguishing element of PixWorld is the geometry perception loss. The paper argues that photometric and perceptual supervision at the 2D image level lacks 3D geometric awareness, and therefore introduces a loss defined in the geometry-aware feature space of a frozen 3D foundation model such as VGGT or $\Pi^{-1}$5 (Gao et al., 6 Jul 2026). With rendered views $\Pi^{-1}$6, ground-truth views $\Pi^{-1}$7, and cameras $\Pi^{-1}$8, the extracted features are

$\Pi^{-1}$9

and the loss is the average per-location cosine distance,

$\mathcal{T}^n$0

The geometry critic $\mathcal{T}^n$1 is frozen, and gradients are stopped on the reference branch $\mathcal{T}^n$2 (Gao et al., 6 Jul 2026).

The model’s unified handling of reconstruction and generation follows directly from the partition $\mathcal{T}^n$3. In reconstruction mode, $\mathcal{T}^n$4: all views are clean, the model predicts $\mathcal{T}^n$5 and $\mathcal{T}^n$6, and rendered outputs are supervised against the observed images through MSE, LPIPS when enabled, and the depth and geometry losses (Gao et al., 6 Jul 2026). In generation mode, both clean and noisy inputs are present: at least one clean conditioning view anchors the scene, while noisy target views are denoised conditionally via the same network and rendered under desired cameras (Gao et al., 6 Jul 2026).

The procedural description in the paper is explicit. During training, each iteration samples $\mathcal{T}^n$7 posed views, partitions them into $\mathcal{T}^n$8 and $\mathcal{T}^n$9, constructs $\hat{\mathcal{G}}$0, predicts $\hat{\mathcal{G}}$1 and $\hat{\mathcal{G}}$2, renders $\hat{\mathcal{G}}$3, computes $\hat{\mathcal{G}}$4, $\hat{\mathcal{G}}$5, and $\hat{\mathcal{G}}$6, and back-propagates through the renderer (Gao et al., 6 Jul 2026). Training alternates a single-view batch of 32 images with a multi-view batch of up to 32 images, using 4–8 views across 4–8 scenes and biasing toward small $\hat{\mathcal{G}}$7 to emphasize conditioned generation while occasionally using $\hat{\mathcal{G}}$8 to ground the geometry head (Gao et al., 6 Jul 2026).

This suggests that the geometry perception term is not an auxiliary aesthetic regularizer but a structural constraint that makes the shared model viable across both tasks.

5. Datasets, protocols, and empirical results

PixWorld is trained from scratch on Re10K and DL3DV-10K, amounting to “~67K scenes,” together with “a single-image branch of 10M images from BLIP-3o as a 2D appearance prior” (Gao et al., 6 Jul 2026). Training uses resolution $\hat{\mathcal{G}}$9, AdamW with initial learning rate $f_\theta$0 linearly decayed to $f_\theta$1, EMA decay $f_\theta$2, gradient clipping at $f_\theta$3, classifier-free text drop rate $f_\theta$4, and 200K steps on 32 NVIDIA A800-SXM4-80G GPUs (Gao et al., 6 Jul 2026).

Evaluation covers RealEstate10K and DL3DV-10K with 200 test scenes each, and WorldScore with the official static split of 2000 scenes (Gao et al., 6 Jul 2026). The protocols comprise four families: reconstruction from 4 or 8 posed views; 1-view generation in First Frame and Bidirectional settings; 2-view generation in Interpolation and Extrapolation settings; and WorldScore with seven official metrics (Gao et al., 6 Jul 2026).

The reconstruction results reported in the paper are as follows:

Dataset / views	PixWorld	Comparator context
RealEstate10K, 4 views	26.21 PSNR / 0.844 SSIM / 0.138 LPIPS	YoNoSplat 25.86 / 0.841 / 0.143
RealEstate10K, 8 views	28.58 / 0.892 / 0.101	YoNoSplat 28.35 / 0.889 / 0.107
DL3DV-10K, 4 views	23.18 / 0.714 / 0.226	YoNoSplat 22.89 / 0.710 / 0.228
DL3DV-10K, 8 views	22.46 / 0.681 / 0.257	YoNoSplat 21.92 / 0.678 / 0.262

The paper states that PixWorld “achieves the best PSNR and LPIPS across settings, and the best SSIM on RealEstate10K, trailing DepthSplat on SSIM for DL3DV-10K” (Gao et al., 6 Jul 2026).

For 1-view generation, averaged over First Frame and Bidirectional, the reported RealEstate10K result is 18.88 PSNR / 0.702 SSIM / 0.325 LPIPS, with I2V Subject 0.979, I2V Background 0.978, Image Quality 0.623, Aesthetic Quality 0.556, and camera-control AUC@30 0.869 / @15 0.798 / @5 0.614 (Gao et al., 6 Jul 2026). On DL3DV-10K the corresponding result is 16.50 / 0.527 / 0.449, with I2V Subject 0.952, I2V Background 0.956, Image Quality 0.631, Aesthetic Quality 0.567, and AUC@30 0.793 / @15 0.706 / @5 0.485 (Gao et al., 6 Jul 2026). The paper states that PixWorld is “top overall” in these 1-view generation settings (Gao et al., 6 Jul 2026).

For 2-view generation, averaged over Interpolation and Extrapolation, the reported RealEstate10K result is 23.54 / 0.815 / 0.210, with I2V Subject 0.974, I2V Background 0.974, Image Quality 0.628, Aesthetic Quality 0.561, and AUC@30 0.880 / @15 0.817 / @5 0.649 (Gao et al., 6 Jul 2026). On DL3DV-10K the result is 19.37 / 0.594 / 0.340, with I2V Subject 0.950, I2V Background 0.956, Image Quality 0.607, Aesthetic Quality 0.565, and AUC@30 0.821 / @15 0.734 / @5 0.534 (Gao et al., 6 Jul 2026).

On WorldScore, PixWorld “tops the average with 71.04,” ranking first in Camera Control (91.08), 3D Consistency (91.39), and Photometric Consistency (93.84) (Gao et al., 6 Jul 2026). These results are used in the paper to support the claim that the pixel-space paradigm improves both cross-view consistency and controllability relative to prior latent-space generation baselines and remains competitive with feed-forward reconstruction baselines (Gao et al., 6 Jul 2026).

6. Ablations, efficiency, limitations, and relation to similarly named systems

The principal ablation concerns the geometry perception loss. On 1-view generation for RealEstate10K, 10K-sequence subset, after 30K steps, the full model reports 19.12 PSNR / 0.717 SSIM / 0.310 LPIPS; I2V Subject 0.972 / Background 0.975; Image Quality 0.619 / Aesthetic 0.561; and AUC@30 0.886 / @15 0.813 / @5 0.642 (Gao et al., 6 Jul 2026). Removing geometry perception yields 17.99 / 0.612 / 0.332; I2V Subject 0.973 / Background 0.974; Image Quality 0.613 / Aesthetic 0.541; and AUC@30 0.847 / @15 0.763 / @5 0.562 (Gao et al., 6 Jul 2026). The paper interprets this as confirmation that photometric and perceptual losses alone do not sufficiently constrain 3D geometry (Gao et al., 6 Jul 2026).

The reported inference speed on a single A100-SXM4-80G GPU is 15 seconds per scene with 8 key frames and NFE 100, compared with FlashWorld at 10 seconds with 24 key frames and NFE 4, Gen3C at 791 seconds with 121 key frames and NFE 70, and Gen3R at 882 seconds with 49 key frames and NFE 100 (Gao et al., 6 Jul 2026). The stated explanation is that PixWorld reconstructs a 3D representation from a small number of key frames and then renders novel views efficiently, rather than denoising a long frame sequence end-to-end (Gao et al., 6 Jul 2026). The authors note that future speedups may come from distillation and quantization (Gao et al., 6 Jul 2026).

The limitations listed in the paper are bounded and domain-specific. Evaluation is conducted on “indoor-like scenes (RealEstate10K, DL3DV-10K),” and broader generalization to “diverse outdoor/object-centric domains” is described as an open direction (Gao et al., 6 Jul 2026). Pixel-space diffusion and differentiable rendering are trained at finite resolution and compute budgets, leaving room for improvement in fine texture fidelity and higher-resolution multi-view scaling (Gao et al., 6 Jul 2026). Errors in camera parameters or ambiguous texture–geometry cues may degrade results (Gao et al., 6 Jul 2026).

Because the name “PixWorld” can be confused with other recent systems, disambiguation is important. PicWorld is “the first comprehensive benchmark that assesses the grasp of implicit world knowledge and physical causal reasoning of T2I models” and is explicitly noted as “sometimes misread as ‘PixWorld’” (Han et al., 23 Nov 2025). PixelWorld, by contrast, is a benchmark for PEAP that renders “natural-language, tabular, mathematical and diagrammatic inputs into a single pixel space” for unified vision-language evaluation (Lyu et al., 31 Jan 2025). These three works share a lexical emphasis on pixels, but they address different technical problems: 3D scene generation and reconstruction in PixWorld, text-to-image world-knowledge benchmarking in PicWorld, and modality unification in vision-language perception in PixelWorld (Gao et al., 6 Jul 2026, Han et al., 23 Nov 2025, Lyu et al., 31 Jan 2025).

Within 3D scene modeling, PixWorld is positioned against NeRF and 3DGS methods with per-scene optimization, SDS-based systems such as DreamFusion, Magic3D, and DreamGaussian, and latent diffusion approaches that learn latent-to-3D decoders (Gao et al., 6 Jul 2026). The paper’s central claim is that direct supervision on rendered images of the 3D representation avoids latent bottlenecks and strengthens scene fidelity, while the geometry perception loss supplies a structural signal that photometric losses alone do not provide (Gao et al., 6 Jul 2026).