Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

Abstract: We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.

• The paper introduces Puffin-World, a unified multimodal model that integrates physics, geometry, and appearance for 3D world modeling, scaling a dataset of 15 million vision-language-camera triplets and 1 million trajectories.
• The Omni-Camera representation combines absolute gravity-aware camera fields with relative ray geometry, improving PSNR from 19.87 to 20.54, SSIM from 0.62 to 0.67, and LPIPS from 0.23 to 0.21, and a gravity camera orientation median error to 0.79° on benchmarks, enhancing both image fidelity and camera alignment in multi-view generation.
• Puffin-World outperforms previous specialized models in camera-to-world understanding, multi-view trajectory generation, and 3D reconstruction, supporting image-to-3D and text-to-3D transformations, demonstrating close-loop camera models for mimic and self-calibrated exploration.

Problem formulation and contribution

“Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States” (2609.04196) proposes a unified multimodal architecture for camera-aware perception, viewpoint-controlled image synthesis, multi-view world generation, and 3D reconstruction. Its central claim is that a useful visual world model must represent more than appearance. Puffin-World therefore models three coupled native world states: physics, represented by gravity direction and latitude; geometry, represented by depth; and appearance, represented by RGB observations.

This formulation addresses a structural limitation in existing systems. Camera-controlled generators generally encode relative motion but lack an absolute physical frame, while camera-calibration systems estimate physical orientation but do not generate persistent multi-view worlds. Conventional unified multimodal models combine visual understanding and generation, but primarily operate over 2D semantics. Puffin-World attempts to integrate these capabilities in one model without external calibration, reconstruction, or perception modules.

The paper makes three principal contributions. First, it introduces the Omni-Camera representation, which combines absolute gravity-aware camera fields with relative ray geometry. Second, it proposes physics propagation, which transfers the absolute physical state inferred from a reference image across future camera poses. Third, it constructs Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories with large roll, pitch, and yaw variation.

Figure 1: Puffin-World jointly supports camera-to-world understanding, camera-controllable generation, native 3D world-state prediction, and 3D reconstruction.

The resulting system is intended to support both isolated tasks and closed-loop compositions. For example, the same model can infer gravity alignment from an image, generate an image under a specified camera action, predict depth for the generated view, and use the resulting view as the reference for the next generation step. This integration is the paper’s principal architectural and experimental thesis.

Native 3D world states and Omni-Camera representation

Puffin-World organizes visual world modeling hierarchically. The physics state provides global orientation relative to gravity; the geometry state describes scene structure through depth; and the appearance state encodes observable image content. The hierarchy is not merely conceptual: the model predicts and consumes these states as explicit conditioning signals.

The Omni-Camera representation is a dense nine-channel per-pixel map. It concatenates a three-channel absolute representation with a six-channel relative ray representation. The absolute component contains a pixel-wise up-vector and latitude angle derived from the gravity direction. The relative component contains the ray origin and normalized viewing direction. Consequently, the representation simultaneously captures:

• Absolute orientation: roll, pitch, gravity direction, and image-plane uprightness.
• Relative geometry: camera translation and per-pixel viewing directions.
• Camera intrinsics: principally vertical field of view, encoded through the perspective field.
• Cross-view motion: changes in ray origins and directions across views.

This design directly targets the ambiguity of relative camera conditioning. A relative trajectory specifies how the camera moves, but not whether the initial camera is upright, tilted, or inverted with respect to the physical world. Omni-Camera supplies the missing anchor. Its significance is therefore not simply that it increases the dimensionality of the camera condition; it combines coordinate systems with different invariance properties in a single spatially aligned signal.

Figure 2: The architecture combines physics perception, diffusion-based generation, geometry reconstruction, Omni-Camera conditioning, and role masks within one multimodal framework.

The network contains a geometry-aligned vision encoder, an LLM, a diffusion transformer, a condition fusion module, and a role mask. The role mask identifies whether a view is a reference, a generated target, an image-conditioned input, or a geometry target. This permits task specification through input composition rather than through separate task-specific subnetworks.

The LLM performs autoregressive understanding, including scene description and camera-parameter prediction. Its hidden states are converted through learnable queries and a six-layer connector into conditioning for the diffusion backbone. RGB and depth are encoded by the same frozen VAE. Depth is first transformed into a bounded three-channel color representation inspired by a 3D Hilbert-curve mapping, allowing the model to use the pretrained RGB latent interface for geometry without introducing a dedicated depth tokenizer or decoder.

Camera features are injected directly into the diffusion latent before patch embedding and are sparsely re-injected into deeper transformer layers. This differs from approaches that separately encode a low-dimensional camera representation and provide it through cross-attention. The proposed mechanism preserves pixel-level alignment between each latent token and its camera geometry, which is particularly important for multi-view generation and spatially varying perspective control.

Physics perception and propagation

For single images, Puffin-World predicts roll, pitch, vertical FoV, and a structured scene description through autoregressive sequence modeling. Camera parameters are generated after semantic and spatial reasoning about features such as horizons, vertical structures, and foreground composition. The paper therefore treats camera calibration as a multimodal reasoning problem rather than direct numerical regression.

The inferred reference-view gravity direction is then propagated through a relative trajectory. If $\mathbf{g}_0$ is the gravity vector in the reference camera frame and $\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$ is the relative rotation to a future view, the propagated gravity state is

$$\mathbf{g}_t=\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}\mathbf{g}_0.$$

The resulting gravity vector is converted back into an absolute perspective field for each target view and concatenated with its relative ray map. This procedure separates two sources of information: perception estimates the initial absolute state, while known camera motion transports that state across the sequence.

The distinction is important because a trajectory dataset can provide highly accurate relative poses without providing absolute orientation relative to gravity. Physics propagation imposes a persistent frame without requiring absolute labels for every future generated view. The model’s claim is therefore stronger than ordinary pose-conditioned synthesis: future images should preserve both the requested relative motion and the physical orientation inherited from the reference.

The ablation supports this mechanism, although the magnitude of the improvement is moderate. Averaged over roll, pitch, and yaw trajectories, physics propagation improves PSNR from 19.87 to 20.54, SSIM from 0.62 to 0.67, and LPIPS from 0.23 to 0.21. Mean roll error decreases from 1.94° to 1.76°, while mean pitch error decreases from 5.14° to 4.75°. The largest gains occur for roll and pitch, where gravity alignment is directly relevant; yaw shows smaller gains because it rotates around the gravity axis.

Figure 3: Physics propagation reduces horizon and camera-orientation inconsistencies in generated rotational trajectories.

These results establish that the proposed physical state is not only a visualization device. Within the reported experiments, it improves both image fidelity and camera alignment. However, the ablation does not isolate all sources of error in the absolute camera estimator: the same Puffin-World branch is used to recover camera parameters from generated frames, so the evaluation remains partly coupled to the model’s own perception subsystem.

Puffin-16M and the motion distribution

The data contribution is designed around the paper’s diagnosis that existing 3D datasets underrepresent absolute orientation and challenging camera rotation. Puffin-Cam-15M expands panoramic imagery into approximately 15 million perspective views with captions, camera parameters, diverse resolutions, and aspect ratios. Roll and pitch are sampled from $[-45^\circ,45^\circ]$, while vertical FoV is sampled from $[20^\circ,105^\circ]$.

Puffin-Traj-1M contains one million trajectories with rotations sampled over broad ranges, including full yaw coverage from $0^\circ$ to $360^\circ$. The trajectories include continuous look-up, look-down, clockwise and counterclockwise rotation, and full surrounding-view exploration. This distribution differs substantially from conventional sequential datasets, where roll is often concentrated near $0^\circ$ and pitch within a narrow range.

Figure 4: Puffin-16M combines large-scale image-language-camera triplets with trajectories containing diverse rotational camera motions.

The dataset also provides a broad annotation release for 28 public datasets, covering approximately 44.5 million images. These annotations include predicted roll, pitch, and vertical FoV, thereby converting conventional recognition, detection, segmentation, and multimodal corpora into camera-grounded resources. The paper reports systematic viewpoint biases: roll is generally concentrated near zero, pitch is often negative, and FoV varies according to the capture regime.

This extension is useful for scaling supervision, but its physical labels should be interpreted carefully. The annotations for existing datasets are generated by Puffin-World rather than obtained from independent ground-truth calibration. They are therefore pseudo-labels whose value depends on the accuracy and calibration of the annotator model. The paper uses reliable sparse depth where available and completes missing depth with DA3 predictions, but this similarly introduces model-derived supervision into the training mixture.

Training and task unification

Training proceeds in four stages. The first stage aligns the vision encoder and LLM with camera-aware descriptions while training the diffusion connector and condition tokenizer. The second stage fine-tunes nearly the full system for single-view understanding and generation. The third stage post-trains the diffusion model for cross-view generation on trajectories. The fourth activates joint RGB-depth generation.

The staged schedule is necessary because appearance and geometry share the diffusion backbone. The geometry objective is initially given zero weight and ramped to one over 3,000 iterations. This prevents early depth gradients from disrupting the pretrained appearance pathway. Asymmetric attention further constrains information flow: geometry tokens can attend to all modalities, whereas appearance and text tokens cannot attend to geometry tokens. A zero-initialized geometry-modality embedding preserves the appearance model at the beginning of geometry training.

For long-horizon generation, views are generated in chunks. The final view of one chunk becomes the reference for the next, with the denoised latent carried across the boundary rather than re-encoding a decoded image. This avoids repeated VAE encode-decode operations and is intended to reduce boundary artifacts. Physics propagation is applied over the entire sequence, maintaining one gravity-aligned frame across chunks.

This is a coherent engineering strategy, but it also introduces a dependency on the latent-space interface. The paper demonstrates qualitative long trajectories, yet does not provide a dedicated quantitative scaling curve for error accumulation as a function of horizon length. The effectiveness of latent carryover beyond the evaluated trajectory lengths therefore remains an open empirical question.

Camera-to-world understanding

Puffin-World is evaluated on Stanford2D3D, MegaDepth, TartanAir, and LaMAR using median roll, pitch, and FoV errors together with AUC at $1^\circ$, $5^\circ$, and $\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$0. It outperforms the listed baselines on all median-error metrics and achieves the best result on most AUC metrics.

Representative results illustrate the magnitude of the improvement:

Benchmark	Roll median error	Pitch median error	FoV median error
Stanford2D3D	0.29°	0.53°	1.62°
MegaDepth	0.28°	1.01°	2.41°
TartanAir	0.31°	0.67°	2.34°
LaMAR	0.26°	0.71°	2.73°

On Stanford2D3D, Puffin-World reaches roll AUC@$\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$1 of 97.4, pitch AUC@$\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$2 of 88.8, and FoV AUC@$\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$3 of 61.1, exceeding the specialized GeoCalib baseline in each of these reported measures. On TartanAir, the improvement over the earlier Puffin model is especially substantial: roll AUC@$\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$4 increases from 71.7 to 80.1, pitch AUC@$\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$5 from 51.0 to 60.2, and FoV AUC@$\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$6 from 16.3 to 26.6.

The result is notable because Puffin-World is not a calibration-only architecture. It uses the same model family for understanding, generation, and 3D world modeling, yet remains competitive with or superior to specialized calibration methods. The implication is that scaling camera-grounded multimodal training can improve a general model’s physical-state perception rather than forcing a strict trade-off against task specialization.

The evaluation nevertheless depends on benchmark distributions that may overlap conceptually with the panorama-derived training data. The paper excludes Stanford2D3D from the source panoramas used to construct Puffin-Cam-15M, which addresses direct contamination for that benchmark, but the broader independence of all evaluation distributions from the synthetic camera-rendering pipeline is less fully established.

Camera-controllable image generation

For free-viewpoint spatial simulation, the paper introduces Puffin-Cam-Bench, containing 600 text-camera pairs spanning scenes, viewpoints, camera poses, FoVs, and aspect ratios. Generated images are evaluated by re-estimating their up-vector, latitude, and gravity fields, and by measuring FID.

Puffin-World substantially outperforms general-purpose image generators and prior camera-controllable models:

Method	Gravity mean error	Gravity median error	FID
GPT Image2	28.83°	29.38°	99.36
FLUX.2-dev	28.31°	28.36°	97.06
PreciseCam	17.07°	14.01°	90.89
Puffin	4.92°	2.87°	80.29
Puffin-World	1.32°	0.79°	75.93

Puffin-World reduces the median gravity error from 2.87° to 0.79° relative to Puffin while also improving FID from 80.29 to 75.93. This combination is important: the model does not obtain camera accuracy by sacrificing image realism. Relative to general-purpose generators, the difference is much larger, with errors around $\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$7 for gravity compared with less than $\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$8 for Puffin-World.

Figure 5: Puffin-World preserves specified camera orientation more accurately than general-purpose and specialized image generators while maintaining visual quality.

The result supports the direct latent-space injection design. General image generators can produce semantically plausible images while violating the requested viewpoint, particularly when camera specifications conflict with learned compositional priors. The Omni-Camera map provides dense geometric supervision at the latent-token level, and the training data explicitly samples camera configurations that are rare in ordinary image corpora.

One evaluation caveat is that camera metrics are estimated using Puffin-World’s own physical-perception branch. This creates a shared-model evaluation pathway: an image may be judged according to the same estimator that supplied the model’s camera representation. Independent calibration systems or geometric measurements would provide a stronger test of absolute viewpoint accuracy.

3D world generation and reconstruction

Puffin-World extends single-view generation to multi-view trajectories through joint attention over reference and target views. All views receive individual Omni-Camera conditions, and a view-axis rotary positional embedding identifies their ordering. Target RGB and depth latents are denoised jointly, allowing the model to produce appearance and geometry from the same cross-view process.

Qualitatively, the system supports image-to-3D generation, text-to-3D generation, arbitrary roll/pitch/yaw control, compound translation-rotation trajectories, native physics-state prediction, and 3D reconstruction.

Figure 6: Puffin-World generates multi-view appearance, depth, and physics states under image, text, and camera-trajectory conditions.

On RealEstate10K, Puffin-World achieves PSNR 17.22, SSIM 0.595, and LPIPS 0.318, improving over the strongest listed baselines in all three metrics. On the more difficult Puffin-Traj-Bench, it achieves PSNR 18.00, LPIPS 0.288, roll error 0.80°, pitch error 1.10°, and FoV error 2.96°. Its roll AUC values at $\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}$9, $$\mathbf{g}_t=\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}\mathbf{g}_0.$$0, and $$\mathbf{g}_t=\mathbf{R}_{t\leftarrow0}^{\mathrm{rel}}\mathbf{g}_0.$$1 are 0.57, 0.79, and 0.88; corresponding pitch AUC values are 0.45, 0.72, and 0.83.

Benchmark	PSNR	SSIM	LPIPS	Roll error	Pitch error	FoV error
RealEstate10K	17.22	0.595	0.318	—	—	—
Puffin-Traj-Bench	18.00	0.613	0.288	0.80°	1.10°	2.96°

These results indicate that explicit absolute camera conditioning is most valuable under challenging rotations. On conventional trajectories, image quality is competitive with established multi-view generators; under broad rotational variation, the model maintains substantially better physical camera control. The paper’s strongest empirical claim is therefore not merely that Puffin-World generates plausible novel views, but that it preserves viewpoint semantics under camera transformations that are poorly represented in standard datasets.

The reconstruction results are generated from the model’s predicted views and depth maps rather than from an independently optimized scene representation. The reported visualizations show coherent surfaces and layouts, but the paper provides limited quantitative analysis of metric depth accuracy, reconstruction completeness, or failure cases in occluded and non-Lambertian regions. Thus, the evidence for geometry is principally joint-generation consistency and qualitative reconstruction quality rather than a comprehensive 3D reconstruction benchmark.

Closed-loop applications

The unified interface enables two closed-loop procedures. In mimic exploration, the model receives an initial view and reproduces a prescribed camera trajectory, generating a corresponding exploration of the inferred world. In self-calibrated exploration, it estimates physical misalignment, predicts corrective camera actions, and generates the observations associated with those actions.

Figure 7: Mimic and self-calibrated exploration combine camera understanding, action reasoning, and view generation within the same model.

These applications demonstrate task composition rather than a new training objective. The model’s ability to switch between understanding and generation is mediated by the role mask, camera conditions, and shared latent representations. This is a meaningful consequence of the architecture: calibration is not an isolated preprocessing step, and generation is not an endpoint detached from perception.

The paper’s wording should nevertheless be treated precisely. The experiments demonstrate closed-loop visual simulation, not validated physical control or embodied deployment. The generated world is conditioned on prescribed camera actions, and the self-calibration loop operates in the model’s imagined environment. Whether the same mechanism remains reliable under dynamic objects, action-induced scene changes, sensor noise, or real camera actuation is not evaluated.

Limitations and open questions

The paper identifies dynamic scenes and richer physical states as future extensions, and these limitations are material to the present results. Puffin-World models gravity, latitude, depth, and appearance, but its trajectory data and objectives are predominantly static-scene formulations. Object motion, articulated agents, illumination changes, contact dynamics, and temporal scene evolution are not explicitly represented.

The absolute physical state is also restricted mainly to gravity-relative orientation and FoV. Gravity propagation assumes that the relative rotations are known or accurately specified and that the reference gravity estimate is sufficiently reliable. Errors in either quantity can persist throughout a trajectory because the propagated state is anchored to the initial estimate rather than periodically corrected by independent measurements.

Several evaluations use model-derived labels or model-derived camera estimates. Puffin-World annotates approximately 44.5 million public-dataset images with its own predictions, and generated-image camera accuracy is assessed using its own perception branch. These choices are practical for scale, but they complicate claims of independent calibration and geometric validity.

Finally, the paper does not fully quantify long-horizon degradation, metric depth accuracy, computational cost, or sensitivity to the number of reference views. It also does not establish whether the unified model’s gains arise primarily from the Omni-Camera representation, the enlarged data distribution, the increased backbone capacity, the staged training recipe, or their interaction. The physics-propagation ablation isolates one component, but a broader factorial ablation would be required to attribute the complete improvement.

Conclusion

Puffin-World (2609.04196) presents a unified formulation of camera-aware perception, controllable image generation, multi-view world modeling, and reconstruction around explicit physics, geometry, and appearance states. Its Omni-Camera representation combines gravity-anchored absolute orientation with relative ray geometry, while physics propagation maintains the reference frame across future views. The model achieves strong camera-understanding results across four benchmarks, reduces median gravity error to 0.79° in camera-controlled generation, and obtains the best reported visual and camera-control results on the paper’s 3D generation benchmarks.

The work’s main technical contribution is the integration of absolute physical grounding into a unified multimodal generation framework. Its central unresolved questions concern independent evaluation, dynamic-world modeling, metric geometric accuracy, and the stability of physical grounding over substantially longer and less controlled trajectories.