Puffin-World: Multimodal Architecture

• Puffin-World is a 2026 multimodal world-modeling framework integrating gravity- and camera-grounded representations to model physical states, spatial geometries, and visual appearances in 3D.
• The architecture offers functionalities supporting single-view understanding, multi-view image synthesis, and RGB-depth reconstruction.
• Evaluations show improvements in PSNR, SSIM, roll, and pitch accuracy for 3D world generation across multiple datasets.

Puffin-World is a unified multimodal architecture for physical understanding, spatial simulation, and 3D world generation and reconstruction. It models three native world states—physics, geometry, and appearance—through gravity- and camera-grounded representations, and couples them with an Omni-Camera representation for single-view understanding, controllable image generation, multi-view synthesis, trajectory generation, and RGB-depth reconstruction. The name is also used in several unrelated research contexts, including an uncertainty compiler, free-electron-laser simulation software, a neural vocoder, vector-index storage in Apache Iceberg, and animal-vision systems. In its formal architectural sense, however, Puffin-World refers to the 2026 multimodal world-modeling framework (Liao et al., 3 Sep 2026).

1. Terminology and research contexts

The formal Puffin-World architecture is distinct from earlier systems named Puffin. “Puffin” denotes an unaveraged broadband free-electron-laser code in which a single monoenergetic electron beam can interact with widely separated, non-harmonic resonances through alternating undulator modules (Campbell et al., 2013). A subsequent FEL study used Puffin to demonstrate simultaneous bunching at $1$ nm and $2.4$ nm, including nonlinear sum- and difference-frequency components (Campbell et al., 2014).

In uncertainty quantification, Puffin is an automatic uncertainty compiler that translates deterministic source code into uncertainty-aware code using intervals, probability distributions, p-boxes, confidence structures, dependence-aware arithmetic, and intrusive UQ algorithms (Gray et al., 2021). Its broader “Puffin-World” concept describes an ecosystem combining an uncertainty language, source-to-source compiler, and uncertainty-aware numerical library.

Other works use the name for unrelated application components. Puffin is a pitch-synchronous neural vocoder that generates full-band $48$ kHz speech on modest AAC devices using low-rate convolutional processing, complex spectral prediction, inverse FFT, and pitch-synchronous overlap-add (Watts et al., 2022). Puffin-backed vector indexes store Vamana or DiskANN graph shards in Apache Iceberg Puffin sidecar files and bind them to snapshots through the statistics-file property (Borycki, 2 Jun 2026). In computer vision, Puffin-related systems address bird reconstruction, point-based aerial animal detection, camera-centric multimodal reasoning, bird identity-preserving generation, and motion-aware penguin detection (Kuang et al., 2023, May et al., 2024, Liao et al., 9 Oct 2025, Sun et al., 4 Dec 2025, Seko et al., 4 Mar 2026).

The formal Puffin-World paper extends the camera-centric model introduced in “Thinking with Camera,” which combined camera-language reasoning, diffusion generation, global camera parameters, and pixel-wise camera maps (Liao et al., 9 Oct 2025). Puffin-World adds native geometry and physics states, trajectory-level camera propagation, and joint RGB-depth generation (Liao et al., 3 Sep 2026).

2. Native world states and camera grounding

Puffin-World represents a world state at view $t$ as three coupled components:

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$

where $\mathcal{P}_t$ is the physics state, $\mathcal{G}_t$ is geometry, and $\mathcal{A}_t$ is appearance.

The physics state is represented by a pixel-wise up-vector field and latitude map:

$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$

The up-vector field expresses the image-plane direction corresponding to world-up. The latitude map describes the elevation of each camera ray relative to the gravity-defined horizontal plane. Geometry is represented by a depth map,

$\mathcal{G}_t=\mathbf{D}_t,$

and appearance by an RGB image,

$2.4$0

“Physics” is used in a restricted camera-centric sense. It denotes gravity, latitude, roll, pitch, vertical field of view, and related absolute orientation cues; it does not constitute a full simulator of forces, materials, object dynamics, or articulated agents. The architecture consequently establishes a hierarchy in which physical orientation anchors the coordinate frame, geometry specifies scene structure, and appearance renders the observed view.

The Omni-Camera representation combines absolute physical information with relative ray geometry. For each pixel $2.4$1,

$2.4$2

where the absolute component is

$2.4$3

and the relative component is

$2.4$4

Here $2.4$5 is the camera center and $2.4$6 is the normalized ray direction associated with pixel $2.4$7. With camera intrinsics $2.4$8, rotation $2.4$9, translation $48$0, and homogeneous pixel coordinate $48$1, the camera center and ray direction are:

$48$2

$48$3

The absolute up-vector is computed from the image displacement induced by moving a 3D point opposite to gravity. If $48$4 is the unit gravity direction and $48$5 is perspective projection,

$48$6

The latitude field is:

$48$7

This combination supports roll, pitch, yaw, translation, field-of-view changes, cross-view synthesis, and camera-conditioned generation. A four-channel role mask specifies whether a view is a generation target, conditioning reference, image-conditioned input, or geometry view.

3. Model architecture and multimodal interaction

Puffin-World contains a vision-language pathway, a diffusion generation pathway, camera-condition fusion, and multi-view sequence processing. A geometry-aligned vision encoder processes images, and an MLP projector maps its features into the language-model embedding space. The LLM produces scene descriptions, spatial reasoning, and camera parameters such as roll, pitch, and vertical field of view.

Camera understanding is formulated as autoregressive sequence prediction. If $48$8 is the structured sequence containing text and camera parameters, the objective is approximately:

$48$9

The diffusion pathway uses 64 learnable queries and a six-layer Transformer connector. The connector produces a pooled conditioning vector and joint-attention conditioning embeddings for a multimodal diffusion transformer initialized from Stable Diffusion 3.5 variants. Images and depth maps are encoded with the same frozen VAE into 16-channel latent tensors.

The camera map and role mask are concatenated and processed by a condition-fusion module $t$0. If the camera map has nine channels and the role mask has four, their concatenation has thirteen channels:

$t$1

The initial camera-conditioned latent representation is:

$t$2

where $t$3 is the noisy image or depth latent and $t$4 is a patch-embedding operator. Camera features are also re-injected at selected transformer layers:

$t$5

The fusion module is initialized near zero, preserving the behavior of the pretrained image generator at the beginning of training.

For a trajectory, reference and target views are placed into one joint-attention sequence. Reference views use clean latents, while target views are noised and denoised jointly. Each view receives its own Omni-Camera map, role mask, view-axis index, and one-dimensional rotary positional embedding along the view axis. The model does not use a separately specified pairwise RGB-consistency loss; cross-view consistency is encouraged through shared attention, camera conditioning, view indexing, and semantic conditioning.

4. Physics propagation and trajectory generation

Absolute camera grounding and relative camera motion serve complementary purposes. Relative transformations describe how a camera moves between views, but they do not guarantee that the trajectory remains upright relative to gravity. Puffin-World propagates the absolute gravity state from an initial reference view through relative rotations.

Let $t$6 be the gravity direction inferred for the reference frame, and let

$t$7

be the relative rotation from the reference camera frame to view $t$8. The propagated gravity direction is:

$t$9

For a trajectory, the propagated physical states are:

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$0

Each propagated gravity direction is combined with the target camera rays to construct the target up-vector and latitude fields. The resulting camera condition is:

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$1

This mechanism is deterministic propagation of camera physics rather than learned object dynamics. It anchors future views to the reference frame’s gravity orientation and reduces roll and pitch drift.

Puffin-World is trained on fixed-length sequences, typically eight views, but can generate longer trajectories autoregressively. It generates a chunk of views, carries the last generated view forward as the next reference, and uses a sliding context window for the subsequent chunk. The carried-over view remains in latent form rather than being decoded to RGB and re-encoded, avoiding repeated VAE encode/decode artifacts.

Two closed-loop applications are emphasized. Mimic world exploration follows a specified target camera trajectory from a shared initial view and generates corresponding RGB and optionally depth views. Self-calibrated world exploration repeatedly estimates the current camera state, predicts a corrective camera action, propagates the physical state, generates the imagined target view, and re-estimates its camera state.

These applications demonstrate camera-conditioned generation and closed-loop interaction, but they do not establish a persistent 3D map, collision-aware navigation, physical action planning, or indefinite temporal consistency.

5. Joint appearance and geometry generation

Puffin-World represents depth through an invertible three-channel encoding inspired by a 3D Hilbert-curve color mapping. If $$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$2 is a scalar depth map and $$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$3 is the encoding,

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$4

with inverse:

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$5

This allows depth to use the same frozen VAE interface as RGB imagery. Depth tokens are appended as a separate modality block and receive the same camera condition, view-axis index, and semantic conditioning as their corresponding RGB views. A geometry-modality embedding distinguishes depth from appearance.

For a clean latent $$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$6 and Gaussian noise $$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$7, flow matching constructs:

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$8

with velocity target:

$$\mathcal{W}_t = \left( \mathcal{P}_t, \mathcal{G}_t, \mathcal{A}_t \right),$$9

For target modalities and views $\mathcal{P}_t$0, the flow-matching objective is:

$\mathcal{P}_t$1

Appearance and geometry losses are separately normalized:

$\mathcal{P}_t$2

where:

$\mathcal{P}_t$3

The reported setup uses $\mathcal{P}_t$4 iterations and $\mathcal{P}_t$5. The geometry loss is ramped up so that early depth gradients do not destabilize the pretrained appearance pathway.

Attention is asymmetric. Appearance and text tokens cannot attend to geometry tokens, whereas geometry tokens can attend to all modalities. This allows depth prediction to exploit RGB, text, camera, and other depth information without allowing depth features to directly contaminate the appearance stream.

The architecture does not report an additional explicit photometric reprojection loss, depth smoothness loss, point-cloud consistency loss, or multi-view geometric consistency loss. Geometry and appearance are coupled through the shared VAE interface, shared diffusion transformer, joint attention, common camera conditioning, and joint flow-matching objective.

6. Training, datasets, results, and limitations

Training proceeds in four stages. The alignment stage trains the MLP projector, learnable queries, connector, and condition tokenizer while freezing the vision encoder, LLM, diffusion model, and VAE. Supervised fine-tuning then unfreezes all modules except the VAE, with vision-encoder gradients scaled by $\mathcal{P}_t$6. Cross-view world modeling freezes the vision encoder and LLM while training the connector, diffusion backbone, and camera-condition fusion on trajectory data. The final stage introduces depth and appearance-geometry reconstruction.

The reported optimizer is AdamW with cosine scheduling, weight decay $\mathcal{P}_t$7, and betas $\mathcal{P}_t$8. Stage-dependent learning rates are:

$\mathcal{P}_t$9

with batch sizes $\mathcal{G}_t$0, $\mathcal{G}_t$1, $\mathcal{G}_t$2, and $\mathcal{G}_t$3. Noise levels follow a logit-normal distribution with $\mathcal{G}_t$4 and $\mathcal{G}_t$5, with resolution-dependent shifts. Classifier-free guidance is implemented by condition dropout, and inference uses greedy decoding for understanding and 50 sampling steps for generation. Reported guidance scales are $\mathcal{G}_t$6 for single-view generation and $\mathcal{G}_t$7 for 3D world modeling.

The principal dataset is Puffin-16M. Puffin-Cam-15M contains approximately 15 million vision-language-camera triplets derived from approximately 900,000 panoramas, including indoor and outdoor scenes, real and synthetic environments, self-driving street views, and scenes from multiple countries. Perspective views are rendered with:

$\mathcal{G}_t$8

and:

$\mathcal{G}_t$9

The dataset includes square, portrait, landscape, and widescreen aspect ratios, including $\mathcal{A}_t$0, $\mathcal{A}_t$1, $\mathcal{A}_t$2, $\mathcal{A}_t$3, $\mathcal{A}_t$4, $\mathcal{A}_t$5, and $\mathcal{A}_t$6.

Puffin-Traj-1M contains one million trajectories and approximately 100 million frames. It samples roll and pitch from $\mathcal{A}_t$7, yaw from $\mathcal{A}_t$8, and vertical field of view from $\mathcal{A}_t$9. Its motion types include continuous look-down, continuous look-up, clockwise and counterclockwise rotation, full $$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$0 surrounding exploration, and long-horizon rotations. Trajectories are produced through single-pass and recursive-pass sampling.

Additional data sources include DL3DV, RealEstate10K, Hypersim, MVS-Synth, TartanAir, and ScanNet. Puffin-World also reports absolute camera annotations for 28 public datasets covering approximately 44.5 million images; these annotations are generated through the camera-understanding pathway and therefore are not necessarily independent human ground truth.

On camera-to-world understanding, the reported median errors are:

Dataset	Roll	Pitch	Vertical FoV
Stanford2D3D	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$1	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$2	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$3
MegaDepth	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$4	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$5	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$6
TartanAir	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$7	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$8	$$\mathcal{P}_t = \left( \mathbf{U}_t, \boldsymbol{\Phi}_t \right).$$9
LaMAR	$\mathcal{G}_t=\mathbf{D}_t,$0	$\mathcal{G}_t=\mathbf{D}_t,$1	$\mathcal{G}_t=\mathbf{D}_t,$2

The paper reports the best median errors across the evaluated datasets and most AUC values relative to the listed camera-calibration baselines.

On Puffin-Cam-Bench, which contains 600 text-camera specification pairs, the reported camera-controlled generation errors are:

• up-vector mean/median error: $\mathcal{G}_t=\mathbf{D}_t,$3;
• latitude mean/median error: $\mathcal{G}_t=\mathbf{D}_t,$4;
• gravity mean/median error: $\mathcal{G}_t=\mathbf{D}_t,$5;
• FID: $\mathcal{G}_t=\mathbf{D}_t,$6.

For comparison, the earlier camera-centric Puffin model reported up-vector error of $\mathcal{G}_t=\mathbf{D}_t,$7, latitude error of $\mathcal{G}_t=\mathbf{D}_t,$8, gravity error of $\mathcal{G}_t=\mathbf{D}_t,$9, and FID $2.4$00 (Liao et al., 9 Oct 2025, Liao et al., 3 Sep 2026).

On 3D world generation, Puffin-World reports:

$2.4$01

on RealEstate10K, and:

$2.4$02

on Puffin-Traj-Bench. On the trajectory benchmark, the reported median camera errors are $2.4$03 for roll, $2.4$04 for pitch, and $2.4$05 for field of view.

The physics-propagation ablation reports the following comparison:

Configuration	PSNR	SSIM	LPIPS	Roll error	Pitch error
Without propagation	$2.4$06	$2.4$07	$2.4$08	$2.4$09	$2.4$10
With propagation	$2.4$11	$2.4$12	$2.4$13	$2.4$14	$2.4$15

The reported improvement is strongest for roll and pitch, which are directly coupled to gravity. Yaw produces smaller appearance gains because it rotates around the gravity axis.

Puffin-World remains limited in several respects. Dynamic scenes are not modeled as explicit world states, and the physical state is restricted primarily to gravity, latitude, and absolute camera orientation. Long-horizon generation may accumulate semantic and geometric drift despite latent sliding-window propagation. The depth representation remains subject to VAE compression, bounded encoding precision, and generative uncertainty. Many absolute camera annotations are inferred by the model itself and may inherit its biases. Closed-loop applications are presented without a complete embodied-control benchmark, success-rate analysis, or long-horizon policy evaluation. The architecture also does not report an explicit multi-view geometric consistency loss, and its cross-view coherence may degrade under occlusion, disocclusion, extreme viewpoint changes, reflective surfaces, dynamic objects, unusual optics, or weak gravity cues.

Puffin-World is therefore best characterized as a camera-grounded multimodal generation and reconstruction framework rather than a complete physical world simulator. Its principal contribution is the integration of absolute physical camera fields, relative ray geometry, RGB appearance, depth, language-mediated reasoning, and trajectory-conditioned diffusion generation within one architecture (Liao et al., 3 Sep 2026).