Puffin-16M Dataset: Description, Components and Real World Application

• Puffin-16M is a multimodal dataset comprising approximately 15 million vision-language-camera triplets and one million camera trajectories, designed for unified 3D world modeling and camera control.
• The dataset includes RGB images, scene descriptions, spatial relationships, camera parameters, geometry, and challenging camera motion to enhance supervised learning and generation of physical models.
• Puffin-16M consists of Puffin-Cam-15M for perception, and Puffin-Traj-1M for challenging camera movements.

Puffin-16M is a multimodal dataset constructed for Puffin-World, a unified model for camera understanding, camera-controlled image generation, multi-view and trajectory generation, geometry reconstruction, and physically grounded 3D world modeling. It combines approximately 15 million vision–language–camera triplets in Puffin-Cam-15M with one million camera trajectories in Puffin-Traj-1M, linking RGB appearance, language, camera state, gravity-related orientation, relative rays, and challenging viewpoint motion. The dataset and associated code and models are released with the Puffin-World project (Liao et al., 3 Sep 2026).

1. Purpose and conceptual scope

Puffin-16M is designed around the premise that conventional image, video, and multi-view datasets provide insufficient supervision for physically grounded world models. Many image datasets lack camera metadata, captions rarely describe spatial or physical relationships, and conventional multi-view datasets often provide relative camera poses without an absolute orientation tied to gravity. These limitations can produce horizon drift, incorrect uprightness, and unstable long-horizon viewpoint generation.

The dataset addresses these deficiencies by jointly representing:

• Appearance: RGB images and image sequences.
• Language: captions, scene descriptions, spatial relationships, and physical reasoning.
• Camera state: intrinsic and extrinsic parameters, field of view, camera height, gravity-related orientation, and relative rays.
• Geometry: principally depth, supplied directly or through the broader training mixture.
• Motion: broad roll, pitch, yaw, and long-horizon camera trajectories, including full $360^\circ$ exploration.

Puffin-16M consists of two complementary subsets:

Subset	Primary unit	Scale	Principal role
Puffin-Cam-15M	Perspective image with language and camera annotations	Approximately 15M triplets	Camera perception and controllable image generation
Puffin-Traj-1M	Camera trajectory	1M trajectories and approximately 100M frames	Cross-view consistency and difficult camera motion

The overall “16M” designation combines the two subsets, although their units differ: Puffin-Cam-15M is counted in individual perspective images, whereas Puffin-Traj-1M is counted in trajectories. The paper reports the aggregate scale as 16M frames or items. It does not provide complete train, validation, and test counts for each subset.

Puffin-16M expands the earlier Puffin-4M dataset from approximately four million to approximately 15 million perspective images, adds multiple resolutions and aspect ratios, introduces explicit camera-height labels, broadens camera-orientation coverage, and adds approximately one million challenging trajectories.

2. Puffin-Cam-15M

Source panoramas

Puffin-Cam-15M is generated from approximately 900,000 panoramas, compared with 200,000 panoramas used by Puffin-4M. The source collection covers indoor and outdoor environments, real and synthetic scenes, self-driving street views, and scenes from different countries.

Stanford2D3D is excluded from the panoramic source data to support fair evaluation on a camera-to-world benchmark. Because consumer-grade $360^\circ$ imagery and autonomous-driving imagery can contain calibration errors, distortion, or gravity misalignment, the construction pipeline applies geometric correction using line segmentation and vanishing-point estimation.

Perspective rendering

The panoramas are converted into perspective images with a virtual pinhole camera. Roll and pitch are sampled according to

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$

while vertical field of view is sampled from

$\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$

The number of perspective crops is selected adaptively according to panorama resolution. Supported aspect ratios include

$$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$

This multi-ratio construction differs from Puffin-4M, whose images were fixed at $512\times512$. Approximately 15 million images are retained for camera-to-world understanding. A curated subset of approximately eight million images is selected for camera-controllable image generation according to the realism and aesthetic quality of the source panoramas. The supplied paper text does not specify the exact filtering score or threshold.

Language supervision

Each image is annotated with language generated by advanced multimodal LLMs, including Qwen3-VL-8B-Instruct and Qwen3-VL-32B-Instruct. The annotations contain detailed scene semantics, spatial relationships, camera parameters, physical reasoning, and structured chain-of-thought-style spatial analysis.

The dataset therefore does not consist simply of image-caption pairs. Its language annotations are intended to connect visual content with camera configuration and spatial reasoning, enabling a model to describe a scene and infer or generate camera states.

Camera-height categories

Each source panorama receives one of five camera-height labels:

1. Underwater shot
2. Low-position shot
3. Eye-level shot
4. High-position shot
5. Aerial shot

The inherited label is assigned to every perspective image rendered from the panorama. These categories provide semantic information about observer elevation in addition to metric camera parameters.

3. Puffin-Traj-1M and challenging motion

Puffin-Traj-1M contains one million generated camera trajectories and approximately 100 million frames. A trajectory is an ordered sequence of perspective views associated with changing camera extrinsics. The paper does not provide a complete trajectory-length histogram or exact train, validation, and test partition counts.

The trajectory subset is intended to address the limited rotational diversity of many existing 3D and sequential datasets. Conventional camera-capture datasets commonly constrain rotation approximately to

$$\text{roll}\in[-5^\circ,5^\circ],\qquad \text{pitch}\in[-10^\circ,10^\circ],$$

because handheld and mobile capture is generally dominated by translation while maintaining an approximately upright camera.

Puffin-Traj-1M instead includes strong look-down and look-up motion, large roll, arbitrary yaw, long rotational trajectories, and full surrounding-view exploration. Its motion families include:

• continuous look-down;
• continuous look-up;
• clockwise rotation;
• counterclockwise rotation;
• full $360^\circ$ exploration.

For trajectory generation, the camera parameters are sampled as

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$

$\psi\sim\mathcal{U}[0^\circ,360^\circ),$

and

$360^\circ$0

One extrinsic parameter is randomly selected to define the trajectory motion pattern. The paper does not report exact proportions for the different motion families.

Two generation strategies are used. Single-pass generation provides long-range scene-exploration trajectories, whereas recursive-pass generation feeds generated frames back into the process to encourage spatial memory and extended 3D consistency. The latter approximates autoregressive exploration, in which a newly generated view becomes context for subsequent views.

Puffin-Traj-1M is generated from panorama-based rendering rather than being a corpus of naturally recorded robot trajectories. Its challenging-motion property is therefore operationalized through broad camera-parameter sampling and predefined motion families.

4. Omni-Camera representation and physical world states

Puffin-World uses a dense, per-pixel Omni-Camera representation rather than encoding camera motion solely as a global pose vector. For a pixel $360^\circ$1, the camera representation is

$360^\circ$2

where

$360^\circ$3

The resulting camera map has dimensions

$360^\circ$4

Relative ray field

Let $360^\circ$5 be a homogeneous pixel coordinate, $360^\circ$6 the intrinsic matrix, and $360^\circ$7 the world-to-camera transformation. The camera center is

$360^\circ$8

The normalized viewing direction is

$360^\circ$9

The relative ray representation is therefore

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$0

The shared ray origin represents camera translation, while the per-pixel directions represent viewpoint geometry.

Absolute gravity-aware field

Let $\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$1 be the unit gravity direction. The absolute field contains the image-plane projection of the direction opposite gravity and the ray latitude relative to the horizontal plane:

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$2

and

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$3

The Omni-Camera consequently combines absolute roll and pitch grounding with relative translation and rotation. This combination is particularly relevant to the trajectory subset, where large rotations must remain consistent with a common physical frame.

Native world states

Puffin-World models three native world states:

1. Physics: gravity field and latitude.
2. Geometry: depth.
3. Appearance: RGB image.

Puffin-16M primarily supplies RGB, camera, and language supervision. Depth-bearing public datasets are incorporated into the broader training mixture for joint appearance–geometry learning.

Depth is encoded as an RGB image using a deterministic, invertible color mapping inspired by a 3D Hilbert-curve representation. The mapped depth is processed by the same frozen VAE used for RGB. Images and mapped depth are represented as 16-channel latent representations. During multi-view training, depth is appended as an additional token block and identified with a geometry channel in a four-channel role mask.

5. Integration with Puffin-World

Puffin-World is a unified multimodal architecture supporting camera understanding, image generation, geometry reconstruction, and trajectory-conditioned world modeling. RGB and depth share a VAE latent space, while camera conditions are represented through the nine-channel Omni-Camera map.

Camera features are fused with latent representations according to

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$4

where $\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$5 is a condition-fusion module, $\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$6 is a patch-embedding operation, and $\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$7 is a four-channel role mask. Camera features are re-injected at selected transformer layers:

$\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$8

The role mask identifies whether a view is a generation target, a conditioning reference, an image-conditioned input, or a geometry view.

The architecture supports:

• image-to-text-camera understanding;
• text-camera-to-image generation;
• image-to-3D generation;
• multi-view appearance generation;
• joint RGB-depth reconstruction;
• long-horizon exploration.

The reported task ratios are:

Stage	Main task ratios
I	Image $\phi,\theta\sim\mathcal{U}[-45^\circ,45^\circ],$9 Text-Camera: 0.5; Text-Camera $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$0 Image: 0.5
II	Image $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$1 Text-Camera: 0.5; Text-Camera $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$2 Image: 0.5
III	Text-Camera $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$3 Image: 0.14; Image $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$4 3D: 0.86
IV	Text-Camera $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$5 Image: 0.09; Image $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$6 3D reconstruction: 0.91

Stage III uses sequences of eight views, with one to three reference views sampled randomly and the remaining views serving as targets. Stage IV uses one reference view. Multi-view scenes are cropped to $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$7, while single-view Puffin-Cam-15M images retain their native resolutions and aspect ratios.

Physics propagation

If the reference state produces gravity direction $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$8, and $\mathrm{FoV}\sim\mathcal{U}[20^\circ,105^\circ].$9 transforms vectors from the reference camera frame to view $$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$0, gravity is propagated as

$$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$1

The propagated gravity direction is used to render the absolute camera field for each future view. This mechanism is intended to preserve a common gravity-aligned world frame during arbitrary relative motion.

The reported physics-propagation ablation is:

Model	PSNR	SSIM	LPIPS	Roll error	Pitch error
Baseline	19.87	0.62	0.23	1.94	5.14
Baseline + physics propagation	20.54	0.67	0.21	1.76	4.75

The paper reports improvements for roll, pitch, and yaw, with smaller yaw gains because yaw rotates around the gravity axis.

Training objectives

Camera understanding is formulated as autoregressive multimodal sequence modeling. The model generates scene descriptions and spatial reasoning followed by camera parameters such as roll, pitch, and field of view, using next-token cross-entropy.

Appearance and geometry generation use flow matching. For target views $$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$2,

$$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$3

with

$$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$4

The total appearance–geometry objective is

$$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$5

where

$$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$6

with $$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$7 and $$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$8. Geometry loss therefore ramps to its full weight during the first 3,000 iterations.

6. Evaluation, extensions, and limitations

Downstream capabilities

Puffin-Cam-15M supports camera-to-world understanding. On Stanford2D3D, Puffin-World reports median errors of $$1{:}1,\quad 2{:}3,\quad 3{:}2,\quad 3{:}4,\quad 4{:}3,\quad 9{:}16,\quad 16{:}9.$$9 for roll, $512\times512$0 for pitch, and $512\times512$1 for field of view.

On the 600-pair Puffin-Cam-Bench, the reported results are:

• up-vector median error: $512\times512$2;
• latitude median error: $512\times512$3;
• gravity median error: $512\times512$4;
• FID: $512\times512$5.

Puffin-Traj-1M supports image-to-3D generation, text-to-3D generation, rotation-controlled generation, compound translation-plus-rotation control, long-horizon exploration, mimic world exploration, and self-calibrated exploration. In self-calibrated exploration, the model estimates its current physical state, predicts corrective camera actions, imagines resulting views, and iteratively reduces gravity misalignment.

For long-horizon generation, trajectory chunks are produced autoregressively. The final view of one chunk becomes the reference for the next under a sliding context window. The next chunk conditions on the carried-over denoised latent rather than re-encoding a decoded image, reducing cumulative VAE artifacts.

Public-data augmentation

The release also contains absolute-camera annotations for approximately 44.5 million images across 28 public datasets, comprising 22 single-image datasets and six sequential or 3D datasets. These annotations include roll, pitch, vertical field of view, gravity fields, and latitude maps.

Examples include ImageNet, CC12M, COCO, Objects365, Open Images, Megalith-10M, EgoObjects, DL3DV, RealEstate10K, HyperSim, MVS-Synth, ScanNet, and TartanAir. These additional annotations are not included in the 16M count.

The paper reports distributional biases in these public datasets: roll is concentrated near $512\times512$6, pitch is generally negative because cameras often face downward, web photographs tend to have narrower fields of view, EgoObjects has a strong downward bias, and TartanAir has unusually high rotational diversity.

Dataset limitations

Several limitations qualify the interpretation of Puffin-16M:

1. Incomplete split documentation: exact train, validation, and test counts for Puffin-Cam-15M and Puffin-Traj-1M are not provided.
2. Synthetic trajectory construction: Puffin-Traj-1M is generated from panoramas and camera-parameter sampling rather than collected from naturally recorded robot trajectories.
3. Limited native depth: the principal Puffin-16M subsets are listed without depth; depth supervision is supplied mainly by additional public datasets.
4. Coarse camera-height labels: the five semantic categories do not replace metric camera-height measurements.
5. Static-world emphasis: dynamic scenes are identified as future work.
6. Restricted physical state space: the native physics representation primarily models gravity and latitude.
7. Potential annotation error: absolute camera fields for public trajectory datasets may inherit errors from the perception system used to generate them.
8. Long-horizon drift: latent carryover reduces cumulative artifacts but does not eliminate errors over extended trajectories.
9. Unspecified licensing: the paper reports release of code, models, and datasets but does not specify separate licenses in the supplied text.
10. Limited motion statistics: exact proportions for roll, pitch, yaw, clockwise, counterclockwise, and full-circle trajectories are not reported.

Puffin-16M is therefore best characterized as a camera-grounded multimodal dataset rather than merely a large image collection. Puffin-Cam-15M provides scale, language, camera perception, and multi-resolution viewpoint supervision, while Puffin-Traj-1M supplies broad rotational motion, cross-view consistency, and long-horizon exploration. Together with public RGB-D and multi-view datasets, these components support Puffin-World’s unified treatment of appearance, geometry, and physical orientation.