Puffin-4M: Camera-Centric Multimodal Dataset

• The dataset introduces unified camera-centric multimodal data with 4 million triplets, integrating imagery, descriptive captions, and detailed camera configurations.
• It employs a novel camera as language paradigm, quantizing camera parameters into professional photographic terms to enhance spatial and geometric reasoning.
• It supports robust model training for scene synthesis and spatial intelligence by providing dense pixel-wise camera maps and comprehensive geometric annotations.

The Puffin-4M dataset is a large-scale, curated resource comprising 4 million vision–language–camera triplets. It is engineered to support unified camera-centric multimodal learning, enabling spatially aware models to integrate scene understanding, spatial reasoning, and generative synthesis through explicit camera control. The dataset features panoramic-derived perspective imagery, descriptive captions embedding spatial cues, and comprehensive camera parameter annotations—including both global and pixel-wise geometric encodings—advancing the modality alignment between vision, language, and geometry.

1. Dataset Composition and Structure

Puffin-4M consists of 4 million entries, each representing a single-view image paired with a textual caption and camera configuration data. Source images are perspective renderings from panoramic scenes obtained through established datasets (OmniPhotos, Stanford2D3D, PanDA) and sources such as Google Street View. Each triplet contains:

• Image: Perspective view generated via pinhole projection from a panorama, corrected geometrically and representing a consistent spatial context.
• Caption: A description, often augmented with spatial reasoning and professional photographic terminology, that annotates visual content and orientation.
• Camera Information:
  • Global parameters: Roll, pitch, vertical field-of-view (FoV), all normalized and expressed both numerically and as mapped language tokens.
  • Pixel-wise camera maps: For each image, dense spatial encodings annotate per-pixel orientation (up vector) and latitude angle, calculated by:

  $$u_x = \lim_{c \to 0} \frac{\mathcal{P}(X - c \cdot g) - \mathcal{P}(X)}{\|\mathcal{P}(X - c \cdot g) - \mathcal{P}(X)\|_2}, \qquad \phi_x = \arcsin \left( \frac{R \cdot g}{\|R\|_2} \right)$$

  where $\mathcal{P}(\cdot)$ projects a 3D point, $g$ is the gravity vector, and $R$ is the light ray for pixel $x$.

2. Methodological Innovations

Puffin-4M incorporates several methodological advances in dataset construction and annotation:

• Camera as Language paradigm: Camera parameters (e.g., roll, pitch, FoV) are quantized and explicitly mapped to professional photographic terms such as "large counterclockwise Dutch angle" and "near level shot". This facilitates intermediate supervision and bridges the gap between raw numerical cues and semantic reasoning, supporting models in integrating geometric understanding with linguistic synthesis.
• Dense geometric encoding: By generating pixel-wise camera maps using analytic formulas within the pinhole model, the dataset embeds fine-grained spatial information alongside the image and caption, enabling conditional generation and robust spatial reasoning in downstream models.
• Caption enrichment pipeline: Advanced vision-LLMs (e.g., Qwen2.5-VL) are leveraged to produce both vivid descriptive captions and "thinking captions" explicitly tied to camera configuration, further strengthening the multimodal alignment.

3. Data Generation and Annotation Pipeline

The multi-stage process underlying Puffin-4M construction involves:

• Panorama collection: Selection of high-quality panoramic images from diverse sources.
• Perspective rendering: Sampling of camera intrinsic parameters (vertical FoV) and extrinsic parameters (roll, pitch sampled uniformly within $[-45^\circ,\,45^\circ]$; FoV in $[20^\circ,\,105^\circ]$). An additional yaw parameter enables extension to cross-view learning scenarios.
• Geometric correction: Application of standard techniques to ensure accurate mapping from panorama to perspective.
• Caption synthesis: Automated scene description and camera-centric reasoning generated via contemporary multimodal models.
• Camera information annotation: Quantization and mapping of parameters to language tokens; calculation of pixel-wise orientation and latitude maps using closed-form analytical expressions.

4. Technical Contributions to Camera-Centric Multimodal Learning

Puffin-4M serves as the principal training and evaluation resource for the Puffin model (Liao et al., 9 Oct 2025). The dataset enables:

• Unified learning objectives: Puffin is trained to both interpret camera parameters from images (camera-centric understanding) and synthesize novel scenes under compositional camera control (camera-controllable generation).
• Reasoning decoupling: By aligning roll, pitch, and FoV both as numerical and semantic entities, models disentangle geometric reasoning from visual feature learning.
• Spatially conditioned generation: Dense pixel-wise camera maps serve as priors for diffusion-based generation modules. This enables nuanced spatial consistency—adjustments to roll, pitch, or FoV reflect in the synthesized image structure and viewpoint, providing precise control.
• Chain-of-thought integration: The captioning and annotation strategy allows models to develop shared reasoning processes that serve both perception and generative tasks, connecting recognition with synthesis via camera-aware conceptual grounding.

5. Broader Impacts and Application Domains

The scope of Puffin-4M extends beyond its direct utility for the Puffin model. Its contributions include:

• Benchmarking spatial intelligence: Puffin-4M establishes a new standard for evaluating models on camera-centric generation, cross-view spatial imagination, and instructable spatial reasoning tasks.
• Modality bridging and geometric reasoning: The dataset's comprehensive approach to camera parameter annotation and professional terminology facilitates research into modality gaps between vision, geometry, and language, relevant for advanced multimodal models.
• Applied impact: A plausible implication is that Puffin-4M can accelerate developments in fields such as robotics (navigation, grasping), AR/VR (viewpoint synthesis, scene rendering), and autonomous systems which require precise spatial control and interpretation.

6. Relationship to Ultra-Long Context Datasets and Models

While Puffin-4M is primarily focused on camera-centric multimodal tasks, insights from ultra-long context LLM training (Xu et al., 8 Apr 2025) underpin its design:

• Curated data composition: Strategies such as document concatenation with special separator tokens, validated in ultra-long context LLMs, inform Puffin-4M's approach to high-fidelity multimodal data fusion.
• Scalability and performance preservation: The demonstrated ability in large context models to extend sequence length while maintaining or enhancing reasoning and instruction-following capacity suggests analogous robustness in multimodal spatial datasets, supporting complex comprehension and generation tasks spanning millions of tokens and pixels.
• Blueprint for future corpus construction: The careful balancing of sequence length, semantic richness, and geometric detail in Puffin-4M reflects best practices in scalable data engineering from language modeling research, informing future large-scale multimodal data resources.

7. Future Directions

The public release of Puffin-4M, with code, models, and pipelines, is expected to galvanize research in multimodal spatial intelligence (Liao et al., 9 Oct 2025). Anticipated advances include:

• New model architectures: Exploiting dense camera maps and semantic alignment to develop more robust unified perception-generation frameworks.
• Extension to time and temporal context: Further research could augment Puffin-4M with video sequences and temporal camera parameters, extending spatial reasoning to four-dimensional domains.
• Cross-domain applicability: Benchmarking spatial intelligence in real-world domains, integrating Puffin-4M-derived capabilities with robotics, remote sensing, and immersive content creation.

Puffin-4M thus marks a pivotal development in principled, scalable multimodal dataset engineering, bridging direct geometric control with linguistic and perceptual reasoning in a unified camera-centric paradigm.