Enhancing In-context Panoramic Generation via Geometric-aware Pretraining

Abstract: In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenarios. On the modeling side, Canvas360 enhances text-to-panorama generation through parallel depth generation, velocity circular padding, and similarity loss regularization, enabling the model to learn geometry-aware representations, capture object distortion details, and improve geometric consistency and global coherence. Furthermore, empowered by strong panoramic priors, Canvas360 enables a unified in-context panoramic generation framework that supports diverse downstream tasks via token-level concatenation, surpassing prior methods in both task coverage and modeling flexibility. Extensive experiments show that Canvas360 improves panoramic image fidelity, achieving particularly strong performance on the panorama-specific FAED metric and competitive or leading results across the reported quantitative evaluations. More information can be found on our project page: https://zry000.github.io/Canvas360/

• The paper introduces Canvas360, which integrates parallel depth supervision and velocity circular padding to significantly enhance panoramic geometric fidelity.
• It employs a two-stage training scheme combining RGB-depth pretraining with transformer-based token-level in-context finetuning for multi-task applications.
• Empirical results demonstrate superior performance in seam continuity and overall geometry, enabling robust style transfer, inpainting, outpainting, and editing.

Enhancing Geometry-aware In-context Panoramic Generation with Canvas360

Introduction and Motivation

Panoramic image generation has advanced notably with the advent of large-scale generative models, yet in-context interactive manipulation across style transfer, inpainting, outpainting, and editing remains challenging due to complex geometric distortions and data scarcity in ERP panoramas. The paper "Enhancing In-context Panoramic Generation via Geometric-aware Pretraining" (2607.08765) introduces Canvas360, a unified architecture explicitly regularized for panoramic geometric structure through depth-aware pretraining and comprehensive in-context dataset construction. Canvas360’s central claim is that incorporating parallel depth supervision and velocity circular padding significantly improves geometric fidelity and seam continuity, properties that are difficult to achieve in previous works relying on piecemeal architectural adaptations or limited training data.

The paper additionally contributes Canvas360Dataset, a 1M-sample panorama corpus, supporting end-to-end in-context training across four canonical tasks. The two-stage framework first instills geometry priors using parallel RGB-depth flow-matching pretraining, then fine-tunes across all downstream contexts using a token-concatenation transformer strategy, producing state-of-the-art results in numbers and visual quality.

Canvas360: Unified Framework and Technical Innovations

Canvas360 is distinguished by its integration of explicit geometric supervision and transformer-based joint modeling. The major components include:

• Parallel Depth Generation: During pretraining, the model jointly generates RGB panorama latents and their corresponding pseudo-depth maps, obtained with DAP. Rather than naively appending depth as a channel, Canvas360 concatenates post-VAE RGB and depth token sequences, applying a positional offset in the RoPE embedding space for branch disambiguation. Independent flow-matching losses are imposed per modality, enhanced by a similarity regularization to discourage degenerate mode collapse where RGB predictions simply replicate depth.
• Velocity Circular Padding: To address the notorious 0/360° longitude seam, Canvas360 introduces ghost tokens with synchronized features at both horizontal boundaries, explicitly encoding spherical adjacency during velocity estimation—in contrast to previous approaches that merely duplicate columns.
• Flow Transformer Backbone: All image and contextual latents are processed by a transformer operating over token sequences derived from a VAE, leveraging continuous-time flow-matching objectives. This design aligns with recent progress in flow-matching models but is uniquely geometrically regularized here.
• Token-level In-context Unification: Finetuning for downstream tasks is performed via token-level concatenation, using simple positional offsets to maintain context/target separation in the positional encoding, enabling the model to handle style transfer, inpainting, outpainting, and editing in a single pass.

  Figure 1: The Canvas360 pipeline: (left) large-scale RGB-depth pretraining including velocity circular padding and parallel depth; (right) unified in-context finetuning over 900K conditional training pairs.

Data Synthesis and Canvas360Dataset

Given the severe lack of relevant large-scale panoramic datasets for in-context tasks, the authors curate Canvas360Dataset, which comprises a 100K annotated RGB-depth corpus (pilot set) and 900K in-context samples distributed as 250K outpainting, 250K inpainting, 200K style transfer, and 200K editing.

• Pseudo-depth Construction: Absolute metric-depth maps are generated using DAP, truncated appropriately, and normalized.
• Style Transfer: Stylized panoramas are synthesized using FLUX.2-dev conditioned on pre-defined style prompts.
• Outpainting and Inpainting: Outpainting is defined via perspective projections and corresponding mask generation; inpainting masks are sampled and paired with global/local captions.
• Editing: Grounded object removal is performed with VLM and vision-based grounding models, with original/edited pairs constructed to support both erasure and addition scenarios.

This scale and granularity notably surpass previous panoramic datasets in coverage, annotation, and in-context pairing, facilitating robust joint-task training.

Figure 2: Schematic of the synthesized data pipeline across four in-context tasks: style transfer, outpainting, inpainting, and editing via multi-model orchestration.

Empirical Results and Analysis

Quantitative Benchmarks

Canvas360 is systematically compared against contemporary panorama generation methods using metrics such as FID, FAED, IS, CS, BRISQUE, NIQE, Q-Align, and task-specific left-right consistency. The model:

• Achieves the lowest FAED (2.33), indicating strong fidelity in panoramic geometry and object placement.
• Scores best on IS (1.76), NIQE (3.70), and QA-aesthetic (4.20), with competitive or second-best values for FID, BRISQUE, QA-quality, and pole/equator splits.
• Further, a user preference study confirms Canvas360’s dominance in boundary continuity, panoramic awareness, and overall quality.

  Figure 3: Qualitative comparison; Canvas360 exhibits reduced seam artifacts and improved geometric adherence relative to both diffusion- and autoregressive-based alternatives.

• In full left-to-right consistency assessment, Canvas360 attains the best LRCE-RGB (0.0063), verifying the efficacy of its seam-aware supervision.
• For all in-context tasks, Canvas360 outperforms strong baselines, achieving superior LPIPS, FAED, and PSNR; its editing and style transfer are characterized by improved structure preservation under geometric distortion.

  Figure 4: Multi-task downstream qualitative results: Canvas360 maintains high fidelity and geometric alignment across inpainting, outpainting, and editing, in contrast to pronounced artifacts in prior methods.

Ablations

Ablation studies clarify the contribution of individual innovations:

• Depth Supervision: Joint RGB-depth pretraining with positional offset and similarity regularization is essential for transferring geometry priors and mitigating mode collapse or artifacts.
• Velocity Circular Padding: Explicit circular velocity encoding reliably improves seam continuity, which is corroborated both qualitatively and in ablation scoring.
• Backbone: The adoption of the FLUX.1-based transformer with added geometry priors is necessary to achieve coherent panoramic completions; a direct FLUX.1 finetuned baseline fails on boundary consistency and artifact suppression.

  Figure 5: Depth branch predictions demonstrate high alignment with 3D scene structure, validating that the supervision branch encodes meaningful geometry cues for generation.

  

  Figure 6: Training with explicit depth supervision produces panoramas with correspondingly consistent estimated depths across cubemap faces, especially at seams and boundaries.

Practical and Theoretical Implications

The Canvas360 methodology sets a precedent for geometry-aware generative modeling in panoramas, bridging the gap between spatial reasoning and unified in-context manipulation. The two-stage training regime enables the model to learn robust panoramic priors transferable to arbitrary downstream tasks, facilitating improved multi-purpose deployment in virtual reality, AR authoring, digital twins, and immersive creative production. Practically, the scalable data pipeline paves the way for continued gains as geometric panoramic datasets expand, especially for rare or unconventional scene categories.

Theoretically, this work demonstrates the efficacy of direct geometric regularization—especially depth-aware flow matching and explicit spherical adjacency encoding—in high-dimensional generative modeling where topology-induced distortions dominate the learning landscape. The paradigm can be extended to other domains requiring topologically-constrained synthesis, including 3D scene manipulation, video panorama generation, and sensor-centric world modeling.

Conclusion

Canvas360 advances in-context panoramic generation by instilling explicit geometry-aware priors through parallel depth pretraining and enforcing seamless boundary conditions with velocity circular padding. Unified fine-tuning over a comprehensive 1M-sample dataset enables one model to support style transfer, inpainting, outpainting, and editing with robust geometric consistency and state-of-the-art metrics. These results underline the importance of panoramic-specific data and architectural priors for next-generation immersive content generation. The framework lays groundwork for future exploration of panoramic or spherical generative models spanning both visual and broader embodied modalities.