Scaling Sequence-to-Sequence Generative Neural Rendering

Abstract: We present Kaleido, a family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido operates on the principle that 3D can be regarded as a specialised sub-domain of video, expressed purely as a sequence-to-sequence image synthesis task. Through a systemic study of scaling sequence-to-sequence generative neural rendering, we introduce key architectural innovations that enable our model to: i) perform generative view synthesis without explicit 3D representations; ii) generate any number of 6-DoF target views conditioned on any number of reference views via a masked autoregressive framework; and iii) seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer. Within this unified framework, Kaleido leverages large-scale video data for pre-training, which significantly improves spatial consistency and reduces reliance on scarce, camera-labelled 3D datasets -- all without any architectural modifications. Kaleido sets a new state-of-the-art on a range of view synthesis benchmarks. Its zero-shot performance substantially outperforms other generative methods in few-view settings, and, for the first time, matches the quality of per-scene optimisation methods in many-view settings.

• The paper introduces Kaleido, a generative model that reformulates neural rendering as a sequence-to-sequence task using masked autoregressive transformers.
• It leverages a unified positional encoding and large-scale video pre-training to enhance spatial consistency and boost PSNR in both few-view and many-view scenarios.
• Empirical results demonstrate state-of-the-art performance in zero-shot view synthesis and 3D reconstruction, setting new benchmarks in photorealistic rendering.

Sequence-to-Sequence Generative Neural Rendering at Scale: The Kaleido Model

Introduction and Motivation

The paper introduces Kaleido, a family of generative models for photorealistic neural rendering that unifies object- and scene-level view synthesis within a single, scalable architecture. The central hypothesis is that 3D rendering can be reframed as a sequence-to-sequence image synthesis problem, analogous to language and video generation, thereby circumventing the need for explicit 3D representations. This approach leverages large-scale video pre-training to impart "visual common sense" and spatial consistency, enabling the model to generalize across diverse rendering tasks and data modalities.

Figure 1: Kaleido is a generative rendering engine capable of synthesizing photorealistic novel views from arbitrary numbers of reference images and camera poses.

Sequence-to-Sequence Formulation for Neural Rendering

Kaleido reformulates neural rendering as a conditional sequence-to-sequence task, where the model predicts a set of target views given a set of reference views and their associated camera poses. This is operationalized via a masked autoregressive transformer, allowing for arbitrary numbers of reference and target views during both training and inference. The model operates in a latent space, using a VAE encoder for spatial compression, and employs a rectified flow objective for generative modeling.

Figure 2: Neural rendering is framed as a sequence-to-sequence task, unifying its design with language and video generation via a transformer that generates image tokens conditioned on spatial positions.

A key architectural innovation is the unified positional encoding scheme, which extends RoPE and GTA to parameterize 2D, 3D, and temporal positions in a relative fashion. This enables seamless processing of both video and multi-view 3D data within a single architecture, without task-specific modifications.

Architectural Design and Scaling Strategies

The paper presents a comprehensive ablation study to identify effective scaling strategies for generative neural rendering. The final Kaleido architecture is a decoder-only transformer with grouped-query attention, SwiGLU activations, and a windowed cross-view attention mechanism to enhance feature exchange between views while maintaining computational efficiency.

Figure 3: Ablation study of architectural and training strategies, highlighting the impact of design choices on PSNR and throughput across different view conditioning settings.

Figure 4: Kaleido architecture: a scalable decoder-only transformer processes sequences of reference and noised target latents, with unified positional encoding and AdaIN timestep conditioning.

Auxiliary features from DINOv2 are concatenated with reference latents to improve depth perception and rendering quality, especially for in-the-wild images. The model is trained with a principled view sampling strategy that emphasizes challenging few-view scenarios, and a noise-biased SNR sampler tailored for the constrained nature of rendering tasks.

A notable empirical finding is the emergence of massive activations in rectified flow transformers, particularly when mixing synthetic and real data. This is mitigated by introducing learnable register tokens in attention layers, stabilizing training at high resolutions.

Video Pre-training and Transfer to 3D

Kaleido leverages large-scale video pre-training to bootstrap spatial understanding, followed by fine-tuning on multi-view 3D datasets. This transfer learning approach yields a 1.3–2.0x improvement in 3D training efficiency, validating the hypothesis that 3D can be treated as a specialized subdomain of video.

Experimental Results

Kaleido is evaluated on a comprehensive suite of view synthesis and 3D reconstruction benchmarks, including object-level (OO3D, GSO-30, RTMV) and scene-level (LLFF, Mip-NeRF 360, Tanks and Temples) datasets. The model achieves state-of-the-art zero-shot performance, with strong scaling properties as model size increases.

• Few-view NVS: Kaleido-Small (0.6B params) matches or outperforms larger baselines (SEVA, SV3D, EscherNet) across all datasets. Kaleido-Large achieves up to +7.8 dB PSNR improvement over EscherNet in many-view settings.
• Many-view NVS: Kaleido is the first zero-shot generative model to match or surpass per-scene optimization methods (Instant-NGP, 3DGS) when provided with all available reference views.
• 3D Reconstruction: By generating multi-view renderings and applying NeuS2, Kaleido outperforms direct image-to-3D and view-synthesis-based baselines, achieving superior Chamfer Distance and Volumetric IoU with as few as two reference views.

Figure 5: Kaleido demonstrates zero-shot generative capabilities on in-the-wild images, synthesizing photorealistic novel views from a single input.

Practical and Theoretical Implications

Kaleido's sequence-to-sequence formulation and unified positional encoding establish a scalable paradigm for general-purpose neural rendering, obviating the need for explicit 3D representations or per-scene optimization. The model's ability to generalize across arbitrary numbers of views and data modalities, combined with its strong scaling properties, positions it as a foundation for future world models and interactive virtual environments.

The empirical analysis of massive activations in rectified flow transformers highlights a previously underexplored stability issue in high-resolution generative models, with practical implications for training large-scale visual transformers.

Limitations and Future Directions

Despite its strong performance, Kaleido exhibits several limitations:

• Texture flickering and sticking in challenging scenes, especially at low resolutions or with few reference views.
• Lack of camera intrinsics modeling, precluding effects such as dolly-zooms.
• Degraded semantic plausibility under extreme viewpoint changes.
• Inference speed is not yet competitive with real-time scene-specific methods.

Future work should address these limitations by integrating camera intrinsics into the positional encoding, incorporating priors from large-scale text-to-image/video models, optimizing inference speed via distillation, and extending the framework to 4D (spatiotemporal) generative modeling.

Conclusion

Kaleido demonstrates that sequence-to-sequence generative modeling, underpinned by unified spatial-temporal encoding and large-scale video pre-training, is a viable and scalable approach for photorealistic neural rendering. The model achieves state-of-the-art results in both view synthesis and 3D reconstruction, and for the first time, matches the quality of per-scene optimization methods in a zero-shot setting. The architectural and empirical insights presented have significant implications for the design of future general-purpose rendering engines and world models in AI.