CausNVS: Autoregressive 3D Novel View Synthesis

• CausNVS is an autoregressive multi-view diffusion model designed for 3D novel view synthesis, overcoming fixed view set limitations.
• The model employs causal masking, per-frame noise conditioning, and Pairwise-Relative Camera Pose Encoding (CaPE) to ensure efficient and robust generation.
• Its inference strategies, such as sliding-window attention and KV caching, enable scalable, streaming-compatible synthesis across diverse scene trajectories.

CausNVS is an autoregressive multi-view diffusion model developed for flexible 3D novel view synthesis that supports arbitrary input–output view configurations and sequentially generates images conditioned on available camera perspectives and synthesized frames (Kong et al., 8 Sep 2025). By addressing limitations of non-autoregressive generative baselines, such as their requirement for fixed-size view sets and prohibitive full-joint denoising inference costs, CausNVS facilitates efficient, streaming-compatible view synthesis and achieves robust visual quality across diverse scene and trajectory conditions.

1. Autoregressive Multi-View Diffusion: Core Principles

CausNVS is built upon a multi-view diffusion framework where novel views are generated causally—one at a time, sequenced so that each new target view leverages only the input views and previous model outputs. To formalize this dependency, the transformer-based attention mechanism employs causal masking, ensuring that, for frame $i$, the prediction depends exclusively on accessible frames $v_{<i}$ (input views and all previously generated frames). This contrasts with non-autoregressive methods that treat all target views symmetrically and jointly, thus requiring every query for new views to be defined at once and processed via simultaneous denoising.

The autoregressive design of CausNVS allows for flexible inference: the collection of input and target views does not need to be fixed a priori. Additional views may be generated on demand, and sequential generation avoids the combinatorial complexity inherent to all-to-all dependencies in non-autoregressive denoising schemes.

2. Training Methodology: Causal Masking, Per-Frame Noise, and CaPE

CausNVS training involves several innovations to align the model’s capabilities with its desired autoregressive inference properties:

• Causal Masking: During transformer attention, the mask enforces that frame $i$ can only attend to frames $1, ..., i-1$, never to itself or to future frames. This mirrors the inference-time requirement that only past and present content is available for generating each new frame.
• Per-Frame Noise Conditioning: For training, $F$ frames are sampled from each scene. Each frame is individually injected with a noise level $t_i$ (sampled per-frame), allowing the model to learn robust denoising where conditioning context may itself be noisy (as will be the case during autoregressive generation when context comes from previous model outputs, not from ground-truth frames).
• Pairwise-Relative Camera Pose Encoding (CaPE): Instead of absolute camera pose encodings, CausNVS adopts CaPE, which encodes the relative geometry between query and context cameras without using any absolute coordinate system. Specifically, CaPE applies a rotation transformation to query and key vectors in attention layers using the relative pose matrix, making attention periodic under rotations and linear for translations. This approach encourages robust 3D consistency and ensures that key-value (KV) caching for attention remains valid even as the “current” time index advances.

3. Inference Strategies: Sliding-Window Attention, KV Caching, and Noise Conditioning

• Spatially Aware Sliding-Window: At inference, CausNVS limits attention in each query to a spatially local window—the $K$ reference views that are closest in pose to the target camera. Since CausNVS is trained with randomly subsampled and unordered sets of view frames, this sliding-window naturally supports both diverse camera trajectories and efficient computation, focusing on relevant context without needing to attend to all frames globally.
• Key–Value (KV) Caching: By using CaPE (encoding only relative positions), frame-level KV pairs from already generated frames can be cached and reused efficiently as the generation window advances. This makes long autoregressive rollouts computationally tractable and supports streaming scenarios.
• Noise Conditioning Augmentation: To address “drift” (compounding errors from long autoregressive generation), previously generated views are treated as noisy—assigned small, nonzero noise levels when recycled as context. This noise augmentation provides robustness, preventing error accumulation by exposing the model to imperfect context during training and inference.

4. Quantitative and Qualitative Performance

CausNVS was evaluated on a range of 3D scene benchmarks, including RealEstate10K, LLFF, and DL3DV:

• Quantitative Metrics: On standard novel view synthesis metrics (e.g., PSNR, SSIM, LPIPS), CausNVS achieves strong results, closely tracking or outperforming both non-autoregressive and generative baselines.
• Scalability: With increasing numbers of input views, performance (PSNR) improves consistently. In experiments with the SEVA benchmark, the model’s flexible context handling allows for adaptive fidelity gains as more information becomes available.
• Long-horizon Generation: Autoregressive rollouts up to 10$\times$ the sequence length seen at training sustain visually coherent and geometrically plausible view synthesis, with minimal degradation.
• Objective: The denoising diffusion loss in the latent image space is:

$$L_{\text{causal}} = \mathbb{E}_{(x_i, p_i), t_i, \epsilon_i} \left[ \sum_i \| \hat{\epsilon}_\theta(v_i | v_{<i}) - \epsilon_i \|_2^2 \right],$$

where $\hat{\epsilon}_\theta$ is the predicted noise for frame $i$ given the causal context.

5. Architectural Features and Design Choices

Component	Function	Role in CausNVS
Causal Masking	Limits frame-wise attention to preceding frames	Enforces autoregressive, causal generation ordering
Per-frame Noise	Samples a noise level per frame at training	Provides robustness to denoising under noisy/auto-regressive context
CaPE (Pairwise-Relative Enc)	Encodes relative query-key pose information	Achieves camera-invariant, drift-free attention and KV caching
Sliding-Window Attention	Restricts context to $K$ spatial neighbors	Enhances efficiency and locality in context aggregation
KV Caching	Stores past query-key-value pairs for reuse	Supports fast, scalable autoregressive rollouts
Noise Conditioning Augment.	Treats autoregressive predictions as noisy context	Mitigates accumulation of errors (“drift”) over long sequences

6. Applications and Broader Implications

• Immersive AR/VR and Streaming: CausNVS’s flexible context handling and autoregressive synthesis support dynamic, streaming “novel view” generation—critical for AR/VR and mixed reality systems where viewpoints may shift unpredictably and incremental results are required in real time.
• World Modeling and 3D Reasoning: The separation of generation context and output, plus the use of explicitly parameterized relative pose encodings, suggests the approach is well-suited for robotics and world-modeling agents where new observations must be synthesized given arbitrary (possibly long) action sequences and trajectories.
• Research Impact: By demonstrating that autoregressive, causal masking combined with relative attention and robust conditioning stabilizes long-horizon generation, CausNVS opens frontiers for broader spatial–temporal generative modeling. This establishes a foundation for extending diffusion models to other structured, sequential data (e.g., spatiotemporal video generation, trajectory-conditioned planning).
• Open Limitations and Future Work: The multi-step denoising diffusion nature of CausNVS may limit real-time applications for high-resolution outputs. The original report highlights that speeding up inference via techniques such as consistency training or knowledge-distillation is a target for further research.

7. Relation to Prior Approaches and Distinctive Advantages

CausNVS distinguishes itself by:

• Supporting arbitrary input–output configurations without retraining or architecture changes.
• Eliminating fixed-size context restrictions and global attention bottlenecks.
• Achieving drift-resilient, stable generation even along extended (long-horizon) camera trajectories due to its causal masking and per-frame noise methodology.
• Providing robust camera pose conditioning using parameter-free, relative encodings, which avoids problems associated with memorizing absolute coordinates or accumulating drift in global pose representations.

In contrast, non-autoregressive baselines require all output views to be specified a priori, scale poorly to long sequences, and cannot efficiently support streaming or interactive generation because the model must perform simultaneous denoising for the entire set of target frames.

---

CausNVS thus defines a state-of-the-art framework for flexible 3D novel view synthesis, integrating autoregressive, causal sequence modeling with pose-aware attention and efficient inference strategies to support a full range of practical applications in dynamic scene generation and spatial world modeling (Kong et al., 8 Sep 2025).