Scalable View Synthesis Model (SVSM)

• The paper introduces SVSM as a transformer-based architecture that fuses image patches with pose embeddings, employing token disentanglement to prevent artifact leakage in novel view synthesis.
• It leverages strong camera-pose encodings and synthetic data augmentation via diffusion models to enhance robustness and achieve state-of-the-art PSNR and LPIPS metrics.
• Empirical results show that SVSM outperforms traditional geometry-based methods and prior LVSMs by efficiently scaling to diverse scenes and large model capacities.

A Scalable View Synthesis Model (SVSM) is a class of neural architectures and training protocols designed to enable efficient, high-fidelity, and generalizable novel view synthesis from sparse inputs, with explicit focus on scaling to diverse data, large model capacities, and broad scene variability. Recent SVSMs leverage transformer or diffusion backbones, strong camera-pose encodings, and data-driven regularization strategies to achieve state-of-the-art generalization, robustness to out-of-domain samples, and computational efficiency in comparison to traditional geometry-based techniques.

1. Architectural Components and Token Processing

SVSMs are typically instantiated as large transformer-based models that ingest a set of posed source images and generate photorealistic renderings from novel camera viewpoints. The canonical processing pipeline includes:

• Patch-based tokenization: Each source image, along with per-pixel Plücker-ray embeddings encoding 3D ray origins and directions, is partitioned into non-overlapping $p\times p$ patches. A linear projection fuses the appearance and geometry information into $D$-dimensional tokens $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$. Target view ray patches $P^t_j$ are similarly embedded as $T_j = \mathrm{Linear}(P^t_j)$.
• Backbone transformer: A deep $\mathcal O(24)$-layer stack processes the concatenated source and target tokens via multi-head self-attention and MLP sublayers. Recent SVSMs (Nair et al., 8 Sep 2025) employ a token-disentanglement mechanism: each token is tagged as source ($\delta=0$) or target ($\delta=1$), and this binary indicator modulates all sublayer operations via learned scale-shift parameters, improving feature separation.
• Output decoding: Target (novel view) tokens are linearly projected to $p\times p\times 3$ RGB patch predictions, which are unpatchified to form the output image.

This process unifies data-driven feature learning with explicit geometric cues, while eschewing hand-crafted 3D representations, such as volumetric rendering or plane-sweep pipelines.

2. Token Disentanglement and Modulation

A signature advance in recent SVSMs is explicit token disentanglement (Nair et al., 8 Sep 2025), addressing artifact propagation when models are trained with synthetic or corrupted data. The strategy is as follows:

• Token role indicator: Each token is assigned a binary $\delta$ based on its provenance (source/target).
• Conditioned modulation: At each transformer layer $D$0, modulation parameters $D$1 are generated from $D$2, and applied to every token $D$3 as: $D$4.
• Attention separation: Attention and MLP sublayers are modulated separately per group (source and target), ensuring minimal leak-through of source-specific artifacts (e.g., diffusion model hallucinations) into the target generation process.

This mechanism replaces the undifferentiated processing in prior LVSMs (Jin et al., 2024), improving both in-distribution and cross-dataset performance without incurring additional loss terms or optimization overhead.

3. Scaling with Synthetic Data and Data Diversity

Real-world scaling of SVSMs demands massive training sets with diverse scenes and poses. However, existing datasets are often insufficient. SVSMs address this by:

• Synthetic data augmentation: Employing state-of-the-art diffusion models (e.g., CAT3D) to generate large volumes of synthetic multi-view data, sampled along random pose trajectories.
• Artifact avoidance: The original real image, rather than a synthetic view, is always used as the reconstruction target during training, mitigating the risk of learning to reproduce data-generation artifacts.
• Scalable generalization: The token-disentanglement transformer, trained on mixed real and synthetic data, demonstrates monotonically increasing PSNR with more synthetic samples (in contrast to baseline LVSMs, which degrade past a certain point) (Nair et al., 8 Sep 2025).
• Domain gap bridging: By disentangling source and target features, SVSMs robustly transfer knowledge from synthetic to real scenes, displaying minimal cross-dataset drop in performance.

4. Training Objectives, Optimization, and Hyperparameters

SVSMs employ a composite objective focusing on both pixel-level and perceptual fidelity:

• Total loss: $D$5, where $D$6 and $D$7 denotes LPIPS perceptual similarity.
• Optimization: AdamW, with $D$8, $D$9, weight decay $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$0 (excluding normalization layers), a peak learning rate $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$1, 2,500-step warmup, and linear decay.
• Exponential moving average of weights (decay $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$2) is critical for convergence and contributes up to $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$3 dB in PSNR (Nair et al., 8 Sep 2025).
• No explicit regularization or token disentanglement loss is required.

5. Scalability, Efficiency, and Data-Compute Laws

SVSMs are designed for both data and model scale, underpinned by systematic studies of compute-optimality (Kim et al., 24 Feb 2026):

• Model size and throughput: For 24-layer, $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$4-dim transformers, forward passes incur $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$5 TFLOPs, with training times dominated by dataset scale and token count ($$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$664 batch on 8$$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$7H100s yields $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$8 ms/sample at $$S_{ij} = \mathrm{Linear}([I^s_{ij}; P^s_{ij}]) \in \mathbb R^D$$9 resolution).
• Performance scaling: PSNR increases as a function of both synthetic data volume and number of source views ($P^t_j$0 dB, $P^t_j$1 dB, $P^t_j$2 dB).
• Compute-effective architecture: Encoder-decoder variants amortize context encoding over all targets, yielding a superior Pareto frontier (LPIPS or PSNR per training FLOP) relative to decoder-only models at fixed compute budgets (Kim et al., 24 Feb 2026).
• Power-law scaling: Test LPIPS and PSNR empirically follow scaling exponents with respect to total training compute. Both SVSM and LVSM exhibit exponents $P^t_j$3, but SVSM is 2–4$P^t_j$4 more compute-efficient in empirical Pareto frontier construction.

6. Empirical Results and Comparative Performance

SVSMs set new state-of-the-art results across both in-distribution and out-of-distribution benchmarks, as shown in the following quantitative comparisons (Nair et al., 8 Sep 2025):

Dataset	GPNR	PixelSplat	MVSplat	DepthSplat	LVSM	SVSM (Tok-D)
RealEstate10k	24.11	25.89	26.39	27.44	28.89	30.02
SSIM	0.793	0.858	0.869	0.887	0.894	0.919
LPIPS	0.255	0.142	0.128	0.119	0.108	0.058

Cross-dataset transfer (train on RealEstate10K, test on ACID/DL3DV) confirms robust generalization and enhanced benefits from synthetic augmentation. Ablations show a further $P^t_j$5 dB gain from full Tok-D modulation, $P^t_j$6 dB from extra real data, and marked advantage over LVSMs in both pixel accuracy and perceptual metrics.

7. Limitations, Open Problems, and Future Directions

Despite demonstrable strengths, current SVSMs have the following limitations and identified research directions (Nair et al., 8 Sep 2025):

• Failure cases: Hallucinations in unseen-geometry regions (e.g., newly visible occluded surfaces), artifacts on highly reflective/sparse regions, and GPU memory bottlenecks (≈1,000 tokens/image at $P^t_j$7 patches, 24 layers).
• Architectural improvements: Hierarchical or windowed transformer blocks, linear-time/state-space attention (e.g., Mamba), and incorporation of explicit geometry only where beneficial (e.g., dynamic plane priors) are proposed for memory and fidelity gains.
• Extended scaling: Future SVSMs may explore mixed sparse attention layouts, data-efficient adaptation strategies, or test-time self-supervision for enhanced generalization beyond the current large-scale data regime.

---

In summary, the Scalable View Synthesis Model paradigm centers on transformer-based architectures, data-driven feature disentanglement, and robust scaling strategies leveraging both real and high-quality synthetic data. Explicit token disentanglement, efficient hyperparameterization, and systematic adherence to scaling laws enable SVSMs to outperform both 3D-biased pipelines and prior feed-forward models under demanding multi-view, open-domain, and cross-dataset scenarios (Nair et al., 8 Sep 2025, Jin et al., 2024, Kim et al., 24 Feb 2026).