FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation

Abstract: The development of generalizable Novel View Synthesis (NVS) models is critically limited by the scarcity of large-scale training data featuring diverse and precise camera trajectories. While real-world captures are photorealistic, they are typically sparse and discrete. Conversely, synthetic data scales but suffers from a domain gap and often lacks realistic semantics. We introduce FreeScale, a novel framework that leverages the power of scene reconstruction to transform limited real-world image sequences into a scalable source of high-quality training data. Our key insight is that an imperfect reconstructed scene serves as a rich geometric proxy, but naively sampling from it amplifies artifacts. To this end, we propose a certainty-aware free-view sampling strategy identifying novel viewpoints that are both semantically meaningful and minimally affected by reconstruction errors. We demonstrate FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR on challenging out-of-distribution benchmarks. Furthermore, we show that the generated data can actively enhance per-scene 3D Gaussian Splatting optimization, leading to consistent improvements across multiple datasets. Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision. Project page: https://mvp-ai-lab.github.io/FreeScale.

• The paper introduces FreeScale, a framework that augments sparse 3D scenes with photorealistic free-views using a certainty-aware sampling technique and view graph filtering.
• It employs a continuous 3D Gaussian Splatting representation combined with diffusion-based rectification to ensure both high image fidelity and precise pose accuracy.
• Experimental results demonstrate significant gains in PSNR, SSIM, and LPIPS for both feed-forward models and per-scene optimizations, all with minimal runtime overhead.

FreeScale: Certainty-Aware Free-View Generation for Scalable 3D Scene Data Augmentation

Introduction

The paper "FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation" (2604.10512) addresses a critical bottleneck in training generalizable Novel View Synthesis (NVS) models: the lack of large-scale, diverse, photorealistic datasets with rich camera trajectory coverage. Existing real-world data captures are typically sparse and suffer from labor-intensive acquisition and calibration, whereas synthetic datasets inherently suffer from a domain gap and often lack realistic semantics and scene geometry diversity. This paper proposes FreeScale, a framework that automatically scales sparse real-world multi-view data into dense, high-fidelity, photorealistic free-views accompanied by accurate, diverse camera poses. The generated data is directly applicable for the training of feed-forward NVS models and can benefit per-scene optimization for explicit scene representations such as 3D Gaussian Splatting (3DGS).

FreeScale Framework and Pipeline

FreeScale fundamentally augments a dataset $\mathcal{D}$ consisting of sparse real-world scenes, where each scene $S_k$ is a set of image-pose pairs. The core procedure comprises three stages:

1. 3D Scene Reconstruction: Given $N_k$ sparse real images with known poses, a continuous 3DGS representation $\mathscr{G}_k$ is optimized as a geometric proxy.
2. Certainty-Aware Free-View Synthesis: Using a scene-certainty grid derived from the 3DGS, viewpoint candidates are densely sampled along various predefined canonical and non-canonical camera trajectories designed to maximize viewpoint diversity and geometric coverage. Each candidate is scored for certainty based on the spatial distribution and density of the underlying Gaussians and pruned using a view graph constructed to ensure high information gain and minimal redundancy.
3. Free-View Refinement and Image Rectification: Rendered candidate images undergo no-reference image quality evaluation (BRISQUE) and geometric content assessment (depth range). Weak candidates are either rejected or rectified via pose interpolation. Subsequently, a single-step diffusion model is used for photorealism rectification, guided by the view graph for optimal reference image selection, mitigating the typical artifact injection observed in distance-based reference selection.

   Figure 1: The FreeScale pipeline generates dense, high-quality free-views from sparse real-world sequences via certainty-guided sampling, view filtering, and diffusion-based rectification.

Certainty-Aware Sampling and View Graph Construction

The hallmark of FreeScale is its certainty-guided, diversity-maximizing candidate viewpoint selection. By constructing a certainty grid over the 3DGS volume with resolution $R^3$ (typically $R=128$), the local scene coverage is quantified, and candidate viewpoints are placed to exhaustively sample under-constrained, information-rich regions. Ten canonical and non-canonical camera trajectory modes (e.g., orbit, spiral, lemniscate, cinematic fly-throughs) generate diverse candidate pools. Pose jittering is applied to further diversify the pool.

A certainty-based view graph enables selection of a maximally informative and minimally redundant set of around 500 free-views per scene. Each node corresponds to a candidate pose, scored by weighted occupancy and visibility of the underlying certainty grid. Weighted intersection-over-union (WIoU) between nodes provides an efficient redundancy filter. Non-Maximum Suppression (NMS) in this graph framework ensures both semantic and geometric coverage.

Figure 2: Predefined camera trajectories guarantee comprehensive scene exploration for candidate viewpoint generation.

Image Rectification and Diffusion-Based Enhancement

To ensure the generated free-views are both photorealistic and geometrically faithful, FreeScale extends diffusion-based image refinement along two axes:

• Quality Assurance and Pose Rectification: Images are subject to strict post-render filtering based on BRISQUE and normalized depth coverage. Failing candidates are iteratively repositioned toward anchor training poses.
• View-Graph Guided Diffusion: Unlike recent diffusion-based enhancement techniques relying on nearest neighbors in pose space (e.g., DIFIX3D+), FreeScale leverages its view graph for reference selection, ensuring maximal overlap and thus minimizing hallucinatory artifacts during rectification.

  Figure 3: The view graph enables robust reference selection for diffusion-based image rectification by guaranteeing shared scene visibility.

Scaling Feed-Forward and Optimization-Based NVS

The certainty-aware free-view images generated by FreeScale can be used in two primary application regimes:

• Generalizable Feed-Forward Model Training: Models like LVSM, constrained by limited training-view diversity, benefit substantially from the inclusion of FreeScale-generated free-views. Graph-guided curriculum learning is employed: initially, batches comprise views with strong WIoU adjacency (fostering stability), gradually introducing batches with larger geometric disparity (amplifying generalization).
• Per-Scene 3DGS Optimization: FreeScale adds selected high-complementarity free-views (lowest WIoU with training set) as pseudo-ground truth during 3DGS optimization. Loss weighting is adapted based on free-view BRISQUE, allowing robust integration even in sparse or incomplete initialization regimes.

Empirical Results

Quantitative benchmarks on DL3DV, MipNeRF360, Nerfbusters, and Tanks and Temples datasets demonstrate that FreeScale-augmented training leads to substantial improvements in PSNR, SSIM, and LPIPS for both feed-forward and 3DGS models. Notably:

• A gain of 2.7 dB in PSNR is observed for LVSM on large camera motion (out-of-domain) benchmarks when training data is augmented by FreeScale.
• In per-scene 3DGS optimization, FreeScale achieves consistently higher PSNR/SSIM and reduces artifact prevalence, with runtime overheads substantially less than non-integrated diffusion-based methods.

  Figure 4: Feed-forward models trained with FreeScale exhibit superior geometric detail and viewpoint generalization in challenging scene regions.

  

  Figure 5: On Nerfbusters, FreeScale produces high-fidelity renders in unobserved regions, suppressing floaters and geometric noise by augmenting with certainty-sampled views.

Ablations confirm the robustness of the framework to data sparsity; even with incomplete initialization (5% input views), FreeScale consistently extracts informative geometry for effective augmentation. Removing the certainty grid or the view-graph-guided sampling leads to degraded generalization and less stable model training. Camera trajectory diversity (beyond orbit-only sampling) is critical for full scene coverage and high-fidelity results.

Figure 6: FreeScale’s view graph mitigates rectification errors seen with simple distance-based reference selection, promoting consistent geometric alignment during photorealistic enhancement.

Limitations

Primary limitations arise in the reliance on the external diffusion prior for photorealism rectification, which can still hallucinate content in severely under-constrained, view-dependent reflection regions or misinterpret 3DGS floaters. Extreme scene sparsity conditions may result in too few valid free-views. Preliminary ablations show that feed-forward models are robust to minor generative inconsistencies, but optimizing the diffusion model specifically for certainty-masked regions is proposed as a future direction.

Figure 7: Failure cases include incorrect handling of complex reflections and oversharpened floaters, highlighting the limitations of external diffusion priors.

Conclusion and Implications

FreeScale provides a scalable, practical solution for the lack of camera trajectory diversity and photorealistic coverage in current multi-view datasets. The certainty-aware, graph-guided pipeline delivers photorealistic, pose-accurate free-views with substantial empirical benefits across both feed-forward and per-scene optimization regimes. By enabling scalable scene-centric data augmentation wholly from sparse real-world images, FreeScale opens a new avenue for training robust, generalizable, and 3D-aware neural scene representations, ultimately reducing reliance on laborious ground-truth data capture and mitigating synthetic domain constraints. Progress in integrating view-dependent certainty directly into diffusion priors or co-training the refinement and sampling pipeline could further enhance the quality and scalability of 3D scene learning systems.