TideGS: Scalable Training of Over One Billion 3D Gaussian Splatting Primitives via Out-of-Core Optimization

Abstract: Training 3D Gaussian Splatting (3DGS) at billion-primitive scale is fundamentally memory-bound: each Gaussian primitive carries a large attribute vector, and the aggregate parameter table quickly exceeds GPU capacity, limiting prior systems to tens of millions of Gaussians on commodity single-GPU hardware. We observe that 3DGS training is inherently sparse and trajectory-conditioned: each iteration activates only the Gaussians visible from the current camera batch, so GPU memory can serve as a working-set cache rather than a persistent parameter store. Building on this insight, we introduce TideGS, an out-of-core training framework that manages parameters across an SSD-CPU-GPU hierarchy via three synergistic techniques: block-virtualized geometry for SSD-aligned spatial locality, a hierarchical asynchronous pipeline to overlap I/O with computation, and trajectory-adaptive differential streaming that transfers only incremental working-set deltas between iterations. Experiments show that TideGS enables training with over one billion Gaussians on a single 24 GB GPU while achieving the best reconstruction quality among evaluated single-GPU baselines on large-scale scenes, scaling beyond prior out-of-core baselines (e.g., approximately 100M Gaussians) and standard in-memory training (e.g., approximately 11M Gaussians).

• The paper presents TideGS, which trains over 1 billion 3D Gaussian splatting primitives on a single GPU by materializing only the dynamic working set during training.
• It introduces a novel out-of-core framework using block virtualization, hierarchical asynchronous execution, and trajectory-adaptive differential streaming to overcome VRAM constraints.
• Experiments demonstrate high reconstruction fidelity and competitive throughput, outperforming prior multi-GPU approaches in cost-effective, large-scale neural rendering.

TideGS: Out-of-Core Billion-Scale 3D Gaussian Splatting on a Single GPU

Overview and Motivation

TideGS presents a system for training over one billion 3D Gaussian Splatting (3DGS) primitives on a single commodity GPU (e.g., 24 GB VRAM), addressing the acute VRAM bottleneck inherent to the memory structure of 3DGS models. The central insight is that only a small, dynamic subset of Gaussians—those visible from a camera's viewpoint during each training iteration—undergo gradient updates, yielding strong sparsity and temporal locality. By virtualizing the parameter table across the SSD–CPU–GPU hierarchy and materializing only the currently active working set in GPU memory, TideGS sidesteps the VRAM limitations that confine prior approaches to $\leq 100$ million primitives.

Figure 1: TideGS enables city-scale 3DGS training on a single GPU by virtualizing the Gaussian parameter table across the SSD–CPU–GPU hierarchy and materializing only the trajectory-activated working set in VRAM.

The system integrates three synergistic strategies: block-virtualized geometry with spatial locality, a hierarchical asynchronous pipeline overlapping I/O with computation, and trajectory-adaptive differential streaming that reduces inter-tier data traffic. TideGS thus enables high-scale, high-fidelity novel view synthesis for city-scale neural rendering using commodity hardware.

Out-of-Core Pipeline and System Design

At its core, TideGS implements a multi-tier memory hierarchy for Gaussian parameter management and scheduling, depicted in the system pipeline:

Figure 2: The TideGS pipeline. (a) SSD-resident blocks are cached in CPU RAM and materialized in GPU VRAM as a working set. (b) Trajectory-adaptive differential streaming stages only incremental working-set deltas, minimizing transfer costs.

Block Virtualization and Visibility Filtering

TideGS partitions the entire parameter table into SSD-aligned spatial blocks, using Morton-ordering to align spatial locality with storage layout. Bounding spheres are constructed for each block, and a coarse CPU-side 6-plane frustum culling test selects blocks likely to be visible to the current camera batch. This reduces redundant data staging from SSD, since only relevant blocks are transferred up the memory hierarchy. Fine-grained per-Gaussian filtering is then performed on the GPU for exact visibility masking.

Figure 3: Block virtualization partitions Gaussians into SSD-aligned blocks, summarized by bounding spheres. Coarse CPU-side culling precedes finer GPU-level filtering for efficient visibility selection.

Hierarchical Asynchronous Execution

To hide I/O latency and saturate throughput, TideGS employs an asynchronous execution model:

• SSD Layer: Employs log-structured append-only segments for efficient sequential writes and reduced SSD wear.
• CPU Layer: Maintains a warm LRU cache, tracking dirty blocks and flushing updates only on eviction or explicit synchronization, thereby batching writes.
• GPU Layer: Capacity-bounded VRAM residency allows rendering and backpropagation exclusively on the current working set, minimizing memory footprint.

Crucially, data transfers up or down the hierarchy (SSD$\to$CPU$\to$GPU) are overlapped with GPU computation, using dedicated streams for each path.

Trajectory-Adaptive Differential Streaming

Standard 3DGS training often samples camera viewpoints randomly, yielding low temporal locality. TideGS instead traverses camera trajectories in a clustered/TSP order, ensuring high overlap among consecutive working sets. A residency scoring mechanism, combining access recency and predicted next-step usefulness, manages the limited GPU working set. Only the delta—the set-difference between successive resident sets—is staged or evicted (incremental streaming), making VRAM traffic scale with working-set change rather than model size.

Experimental Results and Performance Evaluation

Scalability and Memory Bottleneck

On a single 24 GB GPU, previous in-memory or host-offload baselines cap out at 11M (Native 3DGS), $\sim$50M (Naive Offload), or $\sim$100M (CLM) Gaussians before VRAM or rasterization buffer overflows. TideGS decouples VRAM usage from total model size, with memory scaling only in the working set size, enabling training with $\mathbf{>1B}$ Gaussians; the limiting factor becomes SSD/CPU capacity and bandwidth rather than GPU memory.

Overhead and Consistency

In the in-memory regime (scenes $\leq$10M–25M Gaussians), TideGS achieves throughput within 12% of Native 3DGS and matches reconstruction quality (PSNR gap $<0.12$ dB), confirming minimal architectural overhead when parameter virtualization is not required.

Large-Scale Out-of-Core Training

For MatrixCity ($\sim$102M, $\sim$1.1B Gaussians), TideGS alone remains feasible at billion-scale, maintaining high GPU utilization (43–50%) and keeping per-iteration bandwidth and latency competitive even when streaming working-set deltas from the SSD. At $\to$0B Gaussians, TideGS achieves 26.1 dB PSNR, exceeding lower-capacity baselines.

Figure 4: Quality scaling on MatrixCity—TideGS achieves the highest PSNR (26.1 dB) at the billion-primitive scale while previous methods fail with OOM.

Ablations and Impact of Core System Components

Disabling trajectory-adaptive differential streaming increases PCIe traffic by up to 8.5$\to$1; removing asynchronous overlap triples iteration latency. Breaking spatial locality drops cache hit rates from 95% to 42%, further inflating data traffic. These effects degrade efficiency but do not compromise final reconstruction quality, highlighting the orthogonality of the system designs to core optimization objectives.

Training with Dense Initialization

Fixed-size, dense-initialized training (without densification) achieves parity with adaptive densification on both indoor and outdoor scenes, showing that model scaling does not fundamentally depend on incremental primitive growth in this workflow.

Figure 5: Dense-initialized fixed-size training matches densified training on the Mip-NeRF 360 bicycle scene.

Figure 6: Dense-initialized fixed-size training matches densified training on the bonsai scene.

View Ordering and Convergence

Trajectory-based view ordering, while exploiting spatiotemporal locality for data reuse, leads to minimal quality degradation and comparable convergence curves relative to traditional randomly shuffled SGD.

Figure 7: Convergence comparisons for randomized shuffling versus trajectory-based ordering show negligible impact on reconstruction quality.

Comparison with Multi-GPU Approaches

Whereas distributed, in-memory multi-GPU systems such as Grendel-GS and RetinaGS offer improved wall-clock convergence, they impose high hardware and engineering costs with diminishing capacity-per-dollar efficiency. TideGS expands the upper bound of trainable capacity on a single node and fits scenarios where scalable training is needed within commodity hardware constraints.

Figure 8: Wall-clock time-to-convergence—TideGS achieves billion-scale capacity per node with modest single-node hardware cost.

Figure 9: Training-iteration-wise convergence—TideGS effectively amortizes I/O without losing optimization efficiency.

Implications and Future Directions

TideGS demonstrates that out-of-core hierarchical management of 3D point-based or splatting representations can facilitate orders-of-magnitude scaling on commodity hardware without sacrificing reconstruction quality or throughput. Methodologically, this approach motivates sparsity- and locality-aware optimization in other memory-bound architectures, including neural radiance fields, large-scale embedding tables, or geometric transformers. Practically, city-scale and world-scale neural rendering and mapping become feasible for cost-sensitive applications beyond well-resourced research labs.

Potential future developments include:

• Integration of more complex optimizer-state streaming strategies for improved convergence in the high-churn regime.
• Adaptation to general point-based or hybrid representations in neural graphics.
• Closer coupling of differential streaming with task-aware active data selection for semi-supervised or online learning.

Conclusion

TideGS establishes a new paradigm for scalable 3DGS training by virtualizing the parameter table over hierarchical storage, tightly orchestrating data movement, and exploiting the inherent sparsity and locality of the rendering task. It is the first system to achieve billion-scale 3DGS optimization on a single commodity GPU with state-of-the-art reconstruction fidelity, offering a practical blueprint for future out-of-core neural scene representations.

---

Reference: "TideGS: Scalable Training of Over One Billion 3D Gaussian Splatting Primitives via Out-of-Core Optimization" (2605.20150)