Multi-GPU Gaussian Splatting

• Multi-GPU Gaussian Splatting is a distributed 3D rendering approach that uses anisotropic Gaussians to represent scenes, overcoming single-GPU memory constraints.
• It leverages custom PyTorch backends, model sharding, and sparse all-to-all communication to efficiently balance workloads across several GPUs.
• The method achieves scalable, high-fidelity reconstructions for graphics and HPC, preserving quality metrics like PSNR/SSIM in urban and scientific visualizations.

Multi-GPU Gaussian Splatting refers to a set of distributed computing methodologies and software systems enabling the training, optimization, and rendering of large-scale 3D scene representations—composed of tens to hundreds of millions of anisotropic Gaussians—across several GPUs and compute nodes. These approaches remove the single-GPU memory constraint that previously limited Gaussian splatting’s scalability, unlocking city-scale, high-detail reconstructions, rapid scientific visualization, and parallel semantic scene encoding. The field encompasses operator-level distribution via custom PyTorch backends, model/data sharding, sparse communication protocols, workload balancing, and multi-node orchestration for both graphics and scientific/HPC contexts (Cong et al., 9 Jun 2026, Han et al., 15 Sep 2025, Han et al., 5 Sep 2025, Zhao et al., 2024, Szilagyi et al., 12 May 2025).

1. Mathematical and Algorithmic Foundations

Multi-GPU Gaussian splatting frameworks use 3D scenes parameterized as sets of Gaussian primitives:

$$S = \{ (\mu_k, \Sigma_k, a_k) \mid k = 1 \ldots K \}$$

where $\mu_k \in \mathbb{R}^3$ is the center, $\Sigma_k \in \mathbb{R}^{3 \times 3}$ (often anisotropic) the covariance, and $a_k \in \mathbb{R}^{C \times L}$ the view-dependent color (frequently as spherical harmonic coefficients). The spatial density function for each Gaussian is:

$$G_k(x) = \exp \left( -\tfrac{1}{2} (x - \mu_k)^\top \Sigma_k^{-1} (x - \mu_k) \right)$$

Rendering projects each 3D Gaussian into a screen-space elliptical footprint $\Sigma_{2D, k}$ via local Jacobian multiplication, then splats the resulting 2D ellipse on the image plane, weighted by transmittance $T_k(u)$ and color $c_k(\omega_k)$. Rasterization is order-dependent (commonly sorted by depth) and is fused with accumulation in parallel kernels (Cong et al., 9 Jun 2026, Han et al., 5 Sep 2025).

Loss functions blend photometric error over all pixels and images with regularization on Gaussian extents:

$$L(\{\mu_k, \Sigma_k, a_k\}) = \sum_{v=1}^V \sum_{u \in \text{pixels}} \|C_{v,u}^{\mathrm{render}} - C_{v,u}^{\mathrm{gt}}\|^2 + \lambda \sum_{k=1}^M \|\Sigma_k\|_F^2$$

Gradients with respect to $\mu_k$, $\mu_k \in \mathbb{R}^3$0, $\mu_k \in \mathbb{R}^3$1 are computed via automatic differentiation.

For language-augmented splatting, per-Gaussian embeddings $\mu_k \in \mathbb{R}^3$2 are computed as weighted aggregates of per-mask CLIP features extracted from SAM masks, normalized over all image-masked contributions—again, embarrassingly parallel across image batches (Szilagyi et al., 12 May 2025).

2. System Design and Operator Distribution

The primary frameworks for multi-GPU Gaussian splatting include distributed custom backends (e.g., torch-dgx), hybrid model/data parallel strategies, and distributed pipeline orchestration spanning both single-node and multi-node scenarios.

• torch-dgx exposes all available GPUs as a single logical PyTorch device via CUDA unified memory pools and NVLink cross-GPU page migration. The system intercepts all tensor allocations for device="dgx", dispatches operator calls via a kDGX key, and partitions operator workloads by their natural axes of parallelism: per-Gaussian (e.g., spherical harmonics evaluations), per-pixel (e.g., rasterization, SSIM loss), or dense linear algebra (convolution, batch-norm). Synchronization is performed via stream-ordered cudaEvents, with no explicit user-side cross-device communication (Cong et al., 9 Jun 2026).
• Shard-based model/data parallelism splits the global set of Gaussian parameters evenly across GPUs, assigning to each device a subset of Gaussians, image tiles, or views. Some systems further stripe training images or camera views across devices for data parallelism (Han et al., 5 Sep 2025, Zhao et al., 2024).
• Operator-level logic: Custom CUDA kernels underpin both forward and backward passes for Gaussian operations. On each GPU, local kernels process assigned slices and write directly to unified memory; only lightweight cross-device synchronizations are needed to avoid race conditions (Cong et al., 9 Jun 2026).

3. Communication Protocols and Scalability Mechanisms

Scaling to hundreds of millions or even billions of Gaussians necessitates specialized communication and load balancing strategies:

• Unified Memory and NVLink: On a single node, CUDA unified memory with NVLink interconnect provides high-throughput, all-to-all access across multi-GPU memory, allowing Gaussian, image, and tensor data to be transparently read and written by any device (Cong et al., 9 Jun 2026).
• Sparse All-to-All and NCCL: Distributed approaches (e.g., Grendel) perform sparse all-to-all communication of Gaussians by transmitting only those primitives intersecting a given device's image or tile partitions. Allreduce synchronizations aggregate gradients after each local forward-backward computation step, overlapping communication with compute when possible to conceal latency (Zhao et al., 2024, Han et al., 5 Sep 2025). Within a node, NCCL with NVLink optimizes fusion of gradient vectors; between nodes, MPI is used for control and synchronization (Han et al., 15 Sep 2025).
• Dynamic Workload Partitioning: To address load imbalance caused by spatially variable Gaussian density, image tiles or pixel blocks are dynamically repartitioned according to measured execution time, ensuring equalized compute per device (Zhao et al., 2024).
• Ghost-Cell Padding and Merge: For multi-node settings, ghost cells (overlapping boundaries) are introduced at spatial shard interfaces to eliminate seam artifacts during rendering, with final duplicate removal during post-training merges (Han et al., 15 Sep 2025).

4. Implementation Patterns and Framework Integration

Implementations blend PyTorch, custom CUDA, CUDA stream/event primitives, and standard distributed systems libraries:

• Memory Management: torch-dgx replaces all tensor allocations with cudaMallocFromPoolAsync into unified memory pools with stream ordering per device. Free operations mirror this pattern, preventing host-thread stalls (Cong et al., 9 Jun 2026).
• Operator Registration and Execution: C++ operators are registered as fragments under torch's dispatch mechanism; in execution, kernel launches iterate over device count, partition work, and launch per-stream CUDA kernels with offset management (Cong et al., 9 Jun 2026).
• Custom Gaussian Operators: Forward Gaussian splatting first partitions the workload (by Gaussian or pixel), then invokes radix or merge-path sorting and fused rasterization on each device. Backpropagation leverages system-scope atomics for cross-GPU gradient accumulation (Cong et al., 9 Jun 2026).
• Integration Recipes: Replacing "cuda" with "dgx" in PyTorch model code suffices for multi-GPU capacity. Tile sharding, batch size adjustment, Morton sort for densification, and conditional memory prefetching are supported for further memory and throughput tuning (Cong et al., 9 Jun 2026).

5. Performance, Scaling, and Workload Characterization

Empirical studies report linear or super-linear scaling up to 8 A100s (and clusters of 8 nodes × 4 GPUs), provided problem sizes are sufficiently large.

Representative Performance Results

System	Dataset	#GPUs	Gaussians	Time (min)	Speedup
torch-dgx (Cong et al., 9 Jun 2026)	Puerto Rico City	8 A100	1B	N/A	N/A
torch-dgx (Cong et al., 9 Jun 2026)	Gettysburg (13km²)	4×A100+H100	145M	~19.5 hr	N/A
Grendel (Zhao et al., 2024)	Rubble	1	11.2M	N/A	1×
Grendel (Zhao et al., 2024)	Rubble	16	40.4M	N/A	4–6×
Multi-GPU HPC (Han et al., 5 Sep 2025, Han et al., 15 Sep 2025)	Kingsnake	1	4M	48.0	1×
		4	4M	8.5	5.65×
Distributed 3D-GS (Han et al., 15 Sep 2025)	Richtmyer–Meshkov	8 nodes, 32 GPUs	106.7M	10.2	3.1×

Rendering quality, measured by PSNR/SSIM, is preserved at scale. Training time per epoch and throughput (img/s) grow nearly linearly with resource count until communication overhead or compute saturation is reached. Super-linear scaling is occasionally achieved due to improved memory locality or cache effects (Han et al., 5 Sep 2025).

6. Applications and Specialized Pipelines

City-scale and scientific visualization: Multi-GPU splatting supports reconstructions at tens to hundreds of millions of Gaussians for urban scenes, and for scientific visualization of isosurfaces with dense simulation data. Synthetic multispectral or LIDAR data can be directly incorporated, and batched multi-view training is practical (Cong et al., 9 Jun 2026, Han et al., 5 Sep 2025, Zhao et al., 2024, Han et al., 15 Sep 2025).

Language-augmented scene encoding: Systems such as SLAG integrate 2D vision-language features (from SAM and CLIP) into 3D scene Gaussians. Multi-GPU distribution accelerates embedding computation (up to 19× over single-GPU OpenGaussian), with all-reduce fusion used for normalized weighted aggregation of per-Gaussian semantic descriptors (Szilagyi et al., 12 May 2025).

HPC and in situ scientific pipelines: Distributed pipelines on clusters (e.g., Polaris at ALCF) enable real-time and in situ visualization during simulation, using MPI/HDF5 for file and metadata management and allowing deployments tailored to node and GPU count (Han et al., 15 Sep 2025).

7. Best Practices, Limitations, and Outlook

Recommended practices include sharding Gaussians uniformly for transformation phases, assigning spatially contiguous image or pixel blocks per GPU, adopting sparse all-to-all communication wherever feasible, and dynamically rebalancing workloads on measured compute time. For optimization hyperparameters (Adam/SGD), scaling the learning rate by $\mu_k \in \mathbb{R}^3$3 and adjusting momentum terms according to batch size accelerates convergence in batched multi-view contexts (Zhao et al., 2024).

Key limitations stem from bottlenecks in source image masking (e.g., high-resolution SAM grids) and the raw memory demands of very large Gaussian sets. Systems mitigate memory pressure by chunked aggregation and external vector-database storage (Szilagyi et al., 12 May 2025). Communication overhead may become dominant at extreme scale or across weak interconnects, but intra-node unified memory and NVLink keep local sync costs minimal (Cong et al., 9 Jun 2026, Han et al., 5 Sep 2025).

Multi-GPU Gaussian splatting, by exposing operator-level distribution and transparent multi-device address spaces, has enabled high-fidelity, large-scale reconstructions and semantic scene representations that were previously infeasible under single-GPU constraints. Its applicability to both graphics and scientific/HPC workloads continues to expand as corresponding software (e.g., torch-dgx, Grendel, SLAG) matures (Cong et al., 9 Jun 2026, Zhao et al., 2024, Szilagyi et al., 12 May 2025, Han et al., 5 Sep 2025, Han et al., 15 Sep 2025).