---
title: Multi-GPU Gaussian Splatting
url: https://www.emergentmind.com/topics/multi-gpu-gaussian-splatting
type: topic
---

# Multi-GPU Gaussian Splatting

Multi-GPU Gaussian Splatting refers to a set of distributed computing methodologies and software systems enabling the training, optimization, and rendering of large-scale 3D scene representations—composed of tens to hundreds of millions of anisotropic Gaussians—across several GPUs and compute nodes. These approaches remove the single-GPU memory constraint that previously limited Gaussian splatting’s scalability, unlocking city-scale, high-detail reconstructions, rapid scientific visualization, and parallel semantic scene encoding. The field encompasses operator-level distribution via custom PyTorch backends, model/data sharding, sparse communication protocols, workload balancing, and multi-node orchestration for both graphics and scientific/HPC contexts [2606.11390, 2509.12138, 2509.05216, 2406.18533, 2505.08124].

## 1. Mathematical and Algorithmic Foundations

Multi-GPU Gaussian splatting frameworks use 3D scenes parameterized as sets of Gaussian primitives:
$$
S = \{ (\mu_k, \Sigma_k, a_k) \mid k = 1 \ldots K \}
$$
where $\mu_k \in \mathbb{R}^3$ is the center, $\Sigma_k \in \mathbb{R}^{3 \times 3}$ (often anisotropic) the covariance, and $a_k \in \mathbb{R}^{C \times L}$ the view-dependent color (frequently as spherical harmonic coefficients). The spatial density function for each Gaussian is:
$$
G_k(x) = \exp \left( -\tfrac{1}{2} (x - \mu_k)^\top \Sigma_k^{-1} (x - \mu_k) \right)
$$
Rendering projects each 3D Gaussian into a screen-space elliptical footprint $\Sigma_{2D, k}$ via local Jacobian multiplication, then splats the resulting 2D ellipse on the image plane, weighted by transmittance $T_k(u)$ and color $c_k(\omega_k)$. Rasterization is order-dependent (commonly sorted by depth) and is fused with accumulation in parallel kernels [2606.11390, 2509.05216].

Loss functions blend photometric error over all pixels and images with regularization on Gaussian extents:
$$
L(\{\mu_k, \Sigma_k, a_k\}) = \sum_{v=1}^V \sum_{u \in \text{pixels}} \|C_{v,u}^{\mathrm{render}} - C_{v,u}^{\mathrm{gt}}\|^2 + \lambda \sum_{k=1}^M \|\Sigma_k\|_F^2
$$
Gradients with respect to $\mu_k$, $\Sigma_k$, $a_k$ are computed via automatic differentiation.

For language-augmented splatting, per-Gaussian embeddings $E_k \in \mathbb{R}^{512}$ are computed as weighted aggregates of per-mask CLIP features extracted from SAM masks, normalized over all image-masked contributions—again, embarrassingly parallel across image batches [2505.08124].

## 2. System Design and Operator Distribution

The primary frameworks for multi-GPU Gaussian splatting include distributed custom backends (e.g., torch-dgx), hybrid model/data parallel strategies, and distributed pipeline orchestration spanning both single-node and multi-node scenarios.

- **torch-dgx** exposes all available GPUs as a single logical PyTorch device via CUDA unified memory pools and NVLink cross-GPU page migration. The system intercepts all tensor allocations for `device="dgx"`, dispatches operator calls via a `kDGX` key, and partitions operator workloads by their natural axes of parallelism: per-Gaussian (e.g., spherical harmonics evaluations), per-pixel (e.g., rasterization, SSIM loss), or dense linear algebra (convolution, batch-norm). Synchronization is performed via stream-ordered cudaEvents, with no explicit user-side cross-device communication [2606.11390].

- **Shard-based model/data parallelism** splits the global set of Gaussian parameters evenly across GPUs, assigning to each device a subset of Gaussians, image tiles, or views. Some systems further stripe training images or camera views across devices for data parallelism [2509.05216, 2406.18533].

- **Operator-level logic**: Custom CUDA kernels underpin both forward and backward passes for Gaussian operations. On each GPU, local kernels process assigned slices and write directly to unified memory; only lightweight cross-device synchronizations are needed to avoid race conditions [2606.11390].

## 3. Communication Protocols and Scalability Mechanisms

Scaling to hundreds of millions or even billions of Gaussians necessitates specialized communication and load balancing strategies:

- **Unified Memory and NVLink**: On a single node, CUDA unified memory with NVLink interconnect provides high-throughput, all-to-all access across multi-GPU memory, allowing Gaussian, image, and tensor data to be transparently read and written by any device [2606.11390].

- **Sparse All-to-All and NCCL**: Distributed approaches (e.g., Grendel) perform sparse all-to-all communication of Gaussians by transmitting only those primitives intersecting a given device's image or tile partitions. Allreduce synchronizations aggregate gradients after each local forward-backward computation step, overlapping communication with compute when possible to conceal latency [2406.18533, 2509.05216]. Within a node, NCCL with NVLink optimizes fusion of gradient vectors; between nodes, MPI is used for control and synchronization [2509.12138].

- **Dynamic Workload Partitioning**: To address load imbalance caused by spatially variable Gaussian density, image tiles or pixel blocks are dynamically repartitioned according to measured execution time, ensuring equalized compute per device [2406.18533].

- **Ghost-Cell Padding and Merge**: For multi-node settings, ghost cells (overlapping boundaries) are introduced at spatial shard interfaces to eliminate seam artifacts during rendering, with final duplicate removal during post-training merges [2509.12138].

## 4. Implementation Patterns and Framework Integration

Implementations blend PyTorch, custom CUDA, CUDA stream/event primitives, and standard distributed systems libraries:

- **Memory Management**: torch-dgx replaces all tensor allocations with `cudaMallocFromPoolAsync` into unified memory pools with stream ordering per device. Free operations mirror this pattern, preventing host-thread stalls [2606.11390].

- **Operator Registration and Execution**: C++ operators are registered as fragments under torch's dispatch mechanism; in execution, kernel launches iterate over device count, partition work, and launch per-stream CUDA kernels with offset management [2606.11390].

- **Custom Gaussian Operators**: Forward Gaussian splatting first partitions the workload (by Gaussian or pixel), then invokes radix or merge-path sorting and fused rasterization on each device. Backpropagation leverages system-scope atomics for cross-GPU gradient accumulation [2606.11390].

- **Integration Recipes**: Replacing `"cuda"` with `"dgx"` in PyTorch model code suffices for multi-GPU capacity. Tile sharding, batch size adjustment, Morton sort for densification, and conditional memory prefetching are supported for further memory and throughput tuning [2606.11390].

## 5. Performance, Scaling, and Workload Characterization

Empirical studies report linear or super-linear scaling up to 8 A100s (and clusters of 8 nodes × 4 GPUs), provided problem sizes are sufficiently large.

**Representative Performance Results**

| System                            | Dataset           | #GPUs   | Gaussians        | Time (min)   | Speedup   |
|------------------------------------|-------------------|---------|------------------|--------------|-----------|
| torch-dgx [2606.11390]             | Puerto Rico City  | 8 A100  | 1B               | N/A          | N/A       |
| torch-dgx [2606.11390]             | Gettysburg (13km²)| 4×A100+H100| 145M          | ~19.5 hr     | N/A       |
| Grendel [2406.18533]               | Rubble            | 1       | 11.2M            | N/A          | 1×        |
| Grendel [2406.18533]               | Rubble            | 16      | 40.4M            | N/A          | 4–6×      |
| Multi-GPU HPC [2509.05216, 2509.12138]| Kingsnake      | 1       | 4M               | 48.0         | 1×        |
|                                    |                   | 4       | 4M               | 8.5          | 5.65×     |
| Distributed 3D-GS [2509.12138]     | Richtmyer–Meshkov | 8 nodes, 32 GPUs | 106.7M   | 10.2         | 3.1×      |

Rendering quality, measured by PSNR/SSIM, is preserved at scale. Training time per epoch and throughput (img/s) grow nearly linearly with resource count until communication overhead or compute saturation is reached. Super-linear scaling is occasionally achieved due to improved memory locality or cache effects [2509.05216].

## 6. Applications and Specialized Pipelines

**City-scale and scientific visualization**: Multi-GPU splatting supports reconstructions at tens to hundreds of millions of Gaussians for urban scenes, and for scientific visualization of isosurfaces with dense simulation data. Synthetic multispectral or LIDAR data can be directly incorporated, and batched multi-view training is practical [2606.11390, 2509.05216, 2406.18533, 2509.12138].

**Language-augmented scene encoding**: Systems such as SLAG integrate 2D vision-language features (from SAM and CLIP) into 3D scene Gaussians. Multi-GPU distribution accelerates embedding computation (up to 19× over single-GPU OpenGaussian), with all-reduce fusion used for normalized weighted aggregation of per-Gaussian semantic descriptors [2505.08124].

**HPC and in situ scientific pipelines**: Distributed pipelines on clusters (e.g., Polaris at ALCF) enable real-time and in situ visualization during simulation, using MPI/HDF5 for file and metadata management and allowing deployments tailored to node and GPU count [2509.12138].

## 7. Best Practices, Limitations, and Outlook

Recommended practices include sharding Gaussians uniformly for transformation phases, assigning spatially contiguous image or pixel blocks per GPU, adopting sparse all-to-all communication wherever feasible, and dynamically rebalancing workloads on measured compute time. For optimization hyperparameters (Adam/SGD), scaling the learning rate by $\sqrt{B}$ and adjusting momentum terms according to batch size accelerates convergence in batched multi-view contexts [2406.18533].

Key limitations stem from bottlenecks in source image masking (e.g., high-resolution SAM grids) and the raw memory demands of very large Gaussian sets. Systems mitigate memory pressure by chunked aggregation and external vector-database storage [2505.08124]. Communication overhead may become dominant at extreme scale or across weak interconnects, but intra-node unified memory and NVLink keep local sync costs minimal [2606.11390, 2509.05216].

Multi-GPU Gaussian splatting, by exposing operator-level distribution and transparent multi-device address spaces, has enabled high-fidelity, large-scale reconstructions and semantic scene representations that were previously infeasible under single-GPU constraints. Its applicability to both graphics and scientific/HPC workloads continues to expand as corresponding software (e.g., torch-dgx, Grendel, SLAG) matures [2606.11390, 2406.18533, 2505.08124, 2509.05216, 2509.12138].

Source: https://www.emergentmind.com/topics/multi-gpu-gaussian-splatting