--- title: WebGPU Gaussian Splatting url: https://www.emergentmind.com/topics/webgpu-powered-gaussian-splatting-2ac6e70b-94dd-4f48-a031-cc170cd811ee type: topic --- # WebGPU Gaussian Splatting WebGPU-powered Gaussian Splatting refers to a class of fully GPU-resident rendering architectures for 3D Gaussian Splatting (3DGS) that leverage the WebGPU API to deliver interactive neural scene synthesis—typically in a browser—with high throughput, cross-device determinism, and the ability to efficiently handle dynamic, generative, and avatar-driven content. These systems unify neural inference (often via ONNX Runtime in WebAssembly) and high-parallelism rendering in the browser, enabling end-to-end neural graphics pipelines with real-time frame rates and significantly reduced deployment friction compared to legacy WebGL-based approaches [2512.08478, 2602.03207]. ## 1. Mathematical Foundations of Gaussian Splatting Gaussian Splatting is based on the volumetric representation of a scene by a collection of 3D Gaussians, each parameterized by its center $X \in \mathbb{R}^3$, covariance $\Sigma \in \mathbb{R}^{3\times3}$, peak opacity $\sigma$, and color coefficients (often learned SH basis $k_i$). The continuous (unnormalized) density is $$ G(\mathbf{x}; X, \Sigma) = e^{-\frac{1}{2} (\mathbf{x}-X)^T \Sigma^{-1} (\mathbf{x}-X)}. $$ A 3D Gaussian projects to a 2D Gaussian in screen space under the camera model. The per-pixel alpha contribution is $$ \alpha = \sigma \cdot \exp\left(-\frac{1}{2} \mathbf{x}^\prime{}^T \Sigma^{\prime-1} \mathbf{x}^\prime\right), $$ where $\mathbf{x}^\prime$ is the 2D offset and $\Sigma^\prime$ is the projected 2D covariance. Color per splat is view-dependent, typically evaluated via SH as $c_i = \mathrm{SH}(k_i, d)$. Final compositing follows sorted alpha blending: $$ C = \sum_{i=1}^N c_i\,\alpha_i \prod_{j=1}^{i-1} (1-\alpha_j). $$ This formulation underpins both Visionary [2512.08478] and WebSplatter [2602.03207]. ## 2. End-to-End Pipeline Architecture WebGPU-powered 3DGS platforms unify all major pipeline stages—asset loading, neural inference, depth sorting, geometry culling, rasterization, and postprocessing—entirely in the browser with explicit GPU-side memory management: 1. **Asset loading**: Supports mesh assets (e.g., glTF, PLY) and ONNX "Gaussian Generator" models for variants such as classic 3DGS, MLP-based 3DGS, 4DGS, neural avatars, or style enhancement networks [2512.08478]. 2. **Per-frame neural pre-decoding**: Executes ONNX models in WebGPU (via ONNX Runtime WASM provider), computing per-frame Gaussian attributes given camera pose, frame index, and control signals [2512.08478]. 3. **GPU-side preprocessing**: Compute shaders project, cull (via AABB and opacity thresholds), and pre-transform Gaussians into screen-space ellipses and packed depth keys [2602.03207]. 4. **Fully GPU-resident sorting**: Per-frame, back-to-front sorting via radix sort or deterministic hierarchical scans (eliminating the need for global atomics, as in WebSplatter) [2602.03207]. 5. **Rasterization and Compositing**: Instanced splat rasterization with alpha compositing in correct depth order, optionally preceded by a mesh depth pre-pass [2512.08478, 2602.03207]. 6. **Post-processing (optional)**: Feedforward ONNX models for denoising or style transfer can be invoked and applied in a fully GPU-side manner [2512.08478]. ## 3. WebGPU Renderer Design, Sorting, and Memory ### Primitive Sorting Sorting millions of splats every frame by view depth is critical for correct compositing and numerical stability. Visionary implements radix sort in a single WebGPU compute pass, achieving $O(N \log N)$ work; empirical timing is $~0.6$ ms for $6$M splats on an RTX 4090 [2512.08478]. WebSplatter introduces a wait-free hierarchical radix sort adapted to the constraints of WebGPU (no global atomics, no fixed scheduling order). The algorithm decomposes sorting into: - Local histogram computation per workgroup (shared memory). - Hierarchical prefix scan ("HierBlelloch") for cross-workgroup offsets, implemented as a sequence of dispatches with only intra-workgroup barriers. - Global scatter, writing sorted keys/indices without spin-waiting. This approach achieves $O(N)$ work per pass (4 passes), is deterministic, deadlock-free, and supports a wide range of hardware [2602.03207]. ### Memory Management - Gaussian parameters are bit-packed (e.g., FP16 in $u32$ pairs) to halve bandwidth and reduce per-Gaussian memory from $\sim$32B to $\sim$16B [2512.08478]. - Single monolithic buffers reduce allocation overhead and CPU-GPU synchronization. - Visionary uses atomic counters and indirect dispatch buffers to dynamically size output arrays per frame [2512.08478]. - WebSplatter demonstrates GPU memory footprint: $1.20$GB (RTX 3070, "garden" scene, $5.83$M splats), a reduction of $36\%$ to $57\%$ vs. prior viewers [2602.03207]. ## 4. Geometry Culling and Rasterization Optimizations Both frameworks implement geometry-level pruning to minimize overdraw and improve memory efficiency: - **Screen-space AABB culling:** Each Gaussian's screen-projected ellipse is rapidly bounded and checked for viewport intersection. - **Opacity-based culling:** Splats below an opacity threshold ($\sigma < 1/255$) are excluded. Quad sizing is dynamically adjusted by solving $1/255 = \sigma \exp(-r^2)$ for $r$, so only fragments where $\alpha \geq 1/255$ are rasterized [2602.03207]. - Disabling tight quad bounds causes significant performance penalty (e.g., $+18.6\%$ render time on MacBook Air M4) [2602.03207]. The net effect is substantial reduction in fragment shading work, peak GPU memory, and overdraw. ## 5. Neural Inference Integration and Dynamic Content WebGPU-powered platforms support dynamic or generative Gaussian content via ONNX Runtime integration: - **Gaussian Generator Contract:** Fixed ONNX I/O schema with camera/projection matrices, frame index, control signals (inputs); positions $[N\times3]$, covariances $[N\times6]$, colors, and opacities (outputs); plus metadata such as SH degree. This enables per-frame, stateless neural generation and updating of Gaussians [2512.08478]. - **Lifecycle:** Models are exported (e.g., PyTorch to ONNX, possibly with chunking of large ops), loaded and warm-started in the browser, then invoked per frame to generate splat parameters for rendering [2512.08478]. - **Extensibility:** Plug-and-play support for different 3DGS variants (classic, MLP-based, 4DGS, neural avatars, single-shot pixel splat networks). Optional post-processing (diffusion, style transfer) via additional ONNX passes allows direct composable neural graphics [2512.08478]. - **APIs:** TypeScript interfaces (e.g., three.js plugin) permit streamlined integration for web applications; sample code provided for end-to-end inference and rendering [2512.08478]. ## 6. Performance Evaluation and Robustness ### Empirical Benchmarks Benchmarking on commodity GPUs and a variety of browsers yields the following highlights: | #Gaussians | SparkJS sort (ms) | SparkJS total (ms) | Visionary sort (ms) | Visionary total (ms) | |------------|------------------|--------------------|---------------------|----------------------| | 6.062M | 172.87 | 176.90 | 0.58 | 2.09 | | 3.031M | 143.50 | 145.75 | 0.32 | 1.09 | | 0.758M | 33.31 | 33.82 | 0.20 | 0.40 | Visionary achieves up to $135\times$ speedup over SparkJS (WebGL) with comparable visual fidelity (PSNR $27.87$ vs. $27.31$ for MipNeRF360) [2512.08478]. WebSplatter achieves $1.2\times$ to $4.5\times$ speedup over state-of-the-art WebGPU viewers across diverse hardware, with consistent memory savings [2602.03207]. | Device | WebSplatter (ms) | Best prior (ms) | |-------------------|------------------|-----------------| | RTX 3070 (Chrome) | 9.50 | 14.4 | | MacBook Air M4 | 68.6 | 78.5 | | MacBook Pro M1 | 112.0 | 225.2 | | Intel NUC iGPU | 151.2 | 341.7 | | Redmi K70 Pro | 33.6 | 39.5 | ### Sorting and Robustness - Per-frame global sort eliminates artifacts and ensures correct alpha compositing under rapid camera motion, outperforming "lazy sorting" and local partitioning approaches in legacy frameworks [2512.08478]. - Systems sustain interactive performance for up to approximately $6.2$M splats; devices with less than $1$GB VRAM may present constraints [2602.03207]. - ONNX inference latency is typically $7$–$9$ ms for standard models (Scaffold-GS, avatars, 4DGS) at multi-million element scale [2512.08478]. ## 7. Applications, Extensibility, and Deployment WebGPU-powered Gaussian Splatting platforms prove extensible for a variety of upstream and downstream tasks: - **3DGS family variants:** Native support for classic 3DGS, MLP-based methods, dynamic 4DGS, neural avatars (e.g., Gauhuman, R³-Avatar), and efficient single-shot models (PixelSplat, MVSplat). - **Generative post-processing:** Integration of diffusion-based denoisers and style transformers for neural scene editing [2512.08478]. - **Web-native deployment:** "Click-to-run" execution in browsers with static HTML+JS+ONNX hosting—no native dependencies; integration in frameworks such as three.js via concise TypeScript APIs [2512.08478]. - **Future directions:** Out-of-core streaming for larger scenes, mesh/splat hybridization for extremely large worlds, and memory reduction via quantization/compression [2602.03207]. A unified inference and rasterization pipeline in the browser significantly lowers barriers for reproduction, comparison, and deployment of neural graphics research and applications across reconstructive and generative paradigms. --- **Key References**: - "Visionary: The World Model Carrier Built on WebGPU-Powered Gaussian Splatting Platform" [2512.08478] - "WebSplatter: Enabling Cross-Device Efficient Gaussian Splatting in Web Browsers via WebGPU" [2602.03207] Source: https://www.emergentmind.com/topics/webgpu-powered-gaussian-splatting-2ac6e70b-94dd-4f48-a031-cc170cd811ee