ONNX-Gaussian Generator for Real-Time 3DGS

• ONNX-based Gaussian Generator is a neural module in ONNX format that produces 3D Gaussian primitives for rendering applications.
• It integrates dynamic inference with GPU-accelerated pipelines, enabling real-time 3D Gaussian Splatting and eliminating CPU bottlenecks.
• The system leverages strict I/O contracts and efficient pre-processing to achieve sub-millisecond frame times in advanced WebGPU rendering.

An ONNX-based Gaussian Generator is a standardized neural network module, exported in Open Neural Network Exchange (ONNX) format, for producing frame-specific Gaussian primitives within a neural rendering pipeline. Deployed primarily in the context of real-time 3D Gaussian Splatting (3DGS), these generators integrate dynamic inference with GPU-accelerated rendering, eliminating legacy pipeline constraints and CPU bottlenecks. The ONNX-based Gaussian Generator is a core element in platforms such as Visionary, which unifies the inference and rendering of generative and reconstructive world models directly in the browser, leveraging WebGPU for high-throughput interactive synthesis and visualization (Gong et al., 9 Dec 2025).

1. Interface Schema and Data Contract

The ONNX-based Gaussian Generator adheres to a strict I/O schema per animation frame, ensuring compatibility and efficient interoperation with WebGPU-based rendering engines.

Inputs:

• camera_extrinsic: float32[4,4], representing the world-to-camera transformation matrix.
• camera_intrinsic: float324,4, encoding the camera’s projection parameters.
• Optional control variables: sequence index, timestamp, or latent vector, each as float32 scalar or 1D tensor.

Outputs:

• N (int32 scalar): specifies the number of Gaussian primitives produced per frame.
• gaussians: float16[N,13] packed tensor, each row parameterizing a 3D Gaussian with:
  • Mean: $\mu_x, \mu_y, \mu_z$ (3)
  • Covariance (upper-triangular): $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$ (6)
  • RGB color: $color_r, color_g, color_b$ (3)
  • Opacity: $\alpha$ (1)
• meta (optional): Dictionary with fields such as "packed_dtype" ("FP16" or "FP32") and upper bound for N.

The contract enforces that exactly N Gaussian tuples are output per frame. The use of a packed float16 layout minimizes upload bandwidth requirements; unpacking occurs on the GPU at runtime. Covariance is expressed in a compressed 6-value format corresponding to the independent elements of the symmetric $3 \times 3$ matrix.

2. Mathematical Foundations of Gaussian Splatting

Each rendered primitive $G_i$ is modeled as an anisotropic Gaussian in $\mathbb{R}^3$, parameterized by weight $w_i$, mean vector $\mu_i \in \mathbb{R}^3$, and covariance matrix $\Sigma_i \in \mathbb{R}^{3\times 3}$. Under camera projection $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$0, the 3D Gaussian maps to a 2D ellipse in image space:

• Projected mean: $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$1
• Projected covariance: $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$2, where $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$3

The influence on pixel $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$4 is given by:

$$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$5

with view-independent color $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$6. Final color compositing proceeds front-to-back:

$$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$7

The generator network maps its outputs to these parameters: $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$8 (output via sigmoid); $$\Sigma_{xx}, \Sigma_{xy}, \Sigma_{xz}, \Sigma_{yy}, \Sigma_{yz}, \Sigma_{zz}$$9 (3-vector head); $color_r, color_g, color_b$0 (6-vector head, upper-triangular entries, positivity enforced via softplus); $color_r, color_g, color_b$1 (3-vector head, postprocessed if using spherical harmonics coefficients).

3. ONNX Inference Pipeline and Runtime Integration

Model deployment and inference occur directly within WebGPU-enabled browser contexts. The pipeline comprises several phases:

• Model Loading: Performed once at application initialization using onnxruntime-web. The ONNX model is loaded, often with parameters quantized to FP16 and constants baked in at export (opset ≥ 14, constant folding enabled).
• Warm-Up: A dummy inference primes the WebGPU execution graph and cache, optimizing subsequent per-frame executions.
• Per-Frame Schedule:
  • Camera and optional control data are packed into WebGPU buffers.
  • Synchronous/asynchronous forward pass via ONNX runtime, with outputs streamed directly as GPU buffers (avoiding CPU roundtrips).
  • Gaussian buffer is bound as storage for GPU compute pre-processing: transform, cull, and radix-sort the Gaussian splats.
  • Instanced draw call issues one triangle-strip per splat; fragment shader evaluates $color_r, color_g, color_b$2 and performs compositing.

Reusing persistent WebGPU bind groups and ONNX session bindings enables predictable low-latency execution and amortized dispatch costs within the frame budget.

The following table summarizes the operational pipeline:

Step	Action	Technology
Model load	ONNX import, session instantiate	onnxruntime-web
Warm-up	Dummy inference to cache WebGPU execution graph	WebGPU
Per-frame inference	Input assembly, neural decoding, GPU buffer bind	WebGPU, ONNX
Preprocessing	Splat preprocess, cull, radix-sort (O(N))	WebGPU compute
Rendering	Instanced draw, fragment compositing	WebGPU fragment

4. Network Architectures and Training Paradigms

Several network designs are supported:

• MLP-based 3DGS: Follows Scaffold-GS style. Input is per-anchor feature ($color_r, color_g, color_b$3, dimension 256–512) concatenated with view direction ($color_r, color_g, color_b$4, dimension 3). 4–6 fully connected layers with hidden units 256–512 (GELU or ReLU activations). Four output heads: $color_r, color_g, color_b$5 (3 floats), $color_r, color_g, color_b$6 (6 floats via softplus), $color_r, color_g, color_b$7 (3 floats, either direct RGB or SH coefficients), and $color_r, color_g, color_b$8 (1 float via sigmoid). Trained via L2 or L1 pixel loss plus regularization on $color_r, color_g, color_b$9 scale. ONNX export uses dynamic axes for batch/view dimensions.
• 4D Gaussian Splatting: Temporal input (timestamp $\alpha$0) with canonical parameters and multi-scale feature planes. Feature lookup (bilinear sampling) is concatenated and fed to a small MLP (2–3 layers, 128 units). Outputs are delta mean (3), delta quaternion for rotation (4), and delta scale (3). The covariance updates as $\alpha$1.
• Animatable Avatars (e.g., LHM, R3-Avatar): Input is SMPL-X pose $\alpha$2 (72D), shape $\alpha$3 (10D), and optional frame index. Internally, canonical $\alpha$4, $\alpha$5, and per-Gaussian skinning weights $\alpha$6 are used. Forward kinematics and LBS are included within the ONNX graph. Output is the deformed $\alpha$7 and $\alpha$8 in the current observation frame.

Training commonly employs mixed-precision (fp16) weights, opset ≥ 14, and constant folding. Large Concats/Splits are refactored in export to conform with WebGPU operational constraints.

5. Browser-Based Integration and Application Example

Integration is facilitated by a concise TypeScript API compatible with three.js and the “visionary-webgpu” library. The typical workflow involves initializing a WebGPU renderer and Gaussian renderer, loading the ONNX model, warm-up, and a main loop that, per frame, updates camera matrices, performs ONNX inference, streams results to GPU, and dispatches render calls.

$3 \times 3$7

This workflow maintains all data and computation on the GPU, minimizing latency and eliminating CPU-GPU roundtrips.

6. Performance Characteristics and Throughput

The ONNX+WebGPU approach yields high throughput and low latency:

• Per-frame preprocessing—frustum culling, $\alpha$9clip, $3 \times 3$0ellipse axes—runs in a single WebGPU compute pass.
• Depth keys and indices are updated via GPU atomics, and a GPU radix sort organizes splats in $3 \times 3$1 runtime, achieving sub-millisecond times for millions of points.
• Instanced triangle-strip draws and fragment-based Gaussian summation enable efficient compositing and blending.
• Captured and replayed ONNX execution graphs result in stable dispatch overhead, with reported per-frame decoding times:
  • Scaffold-GS: 2.5M splats $3 \times 3$2 9 ms
  • 4DGS: 0.06M splats $3 \times 3$3 8 ms
  • Avatars: 0.2M splats $3 \times 3$4 30 ms
• Aggregate end-to-end frame times for static 6M Gaussians are $3 \times 3$52 ms, representing approximately 100$3 \times 3$6 speedup over CPU-based WebGL sorting.
• The entire pipeline remains within a single-frame time budget, leveraging unified memory architectures for device-local execution and rapid updates (Gong et al., 9 Dec 2025).

A plausible implication is that the ONNX-based Gaussian Generator contract enables real-time, browser-native world model rendering and generative processing suitable for both reconstruction and synthetic content, with substantial practical advantages for reproducibility and deployment.