PocketGS: Mobile 3D Gaussian Splatting

• PocketGS is a mobile-optimized framework for 3D Gaussian Splatting that redefines the training loop using hardware-aware, modular operator design.
• It employs co-designed operators for geometry priors, anisotropic initialization, and differentiable alpha compositing to achieve minute-scale training and sub-3GB memory usage.
• Empirical results demonstrate significant speedups and enhanced visual fidelity over conventional methods on both mobile and desktop systems.

PocketGS is a compact, high-performance framework for 3D Gaussian Splatting (3DGS) designed to enable efficient and high-fidelity 3D scene modeling entirely on resource-constrained devices such as mobile phones. By fundamentally reworking the standard splatting pipeline and incorporating hardware-aware optimizations, PocketGS targets on-device 3DGS training—achieving high perceptual fidelity, sub-3 GB peak memory usage, and minute-scale training times, while remaining compatible with existing 3DGS workflows (Liao, 3 Mar 2025, Guo et al., 24 Jan 2026).

1. Motivations and Constraints

Conventional 3DGS implementations assume unconstrained workstation environments, yielding high quality but prohibitive memory and compute costs for mobile deployment. PocketGS addresses three primary device-imposed constraints:

• Total training time: ≤500 iterations and ≤5 minutes per scene.
• Peak memory: <3 GB, accounting for all transient and persistent buffers.
• Visual fidelity: Preservation of sharp textures and thin structures under noisy, low-quality mobile captures.

Three fundamental system contradictions are thereby exposed:

• Input-Recovery Contradiction: Noisy mobile input (poses, sparse points) requires aggressive point set densification, which naively leads to the explosion of point count $N$, undermining efficiency.
• Initialization-Convergence Contradiction: Isotropic Gaussian initialization converges too slowly under tight iteration budgets.
• Hardware-Differentiability Contradiction: Tile-based GPUs obscure blending intermediates, impeding backpropagation unless prohibitive host-device synchronization is performed (Guo et al., 24 Jan 2026).

2. Modular Operator Design

PocketGS implements a staged operator decomposition, enabling both hardware efficiency and analytical reasoning about each phase. Operators are as follows (Liao, 3 Mar 2025, Guo et al., 24 Jan 2026):

Operator	Purpose	Key Methodology
ClusterCuller	Blockwise culling via Morton codes, frustum AABBs	Batching, spatial locality
ClusterCompactor	Reorders clusters for cache coherence, cut warp divergence	L2-reordering, Morton code sort
Project2D	Camera-space projection of means and covariances	$$\tilde{\mu}_k = P\mu_k,\ \tilde{\Sigma}_k = J\Sigma_kJ^\top$$
TileBinner	Screen-space assignment to tiles using tight AABBs	2D orthogonal axis projection, 8×8 tile grid
TileRasterizer	Parallel rasterization and alpha blending in tiles	CUDA-level fusion, alpha compositing

This modular design underpins both the desktop-optimized and on-device workflows, supporting script-based (Python+autograd) and high-throughput (CUDA/Metal shader) APIs.

3. PocketGS On-Device Methodology

PocketGS re-engineers the end-to-end 3DGS training loop for mobile contexts, introducing three co-designed operators (labeled G, I, T):

3.1 Operator G: Geometry-Faithful Point-Cloud Priors

Operator G builds compact, low-noise geometric priors directly on the GPU. This encompasses:

• Information-Gated Frame Subsampling: Selection of keyframes maximizes view displacement (≥5 cm) and image sharpness, using a windowed sharpness heuristic to prevent redundant or blurry inputs.
• GPU-Native Global Bundle Adjustment: A Levenberg–Marquardt optimizer partitions normal equations with the Schur complement; block-diagonalization of $H_{pp}$ enables $O(J)$ parallel inversion.
• Single-Reference MVS: A cost volume is constructed using one optimal reference frame, selected using appearance-based scoring, followed by semi-global matching and census transform for compact dense point cloud recovery.

3.2 Operator I: Local Surface Statistics Injection

Operator I accelerates convergence:

• Anisotropic Initialization: For each dense point, local covariance is computed over $K=16$ nearest neighbors. Eigen-decomposition yields local surface normals and a scale-normalized disc covariance, improving early geometry alignment.
• Parameter Seeding: Gaussians are initialized as $$(\mu_i = p_i, \Sigma_i, c_i, \alpha_i = \sigma(-2.2) \approx 0.1)$$, with scale and opacity optimized in log-space.

3.3 Operator T: Alpha Compositing Unroll & Gradient Scattering

Operator T enables memory-bounded differentiable rendering:

• Unrolled Blending: Front-to-back alpha compositing is made explicit in the shader, caching only $(C_\text{in}, \alpha)$ per pixel and depth-sorted fragment. All blending intermediates reside on-device.
• Index-Mapped Backpropagation: Gradients are scattered to canonical parameter memory using a depth-sort index map, maintaining optimizer state alignment within GPU buffers and removing costly host-device synchronizations. Adam updates are performed fully on-GPU.

The result is an $O(N+M)$ memory complexity, contrasting with the $O(N\times M)$ cost of naive methodologies ($N =$ Gaussians, $M =$ pixels).

4. Training Pipeline, Algorithms, and Optimization

4.1 End-to-End Pipeline

The training workflow is as follows (Guo et al., 24 Jan 2026):

1. Capture: ARKit-driven frame acquisition, information gate selection of ∼50 keyframes.
2. Geometry Prior Construction (G): GPU-based feature matching, Schur bundle adjustment, and single-reference MVS yield a dense point cloud.
3. Initialization (I): KNN covariance estimation, anisotropic Gaussian seeding.
4. Optimization (T): For each of the ≤500 iterations:
   • Metal shader or fused CUDA kernel executes splatting and forward compositing.
   • Photometric loss $\mathcal{L}$ is evaluated.
   • Backward compositor applies analytic gradients; index-mapped updates are scattered to parameter buffers.
   • Adam optimizer runs on-GPU.
5. Export: Parameters are saved for interactive or further downstream use.

4.2 Gaussian Splatting Equation

For pixel $(x, y)$, the forward compositing is: $$c(x,y) = \sum_{k=1}^K G(x, y | \tilde{\mu}_k, \tilde{\Sigma}_k) \, \alpha_k \, c_k \prod_{j<k} [1 - \alpha_j G(x, y | \tilde{\mu}_j, \tilde{\Sigma}_j)]$$ where

$$G(x, y | \tilde{\mu}, \tilde{\Sigma}) = \exp\left(-\frac{1}{2}(\mathbf{p} - \tilde{\mu})^\top \tilde{\Sigma}^{-1}(\mathbf{p} - \tilde{\mu})\right)$$

(Liao, 3 Mar 2025).

4.3 Key Optimizations

• Kernel fusion and warp-level reduction for backward raster remove over 50% of memory-IO stalls (MIO) and DRAM traffic.
• Memory reordering of parameters increases L2 cache hit rate by 15%.
• Sparse gradient updates and fused Adam delivered ∼10% optimizer acceleration without accuracy penalty.
• Asynchronous transfers of parameter updates overlapped with preceding projections, hiding 5% of latency (Liao, 3 Mar 2025).

5. Quantitative Performance and Evaluation

Empirical testing on commodity mobile (iPhone 15, A16 GPU), workstation GPUs (A100, RTX3090), and multiple datasets demonstrates:

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓	Time ↓ (s)	Gaussian Count
LLFF (avg)	3DGS-SFM-WK	21.01	0.641	0.405	108.0	18k
	3DGS-MVS-WK	19.53	0.637	0.387	313.1	40k
	PocketGS	23.54	0.791	0.222	105.4	33k
NeRF-Synthetic	3DGS-SFM-WK	21.75	0.800	0.243	83.7	12k
	3DGS-MVS-WK	24.47	0.887	0.128	532.1	50k
	PocketGS	24.32	0.858	0.144	101.4	47k
MobileScan	3DGS-SFM-WK	21.16	0.687	0.398	112.8	23k
	3DGS-MVS-WK	20.85	0.781	0.281	534.5	165k
	PocketGS	23.67	0.791	0.225	255.2	168k

On MobileScan, PocketGS achieved 2.1× faster runtime than 3DGS-MVS-WK, used less than half the Gaussian count, and remained within <3 GB peak memory (Guo et al., 24 Jan 2026). On desktop, PocketGS obtained a 3.4× speedup and ≈30% lower GPU memory consumption relative to 3DGS (Liao, 3 Mar 2025).

6. Compatibility, Limitations, and Extensions

PocketGS is engineered for seamless integration:

• Parameter and API compatibility: Preserves 3DGS parameters ($\mu, \Sigma, c, \alpha$) and maintains JSON/CMD interface parity for drop-in adoption.
• Python and CUDA interoperability: Dual API surface mirrors GSplatTrainer, enabling extension and benchmarking under established pipelines.
• Bit-exact output: Ensures PSNR, SSIM, and LPIPS outputs match legacy 3DGS logging and evaluation.

Identified limitations include degraded performance under extreme low-texture or dynamic scenes (single-reference MVS). Potential extensions involve multi-reference or learned MVS, adaptive iteration control, and radiance model hybridization (e.g., spherical Gaussians) (Guo et al., 24 Jan 2026).

7. Broader Implications and Future Directions

PocketGS demonstrates that full 3DGS modeling, including capture, initialization, and optimization, is practical on commodity smartphones. This enables rapid in-field digital twin generation and AR/VR asset creation without cloud upload dependencies. Prospective research includes algorithmic adaptation for non-rigid or non-static scenes, improved robustness to mobile capture artifacts, and integration with emerging radiance and geometry representations (Guo et al., 24 Jan 2026).

---

References:

• "LiteGS: A High-Performance Modular Framework for Gaussian Splatting Training" (Liao, 3 Mar 2025)
• "PocketGS: On-Device Training of 3D Gaussian Splatting for High Perceptual Modeling" (Guo et al., 24 Jan 2026)