Beyond Heuristics: Learnable Density Control for 3D Gaussian Splatting

Abstract: While 3D Gaussian Splatting (3DGS) has demonstrated impressive real-time rendering performance, its efficacy remains constrained by a reliance on heuristic density control. Despite numerous refinements to these handcrafted rules, such methods inherently lack the flexibility to adapt to diverse scenes with complex geometries. In this paper, we propose a paradigm shift for density control from rigid heuristics to fully learnable policies. Specifically, we introduce \textbf{LeGS}, a framework that reformulates density control as a parameterized policy network optimized via Reinforcement Learning (RL). Central to our approach is the tailored effective reward function grounded in sensitivity analysis, which precisely quantifies the marginal contribution of individual Gaussians to reconstruction quality. To maintain computational tractability, we derive a closed-form solution that reduces the complexity of reward calculation from $O(N^2)$ to $O(N)$. Extensive experiments on the Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that \textbf{LeGS} significantly outperforms state-of-the-art methods, striking a superior balance between reconstruction quality and efficiency. The code will be released at https://github.com/AaronNZH/LeGS

• The paper introduces a novel RL-driven framework, LeGS, that replaces heuristic density control with a learned policy for 3D Gaussian Splatting.
• It leverages a closed-form sensitivity score and efficient reward mechanism to optimize Gaussian refinements while reducing computational overhead.
• Experimental results show that LeGS delivers state-of-the-art reconstruction quality using fewer Gaussians compared to traditional methods.

Learnable Density Control for 3D Gaussian Splatting (LeGS): A Paradigm Shift from Heuristics

Problem Statement and Motivation

Recent advances in 3D Gaussian Splatting (3DGS) have established it as the de facto technique for real-time, high-fidelity scene reconstruction and novel view synthesis. Central to 3DGS efficacy is the adaptive control of Gaussian density: managing when and where to densify (clone/split) or prune scene primitives. Historically, this process has been dictated by rigid, manually-crafted heuristics, often involving gradient or opacity thresholding mechanisms and scene-agnostic hyperparameters. However, these hand-tuned approaches lack robustness, generalizability, and scene-adaptive expressiveness, leading to either over-redundancy or under-reconstruction in geometrically or texturally complex regions. Recent heuristic refinements (e.g., Pixel-GS, FastGS, Perceptual-GS) introduce metrics and priors for improved allocation but fundamentally retain threshold-based, non-adaptive designs, resulting in brittle performance under variable scene statistics.

Figure 1: Comparison between LeGS and FastGS. The heuristic paradigm achieves suboptimal results, whereas the learning-based LeGS accurately captures local details.

The LeGS Framework: RL-Driven Density Control

Formalization

LeGS reconceptualizes density control as a learned policy, parameterized by a neural network and optimized with Reinforcement Learning (RL). Gaussian refinement is posed as a Markov decision process, where each 3D Gaussian is a state and the permitted actions are $$\mathcal{A} = \{ \text{maintain}, \text{clone}, \text{split}, \text{prune} \}$$. The policy network $\mathcal{F}_\theta$ ingests Gaussian state features, including geometric, photometric, and novel sensitivity-based scores, and predicts per-Gaussian actions at each control iteration.

A pipeline overview is depicted below.

Figure 2: The LeGS pipeline renders multiple views, computes gradients and sensitivity scores, and invokes a policy network to decide discrete actions per Gaussian.

Reward Construction and Efficient Sensitivity Analysis

Critical to robust RL optimization is precise credit assignment. LeGS introduces a sensitivity-based reward system inspired by CNN occlusion analysis. The sensitivity score for each Gaussian quantifies its causal, marginal contribution to reconstruction quality, computed as the difference in per-pixel error with and without that Gaussian’s participation, aggregated across its visible footprint.

To avoid the prohibitive $O(N^2)$ cost of brute-force rendering for each ablation, a closed-form solution is derived by algebraically inverting the cumulative $\alpha$-blending equation, analytically producing all ablated renderings in $O(N)$ time by caching forward-pass color and transmittance accumulations. This significantly accelerates RL reward computation, yielding negligible overhead.

Figure 3: The fast closed-form sensitivity score calculation achieves linear complexity and negligible runtime overhead compared to the naive per-Gaussian re-rendering approach.

Policy Optimization Strategy

The RL policy is optimized via PPO, with two notable innovations: (1) Constructing a reward from sensitivity-based local improvement for each action and its receptive offspring; (2) Employing a maintain-action-based value baseline for GAE to reduce value estimation instability in high-cardinality, operation-driven reward spaces. This exploits the maintain action as a non-parametric, reliable baseline, avoiding overfitting and excessive redundancy.

Experimental Results

Quantitative and Qualitative Results

Benchmarks on Mip-NeRF 360, Tanks & Temples, and Deep Blending datasets demonstrate that LeGS outperforms all baseline methods under all quality and efficiency metrics (PSNR, SSIM, LPIPS, Gaussian count). Notably, LeGS sustains SOTA reconstruction while using fewer Gaussians compared to most baselines, and always produces more compact or higher-quality representations depending on the dataset and resource regime.

Ablation studies verify the utility of both learnable densification and pruning modules, the sensitivity-based reward, and the maintain-action-based value estimator. The RL policy selects locally optimal actions significantly more often than heuristic methods such as FastGS, as verified by controlled action-to-outcome studies.

Qualitative outputs show that LeGS prevents the characteristic under-/over-densification artifacts of heuristic allocation, adapting to complex textures and varying geometries.

Figure 4: Qualitative comparison on complex scenes shows the improved reconstruction fidelity and Gaussian placement of LeGS versus multiple SOTA methods.

Figure 5: Across a range of Gaussian quantities, LeGS retains high SSIM/PSNR and low LPIPS, evidencing robust scalability and adaptability, in contrast to the fragile performance of heuristic baselines.

Figure 6: Additional qualitative examples on Play Room, Bonsai, Tree Hill, and Dr Johnson confirm LeGS's robustness across diverse, challenging real-world scenes.

Practical and Theoretical Implications

LeGS provides a reproducible, efficient framework for substituting brittle heuristic rules with data-driven, per-scene-adaptive learned policies in 3DGS. The implications are threefold:

• Quality-Efficiency Pareto: By adaptively allocating Gaussians only to structurally or perceptually critical regions via learned policies, LeGS achieves SOTA rendering under a fixed computational budget, or provides highly compact representations for a target quality level.
• Generalization and Robustness: The flexibility and theoretical compositionality of learned policy networks facilitate extension to new scene domains (e.g., highly non-Lambertian or low-data regimes) and integration with additional domain-specific priors (e.g., perceptual, geometric, or physical).
• Foundation for Data-Driven Scene Optimization: Framing geometric primitive control as an RL problem with well-motivated rewards opens a research avenue for scalable, optimization-theoretic approaches to other graphics and vision pipelines reliant on discrete, coupled primitive selection.

Speculation: Future Directions

Potential extensions include coupling LeGS with automatic, cross-scene meta-optimization; integrating additional priors (perceptual, saliency, geometry-aware); and scaling the approach to semi-supervised or unsupervised learning settings. Furthermore, the general RL-based paradigm can be adopted for control in hybrid and hierarchical primitive systems, cross-modal rendering, and adaptive resource allocation tasks in broader neural rendering contexts.

Conclusion

LeGS replaces heuristic, hand-tuned density control in 3D Gaussian Splatting with a compact, efficient, and robust learning-based paradigm. By leveraging RL over a sensitivity-based, analytically efficient reward, the method establishes new standards for scene-adaptive primitive allocation, SOTA reconstruction fidelity, and computational resource economy in 3DGS, providing a foundation for principled, scalable control in neural graphics pipelines (2605.00408).