BEA-GS: BEyond RAdiance Supervision in 3DGS for Precise Object Extraction

Abstract: Most Gaussian Splatting techniques that provide a 3D semantic representation of the scene do not optimize the underlying 3D geometry, making object-level editing or asset extraction challenging. Recent methods, such as COBGS, Trace3D, ObjectGS, acknowledge this limitation and propose approaches that modify the scene's geometry to represent the underlying semantics. We advance this concept further by proposing a novel solution that provides near perfect boundaries in object extraction. We do so by introducing two new losses in the optimization that take care of: 1) a loss that modifies the geometry of visible Gaussians to respect semantic boundaries, and 2) a loss that adjusts the geometry of non-visible Gaussians that appear once the object is extracted. Our first loss propagates gradients directly through the rasterization, allowing for seamless integration within the optimization of the Gaussian parameters. The second loss also propagates gradients to Gaussian parameters but does so without passing through the rasterization, enabling modification of the scene's geometry even when little transmittance reaches a Gaussian (partial or non-visible). Exhaustive comparisons with 12 state of the art methods across 4 datasets, using six metrics, demonstrate that our approach produces overall the best boundary segmentation to date.

• The paper introduces BEA-GS, which combines multiview mask reprojection, a rasterized 2D boundary loss, and a direct 3D occupancy loss to correct both visible and occluded Gaussians.
• BEA-GS ranks first in 21 of 24 evaluations, reaching up to 93.2 IoU and 87.3 BIoU while improving extracted boundaries and changing test-set PSNR by no more than 0.2 dB.
• The method adds about 3,000 refinement iterations and 4–15 minutes per scene, but remains limited by initial reconstruction quality, unreliable masks, transparent surfaces, and missing unseen geometry.

Motivation and problem statement

Object-level editing and asset extraction in 3D Gaussian Splatting (3DGS) scenes require a semantic decomposition in which each Gaussian belongs to exactly one object. A recurring failure mode, documented by prior work such as OpenGaussian and ClickGaussian, is that Gaussians trained with purely photometric objectives contribute to multiple objects. Recent geometry-aware methods — COB-GS, Trace3D, and ObjectGS — address this by constraining Gaussian geometry during optimization, but they share a structural limitation: all of their supervision propagates through the alpha-blending rasterizer, so only Gaussians that receive non-negligible transmittance are updated. Partially or fully occluded Gaussians remain "stuck" beneath object surfaces and resurface as artifacts when the object is extracted and reused elsewhere. This is a consequence of the tight coupling between appearance and geometry in 3DGS (unlike NeRF): under photometric loss alone, a Gaussian may stretch or drift into unseen regions rather than adjust its color parameters.

BEA-GS identifies radiance-gated gradient propagation as the root cause and introduces supervision that operates beyond it (2605.09662).

Method

The framework builds on 2D Gaussian Splatting (2DGS) as the reconstruction backbone and consists of three components.

Multiview reprojection. Starting from SAM2 two-stage masks, per-pixel labels are lifted to 3D via rendered depth, reprojected into all views, and consolidated by majority voting ($M' = \mathrm{argmax}(M_\phi)$). This corrects boundary inconsistencies across views and recovers regions missed by SAM2 in individual frames, provided the object is correctly segmented in at least half of its visible frames.

2D boundary loss. Each Gaussian carries a single non-trainable class parameter $\phi_i$, initialized by contribution-weighted frequency accumulation over the refined masks. The loss penalizes any Gaussian contribution to pixels whose label differs from the Gaussian's class:

$$\mathcal{L}_{bound}(u) = \sum_{i=1}^{N} H_i(u)\,\alpha_i\,\hat{\mathcal{G}}_i(x)\prod_{j=1}^{i-1}\left(1-\alpha_j\hat{\mathcal{G}}_j(x)\right)$$

with $H_i(u)=1$ when $\phi_i \neq M'(u)$. Gradients flow through the rasterizer, encouraging opacity reduction, reshaping, or displacement at boundaries. Compared to COB-GS (binary-only, one model per object) and ObjectGS (one probability channel per class), this requires only a single extra channel regardless of class count, and it activates only on out-of-boundary contributions rather than over the full mask.

3D occupancy loss. To regularize Gaussians receiving little or no transmittance, the method constructs class-specific voxel grids $V_\phi$ from backprojected depth points, with an adaptive voxel size derived from $k$-NN density estimation ($k=2000$). During training, $Z=20$ points are sampled uniformly on each Gaussian's surface; samples falling in voxels (and their 27-voxel neighborhoods) with zero occupancy are penalized proportionally to opacity and density. Crucially, these gradients propagate directly to Gaussian parameters without passing through rasterization, so geometry is corrected even where no radiance reaches the Gaussian. Voxels where classes collide are treated as empty, but neighborhood-based queries mitigate erosion artifacts at object contact interfaces.

The final objective combines the standard 2DGS loss with $\lambda_{bound}=0.5$ and $\phi_i$0; the larger occupancy weight compensates for its normalization by both Gaussian count and sample count. Optimization runs for 3K additional iterations after the standard 30K 2DGS schedule, adding roughly 4–15 minutes per scene on an NVIDIA A100.

Quantitative results

Evaluation covers four datasets (Mip-NeRF 360, LeRF, LLFF, 3DOVS), six metrics (extracted and rendered variants of Acc/IoU/BIoU), and twelve baselines, all trained on identical 2D masks. BEA-GS ranks first in 21 of 24 evaluated cases and is the only method that performs consistently well across datasets and metrics; representative extracted-metric results:

Method	Mip-360 IoU / BIoU	LeRF IoU / BIoU	LLFF IoU / BIoU
FlashSplat	87.4 / 78.6	80.4 / 72.5	91.8 / 76.5
COB-GS	86.9 / 78.0	83.0 / 75.6	91.4 / 75.1
Trace3D	87.1 / 75.4	85.9 / 80.1	86.1 / 60.1
BEA-GS	92.0 / 85.8	89.4 / 83.6	93.0 / 80.7

The gains are largest on BIoU, directly reflecting cleaner extracted boundaries. On 3DOVS, BEA-GS attains 93.2 IoU / 87.3 BIoU (extracted), though ObjectGS retains the best rendered metrics there (95.3 IoU / 90.7 BIoU). Rendering quality is essentially preserved: test-set PSNR changes by at most 0.2 dB (e.g., Mip-NeRF 360 at 29.1/29.1 before/after). In terms of computational cost, the 3K refinement iterations are independent of class count, whereas COB-GS scales linearly with objects (e.g., 29,400 iterations for the Figurines scene versus 3,000 for BEA-GS).

The ablation confirms complementarity of the two losses: the boundary loss dominates fine visible detail, while the occupancy loss is decisive on datasets with occlusion-heavy 360° captures (on LeRF, adding occupancy raises BIoU from 70.1 to 77.2 without the boundary loss). Multiview reprojection helps mainly on Mip-NeRF 360 and LeRF and offers little benefit on frontal-view datasets (LLFF, 3DOVS). Sensitivity analyses show results are robust to $\phi_i$1 and $\phi_i$2, with out-of-memory failures only at $\phi_i$3. Under degraded geometry (backbone retrained without depth/normal regularization), performance drops moderately on LeRF and 3DOVS (e.g., 3DOVS IoU from 93.2 to 89.8) but remains stable on Mip-NeRF 360 and LLFF.

Limitations

The authors state several constraints explicitly. Performance is bounded by the quality of the initial 2DGS reconstruction, and the method does not infer or hallucinate unseen object parts, limiting completeness of extracted assets — although integration with diffusion-based completion methods such as InstaScene is noted as feasible. The approach does not address language grounding or mask ID correlation: it depends on externally supplied ID-consistent 2D masks and cannot determine when two masks refer to the same object, unlike Trace3D. Transparent surfaces cause depth ambiguity that reprojection cannot resolve, and consistently failed SAM2 regions (e.g., low-texture areas) cannot be recovered. Finally, ground-truth "extracted" masks for several benchmarks were created manually by the authors using Trace3D outputs as reference, which introduces a potential evaluation dependency worth noting.

Conclusion

BEA-GS demonstrates that radiance-gated supervision is the principal obstacle to clean object extraction in 3DGS, and that combining a rasterizer-propagated boundary loss with a direct, rasterization-free occupancy loss yields state-of-the-art boundary quality across four benchmarks with negligible rendering degradation and modest overhead. The open questions left by the work concern robustness to weaker reconstructions, handling of transparent materials, and coupling geometric boundary refinement with language grounding and instance association.