3D-GALP: Geometry-Aware Labeling

• 3D-GALP is a design principle where label prediction is constrained by explicit 3D geometry rather than solely by 2D appearance.
• It employs methods such as Gaussian occupancy, geometric attention, and world-space unprojection to enhance label consistency and semantic precision.
• Empirical results across segmentation, localization, and robotics show improved mIoU and robustness by leveraging geometric consistency in label refinement.

Searching arXiv for the cited 3D-geometry-aware label prediction papers and related work. arXiv search query: "TrianguLang geometry-aware semantic consensus pose-free 3D localization" 3D-Geometry Aware Label Prediction (3D-GALP) denotes a family of methods in which label prediction is constrained, parameterized, or regularized by explicit 3D geometric structure rather than by 2D appearance alone. In the narrow sense, the term appears as the robust 3D mask-generation module in RoMaP for part-level editing in 3D Gaussian Splatting; in a broader research sense, it covers systems that predict labels through projected 3D primitives, voxel or point-based geometry, geometry-aware attention, cross-view consistency, or future geometric state prediction (Kim et al., 15 Jul 2025, Grant et al., 9 Mar 2026).

1. Conceptual scope and lineage

Earlier work already established the core premise that labeling quality improves when scene structure is modeled geometrically rather than only photometrically. “GAL: A Global-Attributes Assisted Labeling System for Outdoor Scenes” integrated sky lines, ground lines, horizon, planar surfaces, vertical lines, vanishing lines, solid objects, and porous materials into a CRF over superpixels for 7-way outdoor geometric layout labeling (Ren et al., 2016). “Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations” then treated a reconstructed 3D occupancy grid as an intermediate geometry-aware representation for action-conditioned grasp outcome labels (Yan et al., 2017). “3D Geometry-Aware Semantic Labeling of Outdoor Street Scenes” moved dense semantic labeling into a $UVD$ voxel volume and used 3D convolution instead of treating depth as a fourth image channel (Zhong et al., 2018). “ILabel: Interactive Neural Scene Labelling” later unified geometry, colour, and semantics in a scene-specific neural field trained from sparse user clicks, with semantic rendering tied to the same volumetric weights as depth and colour (Zhi et al., 2021).

Recent work has broadened this logic. Some methods attach labels directly to 3D entities, such as Gaussians, voxels, superpoints, or pointmaps; others use geometry only as a consistency prior over otherwise semantic predictions. This suggests that 3D-GALP is best understood not as a single architecture, but as a design principle: labels should be attached to, propagated through, or checked against a representation whose coordinates, neighborhoods, or correspondences have 3D meaning.

2. Representational forms of geometry-aware labels

A central distinction within 3D-GALP is the form of the predicted label. Some methods predict labels that are explicitly geometric; others predict semantics in 2D but lift them into 3D-consistent outputs.

Representative formulations span several label spaces (Gaudillière et al., 2023, Grant et al., 9 Mar 2026, Kim et al., 15 Jul 2025, Xu et al., 17 Oct 2025, Xu et al., 27 Feb 2026):

Paradigm	Label/output form	Geometric carrier
TrianguLang (Grant et al., 9 Mar 2026)	$\{\mathbf{M}_i\}_{i=1}^N$ and $\mathbf{c}\in\mathbb{R}^3$	Predicted depth, intrinsics, extrinsics, world coordinates
3D-aware Gaussian occupancy (Gaudillière et al., 2023)	$$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$	Projection of a 3D ellipsoid
RoMaP 3D-GALP (Kim et al., 15 Jul 2025)	Per-Gaussian label parameter $\mathbf{r}_i$	3D Gaussians with SH view dependence
3D WSSS refinement (Xu et al., 17 Oct 2025)	$Y\in\{1,\dots,C\}^N$	Superpoints from normal-based graph cut
Action-Geometry Prediction (Xu et al., 27 Feb 2026)	$\mathbf{f}_{t+N}$ and $P_{t+N}\in\mathbb{R}^{H\times W\times 4}$	Future dense pointmap decoded from 3D latent

The projected-label formulation is especially explicit in “3D-Aware Object Localization using Gaussian Implicit Occupancy Function.” There the target label is not a bounding box but a Gaussian occupancy label induced by projecting a 3D ellipsoid. The paper defines $Q^{*}= \operatorname{diag}(a'^2,b'^2,c'^2,-1)$, projects it by $C^{*}=PQ^{*}P^\top$, then extracts the Gaussian mean and covariance, yielding $\{\mathbf{M}_i\}_{i=1}^N$0 (Gaudillière et al., 2023). The label therefore encodes center, anisotropic extent, and in-plane orientation through $\{\mathbf{M}_i\}_{i=1}^N$1, rather than only 2D extent.

Other systems keep 2D outputs but make them geometry-indexed. TrianguLang predicts binary masks in each view and a 3D centroid, with every pixel unprojected into a shared estimated world frame using predicted depth and camera parameters (Grant et al., 9 Mar 2026). RoMaP’s 3D-GALP instead attaches a learnable semantic parameter $\{\mathbf{M}_i\}_{i=1}^N$2 to each 3D Gaussian and renders view-dependent segmentation maps from those per-Gaussian labels (Kim et al., 15 Jul 2025). In 3D weakly supervised semantic segmentation, the output remains dense per-point labels, but geometric plausibility is imposed through superpoints rather than dense reconstruction (Xu et al., 17 Oct 2025). In robotics, the “label” may itself be a future geometric state: Action-Geometry Prediction jointly predicts a future action chunk and a future 3D latent that decodes into a dense pointmap $\{\mathbf{M}_i\}_{i=1}^N$3 (Xu et al., 27 Feb 2026).

3. Mechanisms for injecting geometry into label prediction

The defining mechanisms of 3D-GALP are the operations that make semantic affinity subordinate to geometric compatibility. A direct example is TrianguLang’s unprojection backbone: $\{\mathbf{M}_i\}_{i=1}^N$4 followed by world-space positional encoding and Geometry-Aware Semantic Attention (GASA), which penalizes semantically plausible but 3D-distant token pairs through a learned distance kernel $\{\mathbf{M}_i\}_{i=1}^N$5 (Grant et al., 9 Mar 2026). The same paper’s sheaf consistency loss further turns geometry into an explicit semantic regularizer by enforcing agreement between mask probabilities at pixels whose 3D points are within $\{\mathbf{M}_i\}_{i=1}^N$6 cm.

Projected-shape approaches use geometry even earlier, at label construction time. In the Gaussian occupancy formulation, the geometry-aware step is $\{\mathbf{M}_i\}_{i=1}^N$7, after which the projected conic is converted into Gaussian parameters and then into a pixelwise heatmap. E-DSNT decodes $\{\mathbf{M}_i\}_{i=1}^N$8 and $\{\mathbf{M}_i\}_{i=1}^N$9 from the normalized heatmap by differentiable moment extraction, avoiding direct regression of discontinuous ellipse angles (Gaudillière et al., 2023).

RoMaP’s 3D-GALP addresses a different problem: one 3D Gaussian may contribute to different semantic parts depending on viewpoint. Its solution is to represent each Gaussian’s label via spherical harmonics, $\mathbf{c}\in\mathbb{R}^3$0, then quantify label softness by $\mathbf{c}\in\mathbb{R}^3$1, where $\mathbf{c}\in\mathbb{R}^3$2 is entropy over part-label similarities and $\mathbf{c}\in\mathbb{R}^3$3 is variance across views. Anchor sampling chooses both high-softness and low-softness Gaussians, and a 3D neighbor loss

$\mathbf{c}\in\mathbb{R}^3$4

propagates local consistency in Euclidean 3D space (Kim et al., 15 Jul 2025).

Correspondence-based methods make geometry the transport map for supervision. “Geometry-Aware Score Distillation via 3D Consistent Noising and Gradient Consistency Modeling” uses depth and camera geometry to warp supervision across views, comparing gradients only at 3D-corresponding pixels (Kwak et al., 2024). “3DiffTection” similarly introduces an epipolar warp operator during geometric tuning for novel view synthesis, with target pixels aggregating source features only along epipolar-valid support: $\mathbf{c}\in\mathbb{R}^3$5 This is a geometry-aware alternative to naive cross-view feature transfer and is directly reusable for label or pseudo-label propagation (Xu et al., 2023).

4. Supervision and optimization

3D-GALP methods differ sharply in what they supervise: projected geometry, rendered semantics, pseudo-labels, or future geometric states. Yet their objectives share a common pattern: a semantic term is paired with a geometric or geometry-derived consistency term.

TrianguLang’s full objective is

$\mathbf{c}\in\mathbb{R}^3$6

where $\mathbf{c}\in\mathbb{R}^3$7 combines focal and Dice losses for per-view masks, $\mathbf{c}\in\mathbb{R}^3$8 supervises the 3D centroid, and $\mathbf{c}\in\mathbb{R}^3$9 enforces agreement between mask probabilities at geometrically corresponding pixels (Grant et al., 9 Mar 2026). The method’s reported oracle-to-predicted mIoU gap of only $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$0 point, versus $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$1 points for SAM3, highlights that confidence calibration is part of geometry-aware label prediction rather than an afterthought.

The Gaussian occupancy method uses a distributional supervision regime: $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$2 combining a 2-Wasserstein distance over Gaussian parameters with a Jensen-Shannon divergence over pixelwise heatmaps (Gaudillière et al., 2023). This treats the label as both an explicit geometric parameter set and an implicit occupancy field.

In weakly supervised 3D segmentation, the sophistication lies less in the network loss than in pseudo-label curation. The final 3D model is trained with pointwise cross-entropy on refined labels, but those labels are first filtered by Class-Aware Label Refinement and then by Geometry-Aware Label Refinement, which assigns a superpoint label only if the majority-class dominance ratio

$$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$3

exceeds $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$4 (Xu et al., 17 Oct 2025). Iterative self-training then expands coverage while retaining previously trusted labels.

ILabel offers a different template: geometry, photometric appearance, and semantics are jointly optimized in a single neural field. The rendered depth, colour, and semantic predictions share the same volumetric weights $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$5, and the overall objective combines geometric, photometric, and semantic error terms (Zhi et al., 2021). The implication is that semantics inherit 3D consistency because they are rendered from the same density field that explains RGB-D observations.

5. Domains, tasks, and empirical profile

The empirical scope of 3D-GALP is unusually broad. It includes text-guided localization, weakly supervised segmentation, part-aware editing, monocular or multi-view detection, interactive scene labeling, and robotics.

In pose-free language grounding, TrianguLang is a particularly explicit 3D-GALP system. On ScanNet++ it reaches $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$6 mIoU and $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$7 mAcc, and on uCO3D it reaches $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$8 mIoU and $$L_{obj}^{(Gauss)}=[\mu_x,\mu_y,\Sigma_{xx},\Sigma_{yy},\Sigma_{xy}]$$9 mAcc. In cross-domain transfer, training on ScanNet++ and evaluating on uCO3D gives $\mathbf{r}_i$0 mIoU versus MV-SAM’s $\mathbf{r}_i$1. The ablation is equally diagnostic: on ScanNet++, the baseline with GASA and world-space positional encoding gets $\mathbf{r}_i$2 mIoU; removing GASA drops to $\mathbf{r}_i$3, removing world-space positional encoding to $\mathbf{r}_i$4, and removing both to $\mathbf{r}_i$5. Replacing the learned kernel with a fixed RBF kernel drops performance to $\mathbf{r}_i$6 (Grant et al., 9 Mar 2026).

In weak or low-cost supervision regimes, geometry-aware label prediction often matters most because the raw labels are unreliable. The class-aware and geometry-guided pseudo-label refinement method for 3D WSSS reports $\mathbf{r}_i$7 mIoU on ScanNet validation and $\mathbf{r}_i$8 test with LSeg initialization, as well as $\mathbf{r}_i$9 mIoU on S3DIS. Its ablations show a jump from $Y\in\{1,\dots,C\}^N$0 mIoU for a global top-$Y\in\{1,\dots,C\}^N$1 baseline to $Y\in\{1,\dots,C\}^N$2 with class-aware refinement and $Y\in\{1,\dots,C\}^N$3 with class-aware plus geometry-aware refinement (Xu et al., 17 Oct 2025). RoMaP’s 3D-GALP, evaluated on complex scenes from 3D-OVS, reports average mIoU $Y\in\{1,\dots,C\}^N$4, compared with $Y\in\{1,\dots,C\}^N$5 for LeGaussian and $Y\in\{1,\dots,C\}^N$6 for LangSplat, establishing that view-consistent 3D part masks can be improved substantially by per-Gaussian geometry-aware label modeling (Kim et al., 15 Jul 2025).

In projected-shape labeling, the Gaussian occupancy method demonstrates that geometry-aware labels can outperform conventional box-based targets even when evaluation remains in 2D. On AKM it reaches $Y\in\{1,\dots,C\}^N$7 IoU and $Y\in\{1,\dots,C\}^N$8 MHD; on SPEED+ the best variant achieves $Y\in\{1,\dots,C\}^N$9 IoU and $\mathbf{f}_{t+N}$0 MHD. More importantly for 3D reasoning, reconstruction from predicted ellipses on SPEED+ yields $\mathbf{f}_{t+N}$1 cm position error, $\mathbf{f}_{t+N}$2 orientation error, and $\mathbf{f}_{t+N}$3 cm size error, versus $\mathbf{f}_{t+N}$4 cm / $\mathbf{f}_{t+N}$5 / $\mathbf{f}_{t+N}$6 cm for the direct ellipse-regression baseline (Gaudillière et al., 2023).

In robotics, the same principle appears as future geometry-aware state prediction. Action-Geometry Prediction predicts a future action chunk together with a future 3D latent and dense pointmap, and the ablation shows that removing the “geometric imagination” branch lowers performance from $\mathbf{f}_{t+N}$7 average success to $\mathbf{f}_{t+N}$8 on four hard RoboTwin tasks (Xu et al., 27 Feb 2026). This suggests that 3D-GALP extends naturally beyond semantic segmentation into structured decision-making, where the predicted label is a future geometric state rather than a class.

6. Limitations, misconceptions, and open questions

A common misconception is that geometry-aware label prediction necessarily requires calibrated cameras, explicit point clouds, or dense 3D ground truth. Several systems instead rely on predicted geometry, weak priors, or indirect spatial structure: TrianguLang explicitly targets “no ground-truth camera parameters at any stage, neither training nor inference”; the Gaussian occupancy method needs only coarse object dimensions plus pose labels; RoMaP lifts pseudo 2D attention maps into per-Gaussian labels; and 3D WSSS injects geometry only through superpoints (Grant et al., 9 Mar 2026, Gaudillière et al., 2023, Kim et al., 15 Jul 2025, Xu et al., 17 Oct 2025).

A second misconception is that 3D-GALP always means dense scene-wide semantic labeling. Many high-performing systems are narrower: TrianguLang is optimized for language-guided localization of a queried target rather than fixed-ontology scene parsing; RoMaP predicts part masks for editable Gaussians; the Gaussian occupancy method is category-specific and object-centric; MonoDLGD’s geometry-aware label mechanism is a training-time denoising and reconstruction objective rather than an inference-time label predictor (Grant et al., 9 Mar 2026, Kim et al., 15 Jul 2025, Gaudillière et al., 2023, Lee et al., 17 Nov 2025).

The main failure mode across the literature is geometry quality itself. TrianguLang explicitly identifies reflective surfaces such as mirrors and glass, turntable-style captures with small baselines, and fewer than $\mathbf{f}_{t+N}$9 views as failure cases; its absolute 3D localization is bounded by DA3’s metric error, reported around $P_{t+N}\in\mathbb{R}^{H\times W\times 4}$0 cm (Grant et al., 9 Mar 2026). The Gaussian occupancy pipeline depends on ellipsoidal approximability and known camera intrinsics and relative pose labels, so articulated or non-ellipsoidal objects are outside its comfortable regime (Gaudillière et al., 2023). GAP-MLLM shows that sparse point-level geometry-semantic supervision activates 3D perception, but also reports coarse local structure and blurry semantic boundaries, indicating that sparse geometry-aware labeling does not automatically yield dense precision (Zhang et al., 17 Mar 2026).

This suggests several open directions. One plausible implication is that future 3D-GALP systems will move from query-conditioned or object-centric outputs toward persistent 3D semantic fields, scene graphs, or temporally updated world models. Another is that uncertainty-aware geometry fusion will become central: if geometry is wrong, geometry-aware label prediction becomes geometry-misled label prediction. A third is that the most scalable systems may keep explicit geometry only as an intermediate supervisory or routing signal, while emitting flexible structured outputs—masks, boxes, pointmaps, relations, or future states—from a shared geometry-aware representation.