- The paper introduces QAGaussian, a query-conditioned framework that combines multi-scale Gaussian slots, language-aware graph reasoning, adaptive mask routing, and consistency refinement to handle structured referring expressions.
- The method achieves 47.2 average mIoU and 63.2 average F1 across five benchmarks, outperforming ReferSplat by 2.7 mIoU and 2.9 F1 points while improving part-level and relation-dependent segmentation.
- QAGaussian reduces part-level errors, relation confusion, and target-reference confusion, but increases query time modestly to 0.82 seconds and remains sensitive to reconstruction quality and complex multi-hop relations.
QAGaussian addresses open-vocabulary referring segmentation in 3D Gaussian Splatting (3DGS): given a Gaussian scene and a free-form expression, the model must produce a probability mask over individual Gaussian primitives. The paper's central critique is that existing 3DGS pipelines reduce an expression to a global text embedding and select regions via text-region similarity. This strategy fails on structured queries involving attributes, reference objects, spatial relations, and parts, producing four characteristic failure modes: target-reference confusion (both target and reference object activated), granularity mismatch (whole-object output for part-level queries), part-whole leakage (predicted parts spilling beyond their parent object), and relation violation (semantically plausible masks inconsistent with the stated spatial relation). The paper reframes the task as query-conditioned neural reasoning over primitives rather than static similarity matching, and evaluates specifically on these failure modes rather than only aggregate accuracy.
Method
The framework operates in three stages, with two governing design principles: all intermediate representations remain at the Gaussian or slot level (so masks render from arbitrary viewpoints without box or point-cloud conversion), and all hard decisions are deferred to the final thresholding stage.
Query-conditioned hierarchical slot generation. Gaussian primitives are encoded with appearance features, opacity-aware geometry, positions, scales, and distilled semantic features. A text encoder produces a global embedding zQ, and a lightweight parser extracts soft structured cues {ztar,zattr,zrel,zref,zpart}. Notably, the parser provides no query-type labels; it only decomposes the expression into soft cues. Gaussian tokens are pooled at three scales (fine, middle, coarse) and learnable slot queries are modulated by zQ, then bound via slot attention. Each slot predicts soft Gaussian membership and an activeness score; inactive slots serve as unused capacity, allowing a fixed slot budget (192 total: 96/64/32 across scales) across scenes of varying complexity. This step also reduces the computational burden of downstream reasoning, which operates over slots rather than millions of primitives.
Relation-aware graph reasoning and adaptive routing. A lightweight graph is built over active slots, with edges encoding relative position, overlap, containment, scale ratio, and feature similarity. Edge weights are query-conditioned, aij=σ(MLPrel([rij,zrel])), so the same spatial displacement can be weighted differently under "next to," "beneath," or "in front of." A two-layer graph encoder propagates target-reference, attribute, part-whole, and contextual evidence. A granularity-adaptive router then predicts soft weights over five mask branches—region-level, object-level, part-level, attribute-aware, and relation-aware—and the initial mask is assembled by weighted slot aggregation. Soft routing is a deliberate choice: a compositional query can simultaneously draw on attribute-aware and relation-aware evidence without committing to a single hard query type.
Relation-constrained refinement and training. The routed mask is refined with four lightweight consistency priors—spatial relation, part-whole, attribute compatibility, and geometric continuity—each masked out when the corresponding cue is absent from the expression. Training uses Hungarian matching between predicted slots and ground-truth Gaussian masks (BCE + Dice losses, with activeness regularization for unmatched slots), soft router pseudo-labels derived from expression structure, and an alignment loss over slot-query pairs. The model is pretrained end-to-end on Mosaic3D-5.6M only, with hyperparameters fixed on its validation split and never fine-tuned on evaluation benchmarks.
Main results
Under a unified protocol that converts all annotations and baseline predictions to Gaussian-level masks, QAGaussian achieves 47.2 Avg. mIoU and 63.2 Avg. F1 across ScanRefer, ReferIt3D, Multi3DRefer, ReferSplat, and part-level splits, exceeding the strongest 3DGS referring baseline, ReferSplat, by 2.7 mIoU and 2.9 F1 points. It also attains the best [email protected] (72.0), [email protected] (45.7), precision (62.4), and recall (64.1) among all 23 compared baselines, spanning multi-view 2D projection, 3D grounding, open-vocabulary 3D segmentation, and 3DGS-native methods.
The gains are not uniform, and the paper states this plainly: ReferSplat slightly outperforms QAGaussian on the more object-centric ScanRefer (52.8 vs. 52.0) and ReferIt3D (48.6 vs. 48.0) benchmarks. QAGaussian's advantage concentrates on structurally complex queries—Multi3DRefer (50.7 vs. 46.9), the ReferSplat-style split (41.9 vs. 35.8), and part-level evaluation (43.4 vs. 38.6). The diagnostic results substantiate the claimed failure-mode reductions: Part-mIoU improves from 38.6 to 43.4, Rel-mIoU from 44.4 to 50.8, and target-reference confusion drops from 10.8 to 7.4. Qualitative examples in a fixed kitchen scene confirm granularity adaptation across object-level, attribute-aware, part-level, relation-dependent, and compositional queries within the same scene, isolating query structure as the variable rather than scene content.
Generalization, robustness, and ablations
Cross-dataset generalization is evaluated without any target-benchmark fine-tuning. Pretraining scale matters monotonically: 2D pseudo-mask pretraining reaches 41.2 mIoU, Mosaic3D-1.4M reaches 43.6, 2.8M reaches 45.4, and the full 5.6M reaches 47.2, indicating that Gaussian-level language alignment transfers across scene types and annotation formats. Under language perturbations (synonym replacement, relation paraphrase, attribute insertion, compositional rewrite, distractor-rich rewriting), QAGaussian retains more of its original performance than ReferSplat across all perturbation types, with the largest advantage on compositional rewrites and distractor-rich expressions.
The ablation study confirms that each component contributes, with the largest degradation from removing query-conditioned slots (mIoU falls from 47.2 to 43.8, F1 from 63.2 to 59.5). Removing the relation graph causes the largest Rel-mIoU drop and highest target-reference confusion (12.6), removing multi-scale slots costs 3.3 Part-mIoU, and removing the router, refinement, or activeness filtering each produces consistent but smaller declines. Qualitative ablations on a relation-dependent query show distinct failure signatures per removed component: facade leakage without query-conditioned slots, fragmentation across similar glass regions without the relation graph, and coarser masks without routing.
Efficiency
The reasoning overhead is moderate because slot construction, graph message passing, and refinement operate over 192 slots rather than all primitives. On medium-scale scenes, query time rises from 0.66 s (ReferSplat) to 0.82 s, and memory from 9.1 GB to 9.8 GB, in exchange for the 2.7-point mIoU gain. OpenGaussian remains the fastest method, but the paper positions QAGaussian as trading modest latency for accuracy on complex expressions, which is argued to be the appropriate trade-off for embodied interaction and augmented reality applications.
Limitations and open questions
The paper concedes several limitations at the point they bear on the results. Performance depends on 3DGS reconstruction quality; noisy, incomplete, or inaccurate Gaussians from occlusion, transparency, reflective surfaces, or sparse views degrade the predicted mask. Very small parts and thin structures remain difficult because they require both high-quality geometry and fine semantic alignment. The relation-aware graph does not fully solve multi-hop expressions, ambiguous references, or relations such as "between," "inside," and "aligned with." Additionally, the unified evaluation protocol requires converting heterogeneous annotations (2D masks, boxes, point clouds) into Gaussian-level masks via reconstruction and nearest-neighbor or opacity-weighted assignment, so absolute numbers are protocol-dependent and not directly comparable to original benchmark leaderboards. Open questions left by the paper include uncertainty-aware slot selection, stronger 3D scene graph priors, interactive correction, and extension to dynamic or 4D Gaussian scenes.
Conclusion
QAGaussian demonstrates that treating 3DGS referring segmentation as query-conditioned reasoning—rather than text-region similarity matching—yields measurable improvements precisely where similarity-based methods fail: part-level granularity (43.4 vs. 38.6 Part-mIoU), relation-dependent disambiguation (50.8 vs. 44.4 Rel-mIoU), and target-reference separation (7.4 vs. 10.8 confusion rate). The combination of query-conditioned multi-scale slot generation, language-conditioned graph reasoning, soft granularity routing, and consistency-based refinement achieves state-of-the-art average performance with zero-shot transfer from a single pretraining corpus, at a moderate computational cost. The evidence supports the paper's claim that the gains reflect addressing concrete modeling failures rather than added engineering complexity, though the method's dependence on reconstruction quality and its remaining difficulty with multi-hop relations delineate the current boundary of this approach.