Query-Guided Semantic Aggregation

• Query-guided semantic aggregation is a framework where dynamic, input-adaptive queries direct the selection and weighting of semantic evidence from structured inputs.
• Recent studies demonstrate its effectiveness in tasks like 3D scene segmentation and DETR object detection, achieving improvements such as a 2.5% mAP gain with reduced computational cost.
• The approach separates query construction from aggregation, enabling precise local pooling, residual updates, and semantic-aware sampling to enhance performance and reduce redundancy.

Taken together, the cited papers suggest that query-guided semantic aggregation denotes a family of mechanisms in which a query representation guides the selection, weighting, or reintegration of semantic evidence from structured inputs. Recent formulations span 3D scene instance segmentation, DETR-style object detection, few-shot segmentation, 3D occupancy prediction, knowledge-graph aggregation, and visual place recognition. In these settings, queries are not merely decoder slots: they are scene-adaptive carriers of semantic priors, spatial hypotheses, or retrieval anchors that determine which features, masks, voxels, tokens, or graph paths are aggregated and which are suppressed (Yao et al., 11 Feb 2026, Zhang et al., 2024, Shi et al., 2022, Moon et al., 28 Mar 2025, Wang et al., 2022, Zhu et al., 19 Jan 2026).

1. Core computational pattern

Several recent works instantiate the same high-level pattern with different primitives. LaSSM derives a query set from superpoints by combining semantic cues and spatial distribution, then refines those queries through local aggregation and a coordinate-guided state space model decoder (Yao et al., 11 Feb 2026). SACQ generates content queries from transformer encoder features via self-attention pooling and then aggregates similar predictions before Hungarian matching (Zhang et al., 2024). DCAMA treats every query pixel as a token and predicts its segmentation label as an additive aggregation of support-mask values weighted by cross-image similarity (Shi et al., 2022). StreamOcc recurrently aggregates instance-level features of dynamic objects into corresponding voxel features (Moon et al., 28 Mar 2025). DC-VLAQ aggregates local tokens by their residual responses to learnable queries (Zhu et al., 19 Jan 2026). On knowledge graphs, semantic-aware sampling biases a random walk toward nodes and paths semantically similar to the query predicates, and the aggregate result is then estimated from the sample (Wang et al., 2022).

This recurring structure separates query formation from query-conditioned aggregation. The first stage determines what constitutes a useful hypothesis; the second stage determines how evidence is pooled around that hypothesis. A plausible implication is that the concept is broader than transformer self-attention or cross-attention: it includes local neighbor aggregation, residual pooling, support-mask transfer, voxel updates, and even query-guided sampling-estimation.

2. Query construction and initialization

In LaSSM, query quality is addressed explicitly through a hierarchical semantic-spatial query initializer. Superpoint-level features and coordinates are obtained by superpoint pooling, category probabilities are predicted with a lightweight MLP classifier, superpoints are ranked by semantic score, the top-$m$ candidates are retained, and Farthest Point Sampling (FPS) is applied to ensure spatial coverage. The selected features are then projected into the decoder space:

$$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$

The paper characterizes this procedure as producing queries that are both semantic-driven and well-distributed, with comprehensive scene coverage and accelerated convergence (Yao et al., 11 Feb 2026).

SACQ addresses a related issue in DETR variants: the content query is traditionally zero-initialized or a learnable embedding, which lacks image-adaptive content. Its Self-Adaptive Content Query module uses encoder features to generate query-specific vectors through attention map projection, weighted pooling, and channel reweighting. The pooled feature for query $i$ is

$$F^P_i = \sum_{j=0,k=0}^{h,w} F[:,j,k]\cdot A[i,j,k],$$

followed by

$F^O = \sigma(\mathrm{MLP}(F^P)) \odot F^P.$

The first decoder layer uses global SAPM over encoder feature maps, while subsequent layers use RoI-Align on the previous predicted box to refine the content query locally (Zhang et al., 2024).

StreamOcc also assigns a central role to query formation, though at the instance level rather than the decoder-initialization level. Its image-to-query detector, based on Sparse4Dv3, produces a set of instance queries from multi-view image features; high-confidence queries are recurrently maintained for temporal coherence, and self- and cross-attention refine their object-level content over time (Moon et al., 28 Mar 2025).

These formulations share the premise that the aggregation stage is only as good as the query prior. This suggests that query-guided semantic aggregation often begins by replacing static, learned, or spatially naive query initialization with input-adaptive construction.

3. Aggregation operators

The aggregation operator varies substantially across domains.

LaSSM replaces conventional quadratic decoder attention with local aggregation over spatial neighbors. For each query, $k$-NN is performed in coordinate space, query content is expanded to match the neighbor set, and the interaction is accumulated as

$$h_i = \sum_{j=1}^k K_{i,j} \cdot (Q'_{i,j} \odot F_{i,j}),$$

followed by a residual projection. The decoder is described as attention-free, and the local restriction is intended to avoid noisy information and redundant computation while focusing on geometrically coherent regions (Yao et al., 11 Feb 2026).

DCAMA uses dense pixel-wise cross-query-and-support attention. Every query pixel attends to all support pixels, and the output is a direct aggregation of support-mask values:

$$\hat{M}^q = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V.$$

The formulation is explicitly nonparametric at label-transfer time: both foreground and background support pixels contribute, and the predicted mask value for a query pixel is an additive, similarity-weighted aggregation of support mask labels (Shi et al., 2022).

DC-VLAQ changes the target of aggregation from absolute features to query-aligned residuals. After residual-guided complementary fusion of DINOv2 and CLIP tokens, each token is softly assigned to learnable queries, and the descriptor for query $k$ is

$$\bm{v}_{ik} = \sum_j \alpha_{ijk} \, (\bm{z}_{ij} - \bm{q}_k).$$

The final global descriptor is the concatenation of these residual aggregates. The paper positions this as a query--residual global aggregation scheme that improves stability and preserves fine-grained discriminative cues (Zhu et al., 19 Jan 2026).

StreamOcc uses Dynamic Query Aggregation to update voxel features only where selected dynamic-object queries overlap voxel regions. Query-to-voxel attention and a sigmoid gating mechanism modulate the update so that only voxels with overlapping queries are altered, and the gate prevents noisy or inaccurate query features from corrupting voxel features (Moon et al., 28 Mar 2025).

On knowledge graphs, the aggregation primitive is not a neural decoder but a semantic-aware random walk. Transition probabilities are proportional to predicate similarity in embedding space:

$$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$0

Approximate COUNT, SUM, and AVG are then computed from the resulting random sample using unbiased estimators for COUNT and SUM and a consistent estimator for AVG, with a confidence interval as the accuracy guarantee (Wang et al., 2022).

4. Representative task-level instantiations

In 3D scene instance segmentation, LaSSM combines its initializer with a coordinate-guided SSM decoder and reports first place on the latest ScanNet++ V2 leaderboard, outperforming the previous best method by 2.5% mAP with only 1/3 FLOPs. It also reports competitive performance on ScanNet, ScanNet200, S3DIS, and ScanNet++ V1 with less computational cost (Yao et al., 11 Feb 2026).

In DETR variants, SACQ and query aggregation are presented as a plug-and-play modification. Across six DETR variants and multiple configurations on COCO, the paper reports an average improvement of over 1.0 AP. The detailed summary lists gains including Deformable-DETR $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$1, DAB-DETR $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$2, DN-DETR $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$3 AP, and DINO $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$4 (Zhang et al., 2024).

In few-shot segmentation, DCAMA is evaluated on PASCAL-5$$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$5, COCO-20$$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$6, and FSS-1000 and reports 3.1%, 9.7%, and 3.6% absolute improvements in 1-shot mIoU over previous best records. The method attributes these gains to dense pixel-wise correspondences and direct support-mask aggregation rather than prototype compression (Shi et al., 2022).

In 3D occupancy prediction, StreamOcc combines stream-based voxel aggregation with query-guided aggregation. The abstract states that it achieves state-of-the-art performance in real-time settings while reducing memory usage by more than 50% compared to previous methods. The detailed results report $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$7 mIoU for the StreamAgg baseline and $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$8 mIoU with QueryAgg, with dynamic objects improving from $$\mathbf{Q} = \phi_\text{proj}(\mathbf{f}_s[\mathbf{ids}_f]) \in \mathbb{R}^{q \times d}.$$9 to $i$0, and a comparison against an SCA-based approach at $i$1 mIoU, $i$2 ms, and $i$3 MB versus QueryAgg at $i$4 mIoU, $i$5 ms, and $i$6 MB (Moon et al., 28 Mar 2025).

In knowledge-graph processing, semantic-aware sampling is presented as the first work to provide an approximate aggregate result with an effective accuracy guarantee without relying on factoid queries. The paper reports relative error $i$7 on tested queries, response time $i$8 orders of magnitude less than exhaustive enumeration, and support for FILTER, GROUP-BY, and query shapes such as chain, cycle, star, and flower (Wang et al., 2022).

In visual place recognition, DC-VLAQ is positioned as a representation-centric framework that integrates complementary VFM fusion with robust global aggregation. It reports state-of-the-art performance on benchmarks including Pitts30k, Tokyo24/7, MSLS, Nordland, SPED, and AmsterTime, particularly under domain shifts and long-term appearance changes (Zhu et al., 19 Jan 2026).

5. Efficiency, optimization, and supervision

A major theme in the literature is that query-guided semantic aggregation is often introduced to mitigate optimization or efficiency bottlenecks rather than to add model complexity.

LaSSM explicitly targets the cost of attention-based query decoders. Its local aggregation is described as fast and linear in the number of queries, and the attention-free decoder reduces overhead while restricting the model to geometrically coherent regions (Yao et al., 11 Feb 2026). StreamOcc makes a closely related argument in 3D occupancy prediction: global semantic aggregation over all voxels is computationally heavy and can blur dynamic objects, whereas Dynamic Query Aggregation only enriches voxels associated with selected dynamic-object queries (Moon et al., 28 Mar 2025).

In DETR variants, the bottleneck is optimization under one-to-one Hungarian matching. SACQ improves query quality, but that improvement causes multiple queries to produce very similar high-quality predictions for the same target. Query aggregation therefore merges predictions when category similarity, measured by symmetric KL divergence, is below a threshold and box similarity, measured by IoU, is above a threshold. The paper also notes that the method is sensitive to the box IoU threshold, and that a threshold that is too low leads to performance drops due to over-aggregation of distinct instances (Zhang et al., 2024).

DCAMA addresses a different inefficiency: support compression. Rather than averaging support evidence into one or a few prototypes, it retains dense support pixels and further proposes one-pass inference for $i$9-shot segmentation by collecting pixels of all support images for mask aggregation at once (Shi et al., 2022). The knowledge-graph method likewise rejects exhaustive factoid-query enumeration and instead provides iterative refinement of the aggregate estimate until a user-specified relative error bound is met with the desired confidence level (Wang et al., 2022). DC-VLAQ reframes robustness as a distribution-shift problem: fusion of complementary VFMs alters token distributions, and residual aggregation is introduced precisely because absolute query-response pooling can become unstable under those shifts (Zhu et al., 19 Jan 2026).

6. Related mechanisms, misconceptions, and distinctions

A common misconception is that query-guided semantic aggregation is synonymous with global transformer attention. The recent literature does not support that equivalence. LaSSM uses local aggregation plus a spatial dual-path SSM block rather than conventional self/cross-attention (Yao et al., 11 Feb 2026). StreamOcc performs locally focused query-to-voxel updates only for overlapped voxel regions (Moon et al., 28 Mar 2025). The knowledge-graph work uses semantic-aware random walks and estimators rather than attention modules (Wang et al., 2022).

A second misconception is that semantic aggregation should compress the support or context into a small number of prototypes. DCAMA is explicitly formulated against that assumption: it argues that prototype pooling and foreground-only support use cause information loss, and it instead exploits both foreground and background support information through dense pixel-wise correlations (Shi et al., 2022). A related misconception is that more similar queries should always be kept separate; SACQ shows the opposite failure mode, namely that multiple good queries can be penalized as false positives under Hungarian matching unless they are merged before loss computation (Zhang et al., 2024).

A final distinction concerns the difference between guided aggregation and query-guided semantic aggregation. BiSeNet V2 provides an important adjacent example. Its Guided Aggregation Layer fuses a Detail Branch and a Semantic Branch through sigmoid-gated, element-wise interactions:

$$F^P_i = \sum_{j=0,k=0}^{h,w} F[:,j,k]\cdot A[i,j,k],$$0

The mechanism is guided and selective, but the guidance arises from branch-to-branch modulation rather than explicit query tokens (Yu et al., 2020). This distinction clarifies the broader landscape: not all guided fusion is query-guided, yet many of the same design concerns—selectivity, mutual constraint, preservation of detail, and computational economy—recur across both categories.

Across these works, a consistent picture emerges. Query-guided semantic aggregation is repeatedly used to align semantic evidence with input-adaptive hypotheses, to reduce interference from irrelevant context, and to control computational cost. This suggests an emerging methodological preference for query construction that is semantic-aware, aggregation that is locality-aware or residual-based, and update rules that preserve discriminative structure while avoiding redundant or noisy interactions.