Learning Where to Embed: Noise-Aware Positional Embedding for Query Retrieval in Small-Object Detection

Abstract: Transformer-based detectors have advanced small-object detection, but they often remain inefficient and vulnerable to background-induced query noise, which motivates deep decoders to refine low-quality queries. We present HELP (Heatmap-guided Embedding Learning Paradigm), a noise-aware positional-semantic fusion framework that studies where to embed positional information by selectively preserving positional encodings in foreground-salient regions while suppressing background clutter. Within HELP, we introduce Heatmap-guided Positional Embedding (HPE) as the core embedding mechanism and visualize it with a heatbar for interpretable diagnosis and fine-tuning. HPE is integrated into both the encoder and decoder: it guides noise-suppressed feature encoding by injecting heatmap-aware positional encoding, and it enables high-quality query retrieval by filtering background-dominant embeddings via a gradient-based mask filter before decoding. To address feature sparsity in complex small targets, we integrate Linear-Snake Convolution to enrich retrieval-relevant representations. The gradient-based heatmap supervision is used during training only, incurring no additional gradient computation at inference. As a result, our design reduces decoder layers from eight to three and achieves a 59.4% parameter reduction (66.3M vs. 163M) while maintaining consistent accuracy gains under a reduced compute budget across benchmarks. Code Repository: https://github.com/yidimopozhibai/Noise-Suppressed-Query-Retrieval

• The paper introduces the HELP paradigm with HPE to selectively preserve object-relevant positional embeddings, effectively reducing background noise.
• It utilizes MOHFE and HQ-Retrieval to fuse multi-scale heatmap features and generate compact, discriminative query vectors.
• Experiments on datasets like NWPU VHR-10 and PASCAL VOC demonstrate significant mAP gains and reduced computational complexity.

Learning Where to Embed: Noise-Aware Positional Embedding for Query Retrieval in Small-Object Detection

Introduction

The paper "Learning Where to Embed: Noise-Aware Positional Embedding for Query Retrieval in Small-Object Detection" (2604.15065) presents a precisely targeted advance in transformer-based small-object detection. The motivation is clear: transformer detectors, while advantageous for their global context modeling and set-based paradigm, tend to propagate background noise into positional embeddings, which both dilutes the discriminative power of queries and increases the computational burden due to their reliance on deep, repeated refinement in the decoder. This effect is particularly detrimental in aerial and remote sensing imagery, where background dominates and target features are sparse and fine-grained.

Overview of HELP Paradigm

The proposed Heatmap-guided Embedding Learning Paradigm (HELP) resolves these inefficiencies by introducing a noise-aware, positional-semantic fusion strategy—Heatmap-guided Positional Embedding (HPE)—which is tightly integrated throughout the encoding and decoding pipeline. Rather than full spatial application, positional encodings are selectively preserved in regions with high heatmap activations corresponding to salient objects, while background-dominated regions are explicitly suppressed.

Figure 1: HELP architecture: LSConv backbone enhancement, MOHFE encoder fusing class/box heatmaps for embeddings, HQ-Retrieval for background suppression, and interactive HPE heatbar visualization.

The paradigm comprises three main technical contributions:

• HPE (Heatmap-guided Positional Embedding): A core mechanism for selective, interpretable embedding allocation.
• MOHFE (Multi-Scale ObjectBox-Heatmap Fusion Encoder): Semantic and geometric fusion of multi-scale heatmap features for encoder-side alignment.
• HQ-Retrieval (High-Quality Query Retrieval): Gradient-based filtering of background-dominant embeddings, reducing decoder depth.

Additionally, Linear-Snake Convolution (LSConv) is introduced to enrich retrieval-relevant feature extraction.

Heatmap-Guided Positional Embedding (HPE)

HPE is central to the suppression of noisy embeddings. It operates by generating heatmaps based on higher-order gradients of detection confidences and regression losses, producing spatial masks that precisely control where positional signals are injected. This heatmap, after binarization, determines whether a region's positional code is preserved (object-relevant) or eliminated (background).

Figure 2: HPE visualization—heatmaps modulate encoder/decoder input, with higher activation signifying salient regions for embedding.

Figure 3: HPE-driven query retrieval—binary heatmap mask gates out background embeddings, both during encoding and prior to decoding.

This mechanism fully decouples inference-time efficiency from the additional backward computation required during training (higher-order gradients), ensuring no runtime overhead during deployment.

Encoding and Query Retrieval Pipeline

MOHFE Encoder

MOHFE fuses class-driven and box-driven heatmaps to form heatmap-conditioned encoder keys and values. The encoder’s self-attention operates on these multi-scale, masked embeddings, promoting robust memory for subsequent high-quality query selection.

HQ-Retrieval Decoder

The decoder’s queries are not unfiltered positional tokens but are derived from the mixed heatmap output, resulting in compact, background-suppressed, discriminative query vectors:

• Salient, geometry- and class-relevant locations are prioritized.
• Deformable attention exploits the noise-suppressed encoder memory.

This dramatically reduces the need for iterative decoder refinement and, in practice, allows reduction in depth from eight decoder layers to three.

Backbone Enrichment via LSConv

LSConv addresses the deficiency of standard convolutions for sparse, fine-structure evidence proliferation. It deploys a dual-path, axis-wise decomposition:

• Snake branch: Learns offsets for tracing curvilinear or fragmented object structure.
• Linear branch: Maintains rigid, straight-path context.

Controlled regularization ensures stability in receptive field deformation, jointly capturing non-rigid and rigid spatial evidence.

Figure 4: LSConv dual-path axis sampling, combining linear and snake-like offset prediction for geometric adaptability.

Figure 5: Comparative sampling strategies on $9 \times 9$ grids: LSConv captures both canonical and highly deformed contexts, as opposed to fixed or unconstrained operators.

Experimental Analysis

Quantitative Performance

Across five benchmarks, including NWPU VHR-10 and PASCAL VOC, the HELP paradigm achieves a strong empirical profile:

• NWPU VHR-10: 94.51% [email protected], 67.20% mAP, with 59.4% parameter reduction compared to RT-DETR (66.3M vs. 163M) and 58% GFLOP reduction. Absolute gains of +1.9 ([email protected]) and +6.9 (mAP) were observed over the RT-DETR baseline.
• PASCAL VOC: 70.60% [email protected], 51.59% mAP, consistent improvements over both CNN and transformer baselines.

Ablation Studies

Component analysis demonstrates non-redundant gains from each module, with maximum improvements achieved only when all are enabled. Notably, LSConv yields significant geometric sensitivity, while HQ-Retrieval substantially boosts high-IoU accuracy.

Reducing the DETR decoder from 8 to 3 layers with HELP/HPE does not degrade AP when training is sufficient, demonstrating robust query quality without deep refinement. For limited epochs, layers beyond two still improve AP, confirming that HPE does not overly restrict model expressiveness.

Generalization

Evaluation on large-scale aerial datasets (DOTA, DIOR, VisDrone) shows consistent gains (+6.4~+7.6 AP_50) over standard DETR, verifying the adaptability of the HELP framework to varying scales and densities.

Qualitative Visualization

Figure 6: HPE heatbar visualizations—foreground saliency remains robust in cluttered or clean settings, resulting in localized, stable activations and diminished background confusion.

These visualizations illustrate HPE’s interpretable nature: object centers receive sharp, compact positional signals while backgrounds are uniformized, mitigating false-positive query propagation.

Theoretical and Practical Implications

By reframing positional embedding as an explicit spatial allocation problem, this work demonstrates that query noise can be structurally suppressed prior to decoding, in contrast to reliance on post-hoc iterative correction. This not only reduces label leakage from cluttered backgrounds but also aligns backbone and encoder-decoder by enforcing a unified, task-driven encoding mask.

On a practical level, the fewer layers and smaller weights translate into faster, more memory- and energy-efficient deployments, especially critical for remote sensing and real-time embedded systems. The design naturally enforces higher localization precision, with direct impact on high-IoU detection protocols.

Discussion and Future Directions

The methods in this paper provide new perspectives on joint positional-semantic modeling. Extending selective embedding to other detection and segmentation tasks could yield further efficiency and robustness gains, particularly under cross-dataset and domain shift scenarios. Another avenue involves integrating adaptive, prediction-driven spatial allocation to further decouple the embedding map from reliance on gradient-based saliency, increasing plug-and-play flexibility.

Conclusion

The HELP paradigm and HPE mechanism constitute a precise architectural enrichment for small-object transformers, focusing computational resources specifically on detection-relevant regions. This spatial sparsification leads to improved query quality, localization stability, and resource efficiency, establishing a reliable technical path toward scalable, high-fidelity small-object detection in clutter-prone domains (2604.15065).