MDS-DETR: DETR with Masked Duplicate Suppressor

Abstract: The DEtection TRansformer (DETR) is a powerful end-to-end object detector, yet its one-to-one matching strategy suffers from slow convergence and low recall. A common approach to address this issue is to use one-to-many label assignment to provide more positive samples. However, existing methods that use one-to-many matching as an auxiliary objective lead to increased training costs, with their auxiliary decoders discarded during inference. To address this limitation, we propose MDS-DETR, which leverages both one-to-one and one-to-many supervision within a single decoder. Specifically, we introduce a Masked Duplicate Suppressor (MDS) that injects asymmetry into self-attention via confidence-based causal masking. MDS filters out the duplicates generated by the one-to-many supervised layer, enables explainable, duplicate-free predictions in a fully end-to-end framework. MDS-DETR outperforms existing one-to-many DETR variants such as MS-DETR, MR.DETR and Relation-DETR, without relying on any additional queries or auxiliary decoders. Under a 12-epoch training schedule on MS COCO with a ResNet-50 backbone, MDS-DETR achieves a +2.8 mAP improvement over Deformable-DETR with only a 5\% increase in training time, and outperforms the state-of-the-art MR.DETR by +0.3 mAP while being even 20\% faster in training. Our code and models are available at \href{https://github.com/dcholee/mds-detr}{https://github.com/DChoLee/MDS-DETR}.

• The paper introduces an integrated approach that combines one-to-many and one-to-one supervision using a Masked Duplicate Suppressor to eliminate duplicate predictions.
• It employs confidence-based causal masking and symmetric relative position bias to suppress redundant queries without relying on hand-crafted thresholds.
• Empirical results on the COCO2017 benchmark demonstrate improved mAP and AP_S with minimal computational overhead and faster training compared to existing DETR variants.

MDS-DETR: DETR with Masked Duplicate Suppressor

Motivation and Background

Detection Transformers (DETR) have fundamentally transformed the object detection pipeline by leveraging end-to-end architectures with a one-to-one matching paradigm, removing heuristics like anchor box design or post-processing via Non-Maximum Suppression (NMS). However, the constrained number of positive labels in one-to-one matching induces slow convergence and limits recall. One-to-many supervision using auxiliary decoders, as pursued in Hybrid-DETR and its derivatives, has improved training efficiency but at considerable computational expense and architectural complexity. MDS-DETR introduces a paradigm that integrates both one-to-one and one-to-many supervision within a single decoder, eliminating auxiliary branches and addressing inherent symmetry constraints in the traditional DETR framework.

Architecture and Masked Duplicate Suppressor Mechanism

MDS-DETR deploys an architecture in which the decoder is split into two stages: a stack of one-to-many supervised layers produce dense high-quality candidate queries, followed by a single one-to-one supervised layer that leverages the Masked Duplicate Suppressor (MDS) for end-to-end duplicate-free prediction.

Figure 1: Decoder structure of MDS-DETR showing the sequential arrangement of one-to-many supervised layers followed by a one-to-one supervised layer with the Masked Duplicate Suppressor.

The MDS specifically injects asymmetry into the final decoder layer using confidence-based causal masking. Queries are sorted based on their maximum class probability; masking ensures each query can attend only to queries with higher confidence. This formulation reframes duplicate suppression as a parallelizable task akin to NMS, but without relying on hand-crafted thresholds or hyperparameters. To accommodate true positives, MDS introduces explicit tokens serving as attention sinks, providing explainable pathways for maintaining predictions when no valid suppression targets exist.

Figure 2: Visualization of masked self-attention heatmaps for top one-to-many queries and true positive tokens; true positives attend directly to the token while duplicate queries are suppressed.

Position Bias and Suppression Dynamics

Unlike prior works that rely on absolute positional embeddings, MDS-DETR utilizes a symmetric relative position bias, encoding spatial relationships without introducing scale-asymmetry. This bias is integrated into the masked self-attention, penalizing interactions between distant queries and strengthening suppression among spatially proximate candidates.

Figure 3: Headwise relative position bias as a function of query pair IoU, illustrating adaptive suppression strategies among attention heads.

Empirical Evaluation

On the COCO2017 benchmark, MDS-DETR exhibits superior performance relative to state-of-the-art DETR variants, both in terms of box mAP and AP metrics across object scales. When trained for 12 epochs with 300 queries, MDS-DETR achieves +2.8 mAP over Deformable-DETR with only a 5% increase in training time, and outperforms MR.DETR by +0.3 mAP with 20% faster training. Using a Swin-L backbone, MDS-DETR delivers +0.3 mAP and +1.4 AP_S improvement over Relation-DINO, displaying robust small-object detection. These gains are achieved without auxiliary decoders or additional query embeddings.

Qualitative and Computational Analysis

Qualitative results demonstrate that MDS successfully maps one-to-many query outputs to highly accurate, duplicate-free one-to-one predictions, with attention maps that are intuitive and tractable. On instance segmentation tasks, MDS-DETR continues to outperform MR.DETR using identical matching strategies, indicating generalized suppression efficacy.

Figure 4: Qualitative results for instance segmentation on COCO val2017 using a ResNet-50 backbone, showcasing precise segmentation and duplicate suppression.

Analysis of computational resource usage reveals that MDS-DETR incurs minimal overhead compared to baseline Deformable-DETR++, despite outperforming more complex variants that require multiple auxiliary decoders and excessive queries.

Ablation and Architectural Flexibility

Comprehensive ablations confirm the superiority of strict diagonal self-attention masking, as permitting queries to attend to themselves significantly degrades performance. Experiments varying the number of one-to-many and one-to-one supervised layers demonstrate that the MDS block alone suffices for complete duplicate suppression, enabling flexible decoder configurations that outperform conventional multi-stage setups.

Practical and Theoretical Implications

MDS-DETR presents a scalable and explainable solution to duplicate suppression in transformer-based object detectors. By combining efficient one-to-many supervision with confidence-ranked masking, it achieves high recall and precise detection without sacrificing end-to-end inference efficiency or resorting to auxiliary modules. The tractability and explainability of the attention maps open avenues for principled analysis and further extensions, including integration with mixed query denoising pipelines and adaptation to complex dense prediction tasks.

Conclusion

MDS-DETR advances the state of detection transformers by directly encoding duplicate suppression into the decoder via asymmetric masked self-attention, leveraging one-to-many and one-to-one matching within a unified architecture. It achieves substantial improvements in performance, computational efficiency, and interpretability. Future work may focus on structured integration with denoising and query mixture strategies to enhance generalizability and expand applicability in diverse vision domains.