Masked Duplicate Suppressor in DETR

• MDS is an asymmetric, confidence-aware self-attention module that converts one-to-many query predictions into duplicate-free detections.
• It employs confidence-based causal masking and appended true positive tokens to learn an internal, NMS-like suppression within a unified decoder.
• MDS-DETR achieves improved mAP and faster convergence by integrating one-to-many and one-to-one supervision without extra auxiliary decoders.

Searching arXiv for the target paper and closely related DETR one-to-many variants to ground citations. Masked Duplicate Suppressor (MDS) is an asymmetric, confidence-aware self-attention module introduced in MDS-DETR to convert dense one-to-many query predictions into duplicate-free detections within a single decoder layer. It is designed for DETR-style object detection, where conventional one-to-one Hungarian matching provides end-to-end set prediction but is associated with slow convergence and limited recall because at most one query is matched to each ground-truth object. MDS addresses the complementary requirement—retaining the optimization advantages of one-to-many supervision while preserving duplicate-free end-to-end inference—by imposing confidence-based causal masking, appending true positive tokens, and using a symmetric relative position bias in the final one-to-one layer of the decoder (Lee et al., 22 May 2026).

1. DETR background and the problem MDS targets

Standard DETR and Deformable-DETR use a set prediction formulation with one-to-one Hungarian matching between predicted queries and ground-truth objects. Given object queries

$$Q^{(0)} = \{\mathbf{q}^{(0)}_1, \dots, \mathbf{q}^{(0)}_n\},$$

the decoder produces refined queries $Q^{(L)}$, and each query $\mathbf{q}^{(l)}_i$ predicts class logits $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$ and a box $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$. The one-to-one objective can be written as

$$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$

where $\sigma(\cdot)$ denotes the Hungarian assignment.

This formulation yields duplicate-free predictions by construction, because only one query may match each ground truth. Its limitations are equally direct: few positive queries, slow convergence, and weaker recall. Prior DETR variants therefore introduced one-to-many supervision, allowing each ground-truth object to match multiple queries. The paper situates Hybrid-DETR, MS-DETR, DAC-DETR, Rank-DETR, Relation-DETR, and MR.DETR in this lineage, noting that they generally implement one-to-many matching through auxiliary decoders or auxiliary branches. These approaches improve optimization but increase training cost, add extra queries, and rely on decoder components that are discarded at inference (Lee et al., 22 May 2026).

MDS is formulated as a response to a specific architectural question: whether one-to-many benefits can be obtained inside a single decoder, without auxiliary branches, while still ending with duplicate-free, fully end-to-end predictions.

2. Placement of MDS within MDS-DETR

MDS is not a standalone detector; it is the defining module of the final decoder layer in MDS-DETR. The decoder is reorganized into two parts. Early layers are supervised with one-to-many matching, while the last layer is supervised with one-to-one matching. In the six-layer configuration described in the paper, layers 1–5 are standard Deformable-DETR-style decoder layers with one-to-many supervision, and layer 6 is a special one-to-one layer containing MDS followed by an FFN (Lee et al., 22 May 2026).

A central design decision is that the final one-to-one layer removes cross-attention. Its role is therefore not additional image-conditioned box refinement, but selection among the candidate queries produced by the one-to-many layers. The early layers act as dense candidate generators; the last layer uses MDS to identify true positives and suppress duplicates. The paper explicitly characterizes this as an internal, learned analog of NMS.

Relative to Deformable-DETR++, the architecture retains the same number of queries as the baseline—300 or 900 depending on the setting—and introduces no auxiliary decoders, no hybrid branches, and no extra decoder weights. The early decoder layers use the reversed order of cross-attention and self-attention following MS-DETR, which the paper reports as giving $+0.1$ mAP. The final layer contains only masked self-attention and an FFN, with no cross-attention (Lee et al., 22 May 2026).

This organization is significant because it locates duplicate suppression inside the main prediction path rather than in a discarded auxiliary branch or in post-hoc NMS. The same decoder, including MDS, is used during both training and inference, and no NMS is applied at inference.

3. Attention asymmetry, confidence-based causal masking, and true positive tokens

The core technical claim of MDS is that duplicate suppression requires asymmetry. Standard self-attention in DETR is permutation-equivariant; if two queries are identical, symmetric self-attention produces identical outputs, so the duplicates remain duplicates. MDS injects asymmetry by sorting queries according to confidence and enforcing a causal direction in confidence rank (Lee et al., 22 May 2026).

Let the inputs to the final layer be queries $Q^{(l-1)}$, class scores $S^{(l-1)}$, and boxes $Q^{(L)}$0. Query confidence is defined as

$Q^{(L)}$1

Queries are sorted in descending order of $Q^{(L)}$2. On this sorted sequence, MDS applies the causal mask

$Q^{(L)}$3

Thus a lower-confidence query can attend only to higher-confidence queries. The diagonal is also masked, so self-attention to the same query is disallowed. The paper’s ablation reports that allowing the diagonal to remain unmasked drops mAP by $Q^{(L)}$4, indicating that strict masking is critical (Lee et al., 22 May 2026).

The intended behavior is explicit. If several one-to-many queries correspond to the same object, the highest-confidence one is likely to be the true positive, while the others are duplicates. Since lower-confidence duplicates can attend upward in confidence rank, they are updated toward suppression. Conversely, a higher-confidence query is protected from being influenced by lower-confidence duplicates.

A complication arises for a genuine true positive that has no same-context higher-confidence neighbor. Its causal attention window would otherwise contain only unrelated queries. To address this, MDS appends $Q^{(L)}$5 learned true positive tokens to the key/value set. These are always visible to all queries and function as attention sinks. The paper reports that true positive queries tend to attend to these tokens, while lower-confidence similar boxes attend to higher-confidence queries and are suppressed. The default choice is $Q^{(L)}$6; an ablation shows that $Q^{(L)}$7 or $Q^{(L)}$8 also work well, whereas $Q^{(L)}$9 token underperforms due to insufficient expressivity (Lee et al., 22 May 2026).

The paper also incorporates geometry through a relative position bias. The relative box encoding is

$\mathbf{q}^{(l)}_i$0

Because this encoding is non-commutative, the authors propose the symmetric bias

$\mathbf{q}^{(l)}_i$1

which ensures $\mathbf{q}^{(l)}_i$2. The paper states that the asymmetric scale terms are harmful, especially for small objects, and that the symmetric formulation improves suppression behavior. It also notes that $\mathbf{q}^{(l)}_i$3 is not clamped with ReLU, so it may be negative and can therefore down-weight distant pairs (Lee et al., 22 May 2026).

4. Supervision, matching, losses, and implementation details

The one-to-many layers in MDS-DETR use the DETA-style matcher. For a prediction $\mathbf{q}^{(l)}_i$4 and ground truth $\mathbf{q}^{(l)}_i$5, the matching score is

$\mathbf{q}^{(l)}_i$6

For each ground truth, the top-$\mathbf{q}^{(l)}_i$7 queries are selected by $\mathbf{q}^{(l)}_i$8, and predictions below threshold $\mathbf{q}^{(l)}_i$9 are discarded by swipe-out. The reported hyperparameters are $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$0, $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$1, and $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$2, following DETA, MS-DETR, and MR.DETR configurations (Lee et al., 22 May 2026).

Losses consist of Varifocal loss for classification, with $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$3 and $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$4, together with $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$5 and $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$6 for box regression. The one-to-one supervision applied to the final MDS layer and two-stage encoder proposals uses the ratio

$\mathbf{s}^{(l)}_i \in \mathbb{R}^C$7

whereas one-to-many supervision on earlier layers uses

$\mathbf{s}^{(l)}_i \in \mathbb{R}^C$8

The classification term is therefore intentionally down-weighted in the one-to-many layers so that training remains centered on one-to-one predictions while the earlier layers primarily provide box coverage (Lee et al., 22 May 2026).

The implementation base is Deformable-DETR++ in its two-stage variant. Main experiments use an ImageNet-pretrained ResNet-50 backbone, with additional experiments on Swin-L. The training setup uses COCO 2017, AdamW, learning rate $\mathbf{s}^{(l)}_i \in \mathbb{R}^C$9, weight decay $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$0, and batch size 16 on $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$1 RTX 4090. In the 12-epoch schedule, the learning rate is decayed by $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$2 at epoch 11; in the 24-epoch schedule, it is decayed at epoch 20 (Lee et al., 22 May 2026).

The paper also reports several implementation ablations. Relative position bias alone yields $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$3 mAP, outperforming absolute positional embeddings alone ($\mathbf{b}^{(l)}_i \in \mathbb{R}^4$4) and the combination of both ($\mathbf{b}^{(l)}_i \in \mathbb{R}^4$5). Removing dropout improves performance from $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$6 to $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$7 mAP. Look Forward Twice has negligible effect. Mixed Query Selection, which often helps DETR, reduces performance to $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$8 mAP; the authors hypothesize that its static rank-like embeddings conflict with MDS’s dynamic confidence-based ranking (Lee et al., 22 May 2026).

5. Empirical behavior, efficiency, and interpretability

Under a 12-epoch training schedule on MS COCO with a ResNet-50 backbone, MDS-DETR achieves a $\mathbf{b}^{(l)}_i \in \mathbb{R}^4$9 mAP improvement over Deformable-DETR with only a $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$0 increase in training time, and outperforms MR.DETR by $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$1 mAP while being $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$2 faster in training (Lee et al., 22 May 2026). In the paper’s ResNet-50, 300-query comparison, Deformable-DETR++ attains $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$3 mAP, Hybrid-DETR $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$4, MS-DETR $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$5, MR.DETR $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$6, and MDS-DETR $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$7. In the 900-query, 12-epoch setting, MDS-DETR reaches $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$8 mAP, exceeding MR.DETR at $$\mathcal{L}_{\mathrm{one2one}} = \sum_{i=1}^{n} \left[ \mathcal{L}_{\mathrm{cls}}(\mathbf{s}_i,\bar{\mathbf{s}}_{\sigma(i)}) + \mathcal{L}_{\mathrm{bbox}}(\mathbf{b}_i,\bar{\mathbf{b}}_{\sigma(i)}) \right],$$9, DINO at $\sigma(\cdot)$0, and MS-DETR at $\sigma(\cdot)$1, while Hybrid-MDS-DETR reaches $\sigma(\cdot)$2. In the 24-epoch, 900-query setting, MDS-DETR achieves $\sigma(\cdot)$3 mAP, outperforming MR.DETR ($\sigma(\cdot)$4) and Relation-DINO ($\sigma(\cdot)$5) (Lee et al., 22 May 2026).

The reported training-cost comparison with ResNet-50 and 300 queries shows $\sigma(\cdot)$6 MB memory and $\sigma(\cdot)$7 min/epoch for Deformable-DETR++, versus $\sigma(\cdot)$8 MB and $\sigma(\cdot)$9 min/epoch for MDS-DETR. Hybrid-DETR uses $+0.1$0 MB and $+0.1$1 min/epoch; MS-DETR uses $+0.1$2 MB and $+0.1$3 min/epoch; MR.DETR uses $+0.1$4 MB and $+0.1$5 min/epoch. The paper interprets the additional cost of MDS-DETR as arising from MDS computations and extra supervision rather than from extra decoder stacks (Lee et al., 22 May 2026).

Qualitative analysis is presented as an argument for interpretability. Attention maps for the top 20 one-to-many queries plus 10 true positive tokens show that true positive queries attend predominantly to true positive tokens, whereas lower-confidence queries with similar boxes focus on higher-confidence queries and are suppressed. A further plot of learned relative position bias against IoU indicates that the bias increases with IoU, so overlapping boxes receive stronger attention; different heads learn different slopes and patterns, which the paper describes as different suppression strategies (Lee et al., 22 May 2026).

Layer-wise NMS ablations reinforce the same point. Raw outputs from one-to-many layers 1–5 have low mAP without NMS, approximately $+0.1$6–$+0.1$7, but rise to approximately $+0.1$8–$+0.1$9 with NMS. The final one-to-one layer with MDS reaches $Q^{(l-1)}$0 without NMS, and NMS slightly reduces this to $Q^{(l-1)}$1. This indicates that the last layer is already duplicate-free and that external NMS is unnecessary for final predictions (Lee et al., 22 May 2026).

6. Relation to prior DETR variants, common misconceptions, and stated limitations

MDS is closely related to the one-to-many DETR literature, but its mechanism differs from prior variants in where duplicate suppression occurs. MS-DETR uses one-to-many supervision throughout the decoder without a dedicated duplicate suppressor. MR.DETR uses one main decoder and two auxiliary decoders, together with an instruction token controlling one-to-many versus one-to-one behavior. Relation-DETR and Relation-DINO encode pairwise IoU and relative rank into attention weights but still rely on hybrid branches and auxiliary decoders. By contrast, MDS-DETR keeps one decoder and places suppression directly inside the final one-to-one layer (Lee et al., 22 May 2026).

A common misconception is to treat MDS as equivalent to post-hoc NMS. The paper argues against that equivalence on three grounds. First, MDS is integrated into self-attention and used during both training and inference. Second, its behavior is learned through one-to-one supervision rather than specified by a fixed IoU threshold. Third, the suppression process is inspectable through confidence ordering, causal masking, and relative geometry. This suggests that MDS is better understood as a learned, differentiable, NMS-like operation rather than as a direct replacement of heuristic NMS by the same rule set.

The paper also identifies several limitations. Integrating MDS with DINO-like techniques is non-trivial: Mixed Query Selection degrades performance, and query denoising as used in DN-DETR or DINO is not directly incorporated. Stacking multiple MDS-based one-to-one layers does not help; with $Q^{(l-1)}$2 one-to-many/one-to-one layers the model reaches $Q^{(l-1)}$3 mAP, whereas $Q^{(l-1)}$4 and $Q^{(l-1)}$5 both yield $Q^{(l-1)}$6, which the paper attributes to over-suppression. MDS also assumes that one-to-many layers produce reasonably calibrated confidences, since the causal mask depends on ranking by confidence. Finally, the paper notes that asymmetric relative geometry is harmful for small objects and proposes the symmetric bias partly to mitigate this issue (Lee et al., 22 May 2026).

Beyond object detection, the paper reports an instance segmentation extension obtained by adding a mask head. In that setting, MDS-DETR improves both box and mask mAP over MR.DETR with no extra decoders. The authors further suggest that applying MDS to other transformer-based tasks, including universal segmentation and tracking, could be promising.