---
title: 'Anchor DETR: Interpretable Object Detection'
url: https://www.emergentmind.com/topics/anchor-detr
type: topic
---

# Anchor DETR: Interpretable Object Detection

Anchor DETR is a transformer-based object detection framework that introduces an interpretable, anchor-point-driven query formulation for the DETR family of models. By replacing the opaque, learned object queries of the original DETR with explicit anchor point queries—paralleling strategies from CNN-based detectors—Anchor DETR achieves enhanced optimization, faster convergence, and competitive detection accuracy with reduced computational overhead. This approach establishes a bridge between anchor-based detection paradigms and end-to-end transformer detectors, and has influenced subsequent generations of DETR-like models.

## 1. Anchor-Based Query Design in DETR

Anchor DETR reformulates the object query design in the transformer decoder to embed explicit spatial priors, which allows each query to focus on a physically-defined region in the image. Each anchor point is a normalized two-dimensional coordinate:
\[
\mathrm{Pos}_q \in [0,1]^{N_A \times 2},
\]
where $N_A$ denotes the number of anchor points. Each anchor point is encoded to a query-position embedding via a two-layer MLP applied to the standard sine–cosine encoding:
\[
Q_p = \mathsf{Encode}(\mathrm{Pos}_q) \in \mathbb{R}^{N_A \times C},
\]
where $C$ is the model dimension. To support detection of multiple objects at a single location ("one region, multiple objects"), $N_p$ learnable pattern embeddings are introduced and tiled across all anchors:
\[
N_q = N_A \times N_p,\qquad Q_f^{\mathrm{init}}=[P;\ldots;P]\in \mathbb{R}^{N_q \times C}.
\]
Initial decoder queries are the sum $Q_{\mathrm{init}} = Q_f^{\mathrm{init}} + Q_p \in \mathbb{R}^{N_q \times C}$ [2109.07107].

This formulation ensures that each object query has an explicit spatial focus, improving optimization by imparting a clearer division of labor to the prediction slots and more direct interpretability.

## 2. Row-Column Decoupled Attention (RCDA)

Anchor DETR introduces Row-Column Decoupled Attention (RCDA) as an efficient variant of standard multi-head attention to address the memory bottleneck in high-resolution feature processing. RCDA factorizes attention computation into separate row and column interactions:
- Given feature $K_f\in\mathbb{R}^{H \times W \times C}$, average along rows and columns to get $K_{f,x}$ and $K_{f,y}$.
- Row attention: $A_x = \mathsf{softmax}((Q_x K_x^T)/\sqrt{d_k})\in \mathbb{R}^{N_q \times W}$.
- Intermediate result: $Z_{n,i,c} = \sum_{w=1}^W A_x(n,w)V_f(n,i,w,c)$.
- Column attention: $A_y = \mathsf{softmax}((Q_y K_y^T)/\sqrt{d_k})\in \mathbb{R}^{N_q \times H}$, final output is summed accordingly.

RCDA reduces the memory complexity from $O(N_q\,H\,W\,M)$ (standard) to $O(N_q\,(H\,C + H\,M + W\,M))$, substantially saving memory especially at high feature map resolutions [2109.07107].

## 3. Loss Functions and Training Regime

Anchor DETR adopts the same set-based Hungarian matching paradigm as the original DETR for bipartite assignment between predictions and ground-truth targets. The cost and total loss combine focal classification loss, $\ell_1$ box regression loss, and GIoU loss:
\[
\mathcal{C}(i,j) = -\log p_i(c_j) + \lambda_1 \|b_i - b_j\|_1 + \lambda_2 (1-\mathrm{GIoU}(b_i,b_j)),
\]
\[
\mathcal{L} = \sum_{i=1}^{N_q}\Bigl[\lambda_{\mathrm{cls}}\,\mathrm{Focal}(p_i,\,c_i^*) + \lambda_{\mathrm{box}}\|b_i - b_i^*\|_1 + \lambda_{\mathrm{giou}}(1-\mathrm{GIoU}(b_i,b_i^*))\Bigr].
\]
Typical weights are $\lambda_{\mathrm{cls}}=2.0$, $\lambda_{\mathrm{box}}=5.0$, $\lambda_{\mathrm{giou}}=2.0$ [2109.07107].

Training is performed for 50 epochs on COCO with AdamW and ResNet-50-DC5 as backbone, using standard data augmentations and learning rate scheduling.

## 4. Experimental Performance and Ablation Studies

On the COCO dataset, Anchor DETR attains strong metrics while training with 10× fewer epochs compared to DETR. For single-scale setups with ResNet-50-DC5:

| Method                     | AP    | AP$_{50}$ | AP$_{75}$ | FPS  |
|----------------------------|-------|-----------|-----------|------|
| DETR (500 ep)              | 43.3  | 63.1      | 45.9      | 12   |
| Deformable DETR (50 ep)    | 43.8  | 62.6      | 47.7      | 15   |
| SMCA (50 ep)               | 43.7  | 63.6      | 47.2      | 10   |
| Anchor DETR (50 ep)        | 44.2  | 64.7      | 47.5      | 19   |

Ablation reveals:
- Anchor-based queries alone contribute $+4.9$ AP;
- RCDA adds $+1.0$ AP;
- Using both yields the highest AP and best speed.
Grid vs. learned anchor initialization yields nearly identical AP (44.1 vs. 44.2) [2109.07107].

## 5. Comparative Analysis and Influence on Successors

Anchor DETR marks a pivotal shift by rendering the queries interpretable and physically grounded. Its anchor point formulation and RCDA were adopted and extended in subsequent work:
- DAB-DETR generalizes to dynamic box queries, updating anchor box parameters layer by layer while interpreting their cross-attention as soft ROI pooling [2201.12329].
- Conditional DETR V2 formalizes "box queries" as embeddings of a reference point and learned box transform, explicitly drawing a parallel to anchor box refinement in Faster R-CNN [2207.08914].
- Box-DETR further replaces box centers in conditional queries with head-specific agent points encompassing full box information for each cross-attention head, accelerating convergence and improving AP [2307.08353].

These descendant models empirically show further improvements in both training efficiency and detection AP, substantiating the value of the anchor-inspired query mechanism introduced by Anchor DETR.

## 6. Practical Impact and Implementation Considerations

Anchor DETR achieves end-to-end, anchor-free inference; no non-maximum suppression or region assignment modules are required at test time. The inference pipeline remains as in DETR: all predictions are directly interpretable as detection candidates, assigned via bipartite matching, and scored by a unified loss. Implementation requires only minimal deviation from standard DETR, with public code and trained models available at https://github.com/megvii-research/AnchorDETR [2109.07107].

Anchor DETR demonstrates that explicit anchor-point queries can reconcile the optimization advantages of second-stage CNN detectors with the global, permutation-invariant processing of transformers, resulting in faster training, interpretable queries, and state-of-the-art transformer detection performance with lower computational and memory costs.

Source: https://www.emergentmind.com/topics/anchor-detr