Exemplar Med-DETR: Feature-Based Lesion Detection

• The paper introduces an innovative multi-modal lesion detector that injects class-specific exemplar features into a transformer-based DETR pipeline.
• It employs a three-stage training curriculum with contrastive and feature-consistency losses to achieve significant state-of-the-art mAP improvements across imaging modalities.
• The method refines exemplar generation by integrating positional and textual information, enabling robust feature-matching in challenging clinical settings.

Exemplar Med-DETR (EM-DETR) is a transformer-based lesion detector for medical imaging that introduces class-specific exemplar features into a multi-modal DETR pipeline in order to support feature-based detection rather than reliance on class names alone. The method is presented as a novel multi-modal contrastive detector trained with an iterative strategy and evaluated on mammography, chest X-ray, and angiography. In the reported experiments, it achieves state-of-the-art performance across three distinct imaging modalities from four public datasets, including an mAP of 0.70 for mass detection and 0.55 for calcifications on Vietnamese dense breast mammograms, a more than two-fold increase in true-positive rate on an out-of-distribution Chinese mammography cohort, an mAP of 0.25 for chest X-ray mass detection, and an mAP of 0.37 for coronary stenosis detection (Bhat et al., 25 Jul 2025).

1. Clinical and methodological setting

Detecting abnormalities in medical images poses unique challenges because feature representations vary substantially across modalities, and abnormalities are embedded in an intricate relationship with surrounding anatomical structures. The paper identifies mammography as a particularly difficult case: dense breast tissue can obscure lesions and complicate radiological interpretation. It further argues that existing detection methods, even when they leverage anatomical and semantic context, still struggle to learn effective class-specific features, which limits their applicability across tasks and imaging modalities (Bhat et al., 25 Jul 2025).

EM-DETR is positioned as a response to that limitation. Its central premise is that detection can be improved when the decoder is guided not only by text-derived class embeddings but also by exemplar embeddings derived from lesion-associated image tokens. In the formulation given in the paper, the detector does not discard the standard DETR machinery; instead, it augments a multi-modal DETR framework with exemplar generation, cross-attention over exemplars, and contrastive objectives. This design makes the method simultaneously semantic, via text embeddings, and feature-based, via stored class prototypes.

A common misunderstanding would be to treat EM-DETR as a conventional detector with an auxiliary embedding head. The paper’s formulation is narrower and more specific: exemplar features are injected directly into the decoder query stream, so that the model performs what the paper describes as a feature-matching search for class-specific spatial evidence. That distinction is central to the method’s contribution.

2. Backbone structure and exemplar construction

EM-DETR builds on a multi-modal DETR framework, exemplified in the paper by Grounding DINO, and uses three frozen backbones. The description explicitly includes a Swin Transformer image backbone that extracts multi-scale patch embeddings

$$X' = [x_1, \dots, x_J]^\top,\qquad x_j \in \mathbb{R}^d,$$

together with positional encodings

$P' = [p_1, \dots, p_J]^\top,$

and a text encoder, exemplified by CLIP’s ViT text tower, that produces class-name embeddings $t_k$ for each class $k$ (Bhat et al., 25 Jul 2025).

The exemplar pathway begins from annotated regions. For class $k \in \{1,\dots,N\}$, such as mass, calcification, or background, the model gathers the $M$ tokens whose centers lie inside the ground-truth box:

$$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$

Each positional subset is scaled by a learnable scalar $a_k$ and added to the corresponding visual tokens:

$H_k = X_k + a_k \cdot P_k.$

Two learnable class tokens, a content token $ct_k$ and a positional token $P' = [p_1, \dots, p_J]^\top,$0, both in $P' = [p_1, \dots, p_J]^\top,$1, are then concatenated in front of $P' = [p_1, \dots, p_J]^\top,$2. A self-attention pooling transformer $P' = [p_1, \dots, p_J]^\top,$3 with 2 heads and 4 layers ingests $P' = [p_1, \dots, p_J]^\top,$4 and outputs

$P' = [p_1, \dots, p_J]^\top,$5

To reduce rapid drift, the paper computes a running-average over the last $P' = [p_1, \dots, p_J]^\top,$6 samples of $P' = [p_1, \dots, p_J]^\top,$7, producing the exemplar or prototype embedding $P' = [p_1, \dots, p_J]^\top,$8. A small memory bank stores one $P' = [p_1, \dots, p_J]^\top,$9 per class. At inference time, these stored exemplar features remain fixed and guide the decoder to search for class-specific features. The exemplars are therefore neither generic class tokens nor post hoc summaries; they are class-specific prototype embeddings derived from the annotated image evidence itself.

3. Decoder interaction and contrastive objectives

The decoder uses both textual and exemplar information. In each decoder layer $t_k$0, multi-head queries are formed by linearly projecting the concatenation $t_k$1, while keys and values are obtained from image tokens augmented by positional encodings:

$t_k$2

Attention is then computed with standard scaled dot-product attention:

$t_k$3

By interleaving $t_k$4 with $t_k$5 as decoder queries, the network effectively performs a feature-matching search in which each class exemplar guides attention toward spatial locations whose features resemble the stored prototype (Bhat et al., 25 Jul 2025).

The training objective extends the standard DETR losses for bounding-box regression, IoU, and classification with two additional losses. First, the contrastive-feature loss $t_k$6 is defined over the batchwise matrix of exemplars

$t_k$7

and is described as encouraging mutual orthogonality among class embeddings. The paper gives representative formulations, including

$t_k$8

or an InfoNCE-style objective,

$t_k$9

Second, the feature-consistency loss aligns decoder outputs with stored exemplars. If $k$0 denotes the decoder’s highest-scoring query output for class $k$1, then

$k$2

Together, these losses are intended to make class embeddings distinct in latent space and to make the detector’s proposals resemble the relevant class prototype in that same latent space. The total loss at each stage is

$k$3

with $k$4.

4. Iterative training curriculum

EM-DETR is trained in three stages that progressively sharpen class discrimination (Bhat et al., 25 Jul 2025). In Stage I, the network is trained on all ground-truth annotations. For each batch, image features $k$5 and $k$6 are computed; for each class in the batch, $k$7 is derived by the exemplar-generation procedure; the memory bank is updated through an EMA-style rule; and the DETR losses together with $k$8 and $k$9 are optimized by backpropagation.

Stage II explicitly refines per-class background-versus-foreground discrimination. For each class $k \in \{1,\dots,N\}$0, the method samples $k \in \{1,\dots,N\}$1 background boxes from normal images, matching the average area of the ground-truth regions, and labels these samples as negative class $k \in \{1,\dots,N\}$2. The network is then retrained on the combined set of annotated lesions and sampled background regions.

Stage III performs hard-negative mining. The Stage II model is run on the training set, the top-$k \in \{1,\dots,N\}$3 false-positive boxes are collected, and those false positives are treated as background for the incorrectly assigned class. The model is then fine-tuned for one more epoch.

The reported optimization details are specific. The learning rate is $k \in \{1,\dots,N\}$4 with linear decay; the backbones are frozen, and only $k \in \{1,\dots,N\}$5 and the DETR weights are updated; and the exemplar update uses an EMA window of $k \in \{1,\dots,N\}$6 exemplars. Within the reported design, the curriculum is not an auxiliary refinement but part of the core method, and the ablation results indicate that Stage II and Stage III materially contribute to performance.

5. Evaluation across datasets and modalities

The empirical study covers four public datasets and three imaging modalities (Bhat et al., 25 Jul 2025). VinDR-Mammo consists of 16k training and 4k test mammograms, with mass and calcification detection as the tasks, and approximately 90% dense breasts. CMMD is used as an out-of-distribution Chinese mammography cohort and comprises 100 images, with radiologist-marked lesion centers serving as weak ground truth. VinDR-CXR provides 15k training and 3k test chest X-rays for nodule or mass detection. ARCADE contains 1k training and 300 test angiograms for coronary stenosis detection.

On VinDR-Mammo, the baseline Grounding DINO detector reports mass mAP$k \in \{1,\dots,N\}$7, mass recall $k \in \{1,\dots,N\}$8, calcification mAP$k \in \{1,\dots,N\}$9, and calcification recall $M$0. EM-DETR reports mass mAP$M$1, mass recall $M$2, calcification mAP$M$3, and calcification recall $M$4, with the improvements marked as statistically significant at $M$5. The abstract summarizes these mammography gains as an absolute improvement of 16 percentage points.

On the out-of-distribution CMMD cohort, the true-positive rate increases from $M$6 for the baseline to $M$7 for EM-DETR, which the paper describes as more than two-fold. A board-certified radiologist confirmed the clinical plausibility of the detections.

On VinDR-CXR, the baseline reports mAP$M$8 and recall $M$9 for nodules or masses, whereas EM-DETR reports mAP$$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$0 and recall $$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$1, with $$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$2. On ARCADE, the baseline reports mAP$$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$3 and recall $$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$4 for stenosis detection, whereas EM-DETR reports mAP$$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$5 and recall $$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$6, with $$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$7.

These results are presented as evidence that the same architecture and training recipe can transfer across mammography, chest X-ray, and X-ray angiography without modality-specific redesign.

6. Ablation, robustness, and generalization

The ablation study is reported on VinDR-Mammo and is structured to isolate the contribution of the exemplar module, positional information, auxiliary losses, and the later curriculum stages (Bhat et al., 25 Jul 2025). A t-SNE visualization of exemplar features for mass versus background shows clear cluster separation, supporting the claim that the exemplar embeddings become class-discriminative in latent space.

Variant	Mass mAP$$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$8	Calcification mAP$$X_k \subset X',\qquad P_k \subset P',\qquad |X_k| = M.$$9
Baseline GD	0.48	0.45
+ $a_k$0 pooling only	0.55	0.40
+ positional $a_k$1 in exemplar generation	0.60	0.42
+ losses $a_k$2	0.65	0.43
Stage II refinement	0.68	0.50
Stage III hard negative mining	0.70	0.55

The progression is informative. Adding the pooling transformer alone improves mass detection but decreases calcification detection. Adding positional information and the latent losses recovers and then extends class discrimination. The largest late-stage gains for calcifications occur once Stage II refinement and Stage III hard-negative mining are introduced.

The paper also reports a specific robustness observation: even when calcifications share bounding boxes with masses, EM-DETR recovers finer feature distinctions once Stage II begins. More broadly, the same architecture, hyper-parameters, and three-stage training produce state-of-the-art results on mammography, chest X-ray, and angiography. The paper interprets this as evidence that exemplar-guided feature matching enforces dataset-agnostic class prototypes. This suggests that the method’s main inductive bias lies in class-prototype formation rather than in modality-specific engineering.

7. Limitations and proposed extensions

The paper identifies several limitations (Bhat et al., 25 Jul 2025). First, co-annotation of mass and calcification can force a single exemplar to cover two visually distinct findings. Second, angiography labels may omit unstated stenoses, which can penalize the precision metric. Third, the memory-bank exemplars are static at inference time, so sudden domain shifts may degrade performance until new exemplars are collected.

The future directions follow directly from the exemplar formulation. One proposal is to extend EM-DETR into a true medical foundation detection model by pre-computing exemplars for many lesion types. Another is few-shot or one-shot detection, in which new class exemplars are swapped in on the fly using the same $a_k$3-pooling mechanism without retraining. The paper also proposes exploring joint exemplar and language prompting, exemplified by MedKLIP, for richer clinical instructions.

Taken together, these limitations and extensions delimit the current scope of EM-DETR. The reported system is a lesion detector with fixed inference-time prototypes and a three-stage training curriculum; it is not yet a fully open-ended detector. A plausible implication is that future progress will depend on how stable and transferable the exemplar memory remains when lesion taxonomies expand and domain shifts become more abrupt.