---
title: 'Incremental Object Detection: Advances & Challenges'
url: https://www.emergentmind.com/topics/incremental-object-detection-iod
type: topic
---

# Incremental Object Detection: Advances & Challenges

Incremental object detection (IOD) is a continual learning setting for object detection in which classes arrive sequentially over multiple training phases, and the detector must learn the new classes while retaining the ability to recognize and localize all previously seen classes. A standard formulation splits the overall dataset into phase-specific datasets \(D_t=(x_t,y_t)\) with mutually exclusive class sets \(C_t\), \(C_t \cap C_\tau = \varnothing\) for \(t \neq \tau\), while providing annotations only for the current classes \(C_t\) at phase \(t\). The detector is therefore optimized on partially labeled data but evaluated on all seen classes, which makes catastrophic forgetting, background corruption, and the stability–plasticity tradeoff central design constraints rather than secondary implementation details [2407.21687].

## 1. Formal definition and task variants

In the class-incremental formulation most commonly used in recent work, the class universe is partitioned into disjoint subsets and revealed phase by phase. At phase \(t\), the model has access only to \(\mathcal{D}_t\) or \(D_t\), and annotations cover only \(\mathcal{C}_t\) or \(C_t\). Images may nonetheless contain old objects from \(\mathcal{C}_{1:t-1}\) and future objects from \(\mathcal{C}_{t+1:T}\), so unlabeled foreground is systematically absorbed into background supervision [2407.21687][2304.03110].

This annotation asymmetry has produced several variants of the setting. In the co-occurrence setting, incremental images may contain labeled new-class objects together with unlabeled old-class objects; in the non co-occurrence setting, old-class objects are absent from incremental images, which weakens distillation signals and changes the role of replay or generation [2406.04829]. Semi-supervised incremental object detection (SSIOD) further splits each phase into a small labeled subset and a large unlabeled subset, with old and new class instances coexisting in unlabeled images; this setting was introduced precisely because the assumption of fully annotated new-class data was judged impractical for deployment [2401.05362]. Domain-incremental object detection changes the image distribution rather than the class set, but preserves the same continual-learning concerns about forgetting, transfer, and intransigence [2508.14599].

The literature also distinguishes between protocol design choices. Some protocols allow overlapping images across phases with non-overlapping annotations, whereas revised protocols enforce non-overlapping images except for explicitly stored exemplars [2304.03110]. Later benchmark work argued that repeated-image exposure can create data leakage; LoCo COCO was proposed to assign each image to only one stage using category co-occurrence statistics, eliminating inter-stage leakage and making evaluation more realistic [2512.22973].

## 2. Diagnosing catastrophic forgetting beyond a monolithic view

A central development in IOD research has been the shift from treating the detector as a single forgetting-prone block to identifying architecture-specific failure modes. Several papers argue that catastrophic forgetting is not uniformly distributed across detector components, and that the most effective mitigation depends on where forgetting actually occurs [2503.15295][2502.05540].

For transformer detectors, one influential diagnosis is the “forgetting imbalance” between localization and recognition. DCA reports that localization is class-agnostic and much less forgotten, whereas catastrophic forgetting occurs primarily on recognition: in its analysis, localization recall on old classes drops from \(94\%\) to \(87\%\), while recognition accuracy collapses from \(86\%\) to \(12\%\). This motivates a localization-then-recognition reformulation rather than symmetrical treatment of box regression and classification [2503.15295]. A parallel diagnosis on two-stage detectors reaches a closely related conclusion. In Faster R-CNN, NSGP-RePRE finds that catastrophic forgetting is predominantly localized to the RoI Head classifier, while regressors retain robustness across incremental stages; RPN recall degrades only slightly and contributes little to final detector forgetting [2502.05540].

Other work isolates failure modes created by training dynamics rather than detector modules. ABR identifies foreground shift: replayed old-task images may contain current-task objects that were never annotated, so the same semantic category is treated as foreground in current images and as background in replayed images [2307.12427]. Q-MCMF argues that DETR-like models suffer from background foregrounding, caused by the exhaustive Hungarian matcher forcing every target to match some prediction even when IoU is poor. In a \(70\text{-}10\) setting at the 40th epoch, it reports that **5.7%** of old-class matches and **37.6%** of new-class matches have IoU \(<0.7\), supporting the claim that many assignments are geometrically implausible [2603.01524]. FAS generalizes this line of analysis under the term gradient dilution, decomposed into Signal Dispersion, Assignment Drift, and Support Attrition, corresponding respectively to erosion in gradient magnitude, direction, and feature-space support coverage [2606.15253].

These diagnoses collectively challenge the assumption that generic distillation or generic replay is sufficient. A plausible implication is that IOD has become less a single continual-learning problem than a family of detector-specific optimization problems.

## 3. Distillation- and replay-centered methods in CNN and two-stage detectors

Early and still influential IOD methods concentrated on knowledge distillation from a frozen teacher. “Response-based Distillation for Incremental Object Detection” replaces feature imitation with direct distillation of classification responses and bounding-box regression responses in GFLv1. Its Adaptive Pseudo-label Selection (APS) defines elastic thresholds \(\tau_C=\mu_C+\alpha\sigma_C\) and \(\tau_B=\mu_B+\alpha\sigma_B\), followed by NMS on selected boxes. On MS COCO **40+40**, it reports **36.9 AP**, compared with **34.0** for SID and **40.2** for the upper bound; on last \(40\)+first \(40\), it reports **37.5 AP**, leaving only **2.7 AP** to the upper bound [2110.13471]. ERD reaches a closely related conclusion: selective distillation of classification and regression responses is more effective than feature-level matching, and naïve regression distillation can be harmful. Its ablations report **13.0 AP** for “all reg only,” **23.8 AP** for “all cls only,” and **36.9 AP** for full ERD with Elastic Response Selection [2204.02136].

Replay-based methods in object detection diverged from classification-style rehearsal because replaying full images introduces contradictory supervision. ABR addresses this by storing foreground object boxes rather than whole images, then reinserting them via mixup box replay and mosaic box replay. Its Attentive RoI Distillation aligns spatial attention maps \(A_i^t=\sum_{d=1}^{C}|F_{i,d}^t|^p\) and uses the old model’s attention to weight RoI feature distillation. ABR reports that box replay can store roughly **four times as many object instances** as image replay for the same memory footprint; with **2,000 box instances**, memory use is about **15.5 MB**, compared with **68 MB** in one image-replay comparison [2307.12427]. The method also reports **34.5 mAP@[50:95]** on COCO **40-40** and **31.1 / 52.9 / 32.7** on **70-10**, while on Pascal VOC it shows especially large gains in long sequences, including **19.5 mAP** over MMA on the difficult **5-5** setup [2307.12427].

A different replay line focuses on what is stored rather than only how it is replayed. NSGP-RePRE replays coarse and fine-grained RoI prototypes and constrains backbone and neck updates through Null Space Gradient Projection, so stored prototypes remain aligned with the evolving feature extractor. On COCO it reports **35.4 AP** in **40-40** and **36.5 AP** in **70-10** [2502.05540]. IOR addresses the non co-occurrence case by generating old-class objects through detector inversion rather than a separate generative model. Its inverse-generation objective optimizes \(I_{inv}\) with the original detector’s detection loss plus image-prior and BatchNorm regularizers, then reuses generated objects through augmented replay and high-value distillation. In non co-occurrence **40+40**, IOR reports **30.3 AP**, compared with **27.2** for ERD; the paper also shows that generating **10 objects** already reaches **30.3 AP**, while **100 objects** slightly decreases to **30.0**, which it interprets as evidence of redundancy in over-generation [2406.04829].

## 4. Transformer-based IOD and query-centric reformulations

The shift to DETR-style detectors required methods that respect set prediction, query semantics, and bipartite matching. CL-DETR is a foundational formulation in this line. Its Detector Knowledge Distillation (DKD) first selects old-model foreground predictions \(F\), keeps the top-\(K\) confident ones, filters pseudo-labels whose IoU with current ground-truth boxes exceeds \(\lambda=0.7\), then merges pseudo-labels with current annotations and trains with the standard DETR loss. It complements this with exemplar replay calibrated to preserve the natural category distribution rather than enforcing class balance. On COCO **70+10**, CL-DETR reports **40.1 AP** with Deformable DETR; on **40+40**, it reports **37.5 AP**. In the multi-phase **40+10\times4** setting, its advantage over ERD grows to about **7.4 AP**, indicating that transformer-specific KD and ER become more important as forgetting accumulates [2304.03110].

DyQ-DETR departs from fixed-capacity query pools by explicitly treating object queries as expandable memory units. At phase \(t\), it adds a new learnable query set \(Q_t\) for the current classes and decodes all query groups \(\{Q_1,\dots,Q_t\}\) jointly, with phase-specific outputs \(e_\tau=decoder(f_t,Q_\tau)\). It also introduces disentangled self-attention,
\[
a_{i,j}=
\begin{cases}
\mathrm{Softmax}(Q_{m,i}\cdot Q_{n,j}^T/\sqrt{d}) & m=n,\\
0 & m\neq n,
\end{cases}
\]
and isolated bipartite matching with phase-weighted losses \(w_\tau=\frac{|C_\tau|}{|C_{1:t}|}\). For replay, it uses risk-balanced partial calibration, selecting the “middle 10%” of samples according to the partial loss \(\mathcal{L}_{partial}=\mathcal{L}_t^{DETR}(e_t,y_t)\). On the revised protocol with replay, DyQ-DETR reaches **39.7 AP** on **40+40** and **41.9 AP** on **70+10**; without replay it reports **39.1 AP** and **39.6 AP**, summarized as average improvements of **4.3% AP in non-exemplar scenarios** and **2.9% AP with exemplar replay** over CL-DETR [2407.21687].

Subsequent transformer methods increasingly target the query-assignment mechanism itself. DCA redesigns incremental DETR as a localization-then-recognition pipeline, injects semantic queries from a pre-trained language model such as the CLIP text encoder, and fuses a linear classifier with a semantic classifier through \(\bar p_i=\beta \bar h_i+(1-\beta)\bar s_i\) with \(\beta=0.5\). In long-horizon COCO, it reports a **6.9-point** final AP improvement in the four-step setting and only about **1.6%** added inference cost [2503.15295]. Q-MCMF replaces exhaustive Hungarian assignment with quality-guided min-cost max-flow, using \(\alpha=0.7\) for old targets and \(\beta=0.5\) for new targets. It reports **44.3 AP** on **40-40** and **43.4 AP** on **70-10** under protocol (1), and **42.4 AP** and **43.1 AP** under protocol (2), outperforming DyQ-DETR even when exemplar replay is used [2603.01524].

Another line argues that query correspondence itself must be stabilized across phases. IAQD fixes teacher–student query alignment by index, \(\sigma_i=i\), and distills only proxy queries satisfying
\[
\mathcal{Q}=\left\{n \mid \max_{c \in \mathcal{C}_{1:t-1}} p_{\text{last}}^{(n)}(c)\ge\tau\right\},
\]
with \(\tau=0.1\). It then aligns old-class classification probabilities and full box regression only for those queries. IAQD reports **43.6 AP** on COCO **40+40** and **43.2 AP** on **70+10** under Protocol A, and **44.8 AP** in **40+10\times4** [2508.11339]. FAS extends this perspective by targeting gradient dilution through Prior-Injected Queries, Deterministic Anchor Distillation, and Manifold-Support Replay. In its ablations on COCO **70+10**, performance rises from **32.5 AP** with naïve pseudo labeling to **39.9** with PIQ, **43.0** with DAD, and **45.5** with full FAS, while Forgetting Percentage Point falls from **15.6** to **1.1**. The full system reports **44.8 AP** on **40+40** and **45.5 AP** on **70+10**, and more than **5.0 AP** improvement in the **40+10x4** setting [2606.15253].

A parallel efficiency-oriented branch builds on RT-DETR rather than Deformable DETR. Hier-DETR uses a Hierarchical Neural Collapse tree based on General Orthogonal Frames and applies Proxy-NCA to query features across decoder layers. On COCO it reports **48.4 AP** in **70+10** and **47.1 AP** in **40+40**, explicitly positioning real-time inference as part of the IOD design objective [2506.08562].

## 5. Semantic, prompt-based, and real-time expansions

Vision-language and prompt-based methods extend IOD beyond direct distillation or replay. CASA builds a Class-Agnostic Shared Attribute base inside a frozen OWL-ViT detector. Candidate attributes are generated by GPT-3.5, encoded into an attribute bank \(E_a\), and selected through a sparse assignment matrix \(A^t\) with top-\(C^t\times H_a\) selection, where \(H_a=25\). Previously selected attributes are frozen and reused, while adaptation and refinement losses align attribute embeddings with visual features. CASA reports **42.2 AP**, **61.0 AP\(_{.5}\)**, and **46.6 AP\(_{.75}\)** on **70+10**, with \(FPP \approx -0.01\), and **43.2 AP** on **40+40**. It also adds only about **0.7% parameter storage** [2410.05804].

Prompt-based continual learning for detection has produced mixed findings. A dedicated study of L2P, DualPrompt, and S-Prompt on the D-RICO domain-incremental benchmark reports that prompt-only methods underperform replay baselines. The best prompt-only configuration, DualPrompt with a frozen head, reaches \(\overline{mAP}=23.81\), while **Replay 1% deep prompt** reaches **26.55** and **Replay 10% deep prompt** reaches **31.62**. The same study concludes that deep prompts outperform shallow prompts, and that prompt initialization around \([-10^{-2},10^{-2}]\) is preferable to \([-1,1]\) [2508.14599]. This result has served as a counterweight to claims that prompt tuning transfers directly from classification to detection.

Later prompt-based IOD papers respond by redesigning prompt mechanisms around detection-specific co-occurrence. P\(^2\)IOD replaces a discrete prompt pool with a parameterized prompt generator, then fuses prompt parameters across tasks using importance- and direction-aware rules. It reports **71.1 AP\(_{50}\)** on COCO **40+40** and **71.9 AP\(_{50}\)** on **70+10**, with larger gains in long sequences [2510.27316]. PDP instead uses a dual-pool prompt structure—shared prompts for task-general knowledge and private prompts for task-specific knowledge—together with Prototypical Pseudo-Label Generation. In the COCO multi-step setting it reports final \(mAP@A=59.4\%\) and a **+9.2 AP** improvement over MD-DETR, while in two-phase COCO it reaches **43.8 AP** on **40+40** and **42.9 AP** on **70+10** [2603.02286].

Real-time IOD extends these ideas beyond Faster R-CNN and DETR. YOLO-IOD is built on YOLO-World and diagnoses three conflicts in dense one-stage detection: foreground-background confusion, parameter interference, and misaligned knowledge distillation. Its modules are Conflict-Aware Pseudo-Label Refinement, Importance-based Kernel Selection, and Cross-Stage Asymmetric Knowledge Distillation. It also introduces LoCo COCO, motivated by the observation that standard benchmarks can reuse an image across stages an average of **1.84 stage appearances per image** in the **20+20 4-stage** setting. On conventional COCO-based single-step settings, YOLO-IOD reports **53.0 AP** in **40+40** and **52.4 AP** in **70+10**; in ablations it finds that selecting around **12%** of convolution kernels gives the best incremental-stage tradeoff [2512.22973].

## 6. Benchmarks, specialized regimes, and current directions

MS COCO 2017 remains the dominant benchmark for class-incremental detection, with recurring splits such as **40+40**, **70+10**, **40+20\times2**, and **40+10\times4** [2304.03110][2407.21687]. Pascal VOC 2007 is the other common natural-image benchmark, typically under **19+1**, **15+5**, **10+10**, and multi-step variants [2502.05540]. Standard metrics are \(AP\), \(AP_{50}\), \(AP_{75}\), and scale-specific AP; some transformer papers also report FPP, AbsGap, or RelGap to isolate forgetting [2304.03110][2503.15295].

Beyond these benchmarks, the field has diversified into setting-specific subareas. SSIOD uses labeled and unlabeled data simultaneously; DualTeacher constructs an old teacher and a new teacher and concatenates their predictions,
\[
\hat y_{t,n}^{dt}=f_{old}(x_{t,n}^u)\cup f_{tea}(x_{t,n}^u),
\]
to address coexistence of unlabeled old and new classes. On MS COCO, it reports **17.94 AP** in two-phase **40+40**, **24.34 AP** in **60+20**, and a performance lead of up to **18.28 AP** [2401.05362]. Remote sensing incremental detection introduces severe intra-class scale variation, dense clutter, and missing annotations. STAR-IOD addresses these with Subspace-decoupled Topology Distillation and a Clustering-driven Pseudo-label Generator, and reports gains of **1.7 points** on DIOR-IOD and **2.1 points** on DOTA-IOD, while also introducing the DIOR-IOD and DOTA-IOD benchmarks [2605.20738]. Player detection in sports video defines yet another specialized regime; R\(^2\)D proposes coarse-to-fine region refinement and refined response distillation, introduces NBA-IOD and Volleyball-IOD, and reports improvements over RILOD, SID, and ERD across multi-step splits [2305.00620].

The resulting picture is heterogeneous but coherent. Distillation remains important, but later work increasingly targets where, when, and on which representations distillation should operate. Replay remains important, but later work has largely abandoned the assumption that full-image replay is a neutral operation. Transformer-based IOD has moved toward query expansion, assignment control, semantic anchoring, or gradient-level stabilization, whereas one-stage real-time IOD has emphasized pseudo-label refinement and parameter-efficient adaptation. This suggests that the contemporary understanding of IOD is organized less by a single universal remedy than by architecture-aware analyses of forgetting, annotation incompleteness, and protocol realism [2606.15253][2512.22973][2503.15295].

Source: https://www.emergentmind.com/topics/incremental-object-detection-iod