Incremental Object Detection: Advances & Challenges

• Incremental object detection is a continual learning setting where new object classes are introduced phase by phase while previous classes must be retained.
• It poses challenges like catastrophic forgetting, background corruption, and managing the stability–plasticity trade-off across detector components.
• Recent advances leverage knowledge distillation, replay strategies, and transformer adaptations to effectively balance learning across incremental phases.

Incremental object detection (IOD) is a continual learning setting for object detection in which classes arrive sequentially over multiple training phases, and the detector must learn the new classes while retaining the ability to recognize and localize all previously seen classes. A standard formulation splits the overall dataset into phase-specific datasets $D_t=(x_t,y_t)$ with mutually exclusive class sets $C_t$, $C_t \cap C_\tau = \varnothing$ for $t \neq \tau$, while providing annotations only for the current classes $C_t$ at phase $t$. The detector is therefore optimized on partially labeled data but evaluated on all seen classes, which makes catastrophic forgetting, background corruption, and the stability–plasticity tradeoff central design constraints rather than secondary implementation details (Zhang et al., 2024).

1. Formal definition and task variants

In the class-incremental formulation most commonly used in recent work, the class universe is partitioned into disjoint subsets and revealed phase by phase. At phase $t$, the model has access only to $\mathcal{D}_t$ or $D_t$, and annotations cover only $\mathcal{C}_t$ or $C_t$0. Images may nonetheless contain old objects from $C_t$1 and future objects from $C_t$2, so unlabeled foreground is systematically absorbed into background supervision (Zhang et al., 2024, Liu et al., 2023).

This annotation asymmetry has produced several variants of the setting. In the co-occurrence setting, incremental images may contain labeled new-class objects together with unlabeled old-class objects; in the non co-occurrence setting, old-class objects are absent from incremental images, which weakens distillation signals and changes the role of replay or generation (An et al., 2024). Semi-supervised incremental object detection (SSIOD) further splits each phase into a small labeled subset and a large unlabeled subset, with old and new class instances coexisting in unlabeled images; this setting was introduced precisely because the assumption of fully annotated new-class data was judged impractical for deployment (Yuan et al., 2023). Domain-incremental object detection changes the image distribution rather than the class set, but preserves the same continual-learning concerns about forgetting, transfer, and intransigence (Neuwirth-Trapp et al., 20 Aug 2025).

The literature also distinguishes between protocol design choices. Some protocols allow overlapping images across phases with non-overlapping annotations, whereas revised protocols enforce non-overlapping images except for explicitly stored exemplars (Liu et al., 2023). Later benchmark work argued that repeated-image exposure can create data leakage; LoCo COCO was proposed to assign each image to only one stage using category co-occurrence statistics, eliminating inter-stage leakage and making evaluation more realistic (Zhang et al., 28 Dec 2025).

2. Diagnosing catastrophic forgetting beyond a monolithic view

A central development in IOD research has been the shift from treating the detector as a single forgetting-prone block to identifying architecture-specific failure modes. Several papers argue that catastrophic forgetting is not uniformly distributed across detector components, and that the most effective mitigation depends on where forgetting actually occurs (Zhang et al., 19 Mar 2025, Wu et al., 8 Feb 2025).

For transformer detectors, one influential diagnosis is the “forgetting imbalance” between localization and recognition. DCA reports that localization is class-agnostic and much less forgotten, whereas catastrophic forgetting occurs primarily on recognition: in its analysis, localization recall on old classes drops from $C_t$3 to $C_t$4, while recognition accuracy collapses from $C_t$5 to $C_t$6. This motivates a localization-then-recognition reformulation rather than symmetrical treatment of box regression and classification (Zhang et al., 19 Mar 2025). A parallel diagnosis on two-stage detectors reaches a closely related conclusion. In Faster R-CNN, NSGP-RePRE finds that catastrophic forgetting is predominantly localized to the RoI Head classifier, while regressors retain robustness across incremental stages; RPN recall degrades only slightly and contributes little to final detector forgetting (Wu et al., 8 Feb 2025).

Other work isolates failure modes created by training dynamics rather than detector modules. ABR identifies foreground shift: replayed old-task images may contain current-task objects that were never annotated, so the same semantic category is treated as foreground in current images and as background in replayed images (Yuyang et al., 2023). Q-MCMF argues that DETR-like models suffer from background foregrounding, caused by the exhaustive Hungarian matcher forcing every target to match some prediction even when IoU is poor. In a $C_t$7 setting at the 40th epoch, it reports that 5.7% of old-class matches and 37.6% of new-class matches have IoU $C_t$8, supporting the claim that many assignments are geometrically implausible (Wu et al., 2 Mar 2026). FAS generalizes this line of analysis under the term gradient dilution, decomposed into Signal Dispersion, Assignment Drift, and Support Attrition, corresponding respectively to erosion in gradient magnitude, direction, and feature-space support coverage (Zhang et al., 13 Jun 2026).

These diagnoses collectively challenge the assumption that generic distillation or generic replay is sufficient. A plausible implication is that IOD has become less a single continual-learning problem than a family of detector-specific optimization problems.

3. Distillation- and replay-centered methods in CNN and two-stage detectors

Early and still influential IOD methods concentrated on knowledge distillation from a frozen teacher. “Response-based Distillation for Incremental Object Detection” replaces feature imitation with direct distillation of classification responses and bounding-box regression responses in GFLv1. Its Adaptive Pseudo-label Selection (APS) defines elastic thresholds $C_t$9 and $C_t \cap C_\tau = \varnothing$0, followed by NMS on selected boxes. On MS COCO 40+40, it reports 36.9 AP, compared with 34.0 for SID and 40.2 for the upper bound; on last $C_t \cap C_\tau = \varnothing$1+first $C_t \cap C_\tau = \varnothing$2, it reports 37.5 AP, leaving only 2.7 AP to the upper bound (Feng et al., 2021). ERD reaches a closely related conclusion: selective distillation of classification and regression responses is more effective than feature-level matching, and naïve regression distillation can be harmful. Its ablations report 13.0 AP for “all reg only,” 23.8 AP for “all cls only,” and 36.9 AP for full ERD with Elastic Response Selection (Feng et al., 2022).

Replay-based methods in object detection diverged from classification-style rehearsal because replaying full images introduces contradictory supervision. ABR addresses this by storing foreground object boxes rather than whole images, then reinserting them via mixup box replay and mosaic box replay. Its Attentive RoI Distillation aligns spatial attention maps $C_t \cap C_\tau = \varnothing$3 and uses the old model’s attention to weight RoI feature distillation. ABR reports that box replay can store roughly four times as many object instances as image replay for the same memory footprint; with 2,000 box instances, memory use is about 15.5 MB, compared with 68 MB in one image-replay comparison (Yuyang et al., 2023). The method also reports 34.5 mAP@[50:95] on COCO 40-40 and 31.1 / 52.9 / 32.7 on 70-10, while on Pascal VOC it shows especially large gains in long sequences, including 19.5 mAP over MMA on the difficult 5-5 setup (Yuyang et al., 2023).

A different replay line focuses on what is stored rather than only how it is replayed. NSGP-RePRE replays coarse and fine-grained RoI prototypes and constrains backbone and neck updates through Null Space Gradient Projection, so stored prototypes remain aligned with the evolving feature extractor. On COCO it reports 35.4 AP in 40-40 and 36.5 AP in 70-10 (Wu et al., 8 Feb 2025). IOR addresses the non co-occurrence case by generating old-class objects through detector inversion rather than a separate generative model. Its inverse-generation objective optimizes $C_t \cap C_\tau = \varnothing$4 with the original detector’s detection loss plus image-prior and BatchNorm regularizers, then reuses generated objects through augmented replay and high-value distillation. In non co-occurrence 40+40, IOR reports 30.3 AP, compared with 27.2 for ERD; the paper also shows that generating 10 objects already reaches 30.3 AP, while 100 objects slightly decreases to 30.0, which it interprets as evidence of redundancy in over-generation (An et al., 2024).

4. Transformer-based IOD and query-centric reformulations

The shift to DETR-style detectors required methods that respect set prediction, query semantics, and bipartite matching. CL-DETR is a foundational formulation in this line. Its Detector Knowledge Distillation (DKD) first selects old-model foreground predictions $C_t \cap C_\tau = \varnothing$5, keeps the top-$C_t \cap C_\tau = \varnothing$6 confident ones, filters pseudo-labels whose IoU with current ground-truth boxes exceeds $C_t \cap C_\tau = \varnothing$7, then merges pseudo-labels with current annotations and trains with the standard DETR loss. It complements this with exemplar replay calibrated to preserve the natural category distribution rather than enforcing class balance. On COCO 70+10, CL-DETR reports 40.1 AP with Deformable DETR; on 40+40, it reports 37.5 AP. In the multi-phase 40+10\times4 setting, its advantage over ERD grows to about 7.4 AP, indicating that transformer-specific KD and ER become more important as forgetting accumulates (Liu et al., 2023).

DyQ-DETR departs from fixed-capacity query pools by explicitly treating object queries as expandable memory units. At phase $C_t \cap C_\tau = \varnothing$8, it adds a new learnable query set $C_t \cap C_\tau = \varnothing$9 for the current classes and decodes all query groups $t \neq \tau$0 jointly, with phase-specific outputs $t \neq \tau$1. It also introduces disentangled self-attention,

$t \neq \tau$2

and isolated bipartite matching with phase-weighted losses $t \neq \tau$3. For replay, it uses risk-balanced partial calibration, selecting the “middle 10%” of samples according to the partial loss $t \neq \tau$4. On the revised protocol with replay, DyQ-DETR reaches 39.7 AP on 40+40 and 41.9 AP on 70+10; without replay it reports 39.1 AP and 39.6 AP, summarized as average improvements of 4.3% AP in non-exemplar scenarios and 2.9% AP with exemplar replay over CL-DETR (Zhang et al., 2024).

Subsequent transformer methods increasingly target the query-assignment mechanism itself. DCA redesigns incremental DETR as a localization-then-recognition pipeline, injects semantic queries from a pre-trained LLM such as the CLIP text encoder, and fuses a linear classifier with a semantic classifier through $t \neq \tau$5 with $t \neq \tau$6. In long-horizon COCO, it reports a 6.9-point final AP improvement in the four-step setting and only about 1.6% added inference cost (Zhang et al., 19 Mar 2025). Q-MCMF replaces exhaustive Hungarian assignment with quality-guided min-cost max-flow, using $t \neq \tau$7 for old targets and $t \neq \tau$8 for new targets. It reports 44.3 AP on 40-40 and 43.4 AP on 70-10 under protocol (1), and 42.4 AP and 43.1 AP under protocol (2), outperforming DyQ-DETR even when exemplar replay is used (Wu et al., 2 Mar 2026).

Another line argues that query correspondence itself must be stabilized across phases. IAQD fixes teacher–student query alignment by index, $t \neq \tau$9, and distills only proxy queries satisfying

$C_t$0

with $C_t$1. It then aligns old-class classification probabilities and full box regression only for those queries. IAQD reports 43.6 AP on COCO 40+40 and 43.2 AP on 70+10 under Protocol A, and 44.8 AP in 40+10\times4 (MA et al., 15 Aug 2025). FAS extends this perspective by targeting gradient dilution through Prior-Injected Queries, Deterministic Anchor Distillation, and Manifold-Support Replay. In its ablations on COCO 70+10, performance rises from 32.5 AP with naïve pseudo labeling to 39.9 with PIQ, 43.0 with DAD, and 45.5 with full FAS, while Forgetting Percentage Point falls from 15.6 to 1.1. The full system reports 44.8 AP on 40+40 and 45.5 AP on 70+10, and more than 5.0 AP improvement in the 40+10x4 setting (Zhang et al., 13 Jun 2026).

A parallel efficiency-oriented branch builds on RT-DETR rather than Deformable DETR. Hier-DETR uses a Hierarchical Neural Collapse tree based on General Orthogonal Frames and applies Proxy-NCA to query features across decoder layers. On COCO it reports 48.4 AP in 70+10 and 47.1 AP in 40+40, explicitly positioning real-time inference as part of the IOD design objective (Pham et al., 10 Jun 2025).

5. Semantic, prompt-based, and real-time expansions

Vision-language and prompt-based methods extend IOD beyond direct distillation or replay. CASA builds a Class-Agnostic Shared Attribute base inside a frozen OWL-ViT detector. Candidate attributes are generated by GPT-3.5, encoded into an attribute bank $C_t$2, and selected through a sparse assignment matrix $C_t$3 with top-$C_t$4 selection, where $C_t$5. Previously selected attributes are frozen and reused, while adaptation and refinement losses align attribute embeddings with visual features. CASA reports 42.2 AP, 61.0 AP$C_t$6, and 46.6 AP$C_t$7 on 70+10, with $C_t$8, and 43.2 AP on 40+40. It also adds only about 0.7% parameter storage (Guo et al., 2024).

Prompt-based continual learning for detection has produced mixed findings. A dedicated study of L2P, DualPrompt, and S-Prompt on the D-RICO domain-incremental benchmark reports that prompt-only methods underperform replay baselines. The best prompt-only configuration, DualPrompt with a frozen head, reaches $C_t$9, while Replay 1% deep prompt reaches 26.55 and Replay 10% deep prompt reaches 31.62. The same study concludes that deep prompts outperform shallow prompts, and that prompt initialization around $t$0 is preferable to $t$1 (Neuwirth-Trapp et al., 20 Aug 2025). This result has served as a counterweight to claims that prompt tuning transfers directly from classification to detection.

Later prompt-based IOD papers respond by redesigning prompt mechanisms around detection-specific co-occurrence. P$t$2IOD replaces a discrete prompt pool with a parameterized prompt generator, then fuses prompt parameters across tasks using importance- and direction-aware rules. It reports 71.1 AP$t$3 on COCO 40+40 and 71.9 AP$t$4 on 70+10, with larger gains in long sequences (An et al., 31 Oct 2025). PDP instead uses a dual-pool prompt structure—shared prompts for task-general knowledge and private prompts for task-specific knowledge—together with Prototypical Pseudo-Label Generation. In the COCO multi-step setting it reports final $t$5 and a +9.2 AP improvement over MD-DETR, while in two-phase COCO it reaches 43.8 AP on 40+40 and 42.9 AP on 70+10 (Zhang et al., 2 Mar 2026).

Real-time IOD extends these ideas beyond Faster R-CNN and DETR. YOLO-IOD is built on YOLO-World and diagnoses three conflicts in dense one-stage detection: foreground-background confusion, parameter interference, and misaligned knowledge distillation. Its modules are Conflict-Aware Pseudo-Label Refinement, Importance-based Kernel Selection, and Cross-Stage Asymmetric Knowledge Distillation. It also introduces LoCo COCO, motivated by the observation that standard benchmarks can reuse an image across stages an average of 1.84 stage appearances per image in the 20+20 4-stage setting. On conventional COCO-based single-step settings, YOLO-IOD reports 53.0 AP in 40+40 and 52.4 AP in 70+10; in ablations it finds that selecting around 12% of convolution kernels gives the best incremental-stage tradeoff (Zhang et al., 28 Dec 2025).

6. Benchmarks, specialized regimes, and current directions

MS COCO 2017 remains the dominant benchmark for class-incremental detection, with recurring splits such as 40+40, 70+10, 40+20\times2, and 40+10\times4 (Liu et al., 2023, Zhang et al., 2024). Pascal VOC 2007 is the other common natural-image benchmark, typically under 19+1, 15+5, 10+10, and multi-step variants (Wu et al., 8 Feb 2025). Standard metrics are $t$6, $t$7, $t$8, and scale-specific AP; some transformer papers also report FPP, AbsGap, or RelGap to isolate forgetting (Liu et al., 2023, Zhang et al., 19 Mar 2025).

Beyond these benchmarks, the field has diversified into setting-specific subareas. SSIOD uses labeled and unlabeled data simultaneously; DualTeacher constructs an old teacher and a new teacher and concatenates their predictions,

$t$9

to address coexistence of unlabeled old and new classes. On MS COCO, it reports 17.94 AP in two-phase 40+40, 24.34 AP in 60+20, and a performance lead of up to 18.28 AP (Yuan et al., 2023). Remote sensing incremental detection introduces severe intra-class scale variation, dense clutter, and missing annotations. STAR-IOD addresses these with Subspace-decoupled Topology Distillation and a Clustering-driven Pseudo-label Generator, and reports gains of 1.7 points on DIOR-IOD and 2.1 points on DOTA-IOD, while also introducing the DIOR-IOD and DOTA-IOD benchmarks (Zhang et al., 20 May 2026). Player detection in sports video defines yet another specialized regime; R$t$0D proposes coarse-to-fine region refinement and refined response distillation, introduces NBA-IOD and Volleyball-IOD, and reports improvements over RILOD, SID, and ERD across multi-step splits (Bai et al., 2023).

The resulting picture is heterogeneous but coherent. Distillation remains important, but later work increasingly targets where, when, and on which representations distillation should operate. Replay remains important, but later work has largely abandoned the assumption that full-image replay is a neutral operation. Transformer-based IOD has moved toward query expansion, assignment control, semantic anchoring, or gradient-level stabilization, whereas one-stage real-time IOD has emphasized pseudo-label refinement and parameter-efficient adaptation. This suggests that the contemporary understanding of IOD is organized less by a single universal remedy than by architecture-aware analyses of forgetting, annotation incompleteness, and protocol realism (Zhang et al., 13 Jun 2026, Zhang et al., 28 Dec 2025, Zhang et al., 19 Mar 2025).