CLIP-DCA for Incremental Object Detection

• CLIP-DCA is a method that augments the DCA framework by using CLIP’s text encoder to stabilize recognition and mitigate catastrophic forgetting in incremental object detection.
• It decouples object localization from recognition, leveraging the stable spatial cues of localization while addressing the fragile, class-specific nature of recognition.
• The approach employs a localization-then-recognition pipeline with semantic consistency loss and duplex classifier fusion to maintain robust performance across incremental phases.

CLIP-DCA commonly denotes a CLIP-augmented reading of DCA, the method introduced in “DCA: Dividing and Conquering Amnesia in Incremental Object Detection,” where semantic knowledge from a pre-trained LLM is used to stabilize recognition in transformer-based incremental object detection (IOD) (Zhang et al., 19 Mar 2025). In this formulation, DCA redesigns DETR-like IOD as a localization-then-recognition process, motivated by the observation that forgetting is imbalanced: localization is relatively stable and transferable across phases, whereas recognition undergoes catastrophic forgetting. The main implementation uses the CLIP text encoder offline to derive class semantic features, but the detector itself remains a Deformable DETR system rather than a CLIP visual detector.

1. Nomenclature and task definition

In the DCA formulation, IOD trains a unified detector over $T$ phases without access to labels of old or future classes, so that after phase $t$ the detector handles all seen classes $C_{1:t}$ while learning new classes $C_t$. The dataset is written as $D=\{(x,y)\}$, the class set is partitioned into disjoint subsets $C=C_1 \cup \dots \cup C_T$, and at phase $t$ only the labels in $C_t$ are available (Zhang et al., 19 Mar 2025).

The name “CLIP-DCA” is not the title used in the DCA paper. Within the DCA framework, the CLIP text encoder is used offline as the pre-trained LLM in the main setup, which is why a CLIP-qualified label is sometimes applied to the method. A separate source of ambiguity is “DAC: Domain Aligned CLIP,” a different method for few-shot classification with frozen CLIP encoders; in that context, “CLIP-DCA” appears as a misnomer for DAC rather than for DCA (Gondal et al., 2023).

This distinction is important because the two methods address different problem settings. DCA targets incremental object detection and operates on a detector with localization and classification heads (Zhang et al., 19 Mar 2025). DAC targets few-shot classification, improves intra-modal and inter-modal alignment, and does not modify CLIP’s main encoders (Gondal et al., 2023).

2. Forgetting imbalance in incremental object detection

The central empirical claim behind DCA is a forgetting imbalance between localization and recognition in DETR-based IOD. Localization is described as less prone to forgetting and able to generalize to future classes, while catastrophic forgetting occurs primarily on recognition (Zhang et al., 19 Mar 2025).

The evidence is explicit. A DETR-based model trained on a set of classes can localize future classes with high recall $(94\%)$. After fine-tuning on new data, localization recall on old classes drops slightly from $94\%$ to $t$0, whereas average recognition accuracy drops from $t$1 to $t$2 (Zhang et al., 19 Mar 2025). This quantitative asymmetry motivates the “divide-and-conquer” strategy: retain and transfer the relatively stable localization ability, while treating recognition as the fragile component that requires dedicated stabilization.

The rationale given for this imbalance is that DETR’s localization depends primarily on spatial and objectness cues that are more class-agnostic, whereas recognition depends on fine-grained category-specific attributes and is therefore highly sensitive to missing labels and to sequential distributional shifts. The analysis further states that coupled features in the original DETR make localization features category-specific, while decoupling frees localization features and reveals their class-agnostic nature (Zhang et al., 19 Mar 2025). This suggests that the main failure mode in transformer-based IOD is not uniformly distributed across detection subproblems; it is concentrated in semantic discrimination.

3. Localization-then-recognition architecture

DCA uses Deformable DETR as the base detector, with a CNN backbone and transformer encoder-decoder. The backbone and transformer encoder are retained to produce enhanced feature sequences $t$3, and the decoder is modified to separate localization decoding from semantic-guided recognition decoding. Hungarian matching is used for bipartite assignment (Zhang et al., 19 Mar 2025).

The decoupled data flow is:

1. The backbone and transformer encoder produce feature sequences $t$4.
2. A localization module receives randomly initialized location queries $t$5 and outputs location embeddings $t$6.
3. A regression head predicts boxes $t$7.
4. Recognition queries are formed by concatenating location embeddings with semantic queries from a pre-trained LLM.
5. A recognition decoder produces class embeddings $t$8.
6. Final class probabilities are produced by Duplex Classifier Fusion (Zhang et al., 19 Mar 2025).

The core equations are:

$t$9

$C_{1:t}$0

$C_{1:t}$1

The architectural intention is explicit: DCA redesigns DETR-like IOD into a localization-then-recognition pipeline, explicitly decoupling the two stages to “divide” the less-forgetting localization and “conquer” the fragile recognition (Zhang et al., 19 Mar 2025). In encyclopedic terms, the method is not simply a new distillation head or a replay scheme; it is a decomposition of detector function along the empirically identified fault line of forgetting.

4. Semantic anchoring with CLIP and Duplex Classifier Fusion

The semantic component of DCA uses pre-trained LLMs. In the main implementation, the CLIP text encoder is used offline to produce class semantic features, while robustness is also demonstrated with BERT-M and BERT-S. The method is therefore not restricted to CLIP, but CLIP is the default semantic source in the reported setup (Zhang et al., 19 Mar 2025).

For seen classes $C_{1:t}$2, class names are passed through the CLIP text encoder to obtain text embeddings $C_{1:t}$3. These semantic features are concatenated with the location embeddings:

$C_{1:t}$4

The resulting queries are injected into the recognition decoder’s self-attention, introducing inter-class semantic relations and stabilizing representations of old classes across phases (Zhang et al., 19 Mar 2025).

The semantic-guided recognition decoder has $C_{1:t}$5 blocks, each with self-attention and cross-attention:

$C_{1:t}$6

$C_{1:t}$7

To prevent semantic drift and maintain alignment between decoded semantic features and the original $C_{1:t}$8, DCA applies a semantic consistency loss:

$C_{1:t}$9

This enforces alignment with the pre-trained semantic space across incremental phases (Zhang et al., 19 Mar 2025).

The final classification stage is Duplex Classifier Fusion (DCF), which combines a standard linear classifier with a semantic classifier. The linear branch applies a linear layer to $C_t$0 to obtain class probabilities $C_t$1 with sigmoid activation. The semantic branch projects $C_t$2 to semantic space as $C_t$3 and computes semantic probabilities $C_t$4 by cosine similarity against the pre-trained text embeddings of all known classes $C_t$5. The fused prediction is:

$C_t$6

with $C_t$7 in the reported experiments (Zhang et al., 19 Mar 2025).

A common misconception is that this makes DCA a CLIP detector. It does not. The visual pathway of CLIP is not used; the detector’s vision backbone remains ResNet-50 with a Deformable DETR encoder (Zhang et al., 19 Mar 2025). What CLIP contributes is a stable text-derived semantic frame for recognition.

5. Optimization objectives and incremental protocol

After Hungarian matching $C_t$8 between predictions and ground truths $C_t$9, DCA uses the detection loss

$D=\{(x,y)\}$0

with

$D=\{(x,y)\}$1

(Zhang et al., 19 Mar 2025).

Recognition stabilization is supplemented by Hybrid Knowledge Distillation (HKD), implemented as a teacher-student scheme from the old model to the new model, with pseudo labels used to select old instances in current data. The component losses are:

$D=\{(x,y)\}$2

$D=\{(x,y)\}$3

$D=\{(x,y)\}$4

$D=\{(x,y)\}$5

$D=\{(x,y)\}$6

The full objective is:

$D=\{(x,y)\}$7

During base training, $D=\{(x,y)\}$8; during incremental phases, all components are active (Zhang et al., 19 Mar 2025).

The reported protocol is exemplar-free: old-class images are not stored. Instead, pseudo labels from the old model annotate unlabeled old objects in new data for distillation selection. The architecture uses a ResNet-50 backbone with ImageNet pretraining, a Deformable DETR encoder, shared decoder depth $D=\{(x,y)\}$9, and $C=C_1 \cup \dots \cup C_T$0 location queries. During inference, top-50 detections per image are evaluated (Zhang et al., 19 Mar 2025).

The datasets and splits are stated explicitly. PASCAL VOC uses 20 classes and the 10+10, 15+5, and 19+1 settings. MS COCO uses 80 classes and two-step splits 70+10, 60+20, 50+30, and 40+40, as well as multi-step protocols with a base of 40 classes followed by either a 20-class two-step or a 10-class four-step schedule (Zhang et al., 19 Mar 2025).

6. Empirical performance and ablation structure

DCA is reported to achieve state-of-the-art performance, especially for long-term incremental scenarios (Zhang et al., 19 Mar 2025). On COCO two-step evaluation, the reported AP/AP50 values are:

• 70+10: DCA 41.3/59.2; CL-DETR* 40.4/58.0; ACF 37.6/–
• 60+20: DCA 41.9/54.8; ERD 35.8/52.9; SID 32.7/49.8
• 50+30: DCA 39.9/56.1; ERD 36.6/54.0; SID 33.8/51.0
• 40+40: DCA 42.8/58.4; CL-DETR* 42.0/60.1; ERD 36.9/54.5 (Zhang et al., 19 Mar 2025)

On COCO long-term evaluation, the two-step protocol with base 40 classes yields DCA values of 48.0/68.9 $C=C_1 \cup \dots \cup C_T$1 42.7/59.6 $C=C_1 \cup \dots \cup C_T$2 40.3/54.1 across steps, with AbsGap 2.3/7.3 and RelGap 0.05/0.12. In the four-step protocol with base 40 classes and +10 classes per step, DCA AP is 44.0 $C=C_1 \cup \dots \cup C_T$3 41.1 $C=C_1 \cup \dots \cup C_T$4 39.2 $C=C_1 \cup \dots \cup C_T$5 37.2 and DCA AP50 is 61.2 $C=C_1 \cup \dots \cup C_T$6 56.5 $C=C_1 \cup \dots \cup C_T$7 53.8 $C=C_1 \cup \dots \cup C_T$8 49.6. The final AP improvement over ACF is stated as $C=C_1 \cup \dots \cup C_T$9 (Zhang et al., 19 Mar 2025).

On VOC one-step evaluation, DCA achieves the best non-exemplar results. In 10+10, it improves over ACF by 4.8% mAP; in 19+1, it surpasses PROB, an exemplar-based method, by 1.9% mAP (Zhang et al., 19 Mar 2025).

The ablation sequence on VOC 15+5 isolates the contribution of the main components:

• Baseline fine-tuning + pseudo labeling: 62.5 mAP
• + Decoupled Localization/Recognition (DLR): 65.7
• + Semantic-guided Recognition Decoder (SRD): 64.1
• DLR + SRD: 67.9
• DLR + SRD + Duplex Classifier Fusion (DCF): 68.6
• Full (DLR + SRD + DCF + HKD): 71.2, a gain of +8.7 over baseline (Zhang et al., 19 Mar 2025)

The HKD ablation on VOC 15+5 further reports 68.8 for $t$0 alone, 70.5 for $t$1, 69.0 for $t$2, and 71.2 when all three are combined. Language-model robustness on the same split is CLIP 71.2%, BERT-M 71.0%, and BERT-S 70.5%, all exceeding ACF at 70.2% (Zhang et al., 19 Mar 2025). This supports the narrower claim that the semantic anchoring mechanism is not tied exclusively to one text encoder, even though CLIP is the default instantiation.

7. Interpretive scope, limitations, and relation to other CLIP-based formulations

The analysis attached to DCA attributes its behavior to the differing inductive roles of localization and recognition. Localization in DETR is largely governed by spatial structures and objectness rather than fine-grained semantics, so decoupling allows localization to remain class-agnostic and preserve generalization to future classes. Recognition, by contrast, benefits from semantic anchoring because $t$3 and semantic fusion place classes in a unified semantic space across phases, mitigating feature drift and reducing ambiguity between old and new classes (Zhang et al., 19 Mar 2025).

Within this perspective, a “CLIP-DCA” instantiation is described by replacing or augmenting $t$4 with CLIP text embeddings $t$5 for the seen classes, injecting them into the recognition decoder through

$t$6

and keeping text alignment through

$t$7

Optional extensions include a CLIP-style contrastive term

$t$8

and an image-alignment term

$t$9

which requires extracting CLIP image features offline. The corresponding overall objective is written as

$C_t$0

with $C_t$1 unless the CLIP image encoder is used (Zhang et al., 19 Mar 2025). These expressions define a CLIP-centered extension path, not a separately benchmarked official algorithm name.

The limitations identified for DCA are dependence on semantic quality, possible drift accumulation in very long incremental sequences, and increased overhead if full foundation-model components such as CLIP’s image encoder are introduced. The future directions listed include better fusion strategies such as learned $C_t$2 or class-wise fusion weights, continual adaptation of semantic encoders through lightweight adapters or prompt learning, improved prompt engineering and context templates, and more sophisticated alignment and prototype-updating strategies (Zhang et al., 19 Mar 2025).

A final terminological caution is warranted. In few-shot classification, “DAC: Domain Aligned CLIP” is a distinct method with two variants, DAC-V and DAC-VT, built around a frozen CLIP model, a single linear image adapter, and direct modulation of class text embeddings (Gondal et al., 2023). That method is unrelated to DCA’s localization-then-recognition decomposition. Consequently, the label “CLIP-DCA” is best understood as an informal shorthand for a CLIP-augmented DCA perspective in incremental object detection, not as a standardized title across the CLIP adaptation literature.