Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

Abstract: Adapting Detection Transformers to Incremental Object Detection (IOD) poses a systemic challenge, as set-based optimization is inherently destabilized by sequential learning. In this work, we identify Gradient Dilution as the root cause of performance degradation, wherein optimization signals required to preserve old knowledge are progressively weakened. This phenomenon manifests as a cascading erosion of preservation gradients in magnitude, direction, and support coverage, driven by three tightly coupled factors: Signal Dispersion, where foreground gradients are overwhelmed by background noise; Assignment Drift, where stochastic query-target matching induces inconsistent gradient trajectories; and Support Attrition, where gradients from retained samples insufficiently cover the old-class feature space, weakening decision boundaries under interference from new classes. To counteract this, we propose FAS, a unified framework that Focuses, Aligns, and Sustains gradient flow throughout incremental learning. Specifically, we introduce prior-injected queries to focus discriminative signals by filtering background interference at the source. We further propose deterministic anchor distillation to align query-target assignments and enforce semantic consistency across stages under unstable matching. Finally, we devise manifold-support replay to sustain distributional support of old classes, counteracting representational erosion induced by continual updates. Extensive experiments show that FAS restores robust optimization dynamics and outperforms state-of-the-art methods, achieving over 5.0 AP improvement in the challenging 40+10x4 incremental setting.

• The paper identifies gradient dilution—signal dispersion, assignment drift, and support attrition—as the key challenge causing catastrophic forgetting.
• The FAS framework innovates with Prior-Injected Query Formulation, Deterministic Anchor Distillation, and Manifold-Support Replay to stabilize gradient flow.
• Empirical results on COCO and PASCAL VOC show improved average precision and long-term retention across multi-phase incremental learning sessions.

Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection

Motivation and Identification of Gradient Dilution

The paper "Focus, Align, and Sustain: Counteracting Gradient Dilution in Incremental Object Detection" (2606.15253) analyzes the systemic fragility of DETR-based architectures in Incremental Object Detection (IOD). The central thesis is the explicit identification of Gradient Dilution as the root cause of catastrophic forgetting and suboptimal optimization dynamics during sequential learning. Gradient Dilution manifests as a progressive erosion of preservation gradients—diminished in magnitude, destabilized in direction, and attenuated in support coverage. The authors dissect this into three tightly coupled phenomena: Signal Dispersion, Assignment Drift, and Support Attrition.

Signal Dispersion refers to the overwhelming dominance of background gradients over foreground gradients due to set-based query supervision, severely reducing discriminative signal strength required to retain old knowledge. Assignment Drift quantifies the stochasticity in query-object matching across epochs, leading to incoherent optimization trajectories for old-class concepts. Support Attrition describes the weakening of old-class feature-space geometry, primarily at the manifold boundaries, resulting from insufficient replay coverage. Empirical evidence on COCO demonstrates these effects, with foreground-to-background gradient ratios suppressed and assignment distributions highly scattered.

Figure 1: Evidence of Gradient Dilution during incremental training—foreground gradients are overwhelmed and query assignment is inconsistent.

The FAS Framework: Focus, Align, and Sustain

To counteract Gradient Dilution, the authors propose the Focus–Align–Sustain (FAS) paradigm. The method is designated for DETR-based incremental detectors and encompasses three targeted algorithmic innovations:

1. Focus (Prior-Injected Query Formulation): Queries are initialized using semantic priors derived from class prototypes, generating saliency maps to select only high-foreground regions. This sharply improves the Signal-to-Noise Ratio (SNR) for gradients, as background-dominated queries are actively filtered out before optimization begins.

   Figure 2: The FAS architecture with Prior-Injected Query Formulation, Deterministic Anchor Distillation, and Manifold-Support Replay.

   

   Figure 3: PIQ focuses saliency maps on foreground entities, eliminating noisy background-dominated query distribution.

2. Align (Deterministic Anchor Distillation): Teacher–student alignment is stabilized by anchoring distillation to invariant spatial coordinates from the prior-injected queries. A closed-loop distillation loss aligns semantic interpretation at these deterministic anchors, preventing assignment-induced trajectory incoherence.

   Figure 4: Ablation study on the distillation weight $\lambda$ in Alignment, illustrating stability-plasticity trade-off.

3. Sustain (Manifold-Support Replay): Replay sampling is performed according to manifold support, not merely centroid proximity. Manifold coverage is achieved via clustering (e.g., K-Means), extracting representative anchors across local density peaks. This sustains feature geometry and prevents boundary attrition.

   Figure 5: MSR prevents feature collapse seen in standard herding, maintaining manifold boundary anchors.

Empirical Evaluation and Numerical Results

FAS is validated across two-phase and multi-phase protocols on COCO and PASCAL VOC. On COCO 40+40 and 70+10 splits, FAS yields improvements of up to 5.0 AP over the best prior methods, consistently reducing both Absolute and Relative Gaps from the upper-bound joint training scenario. Across multi-phase settings (e.g., 40+10×4), the method demonstrates superior long-term retention, maintaining high AP throughout incremental sessions. Ablation studies confirm each core component’s criticality: PIQ provides substantial SNR improvement, DAD stabilizes gradient coherence, and MSR mitigates geometric boundary erosion.

Qualitative analysis reveals that FAS delivers more complete and accurate detections, particularly for old-class objects post-incremental phase, compared to baselines like CL-DETR.

Figure 6: FAS delivers more complete and accurate detection than CL-DETR on COCO 70+10.

Figure 7: Failure cases illustrate persistent challenges in occluded, cluttered, and small object scenarios.

Theoretical and Practical Implications

The gradient-centric perspective formalized in this work provides a critical theoretical advance: it exposes structural degradation in set-based incremental architectures that is not addressed by conventional knowledge distillation or naive replay. Practically, the integration of semantic priors, deterministic anchors, and manifold preservation reshapes the optimization dynamics of incremental detectors, promoting both stability and plasticity without excessive reliance on memory or retraining.

This approach opens possibilities for further research in:

• Gradient flow regulation in large-scale, open-world detection
• Generalization to foundation models with parameter-efficient tuning
• Robust replay and distillation mechanisms for continual learning scenarios in vision–language and multi-modal domains

Conclusion

The paper systematically deconstructs the limitations of DETR-based incremental object detection, attributing catastrophic forgetting to Gradient Dilution driven by Signal Dispersion, Assignment Drift, and Support Attrition. The proposed FAS framework addresses these phenomena by focusing discriminative gradient flow, aligning optimization trajectories, and sustaining feature support. The resulting improvements in both retention and adaptation set a new benchmark for incremental object detection and offer a refined blueprint for gradient-centric continual learning architectures.