Index-Aligned Query Distillation

• The paper demonstrates that preserving query identity through fixed same-index pairing mitigates semantic and spatial drift during incremental learning.
• It introduces proxy query selection based on teacher confidence to selectively distill old-category features while allowing adaptation for new classes.
• Empirical results on COCO and PASCAL VOC show that IAQD outperforms Hungarian-matched distillation by better balancing old and new category detection.

Index-Aligned Query Distillation (IAQD) is a distillation method proposed for transformer-based incremental object detection (IOD). In its strict sense, it denotes a query-identity-preserving knowledge distillation strategy in which the query at index $i$ in the current-phase detector is aligned with the query at the same index in the last-phase detector, rather than with a dynamically reassigned query obtained by Hungarian Matching. The method further restricts distillation to a subset of old-category-critical queries, called proxy queries, in order to preserve previous semantic and spatial encoding capabilities without unduly constraining the learning of new categories /compiler-like? no; arXiv format only.

1. Incremental object detection context

IAQD is defined in the setting of incremental object detection, where the category space is partitioned across phases: $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$ The dataset is correspondingly partitioned as

$$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$

and at phase $t$ the detector trains only on images from $\mathcal{D}_t$ with annotations only for categories in $\mathcal{C}_t$. Old-category objects may still appear in current-phase images, but they are unlabeled. The central difficulty is catastrophic forgetting: after adaptation to new categories, performance on previously learned categories may degrade sharply. This difficulty is particularly acute in DETR-family models because detection is mediated by a fixed set of learned object queries that are expected to encode semantic and spatial patterns useful for detection (MA et al., 15 Aug 2025).

Earlier transformer-based IOD methods mitigate forgetting through teacher–student knowledge distillation. A frozen last-phase model serves as teacher, and the current-phase model serves as student. Because DETR-style outputs form unordered sets, prior methods establish cross-model query correspondence with Hungarian Matching and then distill matched outputs such as class probabilities and box predictions. In the formulation summarized by the paper, the generic distillation term is

$$\mathcal{L}_{\text{distill}} = \lambda_{1}\cdot \mathrm{CE}(p_{\text{T}}^{(i)},p_{\text{S}}^{(j)}) + (1-\lambda_{1})\cdot \mathrm{MSE}(\mathbf{b}_{\text{T}}^{(i)},\mathbf{b}_{\text{S}}^{(j)}),$$

where $(i,j)$ is a matched teacher–student pair, $p$ denotes classification probabilities, and $\mathbf b$ denotes predicted boxes. IAQD begins from the claim that this matching mechanism is poorly suited to preserving cross-phase query identity in IOD (MA et al., 15 Aug 2025).

2. Critique of Hungarian-matched query distillation

The defining motivation of IAQD is the observation that Hungarian Matching yields unstable query correspondence across training iterations. A current-phase query may match different previous-phase queries at different steps. The paper illustrates this with a fixed current-phase query whose matched teacher indices vary over 50 epochs. Under this regime, one student query is not consistently supervised by one teacher query; instead it is repeatedly aligned to a moving target (MA et al., 15 Aug 2025).

The paper interprets this instability as producing two forms of drift. The first is semantic drift: a query that previously encoded old-category semantics is repeatedly reshaped toward different teacher-query functions, and, under simultaneous learning of new categories, can be pulled away from its previous old-class specialization. The second is spatial drift: because the matched teacher counterpart changes, localization priors encoded in the query may also shift. The argument is not that Hungarian Matching is wrong for DETR set prediction itself, but that it is a poor mechanism for cross-phase query identity preservation in incremental detection (MA et al., 15 Aug 2025).

This point also clarifies a frequent misconception. IAQD does not remove Hungarian Matching from the detector’s main supervision. Hungarian Matching remains part of the standard DETR detection loss for assigning predictions to labels or pseudo-labels. What changes is only the alignment rule used in the distillation branch. In other words, Hungarian Matching remains in $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$0, whereas index alignment is introduced specifically for $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$1 (MA et al., 15 Aug 2025).

3. Index alignment and proxy queries

IAQD is built on the hypothesis that, because the current-phase detector is initialized from the previous-phase detector, the query at index $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$2 in the student naturally inherits the semantic and spatial role of the query at index $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$3 in the teacher better than any dynamically re-matched partner. The distillation correspondence is therefore fixed as

$$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$4

This converts query distillation from dynamic pairing to identity-preserving same-index supervision (MA et al., 15 Aug 2025).

The method does not distill all queries indiscriminately. It introduces Proxy Query Selection, defining a subset of old-category-critical queries: $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$5 A query is selected when the last-phase model assigns high confidence to at least one old category. The default threshold is $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$6. This means the selection heuristic is a confidence-threshold rule on old-class activation, not top-$$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$7, IoU, objectness, or matching statistics. The rationale is capacity allocation: selected proxy queries retain old knowledge, while non-selected queries remain freer to adapt to new categories (MA et al., 15 Aug 2025).

The IAQD loss is then applied only over $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$8: $$\mathcal{C}=\{1,\ldots,C\},\qquad \{\mathcal{C}_t\}_{t=1}^T,\qquad \mathcal{C}_i\cap\mathcal{C}_j=\varnothing \ (i\neq j).$$9 Here the classification term uses only old-category probabilities, whereas the localization term uses full box predictions. The paper’s interpretation is that the teacher has no knowledge of new categories, so forcing the student to imitate teacher outputs on new classes would be harmful, while box regression is class-agnostic and preserves geometric priors (MA et al., 15 Aug 2025).

4. Training pipeline and optimization structure

The main implementation uses Deformable DETR with a ResNet-50 backbone pretrained on ImageNet, with iterative box refinement and the two-stage mechanism disabled to match prior IOD settings. The method is also shown to transfer to DINO. At phase $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$0, a last-phase model is frozen as teacher, the current-phase model is initialized from it, and training uses

$$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$1

where $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$2 denotes exemplar memory from earlier phases (MA et al., 15 Aug 2025).

Because $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$3 contains no old-class annotations, the teacher first generates pseudo labels for old categories. These pseudo labels are merged with the current phase’s ground-truth annotations for new categories. The student is then optimized with the total loss

$$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$4

The detection term still relies on Hungarian bipartite matching: $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$5 with pairwise cost

$$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$6

This coexistence of Hungarian matching for supervision and index alignment for distillation is one of the method’s defining structural features (MA et al., 15 Aug 2025).

After the main incremental stage, the method performs Exemplar Replay with Label Realignment. In this phase, the current model pseudo-labels missing-category annotations in old exemplars and current data, available ground-truth labels are retained, and fine-tuning uses only $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$7, not $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$8, to reduce cost. Reported hyperparameters are $$\mathcal{D}=\mathcal{D}_1\cup\cdots\cup\mathcal{D}_T,$$9, $t$0, and $t$1; training uses AdamW with learning rate $t$2, weight decay $t$3, an exemplar budget of 10% of the total dataset, pseudo-label threshold 0.4 in the incremental stage and 0.6 in the ER stage, 50 epochs for the initial phase, and 50 incremental plus 20 ER epochs for each new phase, on 4 NVIDIA 4090 GPUs (MA et al., 15 Aug 2025).

5. Empirical results and ablation evidence

The empirical evaluation covers COCO 2017 and PASCAL VOC under two incremental-data protocols. Protocol A selects phase data by presence of current categories and allows image overlap across phases, whereas Protocol B directly splits the entire dataset and avoids image overlap. On COCO, reported metrics include AP, AP$t$4, AP$t$5, AP$t$6, AP$t$7, and AP$t$8 (MA et al., 15 Aug 2025).

On COCO 2017, two-phase results show consistent gains. Under Protocol A, the 40+40 setting improves from 43.0 AP for SDDGR and 42.0 AP for CL-DETR to 43.6 AP for IAQD; the 70+10 setting improves from 40.9 AP for SDDGR and 40.4 AP for CL-DETR to 43.2 AP for IAQD. Under Protocol B, IAQD reaches 39.9 AP in 40+40 versus 37.5 for CL-DETR, and 42.2 AP in 70+10 versus 40.1 for CL-DETR. In multi-phase Protocol A, IAQD reaches 40.5 AP in $t$9 versus 36.8 for SDDGR, and 42.8 AP in $\mathcal{D}_t$0 versus 41.1 for SDDGR. Generalization to DINO is also reported: on COCO 70+10, IAQD improves over CL-DETR from 45.9 to 47.4 under Protocol A and from 45.8 to 46.4 under Protocol B. On PASCAL VOC, gains over CL-DETR include 50.0 versus 44.1 in Protocol A 10+10, 50.1 versus 47.5 in Protocol A 15+5, 45.3 versus 34.0 in Protocol B 10+10, and 47.8 versus 42.1 in Protocol B 15+5 (MA et al., 15 Aug 2025).

The ablation study isolates the roles of index alignment, proxy selection, and replay. In Protocol A 70+10, a baseline detector reports All/Old/New AP of 40.8/41.1/45.2. Adding Index-Aligned Query Distillation changes this to 41.5/41.9/43.8, improving old AP by $\mathcal{D}_t$1 while lowering new AP by $\mathcal{D}_t$2. Adding Proxy Query Selection yields 41.7/42.1/44.4, recovering part of the new-class degradation. Adding ER with Label Realignment yields 43.2/43.6/43.5. The sensitivity analysis further reports best performance at $\mathcal{D}_t$3, $\mathcal{D}_t$4, and $\mathcal{D}_t$5, while performance remains relatively stable for $\mathcal{D}_t$6 up to 0.4, $\mathcal{D}_t$7, and $\mathcal{D}_t$8 (MA et al., 15 Aug 2025).

6. Scope, assumptions, and common misconceptions

IAQD rests on a specific representation hypothesis: DETR queries are treated not as interchangeable unordered outputs for distillation, but as persistent indexed slots whose semantic and spatial roles should be preserved across incremental phases. Its fixed same-index pairing is therefore meaningful only to the extent that student queries inherit prior roles through initialization from the previous-phase model. This is a natural assumption in DETR-like models, but the paper identifies it as an assumption rather than a theorem (MA et al., 15 Aug 2025).

A second misconception is that stronger preservation necessarily means constraining all queries. The paper’s ablations argue against this. Full or broader distillation helps old classes but can reduce new-class AP, which is why Proxy Query Selection is introduced. The method’s intended balance is explicit: selected proxy queries preserve old knowledge, and non-selected queries remain free to adapt to new categories. In this sense, IAQD is simultaneously a preservation mechanism and a capacity-management mechanism (MA et al., 15 Aug 2025).

Several practical dependencies are also explicit. Proxy selection relies on teacher old-class confidence, so poor calibration in the teacher may yield a suboptimal selected set $\mathcal{D}_t$9. Performance also depends on pseudo-label quality and exemplar replay, as in many IOD methods. Finally, IAQD is not an architectural redesign: it adds fixed same-index pairing plus query filtering by old-class confidence, while retaining the surrounding pseudo-labeling and replay framework. This relative simplicity is part of its reported practicality (MA et al., 15 Aug 2025).

7. Related and analogous distillation frameworks

Several other works in the supplied literature are relevant to IAQD, although they do not all use the term. They are best understood as adjacent formulations of alignment between a distilled student and a structured support such as retrieved units, ANN candidates, spatial indices, or discretized output bins.

Work	Aligned support	Relation to IAQD
"Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings" (Xiao et al., 2022)	Query-conditioned relevance over sampled candidate documents for IVF and PQ	Retrieval-oriented analogue
"DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA" (Yin et al., 21 May 2026)	Retrieved message IDs and one-to-one rewritten evidence statements	Close conceptual analogue
"QUILL: Query Intent with LLMs using Retrieval Augmentation and Multi-stage Distillation" (Srinivasan et al., 2022)	Retrieved titles and URLs used to supervise a non-RA teacher and efficient student	Retrieval-augmented but only partially equivalent
"XD-RCDepth: Lightweight Radar-Camera Depth Estimation with Explainability-Aligned and Distribution-Aware Distillation" (Sun et al., 15 Oct 2025)	Spatial saliency positions and depth-bin indices	Strong partial analogy, not literal query distillation

"Distill-VQ" trains IVF and PQ not to minimize document reconstruction distortion alone, but to reproduce a dense retriever’s query-conditioned relevance behavior over sampled candidate documents. Its strongest result is that listwise ranking preservation, especially ListNet, is more effective than score invariance, and that labeled data is no longer a necessity for high-quality vector quantization. This suggests a broader IAQD principle in retrieval systems: preserve teacher query–document ranking behavior rather than only embedding geometry (Xiao et al., 2022).

"DeferMem" formalizes long-memory use as high-recall retrieval followed by query-conditioned evidence distillation, $\mathcal{C}_t$0 and $\mathcal{C}_t$1. Its distiller selects useful message IDs and rewrites them into faithful, self-contained, query-conditioned evidence tied one-to-one to those IDs. The supplied material explicitly characterizes it as a close conceptual analogue to IAQD, though not a fully index-aware model in the strongest sense (Yin et al., 21 May 2026).

"QUILL" uses retrieved URLs and titles to augment sparse queries for a retrieval-augmented Professor model, then compresses this supervision through a non-RA Teacher into a 4-layer efficient Student. The material describes it as highly relevant but only partially equivalent to IAQD: it is retrieval-aware and indirectly index-aware, but it distills retrieval-informed intent predictions rather than index-native retrieval or ranking objectives (Srinivasan et al., 2022).

"XD-RCDepth" does not involve learned object queries, but it implements two explicit aligned distillation mechanisms: Explainability-Aligned Saliency Map Distillation over spatial positions and Depth-Distribution Distillation over shared depth-bin indices. The material characterizes it as a strong partial analogy to IAQD, because the transfer occurs over explicit indexed supports, even though the aligned objects are spatial positions and bins rather than query tokens (Sun et al., 15 Oct 2025).

By contrast, the supplied material for "Knowledge Distillation via Query Selection for Detection Transformer" states that there is insufficient information to determine whether QSKD is a precursor to, variant of, or contrastive baseline for IAQD. The abstract describes Group Query Selection, Attention-Guided Feature Distillation, and Local Alignment Prediction Distillation for DETR compression, but the supplied paper content is explicitly said to be insufficient for establishing a paper-grounded relation to IAQD (Liu et al., 2024).