Query-wise Self-Distillation (QuerySD)

• Query-wise Self-Distillation is a framework that aligns query-indexed teacher signals with student representations for finer and more targeted supervision.
• It shifts the distillation focus from global embeddings to query-conditioned features, improving tasks like segmentation, detection, and language reasoning.
• Examples like S2-UniSeg and FLD illustrate how QuerySD leverages query-specific supervision to boost performance across diverse domains.

Query-wise Self-Distillation (QuerySD) denotes self-distillation schemes in which the distilled object is a query or a query-conditioned representation, rather than only a single global embedding or a final task label. The term is used explicitly in "S2-UniSeg: Fast Universal Agglomerative Pooling for Scalable Segment Anything without Supervision" for a segmentation-oriented pretext task that distills matched segmentation queries between a momentum teacher on a global image and a student on local crops (Xu et al., 9 Aug 2025). Elsewhere, closely related mechanisms appear under different names, including Forward-Looking Distillation, on-policy hindsight self-distillation, offline self-distillation, and consensus-gated trajectory distillation. This suggests a broader pattern: QuerySD is best understood as a family of methods that construct query-aligned teacher signals and transfer them to query-indexed student states through feature-, token-, trajectory-, or evidence-level supervision.

1. Terminological scope and conceptual core

In the strictest sense, QuerySD refers to the S2-UniSeg pretext task, where self-distillation operates over a set of segmentation queries rather than an image-level token (Xu et al., 9 Aug 2025). In a broader research sense, the same label naturally covers methods in which the supervision target is indexed by decoder queries, search-query tokens, per-question sibling trajectories, or query-specific evidence structures. The common denominator is that the teacher signal is not generic; it is organized around the current query, the current query slot, or the current query-conditioned trajectory.

Across domains, this query-wise organization serves a similar purpose. In detection transformers, the same decoder query slots are refined across depth, so later query states can supervise earlier ones. In search-augmented LLMs, the positions inside <search> ... </search> spans are the decision points that most directly determine which evidence becomes available downstream. In memory and retrieval systems, the relevant object is often not a token sequence but a query-specific compressed evidence set or a query-specific surrogate retrieval model. This suggests that QuerySD is less a single algorithm than a design principle: move the distillation target from a global representation to the decision-bearing unit tied to a query.

A second unifying property is privileged conditioning. The teacher is usually stronger not because it is a different architecture, but because it has access to additional context: a global image rather than a crop, a later decoder layer rather than an earlier one, hindsight over sibling rollouts, a document available only at training time, a retrieved skill bank, or an initial response plus its verifier outcome. QuerySD therefore overlaps strongly with self-distillation under privileged information, but it remains distinct in that the privileged information is applied to a query-indexed target rather than to a whole-example label.

2. Canonical formulation in self-supervised segmentation

The explicit QuerySD formulation appears in S2-UniSeg, which argues that standard self-supervised objectives based on a ViT [CLS] token or ResNet global pooling are poorly matched to segmentation because segmentation requires more fine-grained features (Xu et al., 9 Aug 2025). S2-UniSeg instead aggregates each image into a set of object queries and distills them query-wise. The teacher branch receives the original image as a global view, while the student receives local views from multi-crop augmentation. Teacher features are processed by Fast Universal Agglomerative Pooling (UniAP), which generates semantic pseudo-masks, instance pseudo-masks, and their query features. Student local features are processed by a Mask2Former-style decoder with semantic and instance queries.

The segmentation queries are explicit architectural objects. The student decoder partitions object queries into semantic queries and instance queries, distinguished by learnable [SEM] and [INS] tokens. The default configuration uses 50 semantic queries and 150 instance queries. The teacher-side targets are not decoder queries; they are UniAP pooled region embeddings attached to teacher pseudo-masks. Training then crops teacher pseudo-masks to the spatial support of each student local view and performs cropped bipartite matching between teacher masks and student masks. Only mask Dice similarity is used as the matching criterion. After matching, the paper denotes the teacher query embeddings by $\mathbf{Q}_t \in R^{L\times d}$ and the student query embeddings by $\mathbf{Q}_s \in R^{L\times d}$, with $L$ the number of matched query pairs, and states that QuerySD is the sum of self-distillation losses over matched teacher-student query pairs.

This design makes the pretext task explicitly local-to-global. A local crop is trained to recover the teacher query corresponding to the same region in the global image, rather than merely imitate a single global representation. The projection head used for this distillation is a 3-layer MLP with hidden dimension 2048, followed by L2 normalization and a final linear layer of dimension $K$, with $K=512$ in the default setting. Ablations over $K \in \{128,512,1024\}$ show better COCO-UVO instance metrics as $K$ increases from 128 to 1024. The teacher is a momentum teacher with a cosine schedule from 0.996 to 1, and the paper reports that two local crops are retained to save computation because more local crops require more bipartite matching.

S2-UniSeg ties QuerySD to continuous online pretraining. UniAP generates teacher pseudo-masks online in tens of milliseconds for one image, avoiding the multi-stage offline pseudo-mask generation routine criticized in prior self-supervised segmentation pipelines. Under the same setting, S2-UniSeg reports improvements over UnSAM of AP+6.9 on COCO, AR+11.1 on UVO, PixelAcc+4.5 on COCOStuff-27, and RQ+8.0 on Cityscapes, and it further reports additional gains after scaling pretraining to a larger 2M-image subset of SA-1B (Xu et al., 9 Aug 2025).

3. Decoder-query supervision in DETR-style perception

A closely related visual pattern appears in DETR-like models, where distillation is indexed by decoder queries rather than by pixels or image tokens. In MonoVQD, the self-distillation component is called Forward-Looking Distillation (FLD). FLD uses the final decoder layer as the teacher and all preceding layers as students, with Hungarian matching first applied on final-layer predictions to identify the object-aligned query slots. The corresponding queries from earlier layers are then distilled toward the final-layer query representation through a shared MLP and a Smooth L1 loss weighted by teacher 3D IoU:

$$L_{distill} = \sum_{i=1}^{L-1}\mathbf{IoU}_{3D_{Q_L}}\cdot S_{L1}\left(f_Q(Q_i), Q_L\right).$$

The full training objective is

$$L_{overall} = \lambda_1L_{det}+ \lambda_{2}L_{DN}+ \lambda_{3}L_{distill},$$

with implementation weights $\{1,1,0.5\}$ for $\mathbf{Q}_s \in R^{L\times d}$0 to $\mathbf{Q}_s \in R^{L\times d}$1 (Vu et al., 14 Jun 2025).

Several details make FLD a clear query-wise self-distillation mechanism. The supervision source is the model’s own final decoder layer, computed online in the same forward pass. The selected supervision set is matching-aware rather than based on raw query indices alone. The distilled object is the decoder query representation itself, not the explicit class logits or 3D box heads. FLD is also applied to both learnable queries and noisy queries, and it is used only during training, so inference cost, memory consumption, and parameter count remain the same as the base MonoDETR model. On KITTI validation, adding FLD alone improves $\mathbf{Q}_s \in R^{L\times d}$2 from $\mathbf{Q}_s \in R^{L\times d}$3 to $\mathbf{Q}_s \in R^{L\times d}$4 for Easy/Mod./Hard, and a separate ablation shows that both the 3D-IoU weighting and the MLP refinement contribute to the gain (Vu et al., 14 Jun 2025).

A neighboring but non-self-distillation DETR line is "Knowledge Distillation via Query Selection for Detection Transformer," which proposes Group Query Selection for DETR distillation by segmenting queries according to their Generalized Intersection over Union with ground truth and explicitly incorporating hard-negative queries. That framework, QSKD, combines Group Query Selection with Attention-Guided Feature Distillation and Local Alignment Prediction Distillation. The abstract reports that the AP of Conditional DETR ResNet-18 increases from 35.8 to 39.9. Although this is query-selection distillation rather than self-distillation, it reinforces the same broader claim: the granularity of query supervision matters, and hard-negative or non-final queries can carry useful training signal (Liu et al., 2024).

4. Search-query token distillation in search-augmented reasoning

In search-augmented LLMs, QuerySD often becomes literally token-wise supervision at search-query positions. SD-Search defines $\mathbf{Q}_s \in R^{L\times d}$5 as the token positions strictly inside <search> ... </search> tags and restricts its self-distillation loss to those positions. The same model serves as student and teacher, but the teacher receives a hindsight block summarizing search traces and final outcomes from a group of rollouts for the same question. The teacher distribution is stop-gradient, and the distillation loss is

$\mathbf{Q}_s \in R^{L\times d}$6

The total objective is $\mathbf{Q}_s \in R^{L\times d}$7, with default $\mathbf{Q}_s \in R^{L\times d}$8, warmup $\mathbf{Q}_s \in R^{L\times d}$9, top-$L$0, and rollout group size $L$1 (Ma et al., 18 May 2026).

The teacher in SD-Search is stronger only because it sees hindsight: masked search traces and Correct/Incorrect labels from sibling rollouts for the same question. This creates dense query-token supervision inside a standard RL loop without an external teacher or extra annotation pipeline. On seven QA benchmarks, SD-Search-Base reaches 0.428 average EM at 3B, compared with 0.405 for AutoRefine-Base and 0.414 for MR-Search-Base, while SD-Search-Instruct reaches 0.476 at 7B. Ablations show that removing outcome labels, shuffling labels, removing the multi-rollout group, or broadening the alignment scope from $L$2 to all action positions all reduce performance (Ma et al., 18 May 2026).

Search-E1 uses a different but closely related query-wise mechanism. It alternates vanilla GRPO with offline self-distillation (OFSD). After each GRPO round, the policy samples multiple trajectories for the same training question, mines a pair consisting of a more efficient correct sibling $L$3 and a weaker sibling $L$4, and aligns the student’s inference-time distribution on the tokens of $L$5 to its own distribution under a privileged prompt that also contains $L$6. The student prompt is $L$7, the teacher prompt is $L$8, and the OFSD loss is a token-level forward KL over all policy-generated positions except <information> spans. Search-E1 reports 0.440 average EM with Qwen2.5-3B and 0.487 with Qwen2.5-7B, with especially large gains on multi-hop datasets where sibling trajectories diverge more in query quality (Liang et al., 21 May 2026).

5. Privileged context, verifier signals, and intra-query correction

A second major branch of QuerySD uses privileged context and verifier outcomes to construct query-conditioned teachers for reasoning. Self-Distillation Zero (SD-Zero) trains a single model in two roles: a Generator that produces an initial response $L$9, and a Reviser that conditions on the problem $K$0, the generated response $K$1, and a reward-conditioned control prompt $K$2 to produce an improved response. Phase 1 trains self-revision; Phase 2 distills the frozen reviser into the generator by minimizing token-level KL between the generator’s distribution $K$3 and the reviser’s distribution $K$4. The paper emphasizes token-level self-localization, namely that the reviser concentrates its corrective pressure on a small subset of tokens for incorrect responses. On Qwen3-4B-Instruct, average performance across eight math and code benchmarks rises from 49.8 for the base model to 60.3 for SD-Zero (He et al., 13 Apr 2026).

GATES studies an asymmetric privileged-context setting in which the same model acts as a tutor with access to a document $K$5 and a student that sees only the question $K$6. Its distinctive QuerySD element is a per-question consensus gate. For each training question, the tutor samples $K$7 document-grounded rollouts; the question-level gate $K$8 is set to 1 only if at least $K$9 tutor rollouts agree on the same final answer, with the implementation using a $K=512$0 training gate. If consensus is weak, the question is skipped entirely. If consensus is strong, the model distills full tutor reasoning trajectories into the student through off-policy and on-policy losses gated by $K=512$1 and a rollout-level eligibility indicator $K=512$2. Held-out asymmetric evaluation improves from 46.0% to 62.0%, and average maj@8 on public document-free math benchmarks rises from 20.2% to 35.4% (Stein et al., 24 Feb 2026).

Skill-Conditioned Gated Self-Distillation (SGSD) replaces trusted privileged information with a retrieved skill bank. For each query $K=512$3, it retrieves $K=512$4 and $K=512$5, constructs teacher contexts $K=512$6, and lets all skill-conditioned teachers score the same plain-prompt student rollout. The final objective is

$K=512$7

where $K=512$8 is a retrieval-derived teacher weight, $K=512$9 is a verifier-validated polarity that can support, reverse, or suppress a teacher’s stance, and $K \in \{128,512,1024\}$0 is a robust gated token loss. This formulation treats teacher supervision as a hypothesis to validate rather than a target to imitate unconditionally. On Qwen3-1.7B, SGSD reaches 43.7 average accuracy on AIME24, AIME25, and HMMT25, compared with 37.5 for GRPO and 42.0 for OPSD (Huang et al., 27 May 2026).

6. Related paradigms, boundaries, and recurrent design lessons

Several adjacent methods illuminate what QuerySD is and is not. Some are query-specific distillation systems without self-distillation in the narrow sense; others are self-distillation systems whose distilled unit is not a query distribution but a trajectory, evidence set, or prototype.

Method	Relation to QuerySD	Core supervised object
ODIS (MacAvaney et al., 2023)	Query-specific online distillation, not self-distillation in the narrow sense	A sparse lexical student for the current query
Adaptive relevance margins (Gienapp et al., 2024)	Closely related self-distillation retrieval method, not canonical query-list distillation	Query-conditioned triplet margins
DeferMem (Yin et al., 21 May 2026)	Query-conditioned evidence distillation at query time	Selected and rewritten evidence
MQSA-TED (Zhu et al., 2023)	Multi-query architecture, not QuerySD	Transition distributions into item embeddings
SD-AANet (Zhao et al., 2021)	Query-conditioned prototype self-distillation	Support prototype aligned to query-informed teacher
TAPO (Huang et al., 17 Jun 2026)	Intra-query self-distillation by explicit trajectory construction	Micro-reflective corrective trajectories
Self-Verified Distillation (Lee et al., 20 May 2026)	Verification-centric query-wise self-training rather than token-level teacher-student KD	Accepted query-response pairs

These neighbors expose several recurring safeguards. First, teacher quality is rarely assumed. MonoVQD weights distillation by teacher 3D IoU; GATES skips questions with weak consensus; SGSD reverses or suppresses misleading teacher signals; SD-Search uses hindsight labels from rollout groups rather than trusting a single teacher pass. Second, the unit of supervision is often narrowed to the part of the example that actually carries query decisions: matched decoder queries, <search> tokens, selected evidence fields, or the prefix beginning at the first critical mistake. Third, many methods achieve their gains without inference-time overhead because the privileged teacher exists only during training.

A common misconception is that every query-conditioned training rule is QuerySD. The literature summarized here suggests a narrower reading. QuerySD is strongest when a query-indexed student state is explicitly aligned to a stronger query-indexed teacher state. By that standard, S2-UniSeg, FLD, SD-Search, Search-E1, GATES, SD-Zero, and SGSD fit naturally. ODIS and DeferMem are closer to query-time distillation than to self-distillation proper; MQSA-TED is a multi-query architecture with transition-aware embedding distillation but not query-wise self-distillation; Self-Verified Distillation and TAPO move even further toward query-wise synthetic data curation and explicit corrective trajectory construction.

Taken together, these papers suggest that QuerySD is becoming a cross-domain training pattern rather than a domain-specific trick. In vision, it distills matched region or decoder queries. In reasoning, it distills search-query tokens, sibling trajectories, or reward-conditioned revisions. In retrieval and memory systems, it distills query-specific surrogate models or evidence structures. What unifies these settings is the same technical wager: dense supervision becomes more faithful and more useful when it is attached to the query-bearing internal object rather than to a single global target.