Self-Anchored Visual Distillation

• Self-Anchored Visual Distillation is a design principle where a model uses its own visual outputs as a supervisory anchor instead of an external teacher.
• It employs diverse methods—from frozen encoders and weight-sharing teachers to EMA-based online anchors—to enhance learning across modalities.
• Empirical studies show that this approach improves stability, accuracy, and contextual alignment in tasks such as autonomous driving, code generation, and dense prediction.

Self-anchored visual distillation denotes a class of distillation schemes in which the supervisory anchor is derived from the model’s own visual encoder, visual outputs, privileged variants of its own observations, internal feature hierarchy, or student-side saliency signals rather than from an external teacher. In autonomous driving, it appears as a frozen copy of a pre-trained visual encoder that regularizes an unfrozen student with trajectory-guided token weights; in code generation, as a weight-sharing teacher conditioned on the student’s own rendered artifact and detected defects; in 4D perception, as an online EMA teacher operating on richer spatiotemporal context; in fine-grained multimodal LLMs, as a crop-conditioned policy that distills privileged regional perception into a full-image policy; in open-vocabulary dense prediction, as any-to-any self-distillation across the model’s own representation levels; and in think-answer VLM distillation, as student-side masking that forces predictions to remain anchored on visual evidence (Cao et al., 10 Mar 2026, Dong, 9 Jun 2026, Huang et al., 9 Apr 2026, Yuan et al., 18 May 2026, Yeo et al., 10 Jun 2025, Yu et al., 12 May 2026).

1. Domain-specific definitions and scope

Across recent work, the term is used with task-specific meanings. In EvoDriveVLA, self-anchored visual distillation is the perceptual half of a collaborative perception–planning distillation framework, designed to stabilize the visual encoder after unfreezing in a vision-language-action model for autonomous driving (Cao et al., 10 Mar 2026). In Visual-SDPO, it is a visual-feedback self-distillation scheme for code-generated charts, web pages, and slides, where the teacher conditions on rendered screenshots, rubric diagnostics, or execution errors from the student’s own rollout (Dong, 9 Jun 2026). In SelfEvo, the anchor is spatiotemporal context asymmetry: the teacher sees richer context, while the student is trained on reduced context without external annotations (Huang et al., 9 Apr 2026). In Vision-OPD, the anchor is privileged regional evidence, implemented by a crop-conditioned teacher and a full-image-conditioned student instantiated from the same MLLM (Yuan et al., 18 May 2026). In ATAS, self-anchoring refers to using the model’s own internal features across multiple representation levels as anchors for any-to-any consistency (Yeo et al., 10 Jun 2025). In Masking-KD, self-anchoring refers to student-side saliency and self-paced masking budget scheduling, which modify the student’s accessible context during distillation (Yu et al., 12 May 2026).

Work	Anchor source	Distilled target
EvoDriveVLA	frozen, pre-trained visual encoder from Qwen2.5-VL 3B	encoder visual tokens
Visual-SDPO	weight-sharing teacher with privileged visual feedback	rollout-token distributions
SelfEvo	EMA teacher with richer spatiotemporal context	depth, camera, point-map outputs
Vision-OPD	crop-conditioned teacher	full-image next-token distributions
ATAS	internal multi-level visual features	CLIP visual representations
Masking-KD	student saliency and discrepancy signals	masked-student next-token distributions

This suggests that self-anchored visual distillation is not a single algorithmic template but a recurring design principle: supervision is generated from the model’s own stronger visual condition, more stable representation, or more informative internal view, and then transferred to a weaker condition or more plastic student state.

2. Anchor construction and teacher design

A central distinction is whether the anchor is static, online, or context-conditioned. EvoDriveVLA uses a static self-anchor teacher: the frozen, pre-trained visual encoder from the base Qwen2.5-VL 3B architecture is instantiated by copying the student’s visual encoder weights before any driving-specific fine-tuning, and no EMA is used. The teacher receives multi-view camera images $I_t = \{I_t^{(v)}\}_{v=1}^{V}$ and produces teacher visual tokens $z_v^{tea}$; the student processes the same images and produces $z_v^{stu}$ (Cao et al., 10 Mar 2026).

Visual-SDPO uses a weight-sharing teacher rather than a frozen copy. The coding student $\pi_\theta$ generates code $y = (y_1,\ldots,y_T)$ from input $x$, while the teacher $\pi_\theta$ re-scores the same rollout with privileged visual context $v$, which may be the rendered image, a structured rubric, or an execution error trace if rendering failed. The defect detector yields localized defect regions $D$, and a region-to-code mapping $M$ links defective regions back to specific statements in the rollout (Dong, 9 Jun 2026).

SelfEvo and Vision-OPD both use online self-distillation, but with different anchors. SelfEvo defines two model instances, $z_v^{tea}$0 and $z_v^{tea}$1, with the same architecture and initialization. Given an unlabeled clip $z_v^{tea}$2, the teacher input is $z_v^{tea}$3 and the student input is $z_v^{tea}$4, with the teacher updated by EMA according to $z_v^{tea}$5, where $z_v^{tea}$6 for VGGT or $z_v^{tea}$7 for $z_v^{tea}$8 (Huang et al., 9 Apr 2026). Vision-OPD instead instantiates the same MLLM as a crop-conditioned teacher and a full-image-conditioned student, and stabilizes training with EMA regularization at $z_v^{tea}$9 (Yuan et al., 18 May 2026).

ATAS and Masking-KD broaden the notion of anchoring further. ATAS treats patch tokens from early, mid, and late transformer blocks, dense features from a local feature extraction head, the global CLS token, and multi-scale contexts induced by mosaics as an internal feature hierarchy; any two levels may supervise each other (Yeo et al., 10 Jun 2025). Masking-KD is self-anchored in two senses explicitly stated in the paper: the prefixes to hide are identified from the student’s own response-to-response attention map, and the masking scale is adapted using the teacher–student discrepancy measured on the student’s own predictions (Yu et al., 12 May 2026).

An earlier online antecedent appears in MOKD, where each model learns with an EMA teacher of itself and also distills from the other model’s EMA teacher. The cross-attention feature search module injects the student’s global features into the teacher’s transformer head so that the transferred features are semantically aligned to the current student view (Song et al., 2023).

3. Objectives and mathematical forms

The loss used in self-anchored visual distillation depends on what is being anchored. In EvoDriveVLA, the alignment is performed directly on visual tokens, with trajectory-aware token weights:

$z_v^{stu}$0

where $z_v^{stu}$1 is produced by AnchorFormer and $z_v^{stu}$2. The full student objective is

$z_v^{stu}$3

with $z_v^{stu}$4, $z_v^{stu}$5, and $z_v^{stu}$6 (Cao et al., 10 Mar 2026).

Visual-SDPO distills rollout-token distributions with defect-grounded weights. Statement responsibility is defined as

$z_v^{stu}$7

with

$z_v^{stu}$8

The token-level loss is

$z_v^{stu}$9

and it is combined with a sequence-level GRPO term (Dong, 9 Jun 2026).

SelfEvo uses stop-gradient output distillation under context asymmetry:

$\pi_\theta$0

with predictions $\pi_\theta$1 and $\pi_\theta$2. The paper emphasizes output-level distillation and reports that adding $\pi_\theta$3 did not consistently improve results (Huang et al., 9 Apr 2026).

Vision-OPD minimizes token-level divergence along on-policy student rollouts. For a sequence $\pi_\theta$4, the objective is

$\pi_\theta$5

where $\pi_\theta$6 is instantiated as generalized Jensen–Shannon divergence with $\pi_\theta$7 and computed over a top-$\pi_\theta$8 partial softmax with $\pi_\theta$9 (Yuan et al., 18 May 2026).

Masking-KD alters the student’s accessible context during distillation. The student uses a salient reasoning-prefix mask $y = (y_1,\ldots,y_T)$0, while the teacher uses the standard causal mask $y = (y_1,\ldots,y_T)$1. The reverse-KL objective is

$y = (y_1,\ldots,y_T)$2

with temperature $y = (y_1,\ldots,y_T)$3 (Yu et al., 12 May 2026).

MOKD, finally, situates self-anchoring within self-supervised representation learning: each model uses DINO-style self-distillation with its EMA teacher, while cross-distillation uses the other model’s EMA teacher through cross-entropy objectives $y = (y_1,\ldots,y_T)$4 and $y = (y_1,\ldots,y_T)$5, combined as

$y = (y_1,\ldots,y_T)$6

The EMA update follows $y = (y_1,\ldots,y_T)$7 (Song et al., 2023).

4. Spatial, temporal, and reasoning-aware anchoring

A defining property of the recent literature is that the anchor is rarely uniform. EvoDriveVLA makes the anchor explicitly trajectory-aware. AnchorFormer consists of an AnchorLayer, architecturally identical to a single LLM decoder layer and initialized from the model’s final LLM layer, and an AnchorScorer, a single linear layer. Teacher visual tokens $y = (y_1,\ldots,y_T)$8, prompt tokens $y = (y_1,\ldots,y_T)$9, ego-state tokens $x$0, ground-truth waypoint tokens $x$1, and learnable query tokens $x$2 are fused as

$x$3

with $x$4, and token weights are computed as

$x$5

This makes the strongest regularization fall on trajectory-critical regions (Cao et al., 10 Mar 2026).

Visual-SDPO makes the anchor spatially targeted through Visual-Grounded Code Credit Weighting. Defect categories include overlap/collision, clipping, misalignment/position, low contrast/color, and overflow/whitespace/imbalance. Runtime introspection links rendered elements to source statements through tool-specific hooks such as get_window_extent() for matplotlib and getBoundingClientRect() for web/UI, with a Qwen3-VL-based fallback mapping when direct instrumentation is unavailable. The resulting IoU-based responsibility weights amplify gradients on code statements that control defective visual regions (Dong, 9 Jun 2026).

SelfEvo makes the anchor temporally stronger rather than spatially localized. The teacher sees a richer context window with $x$6 frames, while the student operates on $x$7 frames produced by frame dropping. The paper reports that frame dropping is the strongest and most reliable asymmetry, that random student frame selection is best overall, and that freezing the camera decoder provides the best stability–plasticity trade-off (Huang et al., 9 Apr 2026).

Vision-OPD constructs privileged regional evidence through a crop-evidence pipeline. Object recognition and segmentation propose candidate boxes, only small regions with area ratio $x$8 are retained, the crop is resized by $x$9, and the original image is overlaid with the bounding box plus a spatial instruction such as “Only focus on the objects inside the red bounding box.” This yields triplets $\pi_\theta$0; the paper uses 6.2K synthetic samples (Yuan et al., 18 May 2026).

ATAS uses multi-level and multi-scale anchors. Its hierarchy spans patch tokens across transformer blocks, dense features from MaskCLIP-style or ClearCLIP-style heads, the global CLS token, and multi-scale views induced by 2×2, 4×4, and 6×6 mosaics. The stated objective is to enhance semantic coherence and fine-grained vision-language alignment jointly, while preserving global CLIP semantics (Yeo et al., 10 Jun 2025).

Masking-KD anchors reasoning on visual evidence by selectively hiding the student’s own salient reasoning prefixes. Saliency is derived from the response-to-response attention map, top-$\pi_\theta$1 selection is used to form $\pi_\theta$2, the immediate previous token $\pi_\theta$3 is excluded from masking, and the cumulative masking threshold is scheduled as

$\pi_\theta$4

with $\pi_\theta$5 and $\pi_\theta$6. Only response-to-response attention is masked; question and visual tokens remain accessible (Yu et al., 12 May 2026).

5. Reported empirical behavior

In autonomous driving, EvoDriveVLA reports state-of-the-art open-loop performance on nuScenes. Under ST-P3, Avg L2 is $\pi_\theta$7 m and Avg Collision is $\pi_\theta$8; under UniAD, Avg L2 is $\pi_\theta$9 m and Avg Collision is $v$0. The ablation isolating self-anchored visual distillation on UniAD metrics shows Avg L2 improving from approximately $v$1 m to approximately $v$2 m, with 3 s L2 improving from $v$3 to $v$4. In NAVSIM, EvoDriveVLA improves PDMS by 3.4 points over the base 3B model and surpasses larger 8B baselines (Cao et al., 10 Mar 2026).

In code-generated visual artifacts, Visual-SDPO improves over the zero-shot base by more than 10 absolute points in the primary metric on all three reported benchmarks and exceeds GRPO by at least 2.4 points: ChartMimic Overall rises from 67.9 to 78.6, Design2Code Overall from 72.1 to 82.6, and AeSlides Avg from 49.5 to 60.7. The paper also reports that Visual-SDPO achieves GRPO-level performance with about 29% of GRPO’s rollout budget on average, attributing the gain to dense, localized token-level supervision (Dong, 9 Jun 2026).

In 4D perception, SelfEvo reports up to 36.5% relative improvement in video depth estimation and 20.1% in camera estimation without labeled data. On OmniWorld-Game with VGGT, OmniGeo depth Abs Rel (scale) improves from 0.346 to 0.278 and camera AUC@15 from 64.659 to 79.034; online EMA supervision is substantially stronger than offline pseudo-labeling, and similar gains are reported on OmniVideo, BEDLAM2.0, DROID, and unseen domains such as HOI4D (Huang et al., 9 Apr 2026).

In fine-grained multimodal understanding, Vision-OPD-9B reports V* 94.76, ZoomBench 65.80, HR 4K 88.13, HR 8K 85.50, MME-RW Lite 49.56, MME-RW CN 70.46, and Average 75.70. Relative to the base Qwen3.5-9B, the average gain is +6.54 points, with +13.73 on ZoomBench. Vision-OPD-9B also surpasses Qwen3.5-397B on Average 75.70 vs 72.54 and ZoomBench 65.80 vs 57.16, and exceeds Gemini-3.1-Pro on average 75.70 vs 74.74. The paper’s ablations identify EMA teacher regularization, JSD with $v$5, 1024-token rollouts, and top-$v$6 logits distillation as the best-performing configuration (Yuan et al., 18 May 2026).

In open-vocabulary dense prediction, ATAS reports Pascal VOC patch-level classification accuracy via text embeddings of 82.46, compared with 69.48 for CLIP and 70.66 for CLIPSelf. On zero-shot open-vocabulary semantic segmentation with ClearCLIP, VOC20 mIoU rises to 82.4 from 80.9 for CLIP and 74.4 for CLIPSelf. In open-vocabulary object detection, F-ViT + ATAS reports OV-COCO $v$7 of 46.4 versus 42.5 for F-ViT + CLIPSelf, and OV-LVIS mAP of 20.7 versus 20.1 (Yeo et al., 10 Jun 2025).

In think-answer VLM distillation, Masking-KD reports average scores over seven benchmarks of 56.66 for 8B→4B and 47.34 for 8B→2B, compared with 48.89 and 38.26 for naive response distillation. In self-distillation, Masking-KD-8B reaches 58.06 versus 53.65 for the Qwen3-VL-8B-Thinking base, and Masking-KD-4B reaches 56.01 versus 44.11 for the 4B base. The ablations show that masking response prefixes is better than masking visual or question tokens, high-attention prefixes are better than random or low-attention prefixes, reverse KL is better than forward KL or mixed KL, and excluding the immediate previous token is a major stabilizer (Yu et al., 12 May 2026).

Earlier online self-supervised evidence from MOKD is consistent with the same pattern: using EMA-based self-anchors plus semantically aligned cross-distillation improves both models in heterogeneous pairs, for example ResNet-50 + ViT-B improving linear probing from 72.1% to 75.6% for R50 and from 77.0% to 78.0% for ViT-B (Song et al., 2023).

6. Misconceptions, limitations, and open problems

A common misconception is that visual distillation always means copying hidden states from a stronger external teacher. The surveyed methods contradict that assumption. EvoDriveVLA uses the model’s own frozen pre-trained encoder as the teacher; Visual-SDPO uses a weight-sharing teacher with privileged rendered feedback from the student’s own rollout; ATAS uses the model’s own multi-level features; Vision-OPD uses the same MLLM under crop-conditioned and full-image-conditioned views; and Masking-KD shapes the student by modifying its own accessible context rather than by directly matching teacher visual internals (Cao et al., 10 Mar 2026, Dong, 9 Jun 2026, Yeo et al., 10 Jun 2025, Yuan et al., 18 May 2026, Yu et al., 12 May 2026).

The limitations are similarly heterogeneous. EvoDriveVLA can over-constrain adaptation if $v$8 is too large, and its trajectory-guided weighting can be misguided when trajectory supervision is erroneous or low-confidence (Cao et al., 10 Mar 2026). Visual-SDPO depends on the quality of defect detection and region-to-code mapping, and renderer-specific instrumentation may not generalize without re-instrumentation or detector adaptation (Dong, 9 Jun 2026). SelfEvo relies on a non-trivial starting prior, may improve less under static cameras or very low motion, and still faces drift risk under extended training without labels (Huang et al., 9 Apr 2026). Vision-OPD is sensitive to crop selection errors and naive self-distillation collapses when the current policy is used as the teacher without regularization (Yuan et al., 18 May 2026). ATAS incurs computational overhead from any-to-any correspondences and mosaic augmentation, and unlabeled training can create ambiguous positives (Yeo et al., 10 Jun 2025). Masking-KD adds an auxiliary forward pass and full response-to-response attention extraction, making training slower and more memory-intensive; it is also sensitive to teacher quality and over-masking (Yu et al., 12 May 2026). MOKD, finally, can suffer negative transfer when cross-distillation weights are mis-set or when models are too heterogeneous (Song et al., 2023).

Taken together, these works indicate that the distinctive contribution of self-anchored visual distillation is not a fixed loss family but a shift in where supervision is sourced. The anchor can be a frozen pre-trained manifold, privileged visual feedback, richer spatiotemporal context, regional evidence, internal feature correspondences, or student-side saliency. The shared objective is to preserve or transfer the model’s own stronger visual condition into a weaker operating condition without incurring inference-time dependence on the anchor itself (Cao et al., 10 Mar 2026, Dong, 9 Jun 2026, Huang et al., 9 Apr 2026, Yuan et al., 18 May 2026, Yeo et al., 10 Jun 2025, Yu et al., 12 May 2026).