Visual-Language Model Distillation

• Visual-language model knowledge distillation is the process of transferring multimodal competence from a high-capacity teacher to a smaller student by aligning visual and textual signals.
• Key paradigms include output-distribution matching, intermediate representation alignment, and synthetic supervision to selectively capture high-value visual cues.
• Techniques such as Visual-Advantage On-Policy Distillation dynamically weight tokens based on fine-grained visual detail, improving both accuracy and visual grounding.

Searching arXiv for the focal paper and closely related VLM distillation work to ground citations. Searching arXiv for "Visual-Advantage On-Policy Distillation for Vision-LLMs" and related VLM KD papers. Visual-LLM knowledge distillation is the transfer of multimodal competence from a stronger teacher to a smaller or otherwise cheaper student in settings where visual and textual information must be jointly processed. In the literature represented here, the term covers several distinct regimes: compression of region-based visual-linguistic transformers through aligned intermediate-layer distillation (Fang et al., 2021), synthetic-data or “symbolic” distillation for region-conditioned reasoning (Park et al., 2023), multilingual CLIP compression through staged feature distillation and alignment (Zhang et al., 2024), cross-modal alignment distillation for mobile VLMs (Feng et al., 2024), semi-supervised distillation from frozen VLM teachers into compact classifiers (Kang et al., 12 May 2025), and, more recently, on-policy distillation tailored to sparse visual grounding in generative VLMs (Liu et al., 21 May 2026). Across these variants, the central problem is consistent: multimodal knowledge is not uniformly distributed across layers, tokens, modalities, or training examples, so effective distillation requires identifying where the transferable signal actually resides and preventing it from being diluted.

1. Conceptual scope and task regimes

Visual-LLM knowledge distillation is broader than conventional teacher-student compression. In some settings the teacher and student are both multimodal generators or instruction-following models, and the objective is to preserve multimodal reasoning or visual grounding under a large capacity reduction (Liu et al., 21 May 2026). In others, the teacher is a CLIP-style dual encoder and the student is a smaller multilingual or task-specific encoder trained to retain cross-modal alignment with less compute (Zhang et al., 2024). There are also regimes in which the teacher is not a VLM at all: a language-only model can supervise a vision-only student through class-level conceptual signatures (Liang et al., 25 Jun 2026), and a text-only LM can recover linguistic competence in a VLM student via selective cross-modal distillation (Irawan et al., 1 Apr 2026). These variants suggest that “visual-LLM knowledge distillation” is best understood as a family of cross-model transfer procedures centered on multimodal competence rather than a single loss class.

A second axis of variation concerns what is being transferred. Some methods distill output distributions, such as reverse-KL over teacher and student token distributions on student-generated rollouts (Liu et al., 21 May 2026), reverse KL over VLM outputs in compact LLaVA-style students (Gkalelis et al., 11 May 2026), or KL over CLIP-derived class probabilities in semi-supervised classification (Kang et al., 12 May 2025). Other methods distill intermediate representations: attention maps and token-aligned hidden states in region-based VL transformers (Fang et al., 2021), first-layer text-query-to-vision attention and projector outputs in mobile VLMs (Feng et al., 2024), or teacher image and text branch features in multilingual CLIP compression (Zhang et al., 2024). A further class distills through synthetic supervision rather than direct teacher matching, as in localized commonsense QAR generation for region-conditioned visual reasoning (Park et al., 2023) or instance-specific text supervision for long-tail recognition (Zhang et al., 2024).

A third axis is the form of multimodal mismatch. In region-based visual-linguistic transformers, the main obstacle is that teacher and student may process different region proposals, making token-wise hidden-state matching ill-posed unless proposal geometry is aligned (Fang et al., 2021). In modern generative VLMs, the problem can instead be that visually relevant supervision is sparse within long multimodal chains of thought, so uniform token averaging over the whole sequence weakens the grounding signal (Liu et al., 21 May 2026). In multilingual CLIP compression, the difficulty is preserving modality-specific representation quality while re-establishing bilingual image-text alignment after branchwise compression (Zhang et al., 2024). These are different failure modes, but they all imply that multimodal KD cannot usually be reduced to direct imitation of a teacher’s final predictions.

2. Why multimodal distillation is harder than unimodal distillation

The literature repeatedly treats multimodal KD as structurally harder than standard KD because the teacher and student often do not observe commensurate inputs or do not encode the same information in comparable locations. In region-based visual-linguistic transformers, different detectors generate different object proposals, region orderings, and feature sets, so teacher token $i$ and student token $i$ need not correspond semantically (Fang et al., 2021). This is why aligned proposals become a prerequisite for meaningful attention and hidden-state distillation in that setting. The same paper explicitly notes that without semantic correspondence across teacher and student tokens, it is only safe to use classification distillation on masked-token logits (Fang et al., 2021).

In generative VLMs, the obstacle is not detector mismatch but supervision sparsity. “Visual-Advantage On-Policy Distillation” argues that in multimodal chain-of-thought outputs, most tokens are language scaffolding, connective phrases, or algebraic rewrites, whereas only a small minority depend on fine-grained visual detail such as OCR text, diagram labels, symbols, or exact numeric values (Liu et al., 21 May 2026). The paper operationalizes “fine-grained visual detail” by constructing a degraded image $\tilde v$ that preserves coarse structure but destroys OCR-scale and symbol-scale content through downsampling to $10\%$ spatial resolution with bilinear interpolation and nearest-neighbor upsampling (Liu et al., 21 May 2026). It then shows that standard on-policy distillation can raise accuracy while leaving the student’s visual dependence on its own rollouts nearly flat, so better outputs do not necessarily indicate stronger visual grounding (Liu et al., 21 May 2026).

Other settings expose different forms of modality-specific misalignment. MobileVLM distillation work argues that the most important knowledge to transfer is the teacher’s shallow-layer cross-modal alignment mechanism, specifically how text tokens attend to vision tokens in the first language-model layer (Feng et al., 2024). Switch-KD criticizes prior VLM KD for maintaining independent teacher and student forward passes and interacting only at the final loss level, which, in its framing, fails to preserve the teacher’s fused multimodal reasoning process (Sun et al., 16 Apr 2026). LinguDistill identifies a different pathology: during multimodal adaptation, a pretrained LM backbone can lose linguistic competence because visual alignment perturbs the language representation space, and standard continued training on language-heavy data only partially repairs the damage (Irawan et al., 1 Apr 2026). Across these cases, the general lesson is that multimodal KD must reason explicitly about where the multimodal signal is fused and how it is degraded.

3. Principal distillation paradigms

The field can be organized into a small number of recurring paradigms. The differences are not merely implementation details; they reflect incompatible assumptions about what “knowledge” is in a VLM and how it should be transferred.

Paradigm	Representative methods	Distilled signal
Output-distribution distillation	VA-OPD (Liu et al., 21 May 2026), Switch-KD (Sun et al., 16 Apr 2026), LLaVA-CKD (Gkalelis et al., 11 May 2026), DHO (Kang et al., 12 May 2025)	Token or class distributions
Intermediate-representation distillation	DistillVLM (Fang et al., 2021), Align-KD (Feng et al., 2024), DC-CLIP (Zhang et al., 2024)	Attention maps, hidden states, feature branches
Synthetic-supervision distillation	LSKD (Park et al., 2023), VLM-KD (Zhang et al., 2024)	Generated QARs or free-form text
Cross-modal or selective transfer	LinguDistill (Irawan et al., 1 Apr 2026), ARMADA (Sengupta et al., 11 Mar 2026), LaViD (Liang et al., 25 Jun 2026)	Source-conditioned or cross-modal semantic structure

Direct output-distribution matching remains the simplest regime. In VA-OPD, the student samples its own rollouts on-policy and the teacher scores those trajectories token by token; distillation then uses reverse KL on the original image while modulating the weight of rollouts and token groups using Visual Advantage (Liu et al., 21 May 2026). LLaVA-CKD repeats a LLaVA-KD-style distillation pipeline across teacher assistants of increasing capacity, combining autoregressive loss with textual KL, visual KL, and visual correlation alignment, so the student is distilled sequentially by progressively stronger teachers (Gkalelis et al., 11 May 2026). Switch-KD reuses teacher language layers with student visual outputs to construct a shared text-probability supervision space and then applies DBiLD in addition to ground-truth cross-entropy (Sun et al., 16 Apr 2026). DHO, though aimed at semi-supervised classification, distills VLM prompt-based class probabilities into a compact student via a dedicated KD head while a separate supervised head handles labeled data (Kang et al., 12 May 2025).

Intermediate-representation distillation is more common when teacher and student architectures are sufficiently structured to admit correspondence. DistillVLM aligns teacher and student visual tokens by re-extracting teacher features on student detector proposals, then applies MSE on attention maps and a token-wise noise-contrastive hidden loss on aligned hidden states (Fang et al., 2021). Align-KD distills the first-layer text-query-to-vision attention block $A_{1,t-v}$, text-focused top-$K$ visual token embeddings, and all visual projector outputs, arguing that the first layer is where cross-modal alignment primarily occurs (Feng et al., 2024). DC-CLIP instead distills image and text branches separately with Smooth L1 feature regression and then restores bilingual image-text alignment through a second-stage contrastive objective (Zhang et al., 2024).

Synthetic-supervision methods depart further from classical KD. LSKD converts images and regions into textual descriptors, asks ChatGPT to generate localized commonsense QARs, filters them with a critic, and then fine-tunes a student VLM on the resulting corpus (Park et al., 2023). The distilled object is not logits or hidden states but localized symbolic commonsense knowledge tied to regions. VLM-KD similarly uses an off-the-shelf VLM to produce instance-specific text descriptions for each image in long-tail recognition, encodes those texts with a frozen text encoder, and trains a visual classifier with an image-text contrastive alignment loss in addition to long-tail objectives (Zhang et al., 2024).

4. Visual grounding, sparsity, and the VA-OPD formulation

The most explicit treatment of sparse multimodal supervision appears in “Visual-Advantage On-Policy Distillation for Vision-LLMs” (Liu et al., 21 May 2026). The paper studies on-policy KD, where the student first samples $K$ rollouts $\{y^{(k)}\}_{k=1}^{K} \sim \pi(\cdot \mid v,q)$, and the teacher then evaluates those exact student trajectories token by token (Liu et al., 21 May 2026). Standard OPD uses the reverse-KL objective

$$L_{\mathrm{KL}} = \frac{1}{T}\sum_{t=1}^{T} KL\!\left(p(\cdot \mid v, q, y_{<t}) \;\|\; \pi(\cdot \mid v, q, y_{<t})\right).$$

The paper’s diagnosis is that uniform $1/T$ averaging causes visually important tokens to be numerically overwhelmed by the much larger set of weakly visual tokens (Liu et al., 21 May 2026).

To expose which tokens truly depend on fine-grained visual input, the paper defines token-level Visual Advantage as

$i$0

where $i$1 is the original image and $i$2 is the degraded image with fine detail destroyed (Liu et al., 21 May 2026). High $i$3 means the teacher’s probability for the realized student token drops when fine-grained detail is removed; near-zero $i$4 means the token is recoverable from language context or coarse visual structure (Liu et al., 21 May 2026). The paper reports that on held-out student rollouts, the top $i$5 of tokens account for about $i$6 of total VA mass, making the signal strongly heavy-tailed (Liu et al., 21 May 2026).

VA-OPD uses this signal at two granularities. At the rollout level, each rollout receives a trajectory-average VA

$i$7

which is standardized within sibling rollouts and converted into soft weights $i$8 by a temperature-$i$9 softmax (Liu et al., 21 May 2026). At the token level, the top fraction $\tilde v$0 of tokens by VA forms the high-VA set $\tilde v$1, the remainder forms $\tilde v$2, and reverse KL is averaged separately in the two groups: $\tilde v$3 with $\tilde v$4 (Liu et al., 21 May 2026). The full objective for a prompt is

$\tilde v$5

The method is explicitly purely distillation-based: it adds no reward models, preference labels, task rewards, or auxiliary perception losses, only one extra teacher forward pass on $\tilde v$6 to compute VA (Liu et al., 21 May 2026).

The empirical pattern is unusually clean. In the main 8B$\tilde v$72B Qwen3-VL Geometry3K setup, standard OPD reaches Math average 45.4 and Visual average 64.6, while VA-OPD reaches 48.3 and 66.1, with gains on every benchmark: WeMath $\tilde v$8, MathVista $\tilde v$9, MathVerse $10\%$0, HallusionBench $10\%$1, AI2D $10\%$2, MMMU $10\%$3, MMStar $10\%$4, and OCRBench $10\%$5 (Liu et al., 21 May 2026). Gains also increase with teacher size and dataset size: on Geometry3K, 4B$10\%$62B yields $10\%$7 Math Avg and $10\%$8 Visual Avg over OPD, 32B$10\%$92B yields $A_{1,t-v}$0 and $A_{1,t-v}$1, and on ViRL39K with an 8B teacher the gain rises to $A_{1,t-v}$2 and $A_{1,t-v}$3 (Liu et al., 21 May 2026). A masking study further shows that removing KL supervision from the highest-VA $A_{1,t-v}$4 of tokens causes about a $A_{1,t-v}$5-point drop on MathVerse$A_{1,t-v}$6, whereas masking random or lowest-VA tokens has little effect (Liu et al., 21 May 2026). Under standard OPD, accuracy rises while mean rollout VA moves only from $A_{1,t-v}$7 to $A_{1,t-v}$8; under VA-OPD, accuracy rises higher and mean VA increases from $A_{1,t-v}$9 to $K$0, which the paper interprets as improved reliance on visual detail during generation (Liu et al., 21 May 2026).

5. Alignment-centric and structure-aware approaches

A large fraction of VLM distillation work can be read as attempts to preserve or recreate cross-modal alignment structure that is lost under compression. DistillVLM is an early, explicit statement of this problem for region-based VL transformers. Its central move is to adapt the teacher so that it consumes visual tokens indexed by the student detector’s region proposals while still using teacher-side detector features, thereby restoring token-level semantic correspondence (Fang et al., 2021). Once aligned, it becomes meaningful to apply attention MSE,

$K$1

and token-wise contrastive hidden-state alignment,

$K$2

in addition to standard MLM and ITM pretraining losses (Fang et al., 2021). The payoff is substantial in that regime: on COCO captioning, aligned distillation in pretraining improves CIDEr from 110.6 without distillation to 115.6 with adaptation, and the full paper reports 120.8 CIDEr and 69.8 VQA 2.0 accuracy, gains of 5.1 and 0.8 over non-distilled counterparts (Fang et al., 2021).

Align-KD moves the alignment question into the modern mobile-VLM setting. The paper argues that the most important knowledge to distill is the shallow-layer mechanism by which textual queries attend to visual tokens, and it therefore distills only the first-layer text-query-to-vision attention block $K$3, not the full attention matrix or a deep stack of layers (Feng et al., 2024). This selective focus is empirically justified: using the intended $K$4 gives the full 64.4 average on the Short subset, while replacing it with $K$5 drops performance to 43.7 and using full first-layer attention reaches only 63.5 (Feng et al., 2024). Align-KD also uses the same attention map to select the top-$K$6 most relevant visual tokens for projector-level distillation, combining all-token and focused visual losses with output-level reverse KL. On the Short subset, the 1.7B MobileVLM V2 student improves from average 62.4 to 64.4; on the Long subset, from 63.7 to 65.1 (Feng et al., 2024).

DC-CLIP solves a different alignment problem: compressing multilingual CLIP models. Its framework first distills image and text features separately using Smooth L1,

$K$7

and only then performs a bilingual image-text alignment stage with cosine-based bidirectional contrastive loss while freezing the image encoder and tuning only the text branch, following LiT (Zhang et al., 2024). The empirical evidence supports the progressive design: the pre-alignment model DC-CLIP-P is uniformly worse than DC-CLIP across the reported English and Chinese benchmarks, for example English ImageNet 41.912 $K$8 42.782 and Chinese Caltech101 73.59 $K$9 75.44 (Zhang et al., 2024). This suggests that branchwise teacher imitation alone does not preserve multimodal functionality; cross-modal realignment remains necessary after compression.

6. Synthetic supervision, selective transfer, and emerging design principles

A second major current in the literature treats distillation as the construction of a better supervision space rather than direct teacher imitation. LSKD is exemplary. It turns image-level VLMs into region-conditioned commonsense models by verbalizing images and regions, prompting ChatGPT to generate localized Question-Answer-Rationale triples, filtering them with a critic, and then fine-tuning BLIP-2 or Mini-GPT4 students on the resulting localized corpus (Park et al., 2023). The final dataset is about 1.02M QARs over 168,996 images, generated from a 250K-image seed pool and filtered by a critic that raises accepted-data precision from 45% baseline acceptance to about 70% when only the top 20% of instances are kept under a threshold of 0.8 (Park et al., 2023). On zero-shot localized reasoning, BLIP-2 ViT-G + LSKD improves over vanilla BLIP-2 ViT-G on all six tasks, including VCR Q$K$0AR 28.0 $K$1 33.4, Sherlock 19.5 $K$2 29.7, and VisualCOMET 39.0 $K$3 40.3 (Park et al., 2023). The method is notable because the distilled content is explicit localized symbolic commonsense rather than hidden states or logits.

Selective transfer has also become a major theme. LinguDistill uses the original frozen text-only LM backbone as a teacher for an already multimodal student and makes the teacher vision-aware through layer-wise KV-cache sharing, so teacher queries attend over the student’s multimodal keys and values without changing either architecture (Irawan et al., 1 Apr 2026). The paper then applies KD only on language-heavy sources and suppresses it on OCR/document-heavy sources where a text-native teacher is unreliable. In the best selective setting, the average on language-heavy benchmarks rises from 0.471 for the plain full fine-tuning baseline to 0.564, with specific gains such as AI2D 0.416 $K$4 0.507, COCO 0.673 $K$5 0.866, ScienceQA 0.592 $K$6 0.676, ARC Easy 0.540 $K$7 0.621, ARC Challenge 0.279 $K$8 0.318, and HellaSwag 0.326 $K$9 0.394, while preserving OCR/document performance much better than uniform KD (Irawan et al., 1 Apr 2026). The broader implication is that the value of a teacher is task-conditional even within a single multimodal training mixture.

More speculative but still well-supported extensions broaden what can count as the teacher. ARMADA distills from multimodal or generative black-box teachers into language-only students through a learned TS Aligner and shared-manifold alignment, reporting up to 3.4% improvement on language understanding tasks and 2.6% on generative reasoning without multimodal pretraining of the teacher for the student’s downstream tasks (Sengupta et al., 11 Mar 2026). LaViD uses a language-only LLM to generate class-discriminative multiple-choice questions, converts each class into a $\{y^{(k)}\}_{k=1}^{K} \sim \pi(\cdot \mid v,q)$0 conceptual signature, and trains a vision-only classifier to predict that signature as an auxiliary target; on CUB with ResNet-18, the score rises from 63.07 for the independent student to 70.15 under LaViD, and on Waterbirds worst-group accuracy improves from 14.29 to 55.39 (Liang et al., 25 Jun 2026). These examples suggest that multimodal KD increasingly includes cross-modal semantic transfer where the teacher’s modality differs from the student’s, provided the supervision is expressed in a task-relevant shared structure.

Several broader principles emerge from this literature. First, multimodal KD works best when it identifies where the transferable signal is concentrated: a minority of high-VA tokens (Liu et al., 21 May 2026), the first-layer text-query-to-vision attention block (Feng et al., 2024), aligned region proposals (Fang et al., 2021), or localized symbolic QARs rather than generic instructions (Park et al., 2023). Second, multimodal supervision is frequently sparse or heterogeneous, so averaging across all tokens, all teachers, or all tasks is often harmful. This suggests selective normalization, adaptive routing, or source-conditioned weighting as recurring design choices. Third, cross-modal alignment is rarely preserved automatically under compression; it must usually be reintroduced explicitly, whether through grouped token-level KL (Liu et al., 21 May 2026), proposal-aligned attention and hidden-state transfer (Fang et al., 2021), bilingual contrastive realignment (Zhang et al., 2024), or shallow attention distillation (Feng et al., 2024). Finally, many recent methods emphasize zero- or low-inference-overhead deployment: cached text prototypes in CLIP-style systems, no extra inference modules in selective recovery methods, and student-only deployment after teacher-assisted training (Feng et al., 2024, Irawan et al., 1 Apr 2026). A plausible implication is that future VLM distillation will be judged less by whether it uses KD in the narrow sense and more by whether it can expose the right multimodal supervision signal while keeping the deployed student architecturally simple.