Reinforced Multimodal Distillation

• Reinforced multimodal distillation is a training regime that couples teacher–student transfer with reinforcement decisions to optimize module, modality, or attention pathway selection.
• It employs adaptive policies such as multi-armed bandits and REINFORCE to determine the best combination of teacher signals, hidden states, or attention distributions.
• Key applications include model compression, multimodal knowledge graph reasoning, and LLM post-training, demonstrating improved efficiency and cross-modal alignment.

Reinforced multimodal distillation denotes a class of teacher–student training schemes in which multimodal knowledge transfer is coupled to a reinforcement-style decision process. In the recent literature represented by multimodal foundation model compression, multimodal knowledge graph reasoning, and multimodal LLM post-training, the transferred signal is not confined to output logits: it can include hidden states and attention scores at selected modules, soft label structure over target and non-target entities, or latent causal attention distributions along a student’s own trajectory (Liang et al., 2023, Zhao et al., 28 Jul 2025, Li et al., 4 Feb 2026). The unifying pattern is that distillation is made adaptive by optimizing a policy over modules, modalities, teachers, or internal attention.

1. Scope and conceptual variants

The label covers several technically distinct formulations. In Module-wise Adaptive Distillation, a large pretrained multimodal teacher model is compressed into a smaller student by deciding which architectural modules should be distilled more frequently. In DSoM for multimodal knowledge graph reasoning, a unimodal student is distilled from multimodal teachers, while a policy network selects the optimal subset of modality-specific teachers for each triple. In Reinforced Attention Learning, the student is a multimodal LLM whose final-layer causal attention is treated as a policy, and distillation is performed directly on attention distributions rather than only on token logits (Liang et al., 2023, Zhao et al., 28 Jul 2025, Li et al., 4 Feb 2026).

Framework	Distillation target	Reinforcement locus
OPTIMA	Output KL, hidden states, attention scores	Multi-armed bandit over module subsets
DSoM	Teacher logits with neighbor–non-neighbor decoupling	REINFORCE over non-empty teacher subsets
RAL	Causal attention distributions, optionally with GKD	Policy gradient over internal attention

A common misconception is that reinforced distillation is synonymous with reinforcement learning over generated token sequences. The cited work does not support that restriction. In RAL, the optimized action is “attend to position $i$” in the final Transformer layer; in OPTIMA, the action is selection of a module subset; in DSoM, the action is selection of a non-empty subset of teachers. Reinforcement is therefore applied to different control variables, depending on which component of the multimodal system is considered most consequential for transfer.

2. Distillation objectives and transferred signals

The classical point of departure is a task loss plus output-level knowledge distillation. OPTIMA writes the baseline objective as

$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$

and then extends it by matching intermediate hidden states and attention scores at selected layers. For a subset of layers $S$, the module-wise terms are

$$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$

$$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$

and the student loss becomes

$$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$

This makes internal multimodal representations explicit distillation targets rather than auxiliary diagnostics (Liang et al., 2023).

DSoM also begins from logits, but rejects single-target supervision as insufficient for multimodal knowledge graph reasoning. It defines teacher and student softened distributions $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$ and $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$, and replaces undifferentiated KD with Neighbor–Decoupled KD: $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$ Here the logits are decoupled into neighbor entities and non-neighbor entities, so that true-answer correlations and false-tail correlations are treated separately. The paper’s stated motivation is that multimodal soft labels provide rich supervision signals with subtle correlations among both target and non-target entities from multiple perspectives (Zhao et al., 28 Jul 2025).

RAL moves the transferred object even deeper into the model. At each generation step $t$, the teacher attention is $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$0 and the student attention is $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$1. The on-policy attention distillation loss is

$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$2

with $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$3 or $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$4. In the unified objective,

$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$5

attention-level transfer is combined with output-level distillation and reinforcement learning. The explicit claim is that transferring latent attention behaviors yields stronger cross-modal alignment than standard knowledge distillation (Li et al., 4 Feb 2026).

3. Reinforcement mechanisms and policy design

The reinforcement component differs sharply across the three formulations. OPTIMA casts module selection as a non-stationary multi-armed bandit. If a multimodal Transformer is partitioned into $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$6 modules, there are $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$7 arms, each arm corresponding to a nonempty subset of modules and hence a set of layers $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$8. Training is divided into rounds, exactly one arm is selected per round, and the reward is the reduction in the full distillation loss after $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$9 gradient steps: $S$0 Each arm has a Gaussian posterior $S$1, with $S$2 updated by an exponentially weighted moving average $S$3. The resulting Thompson-sampling variant prioritizes modules whose recent distillation has yielded larger loss decrements (Liang et al., 2023).

DSoM uses REINFORCE for sample-wise teacher combination. For each training triple $S$4, the state is the concatenated teacher-logit vector

$S$5

the action space is the set of all non-empty subsets of $S$6, and the policy network is

$S$7

If $S$8, the combined teacher logit is

$S$9

The reward is deliberately asymmetric: $$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$0 A baseline reward from using all teachers is subtracted to reduce variance, and the policy loss is

$$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$1

This design operationalizes the paper’s claim that not every modality helps every triple (Zhao et al., 28 Jul 2025).

RAL formulates final-layer causal attention as a discrete policy over previous positions. With attention logits

$$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$2

the policy is

$$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$3

The overall objective is $$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$4, optimized with REINFORCE and a sequence-level advantage $$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$5 computed via GRPO. RAL also adds an attention-level regularizer

$$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$6

using Jensen–Shannon Divergence. The total RL loss is

$$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$7

The stated shift is from optimizing what to generate to where to attend (Li et al., 4 Feb 2026).

4. Attention distillation in multimodal LLM post-training

RAL is the most direct instance of reinforced multimodal distillation at the level of latent inference behavior. The model assumes a frozen visual encoder $$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$8 and a trainable text transformer $$L_{\text{hidn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(H_t^\ell,H_s^\ell W_{\text{hidn}}^\ell),$$9. Images or video frames are converted into patch features $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$0, text tokens are embedded as $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$1, and the initial context is

$$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$2

At generation step $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$3, the state $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$4 consists of $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$5. The action $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$6 is a previous position in the causal prefix. Rewards are defined by exact answer matching and formatting compliance: $$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$7

$$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$8

$$L_{\text{attn}}^S(\theta_s,\theta_t)=\frac{1}{|S|}\sum_{\ell\in S}\mathrm{MSE}(A_t^\ell,A_s^\ell),$$9

The paper further reports stabilization choices: bounded symmetric JSD for attention divergence, an exponential moving average update for $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$0, gradient clipping for $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$1 and $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$2, and warm-starting from a supervised COT-adapted model. The attention policy uses the final layer’s averaged heads; the student is Qwen-2.5-VL-7B and the teacher in distillation is Qwen-2.5-VL-32B (Li et al., 4 Feb 2026).

Empirically, the reported image QA benchmarks are V* Bench, MMMU-Pro, MME, MuirBench, ChartQA, VizWiz, Blink, and CVBench; the video QA benchmarks are LongVideoBench, NExT-QA, Video-MME, Video-MMMU, LVBench, MVBench, and TempCompass. On selected image tasks, RAL+AttnDistill scores $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$3 on V*, $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$4 on MMMU-Pro, $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$5 on MME, and $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$6 on VizWiz, compared with $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$7, $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$8, $$L_{\text{total}}^S=L_{\text{train}}+\alpha_1 D_{KL}(f_s\parallel f_t)+\alpha_2L_{\text{hidn}}^S+\alpha_3L_{\text{attn}}^S.$$9, and $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$0 for GRPO. On selected video tasks, RAL+AttnDistill reaches $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$1 on LongVideoBench, $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$2 on NExTQA, $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$3 on VideoMME, and $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$4 on MVBench, versus $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$5, $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$6, $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$7, and $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$8 for GRPO. The ablations report stable peaks around $P^{tea}=\mathrm{softmax}(F^{tea}/\tau)$9 and $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$0, increasing margins over GRPO as image-token resolution grows from $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$1 to $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$2 and frame count grows from $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$3 to $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$4, and a “RAL-zero” variant that still outperforms GRPO on $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$5 long-video tasks and $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$6 image tasks. This supports the paper’s claim that pure attention optimization can unlock grounding gains even without explicit rationales.

5. Reinforced multimodal distillation for knowledge graph reasoning

DSoM situates reinforced multimodal distillation in multimodal knowledge graph reasoning, where the goal is to predict missing facts in an incomplete MKG by leveraging structural, visual, and textual information. The multimodal knowledge graph is defined over entities $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$7, relations $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$8, triples $P^{stu}=\mathrm{softmax}(F^{stu}/\tau)$9, and modalities $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$0. Each entity $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$1 has embeddings $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$2: $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$3 is learned from graph structure, $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$4 uses fixed ViT features with a trainable projection, and $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$5 uses fixed BERT features with a trainable projection. Three unimodal teachers, one per modality, are pre-trained with one-hot cross-entropy, while the student is unimodal and never observes raw images or text; it learns only from teacher logits (Zhao et al., 28 Jul 2025).

The framework’s central claim is twofold: dark knowledge from non-target entities is useful, and incompetent modalities can be harmful. The first point is implemented by neighbor–non-neighbor decoupling. For a query $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$6, the neighbor set is $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$7. The average neighbor probabilities define binary vectors $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$8 and $$L_{\text{NDKD}}=\alpha\, KL(\bar b^{tea}\parallel \bar b^{stu})+\beta\, KL(\tilde P^{tea}\parallel \tilde P^{stu}) \equiv \alpha L_{\text{NEKD}}+\beta L_{\text{NNKD}}.$$9, while non-neighbor probabilities are renormalized into $t$0 and $t$1. The resulting $t$2 separates neighbor and non-neighbor correlations. The second point is implemented by reinforced teacher combination, where the policy chooses among the $t$3 non-empty teacher subsets and is rewarded only when the chosen multimodal combination yields better one-hot loss than the student.

The experimental datasets are DB15K, MKG-W, MKG-Y, FB15K-237, WN18, and WN9, with metrics MRR and Hits@$t$4. The reported gains are specific: on DB15K, DSoM achieves $t$5 versus best prior $t$6 $t$7; on MKG-W, $t$8 versus $t$9 $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$00; on MKG-Y, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$01 versus $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$02 $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$03; on FB15K-237, Hits@10 is $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$04 versus $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$05; on WN18, Hits@1 is $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$06 versus best $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$07. In the reinforcement ablation, Teacher-Avg gives $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$08, a meta-learner (MoSE) gives $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$09, and DSoM-RC gives $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$10. In the KD ablation, Only RC yields $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$11, RC + vanilla KD yields $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$12, RC + DKD yields $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$13, and RC + NDKD yields $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$14. Robustness results include graceful degradation under up to $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$15 missing visual or textual modalities and strong parameter efficiency: DSoM with $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$16 $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$17 M params$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$18 outperforms MyGO $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$19 M params$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$20, and even at $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$21 $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$22 M params$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$23 DSoM reaches $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$24 MyGO. The paper further reports that policy reward $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$25 steadily increases over training, with best hyper-parameters around $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$26, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$27, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$28, and $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$29.

6. Adaptive module-wise distillation for multimodal foundation models

OPTIMA presents a different reinforced multimodal distillation regime, aimed at model compression rather than post-training or knowledge graph reasoning. The teacher is CoCa-Large with $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$30 M Transformer-layer parameters and three modules: image encoder $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$31 layers, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$32, text encoder $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$33 layers), and multimodal decoder $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$34 layers). The students are CoCa-Tiny$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$35, with $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$36 Transformer layers arranged $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$37 in image/text/multi and approximately $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$38 M parameters, and CoCa-Tiny$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$39, with $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$40 layers arranged $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$41 and approximately $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$42 M parameters. Student layers are initialized by uniformly sampling from a CoCa-Base pretrained model (Liang et al., 2023).

The technical premise is that some architecture components contribute more significantly to the student’s performance than others, so the frequency of distillation should be controlled adaptively rather than uniformly. Training is split into rounds; in each round exactly one arm $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$43, corresponding to a module subset $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$44, is distilled for $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$45 gradient steps. The reward is the clipped relative decrement of the full distillation loss, averaged across $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$46, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$47, and $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$48. Because arm values are non-stationary, OPTIMA uses modified Thompson sampling with an exponentially weighted reward mean. The paper notes that the frequency with which a module $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$49 is updated is approximately

$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$50

so high-reward modules receive more updates over time.

The experimental tasks are VQA 2.0, SNLI-VE, NLVR2, and MS COCO Caption. Baselines include uniform layerwise distillation, random-arm, fixed-arm, and prior vision-LLMs and distillation methods such as UNITER, OSCAR, ViLT, ALBEF, MiniVLM, DistilVLM, and DIDE. Median over $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$51 seeds, CoCa-Tiny$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$52 + OPTIMA improves over layerwise distillation by $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$53 on VQA, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$54 on SNLI-VE, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$55 on NLVR2, and $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$56 CIDEr on MS COCO Caption. CoCa-Tiny$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$57 shows smaller but consistent gains, including $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$58 on NLVR2. The reported systems also achieve $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$59–$$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$60 inference speedups over CoCa-Large. The paper identifies limitations as well: the assumption that arms are independent ignores covariance among modules, exploration cost grows with $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$61 when $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$62 is large, and hyper-parameters $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$63, $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$64, and $$\min_{\theta_s}\; L_{\text{train}}(\theta_s) + \alpha_1 D_{KL}(f_s(\theta_s)\parallel f_t(\theta_t)),$$65 require tuning for stability.

Taken together, these works show that reinforced multimodal distillation is not a single algorithmic recipe but a design space in which reinforcement-style adaptation determines what multimodal knowledge is transferred and when. One line uses bandits to allocate distillation effort across modules; another uses REINFORCE to exclude unhelpful modalities and exploit dark knowledge in multimodal knowledge graphs; a third distills latent attention policies on the student’s own trajectories. A plausible implication is that the decisive object of distillation in multimodal systems is often not the final prediction alone, but the routing structure—across modules, modalities, or attention paths—that determines how heterogeneous evidence is used.