CompoDistill: Compositional Distillation

• CompoDistill is a dual-domain framework that distills compositional knowledge in both vision-language models and audio-video understanding systems.
• It employs innovative modules such as Visual Attention Alignment (VAT) and compositional contrastive learning to effectively transfer cross-modal features.
• Experimental results demonstrate significant improvements in compositional reasoning and video retrieval, validating its practical impact on multimodal tasks.

CompoDistill refers to two independent lines of research that share the same designation but differ fundamentally in domain, architectural focus, and distillation objectives. The first, "CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMs" (Kim et al., 14 Oct 2025), advances knowledge distillation in vision-LLMs by aligning intermediate visual attention for robust compositional reasoning. The second, "Distilling Audio-Visual Knowledge by Compositional Contrastive Learning" (Chen et al., 2021), introduces compositional contrastive learning for transfer across heterogeneous modalities such as audio, image, and video within the context of video understanding. Both works are central to recent progress in knowledge distillation for multimodal learning, but apply distinct mechanisms and theoretical insights.

1. Motivation and Problem Formulation

Multimodal LLMs: Visual Attention Alignment

Traditional knowledge distillation (KD) approaches in Multimodal LLMs (MLLMs) effectively transfer recognition skills (e.g., object labels), but they are inadequate for compositional reasoning tasks that demand rich visual perception and relational understanding. Systematic analysis reveals that the student’s attention maps in critical intermediate layers are misaligned with those of the teacher—a phenomenon termed visual-attention misalignment. This misalignment is empirically connected to reduced performance on compositional reasoning (CR) benchmarks, as the student attends to irrelevant visual regions despite textual cues (Kim et al., 14 Oct 2025).

Video Understanding: Cross-Modal Knowledge Distillation

In audio-visual video understanding, conventional KD methods are constrained by unimodal assumptions, often failing to exploit complementary information from modalities with weak or no semantic correlation (e.g., background music overlaying an action). The compositional contrastive learning approach of CompoDistill (Chen et al., 2021) bridges this gap by synthesizing privileged representations from heterogeneous teachers (audio and image) into a video student. The centerpiece is a learnable compositional embedding that adapts teacher features to the video domain, facilitating cross-modal transfer even in the absence of tight alignment between modalities.

2. Core Methodological Innovations

Visual Attention Distillation for MLLMs

CompoDistill (Kim et al., 14 Oct 2025) introduces an explicit attention-alignment objective targeting intermediate "visual-understanding" layers (30–70% network depth). The framework extracts the student’s and teacher’s attention over visual-key positions as submatrices $\tilde A_l \in \mathbb{R}^{(N_v+N_t) \times N_v}$ from the transformer attention matrix:

$$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$

Given student (S layers) and teacher (T layers), student layer $l^s_j$ is matched to the average attention of a group of $n=T-S+1$ consecutive teacher layers $G_j$:

$$\bar{A}^t_j = \frac{1}{n}\sum_{l \in G_j} \tilde{A}^t_l$$

$$\mathcal{L}_{ADL} = 1 - \frac{1}{S}\sum_{j=1}^S \mathrm{cosine\_sim}\bigl(\bar{A}^t_j, \tilde{A}^s_{l^s_j}\bigr)$$

This loss is jointly optimized with logit-level KL divergence and language-modeling cross-entropy:

$$\mathcal{L} = \mathcal{L}_{LM} + \mathcal{L}_{KL} + \mathcal{L}_{ADL}$$

Two architectural augmentations are introduced:

1. Visual Attention Alignment (VAT) module: Implements $\mathcal{L}_{ADL}$ over matched student–teacher layer groups.
2. Teacher Adapter Fetch (TAF) module: The student reuses the teacher’s frozen adapter $P^t_{\psi_t}$, adding a trainable MLP $$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$0 for dimensionality matching, thus maintaining invariant feature spaces for attention computation:

$$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$1

Compositional Contrastive Learning for Video

The compositional version of CompoDistill for video (Chen et al., 2021) constructs residual-fusion modules $$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$2 and $$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$3 for audio-video and image-video pairings. Each module fuses the normalized teacher ($$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$4 or $$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$5) and student ($$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$6) embeddings:

$$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$7

$$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$8

A multi-class noise-contrastive estimation (NCE) loss aligns class semantics, while classifier constraints enforce semantic consistency between $$A = \mathrm{softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)$$9, $l^s_j$0, and $l^s_j$1. Prediction-space alignment is added via Jensen–Shannon divergence between output distributions.

$l^s_j$2

At test time, only the video network is used.

3. Training Procedures and Architectural Integration

Three-Stage Training in MLLMs

CompoDistill (Kim et al., 14 Oct 2025) employs a sequential three-stage training pipeline:

1. Distillation Pre-Training (DPT): Freeze vision encoder and LLM. Train only $l^s_j$3 with $l^s_j$4.
2. Distillation Fine-Tuning (DFT): Freeze vision encoder. Fine-tune LLM and $l^s_j$5 with $l^s_j$6.
3. Supervised Fine-Tuning (SFT): Freeze vision encoder. Fine-tune LLM and $l^s_j$7 with only $l^s_j$8 to reinforce instruction-following.

These choices ensure effective propagation of visual perception semantics, while preventing format collapse and maximizing answer probability calibration.

Training Protocol for Video Distillation

In (Chen et al., 2021), all teacher networks are frozen. Only the video student and MLP compositional modules are optimized. Class-aware sampling, batch normalization, and contrastive learning hyperparameters (e.g., temperature, mixing rate $l^s_j$9) are strictly controlled to balance teacher and compositional objectives. Inference requires only the video model, ensuring test-time efficiency.

4. Experimental Results and Quantitative Impact

Multimodal Reasoning in MLLMs

CompoDistill (Kim et al., 14 Oct 2025) demonstrates significant quantitative gains on CR and VQA benchmarks:

Model	VQA avg	CR avg
LLaVA-2B (SFT)	54.9	60.7
LLaVA-KD-2B	61.6	61.5
LLaVA-MoD-2B	58.9	62.6
CompoDistill-2B	61.9	66.7

Notably, CompoDistill closes much of the gap to larger teacher models, improving compositional reasoning by over 5 percentage points and substantially reducing relational hallucination F1 on R-Bench and Reefknot. Ablations confirm the necessity of both VAT and TAF modules, as well as grouped layer-matching and the use of cosine similarity in attention loss.

Scaling experiments show robust gains: doubling training data improves CR by 3.2 points; distillation from larger teachers yields predictable improvements. The approach generalizes across different backbones, for example, raising MobileLLaMA VQA from 49.7 to 53.1 and CR from 43.7 to 48.9.

Audio-Visual Distillation in Video

For cross-modal video understanding (Chen et al., 2021), compositional contrastive CompoDistill outperforms conventional KD and contrastive learning baselines on UCF51 and ActivityNet:

Method	UCF51 Top-1 (%)	ActivityNet Top-1 (%)
No distill (video baseline)	57.5	32.6
Audio-only distillation	64.9	-
Image-only distillation	69.1	-
Audio+Image (CompoDistill)	70.0	47.3

On retrieval (R@1), CompoDistill advances from 57.3% to 67.6% (UCF51) and 29.0% to 39.5% (ActivityNet). Ablations support the necessity of compositional modules, multi-class NCE, and dual-space prediction alignment. Analysis shows superior performance on semantically weakly correlated classes and improved cluster purity in learned embedding space.

5. Mechanistic Analyses and Ablation Findings

• VAT vs TAF (MLLMs): VAT alone boosts CR to 65.0; TAF alone enhances VQA to 61.3 and CR to 63.8; their combination maximizes both.
• Loss configuration (MLLMs): Cosine-similarity outperforms KL or MSE for attention alignment; grouping over intermediate layers is optimal.
• Recipe ablations: Omitting SFT in MLLMs triggers answer formatting collapse; removing DFT and $n=T-S+1$0 eliminates CR advantage.
• Attention–Answer Correlation: Higher teacher–student attention similarity predicts higher answer probability (GQA). Inference-time teacher-attention substitution yields measurable CR improvement.
• Compositional fusion (Video): Removing fusion losses or compositional modules yields consistent drops (~2–4%).

6. Generalizability and Practical Implications

Both CompoDistill frameworks exhibit strong generalizability:

• In MLLMs, compositional distillation is robust to data scale, teacher capacity, and backbone swaps (Kim et al., 14 Oct 2025).
• For video, the recipe works across UCF51, ActivityNet, and VGGSound, with the largest audio–image improvements on the weakly correlated classes (Chen et al., 2021).

For practitioners, CompoDistill enables resource-efficient deployment of highly capable multimodal models without incurring the computational and memory footprint of large teachers. The explicit attention alignment (MLLMs) and residual compositional embeddings (video) are crucial for achieving transfer of deep perception/semantic skills, not just surface-level recognition.

7. Connections to Related Research and Limitations

The attention alignment scheme in (Kim et al., 14 Oct 2025) is not directly comparable to program-of-thought or chain-of-thought distillation in LLMs (e.g., Mixed Distillation (Li et al., 2023)), as it focuses on fine-grained visual-textual cross-representation rather than reasoning step transfer. Compositional contrastive learning (Chen et al., 2021) is likewise distinct from other knowledge fusion techniques by enabling transfer across semantically heterogeneous, unpaired modalities.

Limitations include:

• Evaluation restricted to English and image–text modalities (MLLMs), or RGB–audio–image (video).
• Structural dependence on teacher adapter in MLLMs.
• Absence of systematic analysis on whether these mechanisms apply to non-vision, non-audio modalities or sub-1B models.

A plausible implication is that CompoDistill’s mechanistic focus on intermediate representational alignment could inform future work in multi-modal and multi-step reasoning distillation beyond current vision-language and audio-video pipelines.