EPIC: Progressive Consistency Distillation

• Progressive Consistency Distillation (EPIC) is a framework that progressively adapts to visual token compression in multimodal LLMs to smooth optimization and enhance efficiency.
• It decomposes compression-induced perturbations into token-wise and layer-wise components while utilizing a shared-weight teacher-student setup for dual forward passes.
• The method is architecture-agnostic and parameter-free, employing a curriculum-based approach to maintain accuracy during substantial visual token reduction.

Searching arXiv for the primary paper and closely related progressive distillation work. Progressive Consistency Distillation (EPIC) is a progressive distillation framework for efficient multi-modal LLMs (MLLMs) under visual token compression. It is presented in “Efficient Multi-modal LLMs via Progressive Consistency Distillation” (Wen et al., 1 Oct 2025), where the central problem is that visual tokens consume substantial computational resources and that compression, although beneficial for efficiency, perturbs the feature space in ways that make direct training difficult. EPIC addresses this by decomposing the perturbation induced by token compression along token-wise and layer-wise dimensions, and by using a teacher–student setup in which a single MLLM with shared weights performs two forward passes under different compression ratios. The method is architecture-agnostic and parameter-free, and is designed to reduce optimization instability while preserving accuracy under substantial reductions in the number of visual tokens (Wen et al., 1 Oct 2025).

1. Problem setting and motivation

In MLLMs, the visual modality contributes hundreds to thousands of tokens, and the sequence length becomes the dominant compute and memory bottleneck because attention has quadratic complexity. The motivating example given for LLaVA-v1.5 is 576 tokens for a single CLIP ViT-L/14 image, with the burden becoming more acute for high-resolution or multi-frame inputs (Wen et al., 1 Oct 2025).

Token compression reduces the number of visual tokens fed to the LLM, but it also changes the distribution of the input representation. In the formulation underlying EPIC, this induces perturbations in the model’s feature space and shifts the optimum in parameter space. Large compression ratios induce higher perturbations, making direct training from the original optimum to the new compressed optimum challenging and prone to poor local minima (Wen et al., 1 Oct 2025). This framing motivates a progressive training strategy rather than one-shot adaptation to heavy compression.

The teacher–student structure in EPIC is unusual in that a single MLLM $f_\theta$ plays both roles through shared weights. For each training example, it performs two forward passes with different compression ratios: the teacher uses a slightly lower compression ratio than the student, retaining more visual tokens and thereby supplying an “easier” target. The compression gap between teacher and student is increased progressively during training, which is explicitly linked to the zone of proximal development principle in the source description (Wen et al., 1 Oct 2025).

A related theoretical motivation comes from “Progressive distillation induces an implicit curriculum” (Panigrahi et al., 2024). That work argues that staged supervision from intermediate teacher checkpoints can expose easier subproblems and shorter-context features that are not recoverable from the final teacher alone. This suggests a broader rationale for EPIC’s progressive design: progressive supervision can smooth optimization not only by reducing instantaneous task difficulty, but also by structuring the sequence of learned features (Panigrahi et al., 2024).

2. Core framework and decomposition of perturbations

EPIC decomposes the effect of token compression along two dimensions. The first is token-wise perturbation: changing the sequence length from $N$ to $N_s$ alters token-level context and attention patterns, which shifts the model’s optimum. The second is layer-wise perturbation: the effect of removing visual tokens depends on where in the LLM stack compression is applied, and prior work is cited in the source description as showing that visual tokens are less attended in deeper layers, so compression there induces a smaller effective perturbation (Wen et al., 1 Oct 2025).

This decomposition yields two variants. Token Consistency Distillation (TCD) addresses increasing compression ratios while keeping the compression layer fixed. Layer Consistency Distillation (LCD) addresses the location of compression by progressively moving the compression site from deeper layers to shallower layers over training. In both cases, the distilled signal is the final output distribution over the vocabulary, and the loss is a KL divergence between teacher and student output distributions after temperature-scaled softmax (Wen et al., 1 Oct 2025).

A defining property of EPIC is that distillation is performed at the output distribution level rather than through intermediate token-feature matching. The method explicitly avoids token-level alignment or bipartite matching, which reduces overhead and preserves compatibility with diverse compression operators (Wen et al., 1 Oct 2025). This output-level formulation is one of the reasons the framework is described as plug-and-play with pruning, merging, pooling, and related compression mechanisms.

The architecture-agnostic and parameter-free character of EPIC is central to its positioning. The framework does not modify the MLLM architecture, introduces no new modules, and adds no extra parameters. It is described as compatible with standard vision encoders such as CLIP ViT, modality projectors such as MLPs, any LLM backbone, and plug-and-play token compression operators with negligible integration effort (Wen et al., 1 Oct 2025).

3. Formal definitions and objectives

The notation in the source paper defines image $\mathcal{I}$, prompt $\mathcal{P}$, visual encoder $\mathrm{VE}$, projector $\mathrm{MLP}$, and LLM $\mathrm{LM}$. The visual encoder produces patch features $\mathbf{z}_v \in \mathbb{R}^{N \times d_v}$, which are mapped into the LLM hidden space as visual tokens $\mathbf{e}_v \in \mathbb{R}^{N \times d_h}$:

$N$0

With text embeddings $N$1, the input to the LLM is $N$2. Let $N$3 be the number of transformer layers in the LLM, and let $N$4 denote a token compression operator with compression ratio $N$5 applied at layer $N$6, where a fraction $N$7 of visual tokens is retained (Wen et al., 1 Oct 2025).

For TCD, at training iteration $N$8, the student compression ratio is sampled from a progressive range:

$N$9

where $N_s$0 increases from a small value such as $N_s$1 up to $N_s$2 such as $N_s$3, while $N_s$4 increases more slowly, from $N_s$5 toward at most $N_s$6. The teacher ratio is

$N_s$7

where the teacher–student gap $N_s$8 increases progressively from $N_s$9 to $\mathcal{I}$0 (Wen et al., 1 Oct 2025).

At a fixed transformer layer $\mathcal{I}$1, the shared-weights teacher and student produce logits

$\mathcal{I}$2

which are transformed to distributions

$\mathcal{I}$3

The token-wise consistency objective is

$\mathcal{I}$4

For LCD, the compression layer follows the schedule

$\mathcal{I}$5

so compression starts from the deepest layer and moves progressively to shallower layers. The teacher and student use the same output-level KL divergence as in TCD, but now at a changing compression layer $\mathcal{I}$6 rather than a fixed one (Wen et al., 1 Oct 2025).

The total training objective mixes supervised fine-tuning cross-entropy with either TCD or LCD using a fixed weight $\mathcal{I}$7:

$\mathcal{I}$8

or

$\mathcal{I}$9

with $\mathcal{P}$0 and temperature $\mathcal{P}$1 treated as a hyperparameter. The appendix also describes an integrated variant, ICD, which combines token-wise and layer-wise progressions in a single objective (Wen et al., 1 Oct 2025).

4. Progressive curriculum and theoretical intuition

The progressive schedules in EPIC define an easy-to-hard curriculum. In token-wise progression, the student ratio range shifts toward higher compression over time, and the teacher–student gap grows. In layer-wise progression, compression starts in deeper layers, where visual tokens are said to matter less, and gradually moves toward shallower layers, where perturbations are stronger (Wen et al., 1 Oct 2025).

The source description provides a one-dimensional prototype to motivate why progressive regularization yields a smoother optimization trajectory. Comparing a direct objective

$\mathcal{P}$2

with a progressive objective

$\mathcal{P}$3

the corresponding minimizers are

$\mathcal{P}$4

Under monotonicity, Lipschitz, and convexity assumptions for $\mathcal{P}$5, the progressive path has strictly smaller total variation than the direct path:

$\mathcal{P}$6

with $\mathcal{P}$7 (Wen et al., 1 Oct 2025). The intended interpretation is that progressive distillation regularizes the path of optima, thereby reducing abrupt parameter movement induced by heavy compression.

The companion progressive distillation study (Panigrahi et al., 2024) offers a complementary viewpoint. It argues that intermediate teacher checkpoints create an implicit curriculum because transient signals corresponding to easier subproblems are visible only during training and not in the final converged teacher. In sparse parity, these take the form of correlations with in-support low-degree monomials; in PCFG and language-modeling settings, they take the form of short-context or shorter-span features that later become attenuated in the teacher’s logits. This suggests that EPIC’s use of progressively stronger teacher targets can be understood not only as smoothing feature-space perturbations, but also as staging feature acquisition (Panigrahi et al., 2024).

A common misconception is that a stronger teacher, used in a single-shot distillation step, necessarily yields a stronger student. The background material explicitly rejects this assumption: strong teachers do not always produce strong students when distilled in a single shot, and progressive supervision from intermediate stages can be more effective because it exposes easier subproblems first (Panigrahi et al., 2024). EPIC adopts this broader progressive distillation logic in the specific setting of visual token compression for MLLMs (Wen et al., 1 Oct 2025).

5. Implementation and training configuration

EPIC is implemented on LLaVA-v1.5-7B without architectural changes. The vision encoder is CLIP ViT-L/14 at resolution $\mathcal{P}$8, the projector is the official pretrained MLP projector, and the LLM backbone is Vicuna-v1.5-7B (Wen et al., 1 Oct 2025).

The reported experiments use three representative compression methods: DART, described as duplication-aware pruning and training-free with compatibility with FlashAttention; FastV, described as importance pruning based on attention scores with compression at early language-model layers; and Random pruning, which requires no extra computation (Wen et al., 1 Oct 2025). The framework is also stated to be agnostic to pooling, token merging, top-$\mathcal{P}$9 selection, and saliency pruning.

Training is performed only in the second stage, visual instruction tuning, on LLaVA-665K; the projector remains frozen, and no Stage-1 projector pretraining is performed. The hyperparameters are batch size 128, learning rate $\mathrm{VE}$0 with cosine decay, warmup ratio 0.03, weight decay 0, epoch 1, optimizer AdamW, DeepSpeed stage 3, and maximum tokens 2048. The reported compute cost is approximately 12.2 hours on $\mathrm{VE}$1A100 GPUs (Wen et al., 1 Oct 2025).

The benchmark suite comprises 10 visual understanding datasets: VQA V2, GQA, VizWiz, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, MMBench-CN, and OCRBench (Wen et al., 1 Oct 2025). These datasets are used to assess effectiveness, robustness, and generalization under compression.

The algorithmic sketch provided for each training iteration is concise: sample a student compression ratio, derive the teacher ratio by subtracting the scheduled gap, choose either a fixed layer (TCD) or a scheduled layer (LCD), run shared-weights teacher and student forward passes, compute autoregressive cross-entropy and KL distillation loss, and update $\mathrm{VE}$2 using the mixed objective (Wen et al., 1 Oct 2025). Because distillation operates on the output distribution, no intermediate-feature matching mechanism is required.

6. Empirical performance, ablations, and robustness

The main quantitative results are reported relative to a baseline LLaVA-v1.5-7B using 576 tokens, with average accuracy 61.4%. For EPIC-TCD using DART, the reported averages are 61.7% at 256 tokens, 61.4% at 192 tokens, 61.3% at 128 tokens, 59.4% at 64 tokens, and 57.5% at 36 tokens. For EPIC-LCD using DART, the reported averages are 62.2% at 256 tokens, 62.0% at 192 tokens, 61.3% at 128 tokens, 59.4% at 64 tokens, and 57.6% at 36 tokens (Wen et al., 1 Oct 2025).

At 256 tokens, EPIC-LCD is reported to achieve strong gains on specific benchmarks, including VizWiz at 57.4 versus 52.5 and MME at 1834 versus 1785 (Wen et al., 1 Oct 2025). The general pattern is that EPIC can maintain parity with, or in some settings improve upon, the uncompressed model while using 128–256 visual tokens. The source description interprets this as indicating significant redundancy in visual tokens (Wen et al., 1 Oct 2025).

The efficiency results are reported on POPE with 8,910 samples, using an A100 GPU, compression at the second LLM layer, and retention of 64 tokens. Under these conditions, KV cache memory is reduced from 367.2 MB to 40.9 MB, CUDA time from 1103.5 s to 697.3–749.1 s, and FLOPs from 9.3 T to 1.5 T. The actual speedup is said to approach $\mathrm{VE}$3 for Random compression (Wen et al., 1 Oct 2025).

The ablation studies isolate the role of distillation and progressive schedules. For TCD at 128 tokens, EPIC achieves 61.3%, versus 59.8 without distillation and 59.1 without progressive ratio. For TCD at 64 tokens, EPIC achieves 59.4%, versus 58.1 without distillation and 58.2 without progressive ratio. For LCD at 128 tokens, EPIC achieves 61.3%, versus 60.5 without distillation and 60.3 without progressive layer. For LCD at 64 tokens, EPIC achieves 59.4%, versus 58.7 without distillation and 57.8 without progressive layer (Wen et al., 1 Oct 2025). These results are summarized in the source text as showing that both teacher guidance and progressive schedules matter.

Robustness is evaluated through cross-compressor generalization. Models trained with TCD using one compressor, such as DART, generalize to unseen compressors such as FastV and Random at inference, and EPIC is described as consistently improving performance across benchmarks while reducing performance gaps among compressors (Wen et al., 1 Oct 2025). Even simple Random pruning benefits substantially, which reinforces the claim that the framework’s gains do not depend narrowly on a particular compressor design.

The integrated variant ICD, which combines token-wise and layer-wise progression, achieves average 61.7% at 192 tokens, which is reported as $\mathrm{VE}$4 relative to LLaVA at 61.4%, and is described as comparable or better than MQT-LLaVA and TokenPacker under similar token budgets (Wen et al., 1 Oct 2025).

7. Positioning, limitations, and future directions

EPIC is positioned against two broad families of efficiency methods for MLLMs. Training-free compression methods reduce tokens through importance- or redundancy-based pruning, but often suffer accuracy degradation. Training-aware methods frequently introduce architectural changes, such as Q-Former variants, new projectors, or pre-fusion modules, and may require multi-stage training and additional compute (Wen et al., 1 Oct 2025). EPIC’s distinguishing claim is that it improves efficiency and accuracy under compression without architectural modification, added parameters, or token-level alignment overhead.

Its distinctive methodological contribution is the progressive decomposition of training difficulty along two orthogonal axes. The token-wise axis progressively increases compression ratio with a small teacher–student gap, while the layer-wise axis progressively moves compression from deeper to shallower layers. In both cases, consistency is enforced through output-level distillation by a shared-weights teacher and student (Wen et al., 1 Oct 2025). This suggests a design principle for compression-aware MLLM training in which the perturbation to the optimization landscape is controlled by scheduling the difficulty of compressed inputs rather than confronting the full perturbation at once.

Several limitations and trade-offs are explicitly stated. Extreme compression, such as 1–2 tokens, still degrades performance, and latency gains may diminish when the system becomes memory-bound. Distillation quality depends on the teacher’s easier targets, and schedule hyperparameters including $\mathrm{VE}$5, $\mathrm{VE}$6, $\mathrm{VE}$7, and the ratio ranges may require tuning. EPIC does not align intermediate features; while this reduces overhead, some tasks might benefit from feature-level guidance (Wen et al., 1 Oct 2025).

The method is also reported to be potentially less effective when the inference-time compressor is highly mismatched to the training-time compressor and induces very different token distributions, although the experiments show strong cross-method robustness. Similarly, compressing at very shallow layers early in training can perturb features too strongly, which is given as a rationale for the progressive layer schedule (Wen et al., 1 Oct 2025).

Future directions stated in the source include adaptive scheduling for $\mathrm{VE}$8, $\mathrm{VE}$9, and $\mathrm{MLP}$0 based on validation signals or optimization stability metrics; optional feature-level distillation at the projector or early language-model layers; task-aware compression using prompt or content signals; end-to-end differentiable compressors jointly learned with EPIC curricula; and extension of the framework to pretraining stages rather than only Stage-2 supervised fine-tuning (Wen et al., 1 Oct 2025). A plausible implication is that EPIC is best understood not as a single compressor, but as a general training framework for making heterogeneous token-compression schemes usable within MLLMs without sacrificing compatibility or incurring architectural complexity.