MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruning

Abstract: Vision-LLMs (VLMs) increasingly rely on chain-of-thought (CoT) reasoning to solve complex multimodal tasks, but their large parameter sizes make deployment expensive. Structured pruning offers a natural solution; however, existing methods fail to preserve CoT reasoning accuracy in VLMs. We identify two key reasons: (1) CoT consistency depends on sparse transition points (pivot tokens) in the generation trajectory, while existing pruning methods are CoT-agnostic; and (2) pruning methods designed for unimodal LLMs do not account for activation-distribution differences across visual and textual modalities. Motivated by these observations, we propose MuCRASP, a structured pruning framework that targets reasoning-critical components while preserving cross-modal alignment and accounting for layer-wise sensitivity under a global parameter budget. Experiments on four VLMs across three reasoning benchmarks show that MuCRASP consistently preserves reasoning quality under increasing compression. At 30% pruning on Qwen2.5-VL-7B, MuCRASP achieves an LLM-as-a-Judge score of 8.87 versus 7.32 for the strongest baseline on physical reasoning tasks. Furthermore, MuCRASP maintains high reasoning consistency up to 50% pruning, significantly outperforming prior pruning approaches while exhibiting lower perplexity degradation.

• The paper introduces a training-free structured pruning method that combines pivot-token attribution, cross-modal layer protection, and global knapsack allocation to preserve multimodal chain-of-thought reasoning.
• At 30% pruning, MuCRASP achieves an LLM-Judge score of 8.87 on Qwen2.5-VL-7B physical reasoning versus 7.32 for the best baseline, while maintaining answer exact match and reducing KL divergence to 0.92 nats per token.
• The results show reasoning coherence can remain strong through 50% pruning even as answer precision declines, highlighting the need to evaluate compressed VLMs with logical-validity, grounding, and hallucination metrics rather than perplexity alone.

Motivation and problem statement

Vision-LLMs (VLMs) increasingly rely on chain-of-thought (CoT) prompting to solve multimodal reasoning tasks, but structured pruning methods designed for unimodal LLMs fail to preserve CoT quality when applied to VLMs. The paper identifies two root causes: (i) CoT coherence is governed by a sparse set of transition tokens ("pivot tokens") whose contribution is diluted when importance is aggregated uniformly over all token positions, and (ii) VLM pruning must account for cross-modal activation distribution differences that unimodal methods ignore. The authors propose μCRASP, a training-free structured pruning framework that targets reasoning-critical units while preserving vision–language alignment under a global parameter budget (2605.25842).

The work is framed around three research questions: whether structural pruning can preserve CoT coherence in VLMs (RQ1), why existing methods fail (RQ2), and how pruning should be redesigned to preserve reasoning trajectories under strict budgets (RQ3). A notable empirical claim is that perplexity is a misleading proxy for reasoning preservation: a pruned model can produce fluent, low-perplexity text with logically incoherent reasoning, and vice versa.

Method

μCRASP operates on structural units — SwiGLU MLP neurons, attention heads, and GQA groups — and proceeds in four stages. First, it computes a global first-order Taylor importance score over all token positions on a calibration set of 128 image–question–CoT triples. Second, it computes trajectory pivot attribution by restricting the Taylor loss to local windows ($W{=}8$ tokens) around detected reasoning-transition tokens, identified via surface markers such as step delimiters ("Step 1:", "Final Answer:") and logical connectives ("Therefore", "Thus", "Hence"), with equal-interval segmentation as fallback. The paper argues these transitions are sparse (~5% of tokens) yet disproportionately important.

Third, it profiles each decoder layer with a Cross-Modal Dependency Score (CMDS), a normalized linear-kernel MMD between visual- and text-token activation distributions within the layer, plus an output-magnitude sensitivity proxy. Fourth, it fuses global and pivot scores with a compression-conditioned coefficient $$\gamma_{\text{dyn}} = \gamma_{\text{base}}(1-S)^\rho$$, scales importance by layer protection factors derived from CMDS and sensitivity, and solves a greedy knapsack over all units ranked by value-to-cost efficiency, subject to minimum-retention constraints. The global formulation is motivated by extreme unit-cost heterogeneity: a GQA group in Qwen2.5-VL-7B contains ~5.2M parameters versus ~21K per MLP neuron (~250× disparity).

Experimental setup

Evaluation covers four VLMs spanning three families (Llama 3.2-11B, Qwen2.5-VL-7B/3B, Gemma-3-4B, Qwen2-VL-2B) across three reasoning domains: TDIUC-Physical, TDIUC-Quantitative, and A-OKVQA (commonsense). Ground-truth CoT traces are generated by GPT-4o and manually validated; calibration uses 128 samples shared with all baselines. Beyond perplexity, exact match, and semantic similarity, the paper introduces LLM-as-Judge (GPT-3.5-Turbo scoring logical validity of CoT traces on a 0–10 scale) as its primary metric, complemented by KL divergence against the dense model's output distribution. Baselines include magnitude pruning, attribution-based pruning, LLM-Pruner, Wanda-SP, and FLAP, all adapted to structured pruning without recovery fine-tuning.

Main results

At 30% pruning on Qwen2.5-VL-7B, μCRASP achieves an LLM-Judge score of 8.87 on physical reasoning versus 7.32 for the best baseline (LLM-Pruner), while retaining $EM_a$ of 71.3 versus near-zero for most baselines. On Llama 3.2-11B, baselines collapse entirely at 30% ($EM_a = 0$ for LLM-Pruner, Wanda-SP, and FLAP), whereas μCRASP retains LLM-J of 8.55/7.29/6.66 across the three domains. The method sustains LLM-J > 7.0 up to 50% pruning, a 20–25 percentage-point improvement over existing methods, which exhibit abrupt collapse beyond 25–30% pruning.

A consistent finding is the decoupling of reasoning coherence from answer precision under μCRASP: LLM-J degrades minimally while $EM_a$ collapses earlier (e.g., Qwen2.5-VL-3B at 30% retains LLM-J 8.07 but drops $EM_a$ from 86.0 to 24.8). This separation never occurs in baselines, where both metrics collapse simultaneously. Domain-wise, physical reasoning degrades slowest, quantitative reasoning preserves coherence but loses numerical precision beyond 30%, and commonsense declines most uniformly.

Distributional analysis confirms these gains at the logit level: μCRASP achieves mean per-token KL divergence of 0.92 nats/token at 30% pruning, 1.7× lower than LLM-Pruner and 11.4× lower than Attribution Pruning, with domain-invariant rankings.

Analysis

Three analyses support the design choices. CMDS identifies the cross-modal bottleneck: CMDS peaks in middle layers (11–18) of Qwen2.5-VL-7B, and sliding-window MLP zero-out ablations show that ablating high-CMDS layers triggers catastrophic failure while low-CMDS layers degrade gracefully. μCRASP exploits this by preserving 60–100% of neurons in peak-CMDS regions beyond 40% pruning. Pivot attribution protects transitions: the gap over LLM-Pruner persists despite similar perplexity, indicating the difference lies in which tokens drive importance. Global allocation is indispensable: replacing the knapsack with uniform layerwise ratios collapses LLM-J from 7.43 to 4.40 on Physical at 50% pruning despite identical total parameter counts.

Component ablations at 50% pruning show both mechanisms contribute: removing pivot attribution costs up to 32 points of $EM_a$ on Physical, and removing CMDS causes the steepest semantic-similarity drop (69.9 → 51.1). Critically, a random-pivot control shows random positions recover only partial gains (e.g., $EM_a$ 29.2 vs 38.0 for real pivots on Physical), confirming that correct transition identification — not merely window restriction — drives the advantage. Window-size ablation shows a clear optimum at $W{=}8$, with narrower windows losing context and wider windows diluting the signal toward global attribution.

An additional grounding/hallucination analysis at 50% pruning reports μCRASP achieving a grounding score of 8.6 with hallucination score of 1.8, versus grounding below 5.6 and hallucination above 4.8 for all baselines, indicating that ignoring cross-modal dependencies leads to hallucinated visual content even when text remains fluent.

Limitations and open questions

The paper concedes several limitations. Pivot extraction relies on heuristic regex-based marker matching rather than learned segmentation, leaving robustness to free-form or non-English CoT formats open. Protection-factor hyperparameters are manually tuned, though ablations suggest individual components are not fragile. Evaluation is restricted to English benchmarks and models up to 7B parameters; scaling behavior at larger sizes remains untested. The LLM-as-Judge protocol itself may carry systematic biases and may not capture all reasoning failure modes. The method also depends on synthetic GPT-4o-generated CoT traces for calibration, introducing a dependence on teacher-model trace style. Finally, no post-pruning recovery (distillation, LoRA) is applied, and combining μCRASP with recovery stages is left unexplored.

Conclusion

μCRASP demonstrates that structured pruning of VLMs can preserve chain-of-thought reasoning by jointly targeting sparse reasoning-transition tokens, shielding cross-modal integration layers identified via CMDS, and allocating a global budget through knapsack optimization. Its central empirical contributions are the demonstration that prior methods collapse abruptly beyond moderate compression while μCRASP degrades gracefully to 50%, and the observation that reasoning coherence and answer-extraction accuracy follow separable degradation trajectories — a finding that argues for evaluation protocols beyond perplexity and surface-level metrics when assessing compressed generative models.