Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models

Abstract: Chain-of-Thought (CoT) prompting symbolized a huge improvement of reasoning capabilities of LLMs. However, scaling up test-time computation yields extensive CoT sequences, introducing severe inference latency and key-value (KV) cache memory bottlenecks. While structural pruning offers a fundamental, hardware-aware solution to alleviate static parameter burdens, existing magnitude-based methods may cut off the neurons of CoT: by over-indexing on discrete cross-entropy objectives, these heuristics fall into a \textit{magnitude trap}: they prioritize high-frequency, low-information syntactic tokens and trigger a disappointing reasoning collapse at high sparsities (e.g., 40\%). To overcome this topological phase transition, we propose \textsc{Relative Kinetic Utility} (RKU), a novel theoretical framework that elevates discrete pruning to a continuous kinetic integral over the depth manifold of the model based on Alternating Gradient Flow(AGF). By modifying it with Fisher trace normalization, RKU acts as a lightweight curvature-aware normalization to isolate \textit{kinetic spikes} -- the fundamental structural pathways responsible for high-curvature logical routing. Extensive experiments on Qwen-2.5-7B and LLaMA-3-8B improves performance in the high-sparsity regime around 40\%. RKU attains 13.34\% accuracy on GSM8K at 40\% sparsity, outperforming the strongest baseline, and appears to better preserve reasoning-relevant representations under out-of-distribution evaluation.

• The paper presents a novel pruning criterion, RKU, that leverages continuous kinetic integrals and alternating gradient flow to preserve logical reasoning under high sparsity.
• It demonstrates that RKU outperforms magnitude-based methods by maintaining deep, high-curvature logical pathways, yielding significant accuracy gains on benchmarks like GSM8K and AQuA.
• The methodology introduces Fisher trace normalization to stabilize pruning, delivering hardware acceleration and robust transfer for reasoning-critical applications.

Relative Kinetic Utility for Reasoning-Aware Structural Pruning in LLMs

Introduction and Motivation

Chain-of-Thought (CoT) prompting has produced significant advances in the logical and mathematical reasoning abilities of LLMs. However, the computational costs of deploying such models at scale are dominated by both dynamic inference overhead (e.g., KV cache growth, latency) and static model footprint (parameter count, VRAM). Context and sequence-level compression techniques reduce dynamic cost but do not address hardware constraints imposed by model size. Structural pruning—removing architectural units such as attention heads or FFN dimensions—delivers hardware-level efficiency. Current magnitude-based pruning methods, however, routinely fail to preserve reasoning ability when pushed to high sparsity. The failure mode, termed the "magnitude trap", arises because these approaches over-index on discrete cross-entropy (LCE) objectives, favoring highly activated but semantically superficial neurons, and ultimately inducing catastrophic reasoning collapse at sparsity above 40%.

The "Relative Kinetic Utility" (RKU) framework addresses this limitation by introducing a fundamentally new pruning criterion. RKU leverages continuous kinetic integrals over the model depth, guided by an alternating gradient flow (AGF) and stabilized via Fisher trace normalization. This design is motivated by the empirical and theoretical observation that high-curvature, multi-hop logical routing pathways are essential for reasoning tasks, and are easily lost under traditional pruning regimes.

Theoretical Framework and Methodology

Discrete Objective Gap and the Magnitude Trap

Magnitude-based heuristics (e.g., Wanda, SparseGPT, FLAP) ground their pruning scores in layerwise activation magnitudes or first-order Taylor approximations, which are tightly coupled to discrete LCE over token distributions. These objectives are susceptible to the "magnitude trap": they preferentially preserve neurons aligned with high-frequency, low-information syntactic templates, thereby hollowing out pathways required for complex reasoning, as evidenced by the rapid drop in math benchmark accuracy at high sparsity.

Continuous Kinetic Integral

RKU proposes elevating the pruning score from a discrete, activation-centric view to a continuous physical energy functional. Specifically, it computes a kinetic utility for each structural component as the integral of the squared $L_2$ norm of final hidden states across model depth, operationalized via AGF. This approach inherently targets "kinetic spikes": network units with outsized influence on the high-curvature regions of latent space associated with reasoning.

Riemannian Pre-conditioning via Fisher Trace Normalization

To counteract instability from localized kinetic noise introduced by aggressive pruning in non-convex spaces, RKU normalizes the continuous kinetic integrals using the trace of the empirical Fisher information matrix. This Riemannian manifold pre-conditioning aligns the pruning criterion with a Hessian-root scaling, providing curvature-aware selectivity at essentially zero additional cost.

Empirical Analysis and Results

Extensive evaluations were conducted on Qwen-2.5-7B and LLaMA-3-8B, targeting the critical high-sparsity regime (30–50%). The design of experiments isolates the role of reasoning preservation by focusing on CoT-intensive benchmarks (GSM8K, AQuA).

Key results:

• At 40% sparsity, RKU achieves 13.34% accuracy on GSM8K, outperforming the strongest baseline (Wanda-Struct at 4.85%, Taylor-FO at 6.90%). Similar gains are observed on AQuA (27.56% vs. 24.41% for Wanda-Struct).
• RKU retains the deep logical backbone under sparsity, with qualitative analysis revealing consistent preservation of stepwise deduction and arithmetic chains. By contrast, magnitude-based baselines suffer task format collapse and hallucination under identical conditions.
• SVD-based analysis demonstrates that magnitude heuristics induce representational collapse (low-rank subspaces, loss of orthogonality especially in deep layers). RKU maintains higher-dimensional feature spectra, crucial for out-of-domain (OOD) reasoning generalization.

Ablation confirms the necessity of Fisher Trace normalization: AGF without this normalization cannot stabilize pruning scores and exhibits severe performance collapse (e.g., 0% on AQuA at 40% sparsity).

Knowledge-Logic Trade-off and Structural Plasticity

Evaluation on static retrieval tasks (WinoGrande, PIQA) reveals a deliberate trade-off imposed by RKU. Magnitude-centric pruning modestly outperforms RKU on these tasks at similar sparsity levels, as the latter sacrifices superficial feature preservation to protect topological flow needed for deep logic. However, structural plasticity analysis via minimal LoRA fine-tuning and OOD transfer (e.g., MathQA) shows that RKU-pruned models achieve higher recovery and transfer rates, emphasizing robust logical generalization and strong resistance against shortcut learning.

Hardware Efficiency

RKU translates theoretical sparsity into actual hardware acceleration (1.42x wall-clock speedup for 50% pruned Qwen-2.5-7B in native prefill latency), without requiring specialized inferencing kernels. This matches the primary motivation for structural pruning: enabling practical deployment of reasoning-capable LLMs within resource-constrained environments.

Implications and Future Directions

RKU challenges the dominance of activation-magnitude-based methods for LLM compression in reasoning-centric applications. By formulating pruning as a kinetic, curvature-aware process, it reconceptualizes what it means to "preserve reasoning" under aggressive model compression. The work suggests that future sparsity- and efficiency-aware AI research must contend directly with the topological properties of model representations—not just their ability to recapitulate surface statistics.

From a theoretical perspective, RKU’s framework connects kinetic energy integrals, Riemannian geometry, and neural pruning in a way that could be further extended. Promising directions include more granular curvature diagnostics, per-layer Fisher normalization schedules, and integration with gradient-based or learning-based structured growth approaches. Practically, RKU opens the space for deploying robust, high-sparsity LLMs in domains where logical integrity is critical (mathematics, programming assistance, scientific QA) without the cost explosion associated with larger or denser models.

Conclusion

Relative Kinetic Utility provides a rigorous, theoretically motivated, and empirically validated solution to the challenge of sparsity-tolerant reasoning in LLMs. By moving beyond the constraints of discrete, activation-centric heuristics, RKU explicitly preserves high-curvature logical pathways necessary for complex CoT, while also providing robust hardware efficiency. Its performance on both in-domain and OOD benchmarks—particularly the doubling of survival rates under high sparsity—attests to the importance of topologically-informed structural pruning strategies in the evolution of efficient, reasoning-aligned AI systems.