Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

Abstract: Pruning has emerged as a dominant paradigm for accelerating LLM inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns. Despite sharing the same objective, these pruning approaches induce fundamentally different execution behaviors, causing realized speedups to depend heavily on hardware and kernel implementations. Consequently, the practical acceleration benefits of different pruning families remain poorly understood. In this work, we introduce a GEMM-centric taxonomy that reorganizes existing pruning methods according to the logical \textbf{M}, \textbf{N}, and \textbf{K} dimensions of general matrix multiplication (GEMM). Leveraging this abstraction, we build a unified benchmarking framework that enables implementation-consistent comparison across the pruning design space and systematically characterizes the acceleration--quality Pareto frontier. Our results show that static depth pruning remains the strongest Pareto-optimal baseline and stays closest to its theoretical acceleration upper bound in memory-bounded scenarios. During prefill, the frontier transitions from static depth at low quality loss (0\%--4\%), to dynamic depth at moderate loss (5\%--16\%), and finally to static width pruning at higher loss levels (17\%--26\%). These findings establish the first unified view of the practical limits of pruning-based LLM acceleration and provide guidance for future pruning research.\footnote{Code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/PruningInferSim}

• The paper introduces a unified GEMM-centric taxonomy that categorizes LLM pruning interventions along M, N, and K axes for fair hardware evaluation.
• The paper presents a modular benchmarking suite that replaces operator implementations to accurately measure throughput–quality trade-offs in pruned models.
• The paper finds that static M-pruning consistently delivers the highest realized speedup, while dynamic methods struggle with runtime overheads.

GEMM-Centric Taxonomy and Realized Inference Acceleration in LLM Pruning

Motivation and Problem Statement

Efficient LLM inference remains one of the most urgent deployment challenges due to the disproportionate growth in model size relative to hardware capabilities. While pruning methodologies have proliferated, spanning token, depth, head, and dimension sparsification, evaluation conventions have not kept pace. Notably, realized end-to-end speedup for pruned LLMs is often inferred from proxy metrics such as FLOPs reduction without precise hardware and kernel considerations, creating a reproducibility and fair comparison gap across the literature. The paper "Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy" (2606.09080) addresses this issue by introducing a unified, GEMM-centric taxonomy and a benchmarking suite for consistent evaluation.

GEMM-Centric Taxonomy: Structural and Behavioral Unification

Pruning interventions in LLMs ultimately manifest as changes in the underlying GEMM (General Matrix-Matrix Multiplication) workload, mapping onto the logical axes: $M$ (token/row), $N$ (output/column), $K$ (input/reduction). The paper systematically categorizes mainstream and emergent pruning families as static/dynamic and as M, N, K, or NK axis interventions. This unification separates structural intentions (e.g., layer removal, width decomposition) from propagation and computational impact on the inference pipeline.

A visual illustration of MNK-dimension pruning intervention and propagation through Transformer layers is presented in Figure 1.

Figure 1: An overview of MNK-dimension pruning and their propagation through attention and FFN layers. The empty blocks denote zero-filled elements that make no contributions; lavender blocks indicate deviations compared to the dense baseline.

This abstraction reveals that not all sparsity yields comparable reductions in computation or memory bandwidth. Specifically, M-pruning has persistent effects on downstream GEMMs; N-pruning propagates as K-pruning to subsequent operators, and K-pruning, being reduction-axis, does not propagate. Importantly, the taxonomy is agnostic to whether pruning is realized statically (pre-computed, hardware-aligned) or dynamically (instance-adaptive, runtime-scheduled).

Unified Benchmarking Framework and Implementation

To enable methodologically sound cross-comparisons, the paper proposes a modular benchmarking framework that replaces operator implementations according to the targeted MNK taxonomy. Static K is mapped to either low-rank factorization or semi-structured SpMM (with cuSPARSELt integration, JIT tuning, and dimension alignment). For static M and NK, dependencies are rewired to respect layer and neuron removal and associated mask propagations. Dynamic M/NK leverages mask and index generation with fused gather/scatter and mask preprocessing, implemented via high-level DSLs (Triton and TileLang).

The GEMM-dominated cost profile in LLMs is empirically validated in Figure 2, confirming that GEMM and LM head account for approximately 90% of inference runtime, solidifying the case for a GEMM-centric analytical framework.

Figure 2: Llama3.1-8B's latency breakdown on forward steps for batch size 1 and context length 32,768, highlighting the GEMM-dominated cost.

The pipeline for dynamic M pruning and intra-kernel skipping is depicted in Figure 3.

Figure 3: The execution pipeline of dynamic M, showing tilewise skipping contingent on active tokens.

Empirical Evaluation: Throughput–Quality Pareto Analysis

The unified suite is used to systematically evaluate representative pruning strategies on Llama3.1-8B, with quality measured via perplexity and accuracy across standard benchmarks, and throughput as end-to-end wall-clock speedup (tokens/sec), controlling for batch, context, and hardware.

The main result is the direct, hardware-backed throughput–quality Pareto frontier for LLM pruning (Figure 4).

Figure 4: Throughput--quality trade-off curves at varying sparsity and pruning strategy, showing how practical speedup is strongly taxonomy-dependent.

Contrary to the frequent assumption that pruning sparsity directly determines realized acceleration, the results exhibit taxonomy-dependent frontiers:

• Static M-pruning (depth) consistently achieves the highest realized speedups at low to moderate quality loss, maintaining close proximity to theoretical acceleration upper bounds. Its monotonic performance holds for both prefill and decode.
• Dynamic M-pruning surpasses static M only in a regime of moderate quality loss (5–16%), leveraging instance-adaptive token skipping but with increased overhead from runtime reordering and gathering.
• Static NK-pruning (width, neuron) becomes Pareto-optimal only at sparsity levels inducing higher accuracy degradation (>17%), where its finer granularity softens the utility/quality loss.
• Static K (low-rank, semi-structured) and dynamic NK (attention patterns) consistently underperform in realized speedup and accuracy retention, largely due to overheads from additional launches, pipeline nonuniformity, and lack of propagation to subsequent operators.

Analysis of Latency and Overheads

The realized gap between kernel-level acceleration and theoretical headroom is dissected through kernel and operator breakdown (Figure 5).

Figure 5: Kernel-level wall-clock breakdown, evidencing non-GEMM overheads (e.g., routing, mask reordering), which are taxonomy- and phase-dependent.

Dynamic and fine-grained pruning methods (especially static K and dynamic M) accumulate significant non-GEMM overhead from preprocessing, reorderings, and gather/scatter operations, especially during decode. These overheads offset much of the expected gains from reduced computation footprint, emphasizing the imperative for co-design across pruning algorithm, kernel implementation, and inference scheduler.

Taxonomy-Level and Hardware Implications

The systematic taxonomy-level findings are:

• Static M-pruning is the operational anchor for throughput, saturating realized acceleration under moderate sparsity while suffering sharp accuracy degradation above 37.5% sparsity.
• Dynamic M-pruning offers a more flexible speed/quality trade-off at moderate sparsity and could benefit from further kernel specialization and hardware scheduling support.
• Static NK-pruning is the leading width-pruning strategy but presently fails to dominate at low-sparsity due to search complexity and kernel launch misalignment.
• Other width-pruning (static K, dynamic NK) struggle due to auxiliary and launch overheads, though their quality preservation remains strong.
• Dynamic NK only becomes competitive in very long-context settings, where attention cost dominates.

Combine these findings with Figure 6, which analyzes prefill and decode speedup as well as gap to upper-bound across sparsities and strategies.

Figure 6: Prefill and decode speedups at high sparsity (a-d), illustrating realized-vs-theoretical gap (e-f) tied directly to the pruning taxonomy.

Theoretical and Practical Implications

The paper establishes that nominal sparsity is an unreliable proxy for deployable LLM acceleration. Pruning taxonomy, axis propagation, and kernel/system overheads jointly determine practical gains. Future pruning research must account for hardware-aware implementation details, such as alignment constraints and non-GEMM pipeline stages, and employ hybrid taxonomies (combining dynamic and static, or M and width) to push the realized Pareto frontier.

For generalization, the benchmarking suite exposes that width- and pattern-wise pruning, including semi-structured and low-rank methods, must be co-designed with memory and scheduling strategies to truly unlock their theoretical advantages. Dynamic methods, while promising for fit-to-input computation, await better hardware and software support to minimize their runtime penalty.

Conclusion

This work delivers the first taxonomy-unified, hardware-backed benchmarking framework for inference acceleration in pruned LLMs. The dominant finding is that static depth pruning is consistently the most performant and stable for current deployment scenarios, but its accuracy collapse at high sparsity underscores the need for future hybrid and dynamic-static co-optimized approaches. The dependency of realized speedup on pruning dimension, propagation characteristics, and system overheads, rather than sparsity per se, defines the agenda for the next phase of model compression and acceleration. Systematic benchmarking based on generalized, GEMM-centric taxonomies, as introduced in this work, is now indispensable for fair and actionable comparison of pruning and compression methods.