DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models

Abstract: Existing works on LLM decomposition mainly focus on improving performance on downstream tasks, but they ignore the poor parallel inference performance when trying to scale up the model size. To mitigate this important performance issue, this paper introduces DeInfer, a high-performance inference system dedicated to parallel inference of decomposed LLMs. It consists of multiple optimizations to maximize performance and be compatible with state-of-the-art optimization techniques. Extensive experiments are carried out to evaluate DeInfer's performance, where the results demonstrate its superiority, suggesting it can greatly facilitate the parallel inference of decomposed LLMs.

• The paper introduces a low-rank communication pipeline that reduces bandwidth from O(d) to O(k) and eliminates duplicate self-attention computation.
• The paper details a CUDA Graph–compatible KV cache reconstruction method that minimizes kernel launch overhead through static buffer allocation.
• The paper achieves up to 8.81× speedup and 80% reduction in latency, enabling scalable and efficient inference for decomposed LLMs.

DeInfer: Efficient Parallel Inferencing for Decomposed LLMs

Motivation and Background

Model size scaling in LLMs (e.g., LLaMA-3-70B, OPT-30B) has amplified memory and inference challenges, particularly in multi-GPU deployments. While quantization and pruning techniques have seen extensive development, low-rank decomposition methods—such as SVD, AFM, and feature-based compression—have not been widely adopted for production-scale inference, despite strong memory footprint reductions [fwsvd, SoLA, svd_llm]. The principal barrier is their incompatibility with high-throughput parallel inference, primarily due to increased inter-GPU communication and redundant computation. Notably, state-of-the-art systems (e.g., vLLM, SGLang) do not provide optimized support for decomposed models, resulting in limited scaling and underutilized hardware.

Identified Bottlenecks

The paper provides a detailed analysis of parallel inference bottlenecks specific to decomposed LLMs:

• Communication Overhead: Decomposition of a single weight matrix into two smaller matrices doubles collective operations (e.g., reduce-sum) per forward pass, dramatically increasing bandwidth requirements due to repeated synchronization, especially across attention and MLP sublayers.
• Duplicate Self-Attention Computation: Post-decomposition, KV and output matrices are redundantly broadcast, causing all data parallel workers to recompute identical self-attention, an unnecessary computational burden that does not exist with non-decomposed tensor parallelism.
• CUDA Graph Incompatibility: Decomposition induces dynamic KV cache shapes during decoding, conflicting with CUDA Graph’s requirement for static kernel addresses and shapes. This prevents kernel launch fusion and amortization, causing significant scheduling and launch overhead.

System Design: DeInfer Architecture

The proposed system, DeInfer, systematically targets these bottlenecks.

Low-Rank Communication Pipeline

DeInfer introduces a pipeline reordering innovation allowing collective operations in the low-rank latent space. Unlike traditional all-gather and reduce-sum in the full hidden dimension, the system:

• Transposes all-gather to occur immediately after the downward-projection (rank-$k$) transformation, minimizing communication volume (from $O(d)$ to $O(k)$).
• Ensures local attention and MLP computation avoids full-dimension redundancy, eliminating duplicate self-attention and reducing both bandwidth and FLOPs per layer.

Efficient KV Cache Reconstruction

DeInfer provides a CUDA Graph–compatible KV cache pipeline via static buffer allocation and remapped block tables:

• Performs batch consolidation of logical blocks into contiguous memory before the static graph region, enabling reconstruction with preallocated meta-kernels.
• Ensures that KV cache matrix reconstructions yield fixed-address, dynamically-shaped but preallocated outputs, which enables graph capture and rapid replay during decoding.

Integration and Generalizability

DeInfer does not assume specific model architectures or decomposition ratios. It supports all tensor parallel configurations and is agnostic to the partitioning pattern per-layer, per-matrix, or per-head. This includes efficient inference for models with variable compression ratios across layers and non-GLU/GLU-based MLPs and rotary position embeddings. While MoE-style architectures (e.g., DeepSeek-V2) are not yet directly supported, any decomposition preserving linear matrix factorization principles (without nonlinear intermediate stages) is compatible.

Experimental Evaluation

The evaluation employs LLaMA-3-70B, LLaMA-65B, and OPT-30B on heterogeneous GPU clusters (NVIDIA A800s and A6000s with/without NVLink), using the vLLM benchmark suite for standardization.

Throughput and Latency

• Throughput: Under high-batch or high-sequence regimes, DeInfer consistently provides substantial improvements over a baseline (naive) decomposed model pipeline, with up to ×2.25 speedup with NVLink and ×8.81 without NVLink as parallelism and compression increase. Performance improvements scale positively with higher compression ratios.
• Latency: DeInfer yields up to an 80% reduction in time-to-first-token (TTFT) and inter-token latency (ITL) in both rotary and gated architectures across different LLMs.
• Platform Effects: On bandwidth-limited interconnects (A6000s), gains are more pronounced due to the compounded effect of communication reduction.

Ablation and System Profiling

• Communication/Computation Reduction: Nsight-profiling demonstrates that DeInfer reduces communication from >80% to <10–20% of total transformer timing, with compute time also reduced due to elimination of duplicate attention.
• KV Cache and CUDA Graph: When using static graph execution and paged KV cache, DeInfer achieves an additional performance improvement of up to 66% tokens/sec in NVLink-enabled settings for mild load, demonstrating the necessity for CUDA Graph compatibility.
• Scalability: DeInfer nearly linearly scales with additional GPUs, overcoming the typical saturation observed in decomposed LLMs.

Theoretical and Practical Implications

The approach by DeInfer clarifies that the principal roadblocks to scalable decomposed LLM inference are architectural (collective operation placement and computation duplication) and not inherent to decomposition itself. This shifts decomposition techniques from theoretical compression tools into practical optimizations for production LLM serving. Further, the generalizability allows straightforward extension to more advanced and heterogeneous model topologies, and for integration into state-of-the-art serving engines (vLLM, MLC-LLM, SGLang). Compatibility with CUDA Graphs enables the exploitation of future generations of GPUs with increasingly large kernel fusion and static graph requirements.

The experimental data also indicate that GQA-style architectures (fewer attention heads, e.g., LLaMA-3-70B) are preferred for low-rank cache due to lower reconstruction cost, a design consideration for future LLM architectures optimized for deployed inference.

Conclusion

DeInfer delivers the first parallel inference system targeting decomposed LLMs, resolving the communication and computation bottlenecks that previously limited their deployment. The methods implemented—low-rank collective orchestration, duplicate computation removal, and static-graph friendly KV cache pipelines—contribute to significant throughput and latency improvements. These results allow decomposed LLMs to be deployed efficiently and scalably on both high-bandwidth and commodity GPU clusters without sacrificing inference efficiency.

Future work in decomposed model serving will likely extend DeInfer’s principles for mixture-of-expert (MoE) decomposed LLMs, automatically adaptive compression ratios per data or workload, and system-aware model architecture co-design. The codebase serves as a foundation for further system-level innovations in compressed model serving.

Reference: "DeInfer: Efficient Parallel Inferencing for Decomposed LLMs" (2604.17709)