Opt4GPTQ: Co-Optimizing Memory and Computation for 4-bit GPTQ Quantized LLM Inference on Heterogeneous Platforms

Abstract: The increasing adoption of LLM (LLMs) on heterogeneous computing platforms poses significant challenges for achieving high inference efficiency. To address the low inference efficiency of LLMs across diverse heterogeneous platforms, this paper proposes a practical optimization method, Opt4GPTQ, designed for 4-bit GPTQ quantized LLMs inference on heterogeneous AI accelerators. Built upon the vLLM serving system, Opt4GPTQ integrates three platform-level optimization strategies: Shared Memory Buffering optimization (SMB-Opt), which caches data in shared memory and employs single-threaded writes; Vectorized Memory Loading optimization (VML-Opt), which utilizes vectorized memory operations for efficient data loading; and Inline Assembly optimization (ILAOpt), which directly leverages hardware-native vector halfprecision addition and fused multiply-accumulate instructions for efficient execution. Experimental results show that Opt4GPTQ effectively improves inference performance across different models, achieving up to 84.42% throughput improvement and up to 51.35% latency reduction. This work highlights the critical role of platform-level engineering optimizations in enabling efficient LLMs inference on emerging heterogeneous AI acceleration architectures and provides valuable deployment experience and methodologies for future heterogeneous platform adaptation.