Latency Prediction for LLM Inference on NPU Systems

Abstract: Deploying LLMs requires exploring a large configuration space spanning parallelization strategies, batching techniques, and scheduling policies. Exhaustive measurement across this space is impractical, making latency prediction essential for system optimization. While NPUs have emerged as accelerators designed for LLM inference, no prediction methodology has been established for them. Specifically, applying prior work to LLM inference latency prediction on NPUs faces three challenges: undisclosed microarchitecture of commercial NPUs, unpredictable compiler optimizations, and latency non-linearity induced by bucketing. We present LENS, a latency estimator that predicts NPU inference latency without information on the microarchitecture or compiler, and captures the non-linear latency induced by bucketing. LENS profiles each bucket with two end-to-end (E2E) measurements and composes the results to predict latency for arbitrary input-output length combinations. We validate LENS across NPUs from multiple vendors, several LLMs, and diverse workloads, achieving a mean prediction error of 2.15\%. We further compare LENS against two methodologically related baselines, confirming the validity of its approach.

• The paper introduces LENS, a black-box, bucket-based framework that decomposes end-to-end latency into prefill (TTFT) and per-token (TBT) delays.
• It leverages per-bucket profiling to overcome challenges from non-disclosed NPU architectures, compiler optimizations, and input/output bucketing.
• Experimental results across multiple NPUs and LLMs show LENS achieves a mean error below 2.2% while significantly reducing profiling cost.

Latency Prediction for LLM Inference on NPU Systems: An Expert Analysis

Problem Motivation and Architectural Context

The rapid deployment of LLMs in production settings has driven the demand for inference systems that meet stringent SLOs on response latency and throughput. While GPUs have been the traditional choice for LLM training and inference, architectural mismatches—specifically during autoregressive decode—have made GPUs increasingly ill-suited for inference workloads. NPUs, built for LLM inference, feature dataflow-centric, multi-engine designs that minimize off-chip memory bandwidth bottlenecks and pipeline heterogeneous operations without serializing them on shared execution units.

However, system configuration for LLM inference on NPUs is non-trivial due to several factors: (1) non-disclosure of commercial NPU microarchitecture, which precludes cycle-accurate simulation; (2) unpredictable, aggressive compiler optimizations that leverage physically isolated tensor/vector/scalar engines for kernel fusion and concurrent execution; (3) static compilation requirements, which induce non-linear, stepwise latency profiles via input/output sequence "bucketing." Existing latency prediction methodologies, calibrated for GPUs, are structurally incompatible with this setting.

Challenges in NPU-based Latency Prediction

Three central challenges prevent straightforward adaptation of prior GPU and simulator-based inference latency prediction techniques:

• Architectural Non-transparency: Commercial NPUs (e.g., AWS Inferentia2, Google TPU v4/v5e/v6e) rarely disclose details beyond systolic array dimensions; critical parameters for simulation (HBM timing, network topology, memory hierarchy) are opaque.
• Compiler-driven Latency Non-linearity: On NPUs, compiler optimizations decouple kernel-level profiling measurements from actual E2E latency, hiding or fusing operation latencies in non-obvious ways that violate the additivity assumptions of per-kernel or per-operator summing strategies commonly used in GPU-focused predictors.
• Bucketing-induced Step Functions: Static compilation forces serving systems to use a finite set of input/output length buckets. Latency is constant within a bucket and jumps sharply at bucket boundaries, invalidating any methodology assuming smooth (typically linear or continuous) latency as a function of sequence length.

Contemporary GPU-based approaches, even with operator- or kernel-level granularity (e.g., NeuSight [22], MaverIQ [25]), incur error rates that are catastrophic (up to ~493%) when naively applied to NPUs due to these fundamental issues.

The LENS Methodology

LENS (Latency Estimator for NPU Systems) directly addresses these NPU-specific constraints and behavioral discontinuities by completely eschewing any requirement for microarchitectural or compiler knowledge.

• Black-box Profiling: LENS operates agnostically to the underlying hardware and compiler stack, treating the NPU as a black box. It leverages externally-observed, post-compilation, E2E latency measurements as the primitive unit.
• Per-bucket Profiling: Recognizing the stepwise latency regime induced by bucketing, LENS profiles each bucket with only two E2E measurements at different output lengths. With this, it extracts both the per-bucket prefill latency (TTFT) and per-token generation latency (TBT), capturing the non-linear decode trajectory through bucket transitions.
• Latency Decomposition and Prediction: For arbitrary input-output pairs, LENS decomposes E2E latency into: (i) TTFT determined by the prefill bucket (determined by input length), and (ii) decode latency, which is the sum over buckets traversed during generation, each multiplied by the respective TBT and the number of tokens allocated to that bucket. The methodology generalizes to both single-request and batched inference, accommodating batching-specific bucket assignment and decode trajectory logic.

This approach reduces the required profiling cost on NPUs to O(number of buckets), completely eliminating the exponential cost explosion seen in exhaustive configuration search or kernel decomposition strategies.

Experimental Validation

LENS is evaluated on four production-class NPUs (Inferentia2, TPU v4, v5e, v6e), three LLMs of varying scale (Llama-3.2-1B, Mistral-7B, Qwen3-14B), and four representative workloads (ShareGPT, CNN/DailyMail, arXiv Summarization, WritingPrompts). The evaluation comprises 248 scenario permutations, varying chips, workloads, and batch sizes up to 32.

• Prediction Accuracy: LENS achieves a mean error of 2.15% and a median of 1.69%; 89.9% of all configurations have error rates below 5%. The approach is robust to workload characteristics (input-dominant vs. output-dominant), model scale (1B–14B params), NPU architectural differences, and batch size.
• Baseline Comparison: Compared to LLM Serving Sim 2.0 7 and MaverIQ 25, LENS exhibits drastically lower error, especially in regions where baseline assumptions break due to compiler fusion (errors >200% for operator-sum methods, >70% for global linear TBT approaches on bucket transitions).
• Profiling Cost: LENS dramatically reduces profiling cost. Only two E2E measurements per bucket are required per batch configuration, yielding an order-of-magnitude reduction in both compilation and profiling time.

Empirical Insights and Theoretical Implications

LENS’s findings decisively invalidate several GPU-derived heuristics:

• Larger batch sizes do not always increase throughput; compiler-induced hot/cold paths per batch/bucket shape can make smaller batches optimal.
• Finer bucket granularity does not necessarily yield lower latency due to compiler optimization targeting only specific input shapes.
• Analytical reasoning about optima is infeasible on NPUs; direct measurement is necessary due to complex configuration- and shape-sensitive compiler/hardware interactions.

The practical implication is that optimal configuration search for LLM inference on NPUs must rely on empirical, black-box methods like LENS, as analytical approaches are unreliable.

Limitations and Future Directions

LENS’s approach, while efficient and accurate, is currently limited to the hardware on which profiling is conducted. Unlike analytical or kernel-sum approaches, it cannot directly extrapolate to unseen NPU platforms. Generalizing LENS to support cross-device latency scaling—possibly by identifying persistent bucket-level scaling laws or hardware feature predictors—remains an open research problem. Additionally, LENS presently abstracts away higher-level system factors (queuing, interconnect bandwidth, pipeline parallelism latencies). Extending latency prediction to encompass full-stack, system-level effects is a future direction.

Broader Related Work

LENS diverges fundamentally from prior DNN performance estimation efforts, which focus on operator decomposition and scaling (Habitat [38], Vidur [2]), and from NPU simulation tools (PyTorchSim [36], scaleSim v3 [32], mNPUsim [16]) that require full architectural disclosure. These prior tools are either inapplicable (due to proprietary or opaque hardware) or incur substantial errors for LLM inference on commercial NPUs. LENS’s per-bucket, E2E-only approach directly complements these by offering a practical mechanism for accurate, low-cost inference latency prediction in real-world settings.

Conclusion

LENS constitutes a black-box, bucket-based framework for highly accurate prediction of LLM inference latency on vendor-supplied, commercial NPUs (2606.18042). It resolves core technical challenges—architectural opacity, compiler-driven non-linearity, and bucketing—unaddressed by prior work. Strong experimental evidence demonstrates its accuracy, profiling efficiency, and versatility across architectures and workloads. Empirical case studies show that brute-force measurement is essential under NPUs’ compiler and architecture idiosyncrasies, and LENS makes such exploration tractable. Extending LENS to address hardware transferability and system-level performance estimation defines a natural trajectory for subsequent research in NPU-based LLM serving optimization.