NPUsper: Eliminating Redundant Computation for Real-Time Whisper on Mobile NPUs

Abstract: We present NPUsper, a live transcription system that makes Whisper efficient on mobile NPUs by eliminating redundant computation. To avoid the heavy padding used by prior streaming systems, NPUsper detects hallucinated tokens online from temporal patterns in decoder cross-attention, allowing each inference round to process short audio inputs with minimal carryover. For efficient mobile-NPU execution, we propose controlled unrolling, which executes autoregressive decoding as K-step chunk graphs, removing unnecessary KV-cache computation and reducing graph-dispatch overhead. NPUsper achieves up to 4.84x lower per-word latency, up to 33.2x lower time-to-first-token (TTFT), and up to 88.64% lower average power consumption compared with baselines, while maintaining comparable transcription accuracy. The code is available at https://github.com/npusper/NPUsper.

• The paper introduces NPUsper, which eliminates redundant computation in real-time Whisper transcription on mobile NPUs.
• It employs online cross-attention hallucination detection to avoid excessive padding and overlap, preserving low latency and energy efficiency.
• Controlled unrolling aligns dynamic autoregressive decoding with static-graph NPUs, achieving up to 4.84× lower latency and significant power savings.

NPUsper: Redundant Computation Elimination for Real-Time Whisper on Mobile NPUs

Introduction and Context

NPUsper addresses a central limitation in deploying large foundation models like Whisper for low-latency, real-time speech transcription on mobile devices. As Whisper was originally trained and optimized for offline ASR on 30s audio segments, prior attempts to port it to real-time streaming applications have resorted to extensive input padding and segment overlap, particularly to suppress hallucinations (spurious tokens not grounded in the input). Simultaneously, the autoregressive nature of Whisper's decoding and its dynamic KV-cache growth are fundamentally misaligned with mobile NPUs’ static graph execution paradigm, leading to excessive redundant computation and prohibitively high latency, energy, and resource costs.

Figure 1: Existing streaming systems pad Whisper inputs to 30s and overlap segments, leading to excessive redundant computation.

Whisper and its derivatives thus saturate mobile memory, GPU/NPU compute, and energy budgets, making genuine real-time on-device transcription largely unattainable without architectural or execution-model adaptation.

The NPUsper System

Design Insights

NPUsper's core insight is to remove both sources of inefficiency by:

1. Online hallucination detection based on cross-attention temporal dynamics, removing the need for padding/overlap used in previous work.
2. Controlled unrolling for NPU execution, which matches the static-graph execution model of mobile NPUs to Whisper's otherwise dynamic autoregressive token generation.

Padding, Hallucination, and Computational Overhead

The authors demonstrate that while padding longer inputs suppresses hallucination and reduces WER/CER, it increases latency and compute by multiple factors compared to short, unpadded inputs.

Figure 2: (Left) More padding reduces hallucinations/WER, but at the cost of compute. (Right) NPUs deliver much lower power and better throughput than GPU, but static-graph execution creates inefficiency for dynamic tasks.

Their ablation confirms that hallucinations primarily result not from insufficient acoustic coverage but from failures in token-level temporal grounding, induced by feeding Whisper atypically short audio inputs.

Cross-Attention Based Hallucination Detection

NPUsper leverages the temporal alignment in Whisper's final-layer cross-attention maps: valid content tokens create localized, forward-progressing attention over audio frames, whereas hallucinated tokens are marked by backward shifts or loss of temporal anchoring. By algorithmically analyzing the peaks in attention differences across consecutive tokens—specifically, searching for violations of monotonicity—the system detects, halts, and isolates hallucinated generations online, eliminating the requirement for extensive buffering or overlapping.

Figure 3: Content tokens align cross-attention sharply to their audio span, whereas hallucinated tokens exhibit erratic or backward-shifting patterns.

Controlled Unrolling for Static-Graph NPUs

Autoregressive decoding with incrementally growing KV-caches is incompatible with fully static-graph NPUs (e.g., Qualcomm QNN HTP). NPUsper compiles static chunked decoder graphs (K-step unrolling) that amortize graph-dispatch overhead while only computing the exact minimum KV-cache attention needed per step. This chunked structure enables early-exit decoding, chunk-based hypothesis emission, and reduces redundant attention computations by up to 36× depending on the dynamic length of decoding.

Figure 4: Fixed full-length graphs compute attention over the maximal cache every step. NPUsper’s chunked approach executes only necessary self-attention and reduces dispatches.

This results in a decoding schedule that advances in K-token steps, minimizing both wasted computation and graph-switches—a key bottleneck in mobile NPU implementations.

System Pipeline

Each inference round processes only the newly arrived audio concatenated with minimal validated carryover (as opposed to the entire window or previously seen audio). Input construction and bucketization enable efficient mapping to NPU-supported static input shapes. NPUsper thereby executes Whisper end-to-end within strict energy, latency, and memory constraints of commodity phones and laptops.

Figure 5: Overall NPUsper pipeline: short input extraction, bucketing, cross-attention-based token validation, and dynamic carryover for streaming.

Experimental Results

NPUsper is evaluated on Samsung Galaxy S25 and Snapdragon X Plus devices against four contemporary Whisper streaming systems, including Whisper-Streaming, WhisperFlow, SimulStreaming, and Simul-Whisper. Metrics include per-word latency, time-to-first-token (TTFT), power consumption, and WER/CER.

Results demonstrate:

• 4.84× reduction in per-word latency over baselines.
• 33.2× reduction in TTFT relative to state-of-the-art.
• 88.6% lower average power consumption on S25.
• WER differences remain within a narrow margin (≤1pp), showing NPUsper does not trade accuracy for efficiency.

Figure 6: Per-word latency and WER on Samsung Galaxy S25: NPUsper achieves lowest latency without significant WER degradation.

Figure 7: Average Whisper input lengths per inference: NPUsper consistently uses shortest input, reflecting no redundant audio processing.

Component ablation confirms the substantial role of cross-attention-based hallucination detection and controlled unrolling in delivering these gains.

Practical and Theoretical Implications

Practical Impact: NPUsper enables deployment of full-scale Whisper models in live, real-time, on-device scenarios without custom model retraining, architecture modification, or accuracy compromise, making high-quality speech transcription feasible on unmodified mobile hardware. Power and throughput improvements directly translate to extended device battery life and improved user experiences for privacy-relevant applications.

Systems-Level Insight: The joint design of execution-aware model pipelines, specifically the alignment between speech token temporal structure and hardware constraints (static-graph NPUs), is shown to unlock orders-of-magnitude efficiency gains unavailable to naïve porting or traditional adaptation approaches.

Theoretical Ramifications: The use of cross-attention monotonicity as a universal grounding signal is likely extensible to other encoder-decoder models and generative tasks, with implications for robust model output validation in other domains prone to hallucination. This approach is robust across multiple Whisper model sizes and architectural variants, opening avenues for future work with even larger or more complex models, given sufficient on-device memory.

Future Directions

Areas for continued exploration include:

• Extending this paradigm to very large speech foundation models (e.g., SeamlessM4T, Canary) and further refining subword/punctuation handling in the hallucination validation pipeline.
• Exploring joint pipelining across CPU, GPU, and NPU for even greater resource efficiency.
• Generalizing cross-attention-grounded hallucination detection to other streaming generative tasks (e.g., real-time translation, captioning).

Conclusion

NPUsper establishes a new state-of-the-art for real-time Whisper deployment on mobile NPUs, combining robust token-level validation with hardware-aware execution. It empirically and architecturally resolves the longstanding gap between foundation ASR models’ streaming requirements and mobile resource constraints, setting a template for future streaming model deployment and lightweight, high-fidelity, on-device inference systems.