NPUsper: Efficient Live Transcription on Mobile NPUs

• NPUsper is a live transcription system that enhances Whisper on mobile NPUs by eliminating redundant padded audio processing and decoder computations.
• It employs online hallucination detection via decoder cross-attention and controlled unrolling to significantly reduce latency (up to 4.84x) and power use (up to 88.64%).
• Integrating adaptive chunked decoding and dynamic time warping, NPUsper achieves efficient real-time transcription without altering the original Whisper architecture.

NPUsper is a live transcription system that makes Whisper efficient on mobile NPUs by eliminating redundant computation. It targets two inefficiencies identified in prior Whisper streaming pipelines: long padded and overlapping audio windows used to suppress hallucinations, and the mismatch between autoregressive decoding and the static-graph execution style of mobile NPUs. NPUsper addresses these with online hallucinated-token detection from decoder cross-attention and controlled unrolling for NPU-friendly decoding, and reports up to 4.84x lower per-word latency, up to 33.2x lower time-to-first-token (TTFT), and up to 88.64% lower average power consumption compared with baselines, while maintaining comparable transcription accuracy (Lee et al., 1 Jul 2026).

1. Problem setting and design objective

NPUsper was proposed for live transcription with Whisper under mobile-NPU deployment constraints. The motivating observation is that Whisper behaves poorly on short streaming inputs because it was trained on fixed 30-second segments. Without padding, short inputs often trigger hallucinations—repeated or ungrounded text. Existing systems therefore pad or overlap audio to make the input look more like Whisper’s training distribution. The reported motivation experiment states that this improves WER/CER, but padding to 30 seconds causes about a 3.8× increase in encoding time (Lee et al., 1 Jul 2026).

A second inefficiency arises from the decoder. Mobile NPUs require static graphs with fixed tensor shapes, whereas Whisper decoding is autoregressive and its KV cache grows token by token. The paper contrasts two unsatisfactory deployment options: a fixed full-length graph that wastes computation on a mostly empty KV cache, or step-specific graphs that incur heavy graph-dispatch overhead. Qualcomm’s fixed-shape design is given as an example, with a maximum decode length $M=200$. NPUsper formalizes the resulting waste as

$$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$

For $M=200$, stopping at $T=10$ implies a 36.4× reduction in self-attention MACs compared with the fixed full-length approach.

These two observations define NPUsper’s design objective: reduce redundant padded-audio processing and reduce redundant decoder computation without changing Whisper’s architecture. In that sense, NPUsper is organized as a systems approach rather than a model redesign.

2. Online hallucinated-token detection

NPUsper’s first technical contribution is a lightweight online hallucination detector based on temporal structure in decoder cross-attention. The paper’s key insight is that valid tokens tend to exhibit monotonic temporal grounding: as the transcript advances, the attention focus should generally move forward through the audio. Hallucinated tokens, by contrast, often exhibit backward shifts in attention over time (Lee et al., 1 Jul 2026).

The detector is defined from head-averaged frame-wise cross-attention for token $i$ at decoder layer $l$: $$a_i^{(l)}(f)=\frac{1}{H}\sum_{h=1}^{H}\mathrm{CrossAttn}_i^{(l,h)}(f),$$ where $H$ is the number of heads and $f$ indexes audio frames. NPUsper uses the final decoder layer, which the paper states has the clearest temporal alignment: $\tilde{a}_i(f)=a_i^{(L)}(f).$ For token $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$0, relative to the most recent preceding content token $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$1, it computes

$$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$2

then applies a median filter followed by a moving average: $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$3 The relevant peak locations are

$$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$4

The decision rule is threshold-free: token $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$5 is considered valid if

$$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$6

and hallucinated otherwise: $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$7

Implementation is restricted to content tokens. Punctuation and subword tokens are skipped because they are noisier temporal anchors. The appendix reports that filtering and token skipping materially improve robustness. Operationally, the detector allows NPUsper to stop decoding early when hallucination appears, emit only validated tokens, and update carryover state using dynamic time warping (DTW).

The reported ablations on 250 LibriSpeech samples quantify the effect of this design. Without padding, hallucination rate was 34.4%. Adding backward-shift detection reduced it to 4.4%, and the full configuration with smoothing, median filtering, and skipping punctuation/subwords reached 0.0% hallucination in that study. The same mechanism was also reported on other Whisper variants: on Whisper small, hallucination rate dropped from 24.0% to 2.0%; on Whisper medium, from 36.4% to 6.8%. This suggests that NPUsper is exploiting a temporal alignment property present beyond a single Whisper configuration.

3. Controlled unrolling for static-graph decoding

NPUsper’s second contribution is controlled unrolling, which is designed specifically for NPU-efficient autoregressive decoding. Instead of compiling either one full-length static graph that always computes attention over the maximum cache length or a separate graph per token step, NPUsper uses $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$8-step chunk graphs. Each graph generates up to $$\frac{\sum_{s=1}^{T}\mathrm{MAC}_{\text{fixed}(s)}}{\sum_{s=1}^{T}\mathrm{MAC}_{\text{ours}(s)}}= \frac{2M}{T+1}.$$9 tokens before returning control to the host. After each chunk, the system inspects the cross-attention outputs, applies hallucination detection, emits only valid tokens, and decides whether to stop or dispatch the next chunk (Lee et al., 1 Jul 2026).

The paper attributes two benefits to this organization. First, it improves KV-cache efficiency, because each chunk computes only the attention positions actually needed for the current step range rather than always computing over the maximum length. Second, it improves graph-dispatch efficiency by amortizing multiple decode steps into one graph launch.

The implementation uses schedules such as $M=200$0 for the default path and $M=200$1 when using prompt prefill. The first chunks handle Whisper’s special tokens and prompt tokens, while the later 5-token chunks are shared. The appendix compares chunk sizes 3, 5, and 7 and reports that 5-token chunks give the best latency balance: smaller chunks cause too many graph launches, while larger chunks waste more unused decoder work.

A common misconception is that NPUsper merely ports ordinary autoregressive decoding to an NPU runtime. The reported design is more specific: controlled unrolling changes the decoder execution unit from a token-level loop to a chunk-level static graph. The point is not only compatibility with fixed-shape execution, but also the elimination of unnecessary KV-cache computation and graph-dispatch overhead.

4. End-to-end execution path

At each inference round, NPUsper executes a fixed sequence of operations: it collects newly arrived audio plus carryover context; maps the audio length to a discrete NPU bucket; runs the encoder and decoder using controlled unrolling; retrieves token logits and final-layer cross-attention from each chunk; validates tokens using the temporal attention rule; emits valid tokens immediately; stops early on hallucination or end-of-chunk conditions; and updates carryover using DTW (Lee et al., 1 Jul 2026).

For variable-length streaming inputs, the runtime uses bucketed static graphs with audio buckets $M=200$2 seconds. Padded positions are masked with a finite negative bias of $M=200$3 before softmax so that they do not affect attention. The decoder also uses a dummy leading KV slot to avoid a zero-length-cache special case in compilation.

This execution path is central to NPUsper’s efficiency claim. Short real-audio inputs plus minimal carryover replace the long padded windows used by prior systems, while chunked static graphs replace either fixed full-length decoder graphs or one-graph-per-step execution. A plausible implication is that the two optimizations are interdependent: short-input streaming would not provide the reported benefit if decoder dispatch remained expensive, and chunked decoder graphs would not fully address the redundant work created by heavily padded audio windows.

5. Experimental evaluation

The reported hardware platforms are two commercial devices: Samsung Galaxy S25 and Snapdragon X Plus laptop. NPUsper is compared against four Whisper-based live transcription baselines: Whisper-Streaming (WS), WhisperFlow (WF), SimulStreaming (SS), and Simul-Whisper (SW). WS and SS use 30-second sliding windows; WF reduces input to about 15 seconds via a trained hush word. WS and WF use Local Agreement; SS and SW use AlignAtt. SS keeps tokens from dropped audio segments; SW discards them and uses an integrate-and-fire module for truncation detection. The main datasets are TED-LIUM 3 and Meanwhile, and the main end-to-end results use two representative TED-LIUM talks, EricMead and GaryFlake. Metrics are per-word latency, WER, TTFT, and power and energy (Lee et al., 1 Jul 2026).

Category	Samsung Galaxy S25	Snapdragon X Plus
Per-word latency	1.71–4.62× lower	1.69–4.84× lower
TTFT, Ours	2.2–12.8× lower	2.9–14.2× lower
TTFT, Ours-N	3.9–22.8× lower	6.9–33.2× lower
Average power, Ours / Ours-N	0.56 W / 0.46 W	1.42 W / 0.81 W
Active inference throughput, Ours-N	112.90 tok/s	144.98 tok/s

On S25, the table reports that Ours and Ours-N are 6.3–8.8× lower power than the baselines, and the paper also states that they achieve the best energy-per-inference and energy-per-token. On X Plus, Ours at 1.42 W and Ours-N at 0.81 W are again reported as the lowest among evaluated systems. Power on S25 is measured with a Monsoon High Voltage Power Monitor, and on X Plus with Qualcomm Profiler, subtracting idle power.

The paper emphasizes an accuracy trade-off different from a simple accuracy-maximization narrative. It states that NPUsper maintains comparable transcription quality overall and does not claim perfect accuracy improvements over every baseline in every dataset. The primary headline is that latency, TTFT, and power improve dramatically without a major accuracy sacrifice. This is important because the system’s objective is to remove inefficiency normally associated with aggressive streaming, not to redefine Whisper’s recognition behavior.

6. Relation to NPU systems research, limitations, and interpretation

NPUsper is explicitly described as a systems approach, not a model redesign. It does not change Whisper’s architecture, and its effectiveness depends on two preconditions identified in the paper: the existing temporal alignment behavior in Whisper’s cross-attention and a static-graph runtime infrastructure. Full end-to-end evaluation therefore focuses on Whisper base, while Whisper medium is described as not practical for the real-time setting because of long encoder time (Lee et al., 1 Jul 2026).

The reported limitations are correspondingly concrete. Hallucination detection is robust, but still relies on cross-attention temporal structure and chosen filter parameters. The NPU path needs careful bucket selection, masking, and graph lowering to work well. Controlled unrolling introduces a chunk-size trade-off between dispatch overhead and unused decode computation. These are not incidental details; they are part of the method’s deployment envelope.

Within the broader NPU literature, NPUsper belongs to a line of work that restructures execution to fit accelerator constraints rather than assuming GPU-like runtime flexibility. The OpenPangu speculative decoding study likewise targets NPU execution constraints by turning speculative verification into a static, on-device pipeline with precomputed tensors and zero-copy gathering (Dai et al., 3 Mar 2026). FlexNPU addresses a different layer of the stack, using a transparent user-space virtualization layer for Ascend NPUs to support phase-aware scheduling for LLM serving (Gu et al., 3 Jun 2026). Neu10 addresses cloud multi-tenancy through hardware-assisted NPU virtualization, a vNPU abstraction, and ISA support for fine-grained tensor operator scheduling (Xue et al., 2024). This suggests that NPUsper’s contribution is best understood not as an isolated Whisper optimization, but as part of a broader trend toward hardware-aware algorithm and runtime co-design for NPUs.

A second misconception is that NPUsper’s main result is simply “running Whisper on an NPU.” The paper’s actual claim is narrower and more technical: it eliminates two forms of redundant computation in streaming ASR—redundant padded audio processing and redundant decoder computation. The reported gains in per-word latency, TTFT, power, and throughput follow from that elimination strategy rather than from NPU deployment alone.