AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices

Abstract: Speculative decoding enhances the inference efficiency of LLMs by generating drafts using a small draft LLM (DLM) and verifying them in batches with a large target LLM (TLM). However, adaptive drafting inference on a mobile single-NPU-PIM system faces idle overhead in traditional operator-level synchronous execution and wasted computation in asynchronous execution due to fluctuations in draft length. This paper introduces AHASD, a task-level asynchronous mobile NPU-PIM heterogeneous architecture for speculative decoding. Notably, AHASD achieves parallel drafting on the PIM and verification on a single NPU through task-level DLM-TLM decoupling and specifically, it incorporates Entropy-History-Aware Drafting Control and Time-Aware Pre-Verification Control to dynamically manage adaptive drafting algorithm execution and pre-verification timing, suppressing invalid drafting based on low-confidence drafts. Additionally, AHASD integrates Attention Algorithm Units and Gated Task Scheduling Units within LPDDR5-PIM to enable attention link localization and sub-microsecond task switching on the PIM side. Experimental results for different LLMs and adaptive drafting algorithms show that AHASD achieves up to 4.2$\times$ in throughput and 5.6$\times$ in energy efficiency improvements over a GPU-only baseline, and 1.5$\times$ in throughput and 1.24$\times$ in energy efficiency gains over the state-of-the-art GPU+PIM baseline, with hardware overhead below 3% of the DRAM area.

• The paper introduces AHASD which decouples DLM drafting and TLM verification via asynchronous queues, improving throughput and energy efficiency.
• EDC employs entropy-history modeling to dynamically suppress low-confidence drafts, significantly recovering token acceptance and reducing computational waste.
• TVC uses cycle-level latency predictions to pre-verify tasks, ensuring smooth NPU utilization and achieving up to 4.2× throughput gains over GPUs.

AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices

Introduction

This paper introduces AHASD, an architecture explicitly designed to address inefficient load balance in speculative decoding for LLM inference on resource-constrained mobile devices with heterogeneous NPU-PIM hardware (2604.25326). With the rising demand for local execution of LLMs on edge devices, efficient speculative decoding that minimizes invocation of high-latency target models becomes fundamental. The paper identifies the inefficacy of operator-level synchronous mapping and the underexplored pitfalls of naive asynchronous drafting and verification, especially under adaptive drafting regimes. AHASD's novelty is manifested via strict task-level decoupling, coupled hardware-algorithmic controls (Entropy-History-Aware Drafting Control and Time-Aware Pre-Verification Control), and integration of dedicated attention processing in PIM.

Figure 1: Depiction of speculative decoding and adaptive drafting, illustrating the decoupling of draft generation and verification.

Speculative Decoding on Heterogeneous NPUs and PIM

Traditional speculative decoding approaches employ a lightweight DLM to produce draft tokens, subsequently batch-verified by a heavyweight TLM. The move toward adaptive drafting algorithms leverages runtime statistics (e.g., confidence metrics), dynamically varying draft length to mitigate wasteful computation in low-predictability regimes. On mobile NPU-PIM SoCs, DLM operations are memory-bound, while TLM verification is compute-intensive, suggesting an ideal workload split: DLM drafts on PIM and TLM verification on NPU.

Figure 2: Roofline model quantitatively characterizing speculative decoding, indicating memory intensity for drafting and compute intensity for verification.

Despite this, operator-level synchronous parallelism—the state of the art in efforts like SpecPIM—incurs significant mutual idling due to fluctuating draft length. Empirical system studies reveal severe imbalance and wasted throughput, particularly as the distribution of DLM workload is not well-matched to hardware characteristics under adaptive drafting.

Figure 3: Overhead imbalance analysis in operator-level scheduling under adaptive drafting, highlighting synchronization-induced inefficiency.

Key Innovations in AHASD

AHASD employs task-level decoupling enabled by asynchronous device-bridging queues: unverified drafts, feedback, and pre-verification, thus dissociating DLM and TLM pipeline phases. It directly leverages PIM for DLM, with additional AAU and Gated Task Scheduling units for in-place nonlinearity support and fast rank switching, respectively.

Figure 4: The architectural design of AHASD, highlighting NPU, CPU, asynchronous queues, and LPDDR5-PIM with attention and scheduling logic.

Entropy-History-Aware Drafting Control (EDC)

Adaptive speculative decoding using uncontrolled look-ahead drafting degrades token acceptance rate and amplifies PIM-side compute wastage. EDC utilizes a model of historical batch entropy and drafting lead, updating a Pattern History Table for per-batch decision making. EDC dynamically decides whether to proceed with drafting or suspend based on likelihood of downstream TLM acceptance, significantly suppressing low-confidence draft propagation.

Figure 5: Diagram of Entropy-History-Aware Drafting Control, showing entropy history processing and pattern learning for adaptive suppression.

Time-Aware Pre-Verification Control (TVC)

TVC incorporates online, cycle-granular latency prediction models for both DLM and TLM. When EDC halts speculative drafting due to low confidence, TVC evaluates available NPU-side execution slack and opportunistically launches PIM-side pre-verification on candidate batches. This two-stage gating avoids NPU starvation, enables dynamic insertion of pre-verification tasks, and guarantees the LLM pipeline remains throughput-optimal.

Figure 6: Illustration of Time-Aware Pre-Verification Control, including NPU/PIM latency model and controlled task interleaving.

Quantitative Results

Ablation studies on simulated platforms (OpenPIM and ONNXim) show the impact of each architectural and algorithmic contribution.

• Task-level asynchronous scheduling (NPU+PIM) yields a 2.2× average throughput versus GPU-only baselines but comes at a cost: average draft acceptance rate drops by 25.1%.
• Integrated attention computation (AAU) enables in-memory execution of nonlinearity, further raising throughput while reducing cross-chip transfer.
• EDC recovers acceptance rate (up by 24.6%) and boosts energy efficiency to 4.5× by suppressing invalid drafts on low-confidence prefixes.
• TVC ensures fine-grained non-blocking scheduling, increasing the average throughput gain to 3.8×.

  Figure 7: Results of ablation experiments, detailing the incremental benefits of asynchronous scheduling, AAU, EDC, and TVC.

When benchmarked against SpecPIM and GPU only, AHASD achieves:

• Up to 4.2× throughput and 5.6× energy efficiency improvement over GPU.
• Up to 1.5× throughput and 1.24× energy efficiency over SpecPIM in state-of-the-art adaptive drafting algorithms.

  Figure 8: Comparative evaluation on LLM benchmarks with state-of-the-art baselines, highlighting throughput and energy efficiency benefits.

Practical and Theoretical Implications

AHASD demonstrates that for LLM inference on mobile devices, operator-level parallel partitioning is inherently suboptimal under adaptive drafting regimes. Principle of decoupling at the task level, with feedback-informed suppression and hardware-algorithmic co-optimization, is substantially more effective. The EDC/TVC dual-controller paradigm, grounded in low-overhead statistical modeling and online runtime adaptation, minimizes PIM waste and NPU idling—central bottlenecks in edge LLM deployment.

The introduction of in-situ attention processing in LPDDR5-PIM opens a path toward even greater local model expressivity without substantial DRAM die overhead, as the integration cost remains below 3% of DRAM area. The architecture supports the future trend toward memory-centric AI acceleration.

Speculation and Future Directions

Potential extensions include support for multi-user prompt batching, more sophisticated online draft length predictors (e.g., leveraging full sequence entropy or context score heads), or further specialization of PIM-side compute for large-context LLMs. The principles from AHASD can be generalized to transformer-based vision and multimodal models, and further co-design with quantization schemes or token pruning may yield additional gains.

Hardware co-design for speculative decoding could evolve toward dynamic voltage/frequency scaling in coordination with confidence-based controls, essentially fine-tuning power provisioning based on EDC/TVC evaluations.

Conclusion

AHASD substantiates the performance and energy advantages of task-level asynchronous speculative decoding with hardware-algorithmic co-design for LLM inference on NPU-PIM systems. By combining tailored scheduling, entropy-driven draft suppression, and time-sensitive pre-verification, the architecture materially closes the gap in energy and throughput efficiency on mobile platforms relative to GPU or operator-level heterogeneous baselines. The low area and power overheads make AHASD a practical candidate for future edge AI SoCs targeting robust and responsive LLM inference (2604.25326).