NeutronSparse: Coordinating Heterogeneous Engines for Sparse Matrix Multiplication on NPUs

Abstract: Sparse matrix-matrix multiplication (SpMM) is a fundamental data operation for large-scale sparse data processing. With NPUs increasingly deployed in data centers for their performance and energy efficiency, accelerating SpMM on these platforms is a natural choice. However, high-performance SpMM on NPUs poses a data management challenge, as irregular sparsity demands efficient data organization and scheduling. On Ascend 910B, the official MindSpore implementation achieves only 36.3% of the performance of GPU-based sparse libraries such as cuSPARSE on NVIDIA A100. To this end, we conduct an in-depth architectural analysis of SpMM execution on NPUs versus GPU and identify that the key performance bottleneck for SpMM on NPUs lies in the lack of efficient coordination across heterogeneous compute units under tile-based execution model. Therefore, we propose NeutronSparse, a coordination-first SpMM framework for NPUs. NeutronSparse integrates two key techniques: (i) Sparsity-aware coordination of heterogeneous engines, which adaptively partitions and balances workloads between heterogeneous compute units to keep them busy, and (ii) Locality-aware tile orchestrating, which reorganizes and reuses data tiles to reduce redundant computation and memory movement overhead. Evaluations on Ascend 910B show that NeutronSparse achieves 1.26x-7.78x speedup over NPU baselines and 1.03x-3.07x speedup over leading GPU libraries on NVIDIA A100, revealing untapped potential of NPUs for sparse computation.

• The paper introduces NeutronSparse, a framework that enhances SpMM performance on NPUs by coordinating heterogeneous compute engines and addressing tile-level redundancy.
• It employs a cost model-driven partitioning and global-local tile reordering to balance workloads and maximize runtime concurrency.
• Experimental results demonstrate speedups from 1.26× to 7.78× over NPU baselines and competitive performance against GPU-optimized libraries.

NeutronSparse: Coordinating Heterogeneous Engines for Sparse Matrix Multiplication on NPUs

Introduction

Sparse matrix-matrix multiplication (SpMM) is a central operation in large-scale AI and graph analytics, underpinning workloads such as GNN aggregation and sparse transformer attention. Neural Processing Units (NPUs) are increasingly deployed to accelerate such workloads, but SpMM performance on NPUs such as the Huawei Ascend 910B lags substantially behind that of GPU-optimized libraries like cuSPARSE, despite equivalent compute and bandwidth specifications. The main architectural obstacles are inefficient coordination among heterogeneous compute engines and redundancy introduced by tile-centric SIMD execution in the presence of irregular sparsity.

Architectural Analysis of NPUs vs. GPUs

This work provides a granular analysis of the architectural asymmetries that impede SpMM performance on NPUs. On NVIDIA GPUs, SIMD-style workload partitioning employs small tiles, and core/thread-level mapping enables fine-grained load balance. However, in NPUs, workloads are dispatched to heterogeneous engines (e.g., the AI Cube "AIC" for matrices and AI Vector "AIV" for vectors) that must be explicitly coordinated, and computation executes at a coarse fixed tile size (e.g., $16\times16$ for AIC), leading to significant redundant computation on zeros when sparsity is high. The independence of AIC and AIV pipelines on Ascend enables true concurrency, but prior NPU SpMM implementations either overuse a single engine or fail to balance workloads, resulting in suboptimal resource utilization.

Figure 1: Comparison of SpMM performance between the Ascend 910B and Nvidia A100 highlights the performance gap motivating NeutronSparse.

Figure 2: Resource allocation in NPU AI Cores and GPU SMs, emphasizing differing concurrency and resource contention patterns.

Figure 3: Overview of the Nvidia GPU architecture.

Figure 4: Effective overlap between heterogeneous compute engines is significantly higher on NPU than GPU, supporting concurrent usage.

NeutronSparse Framework: Contributions and Design

NeutronSparse is a coordination-first SpMM framework designed for tile-centric, matrix-centric NPUs such as Ascend 910B. Its chief innovations are:

1. Sparsity-Aware Heterogeneous Coordination: NeutronSparse partitions workloads between AIC and AIV based on an architecture-aware cost model that empirically derives a density threshold $\alpha$ for each tile, aligning the tile assignment with the observed balance of throughput between engines. During runtime, NeutronSparse continuously monitors engine progress and adaptively migrates tiles to maintain load balance and runtime concurrency.
2. Locality-Aware Tile Orchestrating: To address tile-level redundancy and exploit explicit memory controls, NeutronSparse leverages a lightweight global-local tile reordering. This process forms denser tiles by clustering structurally-correlated rows and columns, then refines tile alignment within clusters (row windows), enhancing compute and data locality. Hierarchical tile reuse orchestrates shared $B$ tiles across cores and shapes intra-core tiles to maximize buffer utilization, considering architectural constraints (e.g., double-buffered L0A/L0B/L0C).

   Figure 5: Workflow of NeutronSparse, illustrating partitioning, reordering, and overlap of AIC/AIV.

   

   Figure 6: SpMM execution on heterogeneous AIC and AIV engines; sparse tiles mapped to AIV, dense to AIC.

   

   Figure 7: Heterogeneous workload partitioning. Two-stage row–column extraction separates sparse and dense regions, assigned to AIV and AIC.

   

   Figure 8: Load migration under runtime load imbalance balances pipeline progress between engines.

Key Mechanisms

Heterogeneous Workload Partitioning

NeutronSparse first profiles hardware throughput to empirically derive a cost model: for any tile, AIV runtime is proportional to the number of nonzeros and AIC time is proportional to tile size. The data-driven partitioning is replaced by architecture-aware partitioning, using a threshold $\alpha$ computed as:

$$\alpha = r \cdot \frac{P_{\text{AIV}}}{P_{\text{AIC}}}$$

where $r$ accounts for the ratio of AIV to AIC instances per core. The two-stage extraction first partitions out sparse rows for AIV, then dense-core columns for AIC, minimizing work imbalance.

Adaptive Work Migration

Inter-epoch profiling of AIC/AIV latencies is used to identify progress skew; load migration (at tile granularity) is then dynamically performed to eliminate bubbles and maximize concurrency.

Locality-Aware Tile Orchestrating

Global-local reordering (Figure 9, Figure 10) groups structurally similar rows and columns, then performs intra-cluster refinement by row-window reordering, resulting in increased tile density and improved alignment with SIMD execution.

Figure 9: Global Reordering positions nonzeros into clusters shared by row and column locality.

Figure 10: Local Reordering refined at row-window granularity for tile density.

Hierarchical tile reuse plans the residency of $B$ tiles in the L2 for cross-core sharing and shapes tile sizes to maximize compute under buffer constraints (e.g., optimal $(M,N,K)=(128,256,64)$ from detailed analysis).

Quantitative Evaluation

Extensive experimental comparisons are conducted on an Ascend 910B and NVIDIA A100 (A100 for baseline). NeutronSparse achieves 1.26× to 7.78× speedup over NPU baselines, and 1.03× to 3.07× speedup over state-of-the-art GPU libraries such as cuSPARSE, DTC-SpMM, and HC-SpMM across a wide matrix spectrum.

Figure 11: SpMM performance on Ascend 910B and Nvidia A100 across benchmark datasets.

Ablation studies isolate performance contributions: AIV-AIC coordination alone yields an average of 3.71× speedup over AIV-only and 1.98× over AIC-only; locality-aware tile orchestrating delivers further 2.17× improvement, with reordering and tile reuse addressing tile-level redundancy.

Figure 12: Performance gain of AIV-AIC Coordination on representative datasets.

Convergence of online workload migration is rapid—even under extreme initial imbalance, adaptive migration aligns the pipeline within few epochs, minimizing warmup overhead.

Figure 13: Convergence of online workload migration in extreme initial-skew cases.

Sensitivity analysis shows the cost model–guided threshold for partitioning is robust; final balance is more dependent on runtime migration than minor variations in initial split.

Power and cost analyses show that NeutronSparse delivers higher throughput per watt and per dollar than GPU designs after normalization. When evaluating TFLOPS-normalized SpMM (relative to raw peak) on NVIDIA H100, Ascend 910B with NeutronSparse provides higher efficiency, indicating that efficient sparse computation can be achieved on NPUs even with lower absolute peak compute.

Figure 14: Power draw during training, showing lower consumption for NeutronSparse on Ascend.

Figure 15: Cost-normalized SpMM throughput; NeutronSparse yields the highest performance-per-dollar.

Figure 16: TFLOPS-normalized SpMM performance comparing Ascend 910B and NVIDIA H100.

Portability and Generalization

NeutronSparse’s design—reliant on explicit tile-based execution and heterogeneous engine coordination—is shown to be portable to other NPUs such as Cambricon MLU, with consistent speedups observed across devices once backend-specific scheduling is adapted.

Figure 17: SpMM performance on Cambricon MLU demonstrates cross-NPU portability.

Implications and Future Directions

NeutronSparse demonstrates that efficient SpMM on NPUs demands architecture-aware scheduling beyond standard GPU strategies. The explicit mapping of heterogeneous engines, coupled with tile-centric data orchestration tailored to NPU buffer microarchitecture, is essential to harnessing available compute. NeutronSparse suggests that future NPU programming frameworks should elevate tile-level scheduling and heterogeneous pipeline concurrency to first-class abstractions. Forward-looking developments will likely include learning-based workload partitioning for rapid adaptation in dynamic workloads and further portability layers across emerging NPU designs.

Conclusion

NeutronSparse systematically addresses major NPU SpMM bottlenecks via coordinated heterogeneous engine scheduling and locality-optimized tile management. These techniques close—and in many cases, reverse—the traditional gap between NPU and GPU sparse computation efficiency, with significant implications for the deployment of NPUs in next-generation AI clusters. The methodology is generalizable to other tile-centric, heterogeneous architectures, marking an important direction for the scalable acceleration of irregular, sparse workloads.

[Paper: "NeutronSparse: Coordinating Heterogeneous Engines for Sparse Matrix Multiplication on NPUs" (2606.22482)]