NeutronSparse: Coordinated Sparse SpMM on NPUs

• NeutronSparse is a framework designed for efficient sparse matrix-matrix multiplication on Ascend 910B NPUs by combining sparsity-aware coordination with locality-aware tile orchestrating.
• It addresses heterogeneous engine imbalance and redundant tile computation through adaptive workload partitioning, reordering, and migration mechanisms.
• Evaluations show 1.26×–7.78× speedup over NPU baselines and enhanced energy efficiency compared to leading GPU libraries like cuSPARSE.

NeutronSparse is a coordination-first SpMM framework for NPUs that targets sparse matrix-matrix multiplication under the tile-based execution model of Ascend 910B. It is motivated by the observation that, although SpMM is a fundamental data operation for large-scale sparse data processing and NPUs are increasingly deployed in data centers for their performance and energy efficiency, high-performance SpMM on NPUs poses a data management challenge because irregular sparsity demands efficient data organization and scheduling. The framework is built around an architectural claim: the key performance bottleneck for SpMM on NPUs lies in the lack of efficient coordination across heterogeneous compute units under tile-based execution model. NeutronSparse therefore combines sparsity-aware coordination of heterogeneous engines with locality-aware tile orchestrating, and the reported evaluations on Ascend 910B show 1.26x-7.78x speedup over NPU baselines and 1.03x-3.07x speedup over leading GPU libraries on NVIDIA A100 (Ai et al., 21 Jun 2026).

1. Problem setting and motivation

Sparse matrix-matrix multiplication is the target operation. On Ascend 910B, the official MindSpore implementation achieves only 36.3% of the performance of GPU-based sparse libraries such as cuSPARSE on NVIDIA A100. The paper attributes this gap not to a generic incapacity of NPUs for sparse computation, but to the interaction between irregular sparsity and the NPU execution substrate, where data organization, scheduling, and cross-engine coordination become first-order determinants of throughput (Ai et al., 21 Jun 2026).

The framework is explicitly designed for the case in which sparse and dense work must be mapped onto heterogeneous compute units. Its central premise is that keeping heterogeneous engines busy is at least as important as reducing arithmetic work. This shifts optimization away from a purely kernel-centric view toward a system-level scheduling and orchestration view. A plausible implication is that, for tile-centric accelerators, sparse performance is constrained as much by coordination policy as by raw peak FLOPS.

2. Architectural basis: GPUs, NPUs, and the source of inefficiency

The analysis contrasts GPUs and Ascend NPUs at the execution-model level. GPUs such as NVIDIA A100 use a SIMT model: thousands of lightweight threads and Tensor Cores share SM-local resources, and sparse and dense portions are usually executed serially, for example first CUDA-core kernels on a warp-per-nonzero basis and then Tensor-Core GEMMs on dense blocks, with little true overlap. Ascend NPUs such as 910B use a SIMD-style, tile-centric model. Each AI Core has two independent vector engines (AIV) and one matrix engine (AIC), each with its own instruction stream and local buffers. AIC uses L0A/L0B/L0C, AIV uses UB, and data movement across HBM↔L2/L1/L0 is software-controlled via the Memory Transfer Engine and the FixPipe path (Ai et al., 21 Jun 2026).

Two bottlenecks are identified. The first is heterogeneous-engine imbalance: AIV-only baselines leave AIC idle, while AIC-only baselines leave AIV idle. The second is tile-granularity redundancy: because AIC must process whole $t \times t$ tiles, even tiles with few nonzeros incur full $t^2$ work. The measured zero-fraction inside active tiles rises from $\sim 0.33$ at $16 \times 16$ to $\sim 0.84$ at $64 \times 64$. In other words, tile activation alone is not an adequate proxy for useful work, because active tiles may still be overwhelmingly empty.

This architectural diagnosis is the basis for the framework’s two main components. One addresses engine imbalance by workload partitioning and online rebalancing; the other addresses redundant tile work and memory movement by reordering, reuse, and tile shaping.

3. Sparsity-aware coordination of heterogeneous engines

NeutronSparse builds a two-step pipeline to split workloads by sparsity and hardware balance and then adaptively rebalance at runtime. Its execution models are intentionally asymmetric. The AIV kernel is sparse-vector style: for each nonzero $A[i,j]$, it gathers $B[j,*]$ into UB and performs ScatterAdd into $C[i,*]$, with cost proportional to NNZ. The AIC kernel is dense-tile GEMM style: for each active A-tile of size $M \times K$, it gathers corresponding rows of $t^2$0 into L0B, loads the A-tile into L0A, runs a full MatMul on $t^2$1, and accumulates in L0C, with cost proportional to $t^2$2 even if many zeros are present (Ai et al., 21 Jun 2026).

The static partitioning phase derives a sparsity threshold $t^2$3 that equilibrates AIV and AIC steady-state throughput. Let $t^2$4 be measured fp16-vector throughput in elements/sec and $t^2$5 be measured fp16-tile throughput in elements/sec of tile volume. For a tile of size $t^2$6 with NNZ nonzeros, the costs are

$t^2$7

and

$t^2$8

Accounting for $t^2$9 AIVs versus $\sim 0.33$0 AIC, the framework sets

$\sim 0.33$1

This is converted into a nonzero-length threshold using $\sim 0.33$2 and $\sim 0.33$3. Partitioning then uses two-stage row-column extraction by linear scan: first, all rows whose $\sim 0.33$4 are extracted to the AIV stream, with remaining rows sent to the initial AIC submatrix; second, within the AIC submatrix, $\sim 0.33$5 is computed and sparse columns are extracted back to AIV. The resulting disjoint sets are $\sim 0.33$6 in COO and $\sim 0.33$7 in bitmap/blocked-tile form.

Static partitioning is supplemented by adaptive, epoch-level rebalancing. At the end of each epoch the framework records $\sim 0.33$8 and $\sim 0.33$9, defines $16 \times 16$0, and, if $16 \times 16$1, migrates part of the remaining workload $16 \times 16$2 from the slower engine to the faster one. If AIC is slow, it picks the sparsest tiles in $16 \times 16$3, splits them into nonzero vectors, and offloads them to AIV. If AIV is slow, it finds the densest rows in $16 \times 16$4, merges them into $16 \times 16$5 tile-aligned blocks, and offloads them to AIC. The migrated sub-work $16 \times 16$6 is chosen so that

$16 \times 16$7

This balance condition is intended to keep the two engines balanced and overlapping.

4. Locality-aware tile orchestrating

The second major component is locality-aware tile orchestrating, which aims to reduce redundant computation and memory movement overhead. Rather than fine-grain per-entry shuffles, the framework uses a two-level reordering strategy to maximize tile-level density cheaply. Global reordering applies Rabbit Order on the bipartite graph of rows↔columns in order to coarsely cluster strongly related rows and columns and bring nonzeros into contiguous blocks. Local reordering is then applied within each cluster: the cluster is partitioned into row windows of height $16 \times 16$8; for each window, a seed row is selected; the $16 \times 16$9 most similar rows are chosen greedily using Jaccard over nonzero-column sets; and fixed-shape tiles are built by dropping empty columns (Ai et al., 21 Jun 2026).

Tile reuse is hierarchical. For inter-core reuse via L2, the framework identifies the most-frequently accessed B-rows for each cluster, marks these rows as cacheable in L2, and floods them via the MTE before any core computes that cluster; all AI Cores bypass L2 for other rows. For intra-core reuse via tile shaping, tile dimensions $\sim 0.84$0 are chosen to maximize $\sim 0.84$1 subject to double-buffer L0 constraints,

$\sim 0.84$2

$\sim 0.84$3

$\sim 0.84$4

with $\sim 0.84$5 KB, $\sim 0.84$6 KB, and $\sim 0.84$7. The output-write stride $\sim 0.84$8 bytes is also aligned to the 512-byte FixPipe boundary, which requires $\sim 0.84$9 to be a multiple of 128. The reported default is $64 \times 64$0, which fully utilizes L0A/L0B, saturates L0C, and aligns writes.

Additional kernel optimizations refine the same locality and balance objective. AIV Vector-Tile Merging groups gathered B rows sharing the same C-row into a 128-wide fused vector to boost lane utilization. AIC Row-Window List Reordering interleaves heavy and light row windows in the compute queue per core to reduce stragglers. Double-Buffer Pipelining overlaps MTE→L0 data movement with MatMul in both engines.

5. Implementation on Ascend 910B

The implementation is on Ascend 910B using CANN v8.0.1 and custom HCCL kernels for AIC and AIV. The hardware configuration reported for each AI Core consists of an AIC with a $64 \times 64$1 fp16 Cube Unit and buffers L0A/L0B of 64 KB each plus L0C of 256 KB; an AIV side with two 128-wide fp16 vector units and UB $64 \times 64$2 KB for gather/scatter; and shared resources including L2 cache of 20–32 MB, MTE, and FixPipe (Ai et al., 21 Jun 2026).

The data structures reflect the split execution model. $64 \times 64$3 is stored as bitmap plus list of active tile indices. $64 \times 64$4 is stored in COO as $64 \times 64$5 for gather/scatter. Matrix $64 \times 64$6 is split into hot rows, which are cacheable, and cold rows. Coordination is implemented with epoch timer hooks around the AIC/AIV launch, while a host-side scheduler checks skew and issues a small remainder migration plan before the next epoch.

This implementation profile is significant because the framework is not presented as a purely algorithmic abstraction detached from hardware. Its partitioning, caching, tile shaping, and migration policies are expressed directly in terms of AIC, AIV, L0 buffers, UB, L2, MTE, and FixPipe. This suggests that NeutronSparse is tightly coupled to the execution semantics of the Ascend tile-centric model.

6. Evaluation, ablations, and interpretation

The reported evaluation uses an NPU server with Ascend 910B, 24 AI Cores, 280 TFLOPS fp16, 1.8 TB/s HBM, and Ubuntu 18.04, and a GPU server with NVIDIA A100, 108 SMs, 312 TFLOPS fp16, 2.04 TB/s, and CUDA 12.3. The datasets comprise 4 GNN graphs—cora, ogbn-arxiv, reddit, amazon-product—and 16 SuiteSparse matrices with sizes 2 K–900 M NNZ, density 0.02–49 ×10⁻³, skew, and empty-tile fraction. NPU baselines are MindSporeGL as an AIV-only baseline and an AIC-only design; GPU baselines are cuSPARSE, DTC-SpMM, and HC-SpMM (Ai et al., 21 Jun 2026).

The reported performance results are as follows. Over NPU baselines, speedup is 1.26×–7.78× with average $64 \times 64$7. Over cuSPARSE on A100, speedup is 1.03×–3.07× with average $64 \times 64$8. Over DTC-SpMM and HC-SpMM, average speedup is $64 \times 64$9 and $A[i,j]$0, respectively. Ablation results attribute 3.71× and 1.98× average gains to engine coordination relative to AIV-only and AIC-only baselines. Online migration drives skew toward $A[i,j]$1 within $A[i,j]$2 epochs and yields 3.8× end-to-end speedup over static split. Global–Local Reorder alone gives 2.17× average, and Hierarchical Reuse contributes another 1.42×. The default tile size $A[i,j]$3 is best, with 1.3×–3.4× gains over tile-size alternatives in $A[i,j]$4. Preprocessing overhead is $A[i,j]$5 for partition and $A[i,j]$6 for reorder amortized over 200 epochs, and this is 4×–157× lower than global reorder in DTC-SpMM. Average power draw is 147.7 W versus 166 W for HC-SpMM, and cost efficiency measured as throughput per dollar is 1.43×–2.47× better than GPU baselines.

The paper attributes speedup to four mechanisms: compute utilization through true overlap of AIV and AIC, with overlap rate 2.1%–32.9% versus GPU −3.9%–1.3%; memory efficiency through hot-row L2 caching and double-buffering; reduced redundant work through tile-level clustering, which cuts zero-tile processing by up to 10× density improvement; and better per-core load balance through row-window migration. These results are reported for Ascend 910B, the stated baselines, and the stated datasets. This suggests that the claimed advantage is specifically linked to architecture-aware coordination and tile orchestration under a tile-centric NPU execution model, rather than to a universal dominance of NPUs over GPUs for every SpMM workload.