SegFold: Accelerating Sparse GEMM with a Fine-Grained Dynamic Dataflow

Abstract: Generalized sparse matrix-matrix multiplication (SpGEMM) is critical in many domains. Existing CPUs and GPUs, as well as specialized accelerators, rely on static dataflows (e.g., inner product, outer product, Gustavson, etc.). Each static dataflow sacrifices some data reuse opportunities and imposes constraints on load balance. To address this inefficiency, we extend the typical SpGEMM dataflows by considering dynamism. Specifically, we add fine-grained dynamic scheduling to optimize reuse and reduce resource contention. We also develop dynamic remapping of partially completed work to improve load balance and parallelism. These ideas are formalized into a specific dataflow called Segment. To demonstrate Segment, we codesign a SpGEMM accelerator called SegFold. SegFold includes a memory controller that identifies fine-grained reuse opportunities in a local window of the stationary input array and exploits them through dynamic work assignment. It also incorporates a merge network that routes inputs to appropriate processing elements (PEs) for computation while dynamically remapping the work assigned to each PE to balance load. Across diverse densities and matrix sizes, SegFold achieves a geometric-mean $1.95\times$ speedup over state-of-the-art SpGEMM accelerators and $5.3\times$ over the best static dataflow configuration, demonstrating that adding dynamism to the dataflow design space unlocks reuse and load-balance gains that no static scheduling choice can achieve in isolation.

• The paper introduces SegFold, a novel fine-grained dynamic dataflow for sparse GEMM that integrates dynamic scheduling (SELECTA) and mapping (SEGMENTBC).
• It demonstrates significant performance improvements with a geometric-mean speedup of 5.3× over static dataflows and 1.95× over dynamic baselines.
• Experimental evaluations confirm robust load balancing, enhanced data reuse, and adaptability across varying matrix sizes, densities, and sparsity patterns.

Fine-Grained Dynamic Dataflow for SpGEMM: An Analysis of SegFold

Motivation and Limitations of Existing Approaches

Sparse generalized matrix-matrix multiplication (SpGEMM), where both operands exhibit unstructured sparsity, remains a core workload in areas such as scientific computing, graph analytics, and increasingly in ML applications like pruned DNNs and sparse attention. State-of-the-art SpGEMM accelerators, despite substantial architectural sophistication, have primarily relied on static dataflows—inner product, outer product, or Gustavson—each trading off data reuse, locality, and load balancing in ways that are sensitive to the sparsity patterns of the input matrices. While some accelerators attempt limited dataflow adaptation at the tile level, the scheduling within tiles remains essentially static, inherently sacrificing opportunities for fine-grained reuse and dynamic load balancing.

Static scheduling cannot simultaneously exploit reuse across all operands (A, B, and C), nor flexibly adapt to highly irregular and data-dependent sparsity. This leads to underutilization of compute resources, increased data movement, and poor scaling with unstructured sparsity. The consequence is that, even in highly customized architectures, significant headroom for performance and efficiency improvements remains.

Segment Dataflow: Fine-Grained Dynamism in Sparse GEMM

The SegFold accelerator introduces a new dataflow paradigm, Segment, which is fundamentally dynamic at a fine spatial and temporal granularity within each tile. Segment dataflow integrates two main dynamic strategies:

• Dynamic Scheduling (SELECTA): Rather than processing (m, k) pairs in a static nested order, SELECTA maintains an active window of k values (dimension K) and dynamically selects and reorders (m, k) pairs on the fly to maximize B-row reuse and parallelism, while avoiding C-row contention. This allows the dataflow to exploit both intra- and inter-row sparsity, and to opportunistically schedule work based on runtime patterns.
• Dynamic Mapping (SEGMENTBC): Partial sums are dynamically routed and recombined in a virtual coordinate space representing C, using an adaptive merge network. The virtual mapping evolves as new nonzeros in the output are encountered at runtime, enabling load balancing and locality optimizations unreachable by static schemes. This mapping is backed by a hardware-efficient index-to-PE mapper using binary search for minimal segment traversal length.

Additionally, SegFold adopts architectural mechanisms such as spatial and temporal folding to map irregular logical output rows onto a regular PE array, further boosting hardware utilization.

SegFold Microarchitecture

The microarchitecture realizes the Segment dataflow using a two-dimensional PE array with row-stationary allocation at the PE-row level. Key hardware features include:

• Memory Controller: Maintains active windows and metadata in both A (column-major, bitmask) and B (row-major, compressed), orchestrating SELECTA and SEGMENBC for each iteration.
• Adaptive Merge Network and IPM: Each PE row implements a merge network supporting on-the-fly redistribution of B elements based on their column index, with an IPM for legal and near-optimal segment injection.
• Spatial and Temporal Folding: Logical output rows exceeding PE row capacity are dynamically mapped onto multiple PE rows (spatial folding), while temporal folding uses per-row scratchpads to offload surplus partials, alleviating PE pressure and accommodating unbounded row lengths.
• Vector Multicast Support: Enables concurrent loading and mapping of multiple B rows into different PE rows, exposing higher degrees of inter-row parallelism.

Crucially, control and data-movement logic are designed to scale linearly with the number of PEs, and all dynamic mechanisms operate with small, hardware-realizable state.

Experimental Evaluation and Numerical Results

Comprehensive evaluation using real-world (SuiteSparse) and synthetic datasets highlights several core findings:

• Performance: SegFold achieves a geometric-mean speedup of 1.95× over the dynamic, window-adaptive baseline Spada, and 5.3× over the best-performing static (Flexagon) dataflow on a diverse suite of sparse matrices (including highly nonuniform and unstructured cases).
• Component Attribution: An ablation study demonstrates that dynamic scheduling (SELECTA) is the largest contributor to speedup, especially under high sparsity and irregularity, with dynamic mapping, spatial folding, and optimal mapping logic each providing additive gains.
• Sensitivity: SegFold scales robustly across a wide range of matrix dimensions, densities, and aspect ratios. Sensitivity experiments confirm that window size (for scheduling) and multicast width (for B delivery) both reach diminishing returns with moderate hardware provisioning. Input asymmetry is handled adaptively by operand role selection.
• Hardware Cost: Post-synthesis area (0.16 mm² at 7 nm, ~1.35 mm² projected at 28 nm) and power (386 mW/2.3 W for 256 PEs) are comparable to static architectures, confirming the practicality of the additional dynamic logic.
• Load Balancing and Reuse: Across all patterns, SegFold more fully exploits intra-tile reuse of A, B, and C—no static or tile-static dataflow achieves this combination of properties.
• Contradictory/Bold Claims: The results show that no static dataflow or statically-adaptive scheduling can achieve the observed combination of data reuse and load balancing, and that dynamic techniques are essential to approach optimal PE utilization in the presence of unstructured sparsity.

Implications and Future Prospects

From a theoretical perspective, SegFold demonstrates that the dataflow design space for sparse GEMM is richer than previously formalized, with dynamic scheduling and mapping along spatial and temporal axes representing orthogonal and previously unexploited dimensions. This hints at a broader theory of sparse computation wherein dynamic per-tile reordering provides provably higher bounds on data reuse and resource utilization.

Practically, SegFold offers a concrete path toward efficient sparse arithmetic for workloads with unpredictable or frequently changing sparsity patterns, such as dynamic graphs, online learning, and iterative solvers. Its runtime-only approach is particularly suitable for use cases where pre-processing is expensive or infeasible, and it complements tile-reordering or inter-tile locality-enhancing strategies.

Future work includes extending these techniques to other irregular kernels, such as SpMM and batched operations, hardware/software co-optimization for composability with pre-processing strategies, and co-design with reconfigurable dataflow architectures. Hardware-efficient generalizations for mixed-precision and quantized domains also appear promising.

Conclusion

"SegFold: Accelerating Sparse GEMM with a Fine-Grained Dynamic Dataflow" (2606.26701) introduces a fundamentally new approach to sparse GEMM acceleration centered on dynamic scheduling and adaptive mapping within a tile. Empirical evidence demonstrates substantial and consistent performance gains over both static and adaptively-static baselines, with efficient hardware cost scaling. The results establish the necessity and feasibility of fine-grained dynamism in the architectural and dataflow domains for efficient execution of irregular sparse workloads, and pave the way for future research in dynamic dataflow accelerator design.