SegFold: Dynamic Dataflow for SpGEMM

• SegFold is a dynamic accelerator for SpGEMM that integrates sub-tile scheduling with runtime remapping to improve both data reuse and load balance.
• It employs the Segment dataflow with coordinated SelectA and SegmentBC operations to interleave A and B processing while efficiently reducing C.
• The microarchitectural design on a 16×16 PE array achieves up to 1.95× speedup over existing accelerators by dynamically adapting to sparse matrix patterns.

SegFold is a codesigned dataflow and accelerator for generalized sparse matrix–matrix multiplication (SpGEMM) that introduces fine-grained dynamism into both scheduling and mapping, departing from the three canonical static schedules—inner product, outer product, and Gustavson. In the formulation considered, SpGEMM computes $C_{m,n} = \sum_k A_{m,k} \times B_{k,n}$ for sparse $A$ and $B$. The central claim is that static dataflows commit in advance to a single reuse–imbalance trade-off, whereas SegFold adds sub-tile dynamic scheduling to optimize reuse and dynamic remapping of partially completed work to improve load balance and parallelism (Wu et al., 25 Jun 2026).

1. Problem setting and motivation

SegFold is motivated by the observation that the classical SpGEMM loop orders each privilege one form of locality while constraining another. Inner product, ordered as $M \to N \to K$, reuses $C$ partial sums because $K$ is innermost, but exhibits poor input reuse of $A$ and $B$ and is sensitive to row–column intersection balance. Outer product, ordered as $K \to M \to N$, reuses entire rows or columns of $A$ and $A$0 because $A$1 is outermost, but produces large, irregular partial-sum matrices $A$2 and places $A$3 reuse far apart in time. Gustavson, ordered as $A$4, fully reuses rows of $A$5 and reuses $A$6 within each row, but does not reuse $A$7 and still faces irregular $A$8 reduction patterns (Wu et al., 25 Jun 2026).

The paper’s framing is that each static dataflow chooses a reuse pattern and a load-balance profile ahead of time. Real sparse patterns vary tile-to-tile and even within tiles, so a fixed schedule either loses reuse opportunities or suffers load imbalance. Recent adaptive schemes such as Spada and Flexagon are described as selecting among the three static schedules at tile granularity, while preserving a fixed execution order inside each tile. SegFold’s key insight is that adding dynamism at sub-tile granularity enables two simultaneous effects: reordering work within a window of $A$9 to maximize row-wise reuse of $B$0, and redistributing partial sums of $B$1 across processing elements (PEs) on the fly to balance load (Wu et al., 25 Jun 2026).

A recurrent misconception in this area is to equate “adaptive” execution with any runtime choice among inner product, outer product, and Gustavson. SegFold’s contribution is narrower and more specific: it does not merely switch among preexisting loop nests, but extends the dataflow space by adding dynamic scheduling and dynamic mapping within the tile itself. This suggests that the proposal is best understood not as a new point in the static design space, but as a mechanism for relaxing the commitment to any single static schedule.

2. The Segment dataflow

The dynamic dataflow formalized in the work is called Segment. Rather than fixing a single loop-nest order, Segment interleaves the $B$2 and $B$3 dimensions through two coordinated operations, called SelectA and SegmentBC (Wu et al., 25 Jun 2026).

SelectA performs fine-grained dynamic scheduling over an active window $B$4 of up to $B$5 distinct $B$6-values. At each cycle, it greedily selects up to $B$7 pairs $B$8 from $B$9 under two constraints: no two selected pairs may share the same $M \to N \to K$0, to avoid $M \to N \to K$1-row contention, and as many pairs as possible should share the same $M \to N \to K$2, to maximize reuse of a row of $M \to N \to K$3. The scheduler fills the active window with new $M \to N \to K$4 values until $M \to N \to K$5, tracks completed $M \to N \to K$6 values, and retires any $M \to N \to K$7 whose entire row is done. The effect is to break both the static $M \to N \to K$8 and $M \to N \to K$9 nests and to permit partial-row streaming of $C$0 by segmenting its $C$1 dimension.

SegmentBC performs on-the-fly element-wise redistribution after a batch of $C$2 pairs has been selected. Each selected pair carries a partial row $C$3, and each nonzero $C$4 is injected into a virtual coordinate space $C$5, where $C$6 indexes nonempty rows of $C$7 and $C$8 indexes the current sorted nonzeros within that row. A time-varying injective mapping

$C$9

is required to satisfy four conditions: injectivity, row saturation with no gaps in $K$0, column ordering so that $K$1 implies $K$2, and time-ascending motion in which entries move only to the right over time. When a new term $K$3 arrives, it either matches an existing $K$4 entry and is reduced in place, or is inserted into sorted position with rightward shifts of existing entries. The work defines a segment displacement for each injected element and states that minimizing average displacement reduces network contention (Wu et al., 25 Jun 2026).

The significance of Segment lies in its attempt to combine multiple forms of reuse that are separated in conventional static schedules. The paper characterizes this as dynamic exploitation of reuse across $K$5, $K$6, and $K$7, together with redistribution of reductions for better balance. A plausible implication is that Segment is not only a scheduling policy but also a structural constraint on how partial results may evolve over time, since sortedness, injectivity, and monotonic rightward movement are built into the mapping.

3. Microarchitectural realization in SegFold

SegFold realizes Segment with a 16×16 PE array and a microarchitecture organized around a memory controller, an adaptive merge network, an index-to-PE mapper, and folding mechanisms (Wu et al., 25 Jun 2026).

The memory controller and metadata path store $K$8 in column-major order and $K$9 in row-major order, with DCSR plus a start-pointer for partial rows. The controller maintains the active window $A$0, whose default size is up to 32, and scans the $A$1-bitmask of size $A$2 each cycle using pure combinational logic to drive SelectA. It also coalesces $A$3 and $A$4 accesses to the on-chip cache and then to HBM2.

Within each PE row, the adaptive merge network assigns one virtual row of $A$5 to an ordered sequence of PEs whose stored column indices $A$6 increase from left to right. For an incoming $A$7 element with column index $A$8, the network injects at a start position $A$9 and then propagates right with simple per-PE comparator logic. If $B$0, the element is forwarded rightward. If $B$1, it accumulates and stops. If $B$2, all downstream $B$3 values are shifted one position to the right and $B$4 is inserted in the current slot. This preserves sorted order while simultaneously handling insertion and reduction.

The index-to-PE mapper (IPM) computes an effective injection point. Each PE row implements a pipelined binary-search lookup table over the current $B$5 values in order to choose a start $B$6 such that all $B$7 to the left are smaller than $B$8. The first nonzero in a row of $B$9 drives the IPM, while subsequent elements begin at $K \to M \to N$0. Updates to $K \to M \to N$1 are fed back to the lookup structure, with a write queue buffering port contention, yielding near-optimal starts in $K \to M \to N$2.

The folding mechanism, both spatial and temporal, addresses variation in virtual row lengths for $K \to M \to N$3. When one PE row saturates, newly inserted columns fold into neighboring PE rows according to a 4-way priority—right, then up, then down, then left—while occupancy bits $K \to M \to N$4 record placement. Overflow entries spill into a per-row scratchpad for rows longer than $K \to M \to N$5. Completed partial sums may also spill early through temporal folding to free PEs for subsequent work.

The design perspective of SegFold is that dynamic dataflow and hardware support are inseparable. The memory controller must detect reuse opportunities in a local window of the stationary input array, while the merge network and IPM must support ordered insertion, reduction, and remapping at runtime. This suggests that Segment is not simply a software scheduling abstraction layered atop a conventional accelerator, but a dataflow whose efficiency depends on dedicated microarchitectural support.

4. Complexity, resource model, and operating parameters

The paper gives a concise complexity and resource characterization for the 16×16 instantiation. The memory controller’s scan over the $K \to M \to N$6-bitmask is $K \to M \to N$7 per cycle with fixed $K \to M \to N$8. IPM lookup is $K \to M \to N$9 with $A$0. The merge network has worst-case traversal $A$1, although the expected traversal is described as short. Storage scales with $A$2 lookup-table entries per row, together with a scratchpad per row (Wu et al., 25 Jun 2026).

The PE array is reported at approximately $A$3 and $A$4 at 7 nm, and scaled to approximately $A$5 and $A$6 at 28 nm. The evaluated system uses a cycle-accurate C++ simulator with Ramulator2 modeling HBM2. The principal configuration parameters are a 16×16 PE array, active window size 32, and multicast width 4. The comparison targets are Flexagon, described here as a static inner-product/outer-product/Gustavson design extended to 2D with 128 PEs, and Spada, described as tile-adaptive with static scheduling within the tile (Wu et al., 25 Jun 2026).

The benchmark corpus combines real and synthetic sparse workloads. The real set consists of 15 SuiteSparse matrices with sizes from 1–23K and densities from $A$7 to $A$8. The synthetic set includes square matrices of size 256–1024, density sweep 0.05–1.0, and asymmetric sparsity cases.

These parameters matter because SegFold’s argument is not only algorithmic but also architectural: the proposed dynamism is intended to be implementable with bounded metadata structures, shallow on-chip search, and limited multicast. The explicit $A$9 and multicast width 4 sensitivity points therefore function as evidence that the dynamic mechanisms saturate without unbounded scaling.

5. Quantitative results and sensitivity analyses

Across diverse densities and matrix sizes, SegFold is reported to achieve a geometric-mean $A$00 speedup over state-of-the-art SpGEMM accelerators and $A$01 over the best static dataflow configuration (Wu et al., 25 Jun 2026).

Comparison	Geometric-mean speedup
Over Spada	$A$02
Over best static (Flexagon)	$A$03

On the real SuiteSparse matrices, speedups range from $A$04 to $A$05 over Spada, with one outlier, ca-GrQc, at $A$06. For non-square inputs, tall matrices reach $A$07 over Spada, while wide matrices improve by $A$08–$A$09 if the operands are swapped. In the synthetic density sweep measured in cycles per MAC, SegFold remains flat until matrices become very dense and then degrades gracefully; Spada degrades sharply beyond density 0.4. A static outer-product configuration improves with density but remains below SegFold on sparse cases. For asymmetric sparsity, the paper states that it is best to place the sparser matrix in $A$10, which drives SelectA, except when the density ratio exceeds $A$11–$A$12, in which case the operands should be swapped (Wu et al., 25 Jun 2026).

The ablation study ties the performance gains directly to the dynamic mechanisms. Disabling $A$13-reordering reduces performance to $A$14 of baseline. For mapping policies, the SegFold lookup-table start policy delivers $A$15 over a zero-offset start and lies within $A$16 of an ideal oracle. For hardware knobs, the benefit of multicast width increases up to 4 rows per cycle and then plateaus, while increasing the active window size $A$17 gives benefit up to $A$18 with only marginal returns thereafter (Wu et al., 25 Jun 2026).

Taken together, these results support a specific interpretation of where the gains arise. The $A$19-reordering ablation attributes a large part of the benefit to dynamic reuse of $A$20 rows, while the LUT-versus-zero-offset result attributes another part to reduced displacement in the merge network. The presence of one real-matrix outlier and operand-order sensitivity in asymmetric cases also indicates that the approach is not uniformly dominant; its benefits depend on the interaction between matrix shape, density, and which operand drives SelectA.

6. Interpretive claims, limitations, and proposed extensions

The paper distills three main insights. First, sub-tile dynamism is said to unify the best aspects of inner product, outer product, and Gustavson by reusing $A$21 element-wise, $A$22 row-wise, and $A$23 tensor-wise. Second, on-the-fly discovery of $A$24’s structure removes the need for large $A$25 buffers or offline preprocessing. Third, dynamic mapping together with folding maintains high PE utilization even when rows of $A$26 vary widely in length (Wu et al., 25 Jun 2026).

The limitations are correspondingly concrete. SegFold is reported to work best when the $A$27 dimension of $A$28 is not huge relative to $A$29, because a non-tiled $A$30 can produce imbalance. One operand, namely $A$31, is treated at finer granularity than $A$32, so extreme asymmetry above $A$33 can penalize very low-density $A$34. Stale indices in the IPM, caused by write-port queueing, may occasionally increase displacement, although the measured cost is below $A$35. The dynamic logic also incurs area and power overhead relative to the simplest static arrays (Wu et al., 25 Jun 2026).

The future directions proposed in the work remain within the same conceptual frame. They include $A$36-dimension tiling within the tile for wide and tall extremes; combining offline graph or row reordering methods such as Gamma or Zed with runtime SelectA for static patterns reused across many multiplications; extending dynamic dataflows to other kernels such as SpMV and graph analytics; and a hardware–software co-design in which lightweight sparsity metadata guides window movement (Wu et al., 25 Jun 2026).

A broader implication is that SegFold reframes SpGEMM accelerator design around dynamic dataflow rather than around selection among a small number of canonical loop orders. Within the evidence presented, the system’s novelty is not simply that it is faster on average, but that it treats data reuse and load balancing as runtime-coupled concerns. The paper’s conclusion is therefore narrower than a universal replacement claim: fine-grained dynamism can unlock reuse and load-balance gains that no static scheduling choice can achieve in isolation, while still leaving open operating regimes—especially extreme aspect-ratio and asymmetry cases—where additional tiling or preprocessing may be required (Wu et al., 25 Jun 2026).