3D-Adaptive GEMV-GEMM Ratio Systolic Array

• The paper introduces a 3D-adaptive systolic array that dynamically partitions sub-arrays between GEMM and GEMV modes, reducing fill overhead and achieving up to 2× latency reduction.
• The architecture employs a three-layer heterogeneous integration stack with TSVs and V-Cache to enable a unified 3D dataflow that minimizes memory traffic and maximizes operand reuse.
• The design leverages runtime adaptation and score-aware HBM traffic reduction to optimize resource utilization, resulting in improved throughput and energy efficiency for MoE LLM serving.

Searching arXiv for the specified paper and closely related work to ground the article in current sources. The 3D-Adaptive GEMV-GEMM-ratio systolic array is a compute substrate introduced within A3D-MoE for fine-grained Mixture-of-Experts (MoE) LLM serving, where runtime execution alternates among GEMM-heavy prefill, GEMV-heavy decode, and mixed windows produced by gating sparsity and continuous batching. In this setting, the array is defined by three coupled properties: a 3D heterogeneous integration stack that uses TSVs and hybrid bonding, a unified 3D dataflow that supports both GEMM and GEMV efficiently, and runtime adaptation that partitions the array across modes to match arbitrary GEMV-GEMM ratios while reducing memory traffic through a co-located V-Cache (Huang et al., 25 Jul 2025).

1. Problem setting and motivation

Fine-grained MoE LLM serving exhibits highly variable mixtures of GEMM and GEMV at runtime for three stated reasons: gating sparsifies per-token expert activation; continuous batching such as Sarathi-Serve chunks long prefills and interleaves prefill and decode regions; and decode attention is intrinsically GEMV-dominant because new-token $Q \cdot K^T$ and $S \cdot V$ are vector-matrix products over growing KV caches. The resulting workload is not a stable GEMM stream but a sequence of mixed windows with arbitrary ratios of high-reuse and low-reuse linear algebra (Huang et al., 25 Jul 2025).

The architectural difficulty follows directly from the behavior of conventional $N \times N$ 2D systolic arrays. For GEMV, utilization collapses to approximately $1/N$ because only one row or column is active, while GEMM incurs pipeline fill and flush costs of $O(N)$ cycles per tile. Fixed allocations of GEMM and SIMD or GEMV units therefore underperform in opposite phases: GEMV-heavy decode leaves GEMM capacity idle, whereas GEMM-oriented organizations underutilize hardware during low-arithmetic-intensity expert execution and attention. A further source of inefficiency is that traditional MoE-based scheduling cannot fuse attention operations with MoE operations, which introduces stalls between stages.

A common misconception is that the problem is solely memory bandwidth. The A3D-MoE formulation instead treats it as a joint utilization, scheduling, and data-movement problem: arbitrary GEMV-GEMM ratios reduce PE occupancy, non-fused attention and MoE scheduling increases latency, and expert loading from DRAM remains energy-intensive even when MoE activates fewer weights than dense models. This suggests that the distinctive feature of the 3D-Adaptive GEMV-GEMM-ratio systolic array is not merely 3D stacking, but the coordinated use of vertical injection, runtime repartitioning, and cache-assisted reuse.

2. 3D stack organization and processing-element fabric

The array is embedded in a three-layer heterogeneous integration stack. At the top are multi-tier HBM stacks, described as fine-grained DRAM, connected by TSVs and hybrid bonding with sub-5 $\mu$m pitch to the logic base die. In the middle is the HBM logic base die containing Type-2 SRAM, termed V-Cache, in capacities of 16 MB or 32 MB, co-located with TSV landing pads for high-bandwidth, low-latency staging. At the bottom is the compute logic die, which hosts two compute fabrics: a Normal Systolic Array fed by Type-1 SRAM and the 3D-Adaptive GEMV-GEMM-ratio Systolic Array fed directly by TSVs from V-Cache or HBM, plus light local buffers for interleaving and de-interleaving (Huang et al., 25 Jul 2025).

The 3D-SA core tile is a $16 \times 16$ PE array at 1 GHz in 7 nm. Table II in the source specifies vertical injection widths such as $16 \times 16 \times 512$ or $\times 1024$, which indicate the number of TSV lanes feeding the array in parallel. Each PE is a MAC with two operand latches for interleaved input and weight ingestion, a partial-sum accumulator, and configurable shift registers on east and south outputs for one-cycle per-hop propagation. The interconnect retains a standard systolic mesh but adds ring wrap-around: the row-$N$ output buffer connects to row 1, and the column-$S \cdot V$0 weight buffer connects to column 1, in order to support spatial pre-skew and maintain correct reduction order.

The pipeline is organized as parallel operand injection from V-Cache or HBM via TSVs at $S \cdot V$1, systolic wave propagation from $S \cdot V$2 with either output-stationary or input-stationary behavior, and single-cycle upward TSV ejection at $S \cdot V$3 followed by de-interleaving and write-back. The control logic includes a mode controller that selects among 3D-GEMM-IS, 3D-GEMM-OS, 3D-GEMV, and 3D-GEMV-V-Cache, as well as an adaptation manager that partitions arrays across modes at runtime and configures interleaving offsets in per-column weight buffers and per-row output buffers. Double-buffering on injection and ejection hides TSV latency.

This organization makes the vertical dimension a compute operand path rather than only a memory path. The paper’s own description states that the vertical dimension provides a “third axis” for parallel operand injection and output ejection, so the inner $S \cdot V$4-loop is fed at very high vertical bandwidth rather than serialized across $S \cdot V$5 cycles. In that sense, the defining departure from planar systolic arrays is architectural: the stack reshapes the timing of operand movement as much as it reshapes the memory hierarchy.

3. Unified 3D dataflow and runtime adaptation

For GEMM, the design supports both 3D-GEMM-IS and 3D-GEMM-OS. Given

$S \cdot V$6

the conventional input-stationary systolic schedule loads $S \cdot V$7 across $S \cdot V$8 cycles and streams $S \cdot V$9 over $N \times N$0 cycles, leading to $N \times N$1 cycles per tile. In 3D-GEMM-IS, a tile of $N \times N$2 is injected into all rows via TSVs in one cycle, then spatially pre-skewed $N \times N$3 is injected into all columns in one cycle; subsequent cycles shift $N \times N$4 right and stream partial sums down, with row wrap-around preserving reduction order. In 3D-GEMM-OS, $N \times N$5 and spatially interleaved $N \times N$6 are similarly injected in one cycle each, after which $N \times N$7 shifts right, $N \times N$8 shifts down, and partial sums remain stationary in the PEs (Huang et al., 25 Jul 2025).

For GEMV, the array uses a vector-parallel, interleaved 3D dataflow:

$N \times N$9

Multiple independent GEMVs, such as different tokens or experts, are mapped across rows and columns to fill the $1/N$0 grid. At $1/N$1, $1/N$2 vectors and interleaved columns of $1/N$3 are injected via TSVs, and the subsequent schedule uses weight interleaving to preserve partial-sum alignment while keeping critical paths short as MAC plus register chains, rather than wide SIMD reductions. The 3D-GEMV-V-Cache mode further decomposes low-arithmetic-intensity GEMM tiles into batched GEMVs so that sub-matrices of $1/N$4 are staged in V-Cache and reused across several $1/N$5 vectors with minimal HBM traffic.

Runtime adaptation combines spatial partitioning, reconfigurable dataflow, and temporal multiplexing. The 3D-SA is partitioned into sub-arrays assigned to GEMM or GEMV roles, with ratios set at runtime by the hardware resource-aware operation fusion scheduler according to observed traffic and predicted gating patterns. Each sub-array can switch among 3D-GEMM, 3D-GEMV, and 3D-GEMV-V-Cache. Idle sub-arrays can opportunistically switch modes when counterparts stall, such as when GEMM completes early in prefill while decode attention continues.

The utilization model given for the adaptive array formalizes this behavior. Let $1/N$6 denote the GEMV share of total workload and $1/N$7 the fraction of the 3D-SA allocated to GEMV. Then

$1/N$8

with $1/N$9 and $O(N)$0. The paper contrasts this with the conventional lower bound

$O(N)$1

and states that for $O(N)$2 the GEMV term moves from $O(N)$3 toward approximately $O(N)$4, which explains the reported gains in decode-heavy windows. This suggests that the central function of adaptivity is not simply balancing two kernels, but lifting the GEMV side from the utilization regime that ordinarily dominates decode latency.

4. V-Cache reuse, arithmetic intensity, and vertical bandwidth

The V-Cache is a Type-2 SRAM placed on the HBM logic die in capacities of 16 MB for A3D-MoE-1 and 32 MB for A3D-MoE-2. It is banked to match TSV groups feeding columns and rows of the 3D-SA, with bank interleaving across experts and tokens to minimize bank conflicts and maximize burst utilization. Hybrid bonding and TSVs provide direct V-Cache-to-PE injection and PE-to-V-Cache write-back with one-hop paths (Huang et al., 25 Jul 2025).

The reuse argument is expressed through arithmetic intensity. For GEMM and GEMV,

$O(N)$5

If $O(N)$6 denotes the reuse factor enabled by V-Cache across $O(N)$7 vectors for a fixed $O(N)$8 tile, and $O(N)$9 is fetched once then reused $\mu$0 times, the effective bytes for batched GEMV become approximately

$\mu$1

so that

$\mu$2

As $\mu$3 increases, the paper states that $\mu$4 asymptotically approaches GEMM-like regimes because $\mu$5 is amortized.

Vertical bandwidth is modeled as

$\mu$6

In A3D-MoE, each HBM provides approximately 9600 GB/s vertically. This parallel injection reduces $\mu$7 from approximately $\mu$8 cycles to 2 cycles, which decouples inner-$\mu$9 loop feeding from the lateral mesh. The associated traffic model also states that effective hop count is approximately $16 \times 16$0 when operands traverse TSVs directly, rather than many-hop mesh routing, and therefore the energy per byte for NoC movement drops with $16 \times 16$1.

A useful correction to a common simplification is that the V-Cache is not an ordinary local scratchpad inserted between HBM and a conventional array. Its function is explicitly tied to the 3D dataflow: weight-stationary tiling pins GEMM tiles in V-Cache while multiple $16 \times 16$2-tiles stream vertically, GEMV decomposition keeps a fixed tile of $16 \times 16$3 resident while sweeping multiple vectors temporally, and double-buffering uses two V-Cache sets per sub-array so that one set is consumed by PEs while the other is filled from HBM. The paper’s bandwidth rule, $16 \times 16$4 during GEMV-decomposition phases, further indicates that V-Cache capacity and TSV refill rate are co-scheduled rather than independently provisioned.

5. Fusion scheduling and score-aware HBM traffic reduction

The hardware resource-aware operation fusion scheduler, HR-OFS, overlaps attention and MoE execution. In decode-dominant, memory-bound cases, it prioritizes QKV for high-arithmetic-intensity tokens; once prefill attention completes for a subset, it starts high-AI MoE on GEMM partitions while decode attention continues on GEMV partitions. In prefill-dominant, compute-bound cases, it prioritizes low-AI tokens so that low-AI MoE can overlap with remaining prefill attention. Decode-only execution follows similar patterns. The text summary of the execution timeline is explicit: QKV GEMM runs on 3D-GEMM while decode attention GEMV runs on 3D-GEMV; later, GEMM partitions switch to high-AI MoE for WG/WU/WD while GEMV partitions continue attention; then mid-AI MoE decomposed into GEMV-V-Cache streams overlaps with residual attention GEMV; finally softmax and aggregation complete and gating for next layers drives even-odd expert selection (Huang et al., 25 Jul 2025).

The latency characterization is given as

$16 \times 16$5

without fusion, and

$16 \times 16$6

with fusion on the 3D-SA, where $16 \times 16$7 is a small reconfiguration overhead. The ablation result cited for this change is approximately $16 \times 16$8–$16 \times 16$9 latency reduction.

A second optimization, MoE Score-Aware HBM access reduction with even-odd expert placement, addresses the observation that top-$16 \times 16 \times 512$0 gating scores are skewed and that low-score experts contribute minimally to the weighted sum. The policy is: for experts with normalized gate score below 0.45, fetch FP8 instead of BF16 to reduce HBM bytes while preserving accuracy via LUT-based exponent restoration. Profiling shows that most expert weights have 4-bit exponent range coverage of 99.87–99.94% across models; the system stores a shared “Regular data” map and a rare “outlier” map in HBM, loads them into SRAM at runtime, reconstructs the full 8-bit exponent on the logic die, and zero-pads mantissa to BF16 width for MACs. DRAM row interleaving places FP8 portions in odd rows and residual BF16 bits in even rows, which the paper states achieves full bandwidth regardless of precision mode.

The traffic model for this mechanism is given as follows: if $16 \times 16 \times 512$1 is the fraction of low-score expert accesses, then FP8 halves payload per read, so HBM bytes reduce by approximately

$16 \times 16 \times 512$2

Measured DRAM transactions fall by $16 \times 16 \times 512$3–$16 \times 16 \times 512$4. The paper connects this not only to reduced traffic, but also to increased array utilization and lower NoC energy because less bandwidth is consumed by low-contribution experts.

6. Quantitative characteristics, examples, and limitations

The implementation parameters in Table II describe two A3D-MoE configurations. The 3D-SA uses $16 \times 16 \times 512$5 arrays with 512 vertical TSV lanes per array in A3D-MoE-1 and 1024 in A3D-MoE-2; the Normal Systolic Array uses $16 \times 16 \times 512$6 arrays; frequency is 1 GHz in 7 nm; HBM capacity is 36 GB or 72 GB; per-HBM bandwidth is 9600 GB/s; on-die SRAM is 16–32 MB for both Type-1 and Type-2; and compute-die area is reduced by approximately 10% due to NoC shrink, while HBM logic-die area is re-budgeted from SerDes to TSVs and V-Cache with net-neutral area (Huang et al., 25 Jul 2025).

The reported system-level gains are stated relative to prior baselines. Measured latency, in TBT p99, is approximately $16 \times 16 \times 512$7 versus NeuPIM and up to $16 \times 16 \times 512$8 versus Duplex with liquid cooling. Throughput improves by $16 \times 16 \times 512$9–$\times 1024$0 versus NeuPIM and $\times 1024$1–$\times 1024$2 versus Duplex. Energy is reduced by approximately $\times 1024$3 versus Duplex and approximately $\times 1024$4 versus NeuPIM on average. The abstract summarizes the full range as latency reduction by a factor of $\times 1024$5 to $\times 1024$6, throughput improvement by $\times 1024$7 to $\times 1024$8, and energy reduction by $\times 1024$9 to $N$0.

The case studies clarify how the array behaves on specific kernels. For prefill attention projections $N$1 with $N$2, $N$3, and $N$4, 3D-GEMM-OS is used and the fill overhead drops from approximately $N$5 cycles to 2 cycles; for $N$6, the source states that fill overhead drops by approximately $N$7. For decode attention, both $N$8 and $N$9 are mapped as GEMV, with multiple tokens or heads running concurrently and outputs produced after a small fixed number of cycles per vector in the illustrated example. High-AI experts run as 3D-GEMM with V-Cache pinning WG/WU/WD tiles; mid-AI experts are decomposed into GEMV streams; low-score experts use FP8 via even-odd placement.

The limitations are equally explicit. Extreme GEMV-dominant scenarios are gated by vertical bandwidth and V-Cache capacity; if KV caches or experts overwhelm V-Cache, HR-OFS throttles low-score experts via FP8. More experts or larger sequences require scaling V-Cache banking and TSV width, which is reflected in A3D-MoE-2 doubling vertical channels. HR-OFS begins at layer 4 because early-layer prediction accuracy is low. Thermal constraints remain material in 3D stacking: the thermal limit keeps DRAM dies below 95°C; hybrid bonding improves vertical heat conduction relative to 2.5D; power limits are enforced; and DVFS engages without liquid cooling. These constraints indicate that the array is adaptive within a bounded thermal and bandwidth envelope rather than universally compute-bound.

7. Relation to broader 3D and near-memory systolic-array research

The 3D-Adaptive GEMV-GEMM-ratio systolic array sits at the intersection of two prior lines of work. One is the three-dimensional systolic GEMM architecture on Intel Stratix 10 FPGAs, where the third dimension corresponds to layers along $S \cdot V$00, partial sums are forwarded upward, and the design uses register chains and HyperFlex retiming to avoid routing congestion while achieving above 3 TFLOPS and up to 99.8% of available DSPs (Gorlani et al., 2021). The other is reconfigurable near-memory decoding hardware for 3D-stacked NMP, which argues that high local bandwidth makes many decode operators compute-bound and therefore requires fine-grained reconfigurability in both array shape and dataflow, implemented through a reconfigurable systolic array and multi-core scheduling framework (Ai et al., 5 Apr 2026).

The A3D-MoE design differs from the FPGA 3D systolic formulation in emphasis. The 2021 architecture turns part of accumulation time into spatial depth along $S \cdot V$01 and focuses on GEMM throughput, HLS implementation, and DDR-aware reuse. Its own extension to GEMV is primarily shape rebalancing: shrink one spatial axis, expand $S \cdot V$02-depth, and batch vectors to recover arithmetic intensity. By contrast, the A3D-MoE array is organized around TSV-based parallel injection, one-cycle operand ingress and egress, explicit GEMV and GEMV-V-Cache modes, runtime spatial partitioning, and fusion with MoE scheduling.

It also differs from the 2026 NMP decode design. That work uses a physical $S \cdot V$03 PE fabric per core, SNAKE-like remapping to logical shapes such as $S \cdot V$04 or $S \cdot V$05, switching between input-stationary and output-stationary dataflows based on $S \cdot V$06 versus $S \cdot V$07, and a unified vector core built around a shared 2-read/2-write output buffer. Its principal problem is small-$S \cdot V$08 decode on 3D-stacked NMP, where many operators become compute-bound under 24 TB/s local bandwidth. A3D-MoE instead addresses mixed prefill-decode windows, MoE expert heterogeneity, and score-aware HBM traffic reduction in a stack where each HBM provides approximately 9600 GB/s vertically. The shared theme is that reconfigurability becomes a first-order architectural concern once 3D memory bandwidth removes the simple assumption that decode is purely memory-bound.

Taken together, these works indicate that the phrase “3D-Adaptive GEMV-GEMM-ratio systolic array” denotes more than a generic 3D accelerator for dense linear algebra. In the specific A3D-MoE usage, it refers to a vertically fed, mode-switchable systolic fabric that uses unified 3D dataflow, V-Cache reuse, scheduler-guided partitioning, and score-aware memory reduction to sustain high utilization across arbitrary GEMV-GEMM ratios. A plausible implication is that its main contribution lies in converting workload irregularity—rather than merely matrix multiplication—into an explicit architectural scheduling problem.