--- title: 'ProDF: Programmable DataFlow for SSMs' url: https://www.emergentmind.com/topics/prodf type: topic --- # ProDF: Programmable DataFlow for SSMs ProDF (Programmable DataFlow) is an advanced dataflow architecture developed for efficient execution of structured State-Space Models (SSMs) such as S4 and Liquid-S4 on 2-D systolic arrays, specifically as a core component of EpochCore's LIMA-PE accelerator. By introducing a reconfigurable inter-PE communication strategy—encompassing non-traditional diagonal flows together with standard east-west and north-south interconnects—ProDF enables highly efficient, high-utilization execution of SSM layers for long-sequence inference, while also generalizing to standard DNN workloads. Key innovations in ProDF include fusing recurrence and output stages in a single tiled sweep, weight-stationary per-PE caching, and per-timestep diagonal data streaming, yielding marked improvements in throughput, latency, and energy efficiency [2507.21394]. ## 1. Architectural Definition and Core Objectives ProDF is explicitly designed to address the peculiar requirements of mapping structured SSM layers, which follow the operations - State update: $x_{k+1} = \overline A \odot x_k + \overline B u_k$ - Output computation: $y_k = C x_k + D u_k$ to systolic arrays in a manner that maximizes compute density and minimizes data movement. The architecture encompasses: - **A set of per-PE operating modes:** FRI-MAC, TRI-MAC, BWS-MAC, TOS-MAC, Pass-Through, and Sleep, allowing each processing element (PE) to dynamically switch function according to the dataflow phase. - **A unified inter-PE interconnect:** Supports classical north-south (N→S), west-east (W→E), and critically, a programmable northeast→southwest (NE→SW) data path, with no physical re-wiring necessary. - **A controller-programmed schedule:** Orchestrates the sequence of compute, data-load, and result-read phases in three tightly pipelined stages. The explicit objectives are to (1) fuse the SSM’s recurrence and output stages into a single execution sweep, (2) amortize weight loads through weight-stationary caching, (3) exploit diagonal dataflow for rapid, one-cycle-per-token state updates, and (4) maximize PE utilization and data reuse for both SSM and standard GEMM layers [2507.21394]. ## 2. Mapping, Tiling, and Scheduling Strategy SSM layers with state dimension $N$ (and typically $H = 1$ output head) are mapped onto an $(N+2) \times (N+1)$ tile of LIMA-PEs. ProDF employs a two-phase execution: - **Pre-Load Phase:** Each PE's Stationary and Control Caches are loaded with required weights and control bits in $(N+2)\times(N+1)$ cycles. Specific assignments per tile row govern which weights and modes are held stationary. - **Compute Phase:** At each timestep $k=0,\ldots,T-1$, the input $u_k$ is injected, activating a cascade of operations across diagonally, vertically, and horizontally interconnected PEs: - **Row 0:** BWS-MAC mode multiplies $u_k$ by $\overline B_j$ and dispatches partials diagonally NE→SW into Row 1. - **Row 1:** FRI-MAC (or TRI-MAC for Liquid-S4) computes the SSM recurrence. - **Rows 2 to N+1:** Standard weight-stationary MAC performs the output $C x_k$ aggregate. - **Row N+1, column 0:** Produces final $y_k$ after adding $D u_k$, which is written back to output SRAM. Pre-load of weights and controls is batched and conducted only once per tile, leveraging SRAM's efficiency over repeated DRAM accesses [2507.21394]. ## 3. Data-Movement and Interconnect Patterns ProDF reconfigures per-PE data exchange in real time to match the distinctive data dependency structure of SSMs. The key flows are: - **Row 0:** Receives $u_k$ (W→E), then streams NE→SW partials. - **Row 1:** Receives NE→SW partials from Row 0, then produces $x_{k+1}$, streaming results N→S to Rows 2 to N+1. - **Rows 2–N+1:** Accept $x_{k+1}$ (W→E) and partial sums (N→S) for output accumulation. - **Final Output:** Row N+1, column 0 dispatches $y_k$ directly to output SRAM. Buffer management assigns 16 MB each for weight and input/output SRAMs, with double-buffering to pipeline $u_k$ and $y_k$ at one word per cycle. Stationary caches within PEs retain weights and controls, with moving caches holding $x_k$ and partial sums. This arrangement supports high bandwidth amortization and near-peak bandwidth utilization [2507.21394]. ## 4. Performance Models and Architectural Comparison ProDF delivers marked improvements in PE utilization, memory bandwidth, and throughput over conventional dataflows (Output-Stationary/OS, Weight-Stationary/WS, and Input-Stationary/IS): | Dataflow | Weight Loads per Tile | Outputs per Cycle | Avg PE Utilization | |----------|----------------------|-------------------|--------------------| | OS | N+1 | 1/N | 50% | | WS | N+1 | 1/N | 65% | | IS | N+1 | 1/N | 65% | | ProDF | 1 | 1 | 90–95% | Throughput for a long sequence ($T \gg N$) approaches the clock rate $f_{\rm clk}$ in elements/second; first output emerges after $(N+2)$ cycles, followed by one output per cycle. Effective PE utilization ($\eta$) is measured at $90–95\%$, compared to $50–70\%$ for typical OS/WS mappings. ProDF reduces weight SRAM bandwidth by approximately $30\times$ relative to OS/WS, and halves the number of idle PEs [2507.21394]. ## 5. Energy Efficiency and Empirical Results Experimental evaluation across six LRA sequence modeling tasks (where $T$ ranges from 1,000 to 65,000 and $N=64$) demonstrates: - **Latency:** ProDF/EpochCore achieves $2,000\times$ speedup over Nvidia A100 GPUs, $250\times$ over Sparse-SA, and $25\times$ over 1-D FFT-SA baselines. - **Energy:** Per-inference energy is $45\times$ lower than Sparse-SA, and $10\times$ lower than FFT-SA. - **System-level mixed workload:** Integrating “TPU+EpochCore” (for DNN and SSM, respectively) yields about $30\%$ total system-wide energy reduction, although DNN layers on EpochCore add $+30\%$ overhead offset by SSM gains. - **PE-level metrics:** LIMA-PEs incur a $1.4–2\times$ area and $1.1–1.6\times$ power overhead versus traditional PEs, but dual clock-gating enables $3\times$ energy saving in Sleep versus Accumulation modes [2507.21394]. These results suggest that ProDF's dataflow strategy effectively exploits both the sparsity and banded structure of SSM recurrences, yielding orders-of-magnitude improvements in inference latency and energy. ## 6. Applicability, Generalizability, and Limitations ProDF is explicitly architected to unlock high-performance SSM inference, particularly for structured S4 and Liquid-S4 models, but is also readily extensible to conventional GEMM-based DNN/CNN/Transformer layers, due to its programmable sequencing and PE mode flexibility. The architectural design achieves weight-stationary operation, minimal SRAM bandwidth, and linear per-token throughput for both structured and unstructured layers. A plausible implication is that the principal limitation of ProDF lies in the $2\times$ area and moderate power growth per PE, demanded by its increased microarchitectural flexibility and interconnects. However, these are offset by the substantial throughput and energy advantages, especially for long-sequence, memory-intensive tasks [2507.21394]. ## 7. Summary and Outlook ProDF represents a generalizable, tiled, programmable dataflow that enables ultra-efficient execution of structured SSM layers and conventional deep learning operations within modern systolic arrays. By fusing recurrent and output computations, deploying diagonal NE→SW dataflows, and leveraging per-PE weight-stationarity, ProDF achieves over $2,000\times$ latency speedup, $25–45\times$ energy reduction, and $2\times$ higher utilization compared to prior dataflows, while marginally increasing area and power cost. Its integration in EpochCore's LIMA-PE stacks establishes a template for specialized inference hardware targeting both structured state-space and conventional neural architectures, thus extending the performance envelope of sequence modeling and temporal AI workloads [2507.21394]. Source: https://www.emergentmind.com/topics/prodf