Stream Compute Units: Architectural Insights

• SCUs are hardware mechanisms that reinterpret data movement as explicit, hardware-managed streams, enhancing compute efficiency over traditional load/store operations.
• They are implemented in various forms—such as SSR in RISC-V cores, in-core Streaming Units in Occamy, and on-datapath units in SCENIC—each tailored to specific workloads.
• By offloading address generation, buffering, and data movement to dedicated hardware, SCUs reduce orchestration overhead and boost performance in dense, sparse, and networking tasks.

Searching arXiv for the specified papers to ground the article in the cited literature. arXiv search query: (Scheffler et al., 13 Jan 2025) arXiv search query: (Ramhorst et al., 16 Apr 2026) arXiv search query: (Schuiki et al., 2019) Stream Compute Units (SCUs) are hardware mechanisms that treat data movement as an explicit stream-processing problem rather than as a sequence of conventional load/store instructions or fixed-function packet operations. Across the literature, the term spans several realizations with a common operational principle: compute instructions consume ordered, hardware-managed streams, while address generation, buffering, data movement, and, in some designs, sparse index handling are delegated to dedicated stream hardware. This model appears as Stream Semantic Registers (SSR) in single-issue RISC-V cores, as in-core Streaming Units (SUs) in the manycore Occamy processor, and as reconfigurable on-datapath SCUs in the SCENIC SmartNIC (Schuiki et al., 2019, Scheffler et al., 13 Jan 2025, Ramhorst et al., 16 Apr 2026).

1. Concept and terminology

The term “SCU” is best understood as a generic architectural concept rather than a single canonical block. In Occamy, the processor does not use the term Stream Compute Unit explicitly; instead, each worker core integrates three in-core Streaming Units. These SUs are the concrete realization of the SCU idea: they transform a core from a cache-centric, load/store-driven execution model into a stream/scratchpad-centric engine by turning floating-point register accesses into hardware-managed streams to and from an explicitly managed scratchpad. In SCENIC, by contrast, SCUs are reconfigurable, on-datapath hardware pipelines that process network-carried data streams at line rate while having direct access to host CPU memory, GPU memory, and SSDs. In SSR, the same idea is expressed at ISA level by giving selected registers stream semantics, so that reads cause implicit loads and writes cause implicit stores (Scheffler et al., 13 Jan 2025, Ramhorst et al., 16 Apr 2026, Schuiki et al., 2019).

This framing distinguishes SCUs from several adjacent mechanisms. An SCU is not merely a DMA engine, because it sits on the operand or datapath critical path rather than only moving blocks under software orchestration. It is not equivalent to a tensor core, because Occamy’s SU is not a separate datapath; it is a tightly coupled stream engine that feeds the core’s SIMD FPU and optionally writes results back. It is also not reducible to a P4 parsing pipeline, because SCENIC contrasts SCUs with P4 pipelines by noting that P4 is excellent for parsing, matching, and header edits but less suited for compute-heavy dataflow and tight coupling with transport stacks (Scheffler et al., 13 Jan 2025, Ramhorst et al., 16 Apr 2026).

A recurrent motivation is the same across domains. Traditional CPUs and GPUs rely on caches and general-purpose load/store pipelines; address generation for irregular access and the resulting cache misses depress FPU utilization on stencil and sparse kernels, with “≤10% commonly reported for sparse LA on CPUs/GPUs.” SCUs invert that control structure: streams are installed once, and the loop body becomes mostly or entirely compute, while dedicated hardware sustains the data supply (Scheffler et al., 13 Jan 2025).

2. In-core realizations: from SSR to Occamy’s Streaming Units

SSR provides the minimal processor-side formulation of an SCU. A small subset of integer and FP registers acquires “stream semantics”: reads from those registers cause implicit memory loads and writes cause implicit stores. The register file wrapper detects accesses to designated SSR registers such as t0, t1, ft0, and ft1, and routes them over a stream valid/ready interface instead of to the physical register file array. External data movers then supply those streams through Address Generation Units (AGUs), per-lane FIFOs, direction control, and a repeat mechanism. Each AGU generates affine addresses over up to four nested loops via bound0..3 and stride0..3, and the entire mechanism is enabled or disabled with the CSR ssrcfg at 0x7C0 (Schuiki et al., 2019).

The architectural consequence is that a single-issue core can behave like a stream-oriented compute engine. SSR defines compute utilization as

$$U = \frac{N_{\mathrm{comp}}}{N_{\mathrm{comp}} + N_{\mathrm{mem}}}.$$

For a reduction loop in a conventional load/store ISA, two loads plus one compute operation imply $U \approx 33\%$. With SSR alone, loads and stores vanish from the instruction stream; with hardware loops, the hot loop can reduce to a single compute instruction. The paper gives the asymptotic dot-product expressions

$$\eta_{\mathrm{no\text{-}SSR}} = \lim_{N\to\infty}\frac{N}{2 + 3N} = 33\%, \qquad \eta_{\mathrm{SSR}} = \lim_{N\to\infty}\frac{N}{7 + 1\cdot N} = 100\%.$$

This is the foundational SCU idea in its lightest form: operand movement is architectural, precise, and largely absent from the dynamic instruction stream (Schuiki et al., 2019).

Occamy extends this idea substantially. Each worker core is an RV32G core with a 64-bit-wide SIMD FPU supporting FP64/32/16 with two FP16 encodings and [FP8](https://www.emergentmind.com/topics/floating-point-quantization-fp8) with two FP8 encodings. The FPU implements FMAs, widening sum-dot-products, and three-addend summation for FP8/FP16, and can perform mixed-precision expanding accumulation such as FP8→FP16 and FP16→FP32. Three cooperating SUs per worker core attach to FP registers, with a typical mapping to FT0–FT2; when an instruction such as fmadd.d accesses these registers, the SUs implicitly perform the necessary load/stores and address generation. All three SUs support up to 4D strided streams; two SUs additionally support 8-, 16-, and 32-bit index streams for gather/scatter, and those two indirect SUs can compare indices in-stream to compute intersections and unions for sparse-sparse operations. The third SU can optionally emit the joint index stream for a sparse output. Occamy configures these units through Sparse Stream Semantic Registers, and couples them with a FREP hardware loop so that the inner loop body can be just an FMA or dot instruction while SUs stream operands and results (Scheffler et al., 13 Jan 2025).

This processor-side lineage is important because it shows that an SCU need not be a separate accelerator tile. In SSR and Occamy alike, the compute datapath remains the scalar or SIMD core pipeline; the SCU function is the stream engine that changes how operands arrive and depart.

3. Memory hierarchy, latency tolerance, and sparse scheduling

SCUs are only effective if the memory system can sustain their stream contracts. Occamy therefore organizes computation around explicitly managed scratchpad memories and DMA-overlapped tiling. Each compute cluster provides a 128 KiB [SPM](https://www.emergentmind.com/topics/subspace-power-method-spm), banked into 32 banks with double-word interleaving and a single-cycle logarithmic interconnect, sized to keep 24 SUs (8 cores × 3 SUs) busy with low bank-conflict probability. When all three SUs are active, sustained per-core memory bandwidth reaches up to three double-words per cycle into the shared SPM (Scheffler et al., 13 Jan 2025).

At system level, Occamy uses a hierarchical, latency-tolerant fabric with two orthogonal AXI networks: a 512-bit bulk network for DMA and I$ refills and a 64-bit atomics-capable network for message passing and synchronization. Each chiplet hosts 6 compute groups, each group aggregates four clusters, and each chiplet connects to eight [HBM2E](https://www.emergentmind.com/topics/high-bandwidth-memory-2e-hbm2e) controllers at 47.68 GiB/s each (381.47 GiB/s per chiplet). Cluster DMA is tightly coupled: it sustains 64 B/cycle (64 GiB/s) by accessing SPM “superbanks” through a secondary interconnect and is prioritized to reduce backpressure. Double buffering is explicit: the DMA streams the next tile while the SUs feed the FPU from the current tile. The paper lists representative tilings such as [GEMM](https://www.emergentmind.com/topics/general-matrix-multiply-gemm-pipeline) 48×48, 2D stencils 64×64, and 3D stencils 16×16×16 (Scheffler et al., 13 Jan 2025).

The die-to-die path and NoC are likewise part of the SCU story because they determine whether installed streams remain fed at scale. Occamy’s D2D link is a fully digital, source-synchronous DDR design with effective duplex bandwidths of up to 64 Gbit/s (wide) and 1.33 Gbit/s (narrow), 1.6 pJ/bit (measured) energy efficiency, ~27 cycles latency for narrow host accesses to remote SPM, and ~61 cycles for DMA transfers over the wide segment. The crossbar-based cluster→group→chiplet hierarchy yields symmetric latency/bandwidth views from all clusters, which simplifies scheduling and maintains uniform performance for stream/tiling-based codes (Scheffler et al., 13 Jan 2025).

Sparse scheduling follows the same pattern but with additional index handling. For SpMM, Occamy partitions and reuses the sparse matrix intra-group to reduce traffic; the two indirect SUs handle value and index streams, gather the dense operand, and the FREP loop accumulates results. For SpMSpM, the two indirect SUs perform in-stream index intersection, and the third SU can write result indices, minimizing control overhead. Because an SU can stream any major-axis value-index pair, the design handles CSR, CSC, CSF, and similar formats without assuming density or structure (Scheffler et al., 13 Jan 2025).

4. On-datapath SCUs in SCENIC

SCENIC generalizes the SCU idea beyond processor cores by treating the NIC datapath itself as a stream-compute substrate. SCUs are placed on-path behind the offloaded [RDMA](https://www.emergentmind.com/topics/remote-direct-memory-access-rdma) and TCP/IP transport stacks for payload-centric compute and can also be instantiated on the slow path for arbitrary traffic. A front-end prefilter triages traffic at L2/L3/L4, dispatching RoCEv2 and TCP streams to hardware-offloaded transport stacks and sending unmatched traffic to the slow path via DMA. SCENIC can host up to 16 independent SCUs configured at build time; the default build includes one SCU and the fallback path, with RDMA and memory virtualization always present (Ramhorst et al., 16 Apr 2026).

The timing model is explicit. At 200 Gbps with MTU-sized [RoCE](https://www.emergentmind.com/topics/rotary-camera-encoding-roce) packets, the per-packet budget is

$$T_{\mathrm{pkt}} = \frac{S_{\mathrm{pkt}} \times 8}{B},$$

instantiated in the paper as

$$T_{\mathrm{pkt}} = \frac{4178 \times 8}{200 \times 10^9} \approx 167 \text{ ns}.$$

With a 391 MHz SCU pipeline, this corresponds to roughly 65 cycles. The paper states that this is sufficient for multi-signal telemetry-based algorithms and complex streaming operators, including hardware programmable congestion control (Ramhorst et al., 16 Apr 2026).

SCENIC’s memory hierarchy is organized around streaming kernels rather than host mediation. On-chip memories provide tens of MB of BRAM/URAM, single-cycle access; on-card memory may be HBM (U55C, U280, V80) with hundreds of GB/s or DRAM on platforms such as U250. DMA is provided by XDMA (Gen3x16) on 100G Alveos and QDMA (Gen4x16 or Gen5x8) on V80 with up to 2048 queues. SCUs can directly access GPUs via dma-buf based GPUDirect RDMA, including both NVIDIA and [AMD](https://www.emergentmind.com/topics/adaptive-matching-distillation-amd) GPUs, and NVMe through an in-FPGA NVMe host controller that keeps queues, PRP lists, and data buffers in FPGA memory and rings SSD doorbells directly (Ramhorst et al., 16 Apr 2026).

Programming follows the stream-computation abstraction. Kernels are expressed in RTL (SystemVerilog/VHDL), HLS (C++), SpinalHDL, or P4; the build flow links SCU artifacts with SCENIC, and libstf provides reusable stream blocks such as normalizers, hashers, crossbars, width converters. Host integration is intentionally native: SCENIC is exposed as a standard Linux netdev for arbitrary traffic and as an ibv_device for RDMA verbs. Flow-to-SCU steering is explicit through ibv_create_qp_ex, allowing a flow to be pinned to a specific SCU. Runtime control is available through thin Python and C++ APIs for dynamic partial reconfiguration, CSR writes, event handling, and status polling. Congestion-control modules are hardware pipelines that can be swapped via partial reconfiguration, with average reconfig time ≈8 ms, hidden by the dual-CC design shown in Figure 1 (Ramhorst et al., 16 Apr 2026).

SCENIC also sharpens the distinction between SCUs and other NIC-side compute models. Fixed-function offloads are high-performance but not extensible; embedded Arm or on-path RISC cores are flexible but often off-path or underpowered for per-packet compute at 100–200G; and P4 is less suited to payload-heavy compute. SCENIC positions SCUs as user-defined, line-rate dataflow operators coupled directly to transport semantics and peer-device access (Ramhorst et al., 16 Apr 2026).

5. Performance characteristics and empirical results

The performance of SCUs is reported through utilization, throughput, compute density, and energy efficiency. Occamy defines

$$U_{\mathrm{FPU}} = \frac{\text{Achieved FLOP/s}}{\text{Peak FPU FLOP/s}},$$

dense compute density as

$$D = \frac{\text{Achieved GFLOP/s}}{\text{Area (mm}^2\text{)}},$$

graph/sparse index density as

$D = \frac{\text{GCOMP/s}}{\text{mm}^2},$

and the roofline bound as

$$\text{Performance} \le \min(\text{Peak FLOP/s}, \text{Bandwidth} \times \text{Operational Intensity}).$$

Energy efficiency is defined as throughput over power, for example GFLOP/s/W or GCOMP/s/W (Scheffler et al., 13 Jan 2025).

Occamy’s measured results show how the SCU model behaves across dense, stencil, sparse, and ML workloads. At system scale, Occamy is a 432-core, dual-chiplet, dual-HBM2E system with 12 nm compute chiplets (73 mm^2 each) and system peak 768 DP-GFLOP/s. On dense linear algebra, it reaches 89% peak FPU utilization, 686 DP-GFLOP/s, and 39.8 DP-GFLOP/s/W; averaged across dense kernels, it reports 78% FPU utilization, SU speedups up to 2.7× vs. baseline RV32G, and average 2.4× throughput and 1.5× energy efficiency improvements. On stencil codes, it reaches 83% utilization and up to 571 DP-GFLOP/s, with 11.1 DP-GFLOP/s/mm^2, leading state of the art by 1.7× in utilization and 1.2× in density. On SpMM, it reaches 42% FPU utilization, up to 307 DP-GFLOP/s, up to 16.0 DP-GFLOP/s/W, and 5.95 DP-GFLOP/s/mm^2, surpassing state of the art by 5.2× in utilization and 11× in density. On SpMSpM, it reaches up to 187 GCOMP/s, 17.4 GCOMP/s/W, and 3.63 GCOMP/s/mm^2, with index comparator utilization up to 49%. The same architecture also reports 490 tok/s and 29.3 tok/s/W for FP16 GPT-J, non-autoregressive inference with peak FPU utilization 75%, and on a GCN layer (FP64; 144→144 features) it reports SU speedup 2.2× over optimized RV32G, peak FPU utilization 54%, throughput up to 413 DP-GFLOP/s and 8.01 DP-GFLOP/s/mm^2, and energy efficiency up to 25.0 DP-GFLOP/s/W (Scheffler et al., 13 Jan 2025).

SCENIC evaluates SCUs under networking, storage, and accelerator-coupled workloads. On RDMA microbenchmarks, it saturates link bandwidth for both ib_write_bw and ib_read_bw, with latency slightly above Mellanox CX but close enough for target use cases. On the slow path, ping latency is modestly higher than commercial ASIC NICs, and iperf3 throughput at MTU 1500 is slightly lower, consistent with the design choice that the slow path is not optimized for bulk transfers. For GPU direct, AMD GPUs achieve RDMA [READ](https://www.emergentmind.com/topics/reconstruction-and-alignment-of-text-descriptions-read)/WRITE saturate the link, while NVIDIA GPUs show lower throughput partly because local CPU–FPGA DMA capped at ~20 GB/s instead of 23–25 GB/s on the test node. For direct SSD access via TCP/IP offload, completion latency is 2–3× lower per-completion vs host baseline, including 25.6 μs vs 72.4 μs at 4 KB, and throughput with only 8 outstanding I/Os saturates the NVMe SSD and outperforms the host with 64 outstanding by 1.27–1.47×. In multi-tenant tests, with four SCUs and 128 KiB RDMA READs, aggregate bandwidth saturates the 200G link and is evenly split, with each of N active flows receiving approximately R/N. The network-to-GPU hash-based partitioning use case reports an overall, a 6.7× latency improvement vs CPU baseline, while ACCL+ on SCENIC matches OpenMPI on Mellanox CX5 in BROADCAST and GATHER (Ramhorst et al., 16 Apr 2026).

SSR provides the smallest-scale performance baseline for the concept. The paper reports 2x to 5x architectural speedup across kernels, with sequential [code](https://www.emergentmind.com/topics/karpathy-agent-code) runs 3x faster on a single core, 3x fewer cores are needed in a cluster to achieve the same performance, and the utilization increase to almost 100% leading to a 2x energy efficiency improvement in a multi-core cluster. It also reports up to 3.5x fewer instruction fetches, up to 5.6x lower instruction-cache power consumption, and an area increase of 11% per core (Schuiki et al., 2019).

Taken together, these results indicate that SCUs are effective in domains where explicit operand transport or packet handling would otherwise dominate the schedule: dense kernels benefit because loop bodies shrink toward pure compute, sparse kernels benefit because index handling is moved into specialized stream machinery, and network offloads benefit because transport semantics and line-rate compute are kept in a single datapath.

6. Limitations, trade-offs, and scope

The SCU model does not eliminate bottlenecks; it relocates them. In Occamy, sparse-sparse performance can be bounded by index handling and very short runs between intersections; peak comparator utilization is 49%. For extremely sparse inputs, SU setup overhead and finite index ranges per tile begin to dominate. Although 32 banks reduce conflicts, contention among 24 SUs per cluster can reduce throughput on adversarial access patterns. Mixed-precision operation also introduces overheads: SIMD packing/unpacking, accumulator zeroing, reductions, and conversions reduce utilization, with FP8 expanding GEMM at 66% U_FPU. At physical level, the hierarchical crossbars provide a uniform fabric but incur area/implementation costs (Scheffler et al., 13 Jan 2025).

SSR’s limitations are more fundamental to its minimal design. The evaluated lanes support affine patterns; indirection would require a more complex data mover (two-level load) and more bandwidth. A lane is either read or write until its pattern completes, so read/write interleaving is constrained. Short loops may not amortize setup cost; the paper gives the break-even condition 4d+2 ≤ Σ_i Π_{n=1..i} L_n, and for equal-length loops notes that SSR beats baseline for l^d > 5 (1D), \>4 (2D), ≥1 (3D/4D). The prototype also uses deferred interrupts inside SSR regions, whereas a precise-exception design would require shadowing architectural AGU state, with area/power overhead (Schuiki et al., 2019).

SCENIC’s trade-offs are those of FPGA-resident programmable datapaths. Very complex kernels may be area-bound; P4 SCUs are limited to match-action style processing; and dynamic partial reconfiguration pauses the reconfigured SCU, even though the dual-CC design hides congestion-control swaps. Some conventional NIC features on the slow path, such as TSO/GRO, are not yet in hardware. GPU behavior is also platform-dependent: the lower NVIDIA throughput is explicitly attributed to test-node PCIe settings rather than to a semantic limitation of the SCU model (Ramhorst et al., 16 Apr 2026).

A common misconception is that SCUs are specialized only for regular dense loops. The literature does not support that view. Occamy’s SUs support 4D affine and indirect streams with 8/16/32-bit indices, perform in-stream sparse set ops (intersection/union), and handle CSR, CSC, and CSF; SCENIC’s SCUs support payload-heavy operators such as hashing and partitioning, reductions, stateful flow tracking and rate limiting, lookup tables, encryption/compression, and collective kernels via ACCL+; SSR, by contrast, is explicitly limited to affine patterns in its evaluated form (Scheffler et al., 13 Jan 2025, Ramhorst et al., 16 Apr 2026, Schuiki et al., 2019).

This suggests that “SCU” names a design invariant rather than a single block diagram: the critical property is that addressing, buffering, and movement are hoisted into programmable stream machinery close to the compute or transport path, so that the dynamic schedule is dominated by useful arithmetic or payload transformation rather than by explicit orchestration.