Cerebras CS-3 Wafer-Scale Accelerator
- Cerebras CS-3 is a wafer-scale accelerator built around the WSE-3 chip, featuring approximately 900,000 processing elements, 44 GB of on-chip SRAM, and a 2-D mesh interconnect.
- It employs a spatial programming model with explicit data movement, enabling efficient stencil computations, sparse linear algebra, and low-latency LLM inference.
- Despite its high throughput and energy-efficient design, the system requires specialized packaging, robust power delivery, and careful cost-performance considerations compared to multi-GPU setups.
Cerebras CS-3 is the third-generation Cerebras wafer-scale system built around the Wafer-Scale Engine 3 (WSE-3), a wafer-scale, dataflow-oriented accelerator. In the CS-3 literature, the distinction between system and chip is sometimes explicit—the CS-3 is the host system and the WSE-3 is the accelerator—and sometimes elided, with the names used interchangeably. Across published descriptions, the platform is characterized by roughly processing elements, $44$ GB of distributed on-chip SRAM, a $2$-D mesh interconnect, and direct nearest-neighbor communication, and it is evaluated not only for large-model AI inference but also for stencil-based PDE solvers, tsunami simulation, and sparse linear algebra kernels (Belli et al., 8 May 2026, Shah et al., 30 Apr 2026, Kundu et al., 11 Mar 2025).
1. Architectural identity and reported hardware profile
The CS-3 is presented as a wafer-scale alternative to conventional multi-GPU systems. Rather than scaling by assembling many reticle-limited devices, Cerebras keeps essentially an entire wafer as one processor. A comparative study describes the CS-3 as powered by the WSE-3 chip and reports a process of TSMC $5$ nm, $4$ trillion transistors, AI-optimized cores, $44$ GB of on-chip SRAM, $21$ petabytes per second of memory bandwidth, and a wafer-scale chip area of (Kundu et al., 11 Mar 2025). A stencil-computing study describes the WSE-3 as a wafer-scale, dataflow-oriented processor with processing elements arranged in a $44$0-D mesh, $44$1 GB of distributed on-chip SRAM, and direct nearest-neighbor communication through “wavelets,” with support up to fp32 (Belli et al., 8 May 2026).
| Source | Reported figure | Context |
|---|---|---|
| (Belli et al., 8 May 2026) | $44$2 PEs; $44$3 GB distributed on-chip SRAM; $44$4-D mesh; up to fp32 | WSE-3 as accelerator inside CS-3 |
| (Kundu et al., 11 Mar 2025) | TSMC $44$5 nm; $44$6 trillion transistors; $44$7; $44$8 petabytes per second | Comparative architectural overview |
| (Golden et al., 12 Apr 2026) | Prefill compute $44$9 TFLOPS; decode memory type SRAM; decode capacity $2$0 GB; decode bandwidth $2$1 B/s | LLM inference comparison table |
A central architectural claim is that compute is decoupled from memory. One comparison paper states that model parameters can live in external memory such as MemoryX, so model size is not tightly bound to on-chip SRAM or HBM capacity; the same source reports CS-3 memory capacity ranging from $2$2 TB to $2$3 TB and describes the system as intended to support LLMs up to $2$4 trillion parameters (Kundu et al., 11 Mar 2025). The same paper emphasizes on-wafer die-to-die communication, stating that Cerebras and TSMC repurposed wafer scribe lines as wiring channels. This suggests that the defining system-level idea is not merely large raw silicon area, but a communication regime intended to collapse a substantial fraction of inter-chip overhead into on-wafer transport.
Published descriptions are not numerically identical in their peak-throughput reporting. One source gives $2$5 PFLOPS in an introductory description and $2$6 PFLOPS FP8 and $2$7 PFLOPS FP16 in the system-comparison table used there, while another gives $2$8 TFLOPS prefill compute in an LLM inference table (Kundu et al., 11 Mar 2025, Golden et al., 12 Apr 2026). The discrepancy reflects source-specific reporting conventions rather than a disagreement about the basic architectural profile.
2. Programming model and execution semantics
The CS-3 software model is consistently described as spatial and explicit rather than opaque and cache-centric. In stencil work, the machine is programmed through Cerebras Data Structure Descriptors (DSDs), which describe strided memory regions so that an entire grid or subgrid can be traversed by a single hardware-driven instruction. Stencil updates are then expressed as SIMD-style operations such as @fmuls and @fmacs over DSDs, while halo exchange uses @movs backed by microthreads. Communication differs between Star and Box stencils: Star patterns exchange only cardinal boundaries, while Box patterns require a two-stage store-and-forward scheme because processing elements communicate directly only with cardinal neighbors. The same study notes that Cerebras tasks are non-preemptive, so explicit synchronization is needed before compute resumes (Belli et al., 8 May 2026).
For sparse kernels, the dominant interface is low-level CSL. A study of SpMM and SDDMM states that the CS-3 high-level stack does not support sparse tensors directly, so sparse kernels were designed manually in CSL. The paper emphasizes a task-based execution model with explicit PE-to-PE data movement, and it ties kernel parameters such as max_y_chunk, max_v_per_pe, and max_nonzeros directly to local-memory limits. In that account, local PE memory is treated as a first-class algorithmic constraint rather than a transparent cache (Shah et al., 30 Apr 2026).
At the system-software level, the LLM benchmarking literature presents CSTorch as a specialized PyTorch API that requires tensors to be initialized ahead of time, a step closure to avoid retrieving tensor values too early, and custom template code for individual operations. The same benchmark study states that individual operator benchmarks are not currently possible on CS-3 in the same way as on conventional stacks, and that Cerebras is among the slower-to-build specialized accelerator systems. This is why the literature repeatedly characterizes programmability and stack maturity as weaker than mainstream GPU ecosystems, even when raw hardware metrics are favorable (Golden et al., 12 Apr 2026).
3. Stencil computation, PDE solvers, and planetary-scale simulation
A substantial fraction of the CS-3 research literature treats the machine as a platform for communication-heavy scientific kernels. One line of work introduces Domain Translation, a stencil algorithm that shifts the processor-to-grid mapping by $2$9 grid points per iteration, where $5$0 is the stencil radius, rather than holding the mapping fixed as in conventional domain decomposition. The goal is to replace bidirectional latency-bound boundary exchange with directional pipelined communication in which latency is amortized across the width of a subdomain. On a cluster of $5$1 Cerebras CS-3 systems, this method is reported to run simulations in excess of $5$2 million time steps per second, achieve $5$3 PFLOP/s, and attain $5$4 of peak performance in both single-node and clustered environments. The same study reports $5$5 PFLOP/s on a $5$6-node CS-3 cluster for heat-equation workloads, near-perfect weak scaling from $5$7 to $5$8 nodes, and weak-scaling efficiency ranging from $5$9 to $4$0 in the stated heat-equation cases. Its asymptotic utilization analysis distinguishes workloads: $4$1 for the $4$2-point heat equation, $4$3 for the $4$4-point heat equation, and $4$5 for the shallow-water-equation workload (Oppelstrup et al., 14 Nov 2025).
The same paper uses the shallow water equations to model a planetary-scale tsunami triggered by an asteroid impact. The solver uses Lax-Wendroff spatial discretization and two-stage Runge-Kutta time integration, with a full timestep cost of $4$6 FLOPs per grid point, broken into an even half-step of $4$7 FLOPs and an odd half-step of $4$8 FLOPs. The impact is represented as a sine hump spanning about $4$9, with height up to 0 m and volume 1 trillion 2, corresponding in the paper’s setup to an asteroid of about 3 million kg hitting at 4 km/s. Bathymetry is taken from the GEBCO_2024 Grid at 5 arc-second / 6 m resolution (Oppelstrup et al., 14 Nov 2025).
A complementary CS-3 study focuses on manually written 7-D Jacobi-style stencil computations via a framework called CStencil. The formulation is the standard Jacobi update
8
implemented on the WSE-3 as repeated halo-exchange and compute phases. The input matrix is padded so that it divides evenly across the PE grid, each PE receives one tile, and each tile is padded locally with a zero halo of thickness equal to the stencil radius 9. The evaluation covers Star2d and Box2d patterns with $44$0 and $44$1, over $44$2 iterations and grids from $44$3 up to $44$4. Throughput is reported in
$44$5
In the largest tested Star2d-1r case, CStencil on WSE-3 achieves up to $44$6 speedup over an adapted ConvStencil implementation on an NVIDIA A100. The study also reports near-perfect weak scaling, validates its cycle-accurate simulator against actual CS-3 hardware with nearly all real measurements within a $44$7 confidence interval of simulated values, and computes an arithmetic intensity of about $44$8 for Star2d-1r on a $44$9 grid, with a theoretical peak of $21$0 PFLOP/s for the $21$1 PE subarray used in that roofline analysis (Belli et al., 8 May 2026).
The combined implication of these studies is that the CS-3 is being used as a stencil engine in a literal sense: the primary design match is between regular nearest-neighbor data movement and a wafer-scale mesh with distributed SRAM.
4. Sparse linear algebra kernels
The CS-3 has also been examined as a sparse linear algebra accelerator, especially for workloads associated with GNNs, GATs, linear solvers, recommendation systems, sparse attention, graph analytics, and electronic design automation. The two kernels emphasized are sparse-dense matrix multiplication,
$21$2
and sampled dense-dense matrix multiplication,
$21$3
The SpMM design stores $21$4 in CSR form, distributes $21$5 across PEs, and assigns three roles to PEs: routers, workers, and accumulators. Sparse data arrive as streams of $21$6-bit column indices and $21$7-bit floating-point values, with an END_ROW marker instead of an explicit row-pointer stream. To mitigate I/O bottlenecks, the study introduces a SELLPACK-like sparse format tailored to router and worker rows, then adds a multi-accumulator strategy to reduce serialization in host runtime copy calls. SDDMM uses a different layout: routers on the north and west edges, sparse $21$8 tiles in COO format within workers, and streamed rows and columns of the dense factors $21$9 and 0 (Shah et al., 30 Apr 2026).
The reported speedups are substantial but explicitly conditional on sparsity regime. For SpMM, the paper states that CS-3 can exceed CPU performance by more than 1, with the abstract highlighting 2 speedup at 3 sparsity. For SDDMM, the abstract reports up to 4 speedup at 5 sparsity. At the same time, the study stresses that hyper-sparsity is unfavorable: beyond 6 sparsity, CS-3 SpMM performance degrades enough that it can become slower than CPU. Memory footprint is also a trade-off. At density 7 (8 sparsity), the SELLPACK-like format may require 9–0 more space than CSR; at 1, it uses about 2 GB with myc = 1024 and about 3 GB with myc = 256, whereas CSR for the same matrix is about 4 MB. By contrast, at density 5 (6 sparsity), the SELLPACK-like format is about 7 CSR, and both remain far below the 8 GB required by a dense representation at that matrix size (Shah et al., 30 Apr 2026).
This evidence is important because it refines a common assumption that wafer-scale locality automatically favors all sparse workloads. The CS-3 literature instead indicates a narrower proposition: the machine is strong for large sparse problems when streaming, routing, and buffering are co-designed with the kernel, but not for all sparsity patterns.
5. LLM inference behavior and cross-accelerator comparisons
In the AI-inference literature, the CS-3 is presented as a low-batch, latency-sensitive accelerator with unusually strong communication-energy behavior. A broad cross-platform study compares CS-3 with A100, H100, MI300X, SambaNova SN-40, Groq, Gaudi, and TPUv5e, and reports that for Llama-3.1-8B at low batch size, CS-3 achieves the lowest latency per token at 9 of H100 latency. In the same setting, Groq reaches $44$00 of H100 latency and SN-40 reaches $44$01. The paper further states that CS-3 is optimal in the energy-latency trade-off at low batch sizes for both prefill and decode, but drops off the Pareto curve as batch size grows, especially in prefill, while remaining more competitive in decode than in prefill at higher batch sizes (Golden et al., 12 Apr 2026).
That same study attributes the behavior to the on-wafer memory system summarized in its inference table: prefill compute of $44$02 TFLOPS, decode memory type SRAM, decode capacity $44$03 GB, and decode bandwidth $44$04 B/s. It also emphasizes a distinctive power profile. For CS-3, decode power consumption equals prefill power consumption, with normalized power at $44$05 of TDP for both phases. Idle power is reported as $44$06 of TDP, compared with $44$07 for NVIDIA A100/H100 and AMD MI300, $44$08 for Gaudi, and $44$09 for SambaNova. The same paper states that CS-3 reaches energy-per-token parity with a $44$10-GPU H100 cluster at $44$11 duty cycle (Golden et al., 12 Apr 2026).
The communication-energy result is among the strongest quantitative claims in the CS-3 inference literature. For communication over $44$12 mm, the paper reports that CS-3 uses $44$13 less joules per byte than an H100 system and $44$14 less energy per byte than Groq; a plot caption in the same study states up to $44$15 less energy per byte than H100. This suggests that the CS-3’s comparative advantage is largest when a workload can remain substantially on-wafer rather than fragmenting across many external devices (Golden et al., 12 Apr 2026).
6. Packaging, power delivery, reliability, and limits of generality
The CS-3 literature does not present the machine as costless or universally superior. A comparative architectural paper devotes substantial attention to packaging and thermal engineering. It states that traditional packaging is inadequate for a $44$16 wafer-scale processor, so Cerebras mounts the wafer directly on the board, with a stack consisting of PCB, flexible membrane, WSE-3 die, and heat exchanger. The same source reports a need for more than $44$17 amperes with excellent voltage regulation, more than $44$18 VRMs distributed across the wafer, vertical power delivery from above the wafer, and roughly $44$19 kW of heat extraction via a water-cooled cold plate and copper heat exchanger (Kundu et al., 11 Mar 2025).
The same paper frames fault tolerance as a core enabler of wafer-scale viability. Its yield table reports a fault-tolerant core size of $44$20 for WSE-3 versus $44$21 for H100, and it concludes that WSE-3 is about $44$22 more fault tolerant for cores because each defect affects a much smaller functional unit. At the same time, it explicitly states that the long-term reliability of the custom materials, custom connectors, and custom assembly techniques needs to be looked into (Kundu et al., 11 Mar 2025).
Cost-effectiveness is treated similarly. Under the rack-normalized assumptions used in that comparison, the paper gives price estimates of $44$23M per rack, against $44$24M per rack, and $44$25M per rack. It reports that CS-3 exceeds H100 on raw FP8 and FP16 throughput and is competitive with or ahead of B200 on some throughput and performance-per-watt measures, but also states that B200 is significantly better on performance-per-watt-per-dollar, by about $44$26–$44$27 depending on the metric (Kundu et al., 11 Mar 2025).
Other studies reinforce the point that the CS-3 is not a universal replacement for CPUs or GPUs. The WSE-3 stencil study notes that the A100 is better for small grids because the WSE-3 suffers from underutilization when not enough PEs are active; the sparse-kernel study shows that SpMM can become slower than CPU beyond $44$28 sparsity; and the inference comparison shows that throughput-oriented platforms such as SambaNova, MI300, H100, and TPUv5e become more competitive as batch size increases (Belli et al., 8 May 2026, Shah et al., 30 Apr 2026, Golden et al., 12 Apr 2026). The most balanced reading of the literature is therefore that the CS-3 is highly specialized rather than universally dominant: its strongest regime is one in which locality, regular communication, and high duty cycle align with the wafer-scale dataflow model.