SLDB: System-Level Design Benchmark

• SLDB is a benchmark suite that evaluates LLM performance in full SoC integration, highlighting issues of syntactic and functional correctness over isolated RTL blocks.
• It targets heterogeneous, tile-based designs on the ESP platform, incorporating accelerators like AES, FFT, and LSTM for comprehensive system-level evaluation.
• SLDB provides a synthetic library for creating varied SoC configurations and detailed APS evaluations, focusing on DMA, NoC communication, and interface specifications.

System-Level Design Benchmark (SLDB) is a benchmark suite and dataset introduced to evaluate LLMs on system-level hardware design rather than on isolated RTL blocks. Built on the ESP open-source System-on-Chip platform, it targets heterogeneous, tile-based SoCs containing processors, memories, peripherals, and loosely coupled accelerators (LCAs), and provides the artifacts needed to study accelerator integration, communication setup, driver generation, and full SoC configuration (Alvanaki et al., 8 Jul 2025).

1. Motivation and problem setting

SLDB was introduced in response to a gap in LLM-aided Electronic Design Automation evaluation. Earlier benchmarks such as VerilogEval, RTL-Repo, Koios, MG-Verilog, and MetRex are described as valuable for early research, but they primarily evaluate self-contained hardware components, syntax correctness, or simple functional correctness of isolated RTL blocks. SLDB instead targets the setting in which many heterogeneous components must coexist, architecture-level parameters such as NoC widths and cache configurations matter, IP reuse and integration through standardized protocols are central, and software stacks must interact with hardware via drivers and memory maps (Alvanaki et al., 8 Jul 2025).

A central argument behind SLDB is that pass-rate figures reported for component-level hardware generation can be misleading when extrapolated to full SoCs. The benchmark description emphasizes that existing pass@k-style evaluations do not capture interface specifications, system protocols, inter-component timing, or the integration and configuration tasks that dominate SoC design effort. SLDB is therefore positioned around realistic heterogeneous SoC designs, together with the full system-level artifacts needed to compile, simulate, and, in principle, prototype the resulting systems on FPGA through ESP (Alvanaki et al., 8 Jul 2025).

This framing makes SLDB explicitly different from a benchmark for RTL authoring alone. Its core object of evaluation is whether an LLM can perform system integration and configuration under platform rules, not merely emit syntactically correct Verilog.

2. Benchmark structure and design space

At its core, SLDB contains 10 baseline SoC designs. Each baseline uses a 2×2 tile-based ESP architecture with one CPU tile containing an Ariane 64-bit RISC-V core, one memory tile, one auxiliary tile, and one accelerator tile. The accelerator tile hosts one selected LCA. The ten accelerators are AES Encryption, AES Decryption, SHA-256, SOBEL, FFT, FCDNN, LSTM, SIMPLEDNN, SPMV, and CONVOLUTION (Alvanaki et al., 8 Jul 2025).

Accelerator	Source	Domain
AES Encryption	RTL-Repo	cryptography
AES Decryption	RTL-Repo	cryptography
SHA-256	RTL-Repo	cryptography
SOBEL	RTL-Repo	edge detection
FFT	RTL-Repo	signal processing
FCDNN	RTL-Repo	deep learning
LSTM	Koios	machine learning
SIMPLEDNN	RTL-Repo	deep learning
SPMV	Koios	sparse matrix–vector / MLP
CONVOLUTION	RTL-Repo	CNN

These accelerators were selected for suitability to loosely coupled integration, workload complexity, realistic communication patterns, and open-source availability. The benchmark description also highlights substantial variation in scale, including long-context RTL such as FFT and SPMV with more than 100k lines, and reports that all ten SoCs were simulated for correctness and synthesized/implemented using Vivado 2023.2 to obtain PPA data. The reported LUT utilization ranges roughly from about 98k to about 150k, static power from about 2.86 W to about 5.83 W, and maximum datapath delays from about 3.8 ns for the FFT SoC to about 72.65 ns for the AES Decryption SoC (Alvanaki et al., 8 Jul 2025).

Beyond the baseline systems, SLDB includes a synthetic library that allows the baseline accelerators to be composed into an exponential number of tile-based SoCs. Configurable parameters include FPGA board, CPU type, shared local memory per tile from 64 KB to 4096 KB in powers of 2, accelerator data allocation strategy, cache configuration, SoC size up to $8 \times 8$ tiles, coherence and DMA NoC plane bitwidths from 32 to 1024 bits in powers of 2, and multicast up to 16 destinations (Alvanaki et al., 8 Jul 2025). The benchmark description does not give a closed-form combinatorial count, but it characterizes the configuration space as effectively

$$\text{Configurations} \sim \prod_{\text{tiles}}(\text{tile-type options}) \times \prod_{\text{global params}}(\text{parameter options}).$$

Each baseline instance provides SoC configuration files, Verilog accelerator integration wrappers, communication parameters, accelerator-aware system configurations, and testing-application code in bare-metal and Linux variants. These artifacts are directly usable with ESP generation, simulation, and FPGA synthesis flows (Alvanaki et al., 8 Jul 2025).

3. ESP integration and task decomposition

SLDB is built entirely on ESP, a modular tile-based SoC platform whose tiles connect through a multi-plane NoC and whose sockets provide DMA engines, caches, interrupt controllers, and other platform services. In SLDB, accelerators are integrated as LCAs inside accelerator tiles via ESP sockets, and all benchmark artifacts follow ESP conventions for memory maps and device modeling (Alvanaki et al., 8 Jul 2025).

The benchmark formalizes a design flow in which accelerators are chosen from Koios or RTL-Repo, accelerator configuration parameters are specified, ESP generates templates for C testing applications and Verilog accelerator wrappers together with updated configuration files, the wrapper is filled in and test applications may be edited, the SoC configuration is selected through the ESP GUI, the full SoC RTL is auto-generated, then simulated and synthesized (Alvanaki et al., 8 Jul 2025). In the LLM setting, the manual steps in this flow are the main automation targets.

SLDB decomposes system-level design into modular tasks. These include generating integration wrappers that implement DMA handshaking and the Configure–Load–Compute–Store state machine, extracting configuration registers and memory requirements from accelerator RTL, generating C drivers that program memory-mapped registers and trigger execution, generating system-level SoC configuration parameters such as tile counts and NoC widths, and assembling all artifacts required for end-to-end integration of a new accelerator (Alvanaki et al., 8 Jul 2025).

In the case study described for the benchmark, LLM inputs include accelerator RTL, an ESP wrapper template with comments specifying required functionality, and possibly ESP specifications for DMA and configuration registers. Expected outputs include completed Verilog wrappers with correct DMA handshake logic for Load and Store APS, correct mapping between accelerator ports and DMA/control signals, correct Configure–Load–Compute–Store sequencing, correct extraction of configuration parameters and port directions, and, in some experiments, correct driver code (Alvanaki et al., 8 Jul 2025).

4. Evaluation methodology and empirical results

SLDB evaluates integration through the Accelerator Processing Stages (APS): Configuration, Load, Compute, and Store. Functional correctness is checked stage by stage. Configuration asks whether the wrapper and driver assign parameters correctly and initialize registers; Load checks DMA requests, handshakes, and input delivery; Compute checks port mapping, synchronization, and internal sequencing; Store checks DMA write-back of results with proper synchronization. An integration is fully functionally correct only if all four APSs succeed in simulation against the human-engineered reference design (Alvanaki et al., 8 Jul 2025).

The benchmark reports both syntactic correctness and functional correctness, using pass@1 and pass@3. It also counts successful APS completions per SoC and per model, counts total successful APSs across accelerators and runs, classifies error types, and reports correct designs per SoC type. This finer granularity is important because the reported failure distribution clusters around incorrect DMA signal handling, incorrect DMA state transitions, and specification-adherence errors such as active-low reset polarity (Alvanaki et al., 8 Jul 2025).

Model	pass@1	pass@3
o1	70% syntax, 40% functional	90% syntax, 50% functional
o3-mini-high	100% syntax, 10% functional	100% syntax, 20% functional
GPT-4o	60% syntax, 0% functional	70% syntax, 0% functional
DeepSeek-R1-671B	60% syntax, 0% functional	60% syntax, 30% functional

These results show that syntax is often much easier than end-to-end integration. The benchmark narrative states that Configuration is relatively easy, while Load is significantly harder, with Compute and Store often failing downstream of Load errors. Even the best reported model, o1, reaches only 50% functional pass@3 on the evaluated system-level tasks (Alvanaki et al., 8 Jul 2025).

The appendix also reports an in-context learning study in which three correct wrappers—FCDNN, LSTM, and SOBEL—are supplied as examples and the remaining seven accelerators are targeted. The reported functional results remain low: o1 reaches 0% functional with 100% syntax, o3-mini-high reaches 14.2% functional with 71% syntax, GPT-4o reaches 0% functional with 71% syntax, and DeepSeek-R1 reaches 0% functional with 43% syntax. The APS breakdown indicates some improvement in Configuration and Load counts, but complete Configure–Load–Compute–Store correctness remains rare (Alvanaki et al., 8 Jul 2025).

5. Position within the benchmark landscape

SLDB is described as the first benchmark explicitly positioned as a System-Level Design Benchmark for LLM-aided SoC design. Relative to VerilogEval, RTL-Repo, Koios, MG-Verilog, MetRex, RTLLM, and related RTL-generation datasets, its distinguishing features are a real SoC platform, an end-to-end stack spanning accelerators, wrappers, configuration, drivers, and tests, an emphasis on DMA and NoC communication, support for exponentially many SoCs through a synthetic library, and both fine-grain APS-level and whole-SoC evaluation (Alvanaki et al., 8 Jul 2025).

Later work uses the SLDB notion more broadly. ArchXBench is described as a first concrete step toward a system-level design benchmark for LLM-driven RTL and hardware design, but specifically as a datapath-oriented SLDB rather than a full SoC-integration benchmark (Purini et al., 8 Aug 2025). HSCO-Bench is presented as the first benchmark targeting complete end-to-end hardware-software co-design flow and is described as effectively functioning as a System-Level Design Benchmark in the SoC co-design space, extending beyond integration of a predefined accelerator to kernel selection, accelerator design, software modification, and FPGA deployment (Tsai et al., 19 May 2026). In RTL generation, RealBench’s system-level track is described as a de facto System-Level Design Benchmark for real-world IP hierarchies with 100% line coverage and formal equivalence checking (Jin et al., 22 Jul 2025). In repository-scale software generation, RealBench is described as very close to what would be called a System-Level Design Benchmark, because it evaluates design-to-code performance from UML system designs at repository scale (Li et al., 24 Apr 2026). CircuitSense likewise functions as a System-Level Design Benchmark for hierarchical circuit understanding by linking system-level block diagrams to symbolic transfer-function derivation across multiple abstraction levels (Akbari et al., 26 Sep 2025).

This broader usage suggests that, after the introduction of SLDB as a concrete SoC benchmark, the term began to denote a wider class of benchmarks that test architecture, integration, and cross-component reasoning rather than isolated code generation.

6. Limitations and future directions

The original SLDB has several explicit limitations. It is built around a single SoC framework, ESP, and a tile-based architecture; all accelerators are LCAs, so tightly coupled accelerators are outside scope; only ten baseline accelerators and 2×2 SoCs are fully integrated and characterized; larger SoCs are supported through the synthetic library but are not explicitly enumerated; the evaluation assumes ESP’s NoC, DMA, and socket protocols; and experiments cover a small set of frontier models without fine-tuning or domain-specific training (Alvanaki et al., 8 Jul 2025).

The benchmark description also identifies several directions for extension. These include adding more accelerators and more baseline SoCs, covering additional system-level tasks such as cache and coherence configuration and multi-accelerator dataflow setup, and potentially supporting other SoC platforms beyond ESP. On the LLM side, proposed directions include specification-aware methods, improved long-context handling such as StreamingLLM, more advanced prompt engineering, agentic workflows, tool-feedback loops, and improved modeling of state machines and protocol adherence (Alvanaki et al., 8 Jul 2025).

The reported results imply a clear research frontier. SLDB shows that state-of-the-art LLMs can often generate compilable text artifacts for wrappers and drivers, but remain far from fully automating realistic system-level accelerator integration. A plausible implication is that future progress will depend less on syntax-level code generation alone and more on methods that explicitly reason about interfaces, communication protocols, timing, and cross-layer software-hardware contracts.