---
title: 'SLDB: System-Level Design Benchmark'
url: https://www.emergentmind.com/topics/system-level-design-benchmark-sldb
type: topic
---

# SLDB: System-Level Design Benchmark

System-Level Design Benchmark (SLDB) is a benchmark suite and dataset introduced to evaluate Large Language Models (LLMs) on **system-level** hardware design rather than on isolated RTL blocks. Built on the ESP open-source System-on-Chip platform, it targets heterogeneous, tile-based SoCs containing processors, memories, peripherals, and loosely coupled accelerators (LCAs), and provides the artifacts needed to study accelerator integration, communication setup, driver generation, and full SoC configuration [2507.06376].

## 1. Motivation and problem setting

SLDB was introduced in response to a gap in LLM-aided Electronic Design Automation evaluation. Earlier benchmarks such as VerilogEval, RTL-Repo, Koios, MG-Verilog, and MetRex are described as valuable for early research, but they primarily evaluate **self-contained** hardware components, syntax correctness, or simple functional correctness of isolated RTL blocks. SLDB instead targets the setting in which many heterogeneous components must coexist, architecture-level parameters such as NoC widths and cache configurations matter, IP reuse and integration through standardized protocols are central, and software stacks must interact with hardware via drivers and memory maps [2507.06376].

A central argument behind SLDB is that pass-rate figures reported for component-level hardware generation can be misleading when extrapolated to full SoCs. The benchmark description emphasizes that existing pass@k-style evaluations do not capture **interface specifications**, system protocols, inter-component timing, or the integration and configuration tasks that dominate SoC design effort. SLDB is therefore positioned around realistic heterogeneous SoC designs, together with the full system-level artifacts needed to compile, simulate, and, in principle, prototype the resulting systems on FPGA through ESP [2507.06376].

This framing makes SLDB explicitly different from a benchmark for RTL authoring alone. Its core object of evaluation is whether an LLM can perform system integration and configuration under platform rules, not merely emit syntactically correct Verilog.

## 2. Benchmark structure and design space

At its core, SLDB contains **10 baseline SoC designs**. Each baseline uses a **2×2 tile-based ESP architecture** with one CPU tile containing an Ariane 64-bit RISC-V core, one memory tile, one auxiliary tile, and one accelerator tile. The accelerator tile hosts one selected LCA. The ten accelerators are AES Encryption, AES Decryption, SHA-256, SOBEL, FFT, FCDNN, LSTM, SIMPLEDNN, SPMV, and CONVOLUTION [2507.06376].

| Accelerator | Source | Domain |
|---|---|---|
| AES Encryption | RTL-Repo | cryptography |
| AES Decryption | RTL-Repo | cryptography |
| SHA-256 | RTL-Repo | cryptography |
| SOBEL | RTL-Repo | edge detection |
| FFT | RTL-Repo | signal processing |
| FCDNN | RTL-Repo | deep learning |
| LSTM | Koios | machine learning |
| SIMPLEDNN | RTL-Repo | deep learning |
| SPMV | Koios | sparse matrix–vector / MLP |
| CONVOLUTION | RTL-Repo | CNN |

These accelerators were selected for suitability to **loosely coupled** integration, workload complexity, realistic communication patterns, and open-source availability. The benchmark description also highlights substantial variation in scale, including long-context RTL such as FFT and SPMV with more than 100k lines, and reports that all ten SoCs were simulated for correctness and synthesized/implemented using Vivado 2023.2 to obtain PPA data. The reported LUT utilization ranges roughly from about 98k to about 150k, static power from about 2.86 W to about 5.83 W, and maximum datapath delays from about 3.8 ns for the FFT SoC to about 72.65 ns for the AES Decryption SoC [2507.06376].

Beyond the baseline systems, SLDB includes a **synthetic library** that allows the baseline accelerators to be composed into an exponential number of tile-based SoCs. Configurable parameters include FPGA board, CPU type, shared local memory per tile from 64 KB to 4096 KB in powers of 2, accelerator data allocation strategy, cache configuration, SoC size up to \(8 \times 8\) tiles, coherence and DMA NoC plane bitwidths from 32 to 1024 bits in powers of 2, and multicast up to 16 destinations [2507.06376]. The benchmark description does not give a closed-form combinatorial count, but it characterizes the configuration space as effectively

\[
\text{Configurations} \sim \prod_{\text{tiles}}(\text{tile-type options}) \times \prod_{\text{global params}}(\text{parameter options}).
\]

Each baseline instance provides SoC configuration files, Verilog accelerator integration wrappers, communication parameters, accelerator-aware system configurations, and testing-application code in **bare-metal** and **Linux** variants. These artifacts are directly usable with ESP generation, simulation, and FPGA synthesis flows [2507.06376].

## 3. ESP integration and task decomposition

SLDB is built entirely on ESP, a modular tile-based SoC platform whose tiles connect through a multi-plane NoC and whose sockets provide DMA engines, caches, interrupt controllers, and other platform services. In SLDB, accelerators are integrated as LCAs inside accelerator tiles via ESP sockets, and all benchmark artifacts follow ESP conventions for memory maps and device modeling [2507.06376].

The benchmark formalizes a design flow in which accelerators are chosen from Koios or RTL-Repo, accelerator configuration parameters are specified, ESP generates templates for C testing applications and Verilog accelerator wrappers together with updated configuration files, the wrapper is filled in and test applications may be edited, the SoC configuration is selected through the ESP GUI, the full SoC RTL is auto-generated, then simulated and synthesized [2507.06376]. In the LLM setting, the manual steps in this flow are the main automation targets.

SLDB decomposes system-level design into modular tasks. These include generating integration wrappers that implement DMA handshaking and the Configure–Load–Compute–Store state machine, extracting configuration registers and memory requirements from accelerator RTL, generating C drivers that program memory-mapped registers and trigger execution, generating system-level SoC configuration parameters such as tile counts and NoC widths, and assembling all artifacts required for end-to-end integration of a new accelerator [2507.06376].

In the case study described for the benchmark, LLM inputs include accelerator RTL, an ESP wrapper template with comments specifying required functionality, and possibly ESP specifications for DMA and configuration registers. Expected outputs include completed Verilog wrappers with correct DMA handshake logic for Load and Store APS, correct mapping between accelerator ports and DMA/control signals, correct Configure–Load–Compute–Store sequencing, correct extraction of configuration parameters and port directions, and, in some experiments, correct driver code [2507.06376].

## 4. Evaluation methodology and empirical results

SLDB evaluates integration through the **Accelerator Processing Stages (APS)**: Configuration, Load, Compute, and Store. Functional correctness is checked stage by stage. Configuration asks whether the wrapper and driver assign parameters correctly and initialize registers; Load checks DMA requests, handshakes, and input delivery; Compute checks port mapping, synchronization, and internal sequencing; Store checks DMA write-back of results with proper synchronization. An integration is fully functionally correct only if all four APSs succeed in simulation against the human-engineered reference design [2507.06376].

The benchmark reports both **syntactic correctness** and **functional correctness**, using pass@1 and pass@3. It also counts successful APS completions per SoC and per model, counts total successful APSs across accelerators and runs, classifies error types, and reports correct designs per SoC type. This finer granularity is important because the reported failure distribution clusters around incorrect DMA signal handling, incorrect DMA state transitions, and specification-adherence errors such as active-low reset polarity [2507.06376].

| Model | pass@1 | pass@3 |
|---|---|---|
| o1 | 70% syntax, 40% functional | 90% syntax, 50% functional |
| o3-mini-high | 100% syntax, 10% functional | 100% syntax, 20% functional |
| GPT-4o | 60% syntax, 0% functional | 70% syntax, 0% functional |
| DeepSeek-R1-671B | 60% syntax, 0% functional | 60% syntax, 30% functional |

These results show that syntax is often much easier than end-to-end integration. The benchmark narrative states that **Configuration** is relatively easy, while **Load** is significantly harder, with Compute and Store often failing downstream of Load errors. Even the best reported model, o1, reaches only 50% functional pass@3 on the evaluated system-level tasks [2507.06376].

The appendix also reports an in-context learning study in which three correct wrappers—FCDNN, LSTM, and SOBEL—are supplied as examples and the remaining seven accelerators are targeted. The reported functional results remain low: o1 reaches 0% functional with 100% syntax, o3-mini-high reaches 14.2% functional with 71% syntax, GPT-4o reaches 0% functional with 71% syntax, and DeepSeek-R1 reaches 0% functional with 43% syntax. The APS breakdown indicates some improvement in Configuration and Load counts, but complete Configure–Load–Compute–Store correctness remains rare [2507.06376].

## 5. Position within the benchmark landscape

SLDB is described as the first benchmark explicitly positioned as a **System-Level Design Benchmark** for LLM-aided SoC design. Relative to VerilogEval, RTL-Repo, Koios, MG-Verilog, MetRex, RTLLM, and related RTL-generation datasets, its distinguishing features are a real SoC platform, an end-to-end stack spanning accelerators, wrappers, configuration, drivers, and tests, an emphasis on DMA and NoC communication, support for exponentially many SoCs through a synthetic library, and both fine-grain APS-level and whole-SoC evaluation [2507.06376].

Later work uses the SLDB notion more broadly. ArchXBench is described as a first concrete step toward a **system-level design benchmark** for LLM-driven RTL and hardware design, but specifically as a **datapath-oriented SLDB** rather than a full SoC-integration benchmark [2508.06047]. HSCO-Bench is presented as the first benchmark targeting complete end-to-end hardware-software co-design flow and is described as effectively functioning as a **System-Level Design Benchmark** in the SoC co-design space, extending beyond integration of a predefined accelerator to kernel selection, accelerator design, software modification, and FPGA deployment [2605.19399]. In RTL generation, RealBench’s system-level track is described as a de facto **System-Level Design Benchmark** for real-world IP hierarchies with 100% line coverage and formal equivalence checking [2507.16200]. In repository-scale software generation, RealBench is described as very close to what would be called a **System-Level Design Benchmark**, because it evaluates design-to-code performance from UML system designs at repository scale [2604.22659]. CircuitSense likewise functions as a **System-Level Design Benchmark** for hierarchical circuit understanding by linking system-level block diagrams to symbolic transfer-function derivation across multiple abstraction levels [2509.22339].

This broader usage suggests that, after the introduction of SLDB as a concrete SoC benchmark, the term began to denote a wider class of benchmarks that test architecture, integration, and cross-component reasoning rather than isolated code generation.

## 6. Limitations and future directions

The original SLDB has several explicit limitations. It is built around a single SoC framework, ESP, and a tile-based architecture; all accelerators are LCAs, so tightly coupled accelerators are outside scope; only ten baseline accelerators and 2×2 SoCs are fully integrated and characterized; larger SoCs are supported through the synthetic library but are not explicitly enumerated; the evaluation assumes ESP’s NoC, DMA, and socket protocols; and experiments cover a small set of frontier models without fine-tuning or domain-specific training [2507.06376].

The benchmark description also identifies several directions for extension. These include adding more accelerators and more baseline SoCs, covering additional system-level tasks such as cache and coherence configuration and multi-accelerator dataflow setup, and potentially supporting other SoC platforms beyond ESP. On the LLM side, proposed directions include specification-aware methods, improved long-context handling such as StreamingLLM, more advanced prompt engineering, agentic workflows, tool-feedback loops, and improved modeling of state machines and protocol adherence [2507.06376].

The reported results imply a clear research frontier. SLDB shows that state-of-the-art LLMs can often generate compilable text artifacts for wrappers and drivers, but remain far from fully automating realistic system-level accelerator integration. A plausible implication is that future progress will depend less on syntax-level code generation alone and more on methods that explicitly reason about interfaces, communication protocols, timing, and cross-layer software-hardware contracts.

Source: https://www.emergentmind.com/topics/system-level-design-benchmark-sldb