---
title: 'HardSecBench: Security Code Benchmark'
url: https://www.emergentmind.com/topics/hardsecbench
type: topic
---

# HardSecBench: Security Code Benchmark

HardSecBench is a benchmark for evaluating the **security awareness** of large language models in **hardware and firmware code generation**, with emphasis on the distinction between **functional correctness** and **security correctness** under realistic specifications. Introduced in "HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation" [2601.13864], it targets a failure mode in which generated Verilog RTL or firmware-level C satisfies visible functional requirements while omitting protections such as access-control checks, lock enforcement, or safe state-transition handling. The benchmark is explicitly **CWE-driven**, **execution-grounded**, and constructed through a **multi-agent pipeline** that separates specification, implementation synthesis, and test generation, with security requirements withheld from the evaluated model and used only for scoring.

## 1. Scope and motivating problem

HardSecBench addresses the proposition that hardware and firmware generation should not be evaluated solely by whether code compiles or passes ordinary functional tests. The benchmark is built around the observation that code can be functionally sound yet still embed exploitable weaknesses, and that prior evaluations in hardware-code generation have largely concentrated on functional correctness while paying limited attention to security issues [2601.13864].

The benchmark therefore asks a more specific question than standard code-generation suites: whether a model can generate code that is **not only functionally correct, but also security-aware**. Its task design reflects two premises. First, security is often **underspecified** in LLM-assisted hardware generation; models may satisfy a visible prompt while failing to infer latent protections expected in deployed systems. Second, existing security evaluations for hardware and firmware generation have often been **fragmented and unreliable**, relying on bespoke test sets, manually written assertions, or LLM-based judging rather than execution evidence [2601.13864].

A defining design choice is the separation of **functional intent** from **security intent**. Models receive only the functional requirements, while the hidden security requirements are used during evaluation. This makes HardSecBench a test of whether security knowledge is activated without direct prompting, rather than a test of whether the model can restate explicit safety instructions.

## 2. Benchmark composition and task model

HardSecBench contains **924 validated tasks** after filtering, spanning **599 Verilog tasks** and **325 firmware-level C tasks**, and covering **76 hardware-relevant CWE entries**. The construction process began from **1170 samples**, of which **246 samples** were removed during quality filtering. The final benchmark includes **4425 test cases** [2601.13864].

| Component | Value |
|---|---|
| Final tasks | 924 |
| Verilog RTL tasks | 599 |
| Firmware-level C tasks | 325 |
| Hardware-relevant CWE entries | 76 |
| Initial samples | 1170 |
| Removed in filtering | 246 |
| Final test cases | 4425 |

Each task includes three core artifacts: **a structured specification**, **a secure reference implementation**, and **executable tests / requirement-level harnesses** [2601.13864]. The structured specification is not a single prompt fragment; it separates the requirements of task \(i\) into **functional requirements** \(\mathcal{R}^{f}_{i}\) and **security requirements** \(\mathcal{R}^{s}_{i}\), both organized under a common specification \(P_i\). The functional requirements are written so as to **omit security intent**, while the security requirements encode the protections expected from the relevant CWE guidance.

This separation is central to the benchmark’s interpretation. A model that passes functional checks but fails security checks has not merely produced an incomplete answer; it has exposed a gap between visible-task completion and latent secure-design reasoning. The benchmark’s dual-language coverage also matters. It spans both RTL and firmware-level code, whereas prior hardware benchmarks are described as having focused largely on Verilog RTL generation [2601.13864].

## 3. Multi-agent construction pipeline

HardSecBench uses a **multi-agent pipeline** with explicit information isolation. The process begins with a **Seed Generator** that converts each CWE definition into a seed specifying the implementation language and a short scenario in which the weakness could arise. An **Architect** then expands the seed into the structured specification \(P_i\), including the problem statement, I/O interface specification, \(\mathcal{R}^{f}_{i}\), and \(\mathcal{R}^{s}_{i}\) [2601.13864].

Artifact synthesis is then split into two independent branches. The **Expert** synthesizes the golden implementation, and the **Tester** synthesizes the requirement-level harnesses. Both agents see only \(P_i\), and neither sees the other’s outputs. The benchmark describes this decoupling as a safeguard against logic coupling: harnesses should not accidentally encode implementation details from the reference solution, because that would inflate pass rates and reduce diagnostic value [2601.13864].

An **Arbiter** performs iterative reconciliation when the independently created specification, implementation, and harnesses disagree. The Arbiter uses the structured specification, the golden implementation, the harnesses, and runtime evidence and analysis traces to localize failures and provide targeted feedback. The loop repeats until the artifacts agree [2601.13864].

Before inclusion in the benchmark, tasks pass a quality gate based on **coverage** and **mutation score**. The minimum coverage threshold is **80%**. For C, line coverage is measured using **gcov**; for Verilog, coverage is estimated via static analysis mapping harness signals to executable lines. Retained tasks have **92.5% average coverage**. Mutation filtering uses five operators—**constant change**, **operator swap**, **condition negation**, **stuck-at signal**, and **assignment removal**—with five mutants generated per task, and tasks are retained only if the mutation score is at least **50%**. The reported **mean mutation score** is **70.2%** [2601.13864].

These quality controls are important because execution-grounded evaluation can still be weak if the harnesses do not exercise the right behaviors. HardSecBench treats benchmark construction itself as a security-sensitive engineering problem.

## 4. Evaluation protocol and metrics

HardSecBench evaluates generated code through **execution evidence** rather than subjective natural-language judging. For each sample, requirement-level harnesses emit standardized **PASS/FAIL** traces, and evaluation is determined by whether the generated implementation satisfies the **functional harnesses** and the **security harnesses** [2601.13864]. The benchmark states that the harnesses are designed to **actively trigger security-relevant behaviors**, so that vulnerabilities missed by conventional functional tests can still be exposed.

The paper defines two main evaluation settings. In **single-attempt evaluation**, the model receives only the functional requirements. The output must be compilable; if compilation fails, up to **three compile/fix rounds** are allowed using only compiler error messages. Once a compilable version is obtained, the full functional and security harnesses are run once, and the model does **not** receive the harness results back [2601.13864]. This setting measures initial security intuition rather than post hoc debugging.

In **iterative refinement evaluation**, the model generates a compilable implementation and the functional harnesses are run. A **Collaborator** then provides feedback only on **functional** failures and code issues. Security feedback is withheld throughout the refinement loop, which continues for up to **5 iterations** or until the functional requirements pass; security is then evaluated on the final implementation [2601.13864]. This protocol is intended to separate security awareness from ordinary functional defect repair.

The benchmark reports functional and security performance using a unified **Pass@k-style** requirement-level metric. With \(d \in \{\mathrm{Func}, \mathrm{Sec}\}\), \(N\) tasks, \(M_{i,d}\) atomic requirements for task \(i\) and type \(d\), \(n\) sampled generations per task, and \(c_{i,d,j}\) generations that pass atomic requirement \(j\), the paper defines \(R_d@k\) as a Pass@k-style average over atomic requirements under **sampling without replacement** [2601.13864]. The important methodological point is that the unit of evaluation is the **atomic requirement**, not merely a coarse task-level success flag.

## 5. Empirical findings

The central empirical result is that **functional pass rates are substantially higher than security pass rates across models** [2601.13864]. HardSecBench therefore operationalizes a persistent discrepancy: hardware and firmware code generation systems can produce code that appears correct under the visible task formulation while still failing to implement the security protections implied by the hidden CWE-grounded requirements.

The paper further reports that security performance has only a **weak relationship with general code-generation capability** when no explicit security guidance is provided. This suggests that security awareness is not an automatic by-product of stronger general coding performance [2601.13864]. In the iterative setting, models typically reach high functional correctness after about **1–2 iterations**, but security scores improve only slightly, because the Collaborator provides only functional feedback. Likewise, increasing the sampling budget to **Pass@5** helps function more than security, implying that many security failures are systematic rather than random [2601.13864].

HardSecBench also studies prompting sensitivity through three prompt settings: **Hint 0** (only functional requirements), **Hint 1** (generic security reminders), and **Hint 2** (explicit mention of vulnerability classes). The reported pattern is that **closed-source models** often respond strongly to security reminders, with Hint 1 already producing noticeable gains and Hint 2 yielding larger gains, whereas **smaller open-source models** are less responsive even under explicit hints [2601.13864]. The benchmark interprets this as evidence that stronger proprietary models may possess dormant security knowledge that is not surfaced when only functional requirements are provided.

The paper identifies broad model trends rather than a single definitive leaderboard. Examples cited among the strongest closed-source models include **Claude-4.5-Opus**, **Gemini-3-Pro-Preview**, **GPT-5.1-medium**, and **GPT-5-medium**. Among open-source models, stronger entries include **DeepSeek-V3.2**, **GLM-4.6**, **GPT-OSS-120B**, and **Qwen3-Coder-480B-A35B**, while smaller models such as **Llama-4-Scout-Instruct** and **Qwen3-14B** tend to have lower security performance. Across all model groups, however, **security scores remain far below functional scores** [2601.13864].

The benchmark also reports category-level findings. It maps hardware-relevant sub-CWEs into **12 higher-level categories**, including **Security Flow Issues**, **Integration Issues**, **Privilege Separation and Access Control Issues**, **General Circuit and Logic Design Concerns**, **Core and Compute Issues**, **Memory and Storage Issues**, **Peripherals, On-chip Fabric, and Interface/IO Problems**, **Security Primitives and Cryptography Issues**, **Power, Clock, Thermal, and Reset Concerns**, **Debug and Test Problems**, **Cross-Cutting Problems**, and **Physical Access Issues and Concerns** [2601.13864]. The reported security pass rate is **38.31% for firmware C** and **44.93% for Verilog RTL**, a gap of **6.63 percentage points**. The hardest categories are **power/clock/thermal/reset concerns** and **memory and storage issues**, which the paper associates with temporal behavior, state-machine consistency, cross-clock-domain interactions, and incomplete handling of physical access concerns [2601.13864].

## 6. Position within hardware-security benchmarking

HardSecBench occupies a distinct position within the broader landscape of hardware-security benchmarks. It is a benchmark for **LLM-generated code security** under hidden security requirements, not a benchmark for post-silicon attack detection, Hardware Trojan search, or end-to-end hardware-software co-design. This distinction matters because the name has been used in at least one adjacent context: the benchmark introduced as **Seeker1** in "The Seeker's Dilemma: Realistic Formulation and Benchmarking for Hardware Trojan Detection" [2402.17918] is described in the provided material as being later referred to by the query as **HardSecBench**. That benchmark concerns **unknown-infection** Hardware Trojan detection on restructured netlists, with a mixture of clean and infected circuits and a game-theoretic framing in which the defender does not know whether an HT exists at all [2402.17918]. The modern HardSecBench of [2601.13864], by contrast, evaluates whether LLMs infer hidden secure-design constraints when generating Verilog or firmware from ostensibly functional specifications.

A second relevant comparison concerns **benchmark validity** in the LLM era. "AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs" argues that hardware-security competition benchmarks can be undermined when models exploit **diff-style syntactic clues** rather than genuine vulnerability reasoning, and proposes a **semantics-preserving obfuscation framework** to reduce that shortcut signal [2606.15809]. HardSecBench addresses a different layer of the evaluation problem by grounding scores in executable harnesses and by decoupling benchmark construction into isolated roles [2601.13864]. This suggests that robust hardware-security benchmarking now has at least two separable requirements: **execution-grounded scoring**, as in HardSecBench, and **artifact-level resistance to shortcut exploitation**, as emphasized by AttackonCTF [2606.15809].

A broader implication is that HardSecBench should be read not merely as a dataset of secure-code tasks, but as a methodological claim about evaluation. Functional correctness, security correctness, prompt sensitivity, benchmark construction quality, and leakage-resistant evaluation are treated as separate variables. In that respect, HardSecBench formalizes a narrower but technically consequential question: whether hardware-code generation systems surface security reasoning when security is **not explicitly requested**, but is still required by the underlying design problem [2601.13864].

Source: https://www.emergentmind.com/topics/hardsecbench