HardSecBench: Security Code Benchmark

• HardSecBench is a benchmark that assesses LLM-generated code by measuring both functional and security correctness in hardware and firmware code generation.
• It employs a multi-agent pipeline with isolated roles to create structured specifications, secure implementations, and executable tests based on CWE-driven criteria.
• Empirical results show a significant gap between high functional pass rates and lower security pass rates, revealing systematic security weaknesses in the evaluated models.

HardSecBench is a benchmark for evaluating the security awareness of LLMs in hardware and firmware code generation, with emphasis on the distinction between functional correctness and security correctness under realistic specifications. Introduced in "HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation" (Chen et al., 20 Jan 2026), it targets a failure mode in which generated Verilog RTL or firmware-level C satisfies visible functional requirements while omitting protections such as access-control checks, lock enforcement, or safe state-transition handling. The benchmark is explicitly CWE-driven, execution-grounded, and constructed through a multi-agent pipeline that separates specification, implementation synthesis, and test generation, with security requirements withheld from the evaluated model and used only for scoring.

1. Scope and motivating problem

HardSecBench addresses the proposition that hardware and firmware generation should not be evaluated solely by whether code compiles or passes ordinary functional tests. The benchmark is built around the observation that code can be functionally sound yet still embed exploitable weaknesses, and that prior evaluations in hardware-code generation have largely concentrated on functional correctness while paying limited attention to security issues (Chen et al., 20 Jan 2026).

The benchmark therefore asks a more specific question than standard code-generation suites: whether a model can generate code that is not only functionally correct, but also security-aware. Its task design reflects two premises. First, security is often underspecified in LLM-assisted hardware generation; models may satisfy a visible prompt while failing to infer latent protections expected in deployed systems. Second, existing security evaluations for hardware and firmware generation have often been fragmented and unreliable, relying on bespoke test sets, manually written assertions, or LLM-based judging rather than execution evidence (Chen et al., 20 Jan 2026).

A defining design choice is the separation of functional intent from security intent. Models receive only the functional requirements, while the hidden security requirements are used during evaluation. This makes HardSecBench a test of whether security knowledge is activated without direct prompting, rather than a test of whether the model can restate explicit safety instructions.

2. Benchmark composition and task model

HardSecBench contains 924 validated tasks after filtering, spanning 599 Verilog tasks and 325 firmware-level C tasks, and covering 76 hardware-relevant CWE entries. The construction process began from 1170 samples, of which 246 samples were removed during quality filtering. The final benchmark includes 4425 test cases (Chen et al., 20 Jan 2026).

Component	Value
Final tasks	924
Verilog RTL tasks	599
Firmware-level C tasks	325
Hardware-relevant CWE entries	76
Initial samples	1170
Removed in filtering	246
Final test cases	4425

Each task includes three core artifacts: a structured specification, a secure reference implementation, and executable tests / requirement-level harnesses (Chen et al., 20 Jan 2026). The structured specification is not a single prompt fragment; it separates the requirements of task $i$ into functional requirements $\mathcal{R}^{f}_{i}$ and security requirements $\mathcal{R}^{s}_{i}$, both organized under a common specification $P_i$. The functional requirements are written so as to omit security intent, while the security requirements encode the protections expected from the relevant CWE guidance.

This separation is central to the benchmark’s interpretation. A model that passes functional checks but fails security checks has not merely produced an incomplete answer; it has exposed a gap between visible-task completion and latent secure-design reasoning. The benchmark’s dual-language coverage also matters. It spans both RTL and firmware-level code, whereas prior hardware benchmarks are described as having focused largely on Verilog RTL generation (Chen et al., 20 Jan 2026).

3. Multi-agent construction pipeline

HardSecBench uses a multi-agent pipeline with explicit information isolation. The process begins with a Seed Generator that converts each CWE definition into a seed specifying the implementation language and a short scenario in which the weakness could arise. An Architect then expands the seed into the structured specification $P_i$, including the problem statement, I/O interface specification, $\mathcal{R}^{f}_{i}$, and $\mathcal{R}^{s}_{i}$ (Chen et al., 20 Jan 2026).

Artifact synthesis is then split into two independent branches. The Expert synthesizes the golden implementation, and the Tester synthesizes the requirement-level harnesses. Both agents see only $P_i$, and neither sees the other’s outputs. The benchmark describes this decoupling as a safeguard against logic coupling: harnesses should not accidentally encode implementation details from the reference solution, because that would inflate pass rates and reduce diagnostic value (Chen et al., 20 Jan 2026).

An Arbiter performs iterative reconciliation when the independently created specification, implementation, and harnesses disagree. The Arbiter uses the structured specification, the golden implementation, the harnesses, and runtime evidence and analysis traces to localize failures and provide targeted feedback. The loop repeats until the artifacts agree (Chen et al., 20 Jan 2026).

Before inclusion in the benchmark, tasks pass a quality gate based on coverage and mutation score. The minimum coverage threshold is 80%. For C, line coverage is measured using gcov; for Verilog, coverage is estimated via static analysis mapping harness signals to executable lines. Retained tasks have 92.5% average coverage. Mutation filtering uses five operators—constant change, operator swap, condition negation, stuck-at signal, and assignment removal—with five mutants generated per task, and tasks are retained only if the mutation score is at least 50%. The reported mean mutation score is 70.2% (Chen et al., 20 Jan 2026).

These quality controls are important because execution-grounded evaluation can still be weak if the harnesses do not exercise the right behaviors. HardSecBench treats benchmark construction itself as a security-sensitive engineering problem.

4. Evaluation protocol and metrics

HardSecBench evaluates generated code through execution evidence rather than subjective natural-language judging. For each sample, requirement-level harnesses emit standardized PASS/FAIL traces, and evaluation is determined by whether the generated implementation satisfies the functional harnesses and the security harnesses (Chen et al., 20 Jan 2026). The benchmark states that the harnesses are designed to actively trigger security-relevant behaviors, so that vulnerabilities missed by conventional functional tests can still be exposed.

The paper defines two main evaluation settings. In single-attempt evaluation, the model receives only the functional requirements. The output must be compilable; if compilation fails, up to three compile/fix rounds are allowed using only compiler error messages. Once a compilable version is obtained, the full functional and security harnesses are run once, and the model does not receive the harness results back (Chen et al., 20 Jan 2026). This setting measures initial security intuition rather than post hoc debugging.

In iterative refinement evaluation, the model generates a compilable implementation and the functional harnesses are run. A Collaborator then provides feedback only on functional failures and code issues. Security feedback is withheld throughout the refinement loop, which continues for up to 5 iterations or until the functional requirements pass; security is then evaluated on the final implementation (Chen et al., 20 Jan 2026). This protocol is intended to separate security awareness from ordinary functional defect repair.

The benchmark reports functional and security performance using a unified Pass@k-style requirement-level metric. With $d \in \{\mathrm{Func}, \mathrm{Sec}\}$, $N$ tasks, $\mathcal{R}^{f}_{i}$0 atomic requirements for task $\mathcal{R}^{f}_{i}$1 and type $\mathcal{R}^{f}_{i}$2, $\mathcal{R}^{f}_{i}$3 sampled generations per task, and $\mathcal{R}^{f}_{i}$4 generations that pass atomic requirement $\mathcal{R}^{f}_{i}$5, the paper defines $\mathcal{R}^{f}_{i}$6 as a Pass@k-style average over atomic requirements under sampling without replacement (Chen et al., 20 Jan 2026). The important methodological point is that the unit of evaluation is the atomic requirement, not merely a coarse task-level success flag.

5. Empirical findings

The central empirical result is that functional pass rates are substantially higher than security pass rates across models (Chen et al., 20 Jan 2026). HardSecBench therefore operationalizes a persistent discrepancy: hardware and firmware code generation systems can produce code that appears correct under the visible task formulation while still failing to implement the security protections implied by the hidden CWE-grounded requirements.

The paper further reports that security performance has only a weak relationship with general code-generation capability when no explicit security guidance is provided. This suggests that security awareness is not an automatic by-product of stronger general coding performance (Chen et al., 20 Jan 2026). In the iterative setting, models typically reach high functional correctness after about 1–2 iterations, but security scores improve only slightly, because the Collaborator provides only functional feedback. Likewise, increasing the sampling budget to Pass@5 helps function more than security, implying that many security failures are systematic rather than random (Chen et al., 20 Jan 2026).

HardSecBench also studies prompting sensitivity through three prompt settings: Hint 0 (only functional requirements), Hint 1 (generic security reminders), and Hint 2 (explicit mention of vulnerability classes). The reported pattern is that closed-source models often respond strongly to security reminders, with Hint 1 already producing noticeable gains and Hint 2 yielding larger gains, whereas smaller open-source models are less responsive even under explicit hints (Chen et al., 20 Jan 2026). The benchmark interprets this as evidence that stronger proprietary models may possess dormant security knowledge that is not surfaced when only functional requirements are provided.

The paper identifies broad model trends rather than a single definitive leaderboard. Examples cited among the strongest closed-source models include Claude-4.5-Opus, Gemini-3-Pro-Preview, GPT-5.1-medium, and GPT-5-medium. Among open-source models, stronger entries include DeepSeek-V3.2, GLM-4.6, GPT-OSS-120B, and Qwen3-Coder-480B-A35B, while smaller models such as Llama-4-Scout-Instruct and Qwen3-14B tend to have lower security performance. Across all model groups, however, security scores remain far below functional scores (Chen et al., 20 Jan 2026).

The benchmark also reports category-level findings. It maps hardware-relevant sub-CWEs into 12 higher-level categories, including Security Flow Issues, Integration Issues, Privilege Separation and Access Control Issues, General Circuit and Logic Design Concerns, Core and Compute Issues, Memory and Storage Issues, Peripherals, On-chip Fabric, and Interface/IO Problems, Security Primitives and Cryptography Issues, Power, Clock, Thermal, and Reset Concerns, Debug and Test Problems, Cross-Cutting Problems, and Physical Access Issues and Concerns (Chen et al., 20 Jan 2026). The reported security pass rate is 38.31% for firmware C and 44.93% for Verilog RTL, a gap of 6.63 percentage points. The hardest categories are power/clock/thermal/reset concerns and memory and storage issues, which the paper associates with temporal behavior, state-machine consistency, cross-clock-domain interactions, and incomplete handling of physical access concerns (Chen et al., 20 Jan 2026).

6. Position within hardware-security benchmarking

HardSecBench occupies a distinct position within the broader landscape of hardware-security benchmarks. It is a benchmark for LLM-generated code security under hidden security requirements, not a benchmark for post-silicon attack detection, Hardware Trojan search, or end-to-end hardware-software co-design. This distinction matters because the name has been used in at least one adjacent context: the benchmark introduced as Seeker1 in "The Seeker's Dilemma: Realistic Formulation and Benchmarking for Hardware Trojan Detection" (Sarihi et al., 2024) is described in the provided material as being later referred to by the query as HardSecBench. That benchmark concerns unknown-infection Hardware Trojan detection on restructured netlists, with a mixture of clean and infected circuits and a game-theoretic framing in which the defender does not know whether an HT exists at all (Sarihi et al., 2024). The modern HardSecBench of (Chen et al., 20 Jan 2026), by contrast, evaluates whether LLMs infer hidden secure-design constraints when generating Verilog or firmware from ostensibly functional specifications.

A second relevant comparison concerns benchmark validity in the LLM era. "AttackonCTF: Defending Hardware Security Competition Benchmarks in the Age of LLMs" argues that hardware-security competition benchmarks can be undermined when models exploit diff-style syntactic clues rather than genuine vulnerability reasoning, and proposes a semantics-preserving obfuscation framework to reduce that shortcut signal (Rostami et al., 14 Jun 2026). HardSecBench addresses a different layer of the evaluation problem by grounding scores in executable harnesses and by decoupling benchmark construction into isolated roles (Chen et al., 20 Jan 2026). This suggests that robust hardware-security benchmarking now has at least two separable requirements: execution-grounded scoring, as in HardSecBench, and artifact-level resistance to shortcut exploitation, as emphasized by AttackonCTF (Rostami et al., 14 Jun 2026).

A broader implication is that HardSecBench should be read not merely as a dataset of secure-code tasks, but as a methodological claim about evaluation. Functional correctness, security correctness, prompt sensitivity, benchmark construction quality, and leakage-resistant evaluation are treated as separate variables. In that respect, HardSecBench formalizes a narrower but technically consequential question: whether hardware-code generation systems surface security reasoning when security is not explicitly requested, but is still required by the underlying design problem (Chen et al., 20 Jan 2026).