SPICE Bench: Benchmark Suites in EDA & SWE

• SPICE Bench is a term used for multiple benchmark suites in circuit simulation and software engineering, emphasizing automated SPICE-driven workflows.
• It covers diverse applications from memristor-based logic-in-memory and LTSpice SMPS tasks to analog schematic-to-netlist generation and SWE repair task labeling.
• The benchmarks use rigorous quantitative metrics and closed-loop validation protocols to ensure reproducible and actionable insights in both EDA and SWE research.

Searching arXiv for papers that use or define “SPICE Bench” and closely related benchmark frameworks. Tool call: arxiv_search(query="^^^^^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^^^^^", max_results=^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^, sort_by="^^^^^^^^3relevance3 Searching arXiv for exact and near-exact matches to “SPICE Bench”. Tool call: arxiv_search(query="^^^^^^^^3\3 Bench^^^^^^^^3\3 max_results=^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^, sort_by="^^^^^^^^3relevance3 In recent arXiv usage, SPICE Bench does not denote a single standardized artifact. The name is applied to several distinct benchmark suites and SPICE-centered evaluation workflows across electronic design automation and software engineering. In circuit-design work, it refers to a SPICE-based benchmarking flow for memristor-based logic-in-memory, a ^^^^3\3^^^^56-question LTSpice benchmark for adapting switched-mode power-supply netlists, and an open schematic-to-netlist suite for analog-circuit generation and verification. In software engineering, it denotes a corpus of 6 8^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3\3^^^^ labeled SWE-bench-like repair tasks curated from ^^^^3\3^^^^9^^^^3relevance3^^^^ open-source projects in SWE-Gym (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&, &&&^^^^3relevance3^^^^&&&, &&&^^^^3\3^^^^&&&, Bhatia et al., 12 Jul 2025).

^^^^3relevance3^^^^. Nomenclature and domain scope

The shared label reflects a common emphasis on benchmarked interaction with SPICE artifacts or SPICE-style evaluation loops, but the concrete object being benchmarked varies substantially by paper. In some cases the benchmark target is a circuit or workload executed in a simulator; in others it is an LLM’s ability to generate or modify netlists; in software engineering it is a labeled dataset whose name is unrelated to electronic simulation.

Usage in the literature	Benchmark object	Reported scale
MemSPICE-style “SPICE Bench”	memristor-based LiM evaluation flow	six ISCAS’85 circuits in a single 5^^^^3relevance3\3^^^^-device row
SPICEAssistant “SPICE Bench”	LTSpice-based SMPS adaptation tasks	^^^^3\3^^^^56 questions
Masala-CHAI “SPICE Bench”	analog schematic-to-netlist generation and verification	≈^^^^3\3^^^^,^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ schematics automatically converted into SPICE netlists
SPICE Bench in software engineering	SWE-bench-like repair tasks with automated labels	6 8^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3\3^^^^ instances from ^^^^3\3^^^^9^^^^3relevance3 projects

A recurring source of confusion is that the same label spans at least two unrelated research lineages: SPICE as circuit simulation infrastructure and SPICE as a software-engineering labeling pipeline. This suggests that the phrase functions primarily as a project-specific benchmark name rather than as a field-wide standard.

^^^^3\3^^^^. Test-bench antecedents in circuit simulation

Before the appearance of benchmark suites explicitly named SPICE Bench, SPICE-centered research often used the term test bench to denote a validated simulation environment. A representative example is the LTspice and SimScape power-cycling test bench for accelerated life testing of the ^^^^3relevance3^^^^.^^^^3\3^^^^ kV SiC-MOSFET Cree CAS3^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^M^^^^3relevance3\3^^^^BM in harsh offshore environments (Geraei et al., 2020).

That bench comprises a DC voltage source with PRESERVED_PLACEHOLDER_^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^, a device-under-test subcircuit for the CAS3^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^M^^^^3relevance3\3^^^^BM a series load resistor, gate-drive logic, a thermal network, and measurement and data capture for PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^, PRESERVED_PLACEHOLDER_^^^^3\3^^^^, and $T_j(t)$. The subcircuit uses gradual-channel MOSFET equations with temperature dependence, voltage- and temperature-dependent capacitances $C_{gd}$, $C_{gs}$, and $C_{ds}$, and a thermal network with $R_{\theta JC}=0.15\ \mathrm{K/W}$, $R_{\theta CA}=2.5\ \mathrm{K/W}$, and $C_{\mathrm{th,jc}} \approx 1\ \mathrm{J/K}$. Power-cycling profiles use PRESERVED_PLACEHOLDER_^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^, PRESERVED_PLACEHOLDER_^^^^3relevance3relevance3^^^^, ambient temperatures from PRESERVED_PLACEHOLDER_^^^^3relevance3\3^^^^ to PRESERVED_PLACEHOLDER_^^^^3relevance33^^^^, and failure thresholds such as a ^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% increase in PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^4 (Geraei et al., 2020).

The same work formalizes degradation and lifetime extraction through a power-law drift model,

PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^5

and a Coffin–Manson type relation,

PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^6

with remaining useful life estimated as PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^7. Validation is reported against the datasheet: static PRESERVED_PLACEHOLDER_^^^^3relevance3^^^^8-PRESERVED_PLACEHOLDER_^^^^3relevance3 curves show maximum error below 5%, PRESERVED_PLACEHOLDER_^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ versus temperature matches within PRESERVED_PLACEHOLDER_^^^^3\3relevance3^^^^, and thermal impedance matches Cree’s PRESERVED_PLACEHOLDER_^^^^3\3\3^^^^ specification within ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^%; early hardware data show PRESERVED_PLACEHOLDER_^^^^3\33^^^^ drift trends within ^^^^3relevance3^^^^5% over PRESERVED_PLACEHOLDER_^^^^3\3^^^^4 cycles (Geraei et al., 2020).

This antecedent is significant because it locates the later notion of SPICE Bench within a broader tradition of simulation-grounded, measurement-oriented, and validation-heavy circuit research.

3. MemSPICE and SPICE-level benchmarking for logic-in-memory

In "MemSPICE: Automated Simulation and Energy Estimation Framework for MAGIC-Based Logic-in-Memory" (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&), the term “SPICE Bench” is used for a stepwise benchmarking flow that starts from HDL and ends with per-benchmark energy numbers for memristor-based logic-in-memory on a crossbar.

The flow begins with an automated HDL→SPICE-netlist pipeline. Verilog or VHDL RTL is synthesized with ABC into ^^^^3\3^^^^-input NOR and NOT primitives, then converted by SIMPLER into a JSON mapping that records row size, inputs, outputs, and an ordered execution sequence of INIT, NOR, and NOT steps. A JSON→SPICE translator emits a single-row crossbar subcircuit with memristor instances PRESERVED_PLACEHOLDER_^^^^3\3^^^^5, relay-controlled row and column access, PWL sources for operation pulses, and a transient analysis with .save of device currents (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&).

Automatic testbench generation derives one PWL waveform file per net. INIT drives the target column to PRESERVED_PLACEHOLDER_^^^^3\3^^^^6, exemplified as PRESERVED_PLACEHOLDER_^^^^3\3^^^^7, for PRESERVED_PLACEHOLDER_^^^^3\3^^^^8; NOR and NOT operations apply PRESERVED_PLACEHOLDER_^^^^3\3^^^^9 on input nets and $T_j(t)$^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ on the output net, while selected relays are asserted at $T_j(t)$^^^^3relevance3^^^^. Simulations use the VTEAM memristor model, relay elements with $T_j(t)$^^^^3\3^^^^ and $T_j(t)$3, .tran for transient execution, and optional .dc sweeps for read-level profiling (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&).

Energy estimation is defined directly at the SPICE waveform level. For device $T_j(t)$4,

$T_j(t)$5

and the total energy is

$T_j(t)$6

MemSPICE supports numerical integration through Python or Matlab and also auto-generates a Spectre-compatible .ocn file so that Spectre can print per-device integrals (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&).

The benchmark results reported for six ISCAS’85 circuits, all mapped into a single 5^^^^3relevance3\3^^^^-device row, are: c^^^^3relevance3^^^^7 at $T_j(t)$7, c43^^^^3\3^^^^ at $T_j(t)$8, c499 at $T_j(t)$9, c88^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ at $C_{gd}$^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^, c^^^^3relevance3^^^^9^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^8 at $C_{gd}$^^^^3relevance3^^^^, and c354^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ at $C_{gd}$^^^^3\3^^^^, with total SPICE wall-clock time per benchmark of ^^^^3\3^^^^–^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ minutes on a modern workstation. The paper further notes that average energy per logic operation is typically ^^^^3\3^^^^–3 pJ/op for medium-size designs, although re-initialization overhead can dominate, and that input-pattern variation is $C_{gd}$3–^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&).

Within the EDA literature, this version of SPICE Bench is best understood as a reproducible SPICE workflow rather than merely a dataset.

4. SPICEAssistant and the ^^^^3\3^^^^56-question LTSpice benchmark

"SPICEAssistant: LLM using SPICE Simulation Tools for Schematic Design of Switched-Mode Power Supplies" defines SPICE Bench as a benchmark of ^^^^3\3^^^^56 questions testing an LLM’s ability to adapt circuit netlists to fulfill switched-mode power-supply design tasks (&&&^^^^3relevance3^^^^&&&).

The benchmark spans three topology tiers: Easy with 7^^^^3\3^^^^ cases based on an idealized general buck converter; Medium with 7^^^^3\3^^^^ cases using a buck converter with the LTC34^^^^3relevance3^^^^9 dual-step-down IC; and Hard with ^^^^3relevance3relevance3\3^^^^ cases using a ^^^^3\3^^^^-phase synchronous buck with LTC78^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3\3^^^^ controller. Task categories comprise 3^^^^3\3^^^^ topology-adaptation questions and ^^^^3\3\3^^^^4 parameter-tuning questions. The rationale is to mirror a common SMPS workflow in which one starts from a typical application circuit in a datasheet and then either changes component values or slightly rewires pins to achieve new specifications (&&&^^^^3relevance3^^^^&&&).

All simulations are performed in LTSpice with a transient directive $C_{gs}$9 and, where needed, .AC DEC ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ Hz ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ MHz and .DC Iload ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ ^^^^3\3^^^^ A ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^.^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3relevance3^^^^ A. .OPTIONS POST=^^^^3\3^^^^ NOMOD controls data export. Performance metrics are extracted through .MEAS commands such as average output voltage, peak-to-peak ripple, switching frequency, and settling time to 9^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% of final value. The framework’s Python reading tools invoke these measurements so that the LLM does not parse raw waveforms or plotted images (&&&^^^^3relevance3^^^^&&&).

Evaluation is organized as an LLM↔Simulator loop. The LLM proposes a netlist change; the framework writes the modified netlist, calls LTSpice, extracts requested metrics, and returns numeric feedback; the process repeats for up to $C_{gd}$4 iterations, after which solve rate plateaus. Scoring uses Solve Rate (SR), the Absolute Percentage Error

$C_{gd}$5

and $C_{gd}$6 over non-topology tasks (&&&^^^^3relevance3^^^^&&&).

The reported results show a large separation between a standalone LLM and simulator-assisted operation. On the full ^^^^3\3^^^^56-question benchmark, GPT-4o attains ^^^^3relevance3^^^^4.8% SR and 64.3% APE, whereas SPICEAssistant attains 53.^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% SR and 4.^^^^3\3^^^^% APE. By difficulty level, the Easy suite improves from ^^^^3\3^^^^9.^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% ± ^^^^3relevance3^^^^.^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% to 64.9% ± ^^^^3relevance3^^^^.8% SR; the Medium suite from ^^^^3relevance3\3^^^^.5% to 5^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^.^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^%; and the Hard suite from 7.^^^^3relevance3^^^^% to 47.3%. The paper reports an absolute solve-rate gain of 38.^^^^3\3^^^^ percentage points, a drop of median APE from 64% to 4%, and only marginal improvement from adding RAG alone (&&&^^^^3relevance3^^^^&&&).

This SPICE Bench instantiation is therefore a closed-loop LLM evaluation benchmark defined by repeated simulation, automatic measurement extraction, and tolerance-based success criteria.

5. Analog-circuit netlist generation: Masala-CHAI and related SPICEPilot metrics

In "Masala-CHAI: A Large-Scale SPICE Netlist Dataset for Analog Circuits by Harnessing AI," SPICE Bench denotes a benchmark suite for analog-circuit design and verification built from automatically generated SPICE netlists and their associated schematics (&&&^^^^3\3^^^^&&&).

The benchmark currently comprises ≈^^^^3\3^^^^,^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ analog-circuit schematics automatically converted into SPICE netlists, with an open-source framework that users can grow to ≈7,5^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ or more. Covered topologies range from single-stage RC filters and basic common-source amplifiers with ^^^^3\3^^^^–5 devices and ≤^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ nets to two-stage op-amps, differential amplifiers, LC oscillators, and bandgap references with ^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^–4^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^+ devices and ^^^^3relevance3^^^^5–3^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ nets. Device naming conventions follow standard SPICE forms such as R#, C#, L#, M#, Q#, V#, and I#; node numbering uses integer nets with ground at node ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^; values use unit-suffixed syntax such as ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^k, ^^^^3relevance3^^^^p, and ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^n; and MOSFET instance lines specify W= and L= parameters (&&&^^^^3\3^^^^&&&).

Its three-step workflow consists of labeling analog circuits, prompt tuning for LLMs, and SPICE netlist verification. Labeling uses a YOLOv8 object detector trained on ∼4,3^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ textbook images across ^^^^3relevance3\3^^^^ classes and a Deep Hough Transform plus heuristic clustering with endpoint radius $C_{gd}$7 px for net detection. Prompt tuning addresses component differentiation and net-annotated SPICE generation. Verification checks for floating nets, missing multi-terminal connections, and basic simulation sanity in NGSpice/XYCE, re-prompting the LLM until all checks pass (&&&^^^^3\3^^^^&&&).

Benchmark correctness is formalized through a graph-based netlist metric. For netlist graphs $C_{gd}$8 and $C_{gd}$9, similarity is

$C_{gs}$^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^

where

$C_{gs}$^^^^3relevance3^^^^

On ^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ benchmark designs with ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ samples each, fine-tuning improves correctness from 8/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ to ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ on a common-source amplifier for both GPT-3.5-turbo and GPT-4o-mini, from 6/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ to ^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ for GPT-4o-mini on a single-stage RC low-pass filter, and from ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ to 7/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ for GPT-4o-mini on a ^^^^3\3^^^^-stage op-amp with compensation; the bandgap reference remains at ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^/^^^^3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ (&&&^^^^3\3^^^^&&&).

A closely related but separately named benchmark framework is SPICEPilot, which standardizes evaluation for LLM-based SPICE code generation through the metrics Syntax Correctness Rate (SCR), Netlist Functional Coverage (NFC), Simulation Fidelity (SF), and Pass@k (&&&^^^^3\3relevance3^^^^&&&). SPICEPilot’s dataset contains 6^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ unique circuits across digital and analog categories, with 5^^^^3relevance3^^^^ transistor-based “easy” to “extreme” tasks, transistor counts from 4 up to 88, supply voltage fixed at ^^^^3relevance3^^^^.^^^^3\3 V, and temperature at ^^^^3\3^^^^7 °C. It reports, for example, Pass@^^^^3relevance3^^^^ = 79.8% and Pass@5 = 85.5% for SPICEPilot+Claude on the ^^^^3\3^^^^4-circuit AnalogCoder benchmark, and average Pass@^^^^3relevance3^^^^ = 8^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^% and Pass@3 = 88% for SPICEPilot+GPT-4o on a ^^^^3\3^^^^5-circuit mixed benchmark (&&&^^^^3\3relevance3^^^^&&&).

Taken together, Masala-CHAI and SPICEPilot establish a methodological axis in which SPICE Bench denotes either a curated benchmark corpus or a metric-standardized benchmarking framework for schematic interpretation and netlist synthesis.

6. SPICE Bench in software engineering

A separate usage appears in "SPICE: An Automated SWE-Bench Labeling Pipeline for Issue Clarity, Test Coverage, and Effort Estimation," where SPICE Bench is a dataset of 6 8^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3\3^^^^ labeled “SWE-bench-like” repair tasks drawn from ^^^^3\3^^^^9^^^^3relevance3 real-world open-source projects in SWE-Gym (Bhatia et al., 12 Jul 2025).

Each instance includes the GitHub issue title and body, the golden patch, the test patch, and three automated labels with rationales: Issue Clarity, Test Coverage, and Effort Estimation. The pipeline comprises Context-Aware Code Navigation (via Aider), Rationale-Driven Prompting, and Multi-Pass Consensus (Self-Consistency). Context is restricted to files touched by the golden patch and test patch, with RepoMap using ctags, Tree-sitter, and graph ranking; prompting is derived from ^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ fully-agreed SWE-V instances; and consensus fuses three stochastic runs by majority vote or, if no majority exists, by the median ordinal label (Bhatia et al., 12 Jul 2025).

The paper formalizes binary label discretization from human ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^–3 scores, reports inter-rater agreement with Krippendorff’s $C_{gs}$^^^^3\3^^^^,

$C_{gs}$3

and gives a cost model

$C_{gs}$4

Using $C_{gs}$5, ^^^^^^^^3\3 Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ per ^^^^3relevance3^^^^ ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ instances** (Bhatia et al., 12 Jul 2025).

Relative to SWE-bench Verified, which contains 5^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ verified instances after ^^^^3relevance3^^^^ 699 initially labeled examples, SPICE Bench is approximately ^^^^3relevance33^^^^.6× larger. On a ^^^^3relevance3relevance3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^-instance SWE-V test set, reported accuracies include 87.3% ICA for GPT-4o-mini, 68.^^^^3\3^^^^% TCA for GPT-4o, and 68.5% TCA for DeepSeek-Reasoner. On a 48-instance manual agreement-based evaluation, ICA label correctness is 93.5% and TCA label correctness is 6^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^%. Median per-^^^^3relevance3^^^^ ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^-instance API cost is reported as ^^^^^^^^3\3 for DeepSeek-Reasoner, ^^^^^^^^3\3 for GPT-4o, with the default combined configuration at approximately ^^^^^^^^3\3 Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ per ^^^^3relevance3^^^^ ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ instances; total time for ^^^^3relevance3^^^^ ^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^ instances is ≈^^^^3\3^^^^5 h versus ^^^^3relevance3^^^^ 333 h manual, a ×53 speedup (Bhatia et al., 12 Jul 2025).

This use of the term is conceptually distinct from circuit-simulation SPICE Bench. Here, the benchmark object is a large labeled corpus for SE-focused foundation models, not a simulator-driven electronic-design task.

7. Methodological commonalities and terminological boundaries

Across these otherwise heterogeneous uses, several common patterns recur. First, SPICE Bench almost always involves automation of a translation layer: HDL to SPICE netlists in MemSPICE, natural-language design requests to netlist edits in SPICEAssistant, schematic images to SPICE netlists in Masala-CHAI, and repository state to task labels in the software-engineering SPICE pipeline. Second, the evaluation loop is typically programmatic and tool-mediated, whether through .tran runs and current integration, .MEAS extraction, graph-edit comparison, or consensus fusion over repeated model calls. Third, success is operationalized through explicit quantitative metrics rather than informal inspection (&&&^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^&&&, &&&^^^^3relevance3^^^^&&&, &&&^^^^3\3^^^^&&&, Bhatia et al., 12 Jul 2025).

A plausible implication is that the phrase SPICE Bench now denotes a family of benchmark designs centered on closed-loop validation against executable artifacts. In circuit work, the executable artifact is the SPICE simulation itself; in software engineering, it is a structured instance enriched by automated labels and rationales. This convergence explains why the same name can recur across otherwise unrelated subfields.

The term should also be distinguished from benchmark studies of the Solar Orbiter SPICE instrument. Brooks et al. benchmarked Solar Orbiter/SPICE against Hinode/EIS for plasma-composition measurements in active region AR ^^^^3relevance3\3^^^^78^^^^3relevance3 using spectral-line diagnostics, DEM inversion with PintOfAle, density-sensitive ratios such as Fe XIII ^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3\3^^^^.^^^^3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE3^^^^44/^^^^3\3SPICE Bench benchmark SPICEAssistant Masala-CHAI MemSPICE33^^^^.8^^^^3\3^^^^6 and Mg VIII 77^^^^3\3^^^^.3^^^^3relevance3, and Mg/Ne abundance assessments in which the coronal case reproduced 85–95% of lines within 35% in one region and 85% in another (&&&3^^^^3relevance3^^^^&&&). That study is a benchmark of SPICE as an EUV spectrometer, not a benchmark suite named SPICE Bench.

The current literature therefore supports a precise but non-singular definition: SPICE Bench is a reused benchmark label spanning multiple research programs, most prominently simulator-grounded EDA evaluation and automated dataset construction for software-engineering foundation models.