QuanBench+: Multi-Framework Quantum Code Benchmark

• QuanBench+ is a unified multi-framework benchmark for evaluating LLM-based quantum code generation, aligning tasks across Qiskit, PennyLane, and Cirq.
• It fixes the quantum task intent while varying only the framework details, using rigorous functional tests and metrics like Pass@1/5 and KL-divergence based acceptance.
• Empirical results show models improve substantially with repair feedback, revealing framework-specific performance gaps and advancing cross-framework transfer insights.

Searching arXiv for the benchmark and close comparators. QuanBench+ is a unified multi-framework benchmark for evaluating LLMs on quantum code generation across multiple software ecosystems. It spans Qiskit, PennyLane, and Cirq, and is built from 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. Its defining methodological choice is to keep task intent fixed while changing only the target framework, thereby distinguishing portable quantum reasoning from framework familiarity. The benchmark evaluates generated programs with executable functional tests, reports Pass@1 and Pass@5, uses a KL-divergence-based acceptance rule for probabilistic outputs, and additionally measures Pass@1 after feedback-based repair. The benchmark was introduced in “QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation” (Slim et al., 25 Mar 2026).

1. Definition and benchmark scope

QuanBench+ is defined as a unified multi-framework benchmark spanning Qiskit, PennyLane, and Cirq, constructed from 42 aligned tasks that cover quantum algorithms, gate decomposition, and state preparation (Slim et al., 25 Mar 2026). Its central design principle is that the same underlying quantum-programming objective is posed across multiple frameworks, with only framework-specific details changed. This arrangement is intended to test whether a model succeeds because it understands the quantum task itself or because it is familiar with a particular API, abstraction, or framework idiom.

The benchmark addresses a limitation in prior quantum code-generation evaluations. Existing benchmarks such as Qiskit HumanEval, QHackBench, QCircuitBench, and QuanBench are described as mostly single-framework evaluations. That makes them useful for measuring practical coding ability in one stack, but less suitable for separating conceptual quantum reasoning errors from framework-use errors. QuanBench+ is therefore positioned as a benchmark of cross-framework transfer rather than only within-framework proficiency (Slim et al., 25 Mar 2026).

Two neighboring works are especially relevant for context. QuanBench, introduced earlier, is a Qiskit-based benchmark with 44 tasks spanning quantum algorithms, state preparation, gate decomposition, and quantum machine learning; it evaluates both functional correctness and process fidelity, but it does not define a multi-framework aligned benchmark under the name QuanBench+ (Guo et al., 19 Oct 2025). VeriQBench, by contrast, is a verification-oriented benchmark for quantum circuits that organizes circuits into combinational, dynamic, sequential, and variational categories; it is complementary in scope rather than directly comparable, because its emphasis is verification and design automation rather than code generation (Chen et al., 2022).

2. Task curation and aligned multi-framework design

QuanBench+ is derived from the original QuanBench task set. The authors retained only those tasks that could be graded with a clear numerical or functional correctness criterion across all three frameworks, and then adapted them into framework-specific prompts while preserving the same objective. Two tasks from the original benchmark were removed because they did not support reliable cross-framework grading, leaving a final benchmark of 42 tasks (Slim et al., 25 Mar 2026).

The task distribution is explicitly reported as follows.

Category	Tasks
Quantum Algorithms	31
State Preparation	6
Decomposition	5

This distribution indicates that quantum algorithm tasks dominate the benchmark. The paper explicitly notes this imbalance as a limitation, since it can amplify the influence of multi-step algorithmic reasoning on aggregate scores (Slim et al., 25 Mar 2026).

The benchmark’s curation emphasizes grading reliability. Specific task edits are reported in the benchmark construction:

Task	Modification
Task 5	removed because it did not output a clear probabilistic answer
Task 25	prompt modified to measure the first 3 qubits instead of all qubits
Task 28	prompt modified because the old version did not clearly instruct measuring all qubits
Task 38	removed because the testing procedure for the machine learning task was not clearly specified
Task 41	prompt modified so a randomized input library was replaced by a pre-decided randomly generated input

These changes show that QuanBench+ is not merely a direct port of QuanBench, but a filtered and normalized subset built around cross-framework executable evaluation. This also implies a narrower scope than QuanBench, which includes 44 tasks in its main experimental version and a later “current stage” expansion to 117 tasks, including quantum machine learning tasks (Guo et al., 19 Oct 2025).

The prompt normalization procedure is also explicit. Each model receives the same prompt per task and framework; prompts contain strict instructions for code-only output; prompts specify the expected function interface; framework-specific imports are inserted where needed; for tasks requiring inputs, a random set of non-trivial inputs is generated once and reused across all models and frameworks; and each canonical solution’s output is standardized to a probability array over computational basis states (Slim et al., 25 Mar 2026). This standardized output representation is central because the three frameworks differ in circuit representation, measurement APIs, simulator interfaces, and return formats.

3. Evaluation methodology

QuanBench+ follows the functional-correctness paradigm used in HumanEval-style code evaluation, but adapts it to quantum programs whose outputs may be deterministic or probabilistic. A generated program is correct if it executes and satisfies the task-specific criterion under a fixed harness (Slim et al., 25 Mar 2026).

The primary metrics are Pass@1 and Pass@5. Pass@k is defined as the probability that at least one of the top-$k$ generated solutions is correct:

$$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$

Here $n$ is the number of generated samples and $c$ is the number of correct samples. In the reported setup, Pass@1 corresponds to one generated completion per task under greedy decoding, while Pass@5 evaluates five sampled completions per task (Slim et al., 25 Mar 2026).

Because many quantum tasks produce probabilistic measurement outcomes, QuanBench+ uses a KL-divergence-based acceptance rule rather than fidelity. The KL divergence between canonical distribution $P$ and generated distribution $Q$ is given as

$$D_{\mathrm{KL}}(P \Vert Q) = \sum_x P(x)\,\log\frac{P(x)}{Q(x)}.$$

To avoid undefined values when $Q(x)=0$ for some $x$ with $P(x)>0$, the benchmark applies a small additive smoothing constant $$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$0 to both distributions before renormalization; the exact value of $$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$1 is not specified in the paper excerpt. A generated solution is accepted if

$$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$2

This threshold is calibrated rather than chosen ad hoc. For each probabilistic task $$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$3, the canonical circuit is executed $$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$4 times, producing empirical distributions $$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$5. The reference distribution is the normalized mean

$$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$6

The benchmark then computes null KL values

$$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$7

and selects a global threshold from a high quantile of the pooled values across tasks:

$$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$8

With $$\mathrm{Pass@}k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}.$$9, the pooled threshold is reported as $n$0, and the benchmark uses the slightly more permissive constant $n$1 (Slim et al., 25 Mar 2026).

The paper explicitly rejects process fidelity as the main correctness metric, even though the earlier QuanBench benchmark reported it. The process fidelity formula quoted for comparison is

$n$2

QuanBench+ argues that many circuits can be functionally equivalent for a prompt’s measurement objective while differing substantially at the unitary level, so executable functional tests and output distributions are more appropriate for prompt-level evaluation (Slim et al., 25 Mar 2026). This marks a clear methodological departure from QuanBench, where process fidelity was one of the two primary evaluation metrics alongside Pass@K (Guo et al., 19 Oct 2025).

4. Feedback-based repair and error taxonomy

Beyond one-shot generation, QuanBench+ studies a repair setting reported as Pass@1 (FB). The repair loop proceeds by generating an initial completion, executing it under the same harness, and then returning either the exception trace or the wrong function output to the model so that it can revise the code. The benchmark triggers feedback both on runtime exceptions and on wrong-answer outputs, and allows up to five repair chances per task (Slim et al., 25 Mar 2026).

This design broadens the benchmark from static code generation to a constrained debugging setting. The paper reports that performance increases monotonically with additional feedback attempts, that most gains occur early, especially from attempt 1 to 2, and that returns diminish after about three attempts. It also notes framework-specific dynamics: Qiskit saturates earlier for stronger models, whereas PennyLane and Cirq often improve more gradually through attempts 4–5 (Slim et al., 25 Mar 2026). This suggests that the benchmark measures not only latent coding ability but also the extent to which models can exploit execution feedback.

The benchmark also provides an aggregate error-type analysis before and after repair. Before repair, first-attempt failures are distributed as follows:

Error type	Share before repair
wrong answers	46.7%
logic errors	25.0%
missing methods/gates	11.8%
shape mismatches	8.0%
syntax errors	4.7%
qubit specification errors	3.9%

After repair, the total number of wrong tasks falls from 977 to 665, and residual failures become even more concentrated in semantic issues:

Error type	Share after repair
wrong answers	53.4%
logic errors	22.0%
shape mismatches	12.8%
missing methods/gates	3.8%
syntax errors	1.5%

The paper interprets this as evidence that feedback is effective for fixing visible coding and interface problems, but much less effective for repairing deeper reasoning errors (Slim et al., 25 Mar 2026). A plausible implication is that benchmark performance after repair still depends heavily on the model’s internal representation of the quantum task, not merely on runtime debugging competence.

5. Experimental setup and empirical performance

QuanBench+ evaluates 12 models, including proprietary frontier systems and open or open-weight models. The appendix list includes Claude-3.7-Sonnet, DeepSeek-Chat, DeepSeek-R1, Gemini-2.5-Flash, Gemini-3-Pro, GPT-4.1, GPT-5.1, Llama-4-Maverick, Qwen-2.5-7B-Instruct, MiniMax-M2.1, Z-ai-GLM-4.7, and MoonshotAI-Kimi-K2-Thinking (Slim et al., 25 Mar 2026).

The execution environment is fixed and versioned:

Component	Version
Python	3.10
Qiskit	v0.46.0
Cirq	v1.6.1
PennyLane	v0.43.1

Decoding is standardized: Pass@1 uses greedy decoding with temperature $n$3, while Pass@5 uses $n$4 completions sampled at temperature $n$5 (Slim et al., 25 Mar 2026). The paper also analyzes prefill versus no-prefill conditions. In the appendix, prefill means prompts include required imports, function signature, and minimal boilerplate; no-prefill requires the model to generate the full solution from scratch. The reported conclusion is that prefill mainly reduces setup friction rather than solving semantic reasoning problems.

The headline framework-specific one-shot results are:

Framework	Best Pass@1	Best Pass@1 (FB)
Qiskit	59.5%	83.3%
Cirq	54.8%	76.2%
PennyLane	42.9%	66.7%

These figures establish three central empirical claims. First, strong progress is evident: top models solve a substantial fraction of tasks, especially after repair. Second, reliable multi-framework quantum code generation remains unsolved. Third, there is a stable framework ordering in difficulty: Qiskit is easiest, PennyLane is hardest, and Cirq usually lies between them (Slim et al., 25 Mar 2026).

The paper further reports that there is no universally dominant model across all frameworks. Gemini-3-Pro leads the average one-shot ranking and is strongest on Qiskit and Cirq, while GPT-5.1 achieves the best one-shot score on PennyLane (Slim et al., 25 Mar 2026). This supports the benchmark’s central claim that performance is still significantly driven by framework-specific familiarity.

Pass@5 results illustrate the gap between latent capability and one-shot reliability. The paper reports, for example, that GPT-5.1 reaches 76.2 on Qiskit, 64.3 on Cirq, and 57.1 on PennyLane; DeepSeek-R1 reaches 69.0, 61.9, and 59.5; and Gemini-3-Pro reaches 61.9, 59.5, and 40.5 (Slim et al., 25 Mar 2026). This suggests that correct solutions are often present among a small set of samples even when greedy decoding misses them. A plausible implication is that sampling can reveal latent competence, but not eliminate the framework gap.

6. Relation to neighboring benchmarks and conceptual significance

QuanBench+ is best understood relative to three distinct benchmark lineages. The first is classical executable code-generation evaluation, especially HumanEval-style functional correctness. QuanBench+ adopts that paradigm, but extends it to stochastic quantum outputs through the KL-divergence criterion (Slim et al., 25 Mar 2026).

The second lineage is quantum code-generation benchmarking. In this family, QuanBench is the closest predecessor. QuanBench defines a Qiskit-based benchmark with 44 programming tasks covering quantum algorithms, state preparation, gate decomposition, and quantum machine learning, and evaluates models using both Pass@K and process fidelity (Guo et al., 19 Oct 2025). QuanBench+ removes two tasks from the original set, retains only those that support reliable cross-framework functional grading, and generalizes them across Qiskit, PennyLane, and Cirq (Slim et al., 25 Mar 2026). The relationship is therefore one of selective expansion and methodological reorientation: the “plus” primarily denotes unified multi-framework alignment rather than a larger task set.

The third lineage is broader quantum benchmark design. Benchmarks such as VeriQBench focus on verification-oriented circuit diversity rather than code generation (Chen et al., 2022). Application-level benchmarking efforts such as SupermarQ emphasize workload diversity and hardware-agnostic feature coverage at the system level (Tomesh et al., 2022), while Benchpress targets the classical circuit-processing layer of quantum SDKs rather than LLM-based synthesis (Nation et al., 2024). QuanBench+ differs from both by evaluating model-generated source code through framework-aligned prompts and executable correctness criteria.

The benchmark’s main conceptual contribution is therefore not simply more tasks or more models, but a controlled experimental design that holds quantum task intent fixed while varying framework. This permits a sharper interpretation of failure modes. If a model performs strongly on one framework and weakly on another for the same aligned task, then the difference plausibly reflects framework dependence rather than purely quantum-theoretic misunderstanding (Slim et al., 25 Mar 2026).

7. Limitations and prospective extensions

The paper identifies several limitations. First, the benchmark contains only 42 tasks, so it does not cover the full range of real-world quantum software development (Slim et al., 25 Mar 2026). Second, the category mix is imbalanced toward quantum algorithms, which may disproportionately weight multi-step reasoning. Third, only three metrics are reported in this version: Pass@1, Pass@5, and Pass@1 (FB). The authors note that other perspectives, such as robustness to prompt variation, longer repair horizons, and tool-augmented workflows, would be useful extensions (Slim et al., 25 Mar 2026).

Framework coverage is also limited to Qiskit, PennyLane, and Cirq. The paper explicitly leaves extension to additional frameworks as future work (Slim et al., 25 Mar 2026). This is noteworthy in light of the earlier QuanBench paper, which already identified future expansion to Cirq and PennyLane as a natural next step for a Qiskit-based benchmark (Guo et al., 19 Oct 2025). QuanBench+ realizes that expansion, but does not yet generalize beyond those three ecosystems.

The evaluation also depends on the correctness of canonical solutions and faithful cross-framework adaptation. The authors acknowledge that subtle mismatches could arise even when prompts are intended to be equivalent, though they mitigated this by excluding ambiguous tasks and reviewing canonical implementations for functional equivalence (Slim et al., 25 Mar 2026). They also identify framework versioning as a threat to validity: semantically reasonable code can fail because APIs have changed. The benchmark addresses this only by fixing environment versions, not by solving the version-drift problem.

Finally, contamination analysis is not reported in depth. The paper does not present a train/test split or formal decontamination audit (Slim et al., 25 Mar 2026). This suggests that benchmark conclusions should be interpreted as execution-grounded capability measurements under fixed prompts and environments, rather than as contamination-controlled estimates of abstract generalization.

Taken together, these limitations suggest that QuanBench+ is best viewed as a benchmark of portable framework-conditioned quantum programming ability under executable evaluation, rather than a complete benchmark ecosystem for quantum software generation. A plausible future direction is a broader benchmark family in which aligned multi-framework code generation, tool use, repair, and perhaps verification-oriented tasks coexist as separate but interoperable layers.