QuRE Benchmark: Dual-Domain Analysis

• QuRE Benchmark comprises two distinct artifacts: one for detecting defects in software requirements using weak-word analysis, and the other for estimating quantum computing resources via QEC modeling.
• It employs rigorous methodologies including stratified sampling, few-shot human-LLM collaboration, and precise resource scaling metrics to evaluate performance and error correction costs.
• These benchmarks deliver actionable insights for adaptive workflows in requirements engineering and quantum error correction, influencing technology adoption and optimization strategies.

The QuRE benchmark refers to two distinct research artifacts with the same acronym but serving different domains: (1) the Qualitative Requirements Engineering (QuRE) dataset for defect prediction in software requirements (Unterbusch et al., 5 Jan 2026), and (2) the Quantum Resource Estimator (QuRE) benchmarking framework for quantum error correction resource analysis (Suchara et al., 2013). Each is addressed separately below with rigorous technical detail.

1. QuRE: Qualitative Requirements Engineering Benchmark

1.1 Dataset Scope and Structure

QuRE (Qualitative Requirements Engineering) is a curated benchmark of 2,111 unique requirements extracted from Mercedes-Benz’s industrial specification corpus, each containing at least one instance of 23 pre-defined “weak words” such as “certain” or “appropriate.” Rigorous deduplication yielded a balanced dataset of 1,266 annotated instances (633 labeled “defect,” 633 “not defect”), where each instance is a (requirement, weak word) pair assigned a binary label by expert engineers. Annotation targeted whether the highlighted weak word genuinely impaired requirement clarity or was contextually benign. When requirements contained both defective and non-defective weak words, only defective instances were retained. To eliminate class imbalance and repeated contexts, majority-class examples were undersampled for a 50:50 split.

1.2 Sampling, Splitting, and Design

The 1,266 instances were partitioned into three stratified folds of 422 each. For simulating adaptive human-in-the-loop workflows, nested shot pools were constructed with sizes {20, 40, 80, 160, 320} per fold; smaller pools are subsets of larger ones. Cross-assigned validation ensures the evaluation fold contains no overlap with the in-context pool, implementing a form of stratified three-fold testing. This protocol supports rigorous assessment of incremental and few-shot learning algorithms within a realistic industrial context.

1.3 Evaluation Metrics

Binary model outputs (positive = “defect”) are evaluated using precision, recall, and unweighted F₁-score:

$\mathrm{Precision} = \frac{TP}{TP + FP}$

$\mathrm{Recall} = \frac{TP}{TP + FN}$

$$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

where TP, FP, and FN indicate true positives, false positives, and false negatives, respectively. Due to the operational risk of missed defects, recall is closely scrutinized, but F₁ is reported for balanced assessment.

1.4 Experimental Findings

The benchmark serves as the testbed for Human-LLM Collaboration (HLC), where LLMs rely on human-validated in-context examples and explanations (chain-of-thought, CoT) for rapid, adaptive classification. Experimental results establish that:

• Zero-shot GPT (12 CoT-included demonstrations) attains F₁ ≈ 0.728.
• Incorporating just 20 in-context examples with CoT (“GPT Yes CoT”) yields F₁ = 0.799, whereas label-only (“GPT No CoT”) achieves F₁ = 0.745 at the same pool size—a +0.054 absolute improvement.
• Fine-tuned BERT lags, with F₁ = 0.709 at the maximal (320-shot) pool.
• Performance gains plateau after 40‒80 examples (F₁ ≈ 0.803), confirming that most adaptive capacity is realized early.

Table: Excerpt of main comparative results ((Unterbusch et al., 5 Jan 2026), Table 2):

Pool Size	Approach	CoT?	Precision	Recall	F₁
20	GPT (few-shot)	Yes	0.679	0.972	0.799
20	GPT (few-shot)	No	0.634	0.902	0.745
80	BERT (fine-tuned)	—	0.517	0.754	0.613

1.5 Role of In-context Explanations and Rapid Adaptation

Ablation reveals two major findings:

1. Rapid few-shot adaptation: 20 validated examples suffice for substantial precision and recall gains relative to zero-shot, with diminishing returns beyond 40.
2. Validated explanations critical: CoT-augmented demonstrations prevent recall degradation (sustained at ~0.97) while boosting precision by ~4.5 points compared to label-only prompting at 20 shots.

This supports the conclusion that in-context, human-validated rationales fundamentally optimize LLM defect prediction, enabling project-specific, adaptive workflows with minimal annotation effort.

2. QuRE: Quantum Resource Estimator Framework

2.1 Methodological Roles and Cost Metrics

The Quantum Resource Estimator (QuRE) framework provides a systematic methodology for benchmarking physical resource requirements in fault-tolerant quantum computing. QuRE accepts as input a description of the physical quantum technology (gate times, physical error rates, memory times), an explicit logical circuit (e.g., Shor’s algorithm), and a quantum error correction (QEC) scheme (concatenated or topological). QuRE outputs:

• $Q$: Total number of physical qubits
• $G$: Total number of physical gate operations
• $T$: Total wall-clock runtime

These metrics are evaluated by (a) selecting QEC parameters (concatenation level $L$ or surface-code distance $d$) to bound overall failure probability, (b) determining per-logical-gate physical cost recursively, and (c) projecting to the entire fault-tolerant circuit while accounting for parallelism and QEC logic.

2.2 Error Correction Modeling

A. Concatenated Codes (Bacon–Shor):

• Effective logical error rate after $L$ levels: $\epsilon_L \approx (c p)^{2^L}$, with $\mathrm{Recall} = \frac{TP}{TP + FN}$0; $\mathrm{Recall} = \frac{TP}{TP + FN}$1.
• Required to solve $\mathrm{Recall} = \frac{TP}{TP + FN}$2, where $\mathrm{Recall} = \frac{TP}{TP + FN}$3 is total logical gates.
• Physical qubit count: $\mathrm{Recall} = \frac{TP}{TP + FN}$4, for $\mathrm{Recall} = \frac{TP}{TP + FN}$5 logical qubits.
• Total runtime and gate count computed recursively per circuit layer and QEC step.

B. Surface Codes (Topological):

• Logical error per gate: $\mathrm{Recall} = \frac{TP}{TP + FN}$6, with $\mathrm{Recall} = \frac{TP}{TP + FN}$7, $\mathrm{Recall} = \frac{TP}{TP + FN}$8, $\mathrm{Recall} = \frac{TP}{TP + FN}$9.
• Physical qubits: $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$0(#tiles)$$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$1.
• Syndrome extraction and logical operation timing formulas yield $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$2 and $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$3.

2.3 Application to Algorithmic Benchmarks

For Shor’s 1024-bit factoring algorithm (1.18×10⁹ $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$4, 3.36×10⁸ $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$5, 6,144 logical qubits), QuRE calculates end-to-end metrics across varying physical error rates. Crossover analysis reveals:

• For $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$6, surface code demands fewer resources than concatenated codes.
• For $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$7, concatenated codes outperform surface code due to fewer required correction layers.

Key findings for representative platforms:

Technology	Surface Code $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$8	Bacon–Shor $$F_{1} = 2 \,\frac{\mathrm{Precision}\,\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$9	Surface Code $Q$0	Bacon–Shor $Q$1
Supercond., $Q$2	10.8 hours	5.10 years	$Q$3	$Q$4
Ion Traps, $Q$5	2.22 years	57.98 days	$Q$6	$Q$7

2.4 Gate-Type Composition and Platform Implications

Resource allocation differs by QEC scheme. Bacon–Shor codes are dominated by $Q$8 gates (up to 60%), while surface codes use $Q$9 for syndrome extraction/braiding (>80%). The choice of QEC is thus intertwined with technology’s native gate set and error profile; for instance, superconducting qubits favor surface codes, while ion traps can exploit concatenated codes if error rates are sufficiently low.

2.5 Significance and Impact

QuRE provides both an analytic and empirical framework for quantitatively evaluating quantum algorithm implementation costs across hardware contenders and QEC strategies. The resource scaling relationships and crossover points it identifies serve as decision boundaries for quantum platform selection, circuit compilation, and error-correction design in practice.

3. Comparative Summary and Disambiguation

Both QuRE benchmarks address evaluation and optimization within their respective domains, but do so via distinct methodologies:

Domain	Dataset/Framework	Purpose	Key Metrics
Requirements Engineering	QuRE dataset	Weak-word defect classification	Precision, Recall, F₁
Quantum Computing	QuRE (Q. Res. Est)	QEC resource estimation	Physical qubits, gates, $G$0

Each QuRE instantiation underpins state-of-the-art research: one in adaptive, stakeholder-driven requirements defect detection via LLMs (Unterbusch et al., 5 Jan 2026), and the other in technology-agnostic quantum resource modeling for fault tolerance (Suchara et al., 2013).

4. Limitations and Known Constraints

• The requirements QuRE dataset uses industrial data restricted to Mercedes-Benz and focuses on weak-word smells; its defect/no-defect boundaries are subject to project- and company-specific interpretation.
• The quantum QuRE framework’s cost models rely on canonical QEC code families and do not account for all possible hardware constraints—for example, correlated noise models or emerging QEC protocols are excluded.

A plausible implication is that further adaptation or extension of both QuREs is necessary for cross-domain generalization and robust modeling in dynamic or heterogeneous industrial settings.

5. Prospects for Extension

In requirements engineering, expanding QuRE to more defect classes and open corpora—or integrating richer user feedback modalities—can validate the generality of HLC workflows. In quantum computing, incorporating additional hardware models, QEC codes, or circuit synthesis strategies can further clarify the real-world feasibility boundaries for early, large-scale quantum computers.

These QuRE benchmarks thus provide foundational, extensible testbeds for context-adaptive tool evaluation in their respective technical disciplines.