QuantumQA: Enhancing Scientific Reasoning via Physics-Consistent Dataset and Verification-Aware Reinforcement Learning

Abstract: LLMs show strong capabilities in general reasoning but typically lack reliability in scientific domains like quantum mechanics, which demand strict adherence to physical constraints. This limitation arises from the scarcity of verifiable training resources and the inadequacy of coarse feedback signals in standard alignment paradigms. To address the data challenge, we introduce QuantumQA, a large-scale dataset constructed via a task-adaptive strategy and a hybrid verification protocol that combines deterministic solvers with semantic auditing to guarantee scientific rigor. Building on this foundation, we propose the verification-aware reward model (VRM) tailored for Reinforcement Learning with Verifiable Rewards (RLVR), which employs an adaptive reward fusion (ARF) mechanism to dynamically integrate deterministic signals from a scientific execution suite (SES) with multidimensional semantic evaluations for precise supervision. Experimental results demonstrate that our method consistently outperforms baselines and general-purpose preference models. Notably, our optimized 8B model achieves performance competitive with proprietary models, validating that incorporating verifiable, rule-based feedback into the reinforcement learning loop offers a parameter-efficient alternative to pure scaling.

• The paper introduces a scalable dataset and a novel verification-aware reward model that enhances LLM reasoning in quantum mechanics.
• It details a hybrid dataset construction approach integrating deterministic symbolic solvers, semantic auditing, and human-in-the-loop review to ensure physical consistency.
• Experimental results show significant accuracy improvements and reduced reasoning errors compared to standard fine-tuning methods.

QuantumQA: Physics-Consistent Scientific Reasoning and Verification-Aware Reinforcement Learning

Introduction and Motivation

The development and evaluation of LLMs for high-stakes scientific reasoning tasks, particularly in quantum mechanics, confront two primary bottlenecks: the absence of large-scale, process-supervised and physics-consistent datasets, and the inadequacy of existing alignment techniques that do not robustly enforce physical constraints. Standard preference-based reward modeling, including RLHF, often rewards plausible—but physically spurious—outputs, yielding reward over-optimization and persistent hallucination. The "QuantumQA: Enhancing Scientific Reasoning via Physics-Consistent Dataset and Verification-Aware Reinforcement Learning" (2604.18176) addresses these failures by introducing a scalable data generation and verification pipeline and a VRM architecture that grounds reinforcement learning in deterministic symbolic solvers and multidimensional semantic auditing. The research is situated within a growing literature on process reward models and verifiable RL for mathematical and code-related tasks, extending these frameworks for application to physics-constrained reasoning where dense stepwise verifiers are nontrivial to design.

Physics-Consistent Dataset Construction

The QuantumQA dataset contains 92,749 rigorously validated question-answer pairs spanning five task types (Short Answer, Fill-in-the-Blank, True/False, Multiple Choice, and Problem Solving), with an emphasis on complex, multi-step derivation tasks over rote factual recall. The construction mechanism is an extension of the Seed-Evolve paradigm, but with two critical augmentations for mitigating scientific hallucination:

1. Task-adaptive data construction enforces concise responses for retrieval questions to reduce verbosity biases and mandates detailed CoT derivations for tasks demanding logical rigor.
2. Hybrid verification protocol layers automated symbolic and physical constraint checking (via SES) with semantic LLM-based auditing, followed by batch-level HITL review with a strict rejection threshold ($\tau > 5\%$), ensuring that synthetic generations are physically plausible and logical errors are aggressively suppressed.

   Figure 1: The QuantumQA data construction pipeline integrates semantic deduplication, symbolic verification, LLM auditing, and human-in-the-loop review for high-fidelity sample filtering.

Dataset analysis reveals broad topical coverage, substantial difficulty variability, and balanced annotation across quantum mechanics subfields.

Figure 2: Distribution of question types in QuantumQA demonstrates coverage of both closed and open-ended scientific reasoning tasks.

Figure 3: Top-10 category distribution of QuantumQA, confirming topical heterogeneity across advanced quantum mechanics themes.

Figure 4: The explicit inclusion of difficult, multi-step problems with complex physical reasoning requirements.

Verification-Aware Reward Modeling

Advancing beyond conventional reward models, VRM orchestrates a dual-source reward signal: deterministic, tool-based verification from a scientific execution suite (SES), and dense, multidimensional semantic scores covering mathematical correctness, physical consistency, and instruction adherence.

Figure 5: Model architecture for verification-aware reward integration, jointly ingesting deterministic SES feedback and multidimensional semantic scores with adaptive fusion.

The core methodology is:

• Extraction of verifiable constraints (Corr, Phys) using SES via a robust parser-executor stack, avoiding brittle matching and supporting symbolic computation and physical law validation.
• Generation of soft semantic scores (for Inst and cases where SES is unapplicable) via a LLM-as-a-Judge framework.
• An adaptive reward fusion (ARF) mechanism that dynamically weights these sources at the per-dimension level depending on signal reliability, thus overcoming reward sparsity and guiding reinforcement learning toward verifiable, physically consistent solutions.

The VRM is trained in a supervised multi-task regression regime, distilling soft oracle judgments from a heterogeneous, leakage-controlled prompt-response dataset. The segmentation of the reward signal is critical: deterministic SES signals are privileged when available, but the model can fallback on high-quality semantic scoring when tool-based verification is impossible—mitigating the high rejection rate associated with rigid verifiers in open-ended science.

Experimental Results and Model Analysis

Zero-shot evaluation on QuantumQA and two OOD scientific test sets (SuperGPQA, Physics) demonstrates that VRM-guided RL (RLVR) using the open weight Qwen3-8B backbone yields parameter-efficient performance advantages. For example, on the QuantumQA Problem Solving subset, RLVR achieves an accuracy of 0.680, outperforming ChatGPT-5 (0.642) and DeepSeek-R1-0528 (0.665) despite a massive gap in parameter count. The model's accuracy on representative Physics problems (0.423) similarly exceeds supervised baselines.

• RLVR provides substantial improvement over SFT (Problem Solving: 0.621→0.680 for Qwen3-8B; Short Answer: 0.817→0.897).
• Best-of-N sampling analysis (BoN, $N=32$) indicates VRM's strong inference discrimination power—single-sample RLVR policies match SFT policies with extensive BoN sampling, evidencing efficient internalization of the reward function.

  Figure 6: Best-of-N sampling demonstrates monotonically increasing performance, and RLVR matches the SFT BoN-32 baseline with only a single sample.

RLVR's VRM supervision also inhibits reward hacking through excessive verbosity. Solution quality shows a negative correlation with token length in RLVR, in contrast to the verbosity bias observed in SFT and base models, confirming that model improvements are not artifacts of longer reasoning chains.

Figure 7: RLVR policies exhibit reduced verbosity bias; superior solution quality is obtained with lower average token length.

Comprehensive error analysis (blind human evaluation over 100 complex problem-solving tasks) highlights that RLVR substantially reduces logical and physical violations (error rate drop from 41.0% to 28.0%) and increases correctness relative to SFT.

Figure 8: RLVR reduces logical and physical reasoning errors across training stages, evidenced by lower violation rates after RL with VRM.

Ablation results demonstrate that removing SES signals, especially deterministic math and physics verifiers, leads to sharp accuracy loss ($0.680 \rightarrow 0.636$), stronger than the performance impact of ablating semantic or instruction signals, quantifying the necessity and complementarity of both verification sources.

Theoretical and Practical Implications

The primary implication is that process-level and constraint-grounded reward modeling offers a viable alternative to pure scale in scientific LLM improvement, enabling open-weight models to compete with much larger closed baselines. By fusing symbolic execution, dimension-adaptive weighting, and multidimensional semantic scores, VRM addresses the reward sparsity, semantic coverage, and hallucination pitfalls present in previous RLHF and PRM approaches. Importantly, ablation analysis positions deterministic tool-grounded feedback as the main driver of robust scientific reasoning, with semantic evaluation filling coverage gaps.

The architectural principles of VRM and SES are domain-agnostic and, with suitable adaptation of the symbolic solvers, are immediately transferable to other natural sciences (classical mechanics, chemistry). The empirical cost in both HITL audit (threshold-driven batch refinement) and automated filtering marks a practical tradeoff but establishes a reliable data foundation for further AI-driven scientific automation.

Limitations and Future Directions

QuantumQA, in its current iteration, focuses on textual and symbolic scientific derivations. Integration of multimodal data (e.g., quantum circuit diagrams) is a critical next step. The dataset samples predominantly formalized problem-solution pairs, omitting long-form, error-prone exploratory reasoning (e.g., Long Chain-of-Thought). The hybrid verification strategy, while highly effective for quantum physics, may require domain-specific adaptation of verification scripts for other scientific contexts.

Future directions include:

• Multi-domain extension: Adapting SES/VRM to other hard sciences with bespoke verifiers.
• Augmentation for Long-CoT: Leveraging QuantumQA's problem-solving subset as seed prompts for synthetic Long-CoT trace generation and filtered reward learning.
• Multimodal reasoning: Integrating diagram and code-level verification as part of the VRM-SES pipeline.

Conclusion

This work establishes a rigorous framework for training and evaluating LLMs in scientific domains demanding explicit physical consistency. By unifying automated symbolic verification, adaptive multidimensional semantic auditing, and HITL review, the QuantumQA dataset and VRM architecture provide a transferable reference pipeline for physics-grounded reasoning in LLMs and demonstrate the practical value of domain-specific, process-level reward design over parameter scaling (2604.18176).