Fine-Tuning Large Language Models for Quantum Reasoning

Abstract: LLMs exhibit abilities beyond natural language modelling and text generation. Recent advances in their reasoning capabilities have spurred interest in applying LLMs to complex scientific tasks requiring deep domain expertise and sophisticated reasoning. Quantum computing, as a highly specialised field with significant knowledge barriers and hardware constraints, could greatly benefit from such advancements. However, a key open question that first must be answered is: How can we develop fine-tuning pipelines that instil genuine quantum reasoning in LLMs, rather than task-specific pattern matching? We study this question through quantum circuit simulation as a training objective, where the model must predict the measurement probability distribution resulting from a sequence of quantum gate operations. We propose and compare two fine-tuning pipelines: (1) Supervised Fine-Tuning (SFT) on explicit gate-by-gate state-vector simulation traces, and (2) a two-stage SFT+Group Relative Policy Optimisation (GRPO) approach that sequentially applies SFT followed by GRPO with verifiable rewards. Our findings show that SFT achieves near-perfect in-distribution and gate-count extrapolation accuracy, significantly outperforming both the base model and the GPT-OSS-120B baseline. SFT+GRPO trades some in-distribution precision for better generalisation to larger qubit systems that SFT alone cannot handle. Both pipelines significantly outperform the baselines, demonstrating that targeted fine-tuning on explicit reasoning traces is an effective strategy for advancing quantum reasoning in LLMs.

• The paper introduces two fine-tuning strategies, SFT and SFT+GRPO, that significantly enhance LLMs' quantum reasoning by simulating quantum circuits step-by-step.
• It employs explicit gate-by-gate state tracking with verifiable rewards to achieve high fidelity in predicting quantum measurement distributions.
• While achieving token efficiency and accuracy on small circuits, the method highlights challenges in generalizing to larger qubit systems, suggesting directions for further research.

Fine-Tuning LLMs for Quantum Reasoning: Methodology and Empirical Findings

Introduction and Motivation

Recent advancements in LLMs have demonstrated reasoning capabilities that transcend traditional text generation, prompting their application to highly specialized scientific domains, notably quantum computing. The core question addressed in "Fine-Tuning LLMs for Quantum Reasoning" (2606.21974) is whether LLMs can be fine-tuned to acquire genuine quantum reasoning proficiency, instead of performing task-specific pattern matching. The study frames this investigation through quantum circuit simulation: the LLM must predict measurement probability distributions resulting from sequences of quantum gate operations. The work introduces two fine-tuning strategies for this objective: (1) Supervised Fine-Tuning (SFT) on explicit gate-by-gate state-vector simulation traces and (2) a two-stage approach that combines SFT with Group Relative Policy Optimization (GRPO) using verifiable rewards.

Experimental Pipeline and Datasets

Two circuit datasets are curated: a Non-parameterised set containing discrete gates (H, X, Z, CNOT, CZ, MCX/MCZ), and a Parameterised set that adds $R_x, R_y, R_z$ gates with sampled rotation angles. Both datasets span 1–5 qubits and 1–50 gates. Each circuit is encoded via the Qiskit Python API for token efficiency and covers a diverse span of gate and qubit combinations, including non-trivial multi-controlled samples by actively modifying subspaces before multi-qubit gate application.

The SFT pipeline employs structured reasoning templates for explicit state-vector tracking at every gate application, embedding both symbolic and rounded numeric amplitude representations. Outputs provide measurement distributions for the top 15 states per circuit.

The two-stage SFT+GRPO pipeline starts with SFT on quantum circuit reasoning, mathematical reasoning, and code reasoning tasks to prevent catastrophic forgetting and preserve entropy diversity, followed by GRPO optimization for efficient and accurate probability estimation. GRPO uses binary rewards (format compliance, TVD threshold) and LoRA adaptation to scale fine-tuning efficiently.

Model Architectures and Fine-Tuning Approaches

Both pipelines utilize Qwen3-8B as the base LLM, with LoRA deployed for computationally feasible adaptation. SFT explicitly masks the <circuit_reasoning> token loss to allow mode switching, and GRPO leverages rollouts and on-policy optimization with mean-reward-based advantage computation.

Ablation studies confirm the criticality of explicit reasoning traces for quantum circuit simulation accuracy: training on circuit-task pairs alone yields moderate but suboptimal performance compared to trace-supervised models.

Empirical Results and Analysis

The detailed quantitative evaluation spans three settings: in-distribution (Set 1), gate-count extrapolation (Set 2), and system-size extrapolation (Set 3, circuits with 6–7 qubits). SFT achieves near-perfect F1 and TVD scores in-distribution but collapses on larger qubit counts due to inability to generalize state initialization and reasoning to higher-dimensional Hilbert spaces. SFT+GRPO achieves partial generalization (nonzero F1) for out-of-distribution qubit counts, signaling adaptive strategic behavior under RL incentives.

SFT produces concise, highly accurate outputs (mean 2,101 tokens for Set 1), yielding maximal token efficiency. SFT+GRPO responses are longer but remain significantly more efficient than baseline LLMs (Qwen3-8B, GPT-OSS-120B), which produce excessively verbose and inaccurate distributions, especially post parameterization.

Notable findings:

• Stepwise quantum state fidelity for SFT remains above 99% for non-parameterized circuits and above 95% for parameterized circuits, with degradation most pronounced in 5-qubit systems and deeper circuits.

Figure 1: Step-by-step quantum state fidelity of the SFT model during inference, persistent above 99% on non-parameterized circuits and above 95% for parameterized circuits, indicating robust intermediate quantum state representation.

• GRPO stage rapidly improves reward and reduces entropy and mean response length, especially for non-parameterized sets.

Figure 2: GRPO training dynamics showing rising mean reward and decreasing actor entropy/response length, reflecting convergence toward concise and accurate reasoning.

• Post-GRPO, token limit violations induced by SFT (especially in parameterized circuits) are suppressed, reducing TVD.

Figure 3: TVD and token-limit violation counts across training stages; GRPO suppresses SFT-induced token overflows yielding improved TVD.

• Performance degradation correlates most strongly with qubit number (exponential increase in state space), moderately with gate count, and minimally with circuit depth.

Figure 4: Mean TVD as a function of qubit count, circuit depth, and gate number; qubit count dominates TVD degradation.

Qualitative analysis reveals adaptive reasoning strategies exploited by SFT+GRPO, such as omission of redundant state listings and correct output dimensionality on system-size extrapolation, despite internal reasoning failures in state initialization (token-efficient shortcuts).

Theoretical Implications

The findings establish that fine-tuning LLMs on explicit gate-by-gate reasoning traces substantially enhances quantum reasoning proficiency, outperforming general-purpose reasoning LLMs even at larger scale. The persistent challenge of length generalization—failure to adapt reasoning to exponentially larger state spaces for increased qubit counts—mirrors shortcomings observed in arithmetic and algorithmic LLM research. The ability to generalize reasoning strategies under RL exploitation is promising but incomplete, suggesting the need for architectural and representational advances beyond supervised pipeline augmentation.

Explicit learning of quantum state dynamics is facilitated by stepwise trace supervision, while RL enables adaptive efficiency and partial dimensional generalization. Handling parameterized circuits (continuous gate rotations) is achieved with minimal additional accuracy loss, indicating the core complexity derives not from gate parameterization but Hilbert space scaling.

Practical Implications and Future Outlook

These results indicate the plausibility of using fine-tuned LLMs as quantum circuit simulators and verifiers within research and educational quantum computing environments. They further suggest that grounding LLMs in quantum mechanical principles via explicit reasoning signal is superior to prompt engineering or agentic frameworks in conferring genuine domain understanding.

Potential future directions include:

• Construction of fully general Quantum World Models via trace-supervised LLMs, applicable to circuit synthesis and algorithmic design.
• Improved representational schemes for quantum states (increased symbolic precision, phase tracking).
• Architectural innovations leveraging structured state representations for scalable length generalization.
• Extension to diverse circuit description languages (e.g., OpenQASM, Cirq) for broader benchmark compatibility.

Conclusion

The systematic empirical assessment demonstrates that LLMs fine-tuned on quantum circuit simulation traces acquire robust quantum reasoning capability with superior token efficiency and accuracy compared to both base and large-scale general reasoning models. RL augmentation via GRPO elicits adaptive reasoning strategies and partial system-size generalization but length generalization remains an unresolved challenge central to quantum simulation and broader algorithmic reasoning. The study delineates clear methodological advances and sets forth the trajectory for future LLM-driven quantum research.