Fine-Tuning Small Reasoning Models for Quantum Field Theory

Abstract: Despite the growing application of LLMs to theoretical physics, there is little academic exploration into how domain-specific physics reasoning ability develops while training these models. To investigate this, we perform the first academic fine-tuning study of small (7B-parameter) reasoning models dedicated specifically to theoretical physics. Because open-source verifiable training data required to train such capabilities is scarce, we developed a robust data generation pipeline that can both create synthetic problems and make existing human-authored problems suitable for model training. Selecting Quantum Field Theory (QFT) as our primary domain, we generated over 2,500 synthetic problems alongside a curated collection of human-adapted problems sourced from arXiv and standard pedagogical resources. We conduct both Reinforcement Learning (RL) and Supervised Fine-Tuning (SFT) experiments, benchmarking performance gains as well as generalization to other physics domains. We perform an extensive analysis of model chains-of-though before and after fine-tuning, to understand how reasoning errors evolve during RL and SFT. Finally, we publicly release our data pipeline, verifiable QFT training data, and $\sim$200M tokens of QFT reasoning traces.

• The paper establishes a reproducible QFT fine-tuning pipeline with more than 2,500 synthetic problems, about 800 human-adapted problems, and roughly 195 million released reasoning-trace tokens.
• The paper finds that SFT delivers stronger in-distribution results, while RL generalizes better to external physics benchmarks; Qwen3-30B traces produced the best student, reaching 59.7% Easy-set validation accuracy.
• The paper shows that both methods preferentially reduce factual errors, while mathematical manipulation remains the main residual bottleneck and Hard-set performance stays near zero.

Overview

"Fine-Tuning Small Reasoning Models for Quantum Field Theory" (2604.18936) presents the first academic fine-tuning study of small reasoning models targeted at theoretical physics, using quantum field theory (QFT) as the test domain. The authors—affiliated with the University of Wisconsin–Madison and the Perimeter Institute—fine-tune DeepSeek-R1-Distill-Qwen-7B with both reinforcement learning with verifiable rewards (RLVR/GRPO) and supervised fine-tuning (SFT), and characterize how reasoning errors evolve under each method. Beyond the training experiments, the paper's principal artifacts are infrastructural: a data generation pipeline for code-verifiable physics problems, over 2,500 synthetic QFT problems, roughly 800 human-adapted validation problems, and approximately 195M tokens of rejection-sampled reasoning traces, all released publicly. The work is explicitly framed not as an attempt at frontier performance but as a reproducible academic baseline for studying training dynamics in theoretical physics, a domain where industry labs do not publish training data or intermediate RL rollouts.

Motivation and positioning

The authors argue that while RLVR and SFT principles are publicly known, their implementation at scale is industrial, and the resulting opacity prevents academic study of how domain-specific reasoning develops. QFT is selected for three reasons: abundant pedagogical resources, analytically tractable but tedious calculations, and a well-defined pedagogical progression from scalar fields through non-Abelian gauge theory. The related-work section situates the study against scientific LLMs (Minerva, Galactica, Llemma), formal theorem proving (Ax-Prover), physics benchmarks (TPBench, FrontierMath, CritPt, PhysReason, CMT-Benchmark), reasoning distillation (Orca, Distilling Step-by-Step, DeepSeek-R1 distillation), and GRPO-style RL. Notably, the paper cites a recent result in which a frontier LLM conjectured a closed form for single-minus gluon tree amplitudes that was subsequently formally proven, illustrating that the frontier is shifting from benchmark evaluation toward research collaboration—an ambition far beyond what 7B models can address, but one that motivates building academic training infrastructure.

Dataset construction and verification methodology

The central methodological contribution is a pipeline that produces fully code-verifiable problems. Each problem requires the model to implement its analytic solution in a specified Python function, which is then checked against physically meaningful test cases selected by a frontier model. This automates the code-based verification style of TPBench while avoiding symbolic-expression comparison entirely. The authors define five verifiable task categories—direct calculation, hidden-coefficient derivation, ratio/comparison, categorical classification, and logical consistency checks—and distinguish domain difficulty (depth of background knowledge) from operational difficulty (mechanical and logical demands of execution). This distinction turns out to be one of the paper's most consequential analytical claims, discussed below.

Two generation streams feed the pipeline. Synthetic problems are seeded from a hierarchical topic taxonomy spanning advanced undergraduate to post-graduate levels, with a repetition-avoidance registry of expert summaries. Human-adapted problems are curated from 565 QFT pedagogy seeds (Zee, Peskin & Schroeder, Weinberg, Radovanović, Cheng & Li, MIT OCW) and 397 seeds scraped from roughly 100 arXiv manuscripts via an agentic Gemini-based framework. Quality control is layered: five-metric frontier grading, followed by GPT-5 verification (one attempt for Easy/Medium, three for Hard), with model consensus serving as a proxy for ground truth. The total cost of generation and verification was approximately $2,500, and the filtering statistics are instructive: the Hard set suffers a 58% GPT-5 failure rate versus 18–24% for Easy/Medium, confirming that the difficulty gradient was achieved.

The authors candidly document pipeline failure modes: incorrect frontier solutions (<5% of Gemini-2.5-Pro output), convention leakage that trivialized some problems (detected only after RL training caused an anomalous validation drop from ~60 to 54, after which conventions were corrected), non-physical test cases, and golden-solution code that produced false negatives. They also concede an inductive bias: problems generated and filtered by Gemini/GPT-5 may favor those model families, though they hypothesize the effect is negligible for training models outside those families.

Experiment 1: RL on Easy QFT

Full-parameter GRPO fine-tuning of DeepSeek-7B on the Easy QFT set (640 GPU-hours on 4×H200, following JustRL-style hyperparameters with clip-higher from DAPO and no KL penalty or entropy regularization) yields consistent gains. On the synthetic validation sets, Easy accuracy improves from 40.2% to 54.2%, with strong zero-shot transfer to Medium (26.2% → 44.0%) and a marginal opening of the Hard set (0% → 2.0%). Transfer to out-of-distribution benchmarks is broad: arXiv accuracy rises from 16.6% to 24.6% with gains across every subdomain, QFT pedagogy from 7.8% to 14.2%, and TPBench from 23.5% to 30.5%—with nearly all TPBench gains isolated to Level 2, plus a single solved Level 4 problem on QFT scattering amplitudes.

An anomalous finding deserves emphasis: post-graduate topics within the Easy set are the easiest for both base and fine-tuned models. The authors attribute this to generation bias—ambiguity in what constitutes an "easy" post-graduate problem, and generator overcompensation that defaults to low-complexity reasoning steps on advanced subject matter. They demonstrate this with a spinor-helicity little-group problem whose topic is post-graduate but whose solution is direct substitution.

The authors also report that RL fine-tuning of Qwen3-4B-Thinking-2507 failed to improve beyond base performance, hypothesizing that the model is too optimized and low-capacity to specialize without degrading broad reasoning. This is a negative result stated without extensive investigation.

Experiment 2: SFT with rejection-sampled teacher traces

For SFT, the authors generate reasoning traces from three open-weight teachers (gpt-oss-120b, Qwen3-30B-A3B, Qwen3.5-122B-A10B), apply rejection sampling, and fine-tune DeepSeek-7B with Axolotl. Two exploratory findings shape the main results. First, validation accuracy continues rising through visible SFT overtraining (45% → 60%), so the final checkpoint is used. Second, and more strikingly, Qwen3-30B—the least capable teacher on these tasks—produces the best student (59.7% on Easy validation, versus 50.4% for oss-120b and 47.3% for Qwen3.5-122B). The authors hypothesize that Qwen3-30B's trace style is closest to the student's own reasoning distribution, allowing enhancement rather than replacement of existing patterns; the 100B+ teachers' traces may exceed what a 7B model can consistently emulate. This is consistent with prior distillation literature on "local naturalness" but is here established within a single domain.

Comparing the two methods directly on identical problem sets, SFT outperforms RL in-distribution (Easy: +5.5 points; Medium: +1.2) while RL generalizes better out-of-distribution (QFT pedagogy: +3.9 for RL; +5.8; TPBench: +1.0). The compute asymmetry is severe: RL required 160 hours on 4×H200, whereas SFT dataset generation took 36 hours and training only 75 minutes.

Experiment 3: Narrow-domain RL

Restricting to 35 fermion/spinor problems from the Medium split, LoRA fine-tuning (r=64, α=128, necessary because full fine-tuning was unstable at this data scale) improves in-distribution accuracy on Easy fermion/spinor problems from 14.5% to 25.5% and on Medium from 8.9% to 20.0% (Pass@5 from 22.2 to 55.6), with no gains on Hard and no significant out-of-distribution change. The key conclusion is compartmentalization: targeted learning improves in-distribution performance without degrading general physics reasoning, but also without positive transfer. The authors note this contradicts both of their prior expectations—neither catastrophic forgetting nor transfer to related topics was observed.

Error analysis: the distill-then-classify pipeline

The paper's analytical core is a three-stage chain-of-thought error analysis applied to 100 rollouts per problem on the Easy validation set. The authors first document three failed approaches—whole-CoT analysis (defeated by >10k-token traces and self-correction callbacks), line-by-line analysis (granularity mismatch), and per-problem taxonomies (non-aggregatable)—before arriving at a pipeline that decomposes golden solutions into reference logical steps, deduplicates and distills student CoTs into 5–15 logical steps, and classifies errors against a fixed four-label taxonomy (factual, mathematical, logical, executional) with a dedicated code-comparison stage. Analyzer consistency between gpt-oss-120b and Claude Sonnet 4.6 is checked on 52 incorrect rollouts, with satisfactory aggregate agreement and acknowledged label ambiguity at category boundaries.

The headline finding contradicts a piece of conventional wisdom: RL, not SFT, is the stronger reducer of factual errors. Across the top-20 most-improved problems, RL reduces factual error frequency per incorrect rollout from 0.59 to 0.37 (a 37% reduction), while mathematical, logical, and executional error frequencies remain roughly constant. The prevailing view—SFT teaches knowledge while RL teaches strategy—is inverted here. The SFT analysis shows the same qualitative pattern (factual errors 0.51 → 0.44 per incorrect rollout, with other categories rising as a share of residual errors), suggesting preferential factual-error elimination is a general property of domain-specific fine-tuning rather than an artifact of the RL reward signal. On the 10 overlapping problems, both methods reduce incorrect rollouts by ~67–70%, and residual errors are dominated by mathematical mistakes (RL 0.64, SFT 0.60 per incorrect rollout), identifying algebraic manipulation as the principal remaining bottleneck and motivating tool-use integration.

Trace-level analysis reveals a redistribution rather than a net change: aggregate trace lengths (7,148 vs. 7,136 tokens) and backtracking frequencies (24.2 vs. 23.9) are nearly identical, but the RL model generates substantially longer traces on correct attempts (5,399 vs. 3,594 tokens) and backtracks more when it succeeds (20.6 vs. 12.9) while backtracking less on failures (27.7 vs. 31.4). RL thus reallocates self-correction toward solvable problems. SFT, by contrast, universally suppresses backtracking and inflates incorrect-trace length, which the authors attribute to teacher verbosity overriding the student's native self-correction patterns.

Difficulty analysis

Extending to 100 rollouts per problem across all three synthetic sets, the authors find no consistent correlation between accuracy and domain difficulty. Post-graduate problems are solved more reliably than advanced-undergraduate ones in the Easy set (e.g., 62% RL vs. 51%), and the conclusion is that operational complexity—the length of the derivation chain and the number of implicit constraints—is the primary driver of solve rate, not the depth of specialized knowledge. The authors appropriately caveat that Hard-set performance is too low to support conclusions and that this result is specific to their datasets, where the base model's knowledge base plausibly covers all domain levels roughly evenly.

Limitations and open questions

Several limitations bear directly on the results. The synthetic data carries generator bias, including the demonstrated convention-leakage error and the post-graduate difficulty inversion; residual labeling errors in golden solutions cannot be fully excluded, and the authors rely on frontier-model consensus as a proxy for ground truth. The Hard tier remains essentially unsolved by all models studied (base: 0%, RL: 2.0%), so claims about reasoning at high operational difficulty are extrapolated from near-floor performance, and the authors note that best-of-N test-time scaling was not affordable within their compute budget. The narrow-domain experiment rests on only 9 validation problems, making the in-distribution Medium gains (8.9% → 20.0%) statistically fragile. The error taxonomy involves inherent category ambiguity (e.g., a sign inversion can be labeled mathematical, logical, or factual), which the authors mitigate through minimal label sets and consistency checks but do not eliminate. The claim that RL reduces factual errors rests on aggregated frequencies over the top-20 most-improved problems rather than a pre-registered evaluation. Open questions left by the paper include whether multi-step RL or symbolic-verifier rewards could break the Hard-tier barrier, whether teacher-student style matching can be made principled rather than post hoc, and how the factual-error finding extends to larger models and other physics domains.

Conclusion

This paper establishes a reproducible academic baseline for fine-tuning reasoning models in theoretical physics, with three durable contributions: a verified, code-checkable QFT problem corpus and generation pipeline released at modest cost (~$2,500); a controlled RL-versus-SFT comparison showing SFT's in-distribution advantage against RL's out-of-distribution generalization; and an error-taxonomy analysis indicating that domain-specific fine-tuning—by either method—preferentially eliminates factual errors while leaving mathematical manipulation as the dominant residual failure mode. The negative results (failure to improve Qwen3-4B-Thinking, the compartmentalization of narrow-domain gains, the Hard-tier floor) are reported with the same clarity as the positive ones, and the released ~195M tokens of reasoning traces provide a concrete substrate for subsequent academic studies of physics reasoning in LLMs.