Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision (2509.14234v1)

Abstract: Where do learning signals come from when there is no ground truth in post-training? We propose turning exploration into supervision through Compute as Teacher (CaT), which converts the model's own exploration at inference-time into reference-free supervision by synthesizing a single reference from a group of parallel rollouts and then optimizing toward it. Concretely, the current policy produces a group of rollouts; a frozen anchor (the initial policy) reconciles omissions and contradictions to estimate a reference, turning extra inference-time compute into a teacher signal. We turn this into rewards in two regimes: (i) verifiable tasks use programmatic equivalence on final answers; (ii) non-verifiable tasks use self-proposed rubrics-binary, auditable criteria scored by an independent LLM judge, with reward given by the fraction satisfied. Unlike selection methods (best-of-N, majority, perplexity, or judge scores), synthesis may disagree with the majority and be correct even when all rollouts are wrong; performance scales with the number of rollouts. As a test-time procedure, CaT improves Gemma 3 4B, Qwen 3 4B, and Llama 3.1 8B (up to +27% on MATH-500; +12% on HealthBench). With reinforcement learning (CaT-RL), we obtain further gains (up to +33% and +30%), with the trained policy surpassing the initial teacher signal.

• The paper introduces CaT, a novel method that synthesizes reference answers from diverse rollouts as a teacher signal without human annotation.
• The approach employs both inference-time optimization and an RL loop (CaT-RL), achieving up to 33% gains on MATH-500 and 30% on HealthBench.
• It uses rubric-based reward construction to validate model outputs, outperforming selection-based baselines and even matching expert human rubrics.

Compute as Teacher: Reference-Free Supervision via Inference Compute

Introduction and Motivation

The paper "Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision" (2509.14234) introduces a method, CaT, for post-training LLMs in domains where ground-truth references or programmatic verifiers are unavailable. The central premise is to leverage inference-time compute—specifically, the diversity in parallel rollouts generated by the current policy—to synthesize a reference answer using a frozen anchor model. This synthesized reference is then used as a teacher signal for further optimization, either at inference or within an RL loop (CaT-RL). The approach is designed to be drop-in, requiring no human annotation and minimal domain-specific engineering.

Figure 1: CaT pipeline: exploration via parallel rollouts, synthesis of a reference by a frozen anchor, and reward conversion for verifiable and non-verifiable domains.

Methodology

Synthesis of Reference Answers

For each prompt $q$, the current policy $\pi_t$ generates $G$ parallel rollouts $o_{1:G}$. The anchor policy $\pi_0$ (frozen, typically the initial policy) is conditioned only on these rollouts (not the prompt) and tasked with synthesizing a single reference answer $s$ that reconciles omissions, contradictions, and partial solutions. This synthesis is not a selection among rollouts but a generative reconciliation, enabling the anchor to produce answers that may disagree with all rollouts and even correct errors present in every sample.

Reward Construction

• Verifiable Tasks (e.g., Math): The synthesized reference $s$ is used as a target, and rewards are assigned via programmatic equivalence checks (e.g., string match of boxed answers).
• Non-Verifiable Tasks (e.g., Clinical Guidance): The anchor generates a response-specific rubric $\mathcal{R}$, a set of binary, auditable criteria. An independent LLM judge $\pi_J$ evaluates whether each rollout satisfies each criterion, and the reward is the fraction of criteria met.

  Figure 2: Rubric-based rewards for non-verifiable tasks: anchor generates rubrics, judge LLM scores each criterion, reward is normalized proportion satisfied.

Training Regimes

• Inference-Time CaT: The anchor synthesizes a reference from rollouts at test time, improving output quality without weight updates.
• CaT-RL: The synthesized reference or rubric-based reward is used within a GRPO RL loop to update the policy, enabling further improvement beyond the initial teacher signal.

Empirical Results

Performance Gains

CaT and CaT-RL were evaluated on MATH-500 (verifiable) and HealthBench (non-verifiable) using Gemma 3 4B, Qwen 3 4B, and Llama 3.1 8B. CaT at inference yields up to +27% improvement on MATH-500 and +12.5% on HealthBench. CaT-RL further improves performance, achieving up to +33% and +30% relative gains, with the trained policy often surpassing the initial teacher signal.

Figure 3: CaT and CaT-RL improve models by up to $\sim$30% relative to the initial policy baseline.

Rubric Rewards vs. Alternatives

Self-proposed rubrics outperform model-as-judge semantic equivalence and are competitive with expert human rubrics, demonstrating the efficacy of decomposed, auditable criteria for reward construction in non-verifiable domains.

Figure 4: Left: Self-proposed rubrics rival expert rubrics. Right: RL with rubrics outperforms SFT on non-verifiable tasks.

Comparison to Selection Baselines

CaT outperforms single-sample, best-of-$N$, minimum perplexity, mutual predictability, and majority vote baselines, both in verifiable and non-verifiable domains. Notably, CaT can produce correct answers that disagree with all rollouts, a capability unattainable by selection-based methods.

Figure 5: CaT at inference outperforms all selection-based baselines, with substantial improvements on both HealthBench and MATH-500.

Scaling with Rollout Count

Performance scales monotonically with the number of rollouts $G$ in verifiable domains, plateauing in non-verifiable domains after $G \approx 4$. This scaling property enables a practical trade-off between FLOPs and supervision signal strength.

Figure 6: Left: CaT scales with rollout count. Right: CaT leverages cross-rollout reasoning rather than acting as a new rollout.

Mechanistic Insights

Analysis shows that CaT does not simply act as another rollout; it leverages the diversity and reasoning present in the set of rollouts, reconciling disagreements and synthesizing improved answers. CaT disagrees with majority voting on 14% of questions and with all rollouts on 1%, indicating genuine reconciliation rather than selection.

Limitations and Future Directions

CaT's effectiveness depends on the anchor's ability to synthesize meaningful references; weak base models or domains with insufficient rollout diversity may limit improvement. As the policy converges and rollout diversity decreases, the teacher signal plateaus, bounding further gains. Future work may focus on promoting rollout diversity via exploration rewards or sampling strategies, extending synthesis to reasoning traces, and automating question generation to eliminate reliance on curated datasets.

Figure 7: The teacher signal from CaT-RL synthesis converges with the trained policy as rollout diversity diminishes.

Practical and Theoretical Implications

CaT provides a scalable, annotation-free supervision mechanism for both verifiable and non-verifiable domains, addressing a major bottleneck in specialized LLM development. The method generalizes self-training and knowledge distillation by synthesizing references from model exploration rather than relying on single self-labels or consensus. The use of self-proposed rubrics for reward construction in non-verifiable domains offers a robust alternative to judge-only feedback, mitigating issues of instability and bias.

Theoretically, CaT demonstrates that inference compute can be systematically converted into supervision, suggesting a pathway toward reference-free, potentially superhuman model capabilities unconstrained by human-annotated data. The approach bridges RL, self-training, and ensemble error correction, and opens avenues for further research in reference estimation strategies and reward shaping.

Conclusion

Compute as Teacher (CaT) establishes a principled framework for turning inference compute into reference-free supervision by synthesizing reference answers from parallel rollouts and converting them into rewards for both verifiable and non-verifiable tasks. Empirical results show substantial improvements over baselines and demonstrate the superiority of synthesis over selection. The method is practical, scalable, and broadly applicable, with implications for the future of LLM post-training and the development of models in domains where human supervision is scarce or contested.