FineProofs-SFT: Reward-Aware Fine-Tuning

• The paper introduces FineProofs-SFT, a framework that reframes supervised fine-tuning as an inverse reinforcement learning problem by learning reward signals from expert demonstrations.
• It employs a contrastive approach by comparing expert continuations with self-generated outputs to mitigate overfitting and reduce distribution shift.
• Empirical results on 1B and 7B-scale models show that FineProofs-SFT improves alignment, generalization, and downstream performance over traditional SFT and SPIN methods.

Searching arXiv for the cited paper and closely related work on IRL-based SFT, SPIN, and DPO to ground the article in current research. FineProofs-SFT is a reward-aware supervised fine-tuning framework that treats the supervised fine-tuning stage itself as an inverse reinforcement learning problem in sequence space. Rather than using human demonstrations only through maximum-likelihood imitation, it learns a reward signal from demonstrations and couples that reward to policy updates during supervised fine-tuning. In the formulation introduced in “Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment” (Li et al., 2024), the framework’s central claim is that even when only demonstration data is available, learning a reward model from that data and optimizing the policy against it leads to more robust alignment, better generalization, and improved downstream performance. This reframing places FineProofs-SFT at the intersection of supervised fine-tuning, maximum-entropy inverse reinforcement learning, and KL-regularized soft reinforcement learning (Li et al., 2024).

1. Conceptual definition and problem setting

FineProofs-SFT is defined as a framework that turns human demonstrations into a usable reward signal via inverse reinforcement learning and couples that signal to policy optimization during supervised fine-tuning (Li et al., 2024). In the standard alignment pipeline, supervised fine-tuning learns from demonstration pairs $(x,y)$, and preference learning then fits a reward model from chosen-versus-rejected comparisons before a reinforcement-learning stage further adjusts the policy. FineProofs-SFT moves reward learning into the supervised fine-tuning stage itself, so that demonstrations are no longer treated as targets for pure log-likelihood maximization alone (Li et al., 2024).

The motivation is rooted in two limitations of conventional supervised fine-tuning. First, a pure maximum-likelihood objective is brittle to noisy or low-quality demonstrations, because it pushes the policy toward reproducing observed continuations without distinguishing preferred from non-preferred alternatives in nearby output regions. Second, behavior-cloning-style fine-tuning exacerbates distribution shift, since it can overfit observed trajectories, assign extreme probability mass to demonstrated continuations, and collapse probabilities for unseen but acceptable responses (Li et al., 2024). The framework therefore recasts supervised fine-tuning as a contrastive process in which expert continuations are preferred over model-generated continuations, rather than merely copied.

This formulation suggests a broader reinterpretation of supervised fine-tuning. Instead of viewing SFT as only a conditional density-estimation problem, FineProofs-SFT treats it as a structured preference-learning problem with latent reward inference. A plausible implication is that the framework belongs to a family of post-training methods that attempt to preserve exploration and avoid over-specialization already at the SFT stage, rather than waiting for a downstream RLHF stage to repair the consequences of pure imitation.

2. Formal inverse-reinforcement-learning formulation

The framework is developed in the sequence-modeling setting. Let $p_\phi(y \mid x)$ denote the auto-regressive policy over response sequences $y$ conditioned on prompts $x$, and let $R_\theta(x,y)$ denote a scalar reward for a full sequence. Maximum-entropy inverse reinforcement learning models expert demonstrations $\tau=(x,y)$ with an exponential-family likelihood,

$p_\theta(y \mid x) \propto \exp(R_\theta(x,y)).$

Under an auto-regressive factorization, this corresponds to a maximum causal entropy view in which the sequence reward aggregates token-level rewards, with

$$p_\theta(\tau) \propto \exp\!\left(\sum_t r_\theta(s_t,a_t)\right)$$

(Li et al., 2024).

The central saddle-point objective adapts MaxEnt IRL to sequence space by matching occupancy measures of the expert and the learned policy. Using the joint distribution over prompts and completions,

$d_\pi(x,y)=\rho(x)\pi(y \mid x),$

the reward-policy minimax objective is

$$\max_\theta \min_\phi \left[\mathbb{E}_{(x,y)\sim D}[R_\theta(x,y)]-\mathbb{E}_{x\sim D,\,y\sim p_\phi(\cdot \mid x)}[R_\theta(x,y)]-\lambda H(p_\phi)\right].$$

For fixed reward, the policy subproblem becomes a soft RL objective,

$p_\phi(y \mid x)$0

(Li et al., 2024).

In practice, FineProofs-SFT uses KL regularization to a reference model $p_\phi(y \mid x)$1, usually the pretrained model or an SFT baseline. With KL coefficient $p_\phi(y \mid x)$2, the policy subproblem is

$p_\phi(y \mid x)$3

This has the closed-form solution

$p_\phi(y \mid x)$4

where $p_\phi(y \mid x)$5 is the partition function (Li et al., 2024).

This closed-form policy is important for both theory and implementation. It links the MaxEnt IRL normalization directly to soft RL machinery, while clarifying that the learned reward is not an auxiliary diagnostic object but an operational quantity that directly shapes the supervised fine-tuning policy.

3. Algorithmic realizations: RFT and IRFT

FineProofs-SFT is instantiated through two algorithms, RFT and IRFT (Li et al., 2024). RFT, or Reward-learning Fine-Tune, explicitly trains both a reward model and a policy. Its training loop alternates between reward ascent and policy optimization. In the reward step, expert demonstrations and model-generated continuations are contrasted through the gradient

$p_\phi(y \mid x)$6

which directly maximizes the expert-versus-policy reward gap. In the policy step, the method either sets

$p_\phi(y \mid x)$7

or applies PPO or REINFORCE updates against the learned reward under KL-to-reference regularization (Li et al., 2024).

IRFT, or Implicit Reward-learning Fine-Tune, removes the explicit reward model while remaining IRL-consistent. It relies on the identity

$p_\phi(y \mid x)$8

which turns reward learning into a log-likelihood-ratio objective. The resulting self-generation gradient is

$p_\phi(y \mid x)$9

In implementation, IRFT uses the smooth logistic surrogate

$y$0

applied to the log-ratio gap between expert and self-generated continuations (Li et al., 2024).

The two variants differ primarily in explicitness and cost profile. RFT separates reward learning and policy learning, whereas IRFT trains only the policy. The paper characterizes IRFT as easy to implement inside a standard SFT loop, while RFT remains efficient relative to conventional RLHF because reward learning uses only demonstrations rather than pairwise preference labels (Li et al., 2024).

A useful summary is given below.

Algorithm	Core mechanism	Main training objects
RFT	Alternating reward ascent and policy update	Explicit reward model and policy
IRFT	Implicit IRL objective from log-likelihood ratios	Policy only
Shared principle	Expert continuations contrasted against self-generated continuations	KL-regularized policy improvement

4. Robustness properties and relation to self-play methods

A central claim of the framework is robustness to low-quality demonstrations. The paper attributes this to the contrastive IRL term

$y$1

which teaches the model not merely to imitate demonstrations but to prefer them over its own current outputs (Li et al., 2024). This “expert vs. self” margin regularizes learning, discourages degenerate modes, and reduces the impact of spurious or suboptimal targets.

Two empirical effects are highlighted. First, IRL-based supervised fine-tuning produces less extremal policies than pure SFT: a toy example with one prompt and three actions shows that IRL keeps non-zero mass on unseen actions, whereas SFT collapses all probability onto the single demonstrated action. Second, on Anthropic-HH, training only on chosen responses still increases the log-likelihood gap

$y$2

versus SFT, which often gives higher probability to rejected continuations. The reported explanation is that reward learning explicitly contrasts expert samples with model samples, even though non-preferred training examples were not provided (Li et al., 2024).

The framework also establishes a precise connection to Self-Play Fine-Tune (SPIN). Using the identity

$y$3

the IRL gradient becomes the difference of log-likelihood ratios between expert continuations and self-play continuations, which is described as exactly the SPIN training signal. With $y$4, IRFT recovers SPIN; with $y$5, it generalizes SPIN by generating more frequently and by providing finite-time convergence guarantees to stationary points of the IRL objective (Li et al., 2024).

This connection is conceptually important because it shifts the interpretation of self-play fine-tuning. Rather than treating SPIN only as a two-player game heuristic, FineProofs-SFT derives the same contrastive signal from a single-agent MaxEnt IRL formulation. This suggests that self-play-style supervised tuning can be understood as reward learning from demonstrations, rather than as a disconnected empirical trick.

5. Theoretical guarantees

The framework provides finite-time convergence guarantees to stationary solutions of the IRL problem (Li et al., 2024). The relevant stationarity conditions are stated separately for reward and policy. Reward optimality at $y$6 requires

$y$7

for

$y$8

while policy optimality at $y$9 requires

$x$0

for

$x$1

Under boundedness and smoothness of $x$2, with outer iterations $x$3 and $x$4 inner updates per iteration and stepsize $x$5, both RFT and IRFT achieve the informal convergence rate

$x$6

The analysis explicitly accounts for the bias introduced by reusing generated samples within an inner loop (Li et al., 2024).

The theoretical significance of these results is twofold. First, they place IRL-based supervised fine-tuning on a more formal footing than many heuristic post-training schemes. Second, the KL regularization makes the lower-level policy problem strongly convex in the appropriate dual variables, which is what enables the closed-form optimal policy and the subsequent gradient-bias bounds (Li et al., 2024). The paper does not claim global optimality; convergence is to stationary points, and characterization of the limiting policy is left open.

6. Empirical results, implementation profile, and limitations

The empirical evaluation uses both 1B-scale and 7B-scale models. For explicit reward learning, a 1B reward model based on Pythia-1B is paired with a Pythia-1.4B policy; for larger-scale policy fine-tuning, Zephyr-7B-SFT-Full is used with LoRA due to resource limits (Li et al., 2024). Demonstrations come from Ultrachat200k, with 50k training samples following SPIN’s setup, and from Anthropic-HH preferred continuations, where 10k top-scored responses selected by the PKU beaver-7B reward model are used as demonstrations (Li et al., 2024).

The headline quantitative result reported for Zephyr-7B-SFT-Full on the HuggingFace Open LLM Leaderboard is an average improvement from 59.48 for the base model to 61.03 for IRFT with $x$7 and 4 epochs. Task-wise values reported for this setting are ARC 76.78, TruthfulQA 36.84, Winogrande 77.43, GSM8K 34.34, HellaSwag 83.05, and MMLU 57.72 (Li et al., 2024). For Pythia-1.4B on Ultrachat, IRFT is reported to yield consistent average gains over both SFT and SPIN, with the best settings around $x$8 (Li et al., 2024). On Anthropic-HH reward evaluation, RFT trained only on preferred demonstrations improves average helpfulness/harmlessness scores measured by PKU beaver-7B reward and attains higher win rates than both the base SFT model trained on the full 160k demonstrations and a top-10k SFT baseline (Li et al., 2024).

Implementation details emphasize relatively modest overhead. For IRFT, the reported recipe includes 2–4 epochs per iteration, outer iterations $x$9, RMSProp, peak learning rate $R_\theta(x,y)$0 then $R_\theta(x,y)$1, $R_\theta(x,y)$2, bfloat16 precision, FlashAttention-2, and DeepSpeed ZeRO-3. Pythia-1.4B is trained on $R_\theta(x,y)$3A100-40G with per-device batch size 8, while Zephyr-7B LoRA runs on $R_\theta(x,y)$4A100-40G with per-device batch size 2 (Li et al., 2024). RFT uses PPO via HuggingFace TRL for the policy step (Li et al., 2024).

The reported ablations indicate that moderate generation frequency performs best: IRFT with generation frequency around $R_\theta(x,y)$5 performs best, while larger $R_\theta(x,y)$6 can increase variance. RFT benefits from staging reward alignment and policy alignment, often with $R_\theta(x,y)$7 and large $R_\theta(x,y)$8. Regularization strength $R_\theta(x,y)$9 yields stable updates, though the authors note that empirical tuning may further optimize performance (Li et al., 2024).

The limitations are stated directly. Convergence guarantees are only to stationary points; explicit reward-plus-policy training adds compute relative to IRFT; very noisy demonstrations still limit performance despite improved robustness; and generalization across domains and longer-horizon tasks requires further study (Li et al., 2024). The comparison section also clarifies the method’s position relative to other post-training paradigms: compared with standard SFT, it adds an expert-versus-self reward contrast; compared with RLHF, it moves reward learning into SFT and avoids expensive preference collection; compared with DPO or IPO, IRFT uses self-generated negatives rather than human preference pairs; and compared with SPIN, IRFT recovers the $\tau=(x,y)$0 case while generalizing to $\tau=(x,y)$1 with finite-time guarantees (Li et al., 2024).

Taken together, FineProofs-SFT is best understood as a principled attempt to “get more juice out of the SFT data” by replacing pure imitation with reward-aware contrastive alignment at the demonstration stage itself (Li et al., 2024). Its distinguishing features are the MaxEnt IRL reformulation, the KL-regularized closed-form policy view, the explicit or implicit expert-versus-self contrast, and empirical evidence that these ingredients improve robustness and downstream performance without requiring preference-pair supervision.