Dense-Path REINFORCE

• The paper establishes a formal equivalence between supervised fine-tuning and inverse Q-learning, enabling extraction of implicit, token-level rewards.
• DPR extracts dense rewards by subtracting a reference baseline from SFT logits, mitigating sequence bias and facilitating granular credit assignment.
• Experimental results show DPR improves instruction-following performance over sparse RL methods, yielding consistent win-rate gains across benchmarks.

Dense-Path REINFORCE (DPR) is an algorithmic framework that reconceptualizes supervised fine-tuning (SFT) for LLMs as inverse reinforcement learning (IRL), enabling the extraction of fine-grained, token-level reward signals from an SFT-trained model and their direct utilization for further policy optimization through on-policy reinforcement learning. DPR introduces a methodology for transforming the implicit reward representations encoded within SFT logits into explicit dense rewards, facilitating robust, granular credit assignment and improved downstream policy improvement via the REINFORCE algorithm (Li et al., 2 Oct 2025).

1. Theoretical Underpinnings: SFT as Inverse Q-Learning

DPR is rooted in the formal equivalence between SFT and inverse Q-learning. Standard SFT minimizes the cross-entropy loss over expert datasets as

$$L_\text{SFT}(\theta) = -\mathbb{E}_{(s,a)\sim\rho_e}\big[\log\pi_\theta(a|s)\big]$$

where $\rho_e$ is the empirical distribution of expert token transitions. In the context of maximum-entropy IRL, the core objective seeks a reward $r(s,a)$ such that its induced policy $\pi^*$ recovers expert occupancy, formalized via

$$J^*(Q) = \mathbb{E}_{(s,a)\sim\rho_e}[Q(s,a) - V(s')] - \mathbb{E}_{s_0}[V(s_0)]$$

with $V(s) = \log \sum_a \exp Q(s,a)$ and transitions $s' = f(s,a)$. Through a telescoping argument and leveraging the identity $\log\pi_Q(a|s) = Q(s,a) - V(s)$, it follows:

$$J^*(Q) = \mathbb{E}_{\tau\sim\rho_e}\left[\sum_t \log\pi_Q(a_t|s_t)\right]$$

This reveals that log-likelihood maximization in SFT coincides exactly with the IRL objective, establishing SFT as inverse Q-learning over a token-level MDP ($\gamma=1$). As a result, logits $\rho_e$0 encode an implicit, reward-explaining function for expert demonstrations (Li et al., 2 Oct 2025).

2. Extraction of Dense Token-Level Rewards

From the trained SFT model $\rho_e$1, the soft Bellman identity yields

$\rho_e$2

and the policy-value relation

$\rho_e$3

leads to

$\rho_e$4

Thus, the log-probability under the SFT model, $\rho_e$5, serves as a potential-shaped surrogate for the underlying reward. To ensure reward stability and mitigate bias toward shorter sequences (arising since $\rho_e$6), DPR introduces a baseline-relative signal:

$\rho_e$7

where $\rho_e$8 is the model checkpoint at $\rho_e$9 of SFT training. This baseline subtraction removes end-of-sequence bias and yields unbiased dense token-level rewards suitable for policy optimization (Li et al., 2 Oct 2025).

3. Algorithmic Pipeline and Pseudocode

DPR operates in two stages: initial SFT followed by dense-path reinforcement learning. The core loop, using the dense rewards $r(s,a)$0, is as follows:

$\pi^*$0 A small KL regularization term ($r(s,a)$1) is optionally applied per standard RLHF practice (Li et al., 2 Oct 2025).

4. Hyperparameter Selection and Implementation

DPR employs standardized hyperparameter settings across model backbones (LLaMA-3.1-8B, Qwen-2.5-7B, Mistral-7B-v0.1, Gemma-3-4B):

Phase	Learning Rate	Batch Size	Max Prompt/Gen Length	Warmup Ratio	KL Weight	Discount γ	Optimizer
SFT	$r(s,a)$2	256	1024	0.03	–	–	Adam
DPR RL	$r(s,a)$3	128	–	0.03	$r(s,a)$4	$r(s,a)$5	Adam

Additional RL parameters: rollout temperature 1, rollout batch size 1024, value-clip for KL: 0.2, samples per prompt: 1. The OpenRLHF REINFORCE++ codebase is used with standard clipping and KL regularization. All settings are held fixed across backbones to eliminate model-specific tuning (Li et al., 2 Oct 2025).

5. Empirical Results and Comparative Performance

DPR is evaluated on four instruction-following benchmarks with LLaMA-3.1-8B as representative:

Method	AlpacaEval (%)	Arena-Hard (%)	LIMA (%)	MT-Bench
SPIN	55.2	53.3	53.0	5.81
GSIL	58.1	56.7	61.0	5.92
SR (sparse RL)	57.9	60.3	60.8	5.96
DPR	60.6	62.5	62.7	6.01

Key findings (Li et al., 2 Oct 2025):

• DPR delivers consistent improvements over SFT (single- to low double-digit % win-rate increases).
• Token-level dense returns from DPR outperform sparse, end-of-sequence RL, most notably on challenging benchmarks (e.g. Arena-Hard).
• DPR is competitive with or surpasses methods such as SPIN and GSIL without utilizing self-play or preference data.
• On MT-Bench, DPR yields +0.2–0.5 higher scores than strong baselines.

Ablation studies indicate:

• Including the potential term $r(s,a)$6 in reward harms stability.
• Omitting the checkpoint baseline introduces EOS-bias and performance collapse.
• Discount factor $r(s,a)$7 is optimal; discounting degrades late-token credit.
• The optimal reference baseline checkpoint is at roughly $r(s,a)$8 of SFT.

Visualizations confirm that $r(s,a)$9 correctly assigns low reward to incorrect reasoning tokens and preserves high reward for correct chains.

6. Context and Relation to Contemporary Dense-Reward Methods

DPR redefines the role of SFT in LLM alignment, positioning it as a dense, IRL-derived reward modeling process, not mere imitation. The methodology is characterized by:

• Explicit reward recovery at each token from SFT logits, circumventing the need for preference or external feedback signals.
• Compatibility with standard actor-critic or REINFORCE pipelines common in RLHF.
• Stable, highly granular credit assignment as compared to sparse, binary- or sequence-level reward approaches.

Comparatively, recent work on policy optimization with densified intrinsic signals (e.g., ISPO) augments sparse binary outcome rewards not by extracting SFT-based rewards but by introducing complementary, model-internal sequence- and token-level conditional information-based rewards. In both cases, the trend is towards reward densification: providing fine-grained learning signals that afford stable policy improvement and mitigate RL-specific pathologies such as zero-advantage collapse or overconfident errors (Chen et al., 7 Jun 2026).

7. Significance and Future Directions

DPR’s reframing of SFT as IRL and its effective extraction of token-level dense rewards have several implications:

• It demonstrates that policy improvement from demonstrations is not limited by the conventionally sparse nature of RL rewards.
• Its alignment with the theoretical apparatus of IRL broadens the set of techniques applicable for further refining pretrained models using expert data.
• Dense-Path REINFORCE’s robust empirical gains (across backbones, tasks, and metrics) suggest that internal model reward signals, extracted post-fine-tuning, can be leveraged systematically for improved credit assignment and instruction-following performance.

A plausible implication is that future work may further elaborate the extraction and use of implicit reward structures from models pretrained by alternative objectives or in broader domains, broadening the applicability of dense-path learning in both natural language processing and sequential decision-making tasks (Li et al., 2 Oct 2025).