---
title: 'FinePO Algorithm: Fine-Grained Policy Optimization'
url: https://www.emergentmind.com/topics/finepo-algorithm
type: topic
---

# FinePO Algorithm: Fine-Grained Policy Optimization

FinePO Algorithm

FinePO ("Fine-grained Policy Optimization") is a reinforcement learning algorithm designed to address the limitations of trajectory-level credit assignment in large language models (LLMs) and multimodal large language models (MLLMs). Instead of uniformly propagating a scalar reward across all actions within a generated response, FinePO performs fine-grained, step-level credit assignment by leveraging a learned reward model. This approach improves both the alignment and efficiency of policy optimization in complex multi-step reasoning tasks, particularly those requiring explicit, visible intermediate actions, such as chart understanding and visual reasoning [2601.05688].

## 1. Motivation and High-Level Objectives

FinePO's central aim is to resolve the coarse granularity of reward signal assignment inherent to standard RL for LLMs and MLLMs. In conventional approaches, an overall reward $R(y)$ is computed for a complete trajectory or output $y$, and this reward is typically broadcast identically to each token or decision in the sequence. This uniform assignment constrains the model's ability to distinguish skillful sub-decisions from erroneous ones within a single response.

FinePO introduces step-level differentiation by:
- Decomposing each output trajectory into explicit “intent + action” steps,
- Scoring every individual step using a learned Fine-grained Process Reward Model (FinePRM),
- Redistributing global trajectory-level advantage among steps in proportion to their assessed quality.

This approach enables the model to reinforce correct sub-decisions and penalize flawed ones within the same response, which empirically reduces gradient variance and leads to more robust policy improvements [2601.05688].

## 2. Fine-grained Process Reward Model (FinePRM)

FinePRM (𝒫) is a parameterized evaluator that provides scalar step-level scores essential to FinePO’s per-step advantage computation. Its core characteristics are:
- **Inputs**: For each step $s_j$, the FinePRM receives the step’s textual intent, the drawing action parameters, and the before/after visual context (i.e., the state of the image before and after the action).
- **Architecture**: Built upon an MLLM backbone, FinePRM aggregates these inputs via a prompt template and classifies each step into one of four quality categories (Excellent, Acceptable, Poor, Unacceptable), assigning numeric scores $\{4.0, 3.0, 2.0, 1.0\}$.
- **Dataset construction**: The FinePRM is trained on a dataset of 473K labeled examples sourced through visual-to-text annotation and text-to-image distillation. Annotations are balanced across quality labels (2:4:3:1 ratio).
- **Score aggregation**: For each trajectory, FinePO computes a length-weighted intra-trajectory mean score and calculates deviations $\Delta_j$ from the mean for per-step refinement.

This step-level reward modeling is a prerequisite for effective fine-grained policy optimization in tasks where the intermediate reasoning process is explicitly observable [2601.05688].

## 3. Mathematical Foundation

The FinePO algorithm formalizes fine-grained credit assignment as follows:

1. **Trajectory decomposition**: Each sampled response $y_i$ (from a batch of $k$ candidates for a given prompt) is decomposed into $N_i$ reasoning steps $\{s_{i,1}, ..., s_{i,N_i}\}$.
2. **Cross-trajectory advantage**:
   \[
   A(y_i) = R(y_i) - \frac{1}{k}\sum_{m=1}^{k} R(y_m)
   \]
   measures whether $y_i$ is above or below the cohort mean in final correctness.
3. **Step-level process scoring**:
   \[
   p_{i,j} = \mathcal{P}(\text{intent}_{i,j}, \text{action}_{i,j}, \text{img}_{i,j-1}, \text{img}_{i,j})
   \]
   quantifies the quality of each step.
4. **KL-based action regularization**: To discourage domination by "easy" actions, a regularizer,
   \[
   \mathcal{O}_{\mathrm{KL}}(a) = \mathrm{clip}\left(-\lambda_{KL}\log\frac{P_{\mathrm{curr}}(a) + \epsilon}{Q(a) + \epsilon}, -\gamma, \gamma \right)
   \]
   is added, where $Q(a)$ is the prior distribution, $P_{\mathrm{curr}}(a)$ the current empirical policy, and $\lambda_{KL}, \gamma, \epsilon$ are hyperparameters.
5. **Intra-trajectory normalization and deviation**:
   \[
   \bar{p}_i = \frac{\sum_{j} L_{i,j} p'_{i,j}}{\sum_{j} L_{i,j}}, \quad \Delta_{i,j} = p'_{i,j} - \bar{p}_i
   \]
   with $L_{i,j}$ the step token length.
6. **Fine-grained redistribution of advantage**:
   \[
   k_i = \frac{|A(y_i)|}{\max(\max_j \Delta_{i,j}, 0) + \epsilon}
   \]
   \[
   A'(s_{i,j}) = A(y_i) + \alpha k_i \Delta_{i,j}
   \]
   The advantage is clipped as:
   \[
   A(s_{i,j}) = \begin{cases}
   \mathrm{clip}(A'(s_{i,j}), 0, \beta A(y_i)) & A(y_i) > 0 \\
   \mathrm{clip}(A'(s_{i,j}), \beta A(y_i), 0) & A(y_i) \leq 0
   \end{cases}
   \]
   with $\alpha, \beta$ scaling and bounding per-step correction.
7. **Policy gradient update**:
   \[
   \nabla_\theta J(\theta) \approx \frac{1}{k} \sum_{i=1}^k \sum_{j=1}^{N_i} A(s_{i,j}) \nabla_\theta \log \pi_\theta(a_{i,j} \mid s_{i,<j})
   \]
   Model parameters are updated accordingly.

This formulation allows precise redistribution of cross-trajectory advantage in accordance with local step quality, achieving lower signal-to-noise in the RL signal [2601.05688].

## 4. Algorithmic Workflow and Pseudocode

The FinePO training loop for a batch of prompts consists of the following steps:

1. **Candidate sampling**: For each prompt, sample $k$ candidate responses under the current policy.
2. **Terminal reward evaluation**: Compute the final correctness reward $R(y_i)$ for each response, then the group-wise advantage $A(y_i)$.
3. **Step-level analysis**:
   - Decompose each $y_i$ into its $N_i$ explicit steps.
   - For each $s_{i,j}$:
     - Score using FinePRM: $p_{i,j}$,
     - Compute KL regularization offset: $\mathcal{O}_{\mathrm{KL}}(a_{i,j})$,
     - Aggregate: $p'_{i,j} = p_{i,j} + \mathcal{O}_{\mathrm{KL}}(a_{i,j})$.
   - Calculate mean $\bar{p}_i$, deviations $\Delta_{i,j}$, and per-step advantages $A(s_{i,j})$ as in Section 3.
4. **Loss computation**: Form the policy gradient loss $L = -\sum_{i=1}^{k}\sum_{j=1}^{N_i}A(s_{i,j})\log\pi_\theta(a_{i,j} \mid ...)$.
5. **Parameter update**: Update model weights via stochastic gradient descent.
6. **Statistics update**: Update action history for maintaining $P_{\mathrm{curr}}(a)$.

Key hyperparameters used include $k=24$ samples per prompt, $\lambda_{KL}=0.1$, $\gamma=0.5$, $\alpha=0.2$, $\beta=2.0$, learning rate $10^{-6}$, and temperature $1.0$ [2601.05688].

## 5. Empirical Findings, Ablations, and Theoretical Insights

- **Variance reduction and stability**: By attributing rewards at the step level, FinePO empirically achieves lower policy gradient variance and smoother convergence dynamics compared to trajectory-level RL approaches.
- **Ablation results**:
  - Removal of FinePO (cold start only) results in a significant drop (5–12 points) in benchmark accuracy.
  - Substituting FinePO with naive group-level RL yields 2-point lower ChartQA performance (75.12 vs 77.20).
  - Using random (non-learned) step-level rewards in FinePRM degrades performance by more than one point.
  - Excluding KL regularization induces action imbalance, though its impact on raw scores is modest.
  - Disabling the explicit sketching (externalized steps) leads to catastrophic failures (≈30% decrease in performance).
- **Comparative performance**:
  - SketchVL-7B trained with FinePO outperforms the base Qwen2.5VL-7B model across chart-based reasoning benchmarks, with overall multi-dataset average gain ≈7.23%.
  - On ChartQA: +1.96 (83.96 vs 82.00), on ChartBench: +0.33, on EvoChart-QA: +3.84.
  - Gains on general-purpose tasks (MathVista, MMStar) and for smaller models (SketchVL-3B vs Qwen2.5VL-3B: +15.32 on ChartQA).

These results indicate that fine-grained credit assignment yields consistent improvements over both pure supervised imitation learning and traditional RL approaches [2601.05688].

## 6. Integration into Model Training and Distinctive Features

FinePO is integrated into the broader SketchVL training regime as follows:
- **Cold-start phase**: The model first undergoes supervised "Sketch-CoT" training on 50K reasoning samples constructed to teach multi-step visual reasoning.
- **FinePO reinforcement phase**: Fine-tuning is performed with the FinePO loop on 9K diverse prompts, with decoder LoRA adapters frozen and KL-based action regularization applied to maintain action diversity.
- **Distinctive properties**:
  - Supports explicit, visible step decomposition for per-step reward assignment—a prerequisite in domains like chart understanding where intermediate reasoning can be externally annotated and validated.
  - Policy regularization via KL-divergence to match a prior action distribution, promoting action diversity and discouraging overuse of trivial operations.
  - Generates on-the-fly, token-level advantage tensors rather than broadcasting a scalar across the entire output.

Practical implementation uses distributed action history tracking and integration with the ms-swift framework for scalable RL fine-tuning [2601.05688].

## 7. Context, Practical Guidance, and Implications

FinePO's demonstration, particularly within the SketchVL chart reasoning framework, establishes its practical utility in settings requiring nuanced, step-level credit assignment. The empirical results suggest that:
- FinePO is most beneficial when step-level annotation or evaluation is feasible.
- The learned FinePRM is critical; using random or constant process reward quickly degrades performance.
- KL regularization mainly addresses the action distribution collapse, especially with increasing training steps.
- FinePO’s benefits are consistent across both large (7B) and small (3B) model variants, with larger proportional gains for smaller models.

A plausible implication is that the general principle of explicit intra-trajectory reward redistribution is likely transferable to other structured reasoning and policy optimization domains, whenever stepwise decomposition and step-level reward models are viable [2601.05688].

Source: https://www.emergentmind.com/topics/finepo-algorithm