---
title: R1-Style Finetuning Process for Multimodal Models
url: https://www.emergentmind.com/topics/r1-style-finetuning-process
type: topic
---

# R1-Style Finetuning Process for Multimodal Models

R1-Style Finetuning Process

R1-style finetuning is a reinforcement learning (RL)-based procedure designed to efficiently elicit advanced reasoning capabilities, structured tool use, and robust generalization in large language, vision-language, and multi-modal models. Core to this paradigm is the use of rule-based, deterministic rewards derived directly from ground-truth annotations or explicit task criteria, coupled with Group Relative Policy Optimization (GRPO)—a variant of proximal policy optimization (PPO) that normalizes policy gradients within a group of sampled rollouts to stabilize credit assignment. R1-style pipelines have become central in scientific advances across multimodal reasoning, spatial understanding, action planning, vision-based manipulation, and auto-formatted stepwise chain-of-thought generation.

## 1. General Principles and Motivation

R1-style finetuning extends earlier language-model reinforcement learning (as in DeepSeek-R1) to complex multimodal settings. The defining characteristics include:

- **Rule-based reward models**: Instead of trained reward networks or human preference models, R1 applies deterministic reward functions directly grounded in task logic (e.g., exact match to ground-truth answer, numeric error tolerance, spatial IoU, etc.).
- **Group Relative Policy Optimization (GRPO)**: For each data point, multiple candidate outputs are sampled, evaluated, and their rewards normalized into group-wise advantages, which are then used to form clipped policy gradient estimators with a KL-regularization term to a reference policy.
- **Outcome-centric supervision**: Rewards are assigned on final outcomes (e.g., task accuracy, format compliance) rather than process steps or tool usage policies, minimizing reward hacking pathways.
- **Curriculum integration**: R1-style training typically follows or is interleaved with supervised fine-tuning (SFT) on high-quality seed data to anchor model syntax and feasibility, before RL-based optimization for strategy discovery and reward maximization.

This design aligns with findings that SFT alone may seed valid structure but is fundamentally limited by sample effect and exploration, while R1-style RL, particularly when initialized from SFT, can efficiently yield emergent reasoning and tool-use behaviors that generalize [2505.19255][2505.17988][2501.12948][2504.07615].

## 2. Formal Algorithmic Structure and Policy Objectives

The backbone of R1-style finetuning is the GRPO algorithm with per-group advantage normalization and PPO-style clipping. The canonical objective is (using multimodal notation):

\[
\max_{\theta}\;
\mathbb{E}_{[x] \sim \mathcal D,\, y \sim \pi_\theta(\cdot|x)}
\left[\, r_\phi(x, y) \,\right]
-\beta\, D_{KL}\!\left[\pi_\theta(\cdot | x) || \pi_\mathrm{ref}(\cdot | x)\right]
\]

For each group of $G$ sampled rollouts $\{y_i\}_{i=1}^{G}$, with rewards $r_i$, the normalized advantage is

\[
\hat{A}_i = \frac{r_i - \mathrm{mean}_j r_j}{\mathrm{std}_j r_j}
\]

And the clipped GRPO surrogate is

\[
\mathcal J_{\rm GRPO}(\theta) =
\mathbb{E}_{\{y_i\} \sim \pi_\mathrm{old}}
\left[
\frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|}
\min\left( r_{i,t}(\theta)\, \hat{A}_i, \mathrm{clip}(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon)\, \hat{A}_i \right)
- \beta\, D_{KL}[\pi_\theta \| \pi_\mathrm{ref}]
\right]
\]

where $r_{i,t}(\theta)$ is the token-level likelihood ratio between $\pi_\theta$ and the reference policy.

Pseudocode for the main R1-style loop is as follows [2505.19255]:

```
Algorithm VTool-R1 Finetuning Loop

Input: pretrained VLM π_ref, toolset T, dataset 𝒟, group size G, hyperparams ε, β, lr α
Initialize π_θ ← π_ref
repeat:
  1. Sample minibatch {[I^n, x^n]} from 𝒟
  2. For each example n:
    a. Sample G rollouts {y_i^n} from π_old:
      i. First pass: y′_i ∼ π_old(·|I^n, x^n)
      ii. If y′_i contains tool call, execute I′_i = T(y′_i, I^n); else I′_i=∅
      iii. Second pass: y_i ∼ π_old(·|I^n⊕I′_i, x^n; T)
    b. Evaluate rewards r_i^n = r_ϕ(I^n, x^n, y_i^n)
  3. Normalize {r_i^n} to get advantages ĤA_i^n
  4. Compute gradient ∇_θ 𝓙_GRPO(θ)
  5. Update θ ← θ + α·∇_θ 𝓙_GRPO(θ)
  6. Occasionally sync π_old ← π_θ
until convergence
```

The architecture is agnostic: language-only models, VLMs, or even dense visual representation transformers can all be optimized under this formalism by suitable definition of states, actions, and rewards [2505.19255][2505.24025][2508.09804][2506.18234].

## 3. Reward Engineering and Environment Design

Crucial to R1-style RL is the engineering of verifiable, deterministic reward signals:

- **Task Accuracy**: Binary or scalar, compares model output to ground-truth. E.g., VTool-R1 uses $r_\phi(I,x,y) = 1$ if the final answer matches reference, $0$ otherwise [2505.19255]; Smart-R1 uses the official RealismMeta metric for traffic trajectory rollouts [2509.23993].
- **Format Validation**: Many tasks require strict syntax (e.g., JSON, XML tags, <think>/<answer> block delimitation); format rewards are binary or soft and can be composed with main task accuracy [2504.07615][2506.01371][2507.15509].
- **Numerical Sensitivity**: For chart and visual QA, rewards may softly grade near-miss numeric predictions, e.g., $r_{\text{num}}(p,a) = \max(0, 1 - \min(\delta, 1))$ with $\delta = |p-a|/(0.05 |a|)$ for 5% tolerance [2507.15509].
- **Semantic Similarity**: Some systems quantify answer correctness via sentence-BERT or LLM-based similarity for open-ended or partially structured answers [2506.01371].
- **Spatial/Consistency Rewards**: SVQA-R1 adjusts rewards to penalize failures of spatial symmetry by comparing performance on original and mirror-flipped examples, enhancing spatial grounding [2506.01371].
- **Length or Repetition Regularization**: Encouraging chain-of-thought brevity, logical completeness, and discouraging token repetitions via explicit reward terms [2510.00072][2506.13654].
- **Tool Use and Multi-step Environment**: For tool-augmented models (VTool-R1, Ego-R1), outputs can contain callable code blocks whose effects are post-processed and fed back via the environment for multi-pass interaction [2505.19255][2506.13654].

Outcome-based, rather than process-based, reward is typically emphasized, as process shaping rewards tend to be brittle or encourage reward hacking [2505.19255][2504.07615].

## 4. Multi-stage Training Pipelines and Curricula

R1-style pipelines are almost always multi-phase:

1. **Supervised Fine-Tuning (SFT) Stage**  
   - Trains the model to obey stepwise "chain-of-thought" syntax, form output structure, and ground generation in data distributions via cross-entropy on seed datasets.
   - Typical datasets range from 5 k (DeepSeek-R1 cold-start) to millions of triplets (BigCharts, SMART-R1).
   - After SFT, models produce syntactically valid, interpretable outputs but lack robustness and cannot recover from new-error states [2501.12948][2505.19255][2508.09804][2506.18234].

2. **R1-style RL (GRPO) Stage**  
   - Policy is initialized to the SFT checkpoint and updated to maximize outcome-based rewards.
   - RL is run for 1–3 epochs over held-out data, using current policy to sample groups of outputs, evaluate verifiable rewards, normalize, and apply GRPO update with KL penalty.
   - For certain domains (e.g., SMART-R1 for multi-agent traffic), an alternation SFT→RFT→SFT schedule is empirically optimal, with post-RFT SFT restoring fidelity while preserving the exploration benefits of RL [2509.23993].

3. **Evaluation**  
   - Decoding is performed at temperature 1.0, typically with top-k or nucleus sampling; tool calls are executed "in the loop" and chain-of-thought structure is enforced by the prompt and output parser.
   - Task accuracy, generalization (OOD), and reasoning chain length are all tracked as key indicators.

This curriculum ensures syntactic control, enables efficient RL convergence, and preserves overall capabilities and stability [2501.12948][2504.07615][2503.18013].

## 5. Architectural, Implementation, and Hyperparameter Best Practices

R1-style pipelines require modest or no architecture modification beyond standard VLM or LLM backbones:

- **Frozen Vision and Lightweight Adapters**: For stability, vision towers are often frozen, with LoRA modules attached to the language decoders [2504.07615].
- **No Critic Networks**: GRPO is strictly actor-only; value baselines are estimated intra-group by mean reward normalization [2505.19255][2508.09804][2507.15509].
- **Ablation-Proven Hyperparameters**:
  - Learning rates: $10^{-6}$ (RL phase), $10^{-5}$ (SFT phase).
  - Group sizes: Typical $G=4$ to $8$ per prompt.
  - KL-penalty ($\beta$): $0.01-0.1$ (tuned by grid search or inherited from prior art).
  - PPO clip range: $\epsilon=0.1-0.2$.
  - Optimizers: AdamW, with weight decay $1e-2$.
  - Gradient clipping: $1.0$ (global norm) to reduce GRPO-induced variance [2504.07615][2505.19255][2505.24025].
- **Batch/hardware**: Scaling is typically bottlenecked on long context and image input sizes, with micro-batch sizes primary limited by 16 k token windows and large vision feature maps.
- **Platform Agnostic Action Spaces**: GUI and mobile agent R1-style pipelines employ unified action-space codings, supporting cross-device and OS generalization [2504.10458][2506.20332].

## 6. Empirical Effects, Generalization, and Limitations

R1-style fine-tuning consistently yields state-of-the-art accuracy on reasoning, spatial, or grounded tasks and enables the emergence of multimodal chain-of-thought behaviors. Key empirical effects:

- **Emergence of "aha moments"**: Models transition from simple output enumeration to logical, multi-step reasoning (documented for object detection and chart reasoning) [2504.07615][2508.09804][2507.15509].
- **Generalization and Robustness**: Out-of-distribution (OOD) performance for VQA, chart QA, and spatial tasks is significantly improved compared to SFT-only methods [2508.09804][2506.01371].
- **Reward Hacking Mitigation**: Proper reward design (e.g., output length penalties, soft matching for numerics, KL regularization) is shown to be essential to prevent degenerate solutions [2504.07615][2507.15509].
- **Sample Efficiency**: Theoretical and empirical analysis indicates that SFT alone is bottlenecked by the sample effect of its data, but re-distillation of RL-enhanced outputs matches RL performance at a fraction of cost, provided that high-value-for-learning samples are prioritized [2505.17988].
- **Stability**: SFT seeds output syntax, and KL+PPO-style clipping ensure R1-style RL converges stably even in high-variance multimodal settings [2505.19255][2504.07615].

A plausible implication is that R1-style finetuning, when initialized from high-quality SFT and coupled to precise, rule-based reward design, can both replicate and exceed the capabilities previously achievable only through massive-scale, human-preference RLHF, but in largely deterministic and highly reproducible fashion.

---

**References**:  
- "VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use" [2505.19255]  
- "BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning" [2508.09804]  
- "DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models" [2505.24025]  
- "Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning" [2503.18013]  
- "Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasoner" [2507.15509]  
- "Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning" [2506.18234]  
- "Mobile-R1: Towards Interactive Reinforcement Learning for VLM-Based Mobile Agent via Task-Level Rewards" [2506.20332]  
- "Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning" [2505.17988]  
- "VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model" [2504.07615]  
- "Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?" [2505.09439]  
- "GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents" [2504.10458]  
- "Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning" [2506.13654]  
- "Geo-R1: Unlocking VLM Geospatial Reasoning with Cross-View Reinforcement Learning" [2510.00072]  
- "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" [2501.12948]  
- "Advancing Multi-agent Traffic Simulation via R1-Style Reinforcement Fine-Tuning" [2509.23993]  
- "SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimization" [2506.01371]

Source: https://www.emergentmind.com/topics/r1-style-finetuning-process