---
title: 'Pre-GRPO: Baseline Initialization for GRPO'
url: https://www.emergentmind.com/topics/pre-grpo
type: topic
---

# Pre-GRPO: Baseline Initialization for GRPO

Pre-GRPO refers to the collection of initialization and baseline optimization methods employed before the application of Group Relative Policy Optimization (GRPO) in preference-based or reinforcement learning contexts. Across the literature, pre-GRPO regimes are essential for ensuring stable and effective GRPO, serving as the foundation upon which relative or groupwise policy refinement can proceed. The precise nature of “pre-GRPO” depends on the domain and research problem, but generally encompasses either supervised fine-tuning, maximum-likelihood (cross-entropy) training, or reward-free preference optimization, all prior to the introduction of the groupwise comparative structure central to GRPO.

## 1. Conceptual Overview and Purpose

Pre-GRPO denotes the initial phase in multi-stage optimization pipelines where the model is equipped with sufficient baseline capability or policy competence before groupwise relative optimization begins. Pre-GRPO serves several critical purposes:
- It lifts initial policy accuracy to a regime where the model can produce outputs sufficiently well-formed to support relative comparisons, i.e., so groupwise rewards are nontrivial and gradients do not vanish.
- It provides a reference policy (typically the model at the end of pre-GRPO) against which the GRPO loss is defined in KL-regularized settings.
- In reward-free RLHF, pre-GRPO typically refers to Direct Preference Optimization (DPO), providing a globally-aligned baseline before robust group-specific optimization.

Key metrics and empirical ablations repeatedly show that omitting pre-GRPO leads to severe performance degradation or optimization collapse in GRPO stages [2509.16889].

## 2. Typical Pre-GRPO Procedures by Domain

Pre-GRPO instantiations reflect the requirements of the downstream GRPO stage:

| Domain                              | Pre-GRPO Procedure                                    | Purpose                                                                                  |
|--------------------------------------|------------------------------------------------------|------------------------------------------------------------------------------------------|
| Multimodal table reasoning [2509.16889]   | Supervised fine-tuning (SFT) on perception/reasoning | Increase initial policy accuracy ($p$) to raise reward variance in GRPO                  |
| Spatial planning (AlphaMaze) [2502.14669] | SFT via cross-entropy on tokenized sequence          | Teach step-by-step movement and maze representation for robust GRPO rollouts             |
| Reward-free RLHF [2405.20304]            | Direct Preference Optimization (DPO)                 | Learn average/global preferences without group specificity; robustifies initial alignment|
| Text-to-image RL (Pref-GRPO) [2508.20751]| Maximum-likelihood or SFT followed by standard RL    | Provide a valid prior for policy; avoids collapse/degeneracy in comparative optimization |

In all cases, the pre-GRPO objective is maximum likelihood or preference-based, rather than groupwise-relative.

## 3. Mathematical Formulations and Optimization

Specific pre-GRPO optimization objectives take the following representative forms:

- **Supervised Fine-Tuning (Table Reasoning, AlphaMaze):**
  Given input ($I$, $Q$), and target sequence $S=(s_1,...,s_T)$, optimize
  $$
  L_{\text{warm-up}}(\theta) = - \mathbb{E}_{(I,Q,S)} \sum_{t=1}^T \log \pi_\theta(s_t | s_{<t}, I, Q)
  $$
  This cross-entropy phase is run for a predefined number of steps with AdamW and moderate learning rates, typically with dataset filtering for sequence/image size and batch size adjustments to fit accelerator memory [2509.16889], [2502.14669].

- **Direct Preference Optimization (DPO, reward-free RLHF):**
  Given preference triplets $(x, y_w, y_\ell)$ and reference $\pi_{\mathrm{ref}}$:
  $$
  L(\theta) = -\mathbb{E}_{(x, y_w, y_\ell)\sim D} \log\left( \sigma\left(\beta\,h_{\pi_\theta}(x, y_w, y_\ell)\right)\right)
  $$
  with
  $$
  h_\pi(x, y_w, y_\ell) = \log\frac{\pi(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} - \log\frac{\pi(y_\ell|x)}{\pi_{\mathrm{ref}}(y_\ell|x)}
  $$
  This convex minimization ensures a well-initialized policy and global optimality before transitioning to group-robust GRPO [2405.20304].

## 4. Empirical Role and Necessity of Pre-GRPO

Pre-GRPO is empirically necessary to overcome two critical issues:
- **Reward Variance Collapse:** In preference-based RL (e.g., table understanding), if initial policy accuracy is too low ($p\approx0$) or too high ($p\approx1$), the variance $\operatorname{Var}[R]=p(1-p)$ in groupwise rewards collapses and advantages vanish, leading to nearly zero informative policy gradients. The warm-up phase positions $p$ near $0.5$, maximizing variance and enabling effective GRPO [2509.16889].
- **Alignment and Skill Retention:** For complex reasoning, starting from SFT or DPO guarantees that the policy exhibits baseline skill in the relevant domains (e.g., stepwise spatial planning [2502.14669], or chain-of-thought math reasoning [2507.10616]), on which GRPO can amplify performance.

Ablation studies consistently show substantial performance drops (up to 40%) when pre-GRPO is omitted [2509.16889]. In AlphaMaze, the SFT-only policy already achieves 86% maze-solving accuracy, forming a platform for GRPO to further improve accuracy to 93% [2502.14669].

## 5. Theoretical Properties and Guarantees

Pre-GRPO procedures—particularly DPO—possess beneficial theoretical properties under certain policy classes:
- **Convex Landscapes in DPO:** For log-linear policies, the DPO loss is convex and has Lipschitz continuous gradients, guaranteeing global convergence and avoidance of local minima [2405.20304].
- **Initialization for KL Regularization:** In multi-stage optimization, the pre-GRPO parameters $\theta_{\mathrm{warm}}$ typically initialize both the trainable policy $\pi_\theta$ and the reference policy $\pi_{\mathrm{ref}}$ against which KL-divergence is measured in downstream GRPO stages [2509.16889]. This prevents large, destabilizing policy shifts.

## 6. Practical Configurations and Design Choices

Pre-GRPO protocols are characterized by careful selection of data, optimization schemes, and scaling choices:

- **Data Selection and Processing:** Sequence length caps (e.g., 2048 tokens for structured outputs), image resolution filters, and joint supervised SFT over multiple domains (e.g., perception and reasoning) are all employed to ensure convergence and memory fit [2509.16889].
- **Optimizer and Scheduling:** AdamW with standard hyperparameters ($\beta_1 = 0.9, \beta_2 = 0.999$), prescribed weight decay, and learning-rate warm-ups are commonly used. Epoch and batch-size are selected with respect to domain compute constraints [2502.14669].
- **Reference Policy Inclusion:** The pre-GRPO model state anchors the reference for later comparative policy optimization (e.g., DPO $\rightarrow$ GRPO, or warm-up $\rightarrow$ PA-GRPO/HC-GRPO).
- **No Groupwise Structure:** Pre-GRPO is always non-groupwise; all preference or supervised labels are averaged, with no explicit robustness or subpopulation weighting [2405.20304].

## 7. Limitations of Pre-GRPO and Transition to Robust Groupwise Optimization

Pre-GRPO cannot guarantee equitable alignment across heterogeneous subpopulations or sub-tasks:
- **One-Size-Fits-All Limitation:** In preference optimization, the DPO objective averages all labeler preferences, so the induced policy may be aligned only with the majority (“global average”) and may leave minority subgroups under-optimized ("misalignment") [2405.20304].
- **No Worst-Case Control:** There is no mechanism in pre-GRPO for optimizing or bounding worst-case group loss.

This motivates the transition to groupwise or robust preference optimization (GRPO), which directly targets such limitations by adaptively reweighting subgroups or maximizing minimum group reward.

---

In summary, pre-GRPO represents the set of globally averaged, reference-establishing training procedures that precede group-relative or robust preference optimization. These initialization stages are not only empirically critical for successful downstream GRPO, but also mathematically well-behaved and computationally efficient. Their fundamental limitation—lack of subgroup robustness or worst-case guarantees—directly motivates the need for GRPO approaches in modern RLHF and large-model optimization pipelines.

Source: https://www.emergentmind.com/topics/pre-grpo