---
title: 'GRLO: Generalizable RL in Open-Ended Environments'
url: https://www.emergentmind.com/topics/grlo
type: topic
---

# GRLO: Generalizable RL in Open-Ended Environments

Searching arXiv for recent papers on “GRLO” and closely related GRPO usage to ground the article.
GRLO, short for **Generalizable Reinforcement Learning in Open-Ended Environments from Zero**, denotes a post-training strategy in which **RLHF-style PPO** is applied directly from a **base model** on a **small, diverse, open-ended prompt set**, rather than on a narrow verifier-backed domain. It was introduced to test whether reinforcement learning in open-ended environments can unlock general conversational capabilities that then transfer to downstream mathematical reasoning and code generation. On **Qwen3-4B-Base**, GRLO raises the average across **Math500**, **GPQA**, **HumanEval**, **MBPP**, and **AlpacaEval 2 Length-Controlled Win Rate** from **24.1** to **63.1** with only **5K prompts** and **22.7 GPU hours**, using approximately **46x less data** and about **68x less compute** than a strong in-domain RLVR baseline [2605.15464].

## 1. Conceptual position

GRLO is framed against a split in contemporary RL-based post-training between **RLHF** and **RLVR**. In the paper’s formulation, standard in-domain RLHF optimizes a policy on prompts from a target-domain distribution \(D_{\text{in}}\) with a learned reward model \(r_\phi(x,y)\), while constraining drift from a frozen reference policy \(\pi_{\text{ref}}\):
\[
\max_{\pi_\theta} \; \mathbb{E}_{x \sim D_{\text{in}},\, y \sim \pi_\theta(\cdot|x)} \left[r_\phi(x,y)\right] - \beta \, \mathrm{KL}\!\left(\pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\right).
\]

For verifier-backed domains, the same paper describes RLVR through a representative **GRPO-style** grouped objective over prompts \(x \sim D_{\text{ver}}\):
\[
\max_{\pi_\theta} \; \mathbb{E}_{x \sim D_{\text{ver}},\, \{y_i\}_{i=1}^{G} \sim \pi_\theta} \left[ \sum_{i=1}^{G} \frac{1}{G}\,\hat{A}_i \log \pi_\theta(y_i|x) \right],
\]
with normalized groupwise advantage
\[
\hat{A}_i = \frac{ r_{\text{ver}}(x,y_i) - \frac{1}{G}\sum_{j=1}^{G} r_{\text{ver}}(x,y_j) }{ \mathrm{std}\big(\{r_{\text{ver}}(x,y_j)\}_{j=1}^{G}\big) + \epsilon }.
\]

GRLO keeps the **RLHF-style** structure, but changes the training environment. Its defining objective is
\[
\max_{\pi_\theta} \; \mathbb{E}_{x \sim D_{\text{open}},\, y \sim \pi_\theta(\cdot|x)} \left[r_\phi(x,y)\right] - \beta \, \mathrm{KL}\!\left(\pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\right),
\]
where \(D_{\text{open}}\) is an **open-ended prompt distribution** rather than an in-domain reasoning or coding distribution. The novelty is therefore not a new optimizer, but a deliberate shift in **where** RL is applied [2605.15464].

## 2. Training recipe and optimization stack

The training recipe is intentionally lightweight. The flagship setup starts from **Qwen3-4B-Base** and runs PPO in **Verl** for **15 epochs** with actor learning rate **\(1\times10^{-6}\)**, critic learning rate **\(1\times10^{-5}\)**, batch size **1024**, maximum prompt length **1024 tokens**, and maximum response length **3072 tokens**. The reward model is **Skywork/Skywork-Reward-V2-Llama-3.1-8B**. The paper does not describe training a custom reward model; instead it uses this external reward model as the preference-style scoring function over open-ended responses [2605.15464].

Operationally, the loop is: start from a pretrained base model \(\pi_\theta\), freeze a reference copy \(\pi_{\text{ref}}\), sample prompts \(x\) from the open-ended pool \(D_{\text{open}}\), generate responses \(y \sim \pi_\theta(\cdot|x)\), score them with \(r_\phi(x,y)\), and optimize the policy with PPO under KL regularization toward \(\pi_{\text{ref}}\). The paper does not provide the full PPO clipped surrogate formula, value loss, or GAE equations, and it does not specify the exact KL coefficient \(\beta\).

“From Zero” means that GRLO is run **directly from the base model**, without a preceding task-specific RL stage. A notable exception is **Llama3.2-3B**: because that base model lacks conversational ability, the paper first applies a **brief cold-start SFT with 5K UltraFeedback examples**, then treats that SFT checkpoint as the backbone [2605.15464].

## 3. Open-ended environment and transfer hypothesis

The GRLO environment consists of roughly **5K curated synthetic prompts**. The prompt pool is described as deliberately **open-ended rather than math-centric**, including **scientific analysis**, **humanities/argumentative synthesis**, **conceptual explanation**, **long-form reasoning**, **policy/public finance**, **cybersecurity**, **embedded systems**, **labor economics**, and **animal welfare and ethics**. A heuristic topic audit gives the following distribution: **policy/history 34.8%**, **biomedicine/health 15.7%**, **technology/engineering 15.6%**, **environment/earth systems 14.7%**, **humanities/culture 9.3%**, and **general analysis 9.8%** [2605.15464].

The paper’s central empirical claim is that open-ended RL can improve not only chat, but also reasoning and code. It attributes this to strong pretrained base models already containing latent reasoning and coding capabilities from pretraining, with RL reshaping the output distribution so the model better uses those capabilities. This interpretation is presented as empirical rather than mechanistic.

Three observations are emphasized. First, broad gains appear jointly rather than only on chat. Second, most gains arrive early: when the open-ended data size increases from **0** to **1K prompts**, **Math500** rises from **73.6** to **76.6**, **MBPP** from **0.25** to **62.91**, and **AE2 LC** from **1.6** to **30.2**. Third, the effect is not explained by verbosity. On **Math500**, average output length drops from **933** tokens for **General-Reasoner** to **672** tokens for **GRLO** while quality improves; on **GPQA**, GRLO is also shorter than both base and General-Reasoner. This suggests, in the paper’s phrasing, better **decision quality** and **response organization** rather than merely longer chain-of-thought [2605.15464].

The prompt-source analysis reinforces this interpretation. A variant trained on **5K UltraFeedback prompts** reaches **Math500 78.4**, **GPQA 48.5**, **HumanEval 85.4**, **MBPP 55.4**, **AE2 LC 54.2**, and **Avg 64.4**, which is very close to, and slightly above, the default GRLO average of **63.1**. This suggests that the broad-transfer effect is not tied to one exact dataset source [2605.15464].

## 4. Empirical profile

The main evaluation suite contains **Math500**, **GPQA**, **HumanEval**, **MBPP**, and **AlpacaEval 2 Length-Controlled Win Rate**, with the paper reporting the simple average across these five benchmarks. On **Qwen3-4B-Base**, the per-benchmark trajectory is: **Math500 73.6 \(\to\) 79.2**, **GPQA 38.9 \(\to\) 47.0**, **HumanEval 6.1 \(\to\) 84.8**, **MBPP 0.3 \(\to\) 58.9**, **AE2 LC 1.6 \(\to\) 45.8**, and **Average 24.1 \(\to\) 63.1** after GRLO [2605.15464].

| System | Average | Context |
|---|---:|---|
| Qwen3-4B-Base | 24.1 | Base model |
| Qwen3-4B-GRLO | 63.1 | 5K prompts, 22.7 GPU hours |
| General-Reasoner-4B | 60.6 | 230K training examples |
| Qwen3-4B-GRLO+RLVR | 63.2 | Subsequent in-domain RLVR |
| Qwen3-4B (Non-thinking) | 64.0 | Released official post-trained model |

The comparison to **General-Reasoner-4B** is central. General-Reasoner reports **Math500 77.0**, **GPQA 50.0**, **HumanEval 84.8**, **MBPP 60.2**, **AE2 LC 30.8**, and **Avg 60.6**, whereas GRLO reports **79.2**, **47.0**, **84.8**, **58.9**, **45.8**, and **63.1**. The most pronounced GRLO advantage is on **AE2 LC: 45.8 vs 30.8**, while General-Reasoner is slightly ahead on **GPQA** and **MBPP**. The paper emphasizes that GRLO achieves this with **5K prompts** and **22.7 GPU hours**, compared with **230K training examples** and approximately **68x more compute** for the strong broad-domain baseline [2605.15464].

The method also transfers across backbones. On **Qwen3-8B**, average scores are **Base 33.7**, **Official non-thinking 58.2**, **MathSFT 43.0**, **RLVR baseline 50.6**, **GRLO 67.3**, and **GRLO+RLVR 68.1**. On **Qwen2.5-3B**, the reported averages are **Base 38.1**, **Instruct 48.4**, **MathSFT 25.6**, **RLVR 45.3**, **GRLO 50.7**, and **GRLO+RLVR 49.0**. On **Llama3.2-3B**, using the **5K UltraFeedback SFT** checkpoint as backbone, the numbers are **21.5**, **35.6**, **30.7**, **39.3**, and **40.7** for the SFT backbone, Instruct, RLVR, GRLO, and GRLO+RLVR respectively [2605.15464].

The hard-math analysis is more selective. For **Qwen3-4B-GRLO**, the paper reports **OlympiadBench 43.4**, **AIME24 10.0**, **AIME25 3.3**, **Minerva 42.6**, and hard-math average **24.9**. For **Qwen3-4B-GRLO+RLVR**, the numbers become **42.7**, **16.7**, **10.0**, **41.5**, and **27.7**. The interpretation is that GRLO produces the broad shift, while later RLVR helps mainly on harder competition-math benchmarks [2605.15464].

## 5. Relation to RLVR, PPO, and GRPO

The paper’s most informative ablations compare environment choice and optimizer choice. Replacing the open-ended environment with **math-only in-domain prompts**, while keeping PPO, yields **AE2 LC 10.15**, **Math500 77.20**, and **GPQA 41.92**. Default open-ended PPO-based GRLO yields **AE2 LC 45.83**, **Math500 79.20**, and **GPQA 47.00**. The paper uses this to argue that the **open-ended environment matters more than simply doing RL at all** [2605.15464].

Optimizer choice matters as well. Using **open-ended data with GRPO** instead of PPO yields **AE2 LC 34.89**, **Math500 77.80**, and **GPQA 40.40**. This is better than base, but weaker than default PPO-based GRLO. In the reported setup, **open-ended PPO is best** [2605.15464].

The follow-up RLVR stage is therefore treated as complementary rather than primary. On **Qwen3-4B**, **GRLO 63.1** becomes **GRLO+RLVR 63.2** on the five-benchmark average, so the broad score changes little. The later RLVR stage produces more visible movement on hard-math metrics, particularly **AIME24** and pass@k. Averaged over **AIME24** and **AIME25**, the paper reports **General-Reasoner pass@8 10.0%** and **pass@16 13.3%**, **GRLO pass@8 31.7%** and **pass@16 35.0%**, and **GRLO+RLVR pass@16 40.0%**. This suggests that open-ended PPO delivers the main broad-domain shift, while in-domain RLVR acts as a specialization stage [2605.15464].

Within the broader arXiv literature, this also distinguishes GRLO from **GRPO** itself. GRPO is a grouped relative-reward optimizer used in verifier-backed or reward-scored settings such as **zero-shot TTS** [2509.18798], **visual generation** [2505.07818], and **autoregressive image models** [2603.23086]. GRLO, by contrast, names an **open-ended PPO-based post-training recipe** rather than a generic synonym for group-relative optimization [2605.15464].

## 6. Scope, limitations, and term usage

The paper does not present a dedicated limitations section, but several constraints are explicit or directly implied. The empirical case for transfer is demonstrated rather than theoretically explained. The main suite is five benchmarks plus a handful of harder math sets. The tested model scales are **3B**, **4B**, and **8B**. Environment choice remains important even though the **UltraFeedback 5K** variant shows prompt-source robustness. Safety and alignment tradeoffs are not deeply studied. Reproduction details remain incomplete because some practical PPO settings are omitted, including the exact **KL coefficient**, PPO clipping, and value-loss details [2605.15464].

A recurring source of confusion is acronym overlap. In current arXiv usage, **GRLO** explicitly names **“Generalizable Reinforcement Learning in Open-Ended Environments from Zero”** [2605.15464]. By contrast, **“Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study”** [2605.07366] studies gradient-based LoRA rank allocation under **GRPO**, and **“GRASP LoRA: GRPO Guided Adapter Sparsity Policy for Cross Lingual Transfer”** [2601.06702] uses a GRPO-style controller to learn a global prune ratio. Other papers treat “GRLO” as an alternate or mistaken query for **GRPO**-related work rather than as the name of the open-ended RL method itself [2509.18798]. This suggests that, in technical usage, GRLO should be read as a specific **open-ended RLHF-style PPO recipe**, not as a generic label for all group-relative RL methods.

The central takeaway is therefore narrow and precise. GRLO does not claim a new RL objective; it claims that **RLHF-style PPO directly from a base model on a small, diverse, open-ended prompt pool** can recover a broad post-training capability profile across **chat**, **reasoning**, and **code**, and can do so with markedly lower data and compute than a strong in-domain RLVR baseline. Subsequent RLVR remains useful, but mainly as a selective specialization stage rather than the main source of broad transfer [2605.15464].

Source: https://www.emergentmind.com/topics/grlo