GRLO: Generalizable RL in Open-Ended Environments

• GRLO is a post-training strategy that applies RLHF-style PPO on a small, diverse, open-ended prompt set to unlock general conversational and reasoning capabilities.
• It leverages a lightweight training recipe on base models, achieving significant improvements such as raising average benchmark scores from 24.1 to 63.1 with only 5K prompts and 22.7 GPU hours.
• The method efficiently transfers across tasks like chat, math reasoning, and code generation, using approximately 46× less data and 68× less compute than traditional in-domain RLVR baselines.

Searching arXiv for papers on “GRLO” and closely related GRPO usage to ground the article. GRLO, short for Generalizable Reinforcement Learning in Open-Ended Environments from Zero, denotes a post-training strategy in which RLHF-style PPO is applied directly from a base model on a small, diverse, open-ended prompt set, rather than on a narrow verifier-backed domain. It was introduced to test whether reinforcement learning in open-ended environments can unlock general conversational capabilities that then transfer to downstream mathematical reasoning and code generation. On Qwen3-4B-Base, GRLO raises the average across Math500, GPQA, HumanEval, MBPP, and AlpacaEval 2 Length-Controlled Win Rate from 24.1 to 63.1 with only 5K prompts and 22.7 GPU hours, using approximately 46x less data and about 68x less compute than a strong in-domain RLVR baseline (Yin et al., 14 May 2026).

1. Conceptual position

GRLO is framed against a split in contemporary RL-based post-training between RLHF and RLVR. In the paper’s formulation, standard in-domain RLHF optimizes a policy on prompts from a target-domain distribution $D_{\text{in}}$ with a learned reward model $r_\phi(x,y)$, while constraining drift from a frozen reference policy $\pi_{\text{ref}}$: $$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D_{\text{in}},\, y \sim \pi_\theta(\cdot|x)} \left[r_\phi(x,y)\right] - \beta \, \mathrm{KL}\!\left(\pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\right).$$

For verifier-backed domains, the same paper describes RLVR through a representative GRPO-style grouped objective over prompts $x \sim D_{\text{ver}}$: $$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D_{\text{ver}},\, \{y_i\}_{i=1}^{G} \sim \pi_\theta} \left[ \sum_{i=1}^{G} \frac{1}{G}\,\hat{A}_i \log \pi_\theta(y_i|x) \right],$$ with normalized groupwise advantage

$$\hat{A}_i = \frac{ r_{\text{ver}}(x,y_i) - \frac{1}{G}\sum_{j=1}^{G} r_{\text{ver}}(x,y_j) }{ \mathrm{std}\big(\{r_{\text{ver}}(x,y_j)\}_{j=1}^{G}\big) + \epsilon }.$$

GRLO keeps the RLHF-style structure, but changes the training environment. Its defining objective is

$$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D_{\text{open}},\, y \sim \pi_\theta(\cdot|x)} \left[r_\phi(x,y)\right] - \beta \, \mathrm{KL}\!\left(\pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\right),$$

where $D_{\text{open}}$ is an open-ended prompt distribution rather than an in-domain reasoning or coding distribution. The novelty is therefore not a new optimizer, but a deliberate shift in where RL is applied (Yin et al., 14 May 2026).

2. Training recipe and optimization stack

The training recipe is intentionally lightweight. The flagship setup starts from Qwen3-4B-Base and runs PPO in Verl for 15 epochs with actor learning rate $1\times10^{-6}$, critic learning rate $r_\phi(x,y)$0, batch size 1024, maximum prompt length 1024 tokens, and maximum response length 3072 tokens. The reward model is Skywork/Skywork-Reward-V2-Llama-3.1-8B. The paper does not describe training a custom reward model; instead it uses this external reward model as the preference-style scoring function over open-ended responses (Yin et al., 14 May 2026).

Operationally, the loop is: start from a pretrained base model $r_\phi(x,y)$1, freeze a reference copy $r_\phi(x,y)$2, sample prompts $r_\phi(x,y)$3 from the open-ended pool $r_\phi(x,y)$4, generate responses $r_\phi(x,y)$5, score them with $r_\phi(x,y)$6, and optimize the policy with PPO under KL regularization toward $r_\phi(x,y)$7. The paper does not provide the full PPO clipped surrogate formula, value loss, or GAE equations, and it does not specify the exact KL coefficient $r_\phi(x,y)$8.

“From Zero” means that GRLO is run directly from the base model, without a preceding task-specific RL stage. A notable exception is Llama3.2-3B: because that base model lacks conversational ability, the paper first applies a brief cold-start SFT with 5K UltraFeedback examples, then treats that SFT checkpoint as the backbone (Yin et al., 14 May 2026).

3. Open-ended environment and transfer hypothesis

The GRLO environment consists of roughly 5K curated synthetic prompts. The prompt pool is described as deliberately open-ended rather than math-centric, including scientific analysis, humanities/argumentative synthesis, conceptual explanation, long-form reasoning, policy/public finance, cybersecurity, embedded systems, labor economics, and animal welfare and ethics. A heuristic topic audit gives the following distribution: policy/history 34.8%, biomedicine/health 15.7%, technology/engineering 15.6%, environment/earth systems 14.7%, humanities/culture 9.3%, and general analysis 9.8% (Yin et al., 14 May 2026).

The paper’s central empirical claim is that open-ended RL can improve not only chat, but also reasoning and code. It attributes this to strong pretrained base models already containing latent reasoning and coding capabilities from pretraining, with RL reshaping the output distribution so the model better uses those capabilities. This interpretation is presented as empirical rather than mechanistic.

Three observations are emphasized. First, broad gains appear jointly rather than only on chat. Second, most gains arrive early: when the open-ended data size increases from 0 to 1K prompts, Math500 rises from 73.6 to 76.6, MBPP from 0.25 to 62.91, and AE2 LC from 1.6 to 30.2. Third, the effect is not explained by verbosity. On Math500, average output length drops from 933 tokens for General-Reasoner to 672 tokens for GRLO while quality improves; on GPQA, GRLO is also shorter than both base and General-Reasoner. This suggests, in the paper’s phrasing, better decision quality and response organization rather than merely longer chain-of-thought (Yin et al., 14 May 2026).

The prompt-source analysis reinforces this interpretation. A variant trained on 5K UltraFeedback prompts reaches Math500 78.4, GPQA 48.5, HumanEval 85.4, MBPP 55.4, AE2 LC 54.2, and Avg 64.4, which is very close to, and slightly above, the default GRLO average of 63.1. This suggests that the broad-transfer effect is not tied to one exact dataset source (Yin et al., 14 May 2026).

4. Empirical profile

The main evaluation suite contains Math500, GPQA, HumanEval, MBPP, and AlpacaEval 2 Length-Controlled Win Rate, with the paper reporting the simple average across these five benchmarks. On Qwen3-4B-Base, the per-benchmark trajectory is: Math500 73.6 $r_\phi(x,y)$9 79.2, GPQA 38.9 $\pi_{\text{ref}}$0 47.0, HumanEval 6.1 $\pi_{\text{ref}}$1 84.8, MBPP 0.3 $\pi_{\text{ref}}$2 58.9, AE2 LC 1.6 $\pi_{\text{ref}}$3 45.8, and Average 24.1 $\pi_{\text{ref}}$4 63.1 after GRLO (Yin et al., 14 May 2026).

System	Average	Context
Qwen3-4B-Base	24.1	Base model
Qwen3-4B-GRLO	63.1	5K prompts, 22.7 GPU hours
General-Reasoner-4B	60.6	230K training examples
Qwen3-4B-GRLO+RLVR	63.2	Subsequent in-domain RLVR
Qwen3-4B (Non-thinking)	64.0	Released official post-trained model

The comparison to General-Reasoner-4B is central. General-Reasoner reports Math500 77.0, GPQA 50.0, HumanEval 84.8, MBPP 60.2, AE2 LC 30.8, and Avg 60.6, whereas GRLO reports 79.2, 47.0, 84.8, 58.9, 45.8, and 63.1. The most pronounced GRLO advantage is on AE2 LC: 45.8 vs 30.8, while General-Reasoner is slightly ahead on GPQA and MBPP. The paper emphasizes that GRLO achieves this with 5K prompts and 22.7 GPU hours, compared with 230K training examples and approximately 68x more compute for the strong broad-domain baseline (Yin et al., 14 May 2026).

The method also transfers across backbones. On Qwen3-8B, average scores are Base 33.7, Official non-thinking 58.2, MathSFT 43.0, RLVR baseline 50.6, GRLO 67.3, and GRLO+RLVR 68.1. On Qwen2.5-3B, the reported averages are Base 38.1, Instruct 48.4, MathSFT 25.6, RLVR 45.3, GRLO 50.7, and GRLO+RLVR 49.0. On Llama3.2-3B, using the 5K UltraFeedback SFT checkpoint as backbone, the numbers are 21.5, 35.6, 30.7, 39.3, and 40.7 for the SFT backbone, Instruct, RLVR, GRLO, and GRLO+RLVR respectively (Yin et al., 14 May 2026).

The hard-math analysis is more selective. For Qwen3-4B-GRLO, the paper reports OlympiadBench 43.4, AIME24 10.0, AIME25 3.3, Minerva 42.6, and hard-math average 24.9. For Qwen3-4B-GRLO+RLVR, the numbers become 42.7, 16.7, 10.0, 41.5, and 27.7. The interpretation is that GRLO produces the broad shift, while later RLVR helps mainly on harder competition-math benchmarks (Yin et al., 14 May 2026).

5. Relation to RLVR, PPO, and GRPO

The paper’s most informative ablations compare environment choice and optimizer choice. Replacing the open-ended environment with math-only in-domain prompts, while keeping PPO, yields AE2 LC 10.15, Math500 77.20, and GPQA 41.92. Default open-ended PPO-based GRLO yields AE2 LC 45.83, Math500 79.20, and GPQA 47.00. The paper uses this to argue that the open-ended environment matters more than simply doing RL at all (Yin et al., 14 May 2026).

Optimizer choice matters as well. Using open-ended data with GRPO instead of PPO yields AE2 LC 34.89, Math500 77.80, and GPQA 40.40. This is better than base, but weaker than default PPO-based GRLO. In the reported setup, open-ended PPO is best (Yin et al., 14 May 2026).

The follow-up RLVR stage is therefore treated as complementary rather than primary. On Qwen3-4B, GRLO 63.1 becomes GRLO+RLVR 63.2 on the five-benchmark average, so the broad score changes little. The later RLVR stage produces more visible movement on hard-math metrics, particularly AIME24 and pass@k. Averaged over AIME24 and AIME25, the paper reports General-Reasoner pass@8 10.0% and pass@16 13.3%, GRLO pass@8 31.7% and pass@16 35.0%, and GRLO+RLVR pass@16 40.0%. This suggests that open-ended PPO delivers the main broad-domain shift, while in-domain RLVR acts as a specialization stage (Yin et al., 14 May 2026).

Within the broader arXiv literature, this also distinguishes GRLO from GRPO itself. GRPO is a grouped relative-reward optimizer used in verifier-backed or reward-scored settings such as zero-shot TTS (Liu et al., 23 Sep 2025), visual generation (Xue et al., 12 May 2025), and autoregressive image models (Baran et al., 24 Mar 2026). GRLO, by contrast, names an open-ended PPO-based post-training recipe rather than a generic synonym for group-relative optimization (Yin et al., 14 May 2026).

6. Scope, limitations, and term usage

The paper does not present a dedicated limitations section, but several constraints are explicit or directly implied. The empirical case for transfer is demonstrated rather than theoretically explained. The main suite is five benchmarks plus a handful of harder math sets. The tested model scales are 3B, 4B, and 8B. Environment choice remains important even though the UltraFeedback 5K variant shows prompt-source robustness. Safety and alignment tradeoffs are not deeply studied. Reproduction details remain incomplete because some practical PPO settings are omitted, including the exact KL coefficient, PPO clipping, and value-loss details (Yin et al., 14 May 2026).

A recurring source of confusion is acronym overlap. In current arXiv usage, GRLO explicitly names “Generalizable Reinforcement Learning in Open-Ended Environments from Zero” (Yin et al., 14 May 2026). By contrast, “Gradient-Based LoRA Rank Allocation Under GRPO: An Empirical Study” (Sawant, 8 May 2026) studies gradient-based LoRA rank allocation under GRPO, and “GRASP LoRA: GRPO Guided Adapter Sparsity Policy for Cross Lingual Transfer” (Hassan et al., 10 Jan 2026) uses a GRPO-style controller to learn a global prune ratio. Other papers treat “GRLO” as an alternate or mistaken query for GRPO-related work rather than as the name of the open-ended RL method itself (Liu et al., 23 Sep 2025). This suggests that, in technical usage, GRLO should be read as a specific open-ended RLHF-style PPO recipe, not as a generic label for all group-relative RL methods.

The central takeaway is therefore narrow and precise. GRLO does not claim a new RL objective; it claims that RLHF-style PPO directly from a base model on a small, diverse, open-ended prompt pool can recover a broad post-training capability profile across chat, reasoning, and code, and can do so with markedly lower data and compute than a strong in-domain RLVR baseline. Subsequent RLVR remains useful, but mainly as a selective specialization stage rather than the main source of broad transfer (Yin et al., 14 May 2026).