RL-based Post-training in LLMs

• RL-based post-training is a process that refines pretrained models through reward-guided adaptation, enhancing reasoning, alignment, and efficiency.
• It employs algorithms like PPO, GRPO, and DPO to update policies using token-level and trajectory-level gradients under regularized objectives.
• Applications span large language and multi-modal models, evidencing improved scalability, compositional generalization, and robust integration in complex systems.

Reinforcement learning (RL)-based post-training refers to the stage in the development of large models—especially LLMs and multi-modal systems—where supervised pretraining or fine-tuning is followed by additional policy adaptation using RL algorithms. This process directly maximizes user-defined reward signals (e.g., answer correctness, human preferences, tool-use verification), often subject to regularization constraints such as KL-divergence against a reference policy. The RL post-training paradigm has become central for improving reasoning, compositional generalization, alignment, and efficiency in LLMs, and now applies pervasively to vision-language-action models and multi-modal captioning systems. Recent work highlights the emergence of new structures (e.g., skill trees), role-specific learning dynamics, system-level optimization, and specialized curricula, establishing RL-based post-training as a critical area in foundation model research.

1. Conceptual Foundations and Motivation

RL-based post-training builds upon a pretrained or fine-tuned base model by further adapting its weights to maximize expected rewards under its own generation policy, typically penalized to remain close to a behavior model via KL-divergence. Let $\pi_\theta$ denote the policy and $r(x, y)$ the reward for output $y$ given context $x$; the canonical objective is

$$J(\theta) = \mathbb{E}_{x \sim D,\, y \sim \pi_\theta(\cdot|x)}\left[ r(x, y) - \lambda \mathrm{KL}(\pi_\theta(\cdot|x)\| \pi_{\text{ref}}(\cdot|x)) \right].$$

This paradigm addresses limitations of supervised-only training, which is fixed to next-token likelihood or hard targets, and allows models to learn longer, more reasoning-intensive trajectories (such as chain-of-thought, tool-using sequences) using reward feedback. RL post-training underpins advances in reasoning (mathematical, logical), alignment (RLHF), tool-use, and personalization (Park et al., 1 Dec 2025, Tsilivis et al., 13 Oct 2025, Oh et al., 23 Jun 2025).

2. Algorithms, Objectives, and Skill Composition

The dominant algorithms for RL-based post-training include Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO), Direct Preference Optimization (DPO), and hybrid methods combining RL with Knowledge Distillation (KDRL) (Xu et al., 2 Jun 2025). Typical updates in PPO/GRPO rely on surrogate objectives that involve clipped probability ratios and normalized advantages; DPO focuses on optimizing pairwise preference probabilities. Policy gradients are computed either token-wise or trajectory-wise, with importance weighting and group normalization for variance reduction.

The emergence of compositional generalization is evidenced in formal studies where RL post-training induces the ability to synthesize novel skills by recombining learned subtasks. On the Countdown arithmetic reasoning benchmark, RL induces out-of-distribution (OOD) generalization to unseen tree shapes, with the discovery and mastery of balanced skill trees preceding deep or right-heavy ones (Park et al., 1 Dec 2025). This reveals that RL does more than length generalization—it enables true structural composition, as quantified via tree-shape decomposition and fine-grained per-pattern metrics.

Example Table: RL Post-training Algorithmic Variants

Algorithm	Objective Formulation	Update Mechanism
PPO	Clipped surrogate, advantage	On-policy rollout, token-level update
GRPO	Group-normalized clipped surrogate	Batched group rollouts, group advantage normalization
DPO	Pairwise preference objective	Off-policy, static preference pairs
KDRL	RL + reverse-KL distillation	Joint policy gradient from teacher and reward models

3. System Architectures and Scalability

Modern RL post-training frameworks are engineered for large-scale distributed training, emphasizing asynchronous task separation, resource decoupling, and robust fault tolerance. Systems such as AsyncFlow (Han et al., 2 Jul 2025) and Laminar (Sheng et al., 14 Oct 2025) architect multilayered modules separating resource management, model engine APIs, distributed streaming dataloaders, and producer-consumer asynchronous workflows. These frameworks break global update barriers through per-trajectory asynchrony, relay-based weight broadcasting, and dynamic repack mechanisms, enabling up to 5.48× throughput improvement over synchronous RL baselines.

Fault tolerance is achieved through role-based isolation: systems such as RobustRL (Chen et al., 27 Dec 2025) distinguish between trainer, rollout, and management roles, permitting localized recovery and UCX-based point-to-point weight synchronization. RollMux (Wu et al., 12 Dec 2025) further optimizes cross-cluster orchestration, using co-execution group abstractions and round-robin meta-iteration for maximal resource utilization in synchronous disaggregated workloads.

Example Table: Key System Features

System	Decoupling Strategy	Fault Tolerance	Throughput Gain
AsyncFlow	API-layered, async workflow	Producer-consumer recovery	1.59–2.03×
Laminar	Full trajectory-level async	Relay-based isolation	4.06–5.48×
RollMux	Cluster phase multiplexing	Group locality, warm-start	1.84× cost efficiency

4. Curricula and Data Selection Strategies

Curriculum learning and data selection directly affect RL post-training sample efficiency and convergence. Prompt curriculum learning (PCL) (Gao et al., 1 Oct 2025) utilizes a learned value model to select intermediate-difficulty prompts that maximize group variance and gradient norm, dramatically reducing unnecessary rollouts. Distribution-level curricula such as DUMP (Wang et al., 13 Apr 2025) use distribution-wise policy advantages, scheduling samples from distributions with highest average advantage or low sample counts based on upper confidence bound (UCB) criteria, thus balancing exploitation and exploration.

Problem-level prioritized replay (Fatemi, 6 Jan 2026) utilizes a simple priority score $\omega_j = p_j (1-p_j)$ derived from empirical success rates to sample problems that yield the largest mean squared advantage, focusing training on intermediate-difficulty problems and avoiding manual tiers. Unlike static easy-to-hard schedules, this adaptive process requires no external labels and aligns selection with the dynamics of GRPO updates.

5. Learning Dynamics, Scaling Laws, and Internal Model Changes

RL post-training exhibits characteristic learning dynamics, including confidence sharpening and output diversity reduction. Empirical neural tangent kernel (NTK) analysis (Tomihari, 8 Jan 2026) reveals that RL updates systematically increase model confidence via representation-based similarity, concentrating probability mass on high-reward continuations and reducing output diversity. Classifier-first RL (CF-RL) accelerates optimization by reshaping the classifier matrix prior to standard RL, producing rapid reward improvement without the feature distortion seen in linear-probe supervised fine-tuning.

Scaling law studies (Tan et al., 29 Sep 2025) show that, under fixed compute, larger models trained for fewer steps outperform smaller ones trained longer; larger models have higher sample efficiency for fixed data volume, and repeated reuse of high-quality data is effective until overfitting occurs. These relationships hold across base and instruction-tuned models, and provide practical guidance: maximize model size within compute constraints, employ data reuse, and tune rollout group size for sample efficiency.

Example Table: Scaling Relations

Constraint Type	Optimal Strategy	Empirical Impact
Compute	Larger model, fewer steps	Lower test loss
Data volume	Larger model, high sample reuse	Higher efficiency

6. Domain Specialization, Multi-modal RL, and Joint Optimization

RL post‑training supports domain-specialized adaptation and multi-modal objectives. RedOne 2.0 (Zhao et al., 10 Nov 2025) applies a staged RL–SFT–RL pipeline for social networking tasks, using DAPO to achieve superior data efficiency and stable in-domain gains without sacrificing robustness. RePIC (Oh et al., 23 Jun 2025) employs RL post-training for personalized image captioning, with verifiable object, localization, and identity consistency rewards. RL drives generalization in tasks (2-concept, 4-concept), dramatically outperforming SFT-only approaches especially with limited data.

Hybrid objectives emerge from joint optimization of reward maximization and knowledge distillation (KDRL) (Xu et al., 2 Jun 2025), leveraging reverse-KL distillation and GRPO for mathematical reasoning, achieving higher accuracy and shorter reasoning outputs than either RL or KD alone. Critically, theoretical work on decoupling (Niu et al., 12 Jan 2026) demonstrates that SFT and RL cannot be separated without losing prior performance: SFT increases RL reward but lowers SFT likelihood, and vice versa, motivating future research into unified or constrained policy optimization frameworks.

7. Open Questions, Limitations, and Future Directions

RL-based post-training is subject to several bottlenecks and open questions. Compositional generalization depends on tree-shape bias and structural bottlenecks—skill mastery is easiest in balanced decompositions, but right-heavy trees remain fragile even at equal depth (Park et al., 1 Dec 2025). System scaling is challenged by straggler bubbles, memory residency, and phase synchronization (Wu et al., 12 Dec 2025). In decentralized settings, excessive external experience sharing may destabilize learning (Amico et al., 10 Sep 2025). Finally, the irreversible coupling of SFT and RL objectives suggests that multi-objective or regularized joint pipelines will be necessary to balance memorization and reward-based generalization (Niu et al., 12 Jan 2026).

A plausible implication is that the future of RL post-training will include more principled curricula, fine-grained tracking of skill acquisition order, hybrid objective optimization, and robust, asynchronous infrastructure to support sustained scaling.

---

References: (Park et al., 1 Dec 2025, Han et al., 2 Jul 2025, Gao et al., 1 Oct 2025, Wang et al., 30 Sep 2025, Gao et al., 25 Sep 2025, Chen et al., 27 Dec 2025, Zhang et al., 25 Sep 2025, Amico et al., 10 Sep 2025, Sheng et al., 14 Oct 2025, Wang et al., 13 Apr 2025, Tsilivis et al., 13 Oct 2025, Xu et al., 2 Jun 2025, Zhao et al., 10 Nov 2025, Niu et al., 12 Jan 2026, Wu et al., 12 Dec 2025, Ding et al., 9 Dec 2025, Fatemi, 6 Jan 2026, Tomihari, 8 Jan 2026, Tan et al., 29 Sep 2025, Oh et al., 23 Jun 2025).