GRPO-Style Online RL Pipeline

• The paper introduces a pipeline that leverages group relative policy optimization to enable stable, on-policy RL fine-tuning across modular LM programs.
• It groups LM calls by module and invocation index to ensure consistent gradient updates and effective variance reduction through normalized, group-relative advantage scaling.
• The integration of prompt optimization with RL fine-tuning boosts accuracy by up to 11%, demonstrating its practical value in handling complex, variable-length trajectories.

A GRPO-style Online RL pipeline is an on-policy reinforcement learning framework leveraging Group Relative Policy Optimization (GRPO) for credit assignment and stable post-training, originally proposed for LLMs and extended to modular LLM (LM) programs. The multi-module GRPO (mmGRPO) generalizes the standard algorithm to handle structured LM systems composed of multiple modules, each parameterized by distinct prompt templates and potentially separate LoRA adapter weights. The pipeline yields stable, credit-aware RL fine-tuning across modular and heterogeneous LM programs by grouping LM calls for gradient updates and integrating prompt optimization as a staged curriculum (Ziems et al., 6 Aug 2025).

1. Modular LM Program Architecture and Trajectory Semantics

A modular LM program $\Phi$ consists of a set of Python "modules" $M_1, ..., M_m$, each equipped with:

• Prompt template $\pi_m$ (e.g., chain-of-thought, few-shot).
• Module-specific LM weights $\theta_m$ (commonly sharing a base LM with independent LoRA adapters).

For a structured input $x$, executing $\Phi(x)$ yields a final output $y$ and trajectory $\rho = [\zeta_1, ..., \zeta_T]$, with each $\zeta_t = (M_t, q_t, o_t)$ recording the invoked module, the realized prompt $q_t$ (obtained by instantiating $M_1, ..., M_m$0), and the sampled LM continuation $M_1, ..., M_m$1 from $M_1, ..., M_m$2. The system is designed to support complex LM programs with variable-length, interrupted, or modularly compositional execution traces. During training, $M_1, ..., M_m$3 rollouts $M_1, ..., M_m$4 are collected, optionally mixing student and teacher programs.

The core idea is to align module calls across rollouts by $M_1, ..., M_m$5—identifying, for instance, "the third call to module QueryGenerator"—such that all $M_1, ..., M_m$6's with a shared $M_1, ..., M_m$7 index are grouped for RL optimization.

2. mmGRPO Policy Objective and Credit Assignment

For each module $M_1, ..., M_m$8 and invocation index $M_1, ..., M_m$9, the mmGRPO pipeline forms a group $\pi_m$0, where $\pi_m$1 is the group size. Each tuple receives a program-level reward $\pi_m$2 (where $\pi_m$3 is a chosen metric, such as accuracy). The module parameters $\pi_m$4 are optimized via a clipped, group-relative policy gradient:

$\pi_m$5

where:

• $\pi_m$6 is the token-level importance ratio.
• The group-relative advantage is

$\pi_m$7

• $\pi_m$8 is the PPO-style clipping parameter (default $\pi_m$9).
• $\theta_m$0 controls KL-regularization to a reference policy.

This objective replaces value function estimates with directly normalized within-group advantage scaling, yielding variance reduction and robust credit assignment even under variable trajectory lengths or changing group composition.

3. Handling Variable-Length and Interrupted Trajectories

Real-world modular LM programs produce trajectories where the number of calls per module and their positions may vary due to early termination or errors. To address this:

• Trajectories are bucketed by $\theta_m$1, producing groups of potentially uneven size.
• Two padding modes are supported:
  • "truncate": drop $\theta_m$2 groups whose count is below $\theta_m$3
  • "fill": pad missing entries by duplicating or inserting dummy entries (with a low fallback reward)
• Each group is resampled up/down to a fixed $\theta_m$4 using a diversity-promoting sampler (e.g., SelectK-DiverseElements), favoring broad reward coverage and mitigating overfitting to trivial batch statistics.

4. Prompt Optimization and Curriculum Integration

The mmGRPO pipeline may be preceded by integrated prompt optimization to synthesize improved prompt templates $\theta_m$5. This is typically done using a black-box optimizer (e.g., MIPROv2). The staged protocol is:

1. Run prompt optimization: $\theta_m$6.
2. Fix prompt templates and conduct RL fine-tuning: $\theta_m$7.

Empirically, this BetterTogether(PO, mmGRPO) approach yields stronger initial rollouts and boosts final accuracy across multiple task types, including many-hop search and privacy-preserving delegation (Ziems et al., 6 Aug 2025).

5. End-to-End Pipeline and Pseudocode

The mmGRPO training loop consists of the following steps:

1. Sample a batch of $\theta_m$8 inputs from the training dataset.
2. For each input, generate $\theta_m$9 rollouts (possibly including multiple teacher programs) to build the experience set.
3. Align and group all module invocations across rollouts into $x$0-indexed groups.
4. Pad or truncate groups to the target size $x$1 and sample for diversity.
5. For each group $x$2, compute group-relative advantages $x$3 and take one gradient step on $x$4 using the mmGRPO objective.
6. Repeat for $x$5 training iterations.

The function FormModuleLevelGroups constructs $x$6 groups from trajectory logs, pads/truncates, and applies diversity sampling as required. Trained models can be deployed or evaluated immediately after this loop completes.

6. Key Hyperparameters and Practical Tips

Crucial settings for stable, high-performance mmGRPO-style online RL include:

Parameter	Recommended Value	Effect/Notes
Rollouts per example $x$7	12	Matching group size $x$8 for variance control
Group size $x$9	12	Consistency across modules/groups
Training steps	$\Phi(x)$0
Batch size $\Phi(x)$1	4	$\Phi(x)$2 inputs $\Phi(x)$3 rollouts = $\Phi(x)$4 trajectories/step
Learning rate $\Phi(x)$5	$\Phi(x)$6
Gradient accum.	20	Imitates batch size 20
Weight decay $\Phi(x)$7	0
Clipping $\Phi(x)$8	0.2 (PPO-style)	Trust region enforcement
KL penalty $\Phi(x)$9	0.01–0.04	Proportional to model size/task
Context cutoff	8192 tokens	Drop trajectories exceeding this length
LoRA adapters	rank=16, alpha=64	Layerwise attention projection tuning
Sampling temp.	0.6	Exploration/exploitation balance
Top-p, top-k	0.9–0.95, 20	Search diversity control

Additional strategies include setting pad_mode="fill" to retain group diversity and using group diversity sampling (reward stratification or variance maximization) to prevent collapse.

7. Implementation, Extensibility, and Empirical Impact

mmGRPO is implemented in the DSPy framework under the optimizer dspy.GRPO. Extensibility is supported via pluggable grouping strategies, padding modes, diversity samplers, and rollout teacher mixes. DSPy’s Arbor training library (GRPOTrainer) manages gradient accumulation, KL logging, and checkpointing. The modular APIs allow integration with custom programs built from DSPy’s module and chain-of-thought primitives.

Empirically, mmGRPO composed with automatic prompt optimization yields substantial improvements: up to 11% accuracy gains over post-trained LMs, and +5% improvement versus prompt optimization alone, across diverse tasks. The method stably supports arbitrary modular LM architectures and has become a reference pipeline for robust, credit-aware online RL with complex programmatic LMs (Ziems et al., 6 Aug 2025).