RL Post-Training Builds Compositional Reasoning Strategies

Abstract: Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.

• The paper demonstrates that RL post-training induces the formation of new compositional strategies by consolidating primitive actions into reusable procedural chunks.
• It employs a controlled rewrite-grammar environment to precisely audit model behavior, showing RL's ability to solve tasks unsolvable by base models under high sampling budgets.
• The study highlights the importance of structured pretraining for facilitating procedural chunking and expands our understanding of RL as more than mere reweighting of latent behaviors.

Compositional Reasoning via RL Post-Training in Rewrite-Grammar Environments

Background and Motivation

This paper investigates whether reinforcement learning (RL) post-training on Transformer models extends model capabilities beyond amplifying behaviors already present in a pretrained base model. The context is a fully observable rewrite-grammar environment, where every generated rewrite can be classified and audited precisely, and where the pretraining distribution as well as grammar dynamics are entirely controlled. The central question is whether RL, particularly with outcome-only, binary rewards, can induce the emergence and consolidation of new compositional reasoning strategies—procedural chunks composed of primitive symbol rewrites—that are not directly accessible via extensive sampling of base-model behaviors.

Recent literature explores similar themes with conflicting conclusions. Some argue that post-training, particularly RL with verifiable rewards, primarily reweights and amplifies small-k successes, but does not expand the space of reachable solutions (Yue et al., 2025). Others provide evidence that RL, especially via prolonged training or controlled compositional tasks, unlocks behaviors and compositional reasoning inaccessible to the base model (Liu et al., 2025a; Yuan et al., 2025). Imitation-based methods such as rejection fine-tuning (RFT) are often competitive early on, but their limitations in consolidating procedural abstractions remain an open question.

Controlled Environment and Experimental Setup

The rewrite-grammar environment allows for exhaustive classification of model actions: primitive contractions (direct grammar-rule applications), macro contractions (sequential composition of primitives), parallel contractions (simultaneous contraction of independent substrings), and spurious actions (invalid rewrites). The Transformer model is pretrained from scratch on chained, primitive rewrite trajectories, exposing it exclusively to grammar-consistent local rewrite dynamics, not to the higher-level shortcuts or procedural compositions that are the focus of post-training analyses.

The post-training task involves generating contraction traces that reduce a prompt word to a target symbol, with difficulty defined by whether the minimal primitive solution fits within the model’s generation budget. RL is implemented via Group Relative Policy Optimization (GRPO), and RFT is used as a strong, on-policy imitation baseline.

Emergence and Consolidation of Compositional Strategies

Analysis reveals a phased mechanism underlying RL-induced compositional reasoning. RL initially strengthens primitive reduction behaviors; only after an extended training period does the model begin emitting valid, compressed procedural chunks—macro and parallel contractions—that were never present in the pretraining data. These chunks are systematically reused and consolidated across trajectories, demonstrating the formation of a stable procedural repertoire.

The consolidation dynamic is critical: RL does not rely merely on scattered shortcut discoveries, but stabilizes them, turning discovered macro contractions into reusable options within the policy. This is achieved in the absence of explicit intermediate supervision, relying entirely on binary, final-answer rewards.

Importantly, RFT also generates non-primitive rewrites, but a significant fraction are spurious or invalid; RL, in contrast, filters exploration, concentrating policy update pressure into valid, reward-aligned procedural structure. The selectivity mechanism in GRPO exploits within-prompt group contrast, suppressing features (such as spurious contractions) that are enriched in failed rollouts, while amplifying features linked to successful completions.

Finite-Budget Frontier and Expansion Beyond the Base Model

The auditability of this environment enables precise evaluation of whether RL expands the model's capability frontier. For held-out contraction problems of increasing difficulty—defined by primitive-solution length exceeding the generation budget—RL reliably solves problems unsolved by the base model even under much larger sampling budgets (e.g., pass@1024). The largest RL gains emerge late in training and are concentrated on the hardest buckets, where compressed compositional maneuvers are required. RFT improves early but quickly plateaus. This demonstrates that RL is not merely amplifying low-probability completions from the pretrained distribution, but is reorganizing procedural competence and assembling previously inaccessible strategies.

Role of Pretraining Structure in Strategy Emergence

The paper identifies an essential gating mechanism: compositional strategy emergence via RL is contingent not simply on exposure to primitive contractions in pretraining, but on whether pretraining organizes these contractions into reduction-mode procedures that can later be compressed. Ablations varying the local contraction weight reveal that models exposed to high-frequency, chained contraction sequences during pretraining facilitate earlier and stronger emergence of non-primitive compositional strategies. Controls matching only marginal contraction frequency fail to reproduce these dynamics, underscoring the importance of procedural organization and not merely statistical exposure.

Theoretical and Practical Implications

The findings clarify ongoing debates about the capacity of RL post-training to induce new behaviors in LLMs. By leveraging a fully controllable and audit-friendly grammar, the paper demonstrates mechanistically that RL expands the set of usable reasoning strategies, incrementally constructing procedural abstractions from weakly supported primitives. This stands in opposition to claims that RL post-training generally operates as reward-driven reweighting, without structural innovation (2607.07646).

The results provide formal evidence that RL can serve as a "chunking" mechanism: selecting, compressing, and stabilizing procedural ingredients that pretraining puts in place. This suggests that effective pretraining should focus not only on exposure to relevant facts or actions, but on the structuring of trajectories to facilitate subsequent compositional chunking via RL.

Practically, these insights bear on the design of curriculum and environment structure in RL post-training regimes for LLM reasoning tasks. They motivate the explicit auditing and analysis of trace-level behavior to differentiate between valid compositional strategy emergence and reward exploitation through invalid intermediate steps.

Future Directions

Emergent procedural chunking in RL post-training, as observed in this grammar, invites further exploration in more complex, less structured environments. Future work should focus on extending the auditing framework to natural language tasks, where mechanical checking of intermediate steps is far more challenging. Moreover, the procedural substrate supplied by pretraining, and its role in gating strategy emergence, warrants rigorous characterization across diverse pretraining data and model families. Insights from this work encourage targeted modification of pretraining trajectories to facilitate post-training procedural abstraction.

Conclusion

The paper provides formal, mechanistic evidence that RL post-training in a fully-auditable, controlled grammar environment produces compositional reasoning strategies—macro and parallel contractions—that expand the capability frontier of the base model. This expansion is not attributable solely to amplified exploration or sampling of latent behaviors, but arises from the procedural selection and consolidation enabled by RL. The emergence of such strategies depends critically on the structure—not just frequency—of primitive competence encoded during pretraining. These findings refine the understanding of RL post-training’s role in procedural generalization and contribute concrete guidance for the design of pretraining and post-training regimes in automated reasoning systems.