Cause of pass@k gap narrowing in RL post-training

Ascertain whether the observed narrowing of pass@k performance gaps between reinforcement-learning–post-trained large language models and their base models is explained, at least in part, by evaluating and conducting RL training on tasks for which base models already achieve high pass@k due to pretraining, thereby leaving RL with limited incentive to teach genuinely new skills.

Background

A recurring critique of RL post-training for LLMs is that it seemingly only reranks model outputs, as pass@k gaps shrink when larger k is allowed. The authors argue this may be a measurement artifact driven by training and evaluating on tasks that base models already solve with high pass@k, likely due to pretraining exposure, which would deprive RL of the incentive to learn new abilities.

They introduce a controlled string-transformation framework to isolate compositional skill acquisition and present evidence that RL significantly improves performance on harder compositional problems where base models have near-zero pass@k, suggesting the need to determine the causal role of task difficulty and pretraining in the observed pass@k gap behavior.