Effect of KL regularization on single-adapter majority voting

Determine whether adding KL regularization to GRPO training changes the majority-vote performance of a single LoRA adapter trained with reinforcement learning from verifiable rewards.

Background

The paper studies how allocating a fixed RLVR budget between one deeply trained LoRA adapter and multiple independently trained adapters affects majority voting. Its experiments use GRPO without a KL penalty and find that concentrated training can increase single-sample accuracy while reducing majority-vote performance through more correlated voter errors. Because KL regularization can constrain policy deviation during RLVR training, it may alter the balance between accuracy improvement and sampling diversity, but the paper does not investigate this possibility.