Impact of Large AREW Reweighting Strength on Training Stability

Establish whether large values of the AREW reweighting-strength parameter (Au), which scales the likelihood-margin auxiliary objective used to reweight advantages, amplify the variance of advantage estimates and over-emphasize a small subset of steps, thereby making policy updates brittle and sensitive to noise during reinforcement learning of large language model agents.

Background

The paper introduces AREW, a critique-driven advantage reweighting method that injects stepwise directional critiques through a likelihood-margin auxiliary objective. A scalar parameter (denoted Au in the paper) controls the strength of this reweighting during policy optimization.

Empirically, the authors observe that too-weak reweighting fails to escape the self-locking regime, while too-strong reweighting speeds early optimization but often leads to instability and eventual performance collapse (illustrated in Fig. 5d). Based on these observations, they explicitly conjecture a mechanism linking large Au to increased advantage-variance and overemphasis on a small subset of steps, which would make policy updates brittle and sensitive to noise.