Extend the gradient analysis to stochastic estimators

Extend the population-level gradient analysis of the expected-reward and log-barrier objectives for three-outcome policies to the stochastic gradient estimators used by GracefulRL.

Background

The paper proves gradient suppression for the expected-reward policy-gradient objective and non-vanishing gradients for the log-barrier-based GracefulRL objective at the population-gradient level. The analysis establishes behavior under bounded logit-gradient and non-degeneracy assumptions, but does not cover the randomness introduced by finite rollout samples and stochastic gradient estimation. Establishing analogous results for the actual stochastic estimators would clarify whether the theoretical optimization advantages persist during practical training.

References

It is worth noting that our proofs are only for the population level gradients and not for stochastic estimators which we leave for future work.

— Training Language Models to Fail Gracefully  (2610.20401 - Arora et al., 14 Oct 2026) in Appendix, immediately following the proof roadmap in the section titled “Proof of the Results in Section~\ref{sec:collapse}”