On the Convergence of Self-Improving Online LLM Alignment

Abstract: The Self-Improving Alignment (SAIL) algorithm addresses distribution shift by reducing a bilevel formulation of the problem to an efficient, single-level method. Empirically, SAIL has demonstrated strong performance on this task. However, a formal analysis of its convergence properties has been lacking. We identify a key theoretical challenge: the standard SAIL objective function is not guaranteed to be strongly concave due to unfavorable properties of its Hessian. To address this limitation, we propose a regularized objective, SAIL-RevKL, which incorporates a reverse Kullback-Leibler (KL) divergence penalty to improve the optimization landscape. Our central theoretical contribution is to prove that this regularized objective satisfies the Polyak-Lojasiewicz (PL) condition within a bounded parameter space. We establish global convergence guarantees, achieving a near-linear sample complexity. We further validate the effectiveness and stability of SAIL-RevKL through empirical evaluations, demonstrating that it outperforms the vanilla SAIL on both MuJoCo benchmarks and LLM alignment tasks.

• The paper identifies that unregularized SAIL lacks global curvature, and introduces a RevKL penalty to ensure strong PL geometry.
• It rigorously establishes non-asymptotic convergence guarantees and improved sample complexity for projected stochastic gradient ascent.
• Empirical tests on continuous control and LLM benchmarks demonstrate that SAIL-RevKL achieves more stable, efficient, and robust alignment.

On the Convergence of Self-Improving Online LLM Alignment (2606.31524)

Overview and Motivation

The paper addresses a fundamental issue in online alignment of LLMs with human preferences via reinforcement learning from human feedback (RLHF): the lack of rigorous theoretical guarantees for convergence in modern, practically relevant alignment algorithms such as SAIL (Self-Improving Alignment Learner) (Ding et al., 2024). Whereas existing online RLHF methods typically optimize surrogate objectives derived from bilevel formulations but operate with predominantly heuristic or empirical justifications, this work systematically analyzes the underlying optimization geometry and discovers that the canonical SAIL objective generally fails to yield global curvature properties required for provable convergence.

To resolve the geometry limitations, the authors augment SAIL with a reverse Kullback-Leibler (RevKL) divergence regularizer, yielding the SAIL-RevKL objective. Through precise non-asymptotic analysis, they establish Polyak-Łojasiewicz (PL) and strong concavity conditions over a bounded parameter regime, derive improved global sample complexity guarantees, and empirically validate the stabilized learning dynamics and alignment efficacy of SAIL-RevKL in both continuous-control and LLM alignment benchmarks.

Theoretical Contributions

Failure of Strong Concavity for Unregularized SAIL

The central technical finding is that the unregularized SAIL objective is not strongly concave outside a limited local region determined by the interaction of model initialization, parameter norm, and the KL regularization parameter $\beta$. More specifically, the negative Hessian $-\nabla^2 J(\theta)$ decomposes into competing terms, some contributing negative curvature (notably those associated with the Fisher-Hessian identity), such that beyond a norm ball of critical radius, the objective generally becomes non-concave.

This restriction invalidates global linear convergence guarantees; prior work (e.g., (Gaur et al., 22 Mar 2025)) established only local convergence rates in bilevel policy alignment problems, usually with suboptimal sample complexities and assumptions not directly applicable to the deep RLHF context.

Reverse-KL Regularization for Global PL Geometry

To correct the deficit in global curvature, the authors introduce a reverse-KL penalty:

$$J_\gamma(\theta) = J(\theta) - \gamma\,\mathbb{E}_{x}[D_{\mathrm{KL}}(\pi_{\mathrm{ref}}(\cdot\mid x)\|\pi_{\theta}(\cdot\mid x))]$$

where $\gamma > 0$ and $\pi_{\mathrm{ref}}$ is typically the supervised fine-tuning (SFT) policy. The regularizer directly injects the Fisher information matrix as a positive definite term into the Hessian of the objective, compensating for adverse negative curvature and ensuring strong global concavity and PL geometry within a bounded parameter region. Theoretical results rigorously quantify the parameter regime, sufficient regularization magnitude, and leverage explicit bounds for the PL constant.

Sample Complexity and Convergence Guarantees

With the PL condition established globally on the feasible set, the authors analyze a projected stochastic (mini-batch) gradient ascent algorithm, showing for SAIL-RevKL:

• Gradient mapping convergence: Expected norm squared of the (projected) gradient mapping decays as $\widetilde{\mathcal{O}}(\varepsilon^{-2})$ for $T=\widetilde{\mathcal{O}}(\varepsilon^{-1})$ and batch size $B_s=\widetilde{\mathcal{O}}(\varepsilon^{-1})$.
• Global function value convergence: Sub-optimality gap of the objective function decays at rate $$\widetilde{\mathcal{O}}\big(\frac{1}{\varepsilon}\log\frac{1}{\varepsilon}\big)$$, representing substantial improvement over the $\widetilde{\mathcal{O}}(\varepsilon^{-3})$ rates in general bilevel RL frameworks ((Gaur et al., 22 Mar 2025), [chakraborty2024parlunifiedframeworkpolicy]).
• Regularization bias bounds: The end-to-end suboptimality gap between the SAIL and SAIL-RevKL solutions is explicitly bounded as a function of $-\nabla^2 J(\theta)$0, with a parameter-norm bias no greater than $-\nabla^2 J(\theta)$1 and an objective value gap controlled as $-\nabla^2 J(\theta)$2.

Practical Algorithm

The algorithm deploys projected mini-batch stochastic gradient ascent, maintaining the policy parameter within an $-\nabla^2 J(\theta)$3-bounded ball around $-\nabla^2 J(\theta)$4, with explicit update formulas for step, RevKL computation, and projection. The analysis depends on standard L-smoothness and bounded unbiased variance of the stochastic gradient estimator, as is typical in policy-gradient settings.

Empirical Results

Continuous Control

On MuJoCo benchmarks (Door Open, Walker Walk, Walker Stand, Cheetah Run), SAIL-RevKL demonstrates enhanced learning stability and final reward performance compared to both PEBBLE [lee2021pebblefeedbackefficientinteractivereinforcement] and the original SAIL method. Effect sizes (Cohen's $-\nabla^2 J(\theta)$5) confirm the practical significance of gains, not merely in return but in terms of learning dynamics and robustness across seeds.

LLM Alignment

Extensive experiments on standard LLM alignment benchmarks—including PKU-SafeRLHF ([ji2025pkusaferlhf]) and UltraFeedback ([cui2023ultrafeedback])—demonstrate that SAIL-RevKL consistently yields superior pairwise win rates and closed the performance gap in terms of GPT-4 judge scores, compared with DPO (Rafailov et al., 2023) and the vanilla SAIL method. Importantly, this holds both in practical LoRA-based fine-tuning and in a theory-matched last-layer-only (log-linear) regime. The effect persists across diverse LLM backbones (Qwen 0.5B, Phi-3 3.8B, LLaMA-3 8B).

Implications and Future Directions

The work provides the first global convergence guarantees for a single-level surrogate of the online LLM alignment problem, bridging a gap between empirical advances in scalable RLHF and foundational optimization theory. The reverse-KL regularizer is shown to be an effective mechanism for directly controlling optimization geometry, sidestepping problematic local assumptions prevalent in prior bilevel RLHF theory.

Immediate implications include:

• Guidance for systematic selection of KL weight and feasible set radius in practice.
• Justification for last-layer-only adaptation (robustness to distribution shift), with theoretical support for parameter efficiency.
• Clarity on the tradeoff between regularization bias and curvature stabilization.

Potential future work:

• Extending the theoretical analysis to more general (possibly nonlinear) policy classes and broader families of regularizers beyond RevKL.
• Exploration of adaptive or schedule-based $-\nabla^2 J(\theta)$6 selection based on optimization dynamics.
• Investigating the interplay between RLHF sample efficiency, data collection strategies, and online optimization geometry at scale.
• Application to broader settings such as preference-based reward model learning, complex safety alignment, and distributionally-robust optimization (Xu et al., 4 Feb 2025).
• Analysis of convergence properties under more realistic, noisy, or adversarial preference feedback.

Conclusion

This paper rigorously analyzes the SAIL paradigm for online RLHF-style LLM alignment, revealing that unregularized surrogate optimization lacks global convergence guarantees due to unfavorable curvature. Introducing a reverse-KL regularization both remedies the geometric deficiencies and underpins sharp, global, nonasymptotic convergence rates. Empirical results on both continuous control and LLM alignment domains validate the efficacy and robustness of SAIL-RevKL in practical settings. The work advances both theory and practice in scalable, sample-efficient, and provably well-conditioned LLM alignment.