Unregularized Convergence of Single-Loop, Entropy-Regularized Natural Actor-Critic

Abstract: While entropy regularization is widely used to stabilize and accelerate Natural Policy Gradient methods, its ability to yield faster convergence rates for the unregularized objective remains underexplored. Existing analyses often rely on double-loop architectures and invoke a linear entropy penalty. To bridge the gap between theory and practice, we analyze a single-loop, entropy-regularized Natural Actor-Critic algorithm under compatible linear function approximation. By training an uncentered critic, our critic tracking can remain stable even as the training policy approaches determinism and the Fisher information matrix degenerates. We focus on two primary regimes for the optimization landscape: a Stochastic Regime, where we fuse coupled actor-critic updates into a joint Lyapunov recurrence, and a Deterministic Regime, where we pivot to a Policy Mirror Descent framework to circumvent the collapse of Euclidean geometry. By exploiting a positive Minimal Action Gap in the unregularized Markov decision process, we introduce an Exponential Translation mechanism that maps the regularized gap to the unregularized one up to an exponentially decaying tail. By tuning the fixed temperature, our algorithm achieves accelerated unregularized convergence rates, up to approximation-error terms: $\tilde{\mathcal{O}}(T_{total}<sup>{-1})$ in the Stochastic Regime, and $\tilde{\mathcal{O}}(T_{total}<sup>{-2/3})$ for the average iterate alongside $\tilde{\mathcal{O}}(T_{total}<sup>{-1/3})$ for the last iterate in the Deterministic Regime. Here, $T_{total}$ denotes the total number of stochastic critic updates (or Monte Carlo rollouts). Furthermore, in the tabular setting, our positive-action-gap analysis yields a $\tilde{\mathcal{O}}(T_{total}<sup>{-2/3})$ average-iterate rate, surpassing the $\mathcal{O}(T_{total}<sup>{-1/2})$ worst-case statistical barrier that applies without a positive action margin.

• The paper develops a single-loop, uncentered-feature Natural Actor-Critic method whose critic update directly supplies the natural-gradient direction while remaining stable as policies become deterministic.
• The paper proves that an exponential regularized-to-unregularized translation, enabled by a positive minimal action gap, yields tilde-O(T^-1) convergence in the stochastic regime and tilde-O(T^-2/3) average-iterate convergence in the deterministic regime.
• The paper shows that entropy regularization can accelerate instance-dependent tabular learning beyond the worst-case T^-1/2 barrier, while identifying limitations involving Markovian data, bootstrapped critics, approximation error, and last-iterate guarantees.

Overview and Motivation

Entropy-regularized Natural Policy Gradient (NPG) methods are widely deployed in practice, yet existing theory has largely failed to answer whether the regularization itself can be exploited to obtain faster convergence guarantees on the unregularized objective. Prior analyses either study regularized and unregularized objectives in isolation, or translate between them via a linear $\mathcal{O}(\tau)$ entropy penalty that severely degrades rates. For instance, mapping the entropy-regularized NPG analysis of Cayci et al. (Fraga-Lamas et al., 2024) to the unregularized objective through the linear penalty yields only $\tilde{\mathcal{O}}(T_{total}^{-1/6})$, far slower than the $\tilde{\mathcal{O}}(T_{total}^{-1/4})$ to $\tilde{\mathcal{O}}(T_{total}^{-1/2})$ rates established directly for unregularized NPG (Mejorado et al., 2020, Czelusniak et al., 2021, Turnansky, 2023).

This paper analyzes a single-loop, entropy-regularized Natural Actor-Critic (NAC) algorithm under a log-linear softmax policy and linear compatible critic, with exactly one critic SGD step per actor update ($N=1$). Two algorithmic choices are central. First, the critic is trained on uncentered features $\phi(s,a)$ rather than the action-centered score function. This matters because standard centered projections require a positive-definite Fisher information matrix, which degenerates as the training policy approaches determinism. The uncentered feature moment $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$ can remain positive-definite even for deterministic policies, keeping critic tracking stable throughout training. Second, the NPG direction is realized directly through the critic iterate $\theta_{t+1}$ rather than by solving a least-squares projection inside an inner loop, aligning the analysis with practical single-loop implementations.

The main question the paper answers is: compared with unregularized NPG, what unregularized convergence rates can entropy-regularized NPG achieve via joint temperature and step-size tuning?

The Exponential Translation Mechanism

The central technical device is a translation bound that converts regularized suboptimality into unregularized suboptimality without incurring the linear entropy penalty. Under a Minimal Action Gap assumption — a positive margin $\Delta > 0$ between the optimal and suboptimal unregularized action values at every state — the paper proves that for $\tau$ below a structural threshold,

$\tilde{\mathcal{O}}(T_{total}^{-1/6})$0

where $\tilde{\mathcal{O}}(T_{total}^{-1/6})$1 is the regularized global performance gap and $\tilde{\mathcal{O}}(T_{total}^{-1/6})$2 is a temperature-independent constant. The proof proceeds by bounding the suboptimal probability mass $\tilde{\mathcal{O}}(T_{total}^{-1/6})$3 through a decomposition of the reverse KL divergence into cross-entropy and entropy terms, then inverting a nonlinear inequality via a case split on $\tilde{\mathcal{O}}(T_{total}^{-1/6})$4. A corollary extends this bound to all $\tilde{\mathcal{O}}(T_{total}^{-1/6})$5.

The implication is substantial: because the translation tail decays exponentially in $\tilde{\mathcal{O}}(T_{total}^{-1/6})$6 rather than linearly in $\tilde{\mathcal{O}}(T_{total}^{-1/6})$7, the temperature can be tuned as $\tilde{\mathcal{O}}(T_{total}^{-1/6})$8 instead of a polynomial in $\tilde{\mathcal{O}}(T_{total}^{-1/6})$9. Balancing the exponentially small tail against the regularized optimization error then preserves the regularized convergence rates up to logarithmic factors — bypassing the $\tilde{\mathcal{O}}(T_{total}^{-1/4})$0 linear entropy bottleneck that constrains prior work.

Part I: Stochastic Regime and the Joint Lyapunov Analysis

The paper partitions the optimization landscape into two regimes, distinguished by the behavior of the regularized parametric optimum $\tilde{\mathcal{O}}(T_{total}^{-1/4})$1 as $\tilde{\mathcal{O}}(T_{total}^{-1/4})$2.

The Stochastic Regime assumes $\tilde{\mathcal{O}}(T_{total}^{-1/4})$3 converges to a stochastic unregularized parametric optimum, so the centered feature covariance (Fisher information) at the parametric optimum retains minimum eigenvalue $\tilde{\mathcal{O}}(T_{total}^{-1/4})$4 uniformly in $\tilde{\mathcal{O}}(T_{total}^{-1/4})$5. Under a parametric joint concentrability bound, this curvature is dynamically extended to all training policies: $\tilde{\mathcal{O}}(T_{total}^{-1/4})$6.

Within this regime, the paper establishes a Parameter-Space Polyak-Łojasiewicz (PL) condition — bounding the parametric gap by the squared distance between the temperature-scaled actor parameter and the ideal critic target, $\tilde{\mathcal{O}}(T_{total}^{-1/4})$7, rather than by the objective gradient. Under exact parameterization this removes all dependencies on $\tilde{\mathcal{O}}(T_{total}^{-1/4})$8, initial-state coverage, and minimum action probability that appear in the tabular PL condition of Mei et al. (Romao et al., 2020). Combined with an Actor Progress Bound derived from objective smoothness, this yields coupled linear recurrences for the gap and the critic tracking error $\tilde{\mathcal{O}}(T_{total}^{-1/4})$9. The two recurrences are fused into a joint Lyapunov function, giving an optimal $\tilde{\mathcal{O}}(T_{total}^{-1/2})$0 rate for both average and last iterates on the regularized gap — the last-iterate rate matching the average-iterate rate up to a logarithmic factor.

A notable structural finding is an intrinsic compatibility constraint: the conjunction of the Fisher lower bound, the Minimal Action Gap, and the approximation-error assumption forces $\tilde{\mathcal{O}}(T_{total}^{-1/2})$1 as $\tilde{\mathcal{O}}(T_{total}^{-1/2})$2. Consequently, arbitrarily small approximation error independent of temperature is incompatible with the Part-I assumptions, and the exact-realizability case $\tilde{\mathcal{O}}(T_{total}^{-1/2})$3 is excluded from the global analysis. The stated $\tilde{\mathcal{O}}(T_{total}^{-1/2})$4 bounds are understood over the resulting admissible range of $\tilde{\mathcal{O}}(T_{total}^{-1/2})$5.

Applying the exponential translation with a two-stage temperature $\tilde{\mathcal{O}}(T_{total}^{-1/2})$6 yields the headline result: $\tilde{\mathcal{O}}(T_{total}^{-1/2})$7 unregularized convergence for both average and last iterates, up to $\tilde{\mathcal{O}}(T_{total}^{-1/2})$8 approximation terms. This surpasses the best prior unregularized rates under function approximation, including the $\tilde{\mathcal{O}}(T_{total}^{-1/2})$9 last-iterate rate of Yuan et al. (Turnansky, 2023), which itself requires a highly unbalanced double-loop configuration with $N=1$0.

Part II: Deterministic Regime and Policy Mirror Descent

The Deterministic Regime covers the case where the regularized parametric optimum collapses toward a deterministic policy as $N=1$1 — which necessarily occurs when the log-linear family is expressive enough to contain the deterministic global optimum $N=1$2. Here the Fisher information degenerates ($N=1$3), breaking the Euclidean parameter-space geometry underlying Part I.

The analysis pivots to a Policy Mirror Descent (PMD) framework, measuring progress via KL divergence on the probability simplex. A key observation is that the uncentered NAC update is an exact PMD step under the log-linear parameterization, so the algorithmic progress term can be bounded via the Bregman Three-Point identity evaluated against the current policy — bypassing the variational inequalities used in standard PMD analyses. The resulting single-step inequality contains a restorative entropy force $N=1$4 that provides geometric contraction.

Because the Actor Progress Bound is unavailable, critic target drift is controlled by a worst-case parameter step, decoupling the critic from the actor. The competing noise forces — SGD error $N=1$5 and target drift $N=1$6 — are balanced by the fractional coupling $N=1$7, giving the schedule $N=1$8, $N=1$9, analyzed via a specialized fractional Chung's lemma. This yields a $\phi(s,a)$0 average-iterate and $\phi(s,a)$1 last-iterate unregularized rate. The last-iterate rate is slower because direct extraction from the telescoping bound would amplify the noise floor by $\phi(s,a)$2; instead, the terminal forward KL divergence is contracted pointwise and translated back via Pinsker's inequality, at the cost of a square root.

The warm-start extension unifies single- and double-loop architectures: with warm-started inner loops of size $\phi(s,a)$3, the accelerated $\phi(s,a)$4 rate is invariant for $\phi(s,a)$5, while the linear-penalty $\phi(s,a)$6 rate holds across the wider window $\phi(s,a)$7, connecting the single-loop and balanced double-loop endpoints of Agarwal et al. (Czelusniak et al., 2021).

Tabular Application and the Instance-Dependent Barrier

In the tabular setting with one-hot features and an exploratory restart distribution $\phi(s,a)$8 with $\phi(s,a)$9, the paper combines its Deterministic-Regime theory with an $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$0 approximation analysis that eliminates all concentrability constants. Under the Minimal Action Gap as the only primitive condition, the single-loop algorithm achieves $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$1 average-iterate unregularized convergence — surpassing the $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$2 worst-case statistical barrier for tabular MDPs established by minimax lower bounds (Azoury et al., 2013). This is an instance-dependent rate: the barrier applies to worst-case environments with vanishing margins, whereas the exponential translation exploits the positive action margin $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$3 to escape it. The paper explicitly leaves open whether the tabular last-iterate rate can also be improved beyond $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$4.

Limitations and Open Questions

The paper concedes several restrictions. The analysis assumes conditionally independent sampling from the discounted visitation measure; extension to Markovian trajectories is left as future work. Similarly, the critic is assumed to receive unbiased Monte Carlo estimates; bootstrapped (TD) estimates with their attendant bias are deferred. The Minimal Action Gap is a structural condition that may fail in non-tabular MDPs, and a fallback linear-penalty analysis provides only $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$5 rates when it is violated. In the Stochastic Regime, the approximation-temperature compatibility constraint excludes exact realizability, and the $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$6 analysis identifies a geometric bottleneck — the measure shift needed to transfer Fisher curvature from the parametric optimum to training policies — that prevents full elimination of $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$7 in Part I, though the Deterministic Regime is freed from all concentrability assumptions under $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$8 error. Finally, the last-iterate rates in the Deterministic Regime trail the average-iterate rates, and closing this gap is posed as an open question.

Conclusion

This paper establishes that entropy regularization, properly tuned, is not merely a stabilizer but a source of acceleration for the unregularized objective. Through an exponential translation mechanism driven by the Minimal Action Gap, a single-loop uncentered NAC achieves $$\bar{\Sigma}_{unc}(\pi) = \mathbb{E}[\phi\phi^\top]$$9 unregularized rates in the Stochastic Regime and $\theta_{t+1}$0 average-iterate rates in the Deterministic Regime, with the tabular instance-dependent rate breaking the $\theta_{t+1}$1 worst-case barrier. The regime-dependent methodology — joint Lyapunov analysis under nondegenerate Fisher curvature, PMD geometry under deterministic collapse — provides a template for analyzing entropy-regularized actor-critic methods in the temperature-vanishing limit.