Global Optimality for Constrained Exploration via Penalty Regularization

Abstract: Efficient exploration is a central problem in reinforcement learning and is often formalized as maximizing the entropy of the state-action occupancy measure. While unconstrained maximum-entropy exploration is relatively well understood, real-world exploration is often constrained by safety, resource, or imitation requirements. This constrained setting is particularly challenging because entropy maximization lacks additive structure, rendering Bellman-equation-based methods inapplicable. Moreover, scalable approaches require policy parameterization, inducing non-convexity in both the objective and the constraints. To our knowledge, the only prior model-free policy-gradient approach for this setting under general policy parameterization is due to Ying et al. (2025). Unfortunately, their guarantees are limited to weak regret and ergodic averages, which do not imply that the final output is a single deployable policy that is near-optimal and nearly feasible. In this work we take a different approach to this problem, and propose Policy Gradient Penalty (PGP) method, a single-loop policy-space method that enforces general convex occupancy-measure constraints via quadratic-penalty regularization. PGP constructs pseudo-rewards that yield gradient estimates of the penalized objective, subsequently exploiting the classical Policy Gradient Theorem. We further establish the regularity of the penalized objective, providing the smoothness properties needed to justify the convergence of PGP. Leveraging hidden convexity and strong duality, we then establish global last-iterate convergence guarantees, attaining an $ε$-optimal constrained entropy value with $ε$ bounded constraint violation despite policy-induced non-convexity. We validate PGP through ablations on a grid-world benchmark and further demonstrate scalability on two challenging continuous-control tasks.

• The paper introduces the Policy Gradient Penalty method to enforce constrained maximum-entropy exploration through a quadratic penalty, yielding global convergence guarantees.
• It exploits hidden convexity in the occupancy measure space to derive last-iterate convergence results with a sample complexity of O(ε⁻⁶), validated on grid-world and continuous control tasks.
• Empirical results demonstrate that PGP outperforms primal-dual baselines with lower constraint violations and enhanced exploration coverage under diverse constraints.

Global Optimality for Constrained Exploration via Penalty Regularization

Motivation and Problem Formulation

The paper addresses constrained maximum-entropy exploration in reinforcement learning (RL), where the primary goal is to maximize the entropy of the state-action occupancy measure under general convex constraints, such as safety, resource, and imitation requirements. Unlike unconstrained entropy maximization, the presence of constraints fundamentally alters the optimization landscape, eliminating Bellman-equation decomposability and inducing non-convexity through policy parameterization. Prior approaches either rely on model-based occupancy-measure formulations, which do not scale, or policy gradient primal-dual methods offering only weak regret and ergodic guarantees, insufficient for deploying a single nearly-feasible, near-optimal policy.

The paper proposes to enforce these general constraints via a quadratic penalty regularization schema, enabling direct optimization in policy parameter space and fully leveraging standard stochastic policy gradient estimators. The resulting algorithm, Policy Gradient Penalty (PGP), reformulates constrained maximum-entropy exploration as a penalized objective and exploits hidden convexity in occupancy-measure space to establish global, last-iterate convergence guarantees.

Policy Gradient Penalty Method: Algorithmic Construction

The initial constrained RL problem is formalized as:

$$\max_{\theta \in \Theta} F_1(\theta) := -\mathbb{E}_{(s, a) \sim \lambda^{\pi_\theta}}[\log \lambda^{\pi_\theta}(s, a)] \;\;\mathrm{s.t.}\;\; F_2(\theta) := C(\lambda^{\pi_\theta}) \leq 0$$

where $\lambda^{\pi_\theta}$ denotes the state-action occupancy measure generated by policy $\pi_\theta$ under softmax parameterization.

To sidestep dual variables and nested optimization loops, PGP applies a single-loop quadratic penalty transformation:

$$\min_{\theta \in \Theta} P(\theta) := -F_1(\theta) + \beta [F_2(\theta)]_+^2$$

The policy gradient theorem enables computation of unbiased or controllably biased Monte Carlo estimators for occupancy-measure derivatives. The core algorithm computes pseudo-rewards for both objective and constraints, allowing the use of standard REINFORCE-like estimators, and updates policy parameters via stochastic gradient descent.

Figure 1: Solving constrained maximum-entropy exploration using Policy Gradient Penalty Method (PGP).

Theoretical Guarantees and Complexity Analysis

A crucial analytical insight is the identification of hidden convexity in the occupancy-measure space, which is preserved through penalty regularization. Under mild assumptions, specifically strong duality and local invertibility/Lipschitzness of the occupancy-measure mapping, the penalty formulation is shown to admit $L$-smoothness and global contraction properties, facilitating last-iterate convergence guarantees.

The main results demonstrate that, for properly selected penalty parameter $\beta = \mathcal{O}(\epsilon^{-1})$, batch size $B = \mathcal{O}(\epsilon^{-4})$, and trajectory length $H = \mathcal{O}(\log \epsilon^{-1})$, the single-loop algorithm achieves an $(\epsilon, \epsilon)$-optimal and nearly-feasible policy in $\mathcal{O}(\epsilon^{-6})$ total samples, providing both objective and constraint optimality bounds for the final iterate.

This is the first known $\lambda^{\pi_\theta}$0 sample complexity result for constrained maximum-entropy exploration under general policy parameterization and a single-loop, primal-only SGD approach, robust to non-convexity induced by policy parameterization.

Empirical Validation: Grid World and Continuous Control

The method is tested on both tabular (grid-world "Frozen Lake") and continuous-control environments ("PointMass" and "SafeCartpole"). Extensive ablation studies confirm several strong claims:

• The penalty parameter $\lambda^{\pi_\theta}$1 is highly robust; entropy remains nearly constant across several orders of magnitude once feasibility is attained, while constraint violations decrease monotonically with increasing $\lambda^{\pi_\theta}$2.
• The batch size requirement is conservative in theory; practical runs achieve good performance at much lower sample sizes.
• Compared to primal-dual policy gradient baselines, PGP yields policies with lower constraint violation and improved coverage, while primal-dual iterates often exhibit constraint oscillation and weak regret.

  Figure 2: PointMass: State-Space Coverage. SAC reference policy concentrates visits at center, PGP-trained policies yield increasingly uniform coverage as constraint budget increases.

  

  Figure 3: Entropy as a function of penalty parameter. Entropy is insensitive to $\lambda^{\pi_\theta}$3 after feasibility is achieved.

Additionally, in imitation-constrained continuous environments, PGP enables policies to strike a principled balance between exploration and closeness to reference behavior. Experiments on the SafeCartpole environment, which imposes tight positional constraints, show that PGP delivers feasible swing-up policies maximizing entropy while respecting constraints.

Figure 4: Visualization of the Frozen Lake grid-world environment used for numerical experiments.

Figure 5: PointMass: Control Trajectories. PGP policies deviate increasingly from reference as constraint budget increases, enabling broader exploration.

Figure 6: Policy training evolution on SafeCartpole: learned policy evolves to efficiently swing up within positional safety bounds.

Algorithmic and Analytical Robustness

Ablation studies further support the robustness of PGP to practical hyperparameter selection, particularly penalty parameter and batch size. Theoretical bounds on gradient bias and variance induced by finite trajectory length and Monte Carlo occupancy estimation are matched by practical training curves. The framework generalizes beyond smooth, deterministic constraints to non-smooth settings (e.g., norm constraints, imitation constraints) by instantiating exact penalty functions and relying on a unified hidden convexity analysis.

Figure 7: Unconstrained entropy objective.

Figure 8: Ablation study of the entropy function value for PGP under linear constraints.

Figure 9: Ablation study of constraint function value.

Implications, Limitations, and Future Directions

The work implies that principled, scalable constrained exploration is achievable in RL with global optimality guarantees under broad constraints and general policy classes. The approach is modular, compatible with deep policy networks, and circumvents the operational deficiencies of averaging across parameterized policy iterates.

From a theoretical perspective, the hidden convexity structure elucidated by penalty regularization may unlock further advances in non-convex, stochastic constrained optimization, with implications for safe RL, robust control, and data-driven exploration under general constraints. The framework readily generalizes to multiple constraints and non-smooth settings, connecting RL penalty methods to a broader class of hidden convexity problems.

The main limitation is currently the conservative sample complexity ($\lambda^{\pi_\theta}$4). Closing this theory-practice gap, particularly through variance reduction techniques or adaptive penalty schedules, is a promising direction for future research. Generalization to settings with partial observability or model uncertainty further remains open.

Conclusion

The paper establishes a single-loop penalty regularization methodology for constrained maximum-entropy exploration. Through algorithmic exploitation of pseudo-rewards and hidden convexity structure in occupancy-measure space, global last-iterate convergence guarantees are obtained for non-convex policy parameterizations. Empirical results on grid-world and continuous-control environments confirm robust feasibility and efficient exploration. The work forms a foundation for scalable, principled constrained exploration in RL and motivates further advances in non-convex constrained optimization.