Online Convex Programming with RL

• OCP with RL is a framework that unifies convex optimization and reinforcement learning to address convex rewards, constraints, and regret minimization in dynamic decision processes.
• It leverages primal–dual and mirror descent methods to reformulate policy optimization as tractable saddle-point problems with proven convergence and constraint satisfaction guarantees.
• The approach supports plug-and-play architectures with standard RL subroutines, enabling practical applications in safety, diversity, and multi-objective optimization.

Online Convex Programming (OCP) with Reinforcement Learning is the algorithmic and theoretical unification of sequential convex optimization frameworks—typically formalized through Online Convex Optimization (OCO) or Mirror Descent (OMD)—with the dynamics, exploration, and policy-based learning challenges of Reinforcement Learning (RL). The central objective is to design algorithms that can handle convex rewards or constraints, or more generally convex objectives, while adapting to the temporal and stochastic structure of Markov Decision Processes (MDPs). OCP-based RL methods systematically leverage convexity for tractable saddle-point reformulations, regret analysis, constraint handling, and plug-and-play algorithmic architectures.

1. Problem Formulations: Convexity in RL

Convexity in RL arises in several key settings:

• Convex Constraints: Constraints on expected vectorial measurements, such as safety violations, diversity statistics, or expert imitation, typically enforce that $$Z(\pi) = \mathbb{E}_\pi[\sum_{t=0}^\infty \gamma^t z(s_t,a_t)] \in \mathcal{C}$$, where $\mathcal{C}$ is convex (Miryoosefi et al., 2019).
• Global Convex Objectives: The reward is not scalar but a concave/convex function $g(\bar V_{1:T})$ of vectorial averages (e.g., multi-objective, entropy, fairness) (Cheung, 2019).
• General Convex Losses Across Episodes: Episodic RL settings where, in each episode, the learner faces a convex loss $F^t$ on the induced state–action distribution $d^t$ (Moreno et al., 12 May 2025).

The common mathematical structure enables the underlying MDP optimization problem to be cast as: $$\max_{\pi} \; R(\pi) \quad \text{s.t.} \quad Z(\pi) \in \mathcal{C}$$ or more generally,

$$\max_\pi \; g\bigl(\mathbb{E}[\text{outcomes under}~\pi]\bigr)$$

subject to occupancy, flow, or other convex feasible sets. These problems are not tractable via Bellman equations when $g$ is nonlinear.

2. Primal–Dual and Online Saddle-Point Methods

The OCP paradigm translates constrained or convex RL problems into online primal–dual games:

• Lagrangian & Saddle-Point Structure: Introducing dual variables $x \in \mathcal{C}^\circ$ yields the Lagrangian,

$$\mathcal{L}(\mu, x) = \langle \mu, r \rangle - x^\top \Bigl(\sum_{s,a} \mu(s,a) z(s,a)\Bigr)$$

so that solving the primal-dual min-max game in $(\mu, x)$ or their functional equivalents finds Nash equilibria for the constrained problem (Miryoosefi et al., 2019).

• OGD-based Dual Update: At each round $t$, the dual is updated via

$$x_{t+1} = \operatorname{Proj}_{\mathcal{C}^\circ \cap B}\left(x_t + \eta \hat Z(\pi_t)\right)$$

while the primal solves an RL task with scalarized reward $r(s,a) - x_t^\top z(s,a)$.

These methods guarantee $O(DG/\sqrt{T})$ convergence of constraint violations and objective suboptimality, for suitable problem-dependent constants $D,G$ and step size $\eta$ (Miryoosefi et al., 2019). The mixture of learned policies converges to an approximate Nash equilibrium.

3. Algorithmic Architectures

Several distinct OCP-inspired RL algorithms have been developed:

• OCP Wrappers around RL Subroutines: Black-box access to standard RL solvers (policy gradient, Q-learning, model-based planners) is leveraged as an inner loop, with OGD or mirror descent orchestration handling convex objective/constraint meta-optimization (Miryoosefi et al., 2019).
• Successive Convex Approximation (SCAOPO): Constructs quadratic surrogate functions for objective and constraint costs, solves each via efficient Lagrangian minimization, and applies a two-timescale update—fast averaging for statistics, slow update for policy parameters (Tian et al., 2021).
• Episodic Mirror Descent on Occupancy Measures: For convex objective functions on occupancy measures, each episode solves

$$d^{t+1} = \arg\min_{d \in \mathcal{M}_{\mu_0}^{P^{t+1}}} \{ \tau \langle z^t - b^t, d \rangle + \Gamma(d \| \tilde d^t) \}$$

with entropy mirror maps and exploration bonuses to balance exploitation and model uncertainty (Moreno et al., 12 May 2025).

• Boosted OCP Learners: When the base policy class is large, mixtures of weak online convex learners (e.g., linear predictors) are boosted using Frank–Wolfe style updates and Moreau–Yoshida-smooth loss surrogates to efficiently approximate regret minimization over the convex hull of policies (Hazan et al., 2021).

4. Theoretical Guarantees and Regret Analysis

OCP-RL algorithms have attained state-of-the-art guarantees in both constraint satisfaction and competitive regret:

• General Regret Bounds:
  • $O(1/\sqrt{T})$ for smooth convex objectives under Frank–Wolfe or OGD-based meta-optimization (Miryoosefi et al., 2019, Cheung, 2019).
  • $O(1/T^{1/3})$ (with appropriate mirror maps) for non-smooth or G-Lipschitz convex objectives (Cheung, 2019).
  • For episodic convex RL, regret scales as $\widetilde O(L\,H^3\,|S|^{3/2}\sqrt{|A|T})$ under full information, and as $\widetilde O(T^{3/4})$ in the bandit setting (Moreno et al., 12 May 2025).
• KKT Feasibility: In SCAOPO, under standard compactness and ergodicity assumptions, every limit point $\theta^*$ is almost surely a KKT point of the original CMDP (Tian et al., 2021).
• Boosting Lower Envelope: Boosting with $N$ weak online convex learners achieves regret $O(T/\sqrt{N})$ against the convex hull of policies, with additive contributions from the base learner's regret (Hazan et al., 2021).

5. Exploration–Exploitation and Stationarity Trade-offs

Convex objectives in RL often preclude the sufficiency of stationary policies:

• Multi-Chain MDPs: To optimize global concave objectives over outcome vectors, algorithms must interleave multiple stationary policies, as stationary policies may be strictly suboptimal for certain convex objectives (Cheung, 2019). The need to alternate induces a trade-off: too-frequent policy switching incurs overhead; too-rare switching yields poor vectorial balance.
• Gradient-Thresholding for Policy Switching: A gradient threshold rule regulates policy switching, leveraging OCO oracles (Frank–Wolfe, tuned gradient/mirror descent) and optimistic planning (UCRL2) subroutines to maintain near-optimal balance and exploration (Cheung, 2019).
• Exploration Bonuses: In online convex episodic RL, exploration is enforced via count-based bonuses in the OCP update, replacing explicit optimism and ensuring sufficient state–action coverage for model learning (Moreno et al., 12 May 2025).

6. Practical Considerations and Implementation

OCP-based RL can be implemented with low computational overhead:

• Plug-and-Play Layering: OCP wrappers (e.g., ApproPO) simply reweight RL subproblems with dual variable-modified rewards, and are compatible with both model-free and model-based RL methods (Miryoosefi et al., 2019).
• Off-policy Experience Reuse: Algorithms like SCAOPO maintain rolling buffers of transitions, allow for real-time deployment, and run quadratic surrogate solves and dual subgradient updates with per-iteration cost scaling linearly in the number of network parameters and constraints (Tian et al., 2021).
• Parallelization and Resource Constraints: Primal-dual decompositions are coordinate-separable, enabling vectorized implementations and low memory footprints.
• Example Domains: Practical applications include enforcing safety (limiting unsafe action counts), promoting diversity (uniform exploration), and expert imitation via occupancy similarity metrics (Miryoosefi et al., 2019). Global concave objectives cover multi-objective optimization, fairness, and entropy maximization (Cheung, 2019).

7. Extensions, Limitations, and Connections

OCP with RL is an active research frontier:

• Continuous Spaces: Current methods generally target finite (tabular) or low-dimensional regimes; extension to continuous state–action spaces is an active area (Moreno et al., 12 May 2025).
• Bandit Feedback and Partial Information: Sublinear regret rates remain suboptimal in bandit- and limited-feedback scenarios; $\widetilde O(T^{3/4})$ is known, but closing to optimal $O(\sqrt{T})$ is an open problem (Moreno et al., 12 May 2025).
• Boosting and Oracular RL: Online boosting formalizes and extends OCP to settings where enumerating the policy space is infeasible, capturing contextual and bandit RL models using only black-box access to weak learners (Hazan et al., 2021).
• MDPwK and Other Constraints: Variants such as MDPs with knapsack or resource constraints, as well as RL with non-traditional utilities or fairness criteria, are amenable to these frameworks (Cheung, 2019).

OCP has become a unifying meta-algorithmic principle for modern constrained and general-convex RL, providing both rigorous guarantees and scalable implementations. Continued progress will hinge on improved regret rates for adversarial and bandit feedback, scalable algorithms for function approximation, and adaptive exploration in high-dimensional structured environments.