Complexity-Driven Policy Optimization in Discrete RL

• CDPO is a variant of PPO for discrete-action spaces that replaces the standard entropy bonus with a complexity measure combining Shannon entropy and disequilibrium.
• It encourages structured stochasticity by steering policies away from both deterministic collapse and uniform randomness, enhancing exploration.
• Empirical results show that CDPO is more robust across tasks and less sensitive to regularization hyperparameters than traditional entropy-regularized PPO.

Searching arXiv for the specified paper and closely related work on PPO regularization and policy optimization nomenclature. Complexity-Driven Policy Optimization (CDPO) is a variant of Proximal Policy Optimization (PPO) for discrete-action reinforcement learning that replaces the standard entropy bonus with a complexity bonus defined as Shannon entropy times disequilibrium, where disequilibrium is the squared Euclidean distance from the uniform action distribution. The method is motivated by the claim that entropy maximization conflates exploration with equiprobability: entropy regularization keeps policies stochastic, but its unique maximum is the uniform distribution, which can induce unstructured and sometimes inefficient exploration. CDPO instead regularizes toward policies that are simultaneously stochastic and non-uniform, thereby favoring structured stochasticity over both deterministic collapse and pure randomness (Serfilippi et al., 24 Sep 2025).

1. Motivation and problem formulation

In policy-gradient methods, and in PPO specifically, entropy regularization is commonly used to discourage premature convergence to deterministic actions and to smooth optimization. The entropy-regularized PPO objective is written as

$$L_t(\bm{\theta})=E_{t}\left[L_{t}^{CLIP}(\bm{\theta}) - c_{vf} L_{t}^{VF}(\bm{\theta}) + c_{reg} S\left[\pi_{\bm{\theta}}\right]\!(s_t)\right],$$

where $c_{vf}$ is the value loss coefficient and $c_{reg}$ is the regularization coefficient. For a discrete policy $\pi_\theta(a\mid s)$, Shannon entropy is maximized exactly at the uniform distribution,

$$\pi_\theta(a\mid s)=\frac{1}{|\mathcal A|}, \quad \forall a\in\mathcal A.$$

The critique underlying CDPO is that maximizing entropy always pushes the policy toward equiprobability over actions, but good exploration need not be close to uniform. The paper argues that this can promote unstructured exploration, dominate the task objective when the regularization coefficient is too large, force continued sampling of many poor actions when only a subset is useful, and slow or stall learning in simpler tasks or tasks requiring precise control. The central distinction is therefore not between stochastic and deterministic behavior, but between stochasticity as such and uniformity as an overly crude exploration target (Serfilippi et al., 24 Sep 2025).

The method is motivated particularly by environments in which exploration matters but not all actions are equally useful, rewards may be sparse, the state space may be large, and the appropriate exploration level is difficult to tune. The empirical study is restricted to discrete action spaces and includes Atari games, CoinRun, and a custom multi-cart extension of CartPole called CARTerpillar, where exploration pressure becomes increasingly important as task complexity rises.

2. Complexity regularization

CDPO adopts the López-Ruiz, Mancini, and Calbet (LMC) complexity measure. For a system with $N$ accessible states and probabilities $p_i$, entropy is

$H=-\sum_{i=1}^N p_i \log p_i,$

and disequilibrium is

$D=\sum_{i=1}^N \left(p_i-\frac{1}{N}\right)^2.$

The resulting complexity is

$C=H\cdot D.$

For a discrete-action policy, the corresponding quantities are

$c_{vf}$0

$c_{vf}$1

and

$c_{vf}$2

Because the method is explicitly formulated for finite discrete action sets, the regularizer is evaluated exactly from the categorical action probabilities; no approximation is required in the reported experiments. The paper highlights several qualitative properties. Complexity vanishes for deterministic policies because entropy is zero, and it also vanishes for uniform policies because disequilibrium is zero. It is large only when the policy is both stochastic and structured. Under the simplex constraint $c_{vf}$3, the appendix states that the uniform distribution is a local minimum with $c_{vf}$4, and that there are $c_{vf}$5 maxima that are permutations of the same probability pattern. This differs sharply from entropy, whose maximum is unique at the uniform distribution (Serfilippi et al., 24 Sep 2025).

Regularizer	Expression	Induced tendency
Entropy	$c_{vf}$6	Encourages stochasticity; uniquely maximized by the uniform distribution
Complexity	$c_{vf}$7	Encourages stochasticity and non-uniform structure; vanishes at deterministic and uniform policies

This construction implies that CDPO suppresses both extremes identified in the paper as complete order and maximal disorder. A plausible implication is that the method is best understood not as maximizing randomness, but as maximizing structured stochasticity.

3. From PPO to CDPO

CDPO starts from PPO’s clipped objective

$c_{vf}$8

with value loss

$c_{vf}$9

The defining modification is simply to replace the entropy bonus with the complexity bonus: $c_{reg}$0

The regularizer is applied per sampled state $c_{reg}$1 and then averaged over the minibatch expectation, exactly as entropy is in standard PPO. It is therefore a statewise policy-distribution regularizer, not a trajectory-level complexity term. In implementation, the required modification is minimal: obtain the action probabilities, compute the disequilibrium $c_{reg}$2, multiply by entropy, and negate the minibatch mean as a loss term. The paper presents this as a very lightweight change relative to PPO with entropy regularization, with negligible overhead compared with the neural-network forward and backward passes (Serfilippi et al., 24 Sep 2025).

Algorithmically, CDPO follows standard PPO training: initialize policy and value parameters, collect trajectories $c_{reg}$3, split the rollout into minibatches, compute importance ratios

$c_{reg}$4

form the clipped surrogate, value loss, entropy, disequilibrium, and complexity, and then optimize the total objective over PPO epochs. The paper states that gradients are taken directly through the complexity term; no approximation, surrogate estimator, or stop-gradient device is introduced.

4. Optimization pressure and policy geometry

The theoretical interpretation centers on the distinct gradients induced by entropy, disequilibrium, and their product. Entropy regularization alone favors uniformity: the appendix proves via Lagrange multipliers that entropy is maximized at the uniform distribution, and the gradient sign analysis implies that action probabilities above $c_{reg}$5 are pushed downward while those below $c_{reg}$6 are pushed upward. Disequilibrium behaves in the opposite direction: it is minimized at uniformity, and its gradient pushes above-average probabilities upward and below-average probabilities downward (Serfilippi et al., 24 Sep 2025).

For CDPO, the relevant gradient is

$c_{reg}$7

The paper interprets this as a self-regulating mechanism. Near deterministic policies, $c_{reg}$8, so the entropy-like component dominates and pushes toward greater exploration. Near uniform policies, $c_{reg}$9, so the disequilibrium-driven component dominates and pushes away from pure randomness. In the intermediate regime, both terms contribute, creating pressure toward structured stochasticity rather than either collapse or equiprobability.

This differs conceptually from ordinary entropy regularization in an important way. Entropy favors equalization of action probabilities; complexity favors distributions in which multiple actions remain possible but some are preferred. The paper notes explicitly that a policy may assign many actions near-zero probability and still maintain high complexity if the remaining mass is distributed in a structured stochastic pattern. This is especially relevant in tasks where only a subset of actions is functionally useful.

5. Experimental behavior and robustness

The empirical study evaluates CDPO on discrete action spaces only: CartPole-v1 with 2 actions, CarRacing-v2 with 5, AirRaid with 6, Asteroids with 14, Riverraid with 18, CoinRun with 15, and CARTerpillar with an action space that grows by 2 for each added cart. Baselines are PPO with entropy bonus (PPOwEnt) and PPO without entropy (PPOwoEnt). All experiments use 3 random seeds, report mean return or mean episode reward with standard error, and sweep the regularization coefficient $\pi_\theta(a\mid s)$0. The paper also reports aggregated performance across all non-zero coefficients as a robustness measure rather than focusing only on the single best tuned setting (Serfilippi et al., 24 Sep 2025).

Three qualitative regimes are identified. In CartPole and CarRacing, regularization has minimal impact on final performance: CDPO, PPOwEnt, and PPOwoEnt are broadly comparable, CDPO does not hurt performance when regularization is unnecessary, and the slowdown seen with high entropy coefficients is not observed with CDPO. In CoinRun and AirRaid, aggressive entropy regularization is detrimental: in CoinRun, where only a small subset of actions may be useful, PPOwEnt degrades severely as the entropy coefficient increases and may fail to reach the optimum, whereas CDPO remains stable across all coefficient values and matches or exceeds baseline performance; in AirRaid, high entropy coefficients can completely stall learning, while CDPO remains stable over a wider coefficient range. In harder tasks such as Asteroids and RiverRaid, some regularization is beneficial, PPOwoEnt is insufficient, PPOwEnt helps only with careful tuning, and CDPO attains comparable or better returns across a broader coefficient range.

CARTerpillar is used as a controlled difficulty sweep. For $\pi_\theta(a\mid s)$1, all methods can converge. For $\pi_\theta(a\mid s)$2, PPOwoEnt degrades significantly as difficulty increases; entropy begins to help but becomes sensitive to coefficient choice; CDPO is reported as more robust and consistently better in the harder settings. The appendix further notes that for 6–8 carts all CDPO coefficients reach optimum while some entropy settings struggle; for 9 carts high entropy can make PPO unlearn optimal strategies; and for 10–11 carts only carefully tuned entropy coefficients let PPO converge, whereas CDPO converges for multiple coefficient values. The strongest empirical claim is therefore not merely that CDPO can achieve a favorable tuned score, but that it is less sensitive to $\pi_\theta(a\mid s)$3 than entropy-regularized PPO, especially when exploration must remain structured rather than purely random (Serfilippi et al., 24 Sep 2025).

The coefficient sweeps themselves are reported explicitly. For simpler environments, the tested values are $\pi_\theta(a\mid s)$4; for harder environments, $\pi_\theta(a\mid s)$5; and for additional CARTerpillar results, $\pi_\theta(a\mid s)$6. Other PPO hyperparameters are taken from RL-Baselines3-Zoo, with examples including clip range $\pi_\theta(a\mid s)$7 for most tasks and $\pi_\theta(a\mid s)$8 for Atari, and value coefficient $\pi_\theta(a\mid s)$9.

6. Scope, limitations, and related nomenclature

The most important limitation stated in the paper is that CDPO currently applies only to discrete action spaces, because disequilibrium is defined relative to the discrete uniform distribution over actions. Extension to continuous action spaces is left as future work. The empirical evaluation is also confined to relatively classic reinforcement-learning benchmarks with modest action cardinalities, so whether the same benefits scale to very large discrete action spaces remains open. In addition, $$\pi_\theta(a\mid s)=\frac{1}{|\mathcal A|}, \quad \forall a\in\mathcal A.$$0 still matters: the method is presented as more robust than entropy regularization, not as hyperparameter-free (Serfilippi et al., 24 Sep 2025).

The term “CDPO” is also non-unique in the recent literature. In “Beyond Perceptual Shortcuts: Causal-Inspired Debiasing Optimization for Generalizable Video Reasoning in Lightweight MLLMs,” CDPO denotes “Causal Debiasing Policy Optimization,” a GRPO/PPO-style objective with a repulsive KL term against a frozen bias policy rather than a complexity-based regularizer (Wu et al., 2 May 2026). In “Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning,” CDPO denotes “Conservative Dual Policy Optimization,” a two-stage model-based method motivated by instability under high model uncertainty and complexity, not by entropy–disequilibrium regularization (Zhang, 2022). These acronym collisions are terminological rather than conceptual: the discrete-action PPO variant described here is specifically the method introduced as “Complexity-Driven Policy Optimization” (Serfilippi et al., 24 Sep 2025).

Within the broader PPO-modification landscape, CDPO is most closely related to work that changes the exploration or regularization signal rather than the trust-region mechanism. “Policy Optimization with Model-based Explorations” modifies PPO by injecting a discrepancy-based exploration value into the advantage estimator, using disagreement between model-free and model-based targets as an exploration signal (Pan et al., 2018). By contrast, CDPO leaves the PPO surrogate and value-loss structure intact and changes only the regularizer applied to the policy distribution itself. This suggests a broader taxonomy in which PPO extensions can intervene either at the level of advantage estimation, as in POME, or at the level of statewise policy regularization, as in CDPO.