Sparse-Regularized Group Policy Optimization

• SRPO is a reinforcement learning framework that introduces sparsity-promoting regularization to control exploration and policy complexity, achieving exact zeros in suboptimal actions.
• It employs flexible regularizers like Tsallis entropy and group norms to balance deterministic and exploratory behaviors across discrete and continuous control tasks.
• The framework guarantees theoretical convergence and robust performance through regularized MDP formulations and advanced proximal optimization techniques.

Sparse-Regularized Group Policy Optimization (SRPO) is a framework for reinforcement learning and structured policy optimization which leverages explicit sparsity-promoting regularization to induce sparse and often multimodal policies, as well as group-level structure. SRPO generalizes entropy-regularized and group-sparse RL approaches by formulating policy learning as a regularized Markov decision process (MDP) in which the objective is augmented by a flexible regularizer designed to control exploration, selective action support, and policy complexity. This paradigm encompasses a variety of mathematical principles, proximal algorithms, practical actor-critic implementations, and domain-specific extensions in both discrete and continuous control, including recent advances for LLMs and robotic policy optimization.

1. Mathematical Foundations and Regularized MDPs

The unifying mathematical model for SRPO is the regularized MDP, where the value function is defined as: $$V_\lambda(s) = \max_{\pi} \mathbb{E}\Big[\sum_{t} \gamma^t [r(s_t,a_t) + \lambda \cdot \varphi(\pi(a_t|s_t))]\Big]$$ with $\varphi$ an action-probability-dependent regularization function (e.g. Shannon entropy, Tsallis entropy, group penalties), and $\lambda$ the regularization strength. The optimal policy is characterized by KKT conditions: $$\pi^*_\lambda(a|s) = \max\Big\{g_\varphi\left(\frac{\mu^*_\lambda(s) - Q^*_\lambda(s,a)}{\lambda}\right),\, 0\Big\}$$ where $g_\varphi$ is the inverse of $f_\varphi'(x) = \varphi(x) + x \varphi'(x)$, and the normalization multiplier $\mu^*_\lambda(s)$ ensures a valid distribution over actions. The support and degree of sparsity in $\pi^*_\lambda$ are determined by the properties of $f_\varphi'$, especially its limit at zero. Important instances include Tsallis entropy ($\varphi(x)=\tfrac{1}{2}(1-x)$) and trigonometric/exponential forms, which allow for strict sparsity (actions with zero probability), in contrast to Shannon entropy, which enforces strictly positive probabilities (Li et al., 2019).

2. Inducing Policy Sparsity and Group Structure

SRPO enables explicit control over the sparsity of learnt policies by choice of regularization:

• If $f_\varphi'(0)$ is finite, sparse solutions emerge, allowing for exact zeros in the probability mass assigned to actions or groups.
• The regularization coefficient $\lambda$ modulates the trade-off: small $\lambda$ drives determinism/sparsity; large $\lambda$ yields uniform, exploratory policies.
• Recent advances extend this to group structures, where regularizers introduce penalties on sets (groups) of actions or parameters, promoting entire blocks of zeros. For example, group $\ell_{0}$ or $\ell_{2,0}$ regularization, or mixed $\ell_1/\ell_2$ norms in neural architectures.

Key proximal algorithms (e.g., iterative hard thresholding, DC decomposition, subspace-accelerated proximal methods) provide efficient routines for enforcing both element-wise and group-wise sparsity (Ye et al., 30 May 2025, Li et al., 2021, Curtis et al., 2020), with finite support identification and global convergence results under standard convexity and smoothness assumptions.

3. Algorithmic Strategies and Practical Implementations

SRPO encompasses several classes of algorithms:

• Regularized Policy Iteration (RPI): Alternates between policy evaluation and regularized improvement via solving the regularized convex optimization at each step (Li et al., 2019).
• Regularized Actor-Critic (RAC): Incorporates regularization into off-policy actor-critic updates; often stabilizes learning by using dual Q-functions to reduce overestimation bias.
• Two-stage Stochastic Optimization: Combines stochastic proximal gradient steps for initial localization and subsequently applies aggressive half-space projections for enhanced group sparsity (Chen et al., 2020).
• Fat-to-Thin Policy Optimization (FtTPO): Maintains a “fat” proposal policy with broad support and a “thin” sparse actor policy, transferring off-policy knowledge using weighted actor losses and $q$-Gaussian parametrizations (Zhu et al., 24 Jan 2025). This secures safe policies in high-stakes domains.

Recent LLM applications leverage SRPO variants such as history-resampling, group-based advantage estimation, and cross-domain two-stage training to scale mathematical and code reasoning in large models, demonstrating improved efficiency and robustness (Zhang et al., 19 Apr 2025, Wan et al., 2 Jun 2025).

Table: Regularizers and Policy Sparsity Properties

Regularizer $\varphi(x)$	$f_\varphi'(0)$ finite?	Support of Policy $\pi^*_\lambda$
Shannon: $-\log x$	No	Full (nonzero for all actions)
Tsallis: $0.5(1-x)$	Yes	Sparse (actions with zero probability)
Trigonometric	Yes	Sparse
Group $\ell_{2,0}$, $\ell_1$	Yes	Group-wise sparse

4. Extensions to Continuous Control and Robotics

Group-based regularization is extended to continuous control domains—critical in robotics—via trajectory-based policy clustering, state-aware advantage normalization, and group-normalized PPO objectives (Khanda et al., 25 Jul 2025). Sample and computational complexity are rigorously analyzed, with per-iteration complexity scaling as $O(NT(d_\phi K + d_s + d_\theta))$. Adaptive regularization (e.g., temporal smoothness, inter-group diversity) improves training stability for high-dimensional, temporally extended control tasks.

5. Theoretical Guarantees and Error Bounds

SRPO frameworks provide robust theoretical guarantees:

• The contraction properties of regularized Bellman operators ensure uniqueness and convergence of fixed points.
• Error bounds tightly relate the performance gap between regularized and unregularized optimal values to the regularization strength and properties of $\varphi$, as: $$|V^*_\lambda(s) - V^*(s)| \leq \frac{\lambda}{1-\gamma} \varphi(1/|\mathcal{A}|)$$
• For state-regularized approaches (e.g., under dynamics shift), performance lower bounds are expressed in terms of KL-divergence and bounded dynamics shift (Xue et al., 2023): $$\eta_T(\hat{\pi}) \geq \eta_T(\pi^*_T) - \frac{\lambda_1 \lambda_2 \varepsilon_m + 2\lambda_1 + \sqrt{2}R_{\max} \sqrt{\varepsilon_s}}{1-\gamma}$$

6. Empirical Performance and Applications

SRPO has been validated on a broad suite of benchmarks:

• In RL, Gridworld, Atari, and MuJoCo, SRPO leads to multimodal, sparse policies that outperform entropy-only regularization in both sample efficiency and avoidance of poor actions (Li et al., 2019).
• In deep network compression, half-space projections yield highly sparse networks while maintaining competitive accuracy (Chen et al., 2020).
• Safety-critical domains see stable, robust learning of policies that strictly avoid dangerous actions by enforcing zero probability support (Zhu et al., 24 Jan 2025).
• LLM reasoning tasks demonstrate significant improvement in pass@1 scores and reasoning reflection quality, achieved with a fraction of conventional RL training steps, due to staged, sample-efficient optimizers (Zhang et al., 19 Apr 2025, Wan et al., 2 Jun 2025).

7. Future Directions and Open Questions

SRPO suggests several avenues for continued research:

• Adaptive tuning of regularization parameters ($\lambda$, group penalty weights) based on validation feedback or sparsity progress.
• Integrating more structured sparsity settings (overlapping, hierarchical groups, graph-structured regularizers) to further improve interpretability and computational efficiency.
• Developing scalable switching criteria for multi-stage optimization and refining group clustering methods for continuous and high-dimensional action spaces.
• Deepening theoretical analysis, particularly on sensitivity to simulation budget, regularization scaling, and support identification.
• Extending SRPO principles to multi-agent, meta-learning, and hybrid model-based/model-free frameworks.

SRPO therefore provides a rigorous mathematical and algorithmic foundation for sparse and group-structured policy optimization, bridging reinforcement learning, high-dimensional optimization, and modern policy design for safety, interpretability, and efficiency.