Retry Policy Gradients in Continuous Action Spaces

Abstract: Retry-based objectives such as pass@K and max@K optimize the best return obtained from multiple sampled trajectories, and recent work has shown that they can promote exploration without explicit exploration bonuses. In discrete action spaces, ReMax was shown to do so by adapting to return uncertainty. In this work, we introduce pathwise derivative estimators for retry objectives and use them to extend ReMax to continuous action spaces. We study the resulting learning dynamics and show that, even with deterministic rewards, ReMax can encourage stochastic exploration by reshaping the policy-gradient landscape. In particular, it alters gradients both in direction, biasing updates toward higher policy entropy, and in magnitude, damping gradients and slowing convergence. We further show that Adam's adaptive normalization can mitigate this damping, depending on its numerical stabilization parameter. Empirically, we instantiate this objective as ReMax Actor-Critic (ReMAC), an off-policy actor--critic algorithm that optimizes the ReMax objective using a pathwise derivative estimator. Our experiments show that ReMAC can promote higher policy entropy without entropy regularization and achieves performance comparable to SAC.

• The paper introduces a retry-based objective that maximizes the best-of-M returns to enhance both exploitation and exploration in continuous action reinforcement learning.
• It leverages pathwise gradient estimators for Gaussian policies, enabling unbiased, low-variance policy-gradient computation in continuous domains.
• Empirical evaluations with the ReMAC algorithm show that larger retry budgets increase policy entropy and exploration, matching or exceeding SAC performance.

Retry Policy Gradients in Continuous Action Spaces: Summary and Analysis

Introduction

The paper "Retry Policy Gradients in Continuous Action Spaces" (2606.05888) rigorously establishes theoretical and empirical properties of retry-maximization (ReMax) objectives for reinforcement learning (RL) in continuous action domains. The work generalizes prior analyses in discrete spaces to continuous control, deriving practical stochastic policy-gradient estimators, examining gradient dynamics of the retry-augmented objective, and evaluating resultant exploration behaviors using an instantiated off-policy algorithm, ReMax Actor-Critic (ReMAC). By dissecting the interplay between retry-induced gradients and optimizer normalization, the study provides new insights into implicit exploration mechanisms and optimizer interactions absent explicit entropy regularization.

Theoretical Analysis of the ReMax Objective

Objective Formulation

The ReMax objective generalizes standard expected return maximization: instead of maximizing the expected reward of a single trajectory, it optimizes the expected maximum reward obtained from $M$ independently sampled actions (the "retry budget") per state under a given policy. Formally, for policy $\pi$ and value estimator $Q$,

$$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$

This objective interpolates between standard RL ($M=1$) and a "best-of-$M$" selection, with $M>1$ encouraging exploitative behavior – but, crucially, also promoting exploration by elevating the role of stochasticity.

Policy-Gradient Estimators in Continuous Spaces

Unlike previous results in discrete domains, exact gradient computation for ReMax is intractable for continuous actions. The authors leverage pathwise (reparameterization-based) gradient estimators: for a Gaussian policy, action samples are reparameterized as $a_i = \mu + \sigma \epsilon_i$ with $\epsilon_i \sim \mathcal{N}(0, I)$, enabling unbiased, low-variance differentiation via automatic differentiation. The ReMax objective is estimated by combinatorial aggregation over the batch, as in [walder2025pass], yielding gradients tractable for actor-critic optimization in continuous control.

Gradient Dynamics: Exploration By Objective Shaping

The central theoretical results characterize how the retry-maximization (for $M>1$) reshapes the policy-gradient vector field, resulting in two distinct effects:

1. Directional Entropy Increase: If the policy mean is suboptimal and the entropy is low (small $\pi$0), the policy-gradient direction induced by ReMax actively increases $\pi$1, thereby encouraging stochastic exploration (Figure 1).
2. Gradient Magnitude Damping: Near the deterministic optimum, the effective gradient norm is attenuated as $\pi$2 increases, slowing the collapse of entropy and preventing premature convergence to deterministic behavior.

   Figure 1: Normalized ReMax policy gradients across mean and standard deviation, showing that larger $\pi$3 sharply increases exploration (entropy) away from the optimum and damps gradient magnitude near the optimum.

Analytically, these findings are formalized for isotropic Gaussian policies with strongly convex $\pi$4, demonstrating that for small $\pi$5 and suboptimal $\pi$6, $\pi$7 for $\pi$8, while standard RL ($\pi$9) always drives $Q$0 downward. The result holds in high dimensions and does not require entropy bonuses.

Optimizer Interactions: Effect of Norm-Scaling

The study demonstrates that the use of Adam, with its per-coordinate normalization, can fundamentally alter ReMax's effect on exploration. Specifically, Adam can "undo" the natural gradient damping induced by retry objectives. With small $Q$1 in Adam, the effective step sizes are adaptively re-normalized, accelerating entropy collapse relative to raw SGD, but with larger $Q$2 or pure SGD, the entropy is retained longer due to the unmitigated damping.

Figure 2: Convergence speed of Adam and SGD across $Q$3 and normalization $Q$4, illustrating that Adam with small $Q$5 retains fast convergence even for large $Q$6, while SGD and Adam with large $Q$7 slow down as predicted by the vector field analysis.

Algorithms: ReMax Actor-Critic (ReMAC) Instantiation

A canonical off-policy actor-critic implementation is presented for deep RL. ReMAC uses the pathwise estimator for the ReMax objective in the policy update, and double Q-learning for stability. Crucially, ReMAC is a minimal modification of Soft Actor-Critic (SAC), replacing entropy bonus and temperature adaptation with the retry-gradient policy objective.

Empirical Findings

Return and Entropy Comparisons

Experiments were conducted on a suite of Brax continuous-control benchmarks. The following empirical claims are supported:

• Comparable Performance to SAC: Across environments, ReMAC with $Q$8 matches or slightly exceeds the final episode returns of SAC, and outperforms PPO (Figure 3).
• Increased Policy Entropy With Retry Budget: Larger $Q$9 robustly increases policy entropy in early and mid-training, particularly in environments such as Ant and Swimmer (Figure 4). This is achieved without entropy bonuses, and entropy is maintained for more epochs as $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$0 increases.

  Figure 3: Average return for SAC, PPO, and ReMAC with varying $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$1, showing ReMAC is on par with SAC as $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$2 increases.

  

  Figure 4: Average policy entropy over training for SAC and ReMAC with increasing $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$3; larger $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$4 substantially raises entropy, validating the theoretical directionality analysis.

Empirical ablations verify the dependence of entropy and convergence on the Adam $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$5 parameter, the action sample batch size $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$6, and hyperparameters. The observed patterns are consistent with the theoretical analysis.

Computational Considerations

The sampling complexity of policy evaluation increases proportionally with $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$7 for ReMAC, incurring higher wall-clock training times compared to SAC. However, the policy returns and entropy behavior are robust to reasonable ablations of $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$8.

Implications and Prospective Directions

The results demonstrate an inherently exploration-promoting mechanism arising from the retry-augmented objective in continuous control, independent from auxiliary entropy regularization or explicit exploration bonuses. This provides both a theoretically justified and empirically validated means to sustain stochasticity and mitigate premature policy determinism.

From a practical standpoint, algorithms based on ReMax objectives (e.g., ReMAC) can be seamlessly integrated into existing actor-critic architectures with minimal changes, while controlling exploration via the retry budget $$\mathcal{J}^M(\pi,Q,s) = \mathbb{E}_{a_{1:M} \,\mathrm{i.i.d.}\, \sim\, \pi(\cdot \mid s)}\left[ \max_{m=1}^M Q(s,a_m) \right].$$9 and optimizer settings.

Theoretically, the results suggest that more principled and less heuristic exploration in RL can be induced via objective shaping rather than reward engineering or randomness injection. Population-based extensions, integration with posterior sampling (for deep exploration), or hierarchical retrying are immediate avenues for further study. Fine-grained tuning of optimization hyperparameters (e.g., Adam's $M=1$0) becomes critical, as they qualitatively affect exploration properties in the presence of non-standard objectives like ReMax.

Conclusion

The paper rigorously extends the theory of retry-augmented objectives to continuous action RL, providing detailed gradient analyses, practical estimators, an actor-critic algorithm, and empirical validation of key theoretical phenomena. The findings clarify previously anecdotal observations and introduce a new paradigm for exploration in high-dimensional RL that merges control over policy entropy with minimal intervention in the objective or architecture. These results open new avenues for integrating implicit and explicit exploration strategies in scalable RL.