Autoregressive Policy Network in RL

• Autoregressive policy networks are reinforcement learning policies that use stationary autoregressive processes to generate temporally correlated actions.
• They enhance exploration efficiency and produce smoother, safer control trajectories by tuning noise parameters like α in continuous control settings.
• Integration is straightforward through state augmentation, preserving standard normal marginals while allowing domain-specific tuning of temporal coherence.

An autoregressive policy network, in the context of modern reinforcement learning (RL) and sequential decision-making, systematically generates actions by modeling temporal dependencies directly in the exploration noise or the policy structure itself. This approach contrasts with classical policies in which the action distribution is often independent across time. By leveraging autoregressive stochastic processes, such networks can achieve smoother, more temporally coherent exploration and safely operate in demanding continuous control settings.

1. Motivation: Temporal Coherence in Exploration

In continuous control RL, policies with Gaussian exploration noise—i.e., policies where actions are drawn as $$a_t = \mu_\theta(s_t) + \sigma_\theta(s_t)\epsilon_t$$, $\epsilon_t \sim \mathcal{N}(0,I)$—are the standard due to tractability and analytic convenience. However, the "white noise" generated in this manner is temporally uncorrelated across timesteps, resulting in jerky trajectories, low sample efficiency, and poor exploration, especially at high control rates.

In physical systems, such as robotics, this lack of smoothness not only impedes exploration but also poses safety risks. Trajectory smoothness is crucial for hardware safety and efficient discovery of rewarding behaviors, particularly under sparse reward scenarios and tight action constraints. Thus, a more physically plausible noise process, which incorporates temporal correlation, is desired.

2. Stationary Autoregressive Processes for Exploration

The core advance of the autoregressive policy network is to replace the independent Gaussian noise with samples from a stationary autoregressive (AR) stochastic process of order $p$: $$X_t = \sum_{k=1}^p \phi_k X_{t-k} + Z_t,\;\;\; Z_t \sim \mathcal{N}(0,\sigma_Z^2)$$ The coefficients $\{\phi_k\}$ are chosen so that the process is stationary—all roots of the characteristic polynomial are inside the unit circle—and the marginals are standard normal: $X_t \sim \mathcal{N}(0,1)$. In a specific sub-family, setting

$\phi_k = (-1)^{k+1} \binom{p}{k} \alpha^k$

with scalar $\alpha \in [0,1)$, allows continuous control over the temporal coherence: $\alpha\to 0$ recovers white noise, while $\alpha\to1$ produces highly persistent ("red") noise.

The temporal coherence of process $\{X_t\}$—that is, the autocorrelation between $X_t$ and $X_{t-\Delta}$—is directly controlled by $\alpha$ and $p$. This direct control is essential for tuning the smoothness of the resulting actions and for adapting exploration to problem characteristics.

3. Policy Network Integration and RL Compatibility

The autoregressive policy is implemented by replacing the standard policy noise $\epsilon_t$ with the temporally correlated $X_t$: $a_t = \mu_\theta(s_t) + \sigma_\theta(s_t) X_t$ To ensure $X_t$ is computed in a way compatible with the agent-environment loop, the process is expressed recursively in terms of previous actions: $$a_t = \mu_\theta(s_t) + \sigma_\theta(s_t) \left[ \sum_{k=1}^p \phi_k \frac{a_{t-k} - \mu_\theta(s_{t-k})}{\sigma_\theta(s_{t-k})} + \sigma_Z \epsilon_t \right],\;\; \epsilon_t \sim \mathcal{N}(0, I)$$ Here, the RL problem is formalized over an extended state space, $$s'_t = (s_{t-p}, a_{t-p},\dotsc,s_{t-1},a_{t-1},s_t)$$, making the process Markovian with respect to this augmented state. This ensures compatibility with existing RL methods (including policy gradient estimators) and requires minimal change to the typical agent-environment interface.

This construction is algorithmically simple to integrate with standard on-policy and off-policy RL algorithms; empirical code modifications are generally limited to (i) augmenting the state representation with a window of past states and actions and (ii) substituting the sampling of policy noise.

4. Empirical Results: Exploration Efficiency, Smoothness, and Safety

The stationary AR policy network yields several significant empirical benefits:

• Improved Exploration and Sample Efficiency: In both sparse and dense reward environments, AR policies exhibit faster exploration of the state space and quicker discovery of rewarding regions compared to Gaussian policies. In particular, in tasks such as sparse-reward Mujoco environments, ARPs consistently reached reward thresholds more rapidly and reliably.
• Smooth Trajectories: Temporal correlation in noise directly manifests as smooth trajectories, essential for high-frequency control (e.g., 100–125 Hz). High $\alpha$ values maintain effective exploration even as control frequency increases, whereas Gaussian exploration degrades and gets "clipped" by action bounds.
• Hardware Safety: Real-world robotic experiments (UR5 arm) demonstrate that ARPs avoid the unsafe, jittery exploration characteristic of independent noise, thereby reducing hardware stress and the risk of damage.
• Competitive or Superior Benchmark Results: On standard continuous control benchmarks (e.g., Swimmer-v2 in Mujoco), ARPs matched or outperformed Gaussian policies, especially in environments emphasizing smooth, coordinated actions.

5. Theoretical Properties and Trade-offs

• Standard Normal Marginals: Despite the AR dependence, each $X_t$ is marginally standard normal, preserving the effective scale of exploration and ensuring exploration amplitude comparability with non-AR approaches.
• Tunable Temporal Coherence: The $\alpha$ parameter provides a continuous spectrum from fully decorrelated to fully persistent noise, enabling domain-specific tuning of exploration structure without sacrificing stochasticity.
• Extended Markov Property: Formulating the RL problem over recent history ($p$ timesteps) ensures that core RL theorems (e.g., the policy gradient theorem) apply with minimal modification.

Potential trade-offs include increased memory and computation for high-order AR processes (larger $p$) and possible need for additional tuning of $\alpha$ in highly nonstationary environments. However, these trade-offs are often mitigated by the gains in sample efficiency and safety.

6. Implementation and Deployment Considerations

• Computational Cost: The forward computation at each step includes $O(p)$ additional operations (scaling with AR order) to maintain and update the history buffer.
• State Augmentation: Practical implementation in standard RL frameworks requires extending the state input and tracking previous actions and states for the AR recursion.
• Policy Parameterization: Since the autoregressive noise has the same marginal distribution as in the standard Gaussian case, policy architectures and learning rates do not require wholesale changes.
• Deployment in Physical Systems: Empirical results justify deployment in constraints-sensitive scenarios (robotics), where safety-critical characteristics such as smooth exploration are mandatory, and non-i.i.d. action noise leads to tangibly safer behaviors.

7. Future Directions

Extensions suggested by the paper include:

• Expansion to higher-dimensional and more complex action spaces, exploring richer AR parameterizations.
• Combining ARP noise with other exploration strategies (e.g., parameter-space noise, auxiliary rewards).
• Deeper analysis of the extended MDP induced by ARPs and its relationship to underlying Markovian policies.
• Application to additional RL tasks requiring fine-tuned exploration–exploitation trade-offs, especially those featuring sparse, delayed, or safety-critical rewards.

Further investigation into these directions could yield new methodological advances in exploration for both simulated and real-world RL problems.

---

In summary, autoregressive policy networks introduce temporally correlated stochasticity for exploration by employing stationary AR processes within the policy structure. This modification produces smoother and safer trajectories, improves exploration efficiency, and easily integrates with the broad spectrum of RL algorithms by maintaining standard agent–environment protocols and marginal noise properties. This paradigm is empirically validated across simulated and real-world robotic control benchmarks, establishing ARPs as an effective alternative to classic Gaussian noise-based policies (Korenkevych et al., 2019).