Batch-Constrained Actor-Critic Methods

• The paper introduces NAC-B, a batch-constrained actor-critic algorithm that averages gradient estimates over contiguous transitions to mitigate bias and variance.
• It achieves order-optimal 𝒪̃(√T) regret by setting the batch size as Θ(√T), ensuring controlled error bounds without full ergodicity assumptions.
• The architecture leverages online, sequential data collection without simulator resets, making it effective for unichain MDPs with transient and periodic states.

Batch-constrained actor-critic architectures constitute a class of reinforcement learning (RL) algorithms that structure both actor and critic updates around batches of consecutive Markovian transitions, rather than single-step or fully i.i.d. samples. This architectural constraint addresses bias and variance in gradient estimates that arise in average-reward Markov Decision Processes (MDPs), especially in the absence of standard ergodicity assumptions. The NAC-B (Natural Actor-Critic with Batching) algorithm exemplifies the batch-constrained approach, achieving order-optimal $\tilde{O}(\sqrt{T})$ regret in infinite-horizon average-reward unichain MDPs, a setting that permits both transient states and system periodicity (Ganesh et al., 26 May 2025).

1. Foundations of Batch-Constrained Actor-Critic Methods

Batch-constrained actor-critic methods operate by alternating between actor (policy) and critic (value) updates, each performed over finite batches of $B$ contiguous state-action-reward-next state tuples extracted from a fixed policy roll-out. In contrast to classical actor-critic methods relying on single-step temporal-difference or stochastic policy gradients, batch-constrained variants average gradient estimates over each batch, reducing both transient-state and periodicity-induced biases as well as stochastic variance in both subroutines. NAC-B structures the overall optimization into $K$ outer epochs, each comprising $H$ inner loops for both critic and actor, processing $2HB$ consecutive transitions for each policy parameterization.

2. Algorithmic Structure and Update Mechanisms

The NAC-B architecture is organized as follows:

• Policy Fixation: In each epoch $k$, the current policy $\pi_{\theta_k}$ is fixed.
• Critic Batch Updates: Over $H$ batches of size $B$, critic parameters $\xi_k = [\eta_k, \zeta_k]^T$ are updated using a batched temporal-difference objective, incorporating all $B$0 contiguous transitions at each inner step. The update utilizes

$B$1

where $B$2 and $B$3 encode the policy and feature-dependent matrices and rewards.

• Batched Advantage Estimation: The critic produces advantage estimates $B$4 for use by the actor.
• Actor Batch Updates: Using identical batching and $B$5 additional sub-loops, the actor computes a natural gradient estimate $B$6 via least-squares fitting over the same batch transitions:

$B$7

• Policy Update: Policy parameters are updated as $B$8.
• Data Collection: All batches are contiguous on a single trajectory, avoiding simulator resets and leveraging the Markov property to gather relevant transition statistics.

3. Formal Analysis: Bias, Variance, and Convergence

Classical RL regret analyses often assume geometric mixing (ergodicity) of the Markov chain for tractable bounds on bias and variance in empirical averages. NAC-B relaxes this via the unichain assumption, embracing periodic and transient states. The algorithm introduces two critical constants—$B$9 and $K$0:

• $K$1 quantifies the worst-case expected time to reach the chain’s recurrent class from any initial state.
• $K$2 measures the expected return time to a recurrent-class state sampled from the stationary distribution.

For any batch of $K$3 contiguous transitions, the deviation of empirical state occupancy from the stationary distribution is $K$4 (Lemma 4.1), and variance in empirical mean estimates is $K$5 (Lemma 4.2). By setting $K$6, both bias and variance are driven to $K$7, supporting overall order-optimal $K$8 regret scaling.

4. Pseudocode and Implementation Details

The implementation is formalized in Algorithm 1:

$H$3

Total trajectory length is $K$9. All transitions are contiguous, exploiting the structure of Markovian data and ensuring no resets.

5. Advantages and Limitations in Unichain MDPs

Batching in NAC-B offers several core benefits:

• Mitigation of Non-Ergodicity: Markov transition kernels in unichain MDPs may not mix exponentially fast; batching ensures empirical distributions over $H$0 steps converge to stationary with error $H$1.
• Bias and Variance Control: Simultaneous reduction of transient/periodic bias and sampling variance enables rigorous high-probability error bounds, forming the basis of the regret analysis.
• Elimination of Simulator Resets: The fully online, single-trajectory nature with batching avoids the need for repeated stochastic restarts, characteristic of prior approaches under stronger assumptions.

A plausible implication is that batch-constrained architectures can generalize beyond the unichain assumption if analogous bounds on empirical stationary convergence can be established for broader MDP classes.

6. Theoretical Significance and Connections

NAC-B demonstrates that batch-constrained actor-critic methods can achieve non-asymptotic $H$2 regret in average-reward RL without strict mixing conditions. This approach “batches out” Markov noise similarly to variance reduction techniques in stochastic optimization, but is tailored for dependencies inherent in sequential data. The unichain assumption stands as one of the weakest under which the policy gradient theorem for average-reward remains valid. The batch-constrained framework thereby extends the applicability of scalable RL algorithms to environments where ergodicity is not assured, aligning theoretically justified updates with algorithmic practicality (Ganesh et al., 26 May 2025).