Analysis of Bootstrapping-Based Critic Estimates

Analyze the convergence of the single-loop, entropy-regularized, uncentered Natural Actor-Critic algorithm when the critic uses bootstrapping-based Temporal-Difference estimates instead of unbiased Monte Carlo estimates.

Background

The main theoretical analysis assumes that the critic receives an unbiased estimate of the regularized action-value function, which can be obtained using geometrically stopped Monte Carlo rollouts. The paper also describes a Temporal-Difference alternative based on bootstrapping, but notes that this introduces approximation bias into the critic update.

No convergence analysis is provided for the bootstrapping-based estimator. Establishing guarantees for this practically important variant would require controlling the resulting bias jointly with actor updates, critic tracking, entropy regularization, and the deterministic-regime geometry.