Improving Policy Problem in Reinforcement Learning

• Improving Policy Problem is a specific failure mode in continuous-control actor–critic reinforcement learning where delayed actor updates and noisy Q estimates hinder immediate exploitation of improved critic signals.
• The Instant Retrospect Action (IRA) method mitigates this issue by integrating local representation shaping, greedy action guidance, and instant actor updates into TD3 and DDPG frameworks.
• IRA demonstrates enhanced performance over baselines on eight MuJoCo tasks, achieving up to a 36.9% improvement in average normalized scores while maintaining better control over overestimation bias.

Searching arXiv for the target paper and closely related work on the “Improving Policy Problem.” arXiv search query: (Gao et al., 27 Jan 2026) OR "Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action" OR "Improving Policy Problem" reinforcement learning The “Improving Policy Problem” denotes a specific failure mode in online value-based reinforcement learning in which the actor does not exploit newly improved critic estimates with sufficient timeliness or reliability. In the formulation introduced by “Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Action” (Gao et al., 27 Jan 2026), the problem is not general long-run policy improvement, but the narrower question of how the current critic can immediately and effectively drive the actor toward higher-value actions despite epistemic uncertainty, noisy early $Q$ estimates, and delayed actor updates. The proposed solution, Instant Retrospect Action (IRA), augments standard backbones such as TD3 and DDPG with local representation shaping, explicit greedy policy constraints, and instant actor updates, and is evaluated on eight MuJoCo continuous-control tasks (Gao et al., 27 Jan 2026).

1. Definition and conceptual scope

The motivating setting is standard online value-based actor–critic learning, exemplified by DDPG and TD3, where the critic is first trained by temporal-difference learning and the actor is then improved by ascending the critic (Gao et al., 27 Jan 2026). The problem arises because two bottlenecks slow exploitation. First, exploratory actions in early training often fall into high-uncertainty regions where $Q$ estimates are noisy or overoptimistic, yielding unstable gradients for actor improvement. Second, common implementations such as TD3 delay actor updates, for example by updating the actor every two critic steps, which postpones exploitation of improved value estimates (Gao et al., 27 Jan 2026).

A central distinction in this literature is between general policy improvement and policy exploitation. General policy improvement is the long-run process of making a policy better using learned value functions, advantage functions, or returns. Policy exploitation emphasizes the timeliness and effectiveness with which the current critic immediately drives the actor toward higher-value actions (Gao et al., 27 Jan 2026). In this sense, the Improving Policy Problem is the failure of an online value-based learner to capitalize on critic improvements as soon as they become available.

This formulation is specific to continuous-control actor–critic RL. The same phrase appears elsewhere with different formal meanings. In observational policy learning, for example, the “improving policy problem” is formalized as producing a partial policy that provably improves upon both constant baselines, with abstention when confidence is insufficient (Bastani et al., 3 Jul 2026). In RL with verifiable rewards for LLMs, the analogous problem is framed as the absence of inter-iteration verification that an update actually improved the policy, motivating a closed-loop objective over cumulative policy improvement (Wang et al., 1 Apr 2026). These are related by theme but not by formalism.

2. Formal structure of slow policy exploitation

IRA is instantiated on a value-based online RL backbone with double critics and target networks, following the decomposition

$$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$

where $\phi(\cdot; \theta_+)$ is a nonlinear encoder and $\theta_-$ is a linear projection to the scalar $Q$-value (Gao et al., 27 Jan 2026). The critic is trained with a TD target

$$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$

The paper identifies two concrete sources of slow exploitation. The first is ineffective exploration under epistemic uncertainty: early critic errors make exploratory actions and subsequent policy gradients unreliable. The second is delayed policy updates: if the actor is only updated every $d=2$ critic steps, as in standard TD3, improved value estimates are not converted into action changes immediately (Gao et al., 27 Jan 2026).

IRA addresses the problem by restricting improvement to a local action neighborhood around the actor’s current proposal. It maintains an explored action buffer $A$ of size $n$, storing all past actions. For a state $Q$0 and current actor output $Q$1, it computes Chebyshev distance

$Q$2

sorts actions by this distance, selects the $Q$3 nearest neighbors, and scores them by the double target critics:

$Q$4

The top-ranked neighbor is the local optimum anchor $Q$5, and the second-ranked neighbor is the suboptimal anchor $Q$6 (Gao et al., 27 Jan 2026).

This converts exploitation into a local constrained optimization problem: instead of trusting the critic globally, the actor is guided toward a nearby historically explored action whose value is highest under the target critics.

3. Instant Retrospect Action and its three components

IRA augments TD3 or DDPG with three components: Q-Representation Discrepancy Evolution (RDE), Greedy Action Guidance (GAG), and Instant Policy Update (IPU) (Gao et al., 27 Jan 2026).

Component	Role	Core form
RDE	Makes $Q$7-representations locally discriminative	$Q$8
GAG	Constrains actor toward locally highest-valued neighbor	Uses $Q$9 from $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$0-NN target-critic ranking
IPU	Increases actor update frequency	Sets actor update period $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$1 instead of TD3’s $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$2

RDE is an auxiliary regularizer that enlarges the representational gap between the actor’s current action and a nearby suboptimal action:

$$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$3

Minimizing this inner product reduces similarity, encouraging the critic to distinguish neighboring actions more sharply (Gao et al., 27 Jan 2026).

GAG is an explicit policy constraint implemented through backtracked historical actions and local greedy guidance. The paper presents two equivalent forms. One writes the objective as maximizing $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$4 minus deviation from the retrieved nearest optimal action. The training form is

$$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$5

where $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$6 controls policy constraint strength and is decayed during training, for example from $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$7 to $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$8, to avoid over-conservatism (Gao et al., 27 Jan 2026). The regularizer explicitly projects actor updates toward the highest-valued local neighbor as adjudicated by the target critics.

IPU is a scheduling change rather than a new loss. IRA sets the actor update period to $$Q(s, a; \theta) = \langle \phi(s, a; \theta_+), \theta_- \rangle,$$9, so the actor and target networks are updated at every environment step where an update is scheduled: $n$9 This contrasts with TD3’s delayed-update default $\phi(\cdot; \theta_+)$0 (Gao et al., 27 Jan 2026).

The full critic and actor objectives become

$\phi(\cdot; \theta_+)$1

and

$\phi(\cdot; \theta_+)$2

The critic minimizes $\phi(\cdot; \theta_+)$3, the actor maximizes $\phi(\cdot; \theta_+)$4, and target networks are softly updated with rate $\phi(\cdot; \theta_+)$5 (Gao et al., 27 Jan 2026).

4. Mechanistic interpretation and relation to overestimation bias

IRA’s central claim is that faster exploitation requires not only more frequent actor updates, but also locally trustworthy critic structure. GAG prevents the actor from moving arbitrarily into high-uncertainty action regions, and RDE makes the critic more locally discriminative among neighboring actions (Gao et al., 27 Jan 2026). The method therefore combines a representational intervention, a policy constraint, and an update-schedule intervention.

The paper further argues that IRA’s early-stage conservatism alleviates overestimation bias in value-based RL. In standard value-based learning, function approximation and bootstrapping can create optimistic $\phi(\cdot; \theta_+)$6-targets under epistemic uncertainty; even double $\phi(\cdot; \theta_+)$7 can still leave the actor chasing spurious high values early in training. IRA mitigates this in three ways: GAG confines updates to a local neighborhood, RDE improves the critic’s local discrimination between current and suboptimal actions, and IPU allows exploitation of improved estimates without forcing long delayed jumps (Gao et al., 27 Jan 2026).

The empirical overestimation analysis reports that, with IPU enabled ($\phi(\cdot; \theta_+)$8), standard TD3 accumulates larger predicted-vs-true $\phi(\cdot; \theta_+)$9 discrepancies over training, whereas IRA’s predicted $\theta_-$0 distribution remains closer to true $\theta_-$1 values (Gao et al., 27 Jan 2026). This is notable because increasing actor update frequency alone would ordinarily be expected to worsen instability; IRA instead combines higher frequency with stronger local control.

At the implementation level, the method is computationally dominated by neighbor retrieval. For each sampled state, $\theta_-$2-NN retrieval is $\theta_-$3 for $\theta_-$4 distance plus sorting cost, with memory overhead $\theta_-$5 for the explored action buffer (Gao et al., 27 Jan 2026). In the reported setup, end-to-end runtime is approximately $\theta_-$6 h for IRA versus approximately $\theta_-$7 h for TD3 over $\theta_-$8M steps, with the overhead dominated by $\theta_-$9-NN retrieval (Gao et al., 27 Jan 2026).

5. Empirical profile, benchmarks, and sensitivity

The main empirical study evaluates IRA on eight MuJoCo continuous-control tasks: HalfCheetah-v3, Hopper-v3, Walker2d-v3, Ant-v3, Humanoid-v3, Reacher-v2, InvertedDoublePendulum-v2, and InvertedPendulum-v2 (Gao et al., 27 Jan 2026). Baselines include TD3, DDPG, PPO, PEER, ALH, and MBPO.

IRA achieves state-of-the-art or best among the listed baselines on HalfCheetah ($Q$0), Hopper ($Q$1), Ant ($Q$2), Humanoid ($Q$3), Reacher ($Q$4), and InvertedPendulum ($Q$5). It remains strongly competitive on Walker2d ($Q$6, slightly below ALH’s $Q$7) and InvertedDoublePendulum ($Q$8, close to MBPO’s $Q$9) (Gao et al., 27 Jan 2026). Average normalized scores are reported as IRA $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$0 versus ALH $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$1, PEER $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$2, TD3 $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$3, DDPG $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$4, PPO $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$5, and MBPO $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$6 (Gao et al., 27 Jan 2026). The average improvement over TD3 is $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$7 across the eight tasks (Gao et al., 27 Jan 2026).

Robustness is further analyzed with RLiable metrics, where IRA leads on Mean, IQM, and Median across tasks, indicating both higher returns and robustness (Gao et al., 27 Jan 2026). The method also transfers beyond TD3: substituting TD3 with DDPG still yields consistent gains in HalfCheetah, Humanoid, Reacher, and InvertedDoublePendulum, which the paper interprets as evidence that IRA addresses slow exploitation beyond TD3-specific delayed updates (Gao et al., 27 Jan 2026).

Implementation defaults are tightly specified. The reported backbone uses Adam with actor and critic learning rates $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$8, batch size $$L_Q^{TD}(\theta) = [Q_\theta(s, a) - y]^2,\qquad y = R(s, a) + \min_{i\in\{1,2\}} Q_{\theta'_i}(s', \pi_{\phi'}(s')).$$9, discount $d=2$0, two critics, $d=2$1, and $d=2$2 training steps. Exploration uses Gaussian noise $d=2$3 clipped to $d=2$4. IRA-specific defaults are action buffer size $d=2$5, neighbors $d=2$6, distance metric $d=2$7, RDE coefficient $d=2$8, policy constraint strength $d=2$9 decayed from $A$0 to $A$1, and update frequency $A$2 (Gao et al., 27 Jan 2026).

Sensitivity studies show that $A$3 improves over removing RDE, whereas $A$4 harms stability; $A$5 generally outperforms $A$6; larger action buffers up to $A$7 improve performance, while $A$8 introduces many low-quality actions; and $A$9 improves speed and returns on Hopper, Walker2d, and Ant, though $n$0 can stabilize late training on HalfCheetah (Gao et al., 27 Jan 2026).

6. Position within the broader policy-improvement literature

Within RL, IRA belongs to a family of methods that diagnose policy improvement failures as consequences of distributional or temporal mismatch. “TD-M(PC)$n$1: Improving Temporal Difference MPC Through Policy Constraint” identifies a policy mismatch between planner-generated data and the learned policy prior, linking that mismatch to persistent value overestimation and addressing it with a behavior-likelihood regularizer in the actor loss (Lin et al., 5 Feb 2025). “Align and Filter” analyzes policy lag in asynchronous on-policy RL and mitigates it through advantage realignment and total-variation-based filtering (Honari et al., 2 Mar 2026). “Kalman meets Bellman” improves policy evaluation rather than exploitation by tracking critic uncertainty with an EKF-based optimizer (Shashua et al., 2020). These works differ in mechanism, but all treat policy improvement as limited by the reliability, alignment, or timeliness of the signal sent from evaluator to policy.

Beyond classical control RL, the phrase expands further. In RL with verifiable rewards for reasoning models, Policy Improvement Reinforcement Learning replaces open-loop surrogate reward maximization with an explicit objective over cumulative inter-iteration policy improvement, and PIPO retrospectively verifies whether the last update improved the policy against a sliding-window baseline (Wang et al., 1 Apr 2026). In observational policy learning, a hierarchy has been proposed in which the improving policy problem sits strictly between optimal policy learning and policy-existence testing in sample complexity (Bastani et al., 3 Jul 2026). In recommender systems, “local policy improvement” refers to optimizing a KL-regularized lower bound around a logged policy without importance-ratio correction (Liang et al., 2022). These are not the same object as the RL exploitation problem studied by IRA, but they show that “policy improvement” has become a cross-domain design concern.

IRA’s limitations are correspondingly concrete. The method incurs a $n$2–$n$3 runtime increase relative to vanilla TD3, is sensitive to $n$4, $n$5, $n$6, and action buffer size, depends on the quality of local anchors found in the explored action buffer, and may require action-dimension normalization when $n$7 distance is used with uneven scaling (Gao et al., 27 Jan 2026). Its main advantage is that it does not require offline data and integrates cleanly into standard TD3 and DDPG pipelines (Gao et al., 27 Jan 2026). Its main trade-off is extra computation in exchange for faster and more stable exploitation.

In that sense, the Improving Policy Problem names a particular bottleneck of contemporary online value-based RL: critics may improve, yet actors may still exploit too slowly, too noisily, or in the wrong local direction. IRA’s significance lies in making that bottleneck explicit and turning it into a design target through local $n$8-representation shaping, greedy neighbor-guided constraints, and instant actor updates (Gao et al., 27 Jan 2026).