Monotonic Inference Policy Improvement (MIPI)

• MIPI is a reinforcement learning framework that couples policy improvement with an inference mechanism while preserving monotonic performance guarantees.
• It employs techniques like compatible critics, trust-region constraints, and safe gradient updates to mitigate bias and ensure stable policy enhancement.
• MIPI unifies diverse RL approaches—ranging from surrogate improvement to world models—providing a methodological doctrine for disciplined inference in policy updates.

Searching arXiv for the primary paper and closely related monotonic policy improvement work to ground the article and citations. Monotonic Inference Policy Improvement (MIPI) is an interpretive umbrella for reinforcement-learning results in which a policy-improvement step is coupled to an inference mechanism—typically a critic, an advantage estimator, a world model, a planner, or an observation-based surrogate—while preserving a monotonicity property. Depending on the formulation, monotonicity may refer to the exact return $J(\pi)$, an Approximate Policy Iteration surrogate, an entropy-regularized objective, a high-probability safety condition, a monotone lower-bounding critic, or performance at a fixed initial state. In the supplied literature, the most direct single-policy formulation is the trust-region/API lower bound

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$

together with conditions under which approximate inference of $Q^{\pi_\theta}$ does not bias the surrogate gradient (Tomczak et al., 2019).

1. Surrogate improvement as the basic MIPI template

In the discounted MDP setting $\langle S, A, P, R, \rho_0, \gamma\rangle$, a standard monotonic-improvement construction starts from the API/TRPO-style surrogate

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$

When $\pi_{\tilde\theta}$ is close to $\pi_\theta$, this surrogate approximates $J(\pi_{\tilde\theta})$. The accompanying trust-region lower bound,

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$

with

$$\epsilon = \max_{s,a} |A^{\pi_\theta}(s,a)|, \qquad \alpha = \max_s \frac{1}{2} \int |\pi_\theta(a|s) - \pi_{\tilde\theta}(a|s)| \, da,$$

is the canonical monotonic policy-improvement statement: increasing the surrogate while keeping policy change small controls performance deterioration (Tomczak et al., 2019).

The exact surrogate gradient is

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$0

This identity isolates the MIPI problem in its sharpest form: a policy update is only as trustworthy as the inference mechanism used to approximate $$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$1. If the critic or advantage estimator changes the expectation of this gradient, monotonic ascent of the true surrogate is no longer guaranteed even if a trust region is enforced (Tomczak et al., 2019).

A closely related line replaces worst-case divergence terms by expected divergences. Easy Monotonic Policy Iteration gives

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$2

thereby making the lower bound sample-estimable and differentiable in large-scale settings (Achiam, 2016). An analytical trust-region update derived by calculus of variations yields

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$3

with $$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$4, giving a nonparametric inference-like policy update with a monotonic improvement guarantee (Li et al., 2021).

2. Compatible critics and unbiased surrogate gradients

The primary contribution of "Compatible features for Monotonic Policy Improvement" is to characterize when a parametric critic can be inserted into the API surrogate gradient without introducing bias. This is the point at which “inference” becomes technically central: the critic is not merely a variance-reduction device but part of the monotonic-improvement argument itself (Tomczak et al., 2019).

The paper first recalls the classic policy-gradient compatibility condition. For a differentiable critic $$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$5, unbiasedness of the policy-gradient theorem is recovered when the critic satisfies a least-squares orthogonality condition and

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$6

which implies the familiar structure

$$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$7

The paper’s main step is to derive the analogous condition for the two-policy surrogate $$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$8 (Tomczak et al., 2019).

Two compatible forms are given. The first uses importance-weighted features: $$J(\pi_{\tilde\theta}) \;\ge\; L_{\pi_\theta}(\pi_{\tilde\theta}) - \frac{4 \epsilon \gamma}{(1-\gamma)^2} \alpha^2,$$9 The second, cleaner form removes the inner importance ratio by changing the regression weighting: $Q^{\pi_\theta}$0 Under the corresponding orthogonality condition, the critic-based gradient equals the true surrogate gradient: $Q^{\pi_\theta}$1 Empirically, the second form is realized by the weighted regression

$Q^{\pi_\theta}$2

The baseline choice $Q^{\pi_\theta}$3 preserves unbiasedness and reduces variance. This produces a strict API analogue of Sutton-style compatible function approximation: compatibility is no longer defined relative to $Q^{\pi_\theta}$4, but relative to $Q^{\pi_\theta}$5 (Tomczak et al., 2019).

The theoretical consequence is a direct criticism of mainstream actor-critic practice. The paper states that state-of-the-art methods such as TRPO and PPO generally do not use critics of the compatible form and do not train them with the required weighted regression. As a result, their practical gradients are biased: $Q^{\pi_\theta}$6 A controlled NChain experiment makes the point concrete. A standard linear critic,

$Q^{\pi_\theta}$7

trained by unweighted least squares, retains non-zero bias as rollout count grows, whereas the compatible critic based on

$Q^{\pi_\theta}$8

and importance-weighted regression is unbiased by construction (Tomczak et al., 2019).

3. Safe stochastic-gradient monotonicity

A second major interpretation of MIPI is safety: never deploying a policy whose expected return is worse than before. "Smoothing Policies and Safe Policy Gradients" formulates this directly as

$Q^{\pi_\theta}$9

for all iterations $\langle S, A, P, R, \rho_0, \gamma\rangle$0, under actor-only policy gradient and a $\langle S, A, P, R, \rho_0, \gamma\rangle$1-smoothing policy class (Papini et al., 2019).

The analysis derives global smoothness of $\langle S, A, P, R, \rho_0, \gamma\rangle$2: $\langle S, A, P, R, \rho_0, \gamma\rangle$3 and therefore the standard lower bound

$\langle S, A, P, R, \rho_0, \gamma\rangle$4

For exact gradient ascent,

$\langle S, A, P, R, \rho_0, \gamma\rangle$5

the optimal safe step size is $\langle S, A, P, R, \rho_0, \gamma\rangle$6, yielding deterministic per-iteration monotonic improvement: $\langle S, A, P, R, \rho_0, \gamma\rangle$7

The stochastic case replaces exact gradients by $\langle S, A, P, R, \rho_0, \gamma\rangle$8 and introduces concentration bounds of the form

$\langle S, A, P, R, \rho_0, \gamma\rangle$9

This yields a lower bound $$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$0 on improvement and, after joint optimization of step size and batch size,

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$1

Safe Policy Gradient (SPG) then guarantees, with probability at least $$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$2,

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$3

This is a high-probability MIPI formulation: the inference object is the noisy gradient estimate, and monotonicity is enforced by adaptive control of both step size and sample count (Papini et al., 2019).

The same paper also generalizes strict monotonicity to baseline and milestone constraints such as

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$4

showing that “monotone improvement” can be relaxed to controlled non-degradation relative to a reference trajectory without abandoning the same proof machinery (Papini et al., 2019).

4. KL-regularized, trust-region, and inference-style updates

A third strand of MIPI treats policy improvement itself as an inference operation. "An Analytical Update Rule for General Policy Optimization" derives the closed-form update

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$5

with

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$6

Because this update maximizes a lower bound involving an expected-KL penalty rather than a max-state KL, it carries a monotonic improvement guarantee on the true return $$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$7, not merely on a regularized objective (Li et al., 2021).

"Easy Monotonic Policy Iteration" provides a related average-divergence lower bound,

$$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$8

and shows how this leads to a sample-based monotonic policy-iteration objective that is easier to optimize than bounds involving sup norms (Achiam, 2016). "On- and Off-Policy Monotonic Policy Improvement" further replaces purely on-policy expectations by a mixture of on- and off-policy data, obtaining a lower bound whose penalty depends on both the change from $$L_{\pi_\theta}(\pi_{\tilde\theta}) := J(\pi_\theta) + \mathbb{E}_{s \sim \rho_{\pi_\theta},\, a \sim \pi_{\tilde\theta}(\cdot | s)} \big[ A^{\pi_\theta}(s,a) \big].$$9 to $\pi_{\tilde\theta}$0 and the mismatch between the behavior policy $\pi_{\tilde\theta}$1 and $\pi_{\tilde\theta}$2 (Iwaki et al., 2017).

Entropy-regularized value-based methods make the same idea explicit through cautious interpolation. In "Ensuring Monotonic Policy Improvement in Entropy-regularized Value-based Reinforcement Learning" and "Cautious Policy Programming," the candidate policy $\pi_{\tilde\theta}$3 is not deployed directly; instead one uses

$\pi_{\tilde\theta}$4

with $\pi_{\tilde\theta}$5 selected from an entropy-aware lower bound involving the expected policy advantage $\pi_{\tilde\theta}$6 and the regularization quantity

$\pi_{\tilde\theta}$7

The resulting guarantee is

$\pi_{\tilde\theta}$8

which yields a cautious value-based MIPI scheme that explicitly trades off performance and stability (Zhu et al., 2020, Zhu et al., 2021).

5. Extensions: reliable critics, world models, multi-agent systems, and partial observability

A broad range of later work extends monotonic-improvement logic beyond surrogate critics. "Reliable Critics: Monotonic Improvement and Convergence Guarantees for Reinforcement Learning" moves the constraint from the actor to the evaluator. Reliable Policy Iteration replaces standard projection-based policy evaluation by

$\pi_{\tilde\theta}$9

This yields a coordinate-wise non-decreasing sequence $\pi_\theta$0, each $\pi_\theta$1 lower-bounds the true $\pi_\theta$2, and the model-free penalty-barrier realization can be inserted into DQN and DDPG as a “reliable critic” loss (R. et al., 8 Jun 2025). This is a critic-centric MIPI: the monotonic object is the inferred value function, which in turn supports safer greedy improvement.

Model-based formulations push the same idea into planning. "Theoretically Guaranteed Policy Improvement Distilled from Model-Based Planning" extends SAC policy improvement to a multi-step planning objective

$\pi_\theta$3

defines $\pi_\theta$4, and proves

$\pi_\theta$5

for all $\pi_\theta$6, together with convergence to the maximum-entropy optimal policy (Li et al., 2023). "Deep SPI: Safe Policy Improvement via World Models" then gives an online deep-RL analogue: if policy updates stay within an importance-ratio neighbourhood and local reward and transition losses are small, then

$\pi_\theta$7

so model-based gains dominate real-environment losses whenever the error term $\pi_\theta$8 is small enough (Delgrange et al., 14 Oct 2025).

Multi-agent and partially observable settings require additional structure. In heterogeneous MARL, "Improving monotonic optimization in heterogeneous multi-agent reinforcement learning with optimal marginal deterministic policy gradient" replaces fragile sequential policy-ratio baselines by Optimal Marginal Q-values and a pessimistic Generalized Q Critic, specifically to reconcile monotonic improvement with Partial Parameter-Sharing (Yu et al., 14 Jul 2025). In finite-state discounted MDPs, on-line policy iteration with policy switching guarantees

$\pi_\theta$9

when only the action at the current state is switched, and converges to a local-MDP optimum, or to the original MDP optimum when the MDP is communicating (Chang, 2021). For episodic POMDPs, memoryless policy iteration alternates single-stage observation-based improvements with policy evaluations according to a periodic pattern and proves

$J(\pi_{\tilde\theta})$0

with the computationally optimal minimal-period pattern given by a forward sweep followed by a backward sweep (Zuijlen et al., 11 Dec 2025). In deterministic finite-horizon control with a fixed initial state, on-line policy iteration with trajectory-driven policy generation guarantees

$J(\pi_{\tilde\theta})$1

under a consistency condition on the generator, illustrating a fixed-start-state version of monotonic improvement that applies to trajectory-trained neural policies (Li et al., 16 Apr 2026). A continuous-control analogue, MOTO, derives a lower bound on performance change under an expected KL constraint for linear-Gaussian trajectory policies and uses a locally quadratic $J(\pi_{\tilde\theta})$2-function rather than linearized dynamics (Akrour et al., 2016).

6. Scope, misconceptions, and significance

The supplied literature does not use “Monotonic Inference Policy Improvement” as a uniform algorithm name. Rather, one paper explicitly states that its results “can be interpreted as Monotonic Inference Policy Improvement,” and the broader corpus supports reading the phrase as a unifying description of several monotonic-improvement designs (Tomczak et al., 2019). A common misconception is therefore to treat MIPI as a single standardized method. The literature instead presents a family of guarantees with different monotonic objects, different inference mechanisms, and different admissible approximations.

Another recurrent misconception is that “monotonic improvement” always means exact per-iteration improvement of the true environment return for arbitrary deep implementations. In fact, the guarantee may apply to a surrogate objective, a regularized objective, a high-probability event, a lower-bounding critic sequence, a local-MDP value sequence, or performance at a fixed initial state. Critic bias in TRPO/PPO-like practice is a central example: the primary compatible-features paper argues that arbitrary advantage estimators break the exact surrogate-gradient guarantee, even when trust-region reasoning is otherwise invoked (Tomczak et al., 2019). Reliable critics, world-model SPI, and cautious interpolation are all responses to this same structural problem: inference quality must be aligned with the monotonicity proof, not treated as an independent implementation detail (R. et al., 8 Jun 2025, Delgrange et al., 14 Oct 2025, Zhu et al., 2021).

The significance of the MIPI viewpoint is therefore methodological. It isolates a common design principle across disparate RL subfields: policy improvement is safe only when the inference component—critic regression, value evaluation, gradient estimation, planning, representation learning, multi-agent marginalization, or posterior reconstruction over latent state—satisfies conditions compatible with the improvement theorem being invoked. This suggests that the most durable contribution of the literature is not a single update rule, but a general doctrine: monotonic policy improvement in approximate RL is inseparable from disciplined inference.