Optimality Inheritance in Finite MDPs

• The Optimality Inheritance Theorem is a principle in finite MDPs stating finite-time identification is possible if and only if there is a unique Bellman optimal policy.
• It establishes that across a hierarchy—from average-gain to Blackwell optimality—the certification threshold remains solely determined by Bellman optimality.
• The theorem combines consistent sample estimation with sensitivity to degeneracy, ensuring robust finite-time stopping criteria in practical learning scenarios.

Searching arXiv for the cited papers to ground the article. arxiv_search query: (Boone et al., 15 Oct 2025) In the context of finite Markov Decision Processes, the “Optimality Inheritance Theorem” denotes the central inheritance statement isolated in "Towards Blackwell Optimality: Bellman Optimality Is All You Can Get" (Boone et al., 15 Oct 2025): finite-time identification or stopping is possible if and only if the MDP has a unique Bellman optimal policy, and this threshold is independent of the optimality order being targeted. The result is formulated over a hierarchy running from average gain optimality through bias optimality of arbitrary finite order to Blackwell optimality. Its significance lies in separating asymptotic refinement of policy comparison from the information-theoretic possibility of exact finite-time certification.

1. Definition of the inheritance principle

The paper’s central statement is that finite-time identification/stopping is possible if and only if the MDP has a unique Bellman optimal policy (Boone et al., 15 Oct 2025). The same statement applies whether the objective is average gain optimality, bias optimality of any finite order, or Blackwell optimality. In this sense, identifiability “inherits” from the Bellman level rather than from any stronger refinement criterion.

This use of “inheritance” is structural rather than merely terminological. The paper does not claim that higher-order optimality notions collapse into Bellman optimality as decision criteria. Instead, it shows that the possibility of finite-time certification is already fully determined at the Bellman-optimal level. A plausible implication is that stronger asymptotic ranking criteria may alter which policy is preferred among Bellman-optimal contenders, but they do not alter the threshold at which exact stopping becomes possible.

The theorem is framed for a finite MDP under standard assumptions for average-reward and discounted-optimality analysis. The decisive distinction is between non-degenerate MDPs, which have a unique Bellman optimal policy, and degenerate MDPs, which have multiple Bellman optimal policies (Boone et al., 15 Oct 2025).

2. Optimality hierarchy underlying the theorem

The inheritance result is stated over a hierarchy of increasingly demanding optimality notions. For a policy $\pi$, the gain is the asymptotic average reward

$$g^\pi(s) \;=\; \liminf_{T\to\infty}\frac{1}{T}\,\mathbb{E}_s^\pi\!\left[\sum_{t=0}^{T-1} r_t\right].$$

A policy is gain-optimal if it maximizes this quantity from every state (Boone et al., 15 Oct 2025).

The Bellman or average-reward optimality equation provides the basic comparison principle: $$g^\star + h(s) \;=\; \max_{a\in A(s)} \Bigl(r(s,a) + \sum_{s'} P(s'|s,a)\,h(s')\Bigr),$$ where $g^\star$ is the optimal average reward and $h$ is a bias or differential value function (Boone et al., 15 Oct 2025). A policy that selects only maximizing actions in this equation is Bellman optimal for the average-reward criterion.

The paper then refines average-gain optimality through the bias hierarchy. It considers the asymptotic expansion

$V_T^\pi(s) = T\,g^\pi(s) + h_1^\pi(s) + o(1),$

and, for policies tied at the gain level, compares the next-order term $h_1^\pi$; if still tied, it compares the next term, and so on (Boone et al., 15 Oct 2025). This yields bias optimality of order $k$, producing the strict hierarchy

$$\text{average gain optimal} \;\prec\; \text{1st-order bias optimal} \;\prec\; \cdots \;\prec\; \text{higher-order bias optimal}.$$

At the top of the hierarchy lies Blackwell optimality. A Blackwell optimal policy is optimal for all sufficiently large discount factors $\gamma \in (0,1)$, equivalently in a neighborhood of $$g^\pi(s) \;=\; \liminf_{T\to\infty}\frac{1}{T}\,\mathbb{E}_s^\pi\!\left[\sum_{t=0}^{T-1} r_t\right].$$0 for the discounted problem (Boone et al., 15 Oct 2025). The discounted Bellman operator is

$$g^\pi(s) \;=\; \liminf_{T\to\infty}\frac{1}{T}\,\mathbb{E}_s^\pi\!\left[\sum_{t=0}^{T-1} r_t\right].$$1

The hierarchy relation matters because the theorem asserts that finite-time identifiability is unaffected by movement upward along this refinement chain.

3. Exact content of the theorem

The paper states four linked claims for each order in the hierarchy (Boone et al., 15 Oct 2025).

First, for each target optimality notion, there exists a learning algorithm with vanishing probability of error. Second, there exists a tractable stopping rule that, when coupled with the learning algorithm, stops in finite time whenever finite-time stopping is information-theoretically possible. Third, the MDPs for which such stopping is possible are exactly those with a unique Bellman optimal policy. Fourth, this characterization is unchanged across average gain optimality, any finite-order bias optimality, and Blackwell optimality.

The informal meaning is asymmetrical. If there is exactly one Bellman optimal policy, then the algorithm can eventually certify it and stop after a finite time, with vanishing error probability and a tractable stopping rule (Boone et al., 15 Oct 2025). If there are multiple Bellman optimal policies, no algorithm can in general guarantee finite-time stopping for exact identification.

A common misconception is that moving to a stronger optimality concept should require a stronger uniqueness condition, such as unique Blackwell optimality. The theorem rejects that view. The stopping threshold is not “unique optimal policy at the target level”; it is specifically unique Bellman optimal policy. This suggests that Bellman optimality is the decisive locus of finite-time certifiability even when the final target belongs to a stricter asymptotic refinement class.

4. Proof architecture: consistency, degeneracy, and certification

The proof structure is organized around two components: consistency and the relation between degeneracy and identifiability (Boone et al., 15 Oct 2025).

On the consistency side, the algorithm collects samples from the MDP, estimates transition probabilities and rewards, computes an empirical optimal policy for the desired optimality criterion, and refines the estimate as more data arrive. The resulting procedure is designed to be consistent, in the sense that the probability of outputting a non-optimal policy goes to zero. The justification given is standard concentration of empirical estimates: transition estimates converge to true transitions, reward estimates converge to true rewards, and the induced optimality computations converge (Boone et al., 15 Oct 2025).

On the impossibility side, the paper argues that if multiple Bellman optimal policies exist, then there are arbitrarily small perturbations of the model that change which policy is selected at the higher optimality levels (Boone et al., 15 Oct 2025). Because finite-time certification must be robust to the entire confidence region consistent with observed data, such perturbation sensitivity blocks exact finite-time stopping in the degenerate case.

The converse direction depends on a positive gap. If the Bellman optimal policy is unique, then the gap between the optimal action and the runner-up can eventually be certified after enough samples (Boone et al., 15 Oct 2025). This enables a stopping criterion based on confidence intervals. The stopping rule checks whether the confidence region around the empirical model is narrow enough to certify that the currently identified policy is uniquely optimal at the Bellman level. If the estimated optimal action is separated from competitors by a certified margin, the algorithm can stop and output that policy.

5. Bellman optimality as the maximal inheritance point

The paper’s conceptual conclusion is that Bellman optimality is the maximal point in the hierarchy from which finite-time identifiability inherits (Boone et al., 15 Oct 2025). Higher-order notions such as bias optimality or Blackwell optimality impose more stringent asymptotic requirements, but they do not generate a stronger finite-time certification threshold than Bellman optimality itself.

The underlying intuition is expressed in terms of separation. Finite-time stopping requires a positive separation between the target policy and alternatives; according to the paper, that separation is already determined at the Bellman-optimal level (Boone et al., 15 Oct 2025). Higher-order refinements matter for asymptotic ranking among tied Bellman-optimal policies, but not for the existence of a finite-time certificate. This is why the title’s claim, “Bellman Optimality Is All You Can Get,” has a precise technical role rather than a merely rhetorical one.

A plausible implication is methodological. When the goal is exact identification with stopping, effort spent on increasingly fine asymptotic selection criteria does not relax the fundamental information barrier created by Bellman degeneracy. Conversely, once Bellman uniqueness is available, the same data and stopping structure suffice for the full hierarchy considered in the paper (Boone et al., 15 Oct 2025).

6. Related uses of “optimality” and non-inheritance results in adjacent literatures

The expression “Optimality Inheritance Theorem” is not a standard theorem name across all areas that study optimality. The two additional papers supplied in the record illustrate this by presenting explicitly negative or non-invariant results rather than positive inheritance principles.

In single-agent heuristic search, "The Optimality of Satisficing Solutions" (Hansson et al., 2013) distinguishes p-optimality, where path length is the quality metric, from s-optimality, defined as optimality over arbitrary, subjectively determined criteria. Its central claim is that guaranteeing p-optimality is exponential, while guaranteeing s-optimality is impossible in the real world, because the search process itself consumes time, space, and other resources that belong to solution quality (Hansson et al., 2013). The paper’s strongest formulation is that verification itself destroys the property being verified: if one attempts to verify that a solution is s-optimal, one degrades its quality, guaranteeing that it will not be s-optimal. In that literature, the closest analogue to an inheritance statement is therefore a theorem of non-inheritance.

In reinforcement-learning theory for AIXI, "Bad Universal Priors and Notions of Optimality" (Leike et al., 2015) likewise rejects a robust inheritance principle. It argues that no invariance theorem is known for AIXI under changes of universal Turing machine or universal prior, and its results are “entirely negative” (Leike et al., 2015). The paper shows that Legg-Hutter intelligence and balanced Pareto optimality are subjective, that every policy is Pareto optimal in the class of all computable environments, and that for any computable policy there exists a bad universal prior under which AIXI behaves like that policy (Leike et al., 2015). Here again, the conclusion is not that optimality inherits across representations, but that optimality claims can become vacuous, trivial, or prior-dependent.

Taken together, these contrasts clarify the specificity of the inheritance theorem in finite MDPs. In (Boone et al., 15 Oct 2025), inheritance concerns the finite-time identifiability threshold across a hierarchy of optimality refinements within a fixed finite-MDP setting. In (Hansson et al., 2013) and (Leike et al., 2015), by contrast, the central lesson is that optimality may fail to remain stable once one accounts for search cost, subjective utility, or the choice of prior and reference machine.