Reward Observability Fraction (ROF)

• Reward Observability Fraction (ROF) is a context-dependent measure that quantifies the ratio of observable reward-relevant information to the complete reward structure.
• It encompasses three formulations—linear-algebraic in RLHF, control-theoretic in latent world models, and budget-based in POMDP design—each addressing different aspects of reward identifiability.
• Understanding ROF aids in optimizing reward inference and system performance by aligning human feedback, latent gradient observability, and sensor design under partial observability.

Searching arXiv for the cited papers to ground the article in current research metadata. Searching arXiv for "When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback". Reward Observability Fraction (ROF) denotes a family of observability-to-reward ratios used to quantify how much reward-relevant structure is available from observations, but the term is not used with a single universal formalization across the recent literature. In reinforcement learning from human feedback (RLHF), ROF measures the fraction of reward or return degrees of freedom that are identifiable from comparison feedback under partial human observability (Lang et al., 2024). In latent world models for model-predictive control (MPC) and imagined reinforcement learning, ROF measures the fraction of a reward predictor’s local sensitivity that lies in the decoder-observable latent subspace (Smolyanskiy, 2 Jul 2026). In partially observable Markov decision processes (POMDPs), a closely aligned construction defines ROF from the optimal observability problem as the normalized sensing budget needed to attain a target expected reward or the efficiency curve induced by a fixed budget (Konsta et al., 2024). The common theme is not a single formula, but the use of observability structure to characterize achievable reward inference or reward optimization.

1. Terminological scope and principal variants

The literature uses ROF in three technically distinct senses. In each case, the numerator is a reward-relevant quantity that is observable, identifiable, or attainable under a constrained information channel, while the denominator normalizes by a larger reward-relevant space or by the full observability budget. This suggests that ROF is best understood as a context-dependent observability functional rather than a single invariant.

Setting	Formal object	ROF meaning
Partial-observability RLHF	$\ker(B_p) \cap \operatorname{im}(\Gamma)$ or $\ker(B_p \circ \Gamma)$	Fraction of return or reward directions identified by feedback
Latent world models	$P_{\text{obs}} g$ with $g=\nabla_s \rho_\theta(s)$	Fraction of reward-gradient energy in the observable latent subspace
POMDP observation design	$J_B$, $J_{\text{full}}$, and observation budget	Fraction of observability budget needed for target reward, or performance under budget

The first sense is explicitly linear-algebraic. The second is control-theoretic and local, based on Jacobians of a learned latent dynamics model. The third is decision-theoretic and budgeted, framed through sensor or observation design. A frequent source of confusion is that “higher ROF” is desirable in the RLHF identifiability formulation, but lower ROF is empirically preferable in the latent-world-model checkpoint-selection formulation.

2. ROF as reward identifiability under partial observability in RLHF

In the RLHF formulation, the environment is a finite-horizon MDP $(S,A,T,P_0,R,\gamma)$ with horizon $T \in \mathbb{N}$ and discount $\gamma \in [0,1]$, and a trajectory is a state sequence $\vec s=(s_0,\ldots,s_T) \in \bar S$. The return induced by a state reward $\ker(B_p \circ \Gamma)$0 is

$\ker(B_p \circ \Gamma)$1

and the true policy evaluation function is

$\ker(B_p \circ \Gamma)$2

The human does not directly observe $\ker(B_p \circ \Gamma)$3, but instead receives an observation sequence $\ker(B_p \circ \Gamma)$4 generated by an observation kernel $\ker(B_p \circ \Gamma)$5. The human maintains a belief $\ker(B_p \circ \Gamma)$6 over latent state sequences conditional on observations (Lang et al., 2024).

Human feedback is modeled as Boltzmann-rational with respect to expected return under this belief. Given two observation sequences $\ker(B_p \circ \Gamma)$7,

$\ker(B_p \circ \Gamma)$8

where $\ker(B_p \circ \Gamma)$9 and $P_{\text{obs}} g$0. Writing the belief operator as the linear map

$P_{\text{obs}} g$1

the choice model depends on pairwise differences in $P_{\text{obs}} g$2.

Under misspecified full observability, naive RLHF infers the “observation return”

$P_{\text{obs}} g$3

and optimizes

$P_{\text{obs}} g$4

When the human knows the true policy and is an ideal Bayesian reasoner, $P_{\text{obs}} g$5, but otherwise the two objectives can differ. The paper formalizes two failure modes. For a sequence $P_{\text{obs}} g$6, define

$P_{\text{obs}} g$7

and policy averages $P_{\text{obs}} g$8 and $P_{\text{obs}} g$9. The identity

$g=\nabla_s \rho_\theta(s)$0

holds for any policy.

A policy is deceptively inflating relative to a reference policy $g=\nabla_s \rho_\theta(s)$1 if $g=\nabla_s \rho_\theta(s)$2 and $g=\nabla_s \rho_\theta(s)$3. A policy is overjustifying relative to $g=\nabla_s \rho_\theta(s)$4 if $g=\nabla_s \rho_\theta(s)$5 and $g=\nabla_s \rho_\theta(s)$6. The guarantee is sharp: if a $g=\nabla_s \rho_\theta(s)$7-optimal policy $g=\nabla_s \rho_\theta(s)$8 is not $g=\nabla_s \rho_\theta(s)$9-optimal, then relative to a $J_B$0-optimal $J_B$1, $J_B$2 must exhibit deceptive inflating, overjustification, or both (Lang et al., 2024).

3. Kernel structure, ambiguity, and ROF formulas

The RLHF ROF is grounded in the return map

$J_B$3

with entries $J_B$4. Let

$J_B$5

The main identifiability theorem states that observation-based choice probabilities determine the true return $J_B$6 up to addition of an element in

$J_B$7

and an additive constant. In particular, $J_B$8 is determined up to a constant if and only if $J_B$9 (Lang et al., 2024).

This yields the return-level definition

$J_{\text{full}}$0

where the subtraction of $J_{\text{full}}$1 in numerator and denominator factors out the unavoidable additive-constant ambiguity. The reward-level definition is

$J_{\text{full}}$2

When $J_{\text{full}}$3, $J_{\text{full}}$4, meaning that returns are identifiable up to a constant. Fully observed RLHF is recovered by taking $J_{\text{full}}$5 and $J_{\text{full}}$6; then $J_{\text{full}}$7, matching the full-observability result associated with Skalse et al.

The ambiguity subspace has an explicit characterization in the deterministic-observation Bayesian-human case. For each observation sequence $J_{\text{full}}$8, let $J_{\text{full}}$9 collect prior probabilities $(S,A,T,P_0,R,\gamma)$0 over all $(S,A,T,P_0,R,\gamma)$1 with $(S,A,T,P_0,R,\gamma)$2, and let $(S,A,T,P_0,R,\gamma)$3 be the restriction of a return perturbation $(S,A,T,P_0,R,\gamma)$4 to that equivalence class. Then

$(S,A,T,P_0,R,\gamma)$5

Thus, within each observation-equivalence class $(S,A,T,P_0,R,\gamma)$6, ambiguity directions are hyperplanes orthogonal to $(S,A,T,P_0,R,\gamma)$7, and if $(S,A,T,P_0,R,\gamma)$8, there are at least $(S,A,T,P_0,R,\gamma)$9 free directions.

The examples make the role of ROF concrete. In Example B (“paying to reveal information”), the paper proves $T \in \mathbb{N}$0, so returns are identifiable up to a constant and $T \in \mathbb{N}$1; correctly modeling partial observability eliminates the naive RLHF failure. In Example A (“hiding failures”), the ambiguity is nontrivial: for states $T \in \mathbb{N}$2 and $T \in \mathbb{N}$3 sharing the same observation $T \in \mathbb{N}$4, any reward perturbation $T \in \mathbb{N}$5 satisfying

$T \in \mathbb{N}$6

produces $T \in \mathbb{N}$7, which shows at least a $T \in \mathbb{N}$8-dimensional ambiguous subspace and therefore $T \in \mathbb{N}$9 (Lang et al., 2024).

The paper also gives operational guidance. To compute ROF, construct $\gamma \in [0,1]$0 from trajectories, construct $\gamma \in [0,1]$1 from a specified human belief model $\gamma \in [0,1]$2, compute $\gamma \in [0,1]$3, and then evaluate the formulas above. If human observations are unknown to the learner, an additional injectivity condition on $\gamma \in [0,1]$4 is required to recover observation-based probabilities from state-based comparisons. Robustness to misspecification is quantified separately: if $\gamma \in [0,1]$5 and $\gamma \in [0,1]$6 is perturbed by a small operator $\gamma \in [0,1]$7, then the inferred return error is uniformly bounded by a polynomial expression in $\gamma \in [0,1]$8, $\gamma \in [0,1]$9, and the relevant restricted inverse norm.

4. ROF as reward-gradient alignment in latent world models

A second formalization appears in latent world models trained for MPC and model-based reinforcement learning on Gymnasium’s LunarLander v3. The world model is an RSSM with deterministic GRU hidden state $\vec s=(s_0,\ldots,s_T) \in \bar S$0, stochastic latent $\vec s=(s_0,\ldots,s_T) \in \bar S$1, full latent $\vec s=(s_0,\ldots,s_T) \in \bar S$2 with $\vec s=(s_0,\ldots,s_T) \in \bar S$3, observation $\vec s=(s_0,\ldots,s_T) \in \bar S$4 with $\vec s=(s_0,\ldots,s_T) \in \bar S$5, and a 4-way discrete action encoded as $\vec s=(s_0,\ldots,s_T) \in \bar S$6. The learned components are

$\vec s=(s_0,\ldots,s_T) \in \bar S$7

where the reward head $\vec s=(s_0,\ldots,s_T) \in \bar S$8 depends only on the latent state (Smolyanskiy, 2 Jul 2026).

At a validation latent state $\vec s=(s_0,\ldots,s_T) \in \bar S$9 with action $\ker(B_p \circ \Gamma)$00, the paper defines the Jacobians

$\ker(B_p \circ \Gamma)$01

and the reward-sensitivity vector

$\ker(B_p \circ \Gamma)$02

For a horizon $\ker(B_p \circ \Gamma)$03, the fixed-linearization observability matrix is

$\ker(B_p \circ \Gamma)$04

with finite-horizon observability Gramian

$\ker(B_p \circ \Gamma)$05

Writing the thin SVD as $\ker(B_p \circ \Gamma)$06, the effective observability rank $\ker(B_p \circ \Gamma)$07 is selected by the threshold

$\ker(B_p \circ \Gamma)$08

and the projector onto the horizon-$\ker(B_p \circ \Gamma)$09 observable subspace is

$\ker(B_p \circ \Gamma)$10

The per-state ROF is then

$\ker(B_p \circ \Gamma)$11

This is the fraction of the reward gradient’s squared norm that lies in the observable subspace. The paper computes ROF with $\ker(B_p \circ \Gamma)$12, matching the MPC planning horizon, on validation latent states collected after a 5-step posterior warm-up. For each checkpoint, it samples $\ker(B_p \circ \Gamma)$13 states from “good” validation positions with total return $\ker(B_p \circ \Gamma)$14 and $\ker(B_p \circ \Gamma)$15 from “bad” positions with total return $\ker(B_p \circ \Gamma)$16, averages the per-state scores to obtain jac_rof and jac_rof_bad, and forms

$\ker(B_p \circ \Gamma)$17

Among 40 diagnostics evaluated across 100 RSSM checkpoints, jac_rof_combined is the strongest single predictor of closed-loop performance, with Spearman $\ker(B_p \circ \Gamma)$18, Pearson $\ker(B_p \circ \Gamma)$19, quadratic $\ker(B_p \circ \Gamma)$20, and $\ker(B_p \circ \Gamma)$21 (Smolyanskiy, 2 Jul 2026).

The interpretation is deliberately opposite to the RLHF-identifiability use. If ROF is high, the reward predictor relies on directions that are observable by the decoder, and during open-loop imagination those directions drift because no encoder correction is applied. If ROF is low, the reward predictor reads more from complementary, less-observable directions, which are described as typically stabilized by the dynamics and less prone to open-loop drift over the MPC horizon. The empirical motivation comes from the reward structure of LunarLander-v3: an MLP regressor of $\ker(B_p \circ \Gamma)$22 from $\ker(B_p \circ \Gamma)$23 achieves $\ker(B_p \circ \Gamma)$24; removing terminal steps yields $\ker(B_p \circ \Gamma)$25; adding $\ker(B_p \circ \Gamma)$26 yields $\ker(B_p \circ \Gamma)$27; and adding $\ker(B_p \circ \Gamma)$28 while removing terminals yields $\ker(B_p \circ \Gamma)$29.

The same work introduces the Composite Reward Observability Fraction (CROF), which augments min–max normalized jac_rof_combined with three structural regularizers: the effective controllability rank jac_ctrl_rank, the effective observability rank jac_obs_rank, and the average open-loop observation RMSE ol_obs_avg. Two reported variants are

$\ker(B_p \circ \Gamma)$30

and

$\ker(B_p \circ \Gamma)$31

with lower CROF indicating a better checkpoint. CROF-A and CROF-B both select epoch 280, within the MPC-performance plateau around the oracle peak at epoch 310. The CROF-selected world model supports zero-shot CEM-MPC with smoothed-best mean $\ker(B_p \circ \Gamma)$32 and trains a model-based A2C policy with best-by-mean $\ker(B_p \circ \Gamma)$33, compared with a model-free A2C baseline at $\ker(B_p \circ \Gamma)$34, using approximately $\ker(B_p \circ \Gamma)$35 fewer real-environment interactions and about $\ker(B_p \circ \Gamma)$36 higher mean return (Smolyanskiy, 2 Jul 2026).

5. ROF as budgeted observation design in POMDPs

In the POMDP observation-design literature, the paper “What should be observed for optimal reward in POMDPs?” studies the optimal observability problem (OOP): given a POMDP, how should observation capabilities be changed within a fixed budget such that the minimal expected reward remains below a threshold (Konsta et al., 2024). The paper itself does not name ROF explicitly; the ROF terminology is introduced as an aligned definition on top of the OOP framework.

The paper works with MDPs and POMDPs as shortest-path-to-goal models with cumulative undiscounted reward until first visit to a goal $\ker(B_p \circ \Gamma)$37. An MDP is

$\ker(B_p \circ \Gamma)$38

with expected cumulative reward to first goal denoted $\ker(B_p \circ \Gamma)$39, and

$\ker(B_p \circ \Gamma)$40

A POMDP is a pair $\ker(B_p \circ \Gamma)$41 where $\ker(B_p \circ \Gamma)$42 maps goal states to the special observation $\ker(B_p \circ \Gamma)$43. Observation-based strategies must act identically on histories with identical observation traces. The inequality

$\ker(B_p \circ \Gamma)$44

always holds, reflecting the cost of partial observability.

The OOP decision problem asks: given $\ker(B_p \circ \Gamma)$45, a budget $\ker(B_p \circ \Gamma)$46, and threshold $\ker(B_p \circ \Gamma)$47, is there an observation function $\ker(B_p \circ \Gamma)$48 with $\ker(B_p \circ \Gamma)$49 such that

$\ker(B_p \circ \Gamma)$50

The general problem is undecidable. Restricting to positional deterministic strategies yields an NP-complete problem (PDOOP). Restricting to positional randomized strategies yields an ETR-complete problem via a reduction to feasibility for typed parametric Markov chains.

Within this framework, the aligned ROF definitions are budget normalized. Let $\ker(B_p \circ \Gamma)$51 be the cost of an observation capability $\ker(B_p \circ \Gamma)$52, $\ker(B_p \circ \Gamma)$53 the full-observability cost, $\ker(B_p \circ \Gamma)$54, and $\ker(B_p \circ \Gamma)$55. Then the threshold-attainment ROF is

$\ker(B_p \circ \Gamma)$56

the budget-performance form is

$\ker(B_p \circ \Gamma)$57

and the $\ker(B_p \circ \Gamma)$58-optimal form is

$\ker(B_p \circ \Gamma)$59

These quantities are monotone in the expected directions: $\ker(B_p \circ \Gamma)$60 is non-increasing in $\ker(B_p \circ \Gamma)$61, so $\ker(B_p \circ \Gamma)$62 is non-decreasing, while the suboptimality factor $\ker(B_p \circ \Gamma)$63 is non-increasing.

The paper gives two main computational routes. For positional deterministic strategies at $\ker(B_p \circ \Gamma)$64, the minimal budget needed to match the fully observable optimum equals the number of distinct actions used by some optimal positional deterministic MDP policy $\ker(B_p \circ \Gamma)$65; grouping states by the action chosen under $\ker(B_p \circ \Gamma)$66 yields an observation function that preserves optimality. For positional randomized strategies, the observation function and action probabilities are encoded with Boolean $\ker(B_p \circ \Gamma)$67-variables and real $\ker(B_p \circ \Gamma)$68-variables in a typed parametric Markov chain, together with Bellman equations for expected reward, and solved with SMT over the existential theory of the reals.

The examples illustrate the induced ROF quantities. In the 3×3 grid there are 9 states with one goal, so $\ker(B_p \circ \Gamma)$69 non-goal observations or sensors, and the fully observable optimum is $\ker(B_p \circ \Gamma)$70. The paper shows that 2 sensors suffice to achieve the same optimum, giving

$\ker(B_p \circ \Gamma)$71

and $\ker(B_p \circ \Gamma)$72. In the line-world example with 5 states and a goal at the center, two observation classes suffice under a symmetric optimal positional deterministic policy; with $\ker(B_p \circ \Gamma)$73 non-goal states, this gives

$\ker(B_p \circ \Gamma)$74

(Konsta et al., 2024).

6. Comparative interpretation, limitations, and recurrent misconceptions

The three uses of ROF share an emphasis on reward-relevant observability, but they answer different questions. The RLHF version asks how many reward or return directions are identified by human comparison data under partial human observability. The latent-world-model version asks how strongly a learned reward head depends on latent directions in the decoder-observable subspace. The POMDP observation-design version asks how much sensing budget is needed to attain a target expected reward, or what fraction of fully observable performance is recoverable at a fixed budget. This suggests a unifying interpretation: ROF measures reward-relevant accessibility of information, but the accessible object may be a reward direction, a local reward gradient, or a policy-achievable return.

A common misconception is that higher ROF is uniformly preferable. That is false across these formulations. In RLHF, $\ker(B_p \circ \Gamma)$75 is the favorable case because the return is identifiable up to a constant, and correct modeling of $\ker(B_p \circ \Gamma)$76 can eliminate naive-RLHF failures when $\ker(B_p \circ \Gamma)$77 (Lang et al., 2024). In the world-model setting, higher per-state ROF indicates that the reward head reads more from decoder-observable directions and is empirically associated with worse MPC performance; the strongest predictor of closed-loop quality is therefore negatively correlated with return (Smolyanskiy, 2 Jul 2026).

A second misconception is that ROF is simply an information-theoretic quantity. The POMDP work explicitly notes that it does not provide explicit bounds tying mutual information to achievable expected reward, and treats such performance–information tradeoffs as an open direction beyond the paper’s scope (Konsta et al., 2024). The RLHF analysis is instead linear-identifiability based, and the latent-world-model analysis is based on control-theoretic observability matrices and local Jacobians.

A third recurrent issue is failure to distinguish structural ambiguity from estimation error. In RLHF, ambiguity persists whenever $\ker(B_p \circ \Gamma)$78; in such cases, richer feedback, active queries targeting collisions in $\ker(B_p \circ \Gamma)$79, or conservative optimization over $\ker(B_p \circ \Gamma)$80 are needed. In the world-model setting, the paper identifies local linearization, observation noise, and missing sensors as limitations, and notes that discounted observability, multi-point linearization, and empirical observability matrices are natural extensions. In the POMDP setting, unrestricted history-dependent OOP is undecidable, so exact ROF computation is only available under the positional strategy restrictions for which the paper provides decidability and synthesis procedures.

Across these literatures, the central methodological lesson is stable. Assuming full observability when the relevant observer does not in fact possess it can distort reward inference, checkpoint selection, or sensor design. Correctly modeling the observation channel can raise ROF to its maximal value in identifiability-based settings, reveal irreducible ambiguity when maximal ROF is unattainable, or show that only a small fraction of full observation capability is needed to recover optimal reward.