Composite Reward Observability Fraction (CROF)

• CROF is a metric that quantifies the ratio of observable to generated rewards, linking reward quality with an agent’s observation structure across RL and POMDP settings.
• It integrates methodologies from delayed reward attribution, optimal sensor design, and latent world-model validation to predict performance and inform checkpoint selection.
• By incorporating structural regularizers and addressing observation bottlenecks, CROF improves credit assignment and reduces delay penalties to enhance robust model performance.

Searching arXiv for the cited papers to ground the article. Composite Reward Observability Fraction (CROF) denotes a family of observability–reward metrics that relate what an agent can observe to what reward it can recover, predict, or attribute. In the most explicit current usage, CROF is a validation-time checkpoint-selection score for latent world models, constructed from the Reward Observability Fraction (ROF), controllability rank, observability rank, and open-loop observation error (Smolyanskiy, 2 Jul 2026). In adjacent lines of work, compatible CROF formulations have been proposed for delayed, composite, and partially anonymous reward feedback, where the central issue is how much generated reward is actually visible and credit-assignable to the learner (Mondal et al., 2023), and for observation synthesis in POMDPs, where the issue is how restricted sensing affects achievable reward under a budget (Konsta et al., 2024). This suggests that CROF is presently best understood as a context-dependent observability functional rather than a single universally fixed scalar.

1. Conceptual scope and variants

Across the three settings, CROF always links reward quality to an observation structure, but the object being normalized differs. In delayed composite reward RL, CROF measures realized or attributable reward mass relative to generated reward mass. In observability-constrained POMDPs, it measures how closely the optimal expected reward under an observation design matches the full-observability optimum, optionally combined with an observability-cost term. In latent world models, it is an offline structural score for selecting checkpoints whose reward predictions remain useful under open-loop imagination (Mondal et al., 2023, Konsta et al., 2024, Smolyanskiy, 2 Jul 2026).

Setting	CROF object	Orientation
Delayed, composite, partially anonymous reward	Realization-based or attribution-based fraction of reward mass	Higher is better
POMDP observability design	Reward-normalized or cost-reward composite score under budget	Depends on normalization
Latent world-model validation	Sum of normalized ROF and structural penalties	Lower is better

The resulting terminological overlap is substantive rather than accidental: each formulation addresses the extent to which reward-relevant information survives an observation bottleneck. At the same time, the metrics are not interchangeable. The delayed-reward formulation is tied to mass conservation and credit assignment, the POMDP formulation to optimal expected reward under observation synthesis, and the latent-model formulation to local linearization, observable subspaces, and checkpoint selection. This suggests that any use of the term should specify the dynamical model, the observation channel, the normalization convention, and whether the metric is intended for attribution, performance comparison, or model selection.

2. Delayed, composite, and partially anonymous reward

In the infinite-horizon average-reward MDP studied in "Reinforcement Learning with Delayed, Composite, and Partially Anonymous Reward" (Mondal et al., 2023), the environment is $M \equiv \{\mathcal{S}, \mathcal{A}, r, p\}$ with state space $\mathcal{S}$, action space $\mathcal{A}$, reward function $r:\mathcal{S}\times\mathcal{A}\to[0,1]$, and transition kernel $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$. Taking action $a$ in state $s$ at time $t$ generates a delayed composite reward sequence

$$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$

where $r_{t,\tau}(s,a)\ge 0$ is realized at time $\mathcal{S}$0. The learner does not observe action-tagged components. Instead, at time $\mathcal{S}$1 it observes only the state-indexed aggregate

$\mathcal{S}$2

This is the paper’s partial anonymity condition: realized reward is identifiable by origin state $\mathcal{S}$3, but not by the full $\mathcal{S}$4 source.

Within this model, the detailed synthesis proposes two CROF families. The realization-based form measures how much generated reward mass has become observable. Let

$\mathcal{S}$5

Then the cumulative variant is

$\mathcal{S}$6

Under no delay, $\mathcal{S}$7, every component is realized immediately, and $\mathcal{S}$8 for all $\mathcal{S}$9.

The attribution-based form measures whether realized reward can be uniquely assigned to its generating action. Under DUCRL2’s epoch structure, attribution is unique for state $\mathcal{A}$0 when all components realized at time $\mathcal{A}$1 and belonging to $\mathcal{A}$2 originate in a single epoch that used a single action for that state, with negligible cross-epoch contamination. If $\mathcal{A}$3 denotes realized components or realized mass, and $\mathcal{A}$4 the uniquely attributable subset, then

$\mathcal{A}$5

Using the paper’s epoch decomposition into $\mathcal{A}$6, a conservative lower bound is

$\mathcal{A}$7

The quantity $\mathcal{A}$8 is the key delay parameter. Assumption 2 upper-bounds reward spillover beyond episode or state boundaries, and the paper proves the contamination relation $\mathcal{A}$9. DUCRL2 uses this to derive the reward-estimation bias bound

$r:\mathcal{S}\times\mathcal{A}\to[0,1]$0

That term appears directly in the reward confidence radius and yields the additive delay penalty in the regret bound,

$r:\mathcal{S}\times\mathcal{A}\to[0,1]$1

The structural interpretation is immediate: larger $r:\mathcal{S}\times\mathcal{A}\to[0,1]$2 implies more contamination across epoch boundaries, which lowers attribution quality and widens confidence sets. DUCRL2’s rule of one action per state within an epoch can therefore be read as an explicit attempt to raise $r:\mathcal{S}\times\mathcal{A}\to[0,1]$3 by reducing action mixing inside the effective delay window.

3. Observability-constrained POMDPs

In "What should be observed for optimal reward in POMDPs?" (Konsta et al., 2024), the relevant question is not delayed reward attribution but observation design. A POMDP is given as

$r:\mathcal{S}\times\mathcal{A}\to[0,1]$4

with standard belief update

$r:\mathcal{S}\times\mathcal{A}\to[0,1]$5

and expected return

$r:\mathcal{S}\times\mathcal{A}\to[0,1]$6

The paper studies the Optimal Observability Problem (OOP): given budget $r:\mathcal{S}\times\mathcal{A}\to[0,1]$7 and threshold $r:\mathcal{S}\times\mathcal{A}\to[0,1]$8, determine whether there exists an observation parameterization $r:\mathcal{S}\times\mathcal{A}\to[0,1]$9 with $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$0 such that

$$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$1

Within this framework, the detailed synthesis defines a reward-normalized CROF by comparing optimal expected reward under observation design $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$2 to the full-observability benchmark: $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$3 It also gives two cost terms,

$$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$4

and

$$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$5

depending on whether one wants “fraction saved” or “fraction used.” The composite score is then

$$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$6

This construction is tied to the complexity landscape of OOP. The general optimal observability problem is undecidable. When restricting to positional deterministic strategies, the corresponding decision problem PDOOP is NP-complete. For positional randomized strategies, POP is ETR-complete and therefore decidable in PSPACE. Two algorithmic routes are given. The underlying-MDP-based method computes an optimal positional deterministic strategy $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$7, partitions states by the action used under $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$8, and assigns one observation per action class, thereby preserving the MDP-optimal minimal expected reward when the budget is large enough. The SMT-based method encodes observation assignment and positional policy choice with variables $$p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$$9 and $a$0, Bellman constraints, and budget constraints, and then solves the resulting formula directly.

The line and grid examples show how this POMDP-style CROF encodes a reward–sensor trade-off. In the five-state line example, full observability yields $a$1. A two-observation partition separating the left and right sides also achieves $a$2, so $a$3. With insufficient sensing, the synthesis gives $a$4, hence $a$5. This suggests that, in minimization settings, the directionality of “better” depends on which CROF component is being discussed: raw reward-normalized ratios and cost-aware composites need not order observation designs in the same way.

4. Exact CROF in latent world-model validation

The exact modern use of CROF appears in "Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander" (Smolyanskiy, 2 Jul 2026). The setting is an RSSM world model with latent state

$a$6

where $a$7 and $a$8. Actions are one-hot vectors in $a$9. The transition is $s$0, the observation decoder is $s$1, and the reward head is a separate network $s$2. The paper’s central claim is that conventional validation losses and multi-step RMSE continue to improve long after closed-loop performance has collapsed, so checkpoint selection requires structural diagnostics.

At a validation latent state–action pair $s$3, the paper defines Jacobians

$s$4

and reward gradient $s$5. For horizon $s$6, the fixed-linearization controllability and observability matrices are

$s$7

After SVD, effective ranks $s$8 and $s$9 are defined by the relative singular-value threshold $t$0.

The Reward Observability Fraction is

$t$1

where $t$2 spans the observable subspace. The paper also defines

$t$3

ROF is aggregated over two curated validation subsets: good states with return at least $t$4, and bad states with return at most $t$5. The combined score is

$t$6

CROF then augments ROF with three structural regularizers. Let $t$7, $t$8, $t$9, and $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$0. After min–max normalization across checkpoints,

$$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$1

the paper defines

$$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$2

and

$$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$3

Lower CROF is better. The role of the regularizers is explicitly to guard against early checkpoints whose ROF is deceptively low even though the dynamics are not yet structurally usable.

5. Computation, empirical behavior, and deployment

The latent-world-model formulation is operationalized as a full offline validation pipeline (Smolyanskiy, 2 Jul 2026). Training uses Gymnasium LunarLander-v3 with 872 human-piloted episodes and 180,916 steps, split $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$4 train/validation, sequences of length $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$5 at stride $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$6, and 500 epochs with checkpoints every 5 epochs. For each checkpoint, the pipeline computes 40 metrics: 5 validation losses, 9 open-loop errors, 21 Jacobian-based metrics, 3 empirical sensitivities, and 2 composite scores. The oracle for checkpoint quality is CEM-MPC return averaged over 20 episodes per checkpoint, with a 7-point moving average used as the smoothed target.

Within that benchmark, $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$7 is the strongest single predictor of smoothed MPC mean return, with Spearman $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$8, Pearson $$\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},$$9, and quadratic $r_{t,\tau}(s,a)\ge 0$0 over $r_{t,\tau}(s,a)\ge 0$1 checkpoints. The corresponding good-state and bad-state fixed ROF correlations are $r_{t,\tau}(s,a)\ge 0$2 and $r_{t,\tau}(s,a)\ge 0$3. CROF-B yields $r_{t,\tau}(s,a)\ge 0$4, $r_{t,\tau}(s,a)\ge 0$5, $r_{t,\tau}(s,a)\ge 0$6, while CROF-A yields $r_{t,\tau}(s,a)\ge 0$7, $r_{t,\tau}(s,a)\ge 0$8, $r_{t,\tau}(s,a)\ge 0$9. Standard training metrics are weak or near zero; for example, $\mathcal{S}$00 has $\mathcal{S}$01, and one-step RMSEs are near zero. Open-loop reward RMSE at horizon end, $\mathcal{S}$02, is only mildly informative, with $\mathcal{S}$03.

The checkpoint-selection behavior is correspondingly different. Smoothed MPC performance peaks near epoch 310 at $\mathcal{S}$04, with a plateau over roughly epochs 260–320. The smoothed minimum of ROF alone occurs at epoch 265 and attains MPC $\mathcal{S}$05, but the raw ROF minimum occurs too early, at epoch 250 with $\mathcal{S}$06. Both raw-min CROF-A and raw-min CROF-B select epoch 280, which yields MPC $\mathcal{S}$07 and lies inside the high-performance plateau. Standard criteria such as $\mathcal{S}$08, $\mathcal{S}$09, $\mathcal{S}$10, $\mathcal{S}$11, or maximal $\mathcal{S}$12 pick late checkpoints at or beyond epoch 460 with poor MPC returns ranging from $\mathcal{S}$13 to $\mathcal{S}$14.

The same CROF-selected world model supports both zero-shot planning and imagined-policy training. Using the CROF raw-pick world model at epoch 280, model-based A2C trained entirely in imagination produces a best-by-mean policy with mean $\mathcal{S}$15, worst $\mathcal{S}$16, 79 of 100 perfect landings, and 2 of 100 catastrophic outcomes; the safest-by-worst variant yields mean $\mathcal{S}$17, worst $\mathcal{S}$18, 71 perfect landings, and 0 catastrophic outcomes. A fairly evaluated model-free A2C baseline trained with 50,000 episodes, or about 11.82 million steps, peaks at $\mathcal{S}$19. By contrast, the model-based A2C policy uses only the offline dataset of 180,916 steps, approximately $\mathcal{S}$20 fewer real-environment interactions, and beats the model-free baseline by about 24.5 return points.

The delayed-reward and POMDP formulations also have explicit operational implications. In the delayed composite reward setting, increasing $\mathcal{S}$21 amounts to designing epochs whose action assignment is constant within each state over the effective delay window, exactly the design enforced by DUCRL2 (Mondal et al., 2023). In the POMDP setting, increasing CROF corresponds to choosing observation labels or sensors that separate states requiring different optimal actions; the underlying-MDP algorithm does this by grouping states according to the action used by an optimal positional deterministic policy (Konsta et al., 2024).

6. Interpretation, edge cases, and limitations

Several edge cases clarify what CROF is and is not. In the delayed composite reward model, no delay implies $\mathcal{S}$22, $\mathcal{S}$23, and $\mathcal{S}$24 under partial anonymity because $\mathcal{S}$25 then contains only the component generated at time $\mathcal{S}$26 from $\mathcal{S}$27 (Mondal et al., 2023). If reward observations were fully non-anonymous, with each component tagged by $\mathcal{S}$28, attribution would always be unique. Heavy-tailed delays that violate Assumption 2 can make $\mathcal{S}$29, in which case cumulative realization may remain far below 1 at finite time and the regret guarantees no longer hold.

In the POMDP observability-design setting, the main limitation is computational. General OOP is undecidable, and tractable formulations require restricting the strategy class to positional deterministic or positional randomized strategies (Konsta et al., 2024). The reward-normalized CROF in that setting must also be interpreted carefully because the objective is minimization of expected reward or expected steps to goal. The same synthesis gives both $\mathcal{S}$30 for observation designs matching full observability and $\mathcal{S}$31 for infeasible low-budget designs, while the cost-aware composite may rank a partial-observability design above full observability when sensing cost is sufficiently weighted. This suggests that, in POMDP usage, one must distinguish performance preservation from cost-adjusted desirability.

In the latent-world-model setting, ROF and CROF are explicitly motivated by non-Markovian reward in LunarLander (Smolyanskiy, 2 Jul 2026). The reward has the form

$\mathcal{S}$32

with terminal bonus or penalty $\mathcal{S}$33 depending on flags not present in $\mathcal{S}$34. Predictability experiments show $\mathcal{S}$35 from $\mathcal{S}$36 alone, $\mathcal{S}$37 if terminals are filtered, $\mathcal{S}$38 if $\mathcal{S}$39 is added, and $\mathcal{S}$40 with both $\mathcal{S}$41 and terminal filtering. The paper’s interpretation is that a useful reward head must exploit latent information not recoverable from $\mathcal{S}$42 alone, and that low ROF is structurally aligned with open-loop robustness on that task.

The same paper also gives explicit negative cases. In fully Markovian reward settings such as Gymnasium Reacher, an MLP predicts reward from $\mathcal{S}$43 with $\mathcal{S}$44, and ROF has weak, oppositely signed rank correlation with MPC, $\mathcal{S}$45. Standard prediction metrics suffice there, and there is no late-training collapse for CROF to detect. Other listed failure modes include severe distribution shift between validation rollouts and planner trajectories, non-identifiable dynamics or degenerate latent representations, over-reliance on local linearization, and numerical instability when the reward gradient norm $\mathcal{S}$46 is near zero.

Taken together, these results support a precise but plural interpretation of CROF. In delayed-reward RL, it is a measure of realization or attribution under delayed, composite, partially anonymous feedback. In POMDP observation design, it is a reward-preservation or reward-cost trade-off induced by sensor choices. In latent world models, it is an exact structural diagnostic for offline checkpoint selection. The common core is the same in all three settings: reward is not merely generated, but must be made visible in a form that the agent can exploit.