---
title: Composite Reward Observability Fraction (CROF)
url: https://www.emergentmind.com/topics/composite-reward-observability-fraction-crof
type: topic
---

# Composite Reward Observability Fraction (CROF)

Searching arXiv for the cited papers to ground the article.
Composite Reward Observability Fraction (CROF) denotes a family of observability–reward metrics that relate what an agent can observe to what reward it can recover, predict, or attribute. In the most explicit current usage, CROF is a validation-time checkpoint-selection score for latent world models, constructed from the Reward Observability Fraction (ROF), controllability rank, observability rank, and open-loop observation error [2607.01736]. In adjacent lines of work, compatible CROF formulations have been proposed for delayed, composite, and partially anonymous reward feedback, where the central issue is how much generated reward is actually visible and credit-assignable to the learner [2305.02527], and for observation synthesis in POMDPs, where the issue is how restricted sensing affects achievable reward under a budget [2405.10768]. This suggests that CROF is presently best understood as a context-dependent observability functional rather than a single universally fixed scalar.

## 1. Conceptual scope and variants

Across the three settings, CROF always links reward quality to an observation structure, but the object being normalized differs. In delayed composite reward RL, CROF measures realized or attributable reward mass relative to generated reward mass. In observability-constrained POMDPs, it measures how closely the optimal expected reward under an observation design matches the full-observability optimum, optionally combined with an observability-cost term. In latent world models, it is an offline structural score for selecting checkpoints whose reward predictions remain useful under open-loop imagination [2305.02527][2405.10768][2607.01736].

| Setting | CROF object | Orientation |
|---|---|---|
| Delayed, composite, partially anonymous reward | Realization-based or attribution-based fraction of reward mass | Higher is better |
| POMDP observability design | Reward-normalized or cost-reward composite score under budget | Depends on normalization |
| Latent world-model validation | Sum of normalized ROF and structural penalties | Lower is better |

The resulting terminological overlap is substantive rather than accidental: each formulation addresses the extent to which reward-relevant information survives an observation bottleneck. At the same time, the metrics are not interchangeable. The delayed-reward formulation is tied to mass conservation and credit assignment, the POMDP formulation to optimal expected reward under observation synthesis, and the latent-model formulation to local linearization, observable subspaces, and checkpoint selection. This suggests that any use of the term should specify the dynamical model, the observation channel, the normalization convention, and whether the metric is intended for attribution, performance comparison, or model selection.

## 2. Delayed, composite, and partially anonymous reward

In the infinite-horizon average-reward MDP studied in "Reinforcement Learning with Delayed, Composite, and Partially Anonymous Reward" [2305.02527], the environment is \(M \equiv \{\mathcal{S}, \mathcal{A}, r, p\}\) with state space \(\mathcal{S}\), action space \(\mathcal{A}\), reward function \(r:\mathcal{S}\times\mathcal{A}\to[0,1]\), and transition kernel \(p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})\). Taking action \(a\) in state \(s\) at time \(t\) generates a delayed composite reward sequence
\[
\mathbf{r}_t(s,a)\coloneqq \{r_{t,\tau}(s,a)\}_{\tau=0}^{\infty},
\]
where \(r_{t,\tau}(s,a)\ge 0\) is realized at time \(t+\tau\). The learner does not observe action-tagged components. Instead, at time \(t\) it observes only the state-indexed aggregate
\[
\mathbf{x}_t(s)=\sum_{0<\tau\le t} r_{\tau,t-\tau}(s,a_\tau)\mathbf{1}\{s_\tau=s\}.
\]
This is the paper’s partial anonymity condition: realized reward is identifiable by origin state \(s\), but not by the full \((s,a,\tau)\) source.

Within this model, the detailed synthesis proposes two CROF families. The realization-based form measures how much generated reward mass has become observable. Let
\[
R_{\mathrm{obs}}(s,t)\coloneqq \mathbf{x}_t(s),
\qquad
G_{\mathrm{gen}}(t)\coloneqq \sum_{u\le t}\|\mathbf{r}_u(s_u,a_u)\|_1.
\]
Then the cumulative variant is
\[
\mathrm{CROF}_{\mathrm{real}}^{\mathrm{cum}}(t)\coloneqq
\frac{\mathbb{E}\!\left[\sum_{\tau=1}^{t}\sum_{s\in\mathcal{S}}R_{\mathrm{obs}}(s,\tau)\right]}
{\mathbb{E}\!\left[\sum_{u\le t}\|\mathbf{r}_u(s_u,a_u)\|_1\right]}.
\]
Under no delay, \(d=0\), every component is realized immediately, and \(\mathrm{CROF}_{\mathrm{real}}^{\mathrm{cum}}(t)=1\) for all \(t\).

The attribution-based form measures whether realized reward can be uniquely assigned to its generating action. Under DUCRL2’s epoch structure, attribution is unique for state \(s\) when all components realized at time \(t\) and belonging to \(s\) originate in a single epoch that used a single action for that state, with negligible cross-epoch contamination. If \(N_{\mathrm{real}}(t)\) denotes realized components or realized mass, and \(N_{\mathrm{attr}}(t)\) the uniquely attributable subset, then
\[
\mathrm{CROF}_{\mathrm{attr}}(t)\coloneqq
\frac{\mathbb{E}[N_{\mathrm{attr}}(t)]}{\mathbb{E}[N_{\mathrm{real}}(t)]}.
\]
Using the paper’s epoch decomposition into \(R_0,R_1,R_2,R_3\), a conservative lower bound is
\[
\mathrm{CROF}_{\mathrm{attr}}(s,t)\ge 1-\frac{d}{\mathbb{E}[R_0]}.
\]

The quantity \(d\) is the key delay parameter. Assumption 2 upper-bounds reward spillover beyond episode or state boundaries, and the paper proves the contamination relation \(-d\le R_0-R_2\le d\). DUCRL2 uses this to derive the reward-estimation bias bound
\[
\left|
\hat r_k(s,a)-
\frac{1}{N_k(s,a)}
\sum_{\tau<t_k}\|\mathbf{r}_\tau(s,a)\|_1\mathbf{1}\{s_\tau=s,a_\tau=a\}
\right|
\le
\frac{dE_k(s,a)}{\max\{1,N_k(s,a)\}}.
\]
That term appears directly in the reward confidence radius and yields the additive delay penalty in the regret bound,
\[
\mathrm{Reg}=\tilde{\mathcal O}\!\left(DS\sqrt{AT}+d(SA)^3\right).
\]
The structural interpretation is immediate: larger \(d\) implies more contamination across epoch boundaries, which lowers attribution quality and widens confidence sets. DUCRL2’s rule of one action per state within an epoch can therefore be read as an explicit attempt to raise \(\mathrm{CROF}_{\mathrm{attr}}\) by reducing action mixing inside the effective delay window.

## 3. Observability-constrained POMDPs

In "What should be observed for optimal reward in POMDPs?" [2405.10768], the relevant question is not delayed reward attribution but observation design. A POMDP is given as
\[
M=(S,A,T,\Omega,O,R,\gamma),
\]
with standard belief update
\[
b'(s')=\eta\, O(a,s',o)\sum_{s\in S}T(s,a,s')\,b(s),
\]
and expected return
\[
V^\pi(b_0)=\mathbb{E}\Big[\sum_{t=0}^{\infty}\gamma^tR(s_t,a_t)\Big].
\]
The paper studies the Optimal Observability Problem (OOP): given budget \(B\) and threshold \(\tau\), determine whether there exists an observation parameterization \(\theta\) with \(C(\theta)\le B\) such that
\[
\min_{\sigma\in\mathcal O(M_\theta)} \mathrm{ExpRew}(M_\theta,\sigma)\le \tau.
\]

Within this framework, the detailed synthesis defines a reward-normalized CROF by comparing optimal expected reward under observation design \(\theta\) to the full-observability benchmark:
\[
\mathrm{CROF}_R(\theta)=\frac{V^*(b_0;\theta)}{V^*(b_0;\theta_{\mathrm{full}})}.
\]
It also gives two cost terms,
\[
\mathrm{CROF}_S(\theta)=1-\frac{C(\theta)}{C(\theta_{\mathrm{full}})}
\]
and
\[
\mathrm{CROF}_S(\theta)=\frac{|\text{sensors used under }\theta|}{|\text{sensors in }\theta_{\mathrm{full}}|},
\]
depending on whether one wants “fraction saved” or “fraction used.” The composite score is then
\[
\mathrm{CROF}(\theta;\alpha)=
\alpha\cdot \mathrm{CROF}_R(\theta)
+
(1-\alpha)\cdot
\left(1-\frac{C(\theta)}{C(\theta_{\mathrm{full}})}\right),
\qquad
\alpha\in[0,1].
\]

This construction is tied to the complexity landscape of OOP. The general optimal observability problem is undecidable. When restricting to positional deterministic strategies, the corresponding decision problem PDOOP is NP-complete. For positional randomized strategies, POP is ETR-complete and therefore decidable in PSPACE. Two algorithmic routes are given. The underlying-MDP-based method computes an optimal positional deterministic strategy \(\sigma^*\), partitions states by the action used under \(\sigma^*\), and assigns one observation per action class, thereby preserving the MDP-optimal minimal expected reward when the budget is large enough. The SMT-based method encodes observation assignment and positional policy choice with variables \(y_{s,o}\) and \(x_{o,a}\), Bellman constraints, and budget constraints, and then solves the resulting formula directly.

The line and grid examples show how this POMDP-style CROF encodes a reward–sensor trade-off. In the five-state line example, full observability yields \(V^*(b_0;\theta_{\mathrm{full}})=3/2\). A two-observation partition separating the left and right sides also achieves \(3/2\), so \(\mathrm{CROF}_R=1\). With insufficient sensing, the synthesis gives \(\mathrm{MinExpRew}=\infty\), hence \(\mathrm{CROF}_R=\infty\). This suggests that, in minimization settings, the directionality of “better” depends on which CROF component is being discussed: raw reward-normalized ratios and cost-aware composites need not order observation designs in the same way.

## 4. Exact CROF in latent world-model validation

The exact modern use of CROF appears in "Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander" [2607.01736]. The setting is an RSSM world model with latent state
\[
s_t=[h_t;z_t]\in\mathbb{R}^{272},
\]
where \(d_h=256\) and \(d_z=16\). Actions are one-hot vectors in \(\mathbb{R}^4\). The transition is \(s_{t+1}=f_\theta(s_t,u_t)\), the observation decoder is \(\hat o_t=g_\theta(s_t)\), and the reward head is a separate network \(\hat r_t=\rho_\theta(s_t)\). The paper’s central claim is that conventional validation losses and multi-step RMSE continue to improve long after closed-loop performance has collapsed, so checkpoint selection requires structural diagnostics.

At a validation latent state–action pair \((s,u)\), the paper defines Jacobians
\[
A=\frac{\partial f_\theta}{\partial s}\Big|_{(s,u)},\quad
B=\frac{\partial f_\theta}{\partial u}\Big|_{(s,u)},\quad
C=\frac{\partial g_\theta}{\partial s}\Big|_{s},\quad
R=\frac{\partial \rho_\theta}{\partial s}\Big|_{s},
\]
and reward gradient \(r=R^\top\). For horizon \(H=25\), the fixed-linearization controllability and observability matrices are
\[
\mathcal C_H=[B,AB,A^2B,\ldots,A^{H-1}B],
\qquad
\mathcal O_H=[C;CA;CA^2;\ldots;CA^{H-1}].
\]
After SVD, effective ranks \(k_c\) and \(k_o\) are defined by the relative singular-value threshold \(\tau=10^{-3}\).

The Reward Observability Fraction is
\[
\mathrm{ROF}=
\frac{\|V_o^{(k)\top}r\|_2^2}{\|r\|_2^2}\in[0,1],
\]
where \(V_o^{(k)}\) spans the observable subspace. The paper also defines
\[
\mathrm{RCF}=
\frac{\|U_c^{(k)\top}r\|_2^2}{\|r\|_2^2},
\qquad
\mathrm{OCF}=
\frac{\|CU_c^{(k)}\|_F^2}{\|C\|_F^2}.
\]
ROF is aggregated over two curated validation subsets: good states with return at least \(+100\), and bad states with return at most \(-100\). The combined score is
\[
\texttt{jac\_rof\_combined}
=
\alpha\cdot \texttt{jac\_rof}
+
(1-\alpha)\cdot \texttt{jac\_rof\_bad},
\qquad
\alpha=0.5.
\]

CROF then augments ROF with three structural regularizers. Let \(\mathrm{ROF}:=\texttt{jac\_rof\_combined}\), \(k_c:=\texttt{jac\_ctrl\_rank}\), \(k_o:=\texttt{jac\_obs\_rank}\), and \(e_{\mathrm{obs}}:=\texttt{ol\_obs\_avg}\). After min–max normalization across checkpoints,
\[
\tilde t=\frac{t-\min_{\mathrm{ckpt}} t}{\max_{\mathrm{ckpt}} t-\min_{\mathrm{ckpt}} t},
\]
the paper defines
\[
\mathrm{CROF\text{-}A}
=
\widetilde{\mathrm{ROF}}
+
1.0(1-\tilde k_c)
+
1.0(1-\tilde k_o)
+
1.0\tilde e_{\mathrm{obs}},
\]
and
\[
\mathrm{CROF\text{-}B}
=
\widetilde{\mathrm{ROF}}
+
0.5(1-\tilde k_c)
+
0.5(1-\tilde k_o)
+
0.5\tilde e_{\mathrm{obs}}.
\]
Lower CROF is better. The role of the regularizers is explicitly to guard against early checkpoints whose ROF is deceptively low even though the dynamics are not yet structurally usable.

## 5. Computation, empirical behavior, and deployment

The latent-world-model formulation is operationalized as a full offline validation pipeline [2607.01736]. Training uses Gymnasium LunarLander-v3 with 872 human-piloted episodes and 180,916 steps, split \(750/122\) train/validation, sequences of length \(30\) at stride \(5\), and 500 epochs with checkpoints every 5 epochs. For each checkpoint, the pipeline computes 40 metrics: 5 validation losses, 9 open-loop errors, 21 Jacobian-based metrics, 3 empirical sensitivities, and 2 composite scores. The oracle for checkpoint quality is CEM-MPC return averaged over 20 episodes per checkpoint, with a 7-point moving average used as the smoothed target.

Within that benchmark, \(\texttt{jac\_rof\_combined}\) is the strongest single predictor of smoothed MPC mean return, with Spearman \(\rho_s=-0.710\), Pearson \(r=-0.683\), and quadratic \(R_q^2=0.520\) over \(N=100\) checkpoints. The corresponding good-state and bad-state fixed ROF correlations are \(-0.648\) and \(-0.692\). CROF-B yields \(\rho_s=-0.568\), \(r=-0.495\), \(R_q^2=0.421\), while CROF-A yields \(\rho_s=-0.443\), \(r=-0.375\), \(R_q^2=0.243\). Standard training metrics are weak or near zero; for example, \(\texttt{val\_loss}\) has \(\rho_s\approx +0.090\), and one-step RMSEs are near zero. Open-loop reward RMSE at horizon end, \(\texttt{ol\_rew\_end}\), is only mildly informative, with \(\rho_s=+0.301\).

The checkpoint-selection behavior is correspondingly different. Smoothed MPC performance peaks near epoch 310 at \(+153.0\), with a plateau over roughly epochs 260–320. The smoothed minimum of ROF alone occurs at epoch 265 and attains MPC \(+156.5\), but the raw ROF minimum occurs too early, at epoch 250 with \(+115.8\). Both raw-min CROF-A and raw-min CROF-B select epoch 280, which yields MPC \(+162.8\) and lies inside the high-performance plateau. Standard criteria such as \(\texttt{val\_loss}\), \(\texttt{post\_rew\_rmse}\), \(\texttt{ol\_obs\_end}\), \(\texttt{ol\_cumrew\_err}\), or maximal \(\texttt{jac\_ctrl\_rank}\) pick late checkpoints at or beyond epoch 460 with poor MPC returns ranging from \(+3\) to \(+115.8\).

The same CROF-selected world model supports both zero-shot planning and imagined-policy training. Using the CROF raw-pick world model at epoch 280, model-based A2C trained entirely in imagination produces a best-by-mean policy with mean \(+217.5\), worst \(-178.9\), 79 of 100 perfect landings, and 2 of 100 catastrophic outcomes; the safest-by-worst variant yields mean \(+191.3\), worst \(-39.6\), 71 perfect landings, and 0 catastrophic outcomes. A fairly evaluated model-free A2C baseline trained with 50,000 episodes, or about 11.82 million steps, peaks at \(+193.04\). By contrast, the model-based A2C policy uses only the offline dataset of 180,916 steps, approximately \(65.3\times\) fewer real-environment interactions, and beats the model-free baseline by about 24.5 return points.

The delayed-reward and POMDP formulations also have explicit operational implications. In the delayed composite reward setting, increasing \(\mathrm{CROF}_{\mathrm{attr}}\) amounts to designing epochs whose action assignment is constant within each state over the effective delay window, exactly the design enforced by DUCRL2 [2305.02527]. In the POMDP setting, increasing CROF corresponds to choosing observation labels or sensors that separate states requiring different optimal actions; the underlying-MDP algorithm does this by grouping states according to the action used by an optimal positional deterministic policy [2405.10768].

## 6. Interpretation, edge cases, and limitations

Several edge cases clarify what CROF is and is not. In the delayed composite reward model, no delay implies \(d=0\), \(\mathrm{CROF}_{\mathrm{real}}^{\mathrm{cum}}(t)=1\), and \(\mathrm{CROF}_{\mathrm{attr}}(t)=1\) under partial anonymity because \(\mathbf{x}_t(s)\) then contains only the component generated at time \(t\) from \((s_t,a_t)\) [2305.02527]. If reward observations were fully non-anonymous, with each component tagged by \((s,a,u)\), attribution would always be unique. Heavy-tailed delays that violate Assumption 2 can make \(d=\infty\), in which case cumulative realization may remain far below 1 at finite time and the regret guarantees no longer hold.

In the POMDP observability-design setting, the main limitation is computational. General OOP is undecidable, and tractable formulations require restricting the strategy class to positional deterministic or positional randomized strategies [2405.10768]. The reward-normalized CROF in that setting must also be interpreted carefully because the objective is minimization of expected reward or expected steps to goal. The same synthesis gives both \(\mathrm{CROF}_R=1\) for observation designs matching full observability and \(\mathrm{CROF}_R=\infty\) for infeasible low-budget designs, while the cost-aware composite may rank a partial-observability design above full observability when sensing cost is sufficiently weighted. This suggests that, in POMDP usage, one must distinguish performance preservation from cost-adjusted desirability.

In the latent-world-model setting, ROF and CROF are explicitly motivated by non-Markovian reward in LunarLander [2607.01736]. The reward has the form
\[
r_t=\phi(o_{t+1})-\phi(o_t)-c_{\mathrm{main}}\mathbf{1}[a_t=\mathrm{main}]
-c_{\mathrm{side}}\mathbf{1}[a_t\in\{L,R\}]+r_t^{\mathrm{term}},
\]
with terminal bonus or penalty \(r_t^{\mathrm{term}}\in\{+100,-100,0\}\) depending on flags not present in \(o_t\). Predictability experiments show \(R^2=0.290\) from \((o_t,a_t)\) alone, \(R^2=0.670\) if terminals are filtered, \(R^2=0.595\) if \(o_{t+1}\) is added, and \(R^2=0.971\) with both \(o_{t+1}\) and terminal filtering. The paper’s interpretation is that a useful reward head must exploit latent information not recoverable from \((o_t,a_t)\) alone, and that low ROF is structurally aligned with open-loop robustness on that task.

The same paper also gives explicit negative cases. In fully Markovian reward settings such as Gymnasium Reacher, an MLP predicts reward from \((o_t,a_t)\) with \(R^2=0.9998\), and ROF has weak, oppositely signed rank correlation with MPC, \(\rho_s\approx +0.143\). Standard prediction metrics suffice there, and there is no late-training collapse for CROF to detect. Other listed failure modes include severe distribution shift between validation rollouts and planner trajectories, non-identifiable dynamics or degenerate latent representations, over-reliance on local linearization, and numerical instability when the reward gradient norm \(\|r\|\) is near zero.

Taken together, these results support a precise but plural interpretation of CROF. In delayed-reward RL, it is a measure of realization or attribution under delayed, composite, partially anonymous feedback. In POMDP observation design, it is a reward-preservation or reward-cost trade-off induced by sensor choices. In latent world models, it is an exact structural diagnostic for offline checkpoint selection. The common core is the same in all three settings: reward is not merely generated, but must be made visible in a form that the agent can exploit.

Source: https://www.emergentmind.com/topics/composite-reward-observability-fraction-crof