---
title: Reward Observation Space Evolution
url: https://www.emergentmind.com/topics/reward-observation-space-evolution
type: topic
---

# Reward Observation Space Evolution

Searching arXiv for the cited papers on reward observation space evolution and adjacent topics.
arxiv_search(query="all:2512.13399 OR all:2504.07596 OR all:2606.20858 OR all:2406.01309 OR all:2404.12631 OR all:2302.09601 OR all:2105.08187", max_results=10)
Searching more directly for key papers and topic phrases to ground the article in recent arXiv work.
arxiv_search(query="ti:\"Differentiable Evolutionary Reinforcement Learning\" OR ti:\"Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution\" OR ti:\"Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning\"", max_results=10)
Searching adjacent papers on observation-space optimization, reward-sequence representations, and evolutionary reward systems.
arxiv_search(query="ti:\"Observation Space Matters: Benchmark and Optimization Algorithm\" OR ti:\"Generalization in Visual Reinforcement Learning with the Reward Sequence Distribution\" OR ti:\"Learning to Win, Lose and Cooperate through Reward Signal Evolution\"", max_results=10)
Searching biologically inspired and reward-bottleneck papers connected to evolving reward signals and observation-derived learning.
arxiv_search(query="ti:\"Breaching the Bottleneck: Evolutionary Transition from Reward-Driven Learning to Reward-Agnostic Domain-Adapted Learning in Neuromodulated Neural Nets\" OR ti:\"Evolution of Rewards for Food and Motor Action by Simulating Birth and Death\" OR ti:\"Evolution of Fear and Social Rewards in Prey-Predator Relationship\"", max_results=10)
Reward Observation Space Evolution denotes a family of reinforcement-learning and meta-optimization formulations in which the reward channel available to a learner is not treated as fixed, but is itself searched, parameterized, reshaped, or selectively exposed through structured features. In narrow usage, it refers to the subset of environment states and operations that a reward function is allowed to use; in broader usage, it includes evolving the mapping from observations, trajectories, or observation-derived statistics to scalar reward so that learning receives denser, more task-aligned, or more behaviorally informative feedback over time [2512.13399] [2504.07596] [2105.08187].

## 1. Conceptual scope

A central premise across this literature is that reward is not merely an objective but an *observation channel* for learning. In DERL, changing the reward function changes what the inner agent “observes” about task progress, shifting the agent from sparse outcome-only feedback toward denser and more informative signals derived from trajectory segments, formatting cues, or partial correctness [2512.13399]. In LLM-based reward design, this idea is made explicit through the “Reward Observation Space” (ROS), defined as a subset of all available environment-robot states together with the operations defined on those selected members [2504.07596].

This concept sits adjacent to, and often overlaps with, observation-space design. Work on observation-space optimization treats observation channels as a design variable on par with reward functions and action spaces, showing that changing which channels are exposed to the policy can substantially affect reward learning speed and stability [2011.00756]. Reward-centric representation learning pushes the idea further: instead of predicting observation transitions, it learns latent representations from reward-sequence distributions conditioned on observations and action sequences, thereby using reward structure itself to sculpt an invariant representation space [2302.09601].

Taken together, the topic encompasses at least three related operations. One is direct search over reward functions or reward signals. A second is search over the state features and computations that reward code may access. A third is redesign of observation or latent spaces so that reward-relevant structure becomes easier to infer. The common thread is that the mapping from interaction history to scalar evaluative feedback becomes an object of optimization rather than a fixed hand specification.

## 2. Formalizations and parameterizations

Several distinct parameterizations of reward observation space recur across the literature.

| Setting | What is optimized | Representation |
|---|---|---|
| DERL | Meta-Reward over fixed primitives | Symbolic composition \(R_\omega(o,C)=\mathrm{Func}(g_1,\dots,g_k;\omega)\) |
| LLM ROS | State selection and reward operations | \(\mathrm{ROS}(R)=(\mathrm{ROS}_{st}(R),\mathrm{ROS}_{op}(R))\) |
| Pong reward evolution | Reward signal itself | 3-bit vector \(r_j \in \{0,1\}^3\) |
| Developmental schedules | Time-varying intrinsic weights | 15-dimensional piecewise-linear schedule parameters |
| Ecological reward evolution | Innate reward coefficients | Heritable scalar weights on food, action, social, and threat terms |

In DERL, the reward function is parameterized as a symbolic expression over atomic primitives. For ALFWorld and ScienceWorld these primitives include binary outcome reward and average rewards over the first, middle, and last thirds of a trajectory; for GSM8k and MATH they include correctness, format reward, step-by-step markers, and soft outcome reward. The Meta-Optimizer does not emit raw scalar rewards; it emits a reward configuration \(\omega\) that defines a symbolic Meta-Reward \(R_\omega\) through linear combinations, sums, differences, products, divisions, and scalar weights [2512.13399].

In the LLM ROS formulation, a reward function is decomposed into \(\mathrm{ROS}_{st}(R)\), the selected environment-robot state variables, and \(\mathrm{ROS}_{op}(R)\), the internal operations over them. This makes reward design a search over both *what* the reward can inspect and *how* it computes with those inputs [2504.07596].

Older evolutionary work uses more compact parameter spaces. In Pong, the reward space is \(\mathcal{R}=\{0,1\}^3\), where each bit indicates whether a reward is emitted when the ball is in one of three spatial regions. Despite this small space, evolution discovers qualitatively different behaviors—winning, losing, and cooperating—without hand-designed rewards for those goals [2105.08187].

Temporal reward composition is another axis of parameterization. In developmental reward schedules, the total reward is
$$
r(t)=r_{\mathrm{task}}+\lambda\left[\alpha(t)r_{\mathrm{agency}}+\beta(t)r_{\mathrm{novelty}}+\gamma(t)r_{\mathrm{reactivity}}\right],
$$
where \(\alpha(t),\beta(t),\gamma(t)\) are piecewise-linear, nonnegative weight functions over training time, encoded by a 15-dimensional vector of control points [2606.20858].

Biologically inspired simulations use yet another parameterization: reward functions are linear combinations of observation-derived quantities such as food intake, motor magnitude, conspecific proximity, predator proximity, or pitfall proximity, with the coefficients inherited and mutated across generations. In this setting, evolution changes the valence assigned to specific observation features while leaving the policy-learning algorithm fixed [2406.15016] [2507.09992].

## 3. Search and meta-optimization mechanisms

The main algorithmic distinction in this area is whether reward search is derivative-free, policy-gradient-guided, LLM-mediated, or embedded in ecological birth–death dynamics.

DERL is a bilevel optimization framework. The inner loop trains a policy \(\pi_\theta\) with GRPO under a candidate reward \(R_\omega\). The outer loop trains a Meta-Optimizer \(\pi_\nu^{\mathrm{meta}}\) that proposes reward configurations \(\omega\), receives the inner policy’s validation performance as a scalar reward, and is itself updated by GRPO. This yields an approximate “meta-gradient” of task success without differentiating analytically through the environment or the inner RL process [2512.13399].

REvolve uses a genuinely evolutionary population over explicit reward programs. GPT-4 acts as a reward designer \(G\), mutation operator, and crossover operator; human pairwise comparisons of rollout videos are converted into Elo scores that serve as scalar fitness; and natural-language critiques are fed back into subsequent reward edits. An island model maintains multiple sub-populations, with mutation and crossover probabilities set by \(p_m\) and \(p_c=1-p_m\), and only individuals whose fitness exceeds the island average are retained [2406.01309].

The heuristic ROS-evolution framework is also LLM-based but places stronger structure on the search process. It introduces a state execution table that records historical usage counts and success contributions of environment states, thereby preserving exploration history in a compact tabular form and overcoming the Markovian constraint of ordinary prompt-only dialogues. It also alternates between state-member selection and internal operation editing: odd iterations, or iterations below a success threshold, emphasize \(\mathrm{ROS}_{st}\); later successful iterations emphasize \(\mathrm{ROS}_{op}\) [2504.07596].

Classical reward-signal evolution is simpler. In the Pong framework, a population of reward signals is maintained, each signal trains a DQN, goal-specific fitness functions evaluate the resulting policy, the worst signals are removed, and new bit patterns are introduced by mutation. This is derivative-free evolutionary search over reward signals, with no learned search distribution [2105.08187].

Developmental reward schedules sit between these extremes. A candidate schedule \(\theta\) fixes the entire time course of \(\alpha(t),\beta(t),\gamma(t)\) for one PPO training run; evolutionary algorithms such as CMA-ES, xNES, DE, and L-SHADE optimize \(\theta\) as a black-box outer-loop parameter, using final task performance under sparse extrinsic reward as fitness [2606.20858].

## 4. Empirical findings in engineered domains

In benchmark RL domains, reward observation space evolution is consistently associated with better alignment between feedback density and task structure, although the mechanism differs by setting.

DERL reports state-of-the-art results on ALFWorld and ScienceWorld and improves markedly in out-of-distribution regimes. On ALFWorld L2, outcome reward yields \(29.7\%\) success, while DERL reaches \(65.0\%\) and DERL-pop reaches \(76.4\%\). On ScienceWorld, DERL and DERL-pop again lead the reported baselines, and in mathematical reasoning DERL improves over outcome-only rewards while avoiding the performance degradation that simple heuristic additions can cause on hard MATH problems [2512.13399].

In LLM-mediated reward design, REvolve outperforms expert-designed rewards, Eureka, and automatic variants on the AirSim autonomous-driving benchmark. Under the manually designed fitness used in the paper, REvolve reaches \(0.84\), compared with \(0.65\) for the expert-designed reward; under human Elo evaluation, REvolve ranks second only to human driving, with score \(1534\) versus \(1595\) for the human baseline [2406.01309]. The ROS-evolution framework likewise surpasses Eureka on 9 of 20 Bi-dexterous Manipulation tasks, matches it on 5, and underperforms on 6, with threshold sensitivity identified as the main cause of the underperforming cases [2504.07596].

Evolution over reward schedules produces a different empirical pattern. On DoorKey-6x6, all evolved schedules outperform the extrinsic-only baseline, and L-SHADE achieves an approximate relative mean improvement of \(11.4\%\) over that baseline. On KeyCorridorS3R1, CMA-ES is best overall, while DE and L-SHADE are much less reliable. A notable result is that hand-designed “Developmental” and “Fixed Equal” schedules yield \(0.000 \pm 0.000\) on DoorKey-6x6, showing that simply adding intrinsic components or imposing a biologically inspired ordering is not sufficient [2606.20858].

Discrete reward-signal evolution in Pong offers a compact demonstration of the same principle. The reward signal \(100\) yields the best “winning” behavior, \(000\) yields “losing,” and \(110\) yields the strongest cooperative behavior by keeping the ball in play. These signals are non-intuitive relative to score-based hand design, and some evolved signals show higher training stability than score-based baselines because they provide denser feedback [2105.08187].

Across these domains, a recurrent empirical pattern is that denser or better-structured signals help only when they are *selective*. DERL shows that simple additions such as format reward or average reward can hurt MATH, whereas meta-optimization can learn when and how much these signals should matter [2512.13399]. This suggests that reward observation space evolution is less about indiscriminately adding signals than about reallocating informational bandwidth toward features that correlate with downstream success.

## 5. Observation-side and representation-side extensions

A broader strand of work studies how observation-space redesign and reward-centric representation learning alter what information becomes available for control.

Observation-space optimization in continuous control treats sensor channels as a search object. Starting from a hardware-realistic raw-sensor set, the search algorithm adds semantic groups such as root position or root linear velocity, evaluates them by cumulative reward, and then removes unnecessary or malicious channels with a Dropout-Permutation test. The resulting “Search” observation spaces are better or comparable to OpenAI Gym defaults on Hopper, Walker2D, and InvertedDoublePendulum, with root linear velocity selected almost always in locomotion tasks, while the absolute horizontal root position \(x\) is identified as a harmful channel [2011.00756].

Reward-centric representation learning approaches the same issue from the opposite direction: instead of modifying sensors explicitly, it learns a latent space from reward statistics. The reward sequence distribution conditioned on starting observation and a predefined action sequence, \(p(\mathbf{r}\mid o,\mathbf{a})\), is invariant to visual distractions because it excludes observation-space transition dynamics and captures long-term task-relevant information in both rewards and latent transitions. Its characteristic-function-based instantiation, CRESP-T, improves generalization on Distracting Control Suite tasks relative to DrQ, CURL, DBC, and MISA [2302.09601].

In co-robotic visual exploration, the robot cannot plan directly in raw image space, so it maps images into a low-dimensional semantic space \(\mathcal{Z}\), learns a reward model \(g_\theta(\mathcal{Z})\), and actively requests labels that are expected to minimize trajectory regret. In some bandwidth-limited environments, the regret-based query criterion enables the robot to collect up to \(17\%\) more reward per mission than the next-best criterion, illustrating that reward-model quality depends jointly on observation representation and label-allocation policy [2003.05016].

The reward-mixing MDP formalism introduces a different kind of observation expansion. Here, the environment randomly selects one of two reward models at the start of each episode, but the agent does not observe which one was chosen. The algorithm therefore augments the effective observation space with second-order reward correlations across state-action pairs and proves that an \(\epsilon\)-optimal policy can be learned in \(\tilde{O}(\mathrm{poly}(H,\epsilon^{-1})\cdot S^2A^2)\) episodes, which is also shown to be essentially unavoidable [2110.03743].

A more local version appears in SESTRA for instruction following. Rather than observing reward only for the chosen action, the algorithm evaluates the single-step reward of *all* actions at each visited state and updates the policy by immediate expected reward maximization. On SCONE, this yields absolute accuracy improvements of \(9.8\%\) to \(25.3\%\) over approaches using high-level logical representations [1805.10209]. In all of these cases, the system changes not the environment’s ontology but the learner’s effective access to reward-relevant structure.

## 6. Biological and neuromodulatory interpretations

Biologically inspired work interprets reward observation space evolution at two levels: evolution can reshape the reward function itself, and it can eventually diminish runtime reliance on scalar reward altogether.

EvER fixes a reward aligned with evolutionary fitness rather than learning it by outer-loop evolution. The reward for agent \(i\) is a function of how many agents sharing its genome remain alive, computed through kinship over the whole population; a “sugary” variant additionally weights kinship by food collected. This expands the reward’s input space from local task outcomes to population-level relational observations and allows RL to exploit full trajectories rather than the single scalar fitness used by evolutionary strategies [2004.00048].

In the birth–death ecology of food and motor-action rewards, reward parameters are heritable traits. Positive rewards for food acquisition and negative rewards for motor action can evolve from random initialization, but motor-action rewards diverge into two modes: largely positive and slightly negative. In environments with poor and poisonous foods, the reward weights for less important foods are unstable while rewards for normal foods remain stable, implying that some axes of reward space are tightly constrained and others weakly selected [2406.15016].

The prey–predator study extends this logic to fear and social reward. The prey reward includes coefficients on food, action magnitude, proximity to other prey, and proximity to predators. The main result is that social reward for observing the same species is more important for prey survival, and fear-like negative reward for observing predators evolves only after acquiring social reward. Larger predator mouths amplify fear emergence, while stationary threats can instead produce positive rewards because leftover food accumulates around them [2507.09992].

The most radical reinterpretation appears in the transition from reward-driven learning to reward-agnostic domain-adapted learning. In that work, standard RL is treated as a scalar “reward bottleneck,” and evolution is allowed to introduce neuromodulatory weight updates that use non-reward information directly. Evolved DAL agents show a 300-fold increase in learning speed compared to pure RL agents, and evolution eventually eliminates reliance on reward information altogether, with learning driven exclusively by non-reward signals and local neuromodulation-based updates [2404.12631]. This reframes reward observation space evolution as a possible transition from optimizing the reward channel to bypassing it.

## 7. Limitations, controversies, and future directions

Several limitations recur across the literature. Bilevel and evolutionary reward search is computationally heavy because each outer-loop update can require multiple full inner-loop RL runs; DERL explicitly identifies inner-loop GRPO as the dominant cost, and developmental reward-schedule discovery evaluates full PPO trainings for every candidate schedule [2512.13399] [2606.20858]. Human-in-the-loop systems add another bottleneck: REvolve requires repeated policy training plus pairwise human evaluation, and its storage and coordination costs exceed greedy one-best frameworks [2406.01309].

A second limitation is representational dependence. DERL can compose but not invent new primitives, so reward expressivity is bounded by the chosen atomic set [2512.13399]. ROS-based LLM methods assume that relevant observation variables are already exposed through the environment interface, and threshold selection can strongly affect whether the search explores state membership or reward operations [2504.07596]. Observation-space search likewise depends on predefined semantic groups and can bias solutions toward redundancy; HalfCheetah is identified as a case where that bias can hurt [2011.00756].

The literature also challenges several common simplifications. There is not a unique reward for a given high-level goal: Pong shows that \(100\), \(000\), and \(110\) can induce winning, losing, and cooperating, and some of these rewards are denser and more stable than obvious score-based alternatives [2105.08187]. Adding more heuristic signals is not automatically beneficial: on DoorKey-6x6, fixed-equal and hand-designed developmental schedules collapse to zero performance, and in MATH, naive heuristic reward combinations can hurt compared with outcome-only training [2606.20858] [2512.13399]. Likewise, “obvious” observation channels are not always useful: explicit contact flags are often non-critical in locomotion, whereas a seemingly innocuous variable such as horizontal root position can be actively harmful [2011.00756].

Future directions proposed in the primary sources are correspondingly specific. DERL points to lighter inner-loop algorithms, automatic primitive discovery, and intermediate or hierarchical meta-objectives [2512.13399]. The ROS framework points to adaptive threshold search for balancing state selection against operation refinement [2504.07596]. Developmental reward schedules point to co-evolving the intrinsic scale \(\lambda\), richer intrinsic components, and broader environments [2606.20858]. Observation-space search points to joint optimization of policy parameters and observation space, and the DAL framework points to more complex domains and more biologically plausible routes from reward-driven learning to reward-agnostic adaptation [2011.00756] [2404.12631].

The cumulative picture is therefore neither a single algorithm nor a single definition. Reward Observation Space Evolution is best understood as a research program centered on the controlled transformation of evaluative information: from sparse to dense, from fixed to learned, from local to symbolic or relational, and, in some formulations, from scalar reward dependence toward richer non-reward learning signals. Across symbolic meta-rewards, LLM-generated reward programs, developmental schedules, observation-space search, and ecological reward evolution, the topic studies how changing what the learner can “see” through reward reshapes exploration, generalization, and alignment.

Source: https://www.emergentmind.com/topics/reward-observation-space-evolution