Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning

Abstract: The temporal structure of reward composition in reinforcement learning (RL) is typically hand-designed and held fixed throughout training, leaving the progression of motivational priorities largely unexplored. In this work, we propose an evolutionary framework for discovering developmental reward schedules, in which three distinct biologically inspired motivational components -- agency, novelty, and reactivity -- are combined through time-varying weights that dynamically shift over the course of training. Evaluated on two sparse-reward MiniGrid tasks: DoorKey-6x6 and KeyCorridorS3R1, our framework compares the generalizability of four evolutionary algorithms: CMA-ES, xNES, DE, and L-SHADE against an extrinsically motivated baseline (our main comparison point), and three additional hand-designed methods. On DoorKey-6x6, all evolved methods outperform the non-evolved baselines, with L-SHADE achieving the best performance -- an approximate relative mean improvement of 11.4% over the extrinsic only baseline. On KeyCorridorS3R1, CMA-ES achieves the best overall performance, with the remaining evolved methods showing weaker and less reliable generalization capability compared to the extrinsic only baseline. Interestingly, the discovered schedules diverge from our defined developmental ordering, with novelty consistently emerging as the dominant early signal during training, across both tasks. Collectively, our results position evolutionary optimization as a promising approach for developmental reward schedule discovery in deep reinforcement learning, and suggest that what evolution finds to be optimal in computational settings may differ from what it finds to be optimal in biology. The code for this project can be found at: https://github.com/alannadels/Evolutionary_RL.git.

• The paper demonstrates an evolutionary discovery of time-varying developmental reward schedules that significantly enhance DRL performance in sparse-reward tasks.
• It employs a composite reward function integrating agency, novelty, and reactivity, with evolutionary algorithms optimizing a 15-dimensional parameter space for schedule tuning.
• Experimental results reveal that novelty-dominant, evolved schedules outperform fixed biological orders, underlining the gap between computational optimization and traditional developmental models.

Evolutionary Discovery of Developmental Reward Schedules in Deep Reinforcement Learning

Motivation and Approach

The temporal evolution of motivational priorities is a hallmark of biological learning systems. This work formalizes developmental reward schedules for deep reinforcement learning (DRL), inspired by established stages from cognitive-developmental psychology: agency (effectance), novelty (curiosity-driven exploration), and reactivity (reward sensitivity). These motivational signals are modulated via time-varying weights whose schedules are not fixed but instead evolved for optimal task performance. The framework utilizes four evolutionary algorithms—CMA-ES, xNES, DE, and L-SHADE—to parameterize these weight schedules, while DRL agents are trained using PPO in MiniGrid environments, specifically DoorKey-6x6 and KeyCorridorS3R1, which feature sparse rewards and structured, goal-directed challenges.

Composite Reward Function and Parameterization

The composite reward function $r(t)$ integrates the canonical sparse extrinsic reward with three biologically-inspired intrinsic signals:

• Agency: Binary reward when agent actions lead to observable state change; operationalizes effectance motivation.
• Novelty: Count-based visitation bonus decaying with familiarity, using state-action visit frequencies; formalizes curiosity and exploration.
• Reactivity: Dense proximity-based reward as Manhattan distance to goal; reflects adolescent-like hyper-responsivity to reward proximity.

The core innovation lies in evolving the schedules $\alpha(t),\beta(t),\gamma(t)$ governing the relative weights of these signals. Schedules are parameterized as piecewise-linear functions over 5 control points (per component), collectively forming a 15-dimensional vector $\theta$ which the evolutionary algorithms optimize against cumulative episodic return.

Experimental Design

The pipeline involves a two-phase process:

1. Evolutionary Search: Evolutionary algorithms optimize $\theta$ for each task, with fitness assigned by evaluation of PPO-trained policies under only the sparse task reward.
2. Generalization Testing: Optimal schedule $\theta^*$ is held fixed and PPO training is repeated over 10 different seeds to assess robustness and generalization.

Baselines include: extrinsic only (no intrinsic signal), fixed equal weights, canonical developmental ordering (agency → novelty → reactivity), and reversed developmental ordering. Performance and ablation study results are provided for a comprehensive comparison.

Numerical Results and Schedule Analysis

The empirical findings are acute and task-dependent. On DoorKey-6x6:

• All evolved schedules surpass the extrinsic-only baseline (mean return $0.668$).
• L-SHADE yields highest outcome ($0.744 \pm 0.083$), followed by DE and CMA-ES.
• Enhancement is statistically significant relative to baseline.

On KeyCorridorS3R1:

• CMA-ES exceeds all other evolved schedules and baselines ($0.902$ mean return).
• DE and L-SHADE underperform ($0.266$ mean return), frequently converging to degenerate, non-generalizable schedules.

  Figure 1: Task performance comparison reveals L-SHADE dominance in DoorKey-6x6, and CMA-ES superiority on KeyCorridorS3R1 with high variance in DE/L-SHADE.

Ablations demonstrate that hand-designed developmental orderings and fixed equal weights fail completely—yielding zero return—underscoring that mere presence or naive ordering of motivational signals is insufficient.

Figure 2: Ablation results show non-evolved schedules (fixed, developmental, reversed) severely underperform in comparison to evolved and extrinsic-only baselines.

Evolved weight schedules consistently prioritize novelty as the dominant early signal, irrespective of task or algorithm. Contrary to biological intuition, agency does not lead the schedule, and reactivity is rarely dominant initially; instead, transition to reactivity occurs only in successful schedules (notably L-SHADE in DoorKey-6x6).

Figure 3: Schedule dynamics highlight novelty dominance early, with task-algorithm specific transitions to other signals.

Learning Dynamics and Agent Behavior

Learning curve trajectories reinforce the efficacy of proper schedule discovery. DoorKey-6x6 shows robust convergence in all evolved methods, while KeyCorridorS3R1 displays high variance in DE/L-SHADE due to non-generalizable schedules.

Figure 4: Learning curves reveal stable progression in DoorKey-6x6 but instability and failure for DE/L-SHADE in KeyCorridorS3R1.

Visualizations of agent behavior indicate that evolved schedules yield coherent, goal-directed trajectories and efficient task completion, further validating the success of the reward schedule discovery approach.

Figure 5: The best evolved L-SHADE agent solves DoorKey-6x6 across representative episodes, showcasing goal-focused actions.

Figure 6: The best CMA-ES agent achieves reliable task completion in KeyCorridorS3R1, illustrating behavioral coherence with evolved schedules.

Discussion, Implications, and Future Directions

A critical outcome is the divergence between computationally evolved and biologically prescribed motivational trajectories. All evolutionary methods favor novelty-centric initial weighting, a contradiction to biological developmental staging. This finding suggests that computational optimality in reward composition does not necessarily align with evolutionary biology, even when signals are derived from it. The effectiveness of specific evolutionary families (CMA-ES/xNES versus DE/L-SHADE) is contextually dependent on task difficulty and structure.

The results inform both DRL reward engineering and meta-RL: they emphasize the futility of static or biologically-assumed signal ordering and highlight the necessity of per-task adaptation enabled by evolutionary search. Limitations include fixed intrinsic scaling, restricted environment diversity, and lack of statistical hypothesis testing. Expansion to continuous-control environments, evolving scale parameters, and larger seed pools are proposed as avenues for future investigation.

Conclusion

Evolutionary optimization of developmental reward schedules demonstrably yields superior DRL performance in sparse-reward tasks, provided the search adapts schedules to task structure. Evolved schedules do not recapitulate biological development—novelty emerges as the optimal initial signal in computational settings, displacing agency. Fixed or biologically-informed schedules are insufficient and even detrimental. This method establishes evolutionary schedule discovery as a viable mechanism for robust reward composition in DRL, with practical relevance for automated reward architecture design and theoretical insights into the dissociation between computational and biological optimality.