--- title: Reinforcement Learning from Execution-Based Rewards url: https://www.emergentmind.com/topics/reinforcement-learning-from-execution-based-rewards type: topic --- # Reinforcement Learning from Execution-Based Rewards Reinforcement learning from execution-based rewards refers to a broad class of RL paradigms in which the learning agent does not receive dense, immediate reward feedback for each action, but rather only receives signals at the level of entire executions, partial trajectories, bags of transitions, or outcomes evaluated on completion. These settings arise naturally in domains where reward attribution is unavailable or prohibitively sparse, including robotics, program synthesis, code generation, optimal execution in finance, and multi-turn interaction tasks. The field has developed novel theoretical foundations, general methodologies, and pragmatic algorithms to overcome the challenges posed by the lack of granular reward supervision. ## 1. Formal Frameworks for Execution-Based Reward RL Execution-based rewards are typically formalized by extending or modifying the standard Markov Decision Process (MDP) formulation. A key framework is the **Bagged Reward Markov Decision Process** (BRMDP), defined as follows [2402.03771]: - **Standard MDP**: $M = (S, A, P, r, \mu)$, where $S$ is the state space, $A$ is the action space, $P$ is the transition kernel, $r: S \times A \to \mathbb{R}$ is the immediate reward (unobservable in this setting), and $\mu$ the initial state distribution. - **Bag Partition**: A trajectory $\tau = \{(s_0, a_0), \ldots, (s_{T-1}, a_{T-1})\}$ is partitioned into a sequence of contiguous segments (bags) $B_{i, n_i}$, with $G: \tau \mapsto \{B\}$. - **Bagged Reward Function**: $R(B_{i, n_i}) = \sum_{t=i}^{i+n_i-1} r(s_t, a_t)$. - **BRMDP**: $M_B = (S, A, P, R, \mu)$, with $R$ defined only over bags. The agent never observes the individual $r(s_t, a_t)$ per action, but only the total reward over bags (or entire trajectories). Other approaches formalize outcome-based or non-Markovian reward models using **finite-state reward automata** [2006.15714] or **reward machines** [2408.14871], which map execution traces to scalar rewards. The principal RL objective remains the maximization of expected cumulative reward, where the expectation is now taken over the sum of bag or trajectory-level rewards. ## 2. Theoretical Guarantees and Policy Optimality A foundational theoretical result underpins the feasibility of learning from execution-based rewards: **optimal policy equivalence**. In the BRMDP framework, the set of optimal policies in the original MDP coincides with that in the BRMDP [2402.03771]. Specifically, - Let $\Pi$ be the set of optimal policies for the original MDP, and $\Pi_B$ for the BRMDP. - Then, under standard exploration assumptions, $\Pi = \Pi_B$. The crucial insight is that as long as the sum of redistributed per-step rewards over each bag matches the observed bag reward, any such redistribution yields an equivalent MDP for policy optimization: $$ \forall B_{i, n_i}: \quad \sum_{t=i}^{i+n_i-1} \tilde{r}(s_t, a_t) = R(B_{i, n_i}). $$ Therefore, the RL problem can, in principle, be recast as standard RL using any consistent reward decomposition. A similar guarantee holds in the automaton and reward-machine formalisms. If the non-Markovian reward function is encoded by a finite-state reward automaton, model-based inference (e.g., Angluin's L* algorithm) is guaranteed to recover a minimal automaton that, when combined with the MDP via a product construction, yields a Markovian reward in the augmented state space [2006.15714]. Q-learning on the synchronous product converges to the optimal policy. ## 3. Algorithmic Methodologies for Execution-Based RL Approaches to RL from execution-based rewards can be classified according to how they recover or infer per-step learning signals. ### 3.1 Reward Redistribution The **Reward Bag Transformer** (RBT) is a parametric approach that learns to redistribute bag-level rewards over individual transitions using a Transformer-based model [2402.03771]. RBT operates as follows: - Inputs: A bag of contiguous state-action pairs, $\sigma = [(s_0, a_0), \ldots, (s_{M-1}, a_{M-1})]$. - Transformer Encoding: Apply a causal encoder, followed by a bidirectional attention layer, yielding embeddings for each instance. - Reward Prediction: A head computes per-instance rewards $\hat{r}_t$ using attention across the bag, ensuring context and temporal nuance is captured. - Training Objectives: Minimize bag-level reward prediction loss and, optionally, next-state prediction loss for each transition. - The reward model is iteratively updated from rollouts, after which all transitions in the agent's replay buffer are relabeled with the predicted $\hat{r}_t$. Standard RL algorithms (e.g., SAC) are then applied to the relabeled data. Ablation studies indicate that both bidirectional attention and joint modeling of transition dynamics are critical for accuracy, especially as bag lengths increase. ### 3.2 Learning Reward Automata In domains with inherently non-Markovian, history-dependent rewards, an alternative is to explicitly infer a finite-state machine representing the reward logic [2006.15714, 2408.14871]. The key steps: - Induce a reward automaton via active learning (e.g., L* algorithm based on membership and equivalence queries), or via inductive logic programming techniques even from noisy traces. - Encode episodic or trajectory-level feedback into the automaton structure. - Construct the synchronous product with the environment, yielding a Markovian RL problem in the expanded state space. This methodology provides interpretability, can generalize to noisy or partial labeling, and leverages temporal abstraction. ### 3.3 Outcome-Based, Model-Based, and Adversarial Reward Learning - **Outcome-driven RL via variational inference** directly models the probability of reaching a desired goal state and derives an execution-based reward as $\log p_\psi(g | s, a)$, with dense shaping adaptively provided by the learned dynamics [2104.10190]. - **Observation-based internal modeling** leverages expert trajectories to fit a next-state predictor and defines the reward as negative prediction error between the model's output and observed $s_{t+1}$, supplying a shaped signal for exploration and alignment to demonstrated behavior [1806.01267]. - **Adversarial reward modeling** constructs a binary classifier to discern successful goal states given instructions, training the agent with the classifier's outputs as step rewards, thus decoupling reward provision from the environment [1806.01946]. ### 3.4 Domain-Specific Execution-Based Reward Integration In practical RL for code (e.g., program synthesis or code repair), outcome-based rewards are extracted via test-case execution, while dense and structured feedback is introduced at the level of variable trajectory alignment [2510.18471]. In long-horizon planning tasks (e.g., multi-step software engineering in dockerized sandboxes), **gated reward accumulation** aggregates stepwise rewards only when the outcome (terminal execution) meets a predefined threshold, preventing policy collapse due to reward hacking [2508.10548]. ## 4. Empirical Evidence and Comparative Performance Empirical results across diverse environments demonstrate that principled reward redistribution and reward model learning are essential for robust performance under execution-based feedback. - In continuous-control MuJoCo tasks with increasing bag length, direct application of standard RL algorithms to sparse bag rewards results in collapse or instability. RBT maintains near-oracle performance up to large bag sizes (e.g., bag length 500), outperforming baselines such as IRCR, RRD, LIRPG, and HC [2402.03771]. - In grid-world and Minecraft tasks with non-Markovian rewards, active automaton inference accelerates convergence (50-90% fewer episodes) over logic-inference and policy-optimization baselines [2006.15714]. - Code generation tasks using CodeRL+ show pass@1 rates of 90.9% on HumanEval and large gains across code reasoning and test output tasks, with ablation confirming the critical role of execution-trajectory alignment [2510.18471]. - Gated reward accumulation in long-horizon code-editing raises completion rates from 47.6%→93.8% and modification rates from 19.6%→22.4%, outperforming direct reward accumulation and avoiding policy collapse [2508.10548]. - Internal-model-based reward shaping achieved rapid convergence and higher stability compared to sparse or hand-crafted rewards in pixel-based and continuous-action domains [1806.01267]. - Reward machine inference from noisy traces using probabilistic ILP yields agent performance on par with perfect RM baselines, robust even under high label noise [2408.14871]. ## 5. Challenges, Limitations, and Open Problems Despite progress, execution-based reward RL remains challenging: - **Identifiability and Credit Assignment**: Reward redistribution under long or variable-length bags reduces informational granularity and impedes accurate attribution [2402.03771]. - **Model Dependence**: Approaches such as outcome-driven inference, internal modeling, and adversarial reward shaping rely on accurate, generalizable models or discriminators. Sparse expert data, narrow bag coverage, or model misspecification degrade reward quality. - **Computational Complexity**: Transformer-based redistribution and automaton inference (especially under noise) introduce significant computation and data requirements. - **Exploration**: Execution-based signals are inherently sparse—exploration mechanisms must bridge potentially large gaps between observed signals. - **Assumptions on Bagging**: Most methods assume a known or task-defined bag partition function. Joint learning of partition and reward redistribution is an open problem [2402.03771]. - **Robustness to Noise and Non-IID Feedback**: Automaton and reward machine methods have begun addressing this via Bayesian, belief-tracking, and ILP frameworks, but general robustness remains an area of active research [2408.14871]. ## 6. Emerging Applications and Future Directions - **Structured Program Synthesis and Repair**: Program-level verifiable reward feedback, enhanced with variable-alignment and outcome tracking, is central to RL-driven code assistants [2510.18471]. - **Long-Horizon Planning in Interactive and Multi-Agent Systems**: Execution-verified and gated reward strategies stabilize learning for complex multi-turn domains such as software repositories and robotics [2508.10548]. - **Benchmarking and Monitoring**: In finance, execution-based RL frameworks closely integrate regulatory-style benchmark comparison and retraining for continuous policy deployment [2208.06244]. Anticipated research will focus on joint inference of bag partitions and reward mappings, higher-level temporal abstraction in reward modeling, generalization from sparse or noisy execution feedback, and theoretically grounded approaches to exploration in execution-based reward settings. ## 7. Summary Table: Key Algorithmic Paradigms | Approach | Execution-Based Reward Formulation | Core Method | Principal Citation | |------------------|------------------------------------------------|------------------------------------------------|---------------------| | Reward Redistribution (RBT) | Bag/trajectory-level sum | Transformer model, bidirectional attention | [2402.03771] | | Reward Automata/Reward Machines | Non-Markovian execution traces | Automaton induction (L*, ILP), RL on product MDP | [2006.15714], [2408.14871] | | Adversarial/Outcome-Driven Reward | Discriminator or probability model over terminal/goal states | Discriminator training, variational inference | [1806.01946], [2104.10190] | | Gated Reward Accumulation | Hierarchical outcome + stepwise critics | Policy gradient with reward gating | [2508.10548] | | Execution Semantics Alignment (Code) | Test-case + variable trajectory | On-policy rollouts, dual-advantage policy gradient | [2510.18471] | | Model-Based Internal Reward | Next-state prediction from expert trajectories | RNN/CNN predictive model, negative prediction error | [1806.01267] | This taxonomy illustrates the breadth of algorithmic innovation addressing the fundamental challenge of optimizing policies under execution-based, sparse, and temporally aggregated reward signals. Source: https://www.emergentmind.com/topics/reinforcement-learning-from-execution-based-rewards