---
title: Reward Machines in Reinforcement Learning
url: https://www.emergentmind.com/topics/reward-machines
type: topic
---

# Reward Machines in Reinforcement Learning

A reward machine (RM) is a finite-state automaton designed to encode non-Markovian or temporally extended reward functions in reinforcement learning (RL). By explicitly representing high-level task structure using abstract states and transitions triggered by logical events or observations, RMs allow RL agents to decompose complex objectives into subproblems, utilize dense and shaped rewards for efficient learning, and facilitate modular policy design and transfer across tasks. The reward machine formalism supports a wide range of extensions—including automatic inference from demonstration or policy traces, stacking into hierarchies, accommodation of numeric or stochastic rewards, and automated synthesis via language or planning—which position it as a central tool for the specification, learning, and decomposition of structured RL tasks [2010.03950][2012.14464][2508.06894][2510.14176][2404.19370][2110.00096].

## 1. Formal Definition and Semantics

A reward machine is conventionally formulated as a Mealy-style finite-state machine. Given an underlying MDP $M = \langle S, s_0, A, T, r, \gamma \rangle$, an RM is specified as
$$
\mathcal{R} = (U, u_0, \Sigma, \delta, \rho)
$$
where

- $U$ is a finite set of abstract or automaton states (subtasks, task stages, or logical modes)
- $u_0 \in U$ is the initial RM state
- $\Sigma$ is a finite alphabet of observations, typically subsets of atomic propositions ($2^{AP}$), labels, or symbolic events extracted from the MDP state
- $\delta: U \times \Sigma \to U$ is the deterministic state transition function
- $\rho: U \times \Sigma \to \mathbb{R}$ (or $\mathbb{R}^n$ in multi-agent) is the reward-output function

On each environment transition $(s, a, s')$, the RM observes a label $\sigma = L(s, a, s') \in \Sigma$ (where $L$ is a labeling function), updates $u' = \delta(u, \sigma)$, and emits reward $r = \rho(u, \sigma)$. This effectively augments the MDP with a product state space $S \times U$, transforming a non-Markovian reward structure into a Markovian one over the joint state, and enabling standard RL algorithms to be applied directly [2012.14464][2010.03950][2404.19370].

RMs can natively handle sparse, temporally extended, and logically complex tasks by tracking progression through abstract states and gating rewards on the satisfaction of both logical and temporal task requirements [2010.03950][2205.15752].

## 2. Reward Machine Construction and Shaping

### Structure and Synthesis

RM states correspond to high-level task abstractions, which may be derived from:

- **Atomic Propositions**: Boolean features indicating relevant environment events (e.g., "block-on-goal", "gripper-holding-red") [2012.14464].
- **Demonstrations**: Trajectories from expert behaviour, mapped to feature traces; abstract planning graphs are constructed via clustering and event extraction, from which RM states and transitions are induced [2412.10096][2012.14464].
- **Temporal Logic**: Task specifications given in (sequential) LTL or similar formal languages can be algorithmically compiled into minimal RMs via formula progression and decomposition [2111.09475].

### Reward Shaping

Potential-based reward shaping is integrated by assigning a potential function $\mathrm{Pot}: U \to \mathbb{R}$ over RM states. The shaped reward is
$$
\rho'_{u}(s, a, s') = \rho_{u}(s, a, s') + \gamma \cdot \mathrm{Pot}(u') - \mathrm{Pot}(u),
$$
where $u' = \delta(u, \sigma)$. Optimal "zero-centered" shaping uses $\mathrm{Pot}^*(u) = \gamma^{\mathrm{dist}(u)}$, with $\mathrm{dist}(u)$ the shortest distance in the abstract planning graph to a goal node, to stabilize Q-values and training [2012.14464][2010.03950].

Automated construction and shaping often leverage only expert state sequences, not action labels, and require careful selection of atomic propositions to make goal and non-goal abstract states distinguishable [2012.14464][2412.10096].

## 3. Learning and Inference with Reward Machines

### RL Integration

The canonical approach is to define the product MDP $M' = (S \times U, s_0', A, T', r', \gamma)$, where
- $T'((s,u), a, (s', u')) = T(s, a, s')$ if $u' = \delta(u, L(s, a, s'))$
- $r'((s,u), a, (s',u')) = r(s,a,s') + \rho(u, L(s, a, s'))$

A Q-network or Q-table is maintained over $(s, u)$, and training proceeds as in classical RL. Policy architectures often concatenate visual or state features with a one-hot RM state encoding [2012.14464][2010.03950]. Off-policy techniques such as counterfactual reasoning (CRM) leverage the known RM structure to accelerate learning by updating Q-values for all possible RM states using each transition [2010.03950].

### RM Inference and Learning

When the reward function is not provided as an RM, inference is possible from:

- **Counterexamples**: Iterative learning methods (JIRP) alternate between policy optimization under the current RM hypothesis and RM minimization from observed reward-label inconsistencies, converging to the true minimal RM under sufficient coverage [1909.05912].
- **Partially Observable Policies**: Prefix-tree policies derived from traces suffice to reconstruct RM structure via SAT-based constraint solving, up to policy equivalence [2502.03762].
- **Demonstrations**: Feature clustering and event segmentation from demonstrations induce RM transitions without predefined propositions, particularly effective in vision-based robotic manipulation [2412.10096].

Theoretical bounds establish sufficient episode length and counterexample coverage for minimality and correctness, with practical heuristics for polynomial-time inference in large domains [1909.05912][2502.03762].

## 4. Extensions: Numeric, Stochastic, and Hierarchical Reward Machines

### Numeric and Stochastic Extensions

- **Numeric Reward Machines** extend classical RMs to directly handle real-valued features, enabling dense reward shaping for inherently numeric tasks (e.g., distance-to-goal). Hybrid (numeric-Boolean) RMs use Boolean proxies for numerical changes (e.g., "distance-decreased") [2404.19370].
- **Stochastic Reward Machines (SRMs)** further generalize RMs to model rewards as random variables, parameterized by output-distribution families and inferred from noisy data via constraint-based learning [2510.14837].

These extensions preserve the Markovization property in the product MDP and inherit RM-facilitated sample efficiency.

### Hierarchical and Pushdown Reward Machines

- **Hierarchies of Reward Machines (HRMs):** By allowing RMs to invoke other RMs as subroutines, HRMs enable the modular decomposition of deeply structured tasks. Learning and policy optimization leverages the options framework, matching the expressiveness of flat RMs with potentially exponential gains in compactness and sample efficiency [2205.15752][2403.07005].
- **Pushdown Reward Machines (pdRMs):** By equipping RMs with a (possibly truncated) stack, pdRMs enable the specification of deterministic context-free reward languages, admitting tasks involving nested or retracing patterns that cannot be expressed with regular-language RMs [2508.06894]. Policy space complexity grows exponentially with stack depth, but in many domains only top-$k$ access suffices.

## 5. Applications: Single-Agent, Multi-Agent, and Automated RM Synthesis

### Single-Agent and Vision-Based RL

In vision-based robotic manipulation, RMs constructed from demonstration induce dense reward signals and guide learning over high-level subgoals, drastically improving convergence speed and policy quality compared to purely end-to-end RL [2012.14464][2412.10096]. RMs also support curriculum-guided self-paced RL, enabling rapid learning of long-horizon tasks by focusing curricula and value updates on subtask-relevant context dimensions [2305.16505].

### Multi-Agent RL and Decentralization

In cooperative multi-agent environments, team-level RMs are decomposed (via projection and bisimulation) into local RMs under provable correctness conditions, empowering fully decentralized policy optimization while guaranteeing completion of the global task. Value bounds and sample efficiency improvements of up to an order of magnitude over centralized or independent baselines are reported [2007.01962][2110.00096][2403.07005]. Hierarchical RM structures enable efficient multi-agent learning of temporally and logically entangled subtasks [2403.07005][2205.15752].

### Automated Synthesis and Foundation Model Integration

- **Automated RM Synthesis:** ARM-FM ([2510.14176]) employs foundation models to generate RMs from natural-language task descriptions, with generation, self-critique, and verification in multiple rounds. Event labeling functions and language-aligned state embeddings are produced, supporting sample-efficient compositional RL and zero-shot transfer.
- **Maximally Permissive RMs:** Synthesis from AI planning (e.g., STRIPS domains) using the full set of partial-order plans generates RMs that encode all valid high-level behaviors, provably permitting optimal policies and avoiding the over-constraint of single-plan RMs [2408.08059].

Table: Representative Extensions and Synthesis Methods

| Extension/Method                | Core Feature                                  | Reference     |
|---------------------------------|-----------------------------------------------|---------------|
| Numeric Reward Machines         | Real-valued features and hybrid labels        | [2404.19370]  |
| Stochastic Reward Machines      | Output distributions (CDFs), noise handling   | [2510.14837]  |
| Pushdown Reward Machines        | Stack memory, deterministic context-free Lang.| [2508.06894]  |
| Hierarchical RMs (HRM/MAHRM)    | Compositional, options-based HRL              | [2205.15752], [2403.07005] |
| Automated Synthesis (ARM-FM)    | Foundation models, language-based generation  | [2510.14176]  |
| Planning-based Synthesis (MPRM) | Partial-order, maximally permissive plans     | [2408.08059]  |

## 6. Theoretical Properties, Sample Efficiency, and Limitations

### Theoretical Properties

- RMs admit a regular-language characterization; RMs can encode any temporally extended reward representable by a DFA over event labels [2010.03950][2508.06894].
- The potential-based shaping guarantees preservation of optimal policies under suitable boundedness and potential assignment to goal states [2012.14464][2010.03950].
- Joint RM/policy inference algorithms enjoy convergence guarantees under sufficient exploration and identifiable structure [1909.05912][2510.14837][2502.03762].

### Sample Efficiency and Empirical Findings

Empirical studies report order-of-magnitude speedups in learning and performance, especially in long-horizon, sparse, or compositional RL domains [2012.14464][2010.03950][2412.10096][2111.09475][2205.15752]. Dense reward shaping, counterfactual experience, and accurate progression through abstract states enable rapid policy refinement.

### Limitations

- RM expressiveness is bounded to regular languages, unless extended (e.g., via pdRMs).
- Inference of minimal RMs is computationally hard in general; practical methods rely on heuristics, approximations, or function approximation in large or continuous domains [1909.05912][2404.19370].
- Symbol grounding in high-dimensional or noisy environments requires abstraction models or auxiliary inference modules, with theoretical and practical limitations under partial observability [2406.00120].

The design of atomic propositions and labeling functions remains a bottleneck, although recent foundations-model-based and unsupervised approaches are mitigating this constraint in practice [2510.14176][2412.10096].

## 7. Future Directions and Research Outlook

Principal directions for RM methodology advancement include:

- **Scalable and incremental RM inference**, such as online methods and active learning of event labels and RM transitions [1909.05912][2412.10096].
- **Integration with expressive automata** (e.g., pushdown, stochastic) for complex or partially observed environments [2508.06894][2510.14837].
- **Automated RM synthesis** from language (prompt-based) or planning domains, coupled with foundation model verification and code generation [2510.14176][2408.08059].
- **Deep RL architectures** that are robust to noisy or ambiguous symbol grounding, via belief-based or temporally dependent inference [2406.00120].
- **Multi-agent composition and equilibrium design** using RM-structured incentives, task decompositions, and coordinated or decentralized learning [2408.10074][2305.17372][2403.07005].

Reward machines, through their modular, automaton-based abstraction, are establishing themselves as a central substrate for scalable, interpretable, and theoretically grounded reinforcement learning in temporally and logically elaborate domains.

Source: https://www.emergentmind.com/topics/reward-machines