---
title: Reward Machines in Reinforcement Learning
url: https://www.emergentmind.com/topics/reward-machines-rm
type: topic
---

# Reward Machines in Reinforcement Learning

Reward Machines (RM) are a formal automata-based abstraction for encoding non-Markovian reward functions in reinforcement learning (RL) via finite-state machines whose transitions and outputs depend on high-level events or propositional atoms. Unlike traditional black-box reward functions, RMs expose reward function structure, support reward decomposition at the level of temporally extended subgoals, and enable substantial improvements in sample efficiency, transfer, and lifelong learning by making explicit the stages and logic of task progress [2010.03950, 2111.09475, 2303.14061, 2501.00364]. RMs have been generalized to first-order logic, stochastic and probabilistic settings, and hierarchical and multi-agent contexts, and serve as a foundation for transfer, automated reward shaping, multi-agent decomposition, and compositional RL.

## 1. Formal Structure and Expressive Power

An RM is a tuple $(U, u_0, F, \delta, R)$ with:
- $U$: finite set of internal RM states (corresponding to formulas, subgoals, or abstract progress markers),
- $u_0 \in U$: initial RM state,
- $F \subseteq U$: set of terminal (accepting, success, failure) RM states,
- $\delta: U \times \Sigma \to U$: transition function on RM state and high-level label $\ell \in \Sigma = 2^P$ (where $P$ is the set of propositional atoms / high-level events),
- $R: U \times \Sigma \to \mathbb{R}$: transition-based reward function, e.g., $R(u, \ell) = 1$ if $\delta(u, \ell) \neq u$ and $0$ otherwise or potential-based shaping variants [2011.09475].

RMs augment standard Markov Decision Processes (MDPs) by tracking sufficient automata state to make any regular history-based reward Markovian. RMs subsume Markovian rewards (one-state RM) and encode any reward function over regular event languages, supporting temporally extended objectives such as sequences, branching, conditional logic, arbitrary regular expressions, and specifying temporally extended properties equivalent to co-safe linear temporal logic (LTL) over propositional atoms [2010.03950]. More expressive classes, such as pushdown reward machines, extend to deterministic context-free reward languages [2508.06894].

## 2. Synthesis and Construction of Reward Machines

### 2.1. Construction from Temporal Logic

For tasks specified in LTL or its extensions (e.g., Sequential LTL, SLTL), RMs are synthesized by progression: each RM-state corresponds to a “progressed” subformula, and transitions encode how observed event labels update the remaining specification. The available states $U$ are the fixpoint closure under subformula decomposition and progression $\text{prog}(\psi, \ell)$. For $\psi_1 \sim \psi_2$ (“do $\psi_1$ then $\psi_2$”), if $\psi_1$ is false, the RM moves to $\text{prog}(\psi_2, \ell)$; otherwise, it “progresses” $\psi_1$ [2111.09475]. This supports flexible, automatic RM construction from high-level LTL or SLTL logical task specifications.

### 2.2. Learning RMs from Demonstrations

Reward Machines can also be inferred from demonstration traces. Techniques include density-based clustering over learned high-level features extracted from raw observations (e.g., visual state embeddings via ResNet-50, then DBSCAN clustering) to identify candidate RM states (prototypes), followed by construction of a minimal automaton that encodes observed transitions in demonstration traces [2412.10096]. Without prior knowledge of explicit event labels, unsupervised clustering and inductive logic programming (ILP) frameworks (e.g., ILASP) can synthesize minimal, trace-consistent RMs [2112.09477, 2408.14871].

### 2.3. Automatic Extension and Transfer

Given a growing sequence of tasks, procedures such as ExtendRM($\varphi$) incrementally add to the “memory RM” all new subformulas and transitions needed for a new specification $\varphi$, reusing or composing policies for already learned sub-RMs [2111.09475]. This mechanism supports modular accumulation of reusable skills, state graphs, and Q-functions over the agent’s lifetime.

## 3. Exploiting RM Structure in Reinforcement Learning

RMs expose information that can be used for:
- **Reward shaping**: Design of potential-based shaped rewards $R'(u, \ell) = R(u, \ell) + \gamma \Phi(\delta(u, \ell)) - \Phi(u)$, where $\Phi$ encodes value-to-go in the RM state graph computed by value iteration, proven to preserve policy optimality [2010.03950, 2012.14464, 2307.05209].
- **Task decomposition**: Decomposition of a global RM into options, subtasks, or local RMs enables hierarchical RL, parallel learning of constituent skills, counterfactual off-policy learning, and reuse of learned skills across tasks [2010.03950, 2111.09475].
- **Counterfactual reasoning**: For each real experience $(s, u, a, r, s', u')$, synthetic experiences are generated for hypothetical RM-states $\bar{u}$, allowing massive across-state data efficiency [2010.03950].
- **Multi-agent coordination**: Team-level RMs can be decomposed into agent-specific RMs via event-set projections and automata equivalence (bisimulation), permitting decentralized learning with strong value function bounds and guaranteed compositionality [2007.01962].
- **Deep RL integration**: RM-state is concatenated with raw MDP inputs (e.g., as one-hot or semantic embedding), allowing policy networks to disentangle subtask planning from low-level control, dramatically accelerating convergence and improving sample efficiency [2012.14464, 2510.14176]. Automatic and language-grounded RM generation pipelines leverage foundation models to translate natural-language task specifications into executable RMs [2510.14176].

## 4. Extensions: Expressiveness, Robustness, and Multi-Agent Generalizations

- **First-order RMs (FORMs)** encode transitions using full first-order logic (support for existential/universal quantifiers over object-centric domains), greatly compacting representations for relational tasks such as “visit all yellow objects” (previously exponential in objects; collapses to constant size with quantifiers). FORMs support logical ILP-based learning and multi-agent exploitation, where policy decomposition aligns with subtask automata states [2501.00364].
- **Stochastic RMs** assign cumulative distribution functions to transitions, supporting non-deterministic, noise-tolerant rewards and convergence to optimal policies in expectation under guaranteed equivalence [2510.14837].
- **Robust learning from noise**: Probabilistic ILP and belief-updated reward shaping (PROB-IRM) enable RM learning and exploitation even under noisy or inconsistent event labeling, with Bayesian update of automaton state belief and shaping over belief distributions [2408.14871].
- **Pushdown RMs and beyond**: Pushdown reward machines (pdRMs) use a stack to encode context-free temporal relations, supporting reward specifications outside the regular language class, with well-characterized policy complexity bounds and modular exploitation [2508.06894].
- **Hierarchies and coupling**: Hierarchical reward machines (HRMs) and coupled RMs enable the compact representation of extremely long-horizon, compositional, and unordered subtasks, permitting exponential reductions in automaton state space and linear sample efficiency scaling for highly compositional domains [2205.15752, 2403.07005, 2510.27329].

## 5. Applications, Empirical Findings, and Impact

RMs have been demonstrated to produce substantial gains in sample efficiency, transfer learning, and scalability across various domains:
- **Vision-based robotics**: In pick-and-place manipulation, RM signal and automaton-state input together yield $10$–$50\times$ improvements in success rate and policy convergence versus unstructured DQN [2012.14464, 2412.10096].
- **Lifelong and transfer RL**: Modular augmentation and reuse of RMs enables rapid adaptation to new logical-task specifications, outperforming from-scratch RL by exploiting subtask decomposition and shaping [2111.09475, 2307.05209]. Automatic pre-planning over RM graphs accelerates contextual transfer in deep RL, cutting time-to-threshold by $20$–$40\%$ [2307.05209].
- **Cooperative and decentralized multi-agent systems**: RM-based agent reward decomposition results in an order-of-magnitude faster convergence and improved policy scalability in multi-agent rendezvous, buttons, and navigation benchmarks, outperforming centralized and hierarchical Q-learning baselines [2007.01962, 2303.14061, 2403.07005, 2110.00096]. Hierarchical/coupled RMs address tasks that are intractable for flat automata by exploiting subtask synchronization and modular policy learning [2205.15752, 2403.07005, 2510.27329].
- **Automated reward design and language grounding**: Foundation model pipelines (ARM-FM) automatically generate RMs from natural-language objectives and induce language-grounded state embeddings, supporting multi-task/zero-shot generalization across MiniGrid, Craftium, and MetaWorld robotic domains [2510.14176].
- **Plan synthesis and flexibility**: Maximally permissive RMs, synthesizing all partial-order plans, yield strictly higher expected return than single-plan or single-recipe RMs and remove rigidity in plan adherence, although increasing automaton size and planning cost [2408.08059].

## 6. Limitations, Current Research, and Future Directions

Key limitations include the exponential blow-up in RM state size for domains with many unordered subgoals (mitigated by numeric, agenda, and coupled RMs), the challenge of reward specification and proposition extraction in unstructured domains (partially addressed by learning from demonstration and foundation models), and the computational cost in hierarchical or pushdown automata construction. Ongoing research directions, as evidenced in recent work, focus on:
- Extending RM expressivity to first-order (and richer) logics for relational and compositional environments [2501.00364].
- Learning RMs robustly under noisy, partial, or ambiguous observation and integrating with visual perception [2408.14871, 2412.10096].
- Hierarchical and modular automaton composition for complex, sparse-reward, and long-horizon domains [2205.15752, 2403.07005].
- Automated natural language to reward automaton pipelines and language-conditioned skill transfer [2510.14176].
- Pushdown and context-free reward formalism for tasks demanding rich history dependence [2508.06894].
Theoretical interests include formal analysis of sample complexity under various RM generalizations, optimal policy computation over coupled RM products, automated discovery of propositions/events from raw input, and scaling inference in Inductive Logic Programming frameworks.

---

**References:**
- [2010.03950]: "Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning"
- [2111.09475]: "Lifelong Reinforcement Learning with Temporal Logic Formulas and Reward Machines"
- [2501.00364]: "FORM: Learning Expressive and Transferable First-Order Logic Reward Machines"
- [2408.14871]: "Learning Robust Reward Machines from Noisy Labels"
- [2412.10096]: "Reward Machine Inference for Robotic Manipulation"
- [2012.14464]: "Disentangled Planning and Control in Vision Based Robotics via Reward Machines"
- [2307.05209]: "Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning"
- [2205.15752]: "Hierarchies of Reward Machines"
- [2007.01962]: "Reward Machines for Cooperative Multi-Agent Reinforcement Learning"
- [2112.09477]: "Learning Reward Machines: A Study in Partially Observable Reinforcement Learning"
- [2303.14061]: "Learning Reward Machines in Cooperative Multi-Agent Tasks"
- [2510.14837]: "Reinforcement Learning with Stochastic Reward Machines"
- [2510.27329]: "Reinforcement Learning for Long-Horizon Unordered Tasks: From Boolean to Coupled Reward Machines"
- [2510.14176]: "ARM-FM: Automated Reward Machines via Foundation Models for Compositional Reinforcement Learning"
- [2403.07005]: "Multi-Agent Reinforcement Learning with a Hierarchy of Reward Machines"
- [2508.06894]: "Pushdown Reward Machines for Reinforcement Learning"
- [2408.08059]: "Maximally Permissive Reward Machines"

Source: https://www.emergentmind.com/topics/reward-machines-rm