Symbolic Reward Machines (SRMs)

• Symbolic Reward Machines (SRMs) are automata-based reward representations that use symbolic states, guards, and predicates to convert non-Markovian rewards into Markovian ones.
• They exploit task structure through methods like cross-product state spaces and counterfactual experience to significantly improve learning efficiency and policy optimization.
• Extensions include learning from raw observations, automated reward shaping, and transfer capabilities, making SRMs effective for long-horizon, sparse reward environments.

Symbolic Reward Machines (SRMs) are automata-based reward representations for reinforcement learning that make temporally extended task structure explicit through symbolic states, guards, predicates, or symbolic reward terms. They belong to the broader reward-machine family, in which a finite-state machine tracks task progress and emits intermediate rewards, thereby converting a non-Markovian reward problem into a Markovian one on an augmented state space. In current usage, the term SRM has been used in two closely related ways: as reward machines with predicate-guarded transitions and symbolic reward outputs, especially for inverse reinforcement learning, and as reward machines that consume raw observations directly through symbolic guards instead of relying on an external labeling function (Icarte et al., 2020, Zhou et al., 2022, Krug et al., 3 Mar 2026).

1. Conceptual lineage and terminology

The immediate precursor of SRMs is the standard reward machine (RM), introduced as a finite-state symbolic process over high-level propositional events. In that formulation, a reward function is no longer treated as a black-box scalar map; instead, it is represented by a machine whose states encode “what portion of the task remains,” whose transitions are driven by labels over propositions, and whose outputs encode temporally extended reward structure. This makes it possible to express loops, sequences, conditionals, sequence interleaving, some safety constraints, and temporally extended properties typical of linear temporal logic, provided the reward dependence is regular-language expressible (Icarte et al., 2020, Zheng et al., 2021).

The phrase Symbolic Reward Machines was then specialized in two directions. One direction extends RMs by allowing transitions to carry predicates over trajectories and symbolic reward outputs rather than fixed numeric constants. In that sense, SRMs are structured, interpretable reward functions in which reward magnitudes may remain symbolic “holes” to be inferred from demonstrations, and reward computation may depend on history rather than only on atomic propositions (Zhou et al., 2022). The other direction removes the need for an externally supplied labeling function by letting the machine process the observation directly through symbolic formulas over the environment state. In that formulation, SRMs consume only the standard output of the environment and use guards represented by symbolic formulas, with the explicit goal of adhering to the widely used environment definition while retaining interpretability (Krug et al., 3 Mar 2026).

This terminological split is substantive rather than merely notational. In the first usage, “symbolic” refers to predicate-guarded transitions and symbolic reward terms over trajectories. In the second, it refers to symbolic guards over raw observations that replace task-specific labels. A plausible implication is that SRMs are best understood as a family of automata-based reward representations rather than a single fixed formalism.

2. Formal models and semantics

A standard RM can be defined as a finite automaton over symbolic events. One formulation writes a reward machine as

$\langle U, I, \Sigma, \delta, R, F, L\rangle,$

where $U$ is the finite set of RM states, $I$ is the initial RM state, $\Sigma$ is a finite alphabet of event symbols, $\delta$ is the deterministic transition function, $R$ is the RM reward function, $F$ is the set of terminal RM states, and $L$ is the labeling function mapping environment transitions to symbolic events. The machine updates by

$u_{t+1}=\delta(u_t,e),$

and training proceeds on the cross-product $\mathcal{S}\times\mathcal{U}$ (Castanyer et al., 16 Oct 2025). The earlier RM formalism likewise emphasizes the product construction $U$0, noting that rewards may be non-Markovian relative to the environment state space $U$1 but are Markovian relative to the augmented state (Icarte et al., 2020).

The 2022 SRM formalism generalizes this setup to trajectory predicates and symbolic reward terms. It defines an SRM as

$U$2

where $U$3 is a set of predicates on trajectories, $U$4 is a set of $U$5-terms of type $U$6, and $U$7 consists of transition rules $U$8. A transition has the form

$U$9

is applicable when $I$0, and emits reward $I$1. If no rule applies, the SRM stays in place and emits a default reward such as $I$2 through a dummy transition. Determinism is defined by requiring that, for any state $I$3 and input trajectory $I$4, at most one outgoing guard is true (Zhou et al., 2022).

The 2026 SRM formalism instead defines a machine

$I$5

where $I$6 is a logic with formulas $I$7 and input space $I$8, $I$9 is a finite set of SRM states, $\Sigma$0 is a transition relation, and $\Sigma$1 is a partial output function. A transition is written

$\Sigma$2

and is taken whenever $\Sigma$3. Determinism requires that no two different outgoing transitions from the same SRM state are enabled simultaneously, and completeness requires that for every SRM state and every input there exists some enabled transition. The resulting run semantics map a trajectory of raw observations directly to an SRM-state sequence and reward sequence, without an external labeling function (Krug et al., 3 Mar 2026).

3. Exploiting symbolic structure for learning

The core algorithmic benefit of SRM/RM representations is that they expose reward-function structure to the agent. The foundational RM paper develops three major mechanisms for exploiting this structure: learning on the cross-product state space, counterfactual experience augmentation, and automated potential-based reward shaping. Counterfactual Experiences for Reward Machines (CRM) reuse a single environment transition as a valid training sample for every RM state, and the paper proves that CRM with tabular Q-learning converges to an optimal policy in the limit, assuming every state-action pair is visited infinitely often. Hierarchical Reinforcement Learning for Reward Machines (HRM) extracts options from RM edges, though it may converge to suboptimal policies because it is myopic (Icarte et al., 2020).

Q-learning with SRMs inherits this structure-aware perspective. QSRM maintains one Q-table per SRM state and performs multi-updates over all SRM states, not only the current one. The 2026 paper proves that QSRM converges to an optimal policy under the same conditions as standard Q-learning: bounded rewards, infinite visitation of all state-action pairs, and learning rates satisfying the standard Robbins–Monro conditions. The proof proceeds by constructing the cross-product MDP with state space $\Sigma$4, showing that QSRM is equivalent to Q-learning on that MDP except for parallel multi-updates (Krug et al., 3 Mar 2026).

The same structural principle underlies several transfer-oriented RM methods. In lifelong reinforcement learning with SLTL and RMs, the memory RM grows by progression and decomposition of temporal-logic formulas, and previously learned knowledge is reused by Q-function composition. RM states are formulas, transitions for new states are generated automatically by progression, and the agent selects the “smallest representation” of a new task by minimizing the number of subtasks not yet learned (Zheng et al., 2021). Contextual PRE-Planning on Reward Machine Abstractions goes further by planning inside the RM abstraction, selecting the optimal next abstract transition label

$\Sigma$5

and feeding that symbolic target to the agent as context. The method uses potential-based reward shaping derived from the RM abstraction and improves sample efficiency and few-shot transfer in a variety of domains (Azran et al., 2023). RM-guided self-paced reinforcement learning uses reward machines both in the product MDP for policy learning and in the curriculum update through a reward-machine-context mapping $\Sigma$6, yielding task-structure-aware curricula for long-horizon contextual RL (Koprulu et al., 2023).

4. Specification, concretization, and machine inference

A major SRM research direction concerns how the machine itself is obtained. In the hierarchical Bayesian inverse reinforcement learning approach, the SRM skeleton is designed by a human at a high level, but the symbolic holes in guards and rewards are concretized from demonstrations. The target is a posterior over valid SRM concretizations, approximated by a variational distribution $\Sigma$7 and optimized through an ELBO-style objective. Because SRMs are non-differentiable with respect to symbolic holes, the method uses a sampler over hole assignments, a neural reward discriminator, importance sampling, reparameterization tricks, and a differentiable constraint penalty $\Sigma$8 to enforce symbolic constraints (Zhou et al., 2022).

The 2026 LSRM framework instead learns the SRM structure from counterexamples. It starts from a trivial SRM hypothesis, learns a policy with QSRM or DQSRM, compares environment reward with hypothesis reward, and, when inconsistent, adds the trajectory to a counterexample set and infers a new SRM consistent with all counterexamples. Two variants are given. LSRM-GF assumes a user-provided set of formulas $\Sigma$9 and encodes determinism, state progression, transition consistency, and used-transition constraints in SMT. LSRM-FT replaces fixed formulas by formula templates such as a box template over Linear Real Arithmetic and learns template parameters together with the transition structure (Krug et al., 3 Mar 2026).

Automated specification from natural language appears in ARM-FM, which uses foundation models to generate complete reward-machine specifications from natural-language task descriptions. The pipeline instructs the model to output a concise reward machine in plaintext, Boolean-predicate events derived from environment state, compact transition rules with else self-loops, nonzero rewards only on progress transitions, and separate natural-language instructions and executable labeling functions. The generation process uses iterative self-improvement with paired generator and critic foundation models over $\delta$0 rounds, producing language-aligned reward machines (LARMs) in which each RM state is paired with a natural-language description $\delta$1 and embedding $\delta$2 (Castanyer et al., 16 Oct 2025).

In partially observable settings, inference can target not only reward machines but also complementary transition memory. The DBMM framework introduces Transition Machines (TMs) alongside RMs, unifying both as Dual Behavior Mealy Machines and learning them passively from traces with DB-RPNI. The paper argues that prior automaton representations focus only on reward-based non-Markovianity, whereas in Det-POMDPs some hidden dependence is transition-related rather than reward-related. Experimentally, the method reports speedups of up to three orders of magnitude over state-of-the-art baselines (Wu et al., 3 Aug 2025).

5. Grounding, raw observations, and neurosymbolic variants

A long-standing assumption in RM-style RL is the existence of a correct labeling or symbol-grounding function. Several SRM-adjacent lines of work relax this assumption in different ways. Neural Reward Machines keep the symbolic temporal task specification but replace the fixed grounding function with a probabilistic, learnable symbol grounder. An NRM is defined as

$\delta$3

with probabilistic state updates

$\delta$4

The paper reports that NRM consistently performs much better than RNN baselines and close to the RM upper bound, while also advancing semi-supervised symbol grounding through a groundability analysis that is more efficient than baseline techniques of a factor $\delta$5 (Umili et al., 2024).

Fully Learnable Neural Reward Machines remove even the assumption that the automaton structure is known. FLNRM learns the symbol grounding function, the automaton transition structure, and the reward output structure end-to-end from experience, using temperature-scaled softmax relaxations for both the grounding and automaton tensors. The learned automaton state augments the policy input, turning a non-Markovian task into something closer to a Markovian one for the policy/value learner. The paper reports that FLNRM outperforms previous approaches based on Recurrent Neural Networks, while acknowledging the limitation that the ground-truth number of symbols is still assumed to be known (Dewidar et al., 23 Sep 2025).

Noisy symbolic abstractions provide a different perspective on imperfect grounding. In that setting, the RM state becomes hidden and uncertain because the abstraction from environment state to propositions is noisy. The central claim is that it is better to predict the hidden RM state directly than to infer individual symbols and then reconstruct the RM state. Reward Machine State Modelling (RMSM) therefore learns a belief over RM states from history and conditions the policy on that belief. In MiniGrid deep RL benchmarks, RMSM learns quickly and reaches performance close to Perfect RM, while thresholding and independent belief updating fail in at least one domain (Li et al., 2022).

The explicit 2026 SRM formulation can be read as a symbolic alternative to this neurosymbolic trend: instead of learning a symbolizer, it eliminates the labeling-function requirement by placing symbolic guards directly over raw observations. This suggests a spectrum of approaches, from manually authored symbolic guards, through uncertain or learned grounding, to fully learnable automata (Krug et al., 3 Mar 2026).

6. Transfer, expressivity, empirical status, and limitations

SRM and RM methods are especially prominent in sparse-reward, long-horizon, and compositional settings. ARM-FM evaluates on MiniGrid/BabyAI, Craftium, Meta-World, and XLand-MiniGrid. On DoorKey of increasing size, ARM-FM consistently achieves higher return than all baselines, both on fixed and procedurally generated layouts; on UnlockToUnlock, BlockedUnlockPickup, and KeyCorridor it is the only method that reliably solves the tasks and reaches near-perfect reward; and in XLand-MiniGrid a single Rainbow DQN agent trained on up to 10 simultaneous tasks maintains high success only when both dense LARM rewards and state embeddings are present. The paper also reports zero-shot generalization to a third unseen composite task when its subgoals are semantically familiar (Castanyer et al., 16 Oct 2025).

Comparable structure-exploitation results appear across the broader RM literature. The hierarchical Bayesian SRM-IRL paper reports much faster learning, higher average return, better sample efficiency, and transfer across larger or differently configured versions of DoorKey, KeyCorridor, and ObstructedMaze, in some settings using as few as one demonstration (Zhou et al., 2022). The explicit SRM paper reports that QRM and QSRM show identical behavior, that QRM and QSRM converge to the optimal mean10 score of $\delta$6 in finite tasks, and that LSRM-GF and LSRM-FT also converge to the optimal mean10 score in finite tasks, while deep variants in infinite environments achieve good but not optimal performance (Krug et al., 3 Mar 2026). RM-guided SPRL reduces curriculum-length by roughly one-fourth and lowers curriculum variance by up to four orders of magnitude, and in Customized HalfCheetah it is the only method that succeeds in every independent run (Koprulu et al., 2023).

A major limitation of finite-state reward machines, and therefore of many SRM-style formulations, is expressivity. Standard RMs are limited to regular languages. Counting Reward Automata address this by augmenting the automaton with counters, yielding a machine capable of modelling any reward function expressible as a formal language. The paper proves that any RM can be emulated by a CRA with the same number of states and transitions, while CRAs can also represent count-dependent tasks beyond regular languages. It further reports that the required state machines can be specified from natural language task descriptions using ChatGPT (Bester et al., 2023).

Other limitations recur across the literature. ARM-FM states that the generation pipeline currently may require human verification, that smaller foundation models are less reliable, and that zero-shot transfer depends on the new task being composed of semantically familiar subtasks (Castanyer et al., 16 Oct 2025). The foundational RM paper notes that RMs cannot express reward dependencies beyond regular languages without external memory, that CRM and HRM increase computational cost, and that learning the RM and the labeling function jointly is still open (Icarte et al., 2020). DBMM-based inference from POMDP traces argues that one should not force all hidden history dependence into the reward machine, since some non-Markovianity is transition-based rather than reward-based (Wu et al., 3 Aug 2025).

Taken together, these lines of work position SRMs as a broad research program in which reward specification is treated as a compact symbolic dynamical system rather than a scalar oracle. The central invariant across variants is the same: task progress is represented explicitly by automaton state, and that structure is then exploited for shaping, decomposition, counterfactual reuse, transfer, curriculum generation, interpretability, or automated specification.