---
title: Fully Learnable Reward Machines (FLNRM)
url: https://www.emergentmind.com/topics/fully-learnable-reward-machines-flnrm
type: topic
---

# Fully Learnable Reward Machines (FLNRM)

Fully Learnable Reward Machines (FLNRM) denotes a family of methods in which the finite-state memory needed to represent a non-Markovian task is inferred from data rather than specified manually. In this literature, the learned object is usually a reward machine (RM), but some formulations also infer labeling functions, event detectors, transition-memory automata, stochastic reward outputs, or fully differentiable symbol-grounding modules. The shared objective is to recover a compact automaton that, when composed with the underlying environment, restores a Markovian decision process on an augmented state and thereby enables standard reinforcement learning, inverse reinforcement learning, or control algorithms to operate effectively [2112.09477; 2204.11833; 2508.01947].

## 1. Conceptual scope and historical development

The FLNRM agenda emerged from a recurring limitation in reward-machine-based reinforcement learning: many methods assume that the automaton, its atomic propositions, or the symbol-grounding map are already known. Early work on learning reward machines from experience showed that an RM can itself be inferred and then used to solve partially observable reinforcement-learning problems, but that formulation still required a user-provided labelling function \(L\) over high-level events [2112.09477]. Subsequent work broadened the notion of learnability by treating the truth values of atomic propositions as uncertain and learning a hypothesis RM online while the agent explores and updates proposition beliefs from noisy sensory measurements [2204.11833].

Later formulations pushed the boundary further toward raw-data settings. In robotic manipulation, RMs were inferred directly from visual demonstrations without predefined propositions or prior knowledge of the underlying sparse reward signals; sub-goals were discovered by clustering visual features, and the RM graph was built from the resulting abstracted demonstrations [2412.10096]. Other work removed access to rewards, labels, and machine nodes entirely, asking what trajectory data is sufficient to recover a policy-equivalent RM from raw state and policy information, and then extending the method with active queries [2604.07480]. A related line inferred RMs from partially observed policies through prefix-tree policies and SAT-based synthesis, recovering the exact RM up to an equivalence class when the observed policy is known to sufficient finite depth [2502.03762].

The scope of FLNRM was extended again by introducing Transition Machines (TMs) and the Dual-Behavior Mealy Machine (DBMM), which generalize reward-only memory to a representation that can capture both reward memory and transition memory in POMDPs [2508.01947]. In parallel, fully differentiable variants learned both the symbol-grounding function and the automaton end-to-end from raw states, while stochastic reward machines addressed the case in which reward observations are noisy rather than deterministic [2509.19017; 2510.14837]. This suggests that FLNRM is best understood not as a single algorithm, but as a research program centered on learning the task-relevant memory structure from data.

## 2. Formal models and the Markovization objective

Across the literature, an RM is a finite-state automaton with an initial state, a deterministic transition function over high-level labels, and a reward-output function. One representative formalization is
\[
\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),
\]
where \(\Sigma = 2^{\mathit{AP}}\), \(\delta: U \times \Sigma \to U\), and \(r: U \times \Sigma \to \mathbb{R}\) [2204.11833]. Other works write
\[
M = (U, u_0, AP, \delta_u, \delta_r)
\]
together with a labeling function \(L:S \to 2^{AP}\), or define \(\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle\) over labels produced by \(L:O_0\times A_0\times O \to 2^P\) [2412.10096; 2112.09477]. Despite notational variation, the role of the automaton is stable: it carries the latent history summary required for the reward to become Markovian in the product state.

This product construction is central to FLNRM. In multiple formulations, the environment state is augmented with the automaton state, producing a product MDP over \(S \times U\) or an augmented observation-state such as \((o,u)\) [2204.11833; 2112.09477]. The learned machine is useful only insofar as this augmentation restores the information needed for optimal control. In policy-based identification settings, the target is sometimes weaker than exact automaton recovery: the goal may be a machine that is *policy-equivalent*, meaning that its induced history policy matches the observable behavior of the true product policy [2502.03762; 2604.07480].

The DBMM formalism generalizes this perspective. A DBMM is
\[
DBMM = (V, v_0, I_\alpha, I_\beta, O, T, G),
\]
where \(I_\beta\) contains “beta-inputs” that trigger state changes without outputs, and \(I_\alpha\) contains “alpha-inputs” that trigger outputs without state changes [2508.01947]. In that framework, an RM becomes a DBMM with \(I_\beta = 2^{AP}\), \(I_\alpha = O \times A\), and output space \(\mathbb{R}\), while a TM uses the same input structure but outputs the next observation. This unification matters because it treats reward memory and transition memory as two instances of the same passive automata-learning problem. A plausible implication is that FLNRM in partially observable environments may require learning more than reward structure alone.

## 3. Inference regimes and algorithmic families

Representative FLNRM methods differ primarily in the data they assume and the inference mechanism they use.

| Setting | Observed data | Core learning mechanism |
|---|---|---|
| Experience-based RM learning [2112.09477] | high-level event traces | discrete optimization via MILP, CP, LS, TS |
| Noisy-semantics joint learning [2204.11833] | rewards and noisy proposition measurements | Bayes updates plus counterexample-driven SAT inference |
| Hidden-trigger discovery [2401.11325] | \((s,a,r,s')\) trajectories | ILP over hidden-state assignments |
| Visual RM inference [2412.10096] | raw video demonstrations | pre-trained ResNet-50, DBSCAN, single-pass graph construction |
| Partially observed policy synthesis [2502.03762] | depth-\(l\) prefix-tree policy | SAT plus MaxEnt IRL |
| Raw-trajectory recovery [2604.07480] | raw state and policy information | passive SAT and active query extension |
| POMDP DBMM inference [2508.01947] | interleaved \(\beta\)-inputs, \(\alpha\)-inputs, outputs | DB-RPNI state merging |
| Fully differentiable FLNRM [2509.19017] | raw states and rewards | symbol-grounding network, transition tensor, A2C |
| Noisy-reward structure learning [2510.14837] | label/reward traces | SMT for minimal stochastic reward machines |

One algorithmic family formulates RM inference as exact or approximate discrete optimization. In partially observable RL, the learning problem was posed as minimizing the total log-cardinality of next-event prediction sets, with solvers based on mixed-integer linear programming, constraint programming, local search with restarts, and tabu search [2112.09477]. A related hidden-trigger formulation uses an integer linear program whose variables assign each observed transition to a hidden-state pair \((i,j)\), enforcing determinism in both the transition function and emitted rewards [2401.11325]. SAT-based synthesis appears in several works as well: from counterexample traces in online learning with noisy semantics, from prefix-tree policies in inverse-RL-style settings, and from raw state trajectories via negative examples extracted from differences in action distributions [2204.11833; 2502.03762; 2604.07480].

A second family emphasizes passive automata learning and state merging. In the DBMM setting, DB-RPNI builds a Prefix-Tree Transducer from interleaved \(\beta\)-inputs and \(\alpha\)-inputs, then performs red-blue state merging subject to local compatibility and transitive consistency conditions [2508.01947]. Its preprocessing removes redundant \(\alpha\)-inputs whose outputs are constant across all states and trivial \(\beta\)-inputs known a priori to be self-loops. Under structure completeness, DB-RPNI is guaranteed to return the unique minimal DBMM consistent with the data; the paper also states the running time \(O(|U|\cdot|L|\cdot T\cdot F)\), with worst-case \(O(T^3\cdot F)\) when structure completeness is absent [2508.01947].

A third family learns the symbolic interface itself. In robotic manipulation, visual features \(\phi(\cdot)\) are extracted by a pre-trained ResNet-50, clustered by DBSCAN into sub-goals, converted into proposition detectors by distance thresholding, and then assembled into an RM by a single pass through the abstracted demonstrations [2412.10096]. In fully differentiable neural FLNRM, the symbol-grounding map \(sg_{\theta_{sg}}\), transition tensor \(\mathcal T\), and reward-emission matrix \(\mathcal R\) are optimized jointly with actor-critic learning; the latent automaton state is represented probabilistically as \(\tilde q^{(t)}\in\Delta(\hat Q)\) [2509.19017].

## 4. Integration with reinforcement learning, inverse reinforcement learning, and control

Once an RM or related automaton has been inferred, FLNRM methods typically hand off control to standard learning algorithms on the augmented state space. In the experience-based POMDP formulation, a learned RM \(R=\langle U,u_0,\delta_u,\delta_r\rangle\) induces an augmented state \((o,u)\), and learning can proceed either with a single \(Q(o,u,a)\) network or with Q-learning for reward machines (QRM), which maintains a separate \(Q_u(o,a)\) for each automaton state [2112.09477]. The appeal of this decomposition is off-policy replay across RM states, although the same work notes that experience sharing in partial observability is approximate and can introduce bias if the learned prediction structure is imperfect [2112.09477].

In the noisy-semantics setting, the agent alternates between exploration, Bayesian updates of proposition beliefs, and inference of a hypothesis reward machine from counterexamples [2204.11833]. Q-learning is then performed on the product MDP \(S\times U\), where the RM state evolves according to the current estimated labeling \(\hat L_b\). The paper proves that if the belief converges in finitely many episodes and episode length exceeds \(2^{|S|+1}(|U_{\rm true}|+1)-1\), then the algorithm almost surely recovers the true RM and classical Q-learning converges almost surely to the optimal policy [2204.11833].

For robotic manipulation, the inferred RM is integrated through DQRM: a separate Q-network \(Q_u(s,a)\) is trained for each RM state, experience is stored in state-specific replay buffers, and the reward is shaped by a potential \(\Psi(u)=\gamma^{d_{\text{goal}}(u)}\) so that
\[
\delta_r(u,u')=\delta_r'(u,u')+\gamma\Psi(u')-\Psi(u).
\]
The Q-functions are implemented with two fully-convolutional heads, \(\psi_{\text{pick}}\) and \(\psi_{\text{place}}\), on top of a shared ResNet-50 trunk, and training minimizes the standard TD error with Huber loss [2412.10096].

In POMDPs with separately inferred reward and transition memory, the augmented state becomes \((o,u,q)\), and standard Q-learning on that state was reported to converge to the optimal policy in approximately \(1{,}500\) episodes in a \(25\times25\) environment [2508.01947]. In inverse-reinforcement-learning settings, the pipeline is reversed: SAT is used to recover a transition structure consistent with a depth-\(l\) prefix-tree policy, and a standard entropy-regularized MaxEnt IRL program is then solved on the product MDP to recover \(\delta_r\) [2502.03762]. Fully differentiable FLNRM instead augments the policy input with the soft automaton state \((s^{(t)},\tilde q^{(t)})\) and minimizes
\[
L_{\rm RL}(\phi)+\lambda L_{\rm RM}(\theta_{\rm FLN}),
\]
combining A2C with cross-entropy on reward-machine predictions [2509.19017].

## 5. Empirical profile

Empirical evaluation spans grid-world POMDPs, robotic manipulation, Officeworld-style domains, Breakfastworld, and LTL\(_f\)-derived navigation tasks.

| Study | Setting | Main reported outcome |
|---|---|---|
| [2112.09477] | cookie, symbol, 2-keys POMDPs | FLNRM methods converge to optimal or near-optimal reward; A3C, ACER, PPO, DDQN fail beyond random performance |
| [2412.10096] | five robosuite block-manipulation tasks | inferred RM matches or nearly matches ground-truth RM; Place-3 slightly outperforms; convergence in a few hundred episodes |
| [2508.01947] | \(3\times3\) to \(25\times25\) grid POMDPs | up to three orders of magnitude faster than HMM and ILP baselines; downstream Q-learning converges in \(\approx 1{,}500\) episodes |
| [2401.11325] | Officeworld and Breakfastworld | optimal average return in all four Officeworld tasks; solve times from \(0.1\) s to \(\approx 12{,}000\) s |
| [2604.07480] | two \(4\times4\) stochastic grid-worlds | active querying converges \(2\times\) faster, uses \(100\times\) less memory, runtime \(3.5\) ks vs. \(7.2\) ks on Task B |
| [2509.19017] | Map and Image environments, 8 LTL\(_f\) tasks | FLNRM consistently outperforms LSTM-based A2C; Task 5 reaches \(90\%\) of maximum return in \(\approx 10\)K vs. \(\approx 25\)K steps |
| [2510.14837] | Mining and Harvest | SRMI outperforms existing methods and a naive approach |

Some results are especially informative about computational structure. In DBMM inference, the reported runtime on \(3\times3\) grids was \(1.3\) s for DB-RPNI versus \(104\) s for an HMM baseline and \(5.3\) s for an ILP baseline; on \(4\times4\), \(3.9\) s versus \(>6{,}000\) s and \(5{,}500\) s; on \(5\times5\), DB-RPNI required \(56.1\) s while both baselines timed out after more than \(3\) h [2508.01947]. In the \(25\times25\) ablation, a full pipeline with a ground-truth TM of \(7\) states and RM of \(3\) states recovered \(TM=7, RM=2\) in approximately \(400\) s on \(1{,}000\) traces and approximately \(4{,}200\) s on \(10{,}000\) traces. Omitting observation-supplement increased RM size to \(218\) or \(15\) states and raised runtime to \(26{,}000\)–\(65{,}000\) s, while dropping \(\alpha\)-input or \(\beta\)-input removal caused \(7\times\)–\(15\times\) slowdowns or failure [2508.01947].

In robotic manipulation, only \(1\)–\(6\) demonstrations per task were used across Stack-2, Place-2, Pyramid-3, Stack-3, and Place-3, and agents using the inferred RM matched or nearly matched the ground-truth-RM baseline in reward and final placement error [2412.10096]. In the raw-trajectory active-learning setting, passive SAT at depth \(l=9\) recovered the ground-truth RM up to renaming in both tasks, while active querying with burn-in depth \(l_0=3\) or \(6\), budget \(B=250\), and \(N_a\in\{50,100,200\}\) reduced the hypothesis set to one by depth \(12\)–\(13\) [2604.07480]. These results suggest that FLNRM performance depends not only on the eventual expressiveness of the automaton, but also on the efficiency with which informative distinctions between histories are exposed.

## 6. Identifiability, misconceptions, and research frontiers

A recurrent theoretical theme is that automaton recovery depends on finite-depth or finite-coverage conditions. In partially observed policies, the sufficient depth for identifiability is \(l \ge |S|\cdot N^2\), where \(N\) is an upper bound on the number of RM nodes; beyond that depth, deeper negative examples are redundant for policy-equivalent recovery [2502.03762]. In raw-trajectory recovery, the sufficient and necessary depth is stated as \(l^* = |S|\cdot u_{\max}^2\), and with \(h^{l^*}\) and \(u_{\max}\) the negative-example constraints determine \(\delta\) and \(L\) up to label and node renaming among policy-equivalent RMs [2604.07480]. In DBMM inference, minimality is guaranteed under structure completeness, and in practice the paper states that it suffices that any two truly distinct states exhibit at least one \(\alpha\)-input on which they differ [2508.01947]. These results make clear that “fully learnable” does not imply unconstrained identifiability.

A second recurring misconception is that FLNRM always refers to a single end-to-end differentiable objective. The robotic-manipulation pipeline explicitly does *not* optimize all components via a single likelihood or EM objective; instead it proceeds in four modular stages: pre-trained visual feature extraction, DBSCAN clustering, proposition definition by thresholding, and single-pass automaton construction [2412.10096]. By contrast, neural FLNRM does optimize a joint objective over actor-critic loss and reward-machine prediction loss, but even there the latent alphabet \(\hat P\) and latent automaton size \(\hat Q\) are fixed and user-chosen [2509.19017]. A plausible implication is that the term “fully learnable” is best interpreted relative to which components are no longer hand-engineered, not as a guarantee of a uniform training paradigm.

A third issue is robustness to uncertainty and noise. In partially known semantics, proposition truth values are maintained as posterior probabilities \(\hat b_t(s,p)\) and thresholded at \(0.5\) to define \(\hat L_t(s)\), allowing the automaton to be inferred despite noisy sensors [2204.11833]. For noisy rewards, stochastic reward machines generalize deterministic outputs to transition-indexed reward distributions, and SRMI learns a minimal \(\epsilon_c\)-consistent SRM using SMT constraints over transition structure and output means [2510.14837]. The resulting guarantees are asymptotic and depend on assumptions such as bounded symmetric noise, exploration, and a non-containment condition on reward distributions [2510.14837].

Open directions are stated explicitly in several works. Proposed extensions include stochastic POMDPs and probabilistic DBMMs, active queries, neural parametrization of \(G\) and \(T\), online continual learning via incremental RPNI variants such as IDS, and integration with hierarchical RL through option discovery [2508.01947]. Other papers propose differentiable relaxations of SAT-style synthesis, transfer and composition of learned RMs, and extension beyond tabular MDPs and exact history policies [2502.03762; 2604.07480]. The cumulative picture is that FLNRM has moved from learning automata over engineered labels toward learning symbolic memory directly from raw, noisy, or partially observed data, while leaving open the hard problems of scalability, continuous observation spaces, and precise identifiability under limited coverage.

Source: https://www.emergentmind.com/topics/fully-learnable-reward-machines-flnrm