Fully Learnable Reward Machines (FLNRM)

• FLNRM is a framework that infers finite-state memory structures from data, transforming non-Markovian tasks into augmented Markovian decision processes.
• It utilizes diverse techniques—including MILP, SAT-based synthesis, state merging, and differentiable neural modules—to recover compact automata from various noisy and raw inputs.
• FLNRM integrates with reinforcement, inverse reinforcement, and control algorithms by augmenting state spaces, thereby enabling convergence to optimal policies across different environments.

Fully Learnable Reward Machines (FLNRM) denotes a family of methods in which the finite-state memory needed to represent a non-Markovian task is inferred from data rather than specified manually. In this literature, the learned object is usually a reward machine (RM), but some formulations also infer labeling functions, event detectors, transition-memory automata, stochastic reward outputs, or fully differentiable symbol-grounding modules. The shared objective is to recover a compact automaton that, when composed with the underlying environment, restores a Markovian decision process on an augmented state and thereby enables standard reinforcement learning, inverse reinforcement learning, or control algorithms to operate effectively (Icarte et al., 2021, Verginis et al., 2022, Wu et al., 3 Aug 2025).

1. Conceptual scope and historical development

The FLNRM agenda emerged from a recurring limitation in reward-machine-based reinforcement learning: many methods assume that the automaton, its atomic propositions, or the symbol-grounding map are already known. Early work on learning reward machines from experience showed that an RM can itself be inferred and then used to solve partially observable reinforcement-learning problems, but that formulation still required a user-provided labelling function $L$ over high-level events (Icarte et al., 2021). Subsequent work broadened the notion of learnability by treating the truth values of atomic propositions as uncertain and learning a hypothesis RM online while the agent explores and updates proposition beliefs from noisy sensory measurements (Verginis et al., 2022).

Later formulations pushed the boundary further toward raw-data settings. In robotic manipulation, RMs were inferred directly from visual demonstrations without predefined propositions or prior knowledge of the underlying sparse reward signals; sub-goals were discovered by clustering visual features, and the RM graph was built from the resulting abstracted demonstrations (Baert et al., 2024). Other work removed access to rewards, labels, and machine nodes entirely, asking what trajectory data is sufficient to recover a policy-equivalent RM from raw state and policy information, and then extending the method with active queries (Shehab et al., 8 Apr 2026). A related line inferred RMs from partially observed policies through prefix-tree policies and SAT-based synthesis, recovering the exact RM up to an equivalence class when the observed policy is known to sufficient finite depth (Shehab et al., 6 Feb 2025).

The scope of FLNRM was extended again by introducing Transition Machines (TMs) and the Dual-Behavior Mealy Machine (DBMM), which generalize reward-only memory to a representation that can capture both reward memory and transition memory in POMDPs (Wu et al., 3 Aug 2025). In parallel, fully differentiable variants learned both the symbol-grounding function and the automaton end-to-end from raw states, while stochastic reward machines addressed the case in which reward observations are noisy rather than deterministic (Dewidar et al., 23 Sep 2025, Corazza et al., 16 Oct 2025). This suggests that FLNRM is best understood not as a single algorithm, but as a research program centered on learning the task-relevant memory structure from data.

2. Formal models and the Markovization objective

Across the literature, an RM is a finite-state automaton with an initial state, a deterministic transition function over high-level labels, and a reward-output function. One representative formalization is

$$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$

where $\Sigma = 2^{\mathit{AP}}$, $\delta: U \times \Sigma \to U$, and $r: U \times \Sigma \to \mathbb{R}$ (Verginis et al., 2022). Other works write

$M = (U, u_0, AP, \delta_u, \delta_r)$

together with a labeling function $L:S \to 2^{AP}$, or define $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$ over labels produced by $L:O_0\times A_0\times O \to 2^P$ (Baert et al., 2024, Icarte et al., 2021). Despite notational variation, the role of the automaton is stable: it carries the latent history summary required for the reward to become Markovian in the product state.

This product construction is central to FLNRM. In multiple formulations, the environment state is augmented with the automaton state, producing a product MDP over $S \times U$ or an augmented observation-state such as $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$0 (Verginis et al., 2022, Icarte et al., 2021). The learned machine is useful only insofar as this augmentation restores the information needed for optimal control. In policy-based identification settings, the target is sometimes weaker than exact automaton recovery: the goal may be a machine that is policy-equivalent, meaning that its induced history policy matches the observable behavior of the true product policy (Shehab et al., 6 Feb 2025, Shehab et al., 8 Apr 2026).

The DBMM formalism generalizes this perspective. A DBMM is

$$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$1

where $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$2 contains “beta-inputs” that trigger state changes without outputs, and $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$3 contains “alpha-inputs” that trigger outputs without state changes (Wu et al., 3 Aug 2025). In that framework, an RM becomes a DBMM with $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$4, $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$5, and output space $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$6, while a TM uses the same input structure but outputs the next observation. This unification matters because it treats reward memory and transition memory as two instances of the same passive automata-learning problem. A plausible implication is that FLNRM in partially observable environments may require learning more than reward structure alone.

3. Inference regimes and algorithmic families

Representative FLNRM methods differ primarily in the data they assume and the inference mechanism they use.

Setting	Observed data	Core learning mechanism
Experience-based RM learning (Icarte et al., 2021)	high-level event traces	discrete optimization via MILP, CP, LS, TS
Noisy-semantics joint learning (Verginis et al., 2022)	rewards and noisy proposition measurements	Bayes updates plus counterexample-driven SAT inference
Hidden-trigger discovery (Hyde et al., 2024)	$$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$7 trajectories	ILP over hidden-state assignments
Visual RM inference (Baert et al., 2024)	raw video demonstrations	pre-trained ResNet-50, DBSCAN, single-pass graph construction
Partially observed policy synthesis (Shehab et al., 6 Feb 2025)	depth-$$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$8 prefix-tree policy	SAT plus MaxEnt IRL
Raw-trajectory recovery (Shehab et al., 8 Apr 2026)	raw state and policy information	passive SAT and active query extension
POMDP DBMM inference (Wu et al., 3 Aug 2025)	interleaved $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$9-inputs, $\Sigma = 2^{\mathit{AP}}$0-inputs, outputs	DB-RPNI state merging
Fully differentiable FLNRM (Dewidar et al., 23 Sep 2025)	raw states and rewards	symbol-grounding network, transition tensor, A2C
Noisy-reward structure learning (Corazza et al., 16 Oct 2025)	label/reward traces	SMT for minimal stochastic reward machines

One algorithmic family formulates RM inference as exact or approximate discrete optimization. In partially observable RL, the learning problem was posed as minimizing the total log-cardinality of next-event prediction sets, with solvers based on mixed-integer linear programming, constraint programming, local search with restarts, and tabu search (Icarte et al., 2021). A related hidden-trigger formulation uses an integer linear program whose variables assign each observed transition to a hidden-state pair $\Sigma = 2^{\mathit{AP}}$1, enforcing determinism in both the transition function and emitted rewards (Hyde et al., 2024). SAT-based synthesis appears in several works as well: from counterexample traces in online learning with noisy semantics, from prefix-tree policies in inverse-RL-style settings, and from raw state trajectories via negative examples extracted from differences in action distributions (Verginis et al., 2022, Shehab et al., 6 Feb 2025, Shehab et al., 8 Apr 2026).

A second family emphasizes passive automata learning and state merging. In the DBMM setting, DB-RPNI builds a Prefix-Tree Transducer from interleaved $\Sigma = 2^{\mathit{AP}}$2-inputs and $\Sigma = 2^{\mathit{AP}}$3-inputs, then performs red-blue state merging subject to local compatibility and transitive consistency conditions (Wu et al., 3 Aug 2025). Its preprocessing removes redundant $\Sigma = 2^{\mathit{AP}}$4-inputs whose outputs are constant across all states and trivial $\Sigma = 2^{\mathit{AP}}$5-inputs known a priori to be self-loops. Under structure completeness, DB-RPNI is guaranteed to return the unique minimal DBMM consistent with the data; the paper also states the running time $\Sigma = 2^{\mathit{AP}}$6, with worst-case $\Sigma = 2^{\mathit{AP}}$7 when structure completeness is absent (Wu et al., 3 Aug 2025).

A third family learns the symbolic interface itself. In robotic manipulation, visual features $\Sigma = 2^{\mathit{AP}}$8 are extracted by a pre-trained ResNet-50, clustered by DBSCAN into sub-goals, converted into proposition detectors by distance thresholding, and then assembled into an RM by a single pass through the abstracted demonstrations (Baert et al., 2024). In fully differentiable neural FLNRM, the symbol-grounding map $\Sigma = 2^{\mathit{AP}}$9, transition tensor $\delta: U \times \Sigma \to U$0, and reward-emission matrix $\delta: U \times \Sigma \to U$1 are optimized jointly with actor-critic learning; the latent automaton state is represented probabilistically as $\delta: U \times \Sigma \to U$2 (Dewidar et al., 23 Sep 2025).

4. Integration with reinforcement learning, inverse reinforcement learning, and control

Once an RM or related automaton has been inferred, FLNRM methods typically hand off control to standard learning algorithms on the augmented state space. In the experience-based POMDP formulation, a learned RM $\delta: U \times \Sigma \to U$3 induces an augmented state $\delta: U \times \Sigma \to U$4, and learning can proceed either with a single $\delta: U \times \Sigma \to U$5 network or with Q-learning for reward machines (QRM), which maintains a separate $\delta: U \times \Sigma \to U$6 for each automaton state (Icarte et al., 2021). The appeal of this decomposition is off-policy replay across RM states, although the same work notes that experience sharing in partial observability is approximate and can introduce bias if the learned prediction structure is imperfect (Icarte et al., 2021).

In the noisy-semantics setting, the agent alternates between exploration, Bayesian updates of proposition beliefs, and inference of a hypothesis reward machine from counterexamples (Verginis et al., 2022). Q-learning is then performed on the product MDP $\delta: U \times \Sigma \to U$7, where the RM state evolves according to the current estimated labeling $\delta: U \times \Sigma \to U$8. The paper proves that if the belief converges in finitely many episodes and episode length exceeds $\delta: U \times \Sigma \to U$9, then the algorithm almost surely recovers the true RM and classical Q-learning converges almost surely to the optimal policy (Verginis et al., 2022).

For robotic manipulation, the inferred RM is integrated through DQRM: a separate Q-network $r: U \times \Sigma \to \mathbb{R}$0 is trained for each RM state, experience is stored in state-specific replay buffers, and the reward is shaped by a potential $r: U \times \Sigma \to \mathbb{R}$1 so that

$r: U \times \Sigma \to \mathbb{R}$2

The Q-functions are implemented with two fully-convolutional heads, $r: U \times \Sigma \to \mathbb{R}$3 and $r: U \times \Sigma \to \mathbb{R}$4, on top of a shared ResNet-50 trunk, and training minimizes the standard TD error with Huber loss (Baert et al., 2024).

In POMDPs with separately inferred reward and transition memory, the augmented state becomes $r: U \times \Sigma \to \mathbb{R}$5, and standard Q-learning on that state was reported to converge to the optimal policy in approximately $r: U \times \Sigma \to \mathbb{R}$6 episodes in a $r: U \times \Sigma \to \mathbb{R}$7 environment (Wu et al., 3 Aug 2025). In inverse-reinforcement-learning settings, the pipeline is reversed: SAT is used to recover a transition structure consistent with a depth-$r: U \times \Sigma \to \mathbb{R}$8 prefix-tree policy, and a standard entropy-regularized MaxEnt IRL program is then solved on the product MDP to recover $r: U \times \Sigma \to \mathbb{R}$9 (Shehab et al., 6 Feb 2025). Fully differentiable FLNRM instead augments the policy input with the soft automaton state $M = (U, u_0, AP, \delta_u, \delta_r)$0 and minimizes

$M = (U, u_0, AP, \delta_u, \delta_r)$1

combining A2C with cross-entropy on reward-machine predictions (Dewidar et al., 23 Sep 2025).

5. Empirical profile

Empirical evaluation spans grid-world POMDPs, robotic manipulation, Officeworld-style domains, Breakfastworld, and LTL$M = (U, u_0, AP, \delta_u, \delta_r)$2-derived navigation tasks.

Study	Setting	Main reported outcome
(Icarte et al., 2021)	cookie, symbol, 2-keys POMDPs	FLNRM methods converge to optimal or near-optimal reward; A3C, ACER, PPO, DDQN fail beyond random performance
(Baert et al., 2024)	five robosuite block-manipulation tasks	inferred RM matches or nearly matches ground-truth RM; Place-3 slightly outperforms; convergence in a few hundred episodes
(Wu et al., 3 Aug 2025)	$M = (U, u_0, AP, \delta_u, \delta_r)$3 to $M = (U, u_0, AP, \delta_u, \delta_r)$4 grid POMDPs	up to three orders of magnitude faster than HMM and ILP baselines; downstream Q-learning converges in $M = (U, u_0, AP, \delta_u, \delta_r)$5 episodes
(Hyde et al., 2024)	Officeworld and Breakfastworld	optimal average return in all four Officeworld tasks; solve times from $M = (U, u_0, AP, \delta_u, \delta_r)$6 s to $M = (U, u_0, AP, \delta_u, \delta_r)$7 s
(Shehab et al., 8 Apr 2026)	two $M = (U, u_0, AP, \delta_u, \delta_r)$8 stochastic grid-worlds	active querying converges $M = (U, u_0, AP, \delta_u, \delta_r)$9 faster, uses $L:S \to 2^{AP}$0 less memory, runtime $L:S \to 2^{AP}$1 ks vs. $L:S \to 2^{AP}$2 ks on Task B
(Dewidar et al., 23 Sep 2025)	Map and Image environments, 8 LTL$L:S \to 2^{AP}$3 tasks	FLNRM consistently outperforms LSTM-based A2C; Task 5 reaches $L:S \to 2^{AP}$4 of maximum return in $L:S \to 2^{AP}$5K vs. $L:S \to 2^{AP}$6K steps
(Corazza et al., 16 Oct 2025)	Mining and Harvest	SRMI outperforms existing methods and a naive approach

Some results are especially informative about computational structure. In DBMM inference, the reported runtime on $L:S \to 2^{AP}$7 grids was $L:S \to 2^{AP}$8 s for DB-RPNI versus $L:S \to 2^{AP}$9 s for an HMM baseline and $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$0 s for an ILP baseline; on $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$1, $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$2 s versus $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$3 s and $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$4 s; on $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$5, DB-RPNI required $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$6 s while both baselines timed out after more than $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$7 h (Wu et al., 3 Aug 2025). In the $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$8 ablation, a full pipeline with a ground-truth TM of $$\mathcal{R}_P=\langle U,u_0,\delta_u,\delta_r\rangle$$9 states and RM of $L:O_0\times A_0\times O \to 2^P$0 states recovered $L:O_0\times A_0\times O \to 2^P$1 in approximately $L:O_0\times A_0\times O \to 2^P$2 s on $L:O_0\times A_0\times O \to 2^P$3 traces and approximately $L:O_0\times A_0\times O \to 2^P$4 s on $L:O_0\times A_0\times O \to 2^P$5 traces. Omitting observation-supplement increased RM size to $L:O_0\times A_0\times O \to 2^P$6 or $L:O_0\times A_0\times O \to 2^P$7 states and raised runtime to $L:O_0\times A_0\times O \to 2^P$8–$L:O_0\times A_0\times O \to 2^P$9 s, while dropping $S \times U$0-input or $S \times U$1-input removal caused $S \times U$2–$S \times U$3 slowdowns or failure (Wu et al., 3 Aug 2025).

In robotic manipulation, only $S \times U$4–$S \times U$5 demonstrations per task were used across Stack-2, Place-2, Pyramid-3, Stack-3, and Place-3, and agents using the inferred RM matched or nearly matched the ground-truth-RM baseline in reward and final placement error (Baert et al., 2024). In the raw-trajectory active-learning setting, passive SAT at depth $S \times U$6 recovered the ground-truth RM up to renaming in both tasks, while active querying with burn-in depth $S \times U$7 or $S \times U$8, budget $S \times U$9, and $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$00 reduced the hypothesis set to one by depth $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$01–$$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$02 (Shehab et al., 8 Apr 2026). These results suggest that FLNRM performance depends not only on the eventual expressiveness of the automaton, but also on the efficiency with which informative distinctions between histories are exposed.

6. Identifiability, misconceptions, and research frontiers

A recurrent theoretical theme is that automaton recovery depends on finite-depth or finite-coverage conditions. In partially observed policies, the sufficient depth for identifiability is $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$03, where $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$04 is an upper bound on the number of RM nodes; beyond that depth, deeper negative examples are redundant for policy-equivalent recovery (Shehab et al., 6 Feb 2025). In raw-trajectory recovery, the sufficient and necessary depth is stated as $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$05, and with $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$06 and $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$07 the negative-example constraints determine $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$08 and $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$09 up to label and node renaming among policy-equivalent RMs (Shehab et al., 8 Apr 2026). In DBMM inference, minimality is guaranteed under structure completeness, and in practice the paper states that it suffices that any two truly distinct states exhibit at least one $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$10-input on which they differ (Wu et al., 3 Aug 2025). These results make clear that “fully learnable” does not imply unconstrained identifiability.

A second recurring misconception is that FLNRM always refers to a single end-to-end differentiable objective. The robotic-manipulation pipeline explicitly does not optimize all components via a single likelihood or EM objective; instead it proceeds in four modular stages: pre-trained visual feature extraction, DBSCAN clustering, proposition definition by thresholding, and single-pass automaton construction (Baert et al., 2024). By contrast, neural FLNRM does optimize a joint objective over actor-critic loss and reward-machine prediction loss, but even there the latent alphabet $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$11 and latent automaton size $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$12 are fixed and user-chosen (Dewidar et al., 23 Sep 2025). A plausible implication is that the term “fully learnable” is best interpreted relative to which components are no longer hand-engineered, not as a guarantee of a uniform training paradigm.

A third issue is robustness to uncertainty and noise. In partially known semantics, proposition truth values are maintained as posterior probabilities $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$13 and thresholded at $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$14 to define $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$15, allowing the automaton to be inferred despite noisy sensors (Verginis et al., 2022). For noisy rewards, stochastic reward machines generalize deterministic outputs to transition-indexed reward distributions, and SRMI learns a minimal $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$16-consistent SRM using SMT constraints over transition structure and output means (Corazza et al., 16 Oct 2025). The resulting guarantees are asymptotic and depend on assumptions such as bounded symmetric noise, exploration, and a non-containment condition on reward distributions (Corazza et al., 16 Oct 2025).

Open directions are stated explicitly in several works. Proposed extensions include stochastic POMDPs and probabilistic DBMMs, active queries, neural parametrization of $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$17 and $$\mathcal{R} = \bigl(U,\;u_0,\;\Sigma,\;\delta,\;r,\;F\bigr),$$18, online continual learning via incremental RPNI variants such as IDS, and integration with hierarchical RL through option discovery (Wu et al., 3 Aug 2025). Other papers propose differentiable relaxations of SAT-style synthesis, transfer and composition of learned RMs, and extension beyond tabular MDPs and exact history policies (Shehab et al., 6 Feb 2025, Shehab et al., 8 Apr 2026). The cumulative picture is that FLNRM has moved from learning automata over engineered labels toward learning symbolic memory directly from raw, noisy, or partially observed data, while leaving open the hard problems of scalability, continuous observation spaces, and precise identifiability under limited coverage.