DBMM: Unifying Reward and Transition Memory

• DBMM is a dual-input automaton that distinguishes reward-triggering inputs from state-updating inputs in deterministic partially observable domains.
• It subsumes both Reward Machines and Transition Machines, enabling efficient inference of minimal resolvent models via the DB-RPNI algorithm.
• Empirical results demonstrate that DBMM significantly speeds up automata learning for restoring Markov properties and enhancing RL performance.

Searching arXiv for the specified paper and closely related context. The Dual Behavior Mealy Machine (DBMM) is a unifying automaton formalism introduced for inferring reward-relevant and transition-relevant memory in deterministic partially observable domains. It was proposed in the context of learning from traces in Deterministic Partially Observable Markov Decision Processes (Det-POMDPs), where observations are non-Markovian because the same observation-action pair can correspond to different hidden states and histories. In this setting, DBMM subsumes both Reward Machines (RMs) and Transition Machines (TMs) by separating inputs that trigger outputs from inputs that update internal state, thereby providing a single representation for reward-based and transition-based non-Markovianity (Wu et al., 3 Aug 2025).

1. Conceptual role in partially observable reinforcement learning

The DBMM arises from a specific limitation in automaton-based approaches to partially observable reinforcement learning. In Det-POMDPs, the observation process can fail to preserve the Markov property: identical observation-action pairs may induce different next observations and different rewards depending on latent state and prior history. Existing automaton representations had focused primarily on reward-based non-Markovianity through Reward Machines, but this can force unnatural formulations when the underlying issue is instead transition-based non-Markovianity (Wu et al., 3 Aug 2025).

The motivating example is a sequential task in which an agent must first pick up a key in an “orangeroom” to unlock entry into a “cyanroom,” then visit a “toilet,” and finally sit on a “sofa” in a “limegreenroom” to receive reward. Observations reveal only room area, not hidden variables such as whether the key was acquired or whether the toilet was visited. As a result, the same observation-action pair can lead to different next observations, and whether sitting produces reward depends on earlier events. A minimal RM that tracks only whether the toilet has been reached is sufficient to predict rewards, yet it does not encode the key-dependent transition constraint and is therefore insufficient for policy learning. This is the setting in which the DBMM is introduced as a formal device that can express both kinds of memory in a common framework (Wu et al., 3 Aug 2025).

This suggests that the DBMM is not merely a representational convenience. It is intended to disentangle two different sources of apparent non-Markovianity that would otherwise be conflated if one modeled rewards alone.

2. Formal setting and automaton definition

The formal setting is a deterministic partially observable domain. A Det-POMDP is defined as

$$\mathcal{P} = \langle S, A, s_0, P, R, \gamma, O, Z \rangle,$$

where $S$, $A$, and $O$ are finite sets of states, actions, and observations; $s_0 \in S$ is the initial state; $P: S \times A \rightarrow S$ is a deterministic transition function; $R: S \times A \rightarrow \mathbb{R}$ is the reward function; $\gamma \in [0,1]$ is the discount factor; and $Z: S \rightarrow O$ is the observation function. The framework also assumes a set of atomic propositions $\text{AP}$ and a labeling function $S$0 mapping observations to symbolic events (Wu et al., 3 Aug 2025).

Within this setting, non-Markovian rewards arise when $S$1 depends on history even after conditioning on the current observation $S$2 and action $S$3. Non-Markovian transitions arise when the next observation $S$4 is not uniquely determined by the current observation-action pair $S$5 because the underlying state $S$6 depends on history and is not fully revealed by $S$7 (Wu et al., 3 Aug 2025).

The DBMM is defined as

$S$8

where $S$9 is a finite set of states and $A$0 is the initial state; $A$1 is the alpha input alphabet, whose elements trigger outputs but do not cause state transitions; $A$2 is the beta input alphabet, whose elements cause state transitions but do not trigger outputs; $A$3; $A$4 is the output alphabet; $A$5 is the state transition function; and $A$6 is the output function (Wu et al., 3 Aug 2025).

Its semantics are explicitly dual. The automaton consumes alternating streams of inputs in which $A$7-inputs, corresponding to labels, update internal memory, while $A$8-inputs, corresponding to observation-action pairs, produce outputs conditioned on the current memory state. This Mealy-style decomposition is the defining feature of the model.

3. Relationship to Reward Machines and Transition Machines

A central property of the DBMM is that it subsumes both Reward Machines and Transition Machines. Reward Machines encode reward-related history to restore Markovian rewards, whereas Transition Machines encode transition-related history to restore Markovian transitions (Wu et al., 3 Aug 2025).

A Reward Machine is defined as

$A$9

where $O$0 is a finite set of RM states, $O$1 is the initial RM state, $O$2 is the state-transition function on labels, and $O$3 is the reward-output function. Its semantics are that the automaton consumes label sequences $O$4 through $O$5 and outputs rewards through $O$6, thereby rendering rewards Markovian in the augmented state $O$7 (Wu et al., 3 Aug 2025).

A Transition Machine is defined analogously as

$O$8

where $O$9 is a finite set of TM states, $s_0 \in S$0 is the initial TM state, $s_0 \in S$1 is the state-transition function on labels, and $s_0 \in S$2 is the transition-output function. Its semantics are that the automaton consumes label sequences and predicts next observations via $s_0 \in S$3 (Wu et al., 3 Aug 2025).

The subsumption maps are direct:

Automaton instance	DBMM alphabets and outputs	Interpretation
RM as DBMM	$s_0 \in S$4	$s_0 \in S$5
TM as DBMM	$s_0 \in S$6	$s_0 \in S$7

Formally, RMs and TMs are projections of DBMMs obtained by fixing alphabets and interpreting $s_0 \in S$8 and $s_0 \in S$9 as the transition and output functions appropriate to rewards or next observations. Removing $P: S \times A \rightarrow S$0, or ignoring its outputs, yields a pure transition structure. Fixing $P: S \times A \rightarrow S$1 while retaining $P: S \times A \rightarrow S$2 yields a pure Mealy-style output structure over $P: S \times A \rightarrow S$3 conditioned on label-driven memory (Wu et al., 3 Aug 2025).

A plausible implication is that DBMM provides a common inference target without collapsing the semantic distinction between reward memory and transition memory.

4. Resolvent property, minimality, and Markov restoration

The paper formalizes correctness through the notion of being resolvent to a Det-POMDP. For an automaton with transition function $P: S \times A \rightarrow S$4, let $P: S \times A \rightarrow S$5 denote the extension to sequences, defined by $P: S \times A \rightarrow S$6 and $P: S \times A \rightarrow S$7 (Wu et al., 3 Aug 2025).

An RM $P: S \times A \rightarrow S$8 is resolvent to a Det-POMDP $P: S \times A \rightarrow S$9 if, for every feasible history $R: S \times A \rightarrow \mathbb{R}$0, letting $R: S \times A \rightarrow \mathbb{R}$1 with $R: S \times A \rightarrow \mathbb{R}$2,

$R: S \times A \rightarrow \mathbb{R}$3

A TM $R: S \times A \rightarrow \mathbb{R}$4 is resolvent if, for every feasible history $R: S \times A \rightarrow \mathbb{R}$5, letting $R: S \times A \rightarrow \mathbb{R}$6,

$R: S \times A \rightarrow \mathbb{R}$7

Here $R: S \times A \rightarrow \mathbb{R}$8 denotes the TM output function and coincides conceptually with $R: S \times A \rightarrow \mathbb{R}$9 (Wu et al., 3 Aug 2025).

Minimal resolvent machines are those resolvent machines for which no smaller resolvent machine exists. This notion is central because the learning objective is not only behavioral correctness on feasible histories but also state minimality.

The same work argues that transition-related non-Markovianity can masquerade as reward non-Markovianity, creating spurious states in an inferred RM. To address this, it introduces observation supplement, which augments each observation $\gamma \in [0,1]$0 with the TM state $\gamma \in [0,1]$1 reached before $\gamma \in [0,1]$2, yielding $\gamma \in [0,1]$3. If the TM is resolvent, then the mapping $\gamma \in [0,1]$4 becomes deterministic, so transition-related non-Markovianity is eliminated during RM inference (Wu et al., 3 Aug 2025).

This suggests a layered restoration of Markovianity: first at the transition level through the TM, and then at the reward level through the RM conditioned on supplemented observations.

5. Inference via DB-RPNI

The DBMM is inferred from traces through DB-RPNI, a passive automata learning algorithm that adapts the classical RPNI state-merging procedure to the dual-input DBMM setting (Wu et al., 3 Aug 2025).

Each labeled trajectory

$\gamma \in [0,1]$5

is transformed into two sample sets. For TM inference, the input sequence is

$\gamma \in [0,1]$6

with output sequence

$\gamma \in [0,1]$7

For RM inference, the input sequence is the same alternating sequence, while the output sequence is

$\gamma \in [0,1]$8

Observation-action pairs are $\gamma \in [0,1]$9-inputs and labels are $Z: S \rightarrow O$0-inputs; $Z: S \rightarrow O$1 is a placeholder output for beta inputs (Wu et al., 3 Aug 2025).

The overall pipeline is:

• TMSampleSet ← ReductionRule(TMSampleSet)
• TM ← DB-RPNI(TMSampleSet)
• RMSampleSet ← StateSupp(TM, RMSampleSet)
• RMSampleSet ← ReductionRule(RMSampleSet)
• RM ← DB-RPNI(RMSampleSet)

Two preprocessing rules are specified. Redundant $Z: S \rightarrow O$2-Input Removal removes any $Z: S \rightarrow O$3-input that always produces the same output regardless of state, recording a constant mapping $Z: S \rightarrow O$4. After inference, $Z: S \rightarrow O$5 is assigned for all states. Trivial $Z: S \rightarrow O$6-Input Removal removes beta inputs known a priori not to cause state transitions and later restores them as self-loops $Z: S \rightarrow O$7 for all states (Wu et al., 3 Aug 2025).

DB-RPNI proceeds in two phases. First, Prefix Tree Transducer construction builds an exact automaton from the sample set, with each distinct beta-input prefix inducing a unique state and each observed alpha-input at a state storing its output via $Z: S \rightarrow O$8. Second, red-blue state merging maintains a Red core and a Blue frontier of states reachable by a single beta-transition from Red. Each blue state is merged with compatible red states if possible; otherwise, it is promoted to Red (Wu et al., 3 Aug 2025).

Compatibility is local:

$Z: S \rightarrow O$9

where

$\text{AP}$0

Implied merging, induced by identical beta-successors, is handled through a folding operation that both checks mergeability and constructs the quotient automaton (Wu et al., 3 Aug 2025).

6. Guarantees, empirical results, and reinforcement-learning integration

Under structure completeness, DB-RPNI returns the minimal resolvent DBMM and runs in polynomial time. Let $\text{AP}$1 denote the number of states in the Prefix Tree Transducer, $\text{AP}$2 the number of states in the target automaton, $\text{AP}$3, and

$\text{AP}$4

The main theorem states:

1. If the sample set $\text{AP}$5 is structure complete, DB-RPNI returns the minimal resolvent DBMM.
2. With structure complete $\text{AP}$6, DB-RPNI runs in $\text{AP}$7 time.
3. In the general case, DB-RPNI runs in $\text{AP}$8 time (Wu et al., 3 Aug 2025).

Structure completeness ensures coverage of states and transitions and ensures that any two non-equivalent states eventually conflict by producing different outputs on a common alpha-input, thereby preventing incorrect merges. Although this condition is sufficient rather than necessary, the paper notes that in practice the algorithm succeeds when the data contains enough evidence to block wrong merges (Wu et al., 3 Aug 2025).

Empirically, the method is benchmarked against two state-of-the-art baselines that reduce automaton inference to more expensive mathematical formalisms: HMM-based DFA inference and ILP-based RM inference. On 3×3, 4×4, and 5×5 grid tasks with sequential phases, the proposed method runs substantially faster. For 3×3, it takes 1.3s versus 104.0s and 5.3s. For 4×4, it takes 3.9s versus over 6,000s and about 5,500s. For 5×5, both baselines time out, whereas the proposed method completes in 56.1s. The reported overall speedup is up to three orders of magnitude over the baselines (Wu et al., 3 Aug 2025).

Ablations on a 25×25 grid with complex non-Markovian dynamics further isolate the contribution of the design choices. Removing Observation Supplement causes the RM size to explode, for example to 218 states under low data, and drastically increases runtime. Removing Redundant $\text{AP}$9-Input Removal or Trivial $S$00-Input Removal causes 7×–15× slowdowns, and removing both rules leads to failure at scale (Wu et al., 3 Aug 2025).

Once TM and RM are learned, the agent operates on the augmented state

$S$01

At each step, labels update $S$02 via $S$03 and $S$04 via $S$05; rewards are given by $S$06, and next observations can be predicted by $S$07. Standard RL algorithms such as Q-learning can then be applied over the product state space. In the reported experiment, integrating the learned TM with 7 states and RM with 2 states into a Q-learning agent over $S$08 achieves optimal control (Wu et al., 3 Aug 2025).

7. Illustrative construction, assumptions, and scope

The paper gives a concrete Det-POMDP example with four areas

$S$09

atomic propositions

$S$10

actions

$S$11

and hidden flags indicating whether the key has been acquired and whether the toilet has been visited. Observations reveal only the current area; labels reflect events at that area. Sitting on the sofa yields reward 1 only if the toilet was previously visited, and moving up from corridor to cyanroom succeeds only if the key was previously acquired (Wu et al., 3 Aug 2025).

The minimal RM tracks whether the toilet has been reached. It uses

$S$12

with $S$13 if $S$14 includes toilet and $S$15 otherwise, while $S$16 for all labels. The least TM tracks whether the key has been acquired. It uses

$S$17

with $S$18 if $S$19 includes key and $S$20 otherwise, and $S$21. Its next-observation predictions include

$S$22

The unified DBMM instantiates these as an RM-DBMM with $S$23 and a TM-DBMM with $S$24 (Wu et al., 3 Aug 2025).

On an example trace beginning at corridor without a key, visiting orangeroom to acquire the key, returning to corridor, moving up, visiting the toilet, then reaching limegreenroom and sitting, the TM-DBMM updates state through labels and outputs next observations, while the RM-DBMM updates state through labels and outputs rewards. BuildPTT creates distinct states for each beta-label prefix, and StateMerging merges states that are locally compatible and whose implied merges remain compatible. Observation supplement then prevents the RM from overfitting transition history by augmenting $S$25 to $S$26 (Wu et al., 3 Aug 2025).

The scope of the approach is explicitly limited. It assumes deterministic POMDP dynamics, a known labeling function $S$27, sufficient trace data to avoid incorrect merges, and a passive learning setting in which traces are given rather than obtained through active queries. Limitations include restriction to deterministic environments and sensitivity to data quality. Extending the method to stochastic POMDPs may require probabilistic automata, and handling noise or incomplete traces is left as future work. The preprocessing rules also assume some prior knowledge, such as which beta-inputs are trivial or which alpha-inputs are redundant, although the paper states that this can be justified in many RL environments (Wu et al., 3 Aug 2025).

In this formulation, DBMM provides the common representational layer through which transition-dependent and reward-dependent memory can be inferred separately yet used jointly. Its significance lies in making explicit that reward prediction and transition prediction need not be encoded by the same latent automaton, even when both are learned from the same partially observable traces.