Automaton Constrained Q-Learning (ACQL)

• Automaton Constrained Q-Learning (ACQL) is a reinforcement learning method that augments state spaces with automata to handle non-Markovian rewards, temporal goals, and dynamic safety constraints.
• It constructs an augmented product CMDP in which automata encode stage-wise progress and safety maps, enabling subgoal-conditioned value learning and effective hindsight experience replay.
• Empirical evaluations in continuous-control tasks demonstrate ACQL’s superior success rates and reward performance compared to baseline methods, proving its practical viability.

Automaton Constrained Q-Learning (ACQL) is a reinforcement-learning methodology in which value learning is performed on an automaton-augmented state space so that temporally extended task progress, non-Markovian reward structure, and safety constraints become explicit components of the control problem. As a named method, ACQL was introduced for continuous-control robotics tasks specified by temporal logic, combining goal-conditioned value learning with automaton-guided reinforcement and an explicit safety critic on an augmented product CMDP (Manganaris et al., 6 Oct 2025). Closely related formulations predate the name itself: supervisory-control-based tabular Q-learning augments the MDP state with action-constraint automata and reward-machine memory (Raman et al., 2022), while logically constrained neural fitted Q-iteration constructs an on-the-fly product MDP with a limit-deterministic Büchi automaton for LTL satisfaction (Hasanbeig et al., 2018).

1. Historical emergence and conceptual scope

The term “Automaton Constrained Q-Learning” appears explicitly in the 2025 work on continuous-control tasks with temporally ordered goals and time-varying safety constraints (Manganaris et al., 6 Oct 2025). That formulation presents ACQL as an answer to a gap left by standard RL, standard goal-conditioned RL, and standard safe RL: temporally extended goals are non-Markovian when expressed directly over trajectories, and safety constraints may be non-stationary because what is safe can depend on the current stage of task completion.

Earlier work already contained most of the structural ingredients now associated with ACQL. In “Reinforcement Learning for Task Specifications with Action-Constraints,” the learner operates on a finite-state discounted MDP while admissible actions are filtered online by finite-state automata encoding safe or unsafe action sequences, and reward machines encode non-Markovian state constraints and rewards (Raman et al., 2022). In “Logically-Constrained Neural Fitted Q-Iteration,” an LTL formula is translated into a limit-deterministic Büchi automaton, synchronized with a continuous-state MDP, and optimized by a modified neural fitted Q-iteration algorithm on the product space (Hasanbeig et al., 2018). These precursor lines differ in emphasis—supervisory control, reward-machine decomposition, Büchi acceptance, or deep continuous control—but all instantiate the same general design principle: finite-state symbolic memory is appended to the environment state so that non-Markovian specifications become Markovian in the augmented process.

In the 2025 ACQL formulation, the automaton is not merely a monitor. It explicitly encodes stage-wise goal progression and both stationary and non-stationary safety constraints, and it is paired with subgoal-conditioned value learning so that hindsight experience replay (HER) can densify otherwise sparse automaton rewards (Manganaris et al., 6 Oct 2025). This distinguishes ACQL from product-MDP methods that rely on automaton state alone and from safe RL methods that assume a fixed cost function.

2. Formal architecture of ACQL

The 2025 formulation begins from an unconstrained MDP

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, \gamma),$$

and discusses the standard constrained-RL CMDP

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$

with objective

$$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$

where

$$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$

Temporal task structure is expressed with LTL/STL formulas over atomic propositions, and the implementation uses STL quantitative semantics

$$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$

so atomic propositions can be evaluated as real-valued state functions (Manganaris et al., 6 Oct 2025).

The temporal specification is translated into a deterministic Büchi automaton

$A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$

From each automaton state $q$, the method extracts three derived objects. The safety map $S(q)$ is obtained by identifying outgoing transitions from $q$ that enter non-accepting sink components and conjoining the negations of their predicates. The liveness map $O(q)$ is the disjunction of the remaining progress predicates. A subgoal-list map $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$0 is then produced by filtering $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$1 to those propositions designated as parameterized subgoal propositions (Manganaris et al., 6 Oct 2025).

The resulting control problem is the augmented product CMDP

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$2

whose state space is

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$3

An augmented state is

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$4

where $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$5 is the environment state, $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$6 is the automaton state, and $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$7 is the current list of relevant subgoals. The transition kernel is

$$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$8

The initial distribution fixes the automaton at $$(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)$$9 and the subgoal list at $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$0, and the task reward is sparse: $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$1 This sparse reward is intentionally paired with goal conditioning and HER rather than replaced by handcrafted shaping (Manganaris et al., 6 Oct 2025).

3. Safety, feasibility, and constrained value updates

A defining feature of ACQL is that safety is not treated as a stationary additive penalty. The immediate safety signal is

$$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$2

so safety is the STL robustness of the current state relative to the automaton-stage safety condition (Manganaris et al., 6 Oct 2025). Positive values indicate safe margin, and negative values indicate violation.

Instead of constraining expected discounted cumulative cost, the 2025 ACQL paper uses an expected minimum-safety objective: $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$3 The associated safety state-action function is

$$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$4

The paper argues that this is closer to hard safety than discounted cost accumulation because the salient event is whether the trajectory ever becomes unsafe (Manganaris et al., 6 Oct 2025).

ACQL learns two critics on the augmented state-action space: $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$5 The reward critic is trained by Bellman regression, while the safety critic uses the recursion

$$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$6

A crucial implementation detail is that $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$7 is scheduled during training to asymptotically approach $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$8, which the paper states is necessary for convergence to $$\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,$$9 (Manganaris et al., 6 Oct 2025).

Policy extraction is explicitly feasibility constrained: $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$0 Operationally, this means that the policy is not $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$1, but maximization of reward over actions predicted to preserve minimum safety. The paper recommends $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$2, since safety is normalized to positive and negative values (Manganaris et al., 6 Oct 2025).

The tabular convergence statement is given as Proposition 1. Under finite $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$3, finite $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$4, Robbins–Monro step sizes $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$5 and $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$6 with $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$7, and a slowly updated $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$8 with $$J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].$$9, the safety and reward critics converge almost surely to $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$0 and $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$1. The proof sketch uses a three-timescale stochastic approximation argument: fastest for $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$2, intermediate for $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$3, slowest for $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$4 (Manganaris et al., 6 Oct 2025).

4. Precursor formulations and adjacent variants

The named ACQL formulation sits within a broader family of automaton-structured Q-learning methods.

Formulation	Automaton role	Relation to ACQL
Supervisory Q-learning (Raman et al., 2022)	Filters unsafe action sequences; augments with RM state	Tabular ACQL-style precursor
LCNFQ (Hasanbeig et al., 2018)	Tracks LTL progress via LDBA in product MDP	Neural continuous-state precursor
Deep Constrained Q-learning (Kalweit et al., 2020)	No automata; constrains Bellman max over safe actions	Backup-level analogue
Automaton Distillation (Singireddy et al., 2023)	Stores teacher Q-values on automaton transitions	Soft, transfer-oriented adjacent method
PQLRM (Lequen et al., 17 Jun 2026)	Uses reward machines in multi-objective Q-learning	Automaton-augmented but not hard-constrained

In the supervisory-control formulation, action-sequence constraints are given as finite automata over the action alphabet, and the effective learning state is

$$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$5

where $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$6 tracks safe-sequence automaton state, $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$7 tracks unsafe-sequence automaton state, and $$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$8 tracks reward-machine state (Raman et al., 2022). The admissible action condition is

$$\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},$$9

and the Bellman target is correspondingly restricted to supervisor-enabled actions. This formulation adds a distinctly supervisory-control feature that is absent from many ACQL descriptions: the action alphabet is partitioned into controllable and uncontrollable actions, and enforceability is characterized by the controllability condition

$A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$0

If this condition fails, the best enforceable approximation is the supremal controllable sublanguage $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$1 (Raman et al., 2022).

LCNFQ provides a different precursor path. It translates an LTL formula into a limit-deterministic Büchi automaton, constructs an on-the-fly synchronized product MDP $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$2, and uses an accepting-frontier function to shape rewards toward repeated visits of required accepting sets (Hasanbeig et al., 2018). Its state is the product pair $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$3, and it trains one MLP per automaton state rather than one global network. This can be read as a neural, batch, continuous-state realization of the same principle that later ACQL makes explicit: logical progress is tracked by finite-state automaton memory, and Q-learning proceeds on the synchronized structure.

Deep Constrained Q-learning is not automata based, but it is directly relevant because it embeds feasibility into the Bellman backup itself: $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$4 This paper argues that post hoc safe policy extraction is insufficient because unconstrained backups propagate value through actions that are later masked, yielding safe but potentially suboptimal behavior (Kalweit et al., 2020). That observation closely matches the logic of ACQL, where automaton feasibility must affect value propagation, not only deployment-time action choice.

5. Empirical evaluation and application domains

The 2025 ACQL paper evaluates the method in Brax continuous-control domains with 2D PointMass, Quadcopter, and 8-DOF Ant, and compares it against two online RL baselines for LTL-specified tasks: CRM-RS and LOF (Manganaris et al., 6 Oct 2025). Five task types are studied: a two-subgoal sequential navigation task, a two-subgoal branching navigation task, a single-goal task with an unsafe region, a two-subgoal task with a disappearing safety constraint, and an infinitely looping navigation task with a persistent safety constraint. The reported metrics are reward—defined as the number of steps spent near the final goal of the task automaton—and success rate over 1000-step evaluation episodes, with 5 million environment interactions and 16 evaluation episodes for each of 5 random seeds.

The paper reports that ACQL outperforms both baselines across all tested environments and task types (Manganaris et al., 6 Oct 2025). CRM-RS is described as generally failing because sparse, delayed rewards make stable subgoal achievement difficult without goal conditioning. LOF scales better to multiple goals due to hierarchy and often achieves high logical robustness, but it tends not to obtain high reward because it focuses on option execution for subgoal achievement rather than sustaining the final goal region after task satisfaction. The ablation results are also central: removing HER causes performance to collapse, and replacing the minimum-safety formulation with standard sum-of-cost CMDP constraints likewise causes performance on safety tasks to collapse.

The real-world demonstration uses a 6-DOF UR5e manipulator in a cabinet-like workspace with two geometric obstacles and the task

$A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$5

The action space consists of six discrete end-effector translations along the cardinal directions. The trained ACQL policy achieved mean reward $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$6 and $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$7 success rate over 16 simulated rollouts, and the same learned policy was successfully deployed on the real robot in the real cabinet setup (Manganaris et al., 6 Oct 2025).

The earlier tabular precursor offers a smaller but conceptually revealing experiment. In a pickup-and-delivery grid world with forbidden U-turn substrings $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$8, an uncontrollable drop action $A = (\Sigma, \mathcal{Q}, \delta, q_0, F).$9, and a safe-sequence recovery constraint requiring $q$0 after $q$1, supervised Q-learning is trained for 500,000 episodes with maximum episode length 60, $q$2, $q$3, and $q$4. The learned $q$5-greedy policy completes the task about $q$6 of the time at the end of training, and with $q$7 at inference it completes the task optimally $q$8 of the time (Raman et al., 2022). The evidence is illustrative rather than comparative, but it shows that action-history constraints, uncontrollable events, and non-Markovian task logic can coexist in one tabular RL loop.

6. Limitations, boundary cases, and open directions

The 2025 ACQL paper explicitly restricts its formal support to tasks representable by deterministic Büchi automata, and operationally to STL formulas in the recurrence class that can be translated into a DBA (Manganaris et al., 6 Oct 2025). It also notes that conditioning mainly on the current subgoal set does not fully capture longer-range future task requirements beyond the local stage. Partial observability and broader sim-to-real mismatch remain open problems, and the authors identify richer automata classes, including good-for-MDP nondeterministic Büchi automata, as future directions (Manganaris et al., 6 Oct 2025).

The supervisory-control precursor imposes a different set of limits. It is tabular and finite-state; automata and reward machines must be known or manually specified in advance; full observability is implicitly required; and the paper does not develop function approximation, deep RL, partial observability, or formal convergence and error bounds for the constrained learner (Raman et al., 2022). LCNFQ, while extending automaton-constrained learning to continuous states and neural approximation, does not provide a fully formal theorem in the cited text establishing exact equivalence between maximizing shaped discounted return and maximizing LTL satisfaction probability (Hasanbeig et al., 2018).

A broader boundary of the topic concerns methods that are automaton structured but not hard-constrained. Automaton Distillation uses a DFA as a low-dimensional transfer medium for teacher Q-values, blending automaton-indexed transfer targets with ordinary Bellman targets by an annealed coefficient

$q$9

so the automaton guides early learning but does not permanently constrain action feasibility (Singireddy et al., 2023). PQLRM augments state with joint reward-machine memory in multi-objective RL and exploits deterministic automaton structure to update all automaton states from one transition, but the automata define rewards rather than legal or illegal behavior (Lequen et al., 17 Jun 2026). These lines suggest a wider interpretation in which automata can serve not only as hard constraint monitors, but also as transfer interfaces, sample-efficiency devices, or structured representations of non-Markovian reward.

Taken together, the literature supports a relatively precise characterization. ACQL, in the narrow sense, is Q-learning on an automaton-augmented state space with feasibility or acceptance structure entering the backup and policy extraction. Its central technical commitment is that symbolic task memory should be part of the value-learning state, rather than an external evaluator applied only after learning. Its current practical strength lies in combining temporal-logic task structure, stage-dependent safety, and off-policy goal-conditioned learning in continuous-control settings (Manganaris et al., 6 Oct 2025). Its main unresolved issues concern expressiveness beyond DBA-representable tasks, stronger guarantees under function approximation, and the treatment of partial observability and broader real-world deployment conditions.