---
title: Automaton Constrained Q-Learning (ACQL)
url: https://www.emergentmind.com/topics/automaton-constrained-q-learning-acql
type: topic
---

# Automaton Constrained Q-Learning (ACQL)

Automaton Constrained Q-Learning (ACQL) is a reinforcement-learning methodology in which value learning is performed on an automaton-augmented state space so that temporally extended task progress, non-Markovian reward structure, and safety constraints become explicit components of the control problem. As a named method, ACQL was introduced for continuous-control robotics tasks specified by temporal logic, combining goal-conditioned value learning with automaton-guided reinforcement and an explicit safety critic on an augmented product CMDP [2510.05061]. Closely related formulations predate the name itself: supervisory-control-based tabular Q-learning augments the MDP state with action-constraint automata and reward-machine memory [2201.00286], while logically constrained neural fitted Q-iteration constructs an on-the-fly product MDP with a limit-deterministic Büchi automaton for LTL satisfaction [1809.07823].

## 1. Historical emergence and conceptual scope

The term “Automaton Constrained Q-Learning” appears explicitly in the 2025 work on continuous-control tasks with temporally ordered goals and time-varying safety constraints [2510.05061]. That formulation presents ACQL as an answer to a gap left by standard RL, standard goal-conditioned RL, and standard safe RL: temporally extended goals are non-Markovian when expressed directly over trajectories, and safety constraints may be non-stationary because what is safe can depend on the current stage of task completion.

Earlier work already contained most of the structural ingredients now associated with ACQL. In “Reinforcement Learning for Task Specifications with Action-Constraints,” the learner operates on a finite-state discounted MDP while admissible actions are filtered online by finite-state automata encoding safe or unsafe action sequences, and reward machines encode non-Markovian state constraints and rewards [2201.00286]. In “Logically-Constrained Neural Fitted Q-Iteration,” an LTL formula is translated into a limit-deterministic Büchi automaton, synchronized with a continuous-state MDP, and optimized by a modified neural fitted Q-iteration algorithm on the product space [1809.07823]. These precursor lines differ in emphasis—supervisory control, reward-machine decomposition, Büchi acceptance, or deep continuous control—but all instantiate the same general design principle: finite-state symbolic memory is appended to the environment state so that non-Markovian specifications become Markovian in the augmented process.

In the 2025 ACQL formulation, the automaton is not merely a monitor. It explicitly encodes stage-wise goal progression and both stationary and non-stationary safety constraints, and it is paired with subgoal-conditioned value learning so that hindsight experience replay (HER) can densify otherwise sparse automaton rewards [2510.05061]. This distinguishes ACQL from product-MDP methods that rely on automaton state alone and from safe RL methods that assume a fixed cost function.

## 2. Formal architecture of ACQL

The 2025 formulation begins from an unconstrained MDP
\[
\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, \gamma),
\]
and discusses the standard constrained-RL CMDP
\[
(\mathcal{S}, \mathcal{A}, \mathcal{T}, d_0, r, c, L, \gamma)
\]
with objective
\[
\max_{\pi} \; J_r(\pi) \quad \textnormal{s.t.} \quad J_c(\pi) < L,
\]
where
\[
J_r(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right ], \qquad
J_c(\pi) = \mathbb{E}_{\tau \sim \pi} \left [ \sum_{t=0}^{\infty} \gamma^t c(s_t, a_t) \right ].
\]
Temporal task structure is expressed with LTL/STL formulas over atomic propositions, and the implementation uses STL quantitative semantics
\[
\rho : \Sigma^\omega \times \Phi \rightarrow \mathbb{R},
\]
so atomic propositions can be evaluated as real-valued state functions [2510.05061].

The temporal specification is translated into a deterministic Büchi automaton
\[
A = (\Sigma, \mathcal{Q}, \delta, q_0, F).
\]
From each automaton state \(q\), the method extracts three derived objects. The safety map \(S(q)\) is obtained by identifying outgoing transitions from \(q\) that enter non-accepting sink components and conjoining the negations of their predicates. The liveness map \(O(q)\) is the disjunction of the remaining progress predicates. A subgoal-list map \(G(q)\) is then produced by filtering \(O(q)\) to those propositions designated as parameterized subgoal propositions [2510.05061].

The resulting control problem is the augmented product CMDP
\[
\mathcal{M}^{A} = (\mathcal{S}^{A}, \mathcal{A}^A, \mathcal{T}^{A}, d_0^{A}, r^{A}, c^{A}, \gamma, L),
\]
whose state space is
\[
\mathcal{S}^A = \mathcal{S} \times \mathcal{G}^+ \times \mathcal{Q}.
\]
An augmented state is
\[
s^A = \langle s, g^+, q \rangle,
\]
where \(s\) is the environment state, \(q\) is the automaton state, and \(g^+\) is the current list of relevant subgoals. The transition kernel is
\[
\mathcal{T}^A(\langle s', g^{+}{'}, q' \rangle \mid \langle s, g^+, q \rangle, a)
=
\begin{cases}
\mathcal{T}(s'|s, a) & \text{if } q' = \delta(q, s'), \; g^+{}' = G(q'), \\
0 & \text{otherwise}.
\end{cases}
\]
The initial distribution fixes the automaton at \(q_0\) and the subgoal list at \(G(q_0)\), and the task reward is sparse:
\[
r^A(\langle s, g^+, q \rangle) = \mathbbm{1}_{F}(q).
\]
This sparse reward is intentionally paired with goal conditioning and HER rather than replaced by handcrafted shaping [2510.05061].

## 3. Safety, feasibility, and constrained value updates

A defining feature of ACQL is that safety is not treated as a stationary additive penalty. The immediate safety signal is
\[
c^A(\langle s, g^+, q \rangle, a) = \rho(s, S(q)),
\]
so safety is the STL robustness of the current state relative to the automaton-stage safety condition [2510.05061]. Positive values indicate safe margin, and negative values indicate violation.

Instead of constraining expected discounted cumulative cost, the 2025 ACQL paper uses an expected minimum-safety objective:
\[
\mathbb{E}_{\tau \sim \pi} \left [ \min_{t \in [0, \infty]} c^A(s_t, a_t) \right ] > L.
\]
The associated safety state-action function is
\[
Q^c_\pi(s, a) = \mathbb{E}_{\tau \sim \pi} \left[ \min_{t=0}^{\infty} c^A(s^A_t,a_t) \mid s^A_0=s, a_0=a \right].
\]
The paper argues that this is closer to hard safety than discounted cost accumulation because the salient event is whether the trajectory ever becomes unsafe [2510.05061].

ACQL learns two critics on the augmented state-action space:
\[
Q^r_\psi : \mathcal{S}^A \times \mathcal{A} \to \mathbb{R}, \qquad
Q^c_\theta : \mathcal{S}^A \times \mathcal{A} \to [-1,1].
\]
The reward critic is trained by Bellman regression, while the safety critic uses the recursion
\[
y^c_t =
\gamma_c \min \left\{ c^A(s^A_t, a_t), \; Q^{c}_{\bar{\theta}}(s^A_{t+1}, \pi_j(s^A_{t+1})) \right\}
+
(1 - \gamma_c)\; c^A(s^A_t, a_t).
\]
A crucial implementation detail is that \(\gamma_c\) is scheduled during training to asymptotically approach \(1.0\), which the paper states is necessary for convergence to \(Q^{c*}\) [2510.05061].

Policy extraction is explicitly feasibility constrained:
\[
\pi^*(s^A) = \argmax_{a \; : \; Q^{c*}(s^A, a) > L} Q^{r*}(s^A, a).
\]
Operationally, this means that the policy is not \(\arg\max_a Q^r(s^A,a)\), but maximization of reward over actions predicted to preserve minimum safety. The paper recommends \(L=0\), since safety is normalized to positive and negative values [2510.05061].

The tabular convergence statement is given as Proposition 1. Under finite \(|\mathcal{S}^A|\), finite \(|\mathcal{A}|\), Robbins–Monro step sizes \(a(n)\) and \(b(n)\) with \(b(n)\in o(a(n))\), and a slowly updated \(\gamma_{c_n}\to 1\) with \(c(n)\in o(b(n))\), the safety and reward critics converge almost surely to \(Q^{c*}\) and \(Q^{r*}\). The proof sketch uses a three-timescale stochastic approximation argument: fastest for \(Q^c_n\), intermediate for \(Q^r_n\), slowest for \(\gamma_{c_n}\) [2510.05061].

## 4. Precursor formulations and adjacent variants

The named ACQL formulation sits within a broader family of automaton-structured Q-learning methods.

| Formulation | Automaton role | Relation to ACQL |
|---|---|---|
| Supervisory Q-learning [2201.00286] | Filters unsafe action sequences; augments with RM state | Tabular ACQL-style precursor |
| LCNFQ [1809.07823] | Tracks LTL progress via LDBA in product MDP | Neural continuous-state precursor |
| Deep Constrained Q-learning [2003.09398] | No automata; constrains Bellman max over safe actions | Backup-level analogue |
| Automaton Distillation [2310.19137] | Stores teacher Q-values on automaton transitions | Soft, transfer-oriented adjacent method |
| PQLRM [2606.19134] | Uses reward machines in multi-objective Q-learning | Automaton-augmented but not hard-constrained |

In the supervisory-control formulation, action-sequence constraints are given as finite automata over the action alphabet, and the effective learning state is
\[
(s,q_s,q_h,u_j),
\]
where \(q_s\) tracks safe-sequence automaton state, \(q_h\) tracks unsafe-sequence automaton state, and \(u_j\) tracks reward-machine state [2201.00286]. The admissible action condition is
\[
(\delta_s(a,q_s)\neq s_a)\ \wedge\ (\delta_h(a,q_h)\neq s_a),
\]
and the Bellman target is correspondingly restricted to supervisor-enabled actions. This formulation adds a distinctly supervisory-control feature that is absent from many ACQL descriptions: the action alphabet is partitioned into controllable and uncontrollable actions, and enforceability is characterized by the controllability condition
\[
\overline{K}\Sigma_u \cap L(\mathcal{M}) \subseteq \overline{K}.
\]
If this condition fails, the best enforceable approximation is the supremal controllable sublanguage \(K^{\uparrow}\) [2201.00286].

LCNFQ provides a different precursor path. It translates an LTL formula into a limit-deterministic Büchi automaton, constructs an on-the-fly synchronized product MDP \(M\otimes N\), and uses an accepting-frontier function to shape rewards toward repeated visits of required accepting sets [1809.07823]. Its state is the product pair \((s,q)\), and it trains one MLP per automaton state rather than one global network. This can be read as a neural, batch, continuous-state realization of the same principle that later ACQL makes explicit: logical progress is tracked by finite-state automaton memory, and Q-learning proceeds on the synchronized structure.

Deep Constrained Q-learning is not automata based, but it is directly relevant because it embeds feasibility into the Bellman backup itself:
\[
Q^{\mathcal C}(s_t,a_t)\leftarrow (1-\alpha)Q^{\mathcal C}(s_t,a_t)
+\alpha\left(r_t+\gamma \max_{a\in S_{\mathcal C}(s_{t+1})}Q^{\mathcal C}(s_{t+1},a)\right).
\]
This paper argues that post hoc safe policy extraction is insufficient because unconstrained backups propagate value through actions that are later masked, yielding safe but potentially suboptimal behavior [2003.09398]. That observation closely matches the logic of ACQL, where automaton feasibility must affect value propagation, not only deployment-time action choice.

## 5. Empirical evaluation and application domains

The 2025 ACQL paper evaluates the method in Brax continuous-control domains with 2D PointMass, Quadcopter, and 8-DOF Ant, and compares it against two online RL baselines for LTL-specified tasks: CRM-RS and LOF [2510.05061]. Five task types are studied: a two-subgoal sequential navigation task, a two-subgoal branching navigation task, a single-goal task with an unsafe region, a two-subgoal task with a disappearing safety constraint, and an infinitely looping navigation task with a persistent safety constraint. The reported metrics are reward—defined as the number of steps spent near the final goal of the task automaton—and success rate over 1000-step evaluation episodes, with 5 million environment interactions and 16 evaluation episodes for each of 5 random seeds.

The paper reports that ACQL outperforms both baselines across all tested environments and task types [2510.05061]. CRM-RS is described as generally failing because sparse, delayed rewards make stable subgoal achievement difficult without goal conditioning. LOF scales better to multiple goals due to hierarchy and often achieves high logical robustness, but it tends not to obtain high reward because it focuses on option execution for subgoal achievement rather than sustaining the final goal region after task satisfaction. The ablation results are also central: removing HER causes performance to collapse, and replacing the minimum-safety formulation with standard sum-of-cost CMDP constraints likewise causes performance on safety tasks to collapse.

The real-world demonstration uses a 6-DOF UR5e manipulator in a cabinet-like workspace with two geometric obstacles and the task
\[
\lozenge(p_1 \wedge \circ(\lozenge(p_2 \wedge \circ(\lozenge(p_3))))) \wedge \square(\neg(\text{in\_wall} \vee \text{in\_table})).
\]
The action space consists of six discrete end-effector translations along the cardinal directions. The trained ACQL policy achieved mean reward \(908.4\) and \(100\%\) success rate over 16 simulated rollouts, and the same learned policy was successfully deployed on the real robot in the real cabinet setup [2510.05061].

The earlier tabular precursor offers a smaller but conceptually revealing experiment. In a pickup-and-delivery grid world with forbidden U-turn substrings \(\{ll,rr\}\), an uncontrollable drop action \(d_u\), and a safe-sequence recovery constraint requiring \(d_2d_3p_1p_2p_3\) after \(d_u\), supervised Q-learning is trained for 500,000 episodes with maximum episode length 60, \(\alpha=0.1\), \(\epsilon=0.25\), and \(\gamma=0.9\). The learned \(\epsilon\)-greedy policy completes the task about \(95\%\) of the time at the end of training, and with \(\epsilon=0\) at inference it completes the task optimally \(100\%\) of the time [2201.00286]. The evidence is illustrative rather than comparative, but it shows that action-history constraints, uncontrollable events, and non-Markovian task logic can coexist in one tabular RL loop.

## 6. Limitations, boundary cases, and open directions

The 2025 ACQL paper explicitly restricts its formal support to tasks representable by deterministic Büchi automata, and operationally to STL formulas in the recurrence class that can be translated into a DBA [2510.05061]. It also notes that conditioning mainly on the current subgoal set does not fully capture longer-range future task requirements beyond the local stage. Partial observability and broader sim-to-real mismatch remain open problems, and the authors identify richer automata classes, including good-for-MDP nondeterministic Büchi automata, as future directions [2510.05061].

The supervisory-control precursor imposes a different set of limits. It is tabular and finite-state; automata and reward machines must be known or manually specified in advance; full observability is implicitly required; and the paper does not develop function approximation, deep RL, partial observability, or formal convergence and error bounds for the constrained learner [2201.00286]. LCNFQ, while extending automaton-constrained learning to continuous states and neural approximation, does not provide a fully formal theorem in the cited text establishing exact equivalence between maximizing shaped discounted return and maximizing LTL satisfaction probability [1809.07823].

A broader boundary of the topic concerns methods that are automaton structured but not hard-constrained. Automaton Distillation uses a DFA as a low-dimensional transfer medium for teacher Q-values, blending automaton-indexed transfer targets with ordinary Bellman targets by an annealed coefficient
\[
\beta(\omega,\sigma)=\rho^{\eta_\text{student}(\omega,\sigma)},
\]
so the automaton guides early learning but does not permanently constrain action feasibility [2310.19137]. PQLRM augments state with joint reward-machine memory in multi-objective RL and exploits deterministic automaton structure to update all automaton states from one transition, but the automata define rewards rather than legal or illegal behavior [2606.19134]. These lines suggest a wider interpretation in which automata can serve not only as hard constraint monitors, but also as transfer interfaces, sample-efficiency devices, or structured representations of non-Markovian reward.

Taken together, the literature supports a relatively precise characterization. ACQL, in the narrow sense, is Q-learning on an automaton-augmented state space with feasibility or acceptance structure entering the backup and policy extraction. Its central technical commitment is that symbolic task memory should be part of the value-learning state, rather than an external evaluator applied only after learning. Its current practical strength lies in combining temporal-logic task structure, stage-dependent safety, and off-policy goal-conditioned learning in continuous-control settings [2510.05061]. Its main unresolved issues concern expressiveness beyond DBA-representable tasks, stronger guarantees under function approximation, and the treatment of partial observability and broader real-world deployment conditions.

Source: https://www.emergentmind.com/topics/automaton-constrained-q-learning-acql