Agentic Automata Learning

• Agentic automata learning is a benchmark that recasts DFA learning as an active tool-use problem for LLM agents to infer hidden world models.
• The framework employs membership and equivalence queries to iteratively refine explicit automata hypotheses, enabling controlled exploration of latent environments.
• Empirical findings reveal that LLM agents face scalability challenges, with success rates decreasing as DFA complexity increases, highlighting planning versus reasoning gaps.

Searching arXiv for the cited works and adjacent literature to ground the article. {"query":"Agentic Automata Learning LLM agents infer world models DFA membership queries equivalence queries arXiv", "max_results": 10} Agentic automata learning is a line of research concerned with whether an agent can infer hidden environment structure through interaction, and, more broadly, how agentic systems can learn structured controllers from trajectory data rather than behave as static prompt-response artifacts. In its most precise current usage, the term refers to the active-learning setting introduced in “Can LLM Agents Infer World Models? Evidence from Agentic Automata Learning,” where a tool-calling LLM agent must uncover a hidden deterministic finite automaton (DFA) using membership and equivalence queries (Menaged et al., 15 Jun 2026). A broader, interpretive usage connects this benchmark to work on self-evolving agents, code-level reasoning modules, multi-turn tool-use RL, environment synthesis, and interpretable transition-structure learning, all of which treat agent behavior as stateful, sequential, and revisable rather than as a single stateless text-generation step (Yan et al., 1 Jul 2026).

1. Formal problem setting

The formal core of agentic automata learning is classical active learning of deterministic finite automata, recast as a tool-use problem for LLM agents. The hidden target is a minimal DFA

$A^\star \in \mathcal{D}_\Sigma$

over a known alphabet $\Sigma$, inducing a target language $L^\star \subseteq \Sigma^\star$. A DFA is represented in the usual form

$A = (Q,\Sigma,\delta,q_0,F),$

with finite state set $Q$, transition function $\delta$, initial state $q_0$, and accepting set $F$ (Menaged et al., 15 Jun 2026).

Interaction proceeds through two oracle-access mechanisms. A membership query is

$\mathrm{MQ}:\Sigma^\star \mapsto \{0,1\},$

returning whether a word belongs to the hidden language. An equivalence query is

$$\mathrm{EQ}:\mathcal D_\Sigma \mapsto \{\top\}\cup \Sigma^\star,$$

returning $\Sigma$0 when the proposed DFA is exact and otherwise a counterexample from the symmetric difference. Success is defined by issuing an equivalence query with a hypothesis $\Sigma$1 such that

$\Sigma$2

This makes the task one of exact identification rather than approximate prediction (Menaged et al., 15 Jun 2026).

The benchmark motivation is that a hidden DFA is a controlled proxy for a latent world model. Learning it requires experiment design, evidence integration, latent-state abstraction, and hypothesis revision. This suggests that the task isolates a minimal form of interactive scientific discovery: the agent must infer the environment’s transition structure from intervention outcomes rather than from a static supervised dataset (Menaged et al., 15 Jun 2026).

2. Benchmark protocol and complexity control

The benchmark is built specifically for tool-calling LLM agents. The agent receives two tools: is_word_in_language, which implements membership queries, and evaluate_dfa_candidate, which implements equivalence queries. At each step, the model must emit exactly one TOOL_ACTION JSON block with no extra natural language. Proposed DFAs are represented as explicit JSON objects containing the list of states, alphabet, start state, accept states, and transition list. The interaction history passed to the model includes previous membership queries, previous equivalence queries and returned counterexamples, and previous visible hypothesis DFAs. Chain-of-thought is not preserved across turns; only externally visible outputs are carried forward (Menaged et al., 15 Jun 2026).

Difficulty is controlled by the number of states in the minimal hidden DFA. The experimental setup uses a binary alphabet and 80 task instances, evenly split across four complexity bins: 2–3, 4–5, 6–7, and 8–9 states. Hidden DFAs are generated procedurally by Boltzmann sampling from accessible DFAs over a binary alphabet, followed by rejection sampling to remove non-minimal automata. When an equivalence query fails, the oracle returns a short random counterexample sampled deterministically from lengths between $\Sigma$3 and $\Sigma$4, where $\Sigma$5 is the shortest distinguishing length (Menaged et al., 15 Jun 2026).

The query budget is tied to symbolic baselines. For each hidden DFA, the budget is set to twice the number of queries required by the better of $\Sigma$6 and TTT on that instance. The paper reports the classical upper bounds

$\Sigma$7

for $\Sigma$8 and

$\Sigma$9

for TTT, where $L^\star \subseteq \Sigma^\star$0 is the number of states and $L^\star \subseteq \Sigma^\star$1 the maximum counterexample length. This budget construction guarantees that a correct solution exists within the allowed interaction horizon (Menaged et al., 15 Jun 2026).

The resulting testbed is scalable, procedurally generated, and directly comparable to exact symbolic learners. A plausible implication is that it serves simultaneously as a benchmark for LLM agency and as a diagnostic environment for state-abstraction quality.

3. Empirical performance and scaling behavior

The central empirical finding is that current LLM agents can sometimes perform non-trivial interactive discovery, but degrade sharply as DFA size increases. Classical baselines $L^\star \subseteq \Sigma^\star$2 and TTT solve 100% of instances. Among evaluated LLMs, Gemini 3.1 Pro is strongest, with success rates of 100% on 2–3 states, 85% on 4–5, 65% on 6–7, and 25% on 8–9. DeepSeek-V4-Pro reaches 95%, 70%, 45%, and 10% on the same bins. All remaining evaluated models fall to 0% by 6–7 states, and no model exceeds 25% on 8–9 states (Menaged et al., 15 Jun 2026).

Overall success across all 80 tasks shows the same pattern: Gemini 3.1 Pro reaches 68.75%, DeepSeek-V4-Pro 55.00%, Gemini-3-Flash (thinking) 17.50%, GPT-5.4 without thinking 13.75%, Gemini-3.1-Flash-Lite 7.50%, and Llama-3.3-70B 6.25%. The reasoning versus non-reasoning gap is explicit. At 4–5 states, Gemini 3.1 Pro reaches 85% and DeepSeek-V4-Pro 70%, while GPT-5.4, Flash-Lite, and Llama-3.3 are all at 0% (Menaged et al., 15 Jun 2026).

Query efficiency also remains substantially worse than symbolic algorithms. On successful runs, Gemini 3.1 Pro is close to TTT on small DFAs but much worse on larger ones. At 8–9 states, it averages 25.40 more tool calls than TTT, interpreted in the paper as about 45.8% more queries. DeepSeek-V4-Pro is less efficient across all bins and reaches 39.00 excess tool calls at 8–9 states. The benchmark is also expensive: at larger DFA sizes, runs can reach roughly 20 hours, 30 million tokens, and \$L^\star \subseteq \Sigma^\star$31,200 across 480 runs (Menaged et al., 15 Jun 2026).

Budget matters but does not erase the symbolic gap. Under a budget factor of 2, the overall success rates are those just noted; under a budget factor of 1, Gemini 3.1 Pro drops to 18.75% and DeepSeek-V4-Pro to 0%. This indicates that additional interaction budget helps, but does not convert heuristic agent behavior into exact active-learning performance (Menaged et al., 15 Jun 2026).

4. Trajectory analysis, failure modes, and misconceptions

A distinctive contribution of the benchmark is trajectory-level diagnosis. Failures are partitioned into planning failures and reasoning failures. A planning failure occurs when the model fails and none of the passive learners RPNI, EDSM, or Blue-Fringe can recover the hidden DFA from the examples it collected; the interpretation is that the agent gathered insufficiently informative evidence. A reasoning failure occurs when the model fails but at least one passive learner can recover the hidden DFA from the collected evidence; the interpretation is that the information was available, but the model failed to construct the correct automaton from it. For Gemini 3.1 Pro, the breakdown is 68.75% success, 21.25% reasoning failure, and 10.00% planning failure. For weaker models, planning failures dominate (Menaged et al., 15 Jun 2026).

Non-informative queries expose evidence-integration failures. A membership query is non-informative if it repeats a word already queried; an equivalence query is non-informative if it duplicates a prior hypothesis or proposes a DFA already contradicted by known evidence. These rates rise sharply with step count. Around step 60, DeepSeek-V4-Pro issues non-informative queries about 20% of the time, Gemini 3.1 Pro about 46%, GPT-5.4 about 42%, Gemini-3-Flash roughly 68%, and Flash-Lite and Llama near 98–99%. Classical learners remain at 0% non-informative queries by construction (Menaged et al., 15 Jun 2026).

The benchmark also addresses a common misconception: strong reasoning models are not merely replaying $L^\star \subseteq \Sigma^\star$4 or TTT. The paper reports no exact trajectory matches to classical learners on any task instance. Gemini 3.1 Pro exceeds the classical bound on the number of equivalence queries in 92.5% of instances, and it produces a non-monotonic hypothesis sequence in all interactions, whereas classical learners refine hypotheses monotonically (Menaged et al., 15 Jun 2026).

A second misconception is that agentic automata learning is synonymous with classical automata learning. In the strict benchmark sense, the task is exactly classical active DFA learning recast as an agent tool-use problem. But in adjacent agent-learning work, the learned object is often not an explicit automaton. It may instead be a structured reasoning module, workflow harness, skill library, or multi-surface controller. This distinction is central to the broader literature (Yao et al., 7 Oct 2025).

5. Broader interpretations: structured controllers, workflows, and agent evolution

Several adjacent papers position agent behavior as a structured transition system even when no DFA is explicitly inferred. “ARM: Discovering Agentic Reasoning Modules for Generalizable Multi-Agent Systems” defines a step-generator module

$L^\star \subseteq \Sigma^\star$5

and a meta-policy

$L^\star \subseteq \Sigma^\star$6

then discovers code-level reasoning modules by reflection-guided evolutionary search over valid Python programs. The best discovered ARM performs explicit stages such as generating candidates, critiquing them, comparing them, checking for flaws, and falling back based on a severity threshold. This is not formal automata induction, but it is a reusable reasoning operator with explicit control flow and a stable external interface (Yao et al., 7 Oct 2025).

“Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents” generalizes the structured-policy view from the deployment side. It defines a deployed agent as

$L^\star \subseteq \Sigma^\star$7

and models an agent trajectory as

$L^\star \subseteq \Sigma^\star$8

Here the learned or revised object is not only model weights but also the harness, memory, tool schemas, and governance layer. The paper’s claim is that self-evolution is a control problem over multiple intervention surfaces rather than “always run RL on the base model” (Yan et al., 1 Jul 2026).

Other works reinforce this broader interpretation. MUA-RL formulates multi-turn user-interacting tool use over tool space $L^\star \subseteq \Sigma^\star$9, message space $A = (Q,\Sigma,\delta,q_0,F),$0, and observation space $A = (Q,\Sigma,\delta,q_0,F),$1, with sparse binary reward only for task completion under policy constraints; the learned behavior is described as an implicit state machine over dialogue and tool history (Zhao et al., 26 Aug 2025). EvoTrainer co-evolves policy versions and training harnesses, treating diagnosis, backtesting, and branch promotion as structured control over the training process itself (Chen et al., 2 Jun 2026). AutoForge synthesizes executable environments $A = (Q,\Sigma,\delta,q_0,F),$2, workflow DAGs $A = (Q,\Sigma,\delta,q_0,F),$3, and environment-level RL objectives, making agent training occur in procedurally generated, automata-like interactive worlds (Cai et al., 28 Dec 2025). Agential AI learns explicit discrete temporal structure through BSVs, DSVs, and CSVs, then plans backward over the learned graph; it is closer to symbolic transition-system induction than to LLM prompting (Erden et al., 28 Jan 2025).

Work	Learned or controlled object	Relation to agentic automata learning
(Menaged et al., 15 Jun 2026)	Explicit hidden DFA	Formal active automata-learning benchmark
(Yao et al., 7 Oct 2025)	Code-level reasoning module	Automaton-like control flow without explicit DFA
(Yan et al., 1 Jul 2026)	Composite policy, harness, memory, tools	Structured agent evolution from trajectories
(Zhao et al., 26 Aug 2025)	Multi-turn tool-use policy	Implicit dialogue-state controller
(Cai et al., 28 Dec 2025)	Executable workflow environment	Automata-like environment synthesis

This broader cluster suggests that “agentic automata learning” now names both a strict formal benchmark and a wider research agenda around structured agent policies, though the latter usage remains interpretive in several papers.

6. Limitations, controversies, and likely research directions

The benchmark formulation in (Menaged et al., 15 Jun 2026) is deliberately narrow. It is restricted to deterministic finite automata, synthetic hidden environments, exact oracle feedback, a binary alphabet in experiments, and target classes up to 9 states. Equivalence queries are stronger than the feedback available in many real settings, and the evaluation is expensive in tokens, runtime, and money. The paper explicitly proposes extensions to non-deterministic or stochastic environments, noisy, partial, delayed, or incorrect oracle feedback, weaker forms of supervision than equivalence queries, and use as a training environment via reinforcement learning (Menaged et al., 15 Jun 2026).

In the broader agent-learning literature, the main open issue is how to preserve structured, credit-assignable trajectories and turn them into governed updates. “Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents” argues that enterprise-scale self-evolving agents require three missing pillars: a standardized agent trajectory data protocol, an enterprise-grade governed data proxy, and a unified agent evolution control plane (Yan et al., 1 Jul 2026). A plausible implication is that formal automata-style methods will remain difficult to deploy at scale unless trajectory representation, replayability, and intervention selection are first-class system primitives.

Another recurring issue is evaluator drift. “Towards Agentic Self-Learning LLMs in Search Environment” argues that the source of reward signals and the scale of agent task data are the two key determinants of scalable self-learning, and that a frozen Generative Reward Model induces reward hacking as the task generator moves off-distribution. Continual verifier training and a small late-stage injection of real verification data are reported to mitigate this failure mode (Sun et al., 16 Oct 2025). This suggests that, outside exact oracle settings, agentic automata learning may require adaptive verifiers rather than fixed teachers.

A final unresolved question concerns representation. The strict DFA benchmark offers exactness, strong baselines, and measurable interaction efficiency. Adjacent systems papers offer richer trajectories, tools, users, and long-horizon tasks, but usually stop short of explicit state minimization, equivalence-query learning, or canonical symbolic transition models. The field therefore sits between two poles: exact formal identification of small hidden machines, and optimization-driven learning of large hybrid controllers. Current evidence indicates that LLM agents can perform limited interactive world-model inference on small deterministic environments, but remain less robust and less efficient than classical symbolic learners, while the broader systems literature is building the trajectory protocols, workflow abstractions, and control loops that a more general agentic automata-learning program would likely require (Menaged et al., 15 Jun 2026).