Can LLM Agents Infer World Models? Evidence from Agentic Automata Learning

Abstract: We propose agentic automata learning to evaluate the extent to which tool-calling LLM agents can uncover hidden environments through interaction. In our setup, an agent should uncover a hidden deterministic finite automaton (DFA) by interacting with an oracle through (1) membership queries ("Does this string belong to the target language?") and (2) equivalence queries ("Is this the target DFA?"). This yields a scalable testbed with controlled task complexity, measurable interaction efficiency, and strong baselines (classic automata-learning algorithms). Evaluating state-of-the-art LLMs, we find that performance drops sharply as DFA size increases. Reasoning models are markedly stronger than non-reasoning models, yet trajectory analyses reveal recurring failures in query planning, evidence integration, and hypothesis construction. Overall, our results show that current LLM agents can sometimes perform non-trivial interactive discovery, but remain far less robust and efficient than classic algorithms for the task.

• The paper evaluates LLM agents' ability to infer hidden DFA structures using membership and equivalence queries.
• It demonstrates that performance sharply declines with increased DFA complexity, with reasoning-enabled models faring better.
• The study highlights recurring planning and reasoning errors, contrasting LLM strategies with optimal classical algorithms.

Agentic Automata Learning: Evaluating LLM World Model Inference via DFA Discovery

Problem Formulation and Framework

The paper establishes agentic automata learning as a rigorous, scalable testbed for assessing whether tool-calling LLM agents can infer latent structure of environmental dynamics through interaction. The agent is challenged to reconstruct a hidden DFA by leveraging two query modalities: membership queries (MQs), which probe string acceptance, and equivalence queries (EQs), which submit candidate DFA hypotheses for validation, receiving either confirmation or a counterexample pinpointing behavioral divergence.

Figure 1: Formalized agentic automata learning pipeline, detailing the two query types agentic LLMs employ to uncover an unknown automaton by iterative interaction.

This setup provides controlled complexity (via DFA state count), measurable interaction efficiency, and comparison against classic algorithms (L* and TTT) with polynomial convergence guarantees [angluin1987learning, isberner2014ttt]. Evaluating LLMs within this environment directly tests model-based discovery capabilities beyond simple pattern matching or local heuristics.

Experimental Design

The methodology encompasses procedural generation of binary DFAs of varying complexity (2–9 states), ensuring minimality and accessibility, using Boltzmann sampling [BassinoNicaud2007]. Six models are evaluated: Gemini 3.1 Pro, DeepSeek-V4-Pro, Gemini-3-Flash-Preview (reasoning enabled), GPT-5.4 (non-reasoning), Gemini-3.1-Flash-Lite, and Llama-3.3-70B-Instruct, spanning diverse architectures, parameter scales, and reasoning token configurations.

Performance is quantified via success rate (fraction of instances solved within budget) and discovery efficiency (excess queries over TTT). Auxiliary analyses dissect query informativeness and error typology, distinguishing reasoning from planning failures.

Key Results

Complexity Scaling and Model Gaps

Model success degrades sharply with DFA complexity. For DFAs with 9 states, no model exceeds 25% success rate; classic algorithms consistently achieve 100%. Gemini 3.1 Pro’s query efficiency lags TTT by 45.8% in successful runs at highest complexity. Reasoning-enabled models demonstrate marked superiority: on 4–5 state DFAs, Gemini 3.1 Pro attains 85% success, Gemini-3-Flash-Preview 15%, and all non-reasoning models fail completely.

(Figure 2)

Figure 2: Success rate and query efficiency for all models as DFA complexity increases, highlighting the pronounced gap between LLMs and classical learners.

Error Patterns and Trajectories

Robust error decomposition reveals substantial reasoning and planning deficits across models. Reasoning failures occur despite adequate data acquisition, indicating inability to synthesize evidence into correct hypotheses. Planning failures predominate in weaker models, reflecting inadequate information-gathering strategies.

(Figure 3)

Figure 3: Distribution of planning versus reasoning failures across model classes, demonstrating that stronger models more often fail due to reasoning while weaker models struggle with planning.

Notably, LLM agents increasingly issue non-informative queries as context size grows—after 60 steps, even the most consistent model’s queries are 20% redundant. Classic algorithms remain optimal via structured hypothesis refinement and evidence tracking.

(Figure 4)

Figure 4: Non-informative query rate as interaction proceeds, indicating LLM agent breakdown in using accumulated evidence for planning.

Trajectory Analysis and Algorithmic Divergence

LLMs do not imitate classical algorithms. Query sequences never match L* or TTT; often, Gemini 3.1 Pro proposes candidate DFAs with fewer states than prior hypotheses, violating monotonic refinement. Excessive reliance on equivalence queries, beyond classical bounds, further underscores divergence from formal learning strategies.

Practical and Theoretical Implications

The results decisively demonstrate that advanced LLMs possess limited capacity for non-trivial world model induction through interactive discovery, remaining far below classical learners in both robustness and efficiency. The recurring planning and reasoning errors, together with poor information integration as interaction context expands, expose major bottlenecks in agentic tool-based inference. These deficiencies persist even as query budgets are increased, implicating fundamental limitations in context tracking, credit assignment, and belief updating.

Practically, agentic automata learning provides a scalable, formal benchmark for evaluating agentic reasoning in tool-calling environments. It enables precise measurement of task complexity dependence, query efficiency, and evidence integration, facilitating cross-model and cross-algorithm comparison. The framework can readily incorporate additional complexities: stochasticity, noisy or partial feedback, or weaker oracle signals, making it broadly extensible.

Theoretically, the results suggest that current LLMs lack strong capabilities for integrating sequential evidence and updating explicit latent models—a critical requirement for robust reinforcement learning, planning, and autonomous interaction. Model architectures and training modalities need substantial augmentation to approach the efficiency, memory management, and structured generalization of classical learners.

Directions for Future Research

Several directions emerge for advancing agentic world model inference in LLM settings:

• Extension to Nondeterministic or Partially Observable Settings: Generalizing the benchmark to include NFA or stochastic automata would further stress evidence integration and planning under uncertainty.
• Noisy/Partial Feedback and Weak Oracle Queries: Relaxing the oracle to provide incomplete or unreliable counterexamples would test robustness of interactive learning and belief revision.
• Training Regimes Based on Interactive Discovery: Using agentic automata learning as a reinforcement learning environment could drive explicit model-building in LLM agents.
• Memory and Context Management Architectures: Introducing structured persistent memory, graph-based evidence integration, or model-based state tracking could mitigate non-informative query bloat and reasoning failures.
• Benchmarking Costs and Evaluation Scalability: Analyses of runtime, token usage, and monetary expenditure indicate that agentic evaluation frameworks may require innovation in efficient interaction management and adaptive budget allocation.

Conclusion

Agentic automata learning demonstrably exposes fundamental limitations in tool-calling LLM agents’ capacity for environment model inference via interactive queries. While reasoning-capable models can achieve nontrivial success at moderate complexities, performance collapses as task difficulty grows. Planning, reasoning, and robust use of evidence in discovery trajectories remain unsolved challenges, with LLMs failing to emulate classical learners’ efficiency and reliability. The agentic automata learning framework constitutes a rigorous, extensible paradigm for high-fidelity evaluation and targeted improvement of interactive environment modeling in LLM-based agents (2606.16576).