EvoFSM: Controllable Self-Evolution for Deep Research with Finite State Machines

Abstract: While LLM-based agents have shown promise for deep research, most existing approaches rely on fixed workflows that struggle to adapt to real-world, open-ended queries. Recent work therefore explores self-evolution by allowing agents to rewrite their own code or prompts to improve problem-solving ability, but unconstrained optimization often triggers instability, hallucinations, and instruction drift. We propose EvoFSM, a structured self-evolving framework that achieves both adaptability and control by evolving an explicit Finite State Machine (FSM) instead of relying on free-form rewriting. EvoFSM decouples the optimization space into macroscopic Flow (state-transition logic) and microscopic Skill (state-specific behaviors), enabling targeted improvements under clear behavioral boundaries. Guided by a critic mechanism, EvoFSM refines the FSM through a small set of constrained operations, and further incorporates a self-evolving memory that distills successful trajectories as reusable priors and failure patterns as constraints for future queries. Extensive evaluations on five multi-hop QA benchmarks demonstrate the effectiveness of EvoFSM. In particular, EvoFSM reaches 58.0% accuracy on the DeepSearch benchmark. Additional results on interactive decision-making tasks further validate its generalization.

• The paper introduces EvoFSM, a novel FSM-driven framework that decouples research workflow adaptation into macroscopic Flow and microscopic Skill levels.
• It employs atomic operations and a supervisory critic module to ensure precise, structure-preserving self-evolution while preventing instruction drift.
• Empirical results show significant accuracy gains and consistent performance improvements across diverse benchmarks, underlining its robust, scalable design.

EvoFSM: Structured and Controllable Self-Evolution of LLM Agents via Finite State Machines

Introduction

Recent advances in LLM-based agents for automation of deep research workflows have been impeded by static procedural paradigms and unstructured self-evolution methods. The EvoFSM framework (2601.09465) addresses these deficiencies by formalizing agent workflows as explicit Finite State Machines (FSMs) and implementing a two-dimensional, atomic-operation-based optimization process. EvoFSM decouples workflow adaptation into macroscopic (Flow/topology) and microscopic (Skill/node-specific behavior) levels, enabling targeted, high-precision system self-improvement while minimizing instruction drift and instability characteristic of unconstrained self-modification.

Structured Self-Evolution via FSM Formalism

The key formal innovation of EvoFSM is its rigorous FSM-based behavioral backbone, with states corresponding to specialized action phases (e.g., search, browse, verification) and transitions determined by runtime context and historical trajectories. Each node is equipped with explicit system prompts delineating skill boundaries, and all self-evolution is executed via small sets of atomic operators—ADD_STATE, DELETE_STATE, MODIFY_TRANSITION (on Flow) and REVISE_INSTRUCTION (on Skill). A critic module, instantiated as a supervisory LLM, evaluates outputs to diagnose specific capabilities or reasoning failures, triggering structure-preserving refinements rather than global, opaque rewriting.

Figure 1: Overview of the EvoFSM framework, depicting FSM initialization, structured atomic evolution, and self-evolving memory distillation.

This formulation sharply contrasts with prior unconstrained self-modification (see Figure 2), which often yields untraceable workflow corruption.

Figure 2: Unconstrained self-evolution produces instability and drift, mitigated by EvoFSM's explicit structure and guardrails.

Self-Evolving Memory for Continual Adaptation

EvoFSM complements structured evolution with a self-evolving memory that accumulates task-abstracted strategies and failure patterns. Successes are distilled into prior FSM configurations and atomic operation sequences, while detected failure cases become negative constraints for future evolutionary iterations. Each new query is bootstrapped from the top-$k$ relevant prior trajectories, enabling rapid inheritance of robust sub-workflows and systematic avoidance of previously identified dead-ends or bottlenecks.

Empirical Evaluation and Numerical Results

EvoFSM is benchmarked on five multi-hop QA datasets and outperforms all baselines, including Standard RAG, Agentic RAG, and module-enhanced pipelines such as Search-o1. Notably, EvoFSM achieves:

• 58.0% accuracy on the DeepSearch benchmark (Claude-4 backbone), exceeding the next-best method by 11.0%.
• Consistent improvement across diverse LLMs, demonstrating architecture-agnostic robustness.
• Superior performance generalization in interactive environments such as ALFWorld and WebShop.

  Figure 3: EvoFSM exhibits superior transferability on ALFWorld and WebShop benchmarks, both in final success rates and in average reasoning efficiency.

Ablation studies reveal the intertwined necessity of both structure (FSM topology) and atomic, controlled evolution. Removing either results in substantial drops, particularly for complex settings like DeepSearch, where loss of structured evolution causes a 15% degradation in accuracy relative to the full model.

Illustrative case analyses (see end for referenced cases) confirm that workflow-level bottlenecks (e.g., infinite loops) uniquely require Flow operations, while data precision or evidence extraction improvements are delivered via Skill local refinements. Cases involving legal or technical queries benefit from joint evolution across both axes.

Implications and Future Directions

The explicit modeling of research tasks as evolvable FSMs confers a set of critical theoretical and practical advantages:

• Behavioral stability: Deterministic state transitions and confined evolution operations prevent instruction drift and catastrophic corruption common in free-form rewriting agents.
• Interpretability: Each change is localized, explainable, and reversible, facilitating reliable auditing and diagnosis for production deployment.
• Continual learning: The experience pool's accumulation of both positive and negative examples supports rapid generalization to new domains and cold-start queries.

These attributes position EvoFSM as a credible path forward for LLM-based agents in open-ended research and multi-step reasoning environments. However, several open challenges remain:

• Dependence on large, proprietary LLMs for critic and operation modules limits efficiency and transparency. Training distilled, domain-optimized agents could alleviate this constraint.
• Critic reliability remains a bottleneck; hallucinated or incorrect evaluation cascades into systemic mislearning.
• Scalability of the memory pool requires development of memory consolidation and hierarchy, enabling efficient retrieval and elimination of redundant or obsolete strategies.

Conclusion

EvoFSM delivers a formal framework for controllable, interpretable self-evolution in LLM-based research agents by decomposing optimization into explicit FSM-driven Flow and Skill dimensions. Extensive results on deep research and cross-domain benchmarks demonstrate significant accuracy gains, high transferability, and stability, confirming the efficacy of structured self-evolution. The approach represents an essential advancement over both static pipelines and unconstrained self-improving agents. Open directions include critic robustness, memory management, and scaling to fully autonomous, long-horizon agents.