Memory-Augmented State Machine Prompting (MASMP)

• The paper presents MASMP as a novel framework that combines FSM-like prompt structuring with memory modules to overcome LLM hallucinations and tactical fragmentation.
• It leverages explicit state transitions and hierarchical behavior trees to maintain long-term strategic coherence in complex, real-time environments.
• Empirical validation in StarCraft II shows MASMP’s significant improvements in win rates and tactical consistency compared to traditional LLM agents.

Memory-Augmented State Machine Prompting (MASMP) is a computational framework that unifies the interpretability and strict action mapping of finite state machines with the semantic and adaptive flexibility of LLMs through explicit integration of memory modules. MASMP has emerged as a solution to persistent issues such as hallucinations, fragmented tactical execution, and the “Knowing-Doing Gap” in LLM-based agents, notably within real-time strategy (RTS) environments. By coupling state machine–structured prompt architectures with mechanisms for preserving and updating strategic memory, MASMP enables agents to maintain coherent, long-term behavior across complex decision cycles while preserving the reliability and transparency characteristic of classical symbolic approaches (Qi et al., 21 Oct 2025).

1. Architectural Principles

The foundational architecture of MASMP comprises two synergistic modules:

• State Machine Prompting: The LLM is guided by prompt templates that explicitly encode state transitions, policies, and hierarchical action trees reminiscent of finite state machines (FSMs) and behavior trees. Macro-strategic states (e.g., <defensive>, <aggressive>) are defined alongside natural language transition conditions and action mappings. Behavior trees organize decisions into selectors, sequences, and atomic rules, providing hierarchical control.
• Strategic Memory Module: A lightweight memory is maintained that stores context-dependent variables such as the current tactic, priority units, and temporal decisions from prior cycles. This repository is updated at each timestep, ensuring that long-term plans and strategic consistency persist even as immediate observations shift.

Algorithmically, the decision process extends from simple Markovian inference $a_t \sim$ LLM$_{\text{Generate}}(o_t)$ to a memory-augmented mapping: $$(s_t, a_t) \sim \mathrm{LLM}_{\text{Generate}}(o_t, M_{t-1}, \text{prompt}_{\text{sm}})$$ with the memory updated as $M_t = \text{Update}(M_{t-1}, s_t)$.

2. State Machine Prompting and Structured Decision Mapping

MASMP’s state machine prompting methodology induces the LLM to follow explicit, interpretable policies by simulating FSM-like transitions. Macro-strategic states and transition conditions are encoded in prompts, forming a natural language representation of FSM logic and behavior tree branching. For each game observation or step:

• The last strategic state is retrieved from memory.
• The current observation, state machine prompt, and strategy history are concatenated and input to the LLM.
• Parsed outputs provide both immediate actions and updates to strategic variables.
• The explicit mapping of state to action resolves the “Knowing-Doing Gap,” ensuring the LLM’s plans are faithfully executed.

Pseudocode for the MASMP workflow is:

last_strategy ← memory.get_latest() input_t ← CONCAT(o_t, prompt_sm, last_strategy) output ← LLM_Generate(input_t) strategies ← StrategyExtractor.extract_strategies(output) if strategies ≠ ∅: memory.add_memory(strategies[0], t) execute(output)

This design eliminates the ambiguity and fragmentation common in unconstrained neural agents.

3. Memory Mechanisms and Long-Term Tactical Coherence

The strategic memory module serves as an external repository for long-term contextual data. Unlike pure neural memory buffers, MASMP tracks explicit tactical choices, unit priorities, and scenario-dependent instructions to maintain coherence over extended multi-step planning. Memory is updated after each prompt cycle using extraction mechanisms (e.g., regex-based parsing of strategic outputs), preserving historical context and enabling the agent to adapt plans non-greedily and avoid repeated short-term mistakes.

Such explicit memory management is essential for environments with partial observability (e.g., fog of war) and delayed rewards, where optimal strategies depend on history and accumulated knowledge rather than immediate sensory input.

4. Empirical Validation and Performance Metrics

MASMP has been empirically validated in the StarCraft II environment (LLM-PySC2, Simple64 map), demonstrating significant advantages over baseline LLM agents—particularly regarding tactical execution and strategic adaptation. Experimental data show:

• MASMP achieves a 60% win rate against the hardest built-in AI (Lv7), compared to 0% for baseline LLM agents.
• For intermediate difficulty (Lv6), MASMP’s win rate rises to 80%, retaining 100% win rates at lower levels.
• Case studies reveal robust state transitions (e.g., from <defensive> to <aggressive> and back) and avoidance of greedy local decisions in unit production, supporting both long-term planning and interpretability.

These results establish the approach’s effectiveness in bridging neural semantic understanding with symbolic reliability.

Agent System	Win Rate (Lv7)	Win Rate (Lv6)
MASMP	60%	80%
Baseline LLM (LLM-PySC2)	0%	0%

5. Relationship to Neural-Symbolic and Memory-Augmented Methods

MASMP represents a hybrid neuro-symbolic paradigm, leveraging the creative and semantic generalization of LLMs alongside the deterministic action mapping and transparency of symbolic FSMs. The memory module acts as a bridge between these paradigms, converting learned or inferred strategies into explicit, reusable symbolic entries. MASMP’s strict state–action mapping brings interpretability and reliability to agent decisions, overcoming the instability, hallucinations, and fragmented reasoning endemic to pure neural decision models.

A plausible implication is that similar MASMP frameworks can be generalized to a range of sequential decision-making domains where external memory and structured prompt architectures can reconcile neural flexibility with symbolic control requirements.

6. Broader Applications and Implications

The MASMP approach extends beyond gaming and is applicable to:

• Multi-agent coordination, where shared memory modules can mediate distributed tactical planning.
• Robotics and autonomous control systems that require consistent policy execution with historical state tracking.
• Military, financial, or real-time logistics planning, where strategic context and adaptation over long horizons are crucial.

The method offers a pathway toward interpretable, reliable LLM agents for complex, temporally extended environments, establishing a model for further research into the integration of symbolic and neural mechanisms.

7. Integration with Related Frameworks and Future Directions

MASMP draws upon and integrates advances in memory-augmented agent architectures, retrieval-augmented prompting, and structured memory systems (e.g., hierarchical task trees (Ye, 11 Apr 2025), personalizable retrieval modules (Sarch et al., 2023)). A trend towards increasingly sophisticated memory mechanisms—such as graph-aware extension, meta-optimization with “mistake notebooks” (Wu et al., 26 Aug 2025), and lifelong learning in analog or digital substrates (Mao et al., 2022)—suggests a convergence toward powerful hybrid systems capable of continual adaptation, self-reflection, and robust execution in dynamic environments.

This suggests that future MASMP systems will likely incorporate more advanced memory architectures (e.g., DAG-based state tracking), enhanced retrieval and synthesis methods, and more refined prompt strategies to further improve reliability, efficiency, and contextual adaptation across domains requiring both flexibility and rigorous control.