LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents

Abstract: As LLMs move from text completion toward autonomous agents, they remain constrained by the standard chat interface, which lacks private working memory. This raises a fundamental question: can agents reliably perform interactive tasks that depend on hidden state? We define Private State Interactive Tasks (PSITs), which require agents to generate and maintain hidden information while producing consistent public responses. We show theoretically that any agent restricted to the public conversation history cannot simultaneously preserve secrecy and consistency in PSITs, yielding an impossibility theorem. To empirically validate this limitation, we introduce a self-consistency testing protocol that evaluates whether agents can maintain a hidden secret across forked dialogue branches. Standard chat-based LLMs and retrieval-based memory baselines fail this test regardless of scale, demonstrating that semantic retrieval does not enable true state maintenance. To address this, we propose a novel architecture incorporating an explicit private working memory; we demonstrate that this mechanism restores consistency, establishing private state as a necessary component for interactive language agents.

• The paper proves that public-only chat agents cannot sustain hidden states, resulting in failures for non-trivial private state interactive tasks.
• It introduces a self-consistency testing protocol for Hangman and a Diagnosis Simulator, diagnosing error modes like leakage and over-confirmation.
• The study demonstrates that explicit private working memory architectures achieve significantly higher consistency and memory efficiency compared to retrieval-based models.

Necessity of Private Working Memory for Language Agents: A Technical Perspective on "LLMs Can't Play Hangman"

Formalization of Private State Interactive Tasks (PSITs) and Impossibility Results

This paper rigorously defines a new class of interactive problems—Private State Interactive Tasks (PSITs)—requiring an agent to generate, maintain, and act consistently with hidden state across iterative dialogues. The authors provide formal definitions: a Public-Only Chat Agent (POCA) is strictly constrained to the publicly observed history, with no internal persistent state, and a PSIT mandates that the agent’s behavior is a deterministic function of an initially sampled (but unobservable) secret $s$. The paper proves that POCAs cannot achieve both secrecy and consistency in any non-trivial PSIT. The impossibility result is formalized by demonstrating that, when two secrets remain consistent with the history, responses must diverge, but a POCA can only condition on the public transcript, forcing a contradiction.

This result generalizes to all standard LLM agents, including retrieval-based and memory-augmented systems, as any architecture that augments the LLM merely with public external context or semantically-indexed retrieval inherits the same restriction. Consequently, this impossibility theorem demarcates a sharp architectural boundary: existing LLM agents are structurally incapable of reliable PSIT performance.

Empirical Evaluation: Self-Consistency Testing Protocol

To empirically validate the theoretical limitation, the authors instantiate a self-consistency testing protocol for two representative PSITs: Hangman and a Diagnosis Simulator. The protocol evaluates whether an agent can persist a self-generated secret and act consistently even under controlled probing. After multiple rounds of interaction, the dialogue is forked, prompting the agent for direct confirmation of different candidate secrets that remain compatible with the public history. A consistent agent must affirm its true secret and correctly reject alternatives, ensuring both retention and secrecy.

Figure 1: Illustration of the self-consistency testing protocol for Hangman and the Diagnosis Simulator, and depiction of a memory-augmented architecture enabling persistent private state.

Notably, the design of the SCT protocol enables unambiguous diagnosis of partial or total failure modes. The authors define fine-grained error classes including Leakage (explicit state revelation), Over-Confirmation, State Substitution, and All Denial. This granular analysis exposes the varied failure behaviors of POCAs and existing “memory” agents.

Comparative Analysis of Agent Architectures

The empirical study systematically benchmarks standard LLMs, state-of-the-art memory-augmented agents (Mem0, A-Mem, LightMem, MemoryOS), Private Chain-of-Thought (CoT), and two proposed private working-memory architectures: Autonomous Memory Agents with tool APIs for state manipulation, and deterministic Workflow Agents where update operations are structurally enforced.

Statistical analysis over 50 episodes per condition and model (GPT-OSS and Qwen3 series, up to 235B parameters) demonstrates:

• Vanilla POCAs and retrieval-based agents fail PSITs almost universally, scoring close to zero on SCT.
• Private CoT agents achieve high self-consistency by persistently accumulating all reasoning traces, at the cost of significant context bloat.
• Private Working Memory agents, especially workflow-guided, achieve statistically significant and near-optimal self-consistency (up to 100% in Hangman and 96%+ in Diagnosis Simulator), while maintaining compact state.

  Figure 3: Distribution of failure modes across agent types, revealing varied error propensities including all-denial, leakage, and over-confirmation depending on model size and alignment.

The detailed failure mode landscape confirms that, when deprived of hidden state, LLM agents negotiate a suboptimal trade-off between hallucination and excessive denial, often breaking either secrecy or logical coherence depending on alignment and training biases.

Analysis of Memory Efficiency

A critical analysis is devoted to the context efficiency of private state management. The token cost of maintaining persistent state in Private CoT methods grows linearly (or worse) with dialogue length, due to accumulation of verbose intermediate traces. In contrast, explicit textual private working memory maintains an order-of-magnitude smaller footprint, typically <100 tokens, and does not balloon with continued interaction.

Figure 5: Evolution of private state size (in tokens) over dialogue turns, showing exponential inefficiency of cumulative-reasoning approaches relative to explicit memory updating.

This observation directly addresses scalability, demonstrating that efficient memory management is not possible through context accumulation or external retrieval alone, but requires explicit, persistent, agent-private memory that is both update- and compression-efficient.

Mode Collapse of Internal State Selection

The protocol also reveals that LLMs, even when sampled with stochastic decoding, manifest significant entropy collapse in their nominally "random" secret selection—the distribution of chosen secrets in both Hangman and the diagnosis task collapses sharply to a few domains or word types, reflecting pretraining priors and model biases.

Figure 2: Histograms of secret selection frequencies, demonstrating severe mode collapse towards high-frequency or prototypical concepts, conflicting with uniformity requirements of PSITs.

This has important implications when deploying LLM agents in scenarios requiring uniform or unbiased state generation, and highlights an independent limitation orthogonal to the persistence problem.

Theoretical and Practical Implications

The identification and empirical substantiation of the Generative-Retention Gap establishes that architectural innovation for agentic LLMs must go beyond context and retrieval-based strategies. True agentic interaction requires architectures supporting a full generative-retention-consistency loop, wherein the outputs of deliberation are available for future reasoning yet kept private from the user unless disclosure is necessary.

The necessity of such architectural separation is underscored by the observed tendency of safety-aligned LLMs to default to state leakage in ambiguous contexts—a critical failure in applications requiring robust confidentiality, multi-agent negotiation, adversarial play, or pedagogically honest role-play.

These results further inform cognitive model design: interfaces must facilitate explicit private state in a way amenable to controllable update, accessible retrospection (for auditability), and bounded complexity. Such design constraints align with modular cognitive architectures proposed in the agent literature, yet the paper demonstrates that their literal translation into LLM-based systems remains insufficient unless private state is structurally enforced.

Future Directions and Open Challenges

While the contribution clearly scopes the unsolved core of private-stateful agency, several open avenues remain. Extending SCT evaluation and architectural assessment to PSITs with high-dimensional, non-symbolic, or partially observable states (e.g., collaborative games, negotiation protocols, multi-agent sequential tasks) is an immediate next step. Moreover, balancing transparency for providers/regulators with robust privacy for users in real deployment contexts represents a significant design and governance challenge.

Techniques for private state auditing, checkpointed reasoning logs controllable for explainability, and mechanisms for bounded disclosure warrant further study. Additionally, integrating efficient update and retrieval strategies with private state modules—potentially leveraging neural-symbolic methods—may offer further scalability and robustness improvements.

Conclusion

By establishing the impossibility of solving PSITs with public-only chat agents and demonstrating the practical necessity and efficiency of explicit private working memory, this work places a hard architectural constraint on the design of reliable and consistent LLM agents. The empirical protocol and findings should serve as a reference for any future LLM agentic system deploying in interactive, confidential, or consistency-requiring tasks. Private, explicitly managed working memory is a non-negotiable component for next-generation agent architectures beyond framing tricks or retrieval-augmented context alone.