Belief Stochastic Game Model

• Belief stochastic game model is a framework that separates observable game state and hidden information by managing belief states externally.
• It employs a constraint-based approach using CSPs and a probabilistic extension with BP to efficiently represent and update hidden states.
• Empirical evaluations in games like Mini-Stratego and Goofspiel show that logical filtering often performs comparably to probabilistic refinements.

The belief stochastic game model encompasses a class of stochastic games in which state estimation under partial information is explicitly handled by the game model itself, rather than embedded inside agent logic. This modeling framework facilitates agent design by exposing externally managed belief states, thereby making decision-making algorithms more portable and domain-generic. The model is particularly focused on games with hidden piece identities—such as variants of Stratego or card games—where agents observe only partial information, and uncertainty about the hidden state is crucial for optimal play. Two principal methods for belief representation are considered: a constraint-based approach using Constraint Satisfaction Problems (CSPs) to enumerate feasible hidden states, and a probabilistic extension that employs Belief Propagation (BP) to compute approximate marginals over these states. Comparative evaluations illustrate that constraint-based beliefs are often sufficient for effective agent performance, with probabilistic refinements offering only marginal improvements in several practical settings (Morenville et al., 25 Jul 2025).

1. Model Architecture: Delegated State Estimation

The Belief Stochastic Game model (henceforth, Belief-SG) separates the observable and hidden portions of the game state. The observable component consists of all fully public features (e.g., board layout, revealed cards, or pointer to current player). The hidden component captures uncertainty, typically in the form of unknown piece or card identities. Formally, the game infrastructure maintains a belief state for each agent, representing coherent possibilities for the hidden part of the state given the observed history.

This delegation of state estimation enables a more universal agent design interface:

• Agents receive a belief state (either as a set of feasible assignments or as marginal distributions), abstracting away game-specific inference, counterfactual reasoning, or Bayesian updating.
• The belief update and pruning due to public moves is handled consistently inside the game engine, obviating agent-side logic duplication or mismatches.

This modular approach supports plug-and-play agent models and aligns with contemporary efforts to cleanly separate inference and control in AI architectures.

2. Constraint-Based Belief Representation

The constraint-based approach encodes the uncertainty over hidden identities as a collection of CSPs. Let 𝒫 denote the set of hidden pieces and 𝒯 the set of types, with each type t corresponding to a set of possible identities 𝒱ₜ. Each piece p ∈ 𝒫 is associated, via a type function θ, to its type θ(p) ∈ 𝒯 and a domain Dₚ ⊆ 𝒱₍θ(p)₎ representing feasible identities after applying all public information and inferred constraints.

Crucially, global cardinality constraints (GCC) are imposed to ensure feasibility regarding multiplicities of piece identities—e.g., in Stratego, there can only be a finite number of each rank. The resulting constraint-based belief structure is a set of all assignments to (p, Dₚ) that respect type and multiplicity constraints:

Element	Description	Notation
Hidden pieces	Items with concealed identity	𝒫
Types	Set of types for pieces	𝒯
Domains	Feasible identities per piece	Dₚ ⊆ 𝒱₍θ(p)₎
GCC	Cardinality constraints per type	GCCₜ

As game events occur (moves, captures), the domains Dₚ are pruned accordingly, and constraint propagation through GCCs can trigger additional reductions or determinations.

This representation is combinatorial but non-probabilistic: it compactly specifies which states are still logically feasible but not how likely they are.

3. Probabilistic Belief Representation via Belief Propagation

To augment the CSP-based scheme, the model supports a probabilistic extension through Belief Propagation (BP). By recasting the CSP as a factor graph, variables correspond to hidden pieces and factors to constraints (including GCC decomposed into per-identity count factors). BP then performs message passing to approximate marginal probabilities μₚ(v) for piece p and identity v:

$$\mu_p(v) \propto \prod_{f \in N(p)} m_{f \rightarrow p}(v)$$

where m_{f \rightarrow p}(v) is the message from factor f to variable p about v, and N(p) is the set of adjacent factor nodes.

BP's marginals can inform determinization (sampling of full states for simulation) or provide a prioritization among feasible assignments within an agent’s planning or search algorithm.

Direct enumeration to compute true marginals is infeasible in large games, but BP offers an efficient and typically accurate heuristic for sampling.

4. Integration With General-Purpose Agents

Agents in Belief‑SG are informed by the externally managed belief. Two agent architectures evaluated in the model are:

• Pure Monte Carlo (PMC): Samples multiple hidden state assignments per belief (using either uniform or probability-guided sampling), simulates rollouts for each legal action, and chooses actions based on aggregate outcomes.
• Decoupled UCT (DUCT): For each determinization, constructs separate search trees, aggregates recommendations, and combines them for action selection, adapting the Upper Confidence bounds applied to Trees (UCT) framework to simultaneous-move and uncertainty settings.

When using constraint-based beliefs, assignments are drawn uniformly among all consistent possibilities. Under BP, assignments are sampled proportional to estimated marginals μₚ(v).

This structure allows the agent interface to be decoupled from the underlying game's logic for state estimation, enabling reusable and scalable decision components.

5. Empirical Evaluation and Comparative Findings

The framework was evaluated on two imperfect-information games: Mini-Stratego and Goofspiel. Agents using either constraint-based beliefs ("C") or BP-provided probabilistic beliefs ("P") were compared within both PMC and DUCT planning wrappers, keeping the simulation budget fixed (e.g., 10 determinizations, 1,000 rollouts/action).

Results show that:

• Performance between C and P agents (within either PMC or DUCT) is very similar: win rates and strategy quality are nearly indistinguishable.
• The overall planning paradigm (DUCT vs PMC) dominates the marginal gains from probabilistic refinement: tree search methods outperform flat sampling, but probabilistic state estimation does not yield systematic advantages over precise logical filtering.
• The marginal cost of BP (computation and message passing) is not consistently offset by better planning outcomes for the agent, at least within determinization-based frameworks.

These findings indicate that in many practical scenarios, constraint-based beliefs suffice for effective decision-making, and probabilistic marginalization (via BP) often yields diminishing returns relative to its complexity.

6. Future Directions and Practical Implications

The Belief‑SG formulation provides a clear abstraction layer between game-model-managed uncertainty and agent-side planning. This enables:

• Easy benchmarking of planning algorithms across varied game types without recoding inference or reasoning primitives.
• Efficient knowledge transfer of agent logic to new domains, provided the game exports beliefs in the prescribed format.
• Principled experimentation with advanced inference methods (e.g., BP, MCMC) and assessment of their actual value for performance in complex imperfect-information games.

A plausible implication is that for determinization-based agents and sufficiently rich logical constraints, further probabilistic belief refinement may be unnecessary, as the logical filter provided by CSPs already encodes most relevant information. However, for learning-based or value function approximation agents, or games with noisy observations rather than hidden identities, probabilistic beliefs may have greater impact.

The Belief-SG model sets a foundation for comparative analyses of state estimation representations and for the design of portable, inference-agnostic planning agents in stochastic games with partial information (Morenville et al., 25 Jul 2025).