LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo

Abstract: We introduce LudoBench, a benchmark for evaluating LLM strategic reasoning in Ludo, a stochastic multi-agent board game whose dice mechanics, piece capture, safe-square navigation, and home-path progression introduce meaningful planning complexity. LudoBench comprises 480 handcrafted spot scenarios across 12 behaviorally distinct decision categories, each isolating a specific strategic choice. We additionally contribute a fully functional 4-player Ludo simulator supporting Random, Heuristic, Game-Theory, and LLM agents. The game-theory agent uses Expectiminimax search with depth-limited lookahead to provide a principled strategic ceiling beyond greedy heuristics. Evaluating six models spanning four model families, we find that all models agree with the game-theory baseline only 40-46% of the time. Models split into distinct behavioral archetypes: finishers that complete pieces but neglect development, and builders that develop but never finish. Each archetype captures only half of the game theory strategy. Models also display measurable behavioral shifts under history-conditioned grudge framing on identical board states, revealing prompt-sensitivity as a key vulnerability. LudoBench provides a lightweight and interpretable framework for benchmarking LLM strategic reasoning under uncertainty. All code, the spot dataset (480 entries) and model outputs are available at https://anonymous.4open.science/r/LudoBench-5CBF/

• The paper introduces LudoBench with 480 spot scenarios to assess LLMs' rule compliance and strategic decision-making in a stochastic board game.
• It employs a systematic methodology combining baseline agents and game-theory metrics to benchmark model performance and expose compliance tiers.
• Findings reveal that LLMs, despite high rule adherence, show incomplete strategic alignment with GT optimal play and are vulnerable to prompt-induced shifts.

LudoBench: A Comprehensive Benchmark for Evaluating LLM Strategic Decision-Making in Stochastic Multi-Agent Board Games

Motivation and Context

The assessment of LLMs in controlled environments has traditionally focused on deterministic games (chess, Go), incomplete-information domains (poker), and cooperative settings (Diplomacy), with little systematic evaluation of stochastic multi-agent games requiring multi-asset coordination and probabilistic reasoning. "LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo" (2604.05681) directly addresses this gap, introducing a benchmark situated in Ludo—a dice-driven, multi-agent, multi-piece race game exhibiting complexity absent from prior benchmarks. The framework operationalizes critical experimental controls: a spot-based scenario method that isolates specific decision points, a history-conditioned design to probe prompt sensitivity, and disjoint agent baselines to establish performance boundaries.

Benchmark Construction and Methodology

LudoBench is composed of 480 spot scenarios, meticulously constructed to instrument 12 behaviorally distinct decision categories covering aggression, safety, rule compliance, progress decisions, and sensitivity to history/contextual framing. Each scenario is validated by a full Ludo simulator, ensuring every spot admits a single, interpretable strategic tradeoff grounded in explicit game rules. Legal moves are masked in the prompt to force genuine rule inference.

The framework integrates three non-LLM agents:

• Random (uniform action sampling),
• Heuristic (linear scoring on progress, capture, safety bonuses), and
• Game-Theory (GT), which implements depth-limited Expectiminimax (2-player) and Expectimax-MaxN (3-4-players), equipped with a linear evaluator (Figure 1, right).

Figure 1: The left visualizes the annotated Ludo board configuration; the right displays win rates in head-to-head agent matchups, confirming the skill hierarchy (Random < Heuristic < GT).

Each LLM (six models, five model families) receives natural language prompts encoding full board state, rules, current dice outcome, and optionally a persona or narrative block. For each behavioral decision, models must produce a single piece index and accompanying justification. Every model is evaluated across all combinations of categories and persona instructions, yielding 14,400 spot-level decisions.

Rule Compliance and Model Reliability

Reliability is measured as the pre-fallback invalid move rate (before random fallback is invoked). Notably, only two models (DeepSeek-Chat and Claude-3.5-Haiku) achieve sub-1% invalid rates across categories, forming the “near-perfect” compliance tier, while others (e.g., Qwen-2.5-7B) exhibit severe deficiencies, especially in endgame edge cases like overshoot (up to 43% failure for some spot types). Deviations in compliance are not strictly correlated with model size, highlighting the impact of training scheme and RLHF coverage on grounded task following (Figure 2, left).

Figure 2: Left—average invalid rate and reliability stratify models into three compliance tiers; right—GT (game-theory) alignment scores (dark = high agreement; light = disagreement) stratified by spot category and model.

The separation of rule compliance from behavioral strategy allows the analysis to decouple syntax from semantics—critical for trustworthiness in real-world applications.

Strategic Alignment with Game-Theory Baseline

Comparing model actions to the GT agent reveals universal strategic incompleteness: all LLMs align with the GT only 40–46% of the time. Disaggregated by category, models display pronounced mode collapse: each reliably implements a partial, but not complete, strategy. Category-level heatmaps (Figure 2, right) demonstrate that models occasionally match GT play on single dimensions, but no model approaches holistic strategic competence.

Emergent Behavioral Archetypes

Cluster analysis on move preference surfaces two dominant LLM archetypes—finishers and builders—which are mutually exclusive in most models (Figure 3, left):

• Finishers (e.g., Haiku, Qwen variants) skew toward completing pieces at the expense of board development, mirroring strong goal-seeking at the cost of tactical resource expansion.
• Builders (e.g., DeepSeek-Chat, Gemma) aggressively develop (bring out new pieces) but rarely finish, operationalizing expansion over consolidation.
• Llama-4-Scout adopts a hybrid posture but diverges from GT on critical captures.

The GT agent uniquely occupies the quadrant characterized by both high development and high completion, reflecting the necessity of balancing parallel objectives (Figure 3, right).

Figure 3: Left—models cluster as archetypal finishers or builders relative to the GT's balanced strategy; right—behavioral radar chart shows that while LLMs can match GT on isolated axes, no model replicates the GT’s all-dimensional profile.

Preference Tradeoffs and Subdomain Contradictions

Spot categories forcing direct strategic tradeoffs (e.g., capture vs. finishing a piece) expose where models systematically diverge from the GT optimal policy. For instance, in the capture-vs-home-finish scenario, only finisher models reliably agree with the GT, while builder models select suboptimal captures. In less clear categories (e.g., capture-vs-safe or capture-vs-home), these preferences invert—builders are correct, finishers err (Figure 4, left).

Figure 4: Left—models’ tradeoff preferences relative to the GT optimal choice (dashed line; left/capture, right/alternative); right—persona alignment (how strongly behavior shifts in response to persona instructions), typically weak except for Q7B-aggressive and QP-greedy.

This differential alignment demonstrates that LLMs often “memorize” competing—but incomplete—sub-strategies from their training data, rather than acquiring a coherent planning policy.

History Sensitivity and Prompt Vulnerability

The introduction of grudge scenarios, in which the only variation is a narrative (e.g., “Player 2 captured your piece last turn”), exposes prompt-induced non-Markovian behavior. For models like Qwen-Plus, up to 33% of move choices flip in response to irrelevant narrative context, compared to the GT agent's invariant policy. This indicates a susceptibility to prompt injection and context leakage even in strictly formal environments.

Persona Conditioning and Steering Efficacy

Analysis across five persona conditions (neutral, aggressive, greedy, safe, unforgiving) reveals that natural-language steering signals are generally weak and inconsistent. Most models exhibit persona alignment scores between 0.3–0.5 with strong effect only for specific combinations (Q7B-aggressive: 0.93; QP-greedy: 0.83, Figure 4, right). Paradoxically, some instructions produce inverse effects, underscoring the dangers of behavioral scripting without measurement. This challenges the deployment viability of role-based language agents for safety-critical strategy tasks.

Implications for LLM Evaluation and Future Research

The results establish several robust claims:

• Rule compliance is orthogonal to strategic competence; models may follow instructions perfectly yet implement systematically suboptimal strategies.
• LLM strategic policy is incomplete and archetype-dependent; contemporary instruction methods do not yield balanced reasoning in stochastic, multi-agent, multi-asset games.
• LLMs demonstrate prompt vulnerability in history-sensitive settings, raising risks for both adversarial and benign context shifts.
• Behavioral controls (personas) are unreliable, with their efficacy gated by task specificity and baseline model inductive bias.

In broader AI evaluation, these findings highlight the critical need for multi-factorial, spot-based benchmarks that (a) isolate rule following from reasoning, (b) include prompt/contextual manipulations, and (c) analyze strategic archetype formation. The LudoBench methodology motivates similar benchmarks in other stochastic multi-agent, multi-goal environments, extending to economic simulations, real-time resource allocation, and adversarial risk management.

Conclusion

LudoBench (2604.05681) provides the first systematic, interpretable, and context-sensitive evaluation framework for LLM decision-making in a stochastic, multi-piece board game absent from prior benchmarks. The analysis demonstrates that state-of-the-art LLMs, while competent in rule following, fail to develop robust, GT-aligned strategies and are vulnerable to narrative framing and incomplete behavioral steering. Future research directions include scaling to trajectory-based simulation, integration with reinforcement learning fine-tuning for strategic completeness, and expanding to multiple games with stochastic, multi-agent, and adversarial components.