Guandan AI Agents in Multi-Agent Gaming

• Guandan AI agents are automated decision-making systems using deep RL and neural architectures to tackle the complex, imperfect-information Chinese card game.
• They employ advanced techniques including deep Q-networks, PPO-based policy optimization, and top-k action filtering to manage enormous state and dynamic action spaces.
• Benchmarks via OpenGuanDan show these agents outperform rule-based bots, though challenges remain in sample efficiency, tribute heuristics, and approaching human-expert performance.

Guandan AI agents are automated decision-making systems designed to play the four-player, imperfect-information Chinese card game Guandan at a high level of cooperative and competitive play. These agents confront intricate challenges stemming from Guandan's enormous state and action spaces, highly variable legal move sets, long-horizon credit assignment across multi-round episodes, and the hybrid team-vs-team structure that demands both strategic cooperation and adversarial tactics. As of 2026, advances have produced reinforcement-learning (RL) agents demonstrably superior to historic rule-based bots, with extensive benchmarks and frameworks standardized through the OpenGuanDan platform. Notable directions also include the integration of neural architectures for behavior regulation and the adaptation of LLMs for multi-agent coordination under imperfect information.

1. Formal Definition and Game Theoretic Background

Guandan is formalized as a multi-agent, partially observable stochastic game or, equivalently, an extensive-form imperfect-information game. The system consists of $N=4$ players, each receiving a private hand from two standard 54-card decks, engaging in sequential rounds partitioned into “tricks,” with level-up progression and tribute mechanisms at round boundaries. Each player’s observation $o^i_t$ encodes their hand, records of remaining and played cards, team level state, and constrained legal actions. The central transition, observation, and reward functions are as follows (Li et al., 31 Jan 2026):

• State space $\mathcal{S}$: Complete card configuration, per-player hands, round progression, team levels, tribute/back-tribute status, and order of finished players
• Action set $A^i(s)$: All valid card combinations per rules (solos, pairs, tubes, plates, bombs, flushes, etc.), varying dynamically (often $>5000$ at the trick's outset, as low as $<50$ late in a round (Zhao et al., 2023))
• Observation $O^i(s)$: Non-omniscient; each agent observes only its own hand, the full public history of plays, explicit indicators for wildcards and levels, and summary statistics about partner and opponent states
• Returns and rewards: Zero at intermediates; on round completion, team rewards are +3, +2, +1 (descending by finishing order), negatives for losing teams; more granular scoring and team credit assignment possible in alternative formulations

This formalism supports both RL and game-theoretic approaches, including best-response, extensive-form Nash computation, counterfactual regret minimization (CFR), subgame solving, and a variety of decentralized actor–learner paradigms (Li et al., 31 Jan 2026, Zhao et al., 2023).

2. Agent Architectures and Core Learning Algorithms

The dominant paradigm for Guandan AI agents involves a deep neural value-based function approximator $Q_\theta(s,a)$ or an explicit policy $\pi_\theta(a|s)$ trained with deep RL throughout distributed self-play (Lu et al., 2022, Zhao et al., 2023, Yanggong et al., 2024). Architecturally:

• State and action encodings: Standardized as high-dimensional real vectors concatenating hand representation, unseen cards, trick context, last partner move, opponent statistics, cumulative played cards, team level info, and explicit wild-card flags (active dimensions ranging from 567 to over 2000 depending on architecture depth and inclusion of behavior/history features) (Zhao et al., 2023, Yanggong et al., 2024)
• Q-function approximators: Multi-layer perceptrons (MLPs) with 3–6 layers (512–1024 units per hidden layer), tanh or ReLU activations, trained via mean-squared error against Monte Carlo-target returns
• Policy networks (PPO stage): MLPs with dual heads (policy logits and value); input restricted to top-k pruned actions per state; objective includes clipped surrogate, value, and entropy bonuses

The Deep Monte Carlo (DMC) method is foundational, eschewing bootstrapped target bias for unbiased trajectory returns: $G_t = \sum_{k=0}^{T-t} \gamma^k r_{t+k}$ The Q-network loss is

$o^i_t$0

Distributed frameworks asynchronously coordinate 40–160 actors generating self-play episodes and a central learner performing parameter updates (Zhao et al., 2023, Lu et al., 2022). PPO policy refinement employs DMC-pretrained Q-networks for action candidate selection, drastically reducing sample complexity in the face of immense action spaces (Zhao et al., 2023).

3. Strategies for Large and Dynamic Action Spaces

The cardinality of the legal action set in Guandan commonly exceeds several thousand in opening tricks, rendering naïve policy iteration or softmax computations infeasible. State-of-the-art agent designs employ elaborate top-k action filtering strategies (Zhao et al., 2023, Li et al., 31 Jan 2026):

Top-k Action Pruning:

1. For current state $o^i_t$1, all legal actions $o^i_t$2 are scored via pretrained Q-network $o^i_t$3.
2. The highest-ranked $o^i_t$4 actions (with $o^i_t$5 typically tuned between 2 and 5) are selected.
3. Policy optimization or planning is restricted to this reduced candidate set.

This approach is consistently shown to yield best performance at smaller $o^i_t$6 (e.g., $o^i_t$7 in PPO augmentation), accelerating convergence without substantial loss of optimality, and, for LLM-based agents, is essential for managing context window limitations (Zhao et al., 2023, Yim et al., 2024).

Action Masking and Enumeration: Each agent, prior to neural inference, explicitly constructs a valid move set per hand/trick state, guaranteeing only legal combinations are scored or sampled (Li et al., 31 Jan 2026, Lu et al., 2022).

4. Advanced Modeling: Cooperation, Behavior Regulation, and LLM Integration

Unique to Guandan is the learning of hybrid competitive–cooperative credit assignment. Several agent frameworks embed higher-level behavioral signals directly in the neural encoding:

• Behavior Regulation One-Hots: GuanZero augments the state vector with special one-hot flags indicating inferred or commanded “cooperate,” “dwarf,” or “assist” actions, enabling the Q-network to correlate collaborative intent with long-term returns (Yanggong et al., 2024). Empirically, this produces stable increases (from random baseline cooperation rate ≈33% to ≈60%) in “appropriate” multi-agent tactics, as reflected in large-scale self-play.
• History Features: LSTM modules process rolling action histories, capturing sequential dependencies without explicit tree search (Yanggong et al., 2024).
• LLM Agents: Recent research evaluates GPT-3.5/4, Baichuan, Qwen, ChatGLM variants as partial-information strategists (Yim et al., 2024). These LLM agents receive context-converted state descriptions and action lists, plan and score candidate moves with or without Theory of Mind (ToM) planning modules, and rely on a lightweight RL tool for action pre-filtering. ToM integration allows nested beliefs about other agents’ strategies and is shown to yield incremental performance gains, though state-of-the-art RL agents still retain an overall advantage.

5. Experimental Protocols and Empirical Evaluation

Evaluation of Guandan AI agents is standardized by the OpenGuanDan benchmark and associated protocols (Li et al., 31 Jan 2026). Major components:

• Pairwise Tournaments: Each agent is matched against every other, controlling both seats of one team over 1,000 full games; team win rates and detailed round outcomes are reported.
• Human–AI Matches: Multi-level human teams (beginner, intermediate, advanced) are benchmarked against each agent; current best agents remain below human-superhuman thresholds, with top RL agents (GS2, SDMC, DanZero) achieving win rates up to ≈43% versus advanced human teams.
• Ablation and Feature Importance: Exclusion of critical input features such as wild-card flags or behavioral one-hots yields 10–40 percentage point performance drops versus full models (Lu et al., 2022, Zhao et al., 2023, Yanggong et al., 2024).
• LLM vs. RL Agent Comparison: LLMs leveraging ToM modules and top-k selection approach, but do not surpass, RL agent performance, with GPT-4 (2nd-order ToM) achieving average team scores of −0.88 versus DanZero+ (see Table below) (Yim et al., 2024).

Agent/LLM	Random	Rule-Based	DanZero+	Avg
GPT-3.5 (1st-ToM)	+3.20	+3.00	−3.27	+0.98
GPT-4 (2nd-ToM)	+4.00	+3.40	−0.88	+2.17

Returned win rates for RL agents exceed 90% against rule-based bots and reach up to 71% against graduate student pairs for PPO-enhanced DanZero (Zhao et al., 2023).

6. Major Benchmarks, Limitations, and Open Challenges

The OpenGuanDan benchmark (Li et al., 31 Jan 2026), supporting both efficient simulation and agent evaluation via independent player APIs, codifies the task’s nontrivial properties: large-scale information/action spaces; mixed competitive–cooperative objectives; variable team composition; and long-horizon planning. Limitations reported across current research include:

• Tribute/Back-Tribute Heuristic Dependence: Most agents employ expert- or rule-based submodules for these pre-round phases; learning end-to-end tribute/back-tribute policies is not yet realized (Zhao et al., 2023, Li et al., 31 Jan 2026, Yanggong et al., 2024).
• Sample Efficiency and Compute: DMC-based pipelines require 30 days of continuous training on 160+ CPUs (for DanZero/SDMC), while PPO enhancement can dramatically reduce wall clock, provided a strong Q “teacher” (Zhao et al., 2023).
• No Theoretical Regret Guarantees: Despite empirical dominance, agents lack theoretical bounds on exploitability or worst-case regret.
• Human–Superhuman Gap: All tested learning-based agents, including those integrating subgame solving (GS2), remain below 50% win rates against advanced human teams (Li et al., 31 Jan 2026).

Prominent open research directions include scalable end-to-end learning of all phase decisions, rich opponent modeling under nonstationary or unknown policies, hybrid RL–search architectures (e.g., deep neural MCTS), and automated feature discovery for behavior regulation signals.

7. Summary of Key Contributions and Research Fronts

Guandan AI research marks the convergence of deep reinforcement learning, distributed multi-agent training, neural credit assignment for hybrid objectives, and scalable handling of massive action spaces (Zhao et al., 2023, Lu et al., 2022, Li et al., 31 Jan 2026, Yanggong et al., 2024). Architectures are distinguished by structured input encodings, DMC value learning, PPO-based policy augmentation (requiring only O($o^i_t$8) computation per update), and, increasingly, explicit mechanisms for modeling cooperation and “theory of mind.” Benchmarks demonstrate dramatic superiority over rule-based strategies, but also expose the open challenge of reaching or surpassing human-expert performance and generalizing to broader classes of large-scale multi-agent imperfect-information games. Continued development is catalyzed by open benchmarks (OpenGuanDan), reproducible codebases, and comprehensive empirical standards, positioning Guandan as a flagship problem domain for the study of multi-agent RL and imperfect information game AI (Li et al., 31 Jan 2026, Zhao et al., 2023, Yim et al., 2024).