Causal Reinforcement Learning for Complex Card Games: A Magic The Gathering Benchmark

Abstract: Causal reinforcement learning (RL) lacks benchmarks for complex systems that combine sequential decision making, hidden information, large masked action spaces, and explicit causal structure. We introduce MTG-Causal-RL, a Gymnasium benchmark built on Magic: The Gathering with a 3,077-dimensional partial observation, a 478-action masked discrete action space, five competitive Standard archetypes, three reward schemes, and a hand-specified Structural Causal Model (SCM) over strategic variables. Every episode exposes causal variables, SCM-predicted intervention effects, and per-factor credit traces, making causal credit assignment, leave-one-out cross-archetype transfer, and policy auditability first-class metrics. We adapt a panel of reference baselines: random, heuristic, masked PPO, a causal-world-model PPO variant, and an architecture-matched scalar control. We propose Causal Graph-Factored Advantage PPO (CGFA-PPO) as a reference causal agent that uses SCM parents of win probability as factor-aligned critic targets with an intervention-calibration loss. All comparisons use paired seeds, paired-bootstrap confidence intervals, and Holm-Bonferroni correction within pre-registered families. Masked PPO and CGFA-PPO reach competitive in-distribution win rates and exceed the random baseline; per-factor calibration trajectories and leave-one-out transfer gaps expose diagnostic structure that scalar win rate alone cannot. We release the benchmark, reference-baseline results, and full evaluation protocol openly. By coupling a strategically rich, partially observed domain with an explicit causal interface and statistical protocol, MTG-Causal-RL gives causal-RL, world-model, and LLM-agent research a shared testbed for questions current benchmarks cannot pose together: causal credit assignment under masked action spaces, structural transfer across archetypes, and SCM-grounded policy auditability.

• The paper introduces a novel MTG-Causal-RL benchmark that integrates a structured causal model to enable precise causal credit assignment in complex game settings.
• It employs a sophisticated Gymnasium environment with layered SCMs and state-conditional residual gating to dissect intervention effects in a 478-dimensional action space.
• Experimental results reveal tradeoffs between raw win rates and systematic causal auditability, highlighting cross-archetype transfer and interpretability challenges.

Causal Reinforcement Learning for Complex Card Games: A Structured Benchmark for Magic: The Gathering

Introduction and Motivation

This work presents MTG-Causal-RL, a comprehensive reinforcement learning (RL) benchmark constructed around Magic: The Gathering (MTG), designed to combine strategically rich, partially observable, and high-variance game dynamics with a first-class explicit causal modeling interface. The hallmark of this environment is its integration of a hand-designed Structural Causal Model (SCM) over critical strategic variables, enabling direct evaluation of causal credit assignment and policy auditability. The benchmark is engineered to simultaneously address three open problems in causal RL research: (i) causal credit assignment in large, masked action spaces, (ii) robust generalization via cross-archetype transfer, and (iii) interpretability and auditing through structured intervention diagnostics.

Environment and Structural Causal Model

MTG-Causal-RL is formulated as a Gymnasium environment with a 3,077-dimensional partial observation vector and a 478-dimensional masked discrete action space, capturing the combinatorial complexity of real MTG play. This includes five contemporary competitive archetypes (Mono-Red Aggro, Azorius Control, Dimir Midrange, Domain Ramp, Boros Convoke), each posing distinct strategic demands.

The core innovation lies in the causal interface driven by a hand-engineered SCM, which models the game's strategic structure through layered variables: resources, board state, strategic position, and outcome. The SCM explicitly encodes the six strategic "parents" of win probability—card advantage, board pressure, tempo, life buffer, threat density, and removal availability—allowing each agent action to be decomposed into intervention effects on these factors. This facilitates both potential-based reward shaping and causal credit decomposition, with policy learning and evaluation tightly coupled to interpretable causal traces.

Reference Agents and Causal RL Architectures

A suite of baselines is introduced, including random and heuristic (scripted) agents, vanilla masked PPO (Stable-Baselines3), a causal world-model augmented PPO, and a parameter-count-matched scalar control. The principal causal agent is Causal Graph-Factored Advantage PPO (CGFA-PPO), which augments a standard PPO backbone with:

• Per-factor critics: Value heads $V_k(s)$ aligned with the SCM's strategic factors.
• Learnable mixture weights: $\mathrm{softmax}(\beta)$ initialized from the SCM prior, enabling data-driven reweighting of causal credit channels.
• State-conditional residual gating: $g(s) \in (0,1)$ mediating the tradeoff between scalar and structured advantage updates.
• Intervention-calibration loss: Auxiliary Pearson-correlation alignment between each factor's empirical advantage and the SCM-predicted intervention effect, enforcing causal consistency.

This architecture is designed not primarily for raw performance, but for systematic auditing: every training step exposes per-factor calibration, credit share, and gating metrics, furnishing a diagnostic window into the agent's causal reasoning dynamics.

Experimental Protocol and Benchmarking

A rigorously controlled experimental protocol employs paired seeds, Wilson confidence intervals, paired-bootstrap hypothesis tests, and Holm-Bonferroni correction, ensuring that statistical reporting accounts for the high variance and partial observability typical of card game RL. The benchmark delivers:

• Headline win rates for PPO and CGFA-PPO, both of which substantially exceed random play across all archetypes, but without universal dominance for the causal agent.
• Deck-dependent performance gaps: CGFA-PPO shows superior in-distribution performance on the Azorius Control and Mono-Red Aggro archetypes, but falls behind PPO on others, such as Domain Ramp.
• Diagnostic ablation studies: Disentangling the contributions of parameter count, factor-aligned critics, residual gating, and intervention calibration.

  Figure 1: Win rate comparisons against the in-distribution opponent pool, stratified by archetype, with random and heuristic anchors indicating baseline difficulty.

The ablation suite verifies that causal components such as factor critics and gating mechanisms do not, in the current setup, guarantee scalar win-rate superiority, but instead support model auditability.

Figure 2: Ablation performance on Mono-Red Aggro, separating parameter count, critic structure, residual gating, and calibration mechanisms.

Generalization and Causal Auditability

The environment enables systematic evaluation of cross-archetype transfer, with paired-seed training on four decks and held-out evaluation on the fifth. Both PPO and CGFA-PPO manifest negative transfer gaps, indicating that some held-out archetypes are actually easier under specific matchup mixes—highlighting the necessity of explicit transfer diagnostics rather than scalar win rate reporting alone.

Figure 3: Cross-archetype transfer performance, quantifying the in-distribution vs held-out win rate gap per agent.

A core contribution of CGFA-PPO is its causal auditability—tracking the alignment between per-factor advantages and SCM intervention predictions, the distribution of credit among strategic factors, and the state-conditional use of causal structure.

Figure 4: CGFA-PPO's per-factor calibration trajectory for Mono-Red Aggro, illustrating factor alignment, credit share, and gating evolution over training.

Mechanistic Case Studies

Case-level analyses provide fine-grained visibility into individual episode dynamics: tracking per-factor value contributions, advantage allocations, and real-time calibration against SCM intervention effects. This reveals that, despite only weak win-rate differentiation, CGFA-PPO actively allocates credit in a manner consistent with causal theory, and the residual gate systematically exploits the available causal signal.

Figure 5: Stepwise decomposition of value, advantage, and intervention calibration for one CGFA-PPO episode, illustrating interpretable, factor-driven credit assignment.

Limitations and Broader Implications

The primary limitation is the reliance on a hand-designed SCM, which favors interpretability at the expense of general structural discovery. Furthermore, the benchmark scope—limited to 56 cards and five decks—provides tractable but not exhaustive coverage of the vast MTG decision space. Opponents are fixed rather than adaptive, and rollout costs scale unfavorably with turn cap.

Nevertheless, the open-sourced Gymnasium interface enables extension and integration with LLM-based agents, providing a pathway to evaluate whether LLMs can leverage SCM-induced summaries and interventions for strategic play and decision auditability.

Conclusion

MTG-Causal-RL establishes a challenging testbed for causal RL in strategic, partially observed, high-variance settings with masked actions and structured causal dependencies. Through explicit SCM grounding, paired-seed statistical rigor, and comprehensive agent diagnostics, the benchmark moves beyond performance reporting to enable systematic study of causal regularization, generalization, and interpretability in deep RL.

Future directions include scaling towards larger card pools, learning the SCM structure from interaction data, and assessing the efficacy of language-model-driven agents within this interpretable, intervention-rich environment. The benchmark provides a public anchor for causal RL, model-based RL, and LLM-agent communities to stress-test new methods under conditions that couple high complexity with auditability and structural clarity.