---
title: 'MTG-Causal-RL: A Causal RL Benchmark'
url: https://www.emergentmind.com/topics/mtg-causal-rl
type: topic
---

# MTG-Causal-RL: A Causal RL Benchmark

MTG-Causal-RL is a Gymnasium benchmark for causal reinforcement learning built on **Magic: The Gathering** and designed for settings that combine **sequential decision making**, **hidden information**, **large masked action spaces**, and an **explicit causal interface**. It exposes a **3,077-dimensional partial observation**, a **478-action masked discrete action space**, **five competitive Standard archetypes**, **three reward schemes**, and a **hand-specified Structural Causal Model (SCM)** over strategic variables. During each episode, it publishes causal variables, SCM-predicted intervention effects, and per-factor credit traces, so that **causal credit assignment**, **leave-one-out cross-archetype transfer**, and **policy auditability** become benchmark targets rather than auxiliary diagnostics [2605.06066].

## 1. Conceptual scope and research setting

MTG-Causal-RL was introduced to address a benchmark gap rather than a purely algorithmic one. The motivating claim is that earlier causal-RL settings often isolated only part of the problem—either relatively clean intervention structure in control domains or strategic sequential reasoning in games—whereas Magic: The Gathering naturally combines stochastic draws, hidden information, masked legality constraints, and strategically meaningful abstractions such as mana, board pressure, card advantage, and tempo [2605.06066].

This positioning places MTG-Causal-RL within a broader causal-RL lineage. Earlier work on reinforcement learning of causal variables argued that standard RL can maximize return without identifying the **right abstract variables** to reason about, and proposed learning a mediator \(Z\) that is both **manipulable by a policy** and **predictive of the outcome** via a mediation-analysis objective [2010.15745]. A separate theoretical line argued that online RL is already causal in the sense that, under action sufficiency and self-generated interaction, conditional probabilities estimated from online data can coincide with interventional probabilities; under that view, causal modeling becomes especially important for **offline RL**, **counterfactual queries**, and settings with observation mismatch [2403.04221]. MTG-Causal-RL does not contradict those arguments; a plausible implication is that it shifts emphasis from “are actions causal?” to “can causal structure be surfaced, audited, and exploited under partial observability and action masking?”

## 2. Environment architecture, game structure, and reward design

The benchmark is built around **Standard-format MTG decks**. Each player starts at **20 life** with a **60-card deck** and a **7-card opening hand**, and an episode ends either at lethal damage or at a user-chosen turn cap [2605.06066]. The observation is intentionally partial: the **3,077-dimensional** vector contains visible information about the hand, battlefield, graveyard, life totals, phase, and mana state. The action space is a **478-action masked discrete space** divided into **16 action categories**; at a given state, usually only **2 to 15 actions** are legal. The action layout includes `PASS`, `KEEP`, `MULLIGAN`, `CONFIRM`, `CANCEL`, `AUTO_PAY`, `PLAY_LAND`, `CAST_SORCERY`, `CAST_INSTANT`, `ACTIVATE`, `ATTACK_TOGGLE`, blocking and targeting categories, and `MANA_SOURCE` selection. The environment therefore treats each turn as a sequence of **10 to 20 decisions**, not a single move-selection event [2605.06066].

The benchmark ships with **five fixed Standard 2025 archetypes**, chosen to induce distinct strategic regimes.

| Archetype | Strategic emphasis |
|---|---|
| Mono-Red Aggro | Tempo and immediate pressure |
| Azorius Control | Defensive sequencing and resource management |
| Dimir Midrange | Pressure plus interaction |
| Domain Ramp | Mana acceleration and delayed payoff |
| Boros Convoke | Board development and pressure |

MTG-Causal-RL includes **three reward schemes**. The first is **sparse terminal reward**. The second is a **shaped potential-based reward**
\[
R^{\mathrm{shaped}}_t = \gamma\,\Phi(s_{t+1}) - \Phi(s_t) + R^{\mathrm{terminal}}_t
\]
where \(\Phi(s)\) is a weighted combination of observable strategic variables such as mana, card advantage, board pressure, tempo, and life buffer. The third is a **dense reward variant** that adds per-step signals such as damage, card draw, and creature-related feedback. The shaping potential is computed only from **agent-observable quantities**, which the benchmark presents as preserving policy invariance relative to the observation process [2605.06066].

## 3. Hand-specified SCM and the benchmark’s intervention interface

A distinguishing feature of MTG-Causal-RL is that it exposes a **hand-designed SCM** rather than requiring agents to infer causal structure from scratch. The SCM is organized into four layers: **resource layer**, **board state layer**, **strategic position layer**, and **outcome layer**. Its variables include \(\mathrm{Mana}_t\), \(\mathrm{LandDrop}_t\), \(\mathrm{ManaCreatures}\), \(\mathrm{Mana}_{t+1}\), \(\mathrm{CardCount}\), \(\mathrm{HasRemoval}\), \(\mathrm{BoardPress}\), \(\mathrm{ThreatDensity}\), \(\mathrm{CardAdv}\), \(\mathrm{Tempo}\), \(\mathrm{LifeBuffer}\), \(\mathrm{RemovalAvail}\), and \(\mathrm{WinProb}\) [2605.06066].

The benchmark highlights six strategic parents of \(\mathrm{WinProb}\): **card advantage**, **board pressure**, **tempo**, **life buffer**, **threat density**, and **removal availability**. The outcome equation is
\[
\mathrm{WinProb} = \sigma\!\Big(\mathbf{w}^{\!\top} [\mathrm{CardAdv}, \mathrm{BoardPress}, \mathrm{Tempo}, \mathrm{LifeBuffer}, \mathrm{ThreatDensity}, \mathrm{RemovalAvail}]\Big).
\]
Illustrative structural equations include
\[
\mathrm{Mana}_{t+1} = \mathrm{Mana}_t + \mathbf{1}[\mathrm{LandDrop}_t] + \mathrm{ManaCreatures},
\]
\[
\mathrm{BoardPress} = \sum_{c \in \mathrm{Own}} \mathrm{power}(c) - \sum_{c \in \mathrm{Opp}} \mathrm{power}(c),
\]
and
\[
\mathrm{Tempo} = \operatorname{clip}\!\left(\frac{\mathrm{ManaSpent}_t}{\max(1,\mathrm{Mana}_t)} - \frac{\mathrm{OppManaSpent}_t}{\max(1,\mathrm{OppMana}_t)}, -1, 1\right).
\]
The SCM is explicitly **hand-specified, not learned from scratch**; the stated reason is to prioritize **interpretability** and clear edge orientation [2605.06066].

Within this interface, actions are treated as **interventions**. A canonical example is playing a land, represented as
\[
\mathrm{do}(\mathrm{LandDrop}=1).
\]
The benchmark then computes downstream changes in strategic factors and exposes per-factor intervention effects
\[
\varepsilon_k(s,a) = \phi_k(s') - \phi_k(s),
\]
where \(\phi_k\) is the \(k\)-th strategic factor. During episodes, the environment publishes \(\phi(s_t)\), factor rewards
\[
r^{\mathrm{factor}}_{k,t} = \phi_k(s_{t+1}) - \phi_k(s_t),
\]
the SCM-predicted intervention effects \(\varepsilon_{k,t}\), and factor-level traces for credit assignment [2605.06066]. This means that MTG-Causal-RL is not only a partially observed game environment; it is also an **explicit intervention-analysis interface** over a known strategic abstraction.

## 4. Reference baselines and CGFA-PPO

The benchmark is accompanied by a panel of reference methods rather than a single canonical solver. These include **Random**, **Heuristic**, **Masked PPO**, a **causal-world-model PPO variant**, and an **architecture-matched scalar control** [2605.06066].

| Method | Role in evaluation |
|---|---|
| Random | Uniformly samples legal actions |
| Heuristic | Rule-based script, one per archetype |
| Masked PPO | Standard PPO with invalid-action masking |
| Causal-world-model PPO variant | Auxiliary causal prediction baseline |
| Scalar control / CGFA scalar-only | Capacity-matched non-causal control |

The benchmark’s reference causal agent is **Causal Graph-Factored Advantage PPO (CGFA-PPO)**. It retains a masked PPO actor and scalar critic \(V(s)\), but adds **per-factor critic heads** \(V_k(s)\) trained on factor-specific returns
\[
G_{k,t} = \sum_{i \ge 0} \gamma^i r^{\mathrm{factor}}_{k,t+i},
\qquad
A_{k,t} = G_{k,t} - V_k(s_t).
\]
It also introduces **learnable mixture weights**
\[
w_k = \mathrm{softmax}(\beta)_k
\]
initialized from the SCM’s logistic-regression weights, and a **state-conditional residual gate** \(g(s)\in(0,1)\) that blends scalar and factor-based learning:
\[
A_{\mathrm{used}}(s_t, a_t) =
\bigl(1-g(s_t)\bigr)A^{\mathrm{scalar}}(s_t,a_t)
+ g(s_t)\sum_{k=1}^{K} w_k A_k(s_t,a_t).
\]

A further component is the **intervention-calibration loss**
\[
\mathcal{L}_{\mathrm{cal}} = -\frac{1}{K}\sum_{k=1}^{K} \frac{\operatorname{Cov}_t(A_{k,t}, \varepsilon_{k,t})} {\sigma_{A_k}\sigma_{\varepsilon_k}+\delta},
\]
which encourages alignment between learned factor advantages and SCM-predicted intervention effects. The full objective is
\[
\mathcal{L}_{\mathrm{CGFA}} =
\mathcal{L}_{\mathrm{ppo}}
+ c_v \mathcal{L}_{\mathrm{value}}
- c_H \mathcal{L}_{\mathrm{ent}}
+ c_f \mathcal{L}_{\mathrm{factor}}
+ c_c \mathcal{L}_{\mathrm{cal}}
- c_e \mathcal{L}_{\mathrm{gate}}.
\]
During training, the wrapper logs per-factor Pearson correlation between \(A_k\) and \(\varepsilon_k\), sign agreement, per-factor credit shares, and gate statistics \(g(s)\), making causal usage inspectable rather than implicit [2605.06066].

## 5. Evaluation protocol and empirical findings

MTG-Causal-RL adopts an unusually explicit statistical protocol. The full evaluation uses **at least \(10^6\) environment steps per opponent**, **7 paired seeds** per agent/deck cell, and **300 deterministic evaluation episodes** per agent-seed-opponent cell. Comparisons use **paired bootstrap confidence intervals**, **Wilson confidence intervals** for win rates, **Wilcoxon signed-rank tests**, and **Holm-Bonferroni correction** within pre-registered comparison families. The benchmark also notes that some headline learned-agent results in the main tables use only **\(n_s=2\) paired seeds** and are therefore exploratory [2605.06066].

The main aggregate finding is deliberately restrained: **Masked PPO and CGFA-PPO both beat random play across all five archetypes**, but neither dominates uniformly. In the reported in-distribution comparisons, **CGFA-PPO** exceeds PPO on **Azorius Control** (**25.6% vs 20.6%**) and **Mono-Red Aggro** (**70.6% vs 67.8%**), while PPO is stronger on **Boros Convoke**, **Dimir Midrange**, and **Domain Ramp**. The benchmark also reports that simple heuristics remain strong on some aggressive decks, whereas slower archetypes such as **Azorius Control** and **Domain Ramp** remain difficult under the current training budget [2605.06066].

The benchmark’s causal diagnostics are at least as important as scalar win rate. Calibration trajectories show that several factor correlations become positive over training, that factor credit concentrates heavily on the **life-buffer** factor, and that the residual gate remains active rather than collapsing entirely to scalar PPO. Leave-one-out transfer experiments report similar negative measured gaps for both learned agents:
- **PPO:** \(-4.7\) percentage points  
- **CGFA-PPO:** \(-4.3\) percentage points  

The interpretation given is not that cross-archetype transfer is solved, but that the held-out opponent mix happened to be easier in those runs, which is precisely why paired-seed transfer evaluation is necessary [2605.06066]. A common misconception is therefore that MTG-Causal-RL is primarily a win-rate leaderboard. Its actual diagnostic center of gravity is the joint inspection of **performance**, **factor calibration**, and **transfer gaps**.

## 6. Relation to causal-RL methodology and principal limitations

MTG-Causal-RL is most naturally understood as a benchmark that operationalizes several themes already present in causal RL. One such theme is the search for **coarse-grained causal variables** that are actionable and reward-relevant, rather than merely predictive; mediation-based RL formalized this through variables that are easier to induce under one policy than another and that materially affect return [2010.15745]. Another is the online coupling of **intervention**, **causal structure learning**, and **policy guidance**, where better interventions improve the learned causal graph and the graph in turn constrains policy search [2402.04869]. A third is the representation of environments by **context-dependent causal subgraphs**, as in Meta-Causal Graph world models, where latent meta states determine which causal relations are active [2506.23068]. MTG-Causal-RL does not itself implement those algorithms, but it provides a domain in which comparable questions about intervention effects, structural transfer, and policy auditability can be asked in a strategically rich game setting.

Several limitations follow directly from the benchmark design. First, the causal structure is **not discovered from raw data**; the SCM is **hand-specified**, so the benchmark tests reasoning over a known abstraction rather than causal discovery under full ambiguity [2605.06066]. Second, the causal label does **not** imply uniform empirical superiority: CGFA-PPO is competitive, but the reported benefit is **deck-dependent**, and scalar PPO is better on several archetypes [2605.06066]. Third, transfer remains unresolved: the reported leave-one-out gaps are negative for both PPO and CGFA-PPO, and the benchmark treats that as a diagnostic result rather than a solved generalization story [2605.06066].

Those constraints are integral to what MTG-Causal-RL is meant to measure. It is not a claim that Magic strategies can be reduced to six factors, nor a claim that causal agents should automatically dominate associative baselines. Rather, it is a benchmark in which **masked-action control**, **partial observability**, **SCM-grounded intervention queries**, and **auditability of strategic credit** are exposed together. In that sense, it provides a shared testbed for causal RL, world-model methods, and LLM-agent research on questions that simpler environments do not pose in a single setting [2605.06066].

Source: https://www.emergentmind.com/topics/mtg-causal-rl