Causal Graph-Factored Advantage PPO

• The paper introduces CGFA-PPO, which integrates per-factor critics aligned with a hand-designed SCM to decompose the advantage function in strategic MTG gameplay.
• It details a novel architecture that blends scalar and factorized advantages using a state-conditioned residual gate and intervention-calibration loss for improved credit assignment.
• Experimental diagnostics on the MTG-Causal-RL benchmark highlight both strengths in causal interpretability and limitations in win-rate improvements and transferability.

Causal Graph-Factored Advantage PPO (CGFA-PPO) is a masked Proximal Policy Optimization variant introduced as a reference causal agent for the MTG-Causal-RL benchmark, a partially observable Magic: The Gathering environment with an explicit Structural Causal Model (SCM) over strategic variables (Cunha et al., 7 May 2026). The method augments ordinary masked PPO with per-factor critics aligned to the SCM parents of $\mathrm{WinProb}$, a learnable factor mixture, a state-conditional residual gate, and an auxiliary intervention-calibration loss. Within the benchmark paper, CGFA-PPO is presented not as a replacement for PPO or as a claim of universal win-rate superiority, but as a benchmark-exercising method for causal credit assignment, structural transfer, and policy auditability in a large masked discrete action space.

1. Definition and conceptual role

CGFA-PPO is defined around a narrow but technically specific objective: preserving the standard masked PPO actor and scalar critic while adding structured, factor-aligned credit signals derived from a benchmark SCM (Cunha et al., 7 May 2026). Its central design premise is that a single scalar advantage is often too coarse in domains where actions have interpretable downstream effects on semantically meaningful strategic variables. In the MTG setting, those variables include mana development, board pressure, tempo, life buffer, and removal availability, all of which can mediate eventual win probability.

The method therefore departs from scalar-only actor-critic training in three explicit ways. First, it keeps the ordinary masked PPO actor and scalar critic $V(s)$. Second, it adds per-factor critics $V_k(s)$, one for each strategic causal factor. Third, it forms a residual blend between the standard scalar advantage and a weighted combination of factor-specific advantages, with a state-conditional gate determining how much the factored signal should influence the update in each state.

This positioning distinguishes CGFA-PPO from the paper’s other causal/world-model baseline, the “Causal Agent (CWM-augmented PPO),” which uses auxiliary causal supervision without per-factor advantage decomposition (Cunha et al., 7 May 2026). CGFA-PPO’s distinctive commitment is not merely to causal features, but to factorized advantage estimation over the SCM-defined parents of $\mathrm{WinProb}$, plus explicit calibration of those factorwise signals against SCM-predicted intervention effects.

A broader theoretical context for this emphasis on advantage comes from work on policy confounding in on-policy reinforcement learning. That analysis argues that the advantage function can mitigate policy-induced spurious correlations by downweighting state-action pairs that are common under the current policy, rather than serving only as a variance-reduction device (Suau, 13 Jun 2025). This does not constitute a derivation of CGFA-PPO, but it supplies a direct conceptual rationale for preferring advantage-centered updates when the goal is causal rather than merely observationally predictive credit assignment.

2. Benchmark substrate and causal factorization

CGFA-PPO is inseparable from the benchmark for which it was proposed. MTG-Causal-RL is a Gymnasium benchmark built on Standard-format Magic: The Gathering with a 3,077-dimensional partial observation, a 478-action masked discrete action space, five competitive Standard archetypes, three reward schemes, and a hand-specified SCM over strategic variables (Cunha et al., 7 May 2026). Although the nominal action space contains 478 discrete actions across 16 categories, typically only 2 to 15 actions are legal at a time. Episodes terminate at lethal damage or a user-selected turn cap, and each MTG turn requires roughly 10–20 sequential agent decisions.

The causal interface exposes benchmark causal variables $\mathrm{CV}_t \in \mathbb{R}^K$, default $K=6$, per-step changes $\Delta \mathrm{CV}_t$, and SCM-predicted intervention effects $\varepsilon_{k,t}$. CGFA-PPO uses as its factor basis the six parents of $\mathrm{WinProb}$ in the SCM. In the default benchmark configuration these factors are exactly:

1. $\mathrm{CardAdv}$
2. $V(s)$0
3. $V(s)$1
4. $V(s)$2
5. $V(s)$3
6. $V(s)$4

The strategic factor vector is

$V(s)$5

with $V(s)$6 in the default benchmark. The SCM outcome node is

$V(s)$7

where $V(s)$8 is the sigmoid and $V(s)$9 is learned online by a logistic-regression head.

The benchmark organizes variables into layers. Resources include $V_k(s)$0, $V_k(s)$1, $V_k(s)$2, $V_k(s)$3, $V_k(s)$4, and $V_k(s)$5. Board state includes $V_k(s)$6 and $V_k(s)$7. Strategic position includes $V_k(s)$8, $V_k(s)$9, $\mathrm{WinProb}$0, and $\mathrm{WinProb}$1. The outcome layer is $\mathrm{WinProb}$2. Actions are interpreted as interventions through card mechanics, and the benchmark analytically propagates interventions using Pearl-style $\mathrm{WinProb}$3 semantics implemented via structural equations and descendant enumeration.

This benchmark-specific SCM grounding is essential. CGFA-PPO does not learn its factorization from scratch; it exploits a hand-designed causal abstraction supplied by the environment. A common misconception is therefore to treat the method as a generic graph-learning PPO algorithm. In the reported form, it is instead a PPO variant whose critic decomposition is indexed by known benchmark factors.

3. Architecture, factor critics, and blended advantage

The architecture is explicitly described as a “drop-in replacement” for masked PPO that augments the critic without removing it (Cunha et al., 7 May 2026). It consists of a masked PPO actor $\mathrm{WinProb}$4, a scalar critic $\mathrm{WinProb}$5, a causal value head producing $\mathrm{WinProb}$6 for $\mathrm{WinProb}$7, a state-conditional residual gate $\mathrm{WinProb}$8, and learnable mixture logits $\mathrm{WinProb}$9 with

$\mathrm{CV}_t \in \mathbb{R}^K$0

The scalar critic, factor heads, and gate share critic features or latent representation $\mathrm{CV}_t \in \mathbb{R}^K$1. The inherited PPO backbone is an MLP with layers $\mathrm{CV}_t \in \mathbb{R}^K$2, ReLU activations, and separate policy/value heads. The gate is a small state-conditional MLP with hidden width 32.

The factor critics are trained on factor-specific one-step rewards

$\mathrm{CV}_t \in \mathbb{R}^K$3

and corresponding discounted factor returns

$\mathrm{CV}_t \in \mathbb{R}^K$4

The factor critic loss is

$\mathrm{CV}_t \in \mathbb{R}^K$5

and the per-factor advantage is

$\mathrm{CV}_t \in \mathbb{R}^K$6

CGFA-PPO’s central actor-side object is the blended advantage

$\mathrm{CV}_t \in \mathbb{R}^K$7

If $\mathrm{CV}_t \in \mathbb{R}^K$8, the method reduces to vanilla masked PPO. If $\mathrm{CV}_t \in \mathbb{R}^K$9, policy updates rely entirely on the graph-factored advantage. Because the gate is state dependent, the agent can trust the factorized signal more in some states than in others. The mixture logits $K=6$0 are initialized from the SCM/WinProbLearner logistic-regression weights, providing an initial structural prior over which factors matter more for win probability.

A noteworthy theoretical parallel arises from representation-conditioned advantage analysis in factored MDPs. There, an advantage of the form

$K=6$1

admits a scaling interpretation in which frequent representation-conditional state-action pairs are attenuated by a factor analogous to $K=6$2 (Suau, 13 Jun 2025). This suggests why a graph-factored critic family can be attractive: the baseline is no longer purely monolithic, but conditioned on semantically structured factors that may expose omitted or underweighted causal distinctions.

4. Calibration objective, rollout computation, and optimization

A distinctive component of CGFA-PPO is the intervention-calibration loss, which aligns each factor advantage with the SCM-predicted intervention effect for that factor (Cunha et al., 7 May 2026). The target intervention quantity is defined as

$K=6$3

computed by re-evaluating SCM structural equations under intervention rather than by simulating the environment. In wrapper notation the per-step SCM target is

$K=6$4

The calibration loss is

$K=6$5

with minibatch covariance and standard deviations in the numerator and denominator, and $K=6$6 as a stability constant. This is a negative Pearson-correlation objective: minimizing it encourages positive correlation between learned factorwise advantages and SCM-predicted intervention effects. The appendix additionally states that per-factor standard deviations are clamped to avoid numerical amplification when a factor has near-zero variance in a minibatch.

The full objective is

$K=6$7

Here $K=6$8 is the standard masked PPO clipped surrogate on $K=6$9, $\Delta \mathrm{CV}_t$0 is the scalar critic loss with bootstrapped return $\Delta \mathrm{CV}_t$1, $\Delta \mathrm{CV}_t$2 is a mask-aware policy entropy bonus, and $\Delta \mathrm{CV}_t$3 is an entropy bonus on the gate. The gate term is described as a small per-state entropy bonus that prevents premature collapse to either extreme; the default hyperparameter table sets $\Delta \mathrm{CV}_t$4, with sweep value in ablation. The CGFA-specific hyperparameters reported are $\Delta \mathrm{CV}_t$5 and $\Delta \mathrm{CV}_t$6.

The rollout and update procedure follows the PPO backbone with extra factorwise quantities. At each rollout step the agent reads the legal-action mask, samples

$\Delta \mathrm{CV}_t$7

observes $\Delta \mathrm{CV}_t$8 and $\Delta \mathrm{CV}_t$9, reads $\varepsilon_{k,t}$0, $\varepsilon_{k,t}$1, and $\varepsilon_{k,t}$2 from the wrapper, computes $\varepsilon_{k,t}$3, and stores scalar and factorwise values and gate outputs. After rollout it computes scalar Generalized Advantage Estimation from $\varepsilon_{k,t}$4 using $\varepsilon_{k,t}$5, computes factor returns and factor advantages with the same truncation handling, forms $\varepsilon_{k,t}$6, and then forms $\varepsilon_{k,t}$7.

The PPO backbone hyperparameters are learning rate $\varepsilon_{k,t}$8 linearly annealed to $\varepsilon_{k,t}$9, minibatch size 256, rollout steps $\mathrm{WinProb}$0, 10 epochs per update, discount $\mathrm{WinProb}$1, GAE $\mathrm{WinProb}$2, clip range 0.2, entropy coefficient $\mathrm{WinProb}$3 linearly annealed to 0.005, max gradient norm 0.5, and MLP architecture $\mathrm{WinProb}$4 with separate policy and value heads and ReLU. CGFA-specific settings include $\mathrm{WinProb}$5, initial blend logits from SCM logistic-regression weights, residual gate hidden size 32, initial gate value $\mathrm{WinProb}$6, and default gate entropy coefficient $\mathrm{WinProb}$7.

One implementation detail is especially notable when CGFA-PPO is read alongside the policy-confounding analysis of advantage functions. The benchmark implementation normalizes $\mathrm{WinProb}$8 across the rollout to zero mean and unit variance before PPO updates (Cunha et al., 7 May 2026). By contrast, the causal-advantage analysis argues that advantage normalization can destroy the beneficial occupancy-dependent scaling effect that helps break policy confounding (Suau, 13 Jun 2025). This suggests a possible tension between benchmark-stable PPO practice and anti-confounding advantage semantics.

5. Evaluation protocol, diagnostics, and empirical profile

The benchmark evaluates CGFA-PPO not only by scalar win rate but also by causal diagnostics (Cunha et al., 7 May 2026). Two especially important diagnostics are leave-one-out cross-archetype transfer and per-factor calibration. The transfer gap is defined as

$\mathrm{WinProb}$9

where positive values indicate worse held-out performance and negative values indicate that the held-out opponent pool was easier than the matched training pool. The per-factor diagnostics include Pearson correlation $\mathrm{CardAdv}$0, sign-agreement rate

$\mathrm{CardAdv}$1

restricted to nonzero pairs, per-factor credit share

$\mathrm{CardAdv}$2

and the gate distribution $\mathrm{CardAdv}$3.

In-distribution results are mixed rather than uniformly favorable. Against PPO, CGFA-PPO reports the following per-deck headline comparisons:

• Azorius Control: PPO $\mathrm{CardAdv}$4, 95% CI $\mathrm{CardAdv}$5; CGFA-PPO $\mathrm{CardAdv}$6, 95% CI $\mathrm{CardAdv}$7; $\mathrm{CardAdv}$8 pp; $\mathrm{CardAdv}$9, $V(s)$00.
• Boros Convoke: PPO $V(s)$01, CI $V(s)$02; CGFA-PPO $V(s)$03, CI $V(s)$04; $V(s)$05 pp; $V(s)$06, $V(s)$07.
• Dimir Midrange: PPO $V(s)$08, CI $V(s)$09; CGFA-PPO $V(s)$10, CI $V(s)$11; $V(s)$12 pp; $V(s)$13, $V(s)$14.
• Domain Ramp: PPO $V(s)$15, CI $V(s)$16; CGFA-PPO $V(s)$17, CI $V(s)$18; $V(s)$19 pp; $V(s)$20, $V(s)$21.
• Mono-Red Aggro: PPO $V(s)$22, CI $V(s)$23; CGFA-PPO $V(s)$24, CI $V(s)$25; $V(s)$26 pp; $V(s)$27, $V(s)$28.

The paper repeatedly cautions that these p-values are exploratory because the submitted learned-agent headline comparisons use only $V(s)$29 paired seeds and 30 episodes per opponent. The intended protocol is substantially larger, namely at least $V(s)$30 steps per opponent, 7 paired seeds, and 300 evaluation episodes per seed and opponent.

Relative to simpler baselines, CGFA-PPO clearly exceeds random on all five decks. The heuristic baseline remains strong for proactive decks such as Mono-Red Aggro and Boros Convoke, while learned agents outperform heuristic more clearly on difficult reactive settings such as Azorius Control, although absolute performance remains low. In the Mono-Red diagnostic ablation slice, PPO reports $V(s)$31, CGFA scalar-only $V(s)$32, CGFA-PPO no gate $V(s)$33, CGFA-PPO no calibration $V(s)$34, full CGFA-PPO $V(s)$35, and the CWM-augmented causal agent $V(s)$36. The benchmark therefore separates pure capacity matching, factor critics, gating, and calibration rather than simply rewarding any method labeled causal.

Transfer results also remain diagnostic rather than conclusive. PPO reports in-distribution $V(s)$37, held-out $V(s)$38, $V(s)$39 pp, 95% CI $V(s)$40. CGFA-PPO reports in-distribution $V(s)$41, held-out $V(s)$42, $V(s)$43 pp, 95% CI $V(s)$44. The negative gap indicates that the held-out pool was easier, not that transfer was solved. The benchmark paper is explicit on this point.

The calibration diagnostics are presented as a principal empirical justification for the method. Over training, several factor correlations $V(s)$45 move positive, credit share concentrates heavily on the life-buffer factor, and the residual gate remains active rather than collapsing fully to scalar PPO. These observations support the narrower claim that CGFA-PPO uses the benchmark’s causal interface in an auditable way even when scalar win-rate gains are mixed.

6. Relation to adjacent causal-RL research and principal limitations

CGFA-PPO sits at the intersection of three nearby research directions: advantage-based debiasing, factored causal modeling of non-stationarity, and causal latent-state discovery under partial observability.

The first connection concerns advantage functions and policy confounding. In factored MDPs, policy-induced observational skew can make a compressed state representation appear sufficient on-policy while failing out of trajectory. The analysis in “Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representations” shows that a representation-conditioned baseline can attenuate common policy-favored state-action pairs and relatively amplify rarer informative ones, thereby supporting more causal state representations and better out-of-trajectory generalization (Suau, 13 Jun 2025). CGFA-PPO does not instantiate that theorem directly, but it is closely aligned in spirit because its actor update depends on structured advantages rather than only raw scalar returns.

The second connection concerns graph-factored representation under structured non-stationarity. “Factored Adaptation for Non-Stationary RL” models environment change through latent factors $V(s)$46 with sparse influence on transition, reward, and observation components, learned jointly with graph structure and minimal sufficient policy inputs (Feng et al., 2022). That work does not use PPO, does not provide advantage decomposition, and does not define modular critics, but it is relevant as a front-end perspective: value-relevant structure may be modular rather than monolithic, and control may need only a sparse subset of state and latent factors. This suggests a broader design space in which CGFA-PPO-style factor critics could eventually be paired with learned causal graphs rather than benchmark-specified ones.

The third connection concerns partial observability and causal latent-state recovery. “Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPs” argues that incorporating future information is essential for accurately capturing causal dynamics in factored-POMDPs, and uses a smoothing-style posterior to recover hidden factors and transition graphs more effectively than history-based and typical hindsight-based models (Han et al., 2024). That work is offline and does not address PPO or factorized advantage estimation, but it indicates that a graph-factored PPO method in partially observed settings may require a stronger latent-state inference module than online history-only encoders typically provide.

CGFA-PPO’s own limitations are explicit. The SCM is hand-designed rather than learned. The benchmark scale, although strategically rich, remains smaller than full MTG complexity. Opponents are fixed rather than adaptive. Longer-horizon, slower archetypes remain hard. Current evidence does not show uniform scalar performance gains from causal augmentation. The authors characterize gate and calibration mechanisms more as alignment and interpretability mechanisms than as tuned performance boosters. A revealed failure mode is credit concentration, especially on life buffer, and the paper identifies follow-up directions such as regularizing credit concentration, tuning gate schedules, and testing stronger calibration for difficult control and ramp archetypes.

Taken together, these constraints define CGFA-PPO precisely. It is best understood not as a generic theorem-backed solution to causal RL, but as a benchmark-specific masked PPO variant that turns an explicit SCM interface into trainable and inspectable factorized credit signals. Its significance lies in making causal credit assignment, intervention calibration, and auditability operational inside a difficult partially observed domain, while leaving open the harder problems of learned structure discovery, robust transfer, and consistent scalar performance improvement (Cunha et al., 7 May 2026).