CausalPlan: Causal Framework for Multi-Agent LLM Planning

• CausalPlan is a causality-driven planning framework that integrates explicit structural causal reasoning into multi-agent coordination.
• It employs a two-phase architecture with the Structural Causal Action model to learn causal graphs and reweight candidate actions.
• Empirical evaluations on Overcooked-AI demonstrate significant performance improvements and reduced causally invalid actions across various LLM backbones.

CausalPlan is a causality-driven planning framework for LLM-based multi-agent collaboration. It was introduced to address a recurring failure mode in collaborative environments: LLM agents, especially smaller, open-source models, often generate causally invalid or incoherent actions because they rely on surface-level correlations rather than grounded causal reasoning. The framework integrates explicit structural causal reasoning into the planning loop without requiring fine-tuning of the LLM itself. Its core component is the Structural Causal Action (SCA) model, which learns a causal graph from agent trajectories and uses that graph to score, reweight, or replace candidate actions during inference. The reported evaluation is on Overcooked-AI across five coordination layouts and four open-source LLM backbones—Gemma-7B, Llama-8B, Qwen-14B, and Llama-70B—in both AI-AI and human-AI settings (Nguyen et al., 19 Aug 2025).

1. Problem setting and motivation

CausalPlan is formulated for a two-player Markov Decision Process with factored state and action spaces. At timestep $t$, each agent observes a state $s_t$ that includes the agent’s own state, the partner’s state, and the environment state, and seeks to maximize expected cumulative reward:

$\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$

Within this setting, the framework treats good planning as reasoning about how previous actions and current states cause future valid decisions (Nguyen et al., 19 Aug 2025).

The motivating examples are drawn from Overcooked-style cooperative control. In this domain, action validity depends on temporal and environmental constraints: an agent must pick up an onion before putting it in the pot; a dish can only be filled with soup after the pot is finished; and if a partner already has the relevant item, complementary rather than duplicative action is required. The paper identifies “causally invalid action generation” as the central failure mode of standard LLM agents. Typical errors include proposing “put onion in pot” before holding an onion, “fill dish with soup” when the pot is not finished, or selecting an action already being handled by the partner. The paper positions this difficulty against classic zero-shot coordination RL methods such as self-play, PBT, FCP, MEP, and COLE, which can learn coordination policies but are described as expensive to train, prone to overfitting to seen partners, weak on generalization to unseen collaborators, and not very interpretable (Nguyen et al., 19 Aug 2025).

2. Two-phase architecture

CausalPlan is organized as a two-phase framework. Phase 1 is Causal Action Structure Learning, in which a causal model is learned from trajectories collected by a behavior policy. Phase 2 is Agent Planning with Causal Knowledge, in which the learned causal knowledge is injected into inference-time action selection (Nguyen et al., 19 Aug 2025).

The architecture is centered on the Structural Causal Action model. Rather than modeling rewards or only state transitions, SCA models the causal dependence of the next action on the current state and the previous action. The training buffer is

$$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$

collected by a behavior policy $\pi_\beta$. States and actions are factorized and discretized into binary vectors,

$$s_t = [s_{t,1}, \dots, s_{t,S}] \in \{0,1\}^S,\qquad a_t = [a_{t,1}, \dots, a_{t,A}] \in \{0,1\}^A,$$

where each binary feature indicates whether a particular state predicate or action is active. This yields a learned causal graph whose nodes include state features for the controlled agent, the partner agent, and the environment, together with previous-action features and future-action nodes (Nguyen et al., 19 Aug 2025).

A notable design choice is that causal learning is performed offline once, whereas action selection uses the learned structure online. This separation is operationally important because it lets the same causal model be reused across different LLM backbones. The paper reports this explicitly as part of the framework’s efficiency profile (Nguyen et al., 19 Aug 2025).

3. Structural Causal Action model

The SCA model is presented as an SCM-like model specialized for action prediction and action causality in collaborative control. For each action component $a_i$, the causal mechanism is written as

$$a_i = f_i\left(\mathrm{Pa}_{\mathcal{G}(a_i)}, \varepsilon_{a_i}\right),$$

where $f_i$ is a learned function and $\mathrm{Pa}_{\mathcal{G}(a_i)}$ are the parents of the action node $s_t$0 in the learned graph $s_t$1 (Nguyen et al., 19 Aug 2025).

Learning jointly optimizes generating parameters $s_t$2 and structural parameters $s_t$3 through

$s_t$4

The causal prediction loss is

$s_t$5

and the structural regularization term is

$s_t$6

The appendix, as summarized in the source description, states that optimization alternates between updating $s_t$7 and $s_t$8, using a neural network for each $s_t$9 and a sigmoid on $\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$0 to obtain edge probabilities (Nguyen et al., 19 Aug 2025).

The learned graph encodes directional influences such as pickup_onion → put_onion_in_pot, pot_finished → fill_dish_with_soup, and empty_hand1 → pickup_onion. The paper’s Overcooked examples include state predicates such as holding nothing, onion, dish, or soup; pot occupancy; whether the pot is finished; whether soup has been delivered; and analogous partner-state features. Action features include pickup_onion, put_onion_in_pot, pickup_dish, fill_dish_with_soup, deliver_soup, place_onion_on_counter, and place_dish_on_counter. The paper further notes that some influences emerge as strong while others are weak or absent. To ensure acyclicity, the method compares bidirectional edges and zeroes out the smaller one (Nguyen et al., 19 Aug 2025).

A central derived object is the Causal Action Matrix

$\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$1

Rows correspond to candidate next actions, columns correspond to active state or previous-action features, and each entry represents learned causal strength from a feature to an action. For a candidate action $\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$2, the causal score is

$\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$3

where $\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$4 and $\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$5. In effect, the score measures how strongly the currently active causal context supports the candidate action (Nguyen et al., 19 Aug 2025).

4. Inference-time planning and intervention-consistent action selection

At inference time, CausalPlan uses a two-prompt design. The LLM first receives the observation in an analysis prompt and then in a planning prompt, a separation introduced mainly for cleaner extraction of action proposals. The resulting candidate set is

$\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$6

with LLM probabilities $\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$7. The system then queries the causal matrix for corresponding causal scores $\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$8 (Nguyen et al., 19 Aug 2025).

If the LLM produces valid candidate actions, CausalPlan performs causal-aware action planning by combining the LLM’s belief with causal belief:

$\mathbb{E}\left[\sum_t R(s_t, a_t)\right].$9

The final distribution is

$$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$0

The framework also post-processes redundant action strings, merges duplicates, and resamples from the refined distribution. This is the main mechanism by which candidate proposals are steered toward actions supported by the learned causal structure (Nguyen et al., 19 Aug 2025).

If the LLM fails to produce any valid instructed action, so that

$$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$1

the framework invokes its Causal Backup Action mechanism:

$$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$2

This retrieval-style fallback is one of the design points highlighted in the paper, because it addresses empty, hallucinated, or invalid LLM outputs without reprompting or stalling. The paper describes the overall effect as steering the agent toward intervention-consistent behavior by constraining action choice through a learned causal model of environment dynamics and agent interaction (Nguyen et al., 19 Aug 2025).

5. Empirical evaluation

The reported evaluation uses the Overcooked-AI benchmark, a cooperative cooking environment in which two agents prepare and serve onion soup. Five layouts are used: Cramped Room (CR), Asymmetric Advantages (AA), Coordination Ring (COR), Forced Coordination (FC), and Counter Circuit (CC). The LLM backbones are Gemma-7B, Llama-8B, Qwen-14B, and Llama-70B. The models are integrated into the ProAgent framework with CausalPlan replacing the planning module, while Cohere/command-r (35B via API) is used to generate scenario analysis in the two-prompt setup for speed. RL baselines are SP, PBT, FCP, MEP, and COLE. Reported metrics include average task return or performance score, improvement over the backbone LLM without CausalPlan, invalid-action counts, collaboration performance in AI-AI and human-AI settings, and ablation results; runs are averaged over multiple seeds and 400 timesteps per run (Nguyen et al., 19 Aug 2025).

The paper reports that all backbone models generate causally invalid actions to varying degrees and that CausalPlan significantly reduces these failures. Across all four backbones and all five layouts, performance improves. The reported average gains include about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$3 for Qwen-14B and about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$4 for Llama-70B, with the largest gains in Cramped Room at about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$5 average improvement and Coordination Ring at about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$6. Using Llama-70B + CausalPlan, the method achieves the best score in 3 of 5 layouts, second-best in another, and exceeds the next best baseline by about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$7 in Asymmetric Advantages. In human-proxy partner evaluation, CausalPlan with Llama-70B outperforms all baselines in 8 out of 10 configurations, improves by about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$8 over the LLM without CausalPlan, and beats COLE by about $$B = \left\{ \{(s_t^k, a_t^k) \}_{t=1}^T \right\}_{k=1}^N,$$9 on average (Nguyen et al., 19 Aug 2025).

The ablations isolate several components. The two-prompt setup performs similarly to a one-prompt ProAgent-style design, with slight improvement; the reported conclusion is that the main gains come from the causal module rather than prompt engineering. Removing the backup mechanism reduces performance. The quality of the trajectory data used for causal learning matters: using MEP to collect data yields a better causal matrix and better downstream performance than using trajectories collected by a smaller LLM such as Llama-8B, although even weaker data collection still improves over having no causal reasoning. The balance coefficient $\pi_\beta$0 is reported to work best around

$\pi_\beta$1

Too much trust in either the causal score or the LLM probability degrades performance (Nguyen et al., 19 Aug 2025).

6. Interpretation, efficiency, limitations, and research context

One of the framework’s stated advantages is interpretability. The learned matrix $\pi_\beta$2 provides a human-readable view of which states influence which actions, how partner states affect one’s own decisions, and which action sequences are causally valid. The paper visualizes this with heatmaps and notes that obvious task relations appear clearly, including empty hand $\pi_\beta$3 pickup onion and pot finished $\pi_\beta$4 fill dish with soup. This makes the causal module more transparent than black-box policy refinement. The same section of the source description notes an efficiency trade-off: learning is offline once, taking about 3 hours for the CR environment, but inference becomes slower because the causal matrix is consulted at each step. Reported inference times are approximately 5 minutes without CausalPlan versus 15 minutes with it for Gemma-7B and Llama-8B; 16 minutes versus 41 minutes for Qwen-14B; and 40 minutes versus 68 minutes for Llama-70B (Nguyen et al., 19 Aug 2025).

The limitations acknowledged in the paper are also structural. Performance still falls short of larger closed-source models in some cases; the method has not yet been tested on larger proprietary LLMs; learned causal graphs can contain ambiguous or spurious edges; better causal discovery methods could improve the graph; and the framework adds inference overhead. The generalization claims are correspondingly specific: improvement is reported across multiple LLM sizes, across different Overcooked layouts, across AI-AI and human-AI collaboration, and without fine-tuning the LLM backbone (Nguyen et al., 19 Aug 2025).

Within the broader research landscape, CausalPlan belongs to a long line of work that treats causality as a basis for action selection and policy evaluation, but it applies those ideas to LLM-mediated coordination rather than to classical graphical planning or decision analysis. Dynamic sequential plan identification in causal Bayesian networks reduces plan identification to causal effect identification (Tian, 2012). A decision-based view of causality characterizes causality as structural knowledge needed to predict action outcomes and support rational planning under uncertainty (Heckerman et al., 2013). A decision-theoretic formulation defines Bayesian causality through failures of intervention independence and provides a DAG representation theorem for causal models expressed through choices (Schenone, 2018). This suggests that CausalPlan’s distinct contribution is not a new general theory of intervention, but an operational mechanism for constraining online LLM action proposals with a learned causal model of prior actions, partner state, and environment state.