LLM-MARL: Integrating LLMs with Multi-Agent RL

• LLM-MARL is a framework that integrates large language models into multi-agent reinforcement learning to enhance coordination, communication, and credit assignment.
• It leverages LLMs for strategic guidance, reward synthesis, and structural design through modular patterns like teacher–student distillation and full language-agent policies.
• Empirical results in simulated environments show significant performance gains over standard MARL methods in tasks such as team games and robotic coordination.

LLM–Multi-Agent Reinforcement Learning (LLM‑MARL) denotes a family of methods that integrate LLMs into multi-agent reinforcement learning to improve coordination, communication, credit assignment, generalization, and adaptation in cooperative or mixed cooperative–competitive settings. Across recent work, the LLM is not confined to a single role: it can operate as a strategic planner and communicator above PPO policies, an offline teacher whose decisions are distilled into MARL agents, a generator of reward or credit-assignment structure, a source of graph priors for coordination, or a full policy over textual observations and actions (Li, 1 Jun 2025, Xu et al., 13 May 2025, Jiang et al., 17 Apr 2025, Sarkar et al., 9 Feb 2025).

1. Problem setting and conceptual scope

The canonical motivation for LLM‑MARL is that standard MARL in complex simulations remains bottlenecked by coordination under partial observability, bandwidth-limited communication, brittle generalization beyond the training distribution, long-horizon reasoning and credit assignment under sparse or delayed rewards, and memory of successful strategies from previous episodes or related scenarios (Li, 1 Jun 2025). These issues appear in cooperative 3v3 soccer in Google Research Football, large-scale swarm settings such as MAgent Battle, StarCraft II micromanagement, disaster-area UAV networking, cooperative driving, multi-robot shape assembly, social deduction, and multi-agent search systems for multi-hop question answering (Li, 1 Jun 2025, Xu et al., 13 May 2025, Liu et al., 2024, Zhu et al., 2 Jun 2025, Sarkar et al., 9 Feb 2025, Chen et al., 3 Jun 2025).

A common conceptual distinction in the literature is between using LLMs as low-level controllers and using them as high-level guidance. Several of the strongest systems explicitly keep the core MARL backbone—often PPO or a value-based CTDE method—and place the LLM above or around that backbone as modular guidance, rather than replacing RL with language-only planning (Li, 1 Jun 2025). This design is motivated by the complementary strengths of symbolic and compositional reasoning on the LLM side, and trial-and-error policy optimization on the RL side.

The resulting design space is broad. Some systems use natural language to specify subgoals such as “defend and counterattack,” or symbolic messages such as “collapse on C3 in 5 steps,” or episodic recall such as “lure with Zerglings and flank Medics first” (Li, 1 Jun 2025). Others use the LLM to generate global layouts for UAV swarms, structured action dependency graphs for embodied collaboration, reward functions for cooperative tasks, or executable code for a training-free credit assignment function (Xu et al., 13 May 2025, Jia et al., 13 Mar 2025, Wei et al., 25 Mar 2025, Jiang et al., 17 Apr 2025). This suggests that LLM‑MARL is better understood as an interface layer between linguistic priors and multi-agent optimization than as a single algorithmic template.

2. Architectural patterns

Recent systems cluster into a small number of recurring architectural patterns.

Pattern	Core mechanism	Representative work
Strategic guidance	Coordinator, Communicator, Memory modules layered over CTDE policies	(Li, 1 Jun 2025)
Teacher–student distillation	LLM generates expert decisions or trajectories; MARL imitates and then fine-tunes	(Xu et al., 13 May 2025, Liu et al., 2024, Lou et al., 20 Jul 2025, Zhu et al., 2 Jun 2025)
Structural and algorithm design	LLM generates reward functions, dependency graphs, graph priors, or credit-assignment code	(Jiang et al., 17 Apr 2025, Jia et al., 13 Mar 2025, Wei et al., 25 Mar 2025, Gupta et al., 19 Apr 2026)
Full language-agent policies	The LLM itself acts over textual observations and actions	(Sarkar et al., 9 Feb 2025, Sudhakar, 11 Jun 2025)

In the strategic-guidance pattern, the most explicit unified formulation uses three modules: Coordinator, Communicator, and Memory. The Coordinator transforms a global state summary and high-level task description into per-agent subgoals; the Communicator enables symbolic inter-agent messaging and stores recent messages in communication buffers; the Memory module retrieves language summaries of successful prior episodes and reinjects them into prompts or policy context (Li, 1 Jun 2025). This yields a language-conditioned policy of the form

$$\pi_\theta^i(a_t^i \mid o_t^i, g_t^i, m_t^i, h_t^i),$$

where $g_t^i$, $m_t^i$, and $h_t^i$ are subgoal, message, and memory-context embeddings (Li, 1 Jun 2025).

In the teacher–student pattern, the LLM is often deliberately removed from the real-time control loop. MRLMN uses the LLM as an offline “teacher” that outputs entire multi-UAV deployments on a discretized grid; those outputs are matched to current UAVs by a Hungarian algorithm-based scheme and distilled into PPO agents through a soft action-distribution loss (Xu et al., 13 May 2025). LDPD uses a GPT‑4o-based teacher with a ReAct-style planner and decision-making tools to generate expert-level cooperative driving actions, which are then distilled into multi-agent actor–critic students through a KL-based regularizer (Liu et al., 2024). LAMARL similarly uses the LLM to generate both a prior policy and a reward function for multi-robot shape assembly, then regularizes MADDPG actors toward the prior policy (Zhu et al., 2 Jun 2025).

In the structural-design pattern, the LLM never acts as an environment policy at all. QLLM asks the model to synthesize executable code for a Training-Free Credit Assignment Function that replaces the learned mixing network in value-decomposition MARL (Jiang et al., 17 Apr 2025). LGC‑MARL uses a GPT‑4 planner to decompose a task into subtasks, generate an action dependency graph, and synthesize a modular reward function, while a graph-based collaboration meta policy executes low-latency control (Jia et al., 13 Mar 2025). LERO uses the LLM to generate both hybrid reward decomposition and observation enhancement code, then improves both through an evolutionary outer loop driven by MARL performance (Wei et al., 25 Mar 2025). LLM-derived graph priors work yet another variant: the LLM receives short natural-language descriptions of agent observations and returns a weighted coordination graph that is then used as adjacency in a GNN-based QMIX pipeline (Gupta et al., 19 Apr 2026).

3. Training objectives and optimization interfaces

Most LLM‑MARL systems retain a standard RL core and add language-specific objectives around it. In the unified simulation framework, the MARL backbone is PPO with a central critic, and training combines the RL objective with a subgoal alignment loss: $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{RL}} + \lambda_g \,\mathcal{L}_{\text{subgoal}}.$$ The alignment term nudges the policy toward actions implied by LLM-generated subgoals, while communication buffers can be refined by contrastive or imitation-style losses, and a learned gating policy decides when the LLM should be queried (Li, 1 Jun 2025). The same framework keeps the LLM frozen, does not backpropagate through it, and reports that gating reduces LLM usage by 43% with no performance loss (Li, 1 Jun 2025).

Distillation-based systems instantiate a different interface. MRLMN converts LLM-generated target positions into direction vectors, turns those into a soft action distribution with a cosine-similarity softmax, and minimizes cross-entropy between that target and the MARL policy (Xu et al., 13 May 2025). LDPD uses an actor loss augmented by a KL divergence from teacher to student, with an annealed coefficient so that early training is dominated by imitation and later training by task return (Liu et al., 2024). In LAMARL, the actor objective is modified to maximize

$$Q - \alpha (\mathbf{a} - \mathbf{a}_{\text{prior}})^2,$$

so the learned force-control action remains close to the LLM-generated prior policy while still optimizing the critic’s value (Zhu et al., 2 Jun 2025).

Critic design also varies. Many frameworks retain actor–critic CTDE, but this is not universal. MHGPO, designed for a three-agent LLM-based search system with Rewriter, Reranker, and Answerer roles, removes the critic entirely and estimates relative reward advantages across heterogeneous groups of rollouts. The objective is GRPO-style, group-normalized, and aggregated across agents, with a KL penalty against a frozen reference model; in experiments it consistently outperforms MAPPO in both task performance and computational efficiency, without requiring warm-up (Chen et al., 3 Jun 2025). This suggests that critic-free policy optimization is a viable branch of LLM‑MARL when agents share a large language backbone and act in semantically heterogeneous roles.

4. Communication, memory, and language-grounded interaction

A major branch of LLM‑MARL treats language itself as the medium of coordination. In the unified Coordinator–Communicator–Memory framework, natural-language messages replace opaque vector signaling. Incoming text messages are stored in communication buffers, encoded by a 2-layer Transformer with 4 attention heads, and then injected into the policy. The resulting protocols are interpretable—messages such as “split and pinch,” “cover zone C2,” or “collapse on C3 in 5 steps”—and qualitative analysis shows role specialization and communication-driven tactics (Li, 1 Jun 2025).

Language grounding can also be imposed on otherwise standard communication MARL. LangGround aligns the agents’ continuous communication vectors with an embedding space of human natural language by grounding on synthetic data generated by embodied LLM teammates. The supervised alignment objective is

$$L_{\text{sup}} = \sum_{t \in T} \sum_{i \in \mathcal{I}} \left[1 - \cos\big(c_t^i, D(o_t^i,a_t^i)\big)\right],$$

and the total objective is $L = L_{\text{RL}} + \lambda L_{\text{sup}}$ (Li et al., 2024). The reported effects are twofold: language grounding maintains task performance while accelerating the emergence of communication, and the resulting protocols exhibit zero-shot generalization in ad-hoc teamwork scenarios with unseen teammates and novel task states (Li et al., 2024).

At the opposite extreme, some work makes the LLM itself the agent policy. In the social deduction setting based on Among Us, each crewmate or imposter is a policy implemented by a LLM over a tokenized action–observation history. Communication is decomposed into listening and speaking. Listening is trained by predicting the hidden answer $q$ through a cross-entropy loss $$L_{\text{L}}(\pi,\tau_t^i) = -\log \pi(q \mid \tau_t^i)$$, while speaking is trained with an influence reward

$g_t^i$0

which measures the change in teammates’ belief in the true imposter after an agent speaks (Sarkar et al., 9 Feb 2025). The reported emergent behaviors include accusing suspects and providing evidence, and the method doubles the win rates compared to standard RL (Sarkar et al., 9 Feb 2025).

Memory is the third recurring language mechanism. In the unified simulation framework, episode summaries are logged in language form and retrieved by semantic similarity, enabling few-shot adaptation to new but similar scenarios and long-horizon reuse of tactics without retraining (Li, 1 Jun 2025). In text-based Hanabi, LLMs trained on expert trajectories serve as cooperative policy priors, and their knowledge can be distilled into a Rainbow agent, improving sample efficiency by roughly 5× relative to training from scratch (Sudhakar, 11 Jun 2025).

5. Credit assignment, reward synthesis, and structural priors

LLM‑MARL has also become a vehicle for revisiting one of MARL’s oldest problems: credit assignment. One approach is direct reward shaping. “Speaking the Language of Teamwork” asks an LLM to generate agent-specific preferences over state transitions from natural-language descriptions of the task and team goal, learns per-agent state-scoring models from these preferences, and defines dense potential-based rewards as

$g_t^i$1

This yields faster convergence and higher policy returns than state-of-the-art MARL baselines, while the potential-based formulation reduces the impact of ranking errors (Lin et al., 6 Feb 2025).

Another approach is to let the LLM generate the credit-assignment function itself. QLLM replaces a learned mixing network with a Training-Free Credit Assignment Function: $g_t^i$2 where the state-dependent weights and bias are executable symbolic functions written by the LLM through a coder–evaluator loop (Jiang et al., 17 Apr 2025). The contribution is not merely performance; it also makes the aggregation logic human-readable, and the paper reports that QLLM consistently outperforms QMIX, COMA, MADDPG, RIIT, and Qatten across matrix games, MPE, LBF, and RWARE, while also serving as a drop-in replacement inside RIIT (Jiang et al., 17 Apr 2025).

A parallel line uses the LLM to design reward decomposition and observation enhancement jointly. LERO defines a hybrid reward

$g_t^i$3

together with an observation enhancement function $g_t^i$4, and improves both by an evolutionary outer loop that uses MARL performance as fitness (Wei et al., 25 Mar 2025). LGC‑MARL similarly uses an LLM planner to generate subtasks, an action dependency graph, and a modular reward function, after which a graph-conditioned MARL meta-policy executes the task (Jia et al., 13 Mar 2025).

Structural priors are another important variant. LLM-derived graph priors convert short natural-language descriptions of each agent’s local observation into a weighted adjacency matrix that conditions graph convolution layers inside a QMIX-based pipeline (Gupta et al., 19 Apr 2026). This shifts LLM‑MARL away from “LLM as acting policy” and toward “LLM as coordination substrate.” A plausible implication is that LLM‑MARL includes not only policy augmentation but also automated design of the latent structure through which decentralized policies coordinate.

6. Empirical landscape, applications, and open problems

The empirical center of gravity in current LLM‑MARL work is still simulated control, games, and embodied collaboration, but the application range is already broad. In the unified simulation framework, full LLM‑MARL yields the following win rates:

Method	GRF (3v3)	MAgent Battle	StarCraft II
LLM-MARL	81.2	78.4	83.6
MAPPO	69.4	66.1	74.4
QMIX	61.7	59.8	68.7
RMAPPO	65.3	62.3	70.5
No-LLM	58.5	54.7	66.3
No-Comm	60.2	55.9	69.1
No-Subgoal	68.1	64.2	72.0

These correspond to improvements over MAPPO of +11.8 points in GRF, +12.3 points in MAgent, and +9.2 points in StarCraft II, together with coordination scores of 0.89, 0.86, and 0.91, and a reported >20% improvement in zero-shot generalization on out-of-distribution maps (Li, 1 Jun 2025). The ablations show that removing subgoals or communication substantially degrades performance, confirming that the gains are not explained by architecture size alone (Li, 1 Jun 2025).

Outside games, the same pattern persists. In disaster-area UAV networking, MRLMN’s training curves rise quickly and stabilize >0.8 while baselines saturate around 0.4–0.6; under varying environment size it maintains an average ~27% UE coverage advantage and ~17% available UAV ratio advantage, and under varying UAV count it averages 23% higher UE coverage, 52% higher rate, and 19% higher UAV availability than baselines (Xu et al., 13 May 2025). In cooperative driving, LDPD’s students surpass the teacher, including a hard Scenario 2 success rate of 67% for students versus 30% for the teacher (Liu et al., 2024). In AI2‑THOR, LGC‑MARL reports a Scene1 success rate of 0.92 versus 0.60 for a centralized GPT‑4 controller and 0.68 for multi-LLM dialog, with a normalized token cost of 1.80 versus 6.3 and 10.7 respectively (Jia et al., 13 Mar 2025). In shape assembly, LAMARL’s prior policy improves sample efficiency by an average of 185.9%, and structured prompts with Chain-of-Thought and basic APIs improve function-generation success rates by 28.5%–67.5% (Zhu et al., 2 Jun 2025).

Several limitations recur across the literature. Frozen pretrained LLMs make adaptation indirect and prompt-sensitive; queries are costly and require asynchronous querying, batching, and caching; grounding from natural language to low-level control is often learned only indirectly; and most evaluations remain simulation-focused, leaving real-world deployment, safety guarantees, and scale as open questions (Li, 1 Jun 2025, Xu et al., 13 May 2025, Jia et al., 13 Mar 2025). A common misconception is that LLM‑MARL requires online token-level control. Much of the strongest evidence instead comes from systems that query the LLM intermittently, use it only during training, or restrict it to reward, graph, or credit-assignment design (Xu et al., 13 May 2025, Liu et al., 2024, Jiang et al., 17 Apr 2025, Gupta et al., 19 Apr 2026). Current research directions therefore emphasize scalability and efficiency, robust prompt learning, distillation into smaller local models, multi-modal and embodied interaction, formal analysis of language-guided RL, and explicit treatment of symbolic communication constraints (Li, 1 Jun 2025, Xu et al., 13 May 2025, Wei et al., 25 Mar 2025).

Taken together, these results define LLM‑MARL as a modular research program rather than a single method: LLMs may coordinate, communicate, remember, teach, design rewards, generate graph structure, or instantiate full language-agent policies. The central empirical claim of the field is narrower and more concrete: when language priors are coupled to MARL through carefully designed interfaces, they can materially improve cooperative behavior, sample efficiency, and adaptation in domains where standard MARL alone remains brittle.