Fast Good Codes: MARLÖ Benchmark

• Fast Good Codes are a benchmark framework that evaluates generalization in multi-agent reinforcement learning through parameterized 3D Minecraft tasks.
• The system challenges agents with issues like scalability, non-stationarity, and opponent diversity using varied cooperative and competitive game scenarios.
• By employing hidden task instantiations and mixed-motive games, the benchmark promotes robust, transferable agent performance across diverse gameplay conditions.

The paper presents MARLÖ, short for Multi-Agent Reinforcement Learning in Malmö, as a competition and benchmark designed to push research on general multi-agent reinforcement learning agents rather than agents that overfit to a single fixed game. Its central idea is simple but important: participants build learning agents that must operate in multiple 3D multi-agent games built in Minecraft through the Project Malmo platform, and those agents are then evaluated on held-out task configurations and against varying opponents. In that sense, MARLÖ is not just a game contest; it is intended as a step toward evaluating generalization in multi-agent learning, and the paper explicitly frames that goal as a milestone on the path toward Artificial General Intelligence.

The motivation comes from well-known weaknesses in multi-agent reinforcement learning. The abstract and introduction emphasize that current approaches still struggle with scalability, with general reward settings, and with different opponent types. The paper also highlights classic MARL difficulties such as non-stationarity—when independently learning agents change the environment dynamics for each other—and the fact that adversarial agents can hamper exploration, slowing or destabilizing learning. MARLÖ is therefore constructed to expose agents to exactly those difficulties: multiple games, multiple reward structures, mixed cooperative and competitive interactions, parameterized task variants, and evaluation against a range of other submitted agents and default challenge agents. The intended research target is not merely “can an agent solve task X?” but “can an agent learn robustly across tasks, teammates, and adversaries in a way that transfers?”

The environment platform is Malmo, Microsoft’s research platform built on top of Minecraft. The paper says participants create agents that “will be able to play multiple 3D games within Minecraft as defined in the Malmö platform.” This choice matters for comparison to other MARL benchmarks. Minecraft/Malmo provides a 3D embodied environment, richer and more open-ended than many gridworld or matrix-game MARL testbeds. It supports parameterization of worlds and tasks, which the paper uses heavily. MARLÖ includes three game families in its 2018 edition: Mob Chase, Build Battle, and Treasure Hunt.

Mob Chase is a collaborative game. Two or more agents and a mob move around a small meadow for a limited time. Agents can either cooperate to trap the mob by cornering it so it has no escape route, or abandon the collaborative objective and leave the pen through an exit. The reward structure makes the social dilemma explicit: capturing the mob gives $1$ point, while exiting gives only $0.2$. The paper notes that this is inspired by a stag hunt variant, so the task is designed to expose the tension between high-value cooperation and safe individual action. This makes it a useful benchmark for cooperative coordination, commitment, and handling teammates whose behavior may be uncertain.

Build Battle is described as both competitive and collaborative. Two teams of agents compete to build a specified cuboid structure before the time limit. The low-level action is effectively about block placement and removal. Agents get $0.2$ points for correctly placing a block or removing an incorrectly placed block, and $-0.2$ for incorrectly placing a block or removing a correct block. So this task introduces structured, denser reward feedback than Mob Chase, but still requires coordination within a team and competition across teams. It also appears to test spatial reasoning, division of labor, and perhaps action precision in a 3D block world.

Treasure Hunt is again both competitive and collaborative, but now with role asymmetry inside each team. Each team has collectors, who can pick up treasure, and fighters, who fight enemy entities. The goal is to retrieve treasure and survive. Rewards are team-based: if the collector gets the treasure, all agents on the winning team receive $0.25$; if the collector reaches the exit, they receive $0.5$; the losing team receives the negation of these points. There is also a strong penalty: if anyone on the team dies, all agents on that team receive $-1$. The episode ends when the collector reaches the exit, when a player dies, or when time expires. Compared with the other tasks, Treasure Hunt mixes cooperative role coordination, adversarial pressure, delayed success signals, and a severe shared-failure penalty, making it especially relevant for studying credit assignment, role specialization, and robust teamwork under risk.

The paper does not provide a full formal specification of observations and action spaces. It does say that in stochastic $N$-player games each player interacts with the environment by observing state observations, sending actions, affecting the environment and other players, and receiving rewards. But it does not give exact Malmo observation channels, action discretizations, sensory modalities, or API message formats in the text we have. So if someone wants precise details such as first-person pixel input versus symbolic features, or the exact action set for each game, that information is not provided in this paper excerpt.

What the paper does specify clearly is that the tasks are parameterizable, producing a broad task space rather than a small fixed set of maps. It says examples of parameters include weather, block types, number and position of entities, and size of the playing area. Participants may train on any possible instances of each game, but the final evaluation uses custom configurations designed by the organizers and not revealed before the deadline. This is one of MARLÖ’s most important design decisions. It turns the challenge from pure optimization on known scenarios into a generalization benchmark. Agents must cope not only with the nominal rules of the games but also with unseen instantiations of those games.

Opponent diversity is also central. The paper repeatedly stresses that agents are tested against different opponent types and that entries play against multiple agents in the tournament, which means participants cannot safely overfit to a single known policy. This matters because many MARL systems perform well only against the specific training distribution of opponents they experienced. MARLÖ instead treats robustness to teammate and adversary variation as part of the benchmark itself. For someone comparing MARLÖ to another platform like CrazyMARL, this is a defining axis: MARLÖ evaluates not just policy performance in a fixed environment, but policy adaptability across changing social interaction partners.

The competition format operationalizes these goals. Participants are given a starter kit with instructions to download the framework, develop agents locally, and test them on included games. The kit also includes simple tasks and default challenge agents that serve both as opponents and as examples. The paper does not list the exact baseline algorithms used in those default agents, nor any architecture, hyperparameter, or training details. It only says they exist and are part of the starter resources. Likewise, there are no published benchmark scores or leaderboard summaries in the text provided.

Final rankings are determined by a play-off tournament. Each tournament stage contains the three games, with at least one task from each game, and $N$ teams play a round-robin league. Teams are ranked by the sum of scores obtained across all tasks. The top entries in each group progress to later stages, and the final league determines the winner. The crucial evaluation properties are therefore:

1. agents must perform well on multiple different games, not a single one;
2. they are evaluated on hidden task instantiations;
3. they face multiple other agents in tournament play;
4. ranking is based on aggregate score across tasks, rewarding breadth rather than narrow specialization.

The paper does not give an explicit scoring equation in mathematical notation, but the ranking rule can be paraphrased from the text as total cumulative score over all scheduled tasks in a league. There is no mention of Elo, win-rate weighting, regret, sample-budget normalization, or statistical confidence procedures. Those details are absent.

The mathematical formalization in the paper is minimal but still useful. It states that multi-agent settings can be approached as Stochastic $N$-player Games (SGs), citing Shapley. The exact phrase is that each player “interacts with the game environment by observing state observations, sending actions that in turn affect the state of the environment (and other players) and receiving rewards.” The paper also gives the standard RL objective in words: “The goal of a reinforcement learner is formally to maximize its long-term cumulative reward.” However, it does not provide full notation such as $0.2$0, $0.2$1, $0.2$2, $0.2$3, $0.2$4, $0.2$5, or an explicit objective like

$0.2$6

Since that formula does not appear in the paper, it should not be attributed to it. The only explicit notation directly present is “Stochastic $0.2$7-player Games (SGs)” and the mention of maximizing long-term cumulative reward. So for this section, the key point is that the paper frames MARLÖ within the general SG/RL paradigm but leaves the formalism high-level.

The main technical challenges emphasized are very much the classic pain points of MARL, but made concrete through benchmark design. First is scalability: the abstract says current approaches “still show scalability problems in multiple games with general reward settings and different opponent types.” MARLÖ attacks this by requiring one agent framework to function across three distinct games rather than a single domain. Second is non-stationarity, explicitly discussed in the introduction as a result of independently learning agents. Third is the challenge from adversarial agents affecting exploration, which can slow learning and make discovered behaviors brittle. Fourth is generalization across tasks: because each game is parameterized and the evaluation tasks are hidden, an overfit policy is penalized. Fifth is opponent diversity: agents face varied co-players and opponents, so policy robustness matters. Sixth is reward complexity: while some rewards are fairly dense in Build Battle, others involve stronger delayed coordination incentives or team-level penalties, as in Mob Chase and Treasure Hunt. The paper does not explicitly discuss partial observability, sample efficiency, or delayed rewards in depth, though these are likely present in Minecraft tasks; if so, they are not elaborated in the text. It also does not discuss communication protocols, centralized training/decentralized execution, or explicit credit assignment methods.

On implementation and reproducibility, the paper gives only a partial picture. The software stack is Minecraft via Malmo, and participants use a provided starter kit hosted online. The starter kit includes:

• instructions to download the framework,
• tools to develop and execute agents locally,
• a set of simple tasks,
• code for default challenge agents.

This indicates that the benchmark was intended to have a low barrier to entry and common infrastructure. However, the paper does not specify:

• programming languages supported,
• exact APIs or bindings,
• whether agents communicate over sockets or through a Python client,
• computational budgets,
• hardware constraints,
• training-time restrictions,
• submission packaging details,
• reproducibility protocols,
• seeding or evaluation repeat counts.

So anyone trying to reproduce the exact benchmark from this paper alone would need to consult the starter kit or challenge website.

The paper also does not provide a substantive baseline section in the usual empirical sense. There are mentions of “default challenge agents” and “shared baselines,” but no named algorithms, no implementation assumptions, no comparative results, and no ablation studies. It cites prior competitions as evidence that starter kits and example agents help adoption, but it does not report MARLÖ baseline scores or analyze which standard MARL methods perform best. Thus, in response to the request for “all baseline methods, reference agents, or prior systems discussed,” the honest answer is that the paper does not enumerate or evaluate specific baseline MARL methods. The only prior systems indirectly referenced are prior competitions and the broader RL/MARL literature.

Likewise, there are no empirical findings in the paper excerpt beyond design motivations. Since this appears to be a competition announcement or overview paper rather than a results paper, it does not report experimental comparisons, winning strategies, transfer analyses, or failure modes observed in submitted agents. Therefore:

• it does not say which kinds of agents worked best,
• it does not report whether specialization or generality won in practice,
• it does not quantify transfer across games,
• it does not discuss observed collapse modes or coordination failures in tournament play.

What it does provide are intended lessons and hypotheses embedded in the benchmark design: that robust progress in MARL will require handling multiple games, multiple task variants, and multiple opponent types simultaneously; and that a public competition with common baselines and hidden evaluation can accelerate the field.

For someone comparing MARLÖ to a platform or benchmark such as CrazyMARL, the most useful perspective is that MARLÖ is distinctive along several dimensions. First, it is built around 3D embodied tasks in Minecraft, which gives it more environmental richness and variability than many toy MARL benchmarks. Second, it is explicitly about generalization, not just performance on one fixed map or scenario. Third, it mixes cooperative, competitive, and mixed-motive games within the same benchmark suite. Fourth, it emphasizes opponent and teammate diversity through tournament play, making social robustness a first-class evaluation criterion. Fifth, it is framed not merely as an environment suite but as a competition protocol with hidden test tasks and playoff ranking, which can be more demanding than an offline benchmark with public train/test splits.

So if one were comparing MARLÖ against CrazyMARL or any other MARL benchmark, the right dimensions to compare would be:

• environment richness: 3D Minecraft worlds versus simpler simulators or gridworlds;
• task diversity: one scenario versus multiple game families;
• generalization pressure: fixed tasks versus parameterized hidden tasks;
• agent interaction type: purely cooperative versus mixed cooperative/competitive settings;
• number and heterogeneity of agents: symmetric homogeneous agents versus role-differentiated teams;
• reward structure: dense, sparse, team-level, adversarial, mixed-motive;
• evaluation protocol: static benchmark scoring versus live tournament against varied opponents;
• scalability demands: whether methods must span multiple games and social configurations;
• realism/embodiment: abstract state/action settings versus embodied 3D action in a world model;
• engineering burden: starter kits, APIs, ease of local testing, and reproducibility support.

In that sense, MARLÖ is best understood not as a single benchmark task, but as a hybrid of benchmark suite, competition framework, and AGI-oriented challenge. Its core contribution is less about introducing one new algorithmic idea and more about defining an evaluation regime for general multi-agent learning. That makes it especially relevant for researchers who care about transfer, robustness, and broad capability, and it also makes it a useful comparison point for any newer platform—such as CrazyMARL—that aims to test generality, coordination, competition, or scalability in MARL.

To summarize the paper’s key message in one sentence: MARLÖ matters because it tries to move multi-agent reinforcement learning evaluation away from narrow, fixed, single-task success and toward general, transferable competence across multiple embodied games, hidden task variants, and diverse opponents.