Learning Self-Orchestration in Decentralized Agent Swarms

Abstract: The capabilities of an agent swarm depend on both its individual members and how they coordinate. Yet existing multi-agent systems often train agents within fixed orchestration structures, separating the learning of individual capabilities from the organization of collective behavior. We introduce market-based post-training (MPT), a framework for jointly learning action policies and self-orchestration in decentralized agent swarms. Agents express their willingness to participate through bids, which determine who acts and how collective outcomes provide local training credit. Learning both action and bidding behavior allows individual capabilities and patterns of participation to adapt together without a central coordinating agent. We evaluate MPT on embodied crafting, robot navigation, and deep research. Results show gains over monolithic post-training and confirm that agent participation indeed evolves with learning. With the same post-trained action policies, learned self-orchestration outperforms alternative coordination strategies. Ablations show that both action and bidding benefit from learning, and that learned bids provide more effective action-training signals. Beyond the original training objectives, trained swarms adapt faster to related tasks and achieve better performance with larger populations at inference time. Larger populations also suffer smaller performance losses under agent dropout, demonstrating that learned self-orchestration supports reuse, scaling, and robustness in decentralized agent swarms. Code and trained models are available at https://github.com/Embodied-Minds-Lab/MPT.

• The paper introduces the Market-based Post-Training (MPT) method to jointly train action and bidding policies within decentralized agent swarms, enhancing performance through learned, adaptive orchestration across different domains.
• MPT outperformed a monolithic post-trained agent with matching training FLOPs in three distinct domains (crafting in Minecraft, navigation, and deep research) with mean held-out reward improvements of 113% in crafting, 134% in navigation, and 134% in deep research.
• The paper shows that participation patterns between agents change with their training and skill improvement, indicating that the learned bidding policies adapt over time in conjunction with action policy improvements.

Problem formulation and central contribution

“Learning Self-Orchestration in Decentralized Agent Swarms” (2610.22322) addresses a structural weakness in many multi-agent systems: individual agents may be trainable, while the policy determining which agent acts at each step remains fixed or manually specified. The paper argues that this separation is consequential because orchestration affects both the task trajectory and the distribution of learning signals. An agent’s participation determines which states it observes, which downstream outcomes it influences, and how collective reward should be attributed. Consequently, action learning and coordination learning form a coupled optimization problem.

The proposed solution, market-based post-training (MPT), jointly trains each agent’s action policy and bidding policy. At every decision point, decentralized agents submit bids representing their willingness to act. The highest bidder receives control of the next action, while the bid also determines how much accumulated downstream value is passed backward to preceding participants. This mechanism eliminates the need for a central dispatcher and makes participation itself a learned behavior.

The paper evaluates MPT in three settings with distinct interaction structures: embodied crafting in Minecraft, continuous vision-and-language navigation, and tool-mediated deep research. Across all three, MPT is compared primarily with a monolithic post-trained agent under matched training FLOPs. The central empirical claim is that learning both action and orchestration produces better performance than training a single policy over the combined action space.

Market-based post-training

Each swarm contains $N$ agents with separate parameters. At turn $t$, agent $i$ generates a bid $b_i$ from its local context. The agent with the highest bid acts, and ties are resolved uniformly. A zero bid from every agent causes the turn to be skipped. Agents therefore make two coupled decisions: whether participation is warranted and, if selected, which task-specific action to execute.

The credit-assignment mechanism is defined through a backward value recursion. If $\hat r_t$ is the immediate environment reward, the accumulated value is

$v_t = \hat r_t + b_{t+1}v_{t+1},$

and the acting agent receives local credit

$\tilde r_t = (1-b_t)v_t.$

Thus, the selected agent retains part of the value attributable to its action and passes the remainder backward to the preceding participant. A downstream reward can therefore train earlier actions, with the contribution attenuated by the product of intervening bids. The mechanism resembles a learned bucket-brigade credit-assignment process, but differs from conventional temporal-difference or centralized-critic methods in that the learned participation variable directly controls both action selection and reward propagation.

Action policies are optimized with REINFORCE. Each agent updates only on turns during which it acted, using its local credit and a per-agent mean baseline. This preserves decentralized execution and avoids requiring a centralized value function, although it also makes the quality of learning dependent on sparse and potentially high-variance agent-specific returns.

Bidding policies are learned with counterfactual rollouts. A turn is considered contested when the runner-up submits a nonzero bid. The original trajectory is compared with an alternative branch obtained by restoring the environment and agent states, changing the selected agent’s bid sufficiently to let the runner-up act, and continuing the episode. The branch with the higher post-contest cumulative reward supplies the preferred bid response. The policy is trained using a Bradley–Terry preference loss. Action learning and bidding learning alternate in a four-to-one schedule, allowing the two policies to co-adapt as agent capabilities change.

This procedure introduces a substantial computational requirement: bidding optimization requires environment-state restoration and additional counterfactual rollouts. The paper explicitly includes these costs when matching training FLOPs against the monolithic baseline, which strengthens the comparison, but the same requirement limits applicability to environments that support reliable branching or replay.

Experimental design

The experiments use three agents with complementary roles in each domain.

In embodied crafting, Logger, Miner, and Smith are separate Qwen3-VL-4B-Instruct models. Logger gathers wood, Miner handles underground resource acquisition, and Smith performs crafting and smelting. The task is to obtain an iron pickaxe from an empty Minecraft inventory, which requires a long dependency chain involving logs, planks, crafting stations, multiple pickaxes, cobblestone, iron ore, fuel, and smelting. The agents issue natural-language instructions to a frozen STEVE-1 controller.

In robot navigation, Turner, Mover, and Stopper are separate NaVILA-8B models. Their action spaces are partitioned into rotations, forward movement, and stopping. They operate in VLN-CE R2R episodes within Habitat, using visual histories and route instructions.

In deep research, Searcher, Opener, and Finder are separate gemma-4-E2B-it models. Their tool spaces are partitioned among search, page opening, and exact-text finding, while any agent may submit the final answer. Episodes receive a binary terminal reward from a Qwen-based judge.

The monolithic baseline uses one model trained over the complete role or action space. SFT cold starts are compute-matched: the monolithic model receives demonstrations from all roles, whereas each MPT agent receives role-specific demonstrations. Post-training comparisons also match total LLM computation, including bid generation, action generation, counterfactual rollouts, and parameter updates.

Main performance results

MPT achieves higher training performance and held-out performance in all three domains. The reported held-out results are:

Task	Monolithic RL	MPT
Embodied crafting	$20.3 \pm 4.1$	$41.5 \pm 5.9$
Robot navigation	$0.52 \pm 0.14$	$t$0
Deep research	$t$1	$t$2

The training curves show approximately 41 versus 20 mean reward in crafting, 0.9 versus 0.3 in navigation, and 27% versus 6% accuracy in deep research. These improvements are not confined to the training distribution: evaluation uses 32 unseen Minecraft world seeds, 88 R2R-CE validation-unseen episodes, and 50 held-out research questions.

The crafting result is particularly informative because it contains explicit subtask dependencies and specialized agent roles. MPT more than doubles the held-out reward relative to monolithic RL. Its advantage plausibly reflects both specialization and adaptive scheduling, but the design does not fully disentangle those factors in the main comparison: the monolithic baseline is a single model trained across all roles, whereas MPT uses independently parameterized role-specific models. The later coordination experiments address this issue more directly by holding action policies fixed.

Learned participation changes with competence

The paper provides behavioral evidence that orchestration evolves during training rather than remaining a static role-allocation heuristic. On the Iron Pickaxe task, Logger dominates early decisions both before and after reinforcement learning, because wood acquisition is initially required. Later in the episode, however, Miner and Smith receive increasing participation.

At decision step 120, Logger’s participation falls from approximately 80% after SFT to 45% after RL, while Miner’s share rises from approximately 10% to 40% and Smith’s from approximately 10% to 15%. The implication is that post-training changes not only what agents do but also when they are considered competent or useful enough to act. The observed schedule tracks the crafting dependency structure: early resource gathering gives way to mining and then crafting.

This result supports the paper’s central conceptual claim, but it remains primarily descriptive. Participation proportions demonstrate that allocation changes, not that the changes are globally optimal or causally responsible for the full performance gain. The ablations provide stronger evidence for the latter.

Learned orchestration versus alternative coordination

To isolate coordination quality, the authors freeze the same post-trained action policies and compare MPT’s learned bidding mechanism against Round-Robin, AgentNet, and Graph-of-Agents on the Iron Pickaxe task. The results are:

Coordination strategy	Mean reward
Round-Robin	$t$3
AgentNet	$t$4
Graph-of-Agents	$t$5
MPT	$t$6

MPT exceeds Graph-of-Agents, the strongest alternative, by 63.4% in relative mean reward. Because action policies are held constant, this comparison directly supports the claim that the learned auction-based scheduler is more effective than the evaluated alternative allocation rules.

The result also qualifies the interpretation of specialization. The advantage cannot be attributed solely to role-specific action policies, since all coordination strategies use the same post-trained policies. However, it remains specific to the tested crafting environment and the compared implementations. The experiment does not establish that bidding is universally superior to centralized scheduling, learned critics, or other adaptive routing methods under different communication or action-space assumptions.

Joint learning of action and bidding

A four-way evaluation separates initial and post-trained versions of the bidding and action components. On the Iron Pickaxe task, the results are:

Bidding policy	Action policy	Mean reward
Initial	Initial	$t$7
Trained	Initial	$t$8
Initial	Trained	$t$9
Trained	Trained	$i$0

The fully trained configuration reaches 2.69 times the initial reward. Training action policies alone yields a larger improvement than training bidding alone, but the combination is substantially stronger than either component in isolation. This interaction is consistent with the proposed co-adaptation mechanism: improved action competence changes which agents should participate, while improved bidding changes which agents obtain experience and therefore which action policies receive useful updates.

The result is a strong argument against treating orchestration as a fixed inference-time wrapper around independently trained agents. Nevertheless, the comparison is conducted on one task and one population composition. It does not determine whether the interaction persists when agents have overlapping rather than complementary roles, or when the action space is not explicitly partitioned.

Credit assignment and the role of bids

The paper separately evaluates whether bids are useful merely for turn allocation or also for training credit. In one ablation, turns are assigned by round-robin scheduling. In another, learned bids continue to determine allocation and bidding updates, but random values sampled from $i$1 replace the bids used in credit assignment.

At approximately $i$2 training FLOPs, MPT reaches a mean reward of about 42, compared with approximately 15 for round-robin and 31 for random credit. Round-robin shows little improvement during training, while random credit performs better but remains substantially below MPT.

The gap between learned credit and random credit supports the claim that bids encode useful information for assigning downstream value to prior actions. It also shows that the benefit of MPT is not reducible to simply giving agents a chance to act under a learned scheduler. However, the credit rule is structurally imposed rather than learned end-to-end: bids simultaneously affect selection and the fraction of value propagated backward. This coupling makes causal attribution difficult, because a bid may be beneficial as a participation decision but harmful or beneficial as a credit-allocation coefficient.

Reuse across objectives

The authors test continual adaptation by initializing both MPT and monolithic RL from Iron Pickaxe checkpoints and continuing training on Torch and Stone Bricks objectives. At approximately $i$3 adaptation FLOPs, MPT reaches mean rewards of about 35 on Torch and 38 on Stone Bricks, compared with approximately 17 and 19 for the monolithic baseline.

These results indicate that learned orchestration is not entirely task-specific. A previously trained population can adapt more rapidly when the new objective shares the same crafting domain and role structure. The result is important because it suggests that coordination policies may carry reusable information about division of labor and action dependencies. The evidence, however, concerns closely related crafting objectives rather than a transfer across substantially different environments, role semantics, or model families.

Inference-time population scaling and dropout robustness

MPT supports population expansion at inference time by replicating each trained agent, producing populations of 3, 6, 9, and 12 agents. On five crafting milestones, larger populations improve pass@3 without additional training. For example, increasing the population from 3 to 12 raises pass@3 from approximately 78% to 97% for Wooden Pickaxe, from 69% to 94% for Stone Pickaxe, and from 66% to 91% for Furnace.

This result suggests that the learned bidding policies can exploit additional copies of existing competencies rather than requiring a separately trained scheduler for every population size. The conclusion is narrower than generic population scaling: the added agents are replicas of the same trained role models, so the experiment tests redundancy and sampling diversity rather than an arbitrary increase in heterogeneous capability.

The same redundancy improves tolerance to agent dropout. At dropout probability $i$4, the 12-agent population retains approximately 33 mean reward, compared with approximately 19 for the 3-agent population. This demonstrates a concrete robustness benefit of decentralized participation: no single coordinator or individual agent is indispensable. It does not, however, distinguish whether robustness derives from learned orchestration, mere replication, or both. A replicated population using a fixed scheduler would be needed to isolate those effects.

Limitations and open questions

MPT requires externally evaluated task feedback to train both action and bidding policies. The experiments use staged environmental rewards, navigation distance rewards, or binary correctness judgments. The method’s behavior under subjective, delayed, noisy, or adversarial evaluation remains untested. In deep research specifically, the terminal reward depends on an LLM judge, introducing possible evaluator error and reward-model bias.

Bidding learning also requires counterfactual rollouts from restored environment and agent states. This is feasible in Minecraft, Habitat, and the local research environment, but is not generally available for irreversible physical interactions. The paper therefore does not establish how MPT could be applied when alternative actions cannot be replayed or when counterfactual branches are expensive to simulate.

The study is additionally limited in scale. It uses three-agent swarms and does not evaluate agents above 10B parameters each or substantially larger jointly trained populations. The inference-time replication experiments reach 12 agents, but this is not equivalent to training a 12-agent population with distinct, co-adapting policies. The paper also leaves open whether bid distributions remain calibrated as population size grows, whether auction competition becomes unstable with many similar agents, and how the mechanism behaves when roles overlap or agents have conflicting objectives.

Conclusion

MPT formulates decentralized orchestration as a trainable component of multi-agent post-training. Bids determine both which agent acts and how downstream value is assigned, linking participation, credit assignment, and policy improvement within a single decentralized mechanism. Across crafting, navigation, and deep research, the method outperforms monolithic post-training; on Iron Pickaxe, it also outperforms fixed and alternative learned coordination strategies using the same action policies.

The strongest evidence supports three claims: participation patterns evolve as agents improve, learned bidding and learned action policies are mutually reinforcing, and learned coordination transfers sufficiently to support related-task adaptation, population scaling, and dropout robustness. The method’s principal unresolved constraints are its dependence on verifiable rewards and replayable counterfactual environments, together with limited evidence at larger swarm and model scales.